Compare commits

...

29 Commits

Author SHA1 Message Date
bvandeusen 5366047d55 Merge pull request 'feat(translation): 8h sweep + drain button + detection probe (#1376)' (#207) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m50s
2026-07-08 20:19:33 -04:00
bvandeusen ffdbdbaf07 feat(translation): 8h sweep + drain button + detection probe (#1376)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 46s
CI / integration (push) Successful in 3m55s
Throughput: translate_posts now runs every 8h (was daily) as the
steady-state cadence for newly-imported posts, and the Settings
"Translate now" button runs it in drain mode (run-until-done, no reset)
so one press clears the whole untranslated backlog instead of a single
300-post chunk. The interrupt/backoff re-enqueue now preserves the drain
flag so a bulk drain resumes cleanly after an Interpreter restart.

Misdetection groundwork: surface the detector's confidence from the
Interpreter client (it was in the detectedLanguage payload but discarded)
and add a read-only "Test translation" box — POST /settings/translation/
probe + TranslationCard UI — that shows detected language + confidence +
engine + result for pasted text, without saving. Lets the operator see
why a short/abbreviation-heavy English title gets mis-detected so the
detection guard (min-length + confidence floor) can be tuned from real
numbers. The guard itself follows once the mis-detected cases are probed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
2026-07-08 20:13:49 -04:00
bvandeusen 2923257529 Merge pull request 'fix(ui): tag-chip left-edge clipping + kind-coloured border' (#206) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m45s
2026-07-08 10:37:09 -04:00
bvandeusen a017771621 feat(ui): thin kind-coloured border on tag chips so they don't wash out
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m49s
The tonal fill (esp. character = info) is intentionally faint and blends into
the dark tag rail. Add a thin border in each chip's own kind colour via
color-mix on currentColor (the tonal chip's themed foreground), defining the
edge without changing the fill. Theme-aware in both light and dark.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 10:31:44 -04:00
bvandeusen 25e555cab6 fix(ui): stop leading tag-chip icon clipping on the left edge
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m50s
The larger size=default chip widened Vuetify's negative start-margin on the
leading kind-icon, placing it left of the .v-chip__content box whose
overflow:hidden (the name-truncation guard) then clipped its left edge.
Zero the icon's negative inline-start margin so it sits inside the clip box;
the chip's 12px padding keeps a comfortable left inset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 10:29:28 -04:00
bvandeusen 934731e9ef Merge pull request 'feat(translation): per-field language detection for mixed-language posts' (#205) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 36s
CI / integration (push) Successful in 3m48s
2026-07-08 09:25:23 -04:00
bvandeusen 3c7ab44e74 feat(translation): per-field language detection for mixed-language posts
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m43s
_translate_one translated [title, description] in ONE Interpreter call and keyed
the whole-post passthrough on the aggregate detected_lang (the FIRST item). So an
English title + non-English description detected "en" and marked the post handled,
leaving the description untranslated. Now each field is translated independently
(its own detected_lang / passthrough) and the non-target field is stored on its
own; translated_source_lang reflects the translated field's language.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 23:31:04 -04:00
bvandeusen 4c67c116b2 Merge pull request 'Bolder auto-tag accept/reject + larger tag chips' (#204) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m50s
2026-07-07 23:21:26 -04:00
bvandeusen 06f98acf3e feat(tagging): bolder auto-tag accept/reject + larger tag chips in the rail
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m45s
The in-pill ✓/✗ on unconfirmed auto-tags read muted — a faint colored icon on a
tonal chip (worst on character tags, kind=info) that only lit up on hover. Make
them solid green/red circles with a white glyph (22px, icon 15), mirroring the
Suggestions rail's verdict buttons so accept/reject read identically. Also bump
the applied-tag chips from size=small to default and the leading kind icon to
match — bigger, clearer tags throughout the rail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 23:17:18 -04:00
bvandeusen f8b667604f Merge pull request 'Graceful shutdown + resilience & Explore polish (m147)' (#203) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m48s
2026-07-07 22:53:34 -04:00
bvandeusen 4371ddb7e7 feat(translation): live re-translate progress in the Settings card
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m45s
translation/status now reports `active` (a translate/retranslate sweep is
running, from the TaskRun table) and `last_run` (the most recent finished run's
task + status). The Settings card polls live while a sweep runs, showing a
spinner + "Translating… N remaining" that ticks down, and flags a last run that
ended in error/timeout. No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 22:20:32 -04:00
bvandeusen 40cc11be5b feat(deploy): container healthchecks + Swarm rolling-update auto-rollback
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m45s
web gets a /api/health liveness check; workers a lenient celery-ping check. A
shared deploy policy (update_config order=start-first, failure_action=rollback,
monitor 90s; rollback_config; restart_policy) means a bad image that never goes
healthy is rolled back automatically instead of taking the service down. Ignored
by plain `docker compose up` (deploy: is swarm-only), so the dev override is
unaffected. Assumes prod deploys from this file via docker stack deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 22:15:33 -04:00
bvandeusen 0b78264d62 feat(maintenance): daily janitor for orphaned .part/.partial staging files
Downloads/imports stage into <name>.part / <name>.partial then os.replace() into
place, so a kill mid-write leaves a discardable temp — never a corrupt final.
cleanup_orphaned_temp_files sweeps ones left behind under the images root, only
older than 6h so an in-flight download's staging file is never removed. Daily beat.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 22:15:33 -04:00
bvandeusen 9eae636047 feat(translation): pooled Interpreter session + manual sweep resumes after drain
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m45s
- interpreter_client: shared requests.Session with a connect-only retry
  (connect=2, no status retries — we map 429/5xx ourselves) so a proxy reload
  is smoothed and the keep-alive connection is pooled across the sweep.
- translate_posts: on an interrupt (drain), re-enqueue after the Retry-After
  hint / default backoff instead of waiting for the daily beat; self-terminates
  via the health gate. Steady-state one-chunk-per-run on success is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 22:10:05 -04:00
bvandeusen f8105046dc feat(explore): exclude WIP-tagged work from the Explore rabbit-hole
Explore's neighbour grid (/api/gallery/similar → gallery_service.similar) now
takes an Explore-only exclude_wip flag that drops `wip` system-tagged images
from the candidates, alongside the banner/editor presentation tags. The
gallery's own "similar" button is unchanged (keeps wip, #1274) — only the
Explore store passes exclude_wip=1. The anchor itself may still be a WIP; only
neighbours are filtered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 22:10:05 -04:00
bvandeusen d631ed023c style(translation): use datetime.UTC alias (ruff UP017)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m45s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:04:17 -04:00
bvandeusen c64261593d feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience
CI / lint (push) Failing after 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m42s
Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs
mid-flight. Give in-flight work room to drain and make interrupted work
resume cleanly instead of stalling.

- docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s /
  scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown
  can actually drain before SIGKILL.
- celery_app.py: task_reject_on_worker_lost=True — a task killed past the
  grace window is re-queued (safe: idempotent + chunked, recovery sweeps
  re-drive stragglers).
- interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable
  and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter
  behind a reverse proxy no longer raises an opaque HTTPError.
- translation.py: thread retry_after out of _translate_batch; retranslate_posts
  resumes after the Retry-After hint (or 60s default, capped 900s) on an
  interrupt with _reset_done=True, self-terminating via the health gate.
- tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff.

No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:01:00 -04:00
bvandeusen 11572f469a Merge pull request 'Re-translate on Interpreter model change — artist-scoped + global (m146)' (#202) from dev into main
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 11s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m41s
2026-07-07 20:22:40 -04:00
bvandeusen 1f6d94f51d feat(translation): re-translate on model change — artist-scoped + global re-run (#146)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m41s
retranslate_posts resets the 5 translation columns to NULL for a scoped set of posts (all, or WHERE artist_id IN ids) then reuses the untranslated sweep to re-run them, chasing the tail until drained (run-until-done). Interpreter cache keys on engine_version so a changed model re-translates, an unchanged one is cache-fast. Reset only happens when the service is configured+healthy so translations are never wiped when they can't be rebuilt. New POST /settings/translation/retranslate (artist_id | all=true). UI: per-artist 'Re-translate posts' on the Artist Management tab + 'Re-translate all' in the Settings Translation card, both with confirm dialogs. No migration (reuses m143 columns).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:03:19 -04:00
bvandeusen ba3fd2a118 Merge pull request 'Post translation via Interpreter (milestone 143)' (#201) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m44s
2026-07-07 13:05:18 -04:00
bvandeusen 6a255482ea feat(translation): "Test connection" button — on-demand Interpreter health check (#143)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m43s
New POST /api/settings/translation/test pings /v1/health for a GIVEN base URL (not
the saved one), so the operator can verify a URL before enabling it. TranslationCard
gains a Test-connection button that reports reachable/unreachable inline and
updates the status dot. Endpoint test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 13:00:37 -04:00
bvandeusen af5aa21e45 test(translation): API endpoint tests for status + run (#143 step 6)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m47s
/settings/translation/status defaults (off → no health call) + /run 400-when-
unconfigured + 202-when-configured (monkeypatched .delay). requests is already a
backend dep, so no requirements/ci-requirements change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:39:43 -04:00
bvandeusen 83c1745fd0 feat(ui): translation-forward post text + Settings card (#143 step 5)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m45s
PostCard + modal ProvenancePanel show the English title/description by default when
a translation exists, with a per-card "show original (<lang>)" toggle — translated
bodies render as plain text, originals keep their sanitized HTML. New
TranslationCard in Settings → Ingestion & filters: enable switch, Interpreter base
URL (generic placeholder, no default host), target language, a reachability
indicator + untranslated-posts count + "Translate now".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:38:41 -04:00
bvandeusen ead60978e3 feat(translation): expose translated fields in post feed + provenance payloads (#143 step 4)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m44s
post_feed_service (card + detail) and provenance_service._post_dict now include
post_title_translated, description_translated (card-truncated / detail-uncapped)
and translated_source_lang, keeping the originals for the toggle. Feed
serialization test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:31:46 -04:00
bvandeusen 7a4de7278d feat(translation): backfill sweep + beat + manual trigger (#143 step 3)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m49s
tasks/translation.py — translate_posts: picks untranslated posts (title OR
description non-empty), per-post [title, description] batch via the Interpreter
client, stores translations + detected lang + engine_version; passthrough /
already-target posts are marked handled with no stored translation. 503 or a
connection error interrupts (retry next cycle), 400 stops (fix config), per-post
commit keeps progress; wall-clock bounded. Wired into celery (maintenance_long
lane) + a daily beat. No-op unless enabled + base URL set + healthy. GET
/settings/translation/status + POST .../run for the Settings card. Task tests
(stubbed client, monkeypatched session).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:29:28 -04:00
bvandeusen 7f8073c4c8 feat(translation): Interpreter client — batch translate + health (#143 step 2)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m43s
services/interpreter_client.py: sync (requests) client for the LibreTranslate-
compatible /v1/translate — no new dep, mirrors the platform clients. translate()
maps translatedText[]↔texts (order + length), returns detected_lang +
engine_version (aggregate = first item, fine for a per-post [title, description]
batch); passthrough items come back unchanged in their slot. InterpreterUnavailable
on 503 / connection error (retry later), InterpreterBadRequest on 400. health()
checks /v1/health engines.llm. 10 unit tests with mocked HTTP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:23:49 -04:00
bvandeusen a3bc98a53c feat(translation): Post translation columns + settings + migration (#143 step 1)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m46s
Post gains post_title_translated / description_translated / translated_source_lang
/ translation_engine_version / translated_at — filled by the translate sweep so
viewing is instant. ImportSettings gains translation_enabled (OFF by default),
interpreter_base_url (EMPTY — no default host; the operator points it at their own
Interpreter proxy behind a reverse proxy) and translation_target_lang (en),
exposed + validated via /settings/import. Migration 0083. Settings defaults +
patch + validation test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 12:20:31 -04:00
bvandeusen 9bf095179a Merge pull request 'Auto-tag accept/reject as an in-pill yes/no pair; confirm refocuses the tag input' (#200) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m44s
2026-07-07 01:13:02 -04:00
bvandeusen e7c3f4e9c9 feat(ui): auto-tag accept/reject as an in-pill yes/no pair; confirm refocuses input
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m38s
Fold the auto-tag accept into the chip: a provisional auto-tag now shows a compact
green ✓ / red ✗ pair IN the pill (replacing the ✕), and the "auto" text label is
dropped — the yes/no is signal enough (operator-asked). ✓ confirms (trains +
shields from retraction), ✗ removes (records a negative). The name still
ellipsis-truncates so the pair stays reachable.

onConfirm now returns focus to the tag input like onRemove already does, so the
input is the cursor's resting position after any chip action in both the modal
and Explore views.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-07 00:56:15 -04:00
28 changed files with 2275 additions and 55 deletions
+73
View File
@@ -0,0 +1,73 @@
"""post-text translation via Interpreter (milestone 143) — Post columns + settings
Post gains the translated title/description + the detected source language,
Interpreter engine_version (cache key), and translated_at — filled by the
translate sweep. ImportSettings gains translation_enabled (OFF by default),
interpreter_base_url (EMPTY — the operator sets their own, behind a reverse
proxy), and translation_target_lang (en).
Revision ID: 0083
Revises: 0082
Create Date: 2026-07-07
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0083"
down_revision: Union[str, None] = "0082"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"post", sa.Column("post_title_translated", sa.Text(), nullable=True)
)
op.add_column(
"post", sa.Column("description_translated", sa.Text(), nullable=True)
)
op.add_column(
"post",
sa.Column("translated_source_lang", sa.String(8), nullable=True),
)
op.add_column(
"post",
sa.Column("translation_engine_version", sa.String(128), nullable=True),
)
op.add_column(
"post",
sa.Column("translated_at", sa.DateTime(timezone=True), nullable=True),
)
op.add_column(
"import_settings",
sa.Column(
"translation_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("false"),
),
)
op.add_column(
"import_settings",
sa.Column(
"interpreter_base_url", sa.Text(), nullable=False, server_default="",
),
)
op.add_column(
"import_settings",
sa.Column(
"translation_target_lang", sa.Text(), nullable=False,
server_default="en",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "translation_target_lang")
op.drop_column("import_settings", "interpreter_base_url")
op.drop_column("import_settings", "translation_enabled")
op.drop_column("post", "translated_at")
op.drop_column("post", "translation_engine_version")
op.drop_column("post", "translated_source_lang")
op.drop_column("post", "description_translated")
op.drop_column("post", "post_title_translated")
+5 -1
View File
@@ -145,6 +145,9 @@ async def similar():
filters, _sort = _parse_filters()
except (KeyError, ValueError):
return jsonify({"error": "similar_to query param required"}), 400
# Explore passes exclude_wip=1 to also drop work-in-progress from the
# rabbit-hole; the gallery's own "similar" button omits it (keeps wip, #1274).
exclude_wip = request.args.get("exclude_wip") in ("1", "true", "True")
# post_id is the exclusive post-detail view — not a similarity scope.
# include_hidden is a gallery-browse flag; similar() has its OWN presentation
# exclusion (a similarity-quality concern, #1274), so drop it here (#141).
@@ -154,7 +157,8 @@ async def similar():
async with get_session() as session:
svc = GalleryService(session)
try:
images = await svc.similar(image_id=similar_to, limit=limit, **scope)
images = await svc.similar(
image_id=similar_to, limit=limit, exclude_wip=exclude_wip, **scope)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
if images is None:
+185 -2
View File
@@ -1,12 +1,24 @@
"""Settings API: import filters, system stats."""
import asyncio
import secrets
from quart import Blueprint, jsonify, request
from sqlalchemy import func, select
from sqlalchemy import func, or_, select
from ..extensions import get_session
from ..models import AppSetting, Artist, ImageRecord, ImportBatch, ImportSettings, ImportTask, Tag
from ..models import (
AppSetting,
Artist,
ImageRecord,
ImportBatch,
ImportSettings,
ImportTask,
Post,
Tag,
TaskRun,
)
from ..services import interpreter_client as ic
settings_bp = Blueprint("settings", __name__, url_prefix="/api")
@@ -32,6 +44,9 @@ _EDITABLE_FIELDS = (
"extdl_mediafire_enabled",
"extdl_dropbox_enabled",
"extdl_pixeldrain_enabled",
"translation_enabled",
"interpreter_base_url",
"translation_target_lang",
)
# Per-host external-download toggles — all plain booleans, validated uniformly.
@@ -69,6 +84,9 @@ async def get_import_settings():
"extdl_mediafire_enabled": row.extdl_mediafire_enabled,
"extdl_dropbox_enabled": row.extdl_dropbox_enabled,
"extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
"translation_enabled": row.translation_enabled,
"interpreter_base_url": row.interpreter_base_url,
"translation_target_lang": row.translation_target_lang,
})
@@ -128,6 +146,15 @@ async def update_import_settings():
for tog in _EXTDL_TOGGLE_FIELDS:
if tog in body and not isinstance(body[tog], bool):
return jsonify({"error": f"{tog} must be a boolean"}), 400
# Translation (#143): base URL may be empty (feature off until set — no
# default host; the operator points it at their own Interpreter proxy).
if "translation_enabled" in body and not isinstance(
body["translation_enabled"], bool
):
return jsonify({"error": "translation_enabled must be a boolean"}), 400
for key in ("interpreter_base_url", "translation_target_lang"):
if key in body and not isinstance(body[key], str):
return jsonify({"error": f"{key} must be a string"}), 400
if "series_suggest_threshold" in body:
v = body["series_suggest_threshold"]
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
@@ -270,3 +297,159 @@ async def rotate_extension_api_key():
row.value = new_value
await session.commit()
return jsonify({"key": new_value})
# --- Translation (#143): live status + manual "Translate now" --------------
@settings_bp.route("/settings/translation/status", methods=["GET"])
async def translation_status():
"""For the Settings card: is it on, is a URL set, is the service reachable,
and how many posts still await translation. Health runs the sync client in a
thread so the event loop isn't blocked."""
translation_tasks = (
"backend.app.tasks.translation.translate_posts",
"backend.app.tasks.translation.retranslate_posts",
)
async with get_session() as session:
cfg = await ImportSettings.load(session)
untranslated = (await session.execute(
select(func.count(Post.id))
.where(Post.translated_source_lang.is_(None))
.where(or_(
Post.post_title.is_not(None), Post.description.is_not(None),
))
)).scalar_one()
# Live progress: is a sweep running now, and what did the last one do?
# (run-until-done re-enqueues itself, so `active` stays true across a
# bulk re-translate; `last_run` surfaces a completed run's outcome.)
active = (await session.execute(
select(func.count(TaskRun.id))
.where(TaskRun.task_name.in_(translation_tasks))
.where(TaskRun.status == "running")
)).scalar_one()
last = (await session.execute(
select(TaskRun.task_name, TaskRun.status, TaskRun.finished_at)
.where(TaskRun.task_name.in_(translation_tasks))
.where(TaskRun.finished_at.is_not(None))
.order_by(TaskRun.finished_at.desc())
.limit(1)
)).first()
base_url = (cfg.interpreter_base_url or "").strip()
healthy = await asyncio.to_thread(ic.health, base_url) if base_url else False
return jsonify({
"enabled": cfg.translation_enabled,
"base_url_set": bool(base_url),
"healthy": healthy,
"untranslated_count": int(untranslated),
"active": int(active) > 0,
"last_run": {
"task": last[0].rsplit(".", 1)[-1],
"status": last[1],
"finished_at": last[2].isoformat() if last[2] else None,
} if last else None,
})
@settings_bp.route("/settings/translation/test", methods=["POST"])
async def translation_test():
"""On-demand reachability check for a GIVEN Interpreter base URL (the Settings
'Test connection' button) — pings /v1/health without saving, so the operator
can verify a URL before enabling. Health runs in a thread (sync client)."""
body = await request.get_json()
base_url = ""
if isinstance(body, dict):
base_url = (body.get("base_url") or "").strip()
healthy = await asyncio.to_thread(ic.health, base_url) if base_url else False
return jsonify({"healthy": healthy})
@settings_bp.route("/settings/translation/probe", methods=["POST"])
async def translation_probe():
"""Diagnostic for the Settings 'Test translation' box: translate a pasted
snippet WITHOUT saving anything, returning what Interpreter *detected*
(language + confidence) alongside the result. Lets the operator see why a
given string was (mis-)detected — e.g. a short English title flagged as
another language — so a detection guard can be tuned from real numbers.
Read-only: no post is touched. Uses the currently-saved base URL + target."""
body = await request.get_json(silent=True)
body = body if isinstance(body, dict) else {}
text = (body.get("text") or "").strip()
if not text:
return jsonify({"error": "provide text to translate"}), 400
async with get_session() as session:
cfg = await ImportSettings.load(session)
base_url = (cfg.interpreter_base_url or "").strip()
if not base_url:
return jsonify({"error": "no Interpreter base URL is set"}), 400
target = (cfg.translation_target_lang or "en").strip() or "en"
try:
res = await asyncio.to_thread(
ic.translate, [text], base_url=base_url, target=target,
)
except ic.InterpreterUnavailable as e:
return jsonify({"error": f"Interpreter unavailable: {e}"}), 503
except ic.InterpreterBadRequest as e:
return jsonify({"error": f"Interpreter rejected the request: {e}"}), 400
translations = res.get("translations") or []
return jsonify({
"target": target,
"detected_lang": res.get("detected_lang"),
"detected_confidence": res.get("detected_confidence"),
"engine": res.get("engine"),
"engine_version": res.get("engine_version"),
"translated": translations[0] if translations else None,
})
@settings_bp.route("/settings/translation/run", methods=["POST"])
async def translation_run():
"""Enqueue the translate sweep now (the Settings 'Translate now' button).
Runs in drain mode — run-until-done — so one press chases the whole
untranslated backlog to zero rather than a single 300-post chunk."""
async with get_session() as session:
cfg = await ImportSettings.load(session)
if not cfg.translation_enabled or not (cfg.interpreter_base_url or "").strip():
return jsonify(
{"error": "translation is disabled or no base URL is set"}
), 400
from ..tasks.translation import translate_posts
r = translate_posts.delay(drain=True)
return jsonify({"celery_task_id": r.id}), 202
@settings_bp.route("/settings/translation/retranslate", methods=["POST"])
async def translation_retranslate():
"""Re-translate stored translations after a model change (m146). Body:
``{"artist_id": <int>}`` aims at one artist; ``{"all": true}`` re-runs every
artist. ``all`` must be explicit so an empty/typo body can't wipe everything.
Clears the scoped translations and enqueues the run-until-done retranslate
sweep (the Interpreter cache re-translates on a changed model, is cache-fast
otherwise). Same enabled + base-URL guard as 'Translate now'."""
body = await request.get_json(silent=True)
body = body if isinstance(body, dict) else {}
artist_id = body.get("artist_id")
do_all = bool(body.get("all"))
if artist_id is None and not do_all:
return jsonify(
{"error": "provide artist_id, or all=true to re-translate everything"}
), 400
if artist_id is not None:
try:
artist_id = int(artist_id)
except (TypeError, ValueError):
return jsonify({"error": "artist_id must be an integer"}), 400
async with get_session() as session:
cfg = await ImportSettings.load(session)
if not cfg.translation_enabled or not (cfg.interpreter_base_url or "").strip():
return jsonify(
{"error": "translation is disabled or no base URL is set"}
), 400
from ..tasks.translation import retranslate_posts
# artist_id wins when both are sent; otherwise all=true → None (every artist).
artist_ids = [artist_id] if artist_id is not None else None
r = retranslate_posts.delay(artist_ids=artist_ids)
return jsonify({"celery_task_id": r.id}), 202
+25
View File
@@ -35,6 +35,7 @@ def make_celery() -> Celery:
"backend.app.tasks.backup",
"backend.app.tasks.admin",
"backend.app.tasks.library_audit",
"backend.app.tasks.translation",
],
)
app.conf.update(
@@ -63,9 +64,20 @@ def make_celery() -> Celery:
"backend.app.tasks.backup.*": {"queue": "maintenance_long"},
"backend.app.tasks.admin.*": {"queue": "maintenance_long"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
# Translation backfill hits the LLM (~16s/item) → the long lane so it
# never starves the quick self-healing sweeps (#143).
"backend.app.tasks.translation.*": {"queue": "maintenance_long"},
},
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True,
# Deploy graceful-shutdown safety: with acks_late, a task killed because
# it outran the container's stop-grace window (SIGKILL) is re-queued
# rather than silently lost. Safe because our long tasks are idempotent +
# chunked (translation per-post commit, downloads terminal-status, audits
# chunk) and the 5-min recovery sweeps re-drive anything left non-terminal
# — a re-run resumes cleanly and never corrupts. No redeliver-loop risk:
# heavy GPU work is tombstoned via gpu_queue, not run inline in a worker.
task_reject_on_worker_lost=True,
worker_prefetch_multiplier=1,
# Broker resilience (2026-06-24): a swarm overlay-network blip after a
# redeploy left Redis healthy but transiently unreachable, and a worker
@@ -103,6 +115,11 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.cleanup_old_tasks",
"schedule": 86400.0, # daily
},
"cleanup-orphaned-temp-files": {
"task": "backend.app.tasks.maintenance.cleanup_orphaned_temp_files",
"schedule": 86400.0, # daily — sweep .part/.partial left by a
# download/import killed mid-write (graceful-shutdown fallout)
},
"train-heads-nightly": {
"task": "backend.app.tasks.ml.scheduled_train_heads",
"schedule": 86400.0, # passive cadence; manual retrain stays available
@@ -161,6 +178,14 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.ml.prune_presentation_reviews",
"schedule": 86400.0, # retention: drop resolved review flags >30d
},
"translate-posts-8h": {
"task": "backend.app.tasks.translation.translate_posts",
"schedule": 28800.0, # every 8h: steady-state cadence for the
# trickle of newly-imported posts (no-op unless translation
# configured + healthy). One bounded 300-chunk per fire — the
# one-time backlog drains via the "Translate now" button
# (drain=True, run-until-done), not this sweep.
},
"snapshot-head-metrics-daily": {
"task": "backend.app.tasks.maintenance.snapshot_head_metrics",
"schedule": 86400.0,
+15
View File
@@ -92,6 +92,21 @@ class ImportSettings(Base):
Boolean, nullable=False, default=True, server_default="true",
)
# -- Post-text translation via the Interpreter LAN service (milestone 143).
# Off by default with NO default host — it needs a reachable Interpreter
# service (the operator's, behind a reverse proxy), which not every install
# has; the operator sets the URL and flips it on. Empty base_url OR disabled
# → the translate sweep no-ops.
translation_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, server_default="false",
)
interpreter_base_url: Mapped[str] = mapped_column(
Text, nullable=False, default="", server_default="",
)
translation_target_lang: Mapped[str] = mapped_column(
Text, nullable=False, default="en", server_default="en",
)
@classmethod
async def load(cls, session) -> ImportSettings:
"""The singleton settings row (id=1), via an async session."""
+18
View File
@@ -47,6 +47,24 @@ class Post(Base):
description: Mapped[str | None] = mapped_column(Text, nullable=True)
attachment_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
# -- Post-text translation (milestone 143). Filled by the translate_posts
# sweep via the Interpreter LAN service so viewing is instant.
# translated_source_lang is the DETECTED original language; "en" (or a
# passthrough) means nothing to translate and the *_translated columns stay
# NULL. engine_version keys the Interpreter cache — re-runs are ~1ms and a
# model upgrade re-translates instead of serving stale.
post_title_translated: Mapped[str | None] = mapped_column(Text, nullable=True)
description_translated: Mapped[str | None] = mapped_column(Text, nullable=True)
translated_source_lang: Mapped[str | None] = mapped_column(
String(8), nullable=True
)
translation_engine_version: Mapped[str | None] = mapped_column(
String(128), nullable=True
)
translated_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
downloaded_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+4
View File
@@ -48,6 +48,10 @@ class TagKind(StrEnum):
# content. `wip` is real art: only the training pipelines exclude it.
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
PRESENTATION_SYSTEM_TAGS = ("banner", "editor screenshot")
# `wip` marks real-but-unfinished art. It's kept in the gallery's own "similar"
# results (#1274), but the Explore rabbit-hole opts to hide it (exclude_wip) so a
# browse doesn't keep surfacing work-in-progress (operator, 2026-07-08).
WIP_SYSTEM_TAG = "wip"
image_tag = Table(
"image_tag",
+9 -4
View File
@@ -31,7 +31,7 @@ from ..models import (
Tag,
TagPositiveConfirmation,
)
from ..models.tag import PRESENTATION_SYSTEM_TAGS, image_tag
from ..models.tag import PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG, image_tag
from .pagination import decode_cursor, encode_cursor
from .tag_query import (
fandom_join_alias,
@@ -715,6 +715,7 @@ class GalleryService:
platform: str | None = None,
untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None,
exclude_wip: bool = False,
) -> list[GalleryImage] | None:
"""Visual "more like this": images near `image_id`'s SigLIP embedding
(pgvector, HNSW-indexed — alembic 0036), then DIVERSIFIED so the result
@@ -751,14 +752,18 @@ class GalleryService:
# Presentation images (banner / editor-screenshot system tags, #128)
# cluster on UI chrome rather than content, so near any one of them
# they'd fill the grid. Excluded from CANDIDATES only — the anchor
# itself may be a banner, and `wip` stays surfaced (real art; only
# the training pipelines exclude it).
# itself may be a banner. `wip` stays surfaced here by default (real art;
# only the training pipelines exclude it), but the Explore rabbit-hole
# passes exclude_wip to also drop work-in-progress (operator, 2026-07-08).
excluded_system_tags = PRESENTATION_SYSTEM_TAGS
if exclude_wip:
excluded_system_tags = (*PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG)
presentation = (
select(image_tag.c.image_record_id)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(
Tag.is_system.is_(True),
Tag.name.in_(PRESENTATION_SYSTEM_TAGS),
Tag.name.in_(excluded_system_tags),
)
)
stmt = stmt.where(
+152
View File
@@ -0,0 +1,152 @@
"""Interpreter translation client (milestone 143) — a thin SYNC wrapper over the
self-hosted Interpreter LAN service (LibreTranslate-compatible `/v1/translate`).
Sync (requests) because the only caller is the sync celery translate sweep, and
it mirrors FC's other platform clients. The service knows nothing about Curator —
all Curator logic stays here. base_url is operator-configured (empty until set;
behind a reverse proxy). Full API contract: Scribe note #1347.
"""
from __future__ import annotations
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
class InterpreterUnavailable(Exception):
"""The translation engine is down / unreachable / draining — a connection
error, HTTP 429, or a 5xx (commonly 502/503/504 through a reverse proxy while
the service restarts). Retry later, don't drop the item. ``retry_after`` holds
the server's Retry-After hint in seconds when it sent one, so the caller can
back off exactly as long as it's asked to."""
def __init__(self, message: str, *, retry_after: float | None = None):
super().__init__(message)
self.retry_after = retry_after
class InterpreterBadRequest(Exception):
"""Bad request params (HTTP 400)."""
def _url(base_url: str, path: str) -> str:
return f"{base_url.rstrip('/')}{path}"
def _parse_retry_after(resp) -> float | None:
"""Parse a Retry-After header (RFC 7231: delta-seconds or an HTTP-date) into
non-negative seconds, or None if absent/unparseable — so a gracefully-
draining Interpreter can tell Curator exactly how long to wait before it
tries again."""
raw = (resp.headers.get("Retry-After") or "").strip()
if not raw:
return None
try:
return max(0.0, float(int(raw))) # delta-seconds form
except ValueError:
pass
try: # HTTP-date form
when = parsedate_to_datetime(raw)
except (TypeError, ValueError):
return None
if when is None:
return None
if when.tzinfo is None:
when = when.replace(tzinfo=UTC)
return max(0.0, (when - datetime.now(UTC)).total_seconds())
# A shared session pools the keep-alive connection across the per-post sweep
# calls, and retries CONNECT failures only (connect=2, short backoff) — smoothing
# the instant a reverse proxy reloads. Status codes are deliberately NOT retried
# (status=0, raise_on_status=False): translate() maps 429/5xx → InterpreterUnavailable
# itself, and letting urllib3 retry a draining 503 would defeat the Retry-After
# backoff we honour upstream.
_retry = Retry(
total=None, connect=2, read=0, redirect=0, status=0,
backoff_factor=0.3, raise_on_status=False,
)
session = requests.Session()
_adapter = HTTPAdapter(max_retries=_retry)
session.mount("http://", _adapter)
session.mount("https://", _adapter)
def health(base_url: str, *, timeout: float = 5.0) -> bool:
"""True iff the Interpreter LLM engine is up. Any error (unset URL, network,
non-200, engine down) → False, so the sweep just no-ops rather than raising."""
if not base_url:
return False
try:
r = session.get(_url(base_url, "/v1/health"), timeout=timeout)
except requests.RequestException:
return False
if r.status_code != 200:
return False
engines = (r.json() or {}).get("engines") or {}
return bool(engines.get("llm"))
def translate(
texts: list[str], *, base_url: str, target: str = "en",
source: str = "auto", timeout: float = 120.0,
) -> dict:
"""Translate a batch. Returns::
{"translations": [str, ...], # SAME order & length as `texts`
"detected_lang": str | None, # aggregate: describes the FIRST item
"detected_confidence": float | None, # detector's confidence, if given
"engine": str | None,
"engine_version": str | None}
Interpreter batch metadata is aggregate (first item only) — fine for one
post's ``[title, description]`` batch since they share a language. Passthrough
items (already target-language / emoji-only) come back UNCHANGED in their
slot. Raises InterpreterUnavailable on a connection error / 429 / 5xx (retry
later, honouring Retry-After), InterpreterBadRequest on 400. (Scribe note
#1347.)
"""
if not texts:
return {"translations": [], "detected_lang": None,
"detected_confidence": None,
"engine": None, "engine_version": None}
try:
r = session.post(
_url(base_url, "/v1/translate"),
json={"q": list(texts), "source": source,
"target": target, "engine": "auto"},
timeout=timeout,
)
except requests.RequestException as e:
raise InterpreterUnavailable(str(e)) from e
if r.status_code == 400:
raise InterpreterBadRequest((r.json() or {}).get("error", "bad request"))
# A gracefully-draining service — often behind a reverse proxy — returns 429
# or a 5xx gateway error (502/503/504), not always a clean 503. Treat every
# one as "unavailable, retry later" and honour any Retry-After it sends, so a
# rolling Interpreter restart interrupts the sweep cleanly (instead of raising
# an opaque HTTPError) and resumes exactly when the service says it's ready.
if r.status_code == 429 or r.status_code >= 500:
raise InterpreterUnavailable(
f"interpreter unavailable (HTTP {r.status_code})",
retry_after=_parse_retry_after(r),
)
r.raise_for_status()
body = r.json() or {}
out = body.get("translatedText")
# q was an array → translatedText is an array (same order & length). Guard a
# scalar reply just in case (shouldn't happen for an array q).
if not isinstance(out, list):
out = [out]
interp = body.get("interpreter") or {}
detected = body.get("detectedLanguage") or {}
return {
"translations": out,
"detected_lang": detected.get("language"),
"detected_confidence": detected.get("confidence"),
"engine": interp.get("engine"),
"engine_version": interp.get("engine_version"),
}
+13
View File
@@ -200,6 +200,8 @@ class PostFeedService:
atts_map = await self._attachments_for([post.id])
item = self._to_dict(post, artist, source, thumbs_map, atts_map)
item["description_full"] = html_to_plain(post.description)
# Full (uncapped) translated description for the detail view (#143).
item["description_translated_full"] = post.description_translated
# Sanitized HTML body for faithful (semantic) rendering in the post view;
# detail-only (the feed list stays lightweight plain text). None when the
# post has no body. Inline `<img>` sources are remapped to locally-served
@@ -371,6 +373,12 @@ class PostFeedService:
description_plain, truncated = None, False
else:
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
# Translation (#143): the stored translated description is already plain
# text; truncate it the same way for the card.
desc_trans = post.description_translated
desc_trans_short = (
truncate_at_word(desc_trans, DESCRIPTION_LIMIT)[0] if desc_trans else None
)
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
# `source` is null for filesystem-imported posts with no live
# subscription (alembic 0030). Frontend renders that as a
@@ -384,6 +392,11 @@ class PostFeedService:
"downloaded_at": post.downloaded_at.isoformat(),
"description_plain": description_plain,
"description_truncated": truncated,
# Translation-forward fields (#143): shown by default when present;
# UI toggles back to the originals above. Source lang labels the toggle.
"post_title_translated": post.post_title_translated,
"description_translated": desc_trans_short,
"translated_source_lang": post.translated_source_lang,
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"source": (
{"id": source.id, "platform": source.platform}
@@ -29,6 +29,12 @@ def _post_dict(p: Post) -> dict:
"date": p.post_date.isoformat() if p.post_date else None,
"description_html": sanitize_post_html(p.description),
"attachment_count": p.attachment_count,
# Translation (#143): the English title/description shown by default when
# a translation exists; the UI toggles to the original. Source lang labels
# the original.
"title_translated": p.post_title_translated,
"description_translated": p.description_translated,
"translated_source_lang": p.translated_source_lang,
}
+36
View File
@@ -79,6 +79,14 @@ VERIFY_PAGE = 200
FFPROBE_TIMEOUT_SECONDS = 10
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# Orphaned staging files: downloads/imports stage into <name>.part / <name>.partial
# then os.replace() into place (importer / external_fetch / native_ingest_common /
# attachment_store), so a kill mid-write leaves a discardable temp, never a corrupt
# final. cleanup_orphaned_temp_files sweeps ones left behind; the min-age guard
# keeps it from deleting an in-flight download's staging file mid-write.
IMAGES_ROOT = Path("/images")
TEMP_STAGING_SUFFIXES = (".part", ".partial")
ORPHAN_TEMP_MIN_AGE_HOURS = 6
# Audit 2026-06-02: per-entity recovery sweep thresholds. Each must be
# > the entity's longest legitimate runtime (its task's time_limit + a
@@ -339,6 +347,34 @@ def cleanup_old_tasks() -> int:
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.cleanup_orphaned_temp_files")
def cleanup_orphaned_temp_files() -> int:
"""Delete orphaned .part/.partial staging files under the images root, left by
a download/import killed mid-write. Only removes files older than
ORPHAN_TEMP_MIN_AGE_HOURS so an in-flight download's staging file is never
pulled out from under it. Returns the count removed."""
if not IMAGES_ROOT.is_dir():
return 0
cutoff = datetime.now(UTC).timestamp() - ORPHAN_TEMP_MIN_AGE_HOURS * 3600
removed = 0
for path in IMAGES_ROOT.rglob("*"):
if path.suffix not in TEMP_STAGING_SUFFIXES or not path.is_file():
continue
try:
if path.stat().st_mtime >= cutoff:
continue # too fresh — may be an active download
path.unlink()
removed += 1
except OSError as exc:
log.warning("cleanup_orphaned_temp_files: %s: %s", path, exc)
if removed:
log.info(
"cleanup_orphaned_temp_files: removed %d orphaned staging file(s)",
removed,
)
return removed
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_task_runs")
def recover_stalled_task_runs() -> int:
"""Flip task_run rows stuck in 'running' past their queue-specific
+310
View File
@@ -0,0 +1,310 @@
"""Post-text translation Celery tasks (milestone 143 + re-translate m146).
Backfills non-English post title/description to the target language via the
self-hosted Interpreter LAN service, storing the result + engine_version so
viewing is instant and re-runs are cache-fast. Sync (Celery workers are sync),
same pattern as the other backfill sweeps. No-op unless translation is enabled,
a base URL is set, and the service is healthy.
Two entry points:
- ``translate_posts`` — the periodic untranslated sweep (every 8h;
translated_source_lang IS NULL only), one bounded chunk per fire. The Settings
"Translate now" button calls it with ``drain=True`` to chase the tail
run-until-done and clear the whole backlog in a single press.
- ``retranslate_posts`` — after a model change, resets the stored translation
columns for a scoped set of posts (all, or a given set of artists) so the
untranslated selection re-runs them, then chases the tail until drained
(run-until-done). The Interpreter cache keys on engine_version: a changed
model genuinely re-translates, an unchanged one is ~1ms cache-fast.
"""
import logging
from datetime import UTC, datetime
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import func, or_, select, update
from ..celery_app import celery
from ..models import ImportSettings, Post
from ..services import interpreter_client as ic
from ..utils.text import html_to_plain
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
# Bound one run's work so it commits progress rather than dying at the time
# limit; the rest resumes next cycle (idempotent — only untranslated posts are
# picked, so an interrupted run just re-runs = rule-89 recovery).
_MAX_POSTS_PER_RUN = 300
# Short gap between run-until-done chunks so a big re-translate finishes on its
# own without monopolising the maintenance_long worker in one uninterrupted run.
_RETRANSLATE_COUNTDOWN = 5
# When Interpreter drains mid-chunk (graceful restart), resume the bulk
# re-translate after the service's Retry-After hint if it sent one, else this
# default; capped so a bogus/huge hint can't park the work longer than the daily
# beat's own safety net would.
_INTERRUPT_BACKOFF = 60
_INTERRUPT_BACKOFF_MAX = 900
def _interrupt_backoff(retry_after) -> int:
"""Seconds to wait before resuming after an Interpreter interruption: the
server's Retry-After hint (capped), else the default."""
if retry_after and retry_after > 0:
return int(min(retry_after, _INTERRUPT_BACKOFF_MAX))
return _INTERRUPT_BACKOFF
@celery.task(
name="backend.app.tasks.translation.translate_posts",
soft_time_limit=1800, time_limit=2100,
)
def translate_posts(drain: bool = False) -> str:
"""Translate untranslated non-English post title/description (default target
en) via Interpreter. Per-post [title, description] batch → per-post detected
language. Passthrough / already-target posts are marked handled (source ==
target) with no stored translation. 503 or a connection error interrupts the
run (retry next cycle); 400 stops it (fix config); posts done so far stay
committed. No-op unless configured + healthy. Returns a summary string.
``drain`` (the Settings "Translate now" button) chases the tail
run-until-done: on a clean chunk with work still remaining it re-enqueues
itself until the untranslated backlog is zero, like ``retranslate_posts``. The
periodic beat leaves it False → one bounded chunk per fire (the 8h cadence
drives the rest, enough for the trickle of newly-imported posts)."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
ready = _translation_config(session)
if isinstance(ready, str):
return ready
base_url, target = ready
posts = _select_untranslated(session, None, _MAX_POSTS_PER_RUN)
status, translated, retry_after = _translate_batch(
session, posts, base_url, target,
)
# Drain mode (manual "Translate now"): chase the tail until the backlog is
# zero, so one press clears the whole pile instead of one 300-chunk.
# Termination is guaranteed — every handled post leaves the untranslated
# set, so the remaining count strictly decreases each chunk.
if status == "ok" and drain:
remaining = _count_untranslated(session, None)
if remaining:
translate_posts.apply_async(
(), {"drain": True}, countdown=_RETRANSLATE_COUNTDOWN,
)
log.info(
"translate_posts: draining — %d remaining → re-enqueued",
remaining,
)
# Interpreter drained mid-chunk (graceful restart): don't make a manual
# "Translate now" wait a whole beat cycle — re-enqueue after its
# Retry-After hint (or the default backoff), preserving `drain` so a bulk
# drain resumes cleanly. Self-terminating: once the service is down the
# health gate returns "interpreter unavailable" early.
elif status == "interrupted" and _count_untranslated(session, None):
delay = _interrupt_backoff(retry_after)
translate_posts.apply_async((), {"drain": drain}, countdown=delay)
log.info(
"translate_posts: interrupted (service draining) → retry in %ss",
delay,
)
return _summary(status, translated, len(posts))
@celery.task(
name="backend.app.tasks.translation.retranslate_posts",
soft_time_limit=1800, time_limit=2100,
)
def retranslate_posts(artist_ids=None, _reset_done=False) -> str:
"""Re-translate posts after a model change. On the first call resets the
stored translation columns to NULL for the scoped posts — all artists when
``artist_ids`` is falsy, else ``WHERE artist_id IN artist_ids`` — so the
normal untranslated selection re-runs them through Interpreter. Then handles
one bounded chunk and re-enqueues itself until the scoped backlog is drained
(run-until-done; ``_reset_done`` guards against wiping fresh work on the
follow-up chunks). No reset happens unless the service is configured +
healthy, so translations are never cleared when they can't be rebuilt."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
ready = _translation_config(session)
if isinstance(ready, str):
return ready
base_url, target = ready
if not _reset_done:
n = _reset_translations(session, artist_ids)
log.info(
"retranslate_posts: reset %d post(s) (artist_ids=%s)",
n, artist_ids,
)
posts = _select_untranslated(session, artist_ids, _MAX_POSTS_PER_RUN)
status, translated, retry_after = _translate_batch(
session, posts, base_url, target,
)
# run-until-done: chase the tail when the chunk finished cleanly.
# Termination is guaranteed: every handled post leaves the NULL set, so
# the remaining count strictly decreases each chunk.
if status == "ok":
remaining = _count_untranslated(session, artist_ids)
if remaining:
retranslate_posts.apply_async(
(),
{"artist_ids": artist_ids, "_reset_done": True},
countdown=_RETRANSLATE_COUNTDOWN,
)
log.info(
"retranslate_posts: %d remaining → re-enqueued", remaining,
)
# Interpreter drained mid-chunk (graceful restart / 429 / 5xx): don't
# stall the bulk re-translate until the daily beat — resume it after the
# service's Retry-After hint (or the default backoff), with
# _reset_done=True so the fresh work is never re-wiped. Self-terminating:
# if the service is still down next run, _translation_config's health gate
# returns "interpreter unavailable" early and nothing re-enqueues.
elif status == "interrupted" and _count_untranslated(session, artist_ids):
delay = _interrupt_backoff(retry_after)
retranslate_posts.apply_async(
(),
{"artist_ids": artist_ids, "_reset_done": True},
countdown=delay,
)
log.info(
"retranslate_posts: interrupted (service draining) → retry in %ss",
delay,
)
return _summary(status, translated, len(posts))
def _translation_config(session):
"""Resolve (base_url, target) if translation is enabled + healthy, else a
short status string ("disabled" / "interpreter unavailable") for the caller
to return. Centralises the guard so translate/retranslate agree exactly."""
cfg = ImportSettings.load_sync(session)
if not cfg.translation_enabled or not cfg.interpreter_base_url.strip():
return "disabled"
base_url = cfg.interpreter_base_url.strip()
target = (cfg.translation_target_lang or "en").strip() or "en"
if not ic.health(base_url):
return "interpreter unavailable"
return base_url, target
def _untranslated_filter(stmt, artist_ids):
"""Add the untranslated-post predicate (+ optional artist scope) to a
select/count over Post. Untranslated = translated_source_lang IS NULL with
some text to translate."""
stmt = stmt.where(
Post.translated_source_lang.is_(None)
).where(or_(
Post.post_title.is_not(None), Post.description.is_not(None),
))
if artist_ids:
stmt = stmt.where(Post.artist_id.in_(artist_ids))
return stmt
def _select_untranslated(session, artist_ids, limit):
return session.execute(
_untranslated_filter(select(Post), artist_ids).limit(limit)
).scalars().all()
def _count_untranslated(session, artist_ids) -> int:
return int(session.execute(
_untranslated_filter(select(func.count(Post.id)), artist_ids)
).scalar_one())
def _reset_translations(session, artist_ids) -> int:
"""Clear the stored translation columns for scoped, already-handled posts so
the untranslated sweep re-runs them. Only touches rows that were translated
(translated_source_lang IS NOT NULL) — untranslated rows are already NULL.
Returns the row count reset (commits it)."""
stmt = (
update(Post)
.where(Post.translated_source_lang.is_not(None))
.values(
post_title_translated=None,
description_translated=None,
translated_source_lang=None,
translation_engine_version=None,
translated_at=None,
)
)
if artist_ids:
stmt = stmt.where(Post.artist_id.in_(artist_ids))
n = session.execute(stmt).rowcount
session.commit()
return n
def _translate_batch(session, posts, base_url: str, target: str):
"""Translate each post in the chunk with a per-post commit (an interrupted
run keeps progress). Returns (status, translated, retry_after) where status is
one of "ok" / "interrupted" (unavailable/drain) / "stopped" (400) / "timeout";
retry_after is the server's Retry-After hint in seconds on an interrupt, else
None."""
translated = 0
for post in posts:
try:
translated += _translate_one(session, post, base_url, target)
session.commit()
except ic.InterpreterUnavailable as e:
session.rollback()
return "interrupted", translated, getattr(e, "retry_after", None)
except ic.InterpreterBadRequest as e:
session.rollback()
log.warning("translate bad request: %s", e)
return "stopped", translated, None
except SoftTimeLimitExceeded:
return "timeout", translated, None
return "ok", translated, None
def _summary(status: str, translated: int, scanned: int) -> str:
if status == "interrupted":
return f"interrupted (service down) — translated={translated}"
if status == "stopped":
return f"stopped (bad request) — translated={translated}"
if status == "timeout":
return f"time limit — translated={translated}"
return f"translated={translated} scanned={scanned}"
def _translate_field(text: str, base_url: str, target: str):
"""Translate ONE field independently. Returns (translated, source_lang,
engine_version), or (None, None, None) when there's nothing to store — empty
text, already the target language, or a passthrough (engine "none"). Per-field
(not aggregate first-item) detection so a non-English description still gets
translated when the title is already English (mixed-language posts)."""
if not text:
return None, None, None
res = ic.translate([text], base_url=base_url, target=target)
detected = res["detected_lang"] or target
if detected == target or res["engine"] == "none":
return None, None, None
return res["translations"][0], detected, res["engine_version"]
def _translate_one(session, post, base_url: str, target: str) -> int:
"""Translate a post's title/description in place, each field independently.
Returns 1 if it stored at least one translation, 0 when the post is all
passthrough/empty (still marks it handled so the sweep won't revisit it)."""
title = (post.post_title or "").strip()
desc = (html_to_plain(post.description) if post.description else "") or ""
desc = desc.strip()
title_tr, title_lang, title_ev = _translate_field(title, base_url, target)
desc_tr, desc_lang, desc_ev = _translate_field(desc, base_url, target)
if title_tr is None and desc_tr is None:
# Nothing non-target (or nothing to translate) → handled, store nothing.
post.translated_source_lang = target
return 0
post.post_title_translated = title_tr
post.description_translated = desc_tr
# Source lang = whichever field was actually non-target (for a mixed post,
# the translated field's language, not the already-English one).
post.translated_source_lang = title_lang or desc_lang
post.translation_engine_version = title_ev or desc_ev
post.translated_at = datetime.now(UTC)
return 1
+65
View File
@@ -8,6 +8,35 @@
# run `docker compose up` from this directory and switches images to
# local builds + DEBUG logging.
# Rolling-deploy safety (Swarm / `docker stack deploy`): update one task at a
# time, START the new task before stopping the old (zero-downtime via the ingress
# mesh), and if the new task doesn't reach a healthy state within `monitor`, roll
# back to the previous image automatically. `monitor` is sized above web's
# healthcheck start_period so a broken image that never goes healthy is caught.
# Plain `docker compose up` ignores `deploy:` (it warns + skips), so the dev
# override is unaffected. Referenced by each long-lived service below.
x-deploy-policy: &deploy_policy
update_config:
order: start-first
failure_action: rollback
monitor: 90s
rollback_config:
order: start-first
restart_policy:
condition: any
delay: 10s
# Worker liveness: ping THIS container's celery node over the broker. Lenient
# (60s interval, 3 retries, 60s start_period) so a transient broker blip never
# false-flags a worker into a rollback. `$$HOSTNAME` → `$HOSTNAME` for the shell;
# celery's default node name is celery@<hostname> (the container id).
x-celery-healthcheck: &celery_healthcheck
test: ["CMD-SHELL", "celery -A backend.app.celery_app:celery inspect ping -d celery@$$HOSTNAME --timeout 10 >/dev/null 2>&1"]
interval: 60s
timeout: 15s
retries: 3
start_period: 60s
services:
redis:
image: redis:7-alpine
@@ -47,6 +76,24 @@ services:
web:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["web"]
# Graceful shutdown: give the container time to drain in-flight work on a
# deploy (docker SIGTERMs, then SIGKILLs after this window — default is only
# 10s, far too short for real jobs). Hypercorn/Celery both warm-shut-down on
# SIGTERM; per-lane values sized to typical task length. Anything that still
# outruns the window is re-queued (task_reject_on_worker_lost) and re-driven
# by the 5-min recovery sweeps, so a kill never corrupts. web = short HTTP
# requests + the occasional file download.
stop_grace_period: 30s
# Liveness for rolling deploys: /api/health is a no-DB 200 (just proves the
# app booted + serves HTTP after `alembic upgrade head`). start_period covers
# the migration + boot so a slow start isn't mis-flagged.
healthcheck:
test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/api/health', timeout=5).status==200 else 1)\""]
interval: 15s
timeout: 6s
retries: 3
start_period: 40s
deploy: *deploy_policy
ports:
- "${PORT:-8080}:8080"
environment: &app_env
@@ -77,6 +124,10 @@ services:
worker:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["worker"]
# Drain in-flight import/thumbnail/download tasks before SIGKILL on deploy.
stop_grace_period: 90s
healthcheck: *celery_healthcheck
deploy: *deploy_policy
environment:
<<: *app_env
CELERY_QUEUES: default,import,thumbnail,download
@@ -93,6 +144,10 @@ services:
scheduler:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["scheduler"]
# Quick maintenance/scan lane + beat — short tasks, modest drain window.
stop_grace_period: 60s
healthcheck: *celery_healthcheck
deploy: *deploy_policy
environment:
<<: *app_env
CELERY_QUEUES: maintenance,scan
@@ -110,6 +165,12 @@ services:
maintenance-long:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["worker"]
# Longest lane (DB backups, library audits, translation backfill) — give it
# the most room to finish a chunk gracefully. Chunked + idempotent, so a job
# that still outruns this resumes cleanly next run rather than corrupting.
stop_grace_period: 180s
healthcheck: *celery_healthcheck
deploy: *deploy_policy
environment:
<<: *app_env
CELERY_QUEUES: maintenance_long
@@ -125,6 +186,10 @@ services:
ml-worker:
image: git.fabledsword.com/bvandeusen/fabledcurator-ml:dev
command: ["ml-worker"]
# A single GPU inference pass can run tens of seconds — let it finish.
stop_grace_period: 120s
healthcheck: *celery_healthcheck
deploy: *deploy_policy
environment:
<<: *app_env
volumes:
@@ -101,6 +101,47 @@
</v-card>
</v-dialog>
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Translation</h2>
<p class="fc-muted text-body-2 mb-3">
Re-translate every post by {{ overview.name }} through the current
Interpreter model — clears the stored translations and rebuilds them.
Use this after switching translation models to refresh just this artist.
</p>
<div class="d-flex align-center flex-wrap" style="gap: 10px;">
<v-btn
size="small" variant="tonal" rounded="pill"
prepend-icon="mdi-translate"
:loading="retranslating" :disabled="!translationEnabled"
@click="confirmRetranslate = true"
>Re-translate posts</v-btn>
<span v-if="!translationEnabled" class="fc-muted text-caption">
Translation is off — enable it in Settings → Maintenance.
</span>
</div>
</section>
<!-- m146: per-artist re-translate — clears + rebuilds this artist's stored
translations (used after a translation-model change). -->
<v-dialog v-model="confirmRetranslate" max-width="440">
<v-card>
<v-card-title class="fc-h2">Re-translate {{ overview.name }}?</v-card-title>
<v-card-text class="text-body-2">
Clears the stored translation for every post by
<strong>{{ overview.name }}</strong> and re-runs them through your
current Interpreter model. Unchanged text comes back from the cache
instantly. Runs in the background until done.
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="confirmRetranslate = false">Cancel</v-btn>
<v-btn color="accent" :loading="retranslating" @click="onRetranslate">
Re-translate
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Danger zone</h2>
<ArtistDangerZone
@@ -113,9 +154,11 @@
</template>
<script setup>
import { computed, ref } from 'vue'
import { computed, onMounted, ref } from 'vue'
import { useRouter, RouterLink } from 'vue-router'
import { useApi } from '../../composables/useApi.js'
import { useImportStore } from '../../stores/import.js'
import { useSourcesStore } from '../../stores/sources.js'
import { toast } from '../../utils/toast.js'
import ArtistDangerZone from './ArtistDangerZone.vue'
@@ -125,8 +168,38 @@ const props = defineProps({
})
const router = useRouter()
const api = useApi()
const importStore = useImportStore()
const sources = useSourcesStore()
// m146: per-artist re-translate. Gated on translation being enabled globally
// (the endpoint 400s otherwise) — read the shared import settings once.
const translationEnabled = ref(false)
const retranslating = ref(false)
const confirmRetranslate = ref(false)
onMounted(async () => {
try {
await importStore.loadSettings()
translationEnabled.value = !!importStore.settings?.translation_enabled
} catch { /* non-fatal — the button just stays disabled */ }
})
async function onRetranslate () {
retranslating.value = true
try {
await api.post('/api/settings/translation/retranslate', {
body: { artist_id: props.overview.id },
})
confirmRetranslate.value = false
toast({ text: `Re-translating ${props.overview.name}`, type: 'success' })
} catch (e) {
toast({ text: `Re-translate failed: ${e.message}`, type: 'error' })
} finally {
retranslating.value = false
}
}
// #130: move a source into another artist.
const moveOpen = ref(false)
const moveSource = ref(null)
@@ -42,13 +42,23 @@
· {{ e.post.attachment_count }} files
</span>
</div>
<div v-if="hasTranslation(e)" class="fc-prov__actions">
<a
href="#" @click.prevent="toggleOrig(e.provenance_id)"
>{{ showOrig[e.provenance_id] ? 'Show translation' : `Show original${langLabel(e)}` }}</a>
</div>
<div v-if="e.post.description_html" class="fc-prov__actions">
<a
href="#" @click.prevent="toggleDesc(e.provenance_id)"
>{{ expanded[e.provenance_id] ? 'Hide description ' : 'Show description ' }}</a>
</div>
<!-- Translated body = plain text; original = sanitized HTML (#143). -->
<p
v-if="showTranslated(e) && expanded[e.provenance_id] && e.post.description_translated"
class="fc-prov__desc"
>{{ e.post.description_translated }}</p>
<div
v-if="e.post.description_html && expanded[e.provenance_id]"
v-else-if="e.post.description_html && expanded[e.provenance_id]"
class="fc-prov__desc" v-html="e.post.description_html"
/>
</article>
@@ -117,8 +127,12 @@ const effectiveImage = computed(() => props.image ?? modal.current)
// 2026-05-28. Reset when the viewed image changes.
const expanded = reactive({})
function toggleDesc(id) { expanded[id] = !expanded[id] }
// Per-entry "show the original (untranslated) text" toggle (#143).
const showOrig = reactive({})
function toggleOrig(id) { showOrig[id] = !showOrig[id] }
watch(() => effectiveId.value, () => {
for (const k of Object.keys(expanded)) delete expanded[k]
for (const k of Object.keys(showOrig)) delete showOrig[k]
})
watch(
@@ -157,10 +171,23 @@ const show = computed(() => {
const attachments = computed(() => state.value?.attachments || [])
function postDate(e) { return formatPostDate(e.post.date) }
// Translation-forward (#143): show the English by default when present.
function hasTranslation(e) {
return !!(e.post.title_translated || e.post.description_translated)
}
function showTranslated(e) {
return hasTranslation(e) && !showOrig[e.provenance_id]
}
function langLabel(e) {
return e.post.translated_source_lang ? ` (${e.post.translated_source_lang})` : ''
}
function postTitle(e) {
// Titles can arrive as stored HTML (e.g. "<strong>…</strong>"); render
// as plain text (the CSS makes it bold).
return toPlainText(e.post.title) || `Post ${e.post.external_post_id}`
const t = showTranslated(e) && e.post.title_translated
? e.post.title_translated
: e.post.title
return toPlainText(t) || `Post ${e.post.external_post_id}`
}
function openPost(postId, artistId) {
+59 -41
View File
@@ -1,7 +1,7 @@
<template>
<span class="fc-tag-chip" @mouseenter="onEnter" @mouseleave="onLeave">
<v-chip
size="small" closable
size="default" :closable="!unconfirmedAuto"
:color="store.colorFor(tag.kind)" variant="tonal"
class="fc-tag-chip__nav"
role="link"
@@ -9,27 +9,33 @@
@click="$emit('navigate', tag)"
@click:close="$emit('remove', tag.id)"
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
<v-icon start size="small">{{ iconFor(tag.kind) }}</v-icon>
<span class="fc-tag-chip__name">{{ tag.name }}</span><v-icon
v-if="tag.is_system" end size="x-small" class="fc-tag-chip__system"
title="System tag — tagged items are excluded from training other concepts"
>mdi-shield-outline</v-icon><span
v-if="tag.fandom_id" class="fc-tag-chip__fandom"
:title="tag.fandom_name ? `Fandom: ${tag.fandom_name}` : 'Has a fandom'"
><template v-if="fandomLabel">&nbsp;{{ fandomLabel }}</template></span><span
v-if="unconfirmedAuto" class="fc-tag-chip__auto"
title="Auto-applied — provisional: it won't train the model and can be retracted until you confirm it."
>auto</span>
><template v-if="fandomLabel">&nbsp;{{ fandomLabel }}</template></span>
<!-- Provisional auto-tag: an in-pill yes/no pair REPLACES the (the pair
itself signals "auto" no separate label, operator 2026-07-07). Yes
confirms it (trains + shields from retraction); No removes it (records
a negative). Both hand focus back to the tag input via TagPanel. -->
<span v-if="unconfirmedAuto" class="fc-tag-chip__verdict">
<button
type="button" class="fc-tag-chip__yes"
:title="`Yes — keep “${tag.name}” (trains the model, won't be retracted)`"
:aria-label="`Confirm ${tag.name}`"
@click.stop="$emit('confirm', tag)"
><v-icon size="15">mdi-check</v-icon></button>
<button
type="button" class="fc-tag-chip__no"
:title="`No — remove “${tag.name}”`"
:aria-label="`Reject ${tag.name}`"
@click.stop="$emit('remove', tag.id)"
><v-icon size="15">mdi-close</v-icon></button>
</span>
</v-chip>
<!-- Keep/confirm an auto-applied tag: promotes it to a training positive and
shields it from the retraction sweep (milestone 139). Only shown for
provisional (unconfirmed) auto-tags. -->
<button
v-if="unconfirmedAuto" class="fc-tag-chip__confirm" type="button"
:title="`Keep “${tag.name}” — confirm this auto-tag so it trains the model and won't be retracted`"
:aria-label="`Confirm ${tag.name}`"
@click.stop="$emit('confirm', tag)"
><v-icon size="16">mdi-check</v-icon></button>
<!-- Modal-safe kebab is baked into KebabMenu (this chip lives in the
teleported image modal #711). System tags hide it entirely: rename
is refused server-side (the hygiene machinery keys on the row) and
@@ -71,8 +77,8 @@ const store = useTagStore()
const api = useApi()
// An auto-applied tag the operator hasn't confirmed yet — provisional (milestone
// 139): it doesn't train the model and the retraction sweep can drop it. Shows
// the "auto" badge + a Keep/confirm button. `source`/`confirmed` come from the
// 139): it doesn't train the model and the retraction sweep can drop it. Shows an
// in-pill yes/no pair in place of the ✕. `source`/`confirmed` come from the
// applied-tags payload (list_for_image / get_image_with_tags).
const unconfirmedAuto = computed(() =>
AUTO_SOURCES.includes(props.tag.source) && !props.tag.confirmed
@@ -130,13 +136,26 @@ function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
</script>
<style scoped>
/* A chip must never exceed the rail — a long character+fandom(+AUTO) chip used
to overflow the right edge and clip its close ✕ (operator-flagged 2026-07-06).
The name is the elastic part: it ellipsis-truncates so the ✕, fandom, and AUTO
badge stay reachable. Full name is still available via the chip's hover title. */
/* A chip must never exceed the rail — a long character+fandom chip used to
overflow the right edge and clip its trailing control (operator-flagged
2026-07-06). The name is the elastic part: it ellipsis-truncates so the ✕ (or
the auto-tag yes/no pair) and fandom stay reachable. Full name is on the
chip's hover title. */
.fc-tag-chip { display: inline-flex; align-items: center; gap: 1px; max-width: 100%; min-width: 0; }
.fc-tag-chip__nav { max-width: 100%; min-width: 0; }
/* Tonal chips (esp. character = info) wash out against the dark rail — the fill
is intentionally faint. Give every tag chip a thin border in its OWN kind
colour (currentColor = the tonal chip's themed foreground) so the edge reads
clearly without touching the fill (operator-asked 2026-07-08). Theme-aware:
currentColor tracks the kind colour in either light or dark. */
.fc-tag-chip__nav { border: thin solid color-mix(in srgb, currentColor 55%, transparent); }
.fc-tag-chip__nav :deep(.v-chip__content) { min-width: 0; overflow: hidden; }
/* The bigger size=default chip widens Vuetify's negative start-margin on the
leading kind-icon, pulling it LEFT of the content box above — where that
overflow:hidden then clips the icon's left edge as a vertical slice
(operator-flagged 2026-07-08). Zero the pull so the icon sits inside the clip
box; the chip's own 12px padding keeps a comfortable left inset. */
.fc-tag-chip__nav :deep(.v-icon--start) { margin-inline-start: 0; }
.fc-tag-chip__name {
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
min-width: 0; flex: 0 1 auto;
@@ -152,28 +171,27 @@ function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
.fc-tag-chip__kebab { opacity: 0.7; }
.fc-tag-chip:hover .fc-tag-chip__kebab { opacity: 1; }
.fc-tag-chip__fandom { opacity: 0.7; font-size: 0.85em; }
/* "auto" = provisional (auto-applied, unconfirmed). Quiet pill inside the chip. */
.fc-tag-chip__auto {
display: inline-block; vertical-align: middle; margin-left: 4px;
font-size: 9px; font-weight: 700;
text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
background: rgb(var(--v-theme-surface-light));
border: 1px solid rgb(var(--v-theme-on-surface-variant), 0.3);
padding: 0 4px; border-radius: 999px;
/* Provisional auto-tag: a green ✓ / red ✗ pair in place of the ✕. The yes/no is
obvious enough on its own, so there's no "auto" label (operator-asked
2026-07-07). flex:0 0 auto keeps it visible while the name ellipsis-truncates. */
.fc-tag-chip__verdict {
flex: 0 0 auto; display: inline-flex; align-items: center; gap: 3px;
margin-left: 5px;
}
/* Keep/confirm — the SAME filled green ✓ circle as the suggestion accept button
(.fc-act--yes in SuggestionItem), so "accept this tag" reads identically
wherever it appears (operator-asked 2026-07-06). */
.fc-tag-chip__confirm {
flex: 0 0 auto;
width: 26px; height: 26px; border-radius: 50%; border: none; cursor: pointer;
/* Solid-filled (white glyph on a full success/error circle) so accept/reject
stay well-defined even on a muted tonal chip — e.g. character (info) — instead
of a faint icon that only lit up on hover. Mirrors the Suggestions rail's
verdict buttons so the affordance reads identically (operator-asked 2026-07-08). */
.fc-tag-chip__yes, .fc-tag-chip__no {
display: inline-flex; align-items: center; justify-content: center;
color: #fff; background: rgb(var(--v-theme-success));
opacity: 0.9; transition: transform 0.1s, opacity 0.1s;
width: 22px; height: 22px; padding: 0; border: none; border-radius: 50%;
color: #fff; cursor: pointer; opacity: 0.92;
transition: transform 0.1s, opacity 0.1s;
}
.fc-tag-chip__confirm:hover { opacity: 1; transform: scale(1.1); }
.fc-tag-chip__confirm:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent)); outline-offset: 1px;
.fc-tag-chip__yes { background: rgb(var(--v-theme-success)); }
.fc-tag-chip__no { background: rgb(var(--v-theme-error)); }
.fc-tag-chip__yes:hover, .fc-tag-chip__no:hover { opacity: 1; transform: scale(1.1); }
.fc-tag-chip__yes:focus-visible, .fc-tag-chip__no:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent)); outline-offset: 2px;
}
</style>
@@ -116,6 +116,10 @@ async function onConfirm(tag) {
try {
await api.post(`/api/images/${host.currentImageId}/tags/${tag.id}/confirm`)
await host.reloadTags()
// Return the cursor to the tag input — its resting position in these views,
// matching every other chip action (remove/accept) so the keyboard flow
// never leaves the field (operator-asked 2026-07-07).
focusTagInput()
}
catch (e) { errorMsg.value = e.message }
}
+49 -3
View File
@@ -58,16 +58,23 @@
</div>
<div class="fc-post-card__text">
<h3 v-if="plainTitle" class="fc-post-card__title">{{ plainTitle }}</h3>
<h3 v-if="displayTitle" class="fc-post-card__title">{{ displayTitle }}</h3>
<h3 v-else class="fc-post-card__title fc-post-card__title--missing">
Post {{ post.external_post_id }}
</h3>
<!-- Translation toggle (#143): English shown by default; reveal the
original, labelled with its detected language. -->
<button
v-if="hasTranslation" type="button" class="fc-post-card__lang"
:title="showOriginal ? 'Show the English translation' : `Show the original${sourceLangLabel} text`"
@click.stop="showOriginal = !showOriginal"
>{{ showOriginal ? 'Show translation' : `Show original${sourceLangLabel}` }}</button>
<!-- Faithful (semantic) body render once expanded: backend-sanitized
HTML (headings, lists, links, inline images). Collapsed and the
no-detail fallback stay plain text. -->
<div
v-if="hasDescription && descExpanded && descHtml"
v-if="hasDescription && showHtmlBody"
class="fc-post-card__desc fc-post-card__desc--html"
v-html="descHtml"
/>
@@ -75,7 +82,7 @@
v-else-if="hasDescription" ref="descEl"
class="fc-post-card__desc"
:class="{ 'fc-post-card__desc--clamped': !descExpanded }"
>{{ descText }}</p>
>{{ descTextDisplay }}</p>
<p v-else class="fc-post-card__desc fc-post-card__desc--missing">
(no description)
</p>
@@ -231,6 +238,32 @@ const canExpand = computed(
() => props.post.description_truncated === true || cssOverflow.value,
)
// --- translation-forward (#143): the English is shown by default when a
// translation exists; a per-card toggle reveals the original. Translations are
// plain text, so showTranslated also decides HTML-vs-plain body rendering.
const showOriginal = ref(false)
const hasTranslation = computed(
() => !!(props.post.post_title_translated || props.post.description_translated),
)
const showTranslated = computed(() => hasTranslation.value && !showOriginal.value)
const sourceLangLabel = computed(() =>
props.post.translated_source_lang ? ` (${props.post.translated_source_lang})` : '',
)
const displayTitle = computed(() =>
showTranslated.value && props.post.post_title_translated
? toPlainText(props.post.post_title_translated)
: plainTitle.value,
)
const descTextDisplay = computed(() => {
if (!showTranslated.value) return descText.value
return descExpanded.value
? (detail.value?.description_translated_full || props.post.description_translated)
: props.post.description_translated
})
const showHtmlBody = computed(
() => !showTranslated.value && descExpanded.value && !!descHtml.value,
)
function measureOverflow () {
const el = descEl.value
cssOverflow.value = !!el && el.scrollHeight > el.clientHeight + 1
@@ -467,6 +500,19 @@ function formatBytes (n) {
font-size: 0.85rem; font-weight: 600;
}
.fc-post-card__more:hover { text-decoration: underline; }
/* Translation toggle (#143) — quiet secondary affordance under the title. */
.fc-post-card__lang {
display: block; margin: 2px 0 6px; padding: 0;
background: none; border: 0; cursor: pointer;
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.75rem; font-weight: 600;
}
.fc-post-card__lang:hover {
text-decoration: underline; color: rgb(var(--v-theme-accent));
}
.fc-post-card__lang:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent)); outline-offset: 1px;
}
.fc-post-card__atts {
display: flex; flex-wrap: wrap; gap: 8px;
@@ -13,6 +13,7 @@
</p>
<div class="fc-tile-stack">
<ImportFiltersForm />
<TranslationCard />
</div>
</section>
@@ -69,6 +70,7 @@
import { onMounted, onUnmounted } from 'vue'
import ImportFiltersForm from './ImportFiltersForm.vue'
import TranslationCard from './TranslationCard.vue'
import MLBackfillCard from './MLBackfillCard.vue'
import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import ArchiveReextractCard from './ArchiveReextractCard.vue'
@@ -0,0 +1,312 @@
<template>
<MaintenanceTile
icon="mdi-translate"
title="Post translation (Interpreter)"
blurb="Translate non-English post titles + descriptions to English via your self-hosted Interpreter service, so archived posts read naturally."
>
<div class="d-flex align-center mb-3" style="gap: 10px;">
<span class="fc-section-h">Enabled</span>
<v-switch
v-model="enabled" :loading="busy" hide-details density="compact"
color="success" class="ml-auto" @update:model-value="onToggle"
/>
</div>
<p class="fc-muted text-body-2 mb-3">
Point this at your Interpreter service. It stays off until a URL is set, and
only translates posts whose text isn't already in the target language — the
English is shown by default on post cards, with a toggle back to the original.
</p>
<v-text-field
v-model="baseUrl" label="Interpreter base URL"
placeholder="http://interpreter.your-domain/" density="compact"
hide-details class="mb-3" :disabled="busy" @change="onSave"
/>
<v-text-field
v-model="targetLang" label="Target language" density="compact"
hide-details style="max-width: 160px;" class="mb-3" :disabled="busy"
@change="onSave"
/>
<div class="d-flex align-center flex-wrap mb-3" style="gap: 8px;">
<v-icon size="12" :color="statusColor">mdi-circle</v-icon>
<span class="fc-muted text-body-2">{{ statusText }}</span>
<v-btn
size="x-small" variant="tonal" rounded="pill" class="ml-2"
:loading="testing" :disabled="!baseUrl" @click="onTest"
>Test connection</v-btn>
<span
v-if="testResult !== null" class="text-caption"
:class="testResult ? 'text-success' : 'text-error'"
>{{ testResult ? ' reachable' : ' unreachable' }}</span>
</div>
<div class="d-flex align-center flex-wrap" style="gap: 10px;">
<v-btn
size="small" variant="flat" color="accent" rounded="pill"
prepend-icon="mdi-translate" :loading="running"
:disabled="!enabled || !baseUrl" @click="onRun"
>Translate now</v-btn>
<v-btn
size="small" variant="tonal" rounded="pill"
prepend-icon="mdi-refresh" :loading="retranslating"
:disabled="!enabled || !baseUrl" @click="confirmAll = true"
>Re-translate all</v-btn>
<span
v-if="status && status.active"
class="fc-muted text-caption d-inline-flex align-center" style="gap: 6px;"
>
<v-progress-circular indeterminate size="12" width="2" color="accent" />
Translating… {{ status.untranslated_count }} remaining
</span>
<span v-else-if="status" class="fc-muted text-caption">
{{ status.untranslated_count }}
post{{ status.untranslated_count === 1 ? '' : 's' }} awaiting translation
</span>
</div>
<p
v-if="status && status.last_run && ['error', 'timeout'].includes(status.last_run.status)"
class="text-caption text-error mt-1 mb-0"
>
Last translation run ended with “{{ status.last_run.status }}”. It resumes on
the next sweep; check the Interpreter connection if it persists.
</p>
<p class="fc-muted text-caption mt-2 mb-0">
Use <strong>Re-translate all</strong> after switching the Interpreter model —
it clears every stored translation and re-runs it through the new model
(unchanged text is served from cache, so it's cheap). To refresh just one
artist, use the Re-translate button on that artist's page.
</p>
<v-divider class="my-3" />
<!-- Diagnostic probe (task #1376): paste text → what Interpreter DETECTS
(language + confidence) and returns, without saving. Used to see why a
short/abbreviation-heavy English title gets mis-detected, and to pick a
detection-confidence / min-length guard from real numbers. -->
<div class="mb-1">
<span class="fc-section-h">Test translation</span>
<p class="fc-muted text-caption mt-1 mb-2">
Paste a title or snippet to see what Interpreter detects and returns —
handy for checking why a short string is mis-detected. Nothing is saved.
</p>
</div>
<v-textarea
v-model="probeText" label="Text to test" rows="2" auto-grow
density="compact" hide-details class="mb-2" :disabled="!baseUrl"
/>
<div class="d-flex align-center flex-wrap mb-2" style="gap: 8px;">
<v-btn
size="small" variant="tonal" rounded="pill" prepend-icon="mdi-magnify"
:loading="probing" :disabled="!baseUrl || !probeText.trim()"
@click="onProbe"
>Test translation</v-btn>
</div>
<v-sheet
v-if="probeResult" rounded class="pa-3 mb-1 text-body-2"
color="rgba(var(--v-theme-accent), 0.06)"
>
<div class="d-flex flex-wrap" style="gap: 4px 16px;">
<span>
<span class="fc-muted">Detected:</span>
<strong>{{ probeResult.detected_lang || '' }}</strong>
</span>
<span v-if="probeResult.detected_confidence != null">
<span class="fc-muted">Confidence:</span>
<strong>{{ fmtConf(probeResult.detected_confidence) }}</strong>
</span>
<span>
<span class="fc-muted">Engine:</span>
{{ probeResult.engine || '' }}<template
v-if="probeResult.engine_version"
> ({{ probeResult.engine_version }})</template>
</span>
<span><span class="fc-muted">Target:</span> {{ probeResult.target }}</span>
</div>
<v-divider class="my-2" />
<div class="fc-muted mb-1">
Result<template v-if="probeResult.detected_lang === probeResult.target">
— already {{ probeResult.target }}, so the sweep leaves it as-is</template>:
</div>
<p class="mb-0" style="white-space: pre-wrap;">{{
probeResult.translated || '(no change)'
}}</p>
</v-sheet>
<v-alert
v-if="err" type="error" variant="tonal" density="compact"
class="mt-3" closable @click:close="err = null"
>{{ err }}</v-alert>
<!-- m146: re-translate-everything is a bigger hammer than 'Translate now'
(it clears + rebuilds all translations), so it asks first. -->
<v-dialog v-model="confirmAll" max-width="440">
<v-card>
<v-card-title>Re-translate everything?</v-card-title>
<v-card-text class="text-body-2">
This clears the stored translation for <strong>every</strong> post and
re-runs them through your current Interpreter model. Text that hasn't
changed comes back from the cache instantly; anything the new model
translates differently is refreshed. It runs in the background until
done.
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="confirmAll = false">Cancel</v-btn>
<v-btn
color="accent" :loading="retranslating" @click="onRetranslateAll"
>Re-translate all</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</MaintenanceTile>
</template>
<script setup>
import { computed, onMounted, onUnmounted, ref } from 'vue'
import { toast } from '../../utils/toast.js'
import { useApi } from '../../composables/useApi.js'
import { useImportStore } from '../../stores/import.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
const api = useApi()
const store = useImportStore()
const enabled = ref(false)
const baseUrl = ref('')
const targetLang = ref('en')
const busy = ref(false)
const running = ref(false)
const retranslating = ref(false)
const confirmAll = ref(false)
const testing = ref(false)
const testResult = ref(null) // null = not tested this session; true/false = last result
const status = ref(null)
const err = ref(null)
// #1376 "Test translation" probe: paste text → detected lang/confidence + result.
const probeText = ref('')
const probing = ref(false)
const probeResult = ref(null)
// Confidence scale is Interpreter-defined (0100 or 01) — show it as-is, just
// trimmed to 2 decimals, so the operator reads the true number.
function fmtConf (c) {
return c == null ? '' : Math.round(c * 100) / 100
}
const statusColor = computed(() => {
if (!enabled.value || !baseUrl.value) return 'grey'
return status.value?.healthy ? 'success' : 'error'
})
const statusText = computed(() => {
if (!enabled.value) return 'Off'
if (!baseUrl.value) return 'No base URL set'
if (status.value == null) return 'Checking…'
return status.value.healthy ? 'Interpreter reachable' : 'Interpreter unreachable'
})
let pollTimer = null
async function loadStatus() {
try { status.value = await api.get('/api/settings/translation/status') }
catch { status.value = null }
// Poll live while a sweep is running so the remaining count ticks down; stop
// when idle (a fresh trigger restarts it via its own setTimeout(loadStatus)).
clearTimeout(pollTimer)
if (status.value?.active) pollTimer = setTimeout(loadStatus, 3000)
}
onUnmounted(() => clearTimeout(pollTimer))
async function onTest() {
// Ping /v1/health for the CURRENTLY-typed URL (not the saved one), so a new
// URL can be verified before enabling. Also reflects in the status dot.
testing.value = true
err.value = null
testResult.value = null
try {
const r = await api.post('/api/settings/translation/test', {
body: { base_url: baseUrl.value.trim() },
})
testResult.value = !!r.healthy
if (status.value) status.value.healthy = !!r.healthy
} catch (e) {
err.value = e.message
} finally {
testing.value = false
}
}
async function onProbe () {
probing.value = true
err.value = null
probeResult.value = null
try {
probeResult.value = await api.post('/api/settings/translation/probe', {
body: { text: probeText.value.trim() },
})
} catch (e) {
err.value = e.message
} finally {
probing.value = false
}
}
onMounted(async () => {
try {
await store.loadSettings()
const s = store.settings || {}
enabled.value = !!s.translation_enabled
baseUrl.value = s.interpreter_base_url || ''
targetLang.value = s.translation_target_lang || 'en'
} catch { /* non-fatal */ }
await loadStatus()
})
async function save(patch) {
busy.value = true
err.value = null
try { await store.patchSettings(patch) }
catch (e) { err.value = e.message }
finally { busy.value = false }
}
async function onToggle(val) {
await save({ translation_enabled: !!val })
toast({ text: val ? 'Translation on' : 'Translation off', type: 'success' })
await loadStatus()
}
async function onSave() {
await save({
interpreter_base_url: baseUrl.value.trim(),
translation_target_lang: (targetLang.value || 'en').trim() || 'en',
})
await loadStatus()
}
async function onRun() {
running.value = true
err.value = null
try {
await api.post('/api/settings/translation/run')
toast({ text: 'Translation sweep started', type: 'success' })
} catch (e) {
err.value = e.message
} finally {
running.value = false
setTimeout(loadStatus, 1500) // let the sweep make a dent, then refresh
}
}
async function onRetranslateAll() {
retranslating.value = true
err.value = null
try {
await api.post('/api/settings/translation/retranslate', { body: { all: true } })
confirmAll.value = false
toast({ text: 'Re-translating all posts…', type: 'success' })
} catch (e) {
err.value = e.message
} finally {
retranslating.value = false
setTimeout(loadStatus, 1500) // reset spikes the untranslated count — refresh it
}
}
</script>
+3 -1
View File
@@ -44,7 +44,9 @@ export const useExploreStore = defineStore('explore', () => {
// empty and let the view explain why (anchor.has_embedding === false).
if (detail.has_embedding) {
const body = await api.get('/api/gallery/similar', {
params: { similar_to: numId, limit: NEIGHBOR_LIMIT },
// exclude_wip: keep work-in-progress out of the Explore rabbit-hole
// (the gallery's own "similar" button still shows it) — operator 2026-07-08.
params: { similar_to: numId, limit: NEIGHBOR_LIMIT, exclude_wip: 1 },
})
if (!t.isCurrent()) return
neighbors.value = body.images || []
+200
View File
@@ -28,6 +28,206 @@ async def test_patch_import_settings(client):
assert body["transparency_threshold"] == 0.7
@pytest.mark.asyncio
async def test_translation_settings_defaults_and_patch(client):
# #143: translation is OFF by default with NO default host — the operator
# points it at their own Interpreter proxy and enables it.
body = await (await client.get("/api/settings/import")).get_json()
assert body["translation_enabled"] is False
assert body["interpreter_base_url"] == ""
assert body["translation_target_lang"] == "en"
ok = await client.patch("/api/settings/import", json={
"translation_enabled": True,
"interpreter_base_url": "http://interpreter.lan",
})
assert ok.status_code == 200
out = await ok.get_json()
assert out["translation_enabled"] is True
assert out["interpreter_base_url"] == "http://interpreter.lan"
# Type validation.
assert (await client.patch(
"/api/settings/import", json={"translation_enabled": "yes"})).status_code == 400
assert (await client.patch(
"/api/settings/import", json={"interpreter_base_url": 123})).status_code == 400
@pytest.mark.asyncio
async def test_translation_status_defaults(client):
# Off + no URL → no network health call, healthy False (#143).
body = await (await client.get("/api/settings/translation/status")).get_json()
assert body["enabled"] is False
assert body["base_url_set"] is False
assert body["healthy"] is False
assert "untranslated_count" in body
assert body["active"] is False # no sweep running
assert body["last_run"] is None
@pytest.mark.asyncio
async def test_translation_status_reports_active_and_last_run(client, db):
# A running translate/retranslate TaskRun → active True; the most recent
# finished one → last_run (task basename + status).
from datetime import UTC, datetime
from backend.app.models import TaskRun
db.add(TaskRun(
celery_task_id="r-run", queue="maintenance_long",
task_name="backend.app.tasks.translation.retranslate_posts",
started_at=datetime.now(UTC), status="running",
))
db.add(TaskRun(
celery_task_id="r-done", queue="maintenance_long",
task_name="backend.app.tasks.translation.translate_posts",
started_at=datetime.now(UTC), finished_at=datetime.now(UTC),
status="success",
))
await db.commit()
body = await (await client.get("/api/settings/translation/status")).get_json()
assert body["active"] is True
assert body["last_run"]["task"] == "translate_posts"
assert body["last_run"]["status"] == "success"
@pytest.mark.asyncio
async def test_translation_run_requires_config(client):
resp = await client.post("/api/settings/translation/run")
assert resp.status_code == 400 # disabled / no base URL
@pytest.mark.asyncio
async def test_translation_test_connection(client, monkeypatch):
# Tests a GIVEN url (not the saved one) without persisting anything.
monkeypatch.setattr("backend.app.api.settings.ic.health", lambda *a, **k: True)
resp = await client.post(
"/api/settings/translation/test", json={"base_url": "http://i.lan"})
assert resp.status_code == 200
assert (await resp.get_json())["healthy"] is True
# Empty URL → False, no health call.
empty = await client.post(
"/api/settings/translation/test", json={"base_url": ""})
assert (await empty.get_json())["healthy"] is False
@pytest.mark.asyncio
async def test_translation_run_enqueues_when_configured(client, monkeypatch):
sink = {}
monkeypatch.setattr(
"backend.app.tasks.translation.translate_posts.delay",
lambda *a, **k: sink.update(k) or type("R", (), {"id": "t1"})(),
)
await client.patch("/api/settings/import", json={
"translation_enabled": True, "interpreter_base_url": "http://i.lan",
})
resp = await client.post("/api/settings/translation/run")
assert resp.status_code == 202
assert (await resp.get_json())["celery_task_id"] == "t1"
assert sink["drain"] is True # "Translate now" chases the whole backlog
@pytest.mark.asyncio
async def test_translation_probe_requires_text(client):
resp = await client.post(
"/api/settings/translation/probe", json={"text": " "})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_translation_probe_requires_base_url(client):
# Text given but no Interpreter URL saved → nothing to probe against.
resp = await client.post(
"/api/settings/translation/probe", json={"text": "hello"})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_translation_probe_returns_detection(client, monkeypatch):
# The diagnostic surfaces detected language + confidence + result, unsaved.
monkeypatch.setattr(
"backend.app.api.settings.ic.translate",
lambda texts, **k: {
"translations": ["Work in Progress"],
"detected_lang": "de", "detected_confidence": 71.5,
"engine": "llm", "engine_version": "v1",
},
)
await client.patch("/api/settings/import", json={
"translation_enabled": True, "interpreter_base_url": "http://i.lan",
})
resp = await client.post(
"/api/settings/translation/probe", json={"text": "WIP"})
assert resp.status_code == 200
body = await resp.get_json()
assert body["detected_lang"] == "de"
assert body["detected_confidence"] == 71.5
assert body["translated"] == "Work in Progress"
assert body["target"] == "en"
def _fake_retranslate(monkeypatch, sink):
# Record the kwargs the endpoint enqueues with, return a fake AsyncResult.
monkeypatch.setattr(
"backend.app.tasks.translation.retranslate_posts.delay",
lambda *a, **k: sink.update(k) or type("R", (), {"id": "r1"})(),
)
@pytest.mark.asyncio
async def test_translation_retranslate_requires_config(client):
# Disabled → 400 even with an explicit scope (no reset is enqueued).
resp = await client.post(
"/api/settings/translation/retranslate", json={"all": True})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_translation_retranslate_requires_target(client, monkeypatch):
# 'all' must be explicit: an empty body is a 400, so a stray call can't
# wipe every translation.
_fake_retranslate(monkeypatch, {})
await client.patch("/api/settings/import", json={
"translation_enabled": True, "interpreter_base_url": "http://i.lan",
})
resp = await client.post("/api/settings/translation/retranslate")
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_translation_retranslate_artist_scope(client, monkeypatch):
sink = {}
_fake_retranslate(monkeypatch, sink)
await client.patch("/api/settings/import", json={
"translation_enabled": True, "interpreter_base_url": "http://i.lan",
})
resp = await client.post(
"/api/settings/translation/retranslate", json={"artist_id": 7})
assert resp.status_code == 202
assert (await resp.get_json())["celery_task_id"] == "r1"
assert sink["artist_ids"] == [7]
@pytest.mark.asyncio
async def test_translation_retranslate_all_scope(client, monkeypatch):
sink = {}
_fake_retranslate(monkeypatch, sink)
await client.patch("/api/settings/import", json={
"translation_enabled": True, "interpreter_base_url": "http://i.lan",
})
resp = await client.post(
"/api/settings/translation/retranslate", json={"all": True})
assert resp.status_code == 202
assert sink["artist_ids"] is None
@pytest.mark.asyncio
async def test_translation_retranslate_bad_artist_id(client):
resp = await client.post(
"/api/settings/translation/retranslate", json={"artist_id": "abc"})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_patch_rejects_non_object(client):
resp = await client.patch("/api/settings/import", json=[1, 2, 3])
+24
View File
@@ -98,6 +98,30 @@ async def test_similar_excludes_presentation_tagged_images(db):
assert {i.id for i in res_from_banner} == {src.id, wipped.id, plain.id}
@pytest.mark.asyncio
async def test_similar_exclude_wip_drops_wip_neighbors(db):
"""Explore passes exclude_wip=True to also hide work-in-progress from the
rabbit-hole (banner is always hidden; wip only when asked)."""
src = await _img(db, 1, _vec(1, 0))
bannered = await _img(db, 2, _vec(1, 0.02)) # always hidden
wipped = await _img(db, 3, _vec(1, 0.3)) # hidden only with exclude_wip
plain = await _img(db, 4, _vec(1, 0.6))
banner_tag = (await db.execute(select(Tag).where(
Tag.is_system.is_(True), Tag.name == "banner"))).scalar_one()
wip_tag = (await db.execute(select(Tag).where(
Tag.is_system.is_(True), Tag.name == "wip"))).scalar_one()
await db.execute(image_tag.insert().values(
image_record_id=bannered.id, tag_id=banner_tag.id, source="manual"))
await db.execute(image_tag.insert().values(
image_record_id=wipped.id, tag_id=wip_tag.id, source="manual"))
svc = GalleryService(db)
res = await svc.similar(src.id, limit=10, exclude_wip=True)
assert [i.id for i in res] == [plain.id] # wip + banner both gone
# Default (gallery "similar" button) still keeps wip (#1274).
res_default = await svc.similar(src.id, limit=10)
assert wipped.id in {i.id for i in res_default}
@pytest.mark.asyncio
async def test_similar_composes_with_tag_filter(db):
src = await _img(db, 1, _vec(1, 0))
+142
View File
@@ -0,0 +1,142 @@
"""Interpreter translation client (#143) — unit tests with mocked HTTP (no DB,
no real network)."""
import pytest
from backend.app.services import interpreter_client as ic
class _Resp:
def __init__(self, status_code=200, payload=None, headers=None):
self.status_code = status_code
self._payload = payload or {}
self.headers = headers or {}
def json(self):
return self._payload
def raise_for_status(self):
if self.status_code >= 400:
raise ic.requests.HTTPError(f"status {self.status_code}")
def test_translate_maps_batch_in_order(monkeypatch):
captured = {}
def fake_post(url, json, timeout):
captured["json"] = json
return _Resp(200, {
"translatedText": ["The cat is cute", "Blonde gal!"],
"detectedLanguage": {"language": "ja", "confidence": 0.98},
"interpreter": {"engine": "llm", "engine_version": "ollama:x:12b"},
})
monkeypatch.setattr(ic.session, "post", fake_post)
out = ic.translate(["ねこが可愛い", "金髪ギャル!"], base_url="http://i.lan")
assert out["translations"] == ["The cat is cute", "Blonde gal!"]
assert out["detected_lang"] == "ja"
assert out["detected_confidence"] == 0.98 # surfaced for the detection guard
assert out["engine_version"] == "ollama:x:12b"
assert captured["json"]["q"] == ["ねこが可愛い", "金髪ギャル!"]
assert captured["json"]["target"] == "en"
def test_translate_passthrough_unchanged(monkeypatch):
# Already-English items come back unchanged in their slot (engine "none").
monkeypatch.setattr(ic.session, "post", lambda *a, **k: _Resp(200, {
"translatedText": ["already english"],
"detectedLanguage": {"language": "en"},
"interpreter": {"engine": "none", "engine_version": None},
}))
out = ic.translate(["already english"], base_url="http://i.lan")
assert out["translations"] == ["already english"]
assert out["detected_lang"] == "en"
assert out["engine"] == "none"
def test_translate_503_raises_unavailable(monkeypatch):
monkeypatch.setattr(ic.session, "post", lambda *a, **k: _Resp(503, {}))
with pytest.raises(ic.InterpreterUnavailable):
ic.translate(["x"], base_url="http://i.lan")
def test_translate_connection_error_raises_unavailable(monkeypatch):
def boom(*a, **k):
raise ic.requests.ConnectionError("refused")
monkeypatch.setattr(ic.session, "post", boom)
with pytest.raises(ic.InterpreterUnavailable):
ic.translate(["x"], base_url="http://i.lan")
@pytest.mark.parametrize("code", [429, 500, 502, 503, 504])
def test_translate_429_and_5xx_raise_unavailable(monkeypatch, code):
# A gracefully-draining service behind a reverse proxy returns 429/502/503/504
# — all mean "retry later", not an opaque error.
monkeypatch.setattr(ic.session, "post", lambda *a, **k: _Resp(code, {}))
with pytest.raises(ic.InterpreterUnavailable) as ei:
ic.translate(["x"], base_url="http://i.lan")
assert ei.value.retry_after is None
def test_translate_honours_retry_after_seconds(monkeypatch):
monkeypatch.setattr(
ic.session, "post",
lambda *a, **k: _Resp(503, {}, headers={"Retry-After": "42"}),
)
with pytest.raises(ic.InterpreterUnavailable) as ei:
ic.translate(["x"], base_url="http://i.lan")
assert ei.value.retry_after == 42
def test_translate_retry_after_past_http_date_clamps_to_zero(monkeypatch):
# HTTP-date form; a past date → non-negative clamp to 0 (deterministic).
monkeypatch.setattr(
ic.session, "post",
lambda *a, **k: _Resp(
503, {}, headers={"Retry-After": "Wed, 21 Oct 2015 07:28:00 GMT"}),
)
with pytest.raises(ic.InterpreterUnavailable) as ei:
ic.translate(["x"], base_url="http://i.lan")
assert ei.value.retry_after == 0.0
def test_translate_400_raises_bad_request(monkeypatch):
monkeypatch.setattr(
ic.session, "post", lambda *a, **k: _Resp(400, {"error": "bad target"})
)
with pytest.raises(ic.InterpreterBadRequest):
ic.translate(["x"], base_url="http://i.lan")
def test_translate_empty_is_noop(monkeypatch):
def boom(*a, **k):
raise AssertionError("should not call the service for an empty batch")
monkeypatch.setattr(ic.session, "post", boom)
assert ic.translate([], base_url="http://i.lan")["translations"] == []
def test_health_true_when_llm_up(monkeypatch):
monkeypatch.setattr(
ic.session, "get", lambda *a, **k: _Resp(200, {"engines": {"llm": True}})
)
assert ic.health("http://i.lan") is True
def test_health_false_when_engine_down(monkeypatch):
monkeypatch.setattr(
ic.session, "get", lambda *a, **k: _Resp(200, {"engines": {"llm": False}})
)
assert ic.health("http://i.lan") is False
def test_health_false_on_error(monkeypatch):
def boom(*a, **k):
raise ic.requests.ConnectionError("refused")
monkeypatch.setattr(ic.session, "get", boom)
assert ic.health("http://i.lan") is False
def test_health_false_when_url_empty():
assert ic.health("") is False
+22
View File
@@ -26,6 +26,28 @@ def _make_batch(session) -> int:
return batch.id
def test_cleanup_orphaned_temp_files_removes_stale_only(tmp_path, monkeypatch):
import os
from backend.app.tasks import maintenance as m
monkeypatch.setattr(m, "IMAGES_ROOT", tmp_path)
stale = tmp_path / "artist" / "img.jpg.part" # killed download → orphan
stale.parent.mkdir(parents=True)
stale.write_bytes(b"x")
old = datetime.now(UTC).timestamp() - 8 * 3600 # older than the 6h guard
os.utime(stale, (old, old))
fresh = tmp_path / "in_progress.jpg.partial" # active download → keep
fresh.write_bytes(b"x")
keep = tmp_path / "real.jpg" # a real image → keep
keep.write_bytes(b"x")
assert m.cleanup_orphaned_temp_files() == 1
assert not stale.exists()
assert fresh.exists()
assert keep.exists()
def test_recover_interrupted_only_old(db_sync, monkeypatch):
batch_id = _make_batch(db_sync)
now = datetime.now(UTC)
+24
View File
@@ -283,6 +283,30 @@ async def test_scroll_item_shape_minimal(db):
assert "description_full" not in item
@pytest.mark.asyncio
async def test_scroll_surfaces_translation_fields(db):
# #143: a translated post exposes the translated title/description + source
# lang, with the originals still present for the toggle.
artist = await _seed_artist(db, "alice-tr")
src = await _seed_source(db, artist.id, "pixiv", "https://p/alice-tr")
p = await _seed_post(
db, src.id, external_id="TR", post_date=datetime.now(UTC),
title="ねこ", description="<p>かわいい</p>",
)
p.post_title_translated = "cat"
p.description_translated = "cute"
p.translated_source_lang = "ja"
await db.commit()
page = await PostFeedService(db).scroll(cursor=None, limit=10)
item = page["items"][0]
assert item["post_title_translated"] == "cat"
assert item["description_translated"] == "cute"
assert item["translated_source_lang"] == "ja"
assert item["post_title"] == "ねこ" # original kept for the toggle
assert item["description_plain"] == "かわいい"
@pytest.mark.asyncio
async def test_scroll_description_truncates_long_text(db):
artist = await _seed_artist(db, "alice-long")
+415
View File
@@ -0,0 +1,415 @@
"""translate_posts sweep (#143) — integration, with a stubbed Interpreter client
and the task's session factory pointed at the test's db_sync."""
import pytest
from sqlalchemy import select
from backend.app.models import Artist, ImportSettings, Post
from backend.app.services import interpreter_client as ic
from backend.app.tasks.translation import (
_RETRANSLATE_COUNTDOWN,
retranslate_posts,
translate_posts,
)
pytestmark = pytest.mark.integration
class _Ctx:
def __init__(self, s):
self.s = s
def __enter__(self):
return self.s
def __exit__(self, *a):
return False
def _sf(db_sync):
class _SM:
def __call__(self):
return _Ctx(db_sync)
return _SM()
def _artist(db, name="A", slug="a"):
a = Artist(name=name, slug=slug)
db.add(a)
db.flush()
return a
def _post(db, artist_id, *, title=None, desc=None, ext="p1"):
p = Post(
artist_id=artist_id, external_post_id=ext,
post_title=title, description=desc,
)
db.add(p)
db.flush()
return p
def _mark_translated(p, *, lang="ja", ver="v1"):
"""Simulate a prior (old-model) translation so retranslate has something to
reset."""
p.post_title_translated = "OLD"
p.description_translated = "OLD"
p.translated_source_lang = lang
p.translation_engine_version = ver
def _enable(db, url="http://i.lan"):
cfg = db.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
cfg.translation_enabled = True
cfg.interpreter_base_url = url
db.flush()
def _patch(monkeypatch, db_sync):
monkeypatch.setattr(
"backend.app.tasks.translation._sync_session_factory",
lambda: _sf(db_sync),
)
def test_translate_posts_stores_translation(db_sync, monkeypatch):
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
monkeypatch.setattr(
"backend.app.tasks.translation.ic.translate",
lambda texts, **k: {
"translations": [f"EN:{t}" for t in texts],
"detected_lang": "ja", "engine": "llm", "engine_version": "v1",
},
)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", desc="かわいい")
_enable(db_sync)
db_sync.commit()
assert "translated=1" in translate_posts()
db_sync.refresh(p)
assert p.post_title_translated == "EN:ねこ"
assert p.description_translated == "EN:かわいい"
assert p.translated_source_lang == "ja"
assert p.translation_engine_version == "v1"
def test_translate_posts_passthrough_english_marks_handled(db_sync, monkeypatch):
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
monkeypatch.setattr(
"backend.app.tasks.translation.ic.translate",
lambda texts, **k: {
"translations": list(texts), "detected_lang": "en",
"engine": "none", "engine_version": None,
},
)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="hello world", ext="p2")
_enable(db_sync)
db_sync.commit()
translate_posts()
db_sync.refresh(p)
assert p.translated_source_lang == "en" # marked handled...
assert p.post_title_translated is None # ...but nothing to show
def test_translate_posts_mixed_language_translates_nonenglish_field(db_sync, monkeypatch):
# English title + Japanese description: the description is still translated
# even though the title is already English (per-field detection, not the old
# aggregate first-item bail). Source lang comes from the translated field.
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
def fake_translate(texts, **k):
t = texts[0]
if t.isascii(): # already English → passthrough
return {"translations": [t], "detected_lang": "en",
"engine": "none", "engine_version": None}
return {"translations": [f"EN:{t}"], "detected_lang": "ja",
"engine": "llm", "engine_version": "v9"}
monkeypatch.setattr("backend.app.tasks.translation.ic.translate", fake_translate)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="hello", desc="ねこ")
_enable(db_sync)
db_sync.commit()
assert "translated=1" in translate_posts()
db_sync.refresh(p)
assert p.post_title_translated is None # English title untouched
assert p.description_translated == "EN:ねこ" # Japanese desc translated
assert p.translated_source_lang == "ja" # from the translated field
assert p.translation_engine_version == "v9"
def test_translate_posts_disabled_is_noop(db_sync, monkeypatch):
_patch(monkeypatch, db_sync)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p3")
db_sync.commit() # translation_enabled defaults False
assert translate_posts() == "disabled"
db_sync.refresh(p)
assert p.translated_source_lang is None
def test_translate_posts_service_down_leaves_untranslated(db_sync, monkeypatch):
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: False)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p4")
_enable(db_sync)
db_sync.commit()
assert translate_posts() == "interpreter unavailable"
db_sync.refresh(p)
assert p.translated_source_lang is None # will retry next run
def test_translate_posts_interrupt_reenqueues_after_backoff(db_sync, monkeypatch):
# A drain mid-sweep (health passed, translate 503s w/ Retry-After) re-enqueues
# the daily sweep after the backoff instead of waiting for tomorrow's beat.
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
def _drain(texts, **k):
raise ic.InterpreterUnavailable("draining", retry_after=30)
monkeypatch.setattr("backend.app.tasks.translation.ic.translate", _drain)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.translate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
_post(db_sync, a.id, title="ねこ", ext="p1")
_enable(db_sync)
db_sync.commit()
assert "interrupted" in translate_posts()
assert len(calls) == 1
assert calls[0][1]["countdown"] == 30
def _mock_ok(monkeypatch):
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
monkeypatch.setattr(
"backend.app.tasks.translation.ic.translate",
lambda texts, **k: {
"translations": [f"EN:{t}" for t in texts],
"detected_lang": "ja", "engine": "llm", "engine_version": "v1",
},
)
def test_translate_posts_drain_chases_tail(db_sync, monkeypatch):
# "Translate now" (drain=True): with the chunk capped at 1 and 2 untranslated
# posts, the first chunk re-enqueues itself — drain preserved — to finish the
# backlog rather than waiting for the next beat.
_patch(monkeypatch, db_sync)
_mock_ok(monkeypatch)
monkeypatch.setattr("backend.app.tasks.translation._MAX_POSTS_PER_RUN", 1)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.translate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
_post(db_sync, a.id, title="ねこ", ext="p1")
_post(db_sync, a.id, title="いぬ", ext="p2")
_enable(db_sync)
db_sync.commit()
translate_posts(drain=True)
assert len(calls) == 1 # one re-enqueue for the tail
args, kw = calls[0]
assert args[1] == {"drain": True} # drain flag carried forward
assert kw["countdown"] == _RETRANSLATE_COUNTDOWN
def test_translate_posts_beat_single_chunk_no_tail_chase(db_sync, monkeypatch):
# The periodic beat (drain defaults False) does exactly ONE chunk and never
# re-enqueues — the 8h cadence drives the rest.
_patch(monkeypatch, db_sync)
_mock_ok(monkeypatch)
monkeypatch.setattr("backend.app.tasks.translation._MAX_POSTS_PER_RUN", 1)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.translate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
_post(db_sync, a.id, title="ねこ", ext="p1")
_post(db_sync, a.id, title="いぬ", ext="p2")
_enable(db_sync)
db_sync.commit()
translate_posts() # drain=False
assert calls == [] # no self-re-enqueue
def _mock_new_model(monkeypatch, *, ver="v2"):
"""Interpreter is up and translates with a NEW engine version."""
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
monkeypatch.setattr(
"backend.app.tasks.translation.ic.translate",
lambda texts, **k: {
"translations": [f"NEW:{t}" for t in texts],
"detected_lang": "ja", "engine": "llm", "engine_version": ver,
},
)
def test_retranslate_resets_and_reruns(db_sync, monkeypatch):
# A post translated by the old model gets cleared + re-run through the new
# one (new engine_version proves the reset happened, not a cache short-circuit).
_patch(monkeypatch, db_sync)
_mock_new_model(monkeypatch)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", desc="かわいい")
_mark_translated(p)
_enable(db_sync)
db_sync.commit()
retranslate_posts(artist_ids=[a.id])
db_sync.refresh(p)
assert p.post_title_translated == "NEW:ねこ"
assert p.description_translated == "NEW:かわいい"
assert p.translation_engine_version == "v2"
def test_retranslate_artist_scope_leaves_others(db_sync, monkeypatch):
# Scoping to one artist must not touch another artist's translations.
_patch(monkeypatch, db_sync)
_mock_new_model(monkeypatch)
a1 = _artist(db_sync)
a2 = _artist(db_sync, name="B", slug="b")
p1 = _post(db_sync, a1.id, title="ねこ", ext="p1")
p2 = _post(db_sync, a2.id, title="いぬ", ext="p2")
_mark_translated(p1)
_mark_translated(p2)
_enable(db_sync)
db_sync.commit()
retranslate_posts(artist_ids=[a1.id])
db_sync.refresh(p1)
db_sync.refresh(p2)
assert p1.post_title_translated == "NEW:ねこ" # re-run
assert p1.translation_engine_version == "v2"
assert p2.post_title_translated == "OLD" # untouched
assert p2.translation_engine_version == "v1"
def test_retranslate_runs_until_done(db_sync, monkeypatch):
# With the chunk capped at 1 and 2 posts to redo, the first chunk re-enqueues
# itself (with _reset_done=True so it doesn't wipe the fresh work) to finish.
_patch(monkeypatch, db_sync)
_mock_new_model(monkeypatch)
monkeypatch.setattr("backend.app.tasks.translation._MAX_POSTS_PER_RUN", 1)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.retranslate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
p1 = _post(db_sync, a.id, title="ねこ", ext="p1")
p2 = _post(db_sync, a.id, title="いぬ", ext="p2")
_mark_translated(p1)
_mark_translated(p2)
_enable(db_sync)
db_sync.commit()
retranslate_posts(artist_ids=[a.id])
assert len(calls) == 1 # one re-enqueue for the tail
args, _kw = calls[0]
assert args[1]["_reset_done"] is True
assert args[1]["artist_ids"] == [a.id]
def test_retranslate_disabled_does_not_reset(db_sync, monkeypatch):
# Never wipe translations we can't rebuild — a disabled service leaves them.
_patch(monkeypatch, db_sync)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p1")
_mark_translated(p)
db_sync.commit() # translation_enabled defaults False
assert retranslate_posts(artist_ids=[a.id]) == "disabled"
db_sync.refresh(p)
assert p.translated_source_lang == "ja"
assert p.post_title_translated == "OLD"
def test_retranslate_unhealthy_does_not_reset(db_sync, monkeypatch):
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: False)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p1")
_mark_translated(p)
_enable(db_sync)
db_sync.commit()
assert retranslate_posts(artist_ids=[a.id]) == "interpreter unavailable"
db_sync.refresh(p)
assert p.translated_source_lang == "ja"
assert p.post_title_translated == "OLD"
def test_retranslate_interrupt_resumes_after_retry_after(db_sync, monkeypatch):
# Interpreter drains mid-chunk (health passed, then translate 503s with a
# Retry-After): the bulk re-translate re-enqueues itself after the hinted
# backoff, with _reset_done=True so it never re-wipes fresh work — instead of
# stalling until the daily beat.
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
def _drain(texts, **k):
raise ic.InterpreterUnavailable("draining", retry_after=42)
monkeypatch.setattr("backend.app.tasks.translation.ic.translate", _drain)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.retranslate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p1")
_mark_translated(p)
_enable(db_sync)
db_sync.commit()
assert "interrupted" in retranslate_posts(artist_ids=[a.id])
assert len(calls) == 1 # resume re-enqueued
args, kw = calls[0]
assert args[1]["_reset_done"] is True
assert args[1]["artist_ids"] == [a.id]
assert kw["countdown"] == 42 # honoured the Retry-After
db_sync.refresh(p)
assert p.translated_source_lang is None # reset happened, not re-run
def test_retranslate_interrupt_default_backoff_without_hint(db_sync, monkeypatch):
# No Retry-After header → fall back to the default backoff (60s).
_patch(monkeypatch, db_sync)
monkeypatch.setattr("backend.app.tasks.translation.ic.health", lambda *a, **k: True)
def _drain(texts, **k):
raise ic.InterpreterUnavailable("draining")
monkeypatch.setattr("backend.app.tasks.translation.ic.translate", _drain)
calls = []
monkeypatch.setattr(
"backend.app.tasks.translation.retranslate_posts.apply_async",
lambda *a, **k: calls.append((a, k)),
)
a = _artist(db_sync)
p = _post(db_sync, a.id, title="ねこ", ext="p1")
_mark_translated(p)
_enable(db_sync)
db_sync.commit()
retranslate_posts(artist_ids=[a.id])
assert len(calls) == 1
assert calls[0][1]["countdown"] == 60