Python 3.14 evaluates annotations lazily, so the self-referential return
type needs no forward-ref quotes — matches ImportSettings.load. Fixes the
ruff lint lane on the DRY-pass push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
_lookup_via_api and resolve_display_name shared ~90% of their body (same
endpoint, params, headers, error handling — differing only in which field
they pluck from data[0]). Extract _campaigns_api_first(vanity, cookies_path)
-> dict|None; callers pluck the campaign id vs the display name. Return-value
behavior preserved (the display-name path additionally gains the helper's more
granular warning logs). Covered by the existing test_patreon_resolver.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
recover_stalled_head_training_runs and recover_stalled_head_auto_apply_runs
were near-exact copies (coalesce-flip + keep-last-N prune, differing only in
model + two constants). Extract _recover_stalled_runs(model, stall_minutes,
keep_runs, label); the two tasks become thin wrappers. The other two recover
tasks are deliberately NOT folded in (library-audit has no prune tail; backup
uses a single started_at cutoff). test_recover_stalled_head_runs.py covers
both wrappers (stalled→error, fresh survives) — previously untested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
Consolidate duplication accrued across the ML tagging + settings backend,
behavior-preserving (over-DRY guard applied — the three auto-apply sweep
BODIES stay separate; only their shared inner helpers are extracted).
- _sigmoid / _conflict_scores / _insert_presentation_review (heads.py): the
score→prob transform (6 inlined sites), the presentation conflict signal
(2 sites), and the ring-loud PresentationReview insert (2 sites, single-
sourced so the mode column can't drift on the shared composite PK).
- _applied_or_rejected (training_data.py): the per-tag "applied ∪ rejected"
skip-set, byte-identical at 3 sweep sites (heads.py x2, tasks/ml.py ccip).
- ccip sweep divergence fixes: import ccip._FIGURE_KINDS + training_data._l2norm
instead of local copies that silently drift when the canonical changes.
- MLSettings.load / .load_sync classmethods (mirror ImportSettings); route all
8 scalar_one singleton reads through them (the session.get None-path stays).
- GET serializers for MLSettings + ImportSettings are now table-driven off the
same _EDITABLE tuples PATCH writes, so a new field can't be silently absent
from GET (the split that historically dropped fields).
- AUTO_APPLY_THRESHOLD_MIN/MAX constant single-sources the [0.5,0.999] operating
range across the service clamp + the 5 API validators.
- test_ml_dry_helpers.py pins _applied_or_rejected + _sigmoid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
Symptom (operator-flagged): the extension injected the Add-as-source button on
a Patreon creator you had NOT subscribed to, but it disappeared once you were
subscribed — the opposite of when it's useful.
Root cause (extension logic, not Patreon security): Patreon serves a creator
under three URL shapes — bare patreon.com/Atole, patreon.com/c/Atole, and
patreon.com/cw/Atole (the 'creator workspace' URL you land on once subscribed;
documented in patreon_resolver._VANITY_RE). The button's artist-page gate
(PLATFORM_ARTIST_PATTERNS.patreon in platforms.js) and its byte-mirror probe
pattern (_PLATFORM_PATTERNS in extension_service._derive) only matched the bare
single-segment form and explicitly excluded c/. So the subscribed-view URL
failed the gate → no button. The ingestion resolver already handled all three;
only these two gates were too narrow.
Fix: both regexes now accept optional cw/ and c/ prefixes and drop the strict
single-segment end-anchor, so a creator's inner page (/cw/Atole/posts,
/Atole/membership) also matches — robust to whatever exact shape the subscribed
view uses. Nav-page exclusions (home/search/messages/notifications/library/
settings/posts + post permalinks) preserved. New unit test covers all three
prefixes, sub-paths, and nav-page rejection (both regexes validated identically).
Bump extension 1.0.7→1.0.8 so a fresh signed XPI ships the fix (also exercises
batch-5 web-ext-10's AMO sign path end-to-end on the main build).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Explore walk got stuck in dense signatures — neighbours all too similar, so
forward-arrow couldn't escape and Random was the only exit. Root cause: MMR only
diversifies WITHIN the nearest ~400 pool; in a dense cluster that whole pool is
near-identical, so there's no escape route in it.
- gallery_service.similar(reach=0.0, exclude_ids=None): reach>0 widens the pool
(cap 400→1000) and _reach_sample strides across an outward-growing distance span
so the set handed to MMR spans near→mid-far (guaranteed escape routes), not just
the tight cluster. exclude_ids drops already-walked images. Gallery 'more like
this' (reach=0) is unchanged.
- api/gallery similar: parse reach + exclude_ids.
- explore store: default reach 0.4 (auto-diversifies without touching the dial),
pass the breadcrumb as exclude_ids, setReach action.
- ExploreView: a Near↔Far reach slider in the trail.
- tests: _reach_sample math (deeper ranks with higher reach, near kept); similar
exclude_ids drops walked + reach path runs clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely.
- wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it);
WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg.
- training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and
never trains the wip head (a finished "sketch" can't pollute it). Only the hard
tier (wip_title) + manual train.
- ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087.
- importer: hard tier wins, soft is the fallback (source wip_title_soft).
- backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled.
- heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content
heads, flag ring-loud ones (PresentationReview mode=process) for the review strip —
the operator's "measure if they got falsely tagged" safety.
- api settings toggle; ImportFiltersForm soft toggle.
- tests: soft matcher pos/neg; soft source not a training positive; audit flags
ring-loud + spares quiet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend for the system-tag behavior refactor (milestone #157). editor screenshot
moves from chrome (hidden) to the PROCESS group (shown, like wip); wip+editor gain
provisional auto-apply so they stop needing endless manual identification —
without a runaway loop.
- tag.py: split PRESENTATION_SYSTEM_TAGS → CHROME_SYSTEM_TAGS (banner) +
PROCESS_SYSTEM_TAGS (wip, editor screenshot).
- heads.py: generalize presentation_auto_apply_sweep → system_tag_auto_apply_sweep
(mode chrome|process). Same Guard 1 (skip human/confirmed) + Guard 2 (ring-loud
conflict → PresentationReview). process mode uses source 'process_auto' and does
NOT hide (hide is a gallery-query effect of group membership).
- training_data._AUTO_SOURCES += 'process_auto' → the head never trains on its own
auto-applied output; only wip_title/manual train it (the runaway break).
- ml_settings: process_auto_apply_enabled (OFF, opt-in) + threshold + conflict
threshold. presentation_review.mode ('chrome'|'process'). Migration 0086.
- gallery_service: default-hide reads CHROME only (editor now shows); Explore
neighbors exclude the whole PROCESS group.
- tasks/ml + celery beat: scheduled_process_auto_apply (daily, opt-in); prune
covers both modes.
- api: ml_admin process_* CRUD+validation; hidden-review returns mode.
- tests: rename chrome sweep calls; new test_process_auto_apply (apply, guards,
mode flag, no-self-train); gallery test asserts editor now visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
apply_wip_image_tags relied on result.rowcount, but psycopg reports -1 for a
multi-row INSERT ... ON CONFLICT DO NOTHING (executemany path), so the return
count (and the backfill's reported total) was wrong. Compute the count from a
pre-SELECT of already-tagged ids within the same transaction; keep ON CONFLICT
DO NOTHING as a race-safety belt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ruff I001 fixup for 5719387 — the new .wip_title import belongs after
.thumbnailer (alphabetical), not after .archive_extractor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Auto-apply the `wip` system tag to posts whose TITLE explicitly declares
work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision
complement to the image-based ML `wip` head. WIP images are excluded from the
Explore/gallery browse, so honouring the artist's own label keeps unfinished
pieces out of the main browse.
- services/wip_title.py: precision-first token-anchored matcher (swipe/wiped
never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO
NOTHING, chunked under the psycopg param ceiling).
- importer: live hook on FRESH import only (never on deep-scan/supersede), so a
manually-removed WIP tag is never re-applied by a routine re-scan.
- maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep
(coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a
beat — a periodic re-run would silently undo manual removals.
- ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating
the live hook; GET/PATCH + POST /settings/wip-title/scan.
- Settings UI: toggle + "Scan existing posts" button.
- Tests: pure matcher unit tests + integration (apply idempotency, backfill
precision).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gate at a fixed 0.80 couldn't catch the real pain: Interpreter (fresh ==
cached, verified by probe) confidently mis-detects short ASCII English like
"... WIP Part 1" as German at 0.86 — above the floor — so it was accepted and a
re-translate reproduced it. Confidence alone can't separate the 0.86 collision
(genuine German lands there too), and single-word mis-flags sit at a confident
1.0 no floor catches.
Two operator-approved levers:
- Acceptance floor is now a live Settings value (ImportSettings.
translation_min_confidence, default 0.90; surfaced in the Translation card), so
it's tunable without a redeploy. _accept takes the threshold as a parameter.
- Per-post sticky override (Post.translation_override: auto/force/original).
'force' stores a translation even below the floor (rescue a skipped
legit-foreign title); 'original' keeps the original and clears any stored
translation (kill a confident mis-flag no floor catches). The sweep honors it
on every run and _reset_translations skips 'original', so the choice survives a
Re-translate-all. POST /api/posts/<id>/translation-override applies it
immediately (translate now when the service is up, else queue for the sweep).
UI: PostTranslationControl on the posts-feed card.
Migration 0084 (both columns + a CHECK on the override). The feed + provenance
serializers expose translation_override.
With a stricter floor the rollback finally works: raise it -> Re-translate all ->
the 0.86 mis-flags are rejected and restored to the original; force /
keep-original handle the residual either way.
Tests: gate thresholds against the param (0.86 rejected at 0.90, explicit-floor
cases); sweep force/original + re-translate-skips-original; override endpoint
(validation, original clears, force queues when disabled, feed exposes it);
settings min_confidence default/save/validate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
Every tag suggestion is a canonical DB tag now (tagging-v2 #114: heads + CCIP
score EXISTING concept tags). The pre-heads apparatus for model-predicted tags
that didn't exist in the DB — creates_new_tag / raw_name / via_alias, the
/suggestions/alias endpoint + add_alias_and_accept, AliasPickerDialog, and the
store's aliasAccept/removeAlias — was dead and is removed.
The type-to-add dropdown was TWO row sources (server autocomplete + the image's
ML suggestions) merged with a dedup that dropped the %-bearing suggestion row
when the debounced server hit landed — the operator's "confidence % flickers
then vanishes". Now it's ONE list of DB-tag matches, each annotated with the
model's confidence (join by canonical_tag_id) when the tag was scored for this
image. No dedup, no flicker; picking a suggested tag still records acceptance
via TagPanel.findPending.
Single per-image fetch: score_image now reports above_threshold per row
(computed vs the head's own suggest cut, separate from the inclusion floor), so
the rail makes ONE min=0 request and derives the panel (above_threshold) and the
dropdown (all, text-filtered) client-side — the two /suggestions calls collapse
to one. Manual "Create 'X' as <kind>" (novel typed names) is unchanged; the
alias table + tag-side alias admin + auto-apply alias matching are untouched.
Tests: gate/serializer assertions updated (above_threshold; dropped dead-field
+ alias-endpoint checks); frontend spec seeds via the single load and covers the
byCategory/aboveByCategory split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
Calibrated against fresh probes once Interpreter returned real langdetect
confidence: genuine German detected at 1.0, a correctly-detected but ambiguous
latin string at 0.86. Set _MIN_LATIN_CONFIDENCE to 0.80 (below that band) so
legitimate ambiguous non-English still translates while genuinely-unsure guesses
are rejected. Real langdetect also fixed the original mis-flag at the source, so
this floor is a safety net, not the primary fix. Pin 0.86-accepted in the gate
test to guard against bumping the floor back up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
Interpreter now returns a real per-detection confidence (source stays "auto"),
so curator can reject the mis-detections it was blindly storing — e.g. a short
English title mis-labelled as German and rewritten into the archive.
The gate consumes ONLY Interpreter's own reported detection — curator does no
language detection of its own (Scribe rule 133): a field is stored when the
engine actually translated it AND either the detected language is CJK
(script-detected, reliably high — ja/ko/zh trusted outright, incl. pure-kanji
Japanese that lands as zh ~0.75) or the reported confidence clears a
latin-script floor (_MIN_LATIN_CONFIDENCE = 0.90). A latin detection below the
floor keeps the original and marks the post handled; a missing confidence fails
open. The client already sent source="auto" and parsed confidence, so this is
purely the gate + tests.
Tests: pinned interpreter-client test now asserts source stays "auto"; new
pure-unit gate tests (CJK trusted / latin floor / case-insensitive / fail-open)
in the fast lane; end-to-end reject-low-latin, accept-high-latin,
accept-low-cjk sweeps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
Throughput: translate_posts now runs every 8h (was daily) as the
steady-state cadence for newly-imported posts, and the Settings
"Translate now" button runs it in drain mode (run-until-done, no reset)
so one press clears the whole untranslated backlog instead of a single
300-post chunk. The interrupt/backoff re-enqueue now preserves the drain
flag so a bulk drain resumes cleanly after an Interpreter restart.
Misdetection groundwork: surface the detector's confidence from the
Interpreter client (it was in the detectedLanguage payload but discarded)
and add a read-only "Test translation" box — POST /settings/translation/
probe + TranslationCard UI — that shows detected language + confidence +
engine + result for pasted text, without saving. Lets the operator see
why a short/abbreviation-heavy English title gets mis-detected so the
detection guard (min-length + confidence floor) can be tuned from real
numbers. The guard itself follows once the mis-detected cases are probed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
_translate_one translated [title, description] in ONE Interpreter call and keyed
the whole-post passthrough on the aggregate detected_lang (the FIRST item). So an
English title + non-English description detected "en" and marked the post handled,
leaving the description untranslated. Now each field is translated independently
(its own detected_lang / passthrough) and the non-target field is stored on its
own; translated_source_lang reflects the translated field's language.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
translation/status now reports `active` (a translate/retranslate sweep is
running, from the TaskRun table) and `last_run` (the most recent finished run's
task + status). The Settings card polls live while a sweep runs, showing a
spinner + "Translating… N remaining" that ticks down, and flags a last run that
ended in error/timeout. No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Downloads/imports stage into <name>.part / <name>.partial then os.replace() into
place, so a kill mid-write leaves a discardable temp — never a corrupt final.
cleanup_orphaned_temp_files sweeps ones left behind under the images root, only
older than 6h so an in-flight download's staging file is never removed. Daily beat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- interpreter_client: shared requests.Session with a connect-only retry
(connect=2, no status retries — we map 429/5xx ourselves) so a proxy reload
is smoothed and the keep-alive connection is pooled across the sweep.
- translate_posts: on an interrupt (drain), re-enqueue after the Retry-After
hint / default backoff instead of waiting for the daily beat; self-terminates
via the health gate. Steady-state one-chunk-per-run on success is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Explore's neighbour grid (/api/gallery/similar → gallery_service.similar) now
takes an Explore-only exclude_wip flag that drops `wip` system-tagged images
from the candidates, alongside the banner/editor presentation tags. The
gallery's own "similar" button is unchanged (keeps wip, #1274) — only the
Explore store passes exclude_wip=1. The anchor itself may still be a WIP; only
neighbours are filtered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs
mid-flight. Give in-flight work room to drain and make interrupted work
resume cleanly instead of stalling.
- docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s /
scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown
can actually drain before SIGKILL.
- celery_app.py: task_reject_on_worker_lost=True — a task killed past the
grace window is re-queued (safe: idempotent + chunked, recovery sweeps
re-drive stragglers).
- interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable
and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter
behind a reverse proxy no longer raises an opaque HTTPError.
- translation.py: thread retry_after out of _translate_batch; retranslate_posts
resumes after the Retry-After hint (or 60s default, capped 900s) on an
interrupt with _reset_done=True, self-terminating via the health gate.
- tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff.
No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
retranslate_posts resets the 5 translation columns to NULL for a scoped set of posts (all, or WHERE artist_id IN ids) then reuses the untranslated sweep to re-run them, chasing the tail until drained (run-until-done). Interpreter cache keys on engine_version so a changed model re-translates, an unchanged one is cache-fast. Reset only happens when the service is configured+healthy so translations are never wiped when they can't be rebuilt. New POST /settings/translation/retranslate (artist_id | all=true). UI: per-artist 'Re-translate posts' on the Artist Management tab + 'Re-translate all' in the Settings Translation card, both with confirm dialogs. No migration (reuses m143 columns).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New POST /api/settings/translation/test pings /v1/health for a GIVEN base URL (not
the saved one), so the operator can verify a URL before enabling it. TranslationCard
gains a Test-connection button that reports reachable/unreachable inline and
updates the status dot. Endpoint test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
post_feed_service (card + detail) and provenance_service._post_dict now include
post_title_translated, description_translated (card-truncated / detail-uncapped)
and translated_source_lang, keeping the originals for the toggle. Feed
serialization test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
tasks/translation.py — translate_posts: picks untranslated posts (title OR
description non-empty), per-post [title, description] batch via the Interpreter
client, stores translations + detected lang + engine_version; passthrough /
already-target posts are marked handled with no stored translation. 503 or a
connection error interrupts (retry next cycle), 400 stops (fix config), per-post
commit keeps progress; wall-clock bounded. Wired into celery (maintenance_long
lane) + a daily beat. No-op unless enabled + base URL set + healthy. GET
/settings/translation/status + POST .../run for the Settings card. Task tests
(stubbed client, monkeypatched session).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
services/interpreter_client.py: sync (requests) client for the LibreTranslate-
compatible /v1/translate — no new dep, mirrors the platform clients. translate()
maps translatedText[]↔texts (order + length), returns detected_lang +
engine_version (aggregate = first item, fine for a per-post [title, description]
batch); passthrough items come back unchanged in their slot. InterpreterUnavailable
on 503 / connection error (retry later), InterpreterBadRequest on 400. health()
checks /v1/health engines.llm. 10 unit tests with mocked HTTP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Post gains post_title_translated / description_translated / translated_source_lang
/ translation_engine_version / translated_at — filled by the translate sweep so
viewing is instant. ImportSettings gains translation_enabled (OFF by default),
interpreter_base_url (EMPTY — no default host; the operator points it at their own
Interpreter proxy behind a reverse proxy) and translation_target_lang (en),
exposed + validated via /settings/import. Migration 0083. Settings defaults +
patch + validation test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
scheduled_presentation_auto_apply (daily beat) runs presentation_auto_apply_sweep
— idempotent, so an interrupted run just re-runs next cycle (that's the recovery),
wall-clock bounded by soft/hard task time limits. prune_presentation_reviews
(daily beat) drops RESOLVED review flags older than 30 days (rule 89 retention).
Tests run both tasks via a monkeypatched session factory. Milestone 141 complete:
the presentation-chrome auto-hide + conflict-flagged review is now live end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GET /api/gallery/hidden-review lists unresolved presentation auto-hide flags
(image + presentation tag + conflict tag/score), most-concerning first. POST
.../keep resolves the flag (the tag stays). POST .../unhide removes the
presentation tag (image returns to the gallery), records a TagSuggestionRejection
so the head learns it misfired, and resolves the flag. Tests for list/keep/unhide.
Frontend review strip (shown when Show-hidden is on) next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
presentation_auto_apply_sweep fires banner/editor-screenshot heads at the FLAT
presentation threshold (source=presentation_auto). Two guards: (1) hard-skip any
image already carrying a human/confirmed content tag — you valued it, so the model
can't bury it; (2) if an auto-hide ALSO scores >= presentation_conflict_threshold
on a content head, hide it but record a PresentationReview row (conflict tag +
score) for the Hidden view.
_auto_apply_heads now excludes system tags, so a graduated wip/banner can't fire
via the content path (and wip never auto-applies at all). presentation_auto added
to _AUTO_SOURCES so auto-hidden chrome never self-trains. Tests: applies,
hard-skip valued, conflict-flag, disabled no-op, ignores wip, content-path
excludes system. Settings UI + scheduling land next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) +
presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a
FLAT threshold (decoupled from content-head graduation), plus the "also looks
like content" conflict cut. New presentation_review table (image, presentation
tag, conflict tag + score, created/resolved_at) records auto-hides flagged for
review. Migration 0082 (columns + table), ml_admin API (editable + get_settings
+ _validate bounds), settings roundtrip/bounds test. The sweep that reads these
knobs + the Settings UI land in step 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The /similar route splats **filters into similar(), which doesn't take the new
include_hidden kwarg → TypeError → 500 (test_gallery_similar). Drop it like
post_id; similar() has its own presentation exclusion (#1274), so the
gallery-browse flag doesn't apply there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The default gallery + facets now implicitly exclude images carrying a
presentation system tag (banner / editor screenshot), reusing the tag-scope
EXISTS machinery. Suppressed when the operator explicitly filters FOR a
presentation tag OR passes include_hidden (the Hidden view — step 2). `wip` is
NOT hidden (real, in-progress art). include_hidden threaded through
scroll/timeline/jump_cursor/facets + the gallery API _parse_filters. Test covers
default-hide, include_hidden, explicit-filter-shows, and wip-stays-visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- heads.py: conf_map = dict(conf) instead of a dict comprehension (ruff C416).
- postCard.spec.js: the modal-playlist rename (postImageIds→playlistIds) missed
this frontend test (grep was src-only); update the expected call args.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Completes "no self-training": unconfirmed auto-applied character tags no longer
seed CCIP references — character_references + the prototype builder
(_current_fingerprints/_rebuild_one) gain a shared _positive_char_tag filter
(human-applied OR operator-confirmed), mirroring the head-positive exclusion.
Confirming a tag also has to move the change-detectors, or an incremental
refresh/Retrain right after a confirm wouldn't fold the tag in (only the nightly
full pass would): the CCIP global gate now counts character confirmations, and
the head training fingerprint counts confirmations. Test for the CCIP path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Auto-applied tags are provisional (they don't train the model + can be retracted
until confirmed), so surface and confirm them:
- Backend: list_for_image + get_image_with_tags now include `source` + a
`confirmed` flag on each applied tag (via serialize_tag, image-scoped; defaulted
for autocomplete/directory callers).
- Frontend: TagChip badges an unconfirmed auto-tag with an "auto" pill + a
one-click Keep/confirm (✓) → POST /images/<id>/tags/<id>/confirm, which promotes
it to a training positive and shields it from the retraction sweep; TagPanel
reloads so the badge + button drop once confirmed.
Contract test for the source/confirmed payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Milestone 139 raised head_auto_apply_precision 0.97→0.98; operator confirmed the
general-tag confidence was already well tuned, so revert that. The support floor
(min_positives 30→50) and CCIP match confidence (0.92→0.95) stay. Migration 0081
(not yet deployed) edited to drop the precision bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Makes auto-apply truly "soft" for heads: _ids_with_tag (head positives) and
_eligible_tag_ids (graduation count) now count human-applied + operator-confirmed
tags only, via a shared _AUTO_SOURCES (head_auto/ccip_auto/ml_auto) exclusion.
Unconfirmed auto-applied tags no longer train the head that judges them, so a
misfire can't reinforce itself and the retraction sweep can actually drop it.
Confirming a tag (TagPositiveConfirmation) promotes it to a positive AND protects
it from retraction. sklearn-free tests. CCIP reference exclusion is the companion
piece, next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Daily scheduled_retract_auto_tags re-scores standing auto-applied tags and drops
the ones the model no longer supports:
- retract_auto_applied_heads: per graduated head, re-score its source='head_auto'
images (bounded — only the images already carrying the auto-tag, not the whole
library) and remove ones now < auto_apply_threshold.
- retract_auto_applied_ccip: per source='ccip_auto' character tag, max-cosine the
image's figure vectors vs that character's prototypes; remove ones now below the
ccip auto-apply threshold.
Both SKIP operator-confirmed tags (TagPositiveConfirmation) and are SILENT — a low
score isn't proof the tag was wrong, so no hard negative is recorded (that's
reserved for an operator removal). No-op unless the relevant auto-apply switch is
on. New daily beat. sklearn-free tests for both paths + the disabled no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
head_auto_apply_precision 0.97→0.98, head_auto_apply_min_positives 30→50,
ccip_auto_apply_threshold 0.92→0.95 (operator-asked). Model defaults change for
fresh installs; migration 0081 bumps the existing singleton row IFF still at the
old default (won't clobber a deliberate operator change). ml_admin bounds already
permit these. Fixed a stale comment in the auto-apply test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
train_all_heads is now incremental by default: a per-tag training-data
fingerprint (positive + rejection count/latest-timestamp, stored on
tag_head.train_fingerprint) means a manual Retrain refits ONLY the tags whose
data changed — O(what you touched), not O(all heads). The nightly
scheduled_train_heads passes full=True to reconcile sampled-negative + hygiene
drift across every head. First incremental run after deploy still refits
everyone (NULL fingerprints), stamping them, then it's incremental.
The refit decision + fingerprint are split into sklearn-free helpers
(_head_fingerprints, _heads_needing_retrain) so the incremental logic is
unit-tested directly (train_head itself needs scikit-learn). Migration 0080.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now sources character references from character_prototype via a
per-character in-process cache (_load_prototypes) that reloads ONLY the
characters whose ccip_prototype_state.updated_at advanced — no request-path
rebuild, so the per-accept ~4s stall is gone once the store is populated. Cold
start (store empty pre-first-refresh) falls back to the legacy on-the-fly
reference build, so character suggestions work immediately post-deploy and the
background refresh populates the store within ~15 min. Match math + grounding
are unchanged; existing tests exercise the legacy fallback, and a new test
covers matching from the populated prototype store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- refresh_character_prototypes celery task wraps the incremental builder (sync
ml worker); returns skipped / rebuilt=N removed=N.
- Beat: every ~15 min (cheap global-gate no-op when idle) + a nightly full=True
reconcile as belt-and-suspenders.
- train_heads enqueues it on success, so the Retrain button AND the nightly head
retrain refresh CCIP on the SAME trigger — unified lifecycle, as asked.
The initial (cold) full build loads the whole reference set once in the
background, never on a /suggestions request.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
refresh_character_prototypes (sync, celery ml worker):
- Cheap GLOBAL gate (a few COUNTs) → no-op when nothing that affects references
changed since the last refresh (the operator's "only recompute if something
was tagged" trigger).
- Else a per-character fingerprint diff (one GROUP BY: ref count + max region id)
rebuilds ONLY the characters whose references moved — each capped to
MLSettings.ccip_prototype_cap — and drops characters that lost all refs.
Cost scales with WHAT changed, not library size. Reuses ccip's reference
predicate (single-character, non-hygiene, figure CCIP) so prototypes match the
legacy matcher exactly. The async matcher (next step) will READ the table.
Tests: gate no-op when idle, only-changed-character rebuild, capping,
single-character exclusion, lost-reference cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Foundation for making CCIP character references a precomputed, INCREMENTAL
artifact instead of a request-path rebuild (kills the per-accept ~4s suggestions
stall; cost will scale with change, not library size):
- character_prototype: a character's reference CCIP vectors, capped to
MLSettings.ccip_prototype_cap so match cost doesn't grow with popularity.
- ccip_prototype_state: per-character fingerprint (ref count + max region id) +
updated_at → drives per-character incremental rebuilds and the matcher cache's
reload-only-what-advanced.
- MLSettings.ccip_ref_signature (cheap global change gate) + ccip_prototype_cap.
Migration 0079. Schema + models only — the builder service, refresh task/beat,
and matcher rewrite land in the following steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PreviewDialog.vue was orphaned — nothing mounted it — so the dry-run preview
was unreachable. Rather than wire it up, remove the whole chain: its unique
value (a capped 3-page count of what a backfill would grab) is low, and its
adjacent needs are already covered — auth validation by verify_source_credential,
and actually fetching recent posts by "Check now". Operator decision 2026-07-06.
Removed:
- frontend: PreviewDialog.vue + sources store previewSource()
- backend: POST /api/sources/<id>/preview route, download_backends.preview_source,
IngestCore.preview() + its now-unused NativeIngestError import
- tests: the 3 ingester preview tests
Nothing else referenced the chain (verified). Shared campaign-resolution and
ledger helpers stay — they're used by run()/verify_source_credential.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Applied tags aren't scored live, so compute the grounding on demand: run the
tag's head over the image's max-over-bag (whole-image + concept crops), argmax
→ the region that best explains the tag on this image, mirroring what
score_image records for live suggestions.
- heads.py: extract _image_bag (now shared by score_image) + ground_applied_tag.
Returns (grounding, has_head): has_head False = no head to localize with →
no overlay; grounding None = the whole-image vector won → whole-image frame.
- tags.py: GET /api/images/<id>/tags/<id>/grounding → {grounding, has_head}.
- TagChip/TagPanel: applied chips inject fcSuggestionHover and fetch grounding
on hover (cached per image+tag, race-guarded), reusing Step 3's overlay in
both the modal and Explore. No new frontend overlay code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now tracks WHICH query figure produced the winning cosine per
character (argmax over the per-figure best-reference sim) and attaches its bbox as
grounding {bbox,kind:'figure',detector}. SuggestionService carries it: a CCIP-only
character hit grounds to its figure; a 'both' hit keeps the head's localized crop
if it had one, else falls back to the CCIP figure — so corroborated characters
stay grounded. Test: a character match carries the matched figure's bbox+kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
bvandeusen