Downloads/imports stage into <name>.part / <name>.partial then os.replace() into
place, so a kill mid-write leaves a discardable temp — never a corrupt final.
cleanup_orphaned_temp_files sweeps ones left behind under the images root, only
older than 6h so an in-flight download's staging file is never removed. Daily beat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- interpreter_client: shared requests.Session with a connect-only retry
(connect=2, no status retries — we map 429/5xx ourselves) so a proxy reload
is smoothed and the keep-alive connection is pooled across the sweep.
- translate_posts: on an interrupt (drain), re-enqueue after the Retry-After
hint / default backoff instead of waiting for the daily beat; self-terminates
via the health gate. Steady-state one-chunk-per-run on success is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Explore's neighbour grid (/api/gallery/similar → gallery_service.similar) now
takes an Explore-only exclude_wip flag that drops `wip` system-tagged images
from the candidates, alongside the banner/editor presentation tags. The
gallery's own "similar" button is unchanged (keeps wip, #1274) — only the
Explore store passes exclude_wip=1. The anchor itself may still be a WIP; only
neighbours are filtered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs
mid-flight. Give in-flight work room to drain and make interrupted work
resume cleanly instead of stalling.
- docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s /
scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown
can actually drain before SIGKILL.
- celery_app.py: task_reject_on_worker_lost=True — a task killed past the
grace window is re-queued (safe: idempotent + chunked, recovery sweeps
re-drive stragglers).
- interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable
and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter
behind a reverse proxy no longer raises an opaque HTTPError.
- translation.py: thread retry_after out of _translate_batch; retranslate_posts
resumes after the Retry-After hint (or 60s default, capped 900s) on an
interrupt with _reset_done=True, self-terminating via the health gate.
- tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff.
No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
retranslate_posts resets the 5 translation columns to NULL for a scoped set of posts (all, or WHERE artist_id IN ids) then reuses the untranslated sweep to re-run them, chasing the tail until drained (run-until-done). Interpreter cache keys on engine_version so a changed model re-translates, an unchanged one is cache-fast. Reset only happens when the service is configured+healthy so translations are never wiped when they can't be rebuilt. New POST /settings/translation/retranslate (artist_id | all=true). UI: per-artist 'Re-translate posts' on the Artist Management tab + 'Re-translate all' in the Settings Translation card, both with confirm dialogs. No migration (reuses m143 columns).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New POST /api/settings/translation/test pings /v1/health for a GIVEN base URL (not
the saved one), so the operator can verify a URL before enabling it. TranslationCard
gains a Test-connection button that reports reachable/unreachable inline and
updates the status dot. Endpoint test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
/settings/translation/status defaults (off → no health call) + /run 400-when-
unconfigured + 202-when-configured (monkeypatched .delay). requests is already a
backend dep, so no requirements/ci-requirements change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
post_feed_service (card + detail) and provenance_service._post_dict now include
post_title_translated, description_translated (card-truncated / detail-uncapped)
and translated_source_lang, keeping the originals for the toggle. Feed
serialization test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
tasks/translation.py — translate_posts: picks untranslated posts (title OR
description non-empty), per-post [title, description] batch via the Interpreter
client, stores translations + detected lang + engine_version; passthrough /
already-target posts are marked handled with no stored translation. 503 or a
connection error interrupts (retry next cycle), 400 stops (fix config), per-post
commit keeps progress; wall-clock bounded. Wired into celery (maintenance_long
lane) + a daily beat. No-op unless enabled + base URL set + healthy. GET
/settings/translation/status + POST .../run for the Settings card. Task tests
(stubbed client, monkeypatched session).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
services/interpreter_client.py: sync (requests) client for the LibreTranslate-
compatible /v1/translate — no new dep, mirrors the platform clients. translate()
maps translatedText[]↔texts (order + length), returns detected_lang +
engine_version (aggregate = first item, fine for a per-post [title, description]
batch); passthrough items come back unchanged in their slot. InterpreterUnavailable
on 503 / connection error (retry later), InterpreterBadRequest on 400. health()
checks /v1/health engines.llm. 10 unit tests with mocked HTTP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Post gains post_title_translated / description_translated / translated_source_lang
/ translation_engine_version / translated_at — filled by the translate sweep so
viewing is instant. ImportSettings gains translation_enabled (OFF by default),
interpreter_base_url (EMPTY — no default host; the operator points it at their own
Interpreter proxy behind a reverse proxy) and translation_target_lang (en),
exposed + validated via /settings/import. Migration 0083. Settings defaults +
patch + validation test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
scheduled_presentation_auto_apply (daily beat) runs presentation_auto_apply_sweep
— idempotent, so an interrupted run just re-runs next cycle (that's the recovery),
wall-clock bounded by soft/hard task time limits. prune_presentation_reviews
(daily beat) drops RESOLVED review flags older than 30 days (rule 89 retention).
Tests run both tasks via a monkeypatched session factory. Milestone 141 complete:
the presentation-chrome auto-hide + conflict-flagged review is now live end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GET /api/gallery/hidden-review lists unresolved presentation auto-hide flags
(image + presentation tag + conflict tag/score), most-concerning first. POST
.../keep resolves the flag (the tag stays). POST .../unhide removes the
presentation tag (image returns to the gallery), records a TagSuggestionRejection
so the head learns it misfired, and resolves the flag. Tests for list/keep/unhide.
Frontend review strip (shown when Show-hidden is on) next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
presentation_auto_apply_sweep fires banner/editor-screenshot heads at the FLAT
presentation threshold (source=presentation_auto). Two guards: (1) hard-skip any
image already carrying a human/confirmed content tag — you valued it, so the model
can't bury it; (2) if an auto-hide ALSO scores >= presentation_conflict_threshold
on a content head, hide it but record a PresentationReview row (conflict tag +
score) for the Hidden view.
_auto_apply_heads now excludes system tags, so a graduated wip/banner can't fire
via the content path (and wip never auto-applies at all). presentation_auto added
to _AUTO_SOURCES so auto-hidden chrome never self-trains. Tests: applies,
hard-skip valued, conflict-flag, disabled no-op, ignores wip, content-path
excludes system. Settings UI + scheduling land next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) +
presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a
FLAT threshold (decoupled from content-head graduation), plus the "also looks
like content" conflict cut. New presentation_review table (image, presentation
tag, conflict tag + score, created/resolved_at) records auto-hides flagged for
review. Migration 0082 (columns + table), ml_admin API (editable + get_settings
+ _validate bounds), settings roundtrip/bounds test. The sweep that reads these
knobs + the Settings UI land in step 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The default gallery + facets now implicitly exclude images carrying a
presentation system tag (banner / editor screenshot), reusing the tag-scope
EXISTS machinery. Suppressed when the operator explicitly filters FOR a
presentation tag OR passes include_hidden (the Hidden view — step 2). `wip` is
NOT hidden (real, in-progress art). include_hidden threaded through
scroll/timeline/jump_cursor/facets + the gallery API _parse_filters. Test covers
default-hide, include_hidden, explicit-filter-shows, and wip-stays-visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The confirm-UI change added source + confirmed to serialize_tag; two exact-dict
unit tests in test_tag_query.py failed on the new keys. Add them (default
None/False for rows without image scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Completes "no self-training": unconfirmed auto-applied character tags no longer
seed CCIP references — character_references + the prototype builder
(_current_fingerprints/_rebuild_one) gain a shared _positive_char_tag filter
(human-applied OR operator-confirmed), mirroring the head-positive exclusion.
Confirming a tag also has to move the change-detectors, or an incremental
refresh/Retrain right after a confirm wouldn't fold the tag in (only the nightly
full pass would): the CCIP global gate now counts character confirmations, and
the head training fingerprint counts confirmations. Test for the CCIP path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Auto-applied tags are provisional (they don't train the model + can be retracted
until confirmed), so surface and confirm them:
- Backend: list_for_image + get_image_with_tags now include `source` + a
`confirmed` flag on each applied tag (via serialize_tag, image-scoped; defaulted
for autocomplete/directory callers).
- Frontend: TagChip badges an unconfirmed auto-tag with an "auto" pill + a
one-click Keep/confirm (✓) → POST /images/<id>/tags/<id>/confirm, which promotes
it to a training positive and shields it from the retraction sweep; TagPanel
reloads so the badge + button drop once confirmed.
Contract test for the source/confirmed payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Makes auto-apply truly "soft" for heads: _ids_with_tag (head positives) and
_eligible_tag_ids (graduation count) now count human-applied + operator-confirmed
tags only, via a shared _AUTO_SOURCES (head_auto/ccip_auto/ml_auto) exclusion.
Unconfirmed auto-applied tags no longer train the head that judges them, so a
misfire can't reinforce itself and the retraction sweep can actually drop it.
Confirming a tag (TagPositiveConfirmation) promotes it to a positive AND protects
it from retraction. sklearn-free tests. CCIP reference exclusion is the companion
piece, next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The stricter head_auto_apply_min_positives (30→50, migration 0081) dropped the
_head helper's default n_pos=30 below the support floor, so the "supported head"
sweep tests saw the head as ineligible (n_applied 0). Move the default to 60; the
explicit n_pos=5 under-supported test stays correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Daily scheduled_retract_auto_tags re-scores standing auto-applied tags and drops
the ones the model no longer supports:
- retract_auto_applied_heads: per graduated head, re-score its source='head_auto'
images (bounded — only the images already carrying the auto-tag, not the whole
library) and remove ones now < auto_apply_threshold.
- retract_auto_applied_ccip: per source='ccip_auto' character tag, max-cosine the
image's figure vectors vs that character's prototypes; remove ones now below the
ccip auto-apply threshold.
Both SKIP operator-confirmed tags (TagPositiveConfirmation) and are SILENT — a low
score isn't proof the tag was wrong, so no hard negative is recorded (that's
reserved for an operator removal). No-op unless the relevant auto-apply switch is
on. New daily beat. sklearn-free tests for both paths + the disabled no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
head_auto_apply_precision 0.97→0.98, head_auto_apply_min_positives 30→50,
ccip_auto_apply_threshold 0.92→0.95 (operator-asked). Model defaults change for
fresh installs; migration 0081 bumps the existing singleton row IFF still at the
old default (won't clobber a deliberate operator change). ml_admin bounds already
permit these. Fixed a stale comment in the auto-apply test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
train_all_heads is now incremental by default: a per-tag training-data
fingerprint (positive + rejection count/latest-timestamp, stored on
tag_head.train_fingerprint) means a manual Retrain refits ONLY the tags whose
data changed — O(what you touched), not O(all heads). The nightly
scheduled_train_heads passes full=True to reconcile sampled-negative + hygiene
drift across every head. First incremental run after deploy still refits
everyone (NULL fingerprints), stamping them, then it's incremental.
The refit decision + fingerprint are split into sklearn-free helpers
(_head_fingerprints, _heads_needing_retrain) so the incremental logic is
unit-tested directly (train_head itself needs scikit-learn). Migration 0080.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now sources character references from character_prototype via a
per-character in-process cache (_load_prototypes) that reloads ONLY the
characters whose ccip_prototype_state.updated_at advanced — no request-path
rebuild, so the per-accept ~4s stall is gone once the store is populated. Cold
start (store empty pre-first-refresh) falls back to the legacy on-the-fly
reference build, so character suggestions work immediately post-deploy and the
background refresh populates the store within ~15 min. Match math + grounding
are unchanged; existing tests exercise the legacy fallback, and a new test
covers matching from the populated prototype store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
refresh_character_prototypes (sync, celery ml worker):
- Cheap GLOBAL gate (a few COUNTs) → no-op when nothing that affects references
changed since the last refresh (the operator's "only recompute if something
was tagged" trigger).
- Else a per-character fingerprint diff (one GROUP BY: ref count + max region id)
rebuilds ONLY the characters whose references moved — each capped to
MLSettings.ccip_prototype_cap — and drops characters that lost all refs.
Cost scales with WHAT changed, not library size. Reuses ccip's reference
predicate (single-character, non-hygiene, figure CCIP) so prototypes match the
legacy matcher exactly. The async matcher (next step) will READ the table.
Tests: gate no-op when idle, only-changed-character rebuild, capping,
single-character exclusion, lost-reference cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PreviewDialog.vue was orphaned — nothing mounted it — so the dry-run preview
was unreachable. Rather than wire it up, remove the whole chain: its unique
value (a capped 3-page count of what a backfill would grab) is low, and its
adjacent needs are already covered — auth validation by verify_source_credential,
and actually fetching recent posts by "Check now". Operator decision 2026-07-06.
Removed:
- frontend: PreviewDialog.vue + sources store previewSource()
- backend: POST /api/sources/<id>/preview route, download_backends.preview_source,
IngestCore.preview() + its now-unused NativeIngestError import
- tests: the 3 ingester preview tests
Nothing else referenced the chain (verified). Shared campaign-resolution and
ledger helpers stay — they're used by run()/verify_source_credential.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Applied tags aren't scored live, so compute the grounding on demand: run the
tag's head over the image's max-over-bag (whole-image + concept crops), argmax
→ the region that best explains the tag on this image, mirroring what
score_image records for live suggestions.
- heads.py: extract _image_bag (now shared by score_image) + ground_applied_tag.
Returns (grounding, has_head): has_head False = no head to localize with →
no overlay; grounding None = the whole-image vector won → whole-image frame.
- tags.py: GET /api/images/<id>/tags/<id>/grounding → {grounding, has_head}.
- TagChip/TagPanel: applied chips inject fcSuggestionHover and fetch grounding
on hover (cached per image+tag, race-guarded), reusing Step 3's overlay in
both the modal and Explore. No new frontend overlay code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now tracks WHICH query figure produced the winning cosine per
character (argmax over the per-figure best-reference sim) and attaches its bbox as
grounding {bbox,kind:'figure',detector}. SuggestionService carries it: a CCIP-only
character hit grounds to its figure; a 'both' hit keeps the head's localized crop
if it had one, else falls back to the CCIP figure — so corroborated characters
stay grounded. Test: a character match carries the matched figure's bbox+kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
score_image now keeps the ARGMAX beside the max-over-bag: which bag row won each
head. The region query also selects bbox/kind/detector_version, a parallel
bag_meta maps each row → its region (None for the whole-image vector), and every
hit gains grounding {bbox,kind,detector} (null when the global vector won). Threaded
through SuggestionService (new Suggestion.grounding field) → /api/.../suggestions
payload. This is the data the #1206 hover-overlay draws. CCIP-only hits ground null
for now (figure grounding = step 2). Tests: winning crop grounds the tag with its
bbox+kind; whole-image win → grounding None.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Exposes the detector config (per-proposer enable + weights + confidence, caps,
dedupe IoU) in Settings → Tagging, backed by MLSettings via /api/ml/settings.
ml_admin adds the detector fields to _EDITABLE + GET payload + validation (conf
0..1, caps >=1, IoU 0..1). New CropProposersCard.vue (mirrors HeadsCard) with
working defaults pre-filled, per-field live-save (no restart — the agent picks
changes up on its next lease), weights-format help, switch-revert on error.
Closes milestone #134: all three proposers are on out-of-the-box and tunable in
the UI. Test: detector defaults GET + patch round-trip + range validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The GPU lease now carries the crop-proposer config from MLSettings in a per-job
'detectors' block (same pattern as embed_model_name). The agent's worker builds
its Proposers from the announced config via _effective_cfg (lease block overlaid
on env) + _proposers_for (rebuilds only when a config signature changes) — so an
operator's UI edit takes effect on the next lease with NO restart, and env is now
just the bootstrap fallback until the server announces. enabled-off maps to empty
weights (proposer skipped); dedupe_iou + max_regions also come from the effective
cfg. Test: lease announces the detectors block with the seeded default weights.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Parity with pixiv (operator ask): the extension add now resolves the real
display name for our other native platforms too, not just the URL handle.
patreon_resolver.resolve_display_name reads the campaigns API's
attributes.name; SubscribeStarClient.resolve_display_name pulls the creator
name off the profile page (og:title, else the <title> stripped of the
SubscribeStar suffix). extension_service._resolve_artist_name dispatches per
platform (pixiv=token, patreon/subscribestar=cookies via get_cookies_path),
best-effort in an executor, falling back to the readable URL handle on any
failure. Still all curator core — the extension is unchanged (sends only the
URL). gallery-dl platforms keep the handle (readable, no native client).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Final piece of the artist decoupling. (1) Identity-by-source: quick_add_source
resolves the artist by an existing (platform, url) Source first, so a re-add
reuses the artist even after it was renamed (its frozen slug no longer matches
the name) — a slug-based lookup would have duplicated it. (2) Pixiv naming: a new
pixiv source resolves the real display name via the app API (PixivClient
.resolve_display_name → /v1/user/detail) using the stored token, so the artist is
'Kurotsuchi Machi' not '12345678' — and its name-derived slug matches what a
native download produces, unifying them. Falls back to the numeric id when no
token/crypto. ExtensionService gains the crypto seam; the endpoint passes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator ask: a surface to merge new sources into existing artists (consolidate
the singleton artist a fresh add spins up). Enabled by the #130 slug decoupling —
the storage path is immutable, so re-attribution moves NO files. SourceService
.reassign moves the source, re-points its posts (Post.source_id==S) and the
images it contributed (ImageProvenance via S, scoped to the old artist so shared
images aren't stolen), and deletes the old artist if it's left fully empty (else
clears its subscription flag). POST /api/sources/<id>/reassign. Frontend: a
'Move…' action per source on the artist Management tab → artist-autocomplete
picker → confirm → routes to the target (whose slug is stable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
First step of decoupling artist identity/storage/display. migration 0077 drops
uq_artist_name so the display name is free text (two genuinely different creators
can share a name); the slug stays the immutable, unique storage/identity key (the
on-disk path component — untouched, so nothing moves). ArtistService.rename +
PATCH /api/artists/<id> change the name ONLY. Frontend: inline pencil-edit on the
artist header (mirrors TagCard), slug/route unaffected so no navigation. Fixes the
operator's 'no surface to rename an artist' + the name-collision fragility.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Recapture disk-skips already-downloaded media, and upsert_post_record only
writes Post fields — so a pre-existing image (e.g. one pulled under the old
gallery-dl path, imported bare with no post) stays orphaned even after its post
record is (re)written. Confirmed on the operator's instance: 329 pixiv images
with primary_post_id NULL, 694 pixiv posts with content but no linked images, 0
duplicate posts.
Fix: the recapture relink channel now carries the media's post_id (2- → 3-tuple
path/url/post_id), and phase 3 calls importer.link_existing_image_to_post — match
the on-disk image by path, find its Post by (source, external_post_id), upsert
image_provenance + primary_post_id. Factored the provenance-linking out of
_apply_sidecar into a shared _attach_provenance so the fresh-import and
recapture-backlink paths can't diverge. Idempotent; generic across native
platforms (no-op for already-linked Patreon/SubscribeStar). Re-running recapture
now repairs orphaned images; future walks never orphan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags are kind=general, so their suggestions previously landed in the
General group. Give them their own 'system' suggestion category so the operator
reviews them apart from content tags: _current_heads maps is_system heads to
category 'system' (still trained as general heads, still gated by the 0.65
floor). Frontend: CATEGORY_ORDER/LABELS gain 'system'; SuggestionsPanel renders
a 'System' group first (small, collapsible, open — false positives easy to spot
and reject); the typed-dropdown shows the shield icon for system entries. Safe:
system-tag suggestions always carry a canonical_tag_id, so the create-by-kind
path (which would send 'system' as a TagKind) is never hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The core writes the post record BEFORE extract_media, but the ugoira frame
delays were only memoized DURING extract_media — so write_post_record never saw
them and ugoira_frames was always empty in the record. Extract a memoized
_ugoira_meta (frames + zip url share ONE /v1/ugoira/metadata call regardless of
order) and inject client.fetch_ugoira_frames into the downloader (mirrors
Patreon's content_fetcher) so write_post_record populates the frames itself.
Zero extra API calls — the fetch is shared/memoized with extract_media. A
recapture now backfills the timings onto existing ugoira posts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags (wip/banner/editor) already get heads (kind=general) and aren't
filtered from suggestions, but they surfaced only at each head's precision-tuned
suggest_threshold — high enough to hide the borderline/false-positive guesses the
operator wants to SEE and REJECT (hard-negative mining: 'negatively reinforce
what isn't a system tag'). score_image now uses a flat _SYSTEM_TAG_SUGGEST_FLOOR
(0.65, operator-set) for system-tag heads instead of their auto threshold;
content-tag heads keep their own, and the typed-dropdown threshold_override still
overrides everything. _current_heads carries Tag.is_system into the head meta to
drive it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Two pre-merge fixes:
1. SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on
the .art domain (keeps 302'ing to /age_confirmation_warning even with the
cookie — Elasid #54116), but the same creator is reachable on .adult where
the cookie works. _normalize_ss_host rewrites subscribestar.art →
subscribestar.adult at request time (stored Source.url untouched), logged so
it's visible in walk logs. .com/.adult pass through.
2. Disabling a source now clears its failure state (last_error, error_type,
consecutive_failures) so subs you pause (not paying for) stop lingering as
'failing'. Only the explicit disable clears — an unrelated edit to an
already-disabled source leaves state alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The native downloader used the Windows-safe sanitize_segment, but gallery-dl on
Linux (path-restrict auto→'/', path-remove default control chars, path-strip
auto→'') replaces ONLY '/' and deletes control chars — the Windows-forbidden set
(<>:"|?*) and trailing dots/spaces stay RAW in on-disk titles. Any pixiv title
with those chars would therefore miss the tier-2 disk-skip and re-download the
whole work at cutover (seen-ledger starts empty). Replace sanitize_segment with
gdl_clean_filename, a byte-exact mirror of gallery-dl 1.32.5 build_filename
(verified against path.py). Directory + template already matched; this closes the
last parity gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The real phase-1 ctx has always carried auth_token; the native branch now
threads it into the adapter constructors, so the stub ctx must match the
contract (kept the strict ctx[...] read — it catches exactly this drift).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the
recover/recapture UI actions now route through PixivIngester. Campaign id is
parsed straight from the source URL (numeric user id — no network resolver),
with a platform-aware resolution-failure message. auth_token now rides the
uniform adapter construction (token platforms use it, cookie platforms
accept-and-ignore), and the preview endpoint fetches/threads it. The legacy
gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the
refresh-token config branches in download/verify) per no-legacy policy;
gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The stub payload is PNG bytes regardless of target extension, so the real
validator quarantined the .jpg cases; and extracting the ugoira work hit the
API seam of a fake session with no .post. Validation/quarantine plumbing
stays covered by the Patreon downloader tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar
ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira
— pximg URLs carry no content hash). PixivIngester wires client/downloader/
ledgers into ingest_core with drift label 'Pixiv app API' and the new
body_canary=False opt-out: caption-less pixiv artists are common, so the
zero-bodies #862 alarm would false-positive here — the client's
response-shape drift checks cover that failure class instead. auth_token
joins the uniform adapter constructor (pixiv is the first token-auth native
platform). verify_pixiv_credential = one OAuth refresh, no feed walk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivDownloader writes originals to the exact pre-cutover gallery-dl layout
(<artist_slug>/pixiv/pixiv/{id}_{title[:50]}_{NN}.{ext} — flat, double
platform segment) so tier-2 disk-skip recognizes existing files. Post-first:
per-media sidecar is identity-only; the post record (_post_<id>.json — id
suffix because the flat layout would collide a bare _post.json) carries the
enrichment: tags + EN translations, rating from x_restrict, series,
view/bookmark/comment counts, AI flag, dimensions, author, and ugoira frame
delays (the zip has no timings). i.pximg.net media GETs ride the app-header
profile (403 without the app-api Referer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivClient mirrors gallery-dl 1.32.5's PixivAppAPI request profile exactly
(iOS app headers, OAuth refresh with X-Client-Time/X-Client-Hash,
/v1/user/illusts pagination via next_url — whose query string doubles as the
resumable page cursor). Post-first seams (post_record_key / post_is_gated /
post_meta) + extract_media covering multi-page, single-page, ugoira zip
(600x600→1920x1080 swap, frame delays memoized for the post record), and
limit_* placeholder gating. No PHPSESSID web fallback: FC holds only the
refresh token, same effective coverage as the gallery-dl path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The four remaining run-1895 failures were stale expectations, not
predicate bugs — prune/reset returned the right counts, but these tests
verified no-deletion by counting the ENTIRE tag table (or asserting the
full kind set), which now includes the three seeded hygiene tags that
survive prunes and resets by design. Filter is_system=false with a
pointer to #128 so future system tags cannot re-break them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
bvandeusen