Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got
3-10 character guesses dominated by the most-referenced characters (a 27-ref
character clears a low bar on many images). A sweep showed 0.85 collapses the
noise (noisy multi-matches 47→3) while keeping the confident single-character
matches.
- ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image
reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85.
- Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999).
- Slider in the GPU agent card ("Character-match strictness") — tune live, no
redeploy, same observe-and-tune loop as auto-apply.
Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
So work an agent orphaned gets picked back up quickly, three layers:
- GpuJobService.release(): a graceful agent stop hands its still-leased jobs back
to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease.
- GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a
60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the
queue counts honest even when nothing is leasing.
- Lease TTL 300→180s: still well above any single job (a capped-frame video embed
is tens of seconds, and a live worker heartbeats), but a hard crash recovers
faster once the sweep fires.
Tests: release returns-to-pending (token-scoped), recover_orphaned resets only
expired leases, release API round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
So the work can be checked through an API as the agent fills in vectors (same
pattern as /api/heads/metrics):
- GET /api/ccip/overview: regions by kind, images with figure CCIP vectors, the
per-character reference counts (which characters have enough examples to match
on), and the embedding versions present.
- GET /api/ccip/images/<id>: that image's stored regions (bbox, frame_time,
has_ccip/has_siglip, versions) + the CCIP character matches it would get — for
spot-checking detector + matcher output.
Read-only, no GPU. (Queue depth is already at /api/gpu/status.)
Tests: overview coverage counts + per-character refs; per-image regions + matches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
SuggestionService.for_image now merges CCIP character matches with the SigLIP
head suggestions — they're complementary, not exclusive: CCIP is the identity-
specialized signal but needs a detected figure; the heads work whole-image but
conflate identity with style. Merged by tag: 'both' when they corroborate
(higher score wins), 'ccip' / 'head' otherwise. Cheap when no CCIP vectors exist
yet (match_image returns early without a figure vector), so it's a no-op until
the agent runs. Suggestion.source is now 'head' | 'ccip' | 'both'.
Test: a character with a CCIP reference figure surfaces (source='ccip') on a new
image whose figure matches.
NEXT: the agent container (real CCIP/detector models, hands-on) that produces the
vectors this consumes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The server-side brain that turns stored CCIP vectors into character suggestions
— no GPU. character_references() gathers each character tag's prototype vectors
(figure/face-region CCIP embeddings on images carrying that tag); match_image()
cosine-matches an image's figure vectors against every character (multi-
prototype: best over a character's examples), surfacing those above a tunable
threshold as {tag_id, name, category:'character', score, source:'ccip'},
excluding already-applied characters. v1 = cosine on raw CCIP vectors; the exact
CCIP metric/threshold gets validated against the model in the hands-on eval.
Tests (synthetic vectors): same-character match across images, no-match for an
orthogonal figure, already-applied exclusion, no-figure-vectors empty.
NEXT: merge CCIP character suggestions into the rail; the agent container that
actually produces the vectors (hands-on, GPU — not CI-verifiable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The thin HTTP surface over the queue so the desktop agent stays HTTP-only:
- Agent endpoints (Authorization: Bearer <token>): POST /api/gpu/jobs/lease
(returns jobs + image_url + mime + video frame cadence), /submit (stores
regions via RegionService + closes the job; 409 on a stale lease), /heartbeat,
/fail. Token validated against AppSetting (mirrors the extension-key pattern,
constant-time compare).
- Admin (browser): GET/POST /api/gpu/token[/rotate] (generate + show the agent
token), GET /api/gpu/status (queue counts), POST /api/gpu/backfill → dispatches
enqueue_gpu_backfill.
- enqueue_gpu_backfill(task): one INSERT…SELECT enqueues a job per image lacking
one for the task (scales to the full library; idempotent).
Agent flow: lease over HTTP → fetch pixels via the normal FC image URL → compute
on the GPU → submit. Redis/Postgres never exposed.
Tests: bearer required (+ wrong-token 401), lease→submit round-trip (region+CCIP
vector stored, job done via /status), stale-lease 409, backfill enqueue +
idempotency.
NEXT: the agent container + control UI, then the CCIP detector/embedder + matcher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Answers "how are videos/all media handled by the GPU worker": a job is per ITEM,
but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the
existing cadence), each stored with a timestamp — so a video becomes a BAG of
frame embeddings (fixes the mean-embedding muddle) instead of one washed-out
vector. Stills → frame_time NULL; animated GIF/WebP treated like short video.
- image_region.frame_time (migration 0061, not yet deployed so folded in): the
source frame's seconds for video/animated media; NULL for stills. RegionService
passes it through. A whole frame is just kind='frame'.
- gpu_job + GpuJobService (migration 0062): the durable work list that keeps the
desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE
SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat /
complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues;
the agent leases+submits over the web API; Redis/Postgres stay private.
Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped
heartbeat, complete, fail-requeue-then-error. region test now covers frame_time.
NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the
agent container + control UI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The storage backbone both crop jobs write to and read from. image_region =
normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id;
'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP /
Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The
bbox doubles as grounded-tag provenance. Migration 0061.
RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines
don't clobber each other) + get_regions — the GPU agent's results endpoint will
call the writer; the character matcher + bag scorer read. Server-side, no GPU.
Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The trunk of both crop jobs — CCIP figure-crops and SigLIP concept-crops call
the SAME crop_region(): normalized-bbox crop with optional context padding,
edge-clamping, and the lower-bound size floor (max of a fraction-of-short-side
and an absolute pixel floor) below which a region is too small to embed and
returns None. Only the proposer (where) and embedder (what) differ; the crop is
shared. Pure Pillow — importable + testable anywhere (the GPU agent imports it
for the crop step). Unit-lane tests (no DB): region pixels, floor rejection,
edge clamp, pad expansion, out-size resize.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
dict(session.execute(...)) on a bare Result invokes the mapping protocol (a
Result has .keys() = column names) and subscripts it → "CursorResult is not
subscriptable". Materialize with .all() so dict() consumes rows as key-value
pairs. The API path already did this; the snapshot task missed it. Caught by
test_snapshot_records_timeseries_point (run 1628).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.
Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
Both captured at correction time in TagService.add_to_image/remove_from_image
(source is lost on delete) into durable per-tag counters (head_metric), keyed
by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
(head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
+ head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
head quality, plus the snapshot time-series — the report to tune the precision
target + support floor.
Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.
What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.
NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Graduated heads can now apply their tag without a human — gated so it's safe:
- FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled,
default OFF) is on AND it has >= head_auto_apply_min_positives (default 30)
clean labels. A precise-looking but under-supported low-N head can't spray tags.
- auto_apply_sweep (heads.py): streams every embedded image in chunks, scores
against the eligible heads (numpy, no sklearn), applies each head's tag where
score >= its auto_apply_threshold and the tag isn't already applied/rejected,
with source='head_auto' (distinguishable + reversible). dry_run counts only.
- HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags
task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled)
+ recovery sweep + retention(20).
- API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled),
GET /api/heads/auto-apply (recent runs + per-concept report). Settings
head_auto_apply_enabled + min_positives via /api/ml/settings.
Tests: sweep applies above threshold, dry-run writes nothing, skips under-
supported + ungraduated heads; API disabled/dry-run/conflict guards.
NEXT (slice 2): the observability the operator asked for — per-concept misfire
(auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a
reporting API to tune. Slice 3: the UI (enable, preview, trends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two cadences for keeping heads in sync with your tagging:
- PASSIVE: a nightly `scheduled_train_heads` beat (skips if a run is already
in flight; creates+commits the run row before dispatching train_heads so the
ml worker always finds it). Folds the day's accepts/rejects + newly-eligible
concepts into the heads without anyone clicking.
- ACTIVE: a "Retrain heads" button in the Explore trail bar — bank the +/-
feedback you just gave while walking content, without a trip to Settings.
Shared logic in a new useHeadTraining composable (trigger + poll + start/finish
toasts), used by the Explore button; reflects an already-running run (incl. the
nightly one) on mount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The rail's Suggestions now come from the trained per-concept heads. SuggestionService.for_image scores the image's frozen SigLIP embedding against
every head (heads.score_image) and surfaces concepts above each head's own
suggest threshold; the typed-dropdown's min=0 "show everything" mode maps to a
flat floor so any head-scored concept can still be picked. Already-applied tags
drop; rejected tags stay flagged + reversible (unchanged).
REMOVED from the suggestion path (rule 22, no fallback): the Camie
ImagePrediction candidate/alias/merge pipeline and the per-tag centroid
augmentation, plus the now-dead SuggestionService internals (_load_predictions,
_threshold_for, _settings, self.aliases, self.centroids). Head suggestions are
always canonical tags, so raw_name/via_alias are null/false and the rail's
alias kebab is inert by data (its removal + the Camie ingest-tagger rip are the
flagged follow-up). for_selection (bulk consensus) now aggregates head
suggestions unchanged.
Tests rewritten to the head path: test_ml_suggestions (surfaces/applied/
rejected-reversible/override/no-embedding/no-heads), test_suggestions_bulk
(consensus), test_api_suggestions (get + dropped the Camie-alias roundtrip),
and test_ml_artist_retired (artist not head-eligible via _HEAD_KINDS).
DEPLOY NOTE: after this lands, the rail is empty until you run Train heads
(Settings → Tagging → Concept heads) — deploy, train, then the rail populates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before
I), and drop the inline comment that split heads.py's import block. Pure import
ordering — no behavior change. (run 1601 lint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
A red-✗ dismissal no longer makes the suggestion vanish. The rejected tag
stays in the rail — dimmed, struck-through, with a "rejected" pill and a
one-click undo (↶) in place of the ✗ — so a misclick is recoverable and the
operator can see what they've said no to (operator-asked 2026-06-27).
Backend: SuggestionService.for_image now KEEPS rejected tags, flagged
rejected=True, sorted to the bottom of their category, instead of dropping
them. New AllowlistService.undismiss + POST /suggestions/undismiss clears the
TagSuggestionRejection. Rejected items are still excluded from bulk consensus
(for_selection) and the type-to-add dropdown, whose jobs are unchanged.
Frontend: store.dismiss flags in place (canonical tags) rather than dropping;
new store.undismiss reverts. SuggestionItem renders the rejected state and
swaps ✗→↶; ✓ still accepts (which clears the rejection server-side).
Tests: rejected-surfaced-flagged-then-reversible (service) + undismiss
endpoint idempotency (API).
Completes #1134's reversible-rejection half. Heads-as-suggestion-source is
the remaining piece.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
"Keep" on a doubted positive was a no-op, so the same confirmed-correct images
came back in "head doubts" every run (operator-flagged: reinforcement keeps
surfacing the same images). Add tag_positive_confirmation (mirror of
tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and
the eval excludes confirmed positives from the doubts list — exactly as rejected
items already drop out of the suggest list. The tag stays a positive either way
(confirmation is a "reviewed" marker, not a training change).
- model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent).
- tag_eval: _confirmed_ids + exclude from head_doubts_positive examples.
- store.confirmTag + card "keep" calls it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"head would suggest" drew from the whole negative pool, which INCLUDES the
images the operator rejected. A rejected near-miss (e.g. an orc under "goblin")
is a hard negative that still scores high, so it kept resurfacing as a fresh
suggestion every run (operator-flagged: "same items keep appearing"). Exclude
already-rejected ids from the suggest list — once you've said no, it's gone.
(head doubts = lowest-scoring positives is unchanged; genuinely-hard true
positives legitimately recur there.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two additions driven by "what's the commit threshold?" + "find more tags":
1. High-precision operating point (Bar 4). Per concept, report the threshold that
maximizes recall while holding precision >= a target (default 0.97, configurable
via `precision_target`) — i.e. "could this fire without a human, and how much
would it catch?" `head.auto_apply` = {target, threshold, precision, recall} or
null if the target is unreachable. Surfaced on the card.
2. Server-side concept auto-discovery. `auto_top_n` param unions the explicit
concept list with the N most-tagged general tags (one fast DB query) so the
eval can broaden itself without hand-listing — replaces the slow HTTP directory
paging. Card gains "+ auto-add top-N" and precision-target inputs.
No migration; numpy/sklearn stay lazy. Existing _normalize_params test still
holds (new keys additive; None still falls back to DEFAULT_CONCEPTS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Frontend for #1130. A maintenance tile in Settings → Tagging:
- Editable concept list + "Run eval" → POST /api/tag-eval (one running at a time).
- Rehydrates on mount via the persisted run (getRun by latest id) and polls while
running — so the report SURVIVES navigation (operator-flagged); the task runs
backend-side regardless and the card reconnects to its row.
- Renders the saved report: per-concept head-vs-centroid metrics table (AP/F1/
precision/recall) with Δ AP, the learning curve (AP @ N positives), and
thumbnail galleries (head-would-suggest / head-doubts-positive) for eyeballing.
Backend: _examples now stores thumbnail_urls (not just ids) so the report is a
self-contained artifact that renders without per-id lookups on reload.
No new top-level surface — slots into the existing maintenance area.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cluster tag-gap feature's only UI (Explore's TagGapPanel) was removed in the
3-pane rework, leaving the backend that fed it with no caller. Surgical removal:
- drop the POST /api/images/cluster/tag-gaps route (cluster_tag_gaps)
- drop BulkTagService.tag_gaps (+ the now-unused `import math`)
- drop the tag_gaps tests (test_bulk_tag_service, test_api_bulk_tags)
BulkTagService's common_tags / bulk_add / bulk_remove stay — they still back the
gallery bulk editor. Pure deletion, no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The directory card count regressed to a globally-inflated number (~every
card showed the same ~469): the fandom leg used a doubly-nested correlated
subquery — image_tag.tag_id IN (SELECT member.id WHERE member.fandom_id ==
Tag.id) — whose inner predicate did not correlate the outer Tag, so it
matched EVERY character that has any fandom and counted all their images for
every tag. The gallery scope and cleanup count were unaffected (they pass a
literal tag id, a single-level subquery), which is why only the card diverged
from the gallery.
Rewrite the count as a single-level correlated scalar subquery: join `member`
(the tag applied to the image) and match image_tag.tag_id == Tag.id (direct)
OR member.fandom_id == Tag.id (a character of this fandom). Strengthen the
directory test with a second unrelated fandom/character so a non-correlating
fandom leg fails (count would read 4 instead of 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fandom owns characters via Tag.fandom_id, but every image<->tag query
went purely through direct image_tag rows, so a fandom only surfaced
images literally tagged with it — images carrying one of its characters
were invisible to its browse count, previews, and gallery filter.
Derive membership at query time instead of materializing fandom rows
(which would drift on every reassign/merge/remove). Add one shared
predicate in tag_query.py — image_in_tag_scope / image_in_any_tag_scope:
an image belongs to a tag if tagged with it directly OR (when the tag is
a fandom) carrying a character whose fandom_id is that tag. The character
leg is empty for non-fandom tags, so it applies uniformly with no kind
branching. Route all read sites through it:
- gallery _apply_scope: include, OR-groups, and symmetric exclude
- directory image_count: correlated COUNT(DISTINCT) scalar subquery
- directory previews: UNION direct + via-character, then ROW_NUMBER<=3
- cleanup count_tag_associations: Tier-B delete prompt now reports a
fandom's true blast radius (was 0 for fandoms with no direct rows)
find_unused_tags already protected fandoms via used_via_fandom; left as is.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A swarm overlay-network blip after the :latest redeploy left Redis healthy but
transiently unreachable; a worker starting in that window crash-looped on the
initial broker connect (kombu OperationalError) and needed a manual Redis reset
to recover.
Retry the broker forever on startup + at runtime (broker_connection_max_retries
=None), add redis-transport socket options to the broker (short connect timeout,
TCP keepalive, retry_on_timeout, periodic health check), and mirror the same on
the Redis result backend. Now a transient outage self-heals when overlay routing
returns instead of the worker exiting.
Test pins the key resilience settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster C, milestone #94. BulkTagService.tag_gaps(image_ids, threshold) finds
tags applied to >= threshold fraction of a visual neighbour set but not all of
it (the '7 of 10 share Miku; these 3 don't' signal). Each gap carries the
laggard image ids minus any TagSuggestionRejection rows, so apply-to-cluster
never re-proposes a tag a neighbour dismissed. 100%-common tags and <2-image
sets are excluded. New POST /api/images/cluster/tag-gaps.
Tests: consensus found / common excluded / missing ids; rejected laggard
excluded from missing; tag dropped when all laggards rejected; <2 images empty;
route shape + bad input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the
same counts the apply produces (rule 93 parity) without mutating: images_moving
(source links the apply UPDATEs), images_already_on_target (links it drops),
source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom
compatible flag (surfaced, not raised, so the UI can warn), and up to 6
thumbnails of the moving images. The admin /tags/<dest>/merge route gains a
dry_run flag returning the preview JSON.
Tests: preview moving-count == apply merged_count (parity), incompatible flagged
without raising, self/missing raise, admin dry_run returns preview + no mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. Backend for the allowlist tuning dashboard.
#7a: AllowlistService.coverage(tag_id, threshold) counts distinct images with
a prediction resolving to the tag (raw_name==tag.name OR (raw_name,category) in
the tag's aliases) scoring >= threshold — the gross candidate pool, mirroring
tasks.ml._confidence_for_tag resolution. list_all now carries applied_count
(grouped image_tag count) + coverage_count (at the row's threshold). New
GET /api/tags/<id>/allowlist/coverage?threshold= for the live what-if number.
#7b: /suggestions/accept + /alias return {allowlisted, tag_id, tag_name,
projected_count} (projection at the tag's threshold) instead of 204, so the UI
can show a non-blocking 'auto-applying to ~N images' toast. Apply still runs
async via apply_allowlist_tags — projected_count is an estimate.
Tests: coverage by threshold (direct + alias-with-category), list applied vs
coverage, coverage route (explicit/default/bad threshold), accept/alias payload
(newly-allowlisted vs already-on-list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster A, milestone #97. #5: clicking an image-modal tag chip's body now
closes the modal and opens the gallery filtered for that one tag (fresh
filter); ✕/kebab stay as the explicit remove/rename controls.
#6a (backend of OR/exclude filtering): gallery_service._apply_scope gains a
structured tag model — tag_or_groups (AND-of-OR: one EXISTS(tag_id IN group)
per group) + tag_exclude (NOT EXISTS(tag_id IN exclude)) — layered additively
on the existing tag_ids AND path so cursors/facets/deep-links are untouched.
Threaded through scroll/timeline/jump_cursor/facets/similar + facets common
dict; _require_single_filter rejects post_id combined with OR/exclude. API
parses tag_or (repeatable → one OR-group each) + tag_not (csv exclude).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
DRY pass follow-up (note #1026). Five handlers returned the identical
jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result).
Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives,
trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews.
trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it).
Added route-level tests for all five consumers (these trigger endpoints had no
route coverage before): 202 + task_id via _queued, and the dry_run flag threading
through to dedup/purge-gated. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened
process #594). Five Tier-A endpoints repeated the same get_json -> dry_run ->
run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn,
**kwargs); the five route handlers now delegate. reconcile keeps its source_id
validation and passes it through **kwargs.
The cleanup_service predicates were already shared between preview and apply
(find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no
backend-logic change; this is purely the HTTP-handler boilerplate.
Consumers (all routed through the helper, verified no copy left behind):
prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id),
purge_legacy_tags, reset_content_tagging.
Added route-level tests for prune-bare (apply) and reconcile (apply + source_id
passthrough + invalid-source_id 400) — the two helper consumers that previously
had only service-level coverage, so every consumer is exercised at the route.
Findings B (queued-response helper) and C (store dry-run POST helper) identified
but not applied this pass (operator scoped to A). The card preview->commit state
machine is deferred to a frontend pattern-consistency sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in
eff6427; closing it out after today's #87 work, which added a seam it didn't
account for. _repoint_post_links drops a loser post's ImageProvenance row on the
(image, post) uniqueness collision — and that row may now carry from_attachment_id
(which archive the file was extracted from). For the exact gallery-dl->native
case this targets, the keeper is the native stub (no archive) and the loser is the
gallery-dl row that extracted the member, so a blind delete silently lost the
containing-archive linkage. Carry from_attachment_id onto the keeper's surviving
row (when NULL) before dropping the collision.
The rarer PostAttachment-collision case (both dup posts captured the same archive
blob) doesn't arise in the targeted scenario — the archive lives only on the
gallery-dl post, so it re-points straight to the keeper and the FK stays valid.
Test: collision merge preserves the loser's from_attachment_id on the keeper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A filesystem-imported archive with no adjacent sidecar has no Post, so
_post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced
post.id. _stamp_member_archive already no-ops on a None post_id (no post → no
provenance to stamp); pass None instead of crashing. Caught by the existing
test_reimport_archive_is_idempotent (no-sidecar zip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from 068def2.
- ProvenancePanel: heading pluralizes ('Attachment' for a single file).
- Backfill: re-running reextract_archive_attachments (ArchiveReextractCard)
routes through _import_archive and stamps existing rows — no new code.
Tests: capture stamps on fresh import, nested-archive attribution, per-post
archive on dedup; for_image filters to the containing archive; reextract
backfill stamps the link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Attachments section aggregated PostAttachment rows across EVERY post an
image was pHash-linked to. When one of those was a 'High Resolution Files'
mega-bundle (dozens of unrelated archives), the list ballooned past the
viewport and overwhelmed the modal's right rail.
- for_image() now scopes attachments to ImageRecord.primary_post_id (the post
the file was actually captured from), falling back to all linked posts only
when primary_post_id is unset (older rows / filesystem imports).
- ProvenancePanel wraps the list in a max-height scroll container with a count
in the heading, mirroring the cards' independent-scroll treatment.
Note: FC stores archives as opaque blobs and never records which archive an
extracted image came from, so attachments can't yet be scoped tighter than the
post. Capturing image->archive containment is tracked as separate work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Image posts wrap the post date in an <a> permalink
(<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts
don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY
the unwrapped case, so every image post got a null published_at and sorted to the
top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's
_data_from_post method: text up to the first </, then after the last > — handles
both. Verified against the live raw feed (all 6 dates now parse).
Robust logging (operator request): _parse_posts now logs per-page parse stats
(posts / dated / with-body) and a WARNING canary when posts parse but NONE get a
date or body while the raw markers are present — i.e. our extraction diverged
from the live markup. Makes this failure class diagnosable from the worker log
alone, no authed re-fetch needed.
Test: a permalink-wrapped date parses to ISO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SubscribeStar posts deliver content only through document/audio attachments,
which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them:
- docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the
href URL + doc_preview-title + data-upload-id (kind=attachment).
- audio: scope uploads-audios..post-edit_form, split on audio_preview-data
blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio).
The existing downloader handles them unchanged (plain streaming GET; the file
validator only inspects image/video extensions via is_validatable, so PDFs/zips/
audio pass straight through, no quarantine).
Test covers doc + audio extraction (the cheunart sample has none, so this pins
gallery-dl's documented markup shape).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body
regex either returned empty or over-captured into sibling upload divs and the
'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact
_data_from_post rule — content between the post_content-text wrapper and the
youtube-uploads div (literal markers), then strip the trix editor's
<html><body>…</body></html> document wrapper to its inner. Verified against the
live cheunart sample: clean per-post bodies, empty for genuinely text-less posts.
Also port gallery-dl's _media_from_post preview guard: skip gallery items whose
URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of
the Patreon gated-preview bug (#874); this is why a locked post yields no media.
Tests: body must not bleed into the pagination counter; trix html-document
wrapper stripped; /previews items skipped. Fixture now includes the youtube-
uploads close marker present in real markup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_
subscriber even with valid .adult cookies (confirmed present: _personalization_id
+ _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's
ENTIRE flow — including the base Extractor._init_session I'd not read — the
divergence is the HTTP request profile, not the cookies or parser.
gallery-dl's default (cookies-only) mode sends, on EVERY request including the
first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same-
site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint
is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling
looked unlike a browser → SubscribeStar gated the adult-creator page.
Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept-
Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the
per-request XHR headers (both feed and load-more now use the shared profile).
Test updated to assert the gallery-dl-parity profile instead of the old
navigation-vs-XHR split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native client split the feed on `<div class="post is-shown`, but `is-shown`
is added by SubscribeStar's infinite-scroll JS when a post scrolls into view —
present in a browser-SAVED page (what the Step-0 characterization used) but
ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live
feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError.
Align with gallery-dl's proven `_pagination`: split on the generic
`<div class="post ` (trailing space rules out the hyphenated post-content/
post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating
detection (/verify_subscriber, /age_confirmation_warning => auth, not drift).
Regression tests: raw server markup without is-shown now parses; an age-wall
redirect raises SubscribeStarAuthError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart
still drifts — we're being served a full HTML page that isn't the feed. Add
_describe_page(): the drift error now reports the page <title> + which known
interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha),
so the next run names the actual cause instead of "markup changed". Strong
suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is
UA-locked and our hardcoded UA likely differs from the cookie-capturing browser).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First live run (cheunart) tripped the drift guard: "no posts and no recognizable
feed container". The browser-saved page was normal (6 posts + posts_container-list),
so the parser was fine — our live HTTP fetch got a different response. Cause: the
client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the
initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content-
negotiates an XHR full-page request to a non-HTML body → no container → drift.
Fix: the session now uses browser-like navigation headers (Accept: html, NO
X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on
the "load more" endpoint (which is a genuine XHR). Drift message now reports the
response length + a JSON hint so a recurrence is self-explaining. Regression test
pins the header split (navigation initial GET, XHR load-more).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at
import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester",
_FakeIngester) no longer affected dispatch → the fake's run() never ran →
KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the
dict with a _native_ingester_cls() call-time lookup that reads the module globals,
so monkeypatching the class names works again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.
- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
_NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
/ preview_source / verify_source_credential platform-aware. SubscribeStar's
campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
recover/recapture actions show for subscribestar; download_service's
cursor/mode/post_first + the preview endpoint already key on
uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
verify, gallery_dl_service skip-value, api_sources arm-no-preflight).
post_is_gated stays best-effort (can't cause junk downloads); not gating this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
