e6f128c894ae5db7026423a2b1f81c31d28a3136
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Fable 5
|
e6f128c894 |
feat(ml): training hygiene — system-tagged images are absent from other concepts training
Step 2 of milestone #128. _hygiene_excluded_ids (training_data.py) is the one shared predicate: images carrying any system tag are dropped from every OTHER concepts head training — not positives (a rough wip tagged as a character drags the head toward generic-sketch) and not rejection or sampled negatives (a wip OF character X is not evidence against X). A system tags own head trains on them unfiltered; that is what makes auto-flagging banners work. Selection is split out of train_head as the sklearn-free head_training_ids so CI (no sklearn) can pin the behavior. CCIP: reference prototypes skip hygiene-tagged images — a faceless wip figure region must never become an identity reference — and the ref cache signature now counts hygiene applications, since tagging an image wip changes the reference set without touching character/region counts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
e9891ee9f3 |
feat(tags): system tags — is_system column, seeded hygiene tags, protection guards
Training hygiene step 1 (milestone #128). Migration 0075 adds tag.is_system and seeds wip / banner / editor screenshot (kind=general), ADOPTING an existing same-(name,kind) tag case-insensitively instead of duplicating. These rows drive the upcoming training exclusions, so they are protected: rename and merge-away refuse system tags (merge-INTO stays allowed — folding an operator's old hygiene tag into the system row is the intended move; merge is the only tag-delete path, so that guard covers deletion). is_system rides every tag serialization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
581b778528 |
fix(modal): accept-by-known-id keeps the raw suggestion row identity for the drop
Spreading canonical_tag_id onto a raw suggestion changed its _keyOf identity, so _dropEverywhere missed the actual list row and the panel kept showing an already-accepted suggestion. Pass the resolved id as an option instead; pinned with a raw-suggestion spec. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
6d314d662f |
feat(modal): applied tags drop from search instantly; manual add == accepting a suggestion
Three tag-flow gaps in the view modal (and the Explore workspace, which shares TagPanel): - the type-to-add dropdown now filters both its sections against the imageledger applied tags reactively, so a just-added tag disappears from search the moment the chip rail updates instead of after a modal refresh - manually picking or creating a tag the model also suggested routes through the suggestion-accept flow: the acceptance is recorded for head training and the row leaves the panel, instead of the add silently bypassing the feedback loop - removing a tag reloads the suggestion lists, so a model-suggested tag returns to the suggestions area (flagged rejected, one-click reversible) rather than vanishing until the next modal open Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
b54243a1ff |
fix(subscribestar): inject the 18+ age cookie on every SubscribeStar domain
The cookie was pinned to .subscribestar.adult only; cookies are domain-scoped, so sources on subscribestar.art (Elasid, event #54116) never sent it and every poll 302d to /age_confirmation_warning. Emit one line per domain (.com/.adult/.art) with a per-domain presence check, and admit .art in the platform url_pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
aa12a57f97 |
feat(recovery): surgical re-fetch for deep posts via ExternalLink reset
Operator-flagged: the recovered defective files live DEEP in their artists'
back-catalogues — the normal download cadence (by design, via the seen-gates)
will never re-walk them, so recovery's source re-check alone can't bring them
back. The durable per-post handle is the ExternalLink row, which survives the
image delete:
- services/external_links.refetch_links_for_post: reset settled links to
pending (fresh attempt budget, in-flight left alone) + dispatch their
fetches; sha-dedupe at import discards payload files that still exist, so
only the missing file lands.
- recover_defective_image now captures the image's post ids BEFORE the delete
cascades provenance away and resets those posts' links — future recoveries
are surgical automatically (response gains links_reset; source re-check
stays for gallery-dl-native files within walk reach).
- POST /api/admin/posts/refetch-external {external_post_id, source_id?} — the
manual tool for the three files recovered before this fix existed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
|
||
|
bvandeusenandClaude Fable 5
|
b1cfbcc06a |
fix(agent): sleep mode — an empty queue sheds to one downloader and backs the lease poll off to 15 min
Operator-flagged on the deployed .5 build: the autoscaler grew the pool 1→8 against an EMPTY queue (an empty buffer read as 'GPU starving' regardless of WHY), and every downloader kept polling lease every 10s all night. - New idle signal straight from the lease results: an empty lease sets _idle, any jobs clear it. The occupancy-low branch now distinguishes three cases: queue empty → shed to ONE polling downloader; pinned at the bandwidth cap → shed toward 3; cap headroom + work flowing → grow. - Idle lease polls back off exponentially per downloader to IDLE_POLL_MAX_SECONDS (15 min) and reset the moment work appears — so an idle night costs one HTTP call per 15 min, and new work is noticed within at most ~15 min (operator-accepted trade-off). - UI hint: 'idle — queue empty, lease poll backed off'; /status gains idle. Agent build 2026-07-02.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
f08a64f7ae |
style(ia): wave 4 — one section-header language across every admin surface
The Subscriptions Settings tab's bare text-h6 headers adopt the same uppercase accent section-title + hint convention Maintenance/Cleanup use, with a one-line hint per section (extension / credentials / downloader / external file-hosts / schedule defaults). Every settings-ish surface now reads identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
dc7fa6eae2 |
feat(ia): wave 3 — Subscriptions landing answers 'what needs me, what came in?'
Daily-use reorder of the Subscriptions tab: needs-attention strip first (FailingSourcesCard moves up from below the Downloads fold — a broken subscription was invisible unless you went looking), then a new Recent arrivals card (real downloads only, no-change scans filtered out, artist links), then the source list. Both cards render nothing when there's nothing to say. Retry logic moves into the downloads store (retrySource / retryAllFailing) so the needs-attention card and the Downloads maintenance menu share one implementation — single-retry forces past cooldown, bulk keeps cooldown enforcement, same tally shape. The card's Logs button deep-links into the Downloads tab pre-filtered (?source_id now watched, not just read on mount). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
e039689eff |
feat(ia): wave 2 — Activity becomes the whole-app pulse; Overview gets the health strip
The Activity tab only knew Celery — the GPU agent (the majority of processing) and the download pipeline were invisible there. Two new self-polling panels: - GpuActivityPanel: queue depths + triage verdicts (defects / file-ok / unprobed, top reason buckets) with a jump to Maintenance -> Failed processing. The triage detail refetches only when the error count moves. - DownloadsActivityPanel: 24h stat chips + failing-source names with a jump into Subscriptions. Both panels join the Activity tab under Queues+workers AND double as the Overview health strip (side-by-side grid under the Celery summary) — one component set, so Overview answers 'is everything healthy?' across all systems. SystemStatsCards reviewed: content still accurate, left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
5b34c9221c |
feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home
Settings IA per the approved A3 design (the old layout was the two-app merge fossilized): - Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual /import scans stay API-level; imports arrive via downloads/extension, heal via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm moves to Maintenance → 'Ingestion & filters' and loads its own settings; the import store shrinks to settings-only (no remaining consumers of the scan/task-list machinery). Overview's pending banner now points at Activity. - Maintenance regrouped: Ingestion & filters / GPU agent & embeddings (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders, Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive re-extract demoted last) / Storage. - One extension home: BrowserExtensionCard moves from Settings → Overview to Subscriptions → Settings, above the API key bar it authenticates. - Single-color import filter WIRED: skip_single_color/threshold existed since FC-2 but nothing read them (the audit module's docstring said as much) — now enforced on both import paths via the audit's canonical predicate (tolerance 30, matching the Cleanup card default; animated images exempt like the transparency check). Default stays off; test added. - Dead weight: PlaceholderView (zero refs) and the permanently-disabled 'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed (celery queue docstring, threshold comment citing retired tasks, ml package docstring, HeadsCard 'replaces Camie' blurb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
19b962f1a7 |
feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
7c19ad91ed |
feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback)
Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint says 'holding at the bandwidth cap' and /status reports bw_capped, so the behavior is legible without tests that need the ML stack. Reset content tagging: stays a FULL-instance reset (operator's call), but now lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale counts are rejected server-side). Copy no longer claims suggestions repopulate: it says plainly the heads' training examples are deleted and re-tagging starts fresh. Moved out of TagMaintenanceCard into DangerZoneCard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
eaea4308fc |
chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved)
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
a7abcc41ca |
feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
|
||
|
bvandeusenandClaude Fable 5
|
1f27189b8f |
chore: retire ml-backfill-daily beat + the spent purge-legacy action (operator-approved)
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's daily embed backfill for the same NULL-embedding images at ~100x the cost (B1 audit verdict, milestone #124). The backfill TASK stays — the manual /api/ml/backfill button remains the deliberate CPU fallback pending B3. - purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on the live library before removal (A2 audit, milestone #123). Fully retired per rule 22: tile, store action, route, service fn, tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
95d2ae1d58 |
feat(agent): global bandwidth cap — the agent can't saturate the desktop's network
One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited; live MB/s dial + net readout in the control UI) is charged by every still download (streamed chunk reads) and every ffmpeg video stream (metered from outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget). Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8 unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's browser included). Capping the aggregate keeps the desktop usable and still beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
31c416bc7b |
docs(beat): backfill comments no longer claim errored jobs are retried
Follow-through on the tombstone rule (
|
||
|
bvandeusenandClaude Fable 5
|
09e2772628 |
fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards
The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).
One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.
Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
reason instead of another release; strikes never count while stopping, and
clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
GpuJobService so they can't drift) converts expired leases with >=5 lease
grants and pending jobs with >=10 to 'error', preserving the last stored
failure reason. Backstops old agent builds.
Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
|
||
|
bvandeusenandClaude Fable 5
|
3d6201734c |
fix(settings): maintenance tiles start collapsed; remember manual open state
GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head existed, TagEvalCard whenever a persisted run existed — so a fresh Settings load greeted the operator with several tiles already expanded. All three now force-open only while their task is actually running (the #877 resurface behavior on the busy-driven tiles is untouched). MaintenanceTile additionally persists MANUAL expand/collapse per tile in localStorage, so the section reloads the way the operator left it; a forced open while a task runs stays transient and is never saved as a preference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Fable 5
|
1b1d3732dc |
feat(agent): store ffmpeg's actual failure reason in the job's error field
sample_frames_from_url now returns (frames, reason) — reason carries the SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found", or the timeout) instead of only logging it agent-side. The worker folds it into the failure it reports, so curator's GpuJob.error reads e.g. no frames sampled from video — ffmpeg exit 183: moov atom not found ... instead of the bare "(unprocessable)". The errored-jobs list becomes self-describing: after a retry sweep, surviving errors name their real defect without needing the agent log. Return-value plumbing (not shared state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Fable 5
|
686808d3f3 |
feat(gpu): "Retry errored jobs" — scoped requeue of errors only
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Fable 5
|
3a683d7feb |
fix(agent): short videos failed as "unprocessable" — fps filter emits 0 frames
Root cause of the "no frames sampled from video (unprocessable)" flood
(operator-flagged 2026-07-02, whole 62k-70k image block + others): the
sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4)
frames — which is ZERO for any clip shorter than ~2s. Short animation loops
(0.5s, 1.75s — verified against two originals from different artists) are
complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames,
exited 0, and the agent failed the job as unprocessable. Long videos worked,
so only the short-clip class flooded.
Fix: sample with select ("first frame always, then one per interval of
timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range
yuv420p sources don't trip the mjpeg encoder's full-range strictness
(secondary failure observed on a 4440x2760 clip). Verified locally against
both failing originals (frames extracted, PIL-clean) and a synthetic 15s
video (4 frames at t=0/4/8/12 — long-video behavior unchanged).
Observability (why this hid for weeks): ffmpeg's stderr was discarded, so
every failure logged only "no frames sampled". stderr now goes to a temp
file and its tail is logged on any produced-no-frames/timeout failure — the
log names the actual ffmpeg reason from now on. Also: frames written before
a mid-stream ffmpeg error are now kept (partial > nothing).
VERSION → 2026-07-02.1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
0a618db10c |
fix(agent): Status froze after one update — conchint.textContent destroyed #capn
THE root cause of "the Status section doesn't update" (chased across several
rounds; the backend was always healthy). `#capn` (the max-concurrency number)
was nested inside `#conchint`:
<div id=conchint>… · max <b id=capn>8</b></div>
and applyStatus() ran, every call: `capn.textContent=CAP` AND
`conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces
ALL of conchint's children — destroying the <b id=capn> node. So:
call 1: capn exists → tiles update → conchint.textContent DELETES capn
call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) →
applyStatus throws on its FIRST line → aborts before any tile →
frozen.
This is exactly the observed "ticks a couple times then freezes", and why
/gpu + /logs (which never touch capn) kept updating fine.
The capn write was redundant anyway — conchint.textContent already renders
the max. Remove the nested <b id=capn> element and the capn.textContent line;
the hint still shows "· max N". VERSION → .10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
713a11e394 |
fix(agent): server-side rate metrics + killable-on-stop ffmpeg
Two follow-ups from live debugging of "work/min never populates" and "stopped never reached". 1) jobs/min + downloads/min are now computed in the BACKEND on a fixed cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were derived client-side from poll deltas with a dt<30s guard — but a backgrounded/unfocused browser tab throttles its timers to ~1/min, so every delta exceeded 30s and the guard blanked the rates forever. A server-side rate is independent of how often the tab polls. Frontend just displays s.jobs_per_min / s.downloads_per_min. VERSION → .9. 2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see the stop signal until ffmpeg returned, so Stop detached still-running threads and work kept flowing long after — "stopped" that wasn't really stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a `should_stop` callback every 0.5s, terminating (then killing) the process at once on Stop or the per-video timeout. A stop-killed job is handed back (transient), not failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
6282e753a9 |
fix(agent): real start/stop state machine — kill the stuck "stopping" pill
The Status pill hung on "stopping" forever (operator-flagged 2026-07-01). Root cause: the backend had no lifecycle state — status() only returned running/stopped — so the UI FABRICATED "stopping" in JS as `!running && active>0`. That pill only cleared when the backend's `active` counter hit 0, but stop() (a) blocked the HTTP handler on lease-release calls to curator and (b) left `active>0` whenever a consumer wedged mid-submit/release to an overloaded curator → "stopping" that never resolved. Give the backend a real, truthful state it drives itself: stopped → starting → running → stopping → stopped - start(): → starting; a downloader flips it to running on its FIRST successful lease (so "running" means curator is actually answering, not just "Start was clicked"). If curator's down it honestly stays "starting". - stop(): → stopping; returns immediately (no handler block). A background monitor waits for the worker threads to actually exit, releases leases, then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can NEVER hold the UI in "stopping" again. In-flight work is handed back safely. - Buttons follow the real state (Start only from stopped; both disabled through the transition), so you can't fight a transition. - Log every Start/Stop button press (routes) and every transition (worker), so the Logs panel shows exactly what each button did. Frontend now trusts s.state (drops the active>0 hack); VERSION → .8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
91ea06be79 |
feat(agent): Status shows smoothed jobs/min + downloads/min (replace jumpy gauges)
The buffer / on-GPU / downloader counts flip many times a second, so a 3s
status poll only ever samples noise — the tiles looked frozen (same value
twice) or random (wildly different), reading as "the Status section doesn't
update" when the backend was in fact live (operator-flagged 2026-07-01).
Replace the three instantaneous gauge tiles with two derived RATE tiles:
- jobs / min — GPU throughput, from the monotonic `processed` counter
- downloads / min — fetch throughput, from a new monotonic `downloaded`
counter (bumped when a job is decoded into the buffer)
Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the
reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped
at 0 (agent restart resets the counters), and skip a backgrounded-tab gap.
The still-useful instantaneous state is demoted, not lost: buffer stays as
the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited
out` (transient) gets promoted to a tile.
backend: worker.status() gains `downloaded`; `_bump(downloaded=)`.
frontend: retiled Status + rate math in applyStatus; VERSION → .7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
98b2ac90dd |
refactor(agent): DRY pass on the GPU agent worker package
Consolidate genuine duplication in agent/fc_agent into single-source helpers (behavior-preserving; DRY Pass process #594): worker.py - _fail(jid, image_id, exc, verb) — 4 terminal "fail this job" blocks (downloader HTTP-fault + decode, consumer non-transient + generic). - _release(job_ids) (was _release_owned) — the one lease hand-back path; 6 inline release([jid])+unhold sites now route through it. - _stopped(stop_evt) + _abort_if_stopped(jid, stop_evt) — 4 stop-check -and-release blocks and every bare stop-check. - _timed(stage) contextmanager — ~8 monotonic()/_record() timing pairs; records only on clean exit, matching the old skip-on-raise behavior. - _ewma(prev, x, alpha) module fn — 3 EWMA updates in the autoscaler. client.py - _submit(path, payload) — submit / submit_embedding (retrying session). - _post_quiet(path, payload) — heartbeat / fail / release fire-and-forget. detectors.py - Proposers._top(detector, image, cap) — merges components() and panels(). config.py - _bool_env(name, default) — auto_start / auto_scale env parsing. Left alone (recorded): the xyxy→norm-xywh conversion duplicated across models.py/detectors.py (2 copies, independent wrapper modules — sharing would couple them), and the _ensure_embedder/_ensure_proposers pair (same lock shape, different concepts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
8a0237eeea |
fix(agent): stream videos via ffmpeg-from-URL instead of downloading the whole file
The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE file into memory (r.content) just to sample a few frames, which buffered ~1GB in RAM and — on any slow/contended media store — got cut off mid-download (ChunkedEncodingError), failed, and re-leased forever. Measured the media read at ~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak helps; the file simply shouldn't be fully downloaded. Environment-agnostic fix (works for any deployment, completes even when slow): - media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL. It Range-reads only the video index + up to max_frames of content — never the whole file — and reconnect flags resume a dropped transfer instead of failing. Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over speed. Removes the bytes-based sample_frames (dead once videos stream). - worker._download_decode: videos now stream (no fetch_image, no RAM blowup); stills still download+decode. On an ffmpeg miss, probe curator liveness (client.is_reachable) → fail the job if curator is up (unprocessable file, stops the infinite re-lease) vs release if curator is down (transient, survives a redeploy). Auth header passed so it works whether or not /images is gated. Build marker 2026-07-01.6. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
aa0605585b |
fix(agent): log the REAL fetch/submit failure reason, not "curator unreachable"
"curator unreachable" was printed for every transient error, hiding whether a single file's transfer stalled (ReadTimeout — curator is up, that stream is slow) or curator itself is down (ConnectTimeout/ConnectionError) or errored (HTTP 5xx). Those need completely different fixes, and we've been diagnosing the download slowness blind. Add _transient_reason(exc) → a specific label (HTTP <code>, else the exception class: ReadTimeout / ConnectTimeout / ConnectionError / …) and use it in both transient paths: - downloader: "fetch failed job <id> (image <id>, ReadTimeout) — released, backing off" - consumer: "submit failed job <id> (<reason>) — released, re-lease later" Now the logs say which failure it actually is (and which image), so we can tell a slow/stalled transfer apart from an unreachable curator. Build marker 2026-07-01.5. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
0fe1674753 |
perf(web): stream files in 4 MiB chunks + 4 hypercorn workers (fix 40s downloads)
The image library is on a CIFS/SMB share (mounted rsize=4 MiB, actimeo=1), and Quart's FileBody streams in 8 KiB chunks — so serving one large original was ~19k network round-trips to the storage server, i.e. 30–58s per download (operator-flagged). That's what starved the GPU agent (constant "curator unreachable" backoff) AND slowed the browser: every byte is read off CIFS and streamed through the Python app (no reverse-proxy sendfile), and only 2 hypercorn workers meant the agent + the browser's thumbnail grid queued behind each other. In-container fix, no new service: - Raise FileBody.buffer_size 8 KiB → 4 MiB in create_app, matching the mount's read size: one round-trip per read, ~500× fewer. buffer_size is the MAX read so small thumbnails still read in one gulp, and Range/mime/ETag/conditional handling lives on Response — all preserved. Guarded so a Quart-internal change can't break boot. - HYPERCORN_WORKERS default 2 → 4 so concurrent /images requests stop queuing. Expected: large-file transfers drop from ~40s toward link speed (a few seconds) for the agent and the browser. See issue #1223. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c22f37d64d |
feat(gallery): sort by earliest post date across all posts (new default)
The gallery's newest/oldest sort keys off image_record.effective_date = COALESCE(primary post's post_date, created_at). The primary post is often the repost/download the file came from, so the grid led with download dates rather than when content was first posted (operator-flagged). Add a second materialized sort key, earliest_post_date = MIN(post_date) across ALL of an image's provenance posts (every post it appears in), else created_at — the original publish date. Mirrors the effective_date pattern so the sort stays a forward index scan. - alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill created_at baseline then MIN over image_provenance ⋈ post. - importer: recompute earliest_post_date whenever a dated post is linked (MIN over the image's provenance, which now includes the just-added row). - gallery_service: new sorts posted_new / posted_old key off earliest_post_date; cursor + year/month grouping follow the active column transparently. - api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads with original publish date. newest/oldest (effective_date) still available. - frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post date); existing effective-date sorts relabelled "Newest/Oldest added". - tests: service test asserts posted_new/posted_old key off earliest_post_date; frontend default-sort omission test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
ccbb5cbc9e |
fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse)
Operator hit an outage after the machine slept overnight: the agent showed "curator unreachable" in a loop while curator's API (lease) was actually fine and the browser could still load images — just slowly. Root cause is a feedback loop in the new pipeline: every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for that reason — so the agent piled on more concurrent large-file GETs, saturating curator's web workers + NFS, which slowed curator (and its browser) further and produced more failures → more downloaders. Classic congestion collapse. - Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing — the empty buffer is caused by failures, not the GPU starving. It ramps back up only once downloads succeed again. - DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving path is too many; 8 keeps a fast GPU fed without stampeding curator. - fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a large-but-flowing download still completes, but a stuck/dead connection fails in 60s instead of hanging a downloader for 3 min and piling up stuck requests. Build marker 2026-07-01.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
ef3318aac1 |
feat(explore): more variance in the related rail (stronger MMR diversification)
Operator wants the Explore "related" rail to span more — the #1188 diversifier was tuned conservatively. Push all three knobs so it reaches further across clusters instead of clumping near the anchor: - MMR lam 0.55 → 0.40 — weight the diversity penalty harder (the main dial). - candidate pool min(200, max(limit*5, 60)) → min(400, max(limit*8, 100)) — a wider nearest-cosine pool so MMR has genuinely distinct neighbourhoods to pick from, not just the near-dupes. - pHash dup_threshold 6 → 8 — collapse more near-duplicate reposts/clones, freeing rail slots for distinct picks. Still deterministic (same set per image, just more spread) and relevance-anchored via the lam*sim-to-anchor term. Backend-only; no migration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
7cdce0c474 |
feat(agent): temporal video dedup — drop near-duplicate frames before the GPU
Near-static videos are the dominant GPU load: sampled into up to 64 frames, each re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU perceptual-hash frame dedup upstream of the GPU so the redundant frames are never processed at all (not just their embeds). - media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy keep — a frame survives only if its hash differs from every kept frame by >= min_distance bits (Hamming). A static run collapses to one frame; genuinely distinct scenes all survive. Order + frame_time preserved. - Called in worker._download_decode right after sample_frames, so it runs in the decode stage on the downloader thread (CPU) — the GPU consumers only ever see deduped frames, and buffered video items shrink (less RAM too). - Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M` when it drops any, so video load reduction is visible. Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal axis. Build marker 2026-07-01.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
eaae896858 |
feat(agent): dedupe near-duplicate crops before the SigLIP embed
Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes, but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept ∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping booru head classes on one head), embedding the same region twice and burning a slot against max_regions. Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass over the pending (crop, template) list right before embed_batch — drop boxes that overlap ≥ iou within the same kind, keep the highest score. The high threshold is deliberate: it collapses only true near-identical boxes while preserving intentional nested crops across scopes (a whole figure vs a small head component sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU (≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
afef95a87d |
feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms), so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit) left the fast GPU idle during every download. Rearchitect worker.py into a producer/consumer pipeline: downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU consumers (detect+embed→submit) - Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a 2nd consumer if it has util/VRAM headroom and lifts throughput, else trim). - Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping RAM + lease look-ahead. VRAM pressure sheds a consumer immediately. - Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU; curator's 180s TTL would otherwise reclaim them mid-buffer). - Preserves all resilience: lease exp-backoff, submit-path retry (#169), release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools and releases every held lease at once (single held-set as source of truth). - Consumers SHARE one embedder + proposers instance (a 2nd consumer adds concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots). - UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU · processed · errors, a buffer-occupancy meter, and a consumers/waited-out line; the dial now tunes downloaders. Build marker 2026-07-01.1. Also fix the operator-flagged detector warning: yolo11n + the comic-panel model threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load- time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3 crop proposers and spamming the log per image. Disable that fusion (unfused inference is correct, marginally slower) and permanently self-disable a proposer on the first inference failure instead of re-throwing forever. Refs milestone 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
83f1070a11 |
fix(agent): bound video GPU work — early-exit the frame loop at max_regions
Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the SUBMIT — the GPU work was already spent. Break out of the frame loop once accumulated regions reach max_regions, so a long video costs ~a few frames of GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it mean-pools all frames and returns before this loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c587ac667c |
fix(agent): cap figures + global region cap + reset active on stop
Three safety/robustness fixes from the operator's run logs: - Cap figures per frame (MAX_FIGURES, default 8) like components/panels already are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept (image 81602 looped on 413). This is the acute fix. - Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still exceed the cap (long video), keep the highest-scoring and log the drop, so a submit body can never blow past curator's limit. - Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the reset can't drive it negative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
7e74fa767c |
fix(agent): load huge images — disable PIL decompression-bomb guard
Trusted local library, not an upload surface, so a legitimately large image (90–95M px, operator-flagged) must load. PIL only WARNS at the 89M-px default but RAISES DecompressionBombError at ~179M px, which would fail those jobs. Set Image.MAX_IMAGE_PIXELS = None. (The agent works off individual extracted files — curator's archive_extractor unpacks zip/cbz/rar/7z at import — so this is about big single images, not archives.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
9f1148b110 |
chore(agent): modernise base → Ubuntu 24.04 / Python 3.12 / CUDA 12.9
Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10, CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04: - Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10. - CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13: ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open "Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call, not a hardware limit. - PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed (PEP 668); a single-purpose container owns its environment, so global installs are fine and simplest. - agent/ruff.toml pinned to py312 (was py310) so CI lints against the real runtime; from __future__ import annotations stays (PEP 649 lazy annotations are 3.14, so self-refs still evaluate on 3.12). CI builds the image but has no GPU — validate on the desktop after pull that it starts and loads CUDAExecutionProvider (not CPU fallback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
f01b59f390 |
fix(agent): py3.10 startup crash + submit-path retry; pin agent ruff to py310
The agent container (CUDA base, Python 3.10) crashed on startup with `NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the `from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14 (PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10. CI lint/compile run on 3.14, so it slipped through. - config.py: `from __future__ import annotations` so the self-referential annotation is a string, never evaluated — works on 3.10 and every version. - agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime) and inherit the root rules. Ruff now flags exactly this class as F821, so CI's lint lane catches it instead of shipping a broken image. (CI otherwise lints on 3.14, masking 3.10 issues.) - client.py: submit path now retries in-place. A dedicated session with a urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a momentary blip after the GPU work is done doesn't discard it and force a full re-download + recompute elsewhere. A duplicate submit after a lost response is a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
79269da802 |
fix(agent): prompt stop + lazy curator polling + build marker; add agent to CI
Addresses operator reports: Stop never finishes, the agent polls curator constantly, and stale-cached pages get mistaken for a failed deploy. - Stop is prompt: flip _running BEFORE any lock so /status + worker loops see "stopped" immediately, and add a stop/shrink checkpoint in _process (after decode, before the expensive detect+embed) that releases the job and bails — so a Stop doesn't wait out heavy GPU work. - Lazy curator polling: the queue snapshot is fetched only while a browser is actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence, not a constant background loop. The work loop's own lease/submit is curator's only visitor otherwise — nothing polls just to poll. - Build marker: VERSION is embedded in the page and reported on /status; the UI shows a "reload" banner when they differ, so a browser-cached page can't be mistaken for "the new image didn't deploy" (complements the no-store header). CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the GPU agent is linted + syntax-checked before its image builds (build.yml only `docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
e6a7fe7d03 |
feat(agent): per-stage timing breakdown (lease/download/decode/gpu/submit)
Instrument the job pipeline so we can see where wall-clock actually goes and decide — on data, not theory — whether a download/compute split is worth building. Each stage is timed per job and a rolling breakdown is logged every 30s to the agent console, e.g.: timing/30s — lease 8ms · download 310ms · decode 40ms · gpu 165ms · submit 70ms | wall/job 585ms (214 jobs) - lease timed around client.lease() in the slot loop (per batch). - download = fetch_image; decode = image/frame decode; gpu = detect + CCIP + batched embed; submit = the results POST. One-time model load is excluded from the gpu figure. - Thread-safe accumulator (stage -> [sum, count]) summarised + reset by a small daemon reporter thread; logs only when there was work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
181f1c6a27 |
perf(gpu-queue): partial indexes + two-phase lease so leasing stays O(batch)
The throughput bottleneck was curator-side, not the network. lease() claimed the lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a plain `status` index Postgres walked the primary key from id=1, skipping the entire prefix of already done/error rows before reaching pending ones. As `done` grew (69k+), every lease became an O(done) scan — leasing crawled, the DB saturated, and even /status (the queue GROUP BY count) stalled the agent. - Migration 0070 adds two partial indexes over just the live slice: pending rows indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery + orphan sweep). They stay tiny no matter how large the done/error history. - lease() split into two phases so each uses a partial index: claim pending first (id-ordered, O(batch)); reclaim expired leases only when pending can't fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim). - Model __table_args__ declares the indexes so ORM and schema agree. - Test: a done-prefix at low ids must not stop the lease reaching pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
f0f031782d |
fix(agent): unfreeze status view + smoothed throughput-aware autoscaler + log pane
Operator: the status tiles (state/active/processed) and the Start/Stop buttons freeze while the GPU meters stay live. Root cause: /status made an INLINE blocking curator call (queue_status) on every poll, and with curator buried under a 112k-job backlog that call stalled — freezing the whole status refresh (the GPU bars survived because /gpu is a lock-free local read). Made worse by the old util-band autoscaler, which grew workers toward the 32 cap forever because util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold — piling load onto curator and the agent process. - /status is now a pure in-memory read: worker.status() is lock-free, and the curator queue snapshot is refreshed by a background poller (never inline). - Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES: samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every ~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a grow stops helping it backs off one and holds, re-probing occasionally. No runaway, no flopping. - GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth, exposed on /gpu) when running, else smoothed client-side — so it glides instead of bouncing 0↔99. - act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the now-always-fast /status refresh recovers state regardless. - Log pane: bound the page to the viewport (height:100vh) so the Logs card scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at 400 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
82e1a4e127 |
fix(agent): send Cache-Control: no-store so a new image isn't masked by cache
The control page is a static string served with no cache headers, so after pulling a fresh agent image the browser kept showing the OLD UI until a hard refresh (operator-flagged). Add a no-store middleware covering the page and the status/gpu/logs polls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c2e9157822 |
feat(agent): graceful Start/Stop with starting/stopping states + instant status
Operator: the buttons fire but the status view doesn't reflect the change. Cause: act() ignored the POST's own status response and waited on the separate /status poll (which lags behind the curator queue call). Now: - act() applies the POST's returned status immediately for instant feedback, and shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled) the moment it's clicked. - A stop that still has in-flight jobs draining shows "stopping" until active hits 0, then resolves to "stopped" on its own. - applyStatus() guards the /status-only fields (connection pill + queue) so the lean action response can't blank them — the Start/Stop path deliberately skips the slow curator call to stay snappy. Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll + autoscaler + /status share a single subprocess instead of piling up in the server thread pool (which was what made clicks feel dead under load). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
3b34230fbd |
fix(agent): stable util-band autoscaler + live GPU meters
Two operator-reported issues with the GPU agent: 1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb probed +1, judged it over a too-short noisy throughput window, saw no clear gain and reverted -1 — every tick. Replace it with a GPU-utilization-band controller: HOLD while smoothed util sits in a healthy band, grow only on clear spare capacity (util below the low mark + VRAM headroom), shrink under saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool. Load stays consistent instead of probe/reverting. 2. GPU util/VRAM bars only updated on manual refresh. They rode the /status poll, which blocks on the curator queue call (slow when curator is busy), so the meters froze between refreshes. Give them a dedicated /gpu endpoint (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop the curator queue-status timeout 15s -> 5s so /status itself stays snappy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c259d03618 |
fix(agent): revert full-width page, grow the Logs section to the bottom
Operator meant the LOG section should fill down the viewport (vertical), not the whole page going full-width horizontally. Restore the centered column (820px), make .wrap a full-height flex column, and let the Logs card flex to fill the remaining height to the bottom (drop the fixed 230px log-pane cap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
2713c3f773 |
perf(agent): batch SigLIP crop embeds per image + load truncated images
Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s,
truncated-image failures):
- BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo
components + panels) and embed them in ONE forward pass instead of one
forward+lock per crop. The per-crop path serialised every crop through the
inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating);
batching gives a real GPU-bound workload + far higher throughput. CCIP still
runs per figure inline.
- LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly-
truncated scraped images now load instead of failing the job 3× then erroring
("image file is truncated (N bytes not processed)").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
9eaefac385 |
feat(agent): full-width control page, Copy-logs button, quiet HTTP log noise
- Page fills the viewport horizontally (drop the 780px cap). - Copy button on the Logs card → copies the console (clipboard API on localhost, textarea-execCommand fallback), with a brief "Copied" confirmation. - Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/ ultralytics to WARNING so the console shows agent activity (detector loads, job errors, autoscale moves) instead of per-request HF-download spam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c1b099e5a3 |
feat(agent): in-UI log console + a real styling pass on the control page
- logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root logger; GET /logs serves it; the control page polls it into a console pane — so runs are monitorable without `docker logs`. worker now logs autoscale moves (one line per change, with jobs/s + util + VRAM) and job failures (job + image + reason); detectors already log load/disable. - Restyled the whole control page: a proper dark layout with a header + live connection pill, cards (Control / Status / Logs), a styled Auto switch + worker stepper, status tiles, separate GPU-util and VRAM meters, and the log console. No longer feels like an afterthought; all the existing control hooks are preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
6d7b17b0b5 |
feat(agent): autoscale the worker count (throughput hill-climb), Auto default-on
The new per-job workload (3 detectors + several SigLIP embeds) is far more GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and is hard to guess. Add an Auto mode (default ON) that finds it: - _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the target: grow while throughput keeps improving and VRAM stays under budget, revert a step that doesn't help, back off under memory pressure (VRAM >= 90%), then settle and periodically re-probe (the GPU/IO balance shifts over a run). - A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the control UI re-enables it. status() reports `auto`; the dial reflects the auto-chosen count (read-only) while Auto is on. - AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
359bc5a283 |
feat(ml): default to SigLIP 2 (new installs) + model dropdown, no free-text (#1203)
- Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in) — UPDATE applies ONLY where no image is embedded yet (fresh install), so an existing library is NOT silently invalidated; it switches deliberately via the dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2. - GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512 recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types. - GpuAgentCard: the two name/version text fields → a single model dropdown; Save sets name+version from the picked option (the current model is always selectable even if off-list). - embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1) to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the ml-worker's persistent HF_HOME. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
80f8eb4756 |
feat(gpu): re-process trigger to apply new crop detectors to the existing library (#1202)
The siglip/ccip backfills skip images that already have current-version regions, so adding crop detectors only affected NEW images — the back-catalogue would never be re-cropped. Add a reprocess trigger that resets every done/error job of a task back to pending, so the agent re-runs the FULL pipeline (figure detection + CCIP + concept/panel crops) over the whole library under the current detectors. - reprocess_gpu_jobs(task='ccip') task + POST /api/gpu/reprocess. - gpu store reprocess() + GpuAgentCard "Re-process library (re-detect + re-crop)" button with a confirm (it's heavy). - Test: a done job resets to pending (attempts cleared). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
ce5db5caaf |
chore(agent): default the anatomy + panel proposer weights to working values
booru_yolo yolov11m_aa22.pt (40MB) + mosesb/best-comic-panel-detection::best.pt. Each self-disables if its download fails, so defaulting them on is safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
d5f29f7056 |
feat(agent): crop proposers — booru_yolo anatomy + COCO person + comic panels (#1202)
Better region PROPOSERS feeding the existing crop→SigLIP→max-over-bag heads (no change to the learned-tagging approach; no per-tag cost — propose once, embed each region, all heads in one matmul). - detectors.py: lazy ultralytics YOLO wrapper, each proposer independently optional + guarded (a bad weight spec / inference error self-disables that one, logged, never breaks the worker). Weights resolve from an ultralytics name | http(s) URL | "hf_repo::file", cached under HF_HOME. NMS merge so a figure two detectors both find collapses to one crop. - worker: figure boxes = imgutils detect_person ∪ general COCO person (merged) → CCIP + concept (anime + Western/realistic coverage); booru_yolo anatomy components (head/cat-head/anatomy/…) → concept crops; comic panels → kind= 'panel' concept crops. Capped per frame (MAX_COMPONENTS/MAX_PANELS). - config + compose: PERSON_WEIGHTS (default yolo11n.pt, works OOB), ANATOMY_WEIGHTS + PANEL_WEIGHTS (operator sets booru_yolo URL + mosesb panel hf::file; empty = off). ultralytics added to requirements. - backend: image_region 'kind' doc notes 'panel'; no migration (free String, and the bag scorer keys on a non-null siglip_embedding, not the kind, so any SigLIP region joins the bag automatically). Agent is outside CI — py-compiled here; operator tests on the GPU and checks Western-vs-anime crop quality via /api/ccip observability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
3d7f60a6e3 |
fix(lint): use dict() not a dict-comprehension in tag_stats (C416)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
9a3cda007a |
feat(api): agent-friendly tag analysis endpoints — /tags/top + /tags/<id>/stats (#1136)
Fast, read-only, indexed aggregates shaped for ANALYSIS (not the paged UI directory, which is alphabetical + builds previews and timed out at 10 min on a full count sweep). - GET /api/tags/top — top tags by image count, desc. ?kind, ?limit (cap 500), ?min_count, ?source=all|human|manual|accepted|auto (human=manual+ml_accepted, auto=head_auto+ccip_auto+ml_auto). One GROUP BY over image_tag (indexed on tag_id). - GET /api/tags/<id>/stats — per-tag dataset health: total + per-source counts (manual/accepted/head_auto/ccip_auto), human vs auto rollups, rejection count, and whether a trained head exists. Backs concept-readiness + source-split analysis. Plain-HTTP homelab posture, no auth change. Tests cover ranking, source filter, min_count, the source breakdown, and 404. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
bc6d43d3f2 |
refactor(ml): drop dead tagger/suggestion settings + columns (#1199)
Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
3d97667f5b |
fix(lint): drop unused select import in tags.py after allowlist removal
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
485387ff0b |
refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189)
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
3d77a38a25 |
refactor(ml): remove the dead per-tag centroid subsystem (#1189)
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
4daa3f2790 |
feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
0f472b2f9e |
fix(explore): diversify "more like this" so it stops getting stuck (#1188)
Pure nearest-cosine piled near-identical images into the neighbour grid — a reposted banner filled all 24 slots, and once you wandered into a B&W / comic-panel cluster every neighbour was more of the same with no way back to colour without the Random button (operator-reported, with screenshot). similar() now over-fetches a wide candidate pool (5x the requested limit, cap 200), then diversifies down to `limit`: - pHash near-duplicate collapse: drop candidates within 6 Hamming bits of the anchor or an already-kept candidate, so a repost (and the anchor's own clones) appears at most once. - MMR re-rank: greedily pick for closeness-to-anchor minus similarity-to-already -picked (lambda 0.55), so the result SPANS clusters instead of returning 40 variations of one image. Falls back to nearest-order on any failure / small pool, so existing nearest-first behaviour is unchanged when there's nothing to diversify. Frontend forwardTarget drops the now-redundant skip-nearest-third hack (the list is already diversified server-side) — plain random-over-unvisited gives the variance now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
715e276c03 | Merge pull request 'feat(tagging): SigLIP concept crops + max-over-bag scoring (#114)' (#153) from feat/siglip-concept-crops into dev | ||
|
bvandeusenandClaude Opus 4.8
|
55fa4656ff |
feat(agent): survive + auto-recover when curator is unreachable
For redeploying curator while away with nobody to restart the agent: - _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx, 401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job fault. On a transport error it hands the job back (best effort) and signals the loop to back off — instead of calling fail(), which would burn the job's server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't re-lease forever. - lease loop retries with capped exponential backoff (poll_idle → 60s) and resets on the first successful lease, so a long outage is gentle and recovery is automatic within ≤60s of curator returning. Sleeps are interruptible so Stop / pool-shrink stays responsive. - AUTO_START env (default on in compose) resumes the worker on container start, so a host reboot / crash-restart (restart: unless-stopped) self-heals with nobody at the desktop. - control UI shows a "waited out" counter + an "curator unreachable, holding work" banner so the recovering state reads as recovery, not failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
c6f38b0dac |
feat(tagging): SigLIP concept crops + max-over-bag scoring (#114)
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray, lactation) that the whole-image SigLIP vector washes out: the GPU agent now embeds figure crops with SigLIP too, stored as kind='concept' regions, and the suggestion rail scores each image as a BAG (whole-image + every concept crop), taking each head's MAX over the bag. The whole-image vector is always in the bag, so this can never score lower than before. Model-agnostic by construction: the server ANNOUNCES the embedding model (HF name + version) in the lease, so the agent loads whatever the heads were trained in and stays in lock-step — a model swap is a server setting + a re-embed migration, never an agent change. - agent: model-agnostic CropEmbedder (torch/transformers get_image_features, fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the back-catalogue backfill never churns figure/CCIP regions; torch cu124 + transformers in the image. - server: lease announces embed_model_name/embed_version; score_image is max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill 'siglip' gates on a missing concept region (drains the back-catalogue, retries failures, no double-enqueue); daily siglip-backfill beat; UI button; /api/ccip/overview reports images_with_concept_siglip. - v1 scope: suggestion rail only — auto-apply stays whole-image (conservative; heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
b91a230f12 |
feat(ccip): automation + reference quality — keep identity flowing hands-free (#114)
Works through the optional CCIP ideas + the "keep moving even if I forget" ask:
AUTOMATION (no button needed):
- Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored
ones retried) on their own; the queue never goes idle waiting for a click.
- CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so
identity tags keep flowing. ON by default (opt-out, like head auto-apply);
ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut),
migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips
already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH
/api/ml/settings; auto_applied count in /api/ccip/overview.
REFERENCE QUALITY (the over-fire root cause):
- character_references now draws ONLY from single-character images — on a
multi-character image the tag is image-level, so every figure would otherwise
pollute each character's prototypes (a 2-char image tagged 'Velma' made
Daphne's figure a Velma reference). This is the contamination behind residual
over-firing.
- Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the
reference load isn't redone on every modal open.
Tests: multi-character image not used as a reference; auto-apply tags a confident
match as ccip_auto.
NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot
interesting content").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
74b7ceaf47 |
fix(tags): return focus to the tag input after reject/un-reject too
Accept already re-focused the tag input (so you keep typing without re-clicking); reject (✗) and un-reject (↶) went straight to the store and skipped it. Route them through onDismiss/onUndismiss which emit 'dismissed', and wire that to focusTagInput in TagPanel — same return-to-input behaviour as accept. TagPanel is shared, so this covers both the image modal and the Explore workspace. The field's mobile-focus guard is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
301f2de989 |
fix(explore): variance + no loop-back on → navigation (#94)
Two reports: → sometimes "loops back", and the walk gets stuck on near-identical images. Cause: forwardTarget picked a uniformly-random neighbour from the 24 NEAREST, so it (a) often landed on an image already in the trail — which snaps the cursor back into history and makes → bounce between visited nodes — and (b) only ever offered near-duplicates. forwardTarget now: excludes already-visited neighbours (→ opens something new, no snap-back), and skips the closest third of the (similarity-sorted) pool so the jump favours the more-varied remainder instead of lookalikes. Neighbour pool widened 24→40 for more variety to browse + jump into. The post-← browser-forward walk through visited crumbs is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
625336b6b4 |
feat(ccip): tunable match threshold, default 0.85 (#114)
Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got
3-10 character guesses dominated by the most-referenced characters (a 27-ref
character clears a low bar on many images). A sweep showed 0.85 collapses the
noise (noisy multi-matches 47→3) while keeping the confident single-character
matches.
- ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image
reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85.
- Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999).
- Slider in the GPU agent card ("Character-match strictness") — tune live, no
redeploy, same observe-and-tune loop as auto-apply.
Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
b7fd69815e |
feat(agent): raise worker cap to 32 + size the HTTP pool for it (#114)
At 8 workers the GPU sat at ~5% util / <5GB VRAM — the pipeline is I/O-bound (downloading + decoding images over HTTP), so the GPU starves until many workers overlap that I/O. Raise MAX_CONCURRENCY 8→32 and make the UI worker control a number input (reaching 32 by ±1 was tedious); the cap is reported via /status so the UI clamps to it. Also size the shared requests pool (pool_maxsize=64) — the default 10 would have throttled 32 workers + spammed "connection pool is full". Verified by running; watch GPU util/VRAM climb as you dial up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
3abbe58450 |
fix(agent): flatten transparency onto white before RGB (#114)
A naive convert('RGB') on a palette-with-transparency image (common: character
PNGs on a clear background) lets PIL guess the transparent pixels — black-ish
artifacts that bleed into the crop + the CCIP embedding (and the "should be
converted to RGBA" warning). to_rgb() composites over white first for a clean,
consistent background; used by both stills and video frames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
4a1a9ec5a7 |
feat(agent): GPU load readout + live worker-count tuning (#114)
Control UI gains what the operator asked for: - GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how hard the card is working while you're at the desktop. - Worker count is now a live − / + control (POST /concurrency), not just an env: the worker is a pool of independent slots (shared model, so slots add concurrent inference, not N× VRAM). Dial up for speed, down to free the card. Replaces pause/resume with Start/Stop + the worker dial. - Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back via client.release() so they're re-picked immediately (pairs with the server recovery sweep). Not CI-tested (agent/ outside CI) — verified by running. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
2cb0427868 |
feat(gpu): fast orphan recovery — graceful release + 60s sweep (#114)
So work an agent orphaned gets picked back up quickly, three layers: - GpuJobService.release(): a graceful agent stop hands its still-leased jobs back to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease. - GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a 60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the queue counts honest even when nothing is leasing. - Lease TTL 300→180s: still well above any single job (a capped-frame video embed is tens of seconds, and a live worker heartbeats), but a hard crash recovers faster once the sweep fires. Tests: release returns-to-pending (token-scoped), recover_orphaned resets only expired leases, release API round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
614b6bc52a |
docs(agent): note the NVIDIA Container Toolkit host prereq
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
7b10f4caab |
fix(agent): cuDNN base image so onnxruntime-gpu loads (#114)
onnxruntime-gpu needs cuDNN 9; the plain cuda:12.4.1-runtime image lacks it (libcudnn.so.9 missing → CUDAExecutionProvider falls back to CPU). Switch to the -cudnn-runtime variant which bundles cuDNN 9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
b6b151a500 |
docs(agent): docker-compose for the GPU agent
compose file (pull the published image, GPU reservation, model-cache volume, .env for the token) so the agent runs with `docker compose up -d` instead of a long docker run. A copy + .env template also placed in ~/Documents/fc-gpu-agent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
9449241fc2 |
ci(agent): publish the GPU agent image (build-agent job)
Build + push fabledcurator-agent alongside web/ml (own CUDA + onnxruntime-gpu image, context=agent/, same tag cadence: main → :main/:latest/:c-<sha>, tag → :<version>). So the operator PULLS + runs it on the GPU machine instead of building locally. README switched to docker pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
8419ebd761 |
feat(agent): desktop GPU agent container — CCIP + figure crops over HTTP (#114)
The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP (lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres stay private. New top-level agent/ (outside CI scope — verified by running it): - fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/ stop (stop frees the card; unprocessed leases expire + re-queue). - fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed. The two API seams to verify against the installed dghs-imgutils (flagged). - fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence → per-frame instances (the bag). - fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client. - fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress + queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements + README (token → build → run --gpus all → Start; CPU-fallback path). This completes the CCIP pipeline end to end: agent produces region CCIP vectors → RegionService stores → matcher suggests characters → rail. Verified by running on the desktop (not CI). README calls out the imgutils API + model-string checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
60f26247e9 |
style: alphabetize ccip_bp import (ruff I001)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
de33bab41c |
feat(ccip): read-only observability API for the crop/CCIP work (#114)
So the work can be checked through an API as the agent fills in vectors (same pattern as /api/heads/metrics): - GET /api/ccip/overview: regions by kind, images with figure CCIP vectors, the per-character reference counts (which characters have enough examples to match on), and the embedding versions present. - GET /api/ccip/images/<id>: that image's stored regions (bbox, frame_time, has_ccip/has_siglip, versions) + the CCIP character matches it would get — for spot-checking detector + matcher output. Read-only, no GPU. (Queue depth is already at /api/gpu/status.) Tests: overview coverage counts + per-character refs; per-image regions + matches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
5faf34a3b5 |
feat(suggestions): overlay CCIP character matches onto the rail (#114)
SuggestionService.for_image now merges CCIP character matches with the SigLIP head suggestions — they're complementary, not exclusive: CCIP is the identity- specialized signal but needs a detected figure; the heads work whole-image but conflate identity with style. Merged by tag: 'both' when they corroborate (higher score wins), 'ccip' / 'head' otherwise. Cheap when no CCIP vectors exist yet (match_image returns early without a figure vector), so it's a no-op until the agent runs. Suggestion.source is now 'head' | 'ccip' | 'both'. Test: a character with a CCIP reference figure surfaces (source='ccip') on a new image whose figure matches. NEXT: the agent container (real CCIP/detector models, hands-on) that produces the vectors this consumes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
d57ca847e7 |
feat(ccip): few-shot character matcher (#114 slice 5)
The server-side brain that turns stored CCIP vectors into character suggestions
— no GPU. character_references() gathers each character tag's prototype vectors
(figure/face-region CCIP embeddings on images carrying that tag); match_image()
cosine-matches an image's figure vectors against every character (multi-
prototype: best over a character's examples), surfacing those above a tunable
threshold as {tag_id, name, category:'character', score, source:'ccip'},
excluding already-applied characters. v1 = cosine on raw CCIP vectors; the exact
CCIP metric/threshold gets validated against the model in the hands-on eval.
Tests (synthetic vectors): same-character match across images, no-match for an
orthogonal figure, already-applied exclusion, no-figure-vectors empty.
NEXT: merge CCIP character suggestions into the rail; the agent container that
actually produces the vectors (hands-on, GPU — not CI-verifiable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
d91eef7a4b |
feat(gpu): GPU agent admin card — token, queue, backfill (#114)
The FC-side control surface the operator asked for: Settings → Tagging → "GPU agent". Generate/reveal/copy/rotate the agent bearer token (with the FC URL to point the agent at), see the live job-queue depth (pending/in-flight/done/ errored, polled), and a "Queue character embedding (CCIP)" button that triggers the library backfill. Plain-HTTP-safe copy (copyText resolves on success, throws on fail). Closes the "how do I get the token in the UI" gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
558d965a1c |
fix(gpu): count backfill enqueues via RETURNING, not rowcount
result.rowcount is unreliable for INSERT…SELECT (returned -1), failing the idempotency assert. Use .returning(GpuJob.id) and count the rows. (run 1652) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
f247f9247c |
style(gpu): ruff — split as-import, dict(rows) over comprehension
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
6cabef07a4 |
feat(gpu): HTTP job API + token auth + backfill — the agent's server side (#114 slice 3b)
The thin HTTP surface over the queue so the desktop agent stays HTTP-only: - Agent endpoints (Authorization: Bearer <token>): POST /api/gpu/jobs/lease (returns jobs + image_url + mime + video frame cadence), /submit (stores regions via RegionService + closes the job; 409 on a stale lease), /heartbeat, /fail. Token validated against AppSetting (mirrors the extension-key pattern, constant-time compare). - Admin (browser): GET/POST /api/gpu/token[/rotate] (generate + show the agent token), GET /api/gpu/status (queue counts), POST /api/gpu/backfill → dispatches enqueue_gpu_backfill. - enqueue_gpu_backfill(task): one INSERT…SELECT enqueues a job per image lacking one for the task (scales to the full library; idempotent). Agent flow: lease over HTTP → fetch pixels via the normal FC image URL → compute on the GPU → submit. Redis/Postgres never exposed. Tests: bearer required (+ wrong-token 401), lease→submit round-trip (region+CCIP vector stored, job done via /status), stale-lease 409, backfill enqueue + idempotency. NEXT: the agent container + control UI, then the CCIP detector/embedder + matcher. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
b735432d02 |
feat(gpu): video-ready regions + the HTTP GPU-job queue engine (#114 slice 3)
Answers "how are videos/all media handled by the GPU worker": a job is per ITEM, but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the existing cadence), each stored with a timestamp — so a video becomes a BAG of frame embeddings (fixes the mean-embedding muddle) instead of one washed-out vector. Stills → frame_time NULL; animated GIF/WebP treated like short video. - image_region.frame_time (migration 0061, not yet deployed so folded in): the source frame's seconds for video/animated media; NULL for stills. RegionService passes it through. A whole frame is just kind='frame'. - gpu_job + GpuJobService (migration 0062): the durable work list that keeps the desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat / complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues; the agent leases+submits over the web API; Redis/Postgres stay private. Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped heartbeat, complete, fail-requeue-then-error. region test now covers frame_time. NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the agent container + control UI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
0ea7ecdea5 |
feat(regions): image_region storage + service for the crop pipeline (#114 slice 2)
The storage backbone both crop jobs write to and read from. image_region =
normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id;
'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP /
Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The
bbox doubles as grounded-tag provenance. Migration 0061.
RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines
don't clobber each other) + get_regions — the GPU agent's results endpoint will
call the writer; the character matcher + bag scorer read. Server-side, no GPU.
Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
e8d3400d22 |
feat(crops): shared crop primitive for the region/crop pipeline (#114)
The trunk of both crop jobs — CCIP figure-crops and SigLIP concept-crops call the SAME crop_region(): normalized-bbox crop with optional context padding, edge-clamping, and the lower-bound size floor (max of a fraction-of-short-side and an absolute pixel floor) below which a region is too small to embed and returns None. Only the proposer (where) and embedder (what) differ; the crop is shared. Pure Pillow — importable + testable anywhere (the GPU agent imports it for the crop step). Unit-lane tests (no DB): region pixels, floor rejection, edge clamp, pad expansion, out-size resize. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
f6e10ccc4f |
fix(explore): render videos with VideoCanvas, not ImageCanvas
The Explore center pane hardcoded ImageCanvas, so a video anchor (e.g. a 169 MB
MP4) tried to load the MP4 into an <img> and showed only the alt text — the
thumbnail worked but the "main image" never rendered. Branch on
mime.startsWith('video/') to VideoCanvas (with mime), exactly like the image
modal. The anchor payload already carries mime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusenandClaude Opus 4.8
|
ad2921b4a0 |
fix(tags): allow creating a same-named character in a different fandom
The autocomplete suppressed the Create row whenever any existing tag matched name+kind — but characters are unique by (name, kind, fandom), so a same-named character in a different fandom (e.g. another "Raven") is a valid distinct tag. allowCreate now always offers Create for the character kind; the fandom picker disambiguates and find_or_create is idempotent if the same fandom is re-picked. The Create row reads "Create another \"Raven\" character (different fandom)" when a same-name character already exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
1463794778 |
feat(heads): auto-apply UI on the Concept-heads card (#114 auto-apply C)
Surfaces earned auto-apply + its observability in Settings → Tagging → Concept heads: - Auto-apply section: an on/off switch (writes head_auto_apply_enabled), the precision-target + min-examples-to-fire tuning inputs, a Preview (dry-run → "would apply N", per-concept chips) and Apply-now button, with live run state. - "How auto-apply is landing": per-concept table from /api/heads/metrics — applied volume, misfires, realized misfire rate (green/amber/red), and missed (under-fires) — the signal to tune the precision target from. store: autoApply(dryRun) / autoApplyStatus() / metrics(). Card polls the sweep to completion, then refreshes counts + metrics. Completes the auto-apply task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
a5a95320df |
fix(test): disable switch explicitly now that auto-apply defaults ON
test_auto_apply_disabled_blocks_real_run assumed head_auto_apply_enabled defaulted False; it now defaults True (opt-out), so a real sweep is accepted (202). Set the switch off in the test to exercise the disabled→400 path. (run 1629) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
9326a82b29 |
fix(heads): .all() before dict() in snapshot_head_metrics
dict(session.execute(...)) on a bare Result invokes the mapping protocol (a Result has .keys() = column names) and subscripts it → "CursorResult is not subscriptable". Materialize with .all() so dict() consumes rows as key-value pairs. The API path already did this; the snapshot task missed it. Caught by test_snapshot_records_timeseries_point (run 1628). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
48c8811d69 |
feat(heads): auto-apply observability + on by default (#114 auto-apply B)
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusenandClaude Opus 4.8
|
01933c5b26 |
style(test): drop unused img in ungraduated-head sweep test (ruff F841)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |