1144 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusen	e52090a4cf	Merge pull request 'Agent sleep mode: empty queue sheds to one downloader, lease poll backs off to 15 min' (#188) from dev into main	2026-07-02 20:37:48 -04:00
bvandeusen	b1cfbcc06a	fix(agent): sleep mode — an empty queue sheds to one downloader and backs the lease poll off to 15 min ... Operator-flagged on the deployed .5 build: the autoscaler grew the pool 1→8 against an EMPTY queue (an empty buffer read as 'GPU starving' regardless of WHY), and every downloader kept polling lease every 10s all night. - New idle signal straight from the lease results: an empty lease sets _idle, any jobs clear it. The occupancy-low branch now distinguishes three cases: queue empty → shed to ONE polling downloader; pinned at the bandwidth cap → shed toward 3; cap headroom + work flowing → grow. - Idle lease polls back off exponentially per downloader to IDLE_POLL_MAX_SECONDS (15 min) and reset the moment work appears — so an idle night costs one HTTP call per 15 min, and new work is noticed within at most ~15 min (operator-accepted trade-off). - UI hint: 'idle — queue empty, lease poll backed off'; /status gains idle. Agent build 2026-07-02.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 20:32:11 -04:00
bvandeusen	bfba8045e4	Merge pull request 'Admin IA overhaul, cap-aware autoscaler, optional ml-worker, hardened tag reset' (#187) from dev into main	2026-07-02 18:01:56 -04:00
bvandeusen	f08a64f7ae	style(ia): wave 4 — one section-header language across every admin surface ... The Subscriptions Settings tab's bare text-h6 headers adopt the same uppercase accent section-title + hint convention Maintenance/Cleanup use, with a one-line hint per section (extension / credentials / downloader / external file-hosts / schedule defaults). Every settings-ish surface now reads identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:56:38 -04:00
bvandeusen	dc7fa6eae2	feat(ia): wave 3 — Subscriptions landing answers 'what needs me, what came in?' ... Daily-use reorder of the Subscriptions tab: needs-attention strip first (FailingSourcesCard moves up from below the Downloads fold — a broken subscription was invisible unless you went looking), then a new Recent arrivals card (real downloads only, no-change scans filtered out, artist links), then the source list. Both cards render nothing when there's nothing to say. Retry logic moves into the downloads store (retrySource / retryAllFailing) so the needs-attention card and the Downloads maintenance menu share one implementation — single-retry forces past cooldown, bulk keeps cooldown enforcement, same tally shape. The card's Logs button deep-links into the Downloads tab pre-filtered (?source_id now watched, not just read on mount). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:52:00 -04:00
bvandeusen	e039689eff	feat(ia): wave 2 — Activity becomes the whole-app pulse; Overview gets the health strip ... The Activity tab only knew Celery — the GPU agent (the majority of processing) and the download pipeline were invisible there. Two new self-polling panels: - GpuActivityPanel: queue depths + triage verdicts (defects / file-ok / unprobed, top reason buckets) with a jump to Maintenance -> Failed processing. The triage detail refetches only when the error count moves. - DownloadsActivityPanel: 24h stat chips + failing-source names with a jump into Subscriptions. Both panels join the Activity tab under Queues+workers AND double as the Overview health strip (side-by-side grid under the Celery summary) — one component set, so Overview answers 'is everything healthy?' across all systems. SystemStatsCards reviewed: content still accurate, left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:45:45 -04:00
bvandeusen	5b34c9221c	feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home ... Settings IA per the approved A3 design (the old layout was the two-app merge fossilized): - Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual /import scans stay API-level; imports arrive via downloads/extension, heal via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm moves to Maintenance → 'Ingestion & filters' and loads its own settings; the import store shrinks to settings-only (no remaining consumers of the scan/task-list machinery). Overview's pending banner now points at Activity. - Maintenance regrouped: Ingestion & filters / GPU agent & embeddings (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders, Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive re-extract demoted last) / Storage. - One extension home: BrowserExtensionCard moves from Settings → Overview to Subscriptions → Settings, above the API key bar it authenticates. - Single-color import filter WIRED: skip_single_color/threshold existed since FC-2 but nothing read them (the audit module's docstring said as much) — now enforced on both import paths via the audit's canonical predicate (tolerance 30, matching the Cleanup card default; animated images exempt like the transparency check). Default stays off; test added. - Dead weight: PlaceholderView (zero refs) and the permanently-disabled 'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed (celery queue docstring, threshold comment citing retired tasks, ml package docstring, HeadsCard 'replaces Camie' blurb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:37:21 -04:00
bvandeusen	19b962f1a7	feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch ... The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:53:08 -04:00
bvandeusen	7c19ad91ed	feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback) ... Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint says 'holding at the bandwidth cap' and /status reports bw_capped, so the behavior is legible without tests that need the ML stack. Reset content tagging: stays a FULL-instance reset (operator's call), but now lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale counts are rejected server-side). Copy no longer claims suggestions repopulate: it says plainly the heads' training examples are deleted and re-tagging starts fresh. Moved out of TagMaintenanceCard into DangerZoneCard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:14:48 -04:00
bvandeusen	a708357436	Merge pull request 'Settings defaults, GPU error tombstones + failure triage/recovery, agent bandwidth cap, approved retirements' (#186) from dev into main	2026-07-02 15:48:48 -04:00
bvandeusen	eaea4308fc	chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved) ... The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:41:24 -04:00
bvandeusen	a7abcc41ca	feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3) ... An errored GPU job's stored reason is a suspicion; the file probe is the verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own probe (sha256 + decode) on each errored image ONCE and writes both verdicts: ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' | 'file_ok', migration 0072). Every classification logs at WARNING so it surfaces in Logs/System Activity. - 'defect' rows are excluded from /retry_errors (re-running a known-bad file burns agent time re-minting the tombstone); response now reports defects_kept and the GpuAgentCard toast says so. - GET /api/gpu/errors: triage view — reason buckets (classify_reason), probe verdicts, per-job detail. POST /errors/triage runs the sweep now. - POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern — delete the defective copy + record (full cascade takes the tombstones too) and re-poll its subscription Source so a fresh copy re-imports and re-enters the pipeline; 'no_source' when nothing pollable resolves. - New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts, reason summary, probe-now, defect list with thumbnails + per-image Recover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:36:02 -04:00
bvandeusen	1f27189b8f	chore: retire ml-backfill-daily beat + the spent purge-legacy action (operator-approved) ... - ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's daily embed backfill for the same NULL-embedding images at ~100x the cost (B1 audit verdict, milestone #124). The backfill TASK stays — the manual /api/ml/backfill button remains the deliberate CPU fallback pending B3. - purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on the live library before removal (A2 audit, milestone #123). Fully retired per rule 22: tile, store action, route, service fn, tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 11:24:08 -04:00
bvandeusen	95d2ae1d58	feat(agent): global bandwidth cap — the agent can't saturate the desktop's network ... One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited; live MB/s dial + net readout in the control UI) is charged by every still download (streamed chunk reads) and every ffmpeg video stream (metered from outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget). Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8 unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's browser included). Capping the aggregate keeps the desktop usable and still beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 11:20:45 -04:00
bvandeusen	31c416bc7b	docs(beat): backfill comments no longer claim errored jobs are retried ... Follow-through on the tombstone rule (09e2772): the hourly/daily backfill entries' comments still described the pre-fix retry-errored behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-01 23:09:26 -04:00
bvandeusen	09e2772628	fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards ... The hourly ccip backfill's skip-list lacked 'error' (and the daily siglip/embed variants re-gated failures on their missing results), so every permanently-bad file got a fresh doomed job each run — ~24 duplicate error rows/day per file, the perpetual 'unprocessable' flood. An errored job is now a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via /retry_errors (an errored back-catalogue needs one button press after a model swap). One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements) runs before every backfill and inside /retry_errors: error rows made moot by a later pending/leased/done row go first, then older duplicates (newest reason survives) — so the error count reads as distinct failing files and a retry can't fan one file out into duplicate pending jobs. /retry_errors now returns {requeued, pruned} and the toast shows both. Poison-loop guards (release and lease-expiry burn no attempts, so a job that stalls its transfer or crashes the agent every time cycled forever — operator-observed jobs 99044/125288/131594/143131): - agent: 3 in-session transient bounces (fetch or submit) → fail with the real reason instead of another release; strikes never count while stopping, and clear on submit success. Agent build 2026-07-02.3. - server: the 60s orphan sweep (statements shared between the beat task and GpuJobService so they can't drift) converts expired leases with >=5 lease grants and pending jobs with >=10 to 'error', preserving the last stored failure reason. Backstops old agent builds. Tests: tombstone rule across all three backfill variants, moot-row pruning, poison conversions, and the extended /retry_errors dedupe contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-01 22:52:38 -04:00
bvandeusen	3d6201734c	fix(settings): maintenance tiles start collapsed; remember manual open state ... GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head existed, TagEvalCard whenever a persisted run existed — so a fresh Settings load greeted the operator with several tiles already expanded. All three now force-open only while their task is actually running (the #877 resurface behavior on the busy-driven tiles is untouched). MaintenanceTile additionally persists MANUAL expand/collapse per tile in localStorage, so the section reloads the way the operator left it; a forced open while a task runs stays transient and is never saved as a preference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-01 22:28:52 -04:00
bvandeusen	1b1d3732dc	feat(agent): store ffmpeg's actual failure reason in the job's error field ... sample_frames_from_url now returns (frames, reason) — reason carries the SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found", or the timeout) instead of only logging it agent-side. The worker folds it into the failure it reports, so curator's GpuJob.error reads e.g. no frames sampled from video — ffmpeg exit 183: moov atom not found ... instead of the bare "(unprocessable)". The errored-jobs list becomes self-describing: after a retry sweep, surviving errors name their real defect without needing the agent log. Return-value plumbing (not shared state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-01 22:01:20 -04:00
bvandeusen	d9354ac1e1	Merge pull request 'Agent: short-video sampling fix + "Retry errored jobs" recovery action' (#185) from dev into main	2026-07-01 21:13:48 -04:00
bvandeusen	686808d3f3	feat(gpu): "Retry errored jobs" — scoped requeue of errors only ... After an agent-side fix (e.g. the short-video sampler), the errored jobs (~2.8k) have exhausted their 3 attempts and stay parked: backfill skips images that already have a job, and /reprocess is the nuclear option (it resets the 179k DONE jobs too). There was no way to re-run just the errors. POST /api/gpu/retry_errors resets every status='error' job (all task types) to pending with attempts=0 and the stored error cleared — a small inline UPDATE that returns {requeued: n} so the UI toast can show the count. UI: a "Retry errored jobs" button on the GPU-agent card, right under the queue tiles; disabled when errored==0. With the agent now logging ffmpeg's stderr on failure, retrying also reveals which errors were real vs victims of the fps-filter bug. Test: retry_errors requeues the errored job (fresh attempts, error cleared) and leaves done work untouched; asserts via column selects (Core-DML gotcha), not ORM refresh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-01 21:09:07 -04:00
bvandeusen	3a683d7feb	fix(agent): short videos failed as "unprocessable" — fps filter emits 0 frames ... Root cause of the "no frames sampled from video (unprocessable)" flood (operator-flagged 2026-07-02, whole 62k-70k image block + others): the sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4) frames — which is ZERO for any clip shorter than ~2s. Short animation loops (0.5s, 1.75s — verified against two originals from different artists) are complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames, exited 0, and the agent failed the job as unprocessable. Long videos worked, so only the short-clip class flooded. Fix: sample with select ("first frame always, then one per interval of timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range yuv420p sources don't trip the mjpeg encoder's full-range strictness (secondary failure observed on a 4440x2760 clip). Verified locally against both failing originals (frames extracted, PIL-clean) and a synthetic 15s video (4 frames at t=0/4/8/12 — long-video behavior unchanged). Observability (why this hid for weeks): ffmpeg's stderr was discarded, so every failure logged only "no frames sampled". stderr now goes to a temp file and its tail is logged on any produced-no-frames/timeout failure — the log names the actual ffmpeg reason from now on. Also: frames written before a mid-stream ffmpeg error are now kept (partial > nothing). VERSION → 2026-07-02.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-01 21:00:29 -04:00
bvandeusen	1d48770793	Merge pull request 'Agent: fix Status freeze — conchint.textContent destroyed #capn (root cause)' (#184) from dev into main	2026-07-01 20:46:28 -04:00
bvandeusen	0a618db10c	fix(agent): Status froze after one update — conchint.textContent destroyed #capn ... THE root cause of "the Status section doesn't update" (chased across several rounds; the backend was always healthy). `#capn` (the max-concurrency number) was nested inside `#conchint`: <div id=conchint>… · max <b id=capn>8</b></div> and applyStatus() ran, every call: `capn.textContent=CAP` AND `conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces ALL of conchint's children — destroying the <b id=capn> node. So: call 1: capn exists → tiles update → conchint.textContent DELETES capn call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) → applyStatus throws on its FIRST line → aborts before any tile → frozen. This is exactly the observed "ticks a couple times then freezes", and why /gpu + /logs (which never touch capn) kept updating fine. The capn write was redundant anyway — conchint.textContent already renders the max. Remove the nested <b id=capn> element and the capn.textContent line; the hint still shows "· max N". VERSION → .10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 20:19:37 -04:00
bvandeusen	489e6aaaee	Merge pull request 'Agent: server-side throughput rates + killable-on-stop ffmpeg' (#183) from dev into main	2026-07-01 20:01:00 -04:00
bvandeusen	713a11e394	fix(agent): server-side rate metrics + killable-on-stop ffmpeg ... Two follow-ups from live debugging of "work/min never populates" and "stopped never reached". 1) jobs/min + downloads/min are now computed in the BACKEND on a fixed cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were derived client-side from poll deltas with a dt<30s guard — but a backgrounded/unfocused browser tab throttles its timers to ~1/min, so every delta exceeded 30s and the guard blanked the rates forever. A server-side rate is independent of how often the tab polls. Frontend just displays s.jobs_per_min / s.downloads_per_min. VERSION → .9. 2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see the stop signal until ffmpeg returned, so Stop detached still-running threads and work kept flowing long after — "stopped" that wasn't really stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a `should_stop` callback every 0.5s, terminating (then killing) the process at once on Stop or the per-video timeout. A stop-killed job is handed back (transient), not failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:51:26 -04:00
bvandeusen	ed20df905b	Merge pull request 'Agent: DRY pass + Status rate-metrics + real start/stop state machine' (#182) from dev into main	2026-07-01 16:30:32 -04:00
bvandeusen	6282e753a9	fix(agent): real start/stop state machine — kill the stuck "stopping" pill ... The Status pill hung on "stopping" forever (operator-flagged 2026-07-01). Root cause: the backend had no lifecycle state — status() only returned running/stopped — so the UI FABRICATED "stopping" in JS as `!running && active>0`. That pill only cleared when the backend's `active` counter hit 0, but stop() (a) blocked the HTTP handler on lease-release calls to curator and (b) left `active>0` whenever a consumer wedged mid-submit/release to an overloaded curator → "stopping" that never resolved. Give the backend a real, truthful state it drives itself: stopped → starting → running → stopping → stopped - start(): → starting; a downloader flips it to running on its FIRST successful lease (so "running" means curator is actually answering, not just "Start was clicked"). If curator's down it honestly stays "starting". - stop(): → stopping; returns immediately (no handler block). A background monitor waits for the worker threads to actually exit, releases leases, then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can NEVER hold the UI in "stopping" again. In-flight work is handed back safely. - Buttons follow the real state (Start only from stopped; both disabled through the transition), so you can't fight a transition. - Log every Start/Stop button press (routes) and every transition (worker), so the Logs panel shows exactly what each button did. Frontend now trusts s.state (drops the active>0 hack); VERSION → .8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:20:12 -04:00
bvandeusen	91ea06be79	feat(agent): Status shows smoothed jobs/min + downloads/min (replace jumpy gauges) ... The buffer / on-GPU / downloader counts flip many times a second, so a 3s status poll only ever samples noise — the tiles looked frozen (same value twice) or random (wildly different), reading as "the Status section doesn't update" when the backend was in fact live (operator-flagged 2026-07-01). Replace the three instantaneous gauge tiles with two derived RATE tiles: - jobs / min — GPU throughput, from the monotonic `processed` counter - downloads / min — fetch throughput, from a new monotonic `downloaded` counter (bumped when a job is decoded into the buffer) Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped at 0 (agent restart resets the counters), and skip a backgrounded-tab gap. The still-useful instantaneous state is demoted, not lost: buffer stays as the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited out` (transient) gets promoted to a tile. backend: worker.status() gains `downloaded`; `_bump(downloaded=)`. frontend: retiled Status + rate math in applyStatus; VERSION → .7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 16:01:29 -04:00
bvandeusen	98b2ac90dd	refactor(agent): DRY pass on the GPU agent worker package ... Consolidate genuine duplication in agent/fc_agent into single-source helpers (behavior-preserving; DRY Pass process #594): worker.py - _fail(jid, image_id, exc, verb) — 4 terminal "fail this job" blocks (downloader HTTP-fault + decode, consumer non-transient + generic). - _release(job_ids) (was _release_owned) — the one lease hand-back path; 6 inline release([jid])+unhold sites now route through it. - _stopped(stop_evt) + _abort_if_stopped(jid, stop_evt) — 4 stop-check -and-release blocks and every bare stop-check. - _timed(stage) contextmanager — ~8 monotonic()/_record() timing pairs; records only on clean exit, matching the old skip-on-raise behavior. - _ewma(prev, x, alpha) module fn — 3 EWMA updates in the autoscaler. client.py - _submit(path, payload) — submit / submit_embedding (retrying session). - _post_quiet(path, payload) — heartbeat / fail / release fire-and-forget. detectors.py - Proposers._top(detector, image, cap) — merges components() and panels(). config.py - _bool_env(name, default) — auto_start / auto_scale env parsing. Left alone (recorded): the xyxy→norm-xywh conversion duplicated across models.py/detectors.py (2 copies, independent wrapper modules — sharing would couple them), and the _ensure_embedder/_ensure_proposers pair (same lock shape, different concepts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 14:53:58 -04:00
bvandeusen	5ba9871ef0	Merge pull request 'fix(agent): stream videos via ffmpeg-from-URL (no full download) — env-agnostic' (#181) from dev into main	2026-07-01 14:20:53 -04:00
bvandeusen	8a0237eeea	fix(agent): stream videos via ffmpeg-from-URL instead of downloading the whole file ... The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE file into memory (r.content) just to sample a few frames, which buffered ~1GB in RAM and — on any slow/contended media store — got cut off mid-download (ChunkedEncodingError), failed, and re-leased forever. Measured the media read at ~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak helps; the file simply shouldn't be fully downloaded. Environment-agnostic fix (works for any deployment, completes even when slow): - media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL. It Range-reads only the video index + up to max_frames of content — never the whole file — and reconnect flags resume a dropped transfer instead of failing. Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over speed. Removes the bytes-based sample_frames (dead once videos stream). - worker._download_decode: videos now stream (no fetch_image, no RAM blowup); stills still download+decode. On an ffmpeg miss, probe curator liveness (client.is_reachable) → fail the job if curator is up (unprocessable file, stops the infinite re-lease) vs release if curator is down (transient, survives a redeploy). Auth header passed so it works whether or not /images is gated. Build marker 2026-07-01.6. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 14:15:22 -04:00
bvandeusen	2a820d0848	Merge pull request 'fix(agent): log the real fetch/submit failure reason (diagnose #1225)' (#180) from dev into main	2026-07-01 12:37:45 -04:00
bvandeusen	aa0605585b	fix(agent): log the REAL fetch/submit failure reason, not "curator unreachable" ... "curator unreachable" was printed for every transient error, hiding whether a single file's transfer stalled (ReadTimeout — curator is up, that stream is slow) or curator itself is down (ConnectTimeout/ConnectionError) or errored (HTTP 5xx). Those need completely different fixes, and we've been diagnosing the download slowness blind. Add _transient_reason(exc) → a specific label (HTTP <code>, else the exception class: ReadTimeout / ConnectTimeout / ConnectionError / …) and use it in both transient paths: - downloader: "fetch failed job <id> (image <id>, ReadTimeout) — released, backing off" - consumer: "submit failed job <id> (<reason>) — released, re-lease later" Now the logs say which failure it actually is (and which image), so we can tell a slow/stalled transfer apart from an unreachable curator. Build marker 2026-07-01.5. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 12:33:51 -04:00
bvandeusen	2f9aa3d86c	Merge pull request 'perf(web): 4 MiB file streaming + 4 hypercorn workers (fix 40s downloads)' (#179) from dev into main	2026-07-01 11:55:35 -04:00
bvandeusen	0fe1674753	perf(web): stream files in 4 MiB chunks + 4 hypercorn workers (fix 40s downloads) ... The image library is on a CIFS/SMB share (mounted rsize=4 MiB, actimeo=1), and Quart's FileBody streams in 8 KiB chunks — so serving one large original was ~19k network round-trips to the storage server, i.e. 30–58s per download (operator-flagged). That's what starved the GPU agent (constant "curator unreachable" backoff) AND slowed the browser: every byte is read off CIFS and streamed through the Python app (no reverse-proxy sendfile), and only 2 hypercorn workers meant the agent + the browser's thumbnail grid queued behind each other. In-container fix, no new service: - Raise FileBody.buffer_size 8 KiB → 4 MiB in create_app, matching the mount's read size: one round-trip per read, ~500× fewer. buffer_size is the MAX read so small thumbnails still read in one gulp, and Range/mime/ETag/conditional handling lives on Response — all preserved. Guarded so a Quart-internal change can't break boot. - HYPERCORN_WORKERS default 2 → 4 so concurrent /images requests stop queuing. Expected: large-file transfers drop from ~40s toward link speed (a few seconds) for the agent and the browser. See issue #1223. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 11:51:09 -04:00
bvandeusen	560a5000a2	Merge pull request 'gallery: sort by earliest post date across all posts (new default)' (#178) from dev into main	2026-07-01 10:54:41 -04:00
bvandeusen	c22f37d64d	feat(gallery): sort by earliest post date across all posts (new default) ... The gallery's newest/oldest sort keys off image_record.effective_date = COALESCE(primary post's post_date, created_at). The primary post is often the repost/download the file came from, so the grid led with download dates rather than when content was first posted (operator-flagged). Add a second materialized sort key, earliest_post_date = MIN(post_date) across ALL of an image's provenance posts (every post it appears in), else created_at — the original publish date. Mirrors the effective_date pattern so the sort stays a forward index scan. - alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill created_at baseline then MIN over image_provenance ⋈ post. - importer: recompute earliest_post_date whenever a dated post is linked (MIN over the image's provenance, which now includes the just-added row). - gallery_service: new sorts posted_new / posted_old key off earliest_post_date; cursor + year/month grouping follow the active column transparently. - api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads with original publish date. newest/oldest (effective_date) still available. - frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post date); existing effective-date sorts relabelled "Newest/Oldest added". - tests: service test asserts posted_new/posted_old key off earliest_post_date; frontend default-sort omission test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 10:46:09 -04:00
bvandeusen	7ddad231f8	Merge pull request 'agent: stop downloader pool stampeding a slow curator (congestion collapse)' (#177) from dev into main	2026-07-01 10:45:48 -04:00
bvandeusen	ccbb5cbc9e	fix(agent): stop the downloader pool stampeding a slow curator (congestion collapse) ... Operator hit an outage after the machine slept overnight: the agent showed "curator unreachable" in a loop while curator's API (lease) was actually fine and the browser could still load images — just slowly. Root cause is a feedback loop in the new pipeline: every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for that reason — so the agent piled on more concurrent large-file GETs, saturating curator's web workers + NFS, which slowed curator (and its browser) further and produced more failures → more downloaders. Classic congestion collapse. - Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing — the empty buffer is caused by failures, not the GPU starving. It ramps back up only once downloads succeed again. - DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving path is too many; 8 keeps a fast GPU fed without stampeding curator. - fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a large-but-flowing download still completes, but a stuck/dead connection fails in 60s instead of hanging a downloader for 3 min and piling up stuck requests. Build marker 2026-07-01.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 10:33:18 -04:00
bvandeusen	a78f7eaace	Merge pull request 'explore: more variance in the related rail (stronger MMR diversification)' (#176) from dev into main	2026-07-01 00:50:59 -04:00
bvandeusen	ef3318aac1	feat(explore): more variance in the related rail (stronger MMR diversification) ... Operator wants the Explore "related" rail to span more — the #1188 diversifier was tuned conservatively. Push all three knobs so it reaches further across clusters instead of clumping near the anchor: - MMR lam 0.55 → 0.40 — weight the diversity penalty harder (the main dial). - candidate pool min(200, max(limit*5, 60)) → min(400, max(limit*8, 100)) — a wider nearest-cosine pool so MMR has genuinely distinct neighbourhoods to pick from, not just the near-dupes. - pHash dup_threshold 6 → 8 — collapse more near-duplicate reposts/clones, freeing rail slots for distinct picks. Still deterministic (same set per image, just more spread) and relevance-anchored via the lam*sim-to-anchor term. Backend-only; no migration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 00:46:56 -04:00
bvandeusen	71337b0ba4	Merge pull request 'agent: temporal video dedup — drop near-duplicate frames before the GPU' (#175) from dev into main	2026-07-01 00:38:53 -04:00
bvandeusen	7cdce0c474	feat(agent): temporal video dedup — drop near-duplicate frames before the GPU ... Near-static videos are the dominant GPU load: sampled into up to 64 frames, each re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU perceptual-hash frame dedup upstream of the GPU so the redundant frames are never processed at all (not just their embeds). - media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy keep — a frame survives only if its hash differs from every kept frame by >= min_distance bits (Hamming). A static run collapses to one frame; genuinely distinct scenes all survive. Order + frame_time preserved. - Called in worker._download_decode right after sample_frames, so it runs in the decode stage on the downloader thread (CPU) — the GPU consumers only ever see deduped frames, and buffered video items shrink (less RAM too). - Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M` when it drops any, so video load reduction is visible. Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal axis. Build marker 2026-07-01.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 00:35:03 -04:00
bvandeusen	3996205f3b	Merge pull request 'agent: dedupe near-duplicate crops before the SigLIP embed' (#174) from dev into main	2026-07-01 00:24:39 -04:00
bvandeusen	eaae896858	feat(agent): dedupe near-duplicate crops before the SigLIP embed ... Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes, but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept ∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping booru head classes on one head), embedding the same region twice and burning a slot against max_regions. Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass over the pending (crop, template) list right before embed_batch — drop boxes that overlap ≥ iou within the same kind, keep the highest score. The high threshold is deliberate: it collapses only true near-identical boxes while preserving intentional nested crops across scopes (a whole figure vs a small head component sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU (≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 00:20:40 -04:00
bvandeusen	9f9db01456	Merge pull request 'agent: download/GPU producer-consumer pipeline + detector fuse fix' (#173) from dev into main	2026-06-30 23:39:25 -04:00
bvandeusen	afef95a87d	feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash ... The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms), so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit) left the fast GPU idle during every download. Rearchitect worker.py into a producer/consumer pipeline: downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU consumers (detect+embed→submit) - Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a 2nd consumer if it has util/VRAM headroom and lifts throughput, else trim). - Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping RAM + lease look-ahead. VRAM pressure sheds a consumer immediately. - Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU; curator's 180s TTL would otherwise reclaim them mid-buffer). - Preserves all resilience: lease exp-backoff, submit-path retry (#169), release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools and releases every held lease at once (single held-set as source of truth). - Consumers SHARE one embedder + proposers instance (a 2nd consumer adds concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots). - UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU · processed · errors, a buffer-occupancy meter, and a consumers/waited-out line; the dial now tunes downloaders. Build marker 2026-07-01.1. Also fix the operator-flagged detector warning: yolo11n + the comic-panel model threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load- time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3 crop proposers and spamming the log per image. Disable that fusion (unfused inference is correct, marginally slower) and permanently self-disable a proposer on the first inference failure instead of re-throwing forever. Refs milestone 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 23:34:12 -04:00
bvandeusen	216b7fc743	Merge pull request 'Agent: bound video GPU work (early-exit frame loop at max_regions)' (#172) from dev into main	2026-06-30 22:57:33 -04:00
bvandeusen	83f1070a11	fix(agent): bound video GPU work — early-exit the frame loop at max_regions ... Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the SUBMIT — the GPU work was already spent. Break out of the frame loop once accumulated regions reach max_regions, so a long video costs ~a few frames of GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it mean-pools all frames and returns before this loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 22:52:32 -04:00
bvandeusen	fbb76e6f36	Merge pull request 'Agent: huge-image load + figure/region caps + stale-active fix' (#171) from dev into main	2026-06-30 22:47:48 -04:00