Admin IA overhaul, cap-aware autoscaler, optional ml-worker, hardened tag reset #187

Merged

bvandeusen merged 6 commits from dev into main 2026-07-02 18:01:57 -04:00

Conversation 0 Commits 6 Files Changed 50

+1291 -953

bvandeusen commented 2026-07-02 18:01:49 -04:00

Owner

Copy Link

Copy Source

Six CI-green commits (runs 1863, 1867, 1869–1872):

• 7c19ad9 cap-aware autoscaler (agent 2026-07-02.5): downloaders stop growing (and shed to 3) when the bandwidth cap — not concurrency — is the bottleneck; UI says "holding at the bandwidth cap". Plus the whole-instance tag reset moves to a fenced Danger zone with a preview-derived typed confirm token.
• 19b962f B3 — ml-worker optional: tag_and_embed→embed_image (embed-only role); GPU-queue coordination moved to tasks/gpu_queue.py on the maintenance lane; cpu_embed_enabled toggle (migration 0074); regression test pins that CPU embeds never close ccip/siglip crop work.
• 5b34c92–f08a64f admin IA overhaul (4 waves): Import tab dissolved (filters → Maintenance "Ingestion & filters"; trigger/task-list retired); Maintenance regrouped by system; one extension home in Subscriptions Settings; Activity + Overview get GPU-agent and downloads pulse panels; Subscriptions landing reordered around "what needs me / what came in" (failing-sources card moved up, recent arrivals added, shared retry actions); section-header language unified. Bonus: the single-color import filter is now actually wired (was a dead switch since FC-2), PlaceholderView + the CSV-stub menu item deleted, stale docs fixed.

Deploy notes: migration 0074 (add column, default ON) on web boot; agent re-pull → build 2026-07-02.5. Old-name celery messages (tag_and_embed, ml.enqueue_gpu_backfill, ml.recover_orphaned_gpu_jobs, ml.reprocess_gpu_jobs) in flight at deploy are dropped; the sweeps re-fire under the new names within 60 s/hourly.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

Six CI-green commits (runs 1863, 1867, 1869–1872): - `7c19ad9` **cap-aware autoscaler** (agent 2026-07-02.5): downloaders stop growing (and shed to 3) when the bandwidth cap — not concurrency — is the bottleneck; UI says "holding at the bandwidth cap". Plus the whole-instance tag reset moves to a fenced **Danger zone** with a preview-derived typed confirm token. - `19b962f` **B3 — ml-worker optional**: `tag_and_embed`→`embed_image` (embed-only role); GPU-queue coordination moved to `tasks/gpu_queue.py` on the maintenance lane; `cpu_embed_enabled` toggle (migration 0074); regression test pins that CPU embeds never close ccip/siglip crop work. - `5b34c92`–`f08a64f` **admin IA overhaul (4 waves)**: Import tab dissolved (filters → Maintenance "Ingestion & filters"; trigger/task-list retired); Maintenance regrouped by system; one extension home in Subscriptions Settings; Activity + Overview get GPU-agent and downloads pulse panels; Subscriptions landing reordered around "what needs me / what came in" (failing-sources card moved up, recent arrivals added, shared retry actions); section-header language unified. Bonus: the single-color import filter is now actually wired (was a dead switch since FC-2), PlaceholderView + the CSV-stub menu item deleted, stale docs fixed. Deploy notes: migration 0074 (add column, default ON) on web boot; agent re-pull → build **2026-07-02.5**. Old-name celery messages (`tag_and_embed`, `ml.enqueue_gpu_backfill`, `ml.recover_orphaned_gpu_jobs`, `ml.reprocess_gpu_jobs`) in flight at deploy are dropped; the sweeps re-fire under the new names within 60 s/hourly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

bvandeusen added 6 commits 2026-07-02 18:01:49 -04:00

feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback) ... 7c19ad91ed

Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg
downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real
constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now
gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the
cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint
says 'holding at the bandwidth cap' and /status reports bw_capped, so the
behavior is legible without tests that need the ML stack.

Reset content tagging: stays a FULL-instance reset (operator's call), but now
lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a
preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale
counts are rejected server-side). Copy no longer claims suggestions repopulate:
it says plainly the heads' training examples are deleted and re-tagging starts
fresh. Moved out of TagMaintenanceCard into DangerZoneCard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch ... 19b962f1a7

The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).

Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
  reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
  quick lane — it lived on the 'ml' queue only by module colocation, which made
  the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
  installs keep working): OFF stops the four import hooks queueing embed work
  nothing will consume and no-ops the manual backfill; switch lives on the
  renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
  that removes the container gives those up too.

Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home ... 5b34c9221c

Settings IA per the approved A3 design (the old layout was the two-app merge
fossilized):
- Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual
  /import scans stay API-level; imports arrive via downloads/extension, heal
  via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm
  moves to Maintenance → 'Ingestion & filters' and loads its own settings; the
  import store shrinks to settings-only (no remaining consumers of the
  scan/task-list machinery). Overview's pending banner now points at Activity.
- Maintenance regrouped: Ingestion & filters / GPU agent & embeddings
  (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders,
  Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive
  re-extract demoted last) / Storage.
- One extension home: BrowserExtensionCard moves from Settings → Overview to
  Subscriptions → Settings, above the API key bar it authenticates.
- Single-color import filter WIRED: skip_single_color/threshold existed since
  FC-2 but nothing read them (the audit module's docstring said as much) —
  now enforced on both import paths via the audit's canonical predicate
  (tolerance 30, matching the Cleanup card default; animated images exempt
  like the transparency check). Default stays off; test added.
- Dead weight: PlaceholderView (zero refs) and the permanently-disabled
  'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed
  (celery queue docstring, threshold comment citing retired tasks, ml
  package docstring, HeadsCard 'replaces Camie' blurb).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ia): wave 2 — Activity becomes the whole-app pulse; Overview gets the health strip ... e039689eff

The Activity tab only knew Celery — the GPU agent (the majority of processing)
and the download pipeline were invisible there. Two new self-polling panels:

- GpuActivityPanel: queue depths + triage verdicts (defects / file-ok /
  unprobed, top reason buckets) with a jump to Maintenance -> Failed
  processing. The triage detail refetches only when the error count moves.
- DownloadsActivityPanel: 24h stat chips + failing-source names with a jump
  into Subscriptions.

Both panels join the Activity tab under Queues+workers AND double as the
Overview health strip (side-by-side grid under the Celery summary) — one
component set, so Overview answers 'is everything healthy?' across all
systems. SystemStatsCards reviewed: content still accurate, left as-is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ia): wave 3 — Subscriptions landing answers 'what needs me, what came in?' ... dc7fa6eae2

Daily-use reorder of the Subscriptions tab: needs-attention strip first
(FailingSourcesCard moves up from below the Downloads fold — a broken
subscription was invisible unless you went looking), then a new Recent
arrivals card (real downloads only, no-change scans filtered out, artist
links), then the source list. Both cards render nothing when there's nothing
to say.

Retry logic moves into the downloads store (retrySource / retryAllFailing) so
the needs-attention card and the Downloads maintenance menu share one
implementation — single-retry forces past cooldown, bulk keeps cooldown
enforcement, same tally shape. The card's Logs button deep-links into the
Downloads tab pre-filtered (?source_id now watched, not just read on mount).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

style(ia): wave 4 — one section-header language across every admin surface ... f08a64f7ae

The Subscriptions Settings tab's bare text-h6 headers adopt the same
uppercase accent section-title + hint convention Maintenance/Cleanup use, with
a one-line hint per section (extension / credentials / downloader / external
file-hosts / schedule defaults). Every settings-ish surface now reads
identically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

bvandeusen merged commit bfba8045e4 into main 2026-07-02 18:01:57 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#187