Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to
its live pool; `POST /api/system/workers/<name>` changes it.
NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because
restarting a dead worker needs a socket the web container deliberately does
not have. That holds for restarting a CONTAINER; it does not hold for
changing how much work a RUNNING worker does. celery's pool_grow /
pool_shrink / add_consumer / cancel_consumer send a message over the Redis
the app already uses, and the worker resizes itself. No new privilege, no new
surface, and the security question that deferred this is never raised.
PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a
restart drops every lane to its env concurrency — so a UI that only pushed
would lose the setting on the next deploy with nothing to show for it (lesson
#4202). Storing alone would describe nothing until something restarted. A
failed PUSH is not a failed setting: 200 with `applied: false` and a reason,
so the UI says "saved, not yet live" rather than "that didn't work". Step 3's
reconcile carries it when the lane answers again.
PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH
destination, so while `worker` runs `replicas: 2` a single delta from an
aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY
means — one process's pool — and each replica is driven to it from its OWN
current size, so replicas that drifted apart converge rather than moving in
lockstep. I wrote this wrong first: the docstring claimed per-replica while
the code computed one delta from the max across replicas. LaneLiveState now
carries `pools` per hostname and exposes `pool` as a property.
A replica already at the target is sent nothing at all — the reachable fixed
point step 3's periodic reconcile needs, or it re-issues a grow of zero every
tick forever (lesson #4183). A replica that answered inspect but not stats is
NAMED in the error rather than skipped silently, since otherwise it would run
at a size the UI claims it does not.
`present=False` is not "zero slots", it is "nothing answered" — kept distinct
throughout, because step 3 skips an absent lane rather than correcting it.
/workers now also reports pool size (from `insp.stats()`) and RESERVED count.
Celery prefetches, so tasks that have left the Redis list but not started are
invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker
memory. `pending` is depth + reserved. The UI is misleading without this and
step 7's autoscaler would be simply wrong.
Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES,
whose own comment admitted the coupling ("must match celery_app.task_routes")
and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed.
Now derived from LANES. The rendered order changes to lane grouping, which is
the better shape for a lane-oriented UI.
Separate blueprint rather than folding into system_activity, which states in
its first line that it is read-only and answers a different question — its
/workers is keyed on celery HOSTNAME and reports which nodes answered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
The single endpoint the nav indicator and the System page will both read.
Composing a verdict is this module's job, not the UI's.
Two kinds of part, answered differently. LEARNED — celery roles and the GPU
agent, out of service_seen, where the question is "how long since it checked
in" and the answer can be "it has not". PROBED — Postgres and Redis, always
expected, never learned, because a last-seen for them would be actively
misleading: that Redis answered thirty seconds ago says nothing about now.
**The endpoint must never fail because something it checks has failed.** That
inversion is easy to write by accident and it destroys the feature exactly
when it is needed — a 500 when Redis is down instead of `redis: down`. Every
probe is wrapped, every wait carries a deadline (rule 156), and the roster
refresh swallows its own errors. The worst case is a part reported `unknown`,
which is a true statement about the system.
Postgres is probed first and gates the rest, because if it is unreachable
nothing else can be read — and "the database is down" is the most useful
single thing this can ever say.
The staleness thresholds are the design risk, not the code, and they are
deliberately generous: 90s to doubt, 300s to disbelieve. The constraint is a
deploy rather than a crash — `docker compose up -d` rolls start-first, so a
role is briefly served by two containers and then by neither while the old one
drains. Thresholds tight enough to catch a crash in seconds would paint the
page red on every update, and an alarm that cries wolf on every deploy is one
nobody reads. Tune down only after watching a real deploy pass through. The
numbers ship in the response so the UI can explain a `stale` without keeping a
second copy of them.
States are described in sentences rather than left as chips: "Scheduler has
not checked in for 6 min — treat it as stopped" is what someone needs at the
moment they are deciding whether to go and open Portainer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding +
trained head' spine; the operator accepted the tagging system and dropped the
harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint,
tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry,
TagEvalRun model + table (migration 0073), and its tests.
The eval's data loaders + metric helpers were NOT eval-specific — the nightly
heads trainer runs on them — so they moved verbatim to
services/ml/training_data.py (heads.py import updated; behavior unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
So the work can be checked through an API as the agent fills in vectors (same
pattern as /api/heads/metrics):
- GET /api/ccip/overview: regions by kind, images with figure CCIP vectors, the
per-character reference counts (which characters have enough examples to match
on), and the embedding versions present.
- GET /api/ccip/images/<id>: that image's stored regions (bbox, frame_time,
has_ccip/has_siglip, versions) + the CCIP character matches it would get — for
spot-checking detector + matcher output.
Read-only, no GPU. (Queue depth is already at /api/gpu/status.)
Tests: overview coverage counts + per-character refs; per-image regions + matches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The thin HTTP surface over the queue so the desktop agent stays HTTP-only:
- Agent endpoints (Authorization: Bearer <token>): POST /api/gpu/jobs/lease
(returns jobs + image_url + mime + video frame cadence), /submit (stores
regions via RegionService + closes the job; 409 on a stale lease), /heartbeat,
/fail. Token validated against AppSetting (mirrors the extension-key pattern,
constant-time compare).
- Admin (browser): GET/POST /api/gpu/token[/rotate] (generate + show the agent
token), GET /api/gpu/status (queue counts), POST /api/gpu/backfill → dispatches
enqueue_gpu_backfill.
- enqueue_gpu_backfill(task): one INSERT…SELECT enqueues a job per image lacking
one for the task (scales to the full library; idempotent).
Agent flow: lease over HTTP → fetch pixels via the normal FC image URL → compute
on the GPU → submit. Redis/Postgres never exposed.
Tests: bearer required (+ wrong-token 401), lease→submit round-trip (region+CCIP
vector stored, job done via /status), stale-lease 409, backfill enqueue +
idempotency.
NEXT: the agent container + control UI, then the CCIP detector/embedder + matcher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
trigger enqueues a Celery scan_directory task and returns 202 with the
celery task id (the actual scan happens asynchronously; status surfaces
via /api/import/status and /api/system/stats). list_tasks supports
status filter and cursor pagination. retry-failed resets failed tasks
to queued and re-enqueues. clear-completed deletes finished tasks
older than the supplied age (used by the Settings UI's cleanup control).
mode='deep' is explicitly rejected with 400 until FC-2d ships it; the
frontend's UI in FC-2a only sends 'quick'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Missed in the Task 11 commit because of a stale-cache Edit failure.
Routes /api/settings/import and /api/system/stats are now actually mounted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Thin async blueprint delegating to TagService. Returns 400 with the
TagValidationError message on bad kind/fandom combos so the frontend can
surface the reason. List/add/remove endpoints scoped under
/api/images/<id>/tags follow REST conventions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Thin Quart blueprint that delegates to GalleryService. Cursor-based scroll
returns images + a date_groups summary so the SPA can render year-month
headers without re-grouping client-side. Timeline returns year-month
buckets for the sidebar jump nav. Jump returns a cursor positioned at a
specific year-month so the gallery can scroll to historical periods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Config reads from env vars (12-factor); .env.example documents defaults.
Health endpoint is liveness-only (no DB/Redis touch). Test added for CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
