Compare commits

...

9 Commits

Author SHA1 Message Date
bvandeusen 57e52433d0 feat(agent): idle-unload GPU models to free VRAM when the queue is idle
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 4m3s
The SigLIP embedder + YOLO proposers load lazily then stay resident for the
container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of
VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep
mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded.

Add a monitor thread that unloads the torch-owned models after
cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with
the GPU genuinely idle (active==0, buffer drained, no job completed in the
window), then torch.cuda.empty_cache() to hand the blocks back to the driver.
They reload lazily on the next job via the existing _ensure_embedder /
_proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in
/status (models_loaded) and the agent UI pipe line; the VRAM meter drops too.

Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident
(no clean unload API) — idle VRAM drops substantially, not to zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
2026-07-17 12:57:31 -04:00
bvandeusen ec66ea5f83 refactor(ui): settings-card primitives + fix threshold clamp / card misgroup (#161)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m52s
extension / lint (pull_request) Successful in 10s
Tier-3 frontend DRY for the ML settings cards, plus the F-D2 clamp bug and the
F-D3 card misgrouping.

New primitives (components/common + composables):
- <SettingToggleRow> — the accent-icon + .fc-section-h label + right-aligned
  switch row (HeadsCard x3, CropProposersCard). iconColor prop absorbs the
  on/off dim.
- <SettingNumberField> — compact numeric field that CLAMPS to [min,max] on
  commit. This fixes F-D2: HeadsCard/CropProposersCard previously sent
  Number(raw) straight to the API, so an out-of-range threshold bounced off the
  400 validator (only TranslationCard clamped). density prop for the grid cards.
- useSettingSave(patchFn) — the busy + patch + toast + revert-on-failure flow
  each card hand-rolled (HeadsCard x6 handlers, CropProposersCard, MLBackfillCard,
  VideoEmbeddingCard). Returns ok/false for the optimistic-switch revert.

Adopted in HeadsCard, CropProposersCard, MLBackfillCard (handler only — its
plain labelled switch is a different affordance), VideoEmbeddingCard.

F-D3: MLThresholdSliders.vue actually rendered a "Video embedding" (frame-
sampling) card but sat under "Tagging → Suggestion thresholds". Renamed it
VideoEmbeddingCard.vue and moved it to the "GPU agent & embeddings" section.

Left deliberately (over-DRY guard): TranslationCard uses an inline error ALERT
(not a toast), already clamps its confidence with a NaN fallback, and lives on
the ImportStore — a genuinely different save pattern, so forcing it onto
useSettingSave would change its UX.

Behaviour-preserving refactor; CI has no Vue type-check so this needs a live
UI pass (toggles persist + revert on failure, thresholds clamp on blur, video
card now under Embeddings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 22:28:47 -04:00
bvandeusen e92570a31e refactor(extension): DRY the web-root transform + cookie-export flow (#161)
CI / lint (push) Successful in 2s
extension / lint (push) Successful in 11s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m53s
Behavior-preserving (extension JS has lint-only CI + your manual test):
- api.webRoot() single-sources the baseUrl→web-root transform (strip trailing
  slash + /api) that was copy-pasted in background.js's self-update check and
  OPEN_ARTIST_PAGE, whose comments even cross-referenced each other.
- exportPlatformCookies(key) shares the extract→verify→upload spine between
  EXPORT_COOKIES (single) and EXPORT_ALL_COOKIES; it returns a structured
  outcome so each caller keeps its own response/skip messages verbatim.
- popup.js mutedNote(text) replaces the "centered muted note" div hand-rolled
  in the platform-loading, sources-loading, and empty-sources renderers.

No version bump — no behavior change, so it rides the next real ext release
rather than forcing an AMO re-sign + reinstall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:57:14 -04:00
bvandeusen a2d1ed935d refactor(ui): DRY the settings-card CSS tokens + fix unstyled headers (#161)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m57s
- Promote .fc-section-h to a global token (app.css). It was copied identically
  into 4 cards, and TranslationCard used the class with NO local def — so its
  section headers rendered unstyled. Now fixed everywhere.
- Promote .fc-good / .fc-weak status colours to globals; delete the local copies
  in the GPU/heads cards. (.fc-ok stays local — divergent: on-surface in
  HeadsCard vs success in QueuesTable. .fc-bad stays — different name.)
- Delete 10 identical local .fc-muted redefinitions that crept back after the
  2026-06-09 sweep; the global utility already covers them.
- DbMaintenanceCard: opacity:0.6 muted text → the .fc-muted token (the exact
  anti-pattern that token's comment forbids).
- HeadsCard: collapse byte-identical ratePct() into pct().

CSS-only + one template class swap; no logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:51:23 -04:00
bvandeusen 05df51b749 fix(ml): drop unnecessary quotes on MLSettings.load annotations (UP037)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m51s
Python 3.14 evaluates annotations lazily, so the self-referential return
type needs no forward-ref quotes — matches ImportSettings.load. Fixes the
ruff lint lane on the DRY-pass push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:49:30 -04:00
bvandeusen 099e1e664c refactor(patreon): DRY the campaigns-API request (#161)
CI / lint (push) Failing after 2s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / integration (push) Successful in 4m0s
_lookup_via_api and resolve_display_name shared ~90% of their body (same
endpoint, params, headers, error handling — differing only in which field
they pluck from data[0]). Extract _campaigns_api_first(vanity, cookies_path)
-> dict|None; callers pluck the campaign id vs the display name. Return-value
behavior preserved (the display-name path additionally gains the helper's more
granular warning logs). Covered by the existing test_patreon_resolver.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:34 -04:00
bvandeusen c87f8a1bb3 refactor(maintenance): DRY the recover_stalled head-run twins (#161)
recover_stalled_head_training_runs and recover_stalled_head_auto_apply_runs
were near-exact copies (coalesce-flip + keep-last-N prune, differing only in
model + two constants). Extract _recover_stalled_runs(model, stall_minutes,
keep_runs, label); the two tasks become thin wrappers. The other two recover
tasks are deliberately NOT folded in (library-audit has no prune tail; backup
uses a single started_at cutoff). test_recover_stalled_head_runs.py covers
both wrappers (stalled→error, fresh survives) — previously untested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:34 -04:00
bvandeusen 666b3a2ec8 refactor(ml): DRY pass — shared sweep helpers + table-driven settings (#161)
Consolidate duplication accrued across the ML tagging + settings backend,
behavior-preserving (over-DRY guard applied — the three auto-apply sweep
BODIES stay separate; only their shared inner helpers are extracted).

- _sigmoid / _conflict_scores / _insert_presentation_review (heads.py): the
  score→prob transform (6 inlined sites), the presentation conflict signal
  (2 sites), and the ring-loud PresentationReview insert (2 sites, single-
  sourced so the mode column can't drift on the shared composite PK).
- _applied_or_rejected (training_data.py): the per-tag "applied ∪ rejected"
  skip-set, byte-identical at 3 sweep sites (heads.py x2, tasks/ml.py ccip).
- ccip sweep divergence fixes: import ccip._FIGURE_KINDS + training_data._l2norm
  instead of local copies that silently drift when the canonical changes.
- MLSettings.load / .load_sync classmethods (mirror ImportSettings); route all
  8 scalar_one singleton reads through them (the session.get None-path stays).
- GET serializers for MLSettings + ImportSettings are now table-driven off the
  same _EDITABLE tuples PATCH writes, so a new field can't be silently absent
  from GET (the split that historically dropped fields).
- AUTO_APPLY_THRESHOLD_MIN/MAX constant single-sources the [0.5,0.999] operating
  range across the service clamp + the 5 API validators.
- test_ml_dry_helpers.py pins _applied_or_rejected + _sigmoid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:24 -04:00
bvandeusen d80a5255ed feat(extension): in-app update prompt — popup banner + toolbar badge (#1489)
CI / lint (push) Successful in 3s
extension / lint (push) Successful in 10s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 39s
CI / integration (push) Successful in 3m51s
extension / lint (pull_request) Successful in 10s
The extension is installed per-instance from the operator's FC host, so Firefox's
static update_url can't apply (each instance has a different host) and updates
were fully manual. Add a self-hosted-friendly update surface that reuses the
existing public GET /api/extension/manifest ({version, latest_url, sha256}):

- lib/api.js: getExtensionManifest().
- background.js: checkForUpdateInfo() compares the instance's latest published
  version against runtime.getManifest().version (dotted-numeric compare so
  1.0.10 > 1.0.9); CHECK_UPDATE message handler; refreshUpdateBadge() sets a
  toolbar badge via browser.action; a daily browser.alarms check plus on
  startup/installed. New 'alarms' permission (non-prompting).
- popup: an 'Update available — vX' banner with an Update button that opens the
  signed XPI (web root, /api stripped like OPEN_ARTIST_PAGE) → Firefox's native
  install prompt. Never blocks the popup on a failed check.

No backend changes (endpoint already exists). Bump 1.0.8→1.0.9 so this ships;
from here on updates surface themselves instead of needing a manual reinstall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 18:28:06 -04:00
41 changed files with 852 additions and 547 deletions
+4 -1
View File
@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
# Bump on every agent change. The page embeds this and /status reports it; the UI # Bump on every agent change. The page embeds this and /status reports it; the UI
# warns to reload when they differ — so a stale browser-cached page can't be # warns to reload when they differ — so a stale browser-cached page can't be
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.) # mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
VERSION = "2026-07-02.6 · sleep mode: an empty queue sheds to one downloader and backs the lease poll off to 15 min" VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
logbuf.install() logbuf.install()
cfg = Config.from_env() cfg = Config.from_env()
@@ -334,9 +334,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
waited.textContent=s.transient||0 waited.textContent=s.transient||0
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads // Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
// as live churn rather than a "broken" headline metric. // as live churn rather than a "broken" headline metric.
// '=== false' (not falsy) so a stale page that doesn't send models_loaded shows
// nothing; when the idle monitor unloads, the VRAM meter drops alongside this.
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'')+' · consumers '+(s.consumers!=null?s.consumers:'')+' · on GPU '+(s.active||0) pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'')+' · consumers '+(s.consumers!=null?s.consumers:'')+' · on GPU '+(s.active||0)
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'')+' MB/s' +' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'')+' MB/s'
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'') +(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
+(s.models_loaded===false?' · GPU models unloaded (idle — reload on next job)':'')
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
// Buffer occupancy bar (also driven here so it tracks the /status cadence). // Buffer occupancy bar (also driven here so it tracks the /status cadence).
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max) if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
+10
View File
@@ -51,6 +51,12 @@ class Config:
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
# all downloaders + video streams (0 = unlimited); # all downloaders + video streams (0 = unlimited);
# tunable live from the agent UI # tunable live from the agent UI
idle_unload_seconds: float # after this long with the GPU idle (nothing in
# flight, queue empty or Stopped), unload the
# SigLIP embedder + YOLO proposers to free their
# VRAM; they reload lazily on the next job. A
# 24/7 agent otherwise squats on ~5GB doing
# nothing. 0 disables (keep models warm forever).
@classmethod @classmethod
def from_env(cls) -> Config: def from_env(cls) -> Config:
@@ -87,4 +93,8 @@ class Config:
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0) # link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
# from the agent UI on wired/faster networks. # from the agent UI on wired/faster networks.
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")), bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
# 5 min: long enough that a lull between job bursts doesn't thrash the
# (few-second) reload, short enough that an agent left running with an
# empty queue hands its VRAM back promptly.
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
) )
+15
View File
@@ -170,6 +170,13 @@ class YoloProposer:
)) ))
return out return out
def unload(self) -> None:
"""Drop the loaded YOLO so its VRAM can be reclaimed; detect() reloads it
lazily on the next job. Leaves _ok untouched — a healthy proposer comes
back, but one that self-disabled on a fault stays off."""
with self._lock:
self._model = None
class Proposers: class Proposers:
"""The agent's proposer set, built from config. Each detector is optional — """The agent's proposer set, built from config. Each detector is optional —
@@ -216,3 +223,11 @@ class Proposers:
def panels(self, image): def panels(self, image):
return self._top(self._panel, image, self.cfg.max_panels) return self._top(self._panel, image, self.cfg.max_panels)
def unload(self) -> None:
"""Release every loaded proposer's YOLO (idle VRAM reclaim). The worker
also drops its reference to this Proposers and rebuilds a fresh one via
_proposers_for on the next job, so this is belt-and-braces."""
for p in (self._person, self._anatomy, self._panel):
if p is not None:
p.unload()
+15
View File
@@ -75,3 +75,18 @@ class CropEmbedder:
pooled = out.pooler_output if hasattr(out, "pooler_output") else out pooled = out.pooler_output if hasattr(out, "pooler_output") else out
arr = pooled.float().cpu().numpy().astype(np.float32) arr = pooled.float().cpu().numpy().astype(np.float32)
return [row.reshape(-1).tolist() for row in arr] return [row.reshape(-1).tolist() for row in arr]
def unload(self) -> bool:
"""Drop the loaded model so its VRAM can be reclaimed — the idle monitor
calls this after a spell with no work so an idle agent doesn't squat on
the card; the next embed() reloads it lazily (a few seconds). Held under
BOTH the load and inference locks so it can never race a concurrent load
or an in-flight forward pass. Returns True if a model was actually
released (the caller then runs one empty_cache() to hand the freed blocks
back to the driver)."""
with self._load_lock, self._infer_lock:
if self._model is None:
return False
self._model = None
self._processor = None
return True
+75
View File
@@ -57,6 +57,15 @@ MAX_BACKOFF_SECONDS = 60.0
# up on their own. # up on their own.
IDLE_POLL_MAX_SECONDS = 900.0 IDLE_POLL_MAX_SECONDS = 900.0
# Idle VRAM reclaim (operator 2026-07-17): the SigLIP embedder + YOLO proposers
# load lazily and then stay warm for fast job bursts — but a 24/7 agent with an
# empty queue would otherwise squat on that VRAM (~5GB on the operator's card)
# indefinitely while doing nothing. So a monitor unloads them after
# cfg.idle_unload_seconds with the GPU genuinely idle (nothing in flight, buffer
# drained); they reload lazily on the next job. This is just how often the
# monitor wakes to check — it bounds how soon past the threshold the unload fires.
IDLE_UNLOAD_CHECK_INTERVAL = 30.0
# A job whose fetch dies transiently this many times IN ONE SESSION stops being # A job whose fetch dies transiently this many times IN ONE SESSION stops being
# handed back and is failed instead. Transient handbacks (release) burn no # handed back and is failed instead. Transient handbacks (release) burn no
# attempts on the server, so a poisoned transfer — an original that stalls the # attempts on the server, so a poisoned transfer — an original that stalls the
@@ -268,6 +277,11 @@ class Worker:
self._proposers_sig = None # detector-config signature the current self._proposers_sig = None # detector-config signature the current
# proposers were built for (#134) # proposers were built for (#134)
self._proposers_lock = threading.Lock() self._proposers_lock = threading.Lock()
# Monotonic time of the last GPU activity (a consumer finishing a job).
# The idle monitor unloads the warm models once this goes stale by
# cfg.idle_unload_seconds — see _idle_unload_loop.
self._last_gpu_activity = time.monotonic()
threading.Thread(target=self._idle_unload_loop, daemon=True).start()
# --- held-lease bookkeeping -------------------------------------------- # --- held-lease bookkeeping --------------------------------------------
def _hold(self, job_ids) -> None: def _hold(self, job_ids) -> None:
@@ -608,6 +622,9 @@ class Worker:
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate "net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint) "bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
"idle": self._idle, # queue empty → poll backed off (UI hint) "idle": self._idle, # queue empty → poll backed off (UI hint)
# Whether the GPU models are currently resident (False after an idle
# unload freed their VRAM) — a plain bool read, UI hint only.
"models_loaded": self._embedder is not None or self._proposers is not None,
} }
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0): def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
@@ -788,6 +805,9 @@ class Worker:
self._bump(processed=1) self._bump(processed=1)
finally: finally:
self._bump(active=-1) self._bump(active=-1)
# Mark the GPU busy-until-now so the idle monitor starts its
# unload countdown from when work actually stopped, not before.
self._last_gpu_activity = time.monotonic()
def _ensure_embedder(self, model_name: str): def _ensure_embedder(self, model_name: str):
if self._embedder is not None: if self._embedder is not None:
@@ -845,6 +865,61 @@ class Worker:
self._proposers_sig = sig self._proposers_sig = sig
return self._proposers return self._proposers
def _unload_models(self) -> bool:
"""Release the GPU-resident models (SigLIP embedder + YOLO proposers) so an
idle agent hands their VRAM back instead of squatting on the card. They
reload lazily on the next job (_ensure_embedder / _proposers_for) — a
few seconds' cost paid only when work actually resumes. Dropping the
shared instances under their build locks means a concurrent job either
sees the old instance (before) or rebuilds a fresh one (after); the idle
monitor only calls this with nothing in flight, so no inference is using
them. Returns True if anything was released."""
released = False
with self._embedder_lock:
if self._embedder is not None:
self._embedder.unload()
self._embedder = None
released = True
with self._proposers_lock:
if self._proposers is not None:
self._proposers.unload()
self._proposers = None
self._proposers_sig = None
released = True
if released:
try:
import torch
if torch.cuda.is_available():
# torch's caching allocator holds freed blocks; hand them back
# to the driver so nvidia-smi actually reflects the drop.
torch.cuda.empty_cache()
except Exception: # noqa: BLE001 — torch absent / CPU-only → nothing to free
pass
return released
def _idle_unload_loop(self) -> None:
"""Unload the warm GPU models after a stretch of inactivity so a 24/7
agent with an empty queue doesn't hold ~5GB of VRAM doing nothing. Fires
only when nothing is in flight (active == 0 AND the buffer is drained) and
no job has completed for cfg.idle_unload_seconds — a window long enough
that a brief lull between bursts doesn't thrash reload/unload. Covers BOTH
sleep mode (queue empty, pipeline still running) and a full Stop; the
models reload lazily on the next job. idle_unload_seconds <= 0 disables it."""
idle_after = self.cfg.idle_unload_seconds
if idle_after <= 0:
return
while True:
time.sleep(IDLE_UNLOAD_CHECK_INTERVAL)
if self._embedder is None and self._proposers is None:
continue # nothing loaded → nothing to free
if self._active != 0 or not self._buffer.empty():
continue # work in flight → keep them warm
if time.monotonic() - self._last_gpu_activity < idle_after:
continue # not idle long enough yet
if self._unload_models():
log.info("idle %.0fs — unloaded GPU models, freed VRAM "
"(reload on next job)", idle_after)
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool: def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
"""Detect + embed the decoded frames and submit the result. Returns True """Detect + embed the decoded frames and submit the result. Returns True
when the job was completed (→ count it processed), False otherwise: a when the job was completed (→ count it processed), False otherwise: a
+1 -3
View File
@@ -256,9 +256,7 @@ async def lease():
if not await _agent_authed(session): if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401 return jsonify({"error": "unauthorized"}), 401
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch) jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
ml = ( ml = await MLSettings.load(session)
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
# image rows for url/mime in one shot # image rows for url/mime in one shot
ids = [j.image_record_id for j in jobs] ids = [j.image_record_id for j in jobs]
imgs = { imgs = {
+17 -43
View File
@@ -4,6 +4,7 @@ from quart import Blueprint, jsonify, request
from ..extensions import get_session from ..extensions import get_session
from ..models import MLSettings from ..models import MLSettings
from ..services.ml.heads import AUTO_APPLY_THRESHOLD_MAX, AUTO_APPLY_THRESHOLD_MIN
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml") ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
@@ -83,48 +84,21 @@ async def embedder_models():
@ml_admin_bp.route("/settings", methods=["GET"]) @ml_admin_bp.route("/settings", methods=["GET"])
async def get_settings(): async def get_settings():
from sqlalchemy import select
async with get_session() as session: async with get_session() as session:
s = ( s = await MLSettings.load(session)
await session.execute(select(MLSettings).where(MLSettings.id == 1)) # Table-driven off _EDITABLE (which PATCH also writes) so a new settings field
).scalar_one() # can never be silently absent from GET — the split that historically dropped
return jsonify( # fields. _EDITABLE already includes *_DETECTOR_FIELDS.
{ return jsonify({f: getattr(s, f) for f in _EDITABLE})
"cpu_embed_enabled": s.cpu_embed_enabled,
"video_frame_interval_seconds": s.video_frame_interval_seconds,
"video_max_frames": s.video_max_frames,
"embedder_model_version": s.embedder_model_version,
"head_min_positives": s.head_min_positives,
"head_auto_apply_precision": s.head_auto_apply_precision,
"head_auto_apply_enabled": s.head_auto_apply_enabled,
"head_auto_apply_min_positives": s.head_auto_apply_min_positives,
"ccip_match_threshold": s.ccip_match_threshold,
"ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
"ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
"presentation_auto_apply_enabled": s.presentation_auto_apply_enabled,
"presentation_auto_apply_threshold": s.presentation_auto_apply_threshold,
"presentation_conflict_threshold": s.presentation_conflict_threshold,
"process_auto_apply_enabled": s.process_auto_apply_enabled,
"process_auto_apply_threshold": s.process_auto_apply_threshold,
"process_conflict_threshold": s.process_conflict_threshold,
"embedder_model_name": s.embedder_model_name,
**{f: getattr(s, f) for f in _DETECTOR_FIELDS},
}
)
@ml_admin_bp.route("/settings", methods=["PATCH"]) @ml_admin_bp.route("/settings", methods=["PATCH"])
async def patch_settings(): async def patch_settings():
from sqlalchemy import select
body = await request.get_json() body = await request.get_json()
if not isinstance(body, dict): if not isinstance(body, dict):
return jsonify({"error": "body must be an object"}), 400 return jsonify({"error": "body must be an object"}), 400
async with get_session() as session: async with get_session() as session:
s = ( s = await MLSettings.load(session)
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
# Merge the patch over current values, then validate the result as a # Merge the patch over current values, then validate the result as a
# whole — the store-floor invariant couples three fields, so they # whole — the store-floor invariant couples three fields, so they
@@ -154,24 +128,24 @@ def _validate(p: dict) -> str | None:
# Head training (#114). # Head training (#114).
if int(p["head_min_positives"]) < 1: if int(p["head_min_positives"]) < 1:
return "head_min_positives must be >= 1" return "head_min_positives must be >= 1"
if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999): if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["head_auto_apply_precision"]) <= AUTO_APPLY_THRESHOLD_MAX):
return "head_auto_apply_precision must be between 0.5 and 0.999" return f"head_auto_apply_precision must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if int(p["head_auto_apply_min_positives"]) < 1: if int(p["head_auto_apply_min_positives"]) < 1:
return "head_auto_apply_min_positives must be >= 1" return "head_auto_apply_min_positives must be >= 1"
if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999): if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_match_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return "ccip_match_threshold must be between 0.5 and 0.999" return f"ccip_match_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999): if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return "ccip_auto_apply_threshold must be between 0.5 and 0.999" return f"ccip_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is # Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
# consequential); the conflict cut is a plain probability [0,1]. # consequential); the conflict cut is a plain probability [0,1].
if not (0.5 <= float(p["presentation_auto_apply_threshold"]) <= 0.999): if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["presentation_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return "presentation_auto_apply_threshold must be between 0.5 and 0.999" return f"presentation_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0): if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
return "presentation_conflict_threshold must be between 0 and 1" return "presentation_conflict_threshold must be between 0 and 1"
# Process auto-apply (#1464). wip/editor stay VISIBLE so a false apply is # Process auto-apply (#1464). wip/editor stay VISIBLE so a false apply is
# low-harm (excludes-from-training + a review flag), but keep the same bar. # low-harm (excludes-from-training + a review flag), but keep the same bar.
if not (0.5 <= float(p["process_auto_apply_threshold"]) <= 0.999): if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["process_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return "process_auto_apply_threshold must be between 0.5 and 0.999" return f"process_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (0.0 <= float(p["process_conflict_threshold"]) <= 1.0): if not (0.0 <= float(p["process_conflict_threshold"]) <= 1.0):
return "process_conflict_threshold must be between 0 and 1" return "process_conflict_threshold must be between 0 and 1"
# Embedder model swap (#1190): both must be non-empty. Changing them means a # Embedder model swap (#1190): both must be non-empty. Changing them means a
+3 -28
View File
@@ -66,34 +66,9 @@ _EXTDL_TOGGLE_FIELDS = (
async def get_import_settings(): async def get_import_settings():
async with get_session() as session: async with get_session() as session:
row = await ImportSettings.load(session) row = await ImportSettings.load(session)
return jsonify({ # Table-driven off _EDITABLE_FIELDS (which PATCH also writes) so a new field
"min_width": row.min_width, # can't be silently absent from GET.
"min_height": row.min_height, return jsonify({f: getattr(row, f) for f in _EDITABLE_FIELDS})
"skip_transparent": row.skip_transparent,
"transparency_threshold": row.transparency_threshold,
"skip_single_color": row.skip_single_color,
"single_color_threshold": row.single_color_threshold,
"single_color_tolerance": row.single_color_tolerance,
"phash_threshold": row.phash_threshold,
"download_rate_limit_seconds": row.download_rate_limit_seconds,
"download_validate_files": row.download_validate_files,
"download_schedule_default_seconds": row.download_schedule_default_seconds,
"download_event_retention_days": row.download_event_retention_days,
"download_failure_warning_threshold": row.download_failure_warning_threshold,
"series_suggest_enabled": row.series_suggest_enabled,
"series_suggest_threshold": row.series_suggest_threshold,
"extdl_mega_enabled": row.extdl_mega_enabled,
"extdl_gdrive_enabled": row.extdl_gdrive_enabled,
"extdl_mediafire_enabled": row.extdl_mediafire_enabled,
"extdl_dropbox_enabled": row.extdl_dropbox_enabled,
"extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
"translation_enabled": row.translation_enabled,
"interpreter_base_url": row.interpreter_base_url,
"translation_target_lang": row.translation_target_lang,
"translation_min_confidence": row.translation_min_confidence,
"wip_title_tagging_enabled": row.wip_title_tagging_enabled,
"wip_soft_title_tagging_enabled": row.wip_soft_title_tagging_enabled,
})
@settings_bp.route("/settings/import", methods=["PATCH"]) @settings_bp.route("/settings/import", methods=["PATCH"])
+12
View File
@@ -10,6 +10,7 @@ from sqlalchemy import (
Integer, Integer,
String, String,
func, func,
select,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -212,3 +213,14 @@ class MLSettings(Base):
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
@classmethod
async def load(cls, session) -> MLSettings:
"""The singleton settings row (id=1), via an async session. Mirrors
ImportSettings.load — the shared singleton-loader pattern."""
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
@classmethod
def load_sync(cls, session) -> MLSettings:
"""The singleton settings row (id=1), via a sync session."""
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
@@ -150,9 +150,7 @@ def refresh_character_prototypes(
"""Incrementally refresh the prototype store. `full=True` rebuilds every """Incrementally refresh the prototype store. `full=True` rebuilds every
character regardless of the gate/fingerprints (nightly reconcile). Returns character regardless of the gate/fingerprints (nightly reconcile). Returns
{skipped, rebuilt, removed}; commits.""" {skipped, rebuilt, removed}; commits."""
settings = session.execute( settings = MLSettings.load_sync(session)
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
sig = _global_signature(session) sig = _global_signature(session)
if not full and settings.ccip_ref_signature == sig: if not full and settings.ccip_ref_signature == sig:
return {"skipped": True, "rebuilt": 0, "removed": 0} return {"skipped": True, "rebuilt": 0, "removed": 0}
@@ -204,9 +202,7 @@ def retract_auto_applied_ccip(session: Session) -> int:
n_retracted.""" n_retracted."""
import numpy as np import numpy as np
settings = session.execute( settings = MLSettings.load_sync(session)
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
if not settings.ccip_auto_apply_enabled: if not settings.ccip_auto_apply_enabled:
return 0 return 0
thr = float(settings.ccip_auto_apply_threshold) thr = float(settings.ccip_auto_apply_threshold)
+65 -62
View File
@@ -23,6 +23,7 @@ from datetime import UTC, datetime
from typing import Any from typing import Any
from sqlalchemy import delete, exists, func, select from sqlalchemy import delete, exists, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -42,6 +43,7 @@ from ...models import (
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
from .training_data import ( from .training_data import (
_AUTO_SOURCES, _AUTO_SOURCES,
_applied_or_rejected,
_auto_apply_point, _auto_apply_point,
_hygiene_excluded_ids, _hygiene_excluded_ids,
_ids_with_tag, _ids_with_tag,
@@ -61,6 +63,14 @@ MIN_POSITIVES_FLOOR = 8 # hard floor; settings.head_min_positives can raise
_UNLABELED_POOL = 4000 _UNLABELED_POOL = 4000
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head _EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
# Auto-apply / match confidence operating range. Every graduated auto-apply or
# CCIP-match threshold the operator can set lives in this band, and the head
# precision target is clamped to it: below 0.5 "auto-apply" is meaningless, and
# 1.0 is unachievable so 0.999 is the ceiling. One source shared by the service
# clamp (_normalize_params) and the API validator (ml_admin._validate).
AUTO_APPLY_THRESHOLD_MIN = 0.5
AUTO_APPLY_THRESHOLD_MAX = 0.999
# Only these tag kinds get heads (the surfaced suggestion categories). # Only these tag kinds get heads (the surfaced suggestion categories).
_HEAD_KINDS = (TagKind.general, TagKind.character) _HEAD_KINDS = (TagKind.general, TagKind.character)
# tag.kind -> the suggestion category the rail groups under. # tag.kind -> the suggestion category the rail groups under.
@@ -78,6 +88,38 @@ _CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65 _SYSTEM_TAG_SUGGEST_FLOOR = 0.65
def _sigmoid(z, np):
"""Logistic sigmoid 1/(1+e^-z): the head score→probability transform. One home
for what was inlined at every scoring site (suggest, both sweeps, retract)."""
return 1.0 / (1.0 + np.exp(-z))
def _conflict_scores(Xn, Wc, bc, np):
"""The presentation conflict signal (#141): per row, the MAX content-head
probability and WHICH head produced it. Shared by the system-tag sweep's guard-2
and the soft-wip audit — both ask "does this ALSO look like real content?"."""
cprobs = _sigmoid(Xn @ Wc.T + bc, np)
return cprobs.max(axis=1), cprobs.argmax(axis=1)
def _insert_presentation_review(
session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode,
):
"""Single-source the ring-loud PresentationReview row shape so the two writers
(system-tag sweep guard-2 + soft-wip audit) can't drift on columns or `mode` —
they share the (image_record_id, tag_id) composite PK, so a divergent `mode`
would be a silent first-writer-wins bug."""
session.execute(
pg_insert(PresentationReview)
.values(
image_record_id=image_record_id, tag_id=tag_id,
conflict_tag_id=conflict_tag_id, conflict_score=conflict_score,
mode=mode,
)
.on_conflict_do_nothing()
)
class HeadTrainingAlreadyRunning(Exception): class HeadTrainingAlreadyRunning(Exception):
"""Raised by start_head_training_run when a run is already in flight.""" """Raised by start_head_training_run when a run is already in flight."""
@@ -103,9 +145,7 @@ def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
def _settings(session: Session) -> MLSettings: def _settings(session: Session) -> MLSettings:
return session.execute( return MLSettings.load_sync(session)
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]: def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
@@ -124,7 +164,7 @@ def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[s
except (TypeError, ValueError): except (TypeError, ValueError):
cv_folds = DEFAULT_CV_FOLDS cv_folds = DEFAULT_CV_FOLDS
try: try:
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999) precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), AUTO_APPLY_THRESHOLD_MIN), AUTO_APPLY_THRESHOLD_MAX)
except (TypeError, ValueError): except (TypeError, ValueError):
precision_target = s.head_auto_apply_precision precision_target = s.head_auto_apply_precision
return { return {
@@ -536,7 +576,7 @@ async def score_image(
norms[norms == 0] = 1.0 norms[norms == 0] = 1.0
Xn = X / norms Xn = X / norms
Z = Xn @ heads["W"].T + heads["b"] # (B, H) Z = Xn @ heads["W"].T + heads["b"] # (B, H)
probs_bag = 1.0 / (1.0 + np.exp(-Z)) # (B, H) probs_bag = _sigmoid(Z, np) # (B, H)
probs = probs_bag.max(axis=0) # (H,) best over the bag probs = probs_bag.max(axis=0) # (H,) best over the bag
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds # ARGMAX beside the max: WHICH bag row won each head → the region that grounds
# the tag (bag_meta[win]); None when the whole-image vector won (#1206). # the tag (bag_meta[win]); None when the whole-image vector won (#1206).
@@ -614,9 +654,7 @@ async def ground_applied_tag(
async def _settings_async(session: AsyncSession) -> MLSettings: async def _settings_async(session: AsyncSession) -> MLSettings:
return ( return await MLSettings.load(session)
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
# --- Earned auto-apply (sync, ml worker) --------------------------------- # --- Earned auto-apply (sync, ml worker) ---------------------------------
@@ -687,7 +725,6 @@ def auto_apply_sweep(
embeddings in chunks; commits per chunk on a real run. Returns embeddings in chunks; commits per chunk on a real run. Returns
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}.""" {n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
import numpy as np import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
settings = _settings(session) settings = _settings(session)
rows = _auto_apply_heads( rows = _auto_apply_heads(
@@ -704,18 +741,7 @@ def auto_apply_sweep(
names = [r.name for r in rows] names = [r.name for r in rows]
# Skip images that already carry, or have rejected, each tag. # Skip images that already carry, or have rejected, each tag.
skip = {tid: set() for tid in tag_ids} skip = _applied_or_rejected(session, tag_ids)
for tid in tag_ids:
for (iid,) in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
):
skip[tid].add(iid)
for (iid,) in session.execute(
select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == tid
)
):
skip[tid].add(iid)
applied = [0] * len(rows) applied = [0] * len(rows)
scanned = 0 scanned = 0
@@ -729,7 +755,7 @@ def auto_apply_sweep(
if not cids: if not cids:
continue continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np) Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b))) # (N, H) probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
scanned += len(cids) scanned += len(cids)
for h in range(len(rows)): for h in range(len(rows)):
tid = tag_ids[h] tid = tag_ids[h]
@@ -840,7 +866,6 @@ def system_tag_auto_apply_sweep(
enabled flag is set. numpy-only (no sklearn). Returns {n_applied, n_flagged, enabled flag is set. numpy-only (no sklearn). Returns {n_applied, n_flagged,
concepts}.""" concepts}."""
import numpy as np import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
cfg = _SWEEP_MODES[mode] cfg = _SWEEP_MODES[mode]
settings = _settings(session) settings = _settings(session)
@@ -869,18 +894,7 @@ def system_tag_auto_apply_sweep(
valued = _valued_image_ids(session) valued = _valued_image_ids(session)
# Skip images that already carry, or have rejected, each presentation tag. # Skip images that already carry, or have rejected, each presentation tag.
skip = {tid: set() for tid in pres_tag_ids} skip = _applied_or_rejected(session, pres_tag_ids)
for tid in pres_tag_ids:
for (iid,) in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
):
skip[tid].add(iid)
for (iid,) in session.execute(
select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == tid
)
):
skip[tid].add(iid)
applied = [0] * len(pres) applied = [0] * len(pres)
n_flagged = 0 n_flagged = 0
@@ -895,11 +909,9 @@ def system_tag_auto_apply_sweep(
if not cids: if not cids:
continue continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np) Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = 1.0 / (1.0 + np.exp(-(Xn @ Wp.T + bp))) # (N, P) probs = _sigmoid(Xn @ Wp.T + bp, np) # (N, P)
if Wc is not None: if Wc is not None:
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc))) # (N, C) max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
max_c = cprobs.max(axis=1)
arg_c = cprobs.argmax(axis=1)
scanned += len(cids) scanned += len(cids)
for p in range(len(pres)): for p in range(len(pres)):
tid = pres_tag_ids[p] tid = pres_tag_ids[p]
@@ -924,15 +936,12 @@ def system_tag_auto_apply_sweep(
if Wc is not None and float(max_c[idx]) >= conflict_thr: if Wc is not None and float(max_c[idx]) >= conflict_thr:
n_flagged += 1 n_flagged += 1
if not dry_run: if not dry_run:
session.execute( _insert_presentation_review(
pg_insert(PresentationReview) session,
.values( image_record_id=iid, tag_id=tid,
image_record_id=iid, tag_id=tid, conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
conflict_tag_id=conf_tag_ids[int(arg_c[idx])], conflict_score=float(max_c[idx]),
conflict_score=float(max_c[idx]), mode=mode,
mode=mode,
)
.on_conflict_do_nothing()
) )
if not dry_run: if not dry_run:
session.commit() session.commit()
@@ -956,7 +965,6 @@ def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
NOT remove the tag; the operator decides. No-op when there are no content heads. NOT remove the tag; the operator decides. No-op when there are no content heads.
numpy-only. Returns {n_scanned, n_flagged}.""" numpy-only. Returns {n_scanned, n_flagged}."""
import numpy as np import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id
@@ -993,22 +1001,17 @@ def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
continue continue
scanned += len(cids) scanned += len(cids)
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np) Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc))) max_c, arg_c = _conflict_scores(Xn, Wc, bc, np)
max_c = cprobs.max(axis=1)
arg_c = cprobs.argmax(axis=1)
for k in range(len(cids)): for k in range(len(cids)):
if float(max_c[k]) >= conflict_thr: if float(max_c[k]) >= conflict_thr:
n_flagged += 1 n_flagged += 1
if not dry_run: if not dry_run:
session.execute( _insert_presentation_review(
pg_insert(PresentationReview) session,
.values( image_record_id=cids[k], tag_id=wip_id,
image_record_id=cids[k], tag_id=wip_id, conflict_tag_id=conf_tag_ids[int(arg_c[k])],
conflict_tag_id=conf_tag_ids[int(arg_c[k])], conflict_score=float(max_c[k]),
conflict_score=float(max_c[k]), mode="process",
mode="process",
)
.on_conflict_do_nothing()
) )
if not dry_run: if not dry_run:
session.commit() session.commit()
@@ -1062,7 +1065,7 @@ def retract_auto_applied_heads(session: Session) -> int:
continue continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np) Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
w = np.asarray(weights, dtype=np.float32) w = np.asarray(weights, dtype=np.float32)
probs = 1.0 / (1.0 + np.exp(-(Xn @ w + float(bias)))) probs = _sigmoid(Xn @ w + float(bias), np)
below = [cids[k] for k in np.where(probs < float(thr))[0]] below = [cids[k] for k in np.where(probs < float(thr))[0]]
for iid in below: for iid in below:
session.execute( session.execute(
+18
View File
@@ -94,6 +94,24 @@ def _rejected_ids(session: Session, tag_id: int) -> list[int]:
] ]
def _applied_or_rejected(session: Session, tag_ids) -> dict[int, set[int]]:
"""Per-tag skip set for the auto-apply sweeps: every image that ALREADY carries
the tag (ANY source — not just training positives) OR has rejected it. A sweep
never re-applies to these. Shared by auto_apply_sweep + system_tag_auto_apply_sweep
(heads.py) and scheduled_ccip_auto_apply (tasks/ml.py). Callers mutate the returned
sets in-place to also dedupe within a single run."""
skip: dict[int, set[int]] = {}
for tid in tag_ids:
ids = {
r[0] for r in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
).all()
}
ids.update(_rejected_ids(session, tid))
skip[tid] = ids
return skip
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]: def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are """Random image ids (with an embedding) NOT carrying the tag. Concepts are
sparse, so an untagged image is almost always a true negative.""" sparse, so an untagged image is almost always a true negative."""
+20 -36
View File
@@ -91,48 +91,46 @@ def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
) )
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None: def _campaigns_api_first(vanity: str, cookies_path: str | None) -> dict | None:
"""The first `data` object from Patreon's campaigns API filtered by vanity
(`?filter[vanity]=<vanity>&fields[campaign]=name`), or None on any failure
(network / non-200 / non-JSON / empty). The single request shape shared by
_lookup_via_api (plucks the campaign id) and resolve_display_name (plucks the
display name)."""
jar = _load_cookie_jar(cookies_path) jar = _load_cookie_jar(cookies_path)
headers = {
"User-Agent": _USER_AGENT,
"Accept": "application/vnd.api+json",
}
params = {
"filter[vanity]": vanity,
"fields[campaign]": "name",
}
try: try:
resp = requests.get( resp = requests.get(
_CAMPAIGNS_URL, _CAMPAIGNS_URL,
params=params, params={"filter[vanity]": vanity, "fields[campaign]": "name"},
headers=headers, headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
cookies=jar, cookies=jar,
timeout=_TIMEOUT_SECONDS, timeout=_TIMEOUT_SECONDS,
) )
except requests.RequestException as exc: except requests.RequestException as exc:
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc) log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
return None return None
if resp.status_code != 200: if resp.status_code != 200:
log.warning( log.warning(
"Patreon campaigns API returned HTTP %d for vanity=%s", "Patreon campaigns API returned HTTP %d for vanity=%s",
resp.status_code, vanity, resp.status_code, vanity,
) )
return None return None
try: try:
payload = resp.json() payload = resp.json()
except ValueError as exc: except ValueError as exc:
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc) log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
return None return None
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
return None
return data[0]
if not isinstance(payload, dict):
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
first = _campaigns_api_first(vanity, cookies_path)
if first is None:
return None return None
data = payload.get("data") campaign_id = first.get("id")
if not isinstance(data, list) or not data:
return None
first = data[0] if isinstance(data[0], dict) else None
campaign_id = first.get("id") if first else None
if not isinstance(campaign_id, str) or not campaign_id: if not isinstance(campaign_id, str) or not campaign_id:
return None return None
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id) log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
@@ -144,24 +142,10 @@ def resolve_display_name(vanity: str, cookies_path: str | None) -> str | None:
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None (`fields[campaign]=name`), used to name the Artist at add-time (#130). None
on any failure — the caller falls back to the vanity handle. Sync: call from on any failure — the caller falls back to the vanity handle. Sync: call from
an executor.""" an executor."""
jar = _load_cookie_jar(cookies_path) first = _campaigns_api_first(vanity, cookies_path)
try: if first is None:
resp = requests.get(
_CAMPAIGNS_URL,
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
cookies=jar,
timeout=_TIMEOUT_SECONDS,
)
if resp.status_code != 200:
return None
data = resp.json().get("data")
except (requests.RequestException, ValueError) as exc:
log.warning("Patreon name lookup failed for vanity=%s: %s", vanity, exc)
return None return None
if not isinstance(data, list) or not data or not isinstance(data[0], dict): name = (first.get("attributes") or {}).get("name")
return None
name = (data[0].get("attributes") or {}).get("name")
return name.strip() if isinstance(name, str) and name.strip() else None return name.strip() if isinstance(name, str) and name.strip() else None
+45 -72
View File
@@ -776,89 +776,62 @@ def recover_stalled_library_audit_runs() -> int:
return recovered return recovered
def _recover_stalled_runs(model, *, stall_minutes: int, keep_runs: int, label: str) -> int:
"""Shared recovery + retention sweep for the head run-tracking tables
(HeadTrainingRun / HeadAutoApplyRun, which share the
status/last_progress_at/started_at/finished_at/error/id columns): flip 'running'
rows with no progress past `stall_minutes` to 'error', then prune to the last
`keep_runs` (rule 89). Returns the number recovered. NOTE the two other recover
tasks are deliberately NOT folded in — library-audit has no prune tail and
backup uses a single started_at cutoff."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=stall_minutes)
with SessionLocal() as session:
result = session.execute(
update(model)
.where(model.status == "running")
.where(func.coalesce(model.last_progress_at, model.started_at) < cutoff)
.values(
status="error", finished_at=now,
error=f"stranded by recovery sweep (no progress for {stall_minutes} min)",
)
)
keep = session.execute(
select(model.id).order_by(model.id.desc()).limit(keep_runs)
).scalars().all()
if keep:
session.execute(delete(model).where(model.id.not_in(keep)))
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("%s: recovered %d rows", label, recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs") @celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
def recover_stalled_head_training_runs() -> int: def recover_stalled_head_training_runs() -> int:
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to """Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention, 'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
rule 89). Runs every 5 min on the maintenance lane; no-op when idle.""" rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
SessionLocal = _sync_session_factory() return _recover_stalled_runs(
now = datetime.now(UTC) HeadTrainingRun,
cutoff = now - timedelta(minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES) stall_minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES,
with SessionLocal() as session: keep_runs=HEAD_TRAINING_KEEP_RUNS,
result = session.execute( label="recover_stalled_head_training_runs",
update(HeadTrainingRun) )
.where(HeadTrainingRun.status == "running")
.where(
func.coalesce(
HeadTrainingRun.last_progress_at, HeadTrainingRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_TRAINING_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadTrainingRun.id).order_by(HeadTrainingRun.id.desc())
.limit(HEAD_TRAINING_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadTrainingRun).where(HeadTrainingRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_training_runs: recovered %d rows", recovered
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs") @celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
def recover_stalled_head_auto_apply_runs() -> int: def recover_stalled_head_auto_apply_runs() -> int:
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the """Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane.""" last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
SessionLocal = _sync_session_factory() return _recover_stalled_runs(
now = datetime.now(UTC) HeadAutoApplyRun,
cutoff = now - timedelta(minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES) stall_minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES,
with SessionLocal() as session: keep_runs=HEAD_AUTO_APPLY_KEEP_RUNS,
result = session.execute( label="recover_stalled_head_auto_apply_runs",
update(HeadAutoApplyRun) )
.where(HeadAutoApplyRun.status == "running")
.where(
func.coalesce(
HeadAutoApplyRun.last_progress_at, HeadAutoApplyRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadAutoApplyRun.id).order_by(HeadAutoApplyRun.id.desc())
.limit(HEAD_AUTO_APPLY_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadAutoApplyRun).where(HeadAutoApplyRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_auto_apply_runs: recovered %d rows", recovered
)
return recovered
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends). # Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
+10 -30
View File
@@ -105,9 +105,7 @@ def embed_image(self, image_id: int) -> dict:
record = session.get(ImageRecord, image_id) record = session.get(ImageRecord, image_id)
if record is None: if record is None:
return {"status": "missing", "image_id": image_id} return {"status": "missing", "image_id": image_id}
settings = session.execute( settings = MLSettings.load_sync(session)
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
src = Path(record.path) src = Path(record.path)
is_vid = _is_video(src) is_vid = _is_video(src)
@@ -488,15 +486,10 @@ def scheduled_ccip_auto_apply() -> str:
from sqlalchemy import select as sa_select from sqlalchemy import select as sa_select
from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..models import ImageRegion, MLSettings, Tag, TagKind, TagSuggestionRejection from ..models import ImageRegion, MLSettings, Tag, TagKind
from ..models.tag import image_tag from ..models.tag import image_tag
from ..services.ml.ccip import _FIGURE_KINDS
fig = ("face", "figure") from ..services.ml.training_data import _applied_or_rejected, _l2norm
def _l2(m):
n = np.linalg.norm(m, axis=1, keepdims=True)
n[n == 0] = 1.0
return m / n
SessionLocal = _sync_session_factory() SessionLocal = _sync_session_factory()
with SessionLocal() as session: with SessionLocal() as session:
@@ -521,7 +514,7 @@ def scheduled_ccip_auto_apply() -> str:
) )
.join(Tag, Tag.id == image_tag.c.tag_id) .join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character) .where(Tag.kind == TagKind.character)
.where(ImageRegion.kind.in_(fig)) .where(ImageRegion.kind.in_(_FIGURE_KINDS))
.where(ImageRegion.ccip_embedding.is_not(None)) .where(ImageRegion.ccip_embedding.is_not(None))
.where(ImageRegion.image_record_id.in_(single)) .where(ImageRegion.image_record_id.in_(single))
).all() ).all()
@@ -532,29 +525,16 @@ def scheduled_ccip_auto_apply() -> str:
for tid, vec in ref_rows: for tid, vec in ref_rows:
by_char.setdefault(tid, []).append(vec) by_char.setdefault(tid, []).append(vec)
ref_tags = list(by_char) ref_tags = list(by_char)
mats = [_l2(np.asarray(by_char[t], dtype=np.float32)) for t in ref_tags] mats = [_l2norm(np.asarray(by_char[t], dtype=np.float32), np) for t in ref_tags]
allref = np.vstack(mats) # (total, 768) allref = np.vstack(mats) # (total, 768)
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
# Per character: images that already carry OR rejected the tag — skip. # Per character: images that already carry OR rejected the tag — skip.
skip = {t: set() for t in ref_tags} skip = _applied_or_rejected(session, ref_tags)
for t in ref_tags:
for (iid,) in session.execute(
sa_select(image_tag.c.image_record_id).where(
image_tag.c.tag_id == t
)
):
skip[t].add(iid)
for (iid,) in session.execute(
sa_select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == t
)
):
skip[t].add(iid)
img_ids = list(session.execute( img_ids = list(session.execute(
sa_select(ImageRegion.image_record_id) sa_select(ImageRegion.image_record_id)
.where(ImageRegion.kind.in_(fig), ImageRegion.ccip_embedding.is_not(None)) .where(ImageRegion.kind.in_(_FIGURE_KINDS), ImageRegion.ccip_embedding.is_not(None))
.distinct() .distinct()
).scalars()) ).scalars())
@@ -566,7 +546,7 @@ def scheduled_ccip_auto_apply() -> str:
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding) sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
.where( .where(
ImageRegion.image_record_id.in_(chunk), ImageRegion.image_record_id.in_(chunk),
ImageRegion.kind.in_(fig), ImageRegion.kind.in_(_FIGURE_KINDS),
ImageRegion.ccip_embedding.is_not(None), ImageRegion.ccip_embedding.is_not(None),
) )
).all() ).all()
@@ -574,7 +554,7 @@ def scheduled_ccip_auto_apply() -> str:
for iid, vec in rows: for iid, vec in rows:
by_img.setdefault(iid, []).append(vec) by_img.setdefault(iid, []).append(vec)
for iid, vecs in by_img.items(): for iid, vecs in by_img.items():
q = _l2(np.asarray(vecs, dtype=np.float32)) # (nq, 768) q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768)
colmax = (q @ allref.T).max(axis=0) # (total,) colmax = (q @ allref.T).max(axis=0) # (total,)
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,) charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
for ci in np.where(charmax >= thr)[0]: for ci in np.where(charmax >= thr)[0]:
+92 -30
View File
@@ -31,6 +31,68 @@ browser.runtime.onInstalled.addListener(() => ensureInitialized());
browser.runtime.onStartup.addListener(() => ensureInitialized()); browser.runtime.onStartup.addListener(() => ensureInitialized());
ensureInitialized().catch(e => console.error('init failed:', e)); ensureInitialized().catch(e => console.error('init failed:', e));
// ---- Extension self-update check (#1489) ----
// Installed per-instance from the operator's FC host, so Firefox's static
// update_url can't apply (each instance has a different host). Instead ask the
// configured backend for the latest published version and nudge the operator to
// reinstall the freshly-signed XPI — surfaced as a popup banner (on demand) and
// a toolbar badge (daily). /api/extension/manifest is public and returns
// {version, latest_url, sha256}; the XPI is served from the web root (not /api).
function versionIsNewer(candidate, current) {
// Dotted numeric compare so 1.0.10 > 1.0.9 (a plain string compare wouldn't).
const a = String(candidate).split('.').map(n => parseInt(n, 10) || 0);
const b = String(current).split('.').map(n => parseInt(n, 10) || 0);
for (let i = 0; i < Math.max(a.length, b.length); i++) {
if ((a[i] || 0) !== (b[i] || 0)) return (a[i] || 0) > (b[i] || 0);
}
return false;
}
async function checkForUpdateInfo() {
await ensureInitialized();
if (!api.isConfigured()) return { updateAvailable: false, configured: false };
let info;
try {
info = await api.getExtensionManifest();
} catch (e) {
return { updateAvailable: false, error: e.message };
}
const currentVersion = browser.runtime.getManifest().version;
const latestVersion = info && info.version ? info.version : null;
// latest_url is served from the web root, not the JSON API.
const base = api.webRoot();
return {
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
currentVersion,
latestVersion,
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null,
};
}
async function refreshUpdateBadge() {
let r;
try { r = await checkForUpdateInfo(); } catch { return; }
try {
await browser.action.setBadgeText({ text: r.updateAvailable ? '↑' : '' });
if (r.updateAvailable) {
await browser.action.setBadgeBackgroundColor({ color: '#F4BA7A' });
await browser.action.setTitle({ title: `FabledCurator — update available (v${r.latestVersion})` });
} else {
await browser.action.setTitle({ title: 'FabledCurator' });
}
} catch { /* action API unavailable — non-fatal */ }
}
// Daily proactive check (needs the "alarms" permission). create() is idempotent
// by name, so re-running it on each event-page load is safe.
browser.alarms.create('fc-update-check', { periodInMinutes: 24 * 60, delayInMinutes: 1 });
browser.alarms.onAlarm.addListener((alarm) => {
if (alarm.name === 'fc-update-check') refreshUpdateBadge();
});
browser.runtime.onStartup.addListener(() => refreshUpdateBadge());
browser.runtime.onInstalled.addListener(() => refreshUpdateBadge());
// ---- Discord token capture via webRequest ---- // ---- Discord token capture via webRequest ----
browser.webRequest.onBeforeSendHeaders.addListener( browser.webRequest.onBeforeSendHeaders.addListener(
@@ -148,6 +210,21 @@ browser.webRequest.onBeforeRedirect.addListener(
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] }, { urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
); );
// Extract → verify → upload one cookie-auth platform. Returns a structured
// outcome so the two callers (EXPORT_COOKIES single, EXPORT_ALL_COOKIES) shape
// their own response + skip semantics. Verifies the captured cookies are
// actually live BEFORE uploading, so a confirmed-stale session doesn't overwrite
// good FC-side credentials; platforms with no verify config (v.ok === null) fall
// through to upload.
async function exportPlatformCookies(key) {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { status: 'empty' };
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) return { status: 'stale', reason: v.reason, cookieCount: cookies.length };
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
return { status: 'ok', cookieCount: cookies.length, verified: v.ok === true };
}
// ---- Message router ---- // ---- Message router ----
browser.runtime.onMessage.addListener(async (msg) => { browser.runtime.onMessage.addListener(async (msg) => {
@@ -192,22 +269,14 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (!platform) return { error: `Unknown platform: ${key}` }; if (!platform) return { error: `Unknown platform: ${key}` };
try { try {
if (platform.authType === 'cookies') { if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key); const r = await exportPlatformCookies(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' }; if (r.status === 'empty') return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE if (r.status === 'stale') {
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
return { return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`, error: `Captured ${r.cookieCount} ${platform.name} cookies but they don't appear authenticated (${r.reason}). Log in again in this browser, then retry.`,
}; };
} }
const data = toNetscapeFormat(cookies); return { success: true, cookieCount: r.cookieCount, verified: r.verified };
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
} }
if (key === 'discord') { if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' }; if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -235,18 +304,10 @@ browser.runtime.onMessage.addListener(async (msg) => {
continue; continue;
} }
try { try {
const cookies = await extractCookiesForPlatform(key); const r = await exportPlatformCookies(key);
if (cookies.length === 0) { if (r.status === 'empty') results[key] = { skipped: true, reason: 'no cookies' };
results[key] = { skipped: true, reason: 'no cookies' }; else if (r.status === 'stale') results[key] = { error: `verify failed: ${r.reason}` };
continue; else results[key] = { success: true, cookieCount: r.cookieCount, verified: r.verified };
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
} catch (e) { } catch (e) {
results[key] = { error: e.message }; results[key] = { error: e.message };
} }
@@ -283,11 +344,9 @@ browser.runtime.onMessage.addListener(async (msg) => {
} }
case 'OPEN_ARTIST_PAGE': { case 'OPEN_ARTIST_PAGE': {
// apiUrl is configured with the /api suffix (see // The SPA artist route (/artist/:slug) is served from the web root, not
// options/options.html placeholder); the SPA artist route is // the JSON API — see api.webRoot().
// /artist/:slug, served from the same origin. Strip /api so the const base = api.webRoot();
// browser-level URL hits the Vue router, not the JSON API.
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
const slug = encodeURIComponent(msg.slug || ''); const slug = encodeURIComponent(msg.slug || '');
if (!base || !slug) return { error: 'apiUrl or slug missing' }; if (!base || !slug) return { error: 'apiUrl or slug missing' };
try { try {
@@ -298,6 +357,9 @@ browser.runtime.onMessage.addListener(async (msg) => {
} }
} }
case 'CHECK_UPDATE':
return await checkForUpdateInfo();
default: default:
return { error: `Unknown message type: ${msg.type}` }; return { error: `Unknown message type: ${msg.type}` };
} }
+13
View File
@@ -89,6 +89,19 @@ class FabledCuratorAPI {
const qs = new URLSearchParams({ url }).toString(); const qs = new URLSearchParams({ url }).toString();
return this.request('GET', `/extension/probe?${qs}`); return this.request('GET', `/extension/probe?${qs}`);
} }
// Latest published extension version on this instance — drives the in-app
// update prompt. Public endpoint (no key needed, but request() sends it
// harmlessly). Returns {version, xpi_url, latest_url, sha256}.
getExtensionManifest() {
return this.request('GET', '/extension/manifest');
}
// The web/SPA root: baseUrl with the trailing slash + `/api` suffix stripped.
// Where the Vue router (artist pages) and the served XPI live, NOT the JSON
// API. Used by OPEN_ARTIST_PAGE + the self-update check.
webRoot() {
return (this.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
}
// Connection test = the cheapest read with auth. // Connection test = the cheapest read with auth.
testConnection() { testConnection() {
+3 -2
View File
@@ -1,7 +1,7 @@
{ {
"manifest_version": 3, "manifest_version": 3,
"name": "FabledCurator", "name": "FabledCurator",
"version": "1.0.8", "version": "1.0.9",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.", "description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": { "browser_specific_settings": {
@@ -22,7 +22,8 @@
"tabs", "tabs",
"activeTab", "activeTab",
"webRequest", "webRequest",
"webRequestBlocking" "webRequestBlocking",
"alarms"
], ],
"host_permissions": [ "host_permissions": [
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "fabledcurator-extension", "name": "fabledcurator-extension",
"version": "1.0.8", "version": "1.0.9",
"private": true, "private": true,
"description": "Firefox extension for FabledCurator", "description": "Firefox extension for FabledCurator",
"scripts": { "scripts": {
+11
View File
@@ -72,6 +72,17 @@ body {
.btn.block { display: block; width: 100%; margin-top: 8px; } .btn.block { display: block; width: 100%; margin-top: 8px; }
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; } .btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
.btn.link:hover { color: var(--accent); } .btn.link:hover { color: var(--accent); }
.btn.small { padding: 6px 12px; font-size: 13px; }
/* In-app update prompt (accent-tinted so it reads as an actionable notice). */
.update-banner {
display: flex; align-items: center; gap: 10px;
margin: 10px 10px 0; padding: 10px 12px;
background: rgba(244, 186, 122, 0.12);
border: 1px solid rgba(244, 186, 122, 0.4);
border-radius: 6px;
}
#update-text { flex: 1; font-size: 13px; }
.source-row .play { .source-row .play {
background: none; border: none; color: var(--on-surface-variant); background: none; border: none; color: var(--on-surface-variant);
+5
View File
@@ -20,6 +20,11 @@
</section> </section>
<section id="main-content" class="main hidden"> <section id="main-content" class="main hidden">
<div id="update-banner" class="update-banner hidden">
<span id="update-text"></span>
<button id="update-btn" class="btn primary small">Update</button>
</div>
<nav class="tabs"> <nav class="tabs">
<button class="tab active" data-tab="platforms">Platforms</button> <button class="tab active" data-tab="platforms">Platforms</button>
<button class="tab" data-tab="sources">Sources</button> <button class="tab" data-tab="sources">Sources</button>
+33 -12
View File
@@ -2,6 +2,15 @@ document.addEventListener('DOMContentLoaded', init);
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000; const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
// A centered muted note div — the loading / empty state shared by the platform
// and sources lists.
function mutedNote(text) {
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = text;
return d;
}
async function init() { async function init() {
try { try {
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' }); const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
@@ -14,6 +23,7 @@ async function init() {
setupEventListeners(); setupEventListeners();
showPlatformsLoading(); showPlatformsLoading();
testConnectionIfNeeded(); testConnectionIfNeeded();
checkForUpdate();
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`)); loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
} catch (e) { } catch (e) {
showSetupRequired(); showSetupRequired();
@@ -37,10 +47,7 @@ function showSetupRequired() {
function showPlatformsLoading() { function showPlatformsLoading() {
const c = document.getElementById('platforms-list'); const c = document.getElementById('platforms-list');
c.textContent = ''; c.textContent = '';
const d = document.createElement('div'); c.appendChild(mutedNote('Loading platforms…'));
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading platforms…';
c.appendChild(d);
} }
async function testConnectionIfNeeded() { async function testConnectionIfNeeded() {
@@ -63,6 +70,26 @@ function updateConnectionDot(connected) {
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected'; d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
} }
// Nudge to reinstall when the configured instance publishes a newer signed XPI
// (the extension is self-hosted, so there's no Firefox auto-update). Never
// blocks the popup — a failed check just leaves the banner hidden.
async function checkForUpdate() {
try {
const r = await browser.runtime.sendMessage({ type: 'CHECK_UPDATE' });
if (r && r.updateAvailable && r.xpiUrl) showUpdateBanner(r);
} catch { /* non-fatal */ }
}
function showUpdateBanner(r) {
document.getElementById('update-text').textContent =
`Update available — v${r.latestVersion} (installed v${r.currentVersion})`;
// Opening the signed XPI triggers Firefox's native install prompt.
document.getElementById('update-btn').addEventListener('click', () => {
browser.tabs.create({ url: r.xpiUrl });
});
document.getElementById('update-banner').classList.remove('hidden');
}
async function loadPlatformStatus() { async function loadPlatformStatus() {
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' }); const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
const c = document.getElementById('platforms-list'); const c = document.getElementById('platforms-list');
@@ -162,10 +189,7 @@ async function exportAllCookies() {
async function loadSources() { async function loadSources() {
const c = document.getElementById('sources-list'); const c = document.getElementById('sources-list');
c.textContent = ''; c.textContent = '';
const d = document.createElement('div'); c.appendChild(mutedNote('Loading sources…'));
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading sources…';
c.appendChild(d);
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' }); const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
c.textContent = ''; c.textContent = '';
if (r.error) { if (r.error) {
@@ -176,10 +200,7 @@ async function loadSources() {
return; return;
} }
if (!r.sources || r.sources.length === 0) { if (!r.sources || r.sources.length === 0) {
const empty = document.createElement('div'); c.appendChild(mutedNote('No sources yet.'));
empty.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
empty.textContent = 'No sources yet.';
c.appendChild(empty);
return; return;
} }
for (const src of r.sources) c.appendChild(createSourceRow(src)); for (const src of r.sources) c.appendChild(createSourceRow(src));
@@ -0,0 +1,56 @@
<!--
Canonical settings number field (DRY pass #161): a compact numeric v-text-field
with a built-in clamp to [min,max] on commit. Hand-rolled identically across the
ML settings cards (HeadsCard x6, CropProposersCard, VideoEmbeddingCard).
The clamp is the point: the cards previously sent Number(raw) straight to the
API, so an out-of-range value bounced off the API's 400 validator (only
TranslationCard clamped). This is now the single home for that clamp.
Binds `modelValue` (v-model) and emits `change` on blur/enter AFTER clamping, so
the parent's save reads the already-clamped value same as the prior
`v-model.number` + `@change=save` pattern.
-->
<template>
<v-text-field
:model-value="modelValue"
:label="label"
type="number"
:min="min"
:max="max"
:step="step"
:disabled="disabled"
:density="density" hide-details
:style="{ maxWidth }"
@update:model-value="v => emit('update:modelValue', v)"
@change="onCommit"
/>
</template>
<script setup>
const props = defineProps({
modelValue: { type: [Number, String], default: null },
label: { type: String, default: '' },
min: { type: [Number, String], default: null },
max: { type: [Number, String], default: null },
step: { type: [Number, String], default: 1 },
maxWidth: { type: String, default: '200px' },
density: { type: String, default: 'compact' },
disabled: { type: Boolean, default: false },
})
const emit = defineEmits(['update:modelValue', 'change'])
function onCommit() {
// On blur/enter: coerce to a number and clamp to [min,max] so an out-of-range
// value never reaches the API. props.modelValue reflects the latest keystroke
// (kept in sync by the passthrough above); re-emit the clamped number, then let
// the parent persist.
let n = Number(props.modelValue)
if (!Number.isNaN(n)) {
if (props.min !== null && props.min !== '') n = Math.max(Number(props.min), n)
if (props.max !== null && props.max !== '') n = Math.min(Number(props.max), n)
if (n !== Number(props.modelValue)) emit('update:modelValue', n)
}
emit('change')
}
</script>
@@ -0,0 +1,42 @@
<!--
Canonical settings toggle row (DRY pass #161): an accent icon + an uppercase
.fc-section-h label + a right-aligned switch. Hand-rolled identically in the
ML settings cards (HeadsCard x3, CropProposersCard, MLBackfillCard).
Two-way binds `modelValue` (so the parent switch state stays optimistic) AND
emits `change` with the new boolean, so the parent can persist + revert on
failure matching the prior `v-model` + `@update:model-value=handler` pattern.
-->
<template>
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon v-if="icon" size="18" :color="iconColor">{{ icon }}</v-icon>
<span class="fc-section-h">{{ label }}</span>
<v-switch
:model-value="modelValue"
:loading="loading"
:disabled="disabled"
hide-details density="compact" color="success" class="ml-auto"
@update:model-value="onSwitch"
/>
</div>
</template>
<script setup>
defineProps({
modelValue: { type: Boolean, default: false },
label: { type: String, default: '' },
icon: { type: String, default: '' },
// Icon tint. Default accent; pass null for the theme default (e.g. when a row
// is off). null (not undefined) so the default doesn't override it.
iconColor: { type: String, default: 'accent' },
loading: { type: Boolean, default: false },
disabled: { type: Boolean, default: false },
})
const emit = defineEmits(['update:modelValue', 'change'])
function onSwitch(v) {
const b = !!v
emit('update:modelValue', b)
emit('change', b)
}
</script>
@@ -15,28 +15,23 @@
</p> </p>
<div v-for="p in proposers" :key="p.key" class="fc-proposer"> <div v-for="p in proposers" :key="p.key" class="fc-proposer">
<div class="d-flex align-center mb-1" style="gap: 10px;"> <SettingToggleRow
<v-icon size="18" :color="p.on ? 'accent' : undefined">{{ p.icon }}</v-icon> v-model="p.on" :loading="busy" :icon="p.icon"
<span class="fc-section-h">{{ p.label }}</span> :icon-color="p.on ? 'accent' : null" :label="p.label"
<v-switch @change="v => saveToggle(p, v)"
v-model="p.on" :loading="busy" hide-details density="compact" />
color="success" class="ml-auto"
@update:model-value="v => saveToggle(p, v)"
/>
</div>
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p> <p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
<div class="d-flex flex-wrap mb-4" style="gap: 12px;"> <div class="d-flex flex-wrap mb-4" style="gap: 12px;">
<v-text-field <v-text-field
v-model="p.weights" label="Weights" density="compact" hide-details v-model="p.weights" label="Weights" density="compact" hide-details
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on" style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
placeholder="name | URL | hf_repo::file" placeholder="name | URL | hf_repo::file"
@change="save({ [`detector_${p.key}_weights`]: p.weights })" @change="saveField({ [`detector_${p.key}_weights`]: p.weights })"
/> />
<v-text-field <SettingNumberField
v-model.number="p.conf" label="Confidence" type="number" v-model="p.conf" label="Confidence" :min="0" :max="1" :step="0.05"
min="0" max="1" step="0.05" density="compact" hide-details max-width="140px" :disabled="busy || !p.on"
style="max-width: 140px;" :disabled="busy || !p.on" @change="saveField({ [`detector_${p.key}_conf`]: Number(p.conf) })"
@change="save({ [`detector_${p.key}_conf`]: Number(p.conf) })"
/> />
</div> </div>
</div> </div>
@@ -48,12 +43,12 @@
storage. Dedupe IoU drops near-duplicate crops before embedding. storage. Dedupe IoU drops near-duplicate crops before embedding.
</p> </p>
<div class="d-flex flex-wrap" style="gap: 12px;"> <div class="d-flex flex-wrap" style="gap: 12px;">
<v-text-field <SettingNumberField
v-for="c in caps" :key="c.key" v-for="c in caps" :key="c.key"
v-model.number="c.val" :label="c.label" type="number" v-model="c.val" :label="c.label"
:min="c.min" :max="c.max" :step="c.step || 1" density="compact" :min="c.min" :max="c.max" :step="c.step || 1"
hide-details style="max-width: 165px;" :disabled="busy" max-width="165px" :disabled="busy"
@change="save({ [c.key]: Number(c.val) })" @change="saveField({ [c.key]: Number(c.val) })"
/> />
</div> </div>
</div> </div>
@@ -61,14 +56,16 @@
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js'
import { onMounted, ref } from 'vue' import { onMounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import SettingToggleRow from '../common/SettingToggleRow.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
const mlSettings = useMLStore() const mlSettings = useMLStore()
const busy = ref(false) const { busy, save } = useSettingSave(mlSettings.patchSettings)
const proposers = ref([]) const proposers = ref([])
const caps = ref([]) const caps = ref([])
@@ -111,31 +108,20 @@ onMounted(async () => {
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 })) caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
}) })
async function save(patch, revert) { // Field @change → persist with a "Saved" confirmation. SettingNumberField has
busy.value = true // already clamped numeric values to their [min,max] before this fires.
try { function saveField(patch) {
await mlSettings.patchSettings(patch) save(patch, { successMessage: 'Saved' })
toast({ text: 'Saved', type: 'success' })
} catch (e) {
if (revert) revert()
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
} }
function saveToggle (p, v) { async function saveToggle(p, v) {
// Revert the switch on failure so it never lies about the persisted state. // Revert the switch on failure so it never lies about the persisted state.
save({ [`detector_${p.key}_enabled`]: !!v }, () => { p.on = !v }) const ok = await save({ [`detector_${p.key}_enabled`]: !!v }, { successMessage: 'Saved' })
if (!ok) p.on = !v
} }
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-proposer { .fc-proposer {
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px; border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
} }
@@ -112,7 +112,6 @@ async function onCommit() {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-code { .fc-code {
background: rgb(var(--v-theme-surface-light)); background: rgb(var(--v-theme-surface-light));
border-radius: 4px; padding: 2px 8px; border-radius: 4px; padding: 2px 8px;
@@ -42,7 +42,7 @@
</tr> </tr>
</tbody> </tbody>
</v-table> </v-table>
<p v-else class="text-caption mt-3" style="opacity: 0.6;"> <p v-else class="text-caption mt-3 fc-muted">
No table statistics yet. No table statistics yet.
</p> </p>
</MaintenanceTile> </MaintenanceTile>
@@ -72,6 +72,5 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-bad { color: rgb(var(--v-theme-error)); } .fc-bad { color: rgb(var(--v-theme-error)); }
</style> </style>
@@ -95,7 +95,6 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-cells { display: flex; gap: 28px; } .fc-cells { display: flex; gap: 28px; }
.fc-cell__n { .fc-cell__n {
font-size: 20px; font-weight: 700; line-height: 1.1; font-size: 20px; font-weight: 700; line-height: 1.1;
@@ -105,6 +104,5 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
} }
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-bad { color: rgb(var(--v-theme-error)); } .fc-bad { color: rgb(var(--v-theme-error)); }
</style> </style>
@@ -367,11 +367,6 @@ async function onReprocess() {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-token { .fc-token {
display: flex; align-items: center; gap: 4px; display: flex; align-items: center; gap: 4px;
background: rgb(var(--v-theme-surface-light)); border-radius: 6px; background: rgb(var(--v-theme-surface-light)); border-radius: 6px;
@@ -390,6 +385,4 @@ async function onReprocess() {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
} }
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style> </style>
@@ -155,11 +155,6 @@ async function onRecover(it) {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-queue { display: flex; gap: 24px; } .fc-queue { display: flex; gap: 24px; }
.fc-q__n { .fc-q__n {
font-size: 20px; font-weight: 700; line-height: 1.1; font-size: 20px; font-weight: 700; line-height: 1.1;
@@ -169,8 +164,6 @@ async function onRecover(it) {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
} }
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
.fc-defect { .fc-defect {
display: flex; align-items: center; gap: 12px; display: flex; align-items: center; gap: 12px;
background: rgb(var(--v-theme-surface-light)); border-radius: 8px; background: rgb(var(--v-theme-surface-light)); border-radius: 8px;
+60 -125
View File
@@ -95,14 +95,10 @@
<!-- Earned auto-apply --> <!-- Earned auto-apply -->
<div class="fc-auto mt-6"> <div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;"> <SettingToggleRow
<v-icon size="18" color="accent">mdi-lightning-bolt</v-icon> v-model="autoEnabled" :loading="settingBusy"
<span class="fc-section-h">Auto-apply</span> icon="mdi-lightning-bolt" label="Auto-apply" @change="onToggleAuto"
<v-switch />
v-model="autoEnabled" :loading="settingBusy" hide-details density="compact"
color="success" class="ml-auto" @update:model-value="onToggleAuto"
/>
</div>
<p class="fc-muted text-body-2 mb-3"> <p class="fc-muted text-body-2 mb-3">
Graduated heads (, with {{ autoMinPosInput }} examples) apply their tag Graduated heads (, with {{ autoMinPosInput }} examples) apply their tag
on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}% on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}%
@@ -111,17 +107,14 @@
</p> </p>
<div class="d-flex mb-3" style="gap: 12px;"> <div class="d-flex mb-3" style="gap: 12px;">
<v-text-field <SettingNumberField
v-model.number="autoPrecisionInput" label="Precision target" v-model="autoPrecisionInput" label="Precision target"
type="number" min="0.5" max="0.999" step="0.01" density="compact" :min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveSettings" @change="onSaveSettings"
/> />
<v-text-field <SettingNumberField
v-model.number="autoMinPosInput" label="Min examples to fire" v-model="autoMinPosInput" label="Min examples to fire"
type="number" min="1" density="compact" hide-details :min="1" :disabled="settingBusy" @change="onSaveSettings"
style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveSettings"
/> />
</div> </div>
@@ -161,15 +154,11 @@
<!-- Presentation chrome auto-hide (#141) --> <!-- Presentation chrome auto-hide (#141) -->
<div class="fc-auto mt-6"> <div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;"> <SettingToggleRow
<v-icon size="18" color="accent">mdi-image-off-outline</v-icon> v-model="presentationEnabled" :loading="settingBusy"
<span class="fc-section-h">Hide presentation chrome</span> icon="mdi-image-off-outline" label="Hide presentation chrome"
<v-switch @change="onTogglePresentation"
v-model="presentationEnabled" :loading="settingBusy" hide-details />
density="compact" color="success" class="ml-auto"
@update:model-value="onTogglePresentation"
/>
</div>
<p class="fc-muted text-body-2 mb-3"> <p class="fc-muted text-body-2 mb-3">
Auto-hide <code>banner</code> chrome from the gallery once a head has Auto-hide <code>banner</code> chrome from the gallery once a head has
learned it ( {{ minPositives }} examples) and clears learned it ( {{ minPositives }} examples) and clears
@@ -180,16 +169,14 @@
tag), it's flagged for review instead of buried. Every auto-hide is reversible. tag), it's flagged for review instead of buried. Every auto-hide is reversible.
</p> </p>
<div class="d-flex mb-3" style="gap: 12px;"> <div class="d-flex mb-3" style="gap: 12px;">
<v-text-field <SettingNumberField
v-model.number="presentationThresholdInput" label="Hide confidence" v-model="presentationThresholdInput" label="Hide confidence"
type="number" min="0.5" max="0.999" step="0.01" density="compact" :min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSavePresentation" @change="onSavePresentation"
/> />
<v-text-field <SettingNumberField
v-model.number="presentationConflictInput" label="Flag if content ≥" v-model="presentationConflictInput" label="Flag if content ≥"
type="number" min="0" max="1" step="0.05" density="compact" :min="0" :max="1" :step="0.05" :disabled="settingBusy"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSavePresentation" @change="onSavePresentation"
/> />
</div> </div>
@@ -197,15 +184,11 @@
<!-- Process auto-tagging (#1464): wip / editor screenshot --> <!-- Process auto-tagging (#1464): wip / editor screenshot -->
<div class="fc-auto mt-6"> <div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;"> <SettingToggleRow
<v-icon size="18" color="accent">mdi-progress-wrench</v-icon> v-model="processEnabled" :loading="settingBusy"
<span class="fc-section-h">Auto-tag work-in-progress</span> icon="mdi-progress-wrench" label="Auto-tag work-in-progress"
<v-switch @change="onToggleProcess"
v-model="processEnabled" :loading="settingBusy" hide-details />
density="compact" color="success" class="ml-auto"
@update:model-value="onToggleProcess"
/>
</div>
<p class="fc-muted text-body-2 mb-3"> <p class="fc-muted text-body-2 mb-3">
Auto-tag <code>wip</code> and <code>editor screenshot</code> process art Auto-tag <code>wip</code> and <code>editor screenshot</code> process art
once a head has learned them (≥ {{ minPositives }} examples) and clears once a head has learned them (≥ {{ minPositives }} examples) and clears
@@ -217,16 +200,14 @@
manual tags, never its own guesses so it can't run away. Every tag reversible. manual tags, never its own guesses so it can't run away. Every tag reversible.
</p> </p>
<div class="d-flex mb-3" style="gap: 12px;"> <div class="d-flex mb-3" style="gap: 12px;">
<v-text-field <SettingNumberField
v-model.number="processThresholdInput" label="Tag confidence" v-model="processThresholdInput" label="Tag confidence"
type="number" min="0.5" max="0.999" step="0.01" density="compact" :min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveProcess" @change="onSaveProcess"
/> />
<v-text-field <SettingNumberField
v-model.number="processConflictInput" label="Flag if content ≥" v-model="processConflictInput" label="Flag if content ≥"
type="number" min="0" max="1" step="0.05" density="compact" :min="0" :max="1" :step="0.05" :disabled="settingBusy"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveProcess" @change="onSaveProcess"
/> />
</div> </div>
@@ -256,7 +237,7 @@
<td class="fc-r fc-mono">{{ c.n_auto_applied }}</td> <td class="fc-r fc-mono">{{ c.n_auto_applied }}</td>
<td class="fc-r fc-mono">{{ c.n_misfires }}</td> <td class="fc-r fc-mono">{{ c.n_misfires }}</td>
<td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)"> <td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)">
{{ ratePct(c.misfire_rate) }} {{ pct(c.misfire_rate) }}
</td> </td>
<td class="fc-r fc-mono">{{ c.n_underfires }}</td> <td class="fc-r fc-mono">{{ c.n_underfires }}</td>
</tr> </tr>
@@ -272,6 +253,9 @@ import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref } from 'vue' import { computed, onMounted, onUnmounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import SettingToggleRow from '../common/SettingToggleRow.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
import { useHeadsStore } from '../../stores/heads.js' import { useHeadsStore } from '../../stores/heads.js'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
@@ -285,7 +269,9 @@ let pollTimer = null
const autoEnabled = ref(false) const autoEnabled = ref(false)
const autoPrecisionInput = ref(0.97) const autoPrecisionInput = ref(0.97)
const autoMinPosInput = ref(30) const autoMinPosInput = ref(30)
const settingBusy = ref(false) // Shared settings-save flow (busy + toast + revert); `settingBusy` gates the
// toggles/fields, `save` returns ok/false for the optimistic-switch revert.
const { busy: settingBusy, save } = useSettingSave(mlSettings.patchSettings)
const autoBusy = ref(false) const autoBusy = ref(false)
const autoStatus = ref(null) const autoStatus = ref(null)
const metricsData = ref(null) const metricsData = ref(null)
@@ -395,81 +381,39 @@ function startAutoPoll() {
function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } } function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } }
async function onToggleAuto(val) { async function onToggleAuto(val) {
settingBusy.value = true const ok = await save({ head_auto_apply_enabled: !!val },
try { { successMessage: val ? 'Auto-apply on' : 'Auto-apply off', errorPrefix: 'Could not update' })
await mlSettings.patchSettings({ head_auto_apply_enabled: !!val }) if (!ok) autoEnabled.value = !val // revert the switch
toast({ text: val ? 'Auto-apply on' : 'Auto-apply off', type: 'success' })
} catch (e) {
autoEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
async function onSaveSettings() { async function onSaveSettings() {
settingBusy.value = true await save({
try { head_auto_apply_precision: Number(autoPrecisionInput.value),
await mlSettings.patchSettings({ head_auto_apply_min_positives: Number(autoMinPosInput.value),
head_auto_apply_precision: Number(autoPrecisionInput.value), })
head_auto_apply_min_positives: Number(autoMinPosInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
async function onTogglePresentation(val) { async function onTogglePresentation(val) {
settingBusy.value = true const ok = await save({ presentation_auto_apply_enabled: !!val },
try { { successMessage: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', errorPrefix: 'Could not update' })
await mlSettings.patchSettings({ presentation_auto_apply_enabled: !!val }) if (!ok) presentationEnabled.value = !val // revert the switch
toast({ text: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', type: 'success' })
} catch (e) {
presentationEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
async function onSavePresentation() { async function onSavePresentation() {
settingBusy.value = true await save({
try { presentation_auto_apply_threshold: Number(presentationThresholdInput.value),
await mlSettings.patchSettings({ presentation_conflict_threshold: Number(presentationConflictInput.value),
presentation_auto_apply_threshold: Number(presentationThresholdInput.value), })
presentation_conflict_threshold: Number(presentationConflictInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
async function onToggleProcess(val) { async function onToggleProcess(val) {
settingBusy.value = true const ok = await save({ process_auto_apply_enabled: !!val },
try { { successMessage: val ? 'WIP auto-tag on' : 'WIP auto-tag off', errorPrefix: 'Could not update' })
await mlSettings.patchSettings({ process_auto_apply_enabled: !!val }) if (!ok) processEnabled.value = !val // revert the switch
toast({ text: val ? 'WIP auto-tag on' : 'WIP auto-tag off', type: 'success' })
} catch (e) {
processEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
async function onSaveProcess() { async function onSaveProcess() {
settingBusy.value = true await save({
try { process_auto_apply_threshold: Number(processThresholdInput.value),
await mlSettings.patchSettings({ process_conflict_threshold: Number(processConflictInput.value),
process_auto_apply_threshold: Number(processThresholdInput.value), })
process_conflict_threshold: Number(processConflictInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
} }
function onPreview() { startSweep(true) } function onPreview() { startSweep(true) }
function onApplyNow() { startSweep(false) } function onApplyNow() { startSweep(false) }
@@ -495,7 +439,6 @@ function sweepConcepts(run) {
.sort((a, b) => b.applied - a.applied) .sort((a, b) => b.applied - a.applied)
} }
function sweepTotal(run) { return run?.n_applied ?? 0 } function sweepTotal(run) { return run?.n_applied ?? 0 }
function ratePct(x) { return x == null ? '—' : `${Math.round(x * 100)}%` }
function rateClass(x) { function rateClass(x) {
if (x == null) return '' if (x == null) return ''
if (x <= 0.03) return 'fc-good' if (x <= 0.03) return 'fc-good'
@@ -526,12 +469,6 @@ function relTime(iso) {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-auto { .fc-auto {
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px; border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px;
} }
@@ -588,7 +525,5 @@ function relTime(iso) {
background: rgb(var(--v-theme-surface-light)); background: rgb(var(--v-theme-surface-light));
padding: 1px 6px; border-radius: 999px; padding: 1px 6px; border-radius: 999px;
} }
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-ok { color: rgb(var(--v-theme-on-surface)); } .fc-ok { color: rgb(var(--v-theme-on-surface)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style> </style>
@@ -39,13 +39,14 @@
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { onMounted, ref } from 'vue' import { onMounted, ref } from 'vue'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import { useSettingSave } from '../../composables/useSettingSave.js'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const store = useMLStore() const store = useMLStore()
const { busy: saving, save } = useSettingSave(store.patchSettings)
const busy = ref(false) const busy = ref(false)
const done = ref(false) const done = ref(false)
const enabled = ref(true) const enabled = ref(true)
const saving = ref(false)
onMounted(async () => { onMounted(async () => {
try { try {
await store.loadSettings() await store.loadSettings()
@@ -55,21 +56,12 @@ onMounted(async () => {
} catch { /* non-fatal */ } } catch { /* non-fatal */ }
}) })
async function onToggle() { async function onToggle() {
saving.value = true const ok = await save({ cpu_embed_enabled: enabled.value }, {
try { successMessage: enabled.value
await store.patchSettings({ cpu_embed_enabled: enabled.value }) ? 'CPU embedding on — imports queue embeds for the ml-worker'
toast({ : 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
text: enabled.value })
? 'CPU embedding on — imports queue embeds for the ml-worker' if (!ok) enabled.value = !enabled.value
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
type: 'success',
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
enabled.value = !enabled.value
} finally {
saving.value = false
}
} }
async function run() { async function run() {
busy.value = true busy.value = true
@@ -80,5 +72,4 @@ async function run() {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style> </style>
@@ -24,6 +24,7 @@
the CPU fallback. the CPU fallback.
</p> </p>
<div class="fc-tile-stack"> <div class="fc-tile-stack">
<VideoEmbeddingCard />
<GpuAgentCard /> <GpuAgentCard />
<GpuTriageCard /> <GpuTriageCard />
<MLBackfillCard /> <MLBackfillCard />
@@ -36,7 +37,6 @@
Suggestion thresholds, trained heads and tag aliases. Suggestion thresholds, trained heads and tag aliases.
</p> </p>
<div class="fc-tile-stack"> <div class="fc-tile-stack">
<MLThresholdSliders />
<CropProposersCard /> <CropProposersCard />
<HeadsCard /> <HeadsCard />
<AliasTable /> <AliasTable />
@@ -77,7 +77,7 @@ import ArchiveReextractCard from './ArchiveReextractCard.vue'
import MissingFileRepairCard from './MissingFileRepairCard.vue' import MissingFileRepairCard from './MissingFileRepairCard.vue'
import GpuTriageCard from './GpuTriageCard.vue' import GpuTriageCard from './GpuTriageCard.vue'
import DbMaintenanceCard from './DbMaintenanceCard.vue' import DbMaintenanceCard from './DbMaintenanceCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue' import VideoEmbeddingCard from './VideoEmbeddingCard.vue'
import CropProposersCard from './CropProposersCard.vue' import CropProposersCard from './CropProposersCard.vue'
import HeadsCard from './HeadsCard.vue' import HeadsCard from './HeadsCard.vue'
import GpuAgentCard from './GpuAgentCard.vue' import GpuAgentCard from './GpuAgentCard.vue'
@@ -12,17 +12,17 @@
</div> </div>
<v-row> <v-row>
<v-col cols="12" sm="6"> <v-col cols="12" sm="6">
<v-text-field <SettingNumberField
v-model.number="local.video_frame_interval_seconds" v-model="local.video_frame_interval_seconds"
label="Frame interval (s)" type="number" min="0.5" step="0.5" label="Frame interval (s)" :min="0.5" :step="0.5"
density="comfortable" hide-details @change="save" density="comfortable" max-width="none" @change="onSave"
/> />
</v-col> </v-col>
<v-col cols="12" sm="6"> <v-col cols="12" sm="6">
<v-text-field <SettingNumberField
v-model.number="local.video_max_frames" v-model="local.video_max_frames"
label="Max frames" type="number" min="1" step="1" label="Max frames" :min="1" :step="1"
density="comfortable" hide-details @change="save" density="comfortable" max-width="none" @change="onSave"
/> />
</v-col> </v-col>
</v-row> </v-row>
@@ -32,21 +32,23 @@
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js'
import { reactive, watch } from 'vue' import { reactive, watch } from 'vue'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
const store = useMLStore() const store = useMLStore()
const { save } = useSettingSave(store.patchSettings)
const local = reactive({}) const local = reactive({})
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true }) watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
async function save() { // SettingNumberField clamps interval to 0.5 and max-frames to 1 before this
const patch = { // fires, so an out-of-range value never reaches the API.
video_frame_interval_seconds: local.video_frame_interval_seconds, function onSave() {
video_max_frames: local.video_max_frames save({
} video_frame_interval_seconds: Number(local.video_frame_interval_seconds),
try { await store.patchSettings(patch) } video_max_frames: Number(local.video_max_frames),
catch (e) { toast({ text: e.message, type: 'error' }) } })
} }
</script> </script>
@@ -0,0 +1,33 @@
import { ref } from 'vue'
import { toast } from '../utils/toast.js'
// The shared "persist a settings patch" flow for the ML settings cards. Flips a
// busy flag, calls the store's patch (which rethrows on failure), toasts
// success/error, and returns true/false so a toggle handler can revert its
// optimistic switch on failure. Centralises the try/catch/toast the cards each
// hand-rolled (HeadsCard x6, CropProposersCard, MLBackfillCard) — and where the
// threshold-clamp drifted; the clamp now lives in <SettingNumberField>.
//
// Pass the store's patch fn, e.g. useSettingSave(ml.patchSettings).
export function useSettingSave(patchFn) {
const busy = ref(false)
// opts.successMessage — toast on success (toggles announce their new state;
// silent field-saves omit it). opts.errorPrefix — the failure toast prefix
// ("Could not save" default; toggles used "Could not update").
async function save(patch, { successMessage = '', errorPrefix = 'Could not save' } = {}) {
busy.value = true
try {
await patchFn(patch)
if (successMessage) toast({ text: successMessage, type: 'success' })
return true
} catch (e) {
toast({ text: `${errorPrefix}: ${e.message}`, type: 'error' })
return false
} finally {
busy.value = false
}
}
return { busy, save }
}
+14
View File
@@ -40,6 +40,20 @@
emits, so no specificity/reorder fight — no !important needed. */ emits, so no specificity/reorder fight — no !important needed. */
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); } .fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
/* Section sub-heading in settings cards (DRY pass #161): was redefined
identically in 4 cards, and TranslationCard used the class with NO local def
so its section headers rendered unstyled. Now one global utility. */
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
/* Status text colours (DRY pass #161): fc-good = success, fc-weak = error,
consolidated from the GPU / heads cards. fc-ok is intentionally NOT global —
it means on-surface in HeadsCard but success in QueuesTable. */
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
/* Vuetify 4 dropped its global CSS reset (normalisation moved into each /* Vuetify 4 dropped its global CSS reset (normalisation moved into each
component). FC's layouts assumed the reset zeroed margins on text elements, so component). FC's layouts assumed the reset zeroed margins on text elements, so
restore just that — the "minimal reset" from the v4 upgrade guide — inside restore just that — the "minimal reset" from the v4 upgrade guide — inside
-1
View File
@@ -284,7 +284,6 @@ onUnmounted(() => {
</script> </script>
<style scoped> <style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
/* Full-height workspace under the sticky top nav. --fc-nav-h is the nav's REAL /* Full-height workspace under the sticky top nav. --fc-nav-h is the nav's REAL
measured height (set by TopNav) — a hardcoded 64px here overflowed the measured height (set by TopNav) — a hardcoded 64px here overflowed the
-1
View File
@@ -349,7 +349,6 @@ async function onDeleteTagConfirm() {
.fc-tags__sentinel { .fc-tags__sentinel {
display: flex; justify-content: center; padding: 32px 0; min-height: 60px; display: flex; justify-content: center; padding: 32px 0; min-height: 60px;
} }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-merge-preview { .fc-merge-preview {
padding: 10px 12px; padding: 10px 12px;
border: 1px solid rgba(var(--v-theme-on-surface), 0.12); border: 1px solid rgba(var(--v-theme-on-surface), 0.12);
+59
View File
@@ -0,0 +1,59 @@
"""Shared ML helpers extracted in the DRY pass (milestone #161). These pin the
single sources the auto-apply sweeps now trust, so a future edit can't silently
drift them: `_applied_or_rejected` is the skip-set used by auto_apply_sweep,
system_tag_auto_apply_sweep (heads.py) and scheduled_ccip_auto_apply (tasks/ml.py);
`_sigmoid` is the head score→prob transform used at every scoring site."""
import pytest
from backend.app.models import ImageRecord, Tag, TagKind, TagSuggestionRejection
from backend.app.models.tag import image_tag
from backend.app.services.ml.training_data import _applied_or_rejected
def test_sigmoid_matches_naive_form():
import numpy as np
from backend.app.services.ml.heads import _sigmoid
z = np.array([-3.0, -0.5, 0.0, 1.5, 12.0], dtype=np.float32)
assert np.allclose(_sigmoid(z, np), 1.0 / (1.0 + np.exp(-z)))
assert float(_sigmoid(np.array([0.0]), np)[0]) == pytest.approx(0.5)
@pytest.mark.integration
def test_applied_or_rejected_unions_applied_any_source_and_rejected(db_sync):
a = Tag(name="dry-helper-a", kind=TagKind.general)
b = Tag(name="dry-helper-b", kind=TagKind.general)
db_sync.add_all([a, b])
db_sync.flush()
imgs = []
for i in range(5):
img = ImageRecord(
path=f"/images/dryhelp{i}.jpg", sha256=f"{i:064d}", size_bytes=1,
mime="image/jpeg", width=1, height=1, origin="imported_filesystem",
integrity_status="unknown", siglip_embedding=[0.0] * 1152,
)
db_sync.add(img)
imgs.append(img)
db_sync.flush()
# tag a: applied manually (img0), applied by an AUTO source (img1), rejected (img2).
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[0].id, tag_id=a.id, source="manual"))
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[1].id, tag_id=a.id, source="head_auto"))
db_sync.add(TagSuggestionRejection(image_record_id=imgs[2].id, tag_id=a.id))
# tag b: applied to img3 only.
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[3].id, tag_id=b.id, source="manual"))
db_sync.flush()
skip = _applied_or_rejected(db_sync, [a.id, b.id])
# Applied-under-ANY-source (manual + head_auto) rejected, kept per-tag; the
# untouched image (img4) appears under neither tag.
assert skip[a.id] == {imgs[0].id, imgs[1].id, imgs[2].id}
assert skip[b.id] == {imgs[3].id}
assert imgs[4].id not in skip[a.id]
assert imgs[4].id not in skip[b.id]
+63
View File
@@ -0,0 +1,63 @@
"""recover_stalled_head_training_runs + recover_stalled_head_auto_apply_runs share
one helper (_recover_stalled_runs, DRY pass #161). These pin BOTH wrappers so the
shared source stays correct: a 'running' row with no progress past the stall
threshold flips to 'error'; a fresh 'running' row is left alone."""
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import select
from backend.app.models import HeadAutoApplyRun, HeadTrainingRun
pytestmark = pytest.mark.integration
def test_recover_stalled_head_training_runs_flips_stalled_keeps_fresh(db_sync):
from backend.app.tasks.maintenance import recover_stalled_head_training_runs
stale = HeadTrainingRun(
params={}, status="running",
last_progress_at=datetime.now(UTC) - timedelta(days=1),
)
fresh = HeadTrainingRun(
params={}, status="running", last_progress_at=datetime.now(UTC),
)
db_sync.add_all([stale, fresh])
db_sync.commit()
stale_id, fresh_id = stale.id, fresh.id
assert recover_stalled_head_training_runs.apply().get() == 1
db_sync.expire_all()
assert db_sync.execute(
select(HeadTrainingRun.status).where(HeadTrainingRun.id == stale_id)
).scalar_one() == "error"
assert db_sync.execute(
select(HeadTrainingRun.status).where(HeadTrainingRun.id == fresh_id)
).scalar_one() == "running"
def test_recover_stalled_head_auto_apply_runs_flips_stalled_keeps_fresh(db_sync):
from backend.app.tasks.maintenance import recover_stalled_head_auto_apply_runs
stale = HeadAutoApplyRun(
dry_run=False, params={}, status="running",
last_progress_at=datetime.now(UTC) - timedelta(days=1),
)
fresh = HeadAutoApplyRun(
dry_run=False, params={}, status="running",
last_progress_at=datetime.now(UTC),
)
db_sync.add_all([stale, fresh])
db_sync.commit()
stale_id, fresh_id = stale.id, fresh.id
assert recover_stalled_head_auto_apply_runs.apply().get() == 1
db_sync.expire_all()
assert db_sync.execute(
select(HeadAutoApplyRun.status).where(HeadAutoApplyRun.id == stale_id)
).scalar_one() == "error"
assert db_sync.execute(
select(HeadAutoApplyRun.status).where(HeadAutoApplyRun.id == fresh_id)
).scalar_one() == "running"