Compare commits

..
15 Commits
Author SHA1 Message Date
bvandeusen 8300029741 Merge pull request 'Extension: fix credential-push 405, guard the publish path, add a unit suite' (#234) from dev into main
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 32s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 1m14s
Build images / sign-extension (push) Successful in 2m23s
Build images / build-ml (push) Successful in 3m53s
CI / integration (push) Successful in 3m56s
Build images / build-web (push) Successful in 2m34s
Build images / build-agent (push) Successful in 13m33s
2026-08-03 08:29:09 -04:00
Claude f9111c06a7 test(extension): unit suite for lib/ + version-consistency specs
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 43s
CI / integration (push) Successful in 3m53s
extension / lint (pull_request) Successful in 19s
extension/ had no test harness at all -- web-ext lint was the only signal, so
the URL-normalization fix in 8214afe shipped with nothing exercising it.

Adds vitest (mirroring frontend/vitest.config.js) and three specs:

- url.spec.js       normalizeApiUrl / webRootFromApiUrl, including the #2393
                    regression: instance-root input must reach /api/credentials,
                    idempotence, trailing-slash and whitespace handling, and
                    that empty input never yields a bare "/api" (which
                    isConfigured() would read as configured).
- platforms.spec.js getPlatformFromUrl / isArtistPage, pinning the #1485
                    regression -- all three Patreon creator URL shapes
                    (bare, /c/, /cw/) plus inner pages, with nav pages
                    excluded -- and table-integrity checks.
- version.spec.js   manifest.json and package.json versions in lockstep,
                    AMO-safe version format, and url.js ordered before api.js
                    in background.scripts (classic scripts share one scope, so
                    a reorder is a runtime ReferenceError with no build signal).

Specs load lib/*.js by evaluating the real file as a classic script
(test/helpers/loadLib.js) instead of adding module.exports shims to production
code that would never run in the browser. The suite therefore exercises exactly
the bytes packaged into the XPI.

Two packaging consequences, both handled:

- web-ext would otherwise bundle test/ and vitest.config.js INTO the XPI;
  both are now in --ignore-files across all four web-ext scripts.
- ci.yml's extension-version guard must ignore the same paths, or editing a
  spec would demand a pointless version bump. The dangerous drift direction is
  the opposite one -- a guard exclusion for a file that DOES ship would let a
  real change pass unnoticed -- so version.spec.js asserts every :(exclude) in
  ci.yml appears in --ignore-files.

extension.yml also triggers on ci.yml now, since version.spec.js reads it.

Refs #2393, #2397

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 23:47:18 -04:00
Claude c37a180c3c ci: guard the extension publish path against a missed version bump
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 45s
CI / integration (push) Successful in 3m59s
build.yml's sign-extension keys its AMO-signing cache purely on the version
string in extension/package.json. If an ext-<version> release already has an
XPI, signing is skipped and build-web bakes that OLD signed XPI into :latest.
Nothing in that path inspects whether extension/ actually changed, so a
forgotten bump ships a stale extension on a fully green build -- silently, and
as the default outcome of forgetting. AMO can't backstop it either: it 409s on
re-signing a version, which is precisely why the cache exists.

New extension-version job, pure git + text, no deps or services:

1. Unconditional consistency check. manifest.json and package.json versions
   must match. web-ext sign reads manifest.json (package.json is in
   --ignore-files and isn't even inside the XPI), so AMO signs the manifest
   version; build.yml keys its cache, release tag, XPI filename -- and so the
   version /api/extension/manifest reports to the update prompt -- on
   package.json. Divergence either 409s at AMO or ships an XPI whose update
   prompt lies about what's installed.

2. Changed-without-bump check. If any PACKAGED file under extension/ differs,
   the version must have moved. Exclusions mirror --ignore-files so a Renovate
   web-ext devDep bump in package.json doesn't falsely demand one.

Compared against main rather than the previous push: the publish decision is
made at merge-to-main against whatever ext-<version> exists, so "differs from
main" is the question that matters. Diffing against the previous dev push
would demand a fresh bump on every iteration, inflating the version to buy
nothing.

Bumping stays manual -- making it automatic requires rewriting the version in
CI and committing back to a protected branch, which this workflow deliberately
avoided. This only ensures a missed bump can no longer be silent.

Refs #2393

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 20:20:44 -04:00
Claude 8214afee1e fix(extension): normalize FC URL so credential push doesn't 405
CI / lint (push) Successful in 3s
extension / lint (push) Successful in 38s
CI / frontend-build (push) Successful in 40s
CI / backend-lint-and-test (push) Successful in 2m58s
CI / integration (push) Successful in 4m56s
The stored apiUrl was required to already carry the `/api` suffix, since
api.js builds requests as `${baseUrl}/credentials`. The options label read
"FC base URL", so entering the instance root -- the natural reading --
sent every request one path segment short: POST /credentials hit the Vue
SPA catch-all and came back 405, and GET /extension/manifest 404'd.

Worse, Test Connection reported success on it: the catch-all answers GET
/credentials with 200 HTML, so `r.ok` was true and the only affordance
meant to catch this misconfiguration actively masked it.

Normalize instead of validate (rules 92, 26):

- New lib/url.js: normalizeApiUrl / webRootFromApiUrl, one source shared
  by the background client and the options page. Accepts either the
  instance root or the API root.
- api.js normalizes on read, so configs already stored in the broken form
  heal themselves without the operator reopening Settings.
- options.js stores the canonical form, echoes back what it saved, and
  the test now asserts a JSON content-type -- killing the false green.
- 404/405 in request() now names the URL and points at the setting.
- Options label/placeholder state that both forms work.

Version 1.0.9 -> 1.0.10 in BOTH manifest.json and package.json; build.yml
resolves the release version from package.json, and a stale value there
would hit the cached ext-1.0.9 asset and republish the old XPI unsigned
against the new code.

Refs #2393

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 19:37:38 -04:00
bvandeusen 306de50f61 docs: Fabled-Git, not Forgejo, in ci-requirements
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 52s
CI / frontend-build (push) Successful in 25s
CI / integration (push) Successful in 4m14s
The instance has run Gitea since the migration. Also fixes a dead rulebook
pointer: the topic was renamed forgejo.md -> fabled-git.md, so the
"CI philosophy" reference pointed at a file that no longer exists.

Prose only — no workflow or path change. Scribe issue #2272.
2026-07-31 23:42:47 -04:00
bvandeusen b5b437ca80 Merge pull request 'feat(agent): idle-unload GPU models to free VRAM when the queue is idle' (#233) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 34s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 25s
Build images / build-web (push) Successful in 3m20s
Build images / build-ml (push) Successful in 4m29s
CI / integration (push) Successful in 4m26s
Build images / build-agent (push) Successful in 11m55s
2026-07-17 13:07:15 -04:00
bvandeusenandClaude Opus 4.8 57e52433d0 feat(agent): idle-unload GPU models to free VRAM when the queue is idle
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 4m3s
The SigLIP embedder + YOLO proposers load lazily then stay resident for the
container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of
VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep
mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded.

Add a monitor thread that unloads the torch-owned models after
cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with
the GPU genuinely idle (active==0, buffer drained, no job completed in the
window), then torch.cuda.empty_cache() to hand the blocks back to the driver.
They reload lazily on the next job via the existing _ensure_embedder /
_proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in
/status (models_loaded) and the agent UI pipe line; the VRAM meter drops too.

Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident
(no clean unload API) — idle VRAM drops substantially, not to zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
2026-07-17 12:57:31 -04:00
bvandeusen ce0dac3524 Merge pull request 'DRY + stability pass — extension, ML/settings backend, frontend cards (#161)' (#232) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 12s
extension / lint (push) Successful in 11s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 50s
CI / integration (push) Successful in 3m47s
2026-07-13 23:08:18 -04:00
bvandeusenandClaude Opus 4.8 ec66ea5f83 refactor(ui): settings-card primitives + fix threshold clamp / card misgroup (#161)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m52s
extension / lint (pull_request) Successful in 10s
Tier-3 frontend DRY for the ML settings cards, plus the F-D2 clamp bug and the
F-D3 card misgrouping.

New primitives (components/common + composables):
- <SettingToggleRow> — the accent-icon + .fc-section-h label + right-aligned
  switch row (HeadsCard x3, CropProposersCard). iconColor prop absorbs the
  on/off dim.
- <SettingNumberField> — compact numeric field that CLAMPS to [min,max] on
  commit. This fixes F-D2: HeadsCard/CropProposersCard previously sent
  Number(raw) straight to the API, so an out-of-range threshold bounced off the
  400 validator (only TranslationCard clamped). density prop for the grid cards.
- useSettingSave(patchFn) — the busy + patch + toast + revert-on-failure flow
  each card hand-rolled (HeadsCard x6 handlers, CropProposersCard, MLBackfillCard,
  VideoEmbeddingCard). Returns ok/false for the optimistic-switch revert.

Adopted in HeadsCard, CropProposersCard, MLBackfillCard (handler only — its
plain labelled switch is a different affordance), VideoEmbeddingCard.

F-D3: MLThresholdSliders.vue actually rendered a "Video embedding" (frame-
sampling) card but sat under "Tagging → Suggestion thresholds". Renamed it
VideoEmbeddingCard.vue and moved it to the "GPU agent & embeddings" section.

Left deliberately (over-DRY guard): TranslationCard uses an inline error ALERT
(not a toast), already clamps its confidence with a NaN fallback, and lives on
the ImportStore — a genuinely different save pattern, so forcing it onto
useSettingSave would change its UX.

Behaviour-preserving refactor; CI has no Vue type-check so this needs a live
UI pass (toggles persist + revert on failure, thresholds clamp on blur, video
card now under Embeddings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 22:28:47 -04:00
bvandeusenandClaude Opus 4.8 e92570a31e refactor(extension): DRY the web-root transform + cookie-export flow (#161)
CI / lint (push) Successful in 2s
extension / lint (push) Successful in 11s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m53s
Behavior-preserving (extension JS has lint-only CI + your manual test):
- api.webRoot() single-sources the baseUrl→web-root transform (strip trailing
  slash + /api) that was copy-pasted in background.js's self-update check and
  OPEN_ARTIST_PAGE, whose comments even cross-referenced each other.
- exportPlatformCookies(key) shares the extract→verify→upload spine between
  EXPORT_COOKIES (single) and EXPORT_ALL_COOKIES; it returns a structured
  outcome so each caller keeps its own response/skip messages verbatim.
- popup.js mutedNote(text) replaces the "centered muted note" div hand-rolled
  in the platform-loading, sources-loading, and empty-sources renderers.

No version bump — no behavior change, so it rides the next real ext release
rather than forcing an AMO re-sign + reinstall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:57:14 -04:00
bvandeusenandClaude Opus 4.8 a2d1ed935d refactor(ui): DRY the settings-card CSS tokens + fix unstyled headers (#161)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m57s
- Promote .fc-section-h to a global token (app.css). It was copied identically
  into 4 cards, and TranslationCard used the class with NO local def — so its
  section headers rendered unstyled. Now fixed everywhere.
- Promote .fc-good / .fc-weak status colours to globals; delete the local copies
  in the GPU/heads cards. (.fc-ok stays local — divergent: on-surface in
  HeadsCard vs success in QueuesTable. .fc-bad stays — different name.)
- Delete 10 identical local .fc-muted redefinitions that crept back after the
  2026-06-09 sweep; the global utility already covers them.
- DbMaintenanceCard: opacity:0.6 muted text → the .fc-muted token (the exact
  anti-pattern that token's comment forbids).
- HeadsCard: collapse byte-identical ratePct() into pct().

CSS-only + one template class swap; no logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:51:23 -04:00
bvandeusenandClaude Opus 4.8 05df51b749 fix(ml): drop unnecessary quotes on MLSettings.load annotations (UP037)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m51s
Python 3.14 evaluates annotations lazily, so the self-referential return
type needs no forward-ref quotes — matches ImportSettings.load. Fixes the
ruff lint lane on the DRY-pass push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:49:30 -04:00
bvandeusenandClaude Opus 4.8 099e1e664c refactor(patreon): DRY the campaigns-API request (#161)
CI / lint (push) Failing after 2s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / integration (push) Successful in 4m0s
_lookup_via_api and resolve_display_name shared ~90% of their body (same
endpoint, params, headers, error handling — differing only in which field
they pluck from data[0]). Extract _campaigns_api_first(vanity, cookies_path)
-> dict|None; callers pluck the campaign id vs the display name. Return-value
behavior preserved (the display-name path additionally gains the helper's more
granular warning logs). Covered by the existing test_patreon_resolver.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:34 -04:00
bvandeusenandClaude Opus 4.8 c87f8a1bb3 refactor(maintenance): DRY the recover_stalled head-run twins (#161)
recover_stalled_head_training_runs and recover_stalled_head_auto_apply_runs
were near-exact copies (coalesce-flip + keep-last-N prune, differing only in
model + two constants). Extract _recover_stalled_runs(model, stall_minutes,
keep_runs, label); the two tasks become thin wrappers. The other two recover
tasks are deliberately NOT folded in (library-audit has no prune tail; backup
uses a single started_at cutoff). test_recover_stalled_head_runs.py covers
both wrappers (stalled→error, fresh survives) — previously untested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:34 -04:00
bvandeusenandClaude Opus 4.8 666b3a2ec8 refactor(ml): DRY pass — shared sweep helpers + table-driven settings (#161)
Consolidate duplication accrued across the ML tagging + settings backend,
behavior-preserving (over-DRY guard applied — the three auto-apply sweep
BODIES stay separate; only their shared inner helpers are extracted).

- _sigmoid / _conflict_scores / _insert_presentation_review (heads.py): the
  score→prob transform (6 inlined sites), the presentation conflict signal
  (2 sites), and the ring-loud PresentationReview insert (2 sites, single-
  sourced so the mode column can't drift on the shared composite PK).
- _applied_or_rejected (training_data.py): the per-tag "applied ∪ rejected"
  skip-set, byte-identical at 3 sweep sites (heads.py x2, tasks/ml.py ccip).
- ccip sweep divergence fixes: import ccip._FIGURE_KINDS + training_data._l2norm
  instead of local copies that silently drift when the canonical changes.
- MLSettings.load / .load_sync classmethods (mirror ImportSettings); route all
  8 scalar_one singleton reads through them (the session.get None-path stays).
- GET serializers for MLSettings + ImportSettings are now table-driven off the
  same _EDITABLE tuples PATCH writes, so a new field can't be silently absent
  from GET (the split that historically dropped fields).
- AUTO_APPLY_THRESHOLD_MIN/MAX constant single-sources the [0.5,0.999] operating
  range across the service clamp + the 5 API validators.
- test_ml_dry_helpers.py pins _applied_or_rejected + _sigmoid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi
2026-07-13 21:41:24 -04:00
50 changed files with 1307 additions and 569 deletions
+116
View File
@@ -2,6 +2,7 @@ name: CI
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
# - extension-version: guards the extension publish path (see the job).
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
# - frontend-build: vitest unit + vite build.
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
@@ -41,6 +42,121 @@ jobs:
# catching syntax errors before the image build.
run: python -m compileall -q agent/fc_agent
# Guards the extension publish path, which has no self-correcting behavior.
#
# build.yml's sign-extension job keys its AMO-signing cache purely on the
# version string in extension/package.json: if an `ext-<version>` Forgejo
# release already carries an XPI, signing is SKIPPED and that old signed XPI
# is what build-web bakes into `:latest`. Nothing in that path inspects
# whether extension/ actually changed — so a forgotten version bump ships a
# stale extension on a fully green build, silently. (AMO can't help: it 409s
# on re-signing a version, which is exactly why the cache exists.)
#
# This job makes that case loud, on the dev push, instead of invisible at
# merge-to-main. It is pure git + text work — no deps, no services.
extension-version:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Full history: the check diffs against the push's `before` SHA (or
# the PR base), which a depth-1 clone wouldn't contain.
fetch-depth: 0
- name: Extension version guard
env:
BEFORE: ${{ github.event.before }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
run: |
set -eu
# busybox sh on the act_runner — no bashisms (family rule).
ver() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/'; }
PKG=$(ver extension/package.json)
MAN=$(ver extension/manifest.json)
test -n "$PKG" || { echo "ERROR: no version found in extension/package.json"; exit 1; }
test -n "$MAN" || { echo "ERROR: no version found in extension/manifest.json"; exit 1; }
# (1) Unconditional: the two version strings must agree. `web-ext sign`
# reads manifest.json (package.json sits in --ignore-files and isn't
# even inside the XPI), so AMO signs MAN and Firefox installs MAN.
# build.yml keys its cache, release tag, XPI filename — and therefore
# the version /api/extension/manifest reports to the update prompt —
# on PKG. Divergence either hard-fails at AMO or ships a mislabelled
# XPI whose update prompt lies about what's installed.
if [ "$MAN" != "$PKG" ]; then
echo "ERROR: extension version mismatch."
echo " extension/manifest.json = $MAN <- what AMO signs / Firefox installs"
echo " extension/package.json = $PKG <- what CI caches, names, and reports"
echo "Set both to the same value."
exit 1
fi
# (2) If the SHIPPED extension changed, the version must have moved.
#
# Compare against MAIN, not against the previous push. The publish
# decision is made at merge-to-main against whatever ext-<version>
# already exists, so "differs from main" is the question that matters.
# Diffing against the previous dev push instead would demand a fresh
# bump on every iteration — push, tweak the extension again, and CI
# would insist on a second bump that buys nothing, inflating the
# version for no reason. On a main push there is no "main to compare
# to" yet, so fall back to that push's own before-SHA.
if [ "${GITHUB_REF##*/}" = "main" ]; then
BASE="${BEFORE:-}"
else
BASE=$(git rev-parse --verify -q origin/main 2>/dev/null || git rev-parse --verify -q main 2>/dev/null || echo "")
# PR base is the fallback when main isn't in the clone at all.
[ -n "$BASE" ] || BASE="${PR_BASE:-}"
fi
case "$BASE" in
''|0000000000000000000000000000000000000000)
echo "No usable base ref (no main in clone / first push) — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
;;
esac
if ! git cat-file -e "$BASE^{commit}" 2>/dev/null; then
echo "Base commit $BASE not in this clone — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
# Exclusions mirror --ignore-files in extension/package.json's web-ext
# scripts: these files are not packaged into the XPI, so touching them
# (e.g. Renovate bumping the web-ext devDep, or editing a spec)
# changes nothing shipped and must not demand a version bump.
# KEEP IN SYNC with --ignore-files — a file packaged into the XPI but
# excluded here is exactly the silent-stale-ship this job exists to
# prevent. test/version.spec.js pins the two lists' shared intent.
CHANGED=$(git diff --name-only "$BASE" HEAD -- extension/ \
':(exclude)extension/package.json' \
':(exclude)extension/package-lock.json' \
':(exclude)extension/README.md' \
':(exclude)extension/.gitignore' \
':(exclude)extension/vitest.config.js' \
':(exclude)extension/test/**')
if [ -z "$CHANGED" ]; then
echo "No packaged extension files changed since $BASE — nothing to guard."
echo "OK: extension version $PKG"
exit 0
fi
echo "Packaged extension files changed since $BASE:"
echo "$CHANGED" | sed 's/^/ /'
PKG_OLD=$(git show "$BASE:extension/package.json" 2>/dev/null | grep -E '"version"' | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
if [ -z "$PKG_OLD" ]; then
echo "Could not read the base version — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
if [ "$PKG_OLD" = "$PKG" ]; then
echo "ERROR: packaged extension files changed but the version is still $PKG."
echo "build.yml would find the existing ext-$PKG release, skip AMO signing,"
echo "and bake the OLD signed XPI into :latest — a green build shipping stale code."
echo "Bump the version in BOTH extension/package.json and extension/manifest.json."
exit 1
fi
echo "OK: extension version $PKG_OLD -> $PKG"
backend-lint-and-test:
runs-on: python-ci
container:
+14 -3
View File
@@ -1,5 +1,5 @@
name: extension
# Lint-only workflow. The sign-and-publish dance moved into build.yml's
# Lint + unit tests. The sign-and-publish dance moved into build.yml's
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
# because sign-extension runs as a build-web dependency in the SAME workflow,
# eliminating the prior race between build.yml and a separate extension.yml.
@@ -10,10 +10,15 @@ on:
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
# test/version.spec.js asserts ci.yml's extension-version guard never
# ignores a file web-ext actually packages, so a ci.yml-only edit can
# break this suite and must trigger it.
- '.forgejo/workflows/ci.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
- '.forgejo/workflows/ci.yml'
workflow_dispatch:
jobs:
@@ -23,7 +28,13 @@ jobs:
image: node:24-bookworm-slim
steps:
- uses: actions/checkout@v4
- name: Install web-ext
run: cd extension && npm install --no-save --no-audit --no-fund
# Not --no-save: vitest and web-ext are both real devDependencies now,
# and the suite needs vitest resolvable from node_modules.
- name: Install dev dependencies
run: cd extension && npm install --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
# Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
# package version-consistency checks. No browser, no network.
- name: Unit tests
run: cd extension && npm run test:unit
+4 -1
View File
@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
# Bump on every agent change. The page embeds this and /status reports it; the UI
# warns to reload when they differ — so a stale browser-cached page can't be
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
VERSION = "2026-07-02.6 · sleep mode: an empty queue sheds to one downloader and backs the lease poll off to 15 min"
VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
logbuf.install()
cfg = Config.from_env()
@@ -334,9 +334,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
waited.textContent=s.transient||0
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
// as live churn rather than a "broken" headline metric.
// '=== false' (not falsy) so a stale page that doesn't send models_loaded shows
// nothing; when the idle monitor unloads, the VRAM meter drops alongside this.
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'')+' · consumers '+(s.consumers!=null?s.consumers:'')+' · on GPU '+(s.active||0)
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'')+' MB/s'
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
+(s.models_loaded===false?' · GPU models unloaded (idle — reload on next job)':'')
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
+10
View File
@@ -51,6 +51,12 @@ class Config:
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
# all downloaders + video streams (0 = unlimited);
# tunable live from the agent UI
idle_unload_seconds: float # after this long with the GPU idle (nothing in
# flight, queue empty or Stopped), unload the
# SigLIP embedder + YOLO proposers to free their
# VRAM; they reload lazily on the next job. A
# 24/7 agent otherwise squats on ~5GB doing
# nothing. 0 disables (keep models warm forever).
@classmethod
def from_env(cls) -> Config:
@@ -87,4 +93,8 @@ class Config:
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
# from the agent UI on wired/faster networks.
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
# 5 min: long enough that a lull between job bursts doesn't thrash the
# (few-second) reload, short enough that an agent left running with an
# empty queue hands its VRAM back promptly.
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
)
+15
View File
@@ -170,6 +170,13 @@ class YoloProposer:
))
return out
def unload(self) -> None:
"""Drop the loaded YOLO so its VRAM can be reclaimed; detect() reloads it
lazily on the next job. Leaves _ok untouched — a healthy proposer comes
back, but one that self-disabled on a fault stays off."""
with self._lock:
self._model = None
class Proposers:
"""The agent's proposer set, built from config. Each detector is optional —
@@ -216,3 +223,11 @@ class Proposers:
def panels(self, image):
return self._top(self._panel, image, self.cfg.max_panels)
def unload(self) -> None:
"""Release every loaded proposer's YOLO (idle VRAM reclaim). The worker
also drops its reference to this Proposers and rebuilds a fresh one via
_proposers_for on the next job, so this is belt-and-braces."""
for p in (self._person, self._anatomy, self._panel):
if p is not None:
p.unload()
+15
View File
@@ -75,3 +75,18 @@ class CropEmbedder:
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
arr = pooled.float().cpu().numpy().astype(np.float32)
return [row.reshape(-1).tolist() for row in arr]
def unload(self) -> bool:
"""Drop the loaded model so its VRAM can be reclaimed — the idle monitor
calls this after a spell with no work so an idle agent doesn't squat on
the card; the next embed() reloads it lazily (a few seconds). Held under
BOTH the load and inference locks so it can never race a concurrent load
or an in-flight forward pass. Returns True if a model was actually
released (the caller then runs one empty_cache() to hand the freed blocks
back to the driver)."""
with self._load_lock, self._infer_lock:
if self._model is None:
return False
self._model = None
self._processor = None
return True
+75
View File
@@ -57,6 +57,15 @@ MAX_BACKOFF_SECONDS = 60.0
# up on their own.
IDLE_POLL_MAX_SECONDS = 900.0
# Idle VRAM reclaim (operator 2026-07-17): the SigLIP embedder + YOLO proposers
# load lazily and then stay warm for fast job bursts — but a 24/7 agent with an
# empty queue would otherwise squat on that VRAM (~5GB on the operator's card)
# indefinitely while doing nothing. So a monitor unloads them after
# cfg.idle_unload_seconds with the GPU genuinely idle (nothing in flight, buffer
# drained); they reload lazily on the next job. This is just how often the
# monitor wakes to check — it bounds how soon past the threshold the unload fires.
IDLE_UNLOAD_CHECK_INTERVAL = 30.0
# A job whose fetch dies transiently this many times IN ONE SESSION stops being
# handed back and is failed instead. Transient handbacks (release) burn no
# attempts on the server, so a poisoned transfer — an original that stalls the
@@ -268,6 +277,11 @@ class Worker:
self._proposers_sig = None # detector-config signature the current
# proposers were built for (#134)
self._proposers_lock = threading.Lock()
# Monotonic time of the last GPU activity (a consumer finishing a job).
# The idle monitor unloads the warm models once this goes stale by
# cfg.idle_unload_seconds — see _idle_unload_loop.
self._last_gpu_activity = time.monotonic()
threading.Thread(target=self._idle_unload_loop, daemon=True).start()
# --- held-lease bookkeeping --------------------------------------------
def _hold(self, job_ids) -> None:
@@ -608,6 +622,9 @@ class Worker:
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
"idle": self._idle, # queue empty → poll backed off (UI hint)
# Whether the GPU models are currently resident (False after an idle
# unload freed their VRAM) — a plain bool read, UI hint only.
"models_loaded": self._embedder is not None or self._proposers is not None,
}
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
@@ -788,6 +805,9 @@ class Worker:
self._bump(processed=1)
finally:
self._bump(active=-1)
# Mark the GPU busy-until-now so the idle monitor starts its
# unload countdown from when work actually stopped, not before.
self._last_gpu_activity = time.monotonic()
def _ensure_embedder(self, model_name: str):
if self._embedder is not None:
@@ -845,6 +865,61 @@ class Worker:
self._proposers_sig = sig
return self._proposers
def _unload_models(self) -> bool:
"""Release the GPU-resident models (SigLIP embedder + YOLO proposers) so an
idle agent hands their VRAM back instead of squatting on the card. They
reload lazily on the next job (_ensure_embedder / _proposers_for) — a
few seconds' cost paid only when work actually resumes. Dropping the
shared instances under their build locks means a concurrent job either
sees the old instance (before) or rebuilds a fresh one (after); the idle
monitor only calls this with nothing in flight, so no inference is using
them. Returns True if anything was released."""
released = False
with self._embedder_lock:
if self._embedder is not None:
self._embedder.unload()
self._embedder = None
released = True
with self._proposers_lock:
if self._proposers is not None:
self._proposers.unload()
self._proposers = None
self._proposers_sig = None
released = True
if released:
try:
import torch
if torch.cuda.is_available():
# torch's caching allocator holds freed blocks; hand them back
# to the driver so nvidia-smi actually reflects the drop.
torch.cuda.empty_cache()
except Exception: # noqa: BLE001 — torch absent / CPU-only → nothing to free
pass
return released
def _idle_unload_loop(self) -> None:
"""Unload the warm GPU models after a stretch of inactivity so a 24/7
agent with an empty queue doesn't hold ~5GB of VRAM doing nothing. Fires
only when nothing is in flight (active == 0 AND the buffer is drained) and
no job has completed for cfg.idle_unload_seconds — a window long enough
that a brief lull between bursts doesn't thrash reload/unload. Covers BOTH
sleep mode (queue empty, pipeline still running) and a full Stop; the
models reload lazily on the next job. idle_unload_seconds <= 0 disables it."""
idle_after = self.cfg.idle_unload_seconds
if idle_after <= 0:
return
while True:
time.sleep(IDLE_UNLOAD_CHECK_INTERVAL)
if self._embedder is None and self._proposers is None:
continue # nothing loaded → nothing to free
if self._active != 0 or not self._buffer.empty():
continue # work in flight → keep them warm
if time.monotonic() - self._last_gpu_activity < idle_after:
continue # not idle long enough yet
if self._unload_models():
log.info("idle %.0fs — unloaded GPU models, freed VRAM "
"(reload on next job)", idle_after)
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
"""Detect + embed the decoded frames and submit the result. Returns True
when the job was completed (→ count it processed), False otherwise: a
+1 -3
View File
@@ -256,9 +256,7 @@ async def lease():
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
ml = (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
ml = await MLSettings.load(session)
# image rows for url/mime in one shot
ids = [j.image_record_id for j in jobs]
imgs = {
+17 -43
View File
@@ -4,6 +4,7 @@ from quart import Blueprint, jsonify, request
from ..extensions import get_session
from ..models import MLSettings
from ..services.ml.heads import AUTO_APPLY_THRESHOLD_MAX, AUTO_APPLY_THRESHOLD_MIN
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
@@ -83,48 +84,21 @@ async def embedder_models():
@ml_admin_bp.route("/settings", methods=["GET"])
async def get_settings():
from sqlalchemy import select
async with get_session() as session:
s = (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
return jsonify(
{
"cpu_embed_enabled": s.cpu_embed_enabled,
"video_frame_interval_seconds": s.video_frame_interval_seconds,
"video_max_frames": s.video_max_frames,
"embedder_model_version": s.embedder_model_version,
"head_min_positives": s.head_min_positives,
"head_auto_apply_precision": s.head_auto_apply_precision,
"head_auto_apply_enabled": s.head_auto_apply_enabled,
"head_auto_apply_min_positives": s.head_auto_apply_min_positives,
"ccip_match_threshold": s.ccip_match_threshold,
"ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
"ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
"presentation_auto_apply_enabled": s.presentation_auto_apply_enabled,
"presentation_auto_apply_threshold": s.presentation_auto_apply_threshold,
"presentation_conflict_threshold": s.presentation_conflict_threshold,
"process_auto_apply_enabled": s.process_auto_apply_enabled,
"process_auto_apply_threshold": s.process_auto_apply_threshold,
"process_conflict_threshold": s.process_conflict_threshold,
"embedder_model_name": s.embedder_model_name,
**{f: getattr(s, f) for f in _DETECTOR_FIELDS},
}
)
s = await MLSettings.load(session)
# Table-driven off _EDITABLE (which PATCH also writes) so a new settings field
# can never be silently absent from GET — the split that historically dropped
# fields. _EDITABLE already includes *_DETECTOR_FIELDS.
return jsonify({f: getattr(s, f) for f in _EDITABLE})
@ml_admin_bp.route("/settings", methods=["PATCH"])
async def patch_settings():
from sqlalchemy import select
body = await request.get_json()
if not isinstance(body, dict):
return jsonify({"error": "body must be an object"}), 400
async with get_session() as session:
s = (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
s = await MLSettings.load(session)
# Merge the patch over current values, then validate the result as a
# whole — the store-floor invariant couples three fields, so they
@@ -154,24 +128,24 @@ def _validate(p: dict) -> str | None:
# Head training (#114).
if int(p["head_min_positives"]) < 1:
return "head_min_positives must be >= 1"
if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999):
return "head_auto_apply_precision must be between 0.5 and 0.999"
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["head_auto_apply_precision"]) <= AUTO_APPLY_THRESHOLD_MAX):
return f"head_auto_apply_precision must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if int(p["head_auto_apply_min_positives"]) < 1:
return "head_auto_apply_min_positives must be >= 1"
if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999):
return "ccip_match_threshold must be between 0.5 and 0.999"
if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999):
return "ccip_auto_apply_threshold must be between 0.5 and 0.999"
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_match_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return f"ccip_match_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return f"ccip_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
# consequential); the conflict cut is a plain probability [0,1].
if not (0.5 <= float(p["presentation_auto_apply_threshold"]) <= 0.999):
return "presentation_auto_apply_threshold must be between 0.5 and 0.999"
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["presentation_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return f"presentation_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
return "presentation_conflict_threshold must be between 0 and 1"
# Process auto-apply (#1464). wip/editor stay VISIBLE so a false apply is
# low-harm (excludes-from-training + a review flag), but keep the same bar.
if not (0.5 <= float(p["process_auto_apply_threshold"]) <= 0.999):
return "process_auto_apply_threshold must be between 0.5 and 0.999"
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["process_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
return f"process_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
if not (0.0 <= float(p["process_conflict_threshold"]) <= 1.0):
return "process_conflict_threshold must be between 0 and 1"
# Embedder model swap (#1190): both must be non-empty. Changing them means a
+3 -28
View File
@@ -66,34 +66,9 @@ _EXTDL_TOGGLE_FIELDS = (
async def get_import_settings():
async with get_session() as session:
row = await ImportSettings.load(session)
return jsonify({
"min_width": row.min_width,
"min_height": row.min_height,
"skip_transparent": row.skip_transparent,
"transparency_threshold": row.transparency_threshold,
"skip_single_color": row.skip_single_color,
"single_color_threshold": row.single_color_threshold,
"single_color_tolerance": row.single_color_tolerance,
"phash_threshold": row.phash_threshold,
"download_rate_limit_seconds": row.download_rate_limit_seconds,
"download_validate_files": row.download_validate_files,
"download_schedule_default_seconds": row.download_schedule_default_seconds,
"download_event_retention_days": row.download_event_retention_days,
"download_failure_warning_threshold": row.download_failure_warning_threshold,
"series_suggest_enabled": row.series_suggest_enabled,
"series_suggest_threshold": row.series_suggest_threshold,
"extdl_mega_enabled": row.extdl_mega_enabled,
"extdl_gdrive_enabled": row.extdl_gdrive_enabled,
"extdl_mediafire_enabled": row.extdl_mediafire_enabled,
"extdl_dropbox_enabled": row.extdl_dropbox_enabled,
"extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
"translation_enabled": row.translation_enabled,
"interpreter_base_url": row.interpreter_base_url,
"translation_target_lang": row.translation_target_lang,
"translation_min_confidence": row.translation_min_confidence,
"wip_title_tagging_enabled": row.wip_title_tagging_enabled,
"wip_soft_title_tagging_enabled": row.wip_soft_title_tagging_enabled,
})
# Table-driven off _EDITABLE_FIELDS (which PATCH also writes) so a new field
# can't be silently absent from GET.
return jsonify({f: getattr(row, f) for f in _EDITABLE_FIELDS})
@settings_bp.route("/settings/import", methods=["PATCH"])
+12
View File
@@ -10,6 +10,7 @@ from sqlalchemy import (
Integer,
String,
func,
select,
)
from sqlalchemy.orm import Mapped, mapped_column
@@ -212,3 +213,14 @@ class MLSettings(Base):
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
@classmethod
async def load(cls, session) -> MLSettings:
"""The singleton settings row (id=1), via an async session. Mirrors
ImportSettings.load — the shared singleton-loader pattern."""
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
@classmethod
def load_sync(cls, session) -> MLSettings:
"""The singleton settings row (id=1), via a sync session."""
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
@@ -150,9 +150,7 @@ def refresh_character_prototypes(
"""Incrementally refresh the prototype store. `full=True` rebuilds every
character regardless of the gate/fingerprints (nightly reconcile). Returns
{skipped, rebuilt, removed}; commits."""
settings = session.execute(
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
settings = MLSettings.load_sync(session)
sig = _global_signature(session)
if not full and settings.ccip_ref_signature == sig:
return {"skipped": True, "rebuilt": 0, "removed": 0}
@@ -204,9 +202,7 @@ def retract_auto_applied_ccip(session: Session) -> int:
n_retracted."""
import numpy as np
settings = session.execute(
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
settings = MLSettings.load_sync(session)
if not settings.ccip_auto_apply_enabled:
return 0
thr = float(settings.ccip_auto_apply_threshold)
+65 -62
View File
@@ -23,6 +23,7 @@ from datetime import UTC, datetime
from typing import Any
from sqlalchemy import delete, exists, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import Session
@@ -42,6 +43,7 @@ from ...models import (
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
from .training_data import (
_AUTO_SOURCES,
_applied_or_rejected,
_auto_apply_point,
_hygiene_excluded_ids,
_ids_with_tag,
@@ -61,6 +63,14 @@ MIN_POSITIVES_FLOOR = 8 # hard floor; settings.head_min_positives can raise
_UNLABELED_POOL = 4000
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
# Auto-apply / match confidence operating range. Every graduated auto-apply or
# CCIP-match threshold the operator can set lives in this band, and the head
# precision target is clamped to it: below 0.5 "auto-apply" is meaningless, and
# 1.0 is unachievable so 0.999 is the ceiling. One source shared by the service
# clamp (_normalize_params) and the API validator (ml_admin._validate).
AUTO_APPLY_THRESHOLD_MIN = 0.5
AUTO_APPLY_THRESHOLD_MAX = 0.999
# Only these tag kinds get heads (the surfaced suggestion categories).
_HEAD_KINDS = (TagKind.general, TagKind.character)
# tag.kind -> the suggestion category the rail groups under.
@@ -78,6 +88,38 @@ _CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65
def _sigmoid(z, np):
"""Logistic sigmoid 1/(1+e^-z): the head score→probability transform. One home
for what was inlined at every scoring site (suggest, both sweeps, retract)."""
return 1.0 / (1.0 + np.exp(-z))
def _conflict_scores(Xn, Wc, bc, np):
"""The presentation conflict signal (#141): per row, the MAX content-head
probability and WHICH head produced it. Shared by the system-tag sweep's guard-2
and the soft-wip audit — both ask "does this ALSO look like real content?"."""
cprobs = _sigmoid(Xn @ Wc.T + bc, np)
return cprobs.max(axis=1), cprobs.argmax(axis=1)
def _insert_presentation_review(
session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode,
):
"""Single-source the ring-loud PresentationReview row shape so the two writers
(system-tag sweep guard-2 + soft-wip audit) can't drift on columns or `mode` —
they share the (image_record_id, tag_id) composite PK, so a divergent `mode`
would be a silent first-writer-wins bug."""
session.execute(
pg_insert(PresentationReview)
.values(
image_record_id=image_record_id, tag_id=tag_id,
conflict_tag_id=conflict_tag_id, conflict_score=conflict_score,
mode=mode,
)
.on_conflict_do_nothing()
)
class HeadTrainingAlreadyRunning(Exception):
"""Raised by start_head_training_run when a run is already in flight."""
@@ -103,9 +145,7 @@ def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
def _settings(session: Session) -> MLSettings:
return session.execute(
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
return MLSettings.load_sync(session)
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
@@ -124,7 +164,7 @@ def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[s
except (TypeError, ValueError):
cv_folds = DEFAULT_CV_FOLDS
try:
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999)
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), AUTO_APPLY_THRESHOLD_MIN), AUTO_APPLY_THRESHOLD_MAX)
except (TypeError, ValueError):
precision_target = s.head_auto_apply_precision
return {
@@ -536,7 +576,7 @@ async def score_image(
norms[norms == 0] = 1.0
Xn = X / norms
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
probs_bag = 1.0 / (1.0 + np.exp(-Z)) # (B, H)
probs_bag = _sigmoid(Z, np) # (B, H)
probs = probs_bag.max(axis=0) # (H,) best over the bag
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds
# the tag (bag_meta[win]); None when the whole-image vector won (#1206).
@@ -614,9 +654,7 @@ async def ground_applied_tag(
async def _settings_async(session: AsyncSession) -> MLSettings:
return (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
return await MLSettings.load(session)
# --- Earned auto-apply (sync, ml worker) ---------------------------------
@@ -687,7 +725,6 @@ def auto_apply_sweep(
embeddings in chunks; commits per chunk on a real run. Returns
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
settings = _settings(session)
rows = _auto_apply_heads(
@@ -704,18 +741,7 @@ def auto_apply_sweep(
names = [r.name for r in rows]
# Skip images that already carry, or have rejected, each tag.
skip = {tid: set() for tid in tag_ids}
for tid in tag_ids:
for (iid,) in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
):
skip[tid].add(iid)
for (iid,) in session.execute(
select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == tid
)
):
skip[tid].add(iid)
skip = _applied_or_rejected(session, tag_ids)
applied = [0] * len(rows)
scanned = 0
@@ -729,7 +755,7 @@ def auto_apply_sweep(
if not cids:
continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b))) # (N, H)
probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
scanned += len(cids)
for h in range(len(rows)):
tid = tag_ids[h]
@@ -840,7 +866,6 @@ def system_tag_auto_apply_sweep(
enabled flag is set. numpy-only (no sklearn). Returns {n_applied, n_flagged,
concepts}."""
import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
cfg = _SWEEP_MODES[mode]
settings = _settings(session)
@@ -869,18 +894,7 @@ def system_tag_auto_apply_sweep(
valued = _valued_image_ids(session)
# Skip images that already carry, or have rejected, each presentation tag.
skip = {tid: set() for tid in pres_tag_ids}
for tid in pres_tag_ids:
for (iid,) in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
):
skip[tid].add(iid)
for (iid,) in session.execute(
select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == tid
)
):
skip[tid].add(iid)
skip = _applied_or_rejected(session, pres_tag_ids)
applied = [0] * len(pres)
n_flagged = 0
@@ -895,11 +909,9 @@ def system_tag_auto_apply_sweep(
if not cids:
continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = 1.0 / (1.0 + np.exp(-(Xn @ Wp.T + bp))) # (N, P)
probs = _sigmoid(Xn @ Wp.T + bp, np) # (N, P)
if Wc is not None:
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc))) # (N, C)
max_c = cprobs.max(axis=1)
arg_c = cprobs.argmax(axis=1)
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
scanned += len(cids)
for p in range(len(pres)):
tid = pres_tag_ids[p]
@@ -924,15 +936,12 @@ def system_tag_auto_apply_sweep(
if Wc is not None and float(max_c[idx]) >= conflict_thr:
n_flagged += 1
if not dry_run:
session.execute(
pg_insert(PresentationReview)
.values(
image_record_id=iid, tag_id=tid,
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
conflict_score=float(max_c[idx]),
mode=mode,
)
.on_conflict_do_nothing()
_insert_presentation_review(
session,
image_record_id=iid, tag_id=tid,
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
conflict_score=float(max_c[idx]),
mode=mode,
)
if not dry_run:
session.commit()
@@ -956,7 +965,6 @@ def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
NOT remove the tag; the operator decides. No-op when there are no content heads.
numpy-only. Returns {n_scanned, n_flagged}."""
import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id
@@ -993,22 +1001,17 @@ def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
continue
scanned += len(cids)
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc)))
max_c = cprobs.max(axis=1)
arg_c = cprobs.argmax(axis=1)
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np)
for k in range(len(cids)):
if float(max_c[k]) >= conflict_thr:
n_flagged += 1
if not dry_run:
session.execute(
pg_insert(PresentationReview)
.values(
image_record_id=cids[k], tag_id=wip_id,
conflict_tag_id=conf_tag_ids[int(arg_c[k])],
conflict_score=float(max_c[k]),
mode="process",
)
.on_conflict_do_nothing()
_insert_presentation_review(
session,
image_record_id=cids[k], tag_id=wip_id,
conflict_tag_id=conf_tag_ids[int(arg_c[k])],
conflict_score=float(max_c[k]),
mode="process",
)
if not dry_run:
session.commit()
@@ -1062,7 +1065,7 @@ def retract_auto_applied_heads(session: Session) -> int:
continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
w = np.asarray(weights, dtype=np.float32)
probs = 1.0 / (1.0 + np.exp(-(Xn @ w + float(bias))))
probs = _sigmoid(Xn @ w + float(bias), np)
below = [cids[k] for k in np.where(probs < float(thr))[0]]
for iid in below:
session.execute(
+18
View File
@@ -94,6 +94,24 @@ def _rejected_ids(session: Session, tag_id: int) -> list[int]:
]
def _applied_or_rejected(session: Session, tag_ids) -> dict[int, set[int]]:
"""Per-tag skip set for the auto-apply sweeps: every image that ALREADY carries
the tag (ANY source — not just training positives) OR has rejected it. A sweep
never re-applies to these. Shared by auto_apply_sweep + system_tag_auto_apply_sweep
(heads.py) and scheduled_ccip_auto_apply (tasks/ml.py). Callers mutate the returned
sets in-place to also dedupe within a single run."""
skip: dict[int, set[int]] = {}
for tid in tag_ids:
ids = {
r[0] for r in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
).all()
}
ids.update(_rejected_ids(session, tid))
skip[tid] = ids
return skip
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
sparse, so an untagged image is almost always a true negative."""
+20 -36
View File
@@ -91,48 +91,46 @@ def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
)
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
def _campaigns_api_first(vanity: str, cookies_path: str | None) -> dict | None:
"""The first `data` object from Patreon's campaigns API filtered by vanity
(`?filter[vanity]=<vanity>&fields[campaign]=name`), or None on any failure
(network / non-200 / non-JSON / empty). The single request shape shared by
_lookup_via_api (plucks the campaign id) and resolve_display_name (plucks the
display name)."""
jar = _load_cookie_jar(cookies_path)
headers = {
"User-Agent": _USER_AGENT,
"Accept": "application/vnd.api+json",
}
params = {
"filter[vanity]": vanity,
"fields[campaign]": "name",
}
try:
resp = requests.get(
_CAMPAIGNS_URL,
params=params,
headers=headers,
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
cookies=jar,
timeout=_TIMEOUT_SECONDS,
)
except requests.RequestException as exc:
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
return None
if resp.status_code != 200:
log.warning(
"Patreon campaigns API returned HTTP %d for vanity=%s",
resp.status_code, vanity,
)
return None
try:
payload = resp.json()
except ValueError as exc:
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
return None
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
return None
return data[0]
if not isinstance(payload, dict):
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
first = _campaigns_api_first(vanity, cookies_path)
if first is None:
return None
data = payload.get("data")
if not isinstance(data, list) or not data:
return None
first = data[0] if isinstance(data[0], dict) else None
campaign_id = first.get("id") if first else None
campaign_id = first.get("id")
if not isinstance(campaign_id, str) or not campaign_id:
return None
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
@@ -144,24 +142,10 @@ def resolve_display_name(vanity: str, cookies_path: str | None) -> str | None:
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None
on any failure — the caller falls back to the vanity handle. Sync: call from
an executor."""
jar = _load_cookie_jar(cookies_path)
try:
resp = requests.get(
_CAMPAIGNS_URL,
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
cookies=jar,
timeout=_TIMEOUT_SECONDS,
)
if resp.status_code != 200:
return None
data = resp.json().get("data")
except (requests.RequestException, ValueError) as exc:
log.warning("Patreon name lookup failed for vanity=%s: %s", vanity, exc)
first = _campaigns_api_first(vanity, cookies_path)
if first is None:
return None
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
return None
name = (data[0].get("attributes") or {}).get("name")
name = (first.get("attributes") or {}).get("name")
return name.strip() if isinstance(name, str) and name.strip() else None
+45 -72
View File
@@ -776,89 +776,62 @@ def recover_stalled_library_audit_runs() -> int:
return recovered
def _recover_stalled_runs(model, *, stall_minutes: int, keep_runs: int, label: str) -> int:
"""Shared recovery + retention sweep for the head run-tracking tables
(HeadTrainingRun / HeadAutoApplyRun, which share the
status/last_progress_at/started_at/finished_at/error/id columns): flip 'running'
rows with no progress past `stall_minutes` to 'error', then prune to the last
`keep_runs` (rule 89). Returns the number recovered. NOTE the two other recover
tasks are deliberately NOT folded in — library-audit has no prune tail and
backup uses a single started_at cutoff."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=stall_minutes)
with SessionLocal() as session:
result = session.execute(
update(model)
.where(model.status == "running")
.where(func.coalesce(model.last_progress_at, model.started_at) < cutoff)
.values(
status="error", finished_at=now,
error=f"stranded by recovery sweep (no progress for {stall_minutes} min)",
)
)
keep = session.execute(
select(model.id).order_by(model.id.desc()).limit(keep_runs)
).scalars().all()
if keep:
session.execute(delete(model).where(model.id.not_in(keep)))
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("%s: recovered %d rows", label, recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
def recover_stalled_head_training_runs() -> int:
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
update(HeadTrainingRun)
.where(HeadTrainingRun.status == "running")
.where(
func.coalesce(
HeadTrainingRun.last_progress_at, HeadTrainingRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_TRAINING_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadTrainingRun.id).order_by(HeadTrainingRun.id.desc())
.limit(HEAD_TRAINING_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadTrainingRun).where(HeadTrainingRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_training_runs: recovered %d rows", recovered
)
return recovered
return _recover_stalled_runs(
HeadTrainingRun,
stall_minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES,
keep_runs=HEAD_TRAINING_KEEP_RUNS,
label="recover_stalled_head_training_runs",
)
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
def recover_stalled_head_auto_apply_runs() -> int:
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
update(HeadAutoApplyRun)
.where(HeadAutoApplyRun.status == "running")
.where(
func.coalesce(
HeadAutoApplyRun.last_progress_at, HeadAutoApplyRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadAutoApplyRun.id).order_by(HeadAutoApplyRun.id.desc())
.limit(HEAD_AUTO_APPLY_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadAutoApplyRun).where(HeadAutoApplyRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_auto_apply_runs: recovered %d rows", recovered
)
return recovered
return _recover_stalled_runs(
HeadAutoApplyRun,
stall_minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES,
keep_runs=HEAD_AUTO_APPLY_KEEP_RUNS,
label="recover_stalled_head_auto_apply_runs",
)
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
+10 -30
View File
@@ -105,9 +105,7 @@ def embed_image(self, image_id: int) -> dict:
record = session.get(ImageRecord, image_id)
if record is None:
return {"status": "missing", "image_id": image_id}
settings = session.execute(
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
settings = MLSettings.load_sync(session)
src = Path(record.path)
is_vid = _is_video(src)
@@ -488,15 +486,10 @@ def scheduled_ccip_auto_apply() -> str:
from sqlalchemy import select as sa_select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..models import ImageRegion, MLSettings, Tag, TagKind, TagSuggestionRejection
from ..models import ImageRegion, MLSettings, Tag, TagKind
from ..models.tag import image_tag
fig = ("face", "figure")
def _l2(m):
n = np.linalg.norm(m, axis=1, keepdims=True)
n[n == 0] = 1.0
return m / n
from ..services.ml.ccip import _FIGURE_KINDS
from ..services.ml.training_data import _applied_or_rejected, _l2norm
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
@@ -521,7 +514,7 @@ def scheduled_ccip_auto_apply() -> str:
)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.where(ImageRegion.kind.in_(fig))
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
.where(ImageRegion.ccip_embedding.is_not(None))
.where(ImageRegion.image_record_id.in_(single))
).all()
@@ -532,29 +525,16 @@ def scheduled_ccip_auto_apply() -> str:
for tid, vec in ref_rows:
by_char.setdefault(tid, []).append(vec)
ref_tags = list(by_char)
mats = [_l2(np.asarray(by_char[t], dtype=np.float32)) for t in ref_tags]
mats = [_l2norm(np.asarray(by_char[t], dtype=np.float32), np) for t in ref_tags]
allref = np.vstack(mats) # (total, 768)
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
# Per character: images that already carry OR rejected the tag — skip.
skip = {t: set() for t in ref_tags}
for t in ref_tags:
for (iid,) in session.execute(
sa_select(image_tag.c.image_record_id).where(
image_tag.c.tag_id == t
)
):
skip[t].add(iid)
for (iid,) in session.execute(
sa_select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == t
)
):
skip[t].add(iid)
skip = _applied_or_rejected(session, ref_tags)
img_ids = list(session.execute(
sa_select(ImageRegion.image_record_id)
.where(ImageRegion.kind.in_(fig), ImageRegion.ccip_embedding.is_not(None))
.where(ImageRegion.kind.in_(_FIGURE_KINDS), ImageRegion.ccip_embedding.is_not(None))
.distinct()
).scalars())
@@ -566,7 +546,7 @@ def scheduled_ccip_auto_apply() -> str:
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
.where(
ImageRegion.image_record_id.in_(chunk),
ImageRegion.kind.in_(fig),
ImageRegion.kind.in_(_FIGURE_KINDS),
ImageRegion.ccip_embedding.is_not(None),
)
).all()
@@ -574,7 +554,7 @@ def scheduled_ccip_auto_apply() -> str:
for iid, vec in rows:
by_img.setdefault(iid, []).append(vec)
for iid, vecs in by_img.items():
q = _l2(np.asarray(vecs, dtype=np.float32)) # (nq, 768)
q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768)
colmax = (q @ allref.T).max(axis=0) # (total,)
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
for ci in np.where(charmax >= thr)[0]:
+21 -3
View File
@@ -11,12 +11,22 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
- python 3.14
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
- node (frontend job: `npm install` + vitest + vite build)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Forgejo registry push)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
## Secondary runtime image
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
per `docs/process.md`'s "add deps to the image when used by >1 project".
## Per-job tool installs
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
- `npm install --no-audit --no-fund` — in `frontend-build` job
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
## Notes
@@ -26,12 +36,20 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
"add deps to image when used by >1 project" rule: FC alone is one Python
project, so the deps live in `requirements.txt` and install per-job.
Reconsider when a second Fabled-family Python backend lands.
- Integration uses Forgejo Actions `services:` + socket-discovered bridge IPs
- Integration uses Fabled-Git Actions `services:` + socket-discovered bridge IPs
because `act_runner` (swarm-runner v0.6+) puts services on the default
bridge with no embedded DNS. The pattern is documented in the rulebook's
`forgejo.md` "CI philosophy" section and FC's `ci.yml` is the canonical
`fabled-git.md` "CI philosophy" section and FC's `ci.yml` is the canonical
example.
- No `package-lock.json` is tracked yet (FC's `feedback_no_local_runs`
memory bans `npm install` locally). Using `npm install` rather than
`npm ci` until a lockfile lands.
- No `imagemagick` / `pandoc` per-job installs needed.
- `extension/`'s vitest specs load `lib/*.js` by evaluating the real file as a
classic script (`test/helpers/loadLib.js`) rather than adding `module.exports`
shims to production code — the libs ship as `background.scripts`, not ES
modules, so the specs exercise exactly the bytes packaged into the XPI.
- Extension test files are excluded from the XPI via `--ignore-files` in
`extension/package.json`, and the same paths are excluded from `ci.yml`'s
`extension-version` guard. Those two lists must agree — `test/version.spec.js`
asserts the guard never ignores a file web-ext actually packages.
+29 -33
View File
@@ -60,9 +60,8 @@ async function checkForUpdateInfo() {
}
const currentVersion = browser.runtime.getManifest().version;
const latestVersion = info && info.version ? info.version : null;
// latest_url is served from the web root; strip the /api suffix off baseUrl
// (same transform as OPEN_ARTIST_PAGE).
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
// latest_url is served from the web root, not the JSON API.
const base = api.webRoot();
return {
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
currentVersion,
@@ -211,6 +210,21 @@ browser.webRequest.onBeforeRedirect.addListener(
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
);
// Extract → verify → upload one cookie-auth platform. Returns a structured
// outcome so the two callers (EXPORT_COOKIES single, EXPORT_ALL_COOKIES) shape
// their own response + skip semantics. Verifies the captured cookies are
// actually live BEFORE uploading, so a confirmed-stale session doesn't overwrite
// good FC-side credentials; platforms with no verify config (v.ok === null) fall
// through to upload.
async function exportPlatformCookies(key) {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { status: 'empty' };
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) return { status: 'stale', reason: v.reason, cookieCount: cookies.length };
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
return { status: 'ok', cookieCount: cookies.length, verified: v.ok === true };
}
// ---- Message router ----
browser.runtime.onMessage.addListener(async (msg) => {
@@ -255,22 +269,14 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (!platform) return { error: `Unknown platform: ${key}` };
try {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
const r = await exportPlatformCookies(key);
if (r.status === 'empty') return { error: 'No cookies found — log in first.' };
if (r.status === 'stale') {
return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
error: `Captured ${r.cookieCount} ${platform.name} cookies but they don't appear authenticated (${r.reason}). Log in again in this browser, then retry.`,
};
}
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
return { success: true, cookieCount: r.cookieCount, verified: r.verified };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -298,18 +304,10 @@ browser.runtime.onMessage.addListener(async (msg) => {
continue;
}
try {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
const r = await exportPlatformCookies(key);
if (r.status === 'empty') results[key] = { skipped: true, reason: 'no cookies' };
else if (r.status === 'stale') results[key] = { error: `verify failed: ${r.reason}` };
else results[key] = { success: true, cookieCount: r.cookieCount, verified: r.verified };
} catch (e) {
results[key] = { error: e.message };
}
@@ -346,11 +344,9 @@ browser.runtime.onMessage.addListener(async (msg) => {
}
case 'OPEN_ARTIST_PAGE': {
// apiUrl is configured with the /api suffix (see
// options/options.html placeholder); the SPA artist route is
// /artist/:slug, served from the same origin. Strip /api so the
// browser-level URL hits the Vue router, not the JSON API.
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
// The SPA artist route (/artist/:slug) is served from the web root, not
// the JSON API — see api.webRoot().
const base = api.webRoot();
const slug = encodeURIComponent(msg.slug || '');
if (!base || !slug) return { error: 'apiUrl or slug missing' };
try {
+17 -1
View File
@@ -11,7 +11,10 @@ class FabledCuratorAPI {
async init() {
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
this.baseUrl = cfg.apiUrl || null;
// Normalize on READ, not just on save: configs stored before the options
// page started normalizing are missing the `/api` suffix, and this heals
// them without the operator having to reopen Settings.
this.baseUrl = normalizeApiUrl(cfg.apiUrl) || null;
this.apiKey = cfg.apiKey || null;
return this.isConfigured();
}
@@ -50,6 +53,13 @@ class FabledCuratorAPI {
} catch {
message = `HTTP ${response.status}: ${response.statusText}`;
}
// 404/405 from FC almost always means the request never reached the JSON
// API — it fell through to the SPA catch-all, which serves HTML on GET
// and rejects everything else. Say so, rather than making the operator
// decode "Method Not Allowed" on an endpoint that plainly allows POST.
if (response.status === 404 || response.status === 405) {
message += `${url} isn't the FC API. Check the FC URL in settings.`;
}
const err = new Error(message);
err.status = response.status;
throw err;
@@ -96,6 +106,12 @@ class FabledCuratorAPI {
return this.request('GET', '/extension/manifest');
}
// The web/SPA root: where the Vue router (artist pages) and the served XPI
// live, NOT the JSON API. Used by OPEN_ARTIST_PAGE + the self-update check.
webRoot() {
return webRootFromApiUrl(this.baseUrl);
}
// Connection test = the cheapest read with auth.
testConnection() {
return this.request('GET', '/credentials');
+32
View File
@@ -0,0 +1,32 @@
/**
* Canonical FC endpoint derivation, shared by the background client and the
* options page so a URL entered either way behaves identically.
*
* FC serves two things on one origin: the JSON API under `/api`, and the Vue
* SPA from the root. `api.js` builds requests as `${baseUrl}/credentials`, so
* the stored base URL has to carry the `/api` suffix.
*/
/**
* Accept what an operator would naturally type — the instance root
* (`http://curator.example.com`) or the API root (`.../api`) — and return the
* API root either way.
*
* Worth normalizing rather than validating: a root-form URL doesn't fail
* loudly, it lands on the SPA catch-all, which answers `GET /credentials` with
* 200 HTML and rejects `POST /credentials` with 405. The operator sees a
* working Test Connection and a broken export.
*/
function normalizeApiUrl(raw) {
const trimmed = (raw || '').trim().replace(/\/+$/, '');
if (!trimmed) return '';
return /\/api$/i.test(trimmed) ? trimmed : `${trimmed}/api`;
}
/**
* The SPA root — where the Vue router (artist pages) and the served XPI live,
* NOT the JSON API. Accepts either input form, same as normalizeApiUrl.
*/
function webRootFromApiUrl(raw) {
return normalizeApiUrl(raw).replace(/\/api$/i, '');
}
+2 -2
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.9",
"version": "1.0.10",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
@@ -46,7 +46,7 @@
},
"background": {
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/api.js", "background/background.js"]
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/url.js", "lib/api.js", "background/background.js"]
},
"options_ui": {
+7 -3
View File
@@ -21,9 +21,12 @@
<body>
<h1>FabledCurator extension</h1>
<label for="api-url">FC base URL</label>
<input id="api-url" type="url" placeholder="http://curator.example.com/api" />
<div class="hint">Find this on FC → Settings → Maintenance → Browser extension.</div>
<label for="api-url">FC instance URL</label>
<input id="api-url" type="url" placeholder="http://curator.example.com" />
<div class="hint">
Your FabledCurator address — with or without the trailing <code>/api</code>; both work.
Find it on FC → Settings → Maintenance → Browser extension.
</div>
<label for="api-key">Extension API key</label>
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
@@ -36,6 +39,7 @@
<div id="status" class="status" style="display:none;"></div>
<script src="../lib/url.js"></script>
<script src="options.js"></script>
</body>
</html>
+23 -5
View File
@@ -8,7 +8,7 @@ document.addEventListener('DOMContentLoaded', async () => {
});
async function save() {
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
const apiKey = document.getElementById('api-key').value.trim();
if (!apiUrl || !apiKey) {
showStatus('Both fields are required.', 'err');
@@ -16,11 +16,14 @@ async function save() {
}
await browser.storage.local.set({ apiUrl, apiKey });
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
showStatus('Saved.', 'ok');
// Show what was actually stored — the operator may have typed the instance
// root and it was normalized to the API root.
document.getElementById('api-url').value = apiUrl;
showStatus(`Saved — using ${apiUrl}`, 'ok');
}
async function test() {
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
const apiKey = document.getElementById('api-key').value.trim();
if (!apiUrl || !apiKey) {
showStatus('Fill both fields first.', 'err');
@@ -31,8 +34,23 @@ async function test() {
method: 'GET',
headers: { 'X-Extension-Key': apiKey },
});
if (r.ok) showStatus(`Connected — HTTP ${r.status}.`, 'ok');
else showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
if (!r.ok) {
showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
return;
}
// A 200 is NOT sufficient. If the URL resolves to the Vue SPA instead of
// the JSON API, the catch-all route returns 200 with an HTML document —
// which used to report "Connected" on a config that could not POST at all.
const contentType = r.headers.get('content-type') || '';
if (!contentType.includes('json')) {
showStatus(
`${apiUrl} answered with ${contentType || 'no content-type'}, not JSON `
+ '— that looks like the FC web UI rather than its API.',
'err',
);
return;
}
showStatus(`Connected to ${apiUrl} — HTTP ${r.status}.`, 'ok');
} catch (e) {
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
}
+7 -5
View File
@@ -1,15 +1,17 @@
{
"name": "fabledcurator-extension",
"version": "1.0.9",
"version": "1.0.10",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore",
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --firefox=firefox",
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --overwrite-dest",
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\"",
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --firefox=firefox",
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --overwrite-dest",
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
"test:unit": "vitest run"
},
"devDependencies": {
"vitest": "^4.0.0",
"web-ext": "^10.0.0"
}
}
+12 -12
View File
@@ -2,6 +2,15 @@ document.addEventListener('DOMContentLoaded', init);
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
// A centered muted note div — the loading / empty state shared by the platform
// and sources lists.
function mutedNote(text) {
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = text;
return d;
}
async function init() {
try {
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
@@ -38,10 +47,7 @@ function showSetupRequired() {
function showPlatformsLoading() {
const c = document.getElementById('platforms-list');
c.textContent = '';
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading platforms…';
c.appendChild(d);
c.appendChild(mutedNote('Loading platforms…'));
}
async function testConnectionIfNeeded() {
@@ -183,10 +189,7 @@ async function exportAllCookies() {
async function loadSources() {
const c = document.getElementById('sources-list');
c.textContent = '';
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading sources…';
c.appendChild(d);
c.appendChild(mutedNote('Loading sources…'));
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
c.textContent = '';
if (r.error) {
@@ -197,10 +200,7 @@ async function loadSources() {
return;
}
if (!r.sources || r.sources.length === 0) {
const empty = document.createElement('div');
empty.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
empty.textContent = 'No sources yet.';
c.appendChild(empty);
c.appendChild(mutedNote('No sources yet.'));
return;
}
for (const src of r.sources) c.appendChild(createSourceRow(src));
+29
View File
@@ -0,0 +1,29 @@
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import path from 'node:path'
const LIB_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', '..', 'lib')
/**
* Load an extension lib and hand back the globals it declares.
*
* The files under lib/ are CLASSIC scripts, not ES modules: manifest.json
* lists them in `background.scripts` and options.html pulls them in with a
* plain <script> tag, so they declare bare functions into a shared scope and
* export nothing. Rather than bolt a `module.exports` shim onto production
* code that would never run in the browser, evaluate the real file the same
* way the browser does — as a script body — and pick the declarations back out.
*
* This means the specs exercise the exact bytes that get packaged into the
* XPI. Only usable for libs that touch no browser APIs at load time
* (url.js, platforms.js); cookies.js and api.js reference `browser.*` and
* would need stubbing, which is why they aren't loaded this way.
*
* @param {string} filename e.g. 'url.js'
* @param {string[]} names declarations to return, e.g. ['normalizeApiUrl']
*/
export function loadLib(filename, names) {
const source = readFileSync(path.join(LIB_DIR, filename), 'utf8')
const factory = new Function(`${source}\nreturn { ${names.join(', ')} }`)
return factory()
}
+127
View File
@@ -0,0 +1,127 @@
import { describe, it, expect } from 'vitest'
import { loadLib } from './helpers/loadLib.js'
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
'platforms.js',
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
)
describe('getPlatformFromUrl', () => {
it('identifies each platform from a domain URL', () => {
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
expect(getPlatformFromUrl('https://subscribestar.adult/someone')).toBe('subscribestar')
expect(getPlatformFromUrl('https://www.hentai-foundry.com/user/someone')).toBe('hentaifoundry')
expect(getPlatformFromUrl('https://discord.com/channels/@me')).toBe('discord')
expect(getPlatformFromUrl('https://www.pixiv.net/en/users/123')).toBe('pixiv')
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe('deviantart')
})
it('accepts http as well as https, with or without www', () => {
expect(getPlatformFromUrl('http://patreon.com/Atole')).toBe('patreon')
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
})
it('returns null for unrelated hosts', () => {
expect(getPlatformFromUrl('https://example.com/patreon.com')).toBe(null)
expect(getPlatformFromUrl('https://not-patreon.com/Atole')).toBe(null)
expect(getPlatformFromUrl('')).toBe(null)
})
})
describe('isArtistPage', () => {
// Regression cases from issue #1485: the Add-to-FC button vanished once the
// operator SUBSCRIBED to a creator, because Patreon serves subscribed users
// the /cw/ ("creator workspace") URL and the pattern only matched the bare
// root. All three creator URL shapes must match, plus inner pages — the
// button matters most exactly when you're subscribed.
it('matches all three Patreon creator URL shapes', () => {
expect(isArtistPage('https://www.patreon.com/Atole', 'patreon')).toBe(true)
expect(isArtistPage('https://www.patreon.com/c/Atole', 'patreon')).toBe(true)
expect(isArtistPage('https://www.patreon.com/cw/Atole', 'patreon')).toBe(true)
})
it('matches Patreon creator inner pages', () => {
expect(isArtistPage('https://www.patreon.com/cw/Atole/posts', 'patreon')).toBe(true)
expect(isArtistPage('https://www.patreon.com/Atole/membership', 'patreon')).toBe(true)
})
it('excludes Patreon navigation pages that are not creators', () => {
for (const nav of ['home', 'search', 'messages', 'notifications', 'library', 'settings']) {
expect(isArtistPage(`https://www.patreon.com/${nav}`, 'patreon')).toBe(false)
expect(isArtistPage(`https://www.patreon.com/${nav}/anything`, 'patreon')).toBe(false)
}
})
it('matches SubscribeStar creator roots on both TLDs but not feed pages', () => {
expect(isArtistPage('https://subscribestar.adult/someone', 'subscribestar')).toBe(true)
expect(isArtistPage('https://subscribestar.com/someone', 'subscribestar')).toBe(true)
expect(isArtistPage('https://subscribestar.adult/feed', 'subscribestar')).toBe(false)
expect(isArtistPage('https://subscribestar.adult/messages', 'subscribestar')).toBe(false)
})
it('matches Hentai Foundry user pages only', () => {
expect(isArtistPage('https://www.hentai-foundry.com/user/someone', 'hentaifoundry')).toBe(true)
expect(isArtistPage('https://www.hentai-foundry.com/pictures/popular', 'hentaifoundry')).toBe(
false
)
})
it('matches Pixiv numeric user pages, with or without the /en/ prefix', () => {
expect(isArtistPage('https://www.pixiv.net/users/12345', 'pixiv')).toBe(true)
expect(isArtistPage('https://www.pixiv.net/en/users/12345', 'pixiv')).toBe(true)
expect(isArtistPage('https://www.pixiv.net/en/artworks/999', 'pixiv')).toBe(false)
})
it('excludes DeviantArt navigation roots', () => {
expect(isArtistPage('https://www.deviantart.com/someone', 'deviantart')).toBe(true)
expect(isArtistPage('https://www.deviantart.com/home', 'deviantart')).toBe(false)
expect(isArtistPage('https://www.deviantart.com/watch', 'deviantart')).toBe(false)
})
it('returns false for a platform with no artist pattern (discord)', () => {
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
})
it('returns false for an unknown platform key', () => {
expect(isArtistPage('https://www.patreon.com/Atole', 'nope')).toBe(false)
})
})
describe('platform table integrity', () => {
it('gives every artist pattern a corresponding platform entry', () => {
// A pattern keyed to a platform that no longer exists is dead code that
// silently never fires; the reverse (a platform with no pattern) is the
// legitimate discord case, so only this direction is an error.
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
expect(Object.keys(PLATFORMS)).toContain(key)
}
})
it('gives every platform the fields the popup renders', () => {
for (const [key, platform] of Object.entries(PLATFORMS)) {
expect(platform.name, `${key}.name`).toBeTruthy()
expect(platform.color, `${key}.color`).toMatch(/^#[0-9A-Fa-f]{6}$/)
expect(['cookies', 'token'], `${key}.authType`).toContain(platform.authType)
expect(platform.urlPattern, `${key}.urlPattern`).toBeInstanceOf(RegExp)
expect(Array.isArray(platform.domains), `${key}.domains`).toBe(true)
expect(platform.domains.length, `${key}.domains`).toBeGreaterThan(0)
}
})
it('keeps every artist URL matched by its own platform pattern too', () => {
// isArtistPage is only ever consulted after getPlatformFromUrl resolves a
// key, so an artist pattern matching a URL its platform's urlPattern
// rejects would be unreachable.
const samples = {
patreon: 'https://www.patreon.com/cw/Atole',
subscribestar: 'https://subscribestar.adult/someone',
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
deviantart: 'https://www.deviantart.com/someone',
pixiv: 'https://www.pixiv.net/en/users/12345'
}
for (const [key, url] of Object.entries(samples)) {
expect(isArtistPage(url, key), `${key} artist pattern`).toBe(true)
expect(getPlatformFromUrl(url), `${key} urlPattern`).toBe(key)
}
})
})
+93
View File
@@ -0,0 +1,93 @@
import { describe, it, expect } from 'vitest'
import { loadLib } from './helpers/loadLib.js'
const { normalizeApiUrl, webRootFromApiUrl } = loadLib('url.js', [
'normalizeApiUrl',
'webRootFromApiUrl'
])
describe('normalizeApiUrl', () => {
// The bug this exists for (issue #2393): the instance root was accepted and
// stored verbatim, so every request went to /credentials instead of
// /api/credentials. That path is a Vue router route, so the SPA catch-all
// answered GET with 200 HTML and rejected POST with 405 — which read as a
// backend bug rather than a URL one.
it('appends /api to an instance root', () => {
expect(normalizeApiUrl('http://curator.traefik.internal')).toBe(
'http://curator.traefik.internal/api'
)
})
it('leaves an API root alone rather than doubling the suffix', () => {
expect(normalizeApiUrl('http://curator.traefik.internal/api')).toBe(
'http://curator.traefik.internal/api'
)
})
it('is idempotent', () => {
const once = normalizeApiUrl('http://curator.example.com')
expect(normalizeApiUrl(once)).toBe(once)
})
it('strips trailing slashes before deciding', () => {
expect(normalizeApiUrl('http://curator.example.com/')).toBe('http://curator.example.com/api')
expect(normalizeApiUrl('http://curator.example.com///')).toBe('http://curator.example.com/api')
expect(normalizeApiUrl('http://curator.example.com/api/')).toBe('http://curator.example.com/api')
})
it('trims surrounding whitespace (paste artifacts)', () => {
expect(normalizeApiUrl(' http://curator.example.com ')).toBe(
'http://curator.example.com/api'
)
})
it('matches the /api suffix case-insensitively', () => {
expect(normalizeApiUrl('http://curator.example.com/API')).toBe('http://curator.example.com/API')
})
it('returns empty string for empty/nullish input, never a bare "/api"', () => {
// isConfigured() gates on truthiness, so a bogus '/api' here would read as
// "configured" and produce a request against the options page's own origin.
expect(normalizeApiUrl('')).toBe('')
expect(normalizeApiUrl(' ')).toBe('')
expect(normalizeApiUrl(null)).toBe('')
expect(normalizeApiUrl(undefined)).toBe('')
})
it('does not treat a path merely containing "api" as the suffix', () => {
expect(normalizeApiUrl('http://curator.example.com/apiary')).toBe(
'http://curator.example.com/apiary/api'
)
})
it('preserves a subpath deployment', () => {
expect(normalizeApiUrl('http://host.internal/curator')).toBe('http://host.internal/curator/api')
})
})
describe('webRootFromApiUrl', () => {
// The SPA root, where the Vue router and the served XPI live. Used by
// OPEN_ARTIST_PAGE and the self-update check — NOT the JSON API.
it('strips the /api suffix', () => {
expect(webRootFromApiUrl('http://curator.example.com/api')).toBe('http://curator.example.com')
})
it('accepts an instance root unchanged', () => {
expect(webRootFromApiUrl('http://curator.example.com')).toBe('http://curator.example.com')
})
it('agrees with normalizeApiUrl in both directions', () => {
for (const input of ['http://curator.example.com', 'http://curator.example.com/api']) {
expect(normalizeApiUrl(webRootFromApiUrl(input))).toBe(normalizeApiUrl(input))
}
})
it('preserves a subpath deployment', () => {
expect(webRootFromApiUrl('http://host.internal/curator/api')).toBe('http://host.internal/curator')
})
it('returns empty string for empty/nullish input', () => {
expect(webRootFromApiUrl('')).toBe('')
expect(webRootFromApiUrl(null)).toBe('')
})
})
+71
View File
@@ -0,0 +1,71 @@
import { describe, it, expect } from 'vitest'
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import path from 'node:path'
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
const read = (name) => JSON.parse(readFileSync(path.join(EXT_DIR, name), 'utf8'))
describe('extension version consistency', () => {
// Duplicates check (1) of ci.yml's extension-version job, deliberately.
// That job is the gate that can't be bypassed; this spec is the one that
// fails in a second on the developer's own CI lane with a readable diff.
// The two version strings feed different systems and nothing else reconciles
// them:
// manifest.json -> what `web-ext sign` signs, so what Firefox installs
// (package.json is in --ignore-files, not in the XPI)
// package.json -> build.yml's AMO cache key, the ext-<version> release
// tag, the XPI filename, and therefore the version
// /api/extension/manifest reports to the update prompt
it('keeps manifest.json and package.json in lockstep', () => {
const manifest = read('manifest.json')
const pkg = read('package.json')
expect(manifest.version).toBe(pkg.version)
})
it('uses a plain dotted numeric version AMO will accept', () => {
// AMO rejects exotic version strings, and build.yml embeds this value in a
// release tag and a filename — so anything needing escaping breaks the
// publish path rather than the extension.
expect(read('package.json').version).toMatch(/^\d+(\.\d+)*$/)
})
it('declares manifest v3', () => {
expect(read('manifest.json').manifest_version).toBe(3)
})
it('never lets the CI guard ignore a file that actually ships', () => {
// ci.yml's extension-version job skips its bump check for paths it deems
// non-shipping. If it excludes something web-ext DOES package, a real
// change to shipped code passes the guard unnoticed — precisely the
// silent-stale-ship the guard exists to stop. The reverse drift (guard
// stricter than web-ext) only costs a needless bump, so it isn't asserted.
const ci = readFileSync(path.join(EXT_DIR, '..', '.forgejo', 'workflows', 'ci.yml'), 'utf8')
const lint = read('package.json').scripts.lint
const after = lint.split('--ignore-files')[1] ?? ''
const ignored = new Set(
after
.split(/\s+/)
.filter((tok) => tok && !tok.startsWith('--'))
.map((tok) => tok.replace(/^["']|["']$/g, ''))
)
expect(ignored.size, 'parsed --ignore-files from the lint script').toBeGreaterThan(0)
const guarded = [...ci.matchAll(/:\(exclude\)extension\/(\S+?)'/g)].map((m) => m[1])
expect(guarded.length, 'parsed :(exclude) entries from ci.yml').toBeGreaterThan(0)
for (const entry of guarded) {
expect(ignored, `ci.yml excludes "${entry}" but web-ext packages it`).toContain(entry)
}
})
it('lists every background script that exists, in dependency order', () => {
// url.js must load BEFORE api.js: api.js calls normalizeApiUrl at
// init()-time, and these are classic scripts sharing one scope, so a
// reordering here is a runtime ReferenceError with no build-time signal.
const scripts = read('manifest.json').background.scripts
for (const rel of scripts) {
expect(() => readFileSync(path.join(EXT_DIR, rel)), `missing ${rel}`).not.toThrow()
}
expect(scripts.indexOf('lib/url.js')).toBeLessThan(scripts.indexOf('lib/api.js'))
})
})
+13
View File
@@ -0,0 +1,13 @@
import { defineConfig } from 'vitest/config'
// Mirrors frontend/vitest.config.js, minus the Vue plugin — the extension has
// no SFCs and mounts nothing. Pure-logic specs only, so `node` is enough; the
// libs under test are deliberately the ones with no browser-API surface (see
// test/helpers/loadLib.js).
export default defineConfig({
test: {
environment: 'node',
include: ['test/**/*.spec.js'],
passWithNoTests: true
}
})
@@ -0,0 +1,56 @@
<!--
Canonical settings number field (DRY pass #161): a compact numeric v-text-field
with a built-in clamp to [min,max] on commit. Hand-rolled identically across the
ML settings cards (HeadsCard x6, CropProposersCard, VideoEmbeddingCard).
The clamp is the point: the cards previously sent Number(raw) straight to the
API, so an out-of-range value bounced off the API's 400 validator (only
TranslationCard clamped). This is now the single home for that clamp.
Binds `modelValue` (v-model) and emits `change` on blur/enter AFTER clamping, so
the parent's save reads the already-clamped value same as the prior
`v-model.number` + `@change=save` pattern.
-->
<template>
<v-text-field
:model-value="modelValue"
:label="label"
type="number"
:min="min"
:max="max"
:step="step"
:disabled="disabled"
:density="density" hide-details
:style="{ maxWidth }"
@update:model-value="v => emit('update:modelValue', v)"
@change="onCommit"
/>
</template>
<script setup>
const props = defineProps({
modelValue: { type: [Number, String], default: null },
label: { type: String, default: '' },
min: { type: [Number, String], default: null },
max: { type: [Number, String], default: null },
step: { type: [Number, String], default: 1 },
maxWidth: { type: String, default: '200px' },
density: { type: String, default: 'compact' },
disabled: { type: Boolean, default: false },
})
const emit = defineEmits(['update:modelValue', 'change'])
function onCommit() {
// On blur/enter: coerce to a number and clamp to [min,max] so an out-of-range
// value never reaches the API. props.modelValue reflects the latest keystroke
// (kept in sync by the passthrough above); re-emit the clamped number, then let
// the parent persist.
let n = Number(props.modelValue)
if (!Number.isNaN(n)) {
if (props.min !== null && props.min !== '') n = Math.max(Number(props.min), n)
if (props.max !== null && props.max !== '') n = Math.min(Number(props.max), n)
if (n !== Number(props.modelValue)) emit('update:modelValue', n)
}
emit('change')
}
</script>
@@ -0,0 +1,42 @@
<!--
Canonical settings toggle row (DRY pass #161): an accent icon + an uppercase
.fc-section-h label + a right-aligned switch. Hand-rolled identically in the
ML settings cards (HeadsCard x3, CropProposersCard, MLBackfillCard).
Two-way binds `modelValue` (so the parent switch state stays optimistic) AND
emits `change` with the new boolean, so the parent can persist + revert on
failure matching the prior `v-model` + `@update:model-value=handler` pattern.
-->
<template>
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon v-if="icon" size="18" :color="iconColor">{{ icon }}</v-icon>
<span class="fc-section-h">{{ label }}</span>
<v-switch
:model-value="modelValue"
:loading="loading"
:disabled="disabled"
hide-details density="compact" color="success" class="ml-auto"
@update:model-value="onSwitch"
/>
</div>
</template>
<script setup>
defineProps({
modelValue: { type: Boolean, default: false },
label: { type: String, default: '' },
icon: { type: String, default: '' },
// Icon tint. Default accent; pass null for the theme default (e.g. when a row
// is off). null (not undefined) so the default doesn't override it.
iconColor: { type: String, default: 'accent' },
loading: { type: Boolean, default: false },
disabled: { type: Boolean, default: false },
})
const emit = defineEmits(['update:modelValue', 'change'])
function onSwitch(v) {
const b = !!v
emit('update:modelValue', b)
emit('change', b)
}
</script>
@@ -15,28 +15,23 @@
</p>
<div v-for="p in proposers" :key="p.key" class="fc-proposer">
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon size="18" :color="p.on ? 'accent' : undefined">{{ p.icon }}</v-icon>
<span class="fc-section-h">{{ p.label }}</span>
<v-switch
v-model="p.on" :loading="busy" hide-details density="compact"
color="success" class="ml-auto"
@update:model-value="v => saveToggle(p, v)"
/>
</div>
<SettingToggleRow
v-model="p.on" :loading="busy" :icon="p.icon"
:icon-color="p.on ? 'accent' : null" :label="p.label"
@change="v => saveToggle(p, v)"
/>
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
<div class="d-flex flex-wrap mb-4" style="gap: 12px;">
<v-text-field
v-model="p.weights" label="Weights" density="compact" hide-details
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
placeholder="name | URL | hf_repo::file"
@change="save({ [`detector_${p.key}_weights`]: p.weights })"
@change="saveField({ [`detector_${p.key}_weights`]: p.weights })"
/>
<v-text-field
v-model.number="p.conf" label="Confidence" type="number"
min="0" max="1" step="0.05" density="compact" hide-details
style="max-width: 140px;" :disabled="busy || !p.on"
@change="save({ [`detector_${p.key}_conf`]: Number(p.conf) })"
<SettingNumberField
v-model="p.conf" label="Confidence" :min="0" :max="1" :step="0.05"
max-width="140px" :disabled="busy || !p.on"
@change="saveField({ [`detector_${p.key}_conf`]: Number(p.conf) })"
/>
</div>
</div>
@@ -48,12 +43,12 @@
storage. Dedupe IoU drops near-duplicate crops before embedding.
</p>
<div class="d-flex flex-wrap" style="gap: 12px;">
<v-text-field
<SettingNumberField
v-for="c in caps" :key="c.key"
v-model.number="c.val" :label="c.label" type="number"
:min="c.min" :max="c.max" :step="c.step || 1" density="compact"
hide-details style="max-width: 165px;" :disabled="busy"
@change="save({ [c.key]: Number(c.val) })"
v-model="c.val" :label="c.label"
:min="c.min" :max="c.max" :step="c.step || 1"
max-width="165px" :disabled="busy"
@change="saveField({ [c.key]: Number(c.val) })"
/>
</div>
</div>
@@ -61,14 +56,16 @@
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { onMounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import SettingToggleRow from '../common/SettingToggleRow.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
import { useMLStore } from '../../stores/ml.js'
const mlSettings = useMLStore()
const busy = ref(false)
const { busy, save } = useSettingSave(mlSettings.patchSettings)
const proposers = ref([])
const caps = ref([])
@@ -111,31 +108,20 @@ onMounted(async () => {
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
})
async function save(patch, revert) {
busy.value = true
try {
await mlSettings.patchSettings(patch)
toast({ text: 'Saved', type: 'success' })
} catch (e) {
if (revert) revert()
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
// Field @change → persist with a "Saved" confirmation. SettingNumberField has
// already clamped numeric values to their [min,max] before this fires.
function saveField(patch) {
save(patch, { successMessage: 'Saved' })
}
function saveToggle (p, v) {
async function saveToggle(p, v) {
// Revert the switch on failure so it never lies about the persisted state.
save({ [`detector_${p.key}_enabled`]: !!v }, () => { p.on = !v })
const ok = await save({ [`detector_${p.key}_enabled`]: !!v }, { successMessage: 'Saved' })
if (!ok) p.on = !v
}
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-proposer {
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
}
@@ -112,7 +112,6 @@ async function onCommit() {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-code {
background: rgb(var(--v-theme-surface-light));
border-radius: 4px; padding: 2px 8px;
@@ -42,7 +42,7 @@
</tr>
</tbody>
</v-table>
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
<p v-else class="text-caption mt-3 fc-muted">
No table statistics yet.
</p>
</MaintenanceTile>
@@ -72,6 +72,5 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-bad { color: rgb(var(--v-theme-error)); }
</style>
@@ -95,7 +95,6 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-cells { display: flex; gap: 28px; }
.fc-cell__n {
font-size: 20px; font-weight: 700; line-height: 1.1;
@@ -105,6 +104,5 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-bad { color: rgb(var(--v-theme-error)); }
</style>
@@ -367,11 +367,6 @@ async function onReprocess() {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-token {
display: flex; align-items: center; gap: 4px;
background: rgb(var(--v-theme-surface-light)); border-radius: 6px;
@@ -390,6 +385,4 @@ async function onReprocess() {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style>
@@ -155,11 +155,6 @@ async function onRecover(it) {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-queue { display: flex; gap: 24px; }
.fc-q__n {
font-size: 20px; font-weight: 700; line-height: 1.1;
@@ -169,8 +164,6 @@ async function onRecover(it) {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
.fc-defect {
display: flex; align-items: center; gap: 12px;
background: rgb(var(--v-theme-surface-light)); border-radius: 8px;
+60 -125
View File
@@ -95,14 +95,10 @@
<!-- Earned auto-apply -->
<div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon size="18" color="accent">mdi-lightning-bolt</v-icon>
<span class="fc-section-h">Auto-apply</span>
<v-switch
v-model="autoEnabled" :loading="settingBusy" hide-details density="compact"
color="success" class="ml-auto" @update:model-value="onToggleAuto"
/>
</div>
<SettingToggleRow
v-model="autoEnabled" :loading="settingBusy"
icon="mdi-lightning-bolt" label="Auto-apply" @change="onToggleAuto"
/>
<p class="fc-muted text-body-2 mb-3">
Graduated heads (, with {{ autoMinPosInput }} examples) apply their tag
on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}%
@@ -111,17 +107,14 @@
</p>
<div class="d-flex mb-3" style="gap: 12px;">
<v-text-field
v-model.number="autoPrecisionInput" label="Precision target"
type="number" min="0.5" max="0.999" step="0.01" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
<SettingNumberField
v-model="autoPrecisionInput" label="Precision target"
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
@change="onSaveSettings"
/>
<v-text-field
v-model.number="autoMinPosInput" label="Min examples to fire"
type="number" min="1" density="compact" hide-details
style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveSettings"
<SettingNumberField
v-model="autoMinPosInput" label="Min examples to fire"
:min="1" :disabled="settingBusy" @change="onSaveSettings"
/>
</div>
@@ -161,15 +154,11 @@
<!-- Presentation chrome auto-hide (#141) -->
<div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon size="18" color="accent">mdi-image-off-outline</v-icon>
<span class="fc-section-h">Hide presentation chrome</span>
<v-switch
v-model="presentationEnabled" :loading="settingBusy" hide-details
density="compact" color="success" class="ml-auto"
@update:model-value="onTogglePresentation"
/>
</div>
<SettingToggleRow
v-model="presentationEnabled" :loading="settingBusy"
icon="mdi-image-off-outline" label="Hide presentation chrome"
@change="onTogglePresentation"
/>
<p class="fc-muted text-body-2 mb-3">
Auto-hide <code>banner</code> chrome from the gallery once a head has
learned it ( {{ minPositives }} examples) and clears
@@ -180,16 +169,14 @@
tag), it's flagged for review instead of buried. Every auto-hide is reversible.
</p>
<div class="d-flex mb-3" style="gap: 12px;">
<v-text-field
v-model.number="presentationThresholdInput" label="Hide confidence"
type="number" min="0.5" max="0.999" step="0.01" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
<SettingNumberField
v-model="presentationThresholdInput" label="Hide confidence"
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
@change="onSavePresentation"
/>
<v-text-field
v-model.number="presentationConflictInput" label="Flag if content ≥"
type="number" min="0" max="1" step="0.05" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
<SettingNumberField
v-model="presentationConflictInput" label="Flag if content ≥"
:min="0" :max="1" :step="0.05" :disabled="settingBusy"
@change="onSavePresentation"
/>
</div>
@@ -197,15 +184,11 @@
<!-- Process auto-tagging (#1464): wip / editor screenshot -->
<div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon size="18" color="accent">mdi-progress-wrench</v-icon>
<span class="fc-section-h">Auto-tag work-in-progress</span>
<v-switch
v-model="processEnabled" :loading="settingBusy" hide-details
density="compact" color="success" class="ml-auto"
@update:model-value="onToggleProcess"
/>
</div>
<SettingToggleRow
v-model="processEnabled" :loading="settingBusy"
icon="mdi-progress-wrench" label="Auto-tag work-in-progress"
@change="onToggleProcess"
/>
<p class="fc-muted text-body-2 mb-3">
Auto-tag <code>wip</code> and <code>editor screenshot</code> process art
once a head has learned them (≥ {{ minPositives }} examples) and clears
@@ -217,16 +200,14 @@
manual tags, never its own guesses so it can't run away. Every tag reversible.
</p>
<div class="d-flex mb-3" style="gap: 12px;">
<v-text-field
v-model.number="processThresholdInput" label="Tag confidence"
type="number" min="0.5" max="0.999" step="0.01" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
<SettingNumberField
v-model="processThresholdInput" label="Tag confidence"
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
@change="onSaveProcess"
/>
<v-text-field
v-model.number="processConflictInput" label="Flag if content ≥"
type="number" min="0" max="1" step="0.05" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
<SettingNumberField
v-model="processConflictInput" label="Flag if content ≥"
:min="0" :max="1" :step="0.05" :disabled="settingBusy"
@change="onSaveProcess"
/>
</div>
@@ -256,7 +237,7 @@
<td class="fc-r fc-mono">{{ c.n_auto_applied }}</td>
<td class="fc-r fc-mono">{{ c.n_misfires }}</td>
<td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)">
{{ ratePct(c.misfire_rate) }}
{{ pct(c.misfire_rate) }}
</td>
<td class="fc-r fc-mono">{{ c.n_underfires }}</td>
</tr>
@@ -272,6 +253,9 @@ import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import SettingToggleRow from '../common/SettingToggleRow.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
import { useHeadsStore } from '../../stores/heads.js'
import { useMLStore } from '../../stores/ml.js'
@@ -285,7 +269,9 @@ let pollTimer = null
const autoEnabled = ref(false)
const autoPrecisionInput = ref(0.97)
const autoMinPosInput = ref(30)
const settingBusy = ref(false)
// Shared settings-save flow (busy + toast + revert); `settingBusy` gates the
// toggles/fields, `save` returns ok/false for the optimistic-switch revert.
const { busy: settingBusy, save } = useSettingSave(mlSettings.patchSettings)
const autoBusy = ref(false)
const autoStatus = ref(null)
const metricsData = ref(null)
@@ -395,81 +381,39 @@ function startAutoPoll() {
function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } }
async function onToggleAuto(val) {
settingBusy.value = true
try {
await mlSettings.patchSettings({ head_auto_apply_enabled: !!val })
toast({ text: val ? 'Auto-apply on' : 'Auto-apply off', type: 'success' })
} catch (e) {
autoEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
const ok = await save({ head_auto_apply_enabled: !!val },
{ successMessage: val ? 'Auto-apply on' : 'Auto-apply off', errorPrefix: 'Could not update' })
if (!ok) autoEnabled.value = !val // revert the switch
}
async function onSaveSettings() {
settingBusy.value = true
try {
await mlSettings.patchSettings({
head_auto_apply_precision: Number(autoPrecisionInput.value),
head_auto_apply_min_positives: Number(autoMinPosInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
await save({
head_auto_apply_precision: Number(autoPrecisionInput.value),
head_auto_apply_min_positives: Number(autoMinPosInput.value),
})
}
async function onTogglePresentation(val) {
settingBusy.value = true
try {
await mlSettings.patchSettings({ presentation_auto_apply_enabled: !!val })
toast({ text: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', type: 'success' })
} catch (e) {
presentationEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
const ok = await save({ presentation_auto_apply_enabled: !!val },
{ successMessage: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', errorPrefix: 'Could not update' })
if (!ok) presentationEnabled.value = !val // revert the switch
}
async function onSavePresentation() {
settingBusy.value = true
try {
await mlSettings.patchSettings({
presentation_auto_apply_threshold: Number(presentationThresholdInput.value),
presentation_conflict_threshold: Number(presentationConflictInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
await save({
presentation_auto_apply_threshold: Number(presentationThresholdInput.value),
presentation_conflict_threshold: Number(presentationConflictInput.value),
})
}
async function onToggleProcess(val) {
settingBusy.value = true
try {
await mlSettings.patchSettings({ process_auto_apply_enabled: !!val })
toast({ text: val ? 'WIP auto-tag on' : 'WIP auto-tag off', type: 'success' })
} catch (e) {
processEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
const ok = await save({ process_auto_apply_enabled: !!val },
{ successMessage: val ? 'WIP auto-tag on' : 'WIP auto-tag off', errorPrefix: 'Could not update' })
if (!ok) processEnabled.value = !val // revert the switch
}
async function onSaveProcess() {
settingBusy.value = true
try {
await mlSettings.patchSettings({
process_auto_apply_threshold: Number(processThresholdInput.value),
process_conflict_threshold: Number(processConflictInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
await save({
process_auto_apply_threshold: Number(processThresholdInput.value),
process_conflict_threshold: Number(processConflictInput.value),
})
}
function onPreview() { startSweep(true) }
function onApplyNow() { startSweep(false) }
@@ -495,7 +439,6 @@ function sweepConcepts(run) {
.sort((a, b) => b.applied - a.applied)
}
function sweepTotal(run) { return run?.n_applied ?? 0 }
function ratePct(x) { return x == null ? '—' : `${Math.round(x * 100)}%` }
function rateClass(x) {
if (x == null) return ''
if (x <= 0.03) return 'fc-good'
@@ -526,12 +469,6 @@ function relTime(iso) {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-auto {
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px;
}
@@ -588,7 +525,5 @@ function relTime(iso) {
background: rgb(var(--v-theme-surface-light));
padding: 1px 6px; border-radius: 999px;
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-ok { color: rgb(var(--v-theme-on-surface)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style>
@@ -39,13 +39,14 @@
import { toast } from '../../utils/toast.js'
import { onMounted, ref } from 'vue'
import { useMLStore } from '../../stores/ml.js'
import { useSettingSave } from '../../composables/useSettingSave.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue'
const store = useMLStore()
const { busy: saving, save } = useSettingSave(store.patchSettings)
const busy = ref(false)
const done = ref(false)
const enabled = ref(true)
const saving = ref(false)
onMounted(async () => {
try {
await store.loadSettings()
@@ -55,21 +56,12 @@ onMounted(async () => {
} catch { /* non-fatal */ }
})
async function onToggle() {
saving.value = true
try {
await store.patchSettings({ cpu_embed_enabled: enabled.value })
toast({
text: enabled.value
? 'CPU embedding on — imports queue embeds for the ml-worker'
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
type: 'success',
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
enabled.value = !enabled.value
} finally {
saving.value = false
}
const ok = await save({ cpu_embed_enabled: enabled.value }, {
successMessage: enabled.value
? 'CPU embedding on — imports queue embeds for the ml-worker'
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
})
if (!ok) enabled.value = !enabled.value
}
async function run() {
busy.value = true
@@ -80,5 +72,4 @@ async function run() {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -24,6 +24,7 @@
the CPU fallback.
</p>
<div class="fc-tile-stack">
<VideoEmbeddingCard />
<GpuAgentCard />
<GpuTriageCard />
<MLBackfillCard />
@@ -36,7 +37,6 @@
Suggestion thresholds, trained heads and tag aliases.
</p>
<div class="fc-tile-stack">
<MLThresholdSliders />
<CropProposersCard />
<HeadsCard />
<AliasTable />
@@ -77,7 +77,7 @@ import ArchiveReextractCard from './ArchiveReextractCard.vue'
import MissingFileRepairCard from './MissingFileRepairCard.vue'
import GpuTriageCard from './GpuTriageCard.vue'
import DbMaintenanceCard from './DbMaintenanceCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import VideoEmbeddingCard from './VideoEmbeddingCard.vue'
import CropProposersCard from './CropProposersCard.vue'
import HeadsCard from './HeadsCard.vue'
import GpuAgentCard from './GpuAgentCard.vue'
@@ -12,17 +12,17 @@
</div>
<v-row>
<v-col cols="12" sm="6">
<v-text-field
v-model.number="local.video_frame_interval_seconds"
label="Frame interval (s)" type="number" min="0.5" step="0.5"
density="comfortable" hide-details @change="save"
<SettingNumberField
v-model="local.video_frame_interval_seconds"
label="Frame interval (s)" :min="0.5" :step="0.5"
density="comfortable" max-width="none" @change="onSave"
/>
</v-col>
<v-col cols="12" sm="6">
<v-text-field
v-model.number="local.video_max_frames"
label="Max frames" type="number" min="1" step="1"
density="comfortable" hide-details @change="save"
<SettingNumberField
v-model="local.video_max_frames"
label="Max frames" :min="1" :step="1"
density="comfortable" max-width="none" @change="onSave"
/>
</v-col>
</v-row>
@@ -32,21 +32,23 @@
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { reactive, watch } from 'vue'
import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import SettingNumberField from '../common/SettingNumberField.vue'
import { useSettingSave } from '../../composables/useSettingSave.js'
const store = useMLStore()
const { save } = useSettingSave(store.patchSettings)
const local = reactive({})
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
async function save() {
const patch = {
video_frame_interval_seconds: local.video_frame_interval_seconds,
video_max_frames: local.video_max_frames
}
try { await store.patchSettings(patch) }
catch (e) { toast({ text: e.message, type: 'error' }) }
// SettingNumberField clamps interval to 0.5 and max-frames to 1 before this
// fires, so an out-of-range value never reaches the API.
function onSave() {
save({
video_frame_interval_seconds: Number(local.video_frame_interval_seconds),
video_max_frames: Number(local.video_max_frames),
})
}
</script>
@@ -0,0 +1,33 @@
import { ref } from 'vue'
import { toast } from '../utils/toast.js'
// The shared "persist a settings patch" flow for the ML settings cards. Flips a
// busy flag, calls the store's patch (which rethrows on failure), toasts
// success/error, and returns true/false so a toggle handler can revert its
// optimistic switch on failure. Centralises the try/catch/toast the cards each
// hand-rolled (HeadsCard x6, CropProposersCard, MLBackfillCard) — and where the
// threshold-clamp drifted; the clamp now lives in <SettingNumberField>.
//
// Pass the store's patch fn, e.g. useSettingSave(ml.patchSettings).
export function useSettingSave(patchFn) {
const busy = ref(false)
// opts.successMessage — toast on success (toggles announce their new state;
// silent field-saves omit it). opts.errorPrefix — the failure toast prefix
// ("Could not save" default; toggles used "Could not update").
async function save(patch, { successMessage = '', errorPrefix = 'Could not save' } = {}) {
busy.value = true
try {
await patchFn(patch)
if (successMessage) toast({ text: successMessage, type: 'success' })
return true
} catch (e) {
toast({ text: `${errorPrefix}: ${e.message}`, type: 'error' })
return false
} finally {
busy.value = false
}
}
return { busy, save }
}
+14
View File
@@ -40,6 +40,20 @@
emits, so no specificity/reorder fight — no !important needed. */
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
/* Section sub-heading in settings cards (DRY pass #161): was redefined
identically in 4 cards, and TranslationCard used the class with NO local def
so its section headers rendered unstyled. Now one global utility. */
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
/* Status text colours (DRY pass #161): fc-good = success, fc-weak = error,
consolidated from the GPU / heads cards. fc-ok is intentionally NOT global —
it means on-surface in HeadsCard but success in QueuesTable. */
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
/* Vuetify 4 dropped its global CSS reset (normalisation moved into each
component). FC's layouts assumed the reset zeroed margins on text elements, so
restore just that — the "minimal reset" from the v4 upgrade guide — inside
-1
View File
@@ -284,7 +284,6 @@ onUnmounted(() => {
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
/* Full-height workspace under the sticky top nav. --fc-nav-h is the nav's REAL
measured height (set by TopNav) — a hardcoded 64px here overflowed the
-1
View File
@@ -349,7 +349,6 @@ async function onDeleteTagConfirm() {
.fc-tags__sentinel {
display: flex; justify-content: center; padding: 32px 0; min-height: 60px;
}
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-merge-preview {
padding: 10px 12px;
border: 1px solid rgba(var(--v-theme-on-surface), 0.12);
+59
View File
@@ -0,0 +1,59 @@
"""Shared ML helpers extracted in the DRY pass (milestone #161). These pin the
single sources the auto-apply sweeps now trust, so a future edit can't silently
drift them: `_applied_or_rejected` is the skip-set used by auto_apply_sweep,
system_tag_auto_apply_sweep (heads.py) and scheduled_ccip_auto_apply (tasks/ml.py);
`_sigmoid` is the head score→prob transform used at every scoring site."""
import pytest
from backend.app.models import ImageRecord, Tag, TagKind, TagSuggestionRejection
from backend.app.models.tag import image_tag
from backend.app.services.ml.training_data import _applied_or_rejected
def test_sigmoid_matches_naive_form():
import numpy as np
from backend.app.services.ml.heads import _sigmoid
z = np.array([-3.0, -0.5, 0.0, 1.5, 12.0], dtype=np.float32)
assert np.allclose(_sigmoid(z, np), 1.0 / (1.0 + np.exp(-z)))
assert float(_sigmoid(np.array([0.0]), np)[0]) == pytest.approx(0.5)
@pytest.mark.integration
def test_applied_or_rejected_unions_applied_any_source_and_rejected(db_sync):
a = Tag(name="dry-helper-a", kind=TagKind.general)
b = Tag(name="dry-helper-b", kind=TagKind.general)
db_sync.add_all([a, b])
db_sync.flush()
imgs = []
for i in range(5):
img = ImageRecord(
path=f"/images/dryhelp{i}.jpg", sha256=f"{i:064d}", size_bytes=1,
mime="image/jpeg", width=1, height=1, origin="imported_filesystem",
integrity_status="unknown", siglip_embedding=[0.0] * 1152,
)
db_sync.add(img)
imgs.append(img)
db_sync.flush()
# tag a: applied manually (img0), applied by an AUTO source (img1), rejected (img2).
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[0].id, tag_id=a.id, source="manual"))
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[1].id, tag_id=a.id, source="head_auto"))
db_sync.add(TagSuggestionRejection(image_record_id=imgs[2].id, tag_id=a.id))
# tag b: applied to img3 only.
db_sync.execute(image_tag.insert().values(
image_record_id=imgs[3].id, tag_id=b.id, source="manual"))
db_sync.flush()
skip = _applied_or_rejected(db_sync, [a.id, b.id])
# Applied-under-ANY-source (manual + head_auto) rejected, kept per-tag; the
# untouched image (img4) appears under neither tag.
assert skip[a.id] == {imgs[0].id, imgs[1].id, imgs[2].id}
assert skip[b.id] == {imgs[3].id}
assert imgs[4].id not in skip[a.id]
assert imgs[4].id not in skip[b.id]
+63
View File
@@ -0,0 +1,63 @@
"""recover_stalled_head_training_runs + recover_stalled_head_auto_apply_runs share
one helper (_recover_stalled_runs, DRY pass #161). These pin BOTH wrappers so the
shared source stays correct: a 'running' row with no progress past the stall
threshold flips to 'error'; a fresh 'running' row is left alone."""
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import select
from backend.app.models import HeadAutoApplyRun, HeadTrainingRun
pytestmark = pytest.mark.integration
def test_recover_stalled_head_training_runs_flips_stalled_keeps_fresh(db_sync):
from backend.app.tasks.maintenance import recover_stalled_head_training_runs
stale = HeadTrainingRun(
params={}, status="running",
last_progress_at=datetime.now(UTC) - timedelta(days=1),
)
fresh = HeadTrainingRun(
params={}, status="running", last_progress_at=datetime.now(UTC),
)
db_sync.add_all([stale, fresh])
db_sync.commit()
stale_id, fresh_id = stale.id, fresh.id
assert recover_stalled_head_training_runs.apply().get() == 1
db_sync.expire_all()
assert db_sync.execute(
select(HeadTrainingRun.status).where(HeadTrainingRun.id == stale_id)
).scalar_one() == "error"
assert db_sync.execute(
select(HeadTrainingRun.status).where(HeadTrainingRun.id == fresh_id)
).scalar_one() == "running"
def test_recover_stalled_head_auto_apply_runs_flips_stalled_keeps_fresh(db_sync):
from backend.app.tasks.maintenance import recover_stalled_head_auto_apply_runs
stale = HeadAutoApplyRun(
dry_run=False, params={}, status="running",
last_progress_at=datetime.now(UTC) - timedelta(days=1),
)
fresh = HeadAutoApplyRun(
dry_run=False, params={}, status="running",
last_progress_at=datetime.now(UTC),
)
db_sync.add_all([stale, fresh])
db_sync.commit()
stale_id, fresh_id = stale.id, fresh.id
assert recover_stalled_head_auto_apply_runs.apply().get() == 1
db_sync.expire_all()
assert db_sync.execute(
select(HeadAutoApplyRun.status).where(HeadAutoApplyRun.id == stale_id)
).scalar_one() == "error"
assert db_sync.execute(
select(HeadAutoApplyRun.status).where(HeadAutoApplyRun.id == fresh_id)
).scalar_one() == "running"