Compare commits
207
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ca6d26d236 | ||
|
|
5366047d55 | ||
|
|
2923257529 | ||
|
|
934731e9ef | ||
|
|
4c67c116b2 | ||
|
|
f8b667604f | ||
|
|
11572f469a | ||
|
|
ba3fd2a118 | ||
|
|
9bf095179a | ||
|
|
cefc064e0f | ||
|
|
06757daf80 | ||
|
|
a723ef436b | ||
|
|
d68f39ca50 | ||
|
|
a66a695977 | ||
|
|
5289fa3879 | ||
|
|
ebac34e17b | ||
|
|
4c3ba20198 | ||
|
|
d2acec61ae | ||
|
|
15ae5ef4fa | ||
|
|
e3ceefc820 | ||
|
|
e52090a4cf | ||
|
|
bfba8045e4 | ||
|
|
a708357436 | ||
|
|
d9354ac1e1 | ||
|
|
1d48770793 | ||
|
|
489e6aaaee | ||
|
|
ed20df905b | ||
|
|
5ba9871ef0 | ||
|
|
2a820d0848 | ||
|
|
2f9aa3d86c | ||
|
|
560a5000a2 | ||
|
|
7ddad231f8 | ||
|
|
a78f7eaace | ||
|
|
71337b0ba4 | ||
|
|
3996205f3b | ||
|
|
9f9db01456 | ||
|
|
216b7fc743 | ||
|
|
fbb76e6f36 | ||
|
|
5ef1478ade | ||
|
|
88ff4147e1 | ||
|
|
1ea02ad44c | ||
|
|
937cfb65b4 | ||
|
|
baac851220 | ||
|
|
a28c33281a | ||
|
|
40be0a9323 | ||
|
|
2c6bf26bfc | ||
|
|
3c1e76bd44 | ||
|
|
831c5b1c10 | ||
|
|
76b6af4903 | ||
|
|
92494ec4ed | ||
|
|
bdfc17477c | ||
|
|
6cd3153bf4 | ||
|
|
5f2853168a | ||
|
|
9a979ee808 | ||
|
|
3138f912fd | ||
|
|
9df874e396 | ||
|
|
6915a7590a | ||
|
|
5e8c28236a | ||
|
|
7e3c0f0b74 | ||
|
|
5d0c7ba706 | ||
|
|
18300e1f8a | ||
|
|
d52ac0a0e2 | ||
|
|
401fe8213e | ||
|
|
e8774d7953 | ||
|
|
bc0f00c51b | ||
|
|
1bef68aa29 | ||
|
|
1a4bc2f981 | ||
|
|
862ace69d6 | ||
|
|
abf88b1a15 | ||
|
|
c05dcafbea | ||
|
|
55e8632dab | ||
|
|
825e6b90bf | ||
|
|
fb012c557c | ||
|
|
66593ab895 | ||
|
|
b266a54ad3 | ||
|
|
ad803b646f | ||
|
|
1f5da3d283 | ||
|
|
93034f580d | ||
|
|
9b9b12f410 | ||
|
|
376d310693 | ||
|
|
bc69495a16 | ||
|
|
478f898e72 | ||
|
|
38a5e7f332 | ||
|
|
57fe15c267 | ||
|
|
eb3231ef10 | ||
|
|
e9af459c0d | ||
|
|
6f02806aec | ||
|
|
a1d19bd96a | ||
|
|
26827ff38f | ||
|
|
26dcfaf6c2 | ||
|
|
9b1b0369cc | ||
|
|
18123fb9cb | ||
|
|
2e806f202f | ||
|
|
18d5c05639 | ||
|
|
11ddfc3876 | ||
|
|
2b8ce86622 | ||
|
|
49bee77cdc | ||
|
|
c209e3b37e | ||
|
|
cffdd93418 | ||
|
|
fd84be40dd | ||
|
|
79f510d7f8 | ||
|
|
59181069da | ||
|
|
428ecd8642 | ||
|
|
ed1e04b831 | ||
|
|
f5156bd847 | ||
|
|
dfc3922d24 | ||
|
|
3eb08e926b | ||
|
|
9e81ced359 | ||
|
|
11e9f5af60 | ||
|
|
909fa37b15 | ||
|
|
dfab8f65ff | ||
|
|
618f7cdc36 | ||
|
|
028ea33a7c | ||
|
|
444c1fb075 | ||
|
|
26c68b0a75 | ||
|
|
e75427b19a | ||
|
|
5447fab987 | ||
|
|
bad37e07b2 | ||
|
|
2bfc9936a1 | ||
|
|
4c6406ee18 | ||
|
|
bb47e80b3e | ||
|
|
dc1083b5e0 | ||
|
|
e46893fefd | ||
|
|
0666e15211 | ||
|
|
747390631d | ||
|
|
e0d2a20588 | ||
|
|
1d84f67418 | ||
|
|
91265df3d6 | ||
|
|
11acdb0322 | ||
|
|
2eb9fd5dd0 | ||
|
|
01e5ce1410 | ||
|
|
3bb94674cf | ||
|
|
a75c602175 | ||
|
|
ef8f4f7193 | ||
|
|
ec3d27b219 | ||
|
|
03bd3b2eda | ||
|
|
7395e77d75 | ||
|
|
575d817919 | ||
|
|
2a8f7cd8b6 | ||
|
|
83f8af8090 | ||
|
|
9a2617c1a2 | ||
|
|
81688815a0 | ||
|
|
773128c3bf | ||
|
|
ce7b154ae9 | ||
|
|
9430a9d9c3 | ||
|
|
23aee56ce3 | ||
|
|
711abea567 | ||
|
|
844bb86802 | ||
|
|
a8f6a464aa | ||
|
|
ab9922ad2e | ||
|
|
0533807669 | ||
|
|
279dff3fb6 | ||
|
|
37e66cddc4 | ||
|
|
9cf6b2d363 | ||
|
|
6ef0fed41f | ||
|
|
89b48f8f35 | ||
|
|
d60e0b9494 | ||
|
|
9c27a2d3c7 | ||
|
|
93e37681b7 | ||
|
|
64ca858574 | ||
|
|
9d0c0b7da8 | ||
|
|
8e4d252ae4 | ||
|
|
fdd3e01f56 | ||
|
|
c82fb308b6 | ||
|
|
8cf8d2ca4d | ||
|
|
b1d58bc3b8 | ||
|
|
65386f02a0 | ||
|
|
667b05f14e | ||
|
|
856e9104b4 | ||
|
|
0397642b21 | ||
|
|
237575447d | ||
|
|
ed358757dc | ||
|
|
d181f4afb8 | ||
|
|
2886fa4997 | ||
|
|
f256f587ee | ||
|
|
384d8d5e50 | ||
|
|
319e8c1d18 | ||
|
|
9075d8eadd | ||
|
|
88e53e5b86 | ||
|
|
37e8b796a1 | ||
|
|
4e82208926 | ||
|
|
52fff00353 | ||
|
|
c14338cbce | ||
|
|
8c36dd28b0 | ||
|
|
88cfb3dd02 | ||
|
|
5d4f223b71 | ||
|
|
05090c6e85 | ||
|
|
3a577d5ade | ||
|
|
f4fe02e346 | ||
|
|
e766197d99 | ||
|
|
3872e1dda9 | ||
|
|
9814f3dbaf | ||
|
|
b214460fdb | ||
|
|
ac55d0e8d8 | ||
|
|
89a89e0ded | ||
|
|
4e9aac2c05 | ||
|
|
2879ac6f2b | ||
|
|
b8dce6c483 | ||
|
|
d1c0b82a22 | ||
|
|
5526b8dc78 | ||
|
|
16eb7075c4 | ||
|
|
885dcf64f3 | ||
|
|
f2f6b6d25e | ||
|
|
0822240fde | ||
|
|
27f7f3fd01 | ||
|
|
c5bf564f53 | ||
|
|
602c7d275d |
+25
-188
@@ -2,18 +2,10 @@ name: Build images
|
|||||||
|
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
# `:dev` builds were dropped 2026-05-26 to save a docker build per dev
|
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
|
||||||
# push, on the reasoning that "operator tests from `:latest` after
|
# merge-to-main, not from the dev branch image. Saves one full docker
|
||||||
# merge-to-main". Restored 2026-08-27: that is testing by shipping, and
|
# build per dev push.
|
||||||
# family rules 146/147 now name it directly — `main` IS production, and a
|
branches: [main]
|
||||||
# channel that can only be refreshed by shipping is not a channel. The
|
|
||||||
# pressure to merge in order to try something does not come from
|
|
||||||
# carelessness; it comes from `:dev` being unable to carry the build.
|
|
||||||
#
|
|
||||||
# All three images build on dev, deliberately: a `:dev` web image paired
|
|
||||||
# with a stale `:dev` ml or agent is a worse trap than no dev channel at
|
|
||||||
# all, since the mismatch only shows up as a runtime failure.
|
|
||||||
branches: [main, dev]
|
|
||||||
# Tag-push triggers an immutable per-version image build (e.g.
|
# Tag-push triggers an immutable per-version image build (e.g.
|
||||||
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
|
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
|
||||||
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
|
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
|
||||||
@@ -33,135 +25,28 @@ jobs:
|
|||||||
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
|
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
|
||||||
# Result is uploaded as an Actions artifact for build-web to consume.
|
# Result is uploaded as an Actions artifact for build-web to consume.
|
||||||
#
|
#
|
||||||
# Why this lives in build.yml (not a separate workflow): the image a push
|
# Why this lives in build.yml (not a separate workflow): the merge-commit's
|
||||||
# publishes MUST carry the XPI. A separate sign workflow racing build.yml
|
# docker image tagged `:latest` MUST carry the XPI. A separate sign workflow
|
||||||
# leaves that image without one for ~5min (until the commit-back triggers
|
# racing build.yml leaves `:latest` without the XPI for ~5min (until the
|
||||||
# another build). Inline ordering eliminates the race.
|
# commit-back triggers another build). Inline ordering eliminates the race.
|
||||||
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
|
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
|
||||||
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
|
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
|
||||||
# branch to manage). AMO blocks re-signing the same version (returns 409),
|
# branch to manage). AMO blocks re-signing the same version (returns 409),
|
||||||
# so signing is intentionally one-shot per version.
|
# so signing is intentionally one-shot per version bump.
|
||||||
#
|
|
||||||
# BOTH branches sign (milestone 271 step 6, 2026-08-27). Not two signatures:
|
|
||||||
# the version is the commit TIME of the newest packaged-extension change, so
|
|
||||||
# dev and main derive the SAME number for the same extension source. A dev
|
|
||||||
# push that changes the extension signs it; the merge to main then finds the
|
|
||||||
# ext-<version> release already there, hits the cache, and bundles the
|
|
||||||
# byte-identical XPI into `:latest` with no second AMO call. One signature
|
|
||||||
# per extension CHANGE, shared by both channels — that is what makes two
|
|
||||||
# channels affordable, and it is why step 4 (derived version) had to land
|
|
||||||
# first. Ungating this while the version was still the hand-set 1.0.11 would
|
|
||||||
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
|
|
||||||
# `:dev` — a dev channel confidently serving old code.
|
|
||||||
#
|
|
||||||
# Tags stay excluded: the tag path deliberately skips signing and polls for
|
|
||||||
# the release instead (see build-web's race note, 2026-05-27).
|
|
||||||
sign-extension:
|
sign-extension:
|
||||||
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
|
if: github.ref == 'refs/heads/main'
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
with:
|
|
||||||
# Full history is load-bearing, not a convenience: the version this
|
|
||||||
# job signs is derived from the commit TIME of the newest packaged
|
|
||||||
# extension change. A depth-1 clone sees one commit and derives a
|
|
||||||
# wrong, too-low value rather than failing (ci-requirements.md).
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
# The version is DERIVED, not read from the repo (milestone 271 step 4,
|
- name: Resolve extension version
|
||||||
# cut over 2026-08-27). `packaging.sh version` returns MAJOR.MINOR from
|
|
||||||
# manifest.json plus a patch component that is the commit TIME of the
|
|
||||||
# newest change to a PACKAGED extension file, in minutes since
|
|
||||||
# 2020-01-01 — family rule 149, never a commit count, which orders by
|
|
||||||
# branch rather than by recency.
|
|
||||||
#
|
|
||||||
# The committed "version" in manifest.json / package.json no longer
|
|
||||||
# decides anything: the stamp step below overwrites it in the working
|
|
||||||
# tree before web-ext ever reads it. It is deliberately NOT committed
|
|
||||||
# back — the commit carrying the bump would itself be a change to the
|
|
||||||
# extension and would move the version again. The repo holds the source;
|
|
||||||
# the build derives the label.
|
|
||||||
- name: Derive extension version
|
|
||||||
id: extver
|
id: extver
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
||||||
VERSION=$(sh extension/scripts/packaging.sh version)
|
|
||||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||||
echo "Derived extension version: $VERSION"
|
echo "Resolved extension version: $VERSION"
|
||||||
|
|
||||||
# Firefox refuses a downgrade and AMO never releases a burned version,
|
|
||||||
# so a version that moves BACKWARDS is unrecoverable: it strands every
|
|
||||||
# install that already took the higher one. Two ways it could happen —
|
|
||||||
# a checkout without full history (derives too low), or a rewritten
|
|
||||||
# history that drops the newest packaged commit.
|
|
||||||
#
|
|
||||||
# The test is `derived < highest already signed`, strictly. Equality is
|
|
||||||
# the ORDINARY case, not a fault: an unchanged extension derives the same
|
|
||||||
# version it did last build, which is exactly what lets the ext-<version>
|
|
||||||
# cache hit and holds AMO to one call per extension CHANGE. Only moving
|
|
||||||
# backwards is a failure, so this runs on every path — cache hit
|
|
||||||
# included — rather than only before a sign.
|
|
||||||
- name: Guard — the derived version must never go backwards
|
|
||||||
env:
|
|
||||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
||||||
DERIVED: ${{ steps.extver.outputs.version }}
|
|
||||||
run: |
|
|
||||||
python3 - <<'PY'
|
|
||||||
import json, os, sys, urllib.request
|
|
||||||
|
|
||||||
API = ("https://git.fabledsword.com/api/v1/repos/"
|
|
||||||
"bvandeusen/FabledCurator/releases")
|
|
||||||
headers = {"Authorization": "token " + os.environ["TOKEN"]}
|
|
||||||
|
|
||||||
# Paginated rather than first-page-only: ext-* releases share this
|
|
||||||
# list with the v* release tags, so one page would start missing them
|
|
||||||
# as those accumulate. The bound FAILS rather than silently scanning
|
|
||||||
# part of the list and calling the highest it saw the highest there is.
|
|
||||||
tags = []
|
|
||||||
for page in range(1, 21):
|
|
||||||
req = urllib.request.Request(
|
|
||||||
f"{API}?limit=50&page={page}", headers=headers)
|
|
||||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
|
||||||
batch = json.load(resp)
|
|
||||||
if not batch:
|
|
||||||
break
|
|
||||||
tags += [r.get("tag_name", "") for r in batch]
|
|
||||||
else:
|
|
||||||
sys.exit("guard: >1000 releases — pagination bound reached")
|
|
||||||
|
|
||||||
def parse(v):
|
|
||||||
try:
|
|
||||||
return tuple(int(part) for part in v.split("."))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
derived_s = os.environ["DERIVED"]
|
|
||||||
derived = parse(derived_s)
|
|
||||||
if derived is None:
|
|
||||||
sys.exit(f"guard: derived version {derived_s!r} is not numeric")
|
|
||||||
|
|
||||||
signed = sorted(
|
|
||||||
(v, t) for t in tags if t.startswith("ext-")
|
|
||||||
for v in [parse(t[4:])] if v
|
|
||||||
)
|
|
||||||
if not signed:
|
|
||||||
print("guard: no ext-* release yet — nothing to go backwards from")
|
|
||||||
raise SystemExit(0)
|
|
||||||
|
|
||||||
hi, hi_tag = signed[-1]
|
|
||||||
print(f"guard: derived={derived_s} highest already signed={hi_tag}")
|
|
||||||
if derived < hi:
|
|
||||||
sys.exit(
|
|
||||||
f"REFUSING TO SIGN: derived {derived_s} is OLDER than the "
|
|
||||||
f"already-signed {hi_tag}. Firefox would reject it as a "
|
|
||||||
f"downgrade, and AMO will not release the burned version. "
|
|
||||||
f"First thing to check: did this job check out with "
|
|
||||||
f"fetch-depth: 0?"
|
|
||||||
)
|
|
||||||
print("guard: ok")
|
|
||||||
PY
|
|
||||||
|
|
||||||
- name: Check Forgejo release-asset cache
|
- name: Check Forgejo release-asset cache
|
||||||
id: cache
|
id: cache
|
||||||
@@ -199,29 +84,6 @@ jobs:
|
|||||||
# removal — sign-extension's job is just to ensure the cache
|
# removal — sign-extension's job is just to ensure the cache
|
||||||
# exists on Forgejo; the build-web side reads it independently).
|
# exists on Forgejo; the build-web side reads it independently).
|
||||||
|
|
||||||
# web-ext signs whatever manifest.json says, so the derived value has to
|
|
||||||
# reach the tree before signing. package.json is written too: the two are
|
|
||||||
# required to agree (ci.yml's guard), and a local `npm run build` reads
|
|
||||||
# it. Working tree only — never committed, per the note on the derive
|
|
||||||
# step.
|
|
||||||
- name: Stamp the derived version into manifest.json + package.json
|
|
||||||
env:
|
|
||||||
DERIVED: ${{ steps.extver.outputs.version }}
|
|
||||||
run: |
|
|
||||||
python3 - <<'PY'
|
|
||||||
import json, os
|
|
||||||
|
|
||||||
version = os.environ["DERIVED"]
|
|
||||||
for path in ("extension/manifest.json", "extension/package.json"):
|
|
||||||
with open(path) as fh:
|
|
||||||
doc = json.load(fh)
|
|
||||||
doc["version"] = version
|
|
||||||
with open(path, "w") as fh:
|
|
||||||
json.dump(doc, fh, indent=2)
|
|
||||||
fh.write("\n")
|
|
||||||
print(f"{path}: version -> {version}")
|
|
||||||
PY
|
|
||||||
|
|
||||||
- name: Sign via AMO (cache miss)
|
- name: Sign via AMO (cache miss)
|
||||||
if: steps.cache.outputs.cached != 'true'
|
if: steps.cache.outputs.cached != 'true'
|
||||||
run: |
|
run: |
|
||||||
@@ -248,11 +110,6 @@ jobs:
|
|||||||
# created it so an upload failure below can roll back (don't
|
# created it so an upload failure below can roll back (don't
|
||||||
# leave an empty release tombstone that the next run's
|
# leave an empty release tombstone that the next run's
|
||||||
# cache-check mistakes for a partial-failure state).
|
# cache-check mistakes for a partial-failure state).
|
||||||
#
|
|
||||||
# target_commitish is the signing commit, not a branch name: since
|
|
||||||
# step 6 either branch can create this release, and hard-coding
|
|
||||||
# `main` would tag a dev-signed XPI against a main commit that may
|
|
||||||
# not even contain the extension source it was built from.
|
|
||||||
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
||||||
-H "Authorization: token $TOKEN" \
|
-H "Authorization: token $TOKEN" \
|
||||||
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
||||||
@@ -260,7 +117,7 @@ jobs:
|
|||||||
CREATED_BY_US=false
|
CREATED_BY_US=false
|
||||||
else
|
else
|
||||||
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
||||||
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"$GITHUB_SHA\"}" \
|
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"main\"}" \
|
||||||
-o release.json \
|
-o release.json \
|
||||||
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
|
||||||
CREATED_BY_US=true
|
CREATED_BY_US=true
|
||||||
@@ -303,31 +160,19 @@ jobs:
|
|||||||
|
|
||||||
build-web:
|
build-web:
|
||||||
needs: [sign-extension]
|
needs: [sign-extension]
|
||||||
# sign-extension runs on main and dev, and is skipped on a tag push (which
|
# sign-extension is main-only; on dev it's skipped, build-web still runs.
|
||||||
# polls for the release instead). Either is fine to build on; a FAILED sign
|
|
||||||
# is not — this condition lets success and skipped through, so a failure
|
|
||||||
# skips build-web rather than shipping an image without the XPI.
|
|
||||||
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
|
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
with:
|
|
||||||
# Full history: this job RE-DERIVES the extension version rather than
|
|
||||||
# being handed it, and a depth-1 clone derives a wrong, too-low value
|
|
||||||
# rather than failing — which would 404 the download of a release
|
|
||||||
# that exists perfectly well under its real name.
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- name: Download signed XPI from Forgejo release asset
|
- name: Download signed XPI from Forgejo release asset (main + tags)
|
||||||
# Fires on every trigger shape. dev and main each bundle the XPI their
|
# Fires on main-push AND on tag-push. Tag-push builds re-package the
|
||||||
# own sign-extension just published — that is the whole point of the
|
# same source code as the preceding main-push build but with an
|
||||||
# channel work (milestone 271 step 6): the dev image carries the
|
# immutable version tag — they need the XPI too, otherwise the
|
||||||
# extension being developed, rather than requiring a merge to try it.
|
# versioned image ships without the signed extension.
|
||||||
# Tag-push builds re-package the same source as the preceding main-push
|
|
||||||
# build but with an immutable version tag — they need the XPI too,
|
|
||||||
# otherwise the versioned image ships without the signed extension.
|
|
||||||
#
|
#
|
||||||
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
|
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
|
||||||
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
|
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
|
||||||
@@ -339,18 +184,12 @@ jobs:
|
|||||||
# for up to 10min total) before giving up. Main-push's signing
|
# for up to 10min total) before giving up. Main-push's signing
|
||||||
# eventually wins and tag-push picks the release up on a later
|
# eventually wins and tag-push picks the release up on a later
|
||||||
# iteration.
|
# iteration.
|
||||||
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/')
|
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
|
||||||
env:
|
env:
|
||||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
run: |
|
run: |
|
||||||
set -eux
|
set -eux
|
||||||
# Re-derived, not read from the repo: sign-extension published
|
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
||||||
# ext-<derived>, and the committed version has been inert since
|
|
||||||
# milestone 271 step 4. Both jobs run `packaging.sh version` over the
|
|
||||||
# same commit, so they agree by construction — and if they ever
|
|
||||||
# didn't, this download 404s and the build fails loudly instead of
|
|
||||||
# shipping a stale XPI.
|
|
||||||
VERSION=$(sh extension/scripts/packaging.sh version)
|
|
||||||
# Poll for the ext-<version> release. main-push's sign-extension
|
# Poll for the ext-<version> release. main-push's sign-extension
|
||||||
# step (AMO round-trip, 1-5min) needs to finish + upload before
|
# step (AMO round-trip, 1-5min) needs to finish + upload before
|
||||||
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
|
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
|
||||||
@@ -416,11 +255,9 @@ jobs:
|
|||||||
# rollback unit"). Rollback to any commit
|
# rollback unit"). Rollback to any commit
|
||||||
# becomes `docker pull …:c-<sha>` without a
|
# becomes `docker pull …:c-<sha>` without a
|
||||||
# release ceremony.
|
# release ceremony.
|
||||||
# refs/heads/dev → push to dev: publish :dev, the rolling test
|
# anything else → safety net; shouldn't fire given the `on:`
|
||||||
# channel (family rule 146). Rolling means it may
|
# config above. Tag :dev to surface the
|
||||||
# carry newer contents than the :c-<sha> of the
|
# unexpected run in the registry.
|
||||||
# same commit; it never writes :c-<sha> itself,
|
|
||||||
# because that is the rollback unit (rule 145).
|
|
||||||
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
||||||
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
||||||
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
||||||
|
|||||||
+8
-125
@@ -2,7 +2,6 @@ name: CI
|
|||||||
|
|
||||||
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
|
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
|
||||||
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
|
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
|
||||||
# - extension-version: guards the extension publish path (see the job).
|
|
||||||
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
|
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
|
||||||
# - frontend-build: vitest unit + vite build.
|
# - frontend-build: vitest unit + vite build.
|
||||||
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
|
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
|
||||||
@@ -10,15 +9,10 @@ name: CI
|
|||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
branches: [dev, main]
|
branches: [dev, main]
|
||||||
# Renovate opens PRs from `renovate/*` branches into `dev`. Those branches
|
# pull_request trigger intentionally absent — with branches: [dev, main]
|
||||||
# never push to dev/main, so the push trigger above gives them NO pre-merge
|
# above, every PR commit already fires CI via the push event on dev. Adding
|
||||||
# CI — a bump could only be validated after it was already merged. This
|
# pull_request would duplicate runs on dev→main PRs. FC has no fork PRs
|
||||||
# pull_request trigger (base `dev` only) validates Renovate PRs before merge.
|
# (single-operator Forgejo repo) so push coverage is complete.
|
||||||
# It deliberately does NOT fire on dev→main PRs (base `main`), which still
|
|
||||||
# rely on the dev push run — so no duplicate runs. FC has no fork PRs
|
|
||||||
# (single-operator Forgejo repo), so secrets-on-PR is not a concern.
|
|
||||||
pull_request:
|
|
||||||
branches: [dev]
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
|
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
|
||||||
@@ -42,117 +36,6 @@ jobs:
|
|||||||
# catching syntax errors before the image build.
|
# catching syntax errors before the image build.
|
||||||
run: python -m compileall -q agent/fc_agent
|
run: python -m compileall -q agent/fc_agent
|
||||||
|
|
||||||
# Guards the extension publish path, which has no self-correcting behavior.
|
|
||||||
#
|
|
||||||
# build.yml's sign-extension job keys its AMO-signing cache purely on the
|
|
||||||
# version string in extension/package.json: if an `ext-<version>` Forgejo
|
|
||||||
# release already carries an XPI, signing is SKIPPED and that old signed XPI
|
|
||||||
# is what build-web bakes into `:latest`. Nothing in that path inspects
|
|
||||||
# whether extension/ actually changed — so a forgotten version bump ships a
|
|
||||||
# stale extension on a fully green build, silently. (AMO can't help: it 409s
|
|
||||||
# on re-signing a version, which is exactly why the cache exists.)
|
|
||||||
#
|
|
||||||
# This job makes that case loud, on the dev push, instead of invisible at
|
|
||||||
# merge-to-main. It is pure git + text work — no deps, no services.
|
|
||||||
extension-version:
|
|
||||||
runs-on: python-ci
|
|
||||||
container:
|
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
# Full history: the check diffs against the push's `before` SHA (or
|
|
||||||
# the PR base), which a depth-1 clone wouldn't contain.
|
|
||||||
fetch-depth: 0
|
|
||||||
- name: Extension version guard
|
|
||||||
env:
|
|
||||||
BEFORE: ${{ github.event.before }}
|
|
||||||
PR_BASE: ${{ github.event.pull_request.base.sha }}
|
|
||||||
run: |
|
|
||||||
set -eu
|
|
||||||
# busybox sh on the act_runner — no bashisms (family rule).
|
|
||||||
ver() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/'; }
|
|
||||||
PKG=$(ver extension/package.json)
|
|
||||||
MAN=$(ver extension/manifest.json)
|
|
||||||
test -n "$PKG" || { echo "ERROR: no version found in extension/package.json"; exit 1; }
|
|
||||||
test -n "$MAN" || { echo "ERROR: no version found in extension/manifest.json"; exit 1; }
|
|
||||||
|
|
||||||
# (1) Unconditional: the two version strings must agree. `web-ext sign`
|
|
||||||
# reads manifest.json (package.json sits in --ignore-files and isn't
|
|
||||||
# even inside the XPI), so AMO signs MAN and Firefox installs MAN.
|
|
||||||
# build.yml keys its cache, release tag, XPI filename — and therefore
|
|
||||||
# the version /api/extension/manifest reports to the update prompt —
|
|
||||||
# on PKG. Divergence either hard-fails at AMO or ships a mislabelled
|
|
||||||
# XPI whose update prompt lies about what's installed.
|
|
||||||
if [ "$MAN" != "$PKG" ]; then
|
|
||||||
echo "ERROR: extension version mismatch."
|
|
||||||
echo " extension/manifest.json = $MAN <- what AMO signs / Firefox installs"
|
|
||||||
echo " extension/package.json = $PKG <- what CI caches, names, and reports"
|
|
||||||
echo "Set both to the same value."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# (2) If the SHIPPED extension changed, the version must have moved.
|
|
||||||
#
|
|
||||||
# Compare against MAIN, not against the previous push. The publish
|
|
||||||
# decision is made at merge-to-main against whatever ext-<version>
|
|
||||||
# already exists, so "differs from main" is the question that matters.
|
|
||||||
# Diffing against the previous dev push instead would demand a fresh
|
|
||||||
# bump on every iteration — push, tweak the extension again, and CI
|
|
||||||
# would insist on a second bump that buys nothing, inflating the
|
|
||||||
# version for no reason. On a main push there is no "main to compare
|
|
||||||
# to" yet, so fall back to that push's own before-SHA.
|
|
||||||
if [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
||||||
BASE="${BEFORE:-}"
|
|
||||||
else
|
|
||||||
BASE=$(git rev-parse --verify -q origin/main 2>/dev/null || git rev-parse --verify -q main 2>/dev/null || echo "")
|
|
||||||
# PR base is the fallback when main isn't in the clone at all.
|
|
||||||
[ -n "$BASE" ] || BASE="${PR_BASE:-}"
|
|
||||||
fi
|
|
||||||
case "$BASE" in
|
|
||||||
''|0000000000000000000000000000000000000000)
|
|
||||||
echo "No usable base ref (no main in clone / first push) — skipping the bump check."
|
|
||||||
echo "OK: extension version $PKG"
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
if ! git cat-file -e "$BASE^{commit}" 2>/dev/null; then
|
|
||||||
echo "Base commit $BASE not in this clone — skipping the bump check."
|
|
||||||
echo "OK: extension version $PKG"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
# The exclusion list is NOT written out here — it comes from
|
|
||||||
# extension/scripts/packaging.sh, the one definition of what ships,
|
|
||||||
# shared with web-ext's --ignore-files and the derived-version patch
|
|
||||||
# count. Three hand-kept copies of that fact is how #2397 happened.
|
|
||||||
#
|
|
||||||
# `set -f` is required around the substitution: without it the shell
|
|
||||||
# globs `test/**` against the working tree and silently narrows it.
|
|
||||||
set -f
|
|
||||||
CHANGED=$(git diff --name-only "$BASE" HEAD -- extension/ $(sh extension/scripts/packaging.sh pathspec))
|
|
||||||
set +f
|
|
||||||
if [ -z "$CHANGED" ]; then
|
|
||||||
echo "No packaged extension files changed since $BASE — nothing to guard."
|
|
||||||
echo "OK: extension version $PKG"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
echo "Packaged extension files changed since $BASE:"
|
|
||||||
echo "$CHANGED" | sed 's/^/ /'
|
|
||||||
PKG_OLD=$(git show "$BASE:extension/package.json" 2>/dev/null | grep -E '"version"' | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
|
||||||
if [ -z "$PKG_OLD" ]; then
|
|
||||||
echo "Could not read the base version — skipping the bump check."
|
|
||||||
echo "OK: extension version $PKG"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
if [ "$PKG_OLD" = "$PKG" ]; then
|
|
||||||
echo "ERROR: packaged extension files changed but the version is still $PKG."
|
|
||||||
echo "build.yml would find the existing ext-$PKG release, skip AMO signing,"
|
|
||||||
echo "and bake the OLD signed XPI into :latest — a green build shipping stale code."
|
|
||||||
echo "Bump the version in BOTH extension/package.json and extension/manifest.json."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "OK: extension version $PKG_OLD -> $PKG"
|
|
||||||
|
|
||||||
backend-lint-and-test:
|
backend-lint-and-test:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
@@ -209,10 +92,10 @@ jobs:
|
|||||||
# If we want strict lockfile-based reproducibility later, commit a
|
# If we want strict lockfile-based reproducibility later, commit a
|
||||||
# package-lock.json and flip this back to `npm ci`.
|
# package-lock.json and flip this back to `npm ci`.
|
||||||
- run: npm install --no-audit --no-fund
|
- run: npm install --no-audit --no-fund
|
||||||
# No type-check step: the frontend is pure JS (no .ts files, no JSDoc),
|
# `npm run check` (vue-tsc --noEmit) skipped: the frontend is pure JS
|
||||||
# so a type-checker has nothing to do. The vue-tsc devDep + its `check`
|
# with no .ts files and no JSDoc annotations, so vue-tsc has nothing
|
||||||
# script were dropped 2026-07-11 rather than bumped to v3. If we add
|
# to type-check. Re-enable once we add a tsconfig.json and either
|
||||||
# TS/JSDoc later, re-add a tsconfig.json + vue-tsc + a type-check step.
|
# convert to TS or add JSDoc.
|
||||||
- run: npm run test:unit
|
- run: npm run test:unit
|
||||||
- run: npm run build
|
- run: npm run build
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
name: extension
|
name: extension
|
||||||
# Lint + unit tests. The sign-and-publish dance moved into build.yml's
|
# Lint-only workflow. The sign-and-publish dance moved into build.yml's
|
||||||
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
|
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
|
||||||
# because sign-extension runs as a build-web dependency in the SAME workflow,
|
# because sign-extension runs as a build-web dependency in the SAME workflow,
|
||||||
# eliminating the prior race between build.yml and a separate extension.yml.
|
# eliminating the prior race between build.yml and a separate extension.yml.
|
||||||
@@ -10,73 +10,20 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
- 'extension/**'
|
- 'extension/**'
|
||||||
- '.forgejo/workflows/extension.yml'
|
- '.forgejo/workflows/extension.yml'
|
||||||
# test/version.spec.js asserts ci.yml's extension-version guard never
|
|
||||||
# ignores a file web-ext actually packages, so a ci.yml-only edit can
|
|
||||||
# break this suite and must trigger it.
|
|
||||||
- '.forgejo/workflows/ci.yml'
|
|
||||||
pull_request:
|
pull_request:
|
||||||
branches: [main]
|
branches: [main]
|
||||||
paths:
|
paths:
|
||||||
- 'extension/**'
|
- 'extension/**'
|
||||||
- '.forgejo/workflows/ci.yml'
|
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
lint:
|
lint:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: node:24-bookworm-slim
|
image: node:22-bookworm-slim
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
# Not --no-save: vitest and web-ext are both real devDependencies now,
|
- name: Install web-ext
|
||||||
# and the suite needs vitest resolvable from node_modules.
|
run: cd extension && npm install --no-save --no-audit --no-fund
|
||||||
- name: Install dev dependencies
|
|
||||||
run: cd extension && npm install --no-audit --no-fund
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
run: cd extension && npm run lint
|
run: cd extension && npm run lint
|
||||||
# Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
|
|
||||||
# package version-consistency checks. No browser, no network.
|
|
||||||
- name: Unit tests
|
|
||||||
run: cd extension && npm run test:unit
|
|
||||||
|
|
||||||
# Everything else about packaging is asserted against our own declaration
|
|
||||||
# of what ships. This is the only check that asks web-ext what it ACTUALLY
|
|
||||||
# put in the archive. Until now that was an unverified assumption about
|
|
||||||
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
|
|
||||||
# only because callers `set -f` first, so losing that quoting would
|
|
||||||
# silently start shipping dev files with no other signal.
|
|
||||||
- name: Verify XPI contents
|
|
||||||
run: |
|
|
||||||
set -eu
|
|
||||||
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
|
|
||||||
cd extension
|
|
||||||
npm run build
|
|
||||||
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
|
|
||||||
echo "=== packaged entries in $ZIP ==="
|
|
||||||
unzip -Z1 "$ZIP" | sort
|
|
||||||
echo "=== end ==="
|
|
||||||
ENTRIES=$(unzip -Z1 "$ZIP")
|
|
||||||
fail=0
|
|
||||||
# Must NOT ship: repo infrastructure with no business in a user's browser.
|
|
||||||
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
|
|
||||||
if echo "$ENTRIES" | grep -q "^$pat"; then
|
|
||||||
echo "ERROR: '$pat' was packaged into the XPI but must not be"
|
|
||||||
fail=1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
# Must ship: if an exclusion pattern ever over-matches, the extension
|
|
||||||
# breaks at runtime rather than at build time, so assert presence too.
|
|
||||||
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
|
|
||||||
if ! echo "$ENTRIES" | grep -q "^$req$"; then
|
|
||||||
echo "ERROR: '$req' is missing from the XPI"
|
|
||||||
fail=1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
|
|
||||||
if ! echo "$ENTRIES" | grep -q "^$dir"; then
|
|
||||||
echo "ERROR: nothing from '$dir' was packaged"
|
|
||||||
fail=1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
[ "$fail" -eq 0 ] || exit 1
|
|
||||||
echo "XPI contents verified."
|
|
||||||
|
|||||||
+2
-2
@@ -1,6 +1,6 @@
|
|||||||
# syntax=docker/dockerfile:1.25
|
# syntax=docker/dockerfile:1.7
|
||||||
|
|
||||||
FROM node:24-alpine AS frontend-builder
|
FROM node:22-alpine AS frontend-builder
|
||||||
WORKDIR /build
|
WORKDIR /build
|
||||||
COPY frontend/package.json frontend/package-lock.json* ./
|
COPY frontend/package.json frontend/package-lock.json* ./
|
||||||
# No package-lock.json is tracked yet (we don't run npm locally per
|
# No package-lock.json is tracked yet (we don't run npm locally per
|
||||||
|
|||||||
+1
-1
@@ -1,4 +1,4 @@
|
|||||||
# syntax=docker/dockerfile:1.25
|
# syntax=docker/dockerfile:1.7
|
||||||
|
|
||||||
FROM python:3.14-slim
|
FROM python:3.14-slim
|
||||||
ENV PYTHONUNBUFFERED=1 \
|
ENV PYTHONUNBUFFERED=1 \
|
||||||
|
|||||||
@@ -6,21 +6,7 @@ Combines what was [ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo)
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
In production. `main` is continuously deployed — every merge to `main` builds
|
Pre-v1. Not yet functional.
|
||||||
and publishes `:latest` images, so whatever is on `main` is what is running.
|
|
||||||
Day-to-day work happens on `dev`, which publishes `:dev` images.
|
|
||||||
|
|
||||||
## What's in here
|
|
||||||
|
|
||||||
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
|
|
||||||
|
|
||||||
| Piece | Built from | Image | Role |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
|
|
||||||
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
|
|
||||||
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. |
|
|
||||||
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on `main` only, then bundled into the web image and served from Settings → Maintenance. See `extension/README.md`. |
|
|
||||||
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
|
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -43,37 +29,22 @@ docker compose -f docker-compose.yml up -d
|
|||||||
# (skips the override so containers pull registry images)
|
# (skips the override so containers pull registry images)
|
||||||
```
|
```
|
||||||
|
|
||||||
The GPU agent is deployed separately, on the machine with the card —
|
|
||||||
`agent/docker-compose.yml`, not this stack.
|
|
||||||
|
|
||||||
## Deployment posture
|
## Deployment posture
|
||||||
|
|
||||||
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
|
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
|
||||||
|
|
||||||
## CI / Forgejo setup
|
## CI / Forgejo setup
|
||||||
|
|
||||||
Three workflows: `ci.yml` (lint, extension-version guard, backend unit tests,
|
The repo's workflows expect:
|
||||||
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
|
|
||||||
content verification), and `build.yml` (sign + publish).
|
|
||||||
|
|
||||||
**The toolchain each job runs in is its `container.image`, not its `runs-on`
|
- **Runner label `python-ci`** — a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Both `ci.yml` and `build.yml` use this label. The runner image (`runner-base:python-ci`) is built from `CI-Runner/CI-python/` in the operator's workspace; `make push` from that directory builds and pushes a new image when toolchain pins change.
|
||||||
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
|
- **Repo secret `RELEASE_TOKEN`** — a Forgejo PAT with the following scopes:
|
||||||
then names the image it actually wants. `ci-requirements.md` is the current,
|
|
||||||
authoritative list of images and per-job installs — read that rather than a
|
|
||||||
copy here, so the two can't drift.
|
|
||||||
|
|
||||||
The repo expects one secret:
|
|
||||||
|
|
||||||
- **`RELEASE_TOKEN`** — a Forgejo PAT with:
|
|
||||||
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
|
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
|
||||||
- `write:release` — for the `ext-<version>` releases that cache the signed XPI
|
- `write:release` — for future release-cutting workflows
|
||||||
- `write:issue` — for issue-management automation
|
- `write:issue` — for future issue-management automation
|
||||||
|
|
||||||
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
|
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
|
||||||
|
|
||||||
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on
|
|
||||||
`main` only and is cached per version, since AMO rejects a re-signed version.
|
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
Personal project; use at your own discretion.
|
Personal project; use at your own discretion.
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
|
|||||||
# Bump on every agent change. The page embeds this and /status reports it; the UI
|
# Bump on every agent change. The page embeds this and /status reports it; the UI
|
||||||
# warns to reload when they differ — so a stale browser-cached page can't be
|
# warns to reload when they differ — so a stale browser-cached page can't be
|
||||||
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
|
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
|
||||||
VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
|
VERSION = "2026-07-02.6 · sleep mode: an empty queue sheds to one downloader and backs the lease poll off to 15 min"
|
||||||
|
|
||||||
logbuf.install()
|
logbuf.install()
|
||||||
cfg = Config.from_env()
|
cfg = Config.from_env()
|
||||||
@@ -334,12 +334,9 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
|||||||
waited.textContent=s.transient||0
|
waited.textContent=s.transient||0
|
||||||
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
|
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
|
||||||
// as live churn rather than a "broken" headline metric.
|
// as live churn rather than a "broken" headline metric.
|
||||||
// '=== false' (not falsy) so a stale page that doesn't send models_loaded shows
|
|
||||||
// nothing; when the idle monitor unloads, the VRAM meter drops alongside this.
|
|
||||||
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'—')+' · consumers '+(s.consumers!=null?s.consumers:'—')+' · on GPU '+(s.active||0)
|
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'—')+' · consumers '+(s.consumers!=null?s.consumers:'—')+' · on GPU '+(s.active||0)
|
||||||
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'—')+' MB/s'
|
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'—')+' MB/s'
|
||||||
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
|
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
|
||||||
+(s.models_loaded===false?' · GPU models unloaded (idle — reload on next job)':'')
|
|
||||||
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
|
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
|
||||||
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
|
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
|
||||||
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
|
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
|
||||||
|
|||||||
@@ -51,12 +51,6 @@ class Config:
|
|||||||
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
|
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
|
||||||
# all downloaders + video streams (0 = unlimited);
|
# all downloaders + video streams (0 = unlimited);
|
||||||
# tunable live from the agent UI
|
# tunable live from the agent UI
|
||||||
idle_unload_seconds: float # after this long with the GPU idle (nothing in
|
|
||||||
# flight, queue empty or Stopped), unload the
|
|
||||||
# SigLIP embedder + YOLO proposers to free their
|
|
||||||
# VRAM; they reload lazily on the next job. A
|
|
||||||
# 24/7 agent otherwise squats on ~5GB doing
|
|
||||||
# nothing. 0 disables (keep models warm forever).
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_env(cls) -> Config:
|
def from_env(cls) -> Config:
|
||||||
@@ -93,8 +87,4 @@ class Config:
|
|||||||
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
|
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
|
||||||
# from the agent UI on wired/faster networks.
|
# from the agent UI on wired/faster networks.
|
||||||
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
|
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
|
||||||
# 5 min: long enough that a lull between job bursts doesn't thrash the
|
|
||||||
# (few-second) reload, short enough that an agent left running with an
|
|
||||||
# empty queue hands its VRAM back promptly.
|
|
||||||
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
|
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -170,13 +170,6 @@ class YoloProposer:
|
|||||||
))
|
))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def unload(self) -> None:
|
|
||||||
"""Drop the loaded YOLO so its VRAM can be reclaimed; detect() reloads it
|
|
||||||
lazily on the next job. Leaves _ok untouched — a healthy proposer comes
|
|
||||||
back, but one that self-disabled on a fault stays off."""
|
|
||||||
with self._lock:
|
|
||||||
self._model = None
|
|
||||||
|
|
||||||
|
|
||||||
class Proposers:
|
class Proposers:
|
||||||
"""The agent's proposer set, built from config. Each detector is optional —
|
"""The agent's proposer set, built from config. Each detector is optional —
|
||||||
@@ -223,11 +216,3 @@ class Proposers:
|
|||||||
|
|
||||||
def panels(self, image):
|
def panels(self, image):
|
||||||
return self._top(self._panel, image, self.cfg.max_panels)
|
return self._top(self._panel, image, self.cfg.max_panels)
|
||||||
|
|
||||||
def unload(self) -> None:
|
|
||||||
"""Release every loaded proposer's YOLO (idle VRAM reclaim). The worker
|
|
||||||
also drops its reference to this Proposers and rebuilds a fresh one via
|
|
||||||
_proposers_for on the next job, so this is belt-and-braces."""
|
|
||||||
for p in (self._person, self._anatomy, self._panel):
|
|
||||||
if p is not None:
|
|
||||||
p.unload()
|
|
||||||
|
|||||||
@@ -75,18 +75,3 @@ class CropEmbedder:
|
|||||||
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
|
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
|
||||||
arr = pooled.float().cpu().numpy().astype(np.float32)
|
arr = pooled.float().cpu().numpy().astype(np.float32)
|
||||||
return [row.reshape(-1).tolist() for row in arr]
|
return [row.reshape(-1).tolist() for row in arr]
|
||||||
|
|
||||||
def unload(self) -> bool:
|
|
||||||
"""Drop the loaded model so its VRAM can be reclaimed — the idle monitor
|
|
||||||
calls this after a spell with no work so an idle agent doesn't squat on
|
|
||||||
the card; the next embed() reloads it lazily (a few seconds). Held under
|
|
||||||
BOTH the load and inference locks so it can never race a concurrent load
|
|
||||||
or an in-flight forward pass. Returns True if a model was actually
|
|
||||||
released (the caller then runs one empty_cache() to hand the freed blocks
|
|
||||||
back to the driver)."""
|
|
||||||
with self._load_lock, self._infer_lock:
|
|
||||||
if self._model is None:
|
|
||||||
return False
|
|
||||||
self._model = None
|
|
||||||
self._processor = None
|
|
||||||
return True
|
|
||||||
|
|||||||
@@ -57,15 +57,6 @@ MAX_BACKOFF_SECONDS = 60.0
|
|||||||
# up on their own.
|
# up on their own.
|
||||||
IDLE_POLL_MAX_SECONDS = 900.0
|
IDLE_POLL_MAX_SECONDS = 900.0
|
||||||
|
|
||||||
# Idle VRAM reclaim (operator 2026-07-17): the SigLIP embedder + YOLO proposers
|
|
||||||
# load lazily and then stay warm for fast job bursts — but a 24/7 agent with an
|
|
||||||
# empty queue would otherwise squat on that VRAM (~5GB on the operator's card)
|
|
||||||
# indefinitely while doing nothing. So a monitor unloads them after
|
|
||||||
# cfg.idle_unload_seconds with the GPU genuinely idle (nothing in flight, buffer
|
|
||||||
# drained); they reload lazily on the next job. This is just how often the
|
|
||||||
# monitor wakes to check — it bounds how soon past the threshold the unload fires.
|
|
||||||
IDLE_UNLOAD_CHECK_INTERVAL = 30.0
|
|
||||||
|
|
||||||
# A job whose fetch dies transiently this many times IN ONE SESSION stops being
|
# A job whose fetch dies transiently this many times IN ONE SESSION stops being
|
||||||
# handed back and is failed instead. Transient handbacks (release) burn no
|
# handed back and is failed instead. Transient handbacks (release) burn no
|
||||||
# attempts on the server, so a poisoned transfer — an original that stalls the
|
# attempts on the server, so a poisoned transfer — an original that stalls the
|
||||||
@@ -277,11 +268,6 @@ class Worker:
|
|||||||
self._proposers_sig = None # detector-config signature the current
|
self._proposers_sig = None # detector-config signature the current
|
||||||
# proposers were built for (#134)
|
# proposers were built for (#134)
|
||||||
self._proposers_lock = threading.Lock()
|
self._proposers_lock = threading.Lock()
|
||||||
# Monotonic time of the last GPU activity (a consumer finishing a job).
|
|
||||||
# The idle monitor unloads the warm models once this goes stale by
|
|
||||||
# cfg.idle_unload_seconds — see _idle_unload_loop.
|
|
||||||
self._last_gpu_activity = time.monotonic()
|
|
||||||
threading.Thread(target=self._idle_unload_loop, daemon=True).start()
|
|
||||||
|
|
||||||
# --- held-lease bookkeeping --------------------------------------------
|
# --- held-lease bookkeeping --------------------------------------------
|
||||||
def _hold(self, job_ids) -> None:
|
def _hold(self, job_ids) -> None:
|
||||||
@@ -622,9 +608,6 @@ class Worker:
|
|||||||
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
|
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
|
||||||
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
|
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
|
||||||
"idle": self._idle, # queue empty → poll backed off (UI hint)
|
"idle": self._idle, # queue empty → poll backed off (UI hint)
|
||||||
# Whether the GPU models are currently resident (False after an idle
|
|
||||||
# unload freed their VRAM) — a plain bool read, UI hint only.
|
|
||||||
"models_loaded": self._embedder is not None or self._proposers is not None,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
|
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
|
||||||
@@ -805,9 +788,6 @@ class Worker:
|
|||||||
self._bump(processed=1)
|
self._bump(processed=1)
|
||||||
finally:
|
finally:
|
||||||
self._bump(active=-1)
|
self._bump(active=-1)
|
||||||
# Mark the GPU busy-until-now so the idle monitor starts its
|
|
||||||
# unload countdown from when work actually stopped, not before.
|
|
||||||
self._last_gpu_activity = time.monotonic()
|
|
||||||
|
|
||||||
def _ensure_embedder(self, model_name: str):
|
def _ensure_embedder(self, model_name: str):
|
||||||
if self._embedder is not None:
|
if self._embedder is not None:
|
||||||
@@ -865,61 +845,6 @@ class Worker:
|
|||||||
self._proposers_sig = sig
|
self._proposers_sig = sig
|
||||||
return self._proposers
|
return self._proposers
|
||||||
|
|
||||||
def _unload_models(self) -> bool:
|
|
||||||
"""Release the GPU-resident models (SigLIP embedder + YOLO proposers) so an
|
|
||||||
idle agent hands their VRAM back instead of squatting on the card. They
|
|
||||||
reload lazily on the next job (_ensure_embedder / _proposers_for) — a
|
|
||||||
few seconds' cost paid only when work actually resumes. Dropping the
|
|
||||||
shared instances under their build locks means a concurrent job either
|
|
||||||
sees the old instance (before) or rebuilds a fresh one (after); the idle
|
|
||||||
monitor only calls this with nothing in flight, so no inference is using
|
|
||||||
them. Returns True if anything was released."""
|
|
||||||
released = False
|
|
||||||
with self._embedder_lock:
|
|
||||||
if self._embedder is not None:
|
|
||||||
self._embedder.unload()
|
|
||||||
self._embedder = None
|
|
||||||
released = True
|
|
||||||
with self._proposers_lock:
|
|
||||||
if self._proposers is not None:
|
|
||||||
self._proposers.unload()
|
|
||||||
self._proposers = None
|
|
||||||
self._proposers_sig = None
|
|
||||||
released = True
|
|
||||||
if released:
|
|
||||||
try:
|
|
||||||
import torch
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
# torch's caching allocator holds freed blocks; hand them back
|
|
||||||
# to the driver so nvidia-smi actually reflects the drop.
|
|
||||||
torch.cuda.empty_cache()
|
|
||||||
except Exception: # noqa: BLE001 — torch absent / CPU-only → nothing to free
|
|
||||||
pass
|
|
||||||
return released
|
|
||||||
|
|
||||||
def _idle_unload_loop(self) -> None:
|
|
||||||
"""Unload the warm GPU models after a stretch of inactivity so a 24/7
|
|
||||||
agent with an empty queue doesn't hold ~5GB of VRAM doing nothing. Fires
|
|
||||||
only when nothing is in flight (active == 0 AND the buffer is drained) and
|
|
||||||
no job has completed for cfg.idle_unload_seconds — a window long enough
|
|
||||||
that a brief lull between bursts doesn't thrash reload/unload. Covers BOTH
|
|
||||||
sleep mode (queue empty, pipeline still running) and a full Stop; the
|
|
||||||
models reload lazily on the next job. idle_unload_seconds <= 0 disables it."""
|
|
||||||
idle_after = self.cfg.idle_unload_seconds
|
|
||||||
if idle_after <= 0:
|
|
||||||
return
|
|
||||||
while True:
|
|
||||||
time.sleep(IDLE_UNLOAD_CHECK_INTERVAL)
|
|
||||||
if self._embedder is None and self._proposers is None:
|
|
||||||
continue # nothing loaded → nothing to free
|
|
||||||
if self._active != 0 or not self._buffer.empty():
|
|
||||||
continue # work in flight → keep them warm
|
|
||||||
if time.monotonic() - self._last_gpu_activity < idle_after:
|
|
||||||
continue # not idle long enough yet
|
|
||||||
if self._unload_models():
|
|
||||||
log.info("idle %.0fs — unloaded GPU models, freed VRAM "
|
|
||||||
"(reload on next job)", idle_after)
|
|
||||||
|
|
||||||
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
|
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
|
||||||
"""Detect + embed the decoded frames and submit the result. Returns True
|
"""Detect + embed the decoded frames and submit the result. Returns True
|
||||||
when the job was completed (→ count it processed), False otherwise: a
|
when the job was completed (→ count it processed), False otherwise: a
|
||||||
|
|||||||
@@ -1,51 +0,0 @@
|
|||||||
"""translation strictness setting + per-post translation override (milestone 155)
|
|
||||||
|
|
||||||
ImportSettings gains ``translation_min_confidence`` (the latin-script acceptance
|
|
||||||
floor, now operator-tunable in the UI; default 0.9 — stricter than the old
|
|
||||||
hardcoded 0.8, since Interpreter confidently mis-detects short ASCII English at
|
|
||||||
~0.86). Post gains ``translation_override`` — a sticky per-post choice of
|
|
||||||
auto / force / original so the operator can force a skipped translation on, or
|
|
||||||
knock a wrongly-translated one back to the original, and have it survive a
|
|
||||||
Re-translate-all.
|
|
||||||
|
|
||||||
Revision ID: 0084
|
|
||||||
Revises: 0083
|
|
||||||
Create Date: 2026-07-10
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0084"
|
|
||||||
down_revision: Union[str, None] = "0083"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"import_settings",
|
|
||||||
sa.Column(
|
|
||||||
"translation_min_confidence", sa.Float(), nullable=False,
|
|
||||||
server_default=sa.text("0.9"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"post",
|
|
||||||
sa.Column(
|
|
||||||
"translation_override", sa.String(16), nullable=False,
|
|
||||||
server_default="auto",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.create_check_constraint(
|
|
||||||
"ck_post_translation_override",
|
|
||||||
"post",
|
|
||||||
"translation_override IN ('auto', 'force', 'original')",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_constraint("ck_post_translation_override", "post", type_="check")
|
|
||||||
op.drop_column("post", "translation_override")
|
|
||||||
op.drop_column("import_settings", "translation_min_confidence")
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
"""title-based WIP auto-tagging (task #1458) — ImportSettings toggle
|
|
||||||
|
|
||||||
ImportSettings gains wip_title_tagging_enabled (ON by default): when a freshly
|
|
||||||
imported post's title explicitly declares work-in-progress ("WIP" / "work in
|
|
||||||
progress"), the importer applies the `wip` system tag to its images. No new
|
|
||||||
table — the tag itself is the seeded `wip` system tag (migration 0075) and the
|
|
||||||
application reuses image_tag with source='wip_title'.
|
|
||||||
|
|
||||||
Revision ID: 0085
|
|
||||||
Revises: 0084
|
|
||||||
Create Date: 2026-07-12
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0085"
|
|
||||||
down_revision: Union[str, None] = "0084"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"import_settings",
|
|
||||||
sa.Column(
|
|
||||||
"wip_title_tagging_enabled", sa.Boolean(), nullable=False,
|
|
||||||
server_default=sa.text("true"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("import_settings", "wip_title_tagging_enabled")
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
"""process auto-apply settings + review mode (#1464) — system-tag refactor
|
|
||||||
|
|
||||||
The system-tag behavior refactor gives `wip` / `editor screenshot` (the PROCESS
|
|
||||||
group) their own provisional auto-apply, parallel to the presentation (chrome)
|
|
||||||
sweep. MLSettings gains three knobs: enabled (OFF by default — a new whole-library
|
|
||||||
auto-tagger is opt-in), the flat apply threshold, and the ring-loud conflict
|
|
||||||
threshold. presentation_review gains a `mode` column so one review surface serves
|
|
||||||
both chrome and process flags (existing rows backfill 'chrome'). server_defaults
|
|
||||||
so the existing rows fill cleanly.
|
|
||||||
|
|
||||||
Revision ID: 0086
|
|
||||||
Revises: 0085
|
|
||||||
Create Date: 2026-07-13
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0086"
|
|
||||||
down_revision: Union[str, None] = "0085"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"ml_settings",
|
|
||||||
sa.Column(
|
|
||||||
"process_auto_apply_enabled", sa.Boolean(), nullable=False,
|
|
||||||
server_default=sa.text("false"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"ml_settings",
|
|
||||||
sa.Column(
|
|
||||||
"process_auto_apply_threshold", sa.Float(), nullable=False,
|
|
||||||
server_default="0.90",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"ml_settings",
|
|
||||||
sa.Column(
|
|
||||||
"process_conflict_threshold", sa.Float(), nullable=False,
|
|
||||||
server_default="0.50",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"presentation_review",
|
|
||||||
sa.Column(
|
|
||||||
"mode", sa.String(16), nullable=False,
|
|
||||||
server_default="chrome",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("presentation_review", "mode")
|
|
||||||
op.drop_column("ml_settings", "process_conflict_threshold")
|
|
||||||
op.drop_column("ml_settings", "process_auto_apply_threshold")
|
|
||||||
op.drop_column("ml_settings", "process_auto_apply_enabled")
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
"""soft WIP title tier toggle (#1474) — ImportSettings.wip_soft_title_tagging_enabled
|
|
||||||
|
|
||||||
The soft tier also tags sketch/doodle/scribble titles, but with a provisional source
|
|
||||||
that never trains the head. OFF by default (a lower-precision tier is opt-in).
|
|
||||||
server_default so the existing singleton row (id=1) fills cleanly.
|
|
||||||
|
|
||||||
Revision ID: 0087
|
|
||||||
Revises: 0086
|
|
||||||
Create Date: 2026-07-13
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0087"
|
|
||||||
down_revision: Union[str, None] = "0086"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"import_settings",
|
|
||||||
sa.Column(
|
|
||||||
"wip_soft_title_tagging_enabled", sa.Boolean(), nullable=False,
|
|
||||||
server_default=sa.text("false"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("import_settings", "wip_soft_title_tagging_enabled")
|
|
||||||
@@ -459,22 +459,6 @@ async def trigger_prune_missing_files():
|
|||||||
return _queued(async_result)
|
return _queued(async_result)
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/reclaim-attachments", methods=["POST"])
|
|
||||||
async def trigger_reclaim_attachments():
|
|
||||||
"""Reclaim orphaned attachments (#3068). Body {"dry_run": bool}: dry_run
|
|
||||||
(the DEFAULT here) projects the orphan rows and unreferenced store blobs
|
|
||||||
without touching either; dry_run=false deletes the rows then unlinks every
|
|
||||||
blob no surviving row references. Maintenance queue; operator-triggered
|
|
||||||
only — never an unattended sweep, since the apply unlinks files. Returns the
|
|
||||||
Celery task id — poll /maintenance/task-result/<id> for the summary."""
|
|
||||||
from ..tasks.admin import reclaim_orphaned_attachments_task
|
|
||||||
|
|
||||||
body = await request.get_json(silent=True) or {}
|
|
||||||
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
|
|
||||||
async_result = reclaim_orphaned_attachments_task.delay(dry_run=dry_run)
|
|
||||||
return _queued(async_result)
|
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
||||||
async def trigger_dedup_videos():
|
async def trigger_dedup_videos():
|
||||||
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import hashlib
|
import hashlib
|
||||||
import hmac
|
|
||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -42,15 +41,7 @@ async def _ext_key_required(session) -> bool:
|
|||||||
stored = (await session.execute(
|
stored = (await session.execute(
|
||||||
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
|
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
|
||||||
)).scalar_one_or_none()
|
)).scalar_one_or_none()
|
||||||
if stored is None:
|
return stored is not None and supplied == stored
|
||||||
return False
|
|
||||||
# compare_digest, not `==`: the stored key is a shared secret, and a
|
|
||||||
# short-circuiting compare leaks its prefix through timing. Costs nothing
|
|
||||||
# here — it is not that this route is exposed (#3072). Compared as BYTES:
|
|
||||||
# compare_digest's str form rejects non-ASCII with TypeError, and this
|
|
||||||
# header is attacker-supplied, so a str compare would turn a junk key into
|
|
||||||
# a 500 instead of a 403.
|
|
||||||
return hmac.compare_digest(supplied.encode("utf-8"), stored.encode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_version(xpi_name: str) -> str:
|
def _extract_version(xpi_name: str) -> str:
|
||||||
|
|||||||
@@ -148,17 +148,6 @@ async def similar():
|
|||||||
# Explore passes exclude_wip=1 to also drop work-in-progress from the
|
# Explore passes exclude_wip=1 to also drop work-in-progress from the
|
||||||
# rabbit-hole; the gallery's own "similar" button omits it (keeps wip, #1274).
|
# rabbit-hole; the gallery's own "similar" button omits it (keeps wip, #1274).
|
||||||
exclude_wip = request.args.get("exclude_wip") in ("1", "true", "True")
|
exclude_wip = request.args.get("exclude_wip") in ("1", "true", "True")
|
||||||
# Explore reach (#1476): 0 = nearest (gallery default), →1 reaches into farther
|
|
||||||
# distance bands so the walk can escape a dense cluster. exclude_ids = the
|
|
||||||
# breadcrumb, so already-walked images aren't re-served as neighbours.
|
|
||||||
try:
|
|
||||||
reach = max(0.0, min(1.0, float(request.args.get("reach", "0"))))
|
|
||||||
except ValueError:
|
|
||||||
reach = 0.0
|
|
||||||
exclude_ids = [
|
|
||||||
int(x) for x in request.args.get("exclude_ids", "").split(",")
|
|
||||||
if x.strip().isdigit()
|
|
||||||
] or None
|
|
||||||
# post_id is the exclusive post-detail view — not a similarity scope.
|
# post_id is the exclusive post-detail view — not a similarity scope.
|
||||||
# include_hidden is a gallery-browse flag; similar() has its OWN presentation
|
# include_hidden is a gallery-browse flag; similar() has its OWN presentation
|
||||||
# exclusion (a similarity-quality concern, #1274), so drop it here (#141).
|
# exclusion (a similarity-quality concern, #1274), so drop it here (#141).
|
||||||
@@ -169,8 +158,7 @@ async def similar():
|
|||||||
svc = GalleryService(session)
|
svc = GalleryService(session)
|
||||||
try:
|
try:
|
||||||
images = await svc.similar(
|
images = await svc.similar(
|
||||||
image_id=similar_to, limit=limit, exclude_wip=exclude_wip,
|
image_id=similar_to, limit=limit, exclude_wip=exclude_wip, **scope)
|
||||||
reach=reach, exclude_ids=exclude_ids, **scope)
|
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
return jsonify({"error": str(exc)}), 400
|
return jsonify({"error": str(exc)}), 400
|
||||||
if images is None:
|
if images is None:
|
||||||
@@ -248,10 +236,8 @@ async def jump():
|
|||||||
# content", surfaced in the gallery's Show-hidden review strip. -----------
|
# content", surfaced in the gallery's Show-hidden review strip. -----------
|
||||||
@gallery_bp.route("/hidden-review", methods=["GET"])
|
@gallery_bp.route("/hidden-review", methods=["GET"])
|
||||||
async def hidden_review():
|
async def hidden_review():
|
||||||
"""Unresolved system-tag auto-apply review flags (chrome + process, #1464),
|
"""Unresolved presentation auto-hide flags, most-concerning first (highest
|
||||||
most-concerning first (highest content score) — for the review strip. `mode`
|
content score) — for the gallery's Hidden-view review strip."""
|
||||||
tells the client whether the flagged tag hid the image ('chrome') or left it
|
|
||||||
visible ('process'), which decides the resolve labels (un-hide vs remove-tag)."""
|
|
||||||
ptag = aliased(Tag)
|
ptag = aliased(Tag)
|
||||||
ctag = aliased(Tag)
|
ctag = aliased(Tag)
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
@@ -261,7 +247,6 @@ async def hidden_review():
|
|||||||
PresentationReview.tag_id,
|
PresentationReview.tag_id,
|
||||||
PresentationReview.conflict_tag_id,
|
PresentationReview.conflict_tag_id,
|
||||||
PresentationReview.conflict_score,
|
PresentationReview.conflict_score,
|
||||||
PresentationReview.mode,
|
|
||||||
ImageRecord.path, ImageRecord.thumbnail_path,
|
ImageRecord.path, ImageRecord.thumbnail_path,
|
||||||
ImageRecord.sha256, ImageRecord.mime,
|
ImageRecord.sha256, ImageRecord.mime,
|
||||||
ptag.name.label("tag_name"),
|
ptag.name.label("tag_name"),
|
||||||
@@ -281,7 +266,6 @@ async def hidden_review():
|
|||||||
"conflict_tag_id": r.conflict_tag_id,
|
"conflict_tag_id": r.conflict_tag_id,
|
||||||
"conflict_name": r.conflict_name,
|
"conflict_name": r.conflict_name,
|
||||||
"conflict_score": r.conflict_score,
|
"conflict_score": r.conflict_score,
|
||||||
"mode": r.mode,
|
|
||||||
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
||||||
"image_url": image_url(r.path),
|
"image_url": image_url(r.path),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -256,7 +256,9 @@ async def lease():
|
|||||||
if not await _agent_authed(session):
|
if not await _agent_authed(session):
|
||||||
return jsonify({"error": "unauthorized"}), 401
|
return jsonify({"error": "unauthorized"}), 401
|
||||||
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
|
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
|
||||||
ml = await MLSettings.load(session)
|
ml = (
|
||||||
|
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
||||||
|
).scalar_one()
|
||||||
# image rows for url/mime in one shot
|
# image rows for url/mime in one shot
|
||||||
ids = [j.image_record_id for j in jobs]
|
ids = [j.image_record_id for j in jobs]
|
||||||
imgs = {
|
imgs = {
|
||||||
|
|||||||
+38
-24
@@ -4,7 +4,6 @@ from quart import Blueprint, jsonify, request
|
|||||||
|
|
||||||
from ..extensions import get_session
|
from ..extensions import get_session
|
||||||
from ..models import MLSettings
|
from ..models import MLSettings
|
||||||
from ..services.ml.heads import AUTO_APPLY_THRESHOLD_MAX, AUTO_APPLY_THRESHOLD_MIN
|
|
||||||
|
|
||||||
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
|
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
|
||||||
|
|
||||||
@@ -43,9 +42,6 @@ _EDITABLE = (
|
|||||||
"presentation_auto_apply_enabled",
|
"presentation_auto_apply_enabled",
|
||||||
"presentation_auto_apply_threshold",
|
"presentation_auto_apply_threshold",
|
||||||
"presentation_conflict_threshold",
|
"presentation_conflict_threshold",
|
||||||
"process_auto_apply_enabled",
|
|
||||||
"process_auto_apply_threshold",
|
|
||||||
"process_conflict_threshold",
|
|
||||||
"embedder_model_name",
|
"embedder_model_name",
|
||||||
"embedder_model_version",
|
"embedder_model_version",
|
||||||
*_DETECTOR_FIELDS,
|
*_DETECTOR_FIELDS,
|
||||||
@@ -84,21 +80,45 @@ async def embedder_models():
|
|||||||
|
|
||||||
@ml_admin_bp.route("/settings", methods=["GET"])
|
@ml_admin_bp.route("/settings", methods=["GET"])
|
||||||
async def get_settings():
|
async def get_settings():
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
s = await MLSettings.load(session)
|
s = (
|
||||||
# Table-driven off _EDITABLE (which PATCH also writes) so a new settings field
|
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
||||||
# can never be silently absent from GET — the split that historically dropped
|
).scalar_one()
|
||||||
# fields. _EDITABLE already includes *_DETECTOR_FIELDS.
|
return jsonify(
|
||||||
return jsonify({f: getattr(s, f) for f in _EDITABLE})
|
{
|
||||||
|
"cpu_embed_enabled": s.cpu_embed_enabled,
|
||||||
|
"video_frame_interval_seconds": s.video_frame_interval_seconds,
|
||||||
|
"video_max_frames": s.video_max_frames,
|
||||||
|
"embedder_model_version": s.embedder_model_version,
|
||||||
|
"head_min_positives": s.head_min_positives,
|
||||||
|
"head_auto_apply_precision": s.head_auto_apply_precision,
|
||||||
|
"head_auto_apply_enabled": s.head_auto_apply_enabled,
|
||||||
|
"head_auto_apply_min_positives": s.head_auto_apply_min_positives,
|
||||||
|
"ccip_match_threshold": s.ccip_match_threshold,
|
||||||
|
"ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
|
||||||
|
"ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
|
||||||
|
"presentation_auto_apply_enabled": s.presentation_auto_apply_enabled,
|
||||||
|
"presentation_auto_apply_threshold": s.presentation_auto_apply_threshold,
|
||||||
|
"presentation_conflict_threshold": s.presentation_conflict_threshold,
|
||||||
|
"embedder_model_name": s.embedder_model_name,
|
||||||
|
**{f: getattr(s, f) for f in _DETECTOR_FIELDS},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@ml_admin_bp.route("/settings", methods=["PATCH"])
|
@ml_admin_bp.route("/settings", methods=["PATCH"])
|
||||||
async def patch_settings():
|
async def patch_settings():
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
body = await request.get_json()
|
body = await request.get_json()
|
||||||
if not isinstance(body, dict):
|
if not isinstance(body, dict):
|
||||||
return jsonify({"error": "body must be an object"}), 400
|
return jsonify({"error": "body must be an object"}), 400
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
s = await MLSettings.load(session)
|
s = (
|
||||||
|
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
# Merge the patch over current values, then validate the result as a
|
# Merge the patch over current values, then validate the result as a
|
||||||
# whole — the store-floor invariant couples three fields, so they
|
# whole — the store-floor invariant couples three fields, so they
|
||||||
@@ -128,26 +148,20 @@ def _validate(p: dict) -> str | None:
|
|||||||
# Head training (#114).
|
# Head training (#114).
|
||||||
if int(p["head_min_positives"]) < 1:
|
if int(p["head_min_positives"]) < 1:
|
||||||
return "head_min_positives must be >= 1"
|
return "head_min_positives must be >= 1"
|
||||||
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["head_auto_apply_precision"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999):
|
||||||
return f"head_auto_apply_precision must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
return "head_auto_apply_precision must be between 0.5 and 0.999"
|
||||||
if int(p["head_auto_apply_min_positives"]) < 1:
|
if int(p["head_auto_apply_min_positives"]) < 1:
|
||||||
return "head_auto_apply_min_positives must be >= 1"
|
return "head_auto_apply_min_positives must be >= 1"
|
||||||
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_match_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999):
|
||||||
return f"ccip_match_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
return "ccip_match_threshold must be between 0.5 and 0.999"
|
||||||
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999):
|
||||||
return f"ccip_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
return "ccip_auto_apply_threshold must be between 0.5 and 0.999"
|
||||||
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
|
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
|
||||||
# consequential); the conflict cut is a plain probability [0,1].
|
# consequential); the conflict cut is a plain probability [0,1].
|
||||||
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["presentation_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
if not (0.5 <= float(p["presentation_auto_apply_threshold"]) <= 0.999):
|
||||||
return f"presentation_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
return "presentation_auto_apply_threshold must be between 0.5 and 0.999"
|
||||||
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
|
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
|
||||||
return "presentation_conflict_threshold must be between 0 and 1"
|
return "presentation_conflict_threshold must be between 0 and 1"
|
||||||
# Process auto-apply (#1464). wip/editor stay VISIBLE so a false apply is
|
|
||||||
# low-harm (excludes-from-training + a review flag), but keep the same bar.
|
|
||||||
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["process_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
|
||||||
return f"process_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
|
||||||
if not (0.0 <= float(p["process_conflict_threshold"]) <= 1.0):
|
|
||||||
return "process_conflict_threshold must be between 0 and 1"
|
|
||||||
# Embedder model swap (#1190): both must be non-empty. Changing them means a
|
# Embedder model swap (#1190): both must be non-empty. Changing them means a
|
||||||
# different embedding space — the operator must re-embed + retrain after.
|
# different embedding space — the operator must re-embed + retrain after.
|
||||||
for key in ("embedder_model_name", "embedder_model_version"):
|
for key in ("embedder_model_name", "embedder_model_version"):
|
||||||
|
|||||||
@@ -3,28 +3,12 @@
|
|||||||
from quart import Blueprint, jsonify, request
|
from quart import Blueprint, jsonify, request
|
||||||
|
|
||||||
from ..extensions import get_session
|
from ..extensions import get_session
|
||||||
from ..models import ImportSettings, Post
|
|
||||||
from ..services import interpreter_client as ic
|
|
||||||
from ..services.post_feed_service import PostFeedService
|
from ..services.post_feed_service import PostFeedService
|
||||||
from ..services.source_service import KNOWN_PLATFORMS
|
from ..services.source_service import KNOWN_PLATFORMS
|
||||||
from ..utils.text import html_to_plain
|
|
||||||
from ._responses import error_response as _bad
|
from ._responses import error_response as _bad
|
||||||
|
|
||||||
posts_bp = Blueprint("posts", __name__, url_prefix="/api/posts")
|
posts_bp = Blueprint("posts", __name__, url_prefix="/api/posts")
|
||||||
|
|
||||||
_TRANSLATION_OVERRIDES = ("auto", "force", "original")
|
|
||||||
|
|
||||||
|
|
||||||
def _queue_for_sweep(post: Post) -> None:
|
|
||||||
"""Mark a post untranslated (all translation columns NULL) so the periodic
|
|
||||||
sweep re-runs it under its new override — used when Interpreter is down and we
|
|
||||||
can't translate inline."""
|
|
||||||
post.post_title_translated = None
|
|
||||||
post.description_translated = None
|
|
||||||
post.translated_source_lang = None
|
|
||||||
post.translation_engine_version = None
|
|
||||||
post.translated_at = None
|
|
||||||
|
|
||||||
|
|
||||||
@posts_bp.route("", methods=["GET"])
|
@posts_bp.route("", methods=["GET"])
|
||||||
async def list_posts():
|
async def list_posts():
|
||||||
@@ -98,70 +82,3 @@ async def get_post(post_id: int):
|
|||||||
if item is None:
|
if item is None:
|
||||||
return _bad("not_found", status=404, detail=f"post id={post_id}")
|
return _bad("not_found", status=404, detail=f"post id={post_id}")
|
||||||
return jsonify(item)
|
return jsonify(item)
|
||||||
|
|
||||||
|
|
||||||
@posts_bp.route("/<int:post_id>/translation-override", methods=["POST"])
|
|
||||||
async def set_translation_override(post_id: int):
|
|
||||||
"""Sticky per-post translation override (milestone 155). Body:
|
|
||||||
``{"override": "auto" | "force" | "original"}``.
|
|
||||||
|
|
||||||
'original' keeps the original (clears any stored translation now — no
|
|
||||||
Interpreter needed). 'force'/'auto' translate the post immediately if the
|
|
||||||
service is up (force bypasses the acceptance floor; auto re-runs the gate);
|
|
||||||
if it's down we save the flag and mark the post untranslated so the next sweep
|
|
||||||
applies it. The override persists, so the sweep + Re-translate-all keep
|
|
||||||
honoring it. Returns the updated translation fields + an ``applied`` status."""
|
|
||||||
body = await request.get_json(silent=True) or {}
|
|
||||||
override = body.get("override")
|
|
||||||
if override not in _TRANSLATION_OVERRIDES:
|
|
||||||
return _bad(
|
|
||||||
"invalid_override",
|
|
||||||
detail=f"override must be one of {list(_TRANSLATION_OVERRIDES)}",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Lazy import (mirrors settings.py) so the API module doesn't pull the celery
|
|
||||||
# task graph at import time.
|
|
||||||
from ..tasks.translation import _store_translation, _translate_field
|
|
||||||
|
|
||||||
async with get_session() as session:
|
|
||||||
post = await session.get(Post, post_id)
|
|
||||||
if post is None:
|
|
||||||
return _bad("not_found", status=404, detail=f"post id={post_id}")
|
|
||||||
post.translation_override = override
|
|
||||||
cfg = await ImportSettings.load(session)
|
|
||||||
target = (cfg.translation_target_lang or "en").strip() or "en"
|
|
||||||
|
|
||||||
if override == "original":
|
|
||||||
_store_translation(post, (None, None, None), (None, None, None), target)
|
|
||||||
applied = "cleared"
|
|
||||||
else:
|
|
||||||
base_url = (cfg.interpreter_base_url or "").strip()
|
|
||||||
if cfg.translation_enabled and base_url and ic.health(base_url):
|
|
||||||
force = override == "force"
|
|
||||||
title = (post.post_title or "").strip()
|
|
||||||
desc = (html_to_plain(post.description) if post.description else "") or ""
|
|
||||||
desc = desc.strip()
|
|
||||||
mc = cfg.translation_min_confidence
|
|
||||||
try:
|
|
||||||
title_res = _translate_field(title, base_url, target, mc, force=force)
|
|
||||||
desc_res = _translate_field(desc, base_url, target, mc, force=force)
|
|
||||||
except ic.InterpreterUnavailable:
|
|
||||||
_queue_for_sweep(post)
|
|
||||||
applied = "queued"
|
|
||||||
else:
|
|
||||||
_store_translation(post, title_res, desc_res, target)
|
|
||||||
applied = "translated"
|
|
||||||
else:
|
|
||||||
# Disabled / no URL / unhealthy → let the sweep apply it later.
|
|
||||||
_queue_for_sweep(post)
|
|
||||||
applied = "queued"
|
|
||||||
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({
|
|
||||||
"id": post.id,
|
|
||||||
"translation_override": post.translation_override,
|
|
||||||
"post_title_translated": post.post_title_translated,
|
|
||||||
"description_translated": post.description_translated,
|
|
||||||
"translated_source_lang": post.translated_source_lang,
|
|
||||||
"applied": applied,
|
|
||||||
})
|
|
||||||
|
|||||||
+25
-38
@@ -47,9 +47,6 @@ _EDITABLE_FIELDS = (
|
|||||||
"translation_enabled",
|
"translation_enabled",
|
||||||
"interpreter_base_url",
|
"interpreter_base_url",
|
||||||
"translation_target_lang",
|
"translation_target_lang",
|
||||||
"translation_min_confidence",
|
|
||||||
"wip_title_tagging_enabled",
|
|
||||||
"wip_soft_title_tagging_enabled",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# Per-host external-download toggles — all plain booleans, validated uniformly.
|
# Per-host external-download toggles — all plain booleans, validated uniformly.
|
||||||
@@ -66,9 +63,31 @@ _EXTDL_TOGGLE_FIELDS = (
|
|||||||
async def get_import_settings():
|
async def get_import_settings():
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
row = await ImportSettings.load(session)
|
row = await ImportSettings.load(session)
|
||||||
# Table-driven off _EDITABLE_FIELDS (which PATCH also writes) so a new field
|
return jsonify({
|
||||||
# can't be silently absent from GET.
|
"min_width": row.min_width,
|
||||||
return jsonify({f: getattr(row, f) for f in _EDITABLE_FIELDS})
|
"min_height": row.min_height,
|
||||||
|
"skip_transparent": row.skip_transparent,
|
||||||
|
"transparency_threshold": row.transparency_threshold,
|
||||||
|
"skip_single_color": row.skip_single_color,
|
||||||
|
"single_color_threshold": row.single_color_threshold,
|
||||||
|
"single_color_tolerance": row.single_color_tolerance,
|
||||||
|
"phash_threshold": row.phash_threshold,
|
||||||
|
"download_rate_limit_seconds": row.download_rate_limit_seconds,
|
||||||
|
"download_validate_files": row.download_validate_files,
|
||||||
|
"download_schedule_default_seconds": row.download_schedule_default_seconds,
|
||||||
|
"download_event_retention_days": row.download_event_retention_days,
|
||||||
|
"download_failure_warning_threshold": row.download_failure_warning_threshold,
|
||||||
|
"series_suggest_enabled": row.series_suggest_enabled,
|
||||||
|
"series_suggest_threshold": row.series_suggest_threshold,
|
||||||
|
"extdl_mega_enabled": row.extdl_mega_enabled,
|
||||||
|
"extdl_gdrive_enabled": row.extdl_gdrive_enabled,
|
||||||
|
"extdl_mediafire_enabled": row.extdl_mediafire_enabled,
|
||||||
|
"extdl_dropbox_enabled": row.extdl_dropbox_enabled,
|
||||||
|
"extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
|
||||||
|
"translation_enabled": row.translation_enabled,
|
||||||
|
"interpreter_base_url": row.interpreter_base_url,
|
||||||
|
"translation_target_lang": row.translation_target_lang,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
@settings_bp.route("/settings/import", methods=["PATCH"])
|
@settings_bp.route("/settings/import", methods=["PATCH"])
|
||||||
@@ -136,32 +155,12 @@ async def update_import_settings():
|
|||||||
for key in ("interpreter_base_url", "translation_target_lang"):
|
for key in ("interpreter_base_url", "translation_target_lang"):
|
||||||
if key in body and not isinstance(body[key], str):
|
if key in body and not isinstance(body[key], str):
|
||||||
return jsonify({"error": f"{key} must be a string"}), 400
|
return jsonify({"error": f"{key} must be a string"}), 400
|
||||||
# Acceptance floor (milestone 155): latin-script translations below this
|
|
||||||
# Interpreter confidence are kept as the original.
|
|
||||||
if "translation_min_confidence" in body:
|
|
||||||
v = body["translation_min_confidence"]
|
|
||||||
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
|
|
||||||
return jsonify(
|
|
||||||
{"error": "translation_min_confidence must be a number in [0, 1]"}
|
|
||||||
), 400
|
|
||||||
if "series_suggest_threshold" in body:
|
if "series_suggest_threshold" in body:
|
||||||
v = body["series_suggest_threshold"]
|
v = body["series_suggest_threshold"]
|
||||||
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
|
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
|
||||||
return jsonify(
|
return jsonify(
|
||||||
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
||||||
), 400
|
), 400
|
||||||
if "wip_title_tagging_enabled" in body and not isinstance(
|
|
||||||
body["wip_title_tagging_enabled"], bool
|
|
||||||
):
|
|
||||||
return jsonify(
|
|
||||||
{"error": "wip_title_tagging_enabled must be a boolean"}
|
|
||||||
), 400
|
|
||||||
if "wip_soft_title_tagging_enabled" in body and not isinstance(
|
|
||||||
body["wip_soft_title_tagging_enabled"], bool
|
|
||||||
):
|
|
||||||
return jsonify(
|
|
||||||
{"error": "wip_soft_title_tagging_enabled must be a boolean"}
|
|
||||||
), 400
|
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
row = await ImportSettings.load(session)
|
row = await ImportSettings.load(session)
|
||||||
@@ -173,18 +172,6 @@ async def update_import_settings():
|
|||||||
return await get_import_settings()
|
return await get_import_settings()
|
||||||
|
|
||||||
|
|
||||||
@settings_bp.route("/settings/wip-title/scan", methods=["POST"])
|
|
||||||
async def wip_title_scan():
|
|
||||||
"""Enqueue the back-catalogue WIP-title scan (task #1458 Settings button):
|
|
||||||
apply the `wip` system tag to EXISTING posts whose title declares
|
|
||||||
work-in-progress. New imports are tagged live by the importer; this catches
|
|
||||||
the existing library. Returns the Celery task id (202)."""
|
|
||||||
from ..tasks.maintenance import backfill_wip_title_tags
|
|
||||||
|
|
||||||
r = backfill_wip_title_tags.delay()
|
|
||||||
return jsonify({"celery_task_id": r.id}), 202
|
|
||||||
|
|
||||||
|
|
||||||
@settings_bp.route("/system/stats", methods=["GET"])
|
@settings_bp.route("/system/stats", methods=["GET"])
|
||||||
async def system_stats():
|
async def system_stats():
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
|
|||||||
@@ -11,11 +11,10 @@ suggestions_bp = Blueprint("suggestions", __name__, url_prefix="/api")
|
|||||||
|
|
||||||
@suggestions_bp.route("/images/<int:image_id>/suggestions", methods=["GET"])
|
@suggestions_bp.route("/images/<int:image_id>/suggestions", methods=["GET"])
|
||||||
async def get_suggestions(image_id: int):
|
async def get_suggestions(image_id: int):
|
||||||
# ?min=<float> overrides the per-head suggest thresholds for INCLUSION. The
|
# ?min=<float> overrides the configured per-category thresholds so the typed
|
||||||
# rail sends min=0 in its single per-image fetch to get EVERY head (each row
|
# tag-input dropdown can surface EVERY stored prediction (min=0), including
|
||||||
# still carries above_threshold vs its natural cut), then derives the panel
|
# low-confidence actions/features, in canonical formatting. Omitted → the
|
||||||
# (above_threshold) and the typed dropdown (all, filtered by text) client-side
|
# curated above-threshold list the Suggestions panel uses.
|
||||||
# — no second request. Omitted → only above-threshold rows.
|
|
||||||
override = None
|
override = None
|
||||||
raw_min = request.args.get("min")
|
raw_min = request.args.get("min")
|
||||||
if raw_min is not None:
|
if raw_min is not None:
|
||||||
@@ -37,11 +36,12 @@ async def get_suggestions(image_id: int):
|
|||||||
"category": s.category,
|
"category": s.category,
|
||||||
"score": round(s.score, 4),
|
"score": round(s.score, 4),
|
||||||
"source": s.source,
|
"source": s.source,
|
||||||
# whether the score cleared the head's own suggest cut.
|
"creates_new_tag": s.creates_new_tag,
|
||||||
# The single min=0 fetch returns every head; the panel
|
# raw model key (alias is stored under this) + whether an
|
||||||
# shows above_threshold, the typed dropdown shows all and
|
# operator alias produced this suggestion — drive the
|
||||||
# annotates each match with its score.
|
# modal's "Treat as alias"/"Remove alias" affordances.
|
||||||
"above_threshold": s.above_threshold,
|
"raw_name": s.raw_name,
|
||||||
|
"via_alias": s.via_alias,
|
||||||
# operator dismissed this tag for this image — surfaced
|
# operator dismissed this tag for this image — surfaced
|
||||||
# (not dropped) so the rail can show it rejected + offer
|
# (not dropped) so the rail can show it rejected + offer
|
||||||
# one-click un-reject.
|
# one-click un-reject.
|
||||||
@@ -73,6 +73,26 @@ async def accept_suggestion(image_id: int):
|
|||||||
return jsonify({"accepted": True, "tag_id": tag_id})
|
return jsonify({"accepted": True, "tag_id": tag_id})
|
||||||
|
|
||||||
|
|
||||||
|
@suggestions_bp.route(
|
||||||
|
"/images/<int:image_id>/suggestions/alias", methods=["POST"]
|
||||||
|
)
|
||||||
|
async def alias_suggestion(image_id: int):
|
||||||
|
body = await request.get_json()
|
||||||
|
required = {"alias_string", "alias_category", "canonical_tag_id"}
|
||||||
|
if not body or not required.issubset(body):
|
||||||
|
return jsonify({"error": f"required: {sorted(required)}"}), 400
|
||||||
|
canonical_tag_id = body["canonical_tag_id"]
|
||||||
|
async with get_session() as session:
|
||||||
|
await AllowlistService(session).add_alias_and_accept(
|
||||||
|
image_id,
|
||||||
|
body["alias_string"],
|
||||||
|
body["alias_category"],
|
||||||
|
canonical_tag_id,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
return jsonify({"accepted": True, "tag_id": canonical_tag_id})
|
||||||
|
|
||||||
|
|
||||||
@suggestions_bp.route(
|
@suggestions_bp.route(
|
||||||
"/images/<int:image_id>/suggestions/dismiss", methods=["POST"]
|
"/images/<int:image_id>/suggestions/dismiss", methods=["POST"]
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -171,19 +171,9 @@ def make_celery() -> Celery:
|
|||||||
},
|
},
|
||||||
"presentation-auto-apply-daily": {
|
"presentation-auto-apply-daily": {
|
||||||
"task": "backend.app.tasks.ml.scheduled_presentation_auto_apply",
|
"task": "backend.app.tasks.ml.scheduled_presentation_auto_apply",
|
||||||
"schedule": 86400.0, # auto-hide banner chrome (#141);
|
"schedule": 86400.0, # auto-hide banner/editor chrome (#141);
|
||||||
# no-op unless presentation_auto_apply_enabled
|
# no-op unless presentation_auto_apply_enabled
|
||||||
},
|
},
|
||||||
"process-auto-apply-daily": {
|
|
||||||
"task": "backend.app.tasks.ml.scheduled_process_auto_apply",
|
|
||||||
"schedule": 86400.0, # auto-tag wip/editor process art (#1464);
|
|
||||||
# no-op unless process_auto_apply_enabled (opt-in)
|
|
||||||
},
|
|
||||||
"soft-wip-conflict-audit-daily": {
|
|
||||||
"task": "backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
|
|
||||||
"schedule": 86400.0, # flag ring-loud soft-WIP (sketch/doodle) tags
|
|
||||||
# for review (#1474); no-op with no content heads
|
|
||||||
},
|
|
||||||
"prune-presentation-reviews-daily": {
|
"prune-presentation-reviews-daily": {
|
||||||
"task": "backend.app.tasks.ml.prune_presentation_reviews",
|
"task": "backend.app.tasks.ml.prune_presentation_reviews",
|
||||||
"schedule": 86400.0, # retention: drop resolved review flags >30d
|
"schedule": 86400.0, # retention: drop resolved review flags >30d
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ from .patreon_seen_media import PatreonSeenMedia
|
|||||||
from .pixiv_failed_media import PixivFailedMedia
|
from .pixiv_failed_media import PixivFailedMedia
|
||||||
from .pixiv_seen_media import PixivSeenMedia
|
from .pixiv_seen_media import PixivSeenMedia
|
||||||
from .post import Post
|
from .post import Post
|
||||||
from .post_attachment import PostAttachment, attachment_download_url
|
from .post_attachment import PostAttachment
|
||||||
from .presentation_review import PresentationReview
|
from .presentation_review import PresentationReview
|
||||||
from .series_chapter import SeriesChapter
|
from .series_chapter import SeriesChapter
|
||||||
from .series_page import SeriesPage
|
from .series_page import SeriesPage
|
||||||
@@ -58,7 +58,6 @@ __all__ = [
|
|||||||
"SubscribeStarSeenMedia",
|
"SubscribeStarSeenMedia",
|
||||||
"Post",
|
"Post",
|
||||||
"PostAttachment",
|
"PostAttachment",
|
||||||
"attachment_download_url",
|
|
||||||
"PresentationReview",
|
"PresentationReview",
|
||||||
"SeriesChapter",
|
"SeriesChapter",
|
||||||
"SeriesPage",
|
"SeriesPage",
|
||||||
|
|||||||
@@ -106,33 +106,6 @@ class ImportSettings(Base):
|
|||||||
translation_target_lang: Mapped[str] = mapped_column(
|
translation_target_lang: Mapped[str] = mapped_column(
|
||||||
Text, nullable=False, default="en", server_default="en",
|
Text, nullable=False, default="en", server_default="en",
|
||||||
)
|
)
|
||||||
# The latin-script acceptance floor for the translation gate: a translation
|
|
||||||
# whose Interpreter-reported confidence is below this is kept as the original
|
|
||||||
# (operator-tunable, milestone 155). Default 0.9 — stricter than the old
|
|
||||||
# hardcoded 0.8, because Interpreter confidently mis-detects short ASCII
|
|
||||||
# English (e.g. "… WIP Part 1") as a European language at ~0.86. CJK stays
|
|
||||||
# trusted regardless (script-detected). Per-post overrides handle the misses.
|
|
||||||
translation_min_confidence: Mapped[float] = mapped_column(
|
|
||||||
Float, nullable=False, default=0.9, server_default="0.9",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Title-based WIP auto-tagging (task #1458). When a freshly-imported post's
|
|
||||||
# TITLE explicitly declares work-in-progress ("WIP" / "work in progress"),
|
|
||||||
# the importer applies the `wip` system tag to its images — the artist's own
|
|
||||||
# label, used to keep unfinished pieces out of the Explore/gallery browse. ON
|
|
||||||
# by default (rule 26 — the feature works out of the box). Gates only the
|
|
||||||
# LIVE import hook; the existing catalogue is caught by the operator-triggered
|
|
||||||
# "Scan existing posts" backfill (which runs regardless of this flag).
|
|
||||||
wip_title_tagging_enabled: Mapped[bool] = mapped_column(
|
|
||||||
Boolean, nullable=False, default=True, server_default="true",
|
|
||||||
)
|
|
||||||
# Soft WIP title tier (#1474): also tag sketch/doodle/scribble titles, but with
|
|
||||||
# a PROVISIONAL source (`wip_title_soft`) that never trains the head, since these
|
|
||||||
# are lower-precision (a finished "sketch" isn't WIP). OFF by default — a lower-
|
|
||||||
# precision tier is opt-in (the ring-loud audit surfaces false positives).
|
|
||||||
wip_soft_title_tagging_enabled: Mapped[bool] = mapped_column(
|
|
||||||
Boolean, nullable=False, default=False, server_default="false",
|
|
||||||
)
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
async def load(cls, session) -> ImportSettings:
|
async def load(cls, session) -> ImportSettings:
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ from sqlalchemy import (
|
|||||||
Integer,
|
Integer,
|
||||||
String,
|
String,
|
||||||
func,
|
func,
|
||||||
select,
|
|
||||||
)
|
)
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
@@ -86,14 +85,12 @@ class MLSettings(Base):
|
|||||||
Float, nullable=False, default=0.95
|
Float, nullable=False, default=0.95
|
||||||
)
|
)
|
||||||
# -- Presentation chrome auto-hide (#141) -------------------------------
|
# -- Presentation chrome auto-hide (#141) -------------------------------
|
||||||
# `banner` (chrome — clusters on UI, not content) auto-applies on the sweep
|
# banner / editor screenshot auto-apply on the sweep with their OWN flat
|
||||||
# with its OWN flat threshold (decoupled from content-head graduation) and is
|
# threshold (decoupled from content-head graduation). Hiding is consequential
|
||||||
# HIDDEN from the gallery. Hiding is consequential so it runs HIGH. When an
|
# so it runs HIGH. `wip` is never auto-applied. When an image would be
|
||||||
# image would be auto-hidden but ALSO scores >= presentation_conflict_threshold
|
# auto-hidden but ALSO scores >= presentation_conflict_threshold on a content
|
||||||
# on a content head, it's still hidden but flagged for review
|
# head, it's still hidden but flagged for review (PresentationReview) instead
|
||||||
# (PresentationReview, mode='chrome') instead of buried silently. ON by default
|
# of buried silently. ON by default (opt-out); every auto-tag is reversible.
|
||||||
# (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor
|
|
||||||
# screenshot` are no longer chrome — they went to the PROCESS path below.
|
|
||||||
presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
|
presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||||
Boolean, nullable=False, default=True
|
Boolean, nullable=False, default=True
|
||||||
)
|
)
|
||||||
@@ -103,26 +100,6 @@ class MLSettings(Base):
|
|||||||
presentation_conflict_threshold: Mapped[float] = mapped_column(
|
presentation_conflict_threshold: Mapped[float] = mapped_column(
|
||||||
Float, nullable=False, default=0.50
|
Float, nullable=False, default=0.50
|
||||||
)
|
)
|
||||||
# -- Process auto-apply (#1464) ----------------------------------------
|
|
||||||
# `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program
|
|
||||||
# screenshots that must stay OUT of head/CCIP training but, unlike chrome,
|
|
||||||
# remain VISIBLE in the gallery (operator 2026-07-12). They auto-apply on the
|
|
||||||
# sweep with their OWN flat threshold and a PROVISIONAL source (`process_auto`,
|
|
||||||
# in training_data._AUTO_SOURCES) so the head NEVER trains on its own output —
|
|
||||||
# it learns only from title (`wip_title`) + manual labels, which breaks the
|
|
||||||
# runaway loop. When a process tag would be applied but the image ALSO scores
|
|
||||||
# >= process_conflict_threshold on a content head, it's flagged for review
|
|
||||||
# (PresentationReview, mode='process') rather than silently marked. OFF by
|
|
||||||
# default — a new whole-library auto-tagger is opt-in; every auto-tag reversible.
|
|
||||||
process_auto_apply_enabled: Mapped[bool] = mapped_column(
|
|
||||||
Boolean, nullable=False, default=False
|
|
||||||
)
|
|
||||||
process_auto_apply_threshold: Mapped[float] = mapped_column(
|
|
||||||
Float, nullable=False, default=0.90
|
|
||||||
)
|
|
||||||
process_conflict_threshold: Mapped[float] = mapped_column(
|
|
||||||
Float, nullable=False, default=0.50
|
|
||||||
)
|
|
||||||
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
||||||
# existing libraries keep their stored value until the operator re-embeds.
|
# existing libraries keep their stored value until the operator re-embeds.
|
||||||
embedder_model_version: Mapped[str] = mapped_column(
|
embedder_model_version: Mapped[str] = mapped_column(
|
||||||
@@ -213,14 +190,3 @@ class MLSettings(Base):
|
|||||||
updated_at: Mapped[datetime] = mapped_column(
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
|
||||||
async def load(cls, session) -> MLSettings:
|
|
||||||
"""The singleton settings row (id=1), via an async session. Mirrors
|
|
||||||
ImportSettings.load — the shared singleton-loader pattern."""
|
|
||||||
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def load_sync(cls, session) -> MLSettings:
|
|
||||||
"""The singleton settings row (id=1), via a sync session."""
|
|
||||||
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
|
|
||||||
|
|||||||
@@ -8,17 +8,7 @@ artist-filter queries don't depend on the Source detour).
|
|||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import (
|
from sqlalchemy import JSON, DateTime, ForeignKey, Integer, String, Text, UniqueConstraint, func
|
||||||
JSON,
|
|
||||||
CheckConstraint,
|
|
||||||
DateTime,
|
|
||||||
ForeignKey,
|
|
||||||
Integer,
|
|
||||||
String,
|
|
||||||
Text,
|
|
||||||
UniqueConstraint,
|
|
||||||
func,
|
|
||||||
)
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from .base import Base
|
from .base import Base
|
||||||
@@ -33,10 +23,6 @@ class Post(Base):
|
|||||||
# (created in alembic 0030) covers that case via
|
# (created in alembic 0030) covers that case via
|
||||||
# (artist_id, external_post_id).
|
# (artist_id, external_post_id).
|
||||||
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
||||||
CheckConstraint(
|
|
||||||
"translation_override IN ('auto', 'force', 'original')",
|
|
||||||
name="ck_post_translation_override",
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
@@ -78,16 +64,6 @@ class Post(Base):
|
|||||||
translated_at: Mapped[datetime | None] = mapped_column(
|
translated_at: Mapped[datetime | None] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=True
|
DateTime(timezone=True), nullable=True
|
||||||
)
|
)
|
||||||
# Sticky per-post override of the translation decision (milestone 155):
|
|
||||||
# 'auto' = the acceptance gate decides; 'force' = always store Interpreter's
|
|
||||||
# translation even below the confidence floor (rescue a skipped legit-foreign
|
|
||||||
# title); 'original' = never translate, keep the original (kill a confidently
|
|
||||||
# mis-flagged one the floor can't catch). The sweep reads this on every run,
|
|
||||||
# and re-translate leaves 'original' posts alone, so the choice survives a
|
|
||||||
# Re-translate-all.
|
|
||||||
translation_override: Mapped[str] = mapped_column(
|
|
||||||
String(16), nullable=False, default="auto", server_default="auto",
|
|
||||||
)
|
|
||||||
|
|
||||||
downloaded_at: Mapped[datetime] = mapped_column(
|
downloaded_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
|
|||||||
@@ -65,15 +65,3 @@ class PostAttachment(Base):
|
|||||||
captured_at: Mapped[datetime] = mapped_column(
|
captured_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def attachment_download_url(attachment_id: int) -> str:
|
|
||||||
"""The path that streams this attachment's bytes.
|
|
||||||
|
|
||||||
Both serializers that expose an attachment to the frontend
|
|
||||||
(`provenance_service`, `post_feed_service`) built this literal themselves,
|
|
||||||
so changing the route in `api/attachments.py` meant two edits and only one
|
|
||||||
would be remembered (#3072). `test_attachment_download_url` pins it against
|
|
||||||
the app's registered rule, so the drift is caught rather than trusted to.
|
|
||||||
"""
|
|
||||||
return f"/api/attachments/{attachment_id}/download"
|
|
||||||
|
|||||||
@@ -1,17 +1,15 @@
|
|||||||
"""PresentationReview — a system-tag the auto-apply sweep applied that ALSO looked
|
"""PresentationReview — an auto-hidden presentation tag that ALSO looked like
|
||||||
like real content, flagged for operator review (milestone 141 + #1464).
|
real content, flagged for operator review (milestone 141).
|
||||||
|
|
||||||
When a sweep applies a system tag but the image ALSO scores highly on a content
|
When the auto-apply sweep hides an image as chrome (banner / editor screenshot)
|
||||||
head, it still applies the tag but records this row so a review strip can surface
|
but the image ALSO scores highly on a content head, it still hides it but records
|
||||||
it ("⚠ also looks like <conflict tag>"). Two modes (#1464): 'chrome' (banner —
|
this row so the Hidden view can surface it ("⚠ also looks like <conflict tag>")
|
||||||
image is HIDDEN, review is keep-hidden / un-hide) and 'process' (wip / editor
|
for a keep-hidden / un-hide decision. Resolved rows are pruned by retention.
|
||||||
screenshot — image stays VISIBLE, review is confirm / remove-tag). Resolved rows
|
|
||||||
are pruned by retention.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import DateTime, Float, ForeignKey, String, func
|
from sqlalchemy import DateTime, Float, ForeignKey, func
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from .base import Base
|
from .base import Base
|
||||||
@@ -33,12 +31,6 @@ class PresentationReview(Base):
|
|||||||
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True
|
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True
|
||||||
)
|
)
|
||||||
conflict_score: Mapped[float] = mapped_column(Float, nullable=False)
|
conflict_score: Mapped[float] = mapped_column(Float, nullable=False)
|
||||||
# Which sweep flagged this (#1464): 'chrome' (banner, hidden) or 'process'
|
|
||||||
# (wip / editor screenshot, shown). Drives which review strip surfaces it and
|
|
||||||
# what "resolve" means (un-hide vs remove-tag). Existing rows backfill 'chrome'.
|
|
||||||
mode: Mapped[str] = mapped_column(
|
|
||||||
String(16), nullable=False, default="chrome", server_default="chrome"
|
|
||||||
)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -43,19 +43,14 @@ class TagKind(StrEnum):
|
|||||||
# to keep historic tag rows queryable.
|
# to keep historic tag rows queryable.
|
||||||
|
|
||||||
|
|
||||||
# The seeded system tags (migration 0075). Two behavior groups (#1464):
|
# The seeded system tags (migration 0075). PRESENTATION tags additionally
|
||||||
# CHROME (banner): clusters on UI chrome, not content → HIDDEN from the default
|
# hide from whole-image similarity results — they cluster on UI chrome, not
|
||||||
# gallery + from similarity; auto-applied via the sweep's chrome mode.
|
# content. `wip` is real art: only the training pipelines exclude it.
|
||||||
# PROCESS (wip, editor screenshot): real-but-unfinished art / program screenshots
|
|
||||||
# → SHOWN in the gallery (operator 2026-07-12), but excluded from the Explore
|
|
||||||
# rabbit-hole; auto-applied via the sweep's process mode (provisional source,
|
|
||||||
# ring-loud review guard).
|
|
||||||
# ALL three are excluded from OTHER concepts' head/CCIP training (training-hygiene,
|
|
||||||
# keyed on is_system); a system tag's OWN head trains on them — that's what makes
|
|
||||||
# auto-flagging work.
|
|
||||||
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
|
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
|
||||||
CHROME_SYSTEM_TAGS = ("banner",)
|
PRESENTATION_SYSTEM_TAGS = ("banner", "editor screenshot")
|
||||||
PROCESS_SYSTEM_TAGS = ("wip", "editor screenshot")
|
# `wip` marks real-but-unfinished art. It's kept in the gallery's own "similar"
|
||||||
|
# results (#1274), but the Explore rabbit-hole opts to hide it (exclude_wip) so a
|
||||||
|
# browse doesn't keep surfacing work-in-progress (operator, 2026-07-08).
|
||||||
WIP_SYSTEM_TAG = "wip"
|
WIP_SYSTEM_TAG = "wip"
|
||||||
|
|
||||||
image_tag = Table(
|
image_tag = Table(
|
||||||
|
|||||||
@@ -48,47 +48,6 @@ log = logging.getLogger(__name__)
|
|||||||
_VIDEO_DURATION_UNKNOWN = -1.0
|
_VIDEO_DURATION_UNKNOWN = -1.0
|
||||||
|
|
||||||
|
|
||||||
# -- artist-cascade predicates (rule 93: ONE definition, preview + apply) ---
|
|
||||||
# project_artist_cascade (preview) and delete_artist_cascade (apply) both build
|
|
||||||
# their queries from these. The preview used to re-derive its own — which is how
|
|
||||||
# it came to count images and stay silent about posts and attachments while the
|
|
||||||
# apply destroyed both. Same failure shape as the 2026-06-08 fandom-tag
|
|
||||||
# deletion, where a re-implemented delete predicate diverged from the preview's.
|
|
||||||
# Returned as condition LISTS spread into `.where(*conds)`, matching
|
|
||||||
# _unused_tag_conditions / _bare_post_conditions below.
|
|
||||||
|
|
||||||
|
|
||||||
def _artist_images_conditions(artist_id: int) -> list:
|
|
||||||
"""Images the cascade deletes (rows AND their on-disk files)."""
|
|
||||||
return [ImageRecord.artist_id == artist_id]
|
|
||||||
|
|
||||||
|
|
||||||
def _artist_posts_conditions(artist_id: int) -> list:
|
|
||||||
"""Posts the cascade destroys. The apply never names these — post.artist_id
|
|
||||||
is ondelete=CASCADE, so Postgres takes them when the artist row goes — which
|
|
||||||
is exactly why the preview has to name them: an artist whose posts are
|
|
||||||
body-only (no images) otherwise previews as `images: 0` and reads as an
|
|
||||||
empty artist, while every captured body/description/external-link set is
|
|
||||||
destroyed."""
|
|
||||||
return [Post.artist_id == artist_id]
|
|
||||||
|
|
||||||
|
|
||||||
def _artist_attachments_conditions(artist_id: int) -> list:
|
|
||||||
"""Attachments the cascade deletes. Matched by artist_id OR by the owning
|
|
||||||
post's artist: artist_id is nullable (_capture_attachment leaves it NULL
|
|
||||||
when no artist resolved), so neither arm alone covers every row. The
|
|
||||||
sha-addressed blobs are NOT unlinked (one blob backs many rows) — these are
|
|
||||||
row counts, and the bytes are not part of this operation's footprint."""
|
|
||||||
return [
|
|
||||||
or_(
|
|
||||||
PostAttachment.artist_id == artist_id,
|
|
||||||
PostAttachment.post_id.in_(
|
|
||||||
select(Post.id).where(*_artist_posts_conditions(artist_id))
|
|
||||||
),
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||||
"""Read-only projection of what delete_artist_cascade would touch.
|
"""Read-only projection of what delete_artist_cascade would touch.
|
||||||
|
|
||||||
@@ -97,17 +56,12 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
|||||||
"artist": {"id": int, "name": str, "slug": str},
|
"artist": {"id": int, "name": str, "slug": str},
|
||||||
"projected": {
|
"projected": {
|
||||||
"images": int,
|
"images": int,
|
||||||
"posts": int, # hard-deleted by the post.artist_id CASCADE
|
|
||||||
"attachments": int, # rows deleted; the sha-addressed blobs stay
|
|
||||||
"sources": int,
|
"sources": int,
|
||||||
"thumbs": int, # images with a thumbnail_path set
|
"thumbs": int, # images with a thumbnail_path set
|
||||||
"import_tasks": int, # ImportTask rows referencing the artist's images
|
"import_tasks": int, # ImportTask rows referencing the artist's images
|
||||||
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
|
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
Every count is built from the shared `_artist_*_conditions` predicates the
|
|
||||||
apply uses, so the two halves cannot drift (rule 93).
|
|
||||||
|
|
||||||
Raises LookupError if slug not found. No mutations.
|
Raises LookupError if slug not found. No mutations.
|
||||||
"""
|
"""
|
||||||
from ..models.import_task import ImportTask
|
from ..models.import_task import ImportTask
|
||||||
@@ -119,49 +73,36 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
|||||||
if artist is None:
|
if artist is None:
|
||||||
raise LookupError(f"artist slug not found: {slug!r}")
|
raise LookupError(f"artist slug not found: {slug!r}")
|
||||||
|
|
||||||
images_conds = _artist_images_conditions(artist.id)
|
|
||||||
|
|
||||||
images_count = session.execute(
|
images_count = session.execute(
|
||||||
select(func.count(ImageRecord.id)).where(*images_conds)
|
select(func.count(ImageRecord.id))
|
||||||
|
.where(ImageRecord.artist_id == artist.id)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
posts_count = session.execute(
|
|
||||||
select(func.count(Post.id))
|
|
||||||
.where(*_artist_posts_conditions(artist.id))
|
|
||||||
).scalar_one()
|
|
||||||
attachments_count = session.execute(
|
|
||||||
select(func.count(PostAttachment.id))
|
|
||||||
.where(*_artist_attachments_conditions(artist.id))
|
|
||||||
).scalar_one()
|
|
||||||
# Sources have no shared predicate: the apply never queries them either, it
|
|
||||||
# gets them from the Artist.sources ORM cascade. Counted directly here.
|
|
||||||
sources_count = session.execute(
|
sources_count = session.execute(
|
||||||
select(func.count(Source.id))
|
select(func.count(Source.id))
|
||||||
.where(Source.artist_id == artist.id)
|
.where(Source.artist_id == artist.id)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
thumbs_count = session.execute(
|
thumbs_count = session.execute(
|
||||||
select(func.count(ImageRecord.id))
|
select(func.count(ImageRecord.id))
|
||||||
.where(*images_conds)
|
.where(ImageRecord.artist_id == artist.id)
|
||||||
.where(ImageRecord.thumbnail_path.is_not(None))
|
.where(ImageRecord.thumbnail_path.is_not(None))
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
import_tasks_count = session.execute(
|
import_tasks_count = session.execute(
|
||||||
select(func.count(ImportTask.id))
|
select(func.count(ImportTask.id))
|
||||||
.where(
|
.where(
|
||||||
ImportTask.result_image_id.in_(
|
ImportTask.result_image_id.in_(
|
||||||
select(ImageRecord.id).where(*images_conds)
|
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
bytes_on_disk = session.execute(
|
bytes_on_disk = session.execute(
|
||||||
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
|
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
|
||||||
.where(*images_conds)
|
.where(ImageRecord.artist_id == artist.id)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
||||||
"projected": {
|
"projected": {
|
||||||
"images": images_count,
|
"images": images_count,
|
||||||
"posts": posts_count,
|
|
||||||
"attachments": attachments_count,
|
|
||||||
"sources": sources_count,
|
"sources": sources_count,
|
||||||
"thumbs": thumbs_count,
|
"thumbs": thumbs_count,
|
||||||
"import_tasks": import_tasks_count,
|
"import_tasks": import_tasks_count,
|
||||||
@@ -336,10 +277,6 @@ def delete_artist_cascade(
|
|||||||
series_page / tag_suggestion_rejection from ImageRecord delete,
|
series_page / tag_suggestion_rejection from ImageRecord delete,
|
||||||
and source / post / download_event / etc. from Artist delete
|
and source / post / download_event / etc. from Artist delete
|
||||||
(via Artist.sources cascade="all, delete-orphan").
|
(via Artist.sources cascade="all, delete-orphan").
|
||||||
|
|
||||||
The artist's post_attachment rows are cleared EXPLICITLY before the
|
|
||||||
artist row goes — see the comment at that step; leaving them to the
|
|
||||||
cascade aborts the whole delete on a unique violation.
|
|
||||||
"""
|
"""
|
||||||
artist = session.get(Artist, artist_id)
|
artist = session.get(Artist, artist_id)
|
||||||
if artist is None:
|
if artist is None:
|
||||||
@@ -350,22 +287,11 @@ def delete_artist_cascade(
|
|||||||
"files_deleted": 0,
|
"files_deleted": 0,
|
||||||
"thumbs_deleted": 0,
|
"thumbs_deleted": 0,
|
||||||
"import_tasks_nulled": 0,
|
"import_tasks_nulled": 0,
|
||||||
"posts_deleted": 0,
|
|
||||||
"attachments_deleted": 0,
|
|
||||||
"files_failed": 0,
|
"files_failed": 0,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
|
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
|
||||||
|
|
||||||
# Counted BEFORE the delete: Postgres takes these via the post.artist_id
|
|
||||||
# CASCADE when the artist row goes, so afterwards there is nothing left to
|
|
||||||
# count. Reported so the summary can be checked against the preview's
|
|
||||||
# `posts` — the parity rule 93 asks for is only testable if both halves
|
|
||||||
# actually state the number.
|
|
||||||
posts_deleted = session.execute(
|
|
||||||
select(func.count(Post.id)).where(*_artist_posts_conditions(artist.id))
|
|
||||||
).scalar_one()
|
|
||||||
|
|
||||||
images_deleted = 0
|
images_deleted = 0
|
||||||
files_deleted = 0
|
files_deleted = 0
|
||||||
thumbs_deleted = 0
|
thumbs_deleted = 0
|
||||||
@@ -374,7 +300,7 @@ def delete_artist_cascade(
|
|||||||
while True:
|
while True:
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
select(ImageRecord)
|
select(ImageRecord)
|
||||||
.where(*_artist_images_conditions(artist.id))
|
.where(ImageRecord.artist_id == artist.id)
|
||||||
.limit(500)
|
.limit(500)
|
||||||
).scalars().all()
|
).scalars().all()
|
||||||
if not rows:
|
if not rows:
|
||||||
@@ -397,28 +323,6 @@ def delete_artist_cascade(
|
|||||||
# source_path_prefix matching that's out of scope here.
|
# source_path_prefix matching that's out of scope here.
|
||||||
import_tasks_nulled = 0
|
import_tasks_nulled = 0
|
||||||
|
|
||||||
# Clear the artist's attachments BEFORE the artist row, or the delete below
|
|
||||||
# aborts. Deleting an artist CASCADEs to Post (post.artist_id is
|
|
||||||
# ondelete=CASCADE), which SET NULLs post_attachment.post_id — and
|
|
||||||
# `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE
|
|
||||||
# WHERE post_id IS NULL, so any two of this artist's attachments sharing a
|
|
||||||
# sha collapse onto one another and raise. That is an ORDINARY shape, not a
|
|
||||||
# corrupt one: _capture_attachment deliberately writes one row per post over
|
|
||||||
# a single sha-addressed blob (a creator who attaches the same pdf to two
|
|
||||||
# posts has two rows), and a pre-existing filesystem-import row with the same
|
|
||||||
# sha and a NULL post_id collides on its own. Migration 0043 reasoned only
|
|
||||||
# about upgrade-time safety and never about this later SET NULL.
|
|
||||||
# _repoint_post_links guards the identical collision class in the reconcile
|
|
||||||
# path; this is its artist-cascade counterpart.
|
|
||||||
#
|
|
||||||
# Which rows count as the artist's — and why the blobs are left on disk —
|
|
||||||
# is _artist_attachments_conditions, shared with the preview.
|
|
||||||
attachments_deleted = session.execute(
|
|
||||||
delete(PostAttachment)
|
|
||||||
.where(*_artist_attachments_conditions(artist.id))
|
|
||||||
).rowcount or 0
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
session.delete(artist)
|
session.delete(artist)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
@@ -429,8 +333,6 @@ def delete_artist_cascade(
|
|||||||
"files_deleted": files_deleted,
|
"files_deleted": files_deleted,
|
||||||
"thumbs_deleted": thumbs_deleted,
|
"thumbs_deleted": thumbs_deleted,
|
||||||
"import_tasks_nulled": import_tasks_nulled,
|
"import_tasks_nulled": import_tasks_nulled,
|
||||||
"posts_deleted": posts_deleted,
|
|
||||||
"attachments_deleted": attachments_deleted,
|
|
||||||
"files_failed": files_failed,
|
"files_failed": files_failed,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1592,155 +1494,3 @@ def purge_gated_previews(
|
|||||||
"ledger_cleared": ledger_cleared,
|
"ledger_cleared": ledger_cleared,
|
||||||
"posts_deleted": posts_deleted,
|
"posts_deleted": posts_deleted,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
# -- orphaned attachment reclamation ---------------------------------------
|
|
||||||
# PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or
|
|
||||||
# artist leaves the row behind rather than taking it. Nothing ever pruned those
|
|
||||||
# rows, and nothing has ever unlinked a file under the attachment store — so
|
|
||||||
# both rows and bytes accumulated permanently and were invisible to every
|
|
||||||
# existing diagnostic.
|
|
||||||
#
|
|
||||||
# Why this is a DISK->DB reconciliation rather than a row sweep: the store is
|
|
||||||
# sha-addressed and idempotent (attachment_store.store), so ONE blob backs MANY
|
|
||||||
# rows. Deleting a row therefore does not free its blob, and — since the artist
|
|
||||||
# cascade now deletes its attachment rows outright — a freed blob has no DB
|
|
||||||
# pointer left to find it by. Walking the store and asking "does any row still
|
|
||||||
# reference this sha?" catches orphans from every cause, including ones no
|
|
||||||
# future delete path will think to report.
|
|
||||||
|
|
||||||
# A blob is written by attachment_store.store BEFORE its row is inserted and
|
|
||||||
# committed, so a just-stored file legitimately has no referencing row for a
|
|
||||||
# moment. Same guard, same reasoning as ORPHAN_TEMP_MIN_AGE_HOURS in
|
|
||||||
# tasks/maintenance.py: never judge a file younger than this.
|
|
||||||
_ATTACHMENT_ORPHAN_MIN_AGE_HOURS = 6
|
|
||||||
|
|
||||||
# Wall-clock budget for the store walk (rule 89). A library with a large
|
|
||||||
# attachment store shouldn't be able to run this past its soft time limit; on
|
|
||||||
# exhaustion it reports partial=True and the operator re-runs to finish.
|
|
||||||
_ATTACHMENT_RECLAIM_BUDGET_SECONDS = 900
|
|
||||||
|
|
||||||
# The store names files `<sha256><ext>`. Parse the sha as the first 64 chars
|
|
||||||
# rather than via Path.stem: store() takes the extension straight from the
|
|
||||||
# source filename, and a URL-encoded basename yields a multi-dot "suffix"
|
|
||||||
# (see [[reference_url_encoded_basename_suffix]]) that would make stem eat part
|
|
||||||
# of the sha. Validating the 64 chars as hex also skips anything else in the
|
|
||||||
# tree that isn't a stored blob.
|
|
||||||
_SHA256_HEX_LEN = 64
|
|
||||||
|
|
||||||
|
|
||||||
def _orphan_attachment_conditions() -> list:
|
|
||||||
"""PostAttachment rows belonging to nothing: both FKs nulled by a deleted
|
|
||||||
post AND a deleted artist. A row with post_id NULL but an artist_id is the
|
|
||||||
deliberate filesystem-import case (importer._capture_attachment writes it
|
|
||||||
that way) and is NOT an orphan — it is still attributed."""
|
|
||||||
return [
|
|
||||||
PostAttachment.post_id.is_(None),
|
|
||||||
PostAttachment.artist_id.is_(None),
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def _is_sha_named(name: str) -> bool:
|
|
||||||
"""True when `name` starts with a 64-char lowercase-hex sha256."""
|
|
||||||
if len(name) < _SHA256_HEX_LEN:
|
|
||||||
return False
|
|
||||||
head = name[:_SHA256_HEX_LEN]
|
|
||||||
return all(c in "0123456789abcdef" for c in head)
|
|
||||||
|
|
||||||
|
|
||||||
def reclaim_orphaned_attachments(
|
|
||||||
session: Session, *, images_root: Path, dry_run: bool = False,
|
|
||||||
) -> dict:
|
|
||||||
"""Prune unattributed PostAttachment rows, then unlink store blobs that no
|
|
||||||
surviving row references.
|
|
||||||
|
|
||||||
Returns (same discovery keys either way, so the UI renders one shape):
|
|
||||||
{"rows": int, # orphan rows found / deleted
|
|
||||||
"files": int, # unreferenced blobs found / unlinked
|
|
||||||
"bytes": int, # their total size
|
|
||||||
"scanned": int, # blobs examined
|
|
||||||
"skipped_recent": int, # blobs under the min-age guard
|
|
||||||
"files_failed": int, # unlink raised (apply only)
|
|
||||||
"partial": bool} # walk hit the time budget
|
|
||||||
|
|
||||||
dry_run computes exactly what the apply would do and mutates nothing — the
|
|
||||||
surviving-sha set is derived by NEGATING the same orphan predicate the
|
|
||||||
delete uses, so the preview cannot disagree with the apply (rule 93).
|
|
||||||
"""
|
|
||||||
started = time.monotonic()
|
|
||||||
orphan_conds = _orphan_attachment_conditions()
|
|
||||||
|
|
||||||
if dry_run:
|
|
||||||
rows = session.execute(
|
|
||||||
select(func.count(PostAttachment.id)).where(*orphan_conds)
|
|
||||||
).scalar_one()
|
|
||||||
else:
|
|
||||||
rows = session.execute(
|
|
||||||
delete(PostAttachment).where(*orphan_conds)
|
|
||||||
).rowcount or 0
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
# Shas that still have a home. In the apply path the orphan rows are already
|
|
||||||
# gone, so `NOT orphan` is redundant but harmless; in the dry-run path it is
|
|
||||||
# what makes the projection honest about blobs the delete would free. One
|
|
||||||
# predicate, one query, both modes.
|
|
||||||
surviving_shas = set(session.execute(
|
|
||||||
select(PostAttachment.sha256).where(~and_(*orphan_conds)).distinct()
|
|
||||||
).scalars())
|
|
||||||
|
|
||||||
root = Path(images_root) / "attachments"
|
|
||||||
cutoff = (
|
|
||||||
datetime.now(UTC).timestamp()
|
|
||||||
- _ATTACHMENT_ORPHAN_MIN_AGE_HOURS * 3600
|
|
||||||
)
|
|
||||||
files = 0
|
|
||||||
freed_bytes = 0
|
|
||||||
scanned = 0
|
|
||||||
skipped_recent = 0
|
|
||||||
files_failed = 0
|
|
||||||
partial = False
|
|
||||||
|
|
||||||
if root.is_dir():
|
|
||||||
for path in root.rglob("*"):
|
|
||||||
if time.monotonic() - started >= _ATTACHMENT_RECLAIM_BUDGET_SECONDS:
|
|
||||||
partial = True
|
|
||||||
break
|
|
||||||
# .partial staging files belong to cleanup_orphaned_temp_files —
|
|
||||||
# leave them alone rather than racing an in-flight store().
|
|
||||||
if path.suffix in (".part", ".partial") or not path.is_file():
|
|
||||||
continue
|
|
||||||
if not _is_sha_named(path.name):
|
|
||||||
continue
|
|
||||||
scanned += 1
|
|
||||||
sha = path.name[:_SHA256_HEX_LEN]
|
|
||||||
if sha in surviving_shas:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
st = path.stat()
|
|
||||||
if st.st_mtime >= cutoff:
|
|
||||||
skipped_recent += 1
|
|
||||||
continue
|
|
||||||
size = st.st_size
|
|
||||||
if not dry_run:
|
|
||||||
path.unlink()
|
|
||||||
files += 1
|
|
||||||
freed_bytes += size
|
|
||||||
except OSError as exc:
|
|
||||||
files_failed += 1
|
|
||||||
log.warning("reclaim_orphaned_attachments: %s: %s", path, exc)
|
|
||||||
|
|
||||||
if not dry_run and (rows or files):
|
|
||||||
log.info(
|
|
||||||
"attachment reclaim: %d orphan row(s) deleted, %d blob(s) unlinked "
|
|
||||||
"(%d bytes), %d failed, partial=%s",
|
|
||||||
rows, files, freed_bytes, files_failed, partial,
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"rows": rows,
|
|
||||||
"files": files,
|
|
||||||
"bytes": freed_bytes,
|
|
||||||
"scanned": scanned,
|
|
||||||
"skipped_recent": skipped_recent,
|
|
||||||
"files_failed": files_failed,
|
|
||||||
"partial": partial,
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -181,7 +181,7 @@ def _augment_cookies(platform: str, netscape: str) -> str:
|
|||||||
"""Delegate to the platform's `augment_cookies` hook if one is
|
"""Delegate to the platform's `augment_cookies` hook if one is
|
||||||
registered (subscribestar, hentaifoundry, etc. — see
|
registered (subscribestar, hentaifoundry, etc. — see
|
||||||
`services/platforms/<name>.py`). No-op when the platform doesn't
|
`services/platforms/<name>.py`). No-op when the platform doesn't
|
||||||
register a hook (Patreon, Discord). Centralizing the
|
register a hook (Patreon, DeviantArt). Centralizing the
|
||||||
quirks-per-platform in the platforms package means adding a new
|
quirks-per-platform in the platforms package means adding a new
|
||||||
platform's cookie quirks doesn't require touching this file."""
|
platform's cookie quirks doesn't require touching this file."""
|
||||||
info = PLATFORMS.get(platform)
|
info = PLATFORMS.get(platform)
|
||||||
|
|||||||
@@ -31,8 +31,9 @@ from .pixiv_ingester import PixivIngester
|
|||||||
from .subscribestar_ingester import SubscribeStarIngester
|
from .subscribestar_ingester import SubscribeStarIngester
|
||||||
|
|
||||||
# Platforms whose download + verify go through the native ingester rather than
|
# Platforms whose download + verify go through the native ingester rather than
|
||||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
|
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord,
|
||||||
# they migrate too.
|
# deviantart — the latter slated for retirement, not migration) until they
|
||||||
|
# migrate too.
|
||||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
|
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
|
||||||
|
|
||||||
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
||||||
|
|||||||
@@ -35,14 +35,9 @@ class InvalidUrlError(Exception):
|
|||||||
# reviewers catch drift.
|
# reviewers catch drift.
|
||||||
_PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
_PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||||
("patreon", re.compile(
|
("patreon", re.compile(
|
||||||
# Three creator URL shapes — bare (patreon.com/Atole), `c/`, and `cw/`
|
|
||||||
# (the "creator workspace" URL served once subscribed, see
|
|
||||||
# patreon_resolver._VANITY_RE). A trailing sub-path is allowed so a
|
|
||||||
# creator's inner page still derives the slug. Nav pages stay excluded.
|
|
||||||
r"^https?://(?:www\.)?patreon\.com/"
|
r"^https?://(?:www\.)?patreon\.com/"
|
||||||
r"(?:cw/|c/)?"
|
r"(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c/)"
|
||||||
r"(?!(?:home|search|messages|notifications|library|settings|posts)(?:[/?#]|$))"
|
r"(?P<slug>[^/?#]+)/?$",
|
||||||
r"(?P<slug>[^/?#]+)",
|
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
)),
|
)),
|
||||||
("subscribestar", re.compile(
|
("subscribestar", re.compile(
|
||||||
@@ -55,6 +50,12 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
|||||||
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
|
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
)),
|
)),
|
||||||
|
("deviantart", re.compile(
|
||||||
|
r"^https?://(?:www\.)?deviantart\.com/"
|
||||||
|
r"(?!home$|watch\b|tag\b|browse\b)"
|
||||||
|
r"(?P<slug>[^/?#]+)/?$",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)),
|
||||||
("pixiv", re.compile(
|
("pixiv", re.compile(
|
||||||
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
|
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
|
|||||||
@@ -299,9 +299,8 @@ class GalleryDLService:
|
|||||||
# (services/patreon_ingester.py), not gallery-dl.
|
# (services/patreon_ingester.py), not gallery-dl.
|
||||||
PLATFORM_DEFAULTS = {
|
PLATFORM_DEFAULTS = {
|
||||||
# subscribestar removed — native-ingester platform now (#71); pixiv
|
# subscribestar removed — native-ingester platform now (#71); pixiv
|
||||||
# removed likewise (#129); deviantart removed at #3069 as a dropped
|
# removed likewise (#129). The remaining entries are the gallery-dl
|
||||||
# platform, not a migrated one. The remaining entries are the
|
# platforms not yet migrated.
|
||||||
# gallery-dl platforms not yet migrated.
|
|
||||||
"hentaifoundry": {
|
"hentaifoundry": {
|
||||||
"content_types": ["all"],
|
"content_types": ["all"],
|
||||||
"directory": [],
|
"directory": [],
|
||||||
@@ -317,6 +316,15 @@ class GalleryDLService:
|
|||||||
"reactions": False,
|
"reactions": False,
|
||||||
"threads": True,
|
"threads": True,
|
||||||
},
|
},
|
||||||
|
"deviantart": {
|
||||||
|
"content_types": ["all"],
|
||||||
|
"directory": [],
|
||||||
|
"filename": "{index:>03}_{title[:50]}.{extension}",
|
||||||
|
"flat": True,
|
||||||
|
"original": True,
|
||||||
|
"mature": True,
|
||||||
|
"metadata": True,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ from ..models import (
|
|||||||
Tag,
|
Tag,
|
||||||
TagPositiveConfirmation,
|
TagPositiveConfirmation,
|
||||||
)
|
)
|
||||||
from ..models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
|
from ..models.tag import PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG, image_tag
|
||||||
from .pagination import decode_cursor, encode_cursor
|
from .pagination import decode_cursor, encode_cursor
|
||||||
from .tag_query import (
|
from .tag_query import (
|
||||||
fandom_join_alias,
|
fandom_join_alias,
|
||||||
@@ -396,25 +396,6 @@ def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
|
|||||||
return [kept[i] for i in order]
|
return [kept[i] for i in order]
|
||||||
|
|
||||||
|
|
||||||
def _reach_sample(rows, limit, reach):
|
|
||||||
"""From a distance-sorted candidate pool (nearest first), pick a spread of ranks
|
|
||||||
that MIXES near (tag the current cluster) and mid-far (escape it) BEFORE dedup +
|
|
||||||
MMR — the Explore "reach" dial (#1476).
|
|
||||||
|
|
||||||
reach in (0, 1]: the sampled span grows outward from the anchor (0.25→1.0 of the
|
|
||||||
pool), evenly strided from rank 0 so the nearest are still represented. In a
|
|
||||||
dense signature the nearest ranks are near-identical, so reaching farther is the
|
|
||||||
only way to hand MMR genuinely different content — MMR alone can't escape a pool
|
|
||||||
that's already all-near. reach<=0 or a small pool passes through unchanged."""
|
|
||||||
n = len(rows)
|
|
||||||
want = max(limit * 8, 100)
|
|
||||||
if reach <= 0 or n <= want:
|
|
||||||
return rows
|
|
||||||
span = int(min(1.0, 0.25 + 0.75 * reach) * n)
|
|
||||||
idx = sorted({min(int(i * span / want), n - 1) for i in range(want)})
|
|
||||||
return [rows[i] for i in idx]
|
|
||||||
|
|
||||||
|
|
||||||
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
||||||
"""Map image_id -> {"name","slug"} via the canonical
|
"""Map image_id -> {"name","slug"} via the canonical
|
||||||
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
||||||
@@ -438,17 +419,16 @@ class GalleryService:
|
|||||||
async def _hidden_tag_ids(
|
async def _hidden_tag_ids(
|
||||||
self, include_hidden, tag_ids, tag_or_groups,
|
self, include_hidden, tag_ids, tag_or_groups,
|
||||||
) -> list[int] | None:
|
) -> list[int] | None:
|
||||||
"""Chrome (banner) tag ids to implicitly exclude from a gallery query, or
|
"""Presentation-chrome tag ids to implicitly exclude from a gallery query,
|
||||||
None. None when the caller asked to include hidden, when the operator is
|
or None. None when the caller asked to include hidden, when the operator
|
||||||
explicitly filtering FOR a chrome tag (they clearly want to see it), or when
|
is explicitly filtering FOR a presentation tag (they clearly want to see
|
||||||
no chrome tags exist. (milestone 141; #1464: editor screenshot is now PROCESS
|
it), or when no presentation tags exist. (milestone 141)"""
|
||||||
— shown — so only `banner` hides here.)"""
|
|
||||||
if include_hidden:
|
if include_hidden:
|
||||||
return None
|
return None
|
||||||
rows = await self.session.execute(
|
rows = await self.session.execute(
|
||||||
select(Tag.id).where(
|
select(Tag.id).where(
|
||||||
Tag.is_system.is_(True),
|
Tag.is_system.is_(True),
|
||||||
Tag.name.in_(CHROME_SYSTEM_TAGS),
|
Tag.name.in_(PRESENTATION_SYSTEM_TAGS),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
pres = [r[0] for r in rows]
|
pres = [r[0] for r in rows]
|
||||||
@@ -736,7 +716,6 @@ class GalleryService:
|
|||||||
untagged: bool = False, no_artist: bool = False,
|
untagged: bool = False, no_artist: bool = False,
|
||||||
date_from: datetime | None = None, date_to: datetime | None = None,
|
date_from: datetime | None = None, date_to: datetime | None = None,
|
||||||
exclude_wip: bool = False,
|
exclude_wip: bool = False,
|
||||||
reach: float = 0.0, exclude_ids: list[int] | None = None,
|
|
||||||
) -> list[GalleryImage] | None:
|
) -> list[GalleryImage] | None:
|
||||||
"""Visual "more like this": images near `image_id`'s SigLIP embedding
|
"""Visual "more like this": images near `image_id`'s SigLIP embedding
|
||||||
(pgvector, HNSW-indexed — alembic 0036), then DIVERSIFIED so the result
|
(pgvector, HNSW-indexed — alembic 0036), then DIVERSIFIED so the result
|
||||||
@@ -765,27 +744,20 @@ class GalleryService:
|
|||||||
# wide pool there's nothing but the near-dupes to choose from. Widened
|
# wide pool there's nothing but the near-dupes to choose from. Widened
|
||||||
# (5×→8×, cap 200→400) so the stronger MMR has genuinely distinct
|
# (5×→8×, cap 200→400) so the stronger MMR has genuinely distinct
|
||||||
# neighbourhoods to reach into for more variance (operator, 2026-07-01).
|
# neighbourhoods to reach into for more variance (operator, 2026-07-01).
|
||||||
# Explore's reach>0 (#1476) widens it a LOT more: in a dense signature the
|
pool_n = min(400, max(limit * 8, 100))
|
||||||
# nearest few hundred are all near-identical, so far-enough candidates only
|
|
||||||
# exist deeper in the ranked pool. _reach_sample then strides across them.
|
|
||||||
if reach > 0:
|
|
||||||
pool_n = min(1000, max(limit * 25, 100))
|
|
||||||
else:
|
|
||||||
pool_n = min(400, max(limit * 8, 100))
|
|
||||||
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
|
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
|
||||||
eff = _effective_date_col()
|
eff = _effective_date_col()
|
||||||
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
||||||
stmt = _outer_join_primary_post(stmt)
|
stmt = _outer_join_primary_post(stmt)
|
||||||
# Chrome (banner, #128) clusters on UI rather than content, so near any one
|
# Presentation images (banner / editor-screenshot system tags, #128)
|
||||||
# of them they'd fill the grid → excluded from CANDIDATES always (the anchor
|
# cluster on UI chrome rather than content, so near any one of them
|
||||||
# itself may be a banner). PROCESS art (wip / editor screenshot) stays
|
# they'd fill the grid. Excluded from CANDIDATES only — the anchor
|
||||||
# surfaced here by default (real content; only the training pipelines exclude
|
# itself may be a banner. `wip` stays surfaced here by default (real art;
|
||||||
# it), but the Explore rabbit-hole passes exclude_wip to also drop the whole
|
# only the training pipelines exclude it), but the Explore rabbit-hole
|
||||||
# process group so a browse doesn't keep surfacing work-in-progress
|
# passes exclude_wip to also drop work-in-progress (operator, 2026-07-08).
|
||||||
# (operator, 2026-07-08; #1464 — editor now rides with wip here).
|
excluded_system_tags = PRESENTATION_SYSTEM_TAGS
|
||||||
excluded_system_tags = CHROME_SYSTEM_TAGS
|
|
||||||
if exclude_wip:
|
if exclude_wip:
|
||||||
excluded_system_tags = (*CHROME_SYSTEM_TAGS, *PROCESS_SYSTEM_TAGS)
|
excluded_system_tags = (*PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG)
|
||||||
presentation = (
|
presentation = (
|
||||||
select(image_tag.c.image_record_id)
|
select(image_tag.c.image_record_id)
|
||||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||||
@@ -799,10 +771,6 @@ class GalleryService:
|
|||||||
ImageRecord.id != image_id,
|
ImageRecord.id != image_id,
|
||||||
ImageRecord.id.not_in(presentation),
|
ImageRecord.id.not_in(presentation),
|
||||||
)
|
)
|
||||||
# Anti-revisit (#1476): the Explore walk passes its breadcrumb so already-
|
|
||||||
# walked images aren't re-served as neighbours — → can't loop you back in.
|
|
||||||
if exclude_ids:
|
|
||||||
stmt = stmt.where(ImageRecord.id.not_in(exclude_ids))
|
|
||||||
stmt = _apply_scope(
|
stmt = _apply_scope(
|
||||||
stmt, tag_ids=tag_ids, post_id=None,
|
stmt, tag_ids=tag_ids, post_id=None,
|
||||||
artist_id=artist_id, media_type=media_type,
|
artist_id=artist_id, media_type=media_type,
|
||||||
@@ -812,10 +780,6 @@ class GalleryService:
|
|||||||
)
|
)
|
||||||
stmt = stmt.order_by(distance.asc()).limit(pool_n)
|
stmt = stmt.order_by(distance.asc()).limit(pool_n)
|
||||||
rows = (await self.session.execute(stmt)).all()
|
rows = (await self.session.execute(stmt)).all()
|
||||||
# Explore reach: stride across an outward-growing distance span so the pool
|
|
||||||
# handed to MMR spans near→mid-far, not just the tight cluster (#1476).
|
|
||||||
if reach > 0:
|
|
||||||
rows = _reach_sample(rows, limit, reach)
|
|
||||||
rows = _diversify_similar(src, rows, limit)
|
rows = _diversify_similar(src, rows, limit)
|
||||||
artists = await _artists_for(self.session, [r[0].id for r in rows])
|
artists = await _artists_for(self.session, [r[0].id for r in rows])
|
||||||
return _gallery_images(rows, artists)
|
return _gallery_images(rows, artists)
|
||||||
|
|||||||
@@ -1,51 +0,0 @@
|
|||||||
"""Bulk, idempotent writes to the ``image_tag`` association table.
|
|
||||||
|
|
||||||
Three writers attach tags to images in bulk: the WIP-title backfill
|
|
||||||
(`wip_title.apply_wip_image_tags`), the concept-head auto-apply sweep and the
|
|
||||||
system-tag auto-apply sweep (both in `ml/heads.py`). The two sweeps used to
|
|
||||||
issue ONE INSERT PER ROW from inside their per-image loop — fine in steady
|
|
||||||
state, but a first pass over a back-catalogue is tens of thousands of
|
|
||||||
individual round-trips (#3072). All three share this one chunked multi-row
|
|
||||||
insert now.
|
|
||||||
|
|
||||||
Sync only: every caller runs on a sync ``Session`` (the Celery task path). No
|
|
||||||
async service writes image_tag in bulk, so there is no async sibling to keep in
|
|
||||||
step — unlike `db_helpers.get_or_create`, which does have one.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from ..models.tag import image_tag
|
|
||||||
|
|
||||||
# 5000 rows x 3 bound params = 15000, comfortably inside Postgres' 65535-param
|
|
||||||
# ceiling for a single statement. Raising this past ~21000 rows would exceed it.
|
|
||||||
INSERT_CHUNK = 5000
|
|
||||||
|
|
||||||
|
|
||||||
def insert_image_tags(
|
|
||||||
session: Session, rows: list[dict], *, chunk: int = INSERT_CHUNK
|
|
||||||
) -> None:
|
|
||||||
"""Attach ``rows`` to their images, skipping any tag already on one.
|
|
||||||
|
|
||||||
Each row is ``{"image_record_id": int, "tag_id": int, "source": str}``.
|
|
||||||
Does NOT commit — the caller owns the transaction.
|
|
||||||
|
|
||||||
ON CONFLICT DO NOTHING against the (image_record_id, tag_id) primary key,
|
|
||||||
so an existing tag keeps its ORIGINAL ``source``: re-running a sweep can
|
|
||||||
never re-stamp a tag the operator applied by hand as machine-applied.
|
|
||||||
|
|
||||||
Returns nothing on purpose. psycopg reports ``rowcount`` -1 for a multi-row
|
|
||||||
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so a count
|
|
||||||
taken from the statement would be a lie rather than an approximation.
|
|
||||||
Callers that need an accurate count derive it themselves — see
|
|
||||||
`wip_title.apply_wip_image_tags`' pre-SELECT, and the sweeps' `skip` sets.
|
|
||||||
"""
|
|
||||||
for start in range(0, len(rows), chunk):
|
|
||||||
session.execute(
|
|
||||||
pg_insert(image_tag)
|
|
||||||
.values(rows[start:start + chunk])
|
|
||||||
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
|
|
||||||
)
|
|
||||||
@@ -47,21 +47,9 @@ from .attachment_store import AttachmentStore
|
|||||||
from .audits import single_color
|
from .audits import single_color
|
||||||
from .link_extract import extract_external_links
|
from .link_extract import extract_external_links
|
||||||
from .thumbnailer import Thumbnailer
|
from .thumbnailer import Thumbnailer
|
||||||
from .wip_title import (
|
|
||||||
WIP_TITLE_SOFT_SOURCE,
|
|
||||||
WIP_TITLE_SOURCE,
|
|
||||||
apply_wip_image_tags,
|
|
||||||
matches_soft_wip_title,
|
|
||||||
matches_wip_title,
|
|
||||||
resolve_wip_tag_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
# Sentinel for the lazily-resolved wip tag id (distinguishes "not resolved yet"
|
|
||||||
# from a genuine None = tag absent, so absence is cached and not re-queried).
|
|
||||||
_UNSET = object()
|
|
||||||
|
|
||||||
|
|
||||||
class SkipReason(StrEnum):
|
class SkipReason(StrEnum):
|
||||||
too_small = "too_small"
|
too_small = "too_small"
|
||||||
@@ -195,10 +183,6 @@ class Importer:
|
|||||||
# invalidated mid-Importer (Importer instances are per-task /
|
# invalidated mid-Importer (Importer instances are per-task /
|
||||||
# per-archive-import so cross-instance staleness is harmless).
|
# per-archive-import so cross-instance staleness is harmless).
|
||||||
self._phash_candidates: list[tuple] | None = None
|
self._phash_candidates: list[tuple] | None = None
|
||||||
# Lazily-resolved `wip` system tag id for title-based WIP auto-tagging
|
|
||||||
# (task #1458). Sentinel _UNSET so a genuine None (tag absent) is cached
|
|
||||||
# and not re-queried per media. Importer is per-task, so this can't stale.
|
|
||||||
self._wip_tag_id: int | None = _UNSET
|
|
||||||
|
|
||||||
def _phash_candidates_cache(self) -> list[tuple]:
|
def _phash_candidates_cache(self) -> list[tuple]:
|
||||||
"""Cached `(phash, width, height, id)` rows from image_record.
|
"""Cached `(phash, width, height, id)` rows from image_record.
|
||||||
@@ -949,10 +933,6 @@ class Importer:
|
|||||||
# Thumbnail is queued separately by the calling task; the importer
|
# Thumbnail is queued separately by the calling task; the importer
|
||||||
# does not generate thumbnails inline so the import queue stays moving.
|
# does not generate thumbnails inline so the import queue stays moving.
|
||||||
|
|
||||||
# Title-based WIP auto-tag (task #1458): fresh import only, after the
|
|
||||||
# sidecar has linked the post so record.primary_post_id / its title exist.
|
|
||||||
self._maybe_apply_wip_title(record)
|
|
||||||
|
|
||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="imported", image_id=record.id)
|
return ImportResult(status="imported", image_id=record.id)
|
||||||
|
|
||||||
@@ -996,47 +976,6 @@ class Importer:
|
|||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="refreshed", image_id=existing.id)
|
return ImportResult(status="refreshed", image_id=existing.id)
|
||||||
|
|
||||||
def _maybe_apply_wip_title(self, record: ImageRecord) -> None:
|
|
||||||
"""Auto-apply the `wip` system tag to a FRESHLY-imported image when its
|
|
||||||
primary post's TITLE explicitly declares work-in-progress (task #1458 —
|
|
||||||
the artist's own "WIP" / "work in progress" label).
|
|
||||||
|
|
||||||
Called ONLY from the two new-record paths (never deep-scan / supersede),
|
|
||||||
so a manually-removed WIP tag is never re-applied by a routine re-scan —
|
|
||||||
removal sticks. The existing catalogue is covered separately by the
|
|
||||||
operator-triggered backfill sweep. Gated by the settings toggle, and
|
|
||||||
best-effort: any failure is logged, never allowed to fail the import."""
|
|
||||||
hard_on = self.settings.wip_title_tagging_enabled
|
|
||||||
soft_on = self.settings.wip_soft_title_tagging_enabled
|
|
||||||
if not (hard_on or soft_on):
|
|
||||||
return
|
|
||||||
if record.primary_post_id is None:
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
title = self.session.execute(
|
|
||||||
select(Post.post_title).where(Post.id == record.primary_post_id)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
# HARD tier ("WIP"/"work in progress") wins — higher precision, and it
|
|
||||||
# trains the head; SOFT (sketch/doodle, #1474) is the provisional fallback
|
|
||||||
# that never trains (source wip_title_soft).
|
|
||||||
if hard_on and matches_wip_title(title):
|
|
||||||
source = WIP_TITLE_SOURCE
|
|
||||||
elif soft_on and matches_soft_wip_title(title):
|
|
||||||
source = WIP_TITLE_SOFT_SOURCE
|
|
||||||
else:
|
|
||||||
return
|
|
||||||
if self._wip_tag_id is _UNSET:
|
|
||||||
self._wip_tag_id = resolve_wip_tag_id(self.session)
|
|
||||||
if self._wip_tag_id is None:
|
|
||||||
return
|
|
||||||
apply_wip_image_tags(
|
|
||||||
self.session, [record.id], self._wip_tag_id, source=source
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001 — a tag must never fail an import
|
|
||||||
log.warning(
|
|
||||||
"wip-title auto-tag failed for image %s: %s", record.id, exc
|
|
||||||
)
|
|
||||||
|
|
||||||
def _apply_post_fields(self, post: Post, sd) -> None:
|
def _apply_post_fields(self, post: Post, sd) -> None:
|
||||||
"""Write a parsed sidecar's post-level fields onto a Post — the SINGLE
|
"""Write a parsed sidecar's post-level fields onto a Post — the SINGLE
|
||||||
predicate shared by BOTH ingest paths: the per-media path (_apply_sidecar)
|
predicate shared by BOTH ingest paths: the per-media path (_apply_sidecar)
|
||||||
@@ -1314,10 +1253,6 @@ class Importer:
|
|||||||
# per-post Source row.
|
# per-post Source row.
|
||||||
self._apply_sidecar(record, path, artist, explicit_source=source)
|
self._apply_sidecar(record, path, artist, explicit_source=source)
|
||||||
|
|
||||||
# Title-based WIP auto-tag (task #1458): fresh import only, see the
|
|
||||||
# matching call in _import_media.
|
|
||||||
self._maybe_apply_wip_title(record)
|
|
||||||
|
|
||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="imported", image_id=record.id)
|
return ImportResult(status="imported", image_id=record.id)
|
||||||
|
|
||||||
|
|||||||
@@ -12,11 +12,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||||||
|
|
||||||
from ...models import TagSuggestionRejection
|
from ...models import TagSuggestionRejection
|
||||||
from ...models.tag import image_tag
|
from ...models.tag import image_tag
|
||||||
|
from .aliases import AliasService
|
||||||
|
|
||||||
|
|
||||||
class AllowlistService:
|
class AllowlistService:
|
||||||
def __init__(self, session: AsyncSession):
|
def __init__(self, session: AsyncSession):
|
||||||
self.session = session
|
self.session = session
|
||||||
|
self.aliases = AliasService(session)
|
||||||
|
|
||||||
async def _apply_image_tag(self, image_id: int, tag_id: int, source: str):
|
async def _apply_image_tag(self, image_id: int, tag_id: int, source: str):
|
||||||
stmt = insert(image_tag).values(
|
stmt = insert(image_tag).values(
|
||||||
@@ -39,6 +41,18 @@ class AllowlistService:
|
|||||||
await self._apply_image_tag(image_id, tag_id, source="ml_accepted")
|
await self._apply_image_tag(image_id, tag_id, source="ml_accepted")
|
||||||
await self._clear_rejection(image_id, tag_id)
|
await self._clear_rejection(image_id, tag_id)
|
||||||
|
|
||||||
|
async def add_alias_and_accept(
|
||||||
|
self,
|
||||||
|
image_id: int,
|
||||||
|
alias_string: str,
|
||||||
|
alias_category: str,
|
||||||
|
canonical_tag_id: int,
|
||||||
|
) -> None:
|
||||||
|
await self.aliases.create(
|
||||||
|
alias_string, alias_category, canonical_tag_id
|
||||||
|
)
|
||||||
|
await self.accept(image_id, canonical_tag_id)
|
||||||
|
|
||||||
async def dismiss(self, image_id: int, tag_id: int) -> None:
|
async def dismiss(self, image_id: int, tag_id: int) -> None:
|
||||||
stmt = insert(TagSuggestionRejection).values(
|
stmt = insert(TagSuggestionRejection).values(
|
||||||
image_record_id=image_id, tag_id=tag_id
|
image_record_id=image_id, tag_id=tag_id
|
||||||
|
|||||||
@@ -150,7 +150,9 @@ def refresh_character_prototypes(
|
|||||||
"""Incrementally refresh the prototype store. `full=True` rebuilds every
|
"""Incrementally refresh the prototype store. `full=True` rebuilds every
|
||||||
character regardless of the gate/fingerprints (nightly reconcile). Returns
|
character regardless of the gate/fingerprints (nightly reconcile). Returns
|
||||||
{skipped, rebuilt, removed}; commits."""
|
{skipped, rebuilt, removed}; commits."""
|
||||||
settings = MLSettings.load_sync(session)
|
settings = session.execute(
|
||||||
|
select(MLSettings).where(MLSettings.id == 1)
|
||||||
|
).scalar_one()
|
||||||
sig = _global_signature(session)
|
sig = _global_signature(session)
|
||||||
if not full and settings.ccip_ref_signature == sig:
|
if not full and settings.ccip_ref_signature == sig:
|
||||||
return {"skipped": True, "rebuilt": 0, "removed": 0}
|
return {"skipped": True, "rebuilt": 0, "removed": 0}
|
||||||
@@ -202,7 +204,9 @@ def retract_auto_applied_ccip(session: Session) -> int:
|
|||||||
n_retracted."""
|
n_retracted."""
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
settings = MLSettings.load_sync(session)
|
settings = session.execute(
|
||||||
|
select(MLSettings).where(MLSettings.id == 1)
|
||||||
|
).scalar_one()
|
||||||
if not settings.ccip_auto_apply_enabled:
|
if not settings.ccip_auto_apply_enabled:
|
||||||
return 0
|
return 0
|
||||||
thr = float(settings.ccip_auto_apply_threshold)
|
thr = float(settings.ccip_auto_apply_threshold)
|
||||||
|
|||||||
@@ -23,7 +23,6 @@ from datetime import UTC, datetime
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy import delete, exists, func, select
|
from sqlalchemy import delete, exists, func, select
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
@@ -40,11 +39,9 @@ from ...models import (
|
|||||||
TagPositiveConfirmation,
|
TagPositiveConfirmation,
|
||||||
TagSuggestionRejection,
|
TagSuggestionRejection,
|
||||||
)
|
)
|
||||||
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
|
from ...models.tag import PRESENTATION_SYSTEM_TAGS, image_tag
|
||||||
from ..image_tag_apply import insert_image_tags
|
|
||||||
from .training_data import (
|
from .training_data import (
|
||||||
_AUTO_SOURCES,
|
_AUTO_SOURCES,
|
||||||
_applied_or_rejected,
|
|
||||||
_auto_apply_point,
|
_auto_apply_point,
|
||||||
_hygiene_excluded_ids,
|
_hygiene_excluded_ids,
|
||||||
_ids_with_tag,
|
_ids_with_tag,
|
||||||
@@ -64,14 +61,6 @@ MIN_POSITIVES_FLOOR = 8 # hard floor; settings.head_min_positives can raise
|
|||||||
_UNLABELED_POOL = 4000
|
_UNLABELED_POOL = 4000
|
||||||
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
|
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
|
||||||
|
|
||||||
# Auto-apply / match confidence operating range. Every graduated auto-apply or
|
|
||||||
# CCIP-match threshold the operator can set lives in this band, and the head
|
|
||||||
# precision target is clamped to it: below 0.5 "auto-apply" is meaningless, and
|
|
||||||
# 1.0 is unachievable so 0.999 is the ceiling. One source shared by the service
|
|
||||||
# clamp (_normalize_params) and the API validator (ml_admin._validate).
|
|
||||||
AUTO_APPLY_THRESHOLD_MIN = 0.5
|
|
||||||
AUTO_APPLY_THRESHOLD_MAX = 0.999
|
|
||||||
|
|
||||||
# Only these tag kinds get heads (the surfaced suggestion categories).
|
# Only these tag kinds get heads (the surfaced suggestion categories).
|
||||||
_HEAD_KINDS = (TagKind.general, TagKind.character)
|
_HEAD_KINDS = (TagKind.general, TagKind.character)
|
||||||
# tag.kind -> the suggestion category the rail groups under.
|
# tag.kind -> the suggestion category the rail groups under.
|
||||||
@@ -89,38 +78,6 @@ _CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
|
|||||||
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65
|
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65
|
||||||
|
|
||||||
|
|
||||||
def _sigmoid(z, np):
|
|
||||||
"""Logistic sigmoid 1/(1+e^-z): the head score→probability transform. One home
|
|
||||||
for what was inlined at every scoring site (suggest, both sweeps, retract)."""
|
|
||||||
return 1.0 / (1.0 + np.exp(-z))
|
|
||||||
|
|
||||||
|
|
||||||
def _conflict_scores(Xn, Wc, bc, np):
|
|
||||||
"""The presentation conflict signal (#141): per row, the MAX content-head
|
|
||||||
probability and WHICH head produced it. Shared by the system-tag sweep's guard-2
|
|
||||||
and the soft-wip audit — both ask "does this ALSO look like real content?"."""
|
|
||||||
cprobs = _sigmoid(Xn @ Wc.T + bc, np)
|
|
||||||
return cprobs.max(axis=1), cprobs.argmax(axis=1)
|
|
||||||
|
|
||||||
|
|
||||||
def _insert_presentation_review(
|
|
||||||
session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode,
|
|
||||||
):
|
|
||||||
"""Single-source the ring-loud PresentationReview row shape so the two writers
|
|
||||||
(system-tag sweep guard-2 + soft-wip audit) can't drift on columns or `mode` —
|
|
||||||
they share the (image_record_id, tag_id) composite PK, so a divergent `mode`
|
|
||||||
would be a silent first-writer-wins bug."""
|
|
||||||
session.execute(
|
|
||||||
pg_insert(PresentationReview)
|
|
||||||
.values(
|
|
||||||
image_record_id=image_record_id, tag_id=tag_id,
|
|
||||||
conflict_tag_id=conflict_tag_id, conflict_score=conflict_score,
|
|
||||||
mode=mode,
|
|
||||||
)
|
|
||||||
.on_conflict_do_nothing()
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class HeadTrainingAlreadyRunning(Exception):
|
class HeadTrainingAlreadyRunning(Exception):
|
||||||
"""Raised by start_head_training_run when a run is already in flight."""
|
"""Raised by start_head_training_run when a run is already in flight."""
|
||||||
|
|
||||||
@@ -146,7 +103,9 @@ def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _settings(session: Session) -> MLSettings:
|
def _settings(session: Session) -> MLSettings:
|
||||||
return MLSettings.load_sync(session)
|
return session.execute(
|
||||||
|
select(MLSettings).where(MLSettings.id == 1)
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
|
|
||||||
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
|
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||||
@@ -165,7 +124,7 @@ def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[s
|
|||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
cv_folds = DEFAULT_CV_FOLDS
|
cv_folds = DEFAULT_CV_FOLDS
|
||||||
try:
|
try:
|
||||||
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), AUTO_APPLY_THRESHOLD_MIN), AUTO_APPLY_THRESHOLD_MAX)
|
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999)
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
precision_target = s.head_auto_apply_precision
|
precision_target = s.head_auto_apply_precision
|
||||||
return {
|
return {
|
||||||
@@ -541,19 +500,13 @@ async def score_image(
|
|||||||
session: AsyncSession, image_id: int, threshold_override: float | None = None,
|
session: AsyncSession, image_id: int, threshold_override: float | None = None,
|
||||||
) -> list[dict]:
|
) -> list[dict]:
|
||||||
"""Suggestions for one image from the trained heads: [{tag_id, name,
|
"""Suggestions for one image from the trained heads: [{tag_id, name,
|
||||||
category, score, above_threshold, grounding}], ranked. A concept is INCLUDED
|
category, score}], ranked. A concept surfaces when its score clears the
|
||||||
when its score clears the head's own suggest_threshold — or, when
|
head's own suggest_threshold — or, when threshold_override is given (the
|
||||||
threshold_override is given (the typed-dropdown "show everything" mode), that
|
typed-dropdown "show everything" mode), that flat floor instead (0 → every
|
||||||
flat floor instead (0 → every head). System-tag heads (wip/banner/editor)
|
head). System-tag heads (wip/banner/editor) instead use a flat
|
||||||
instead use a flat _SYSTEM_TAG_SUGGEST_FLOOR so their false positives surface
|
_SYSTEM_TAG_SUGGEST_FLOOR so their false positives surface for rejection
|
||||||
for rejection (still overridden by threshold_override). Empty if the image has
|
(still overridden by threshold_override). Empty if the image has no
|
||||||
no embedding or no heads exist yet.
|
embedding or no heads exist yet.
|
||||||
|
|
||||||
``above_threshold`` is reported SEPARATELY from inclusion: it's always whether
|
|
||||||
the score cleared the head's NATURAL cut (suggest_threshold, or the system
|
|
||||||
floor), regardless of any override. So the single min=0 fetch returns every
|
|
||||||
head, and the caller can split panel (above_threshold) from dropdown (all)
|
|
||||||
without a second request.
|
|
||||||
|
|
||||||
MAX-OVER-BAG: the image is scored as a BAG of embeddings — the whole-image
|
MAX-OVER-BAG: the image is scored as a BAG of embeddings — the whole-image
|
||||||
vector PLUS every concept-region crop the agent embedded (same model
|
vector PLUS every concept-region crop the agent embedded (same model
|
||||||
@@ -577,32 +530,28 @@ async def score_image(
|
|||||||
norms[norms == 0] = 1.0
|
norms[norms == 0] = 1.0
|
||||||
Xn = X / norms
|
Xn = X / norms
|
||||||
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
|
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
|
||||||
probs_bag = _sigmoid(Z, np) # (B, H)
|
probs_bag = 1.0 / (1.0 + np.exp(-Z)) # (B, H)
|
||||||
probs = probs_bag.max(axis=0) # (H,) best over the bag
|
probs = probs_bag.max(axis=0) # (H,) best over the bag
|
||||||
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds
|
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds
|
||||||
# the tag (bag_meta[win]); None when the whole-image vector won (#1206).
|
# the tag (bag_meta[win]); None when the whole-image vector won (#1206).
|
||||||
winners = probs_bag.argmax(axis=0) # (H,)
|
winners = probs_bag.argmax(axis=0) # (H,)
|
||||||
out = []
|
out = []
|
||||||
for i, p in enumerate(probs):
|
for i, p in enumerate(probs):
|
||||||
m = heads["meta"][i]
|
if threshold_override is not None:
|
||||||
# The head's NATURAL suggest cut — system tags use the flat floor (see
|
cut = threshold_override
|
||||||
# _SYSTEM_TAG_SUGGEST_FLOOR) so their false positives show up for the
|
elif heads["meta"][i]["is_system"]:
|
||||||
# operator to reject; content heads use their own precision-tuned
|
# System tags surface at the flat floor (see _SYSTEM_TAG_SUGGEST_FLOOR)
|
||||||
# threshold. This is what "above threshold" means (drives the panel).
|
# so their false positives show up for the operator to reject.
|
||||||
natural = (
|
cut = _SYSTEM_TAG_SUGGEST_FLOOR
|
||||||
_SYSTEM_TAG_SUGGEST_FLOOR if m["is_system"] else float(heads["thr"][i])
|
else:
|
||||||
)
|
cut = heads["thr"][i]
|
||||||
# INCLUSION is looser under threshold_override (dropdown show-all,
|
|
||||||
# override=0): every head comes back so a low-confidence concept can still
|
|
||||||
# be typed + picked, each carrying its own above_threshold flag.
|
|
||||||
cut = threshold_override if threshold_override is not None else natural
|
|
||||||
if p >= cut:
|
if p >= cut:
|
||||||
|
m = heads["meta"][i]
|
||||||
out.append({
|
out.append({
|
||||||
"tag_id": m["tag_id"],
|
"tag_id": m["tag_id"],
|
||||||
"name": m["name"],
|
"name": m["name"],
|
||||||
"category": m["category"],
|
"category": m["category"],
|
||||||
"score": float(p),
|
"score": float(p),
|
||||||
"above_threshold": bool(p >= natural),
|
|
||||||
"grounding": bag_meta[int(winners[i])],
|
"grounding": bag_meta[int(winners[i])],
|
||||||
})
|
})
|
||||||
out.sort(key=lambda d: d["score"], reverse=True)
|
out.sort(key=lambda d: d["score"], reverse=True)
|
||||||
@@ -655,7 +604,9 @@ async def ground_applied_tag(
|
|||||||
|
|
||||||
|
|
||||||
async def _settings_async(session: AsyncSession) -> MLSettings:
|
async def _settings_async(session: AsyncSession) -> MLSettings:
|
||||||
return await MLSettings.load(session)
|
return (
|
||||||
|
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
|
|
||||||
# --- Earned auto-apply (sync, ml worker) ---------------------------------
|
# --- Earned auto-apply (sync, ml worker) ---------------------------------
|
||||||
@@ -726,6 +677,7 @@ def auto_apply_sweep(
|
|||||||
embeddings in chunks; commits per chunk on a real run. Returns
|
embeddings in chunks; commits per chunk on a real run. Returns
|
||||||
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
|
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
|
||||||
settings = _settings(session)
|
settings = _settings(session)
|
||||||
rows = _auto_apply_heads(
|
rows = _auto_apply_heads(
|
||||||
@@ -742,7 +694,18 @@ def auto_apply_sweep(
|
|||||||
names = [r.name for r in rows]
|
names = [r.name for r in rows]
|
||||||
|
|
||||||
# Skip images that already carry, or have rejected, each tag.
|
# Skip images that already carry, or have rejected, each tag.
|
||||||
skip = _applied_or_rejected(session, tag_ids)
|
skip = {tid: set() for tid in tag_ids}
|
||||||
|
for tid in tag_ids:
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
||||||
|
):
|
||||||
|
skip[tid].add(iid)
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
select(TagSuggestionRejection.image_record_id).where(
|
||||||
|
TagSuggestionRejection.tag_id == tid
|
||||||
|
)
|
||||||
|
):
|
||||||
|
skip[tid].add(iid)
|
||||||
|
|
||||||
applied = [0] * len(rows)
|
applied = [0] * len(rows)
|
||||||
scanned = 0
|
scanned = 0
|
||||||
@@ -756,12 +719,8 @@ def auto_apply_sweep(
|
|||||||
if not cids:
|
if not cids:
|
||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
|
probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b))) # (N, H)
|
||||||
scanned += len(cids)
|
scanned += len(cids)
|
||||||
# Collected across every head, then written as ONE insert below. Was an
|
|
||||||
# insert per applied tag from inside this loop, which on a first sweep
|
|
||||||
# over a back-catalogue is tens of thousands of round-trips (#3072).
|
|
||||||
pending: list[dict] = []
|
|
||||||
for h in range(len(rows)):
|
for h in range(len(rows)):
|
||||||
tid = tag_ids[h]
|
tid = tag_ids[h]
|
||||||
for idx in np.where(probs[:, h] >= thr[h])[0]:
|
for idx in np.where(probs[:, h] >= thr[h])[0]:
|
||||||
@@ -771,12 +730,12 @@ def auto_apply_sweep(
|
|||||||
skip[tid].add(iid)
|
skip[tid].add(iid)
|
||||||
applied[h] += 1
|
applied[h] += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
pending.append({
|
session.execute(
|
||||||
"image_record_id": iid, "tag_id": tid,
|
pg_insert(image_tag)
|
||||||
"source": "head_auto",
|
.values(image_record_id=iid, tag_id=tid, source="head_auto")
|
||||||
})
|
.on_conflict_do_nothing()
|
||||||
|
)
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
insert_image_tags(session, pending)
|
|
||||||
session.commit()
|
session.commit()
|
||||||
run.last_progress_at = datetime.now(UTC)
|
run.last_progress_at = datetime.now(UTC)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -790,42 +749,18 @@ def auto_apply_sweep(
|
|||||||
|
|
||||||
|
|
||||||
_PRESENTATION_SOURCE = "presentation_auto"
|
_PRESENTATION_SOURCE = "presentation_auto"
|
||||||
_PROCESS_SOURCE = "process_auto"
|
|
||||||
|
|
||||||
# System-tag auto-apply modes (#1464). Both modes run the identical sweep — apply
|
|
||||||
# a system tag at a flat threshold with a PROVISIONAL source + a ring-loud review
|
|
||||||
# guard — and differ ONLY in which tags, which settings knobs, and which
|
|
||||||
# source/review-mode. 'chrome' (banner) is HIDDEN from the gallery; 'process'
|
|
||||||
# (wip / editor screenshot) stays VISIBLE (the hide is a gallery-query effect of
|
|
||||||
# the tag's group membership, not of this sweep).
|
|
||||||
_SWEEP_MODES = {
|
|
||||||
"chrome": {
|
|
||||||
"names": CHROME_SYSTEM_TAGS,
|
|
||||||
"enabled": "presentation_auto_apply_enabled",
|
|
||||||
"threshold": "presentation_auto_apply_threshold",
|
|
||||||
"conflict": "presentation_conflict_threshold",
|
|
||||||
"source": _PRESENTATION_SOURCE,
|
|
||||||
},
|
|
||||||
"process": {
|
|
||||||
"names": PROCESS_SYSTEM_TAGS,
|
|
||||||
"enabled": "process_auto_apply_enabled",
|
|
||||||
"threshold": "process_auto_apply_threshold",
|
|
||||||
"conflict": "process_conflict_threshold",
|
|
||||||
"source": _PROCESS_SOURCE,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _system_tag_heads(session: Session, embedding_version: str, names):
|
def _presentation_heads(session: Session, embedding_version: str):
|
||||||
"""Trained heads for a system-tag group (chrome banner / process wip+editor).
|
"""Trained heads for the presentation chrome tags (banner / editor screenshot).
|
||||||
They fire at the group's FLAT threshold regardless of graduation — a head
|
They fire at the FLAT presentation threshold regardless of graduation — a head
|
||||||
exists once the operator has labelled enough (head_min_positives)."""
|
exists once the operator has labelled enough chrome (head_min_positives)."""
|
||||||
return session.execute(
|
return session.execute(
|
||||||
select(TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias)
|
select(TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias)
|
||||||
.join(Tag, Tag.id == TagHead.tag_id)
|
.join(Tag, Tag.id == TagHead.tag_id)
|
||||||
.where(TagHead.embedding_version == embedding_version)
|
.where(TagHead.embedding_version == embedding_version)
|
||||||
.where(Tag.is_system.is_(True))
|
.where(Tag.is_system.is_(True))
|
||||||
.where(Tag.name.in_(names))
|
.where(Tag.name.in_(PRESENTATION_SYSTEM_TAGS))
|
||||||
).all()
|
).all()
|
||||||
|
|
||||||
|
|
||||||
@@ -857,32 +792,27 @@ def _valued_image_ids(session: Session) -> set[int]:
|
|||||||
return {r[0] for r in rows}
|
return {r[0] for r in rows}
|
||||||
|
|
||||||
|
|
||||||
def system_tag_auto_apply_sweep(
|
def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> dict:
|
||||||
session: Session, *, mode: str, dry_run: bool = False
|
"""Auto-hide presentation chrome (banner / editor screenshot) at the FLAT
|
||||||
) -> dict:
|
presentation threshold (#141) — NOT the per-head graduated threshold. Two
|
||||||
"""Auto-apply a system-tag group at its FLAT threshold. mode='chrome' (banner,
|
guards keep it safe: (1) never hide an image carrying a human/confirmed content
|
||||||
#141) hides the image; mode='process' (wip / editor screenshot, #1464) keeps it
|
tag; (2) if an image about to be hidden ALSO scores >= the conflict threshold
|
||||||
VISIBLE — the ONLY difference is the tag group's gallery membership, not this
|
on a content head, still hide it but flag it (PresentationReview) so the Hidden
|
||||||
sweep. Two guards keep it safe: (1) never touch an image carrying a
|
view surfaces "also looks like <X>" for review. No-op unless
|
||||||
human/confirmed content tag; (2) if the image ALSO scores >= the conflict
|
presentation_auto_apply_enabled. numpy-only (no sklearn). Returns
|
||||||
threshold on a content head, still apply but flag it (PresentationReview,
|
{n_applied, n_flagged, concepts}."""
|
||||||
mode=<mode>) so the review strip surfaces "also looks like <X>". The source is
|
|
||||||
PROVISIONAL so the head never trains on its own output. No-op unless the mode's
|
|
||||||
enabled flag is set. numpy-only (no sklearn). Returns {n_applied, n_flagged,
|
|
||||||
concepts}."""
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
|
||||||
cfg = _SWEEP_MODES[mode]
|
|
||||||
settings = _settings(session)
|
settings = _settings(session)
|
||||||
if not dry_run and not getattr(settings, cfg["enabled"]):
|
if not dry_run and not settings.presentation_auto_apply_enabled:
|
||||||
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
||||||
ver = settings.embedder_model_version
|
ver = settings.embedder_model_version
|
||||||
pres = _system_tag_heads(session, ver, cfg["names"])
|
pres = _presentation_heads(session, ver)
|
||||||
if not pres:
|
if not pres:
|
||||||
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
||||||
thr = float(getattr(settings, cfg["threshold"]))
|
thr = float(settings.presentation_auto_apply_threshold)
|
||||||
conflict_thr = float(getattr(settings, cfg["conflict"]))
|
conflict_thr = float(settings.presentation_conflict_threshold)
|
||||||
source = cfg["source"]
|
|
||||||
|
|
||||||
Wp = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in pres])
|
Wp = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in pres])
|
||||||
bp = np.asarray([r.bias for r in pres], dtype=np.float32)
|
bp = np.asarray([r.bias for r in pres], dtype=np.float32)
|
||||||
@@ -899,7 +829,18 @@ def system_tag_auto_apply_sweep(
|
|||||||
valued = _valued_image_ids(session)
|
valued = _valued_image_ids(session)
|
||||||
|
|
||||||
# Skip images that already carry, or have rejected, each presentation tag.
|
# Skip images that already carry, or have rejected, each presentation tag.
|
||||||
skip = _applied_or_rejected(session, pres_tag_ids)
|
skip = {tid: set() for tid in pres_tag_ids}
|
||||||
|
for tid in pres_tag_ids:
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
||||||
|
):
|
||||||
|
skip[tid].add(iid)
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
select(TagSuggestionRejection.image_record_id).where(
|
||||||
|
TagSuggestionRejection.tag_id == tid
|
||||||
|
)
|
||||||
|
):
|
||||||
|
skip[tid].add(iid)
|
||||||
|
|
||||||
applied = [0] * len(pres)
|
applied = [0] * len(pres)
|
||||||
n_flagged = 0
|
n_flagged = 0
|
||||||
@@ -914,15 +855,12 @@ def system_tag_auto_apply_sweep(
|
|||||||
if not cids:
|
if not cids:
|
||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
probs = _sigmoid(Xn @ Wp.T + bp, np) # (N, P)
|
probs = 1.0 / (1.0 + np.exp(-(Xn @ Wp.T + bp))) # (N, P)
|
||||||
if Wc is not None:
|
if Wc is not None:
|
||||||
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
|
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc))) # (N, C)
|
||||||
|
max_c = cprobs.max(axis=1)
|
||||||
|
arg_c = cprobs.argmax(axis=1)
|
||||||
scanned += len(cids)
|
scanned += len(cids)
|
||||||
# Same batching as auto_apply_sweep (#3072): collect the chunk's rows
|
|
||||||
# and write them once, below. The PresentationReview rows stay per-row —
|
|
||||||
# they FK to image_record/tag, not to image_tag, so writing the tags
|
|
||||||
# after them is safe, and a flagged conflict is rare by construction.
|
|
||||||
pending: list[dict] = []
|
|
||||||
for p in range(len(pres)):
|
for p in range(len(pres)):
|
||||||
tid = pres_tag_ids[p]
|
tid = pres_tag_ids[p]
|
||||||
for idx in np.where(probs[:, p] >= thr)[0]:
|
for idx in np.where(probs[:, p] >= thr)[0]:
|
||||||
@@ -932,25 +870,28 @@ def system_tag_auto_apply_sweep(
|
|||||||
skip[tid].add(iid)
|
skip[tid].add(iid)
|
||||||
applied[p] += 1
|
applied[p] += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
pending.append({
|
session.execute(
|
||||||
"image_record_id": iid, "tag_id": tid,
|
pg_insert(image_tag)
|
||||||
"source": source,
|
.values(
|
||||||
})
|
image_record_id=iid, tag_id=tid,
|
||||||
# Guard 2: also looks like real content → still apply, but flag it
|
source=_PRESENTATION_SOURCE,
|
||||||
# for the review strip instead of silently marking (chrome hides,
|
)
|
||||||
# process stays visible — either way the operator gets a heads-up).
|
.on_conflict_do_nothing()
|
||||||
|
)
|
||||||
|
# Guard 2: also looks like content → hide but flag for review.
|
||||||
if Wc is not None and float(max_c[idx]) >= conflict_thr:
|
if Wc is not None and float(max_c[idx]) >= conflict_thr:
|
||||||
n_flagged += 1
|
n_flagged += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
_insert_presentation_review(
|
session.execute(
|
||||||
session,
|
pg_insert(PresentationReview)
|
||||||
image_record_id=iid, tag_id=tid,
|
.values(
|
||||||
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
|
image_record_id=iid, tag_id=tid,
|
||||||
conflict_score=float(max_c[idx]),
|
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
|
||||||
mode=mode,
|
conflict_score=float(max_c[idx]),
|
||||||
|
)
|
||||||
|
.on_conflict_do_nothing()
|
||||||
)
|
)
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
insert_image_tags(session, pending)
|
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
concepts = [
|
concepts = [
|
||||||
@@ -963,68 +904,6 @@ def system_tag_auto_apply_sweep(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
|
|
||||||
"""Ring-loud audit for the SOFT WIP-title cohort (#1474). Images auto-tagged
|
|
||||||
`wip` from a low-precision sketch/doodle title (source='wip_title_soft') that ALSO
|
|
||||||
score >= the process conflict threshold on a content head are probably FINISHED
|
|
||||||
art mis-tagged as process — flag them (PresentationReview, mode='process') so the
|
|
||||||
review strip surfaces them ("also looks like <X>", Keep tag / Remove tag). Does
|
|
||||||
NOT remove the tag; the operator decides. No-op when there are no content heads.
|
|
||||||
numpy-only. Returns {n_scanned, n_flagged}."""
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id
|
|
||||||
|
|
||||||
settings = _settings(session)
|
|
||||||
ver = settings.embedder_model_version
|
|
||||||
conflict_thr = float(settings.process_conflict_threshold)
|
|
||||||
conf = _conflict_heads(session, ver)
|
|
||||||
wip_id = resolve_wip_tag_id(session)
|
|
||||||
if not conf or wip_id is None:
|
|
||||||
return {"n_scanned": 0, "n_flagged": 0}
|
|
||||||
Wc = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in conf])
|
|
||||||
bc = np.asarray([r.bias for r in conf], dtype=np.float32)
|
|
||||||
conf_tag_ids = [r.tag_id for r in conf]
|
|
||||||
|
|
||||||
soft_ids = [iid for (iid,) in session.execute(
|
|
||||||
select(image_tag.c.image_record_id)
|
|
||||||
.where(image_tag.c.tag_id == wip_id)
|
|
||||||
.where(image_tag.c.source == WIP_TITLE_SOFT_SOURCE)
|
|
||||||
)]
|
|
||||||
# Skip images already flagged for this tag (idempotent re-runs).
|
|
||||||
flagged = {iid for (iid,) in session.execute(
|
|
||||||
select(PresentationReview.image_record_id)
|
|
||||||
.where(PresentationReview.tag_id == wip_id)
|
|
||||||
)}
|
|
||||||
soft_ids = [i for i in soft_ids if i not in flagged]
|
|
||||||
|
|
||||||
n_flagged = 0
|
|
||||||
scanned = 0
|
|
||||||
for start in range(0, len(soft_ids), _AUTO_APPLY_CHUNK):
|
|
||||||
chunk = soft_ids[start:start + _AUTO_APPLY_CHUNK]
|
|
||||||
emb = _load_embeddings(session, chunk)
|
|
||||||
cids = [i for i in chunk if i in emb]
|
|
||||||
if not cids:
|
|
||||||
continue
|
|
||||||
scanned += len(cids)
|
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
|
||||||
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np)
|
|
||||||
for k in range(len(cids)):
|
|
||||||
if float(max_c[k]) >= conflict_thr:
|
|
||||||
n_flagged += 1
|
|
||||||
if not dry_run:
|
|
||||||
_insert_presentation_review(
|
|
||||||
session,
|
|
||||||
image_record_id=cids[k], tag_id=wip_id,
|
|
||||||
conflict_tag_id=conf_tag_ids[int(arg_c[k])],
|
|
||||||
conflict_score=float(max_c[k]),
|
|
||||||
mode="process",
|
|
||||||
)
|
|
||||||
if not dry_run:
|
|
||||||
session.commit()
|
|
||||||
return {"n_scanned": scanned, "n_flagged": n_flagged}
|
|
||||||
|
|
||||||
|
|
||||||
def retract_auto_applied_heads(session: Session) -> int:
|
def retract_auto_applied_heads(session: Session) -> int:
|
||||||
"""Soft auto-apply (milestone 139): re-score every standing source='head_auto'
|
"""Soft auto-apply (milestone 139): re-score every standing source='head_auto'
|
||||||
tag against its CURRENT head and REMOVE the ones now BELOW the head's
|
tag against its CURRENT head and REMOVE the ones now BELOW the head's
|
||||||
@@ -1072,7 +951,7 @@ def retract_auto_applied_heads(session: Session) -> int:
|
|||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
w = np.asarray(weights, dtype=np.float32)
|
w = np.asarray(weights, dtype=np.float32)
|
||||||
probs = _sigmoid(Xn @ w + float(bias), np)
|
probs = 1.0 / (1.0 + np.exp(-(Xn @ w + float(bias))))
|
||||||
below = [cids[k] for k in np.where(probs < float(thr))[0]]
|
below = [cids[k] for k in np.where(probs < float(thr))[0]]
|
||||||
for iid in below:
|
for iid in below:
|
||||||
session.execute(
|
session.execute(
|
||||||
|
|||||||
@@ -22,20 +22,22 @@ from .heads import score_image
|
|||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class Suggestion:
|
class Suggestion:
|
||||||
# Every suggestion is a canonical Tag: heads/CCIP only score EXISTING concept
|
# canonical_tag_id is None when this is a raw Camie tag with no alias and
|
||||||
# tags (tagging-v2, #114). The old raw-model-key / creates-new / alias-remap
|
# no existing Tag row — accepting it will create the tag.
|
||||||
# cases are gone — a suggestion always maps to a real tag id.
|
canonical_tag_id: int | None
|
||||||
canonical_tag_id: int
|
|
||||||
display_name: str
|
display_name: str
|
||||||
category: str
|
category: str
|
||||||
score: float
|
score: float
|
||||||
source: str # 'head' | 'ccip' | 'both' (Camie tagger/centroid removed in v2)
|
source: str # 'head' | 'ccip' | 'both' (Camie tagger/centroid removed in v2)
|
||||||
# above_threshold = the score cleared the head's own suggest cut (or the
|
creates_new_tag: bool
|
||||||
# system floor). The Suggestions PANEL shows only these; the typed-tag
|
# raw_name = the booru model vocab key behind this suggestion. It's the key
|
||||||
# dropdown fetches ALL suggestions (every head, min=0) and just annotates each
|
# an alias MUST be stored under (resolution looks up the raw key), so the
|
||||||
# matching row with its score, so a low-confidence concept can still be typed
|
# modal needs it to author an alias correctly. None for centroid-only hits
|
||||||
# and picked. CCIP character matches are always above their match threshold.
|
# (no underlying prediction → nothing to alias).
|
||||||
above_threshold: bool
|
raw_name: str | None = None
|
||||||
|
# via_alias = this suggestion was surfaced because an operator alias remapped
|
||||||
|
# the raw prediction to this canonical tag. Lets the UI mark it + offer undo.
|
||||||
|
via_alias: bool = False
|
||||||
# rejected = the operator dismissed this tag for this image (a stored
|
# rejected = the operator dismissed this tag for this image (a stored
|
||||||
# TagSuggestionRejection). It stays in the list — flagged, not dropped — so
|
# TagSuggestionRejection). It stays in the list — flagged, not dropped — so
|
||||||
# the rejection is VISIBLE and REVERSIBLE in the rail (misclick recovery,
|
# the rejection is VISIBLE and REVERSIBLE in the rail (misclick recovery,
|
||||||
@@ -106,7 +108,6 @@ class SuggestionService:
|
|||||||
for h in hits:
|
for h in hits:
|
||||||
merged[(h["category"], h["tag_id"])] = {
|
merged[(h["category"], h["tag_id"])] = {
|
||||||
"name": h["name"], "score": h["score"], "source": "head",
|
"name": h["name"], "score": h["score"], "source": "head",
|
||||||
"above_threshold": h["above_threshold"],
|
|
||||||
"grounding": h.get("grounding"),
|
"grounding": h.get("grounding"),
|
||||||
}
|
}
|
||||||
for c in ccip_hits:
|
for c in ccip_hits:
|
||||||
@@ -115,16 +116,12 @@ class SuggestionService:
|
|||||||
if ex is not None:
|
if ex is not None:
|
||||||
ex["source"] = "both"
|
ex["source"] = "both"
|
||||||
ex["score"] = max(ex["score"], c["score"])
|
ex["score"] = max(ex["score"], c["score"])
|
||||||
# CCIP only returns matches above its own threshold, so a CCIP
|
|
||||||
# corroboration always makes the merged suggestion above-threshold.
|
|
||||||
ex["above_threshold"] = True
|
|
||||||
# Keep the head's localized crop if it had one; else fall back to
|
# Keep the head's localized crop if it had one; else fall back to
|
||||||
# the CCIP figure so a corroborated character still grounds (#1206).
|
# the CCIP figure so a corroborated character still grounds (#1206).
|
||||||
ex["grounding"] = ex.get("grounding") or c.get("grounding")
|
ex["grounding"] = ex.get("grounding") or c.get("grounding")
|
||||||
else:
|
else:
|
||||||
merged[key] = {
|
merged[key] = {
|
||||||
"name": c["name"], "score": c["score"], "source": "ccip",
|
"name": c["name"], "score": c["score"], "source": "ccip",
|
||||||
"above_threshold": True,
|
|
||||||
"grounding": c.get("grounding"),
|
"grounding": c.get("grounding"),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -139,7 +136,7 @@ class SuggestionService:
|
|||||||
category=cat,
|
category=cat,
|
||||||
score=m["score"],
|
score=m["score"],
|
||||||
source=m["source"],
|
source=m["source"],
|
||||||
above_threshold=m["above_threshold"],
|
creates_new_tag=False,
|
||||||
rejected=tag_id in rejected,
|
rejected=tag_id in rejected,
|
||||||
grounding=m.get("grounding"),
|
grounding=m.get("grounding"),
|
||||||
)
|
)
|
||||||
@@ -160,7 +157,8 @@ class SuggestionService:
|
|||||||
was suggested for (or already applied to) >= threshold fraction of
|
was suggested for (or already applied to) >= threshold fraction of
|
||||||
the selection AND was acceptable on >= 1 image. Confidence is the
|
the selection AND was acceptable on >= 1 image. Confidence is the
|
||||||
mean over images where it was suggested. Aggregated by
|
mean over images where it was suggested. Aggregated by
|
||||||
canonical_tag_id (every suggestion is a canonical tag now)."""
|
canonical_tag_id; creates-new (no canonical id) suggestions are
|
||||||
|
skipped (bulk Accept applies by tag id)."""
|
||||||
if not image_ids:
|
if not image_ids:
|
||||||
return {}
|
return {}
|
||||||
threshold = min(1.0, max(0.0, threshold))
|
threshold = min(1.0, max(0.0, threshold))
|
||||||
@@ -171,6 +169,8 @@ class SuggestionService:
|
|||||||
sl = await self.for_image(image_id)
|
sl = await self.for_image(image_id)
|
||||||
for category, items in sl.by_category.items():
|
for category, items in sl.by_category.items():
|
||||||
for s in items:
|
for s in items:
|
||||||
|
if s.canonical_tag_id is None or s.creates_new_tag:
|
||||||
|
continue
|
||||||
# for_image keeps rejected tags (flagged) for the rail;
|
# for_image keeps rejected tags (flagged) for the rail;
|
||||||
# bulk consensus must still ignore them — a tag dismissed on
|
# bulk consensus must still ignore them — a tag dismissed on
|
||||||
# an image isn't a suggestion for that image.
|
# an image isn't a suggestion for that image.
|
||||||
|
|||||||
@@ -29,15 +29,7 @@ from ...models.tag import image_tag
|
|||||||
# a CCIP reference) unless the operator confirms them (milestone 139). Keeping
|
# a CCIP reference) unless the operator confirms them (milestone 139). Keeping
|
||||||
# auto-applied predictions out of training is what makes them "soft" — a misfire
|
# auto-applied predictions out of training is what makes them "soft" — a misfire
|
||||||
# can't reinforce itself, so the retraction sweep can actually drop it.
|
# can't reinforce itself, so the retraction sweep can actually drop it.
|
||||||
# `process_auto` (#1464): wip/editor screenshot applied by the process sweep are
|
_AUTO_SOURCES = ("head_auto", "ccip_auto", "ml_auto", "presentation_auto")
|
||||||
# ALSO provisional — the head must learn only from title (`wip_title`) + manual
|
|
||||||
# labels, never its own auto-applied output, or it would runaway (operator 2026-07-12).
|
|
||||||
# `wip_title_soft` (#1474): the soft title tier (sketch/doodle) is LOW-precision, so
|
|
||||||
# it's provisional too — a finished piece titled "sketch" must not train the wip head.
|
|
||||||
_AUTO_SOURCES = (
|
|
||||||
"head_auto", "ccip_auto", "ml_auto", "presentation_auto", "process_auto",
|
|
||||||
"wip_title_soft",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _hygiene_excluded_ids(session: Session) -> set[int]:
|
def _hygiene_excluded_ids(session: Session) -> set[int]:
|
||||||
@@ -94,24 +86,6 @@ def _rejected_ids(session: Session, tag_id: int) -> list[int]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def _applied_or_rejected(session: Session, tag_ids) -> dict[int, set[int]]:
|
|
||||||
"""Per-tag skip set for the auto-apply sweeps: every image that ALREADY carries
|
|
||||||
the tag (ANY source — not just training positives) OR has rejected it. A sweep
|
|
||||||
never re-applies to these. Shared by auto_apply_sweep + system_tag_auto_apply_sweep
|
|
||||||
(heads.py) and scheduled_ccip_auto_apply (tasks/ml.py). Callers mutate the returned
|
|
||||||
sets in-place to also dedupe within a single run."""
|
|
||||||
skip: dict[int, set[int]] = {}
|
|
||||||
for tid in tag_ids:
|
|
||||||
ids = {
|
|
||||||
r[0] for r in session.execute(
|
|
||||||
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
|
||||||
).all()
|
|
||||||
}
|
|
||||||
ids.update(_rejected_ids(session, tid))
|
|
||||||
skip[tid] = ids
|
|
||||||
return skip
|
|
||||||
|
|
||||||
|
|
||||||
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
|
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
|
||||||
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
|
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
|
||||||
sparse, so an untagged image is almost always a true negative."""
|
sparse, so an untagged image is almost always a true negative."""
|
||||||
|
|||||||
@@ -91,46 +91,48 @@ def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _campaigns_api_first(vanity: str, cookies_path: str | None) -> dict | None:
|
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
||||||
"""The first `data` object from Patreon's campaigns API filtered by vanity
|
|
||||||
(`?filter[vanity]=<vanity>&fields[campaign]=name`), or None on any failure
|
|
||||||
(network / non-200 / non-JSON / empty). The single request shape shared by
|
|
||||||
_lookup_via_api (plucks the campaign id) and resolve_display_name (plucks the
|
|
||||||
display name)."""
|
|
||||||
jar = _load_cookie_jar(cookies_path)
|
jar = _load_cookie_jar(cookies_path)
|
||||||
|
headers = {
|
||||||
|
"User-Agent": _USER_AGENT,
|
||||||
|
"Accept": "application/vnd.api+json",
|
||||||
|
}
|
||||||
|
params = {
|
||||||
|
"filter[vanity]": vanity,
|
||||||
|
"fields[campaign]": "name",
|
||||||
|
}
|
||||||
try:
|
try:
|
||||||
resp = requests.get(
|
resp = requests.get(
|
||||||
_CAMPAIGNS_URL,
|
_CAMPAIGNS_URL,
|
||||||
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
|
params=params,
|
||||||
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
|
headers=headers,
|
||||||
cookies=jar,
|
cookies=jar,
|
||||||
timeout=_TIMEOUT_SECONDS,
|
timeout=_TIMEOUT_SECONDS,
|
||||||
)
|
)
|
||||||
except requests.RequestException as exc:
|
except requests.RequestException as exc:
|
||||||
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
|
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
if resp.status_code != 200:
|
if resp.status_code != 200:
|
||||||
log.warning(
|
log.warning(
|
||||||
"Patreon campaigns API returned HTTP %d for vanity=%s",
|
"Patreon campaigns API returned HTTP %d for vanity=%s",
|
||||||
resp.status_code, vanity,
|
resp.status_code, vanity,
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
try:
|
try:
|
||||||
payload = resp.json()
|
payload = resp.json()
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
|
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
|
||||||
return None
|
return None
|
||||||
data = payload.get("data") if isinstance(payload, dict) else None
|
|
||||||
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
|
|
||||||
return None
|
|
||||||
return data[0]
|
|
||||||
|
|
||||||
|
if not isinstance(payload, dict):
|
||||||
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
|
||||||
first = _campaigns_api_first(vanity, cookies_path)
|
|
||||||
if first is None:
|
|
||||||
return None
|
return None
|
||||||
campaign_id = first.get("id")
|
data = payload.get("data")
|
||||||
|
if not isinstance(data, list) or not data:
|
||||||
|
return None
|
||||||
|
first = data[0] if isinstance(data[0], dict) else None
|
||||||
|
campaign_id = first.get("id") if first else None
|
||||||
if not isinstance(campaign_id, str) or not campaign_id:
|
if not isinstance(campaign_id, str) or not campaign_id:
|
||||||
return None
|
return None
|
||||||
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
|
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
|
||||||
@@ -142,10 +144,24 @@ def resolve_display_name(vanity: str, cookies_path: str | None) -> str | None:
|
|||||||
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None
|
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None
|
||||||
on any failure — the caller falls back to the vanity handle. Sync: call from
|
on any failure — the caller falls back to the vanity handle. Sync: call from
|
||||||
an executor."""
|
an executor."""
|
||||||
first = _campaigns_api_first(vanity, cookies_path)
|
jar = _load_cookie_jar(cookies_path)
|
||||||
if first is None:
|
try:
|
||||||
|
resp = requests.get(
|
||||||
|
_CAMPAIGNS_URL,
|
||||||
|
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
|
||||||
|
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
|
||||||
|
cookies=jar,
|
||||||
|
timeout=_TIMEOUT_SECONDS,
|
||||||
|
)
|
||||||
|
if resp.status_code != 200:
|
||||||
|
return None
|
||||||
|
data = resp.json().get("data")
|
||||||
|
except (requests.RequestException, ValueError) as exc:
|
||||||
|
log.warning("Patreon name lookup failed for vanity=%s: %s", vanity, exc)
|
||||||
return None
|
return None
|
||||||
name = (first.get("attributes") or {}).get("name")
|
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
|
||||||
|
return None
|
||||||
|
name = (data[0].get("attributes") or {}).get("name")
|
||||||
return name.strip() if isinstance(name, str) and name.strip() else None
|
return name.strip() if isinstance(name, str) and name.strip() else None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -8,10 +8,9 @@ PLATFORMS below. Sidecar parsing, cookie materialization, and
|
|||||||
|
|
||||||
Lifted from GallerySubscriber's
|
Lifted from GallerySubscriber's
|
||||||
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
|
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
|
||||||
and ~/.../extension/lib/platforms.js. Five platforms; auth_type and
|
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
|
||||||
URL patterns match GS exactly so the existing browser extension
|
URL patterns match GS exactly so the existing browser extension
|
||||||
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
|
hits FC unmodified.
|
||||||
FC downloaders are art-dedicated services only.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from .base import (
|
from .base import (
|
||||||
@@ -19,6 +18,7 @@ from .base import (
|
|||||||
DEFAULT_EXTERNAL_POST_ID_KEYS,
|
DEFAULT_EXTERNAL_POST_ID_KEYS,
|
||||||
PlatformInfo,
|
PlatformInfo,
|
||||||
)
|
)
|
||||||
|
from .deviantart import INFO as _DEVIANTART
|
||||||
from .discord import INFO as _DISCORD
|
from .discord import INFO as _DISCORD
|
||||||
from .hentaifoundry import INFO as _HENTAIFOUNDRY
|
from .hentaifoundry import INFO as _HENTAIFOUNDRY
|
||||||
from .patreon import INFO as _PATREON
|
from .patreon import INFO as _PATREON
|
||||||
@@ -33,6 +33,7 @@ PLATFORMS: dict[str, PlatformInfo] = {
|
|||||||
_HENTAIFOUNDRY,
|
_HENTAIFOUNDRY,
|
||||||
_DISCORD,
|
_DISCORD,
|
||||||
_PIXIV,
|
_PIXIV,
|
||||||
|
_DEVIANTART,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ class PlatformInfo:
|
|||||||
# Synthesize a post permalink from sidecar data. Required when
|
# Synthesize a post permalink from sidecar data. Required when
|
||||||
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
||||||
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
||||||
# `url` field (patreon).
|
# `url` field (patreon, deviantart).
|
||||||
derive_post_url: Callable[[dict], str | None] | None = None
|
derive_post_url: Callable[[dict], str | None] | None = None
|
||||||
|
|
||||||
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
"""DeviantArt — no exercised quirks yet.
|
||||||
|
|
||||||
|
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
|
||||||
|
audit, so we don't know yet whether DA's gallery-dl sidecars are
|
||||||
|
well-behaved or have their own quirks. When DA gets exercised for the
|
||||||
|
first time, add `derive_post_url` / `augment_cookies` here as needed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .base import GD_DEFAULTS, PlatformInfo
|
||||||
|
|
||||||
|
INFO = PlatformInfo(
|
||||||
|
key="deviantart",
|
||||||
|
name="DeviantArt",
|
||||||
|
description="Download artwork from DeviantArt artists",
|
||||||
|
auth_type="cookies",
|
||||||
|
requires_auth=False,
|
||||||
|
url_pattern=r"^https?://(www\.)?deviantart\.com/",
|
||||||
|
url_examples=[
|
||||||
|
"https://www.deviantart.com/example-artist",
|
||||||
|
"https://www.deviantart.com/example-artist/gallery",
|
||||||
|
],
|
||||||
|
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
|
||||||
|
)
|
||||||
@@ -24,7 +24,6 @@ from ..models import (
|
|||||||
Post,
|
Post,
|
||||||
PostAttachment,
|
PostAttachment,
|
||||||
Source,
|
Source,
|
||||||
attachment_download_url,
|
|
||||||
)
|
)
|
||||||
from ..utils.html_sanitize import (
|
from ..utils.html_sanitize import (
|
||||||
extract_img_srcs,
|
extract_img_srcs,
|
||||||
@@ -361,7 +360,7 @@ class PostFeedService:
|
|||||||
"ext": att.ext,
|
"ext": att.ext,
|
||||||
"mime": att.mime,
|
"mime": att.mime,
|
||||||
"size_bytes": att.size_bytes,
|
"size_bytes": att.size_bytes,
|
||||||
"download_url": attachment_download_url(att.id),
|
"download_url": f"/api/attachments/{att.id}/download",
|
||||||
})
|
})
|
||||||
return out
|
return out
|
||||||
|
|
||||||
@@ -398,8 +397,6 @@ class PostFeedService:
|
|||||||
"post_title_translated": post.post_title_translated,
|
"post_title_translated": post.post_title_translated,
|
||||||
"description_translated": desc_trans_short,
|
"description_translated": desc_trans_short,
|
||||||
"translated_source_lang": post.translated_source_lang,
|
"translated_source_lang": post.translated_source_lang,
|
||||||
# Sticky per-post translation choice (auto/force/original, #155).
|
|
||||||
"translation_override": post.translation_override,
|
|
||||||
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
||||||
"source": (
|
"source": (
|
||||||
{"id": source.id, "platform": source.platform}
|
{"id": source.id, "platform": source.platform}
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ from ..models import (
|
|||||||
Post,
|
Post,
|
||||||
PostAttachment,
|
PostAttachment,
|
||||||
Source,
|
Source,
|
||||||
attachment_download_url,
|
|
||||||
)
|
)
|
||||||
from ..utils.html_sanitize import sanitize_post_html
|
from ..utils.html_sanitize import sanitize_post_html
|
||||||
|
|
||||||
@@ -36,7 +35,6 @@ def _post_dict(p: Post) -> dict:
|
|||||||
"title_translated": p.post_title_translated,
|
"title_translated": p.post_title_translated,
|
||||||
"description_translated": p.description_translated,
|
"description_translated": p.description_translated,
|
||||||
"translated_source_lang": p.translated_source_lang,
|
"translated_source_lang": p.translated_source_lang,
|
||||||
"translation_override": p.translation_override,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -54,7 +52,7 @@ def _attachment_dict(a: PostAttachment) -> dict:
|
|||||||
"original_filename": a.original_filename,
|
"original_filename": a.original_filename,
|
||||||
"size_bytes": a.size_bytes,
|
"size_bytes": a.size_bytes,
|
||||||
"ext": a.ext,
|
"ext": a.ext,
|
||||||
"download_url": attachment_download_url(a.id),
|
"download_url": f"/api/attachments/{a.id}/download",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,121 +0,0 @@
|
|||||||
"""Title-based WIP auto-tagging (task #1458).
|
|
||||||
|
|
||||||
Deterministic heuristic: when a post's TITLE explicitly declares work-in-progress
|
|
||||||
(the artist's own "WIP" / "work in progress" label), the ``wip`` system tag is
|
|
||||||
applied to that post's images — a cheap, high-precision complement to the
|
|
||||||
image-based ML ``wip`` head. WIP images are excluded from the Explore/gallery
|
|
||||||
browse (see gallery_service ``excluded_system_tags``), so honouring the artist's
|
|
||||||
own label keeps unfinished pieces out of the main browse right at import.
|
|
||||||
|
|
||||||
Precision over recall — a false WIP tag HIDES a finished post — so matching is
|
|
||||||
token-anchored: ``swipe`` / ``wiped`` / ``wiping`` never trip it (a letter on the
|
|
||||||
boundary blocks the match).
|
|
||||||
|
|
||||||
Sync-only: both consumers (the importer and the backfill Celery task) run on a
|
|
||||||
sync Session. Application is idempotent-additive (ON CONFLICT DO NOTHING) and
|
|
||||||
stamps a distinct ``image_tag.source`` so a later pass can tell where a wip tag
|
|
||||||
came from — the "manual" / "head_auto" / "ccip_auto" / "ml_accepted" provenance
|
|
||||||
family gains one member.
|
|
||||||
"""
|
|
||||||
import re
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from ..models.tag import WIP_SYSTEM_TAG, Tag, image_tag
|
|
||||||
from .image_tag_apply import insert_image_tags
|
|
||||||
|
|
||||||
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other
|
|
||||||
# apply sources so provenance stays legible and a future undo can target only these.
|
|
||||||
# HARD tier ("WIP"/"work in progress") is high-precision → trains the wip head.
|
|
||||||
WIP_TITLE_SOURCE = "wip_title"
|
|
||||||
# SOFT tier (sketch/doodle/scribble, #1474) is LOWER-precision — a finished "sketch"
|
|
||||||
# is often not WIP. This source is PROVISIONAL (in training_data._AUTO_SOURCES) so it
|
|
||||||
# NEVER trains the wip head; a soft-tagged image that also looks like real content is
|
|
||||||
# surfaced by the ring-loud audit for review.
|
|
||||||
WIP_TITLE_SOFT_SOURCE = "wip_title_soft"
|
|
||||||
|
|
||||||
# A standalone "WIP" / "W.I.P" token, or the phrase "work in progress"
|
|
||||||
# (space/underscore/hyphen separated). The letter-boundary lookarounds are what
|
|
||||||
# make this precision-first: `s|wip|e`, `|wip|ed`, `|wip|ing` all have a letter
|
|
||||||
# abutting the token, so they're rejected. A trailing digit is allowed so
|
|
||||||
# "WIP2" (= WIP part 2) still matches.
|
|
||||||
_WIP_RE = re.compile(
|
|
||||||
r"(?<![A-Za-z])(?:w\.?i\.?p\.?|work[\s_-]+in[\s_-]+progress)(?![A-Za-z])",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Soft tier: sketch / doodle / scribble (+ plurals), letter-boundary anchored so
|
|
||||||
# "sketchbook" / "kadoodle" don't trip it. Deliberately conservative — recall is
|
|
||||||
# secondary because the soft source doesn't train the head and the ring-loud audit
|
|
||||||
# catches false positives.
|
|
||||||
_SOFT_WIP_RE = re.compile(
|
|
||||||
r"(?<![A-Za-z])(?:sketch|sketches|doodle|doodles|scribble|scribbles)(?![A-Za-z])",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Coarse SQL prefilters for the backfill sweep — narrow the post scan to rows that
|
|
||||||
# COULD match before the precise regex confirms. Case-insensitive ILIKE patterns.
|
|
||||||
# Each MUST stay a SUPERSET of its regex or the sweep would silently miss posts.
|
|
||||||
WIP_TITLE_SQL_PREFILTER = ("%wip%", "%work%progress%")
|
|
||||||
SOFT_WIP_TITLE_SQL_PREFILTER = ("%sketch%", "%doodle%", "%scribble%")
|
|
||||||
|
|
||||||
# Chunk bulk inserts so a large sweep can't blow past psycopg's 65535-parameter
|
|
||||||
# ceiling (3 params/row → ~21k rows max; 5k stays comfortably under).
|
|
||||||
_INSERT_CHUNK = 5000
|
|
||||||
|
|
||||||
|
|
||||||
def matches_wip_title(title: str | None) -> bool:
|
|
||||||
"""True when a post title explicitly marks it work-in-progress (HARD tier)."""
|
|
||||||
if not title:
|
|
||||||
return False
|
|
||||||
return _WIP_RE.search(title) is not None
|
|
||||||
|
|
||||||
|
|
||||||
def matches_soft_wip_title(title: str | None) -> bool:
|
|
||||||
"""True when a title carries a SOFT WIP cue (sketch/doodle/scribble, #1474)."""
|
|
||||||
if not title:
|
|
||||||
return False
|
|
||||||
return _SOFT_WIP_RE.search(title) is not None
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_wip_tag_id(session: Session) -> int | None:
|
|
||||||
"""The seeded ``wip`` system tag's id (migration 0075), or None if absent."""
|
|
||||||
return session.execute(
|
|
||||||
select(Tag.id).where(Tag.name == WIP_SYSTEM_TAG, Tag.is_system.is_(True))
|
|
||||||
).scalar_one_or_none()
|
|
||||||
|
|
||||||
|
|
||||||
def apply_wip_image_tags(
|
|
||||||
session: Session, image_ids, tag_id: int, *, source: str = WIP_TITLE_SOURCE
|
|
||||||
) -> int:
|
|
||||||
"""Attach ``tag_id`` (stamped with ``source``) to each image id, idempotently —
|
|
||||||
never disturbs an existing tag or its source. Returns the number of image_tag
|
|
||||||
rows newly inserted. Does NOT commit.
|
|
||||||
|
|
||||||
The insert count is computed from a pre-SELECT of already-tagged ids rather
|
|
||||||
than the statement's ``rowcount``: psycopg reports -1 for a multi-row
|
|
||||||
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so rowcount
|
|
||||||
is unusable here. The SELECT is accurate within this single transaction (no
|
|
||||||
concurrent writer touches these (image, wip) rows); ON CONFLICT DO NOTHING
|
|
||||||
stays as a race-safety belt so a rare concurrent insert can't error."""
|
|
||||||
ids = list({int(i) for i in image_ids})
|
|
||||||
if not ids:
|
|
||||||
return 0
|
|
||||||
inserted = 0
|
|
||||||
for start in range(0, len(ids), _INSERT_CHUNK):
|
|
||||||
chunk = ids[start:start + _INSERT_CHUNK]
|
|
||||||
already = set(session.execute(
|
|
||||||
select(image_tag.c.image_record_id)
|
|
||||||
.where(image_tag.c.tag_id == tag_id)
|
|
||||||
.where(image_tag.c.image_record_id.in_(chunk))
|
|
||||||
).scalars())
|
|
||||||
to_insert = [iid for iid in chunk if iid not in already]
|
|
||||||
if not to_insert:
|
|
||||||
continue
|
|
||||||
insert_image_tags(session, [
|
|
||||||
{"image_record_id": iid, "tag_id": tag_id, "source": source}
|
|
||||||
for iid in to_insert
|
|
||||||
])
|
|
||||||
inserted += len(to_insert)
|
|
||||||
return inserted
|
|
||||||
@@ -409,31 +409,3 @@ def rescan_series_suggestions_task(self, after_post_id: int = 0) -> dict:
|
|||||||
)
|
)
|
||||||
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
||||||
return summary
|
return summary
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.admin.reclaim_orphaned_attachments_task",
|
|
||||||
bind=True,
|
|
||||||
autoretry_for=(OperationalError, DBAPIError),
|
|
||||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
|
||||||
# The service stops walking at its own 900s budget and reports partial, so
|
|
||||||
# these limits are the backstop for a wedged filesystem (NFS stall), not the
|
|
||||||
# expected exit. Comfortably above the budget so a normal run always returns
|
|
||||||
# its summary rather than being killed mid-walk.
|
|
||||||
soft_time_limit=1200, time_limit=1500, # 20 min / 25 min
|
|
||||||
)
|
|
||||||
def reclaim_orphaned_attachments_task(self, dry_run: bool = True) -> dict:
|
|
||||||
"""Reclaim unattributed PostAttachment rows and the store blobs nothing
|
|
||||||
references any more (#3068). dry_run (the default) returns the projection
|
|
||||||
without touching rows or files; apply deletes the orphan rows, then unlinks
|
|
||||||
every blob no surviving row references.
|
|
||||||
|
|
||||||
Defaults to the SAFE preview — unlike the other tasks here, whose apply is
|
|
||||||
reversible-ish or scoped; this one deletes files. Operator-triggered only,
|
|
||||||
never on a beat: an unattended sweep that unlinks blobs is not something to
|
|
||||||
run without someone reading the projection first."""
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
with SessionLocal() as session:
|
|
||||||
return cleanup_service.reclaim_orphaned_attachments(
|
|
||||||
session, images_root=IMAGES_ROOT, dry_run=dry_run,
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -76,8 +76,6 @@ DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
|
|||||||
OLD_TASK_DAYS = 7
|
OLD_TASK_DAYS = 7
|
||||||
PHASH_PAGE = 500
|
PHASH_PAGE = 500
|
||||||
VERIFY_PAGE = 200
|
VERIFY_PAGE = 200
|
||||||
# Title-based WIP backfill (task #1458): posts scanned per keyset page.
|
|
||||||
WIP_BACKFILL_PAGE = 500
|
|
||||||
FFPROBE_TIMEOUT_SECONDS = 10
|
FFPROBE_TIMEOUT_SECONDS = 10
|
||||||
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
|
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
|
||||||
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
||||||
@@ -173,12 +171,6 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
|||||||
# task-name override beats the queue threshold whatever queue the row records
|
# task-name override beats the queue threshold whatever queue the row records
|
||||||
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
||||||
"backend.app.tasks.external.fetch_external_link": 65,
|
"backend.app.tasks.external.fetch_external_link": 65,
|
||||||
# Attachment reclaim walks the whole sha-addressed store; the service caps
|
|
||||||
# itself at a 900s budget and reports partial, but the task's hard limit is
|
|
||||||
# 25 min for a wedged filesystem (NFS stall). Same phantom-flag class as the
|
|
||||||
# external-fetch entry above — without an override a healthy in-flight walk
|
|
||||||
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
|
|
||||||
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -782,62 +774,89 @@ def recover_stalled_library_audit_runs() -> int:
|
|||||||
return recovered
|
return recovered
|
||||||
|
|
||||||
|
|
||||||
def _recover_stalled_runs(model, *, stall_minutes: int, keep_runs: int, label: str) -> int:
|
|
||||||
"""Shared recovery + retention sweep for the head run-tracking tables
|
|
||||||
(HeadTrainingRun / HeadAutoApplyRun, which share the
|
|
||||||
status/last_progress_at/started_at/finished_at/error/id columns): flip 'running'
|
|
||||||
rows with no progress past `stall_minutes` to 'error', then prune to the last
|
|
||||||
`keep_runs` (rule 89). Returns the number recovered. NOTE the two other recover
|
|
||||||
tasks are deliberately NOT folded in — library-audit has no prune tail and
|
|
||||||
backup uses a single started_at cutoff."""
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
now = datetime.now(UTC)
|
|
||||||
cutoff = now - timedelta(minutes=stall_minutes)
|
|
||||||
with SessionLocal() as session:
|
|
||||||
result = session.execute(
|
|
||||||
update(model)
|
|
||||||
.where(model.status == "running")
|
|
||||||
.where(func.coalesce(model.last_progress_at, model.started_at) < cutoff)
|
|
||||||
.values(
|
|
||||||
status="error", finished_at=now,
|
|
||||||
error=f"stranded by recovery sweep (no progress for {stall_minutes} min)",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
keep = session.execute(
|
|
||||||
select(model.id).order_by(model.id.desc()).limit(keep_runs)
|
|
||||||
).scalars().all()
|
|
||||||
if keep:
|
|
||||||
session.execute(delete(model).where(model.id.not_in(keep)))
|
|
||||||
session.commit()
|
|
||||||
recovered = result.rowcount or 0
|
|
||||||
if recovered:
|
|
||||||
log.info("%s: recovered %d rows", label, recovered)
|
|
||||||
return recovered
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
|
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
|
||||||
def recover_stalled_head_training_runs() -> int:
|
def recover_stalled_head_training_runs() -> int:
|
||||||
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
|
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
|
||||||
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
|
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
|
||||||
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
|
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
|
||||||
return _recover_stalled_runs(
|
SessionLocal = _sync_session_factory()
|
||||||
HeadTrainingRun,
|
now = datetime.now(UTC)
|
||||||
stall_minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES,
|
cutoff = now - timedelta(minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES)
|
||||||
keep_runs=HEAD_TRAINING_KEEP_RUNS,
|
with SessionLocal() as session:
|
||||||
label="recover_stalled_head_training_runs",
|
result = session.execute(
|
||||||
)
|
update(HeadTrainingRun)
|
||||||
|
.where(HeadTrainingRun.status == "running")
|
||||||
|
.where(
|
||||||
|
func.coalesce(
|
||||||
|
HeadTrainingRun.last_progress_at, HeadTrainingRun.started_at
|
||||||
|
)
|
||||||
|
< cutoff
|
||||||
|
)
|
||||||
|
.values(
|
||||||
|
status="error", finished_at=now,
|
||||||
|
error=(
|
||||||
|
f"stranded by recovery sweep (no progress for "
|
||||||
|
f"{HEAD_TRAINING_STALL_THRESHOLD_MINUTES} min)"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
keep = session.execute(
|
||||||
|
select(HeadTrainingRun.id).order_by(HeadTrainingRun.id.desc())
|
||||||
|
.limit(HEAD_TRAINING_KEEP_RUNS)
|
||||||
|
).scalars().all()
|
||||||
|
if keep:
|
||||||
|
session.execute(
|
||||||
|
delete(HeadTrainingRun).where(HeadTrainingRun.id.not_in(keep))
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
recovered = result.rowcount or 0
|
||||||
|
if recovered:
|
||||||
|
log.info(
|
||||||
|
"recover_stalled_head_training_runs: recovered %d rows", recovered
|
||||||
|
)
|
||||||
|
return recovered
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
|
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
|
||||||
def recover_stalled_head_auto_apply_runs() -> int:
|
def recover_stalled_head_auto_apply_runs() -> int:
|
||||||
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
|
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
|
||||||
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
|
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
|
||||||
return _recover_stalled_runs(
|
SessionLocal = _sync_session_factory()
|
||||||
HeadAutoApplyRun,
|
now = datetime.now(UTC)
|
||||||
stall_minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES,
|
cutoff = now - timedelta(minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES)
|
||||||
keep_runs=HEAD_AUTO_APPLY_KEEP_RUNS,
|
with SessionLocal() as session:
|
||||||
label="recover_stalled_head_auto_apply_runs",
|
result = session.execute(
|
||||||
)
|
update(HeadAutoApplyRun)
|
||||||
|
.where(HeadAutoApplyRun.status == "running")
|
||||||
|
.where(
|
||||||
|
func.coalesce(
|
||||||
|
HeadAutoApplyRun.last_progress_at, HeadAutoApplyRun.started_at
|
||||||
|
)
|
||||||
|
< cutoff
|
||||||
|
)
|
||||||
|
.values(
|
||||||
|
status="error", finished_at=now,
|
||||||
|
error=(
|
||||||
|
f"stranded by recovery sweep (no progress for "
|
||||||
|
f"{HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES} min)"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
keep = session.execute(
|
||||||
|
select(HeadAutoApplyRun.id).order_by(HeadAutoApplyRun.id.desc())
|
||||||
|
.limit(HEAD_AUTO_APPLY_KEEP_RUNS)
|
||||||
|
).scalars().all()
|
||||||
|
if keep:
|
||||||
|
session.execute(
|
||||||
|
delete(HeadAutoApplyRun).where(HeadAutoApplyRun.id.not_in(keep))
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
recovered = result.rowcount or 0
|
||||||
|
if recovered:
|
||||||
|
log.info(
|
||||||
|
"recover_stalled_head_auto_apply_runs: recovered %d rows", recovered
|
||||||
|
)
|
||||||
|
return recovered
|
||||||
|
|
||||||
|
|
||||||
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
|
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
|
||||||
@@ -1026,96 +1045,6 @@ def cleanup_old_download_events() -> int:
|
|||||||
return result.rowcount or 0
|
return result.rowcount or 0
|
||||||
|
|
||||||
|
|
||||||
def _backfill_wip_tier(session, tag_id, prefilter, matcher, source) -> int:
|
|
||||||
"""One keyset-paginated pass over posts whose title matches a WIP tier, applying
|
|
||||||
`tag_id` (stamped `source`) to their images. Shared by the hard + soft tiers
|
|
||||||
(#1458 / #1474). Coarse `prefilter` (ILIKE superset) narrows the scan; the precise
|
|
||||||
`matcher` confirms. Idempotent-additive (ON CONFLICT DO NOTHING). Returns the row
|
|
||||||
count newly applied."""
|
|
||||||
from ..models import Post
|
|
||||||
from ..models.image_provenance import ImageProvenance
|
|
||||||
from ..services.wip_title import apply_wip_image_tags
|
|
||||||
|
|
||||||
applied = 0
|
|
||||||
last_id = 0
|
|
||||||
while True:
|
|
||||||
rows = session.execute(
|
|
||||||
select(Post.id, Post.post_title)
|
|
||||||
.where(Post.id > last_id)
|
|
||||||
.where(Post.post_title.is_not(None))
|
|
||||||
.where(or_(*[Post.post_title.ilike(p) for p in prefilter]))
|
|
||||||
.order_by(Post.id.asc())
|
|
||||||
.limit(WIP_BACKFILL_PAGE)
|
|
||||||
).all()
|
|
||||||
if not rows:
|
|
||||||
break
|
|
||||||
last_id = rows[-1][0]
|
|
||||||
match_ids = [pid for pid, title in rows if matcher(title)]
|
|
||||||
if match_ids:
|
|
||||||
image_ids = session.execute(
|
|
||||||
select(ImageProvenance.image_record_id)
|
|
||||||
.where(ImageProvenance.post_id.in_(match_ids))
|
|
||||||
).scalars().all()
|
|
||||||
applied += apply_wip_image_tags(session, image_ids, tag_id, source=source)
|
|
||||||
session.commit()
|
|
||||||
return applied
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.maintenance.backfill_wip_title_tags",
|
|
||||||
# Coarse-prefiltered scan over posts; the candidate set is small on a typical
|
|
||||||
# library, but bound it like the other full-library sweeps.
|
|
||||||
soft_time_limit=1800, time_limit=2100,
|
|
||||||
)
|
|
||||||
def backfill_wip_title_tags() -> int:
|
|
||||||
"""Scan EXISTING posts for WIP titles and apply the `wip` system tag to their
|
|
||||||
images — the operator-triggered back-catalogue catch-up (task #1458 hard tier +
|
|
||||||
#1474 soft tier). New imports are tagged live by the importer; this covers the
|
|
||||||
existing library.
|
|
||||||
|
|
||||||
HARD tier ("WIP"/"work in progress") always runs (the operator triggered the
|
|
||||||
scan); the SOFT tier (sketch/doodle, provisional source) runs only when
|
|
||||||
wip_soft_title_tagging_enabled, AFTER hard so a title matching both keeps the
|
|
||||||
trained hard tag (ON CONFLICT DO NOTHING). Keyset-paginated, restart-safe.
|
|
||||||
|
|
||||||
Deliberately NOT scheduled as a beat: a periodic re-run would re-apply to matching
|
|
||||||
posts and silently undo a manual WIP removal, so it stays an explicit operator
|
|
||||||
action (Settings → "Scan existing posts for WIP titles"). Returns rows applied.
|
|
||||||
"""
|
|
||||||
from ..models import ImportSettings
|
|
||||||
from ..services.wip_title import (
|
|
||||||
SOFT_WIP_TITLE_SQL_PREFILTER,
|
|
||||||
WIP_TITLE_SOFT_SOURCE,
|
|
||||||
WIP_TITLE_SOURCE,
|
|
||||||
WIP_TITLE_SQL_PREFILTER,
|
|
||||||
matches_soft_wip_title,
|
|
||||||
matches_wip_title,
|
|
||||||
resolve_wip_tag_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
with SessionLocal() as session:
|
|
||||||
tag_id = resolve_wip_tag_id(session)
|
|
||||||
if tag_id is None:
|
|
||||||
log.warning(
|
|
||||||
"backfill_wip_title_tags: no `wip` system tag present; nothing to do"
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
settings = ImportSettings.load_sync(session)
|
|
||||||
applied = _backfill_wip_tier(
|
|
||||||
session, tag_id, WIP_TITLE_SQL_PREFILTER, matches_wip_title,
|
|
||||||
WIP_TITLE_SOURCE,
|
|
||||||
)
|
|
||||||
if settings.wip_soft_title_tagging_enabled:
|
|
||||||
applied += _backfill_wip_tier(
|
|
||||||
session, tag_id, SOFT_WIP_TITLE_SQL_PREFILTER, matches_soft_wip_title,
|
|
||||||
WIP_TITLE_SOFT_SOURCE,
|
|
||||||
)
|
|
||||||
if applied:
|
|
||||||
log.info("backfill_wip_title_tags: applied wip to %d image(s)", applied)
|
|
||||||
return applied
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
|
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
|
||||||
def vacuum_analyze() -> dict:
|
def vacuum_analyze() -> dict:
|
||||||
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
|
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
|
||||||
|
|||||||
+36
-52
@@ -105,7 +105,9 @@ def embed_image(self, image_id: int) -> dict:
|
|||||||
record = session.get(ImageRecord, image_id)
|
record = session.get(ImageRecord, image_id)
|
||||||
if record is None:
|
if record is None:
|
||||||
return {"status": "missing", "image_id": image_id}
|
return {"status": "missing", "image_id": image_id}
|
||||||
settings = MLSettings.load_sync(session)
|
settings = session.execute(
|
||||||
|
select(MLSettings).where(MLSettings.id == 1)
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
src = Path(record.path)
|
src = Path(record.path)
|
||||||
is_vid = _is_video(src)
|
is_vid = _is_video(src)
|
||||||
@@ -486,10 +488,15 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
from sqlalchemy import select as sa_select
|
from sqlalchemy import select as sa_select
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
|
||||||
from ..models import ImageRegion, MLSettings, Tag, TagKind
|
from ..models import ImageRegion, MLSettings, Tag, TagKind, TagSuggestionRejection
|
||||||
from ..models.tag import image_tag
|
from ..models.tag import image_tag
|
||||||
from ..services.ml.ccip import _FIGURE_KINDS
|
|
||||||
from ..services.ml.training_data import _applied_or_rejected, _l2norm
|
fig = ("face", "figure")
|
||||||
|
|
||||||
|
def _l2(m):
|
||||||
|
n = np.linalg.norm(m, axis=1, keepdims=True)
|
||||||
|
n[n == 0] = 1.0
|
||||||
|
return m / n
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
@@ -514,7 +521,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
)
|
)
|
||||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||||
.where(Tag.kind == TagKind.character)
|
.where(Tag.kind == TagKind.character)
|
||||||
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
|
.where(ImageRegion.kind.in_(fig))
|
||||||
.where(ImageRegion.ccip_embedding.is_not(None))
|
.where(ImageRegion.ccip_embedding.is_not(None))
|
||||||
.where(ImageRegion.image_record_id.in_(single))
|
.where(ImageRegion.image_record_id.in_(single))
|
||||||
).all()
|
).all()
|
||||||
@@ -525,16 +532,29 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
for tid, vec in ref_rows:
|
for tid, vec in ref_rows:
|
||||||
by_char.setdefault(tid, []).append(vec)
|
by_char.setdefault(tid, []).append(vec)
|
||||||
ref_tags = list(by_char)
|
ref_tags = list(by_char)
|
||||||
mats = [_l2norm(np.asarray(by_char[t], dtype=np.float32), np) for t in ref_tags]
|
mats = [_l2(np.asarray(by_char[t], dtype=np.float32)) for t in ref_tags]
|
||||||
allref = np.vstack(mats) # (total, 768)
|
allref = np.vstack(mats) # (total, 768)
|
||||||
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
|
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
|
||||||
|
|
||||||
# Per character: images that already carry OR rejected the tag — skip.
|
# Per character: images that already carry OR rejected the tag — skip.
|
||||||
skip = _applied_or_rejected(session, ref_tags)
|
skip = {t: set() for t in ref_tags}
|
||||||
|
for t in ref_tags:
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
sa_select(image_tag.c.image_record_id).where(
|
||||||
|
image_tag.c.tag_id == t
|
||||||
|
)
|
||||||
|
):
|
||||||
|
skip[t].add(iid)
|
||||||
|
for (iid,) in session.execute(
|
||||||
|
sa_select(TagSuggestionRejection.image_record_id).where(
|
||||||
|
TagSuggestionRejection.tag_id == t
|
||||||
|
)
|
||||||
|
):
|
||||||
|
skip[t].add(iid)
|
||||||
|
|
||||||
img_ids = list(session.execute(
|
img_ids = list(session.execute(
|
||||||
sa_select(ImageRegion.image_record_id)
|
sa_select(ImageRegion.image_record_id)
|
||||||
.where(ImageRegion.kind.in_(_FIGURE_KINDS), ImageRegion.ccip_embedding.is_not(None))
|
.where(ImageRegion.kind.in_(fig), ImageRegion.ccip_embedding.is_not(None))
|
||||||
.distinct()
|
.distinct()
|
||||||
).scalars())
|
).scalars())
|
||||||
|
|
||||||
@@ -546,7 +566,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
|
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
|
||||||
.where(
|
.where(
|
||||||
ImageRegion.image_record_id.in_(chunk),
|
ImageRegion.image_record_id.in_(chunk),
|
||||||
ImageRegion.kind.in_(_FIGURE_KINDS),
|
ImageRegion.kind.in_(fig),
|
||||||
ImageRegion.ccip_embedding.is_not(None),
|
ImageRegion.ccip_embedding.is_not(None),
|
||||||
)
|
)
|
||||||
).all()
|
).all()
|
||||||
@@ -554,7 +574,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
for iid, vec in rows:
|
for iid, vec in rows:
|
||||||
by_img.setdefault(iid, []).append(vec)
|
by_img.setdefault(iid, []).append(vec)
|
||||||
for iid, vecs in by_img.items():
|
for iid, vecs in by_img.items():
|
||||||
q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768)
|
q = _l2(np.asarray(vecs, dtype=np.float32)) # (nq, 768)
|
||||||
colmax = (q @ allref.T).max(axis=0) # (total,)
|
colmax = (q @ allref.T).max(axis=0) # (total,)
|
||||||
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
|
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
|
||||||
for ci in np.where(charmax >= thr)[0]:
|
for ci in np.where(charmax >= thr)[0]:
|
||||||
@@ -579,54 +599,18 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
soft_time_limit=1800, time_limit=2100,
|
soft_time_limit=1800, time_limit=2100,
|
||||||
)
|
)
|
||||||
def scheduled_presentation_auto_apply() -> str:
|
def scheduled_presentation_auto_apply() -> str:
|
||||||
"""Auto-hide presentation chrome (banner) on a daily passive sweep (#141).
|
"""Auto-hide presentation chrome (banner / editor screenshot) on a daily
|
||||||
No-op unless presentation_auto_apply_enabled. Idempotent — already-tagged images
|
passive sweep (#141). No-op unless presentation_auto_apply_enabled. Idempotent
|
||||||
are skipped — so an interrupted run simply re-runs next cycle (that IS the
|
— already-hidden images are skipped — so an interrupted run simply re-runs next
|
||||||
recovery). Wall-clock bounded by the task time limits."""
|
cycle (that IS the recovery). Wall-clock bounded by the task time limits."""
|
||||||
from ..services.ml.heads import system_tag_auto_apply_sweep
|
from ..services.ml.heads import presentation_auto_apply_sweep
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
result = system_tag_auto_apply_sweep(session, mode="chrome")
|
result = presentation_auto_apply_sweep(session)
|
||||||
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.ml.scheduled_process_auto_apply",
|
|
||||||
soft_time_limit=1800, time_limit=2100,
|
|
||||||
)
|
|
||||||
def scheduled_process_auto_apply() -> str:
|
|
||||||
"""Auto-apply the PROCESS system tags (wip / editor screenshot) on a daily
|
|
||||||
passive sweep (#1464) — provisional source, ring-loud review guard, image stays
|
|
||||||
VISIBLE. No-op unless process_auto_apply_enabled (opt-in). Idempotent —
|
|
||||||
already-tagged/rejected images are skipped — so an interrupted run just re-runs
|
|
||||||
next cycle (the recovery). Wall-clock bounded by the task time limits."""
|
|
||||||
from ..services.ml.heads import system_tag_auto_apply_sweep
|
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
with SessionLocal() as session:
|
|
||||||
result = system_tag_auto_apply_sweep(session, mode="process")
|
|
||||||
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
|
|
||||||
soft_time_limit=1800, time_limit=2100,
|
|
||||||
)
|
|
||||||
def scheduled_soft_wip_conflict_audit() -> str:
|
|
||||||
"""Ring-loud audit over the SOFT WIP-title cohort (#1474) — flag sketch/doodle
|
|
||||||
auto-tags that ALSO look like real content for review. No-op when there are no
|
|
||||||
content heads; idempotent (already-flagged images skipped). Runs regardless of
|
|
||||||
the process-sweep toggle, since soft-title tags come from the importer, not that
|
|
||||||
sweep. Wall-clock bounded by the task time limits."""
|
|
||||||
from ..services.ml.heads import soft_wip_conflict_audit
|
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
with SessionLocal() as session:
|
|
||||||
result = soft_wip_conflict_audit(session)
|
|
||||||
return f"scanned={result['n_scanned']} flagged={result['n_flagged']}"
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.ml.prune_presentation_reviews")
|
@celery.task(name="backend.app.tasks.ml.prune_presentation_reviews")
|
||||||
def prune_presentation_reviews() -> str:
|
def prune_presentation_reviews() -> str:
|
||||||
"""Retention (rule 89): drop RESOLVED presentation-review flags older than 30
|
"""Retention (rule 89): drop RESOLVED presentation-review flags older than 30
|
||||||
|
|||||||
@@ -60,17 +60,14 @@ _INTERRUPT_BACKOFF_MAX = 900
|
|||||||
# language scores low; require it to clear this floor, else keep the original.
|
# language scores low; require it to clear this floor, else keep the original.
|
||||||
# Calibrated 2026-07-09 against fresh (uncached) probes once Interpreter returned
|
# Calibrated 2026-07-09 against fresh (uncached) probes once Interpreter returned
|
||||||
# real langdetect confidence: genuine German detected at 1.0, while a correctly-
|
# real langdetect confidence: genuine German detected at 1.0, while a correctly-
|
||||||
# detected but ambiguous latin string dipped to 0.86 — so a single constant can't win.
|
# detected but ambiguous latin string dipped to 0.86 — so the floor sits at 0.80,
|
||||||
# The floor is now operator-tunable (ImportSettings.translation_min_confidence,
|
# below that band, to keep legitimate ambiguous non-English while still rejecting
|
||||||
# milestone 155) with a stricter 0.90 default; per-post overrides
|
# genuinely-unsure guesses. Real langdetect also fixed the original mis-flag at
|
||||||
# (Post.translation_override) rescue the false negatives it skips. Genuine German
|
# the source (short English titles now detect as English → passthrough), so this
|
||||||
# and mis-flagged short English both land ~0.86, and single-word mis-flags sit at
|
|
||||||
# a confident 1.0 no floor catches. Real langdetect fixed some cases at the
|
|
||||||
# source (some short English titles now detect as English → passthrough), so this
|
|
||||||
# floor is a safety net, not the primary fix. Re-tune via the "Test translation"
|
# floor is a safety net, not the primary fix. Re-tune via the "Test translation"
|
||||||
# box (send fresh text — cache hits report 1.0).
|
# box (send fresh text — cache hits report 1.0).
|
||||||
_CJK_LANGS = frozenset({"ja", "ko", "zh"})
|
_CJK_LANGS = frozenset({"ja", "ko", "zh"})
|
||||||
_DEFAULT_MIN_LATIN_CONFIDENCE = 0.90
|
_MIN_LATIN_CONFIDENCE = 0.80
|
||||||
|
|
||||||
|
|
||||||
def _interrupt_backoff(retry_after) -> int:
|
def _interrupt_backoff(retry_after) -> int:
|
||||||
@@ -103,10 +100,10 @@ def translate_posts(drain: bool = False) -> str:
|
|||||||
ready = _translation_config(session)
|
ready = _translation_config(session)
|
||||||
if isinstance(ready, str):
|
if isinstance(ready, str):
|
||||||
return ready
|
return ready
|
||||||
base_url, target, min_confidence = ready
|
base_url, target = ready
|
||||||
posts = _select_untranslated(session, None, _MAX_POSTS_PER_RUN)
|
posts = _select_untranslated(session, None, _MAX_POSTS_PER_RUN)
|
||||||
status, translated, retry_after = _translate_batch(
|
status, translated, retry_after = _translate_batch(
|
||||||
session, posts, base_url, target, min_confidence,
|
session, posts, base_url, target,
|
||||||
)
|
)
|
||||||
# Drain mode (manual "Translate now"): chase the tail until the backlog is
|
# Drain mode (manual "Translate now"): chase the tail until the backlog is
|
||||||
# zero, so one press clears the whole pile instead of one 300-chunk.
|
# zero, so one press clears the whole pile instead of one 300-chunk.
|
||||||
@@ -155,7 +152,7 @@ def retranslate_posts(artist_ids=None, _reset_done=False) -> str:
|
|||||||
ready = _translation_config(session)
|
ready = _translation_config(session)
|
||||||
if isinstance(ready, str):
|
if isinstance(ready, str):
|
||||||
return ready
|
return ready
|
||||||
base_url, target, min_confidence = ready
|
base_url, target = ready
|
||||||
|
|
||||||
if not _reset_done:
|
if not _reset_done:
|
||||||
n = _reset_translations(session, artist_ids)
|
n = _reset_translations(session, artist_ids)
|
||||||
@@ -166,7 +163,7 @@ def retranslate_posts(artist_ids=None, _reset_done=False) -> str:
|
|||||||
|
|
||||||
posts = _select_untranslated(session, artist_ids, _MAX_POSTS_PER_RUN)
|
posts = _select_untranslated(session, artist_ids, _MAX_POSTS_PER_RUN)
|
||||||
status, translated, retry_after = _translate_batch(
|
status, translated, retry_after = _translate_batch(
|
||||||
session, posts, base_url, target, min_confidence,
|
session, posts, base_url, target,
|
||||||
)
|
)
|
||||||
|
|
||||||
# run-until-done: chase the tail when the chunk finished cleanly.
|
# run-until-done: chase the tail when the chunk finished cleanly.
|
||||||
@@ -204,19 +201,17 @@ def retranslate_posts(artist_ids=None, _reset_done=False) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _translation_config(session):
|
def _translation_config(session):
|
||||||
"""Resolve (base_url, target, min_confidence) if translation is enabled +
|
"""Resolve (base_url, target) if translation is enabled + healthy, else a
|
||||||
healthy, else a short status string ("disabled" / "interpreter unavailable")
|
short status string ("disabled" / "interpreter unavailable") for the caller
|
||||||
for the caller to return. Centralises the guard so translate/retranslate agree
|
to return. Centralises the guard so translate/retranslate agree exactly."""
|
||||||
exactly, including the operator-tunable acceptance floor."""
|
|
||||||
cfg = ImportSettings.load_sync(session)
|
cfg = ImportSettings.load_sync(session)
|
||||||
if not cfg.translation_enabled or not cfg.interpreter_base_url.strip():
|
if not cfg.translation_enabled or not cfg.interpreter_base_url.strip():
|
||||||
return "disabled"
|
return "disabled"
|
||||||
base_url = cfg.interpreter_base_url.strip()
|
base_url = cfg.interpreter_base_url.strip()
|
||||||
target = (cfg.translation_target_lang or "en").strip() or "en"
|
target = (cfg.translation_target_lang or "en").strip() or "en"
|
||||||
min_confidence = cfg.translation_min_confidence
|
|
||||||
if not ic.health(base_url):
|
if not ic.health(base_url):
|
||||||
return "interpreter unavailable"
|
return "interpreter unavailable"
|
||||||
return base_url, target, min_confidence
|
return base_url, target
|
||||||
|
|
||||||
|
|
||||||
def _untranslated_filter(stmt, artist_ids):
|
def _untranslated_filter(stmt, artist_ids):
|
||||||
@@ -249,13 +244,10 @@ def _reset_translations(session, artist_ids) -> int:
|
|||||||
"""Clear the stored translation columns for scoped, already-handled posts so
|
"""Clear the stored translation columns for scoped, already-handled posts so
|
||||||
the untranslated sweep re-runs them. Only touches rows that were translated
|
the untranslated sweep re-runs them. Only touches rows that were translated
|
||||||
(translated_source_lang IS NOT NULL) — untranslated rows are already NULL.
|
(translated_source_lang IS NOT NULL) — untranslated rows are already NULL.
|
||||||
Skips 'keep original' posts (translation_override = 'original') so that choice
|
Returns the row count reset (commits it)."""
|
||||||
survives a Re-translate-all (milestone 155). Returns the row count reset
|
|
||||||
(commits it)."""
|
|
||||||
stmt = (
|
stmt = (
|
||||||
update(Post)
|
update(Post)
|
||||||
.where(Post.translated_source_lang.is_not(None))
|
.where(Post.translated_source_lang.is_not(None))
|
||||||
.where(Post.translation_override != "original")
|
|
||||||
.values(
|
.values(
|
||||||
post_title_translated=None,
|
post_title_translated=None,
|
||||||
description_translated=None,
|
description_translated=None,
|
||||||
@@ -271,7 +263,7 @@ def _reset_translations(session, artist_ids) -> int:
|
|||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
def _translate_batch(session, posts, base_url: str, target: str, min_confidence: float):
|
def _translate_batch(session, posts, base_url: str, target: str):
|
||||||
"""Translate each post in the chunk with a per-post commit (an interrupted
|
"""Translate each post in the chunk with a per-post commit (an interrupted
|
||||||
run keeps progress). Returns (status, translated, retry_after) where status is
|
run keeps progress). Returns (status, translated, retry_after) where status is
|
||||||
one of "ok" / "interrupted" (unavailable/drain) / "stopped" (400) / "timeout";
|
one of "ok" / "interrupted" (unavailable/drain) / "stopped" (400) / "timeout";
|
||||||
@@ -280,7 +272,7 @@ def _translate_batch(session, posts, base_url: str, target: str, min_confidence:
|
|||||||
translated = 0
|
translated = 0
|
||||||
for post in posts:
|
for post in posts:
|
||||||
try:
|
try:
|
||||||
translated += _translate_one(session, post, base_url, target, min_confidence)
|
translated += _translate_one(session, post, base_url, target)
|
||||||
session.commit()
|
session.commit()
|
||||||
except ic.InterpreterUnavailable as e:
|
except ic.InterpreterUnavailable as e:
|
||||||
session.rollback()
|
session.rollback()
|
||||||
@@ -304,36 +296,28 @@ def _summary(status: str, translated: int, scanned: int) -> str:
|
|||||||
return f"translated={translated} scanned={scanned}"
|
return f"translated={translated} scanned={scanned}"
|
||||||
|
|
||||||
|
|
||||||
def _accept(
|
def _accept(language: str, confidence) -> bool:
|
||||||
language: str, confidence, min_confidence: float = _DEFAULT_MIN_LATIN_CONFIDENCE,
|
|
||||||
) -> bool:
|
|
||||||
"""Should curator store Interpreter's translation, or keep the original?
|
"""Should curator store Interpreter's translation, or keep the original?
|
||||||
Consumes ONLY Interpreter's own detection (curator does none, Scribe rule
|
Consumes ONLY Interpreter's own detection (curator does none, Scribe rule
|
||||||
133): CJK languages are trusted regardless of the reported number; a
|
133): CJK languages are trusted regardless of the reported number; a
|
||||||
latin-script detection must clear ``min_confidence`` (the operator-tunable
|
latin-script detection must clear ``_MIN_LATIN_CONFIDENCE``. A missing
|
||||||
ImportSettings.translation_min_confidence). A missing confidence fails OPEN
|
confidence fails OPEN (accept), so a service that omits the field never causes
|
||||||
(accept), so a service that omits the field never silently drops a
|
translations to be silently dropped."""
|
||||||
translation. Per-post ``force`` overrides bypass this entirely (see
|
|
||||||
``_translate_one``)."""
|
|
||||||
if (language or "").split("-", 1)[0].lower() in _CJK_LANGS:
|
if (language or "").split("-", 1)[0].lower() in _CJK_LANGS:
|
||||||
return True
|
return True
|
||||||
if confidence is None:
|
if confidence is None:
|
||||||
return True
|
return True
|
||||||
return confidence >= min_confidence
|
return confidence >= _MIN_LATIN_CONFIDENCE
|
||||||
|
|
||||||
|
|
||||||
def _translate_field(
|
def _translate_field(text: str, base_url: str, target: str):
|
||||||
text: str, base_url: str, target: str, min_confidence: float, force: bool = False,
|
|
||||||
):
|
|
||||||
"""Translate ONE field independently. Returns (translated, source_lang,
|
"""Translate ONE field independently. Returns (translated, source_lang,
|
||||||
engine_version), or (None, None, None) when there's nothing to store — empty
|
engine_version), or (None, None, None) when there's nothing to store — empty
|
||||||
text, already the target language, a passthrough (engine "none"), or a
|
text, already the target language, a passthrough (engine "none"), or a
|
||||||
latin-script detection the acceptance gate rejects as too low-confidence (kept
|
latin-script detection the acceptance gate rejects as too low-confidence (kept
|
||||||
as the original). ``force`` (a per-post 'force' override) bypasses the
|
as the original). Per-field (not aggregate first-item) detection so a
|
||||||
confidence gate so a legitimately-foreign title Interpreter is only ~0.86 sure
|
non-English description still gets translated when the title is already
|
||||||
about is still stored; a genuine passthrough stores nothing regardless (there
|
English (mixed-language posts)."""
|
||||||
is no translation to force). Per-field (not aggregate first-item) detection so
|
|
||||||
a non-English description still translates when the title is already English."""
|
|
||||||
if not text:
|
if not text:
|
||||||
return None, None, None
|
return None, None, None
|
||||||
res = ic.translate([text], base_url=base_url, target=target)
|
res = ic.translate([text], base_url=base_url, target=target)
|
||||||
@@ -343,30 +327,23 @@ def _translate_field(
|
|||||||
# Trust only Interpreter's own detection (Scribe rule 133): keep the original
|
# Trust only Interpreter's own detection (Scribe rule 133): keep the original
|
||||||
# when a latin-script detection doesn't clear the confidence floor, so a short
|
# when a latin-script detection doesn't clear the confidence floor, so a short
|
||||||
# English title mis-labelled as e.g. German isn't rewritten into the archive.
|
# English title mis-labelled as e.g. German isn't rewritten into the archive.
|
||||||
# A 'force' override skips the gate — the operator is overriding a rejection.
|
if not _accept(detected, res.get("detected_confidence")):
|
||||||
if not force and not _accept(
|
|
||||||
detected, res.get("detected_confidence"), min_confidence,
|
|
||||||
):
|
|
||||||
return None, None, None
|
return None, None, None
|
||||||
return res["translations"][0], detected, res["engine_version"]
|
return res["translations"][0], detected, res["engine_version"]
|
||||||
|
|
||||||
|
|
||||||
def _store_translation(post, title_res, desc_res, target: str) -> int:
|
def _translate_one(session, post, base_url: str, target: str) -> int:
|
||||||
"""Apply per-field (translated, source_lang, engine_version) results to a post
|
"""Translate a post's title/description in place, each field independently.
|
||||||
in place. Returns 1 if a translation was stored, 0 when the post is all
|
Returns 1 if it stored at least one translation, 0 when the post is all
|
||||||
passthrough/empty — in which case the translated columns are CLEARED and the
|
passthrough/empty (still marks it handled so the sweep won't revisit it)."""
|
||||||
post is marked handled (translated_source_lang = target). Clearing (not just
|
title = (post.post_title or "").strip()
|
||||||
leaving) matters on re-apply: a 'keep original' override, or a re-translate
|
desc = (html_to_plain(post.description) if post.description else "") or ""
|
||||||
that now rejects a previously-accepted mis-flag, must remove the stale
|
desc = desc.strip()
|
||||||
translation, not keep it."""
|
title_tr, title_lang, title_ev = _translate_field(title, base_url, target)
|
||||||
title_tr, title_lang, title_ev = title_res
|
desc_tr, desc_lang, desc_ev = _translate_field(desc, base_url, target)
|
||||||
desc_tr, desc_lang, desc_ev = desc_res
|
|
||||||
if title_tr is None and desc_tr is None:
|
if title_tr is None and desc_tr is None:
|
||||||
post.post_title_translated = None
|
# Nothing non-target (or nothing to translate) → handled, store nothing.
|
||||||
post.description_translated = None
|
|
||||||
post.translated_source_lang = target
|
post.translated_source_lang = target
|
||||||
post.translation_engine_version = None
|
|
||||||
post.translated_at = None
|
|
||||||
return 0
|
return 0
|
||||||
post.post_title_translated = title_tr
|
post.post_title_translated = title_tr
|
||||||
post.description_translated = desc_tr
|
post.description_translated = desc_tr
|
||||||
@@ -376,23 +353,3 @@ def _store_translation(post, title_res, desc_res, target: str) -> int:
|
|||||||
post.translation_engine_version = title_ev or desc_ev
|
post.translation_engine_version = title_ev or desc_ev
|
||||||
post.translated_at = datetime.now(UTC)
|
post.translated_at = datetime.now(UTC)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
|
|
||||||
def _translate_one(
|
|
||||||
session, post, base_url: str, target: str, min_confidence: float,
|
|
||||||
) -> int:
|
|
||||||
"""Translate a post's title/description in place, each field independently,
|
|
||||||
honoring the per-post override (milestone 155): 'original' keeps the original
|
|
||||||
(clears any stored translation, no Interpreter call); 'force' translates even
|
|
||||||
below the confidence floor; 'auto' (default) runs the acceptance gate. Returns
|
|
||||||
1 if it stored a translation, else 0 (still marks the post handled so the sweep
|
|
||||||
won't revisit it)."""
|
|
||||||
if post.translation_override == "original":
|
|
||||||
return _store_translation(post, (None, None, None), (None, None, None), target)
|
|
||||||
force = post.translation_override == "force"
|
|
||||||
title = (post.post_title or "").strip()
|
|
||||||
desc = (html_to_plain(post.description) if post.description else "") or ""
|
|
||||||
desc = desc.strip()
|
|
||||||
title_res = _translate_field(title, base_url, target, min_confidence, force=force)
|
|
||||||
desc_res = _translate_field(desc, base_url, target, min_confidence, force=force)
|
|
||||||
return _store_translation(post, title_res, desc_res, target)
|
|
||||||
|
|||||||
+3
-39
@@ -11,26 +11,12 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
|
|||||||
- python 3.14
|
- python 3.14
|
||||||
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
|
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
|
||||||
- node (frontend job: `npm install` + vitest + vite build)
|
- node (frontend job: `npm install` + vitest + vite build)
|
||||||
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
|
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Forgejo registry push)
|
||||||
|
|
||||||
## Secondary runtime image
|
|
||||||
|
|
||||||
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
|
|
||||||
|
|
||||||
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
|
|
||||||
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
|
|
||||||
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
|
|
||||||
per `docs/process.md`'s "add deps to the image when used by >1 project".
|
|
||||||
|
|
||||||
## Per-job tool installs
|
## Per-job tool installs
|
||||||
|
|
||||||
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
|
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
|
||||||
- `npm install --no-audit --no-fund` — in `frontend-build` job
|
- `npm install --no-audit --no-fund` — in `frontend-build` job
|
||||||
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
|
|
||||||
- `unzip` — in `extension.yml`'s "Verify XPI contents" step, installed via apt
|
|
||||||
only when absent (`node:24-bookworm-slim` may or may not carry it). Debian
|
|
||||||
package, ~2s. Not worth baking into a shared image for a single consumer, per
|
|
||||||
`docs/process.md`'s ">1 project" rule.
|
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
@@ -40,34 +26,12 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
|
|||||||
"add deps to image when used by >1 project" rule: FC alone is one Python
|
"add deps to image when used by >1 project" rule: FC alone is one Python
|
||||||
project, so the deps live in `requirements.txt` and install per-job.
|
project, so the deps live in `requirements.txt` and install per-job.
|
||||||
Reconsider when a second Fabled-family Python backend lands.
|
Reconsider when a second Fabled-family Python backend lands.
|
||||||
- Integration uses Fabled-Git Actions `services:` + socket-discovered bridge IPs
|
- Integration uses Forgejo Actions `services:` + socket-discovered bridge IPs
|
||||||
because `act_runner` (swarm-runner v0.6+) puts services on the default
|
because `act_runner` (swarm-runner v0.6+) puts services on the default
|
||||||
bridge with no embedded DNS. The pattern is documented in the rulebook's
|
bridge with no embedded DNS. The pattern is documented in the rulebook's
|
||||||
`fabled-git.md` "CI philosophy" section and FC's `ci.yml` is the canonical
|
`forgejo.md` "CI philosophy" section and FC's `ci.yml` is the canonical
|
||||||
example.
|
example.
|
||||||
- No `package-lock.json` is tracked yet (FC's `feedback_no_local_runs`
|
- No `package-lock.json` is tracked yet (FC's `feedback_no_local_runs`
|
||||||
memory bans `npm install` locally). Using `npm install` rather than
|
memory bans `npm install` locally). Using `npm install` rather than
|
||||||
`npm ci` until a lockfile lands.
|
`npm ci` until a lockfile lands.
|
||||||
- No `imagemagick` / `pandoc` per-job installs needed.
|
- No `imagemagick` / `pandoc` per-job installs needed.
|
||||||
- `extension/`'s vitest specs load `lib/*.js` by evaluating the real file as a
|
|
||||||
classic script (`test/helpers/loadLib.js`) rather than adding `module.exports`
|
|
||||||
shims to production code — the libs ship as `background.scripts`, not ES
|
|
||||||
modules, so the specs exercise exactly the bytes packaged into the XPI.
|
|
||||||
- **`extension/scripts/packaging.sh` is the single definition of what ships
|
|
||||||
inside the XPI.** Three consumers read from it rather than keeping their own
|
|
||||||
copy: web-ext's `--ignore-files` (`extension/package.json`), the `:(exclude)`
|
|
||||||
pathspec in `ci.yml`'s `extension-version` guard, and the `git log` pathspec
|
|
||||||
that derives the extension version. Three hand-kept copies of that one fact
|
|
||||||
is what allowed issue #2397.
|
|
||||||
- Jobs that derive the extension version check out with `fetch-depth: 0`. The
|
|
||||||
version is the commit TIME of the newest packaged-extension change (minutes
|
|
||||||
since 2020-01-01, per family rule 149 — never a commit count, which orders
|
|
||||||
by branch rather than by recency). A depth-1 clone sees one commit and
|
|
||||||
derives a wrong, too-low value rather than failing, so the full-history
|
|
||||||
checkout is load-bearing wherever `packaging.sh version` is called.
|
|
||||||
- Callers MUST `set -f` before substituting the script's output. Without it the
|
|
||||||
shell expands `test/**` against the working tree and silently narrows the
|
|
||||||
pattern to whatever files exist at that moment — a failure that looks like
|
|
||||||
nothing until dev files start appearing in the XPI. `test/version.spec.js`
|
|
||||||
asserts every `--ignore-files` consumer sets it, and that no consumer has
|
|
||||||
quietly reinstated a hardcoded list.
|
|
||||||
|
|||||||
+3
-3
@@ -1,9 +1,9 @@
|
|||||||
# FabledCurator Firefox Extension
|
# FabledCurator Firefox Extension
|
||||||
|
|
||||||
Self-hosted Firefox extension that pushes session cookies from supported
|
Self-hosted Firefox extension that pushes session cookies from supported
|
||||||
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv)
|
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv,
|
||||||
into FabledCurator, and lets you add a creator as a Source from their
|
DeviantArt) into FabledCurator, and lets you add a creator as a Source
|
||||||
page in one click.
|
from their page in one click.
|
||||||
|
|
||||||
## Install (operator)
|
## Install (operator)
|
||||||
|
|
||||||
|
|||||||
@@ -31,68 +31,6 @@ browser.runtime.onInstalled.addListener(() => ensureInitialized());
|
|||||||
browser.runtime.onStartup.addListener(() => ensureInitialized());
|
browser.runtime.onStartup.addListener(() => ensureInitialized());
|
||||||
ensureInitialized().catch(e => console.error('init failed:', e));
|
ensureInitialized().catch(e => console.error('init failed:', e));
|
||||||
|
|
||||||
// ---- Extension self-update check (#1489) ----
|
|
||||||
// Installed per-instance from the operator's FC host, so Firefox's static
|
|
||||||
// update_url can't apply (each instance has a different host). Instead ask the
|
|
||||||
// configured backend for the latest published version and nudge the operator to
|
|
||||||
// reinstall the freshly-signed XPI — surfaced as a popup banner (on demand) and
|
|
||||||
// a toolbar badge (daily). /api/extension/manifest is public and returns
|
|
||||||
// {version, latest_url, sha256}; the XPI is served from the web root (not /api).
|
|
||||||
|
|
||||||
function versionIsNewer(candidate, current) {
|
|
||||||
// Dotted numeric compare so 1.0.10 > 1.0.9 (a plain string compare wouldn't).
|
|
||||||
const a = String(candidate).split('.').map(n => parseInt(n, 10) || 0);
|
|
||||||
const b = String(current).split('.').map(n => parseInt(n, 10) || 0);
|
|
||||||
for (let i = 0; i < Math.max(a.length, b.length); i++) {
|
|
||||||
if ((a[i] || 0) !== (b[i] || 0)) return (a[i] || 0) > (b[i] || 0);
|
|
||||||
}
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function checkForUpdateInfo() {
|
|
||||||
await ensureInitialized();
|
|
||||||
if (!api.isConfigured()) return { updateAvailable: false, configured: false };
|
|
||||||
let info;
|
|
||||||
try {
|
|
||||||
info = await api.getExtensionManifest();
|
|
||||||
} catch (e) {
|
|
||||||
return { updateAvailable: false, error: e.message };
|
|
||||||
}
|
|
||||||
const currentVersion = browser.runtime.getManifest().version;
|
|
||||||
const latestVersion = info && info.version ? info.version : null;
|
|
||||||
// latest_url is served from the web root, not the JSON API.
|
|
||||||
const base = api.webRoot();
|
|
||||||
return {
|
|
||||||
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
|
|
||||||
currentVersion,
|
|
||||||
latestVersion,
|
|
||||||
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
async function refreshUpdateBadge() {
|
|
||||||
let r;
|
|
||||||
try { r = await checkForUpdateInfo(); } catch { return; }
|
|
||||||
try {
|
|
||||||
await browser.action.setBadgeText({ text: r.updateAvailable ? '↑' : '' });
|
|
||||||
if (r.updateAvailable) {
|
|
||||||
await browser.action.setBadgeBackgroundColor({ color: '#F4BA7A' });
|
|
||||||
await browser.action.setTitle({ title: `FabledCurator — update available (v${r.latestVersion})` });
|
|
||||||
} else {
|
|
||||||
await browser.action.setTitle({ title: 'FabledCurator' });
|
|
||||||
}
|
|
||||||
} catch { /* action API unavailable — non-fatal */ }
|
|
||||||
}
|
|
||||||
|
|
||||||
// Daily proactive check (needs the "alarms" permission). create() is idempotent
|
|
||||||
// by name, so re-running it on each event-page load is safe.
|
|
||||||
browser.alarms.create('fc-update-check', { periodInMinutes: 24 * 60, delayInMinutes: 1 });
|
|
||||||
browser.alarms.onAlarm.addListener((alarm) => {
|
|
||||||
if (alarm.name === 'fc-update-check') refreshUpdateBadge();
|
|
||||||
});
|
|
||||||
browser.runtime.onStartup.addListener(() => refreshUpdateBadge());
|
|
||||||
browser.runtime.onInstalled.addListener(() => refreshUpdateBadge());
|
|
||||||
|
|
||||||
// ---- Discord token capture via webRequest ----
|
// ---- Discord token capture via webRequest ----
|
||||||
|
|
||||||
browser.webRequest.onBeforeSendHeaders.addListener(
|
browser.webRequest.onBeforeSendHeaders.addListener(
|
||||||
@@ -210,21 +148,6 @@ browser.webRequest.onBeforeRedirect.addListener(
|
|||||||
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
|
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
|
||||||
);
|
);
|
||||||
|
|
||||||
// Extract → verify → upload one cookie-auth platform. Returns a structured
|
|
||||||
// outcome so the two callers (EXPORT_COOKIES single, EXPORT_ALL_COOKIES) shape
|
|
||||||
// their own response + skip semantics. Verifies the captured cookies are
|
|
||||||
// actually live BEFORE uploading, so a confirmed-stale session doesn't overwrite
|
|
||||||
// good FC-side credentials; platforms with no verify config (v.ok === null) fall
|
|
||||||
// through to upload.
|
|
||||||
async function exportPlatformCookies(key) {
|
|
||||||
const cookies = await extractCookiesForPlatform(key);
|
|
||||||
if (cookies.length === 0) return { status: 'empty' };
|
|
||||||
const v = await verifyCookiesForPlatform(key);
|
|
||||||
if (v.ok === false) return { status: 'stale', reason: v.reason, cookieCount: cookies.length };
|
|
||||||
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
|
|
||||||
return { status: 'ok', cookieCount: cookies.length, verified: v.ok === true };
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---- Message router ----
|
// ---- Message router ----
|
||||||
|
|
||||||
browser.runtime.onMessage.addListener(async (msg) => {
|
browser.runtime.onMessage.addListener(async (msg) => {
|
||||||
@@ -269,14 +192,22 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
if (!platform) return { error: `Unknown platform: ${key}` };
|
if (!platform) return { error: `Unknown platform: ${key}` };
|
||||||
try {
|
try {
|
||||||
if (platform.authType === 'cookies') {
|
if (platform.authType === 'cookies') {
|
||||||
const r = await exportPlatformCookies(key);
|
const cookies = await extractCookiesForPlatform(key);
|
||||||
if (r.status === 'empty') return { error: 'No cookies found — log in first.' };
|
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
|
||||||
if (r.status === 'stale') {
|
// Verify the captured cookies are actually live BEFORE
|
||||||
|
// uploading. Skips upload on confirmed-stale sessions so we
|
||||||
|
// don't overwrite FC-side credentials with garbage. Platforms
|
||||||
|
// without a verify config (verify.ok === null) fall through
|
||||||
|
// to upload as before.
|
||||||
|
const v = await verifyCookiesForPlatform(key);
|
||||||
|
if (v.ok === false) {
|
||||||
return {
|
return {
|
||||||
error: `Captured ${r.cookieCount} ${platform.name} cookies but they don't appear authenticated (${r.reason}). Log in again in this browser, then retry.`,
|
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
return { success: true, cookieCount: r.cookieCount, verified: r.verified };
|
const data = toNetscapeFormat(cookies);
|
||||||
|
await api.uploadCredentials(key, 'cookies', data);
|
||||||
|
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
|
||||||
}
|
}
|
||||||
if (key === 'discord') {
|
if (key === 'discord') {
|
||||||
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
|
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
|
||||||
@@ -304,10 +235,18 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
const r = await exportPlatformCookies(key);
|
const cookies = await extractCookiesForPlatform(key);
|
||||||
if (r.status === 'empty') results[key] = { skipped: true, reason: 'no cookies' };
|
if (cookies.length === 0) {
|
||||||
else if (r.status === 'stale') results[key] = { error: `verify failed: ${r.reason}` };
|
results[key] = { skipped: true, reason: 'no cookies' };
|
||||||
else results[key] = { success: true, cookieCount: r.cookieCount, verified: r.verified };
|
continue;
|
||||||
|
}
|
||||||
|
const v = await verifyCookiesForPlatform(key);
|
||||||
|
if (v.ok === false) {
|
||||||
|
results[key] = { error: `verify failed: ${v.reason}` };
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
|
||||||
|
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
results[key] = { error: e.message };
|
results[key] = { error: e.message };
|
||||||
}
|
}
|
||||||
@@ -344,9 +283,11 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case 'OPEN_ARTIST_PAGE': {
|
case 'OPEN_ARTIST_PAGE': {
|
||||||
// The SPA artist route (/artist/:slug) is served from the web root, not
|
// apiUrl is configured with the /api suffix (see
|
||||||
// the JSON API — see api.webRoot().
|
// options/options.html placeholder); the SPA artist route is
|
||||||
const base = api.webRoot();
|
// /artist/:slug, served from the same origin. Strip /api so the
|
||||||
|
// browser-level URL hits the Vue router, not the JSON API.
|
||||||
|
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
|
||||||
const slug = encodeURIComponent(msg.slug || '');
|
const slug = encodeURIComponent(msg.slug || '');
|
||||||
if (!base || !slug) return { error: 'apiUrl or slug missing' };
|
if (!base || !slug) return { error: 'apiUrl or slug missing' };
|
||||||
try {
|
try {
|
||||||
@@ -357,9 +298,6 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
case 'CHECK_UPDATE':
|
|
||||||
return await checkForUpdateInfo();
|
|
||||||
|
|
||||||
default:
|
default:
|
||||||
return { error: `Unknown message type: ${msg.type}` };
|
return { error: `Unknown message type: ${msg.type}` };
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-23
@@ -11,10 +11,7 @@ class FabledCuratorAPI {
|
|||||||
|
|
||||||
async init() {
|
async init() {
|
||||||
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
|
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
|
||||||
// Normalize on READ, not just on save: configs stored before the options
|
this.baseUrl = cfg.apiUrl || null;
|
||||||
// page started normalizing are missing the `/api` suffix, and this heals
|
|
||||||
// them without the operator having to reopen Settings.
|
|
||||||
this.baseUrl = normalizeApiUrl(cfg.apiUrl) || null;
|
|
||||||
this.apiKey = cfg.apiKey || null;
|
this.apiKey = cfg.apiKey || null;
|
||||||
return this.isConfigured();
|
return this.isConfigured();
|
||||||
}
|
}
|
||||||
@@ -53,13 +50,6 @@ class FabledCuratorAPI {
|
|||||||
} catch {
|
} catch {
|
||||||
message = `HTTP ${response.status}: ${response.statusText}`;
|
message = `HTTP ${response.status}: ${response.statusText}`;
|
||||||
}
|
}
|
||||||
// 404/405 from FC almost always means the request never reached the JSON
|
|
||||||
// API — it fell through to the SPA catch-all, which serves HTML on GET
|
|
||||||
// and rejects everything else. Say so, rather than making the operator
|
|
||||||
// decode "Method Not Allowed" on an endpoint that plainly allows POST.
|
|
||||||
if (response.status === 404 || response.status === 405) {
|
|
||||||
message += ` — ${url} isn't the FC API. Check the FC URL in settings.`;
|
|
||||||
}
|
|
||||||
const err = new Error(message);
|
const err = new Error(message);
|
||||||
err.status = response.status;
|
err.status = response.status;
|
||||||
throw err;
|
throw err;
|
||||||
@@ -99,18 +89,6 @@ class FabledCuratorAPI {
|
|||||||
const qs = new URLSearchParams({ url }).toString();
|
const qs = new URLSearchParams({ url }).toString();
|
||||||
return this.request('GET', `/extension/probe?${qs}`);
|
return this.request('GET', `/extension/probe?${qs}`);
|
||||||
}
|
}
|
||||||
// Latest published extension version on this instance — drives the in-app
|
|
||||||
// update prompt. Public endpoint (no key needed, but request() sends it
|
|
||||||
// harmlessly). Returns {version, xpi_url, latest_url, sha256}.
|
|
||||||
getExtensionManifest() {
|
|
||||||
return this.request('GET', '/extension/manifest');
|
|
||||||
}
|
|
||||||
|
|
||||||
// The web/SPA root: where the Vue router (artist pages) and the served XPI
|
|
||||||
// live, NOT the JSON API. Used by OPEN_ARTIST_PAGE + the self-update check.
|
|
||||||
webRoot() {
|
|
||||||
return webRootFromApiUrl(this.baseUrl);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Connection test = the cheapest read with auth.
|
// Connection test = the cheapest read with auth.
|
||||||
testConnection() {
|
testConnection() {
|
||||||
|
|||||||
+12
-10
@@ -68,6 +68,16 @@ const PLATFORMS = {
|
|||||||
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
|
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
|
||||||
note: 'Click to authenticate via OAuth',
|
note: 'Click to authenticate via OAuth',
|
||||||
},
|
},
|
||||||
|
deviantart: {
|
||||||
|
name: 'DeviantArt',
|
||||||
|
domains: ['.deviantart.com', 'www.deviantart.com', 'deviantart.com'],
|
||||||
|
authType: 'cookies',
|
||||||
|
color: '#05CC47',
|
||||||
|
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
|
||||||
|
// DA's logged-in-only endpoints sit behind their internal _napi
|
||||||
|
// namespace which shifts; skipping verify until a stable check
|
||||||
|
// surfaces. Same posture as SubscribeStar.
|
||||||
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -76,18 +86,10 @@ const PLATFORMS = {
|
|||||||
* script to decide whether to show the floating "Add as source" button.
|
* script to decide whether to show the floating "Add as source" button.
|
||||||
*/
|
*/
|
||||||
const PLATFORM_ARTIST_PATTERNS = {
|
const PLATFORM_ARTIST_PATTERNS = {
|
||||||
// Patreon serves the same creator under three URL shapes (see backend
|
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c\/)[^/?#]+\/?$/i,
|
||||||
// patreon_resolver._VANITY_RE): bare `patreon.com/Atole`, `c/` prefix, and
|
|
||||||
// `cw/` "creator workspace" — the last is the URL you land on once you're
|
|
||||||
// SUBSCRIBED, which is exactly when the button matters. Match all three, and
|
|
||||||
// drop the single-segment end-anchor so a creator's inner page
|
|
||||||
// (…/cw/Atole/posts, …/Atole/membership) also injects the button. Nav pages
|
|
||||||
// (home/search/…/posts permalink) stay excluded. Mirrors extension_service
|
|
||||||
// ._PLATFORM_PATTERNS — keep in sync (operator-flagged 2026-07-13: button
|
|
||||||
// vanished once subscribed because the old pattern only matched the bare root).
|
|
||||||
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?:cw\/|c\/)?(?!(?:home|search|messages|notifications|library|settings|posts)(?:[\/?#]|$))[^/?#]+/i,
|
|
||||||
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
|
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
|
||||||
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
|
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
|
||||||
|
deviantart: /^https?:\/\/(www\.)?deviantart\.com\/(?!home$|watch\b|tag\b|browse\b)[^/?#]+\/?$/i,
|
||||||
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
|
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -1,32 +0,0 @@
|
|||||||
/**
|
|
||||||
* Canonical FC endpoint derivation, shared by the background client and the
|
|
||||||
* options page so a URL entered either way behaves identically.
|
|
||||||
*
|
|
||||||
* FC serves two things on one origin: the JSON API under `/api`, and the Vue
|
|
||||||
* SPA from the root. `api.js` builds requests as `${baseUrl}/credentials`, so
|
|
||||||
* the stored base URL has to carry the `/api` suffix.
|
|
||||||
*/
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Accept what an operator would naturally type — the instance root
|
|
||||||
* (`http://curator.example.com`) or the API root (`.../api`) — and return the
|
|
||||||
* API root either way.
|
|
||||||
*
|
|
||||||
* Worth normalizing rather than validating: a root-form URL doesn't fail
|
|
||||||
* loudly, it lands on the SPA catch-all, which answers `GET /credentials` with
|
|
||||||
* 200 HTML and rejects `POST /credentials` with 405. The operator sees a
|
|
||||||
* working Test Connection and a broken export.
|
|
||||||
*/
|
|
||||||
function normalizeApiUrl(raw) {
|
|
||||||
const trimmed = (raw || '').trim().replace(/\/+$/, '');
|
|
||||||
if (!trimmed) return '';
|
|
||||||
return /\/api$/i.test(trimmed) ? trimmed : `${trimmed}/api`;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* The SPA root — where the Vue router (artist pages) and the served XPI live,
|
|
||||||
* NOT the JSON API. Accepts either input form, same as normalizeApiUrl.
|
|
||||||
*/
|
|
||||||
function webRootFromApiUrl(raw) {
|
|
||||||
return normalizeApiUrl(raw).replace(/\/api$/i, '');
|
|
||||||
}
|
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"manifest_version": 3,
|
"manifest_version": 3,
|
||||||
"name": "FabledCurator",
|
"name": "FabledCurator",
|
||||||
"version": "1.0.11",
|
"version": "1.0.7",
|
||||||
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
|
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
|
||||||
|
|
||||||
"browser_specific_settings": {
|
"browser_specific_settings": {
|
||||||
@@ -22,8 +22,7 @@
|
|||||||
"tabs",
|
"tabs",
|
||||||
"activeTab",
|
"activeTab",
|
||||||
"webRequest",
|
"webRequest",
|
||||||
"webRequestBlocking",
|
"webRequestBlocking"
|
||||||
"alarms"
|
|
||||||
],
|
],
|
||||||
|
|
||||||
"host_permissions": [
|
"host_permissions": [
|
||||||
@@ -33,6 +32,7 @@
|
|||||||
"*://*.hentai-foundry.com/*",
|
"*://*.hentai-foundry.com/*",
|
||||||
"*://*.discord.com/*",
|
"*://*.discord.com/*",
|
||||||
"*://*.pixiv.net/*",
|
"*://*.pixiv.net/*",
|
||||||
|
"*://*.deviantart.com/*",
|
||||||
"*://app-api.pixiv.net/*",
|
"*://app-api.pixiv.net/*",
|
||||||
"*://oauth.secure.pixiv.net/*",
|
"*://oauth.secure.pixiv.net/*",
|
||||||
"*://*/*"
|
"*://*/*"
|
||||||
@@ -45,7 +45,7 @@
|
|||||||
},
|
},
|
||||||
|
|
||||||
"background": {
|
"background": {
|
||||||
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/url.js", "lib/api.js", "background/background.js"]
|
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/api.js", "background/background.js"]
|
||||||
},
|
},
|
||||||
|
|
||||||
"options_ui": {
|
"options_ui": {
|
||||||
@@ -60,6 +60,7 @@
|
|||||||
"*://*.subscribestar.com/*",
|
"*://*.subscribestar.com/*",
|
||||||
"*://*.subscribestar.adult/*",
|
"*://*.subscribestar.adult/*",
|
||||||
"*://*.hentai-foundry.com/*",
|
"*://*.hentai-foundry.com/*",
|
||||||
|
"*://*.deviantart.com/*",
|
||||||
"*://*.pixiv.net/*"
|
"*://*.pixiv.net/*"
|
||||||
],
|
],
|
||||||
"js": ["lib/platforms.js", "content/content-script.js"],
|
"js": ["lib/platforms.js", "content/content-script.js"],
|
||||||
|
|||||||
@@ -21,12 +21,9 @@
|
|||||||
<body>
|
<body>
|
||||||
<h1>FabledCurator extension</h1>
|
<h1>FabledCurator extension</h1>
|
||||||
|
|
||||||
<label for="api-url">FC instance URL</label>
|
<label for="api-url">FC base URL</label>
|
||||||
<input id="api-url" type="url" placeholder="http://curator.example.com" />
|
<input id="api-url" type="url" placeholder="http://curator.example.com/api" />
|
||||||
<div class="hint">
|
<div class="hint">Find this on FC → Settings → Maintenance → Browser extension.</div>
|
||||||
Your FabledCurator address — with or without the trailing <code>/api</code>; both work.
|
|
||||||
Find it on FC → Settings → Maintenance → Browser extension.
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<label for="api-key">Extension API key</label>
|
<label for="api-key">Extension API key</label>
|
||||||
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
|
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
|
||||||
@@ -39,7 +36,6 @@
|
|||||||
|
|
||||||
<div id="status" class="status" style="display:none;"></div>
|
<div id="status" class="status" style="display:none;"></div>
|
||||||
|
|
||||||
<script src="../lib/url.js"></script>
|
|
||||||
<script src="options.js"></script>
|
<script src="options.js"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
async function save() {
|
async function save() {
|
||||||
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
|
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
|
||||||
const apiKey = document.getElementById('api-key').value.trim();
|
const apiKey = document.getElementById('api-key').value.trim();
|
||||||
if (!apiUrl || !apiKey) {
|
if (!apiUrl || !apiKey) {
|
||||||
showStatus('Both fields are required.', 'err');
|
showStatus('Both fields are required.', 'err');
|
||||||
@@ -16,14 +16,11 @@ async function save() {
|
|||||||
}
|
}
|
||||||
await browser.storage.local.set({ apiUrl, apiKey });
|
await browser.storage.local.set({ apiUrl, apiKey });
|
||||||
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
|
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
|
||||||
// Show what was actually stored — the operator may have typed the instance
|
showStatus('Saved.', 'ok');
|
||||||
// root and it was normalized to the API root.
|
|
||||||
document.getElementById('api-url').value = apiUrl;
|
|
||||||
showStatus(`Saved — using ${apiUrl}`, 'ok');
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function test() {
|
async function test() {
|
||||||
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
|
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
|
||||||
const apiKey = document.getElementById('api-key').value.trim();
|
const apiKey = document.getElementById('api-key').value.trim();
|
||||||
if (!apiUrl || !apiKey) {
|
if (!apiUrl || !apiKey) {
|
||||||
showStatus('Fill both fields first.', 'err');
|
showStatus('Fill both fields first.', 'err');
|
||||||
@@ -34,23 +31,8 @@ async function test() {
|
|||||||
method: 'GET',
|
method: 'GET',
|
||||||
headers: { 'X-Extension-Key': apiKey },
|
headers: { 'X-Extension-Key': apiKey },
|
||||||
});
|
});
|
||||||
if (!r.ok) {
|
if (r.ok) showStatus(`Connected — HTTP ${r.status}.`, 'ok');
|
||||||
showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
|
else showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
|
||||||
return;
|
|
||||||
}
|
|
||||||
// A 200 is NOT sufficient. If the URL resolves to the Vue SPA instead of
|
|
||||||
// the JSON API, the catch-all route returns 200 with an HTML document —
|
|
||||||
// which used to report "Connected" on a config that could not POST at all.
|
|
||||||
const contentType = r.headers.get('content-type') || '';
|
|
||||||
if (!contentType.includes('json')) {
|
|
||||||
showStatus(
|
|
||||||
`${apiUrl} answered with ${contentType || 'no content-type'}, not JSON `
|
|
||||||
+ '— that looks like the FC web UI rather than its API.',
|
|
||||||
'err',
|
|
||||||
);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
showStatus(`Connected to ${apiUrl} — HTTP ${r.status}.`, 'ok');
|
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
|
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,18 +1,15 @@
|
|||||||
{
|
{
|
||||||
"name": "fabledcurator-extension",
|
"name": "fabledcurator-extension",
|
||||||
"version": "1.0.11",
|
"version": "1.0.7",
|
||||||
"private": true,
|
"private": true,
|
||||||
"description": "Firefox extension for FabledCurator",
|
"description": "Firefox extension for FabledCurator",
|
||||||
"comment_ignore_files": "The --ignore-files list comes from scripts/packaging.sh, the single source of truth shared with ci.yml's guard and the derived-version patch count. `set -f` is REQUIRED before the substitution: without it the shell globs `test/**` against the working tree and silently narrows the pattern to whatever files happen to exist.",
|
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"lint": "set -f; web-ext lint --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore)",
|
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore",
|
||||||
"start": "set -f; web-ext run --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --firefox=firefox",
|
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --firefox=firefox",
|
||||||
"build": "set -f; web-ext build --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --overwrite-dest",
|
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --overwrite-dest",
|
||||||
"sign": "set -f; web-ext sign --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
|
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
|
||||||
"test:unit": "vitest run"
|
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"vitest": "^4.0.0",
|
"web-ext": "^8.0.0"
|
||||||
"web-ext": "^10.0.0"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -72,17 +72,6 @@ body {
|
|||||||
.btn.block { display: block; width: 100%; margin-top: 8px; }
|
.btn.block { display: block; width: 100%; margin-top: 8px; }
|
||||||
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
|
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
|
||||||
.btn.link:hover { color: var(--accent); }
|
.btn.link:hover { color: var(--accent); }
|
||||||
.btn.small { padding: 6px 12px; font-size: 13px; }
|
|
||||||
|
|
||||||
/* In-app update prompt (accent-tinted so it reads as an actionable notice). */
|
|
||||||
.update-banner {
|
|
||||||
display: flex; align-items: center; gap: 10px;
|
|
||||||
margin: 10px 10px 0; padding: 10px 12px;
|
|
||||||
background: rgba(244, 186, 122, 0.12);
|
|
||||||
border: 1px solid rgba(244, 186, 122, 0.4);
|
|
||||||
border-radius: 6px;
|
|
||||||
}
|
|
||||||
#update-text { flex: 1; font-size: 13px; }
|
|
||||||
|
|
||||||
.source-row .play {
|
.source-row .play {
|
||||||
background: none; border: none; color: var(--on-surface-variant);
|
background: none; border: none; color: var(--on-surface-variant);
|
||||||
|
|||||||
@@ -20,11 +20,6 @@
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="main-content" class="main hidden">
|
<section id="main-content" class="main hidden">
|
||||||
<div id="update-banner" class="update-banner hidden">
|
|
||||||
<span id="update-text"></span>
|
|
||||||
<button id="update-btn" class="btn primary small">Update</button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<nav class="tabs">
|
<nav class="tabs">
|
||||||
<button class="tab active" data-tab="platforms">Platforms</button>
|
<button class="tab active" data-tab="platforms">Platforms</button>
|
||||||
<button class="tab" data-tab="sources">Sources</button>
|
<button class="tab" data-tab="sources">Sources</button>
|
||||||
|
|||||||
+12
-33
@@ -2,15 +2,6 @@ document.addEventListener('DOMContentLoaded', init);
|
|||||||
|
|
||||||
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
|
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
|
||||||
|
|
||||||
// A centered muted note div — the loading / empty state shared by the platform
|
|
||||||
// and sources lists.
|
|
||||||
function mutedNote(text) {
|
|
||||||
const d = document.createElement('div');
|
|
||||||
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
|
||||||
d.textContent = text;
|
|
||||||
return d;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function init() {
|
async function init() {
|
||||||
try {
|
try {
|
||||||
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
|
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
|
||||||
@@ -23,7 +14,6 @@ async function init() {
|
|||||||
setupEventListeners();
|
setupEventListeners();
|
||||||
showPlatformsLoading();
|
showPlatformsLoading();
|
||||||
testConnectionIfNeeded();
|
testConnectionIfNeeded();
|
||||||
checkForUpdate();
|
|
||||||
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
|
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
showSetupRequired();
|
showSetupRequired();
|
||||||
@@ -47,7 +37,10 @@ function showSetupRequired() {
|
|||||||
function showPlatformsLoading() {
|
function showPlatformsLoading() {
|
||||||
const c = document.getElementById('platforms-list');
|
const c = document.getElementById('platforms-list');
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
c.appendChild(mutedNote('Loading platforms…'));
|
const d = document.createElement('div');
|
||||||
|
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
||||||
|
d.textContent = 'Loading platforms…';
|
||||||
|
c.appendChild(d);
|
||||||
}
|
}
|
||||||
|
|
||||||
async function testConnectionIfNeeded() {
|
async function testConnectionIfNeeded() {
|
||||||
@@ -70,26 +63,6 @@ function updateConnectionDot(connected) {
|
|||||||
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
|
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
|
||||||
}
|
}
|
||||||
|
|
||||||
// Nudge to reinstall when the configured instance publishes a newer signed XPI
|
|
||||||
// (the extension is self-hosted, so there's no Firefox auto-update). Never
|
|
||||||
// blocks the popup — a failed check just leaves the banner hidden.
|
|
||||||
async function checkForUpdate() {
|
|
||||||
try {
|
|
||||||
const r = await browser.runtime.sendMessage({ type: 'CHECK_UPDATE' });
|
|
||||||
if (r && r.updateAvailable && r.xpiUrl) showUpdateBanner(r);
|
|
||||||
} catch { /* non-fatal */ }
|
|
||||||
}
|
|
||||||
|
|
||||||
function showUpdateBanner(r) {
|
|
||||||
document.getElementById('update-text').textContent =
|
|
||||||
`Update available — v${r.latestVersion} (installed v${r.currentVersion})`;
|
|
||||||
// Opening the signed XPI triggers Firefox's native install prompt.
|
|
||||||
document.getElementById('update-btn').addEventListener('click', () => {
|
|
||||||
browser.tabs.create({ url: r.xpiUrl });
|
|
||||||
});
|
|
||||||
document.getElementById('update-banner').classList.remove('hidden');
|
|
||||||
}
|
|
||||||
|
|
||||||
async function loadPlatformStatus() {
|
async function loadPlatformStatus() {
|
||||||
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
|
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
|
||||||
const c = document.getElementById('platforms-list');
|
const c = document.getElementById('platforms-list');
|
||||||
@@ -189,7 +162,10 @@ async function exportAllCookies() {
|
|||||||
async function loadSources() {
|
async function loadSources() {
|
||||||
const c = document.getElementById('sources-list');
|
const c = document.getElementById('sources-list');
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
c.appendChild(mutedNote('Loading sources…'));
|
const d = document.createElement('div');
|
||||||
|
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
||||||
|
d.textContent = 'Loading sources…';
|
||||||
|
c.appendChild(d);
|
||||||
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
|
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
if (r.error) {
|
if (r.error) {
|
||||||
@@ -200,7 +176,10 @@ async function loadSources() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (!r.sources || r.sources.length === 0) {
|
if (!r.sources || r.sources.length === 0) {
|
||||||
c.appendChild(mutedNote('No sources yet.'));
|
const empty = document.createElement('div');
|
||||||
|
empty.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
||||||
|
empty.textContent = 'No sources yet.';
|
||||||
|
c.appendChild(empty);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (const src of r.sources) c.appendChild(createSourceRow(src));
|
for (const src of r.sources) c.appendChild(createSourceRow(src));
|
||||||
|
|||||||
@@ -1,130 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# Single source of truth for "what ships inside the XPI", plus the version
|
|
||||||
# derived from it.
|
|
||||||
#
|
|
||||||
# Three consumers used to hand-maintain their own copy of this list, and
|
|
||||||
# keeping three copies of one fact in sync by hand is how issue #2397 happened:
|
|
||||||
#
|
|
||||||
# 1. web-ext's --ignore-files (extension/package.json's four scripts)
|
|
||||||
# 2. the :(exclude) pathspec (ci.yml's extension-version guard)
|
|
||||||
# 3. the git-log pathspec (the derived version, below)
|
|
||||||
#
|
|
||||||
# They now all read from here. POSIX sh only — CI's run shell is busybox.
|
|
||||||
#
|
|
||||||
# -f (no pathname expansion) is set for the whole script and is load-bearing:
|
|
||||||
# the lists below are iterated with deliberate word-splitting, and without -f
|
|
||||||
# the shell would also GLOB them, expanding `test/**` into whatever files
|
|
||||||
# happen to exist and corrupting the output. A caller's own `set -f` does not
|
|
||||||
# help here — this runs as a separate sh process and does not inherit it.
|
|
||||||
# Callers still need their own `set -f` for the substituted result; the two
|
|
||||||
# guards protect different expansions.
|
|
||||||
set -euf
|
|
||||||
|
|
||||||
# Paths under extension/ that are NOT packaged into the XPI.
|
|
||||||
#
|
|
||||||
# Split by whether git tracks them: node_modules and web-ext-artifacts are
|
|
||||||
# build/dependency output that never appears in a commit, so they belong in
|
|
||||||
# web-ext's ignore list but would be meaningless in a git pathspec.
|
|
||||||
#
|
|
||||||
# Directories need BOTH forms. `test/**` matches the files inside, but not the
|
|
||||||
# directory entry itself — web-ext writes an entry for the directory too, so
|
|
||||||
# with only the glob the XPI ends up carrying empty `test/` and `scripts/`
|
|
||||||
# entries (caught by the XPI-content check on 2026-08-03). The bare name alone
|
|
||||||
# is not enough either: minimatch's `test` does not match `test/url.spec.js`,
|
|
||||||
# so dropping the glob would ship the contents. Keep both.
|
|
||||||
NOT_PACKAGED_TRACKED='package.json package-lock.json README.md .gitignore vitest.config.js scripts scripts/** test test/**'
|
|
||||||
NOT_PACKAGED_BUILD='web-ext-artifacts node_modules'
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
echo "usage: packaging.sh {ignore|pathspec|version|major-minor|patch}" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
|
|
||||||
# web-ext --ignore-files values, space-separated.
|
|
||||||
#
|
|
||||||
# Callers MUST disable pathname expansion first (`set -f`), or the shell will
|
|
||||||
# glob `test/**` against the working tree before web-ext ever sees the pattern
|
|
||||||
# and silently narrow it to whatever happens to exist right now.
|
|
||||||
cmd_ignore() {
|
|
||||||
echo "$NOT_PACKAGED_TRACKED $NOT_PACKAGED_BUILD"
|
|
||||||
}
|
|
||||||
|
|
||||||
# git pathspec excluding the non-packaged tracked files, e.g.
|
|
||||||
# :(exclude)extension/package.json :(exclude)extension/test/**
|
|
||||||
# Same `set -f` requirement as above.
|
|
||||||
cmd_pathspec() {
|
|
||||||
for entry in $NOT_PACKAGED_TRACKED; do
|
|
||||||
printf ':(exclude)extension/%s ' "$entry"
|
|
||||||
done
|
|
||||||
echo
|
|
||||||
}
|
|
||||||
|
|
||||||
# MAJOR.MINOR stays hand-set in manifest.json — it's the part that carries
|
|
||||||
# deliberate meaning. Only the patch component is derived.
|
|
||||||
cmd_major_minor() {
|
|
||||||
root=$(git rev-parse --show-toplevel)
|
|
||||||
grep -E '"version"' "$root/extension/manifest.json" \
|
|
||||||
| head -1 \
|
|
||||||
| sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([0-9]+)\.([0-9]+).*/\1.\2/'
|
|
||||||
}
|
|
||||||
|
|
||||||
# 2020-01-01T00:00:00Z — the anchor for the derived patch component. Fixed
|
|
||||||
# forever; moving it would renumber every version downwards.
|
|
||||||
VERSION_EPOCH=1577836800
|
|
||||||
|
|
||||||
# Minutes since VERSION_EPOCH of the LATEST commit that touched a PACKAGED
|
|
||||||
# extension file.
|
|
||||||
#
|
|
||||||
# Time-derived, per family rule 149: an artifact's ordering key must never be a
|
|
||||||
# commit count. A count is per-branch — `dev` and `main` count different
|
|
||||||
# histories of the same code — so the moment BOTH channels publish, their
|
|
||||||
# versions order by which branch accumulated more commits rather than by which
|
|
||||||
# is newer. A squash-merge makes that permanent: main gains one commit where dev
|
|
||||||
# gained five, so dev climbs away from main and a dev install can never cross
|
|
||||||
# back. That is Roundtable's 2026-08-24 incident (`versionCode` was the branch's
|
|
||||||
# commit count) in a different repo. Measured here on 2026-08-27: main=23,
|
|
||||||
# dev=24 under the old formula — one apart, which is exactly how the inversion
|
|
||||||
# stays invisible until it strands somebody.
|
|
||||||
#
|
|
||||||
# Why the commit's time and not the build's:
|
|
||||||
# * MONOTONIC — max() over a set that only ever gains members. Verified
|
|
||||||
# across all 24 extension-touching commits: zero non-monotonic steps.
|
|
||||||
# * STABLE while the extension is unchanged, so an unchanged extension keeps
|
|
||||||
# its version, the ext-<version> signature cache still hits, and AMO is
|
|
||||||
# called once per extension CHANGE rather than once per push. Build-time
|
|
||||||
# minutes would re-sign on every push and never let two channels share a
|
|
||||||
# signature.
|
|
||||||
# * SHARED ACROSS CHANNELS — after a merge, `main` sees the same commit and
|
|
||||||
# derives the same number, so `:latest` reuses the signature `:dev` already
|
|
||||||
# produced for byte-identical code. Same code, same version, one signing.
|
|
||||||
# * REPRODUCIBLE — any checkout of a commit yields that commit's version.
|
|
||||||
#
|
|
||||||
# Requires real history: a depth-1 clone sees one commit and will derive a wrong
|
|
||||||
# (too low) value. Every consumer must check out with fetch-depth: 0.
|
|
||||||
cmd_patch() {
|
|
||||||
root=$(git rev-parse --show-toplevel)
|
|
||||||
# Unquoted on purpose: the pathspec must word-split into separate args.
|
|
||||||
# Globbing is already off script-wide (set -euf above).
|
|
||||||
# shellcheck disable=SC2046
|
|
||||||
ts=$(cd "$root" && git log --format=%ct HEAD -- extension/ $(cmd_pathspec) \
|
|
||||||
| sort -n | tail -1)
|
|
||||||
if [ -z "$ts" ]; then
|
|
||||||
echo "packaging.sh: no commit touches a packaged extension file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo $(( (ts - VERSION_EPOCH) / 60 ))
|
|
||||||
}
|
|
||||||
|
|
||||||
cmd_version() {
|
|
||||||
echo "$(cmd_major_minor).$(cmd_patch)"
|
|
||||||
}
|
|
||||||
|
|
||||||
[ $# -ge 1 ] || usage
|
|
||||||
case "$1" in
|
|
||||||
ignore) cmd_ignore ;;
|
|
||||||
pathspec) cmd_pathspec ;;
|
|
||||||
version) cmd_version ;;
|
|
||||||
major-minor) cmd_major_minor ;;
|
|
||||||
patch) cmd_patch ;;
|
|
||||||
*) usage ;;
|
|
||||||
esac
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
import { readFileSync } from 'node:fs'
|
|
||||||
import { fileURLToPath } from 'node:url'
|
|
||||||
import path from 'node:path'
|
|
||||||
|
|
||||||
const LIB_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', '..', 'lib')
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Load an extension lib and hand back the globals it declares.
|
|
||||||
*
|
|
||||||
* The files under lib/ are CLASSIC scripts, not ES modules: manifest.json
|
|
||||||
* lists them in `background.scripts` and options.html pulls them in with a
|
|
||||||
* plain <script> tag, so they declare bare functions into a shared scope and
|
|
||||||
* export nothing. Rather than bolt a `module.exports` shim onto production
|
|
||||||
* code that would never run in the browser, evaluate the real file the same
|
|
||||||
* way the browser does — as a script body — and pick the declarations back out.
|
|
||||||
*
|
|
||||||
* This means the specs exercise the exact bytes that get packaged into the
|
|
||||||
* XPI. Only usable for libs that touch no browser APIs at load time
|
|
||||||
* (url.js, platforms.js); cookies.js and api.js reference `browser.*` and
|
|
||||||
* would need stubbing, which is why they aren't loaded this way.
|
|
||||||
*
|
|
||||||
* @param {string} filename e.g. 'url.js'
|
|
||||||
* @param {string[]} names declarations to return, e.g. ['normalizeApiUrl']
|
|
||||||
*/
|
|
||||||
export function loadLib(filename, names) {
|
|
||||||
const source = readFileSync(path.join(LIB_DIR, filename), 'utf8')
|
|
||||||
const factory = new Function(`${source}\nreturn { ${names.join(', ')} }`)
|
|
||||||
return factory()
|
|
||||||
}
|
|
||||||
@@ -1,190 +0,0 @@
|
|||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { readFileSync } from 'node:fs'
|
|
||||||
import { fileURLToPath } from 'node:url'
|
|
||||||
import path from 'node:path'
|
|
||||||
import { loadLib } from './helpers/loadLib.js'
|
|
||||||
|
|
||||||
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
|
||||||
const manifest = JSON.parse(readFileSync(path.join(EXT_DIR, 'manifest.json'), 'utf8'))
|
|
||||||
|
|
||||||
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
|
|
||||||
'platforms.js',
|
|
||||||
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
|
|
||||||
)
|
|
||||||
|
|
||||||
describe('getPlatformFromUrl', () => {
|
|
||||||
it('identifies each platform from a domain URL', () => {
|
|
||||||
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
|
|
||||||
expect(getPlatformFromUrl('https://subscribestar.adult/someone')).toBe('subscribestar')
|
|
||||||
expect(getPlatformFromUrl('https://www.hentai-foundry.com/user/someone')).toBe('hentaifoundry')
|
|
||||||
expect(getPlatformFromUrl('https://discord.com/channels/@me')).toBe('discord')
|
|
||||||
expect(getPlatformFromUrl('https://www.pixiv.net/en/users/123')).toBe('pixiv')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('accepts http as well as https, with or without www', () => {
|
|
||||||
expect(getPlatformFromUrl('http://patreon.com/Atole')).toBe('patreon')
|
|
||||||
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns null for unrelated hosts', () => {
|
|
||||||
expect(getPlatformFromUrl('https://example.com/patreon.com')).toBe(null)
|
|
||||||
expect(getPlatformFromUrl('https://not-patreon.com/Atole')).toBe(null)
|
|
||||||
expect(getPlatformFromUrl('')).toBe(null)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns null for deviantart, retired at #3069', () => {
|
|
||||||
// The 2026-07-05 product decision (FC downloaders = art-dedicated services
|
|
||||||
// only) left deviantart wired for seven weeks. Asserting the negative is
|
|
||||||
// what keeps a partial retirement from being re-completed by accident.
|
|
||||||
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe(null)
|
|
||||||
expect(PLATFORMS.deviantart).toBeUndefined()
|
|
||||||
expect(PLATFORM_ARTIST_PATTERNS.deviantart).toBeUndefined()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('isArtistPage', () => {
|
|
||||||
// Regression cases from issue #1485: the Add-to-FC button vanished once the
|
|
||||||
// operator SUBSCRIBED to a creator, because Patreon serves subscribed users
|
|
||||||
// the /cw/ ("creator workspace") URL and the pattern only matched the bare
|
|
||||||
// root. All three creator URL shapes must match, plus inner pages — the
|
|
||||||
// button matters most exactly when you're subscribed.
|
|
||||||
it('matches all three Patreon creator URL shapes', () => {
|
|
||||||
expect(isArtistPage('https://www.patreon.com/Atole', 'patreon')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.patreon.com/c/Atole', 'patreon')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.patreon.com/cw/Atole', 'patreon')).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('matches Patreon creator inner pages', () => {
|
|
||||||
expect(isArtistPage('https://www.patreon.com/cw/Atole/posts', 'patreon')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.patreon.com/Atole/membership', 'patreon')).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('excludes Patreon navigation pages that are not creators', () => {
|
|
||||||
for (const nav of ['home', 'search', 'messages', 'notifications', 'library', 'settings']) {
|
|
||||||
expect(isArtistPage(`https://www.patreon.com/${nav}`, 'patreon')).toBe(false)
|
|
||||||
expect(isArtistPage(`https://www.patreon.com/${nav}/anything`, 'patreon')).toBe(false)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('matches SubscribeStar creator roots on both TLDs but not feed pages', () => {
|
|
||||||
expect(isArtistPage('https://subscribestar.adult/someone', 'subscribestar')).toBe(true)
|
|
||||||
expect(isArtistPage('https://subscribestar.com/someone', 'subscribestar')).toBe(true)
|
|
||||||
expect(isArtistPage('https://subscribestar.adult/feed', 'subscribestar')).toBe(false)
|
|
||||||
expect(isArtistPage('https://subscribestar.adult/messages', 'subscribestar')).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('matches Hentai Foundry user pages only', () => {
|
|
||||||
expect(isArtistPage('https://www.hentai-foundry.com/user/someone', 'hentaifoundry')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.hentai-foundry.com/pictures/popular', 'hentaifoundry')).toBe(
|
|
||||||
false
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('matches Pixiv numeric user pages, with or without the /en/ prefix', () => {
|
|
||||||
expect(isArtistPage('https://www.pixiv.net/users/12345', 'pixiv')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.pixiv.net/en/users/12345', 'pixiv')).toBe(true)
|
|
||||||
expect(isArtistPage('https://www.pixiv.net/en/artworks/999', 'pixiv')).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns false for a platform with no artist pattern (discord)', () => {
|
|
||||||
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns false for an unknown platform key', () => {
|
|
||||||
expect(isArtistPage('https://www.patreon.com/Atole', 'nope')).toBe(false)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('platform table integrity', () => {
|
|
||||||
it('gives every artist pattern a corresponding platform entry', () => {
|
|
||||||
// A pattern keyed to a platform that no longer exists is dead code that
|
|
||||||
// silently never fires; the reverse (a platform with no pattern) is the
|
|
||||||
// legitimate discord case, so only this direction is an error.
|
|
||||||
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
|
|
||||||
expect(Object.keys(PLATFORMS)).toContain(key)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('gives every platform the fields the popup renders', () => {
|
|
||||||
for (const [key, platform] of Object.entries(PLATFORMS)) {
|
|
||||||
expect(platform.name, `${key}.name`).toBeTruthy()
|
|
||||||
expect(platform.color, `${key}.color`).toMatch(/^#[0-9A-Fa-f]{6}$/)
|
|
||||||
expect(['cookies', 'token'], `${key}.authType`).toContain(platform.authType)
|
|
||||||
expect(platform.urlPattern, `${key}.urlPattern`).toBeInstanceOf(RegExp)
|
|
||||||
expect(Array.isArray(platform.domains), `${key}.domains`).toBe(true)
|
|
||||||
expect(platform.domains.length, `${key}.domains`).toBeGreaterThan(0)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('keeps every artist URL matched by its own platform pattern too', () => {
|
|
||||||
// isArtistPage is only ever consulted after getPlatformFromUrl resolves a
|
|
||||||
// key, so an artist pattern matching a URL its platform's urlPattern
|
|
||||||
// rejects would be unreachable.
|
|
||||||
const samples = {
|
|
||||||
patreon: 'https://www.patreon.com/cw/Atole',
|
|
||||||
subscribestar: 'https://subscribestar.adult/someone',
|
|
||||||
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
|
|
||||||
pixiv: 'https://www.pixiv.net/en/users/12345'
|
|
||||||
}
|
|
||||||
for (const [key, url] of Object.entries(samples)) {
|
|
||||||
expect(isArtistPage(url, key), `${key} artist pattern`).toBe(true)
|
|
||||||
expect(getPlatformFromUrl(url), `${key} urlPattern`).toBe(key)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('manifest.json agrees with the platform table', () => {
|
|
||||||
// #3069: deviantart was dropped from the product in July but survived in
|
|
||||||
// manifest.json until late August, because NOTHING tied the manifest's
|
|
||||||
// domain lists back to PLATFORMS. These two specs are that tie. Both
|
|
||||||
// directions matter: a stale match ships host access the product decided
|
|
||||||
// not to use, and a missing one silently kills the Add-to-FC button.
|
|
||||||
const matches = manifest.content_scripts[0].matches
|
|
||||||
// '*://*.patreon.com/*' -> '.patreon.com', the form PLATFORMS.domains uses.
|
|
||||||
const hostOf = (m) => m.replace(/^\*:\/\/\*/, '').replace(/\/\*$/, '')
|
|
||||||
|
|
||||||
it('injects the content script only on domains a platform claims', () => {
|
|
||||||
for (const m of matches) {
|
|
||||||
const host = hostOf(m)
|
|
||||||
const owner = Object.entries(PLATFORMS).find(
|
|
||||||
([, p]) => p.domains.includes(host)
|
|
||||||
)
|
|
||||||
expect(owner, `no platform claims content-script match "${m}"`).toBeTruthy()
|
|
||||||
// The content script exists to draw the Add-as-source button, so a
|
|
||||||
// platform with no artist pattern (discord) has no business here.
|
|
||||||
expect(
|
|
||||||
PLATFORM_ARTIST_PATTERNS[owner[0]],
|
|
||||||
`"${m}" injects for ${owner[0]}, which has no artist pattern`
|
|
||||||
).toBeTruthy()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('injects on every platform that has an artist pattern', () => {
|
|
||||||
const covered = new Set(
|
|
||||||
matches
|
|
||||||
.map(hostOf)
|
|
||||||
.map((h) => Object.entries(PLATFORMS).find(([, p]) => p.domains.includes(h)))
|
|
||||||
.filter(Boolean)
|
|
||||||
.map(([key]) => key)
|
|
||||||
)
|
|
||||||
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
|
|
||||||
expect(covered, `${key} has an artist pattern but no content-script match`).toContain(key)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('requests no host permission for a domain no platform claims', () => {
|
|
||||||
// '*://*/*' is the deliberate exception: FC is self-hosted at an arbitrary
|
|
||||||
// operator-chosen URL, so the extension cannot enumerate its own backend.
|
|
||||||
// Every OTHER entry is a platform domain and must still have an owner.
|
|
||||||
for (const h of manifest.host_permissions) {
|
|
||||||
if (h === '*://*/*') continue
|
|
||||||
const host = hostOf(h)
|
|
||||||
// pixiv's OAuth/API hosts are pixiv infrastructure, not creator pages,
|
|
||||||
// so they are matched by suffix rather than by the domains list.
|
|
||||||
const claimed = Object.values(PLATFORMS).some(
|
|
||||||
(p) => p.domains.includes(host) || p.domains.some((d) => host.endsWith(d))
|
|
||||||
)
|
|
||||||
expect(claimed, `host permission "${h}" belongs to no platform`).toBe(true)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { loadLib } from './helpers/loadLib.js'
|
|
||||||
|
|
||||||
const { normalizeApiUrl, webRootFromApiUrl } = loadLib('url.js', [
|
|
||||||
'normalizeApiUrl',
|
|
||||||
'webRootFromApiUrl'
|
|
||||||
])
|
|
||||||
|
|
||||||
describe('normalizeApiUrl', () => {
|
|
||||||
// The bug this exists for (issue #2393): the instance root was accepted and
|
|
||||||
// stored verbatim, so every request went to /credentials instead of
|
|
||||||
// /api/credentials. That path is a Vue router route, so the SPA catch-all
|
|
||||||
// answered GET with 200 HTML and rejected POST with 405 — which read as a
|
|
||||||
// backend bug rather than a URL one.
|
|
||||||
it('appends /api to an instance root', () => {
|
|
||||||
expect(normalizeApiUrl('http://curator.traefik.internal')).toBe(
|
|
||||||
'http://curator.traefik.internal/api'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('leaves an API root alone rather than doubling the suffix', () => {
|
|
||||||
expect(normalizeApiUrl('http://curator.traefik.internal/api')).toBe(
|
|
||||||
'http://curator.traefik.internal/api'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('is idempotent', () => {
|
|
||||||
const once = normalizeApiUrl('http://curator.example.com')
|
|
||||||
expect(normalizeApiUrl(once)).toBe(once)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('strips trailing slashes before deciding', () => {
|
|
||||||
expect(normalizeApiUrl('http://curator.example.com/')).toBe('http://curator.example.com/api')
|
|
||||||
expect(normalizeApiUrl('http://curator.example.com///')).toBe('http://curator.example.com/api')
|
|
||||||
expect(normalizeApiUrl('http://curator.example.com/api/')).toBe('http://curator.example.com/api')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('trims surrounding whitespace (paste artifacts)', () => {
|
|
||||||
expect(normalizeApiUrl(' http://curator.example.com ')).toBe(
|
|
||||||
'http://curator.example.com/api'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('matches the /api suffix case-insensitively', () => {
|
|
||||||
expect(normalizeApiUrl('http://curator.example.com/API')).toBe('http://curator.example.com/API')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns empty string for empty/nullish input, never a bare "/api"', () => {
|
|
||||||
// isConfigured() gates on truthiness, so a bogus '/api' here would read as
|
|
||||||
// "configured" and produce a request against the options page's own origin.
|
|
||||||
expect(normalizeApiUrl('')).toBe('')
|
|
||||||
expect(normalizeApiUrl(' ')).toBe('')
|
|
||||||
expect(normalizeApiUrl(null)).toBe('')
|
|
||||||
expect(normalizeApiUrl(undefined)).toBe('')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('does not treat a path merely containing "api" as the suffix', () => {
|
|
||||||
expect(normalizeApiUrl('http://curator.example.com/apiary')).toBe(
|
|
||||||
'http://curator.example.com/apiary/api'
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('preserves a subpath deployment', () => {
|
|
||||||
expect(normalizeApiUrl('http://host.internal/curator')).toBe('http://host.internal/curator/api')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('webRootFromApiUrl', () => {
|
|
||||||
// The SPA root, where the Vue router and the served XPI live. Used by
|
|
||||||
// OPEN_ARTIST_PAGE and the self-update check — NOT the JSON API.
|
|
||||||
it('strips the /api suffix', () => {
|
|
||||||
expect(webRootFromApiUrl('http://curator.example.com/api')).toBe('http://curator.example.com')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('accepts an instance root unchanged', () => {
|
|
||||||
expect(webRootFromApiUrl('http://curator.example.com')).toBe('http://curator.example.com')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('agrees with normalizeApiUrl in both directions', () => {
|
|
||||||
for (const input of ['http://curator.example.com', 'http://curator.example.com/api']) {
|
|
||||||
expect(normalizeApiUrl(webRootFromApiUrl(input))).toBe(normalizeApiUrl(input))
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('preserves a subpath deployment', () => {
|
|
||||||
expect(webRootFromApiUrl('http://host.internal/curator/api')).toBe('http://host.internal/curator')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('returns empty string for empty/nullish input', () => {
|
|
||||||
expect(webRootFromApiUrl('')).toBe('')
|
|
||||||
expect(webRootFromApiUrl(null)).toBe('')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
@@ -1,112 +0,0 @@
|
|||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { readFileSync } from 'node:fs'
|
|
||||||
import { execFileSync } from 'node:child_process'
|
|
||||||
import { fileURLToPath } from 'node:url'
|
|
||||||
import path from 'node:path'
|
|
||||||
|
|
||||||
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
|
||||||
const read = (name) => JSON.parse(readFileSync(path.join(EXT_DIR, name), 'utf8'))
|
|
||||||
const readText = (...seg) => readFileSync(path.join(EXT_DIR, ...seg), 'utf8')
|
|
||||||
|
|
||||||
// Only the git-free subcommands are exercised here: `version`/`patch` shell out
|
|
||||||
// to git, and the extension lane runs on node:24-bookworm-slim which may not
|
|
||||||
// ship it. Those two are covered where git is guaranteed — ci.yml and build.yml
|
|
||||||
// run on ci-python.
|
|
||||||
const packaging = (cmd) =>
|
|
||||||
execFileSync('sh', [path.join(EXT_DIR, 'scripts', 'packaging.sh'), cmd], {
|
|
||||||
cwd: EXT_DIR,
|
|
||||||
encoding: 'utf8'
|
|
||||||
})
|
|
||||||
.trim()
|
|
||||||
.split(/\s+/)
|
|
||||||
.filter(Boolean)
|
|
||||||
|
|
||||||
describe('packaging.sh — the single definition of what ships', () => {
|
|
||||||
it('emits an ignore list and a pathspec that agree on the tracked files', () => {
|
|
||||||
const ignore = packaging('ignore')
|
|
||||||
const pathspec = packaging('pathspec').map((e) => e.replace(':(exclude)extension/', ''))
|
|
||||||
|
|
||||||
// Every git-excluded path must also be hidden from web-ext. The reverse is
|
|
||||||
// not required: node_modules and web-ext-artifacts are build output git
|
|
||||||
// never tracks, so they appear only in the ignore list.
|
|
||||||
for (const entry of pathspec) {
|
|
||||||
expect(ignore, `pathspec has "${entry}" but --ignore-files does not`).toContain(entry)
|
|
||||||
}
|
|
||||||
expect(pathspec.length).toBeGreaterThan(0)
|
|
||||||
expect(ignore).toContain('node_modules')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('emits glob patterns literally, never expanded against the working tree', () => {
|
|
||||||
// The script iterates its lists with deliberate word-splitting, so it must
|
|
||||||
// run with pathname expansion off. Without that, invoking it from a cwd
|
|
||||||
// where test/ exists (exactly how ci.yml and vitest call it) expands
|
|
||||||
// `test/**` into the individual spec files, and the pathspec silently stops
|
|
||||||
// covering anything added later.
|
|
||||||
const pathspec = packaging('pathspec')
|
|
||||||
expect(pathspec).toContain(':(exclude)extension/test/**')
|
|
||||||
expect(pathspec).toContain(':(exclude)extension/scripts/**')
|
|
||||||
expect(pathspec.some((e) => e.includes('.spec.js'))).toBe(false)
|
|
||||||
expect(pathspec.some((e) => e.includes('helpers'))).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('keeps its own scripts and specs out of the XPI', () => {
|
|
||||||
// Both are repo infrastructure. web-ext packages everything not ignored, so
|
|
||||||
// omitting either would ship dev tooling to users -- and `test/**` in
|
|
||||||
// particular only survives because callers `set -f` before substituting it.
|
|
||||||
const ignore = packaging('ignore')
|
|
||||||
expect(ignore).toContain('vitest.config.js')
|
|
||||||
// Both forms per directory. The glob covers the contents; the bare name
|
|
||||||
// covers the directory ENTRY, which web-ext writes separately — with only
|
|
||||||
// the glob, the XPI carries an empty `test/` and `scripts/`.
|
|
||||||
for (const dir of ['test', 'scripts']) {
|
|
||||||
expect(ignore, `${dir} contents`).toContain(`${dir}/**`)
|
|
||||||
expect(ignore, `${dir} directory entry`).toContain(dir)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('consumers delegate rather than keeping their own copy', () => {
|
|
||||||
// These assertions are the actual anti-regression value: it is easy for a
|
|
||||||
// future edit to "simplify" by inlining a literal list again, which silently
|
|
||||||
// reintroduces the drift that issue #2397 was about.
|
|
||||||
it('package.json derives --ignore-files from the script', () => {
|
|
||||||
for (const [name, script] of Object.entries(read('package.json').scripts)) {
|
|
||||||
if (!script.includes('--ignore-files')) continue
|
|
||||||
expect(script, `${name} should call packaging.sh`).toContain('scripts/packaging.sh ignore')
|
|
||||||
expect(script, `${name} must set -f before the substitution`).toMatch(/set -f;/)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('ci.yml derives its pathspec from the script and hardcodes none', () => {
|
|
||||||
const ci = readText('..', '.forgejo', 'workflows', 'ci.yml')
|
|
||||||
expect(ci).toContain('extension/scripts/packaging.sh pathspec')
|
|
||||||
// A literal :(exclude)extension/... in the workflow means someone bypassed
|
|
||||||
// the shared definition.
|
|
||||||
expect(ci).not.toMatch(/:\(exclude\)extension\//)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('extension version', () => {
|
|
||||||
it('keeps manifest.json and package.json in lockstep', () => {
|
|
||||||
expect(read('manifest.json').version).toBe(read('package.json').version)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('uses a plain dotted numeric version AMO will accept', () => {
|
|
||||||
expect(read('package.json').version).toMatch(/^\d+(\.\d+)*$/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('declares manifest v3', () => {
|
|
||||||
expect(read('manifest.json').manifest_version).toBe(3)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('lists every background script that exists, in dependency order', () => {
|
|
||||||
// url.js must load BEFORE api.js: api.js calls normalizeApiUrl at
|
|
||||||
// init()-time, and these are classic scripts sharing one scope, so a
|
|
||||||
// reordering here is a runtime ReferenceError with no build-time signal.
|
|
||||||
const scripts = read('manifest.json').background.scripts
|
|
||||||
for (const rel of scripts) {
|
|
||||||
expect(() => readFileSync(path.join(EXT_DIR, rel)), `missing ${rel}`).not.toThrow()
|
|
||||||
}
|
|
||||||
expect(scripts.indexOf('lib/url.js')).toBeLessThan(scripts.indexOf('lib/api.js'))
|
|
||||||
})
|
|
||||||
})
|
|
||||||
@@ -1,13 +0,0 @@
|
|||||||
import { defineConfig } from 'vitest/config'
|
|
||||||
|
|
||||||
// Mirrors frontend/vitest.config.js, minus the Vue plugin — the extension has
|
|
||||||
// no SFCs and mounts nothing. Pure-logic specs only, so `node` is enough; the
|
|
||||||
// libs under test are deliberately the ones with no browser-API surface (see
|
|
||||||
// test/helpers/loadLib.js).
|
|
||||||
export default defineConfig({
|
|
||||||
test: {
|
|
||||||
environment: 'node',
|
|
||||||
include: ['test/**/*.spec.js'],
|
|
||||||
passWithNoTests: true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
+13
-11
@@ -4,28 +4,30 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=24"
|
"node": ">=22"
|
||||||
},
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
"build": "vite build",
|
"build": "vite build",
|
||||||
"preview": "vite preview",
|
"preview": "vite preview",
|
||||||
"test:unit": "vitest run"
|
"test:unit": "vitest run",
|
||||||
|
"check": "vue-tsc --noEmit"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"vue": "^3.5.0",
|
"vue": "^3.4.0",
|
||||||
"vue-router": "^5.0.0",
|
"vue-router": "^4.3.0",
|
||||||
"pinia": "^3.0.0",
|
"pinia": "^2.1.0",
|
||||||
"vuetify": "^4.0.0",
|
"vuetify": "^3.5.0",
|
||||||
"@mdi/font": "^7.4.0"
|
"@mdi/font": "^7.4.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@vitejs/plugin-vue": "^6.0.0",
|
"@vitejs/plugin-vue": "^5.0.0",
|
||||||
"vite": "^8.0.0",
|
"vite": "^5.2.0",
|
||||||
"vite-plugin-vuetify": "^2.1.0",
|
"vue-tsc": "^2.0.0",
|
||||||
|
"vite-plugin-vuetify": "^2.0.0",
|
||||||
"sass": "^1.71.0",
|
"sass": "^1.71.0",
|
||||||
"vitest": "^4.0.0",
|
"vitest": "^2.1.0",
|
||||||
"@vue/test-utils": "^2.4.0",
|
"@vue/test-utils": "^2.4.0",
|
||||||
"happy-dom": "^20.0.0"
|
"happy-dom": "^15.0.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,11 +18,11 @@ const route = useRoute()
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-content {
|
.fc-content {
|
||||||
min-height: 100vh;
|
min-height: 100vh;
|
||||||
/* NO padding-top: the TopNav is position:sticky, so it already reserves its
|
/* Push initial viewport content below the sticky TopNav. Without
|
||||||
own space in the v-app flex column — content flows directly below it. The
|
this, some views' first rows / form fields / table headers can
|
||||||
old 64px padding-top was a leftover from a FIXED navbar and double-counted
|
end up obscured by the navbar (depending on parent overflow
|
||||||
that space, leaving a large empty band at the top of EVERY view (and pushing
|
context interacting with position: sticky). Scrolled-down content
|
||||||
the full-height calc(100vh - 64px) views down so they overflowed). Removed
|
still slides under the nav — the gradient-fade design is intact. */
|
||||||
2026-07-13. Scrolled content still slides under the sticky nav as before. */
|
padding-top: 64px;
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<template>
|
<template>
|
||||||
<v-snackbar
|
<v-snackbar
|
||||||
v-model="show" :color="color" location="bottom right" timeout="4000"
|
v-model="show" :color="color" location="bottom right" timeout="4000"
|
||||||
min-height="68" elevation="4"
|
multi-line elevation="4"
|
||||||
>
|
>
|
||||||
{{ message }}
|
{{ message }}
|
||||||
<template #actions>
|
<template #actions>
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<template>
|
<template>
|
||||||
<header ref="navEl" class="fc-topnav" :class="{ 'fc-topnav--chrome': hasStickyChrome }">
|
<header class="fc-topnav">
|
||||||
<div class="fc-nav-left">
|
<div class="fc-nav-left">
|
||||||
<RouterLink :to="FRONT_DOOR" class="fc-brand" aria-label="FabledCurator home">
|
<RouterLink :to="FRONT_DOOR" class="fc-brand" aria-label="FabledCurator home">
|
||||||
<img src="/favicon.svg" alt="" class="fc-brand__glyph" width="22" height="22" />
|
<img src="/favicon.svg" alt="" class="fc-brand__glyph" width="22" height="22" />
|
||||||
@@ -64,39 +64,13 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, onBeforeUnmount, onMounted, ref } from 'vue'
|
import { computed, onMounted } from 'vue'
|
||||||
import { useRoute } from 'vue-router'
|
|
||||||
import router, { FRONT_DOOR } from '../router.js'
|
import router, { FRONT_DOOR } from '../router.js'
|
||||||
import { useSystemStore } from '../stores/system.js'
|
import { useSystemStore } from '../stores/system.js'
|
||||||
import PipelineStatusChip from './PipelineStatusChip.vue'
|
import PipelineStatusChip from './PipelineStatusChip.vue'
|
||||||
|
|
||||||
const system = useSystemStore()
|
const system = useSystemStore()
|
||||||
|
onMounted(() => system.refreshHealth())
|
||||||
// Publish the nav's REAL height as --fc-nav-h so full-height workspaces
|
|
||||||
// (Explore/Subscriptions) and sticky sub-headers pin to it exactly instead of a
|
|
||||||
// hardcoded 64px that Vuetify 4's MD3 sizing broke — the Explore breadcrumb was
|
|
||||||
// tucking under a taller nav (#1481). ResizeObserver keeps it live as the nav
|
|
||||||
// reflows (per-view teleported actions, mobile breakpoint, chip state changes).
|
|
||||||
const navEl = ref(null)
|
|
||||||
let navRO = null
|
|
||||||
onMounted(() => {
|
|
||||||
system.refreshHealth()
|
|
||||||
if (navEl.value && 'ResizeObserver' in window) {
|
|
||||||
navRO = new ResizeObserver(() => {
|
|
||||||
const h = navEl.value?.offsetHeight
|
|
||||||
if (h) document.documentElement.style.setProperty('--fc-nav-h', `${h}px`)
|
|
||||||
})
|
|
||||||
navRO.observe(navEl.value)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
onBeforeUnmount(() => { navRO?.disconnect() })
|
|
||||||
|
|
||||||
// Views that pin a sticky sub-header (filter bar / tabs) directly under the nav
|
|
||||||
// declare `meta.stickyChrome`. On those, the nav doesn't fade to transparent at
|
|
||||||
// its bottom — it hands off at the shared seam alpha so the sub-header can
|
|
||||||
// continue the SAME fade (see .fc-chrome-continues in app.css). One gradient.
|
|
||||||
const route = useRoute()
|
|
||||||
const hasStickyChrome = computed(() => !!route.meta?.stickyChrome)
|
|
||||||
|
|
||||||
// Every route with a meta.title is a nav entry. Order by meta.navOrder —
|
// Every route with a meta.title is a nav entry. Order by meta.navOrder —
|
||||||
// router.getRoutes() does NOT guarantee declaration order, so explicit numbers
|
// router.getRoutes() does NOT guarantee declaration order, so explicit numbers
|
||||||
@@ -145,35 +119,16 @@ const health = computed(() => {
|
|||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 1rem;
|
gap: 1rem;
|
||||||
padding: 0.75rem 1rem;
|
padding: 0.75rem 1rem;
|
||||||
/* Obsidian (#14171A) fade — content scrolls under it. Holds high (0.92 →
|
/* Obsidian (#14171A = 20,23,26) gradient fade — content scrolls under it. */
|
||||||
0.84) through the top half, then eases to transparent over the bottom
|
|
||||||
quarter so it tails off softly instead of a straight line to a hard edge
|
|
||||||
(operator 2026-07-13). Shared --fc-chrome-rgb keeps it in sync with the
|
|
||||||
sub-header continuation. */
|
|
||||||
background: linear-gradient(
|
background: linear-gradient(
|
||||||
to bottom,
|
to bottom,
|
||||||
rgba(var(--fc-chrome-rgb), 0.92) 0%,
|
rgba(20, 23, 26, 0.92) 0%,
|
||||||
rgba(var(--fc-chrome-rgb), 0.84) 50%,
|
rgba(20, 23, 26, 0.65) 60%,
|
||||||
rgba(var(--fc-chrome-rgb), 0.55) 75%,
|
rgba(20, 23, 26, 0) 100%
|
||||||
rgba(var(--fc-chrome-rgb), 0) 100%
|
|
||||||
);
|
);
|
||||||
backdrop-filter: blur(2px);
|
backdrop-filter: blur(2px);
|
||||||
-webkit-backdrop-filter: blur(2px);
|
-webkit-backdrop-filter: blur(2px);
|
||||||
}
|
}
|
||||||
/* On a view with a sticky sub-header pinned beneath (meta.stickyChrome), the nav
|
|
||||||
stops fading at the shared seam alpha instead of going fully transparent — the
|
|
||||||
sub-header (.fc-chrome-continues) picks the fade up from there, so the two read
|
|
||||||
as one continuous gradient. Compound selector out-specifies .fc-topnav so it
|
|
||||||
wins regardless of Vite's production CSS ordering. --fc-chrome-* come from the
|
|
||||||
global :root in app.css (custom props inherit into scoped styles). */
|
|
||||||
.fc-topnav.fc-topnav--chrome {
|
|
||||||
background: linear-gradient(
|
|
||||||
to bottom,
|
|
||||||
rgba(var(--fc-chrome-rgb), 0.92) 0%,
|
|
||||||
rgba(var(--fc-chrome-rgb), 0.84) 60%,
|
|
||||||
rgba(var(--fc-chrome-rgb), var(--fc-chrome-seam)) 100%
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
.fc-brand {
|
.fc-brand {
|
||||||
display: flex;
|
display: flex;
|
||||||
|
|||||||
@@ -50,19 +50,14 @@ const projected = ref(null)
|
|||||||
|
|
||||||
const projectedCounts = computed(() => projected.value?.projected || null)
|
const projectedCounts = computed(() => projected.value?.projected || null)
|
||||||
|
|
||||||
// `posts` is named here, not left to the counts grid below it: an artist whose
|
const modalDescription = computed(
|
||||||
// posts are body-only previews as `images: 0`, and a summary line that says
|
() => projected.value
|
||||||
// only "0 images" reads as "this artist is empty" while the apply destroys
|
? `Artist “${props.artistName}” — `
|
||||||
// every captured post body (#3067). Attachments stay in the grid — the grid
|
+ `${projected.value.projected.images} images, `
|
||||||
// renders every key, so this line carries only what changes the read.
|
+ `${projected.value.projected.sources} sources, `
|
||||||
const modalDescription = computed(() => {
|
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
|
||||||
const p = projectedCounts.value
|
: '',
|
||||||
return p
|
)
|
||||||
? `Artist “${props.artistName}” — ${p.images} images, `
|
|
||||||
+ `${p.posts} posts, ${p.sources} sources, `
|
|
||||||
+ `${Math.round(p.bytes_on_disk / 1_048_576)} MiB on disk`
|
|
||||||
: ''
|
|
||||||
})
|
|
||||||
|
|
||||||
async function onClick() {
|
async function onClick() {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
filter, applied retroactively to the existing library.
|
filter, applied retroactively to the existing library.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<v-row density="compact">
|
<v-row dense>
|
||||||
<v-col cols="6">
|
<v-col cols="6">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model.number="minW" label="Min width (px)" type="number"
|
v-model.number="minW" label="Min width (px)" type="number"
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
cadence as the transparency audit.
|
cadence as the transparency audit.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<v-row density="compact">
|
<v-row dense>
|
||||||
<v-col cols="6">
|
<v-col cols="6">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model.number="threshold" label="Threshold (0–1)"
|
v-model.number="threshold" label="Threshold (0–1)"
|
||||||
|
|||||||
@@ -1,56 +0,0 @@
|
|||||||
<!--
|
|
||||||
Canonical settings number field (DRY pass #161): a compact numeric v-text-field
|
|
||||||
with a built-in clamp to [min,max] on commit. Hand-rolled identically across the
|
|
||||||
ML settings cards (HeadsCard x6, CropProposersCard, VideoEmbeddingCard).
|
|
||||||
|
|
||||||
The clamp is the point: the cards previously sent Number(raw) straight to the
|
|
||||||
API, so an out-of-range value bounced off the API's 400 validator (only
|
|
||||||
TranslationCard clamped). This is now the single home for that clamp.
|
|
||||||
|
|
||||||
Binds `modelValue` (v-model) and emits `change` on blur/enter AFTER clamping, so
|
|
||||||
the parent's save reads the already-clamped value — same as the prior
|
|
||||||
`v-model.number` + `@change=save` pattern.
|
|
||||||
-->
|
|
||||||
<template>
|
|
||||||
<v-text-field
|
|
||||||
:model-value="modelValue"
|
|
||||||
:label="label"
|
|
||||||
type="number"
|
|
||||||
:min="min"
|
|
||||||
:max="max"
|
|
||||||
:step="step"
|
|
||||||
:disabled="disabled"
|
|
||||||
:density="density" hide-details
|
|
||||||
:style="{ maxWidth }"
|
|
||||||
@update:model-value="v => emit('update:modelValue', v)"
|
|
||||||
@change="onCommit"
|
|
||||||
/>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
const props = defineProps({
|
|
||||||
modelValue: { type: [Number, String], default: null },
|
|
||||||
label: { type: String, default: '' },
|
|
||||||
min: { type: [Number, String], default: null },
|
|
||||||
max: { type: [Number, String], default: null },
|
|
||||||
step: { type: [Number, String], default: 1 },
|
|
||||||
maxWidth: { type: String, default: '200px' },
|
|
||||||
density: { type: String, default: 'compact' },
|
|
||||||
disabled: { type: Boolean, default: false },
|
|
||||||
})
|
|
||||||
const emit = defineEmits(['update:modelValue', 'change'])
|
|
||||||
|
|
||||||
function onCommit() {
|
|
||||||
// On blur/enter: coerce to a number and clamp to [min,max] so an out-of-range
|
|
||||||
// value never reaches the API. props.modelValue reflects the latest keystroke
|
|
||||||
// (kept in sync by the passthrough above); re-emit the clamped number, then let
|
|
||||||
// the parent persist.
|
|
||||||
let n = Number(props.modelValue)
|
|
||||||
if (!Number.isNaN(n)) {
|
|
||||||
if (props.min !== null && props.min !== '') n = Math.max(Number(props.min), n)
|
|
||||||
if (props.max !== null && props.max !== '') n = Math.min(Number(props.max), n)
|
|
||||||
if (n !== Number(props.modelValue)) emit('update:modelValue', n)
|
|
||||||
}
|
|
||||||
emit('change')
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
<!--
|
|
||||||
Canonical settings toggle row (DRY pass #161): an accent icon + an uppercase
|
|
||||||
.fc-section-h label + a right-aligned switch. Hand-rolled identically in the
|
|
||||||
ML settings cards (HeadsCard x3, CropProposersCard, MLBackfillCard).
|
|
||||||
|
|
||||||
Two-way binds `modelValue` (so the parent switch state stays optimistic) AND
|
|
||||||
emits `change` with the new boolean, so the parent can persist + revert on
|
|
||||||
failure — matching the prior `v-model` + `@update:model-value=handler` pattern.
|
|
||||||
-->
|
|
||||||
<template>
|
|
||||||
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
|
||||||
<v-icon v-if="icon" size="18" :color="iconColor">{{ icon }}</v-icon>
|
|
||||||
<span class="fc-section-h">{{ label }}</span>
|
|
||||||
<v-switch
|
|
||||||
:model-value="modelValue"
|
|
||||||
:loading="loading"
|
|
||||||
:disabled="disabled"
|
|
||||||
hide-details density="compact" color="success" class="ml-auto"
|
|
||||||
@update:model-value="onSwitch"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
defineProps({
|
|
||||||
modelValue: { type: Boolean, default: false },
|
|
||||||
label: { type: String, default: '' },
|
|
||||||
icon: { type: String, default: '' },
|
|
||||||
// Icon tint. Default accent; pass null for the theme default (e.g. when a row
|
|
||||||
// is off). null (not undefined) so the default doesn't override it.
|
|
||||||
iconColor: { type: String, default: 'accent' },
|
|
||||||
loading: { type: Boolean, default: false },
|
|
||||||
disabled: { type: Boolean, default: false },
|
|
||||||
})
|
|
||||||
const emit = defineEmits(['update:modelValue', 'change'])
|
|
||||||
|
|
||||||
function onSwitch(v) {
|
|
||||||
const b = !!v
|
|
||||||
emit('update:modelValue', b)
|
|
||||||
emit('change', b)
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
@@ -14,10 +14,10 @@
|
|||||||
@update:search="onSearch"
|
@update:search="onSearch"
|
||||||
@update:model-value="onPick"
|
@update:model-value="onPick"
|
||||||
>
|
>
|
||||||
<template #item="{ props: itemProps, internalItem }">
|
<template #item="{ props: itemProps, item }">
|
||||||
<v-list-item v-bind="itemProps" :title="internalItem.raw.name">
|
<v-list-item v-bind="itemProps" :title="item.raw.name">
|
||||||
<template #subtitle>
|
<template #subtitle>
|
||||||
{{ internalItem.raw.fandom_name ? `character · ${internalItem.raw.fandom_name}` : internalItem.raw.kind }}
|
{{ item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind }}
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
</template>
|
</template>
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<template>
|
<template>
|
||||||
<div class="fc-filterbar-wrap fc-chrome-continues">
|
<div class="fc-filterbar-wrap">
|
||||||
<div class="fc-filterbar">
|
<div class="fc-filterbar">
|
||||||
<v-autocomplete
|
<v-autocomplete
|
||||||
v-model="selected"
|
v-model="selected"
|
||||||
@@ -13,14 +13,14 @@
|
|||||||
@update:search="onSearch"
|
@update:search="onSearch"
|
||||||
@update:model-value="onPick"
|
@update:model-value="onPick"
|
||||||
>
|
>
|
||||||
<template #item="{ props: itemProps, internalItem }">
|
<template #item="{ props: itemProps, item }">
|
||||||
<v-list-item v-bind="itemProps" :title="internalItem.raw.name">
|
<v-list-item v-bind="itemProps" :title="item.raw.name">
|
||||||
<template #prepend>
|
<template #prepend>
|
||||||
<v-icon size="small">{{ iconFor(internalItem.raw) }}</v-icon>
|
<v-icon size="small">{{ iconFor(item.raw) }}</v-icon>
|
||||||
</template>
|
</template>
|
||||||
<template #subtitle>
|
<template #subtitle>
|
||||||
{{ internalItem.raw.kind === 'artist' ? 'artist'
|
{{ item.raw.kind === 'artist' ? 'artist'
|
||||||
: (internalItem.raw.fandom_name ? `character · ${internalItem.raw.fandom_name}` : internalItem.raw.kind) }}
|
: (item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind) }}
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
</template>
|
</template>
|
||||||
@@ -306,17 +306,27 @@ function pushFilter(mutate) {
|
|||||||
frosted block pinned directly under the 64px TopNav and continuous with it. */
|
frosted block pinned directly under the 64px TopNav and continuous with it. */
|
||||||
.fc-filterbar-wrap {
|
.fc-filterbar-wrap {
|
||||||
position: sticky;
|
position: sticky;
|
||||||
top: var(--fc-nav-h, 64px); /* pins at the nav's real measured bottom (#1481) */
|
top: 64px;
|
||||||
z-index: 5;
|
z-index: 5;
|
||||||
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
|
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
|
||||||
so the bar sits flush at 64px even at scroll 0 — without this it detaches
|
so the bar sits flush at 64px even at scroll 0 — without this it detaches
|
||||||
and a gap shows through when scrolled to the top. */
|
and a gap shows through when scrolled to the top. */
|
||||||
margin-top: -8px;
|
margin-top: -8px;
|
||||||
margin-bottom: 12px;
|
margin-bottom: 12px;
|
||||||
/* The frost itself (obsidian fade + blur) is the shared .fc-chrome-continues
|
/* EXACT same gradiated obsidian (#14171A = 20,23,26) frost as the TopNav so
|
||||||
primitive: it CONTINUES the TopNav's fade from the seam alpha to transparent
|
the two read as one continuous piece of chrome — images scroll visibly
|
||||||
rather than re-darkening, so the nav + bar read as one gradient (operator
|
under both. The nav's gradient fades to transparent at ITS bottom; this
|
||||||
2026-07-13). This block only owns the sticky positioning now. */
|
bar re-darkens at its top, so a faint seam (the page/image showing through
|
||||||
|
the nav's transparent edge) separates them when scrolled to the very top,
|
||||||
|
while under-scroll they frost as one. */
|
||||||
|
background: linear-gradient(
|
||||||
|
to bottom,
|
||||||
|
rgba(20, 23, 26, 0.92) 0%,
|
||||||
|
rgba(20, 23, 26, 0.65) 60%,
|
||||||
|
rgba(20, 23, 26, 0) 100%
|
||||||
|
);
|
||||||
|
backdrop-filter: blur(2px);
|
||||||
|
-webkit-backdrop-filter: blur(2px);
|
||||||
}
|
}
|
||||||
.fc-filterbar {
|
.fc-filterbar {
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -331,20 +341,6 @@ function pushFilter(mutate) {
|
|||||||
.fc-filterbar-wrap :deep(.v-btn-group) {
|
.fc-filterbar-wrap :deep(.v-btn-group) {
|
||||||
background-color: rgba(20, 23, 26, 0.72);
|
background-color: rgba(20, 23, 26, 0.72);
|
||||||
}
|
}
|
||||||
/* Media toggle (All / Images / Videos) as ONE cohesive segmented control.
|
|
||||||
FC's global VBtn { rounded: 'pill' } default made Vuetify 4 pill-round each
|
|
||||||
SEGMENT individually, so the rounded ends collided at the joins — the shapes
|
|
||||||
landed awkwardly on the button edges (operator 2026-07-13). Square the inner
|
|
||||||
segments (over the pill utility's !important) and clip the group to a single
|
|
||||||
8px outline (matches the chips/tiles rounding elsewhere in the app). Radius
|
|
||||||
only — no height change, so the bar height and nav offset are untouched. */
|
|
||||||
.fc-filterbar-wrap :deep(.v-btn-toggle) {
|
|
||||||
border-radius: 8px;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
.fc-filterbar-wrap :deep(.v-btn-toggle .v-btn) {
|
|
||||||
border-radius: 0 !important;
|
|
||||||
}
|
|
||||||
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
|
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
|
||||||
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
||||||
/* The tag chips' bodies toggle include/exclude — signal they're clickable. */
|
/* The tag chips' bodies toggle include/exclude — signal they're clickable. */
|
||||||
|
|||||||
@@ -139,9 +139,9 @@ function onThumbError() { thumbError.value = true }
|
|||||||
position: absolute; top: 8px; left: 8px;
|
position: absolute; top: 8px; left: 8px;
|
||||||
width: 22px; height: 22px; border-radius: 4px;
|
width: 22px; height: 22px; border-radius: 4px;
|
||||||
border: 2px solid rgba(232, 228, 216, 0.8);
|
border: 2px solid rgba(232, 228, 216, 0.8);
|
||||||
background: rgba(var(--v-theme-background), 0.45);
|
background: rgba(20, 23, 26, 0.45);
|
||||||
display: grid; place-items: center;
|
display: grid; place-items: center;
|
||||||
color: rgb(var(--v-theme-background)); z-index: 11;
|
color: #14171A; z-index: 11;
|
||||||
}
|
}
|
||||||
.fc-gallery-item__checkbox.on {
|
.fc-gallery-item__checkbox.on {
|
||||||
background: rgb(var(--v-theme-accent));
|
background: rgb(var(--v-theme-accent));
|
||||||
@@ -152,7 +152,7 @@ function onThumbError() { thumbError.value = true }
|
|||||||
min-width: 22px; height: 22px; padding: 0 5px;
|
min-width: 22px; height: 22px; padding: 0 5px;
|
||||||
border-radius: 11px;
|
border-radius: 11px;
|
||||||
background: rgb(var(--v-theme-accent));
|
background: rgb(var(--v-theme-accent));
|
||||||
color: rgb(var(--v-theme-background)); font-size: 12px; font-weight: 700;
|
color: #14171A; font-size: 12px; font-weight: 700;
|
||||||
display: grid; place-items: center; z-index: 11;
|
display: grid; place-items: center; z-index: 11;
|
||||||
pointer-events: none;
|
pointer-events: none;
|
||||||
}
|
}
|
||||||
@@ -160,8 +160,7 @@ function onThumbError() { thumbError.value = true }
|
|||||||
position: absolute; left: 0; right: 0; bottom: 0;
|
position: absolute; left: 0; right: 0; bottom: 0;
|
||||||
padding: 14px 8px 6px;
|
padding: 14px 8px 6px;
|
||||||
background: linear-gradient(
|
background: linear-gradient(
|
||||||
to top, rgba(var(--v-theme-background), 0.78),
|
to top, rgba(20, 23, 26, 0.78), rgba(20, 23, 26, 0)
|
||||||
rgba(var(--v-theme-background), 0)
|
|
||||||
);
|
);
|
||||||
font-size: 12px; line-height: 1.2;
|
font-size: 12px; line-height: 1.2;
|
||||||
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||||
|
|||||||
@@ -1,15 +1,14 @@
|
|||||||
<template>
|
<template>
|
||||||
<!-- System-tag auto-applies (chrome hides / process WIP tags) that ALSO looked
|
<!-- Auto-hidden chrome that ALSO looked like real content — surfaced PROACTIVELY
|
||||||
like real content — surfaced PROACTIVELY atop the gallery whenever there's
|
atop the gallery whenever there's something to review (NOT gated on the
|
||||||
something to review (NOT gated on the Show-hidden toggle, so misfires can't
|
Show-hidden toggle, so misfires can't go unnoticed), most-concerning first,
|
||||||
go unnoticed), most-concerning first, with keep / remove (#141, #1464).
|
with keep / un-hide (#141). Renders nothing when there's nothing to review. -->
|
||||||
Renders nothing when there's nothing to review. -->
|
<section v-if="items.length" class="fc-review" aria-label="Hidden images to review">
|
||||||
<section v-if="items.length" class="fc-review" aria-label="Auto-tagged images to review">
|
|
||||||
<div class="fc-review__head">
|
<div class="fc-review__head">
|
||||||
<v-icon size="18" color="warning">mdi-alert-outline</v-icon>
|
<v-icon size="18" color="warning">mdi-alert-outline</v-icon>
|
||||||
<span class="fc-review__title">
|
<span class="fc-review__title">
|
||||||
{{ items.length }} auto-tagged {{ items.length === 1 ? 'image' : 'images' }}
|
{{ items.length }} auto-hidden {{ items.length === 1 ? 'image' : 'images' }}
|
||||||
may be real content — review
|
may be real content — review before they stay hidden
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="fc-review__cards">
|
<div class="fc-review__cards">
|
||||||
@@ -27,16 +26,16 @@
|
|||||||
>
|
>
|
||||||
also looks like <strong>{{ it.conflict_name || 'content' }}</strong>
|
also looks like <strong>{{ it.conflict_name || 'content' }}</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="fc-review-card__tag">{{ tagLine(it) }}</div>
|
<div class="fc-review-card__tag">hidden as {{ it.tag_name }}</div>
|
||||||
<div class="fc-review-card__acts">
|
<div class="fc-review-card__acts">
|
||||||
<button
|
<button
|
||||||
type="button" class="fc-review-btn fc-review-btn--keep"
|
type="button" class="fc-review-btn fc-review-btn--keep"
|
||||||
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')"
|
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')"
|
||||||
>{{ keepLabel(it) }}</button>
|
>Keep hidden</button>
|
||||||
<button
|
<button
|
||||||
type="button" class="fc-review-btn fc-review-btn--unhide"
|
type="button" class="fc-review-btn fc-review-btn--unhide"
|
||||||
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')"
|
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')"
|
||||||
>{{ removeLabel(it) }}</button>
|
>Un-hide</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -55,11 +54,6 @@ const items = ref([])
|
|||||||
const busy = ref([])
|
const busy = ref([])
|
||||||
|
|
||||||
function keyOf(it) { return `${it.image_id}:${it.tag_id}` }
|
function keyOf(it) { return `${it.image_id}:${it.tag_id}` }
|
||||||
// Chrome flags hide the image (keep-hidden / un-hide); process flags leave it
|
|
||||||
// visible and just tagged (keep-tag / remove-tag). Same endpoints, different words.
|
|
||||||
function tagLine(it) { return (it.mode === 'process' ? 'auto-tagged ' : 'hidden as ') + it.tag_name }
|
|
||||||
function keepLabel(it) { return it.mode === 'process' ? 'Keep tag' : 'Keep hidden' }
|
|
||||||
function removeLabel(it) { return it.mode === 'process' ? 'Remove tag' : 'Un-hide' }
|
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
// Fetched unconditionally on mount — the strip prompts for pending misfires
|
// Fetched unconditionally on mount — the strip prompts for pending misfires
|
||||||
@@ -77,8 +71,7 @@ async function resolve(it, action) {
|
|||||||
await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`)
|
await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`)
|
||||||
items.value = items.value.filter((x) => keyOf(x) !== k)
|
items.value = items.value.filter((x) => keyOf(x) !== k)
|
||||||
if (action === 'unhide') {
|
if (action === 'unhide') {
|
||||||
const verb = it.mode === 'process' ? 'Removed' : 'Un-hidden'
|
toast({ text: `Un-hidden — “${it.tag_name}” removed; it'll train the head`, type: 'success' })
|
||||||
toast({ text: `${verb} — “${it.tag_name}” removed; it'll train the head`, type: 'success' })
|
|
||||||
}
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
toast({
|
toast({
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
<template>
|
||||||
|
<v-card>
|
||||||
|
<v-card-title>Treat as alias for…</v-card-title>
|
||||||
|
<v-card-text>
|
||||||
|
<p class="text-caption mb-2">
|
||||||
|
Pick the existing tag the model's prediction should map to. All
|
||||||
|
future predictions of this name will resolve to your tag.
|
||||||
|
</p>
|
||||||
|
<v-autocomplete
|
||||||
|
v-model="selectedId"
|
||||||
|
v-model:menu="menuOpen"
|
||||||
|
:items="results"
|
||||||
|
:item-title="(t) => t.fandom_name ? `${t.name} — ${t.fandom_name}` : t.name"
|
||||||
|
:item-value="(t) => t.id"
|
||||||
|
:loading="loading"
|
||||||
|
label="Canonical tag"
|
||||||
|
no-filter clearable density="compact"
|
||||||
|
@update:search="onSearch"
|
||||||
|
@keydown.enter.capture="onEnter"
|
||||||
|
/>
|
||||||
|
</v-card-text>
|
||||||
|
<v-card-actions>
|
||||||
|
<v-spacer />
|
||||||
|
<v-btn @click="$emit('cancel')">Cancel</v-btn>
|
||||||
|
<v-btn
|
||||||
|
color="primary" rounded="pill" :disabled="!selectedId"
|
||||||
|
@click="$emit('confirm', selectedId)"
|
||||||
|
>Use this tag</v-btn>
|
||||||
|
</v-card-actions>
|
||||||
|
</v-card>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { ref } from 'vue'
|
||||||
|
import { useApi } from '../../composables/useApi.js'
|
||||||
|
import { useAcceptOnEnter } from '../../composables/useAcceptOnEnter.js'
|
||||||
|
|
||||||
|
const props = defineProps({ category: { type: String, required: true } })
|
||||||
|
const emit = defineEmits(['confirm', 'cancel'])
|
||||||
|
|
||||||
|
const api = useApi()
|
||||||
|
const results = ref([])
|
||||||
|
const loading = ref(false)
|
||||||
|
const selectedId = ref(null)
|
||||||
|
let debounce = null
|
||||||
|
|
||||||
|
// Enter on the closed dropdown confirms the selection instead of re-opening it.
|
||||||
|
const { menuOpen, onEnter } = useAcceptOnEnter(() => {
|
||||||
|
if (selectedId.value != null) emit('confirm', selectedId.value)
|
||||||
|
})
|
||||||
|
|
||||||
|
function onSearch(q) {
|
||||||
|
if (debounce) clearTimeout(debounce)
|
||||||
|
debounce = setTimeout(async () => {
|
||||||
|
const query = (q || '').trim()
|
||||||
|
if (!query) { results.value = []; return }
|
||||||
|
loading.value = true
|
||||||
|
try {
|
||||||
|
// Scope the autocomplete to the prediction's category where it
|
||||||
|
// maps to a tag kind. Only 'character' surfaces as both a
|
||||||
|
// suggestion category and a tag kind now ('artist' + 'copyright'
|
||||||
|
// retired); other categories search unscoped.
|
||||||
|
const kind = props.category === 'character' ? 'character' : null
|
||||||
|
const params = { q: query, limit: 20 }
|
||||||
|
if (kind) params.kind = kind
|
||||||
|
results.value = await api.get('/api/tags/autocomplete', { params })
|
||||||
|
} finally {
|
||||||
|
loading.value = false
|
||||||
|
}
|
||||||
|
}, 200)
|
||||||
|
}
|
||||||
|
</script>
|
||||||
@@ -11,6 +11,10 @@
|
|||||||
{{ suggestion.display_name }}
|
{{ suggestion.display_name }}
|
||||||
<span v-if="suggestion.rejected" class="fc-suggestion__rejected-tag"
|
<span v-if="suggestion.rejected" class="fc-suggestion__rejected-tag"
|
||||||
title="You rejected this for this image — un-reject to recover">rejected</span>
|
title="You rejected this for this image — un-reject to recover">rejected</span>
|
||||||
|
<span v-else-if="suggestion.creates_new_tag" class="fc-suggestion__new"
|
||||||
|
title="No matching tag yet — accepting creates it">+ new</span>
|
||||||
|
<span v-else-if="suggestion.via_alias" class="fc-suggestion__alias"
|
||||||
|
:title="`Mapped from the tagger's “${suggestion.raw_name}” via an alias`">alias</span>
|
||||||
</span>
|
</span>
|
||||||
<span class="fc-suggestion__score">{{ scorePct }}</span>
|
<span class="fc-suggestion__score">{{ scorePct }}</span>
|
||||||
<!-- Green ✓ / red ✗ pair (operator-asked 2026-06-28) mirrors the eval
|
<!-- Green ✓ / red ✗ pair (operator-asked 2026-06-28) mirrors the eval
|
||||||
@@ -42,14 +46,40 @@
|
|||||||
@click="$emit('dismiss', suggestion)"
|
@click="$emit('dismiss', suggestion)"
|
||||||
><v-icon size="16">mdi-close</v-icon></button>
|
><v-icon size="16">mdi-close</v-icon></button>
|
||||||
</div>
|
</div>
|
||||||
|
<!-- Modal-safe kebab is baked into KebabMenu (this row lives in the
|
||||||
|
teleported image modal — #711). Only rendered when an alias action
|
||||||
|
applies — dismiss now lives on the red ✗, so a centroid hit with no
|
||||||
|
alias option has no menu. -->
|
||||||
|
<KebabMenu
|
||||||
|
v-if="hasMenu"
|
||||||
|
class="fc-suggestion__menu" size="small" variant="outlined"
|
||||||
|
:label="`More actions for ${suggestion.display_name}`"
|
||||||
|
>
|
||||||
|
<!-- Alias is a tagger-prediction remap, so only offer it for tagger
|
||||||
|
suggestions with a raw model key that aren't already aliased.
|
||||||
|
Centroid hits (raw_name null) have nothing to alias. -->
|
||||||
|
<v-list-item
|
||||||
|
v-if="suggestion.raw_name && !suggestion.via_alias"
|
||||||
|
@click="$emit('alias', suggestion)"
|
||||||
|
>
|
||||||
|
<v-list-item-title>Treat as alias for…</v-list-item-title>
|
||||||
|
</v-list-item>
|
||||||
|
<v-list-item
|
||||||
|
v-if="suggestion.via_alias"
|
||||||
|
@click="$emit('remove-alias', suggestion)"
|
||||||
|
>
|
||||||
|
<v-list-item-title>Remove alias</v-list-item-title>
|
||||||
|
</v-list-item>
|
||||||
|
</KebabMenu>
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, inject } from 'vue'
|
import { computed, inject } from 'vue'
|
||||||
|
import KebabMenu from '../common/KebabMenu.vue'
|
||||||
|
|
||||||
const props = defineProps({ suggestion: { type: Object, required: true } })
|
const props = defineProps({ suggestion: { type: Object, required: true } })
|
||||||
defineEmits(['accept', 'dismiss', 'undismiss'])
|
defineEmits(['accept', 'alias', 'remove-alias', 'dismiss', 'undismiss'])
|
||||||
|
|
||||||
// #1206: on hover, tell the image viewer which crop produced this suggestion so
|
// #1206: on hover, tell the image viewer which crop produced this suggestion so
|
||||||
// it highlights that region. Provided by ImageViewer/Explore; a no-op elsewhere.
|
// it highlights that region. Provided by ImageViewer/Explore; a no-op elsewhere.
|
||||||
@@ -62,6 +92,12 @@ function onLeave () {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
||||||
|
// Kebab now only carries alias actions: show it when this suggestion can be
|
||||||
|
// aliased (raw model key, not yet aliased) or is already aliased (so it can be
|
||||||
|
// un-aliased). Centroid hits (no raw_name, no alias) have an empty menu → hide.
|
||||||
|
const hasMenu = computed(() =>
|
||||||
|
Boolean(props.suggestion.raw_name) || Boolean(props.suggestion.via_alias)
|
||||||
|
)
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
@@ -83,6 +119,26 @@ const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
|||||||
color: rgb(var(--v-theme-on-surface));
|
color: rgb(var(--v-theme-on-surface));
|
||||||
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||||
}
|
}
|
||||||
|
.fc-suggestion__new {
|
||||||
|
display: inline-block;
|
||||||
|
font-size: 10px; font-weight: 600;
|
||||||
|
color: rgb(var(--v-theme-accent));
|
||||||
|
background: rgba(var(--v-theme-accent), 0.12);
|
||||||
|
border: 1px solid rgb(var(--v-theme-accent), 0.4);
|
||||||
|
padding: 1px 6px; border-radius: 999px;
|
||||||
|
margin-left: 6px;
|
||||||
|
text-transform: uppercase; letter-spacing: 0.04em;
|
||||||
|
}
|
||||||
|
.fc-suggestion__alias {
|
||||||
|
display: inline-block;
|
||||||
|
font-size: 10px; font-weight: 600;
|
||||||
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
|
background: rgb(var(--v-theme-surface-light));
|
||||||
|
border: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
|
padding: 1px 6px; border-radius: 999px;
|
||||||
|
margin-left: 6px;
|
||||||
|
text-transform: uppercase; letter-spacing: 0.04em;
|
||||||
|
}
|
||||||
.fc-suggestion__score {
|
.fc-suggestion__score {
|
||||||
flex: 0 0 auto; min-width: 38px; text-align: right;
|
flex: 0 0 auto; min-width: 38px; text-align: right;
|
||||||
font-size: 11px;
|
font-size: 11px;
|
||||||
@@ -110,6 +166,10 @@ const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
|||||||
background: transparent; color: rgb(var(--v-theme-on-surface-variant));
|
background: transparent; color: rgb(var(--v-theme-on-surface-variant));
|
||||||
border: 1px solid rgb(var(--v-theme-on-surface-variant), 0.5);
|
border: 1px solid rgb(var(--v-theme-on-surface-variant), 0.5);
|
||||||
}
|
}
|
||||||
|
.fc-suggestion__menu {
|
||||||
|
flex: 0 0 auto;
|
||||||
|
}
|
||||||
|
|
||||||
/* Rejected state: the row stays put (recovery), dimmed + red-edged so it
|
/* Rejected state: the row stays put (recovery), dimmed + red-edged so it
|
||||||
reads as "handled, negative" without shouting over live suggestions. */
|
reads as "handled, negative" without shouting over live suggestions. */
|
||||||
.fc-suggestion--rejected {
|
.fc-suggestion--rejected {
|
||||||
|
|||||||
@@ -26,6 +26,8 @@
|
|||||||
v-for="(s, i) in items" :key="`${s.display_name}-${i}`"
|
v-for="(s, i) in items" :key="`${s.display_name}-${i}`"
|
||||||
:suggestion="s"
|
:suggestion="s"
|
||||||
@accept="$emit('accept', $event)"
|
@accept="$emit('accept', $event)"
|
||||||
|
@alias="$emit('alias', $event)"
|
||||||
|
@remove-alias="$emit('remove-alias', $event)"
|
||||||
@dismiss="$emit('dismiss', $event)"
|
@dismiss="$emit('dismiss', $event)"
|
||||||
@undismiss="$emit('undismiss', $event)"
|
@undismiss="$emit('undismiss', $event)"
|
||||||
/>
|
/>
|
||||||
@@ -43,7 +45,7 @@ const props = defineProps({
|
|||||||
collapsible: { type: Boolean, default: false },
|
collapsible: { type: Boolean, default: false },
|
||||||
defaultOpen: { type: Boolean, default: true }
|
defaultOpen: { type: Boolean, default: true }
|
||||||
})
|
})
|
||||||
defineEmits(['accept', 'dismiss', 'undismiss', 'reject-all'])
|
defineEmits(['accept', 'alias', 'remove-alias', 'dismiss', 'undismiss', 'reject-all'])
|
||||||
|
|
||||||
// Still-unhandled suggestions (not yet rejected) — how many "Reject rest" clears.
|
// Still-unhandled suggestions (not yet rejected) — how many "Reject rest" clears.
|
||||||
const rejectableCount = computed(() => props.items.filter((s) => !s.rejected).length)
|
const rejectableCount = computed(() => props.items.filter((s) => !s.rejected).length)
|
||||||
|
|||||||
@@ -20,39 +20,48 @@
|
|||||||
first, so false positives are easy to spot and reject (reject-to-train
|
first, so false positives are easy to spot and reject (reject-to-train
|
||||||
for these heads). Small set — collapsible, open by default. -->
|
for these heads). Small set — collapsible, open by default. -->
|
||||||
<SuggestionsCategoryGroup
|
<SuggestionsCategoryGroup
|
||||||
v-if="store.aboveByCategory.system && store.aboveByCategory.system.length"
|
v-if="store.byCategory.system && store.byCategory.system.length"
|
||||||
label="System" :items="store.aboveByCategory.system"
|
label="System" :items="store.byCategory.system"
|
||||||
collapsible :default-open="true"
|
collapsible :default-open="true"
|
||||||
@accept="onAccept"
|
@accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias"
|
||||||
@dismiss="onDismiss" @undismiss="onUndismiss"
|
@dismiss="onDismiss" @undismiss="onUndismiss"
|
||||||
@reject-all="onRejectAll('system')"
|
@reject-all="onRejectAll('system')"
|
||||||
/>
|
/>
|
||||||
<SuggestionsCategoryGroup
|
<SuggestionsCategoryGroup
|
||||||
v-for="cat in peopleCats" :key="cat"
|
v-for="cat in peopleCats" :key="cat"
|
||||||
v-show="store.aboveByCategory[cat] && store.aboveByCategory[cat].length"
|
v-show="store.byCategory[cat] && store.byCategory[cat].length"
|
||||||
:label="labelFor(cat)" :items="store.aboveByCategory[cat] || []"
|
:label="labelFor(cat)" :items="store.byCategory[cat] || []"
|
||||||
@accept="onAccept"
|
@accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias"
|
||||||
@dismiss="onDismiss" @undismiss="onUndismiss"
|
@dismiss="onDismiss" @undismiss="onUndismiss"
|
||||||
@reject-all="onRejectAll(cat)"
|
@reject-all="onRejectAll(cat)"
|
||||||
/>
|
/>
|
||||||
<SuggestionsCategoryGroup
|
<SuggestionsCategoryGroup
|
||||||
v-if="store.aboveByCategory.general && store.aboveByCategory.general.length"
|
v-if="store.byCategory.general && store.byCategory.general.length"
|
||||||
label="General" :items="store.aboveByCategory.general"
|
label="General" :items="store.byCategory.general"
|
||||||
collapsible :default-open="true"
|
collapsible :default-open="true"
|
||||||
@accept="onAccept"
|
@accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias"
|
||||||
@dismiss="onDismiss" @undismiss="onUndismiss"
|
@dismiss="onDismiss" @undismiss="onUndismiss"
|
||||||
@reject-all="onRejectAll('general')"
|
@reject-all="onRejectAll('general')"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<v-dialog v-model="aliasDialog" max-width="480">
|
||||||
|
<AliasPickerDialog
|
||||||
|
v-if="aliasTarget"
|
||||||
|
:category="aliasTarget.category"
|
||||||
|
@confirm="onAliasConfirm" @cancel="aliasDialog = false"
|
||||||
|
/>
|
||||||
|
</v-dialog>
|
||||||
</section>
|
</section>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { toast } from '../../utils/toast.js'
|
import { toast } from '../../utils/toast.js'
|
||||||
import { computed, watch } from 'vue'
|
import { computed, ref, watch } from 'vue'
|
||||||
import { useSuggestionsStore, CATEGORY_LABELS } from '../../stores/suggestions.js'
|
import { useSuggestionsStore, CATEGORY_LABELS } from '../../stores/suggestions.js'
|
||||||
import { useModalStore } from '../../stores/modal.js'
|
import { useModalStore } from '../../stores/modal.js'
|
||||||
import SuggestionsCategoryGroup from './SuggestionsCategoryGroup.vue'
|
import SuggestionsCategoryGroup from './SuggestionsCategoryGroup.vue'
|
||||||
|
import AliasPickerDialog from './AliasPickerDialog.vue'
|
||||||
|
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
imageId: { type: Number, required: true },
|
imageId: { type: Number, required: true },
|
||||||
@@ -91,14 +100,13 @@ const peopleCats = ['character']
|
|||||||
function labelFor(c) { return CATEGORY_LABELS[c] || c }
|
function labelFor(c) { return CATEGORY_LABELS[c] || c }
|
||||||
|
|
||||||
const isEmpty = computed(() =>
|
const isEmpty = computed(() =>
|
||||||
Object.values(store.aboveByCategory).every(list => !list || list.length === 0)
|
Object.values(store.byCategory).every(list => !list || list.length === 0)
|
||||||
)
|
)
|
||||||
|
|
||||||
watch(() => props.imageId, (id) => {
|
watch(() => props.imageId, (id) => {
|
||||||
if (id == null) return
|
if (id == null) return
|
||||||
// One fetch (min=0) backs both surfaces: the panel reads aboveByCategory, the
|
store.load(id) // panel: curated, ≥ threshold
|
||||||
// typed tag-input dropdown reads the full set. No second request.
|
store.loadAll(id) // dropdown: full prediction set (low-confidence included)
|
||||||
store.load(id)
|
|
||||||
}, { immediate: true })
|
}, { immediate: true })
|
||||||
|
|
||||||
// After a successful accept/alias-accept, refresh the modal's current
|
// After a successful accept/alias-accept, refresh the modal's current
|
||||||
@@ -117,6 +125,31 @@ async function onAccept(s) {
|
|||||||
toast({ text: `Accept failed: ${e.message}`, type: 'error' })
|
toast({ text: `Accept failed: ${e.message}`, type: 'error' })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const aliasDialog = ref(false)
|
||||||
|
const aliasTarget = ref(null)
|
||||||
|
function onAlias(s) { aliasTarget.value = s; aliasDialog.value = true }
|
||||||
|
async function onAliasConfirm(canonicalTagId) {
|
||||||
|
try {
|
||||||
|
await store.aliasAccept(aliasTarget.value, canonicalTagId)
|
||||||
|
aliasDialog.value = false
|
||||||
|
await host.reloadTags()
|
||||||
|
emit('accepted')
|
||||||
|
} catch (e) {
|
||||||
|
toast({ text: `Alias failed: ${e.message}`, type: 'error' })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Undo the model-key→tag mapping behind an aliased suggestion. The store
|
||||||
|
// reloads suggestions so the prediction reverts to its raw form; the applied
|
||||||
|
// canonical tag (if any) stays, so no tag-rail reload is needed.
|
||||||
|
async function onRemoveAlias(s) {
|
||||||
|
try {
|
||||||
|
await store.removeAlias(s)
|
||||||
|
} catch (e) {
|
||||||
|
toast({ text: `Remove alias failed: ${e.message}`, type: 'error' })
|
||||||
|
}
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
|
|||||||
@@ -17,14 +17,7 @@
|
|||||||
density="compact" class="fc-tag-autocomplete__list"
|
density="compact" class="fc-tag-autocomplete__list"
|
||||||
>
|
>
|
||||||
<template v-for="(row, idx) in rows" :key="row.key">
|
<template v-for="(row, idx) in rows" :key="row.key">
|
||||||
<!-- One unified row per DB tag (server autocomplete). Every suggestion is
|
<!-- Existing tag match (server autocomplete). -->
|
||||||
a canonical tag now, so when the model ALSO scored this tag for the
|
|
||||||
image the row just carries its confidence (🔧 %) in place of the
|
|
||||||
redundant kind label — the coloured leading icon already shows kind.
|
|
||||||
That means a searched tag that's also a suggestion shows its % on ONE
|
|
||||||
row, with no dedup and no flicker. Picking it emits pick-existing;
|
|
||||||
TagPanel.findPending routes a matching suggestion through accept() so
|
|
||||||
the acceptance is still recorded + it drops from the panel. -->
|
|
||||||
<v-list-item
|
<v-list-item
|
||||||
v-if="row.type === 'hit'"
|
v-if="row.type === 'hit'"
|
||||||
:active="idx === highlight" @click="onPickRow(row)"
|
:active="idx === highlight" @click="onPickRow(row)"
|
||||||
@@ -39,18 +32,32 @@
|
|||||||
<span v-if="row.hit.fandom_name" class="text-caption">— {{ row.hit.fandom_name }}</span>
|
<span v-if="row.hit.fandom_name" class="text-caption">— {{ row.hit.fandom_name }}</span>
|
||||||
</v-list-item-title>
|
</v-list-item-title>
|
||||||
<template #append>
|
<template #append>
|
||||||
<span
|
<span class="text-caption">{{ row.hit.kind }}</span>
|
||||||
v-if="row.sugg"
|
</template>
|
||||||
class="fc-tag-autocomplete__sugg-tag"
|
</v-list-item>
|
||||||
:class="{ 'fc-tag-autocomplete__sugg-tag--below': !row.sugg.above_threshold }"
|
|
||||||
:title="row.sugg.above_threshold
|
<!-- ML suggestion for THIS image that matches the typed query. Picking
|
||||||
? 'The model suggests this tag for this image'
|
it routes through the same accept path as the Suggestions panel, so
|
||||||
: 'The model scored this tag below its suggest threshold'"
|
it's recorded + drops out of the panel (operator-asked 2026-06-07). -->
|
||||||
>
|
<v-list-item
|
||||||
|
v-else-if="row.type === 'suggestion'"
|
||||||
|
:active="idx === highlight" @click="onPickRow(row)"
|
||||||
|
class="fc-tag-autocomplete__sugg"
|
||||||
|
>
|
||||||
|
<template #prepend>
|
||||||
|
<v-icon size="small" :color="store.colorFor(row.sugg.category)">
|
||||||
|
{{ iconFor(row.sugg.category) }}
|
||||||
|
</v-icon>
|
||||||
|
</template>
|
||||||
|
<v-list-item-title>
|
||||||
|
{{ row.sugg.display_name }}
|
||||||
|
<span v-if="row.sugg.creates_new_tag" class="text-caption">— new</span>
|
||||||
|
</v-list-item-title>
|
||||||
|
<template #append>
|
||||||
|
<span class="fc-tag-autocomplete__sugg-tag">
|
||||||
<v-icon size="x-small">mdi-auto-fix</v-icon>
|
<v-icon size="x-small">mdi-auto-fix</v-icon>
|
||||||
{{ scorePct(row.sugg) }}
|
{{ scorePct(row.sugg) }}
|
||||||
</span>
|
</span>
|
||||||
<span v-else class="text-caption">{{ row.hit.kind }}</span>
|
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
|
|
||||||
@@ -93,10 +100,11 @@ import { useSuggestionsStore } from '../../stores/suggestions.js'
|
|||||||
import { useInflightToken } from '../../composables/useInflightToken.js'
|
import { useInflightToken } from '../../composables/useInflightToken.js'
|
||||||
import FandomPicker from './FandomPicker.vue'
|
import FandomPicker from './FandomPicker.vue'
|
||||||
|
|
||||||
const emit = defineEmits(['pick-existing', 'pick-new', 'cancel'])
|
const emit = defineEmits(['pick-existing', 'pick-new', 'accept-suggestion', 'cancel'])
|
||||||
// The host surface's applied tags (host.current?.tags). The dropdown filters
|
// The host surface's applied tags (host.current?.tags). Both dropdown sections
|
||||||
// against this so a just-added tag drops out of the search the moment the chip
|
// filter against this so a just-added tag drops out of the search the moment
|
||||||
// rail updates, not after the next modal refresh (operator-asked 2026-07-03).
|
// the chip rail updates, not after the next modal refresh (operator-asked
|
||||||
|
// 2026-07-03).
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
appliedTags: { type: Array, default: () => [] },
|
appliedTags: { type: Array, default: () => [] },
|
||||||
})
|
})
|
||||||
@@ -230,6 +238,9 @@ const createLabel = computed(() =>
|
|||||||
function scorePct (s) { return `${Math.round(s.score * 100)}%` }
|
function scorePct (s) { return `${Math.round(s.score * 100)}%` }
|
||||||
|
|
||||||
const appliedIds = computed(() => new Set(props.appliedTags.map(t => t.id)))
|
const appliedIds = computed(() => new Set(props.appliedTags.map(t => t.id)))
|
||||||
|
const appliedNames = computed(() =>
|
||||||
|
new Set(props.appliedTags.map(t => `${t.kind}:${(t.name || '').toLowerCase()}`)),
|
||||||
|
)
|
||||||
|
|
||||||
// Server matches minus the image's applied tags. allowCreate/sameNameCharExists
|
// Server matches minus the image's applied tags. allowCreate/sameNameCharExists
|
||||||
// keep reading the RAW hits — they reason about the tag universe (does this tag
|
// keep reading the RAW hits — they reason about the tag universe (does this tag
|
||||||
@@ -238,29 +249,49 @@ const visibleHits = computed(() =>
|
|||||||
hits.value.filter(h => !appliedIds.value.has(h.id)),
|
hits.value.filter(h => !appliedIds.value.has(h.id)),
|
||||||
)
|
)
|
||||||
|
|
||||||
// This image's suggestions keyed by canonical tag id, so a matching DB-tag row
|
// This image's suggestions that match the typed query, minus any the server
|
||||||
// can show the model's confidence inline. Every suggestion is a canonical tag
|
// autocomplete already returned (same name+kind) so a tag never shows twice.
|
||||||
// now (tagging-v2), so the id is the join key — no name/kind matching, no dedup,
|
// Sources the FULL prediction set (allByCategory, down to the store floor) — NOT
|
||||||
// no flicker. Rejected suggestions are excluded: a dismissed tag shouldn't
|
// the threshold-filtered panel list — so a low-confidence action/feature the
|
||||||
// advertise a score in the type-to-add dropdown (un-reject lives in the panel).
|
// model saw can be typed and accepted in canonical formatting instead of being
|
||||||
const suggByTagId = computed(() => {
|
// hand-entered as a custom tag (operator-asked 2026-06-09). The typed query is
|
||||||
const m = new Map()
|
// the only filter; the threshold no longer hides anything here.
|
||||||
for (const list of Object.values(suggestions.byCategory)) {
|
const suggestionHits = computed(() => {
|
||||||
|
const q = parsedName.value.toLowerCase()
|
||||||
|
if (!q) return []
|
||||||
|
const seen = new Set(visibleHits.value.map(h => `${h.kind}:${h.name.toLowerCase()}`))
|
||||||
|
const out = []
|
||||||
|
for (const list of Object.values(suggestions.allByCategory)) {
|
||||||
for (const s of list || []) {
|
for (const s of list || []) {
|
||||||
|
// Rejected suggestions now stay in allByCategory (flagged) so the panel
|
||||||
|
// can show + un-reject them; keep them OUT of the type-to-add dropdown,
|
||||||
|
// whose job is finding a tag to ADD (un-reject lives in the panel).
|
||||||
if (s.rejected) continue
|
if (s.rejected) continue
|
||||||
if (s.canonical_tag_id != null) m.set(s.canonical_tag_id, s)
|
// Already on the image (matched by canonical id or name+kind): nothing
|
||||||
|
// to add. Covers the window between an add and the next suggestions
|
||||||
|
// fetch, where the stale row would otherwise still be pickable.
|
||||||
|
if (s.canonical_tag_id != null && appliedIds.value.has(s.canonical_tag_id)) continue
|
||||||
|
const key = `${s.category}:${s.display_name.toLowerCase()}`
|
||||||
|
if (appliedNames.value.has(key)) continue
|
||||||
|
if (!s.display_name.toLowerCase().includes(q)) continue
|
||||||
|
if (seen.has(key)) continue
|
||||||
|
seen.add(key)
|
||||||
|
out.push(s)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return m
|
// Best matches first; cap generously so a specific typed query surfaces its
|
||||||
|
// matches even when many predictions exist, while the list stays scrollable.
|
||||||
|
out.sort((a, b) => b.score - a.score)
|
||||||
|
return out.slice(0, 20)
|
||||||
})
|
})
|
||||||
|
|
||||||
// One ordered list backing both the rendered dropdown and keyboard nav, so the
|
// One ordered list backing both the rendered dropdown and keyboard nav, so the
|
||||||
// highlight index maps 1:1 to a row. Each hit is annotated with its suggestion
|
// highlight index maps 1:1 to a row regardless of which section it's in.
|
||||||
// (score) when the model scored that tag for this image.
|
|
||||||
const rows = computed(() => {
|
const rows = computed(() => {
|
||||||
const r = visibleHits.value.map(h => ({
|
const r = visibleHits.value.map(h => ({ type: 'hit', key: `h${h.id}`, hit: h }))
|
||||||
type: 'hit', key: `h${h.id}`, hit: h, sugg: suggByTagId.value.get(h.id) || null,
|
for (const s of suggestionHits.value) {
|
||||||
}))
|
r.push({ type: 'suggestion', key: `s${s.category}:${s.display_name}`, sugg: s })
|
||||||
|
}
|
||||||
if (allowCreate.value) r.push({ type: 'create', key: 'create' })
|
if (allowCreate.value) r.push({ type: 'create', key: 'create' })
|
||||||
return r
|
return r
|
||||||
})
|
})
|
||||||
@@ -282,9 +313,16 @@ function moveHighlight (delta) {
|
|||||||
// Dispatch a chosen dropdown row by its type.
|
// Dispatch a chosen dropdown row by its type.
|
||||||
function onPickRow (row) {
|
function onPickRow (row) {
|
||||||
if (row.type === 'hit') { emit('pick-existing', row.hit); reset() }
|
if (row.type === 'hit') { emit('pick-existing', row.hit); reset() }
|
||||||
|
else if (row.type === 'suggestion') { onPickSuggestion(row.sugg) }
|
||||||
else { onCreate() }
|
else { onCreate() }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Picking an ML suggestion hands it to the parent, which runs the same accept
|
||||||
|
// flow as the Suggestions panel (creates the tag if raw, records acceptance,
|
||||||
|
// drops it from the panel). reset() clears the query; the panel-drop makes it
|
||||||
|
// fall out of suggestionHits reactively.
|
||||||
|
function onPickSuggestion (s) { emit('accept-suggestion', s); reset() }
|
||||||
|
|
||||||
function onCreate () {
|
function onCreate () {
|
||||||
const name = parsedName.value
|
const name = parsedName.value
|
||||||
const kind = parsedKind.value
|
const kind = parsedKind.value
|
||||||
@@ -351,16 +389,14 @@ function reset () { query.value = ''; hits.value = []; highlight.value = 0 }
|
|||||||
border: 1px solid rgb(var(--v-theme-surface-light));
|
border: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
}
|
}
|
||||||
/* The model-confidence badge on a row whose tag the model also scored. Accent
|
/* Mark ML-suggestion rows so they read as distinct from typed/known matches. */
|
||||||
when above the head's suggest threshold; muted when below (a low-confidence
|
.fc-tag-autocomplete__sugg {
|
||||||
match you can still type + pick). */
|
border-left: 2px solid rgb(var(--v-theme-accent), 0.5);
|
||||||
|
}
|
||||||
.fc-tag-autocomplete__sugg-tag {
|
.fc-tag-autocomplete__sugg-tag {
|
||||||
display: inline-flex; align-items: center; gap: 2px;
|
display: inline-flex; align-items: center; gap: 2px;
|
||||||
font-size: 11px;
|
font-size: 11px;
|
||||||
font-family: 'JetBrains Mono', monospace;
|
font-family: 'JetBrains Mono', monospace;
|
||||||
color: rgb(var(--v-theme-accent));
|
color: rgb(var(--v-theme-accent));
|
||||||
}
|
}
|
||||||
.fc-tag-autocomplete__sugg-tag--below {
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
}
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -17,6 +17,7 @@
|
|||||||
ref="tagInputRef"
|
ref="tagInputRef"
|
||||||
:applied-tags="host.current?.tags || []"
|
:applied-tags="host.current?.tags || []"
|
||||||
@pick-existing="onPickExisting" @pick-new="onPickNew"
|
@pick-existing="onPickExisting" @pick-new="onPickNew"
|
||||||
|
@accept-suggestion="onAcceptSuggestion"
|
||||||
/>
|
/>
|
||||||
|
|
||||||
<v-alert v-if="errorMsg" type="error" variant="tonal" class="mt-2" closable>
|
<v-alert v-if="errorMsg" type="error" variant="tonal" class="mt-2" closable>
|
||||||
@@ -133,6 +134,7 @@ async function onRemove(tagId) {
|
|||||||
// until the next modal open (operator-asked 2026-07-03).
|
// until the next modal open (operator-asked 2026-07-03).
|
||||||
if (host.currentImageId != null) {
|
if (host.currentImageId != null) {
|
||||||
suggestions.load(host.currentImageId)
|
suggestions.load(host.currentImageId)
|
||||||
|
suggestions.loadAll(host.currentImageId)
|
||||||
}
|
}
|
||||||
focusTagInput()
|
focusTagInput()
|
||||||
}
|
}
|
||||||
@@ -176,6 +178,18 @@ async function onPickNew(payload) {
|
|||||||
}
|
}
|
||||||
catch (e) { errorMsg.value = e.message }
|
catch (e) { errorMsg.value = e.message }
|
||||||
}
|
}
|
||||||
|
// A suggestion picked from the autocomplete dropdown runs the SAME path as the
|
||||||
|
// Suggestions panel's Accept: the store creates the tag if it's raw, records the
|
||||||
|
// acceptance, and drops it from the panel; then we refresh the chip rail.
|
||||||
|
async function onAcceptSuggestion(s) {
|
||||||
|
errorMsg.value = null
|
||||||
|
try {
|
||||||
|
await suggestions.accept(s)
|
||||||
|
await host.reloadTags()
|
||||||
|
focusTagInput()
|
||||||
|
} catch (e) { errorMsg.value = e.message }
|
||||||
|
}
|
||||||
|
|
||||||
const renameDialog = ref(false)
|
const renameDialog = ref(false)
|
||||||
const renameTarget = ref(null)
|
const renameTarget = ref(null)
|
||||||
|
|
||||||
|
|||||||
@@ -70,14 +70,6 @@
|
|||||||
@click.stop="showOriginal = !showOriginal"
|
@click.stop="showOriginal = !showOriginal"
|
||||||
>{{ showOriginal ? 'Show translation' : `Show original${sourceLangLabel}` }}</button>
|
>{{ showOriginal ? 'Show translation' : `Show original${sourceLangLabel}` }}</button>
|
||||||
|
|
||||||
<!-- Per-post translation override (#155): force a skipped translation on,
|
|
||||||
or keep the original for a confidently mis-flagged title. Only where
|
|
||||||
there's text to translate. -->
|
|
||||||
<PostTranslationControl
|
|
||||||
v-if="post.post_title || post.description_plain"
|
|
||||||
:post="post"
|
|
||||||
/>
|
|
||||||
|
|
||||||
<!-- Faithful (semantic) body render once expanded: backend-sanitized
|
<!-- Faithful (semantic) body render once expanded: backend-sanitized
|
||||||
HTML (headings, lists, links, inline images). Collapsed and the
|
HTML (headings, lists, links, inline images). Collapsed and the
|
||||||
no-detail fallback stay plain text. -->
|
no-detail fallback stay plain text. -->
|
||||||
@@ -144,7 +136,6 @@ import { useModalStore } from '../../stores/modal.js'
|
|||||||
import { usePostsStore } from '../../stores/posts.js'
|
import { usePostsStore } from '../../stores/posts.js'
|
||||||
import { toPlainText } from '../../utils/htmlSanitize.js'
|
import { toPlainText } from '../../utils/htmlSanitize.js'
|
||||||
import PostSeriesMenu from './PostSeriesMenu.vue'
|
import PostSeriesMenu from './PostSeriesMenu.vue'
|
||||||
import PostTranslationControl from './PostTranslationControl.vue'
|
|
||||||
|
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
post: { type: Object, required: true },
|
post: { type: Object, required: true },
|
||||||
|
|||||||
@@ -1,102 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- Sticky per-post translation override (#155). Quiet inline control under the
|
|
||||||
post title: force a skipped legit-foreign title on, or knock a wrongly
|
|
||||||
mis-translated one back to the original. The choice sticks through a
|
|
||||||
Re-translate-all. -->
|
|
||||||
<div class="fc-post-tx">
|
|
||||||
<span class="fc-post-tx__label">Translation:</span>
|
|
||||||
<v-menu :disabled="busy">
|
|
||||||
<template #activator="{ props: menuProps }">
|
|
||||||
<button
|
|
||||||
type="button" class="fc-post-tx__btn" v-bind="menuProps" :disabled="busy"
|
|
||||||
:aria-label="`Translation handling: ${currentLabel}`"
|
|
||||||
>
|
|
||||||
{{ currentLabel }}
|
|
||||||
<v-icon size="14">mdi-menu-down</v-icon>
|
|
||||||
</button>
|
|
||||||
</template>
|
|
||||||
<v-list density="compact" min-width="240" class="fc-post-tx__list">
|
|
||||||
<v-list-item
|
|
||||||
v-for="opt in OPTIONS" :key="opt.value"
|
|
||||||
:active="opt.value === current" @click="choose(opt.value)"
|
|
||||||
>
|
|
||||||
<template #prepend>
|
|
||||||
<v-icon size="small">{{ opt.icon }}</v-icon>
|
|
||||||
</template>
|
|
||||||
<v-list-item-title>{{ opt.label }}</v-list-item-title>
|
|
||||||
<v-list-item-subtitle>{{ opt.help }}</v-list-item-subtitle>
|
|
||||||
</v-list-item>
|
|
||||||
</v-list>
|
|
||||||
</v-menu>
|
|
||||||
<v-progress-circular v-if="busy" indeterminate size="13" width="2" />
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed, ref } from 'vue'
|
|
||||||
|
|
||||||
import { usePostsStore } from '../../stores/posts.js'
|
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
|
|
||||||
const props = defineProps({ post: { type: Object, required: true } })
|
|
||||||
const posts = usePostsStore()
|
|
||||||
const busy = ref(false)
|
|
||||||
|
|
||||||
const OPTIONS = [
|
|
||||||
{ value: 'auto', label: 'Auto', icon: 'mdi-cog-outline',
|
|
||||||
help: 'Translate only when confident enough' },
|
|
||||||
{ value: 'force', label: 'Force translate', icon: 'mdi-translate',
|
|
||||||
help: 'Always translate, even at low confidence' },
|
|
||||||
{ value: 'original', label: 'Keep original', icon: 'mdi-translate-off',
|
|
||||||
help: 'Never translate — keep the original text' },
|
|
||||||
]
|
|
||||||
|
|
||||||
const current = computed(() => props.post.translation_override || 'auto')
|
|
||||||
const currentLabel = computed(
|
|
||||||
() => (OPTIONS.find((o) => o.value === current.value) || OPTIONS[0]).label,
|
|
||||||
)
|
|
||||||
|
|
||||||
async function choose (value) {
|
|
||||||
if (value === current.value || busy.value) return
|
|
||||||
busy.value = true
|
|
||||||
try {
|
|
||||||
const res = await posts.applyTranslationOverride(props.post.id, value)
|
|
||||||
if (res.applied === 'queued') {
|
|
||||||
toast({
|
|
||||||
text: 'Saved — this post will translate on the next sweep (service offline).',
|
|
||||||
type: 'info',
|
|
||||||
})
|
|
||||||
}
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: `Couldn't update translation: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-post-tx {
|
|
||||||
display: inline-flex; align-items: center; gap: 6px;
|
|
||||||
margin: 2px 0 6px;
|
|
||||||
}
|
|
||||||
.fc-post-tx__label {
|
|
||||||
font-size: 0.72rem;
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
}
|
|
||||||
.fc-post-tx__btn {
|
|
||||||
display: inline-flex; align-items: center; gap: 1px;
|
|
||||||
padding: 0; background: none; border: 0; cursor: pointer;
|
|
||||||
font-size: 0.72rem; font-weight: 700;
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
}
|
|
||||||
.fc-post-tx__btn:hover { color: rgb(var(--v-theme-accent)); }
|
|
||||||
.fc-post-tx__btn:focus-visible {
|
|
||||||
outline: 2px solid rgb(var(--v-theme-accent)); outline-offset: 1px;
|
|
||||||
border-radius: 2px;
|
|
||||||
}
|
|
||||||
.fc-post-tx__btn:disabled { cursor: default; opacity: 0.6; }
|
|
||||||
.fc-post-tx__list :deep(.v-list-item-subtitle) {
|
|
||||||
font-size: 0.7rem;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -1,127 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- #3068: attachment reclamation. PostAttachment's FKs are both SET NULL, so
|
|
||||||
a deleted post or artist leaves the row behind; and the store is
|
|
||||||
sha-addressed, so one blob backs many rows and deleting a row never freed
|
|
||||||
its file. Nothing swept either. Preview first, then apply (destructive:
|
|
||||||
unlinks files). -->
|
|
||||||
<MaintenanceTile
|
|
||||||
icon="mdi-paperclip-off"
|
|
||||||
title="Reclaim orphaned attachments"
|
|
||||||
blurb="Remove attachment records belonging to nothing, and the files nothing references."
|
|
||||||
destructive
|
|
||||||
:open="applying || previewing"
|
|
||||||
>
|
|
||||||
<p class="text-body-2 mb-3">
|
|
||||||
Attachment records survive the post and artist they belonged to, and the
|
|
||||||
files behind them are shared between records — so a deleted record never
|
|
||||||
freed its file on its own. This finds records attributed to
|
|
||||||
<strong>neither</strong> a post nor an artist, and files in the attachment
|
|
||||||
store that <strong>no remaining record</strong> points at.
|
|
||||||
<strong>Preview</strong> first; <strong>Apply</strong> deletes those
|
|
||||||
records and unlinks those files. Files written in the last few hours are
|
|
||||||
always left alone, so an in-progress download is never caught mid-write.
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
|
|
||||||
<v-btn
|
|
||||||
color="primary" variant="tonal" rounded="pill"
|
|
||||||
:loading="previewing" :disabled="applying" @click="preview"
|
|
||||||
>
|
|
||||||
<v-icon start>mdi-magnify</v-icon> Preview
|
|
||||||
</v-btn>
|
|
||||||
<v-btn
|
|
||||||
color="error" rounded="pill"
|
|
||||||
:loading="applying"
|
|
||||||
:disabled="previewing || !canApply"
|
|
||||||
@click="confirmOpen = true"
|
|
||||||
>
|
|
||||||
<v-icon start>mdi-paperclip-off</v-icon> Apply
|
|
||||||
</v-btn>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-alert
|
|
||||||
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
|
|
||||||
density="comfortable"
|
|
||||||
>
|
|
||||||
<span v-if="applied">
|
|
||||||
Deleted {{ summary.rows }} orphaned record(s) and unlinked
|
|
||||||
{{ summary.files }} file(s), reclaiming {{ humanBytes(summary.bytes) }}.
|
|
||||||
</span>
|
|
||||||
<span v-else-if="hasWork">
|
|
||||||
{{ summary.rows }} orphaned record(s) and {{ summary.files }}
|
|
||||||
unreferenced file(s) — {{ humanBytes(summary.bytes) }} reclaimable.
|
|
||||||
Click <strong>Apply</strong> to remove them.
|
|
||||||
</span>
|
|
||||||
<span v-else>Nothing to reclaim — every attachment is accounted for.</span>
|
|
||||||
|
|
||||||
<!-- Both of these change what the numbers MEAN, so they are stated
|
|
||||||
whenever they are non-zero rather than hidden in a tooltip. -->
|
|
||||||
<div v-if="summary.files_failed" class="mt-1 text-caption">
|
|
||||||
{{ summary.files_failed }} file(s) could not be read or removed — see
|
|
||||||
the worker log.
|
|
||||||
</div>
|
|
||||||
<div v-if="summary.partial" class="mt-1 text-caption">
|
|
||||||
Stopped early at the time limit; some of the store was not examined.
|
|
||||||
Run it again to continue.
|
|
||||||
</div>
|
|
||||||
</v-alert>
|
|
||||||
|
|
||||||
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance" />
|
|
||||||
|
|
||||||
<v-dialog v-model="confirmOpen" max-width="440">
|
|
||||||
<v-card>
|
|
||||||
<v-card-title>Reclaim orphaned attachments?</v-card-title>
|
|
||||||
<v-card-text class="text-body-2">
|
|
||||||
This permanently deletes
|
|
||||||
<strong>{{ summary?.rows ?? 0 }}</strong> attachment record(s) and
|
|
||||||
unlinks <strong>{{ summary?.files ?? 0 }}</strong> file(s)
|
|
||||||
({{ humanBytes(summary?.bytes) }}). Only files that no remaining
|
|
||||||
record points at are removed, so nothing still attached to a post
|
|
||||||
is affected.
|
|
||||||
</v-card-text>
|
|
||||||
<v-card-actions>
|
|
||||||
<v-spacer />
|
|
||||||
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
|
|
||||||
<v-btn color="error" @click="apply">Reclaim</v-btn>
|
|
||||||
</v-card-actions>
|
|
||||||
</v-card>
|
|
||||||
</v-dialog>
|
|
||||||
</MaintenanceTile>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed, ref } from 'vue'
|
|
||||||
|
|
||||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
|
||||||
import { humanBytes } from '../../utils/bytes.js'
|
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
|
||||||
|
|
||||||
const confirmOpen = ref(false)
|
|
||||||
|
|
||||||
// Walks the whole attachment store, so it can run for minutes on a large
|
|
||||||
// library — the service caps itself at 900s and reports `partial`. 150 polls
|
|
||||||
// × 2s ≈ 5m of foreground waiting; past that the composable hands off to the
|
|
||||||
// task dashboard rather than spinning forever.
|
|
||||||
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
|
|
||||||
endpoint: '/api/admin/maintenance/reclaim-attachments',
|
|
||||||
storageKey: 'fc.maint.reclaimAttachments',
|
|
||||||
appliedToast: 'Orphaned attachments reclaimed',
|
|
||||||
maxPolls: 150,
|
|
||||||
})
|
|
||||||
|
|
||||||
const hasWork = computed(
|
|
||||||
() => !!summary.value && (summary.value.rows > 0 || summary.value.files > 0),
|
|
||||||
)
|
|
||||||
const canApply = computed(() => hasWork.value && !applied.value)
|
|
||||||
const summaryType = computed(() => {
|
|
||||||
if (applied.value) return 'success'
|
|
||||||
return hasWork.value ? 'info' : 'success'
|
|
||||||
})
|
|
||||||
|
|
||||||
// The confirm dialog gates the destructive apply; close it, then run.
|
|
||||||
function apply () {
|
|
||||||
confirmOpen.value = false
|
|
||||||
applyTask()
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
@@ -9,7 +9,7 @@
|
|||||||
<v-card-text>
|
<v-card-text>
|
||||||
<p class="fc-muted text-body-2">
|
<p class="fc-muted text-body-2">
|
||||||
Pushes session cookies from supported platforms
|
Pushes session cookies from supported platforms
|
||||||
(patreon, subscribestar, hentaifoundry, discord, pixiv)
|
(patreon, subscribestar, hentaifoundry, discord, pixiv, deviantart)
|
||||||
into FabledCurator, and lets you add a creator as a source from
|
into FabledCurator, and lets you add a creator as a source from
|
||||||
their page in one click.
|
their page in one click.
|
||||||
</p>
|
</p>
|
||||||
|
|||||||
@@ -15,23 +15,28 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div v-for="p in proposers" :key="p.key" class="fc-proposer">
|
<div v-for="p in proposers" :key="p.key" class="fc-proposer">
|
||||||
<SettingToggleRow
|
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
||||||
v-model="p.on" :loading="busy" :icon="p.icon"
|
<v-icon size="18" :color="p.on ? 'accent' : undefined">{{ p.icon }}</v-icon>
|
||||||
:icon-color="p.on ? 'accent' : null" :label="p.label"
|
<span class="fc-section-h">{{ p.label }}</span>
|
||||||
@change="v => saveToggle(p, v)"
|
<v-switch
|
||||||
/>
|
v-model="p.on" :loading="busy" hide-details density="compact"
|
||||||
|
color="success" class="ml-auto"
|
||||||
|
@update:model-value="v => saveToggle(p, v)"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
|
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
|
||||||
<div class="d-flex flex-wrap mb-4" style="gap: 12px;">
|
<div class="d-flex flex-wrap mb-4" style="gap: 12px;">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model="p.weights" label="Weights" density="compact" hide-details
|
v-model="p.weights" label="Weights" density="compact" hide-details
|
||||||
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
|
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
|
||||||
placeholder="name | URL | hf_repo::file"
|
placeholder="name | URL | hf_repo::file"
|
||||||
@change="saveField({ [`detector_${p.key}_weights`]: p.weights })"
|
@change="save({ [`detector_${p.key}_weights`]: p.weights })"
|
||||||
/>
|
/>
|
||||||
<SettingNumberField
|
<v-text-field
|
||||||
v-model="p.conf" label="Confidence" :min="0" :max="1" :step="0.05"
|
v-model.number="p.conf" label="Confidence" type="number"
|
||||||
max-width="140px" :disabled="busy || !p.on"
|
min="0" max="1" step="0.05" density="compact" hide-details
|
||||||
@change="saveField({ [`detector_${p.key}_conf`]: Number(p.conf) })"
|
style="max-width: 140px;" :disabled="busy || !p.on"
|
||||||
|
@change="save({ [`detector_${p.key}_conf`]: Number(p.conf) })"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -43,12 +48,12 @@
|
|||||||
storage. Dedupe IoU drops near-duplicate crops before embedding.
|
storage. Dedupe IoU drops near-duplicate crops before embedding.
|
||||||
</p>
|
</p>
|
||||||
<div class="d-flex flex-wrap" style="gap: 12px;">
|
<div class="d-flex flex-wrap" style="gap: 12px;">
|
||||||
<SettingNumberField
|
<v-text-field
|
||||||
v-for="c in caps" :key="c.key"
|
v-for="c in caps" :key="c.key"
|
||||||
v-model="c.val" :label="c.label"
|
v-model.number="c.val" :label="c.label" type="number"
|
||||||
:min="c.min" :max="c.max" :step="c.step || 1"
|
:min="c.min" :max="c.max" :step="c.step || 1" density="compact"
|
||||||
max-width="165px" :disabled="busy"
|
hide-details style="max-width: 165px;" :disabled="busy"
|
||||||
@change="saveField({ [c.key]: Number(c.val) })"
|
@change="save({ [c.key]: Number(c.val) })"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -56,16 +61,14 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
|
import { toast } from '../../utils/toast.js'
|
||||||
import { onMounted, ref } from 'vue'
|
import { onMounted, ref } from 'vue'
|
||||||
|
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
import SettingNumberField from '../common/SettingNumberField.vue'
|
|
||||||
import SettingToggleRow from '../common/SettingToggleRow.vue'
|
|
||||||
import { useSettingSave } from '../../composables/useSettingSave.js'
|
|
||||||
import { useMLStore } from '../../stores/ml.js'
|
import { useMLStore } from '../../stores/ml.js'
|
||||||
|
|
||||||
const mlSettings = useMLStore()
|
const mlSettings = useMLStore()
|
||||||
const { busy, save } = useSettingSave(mlSettings.patchSettings)
|
const busy = ref(false)
|
||||||
const proposers = ref([])
|
const proposers = ref([])
|
||||||
const caps = ref([])
|
const caps = ref([])
|
||||||
|
|
||||||
@@ -108,20 +111,31 @@ onMounted(async () => {
|
|||||||
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
|
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
|
||||||
})
|
})
|
||||||
|
|
||||||
// Field @change → persist with a "Saved" confirmation. SettingNumberField has
|
async function save(patch, revert) {
|
||||||
// already clamped numeric values to their [min,max] before this fires.
|
busy.value = true
|
||||||
function saveField(patch) {
|
try {
|
||||||
save(patch, { successMessage: 'Saved' })
|
await mlSettings.patchSettings(patch)
|
||||||
|
toast({ text: 'Saved', type: 'success' })
|
||||||
|
} catch (e) {
|
||||||
|
if (revert) revert()
|
||||||
|
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
||||||
|
} finally {
|
||||||
|
busy.value = false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function saveToggle(p, v) {
|
function saveToggle (p, v) {
|
||||||
// Revert the switch on failure so it never lies about the persisted state.
|
// Revert the switch on failure so it never lies about the persisted state.
|
||||||
const ok = await save({ [`detector_${p.key}_enabled`]: !!v }, { successMessage: 'Saved' })
|
save({ [`detector_${p.key}_enabled`]: !!v }, () => { p.on = !v })
|
||||||
if (!ok) p.on = !v
|
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
|
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
||||||
|
.fc-section-h {
|
||||||
|
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
|
||||||
|
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
|
||||||
|
}
|
||||||
.fc-proposer {
|
.fc-proposer {
|
||||||
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
|
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -112,6 +112,7 @@ async function onCommit() {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
|
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
||||||
.fc-code {
|
.fc-code {
|
||||||
background: rgb(var(--v-theme-surface-light));
|
background: rgb(var(--v-theme-surface-light));
|
||||||
border-radius: 4px; padding: 2px 8px;
|
border-radius: 4px; padding: 2px 8px;
|
||||||
|
|||||||
@@ -42,7 +42,7 @@
|
|||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</v-table>
|
</v-table>
|
||||||
<p v-else class="text-caption mt-3 fc-muted">
|
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
|
||||||
No table statistics yet.
|
No table statistics yet.
|
||||||
</p>
|
</p>
|
||||||
</MaintenanceTile>
|
</MaintenanceTile>
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user