Compare commits
63
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fb2c4d5b80 | ||
|
|
609bc82acc | ||
|
|
7a20c55441 | ||
|
|
0c43fa3eb2 | ||
|
|
cf06c81db9 | ||
|
|
0db38cc111 | ||
|
|
a7e626a67a | ||
|
|
fe48e77821 | ||
|
|
9eb946b21b | ||
|
|
5447a40e97 | ||
|
|
239b1ed8d9 | ||
|
|
cd5444e3ae | ||
|
|
5a0e1bbd03 | ||
|
|
1ac448d881 | ||
|
|
bfc5135f19 | ||
|
|
89155478a8 | ||
|
|
516521e7b0 | ||
|
|
ddf896078c | ||
|
|
2e0f8f8c61 | ||
|
|
2ce467e347 | ||
|
|
39cf81aea6 | ||
|
|
11dd324f89 | ||
|
|
1c6452e10e | ||
|
|
597b91d29b | ||
|
|
f9111c06a7 | ||
|
|
c37a180c3c | ||
|
|
8214afee1e | ||
|
|
306de50f61 | ||
|
|
57e52433d0 | ||
|
|
ec66ea5f83 | ||
|
|
e92570a31e | ||
|
|
a2d1ed935d | ||
|
|
05df51b749 | ||
|
|
099e1e664c | ||
|
|
c87f8a1bb3 | ||
|
|
666b3a2ec8 | ||
|
|
d80a5255ed | ||
|
|
69b5637bd6 | ||
|
|
51749e05db | ||
|
|
50d6c42207 | ||
|
|
67c7ca8603 | ||
|
|
eed42a260a | ||
|
|
61b14e8f65 | ||
|
|
447bf73519 | ||
|
|
6104452d2e | ||
|
|
b59828635e | ||
|
|
fac5ae6ce5 | ||
|
|
af0d39ed52 | ||
|
|
d9a14e890d | ||
|
|
ad2a5fc5fe | ||
|
|
0da0e47784 | ||
|
|
503c8854bc | ||
|
|
571938781a | ||
|
|
0cf3a02797 | ||
|
|
6ab495292a | ||
|
|
c98db303d0 | ||
|
|
2d0fca8729 | ||
|
|
a2858892e9 | ||
|
|
7f5e0603de | ||
|
|
49f6765326 | ||
|
|
b23b19bf58 | ||
|
|
b6c5638eab | ||
|
|
466ee898ab |
+722
-78
@@ -2,10 +2,18 @@ name: Build images
|
|||||||
|
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
|
# `:dev` builds were dropped 2026-05-26 to save a docker build per dev
|
||||||
# merge-to-main, not from the dev branch image. Saves one full docker
|
# push, on the reasoning that "operator tests from `:latest` after
|
||||||
# build per dev push.
|
# merge-to-main". Restored 2026-08-27: that is testing by shipping, and
|
||||||
branches: [main]
|
# family rules 146/147 now name it directly — `main` IS production, and a
|
||||||
|
# channel that can only be refreshed by shipping is not a channel. The
|
||||||
|
# pressure to merge in order to try something does not come from
|
||||||
|
# carelessness; it comes from `:dev` being unable to carry the build.
|
||||||
|
#
|
||||||
|
# All three images build on dev, deliberately: a `:dev` web image paired
|
||||||
|
# with a stale `:dev` ml or agent is a worse trap than no dev channel at
|
||||||
|
# all, since the mismatch only shows up as a runtime failure.
|
||||||
|
branches: [main, dev]
|
||||||
# Tag-push triggers an immutable per-version image build (e.g.
|
# Tag-push triggers an immutable per-version image build (e.g.
|
||||||
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
|
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
|
||||||
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
|
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
|
||||||
@@ -25,28 +33,156 @@ jobs:
|
|||||||
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
|
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
|
||||||
# Result is uploaded as an Actions artifact for build-web to consume.
|
# Result is uploaded as an Actions artifact for build-web to consume.
|
||||||
#
|
#
|
||||||
# Why this lives in build.yml (not a separate workflow): the merge-commit's
|
# Why this lives in build.yml (not a separate workflow): the image a push
|
||||||
# docker image tagged `:latest` MUST carry the XPI. A separate sign workflow
|
# publishes MUST carry the XPI. A separate sign workflow racing build.yml
|
||||||
# racing build.yml leaves `:latest` without the XPI for ~5min (until the
|
# leaves that image without one for ~5min (until the commit-back triggers
|
||||||
# commit-back triggers another build). Inline ordering eliminates the race.
|
# another build). Inline ordering eliminates the race.
|
||||||
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
|
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
|
||||||
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
|
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
|
||||||
# branch to manage). AMO blocks re-signing the same version (returns 409),
|
# branch to manage). AMO blocks re-signing the same version (returns 409),
|
||||||
# so signing is intentionally one-shot per version bump.
|
# so signing is intentionally one-shot per version.
|
||||||
|
#
|
||||||
|
# BOTH branches sign (milestone 271 step 6, 2026-08-27). Not two signatures:
|
||||||
|
# the version is the commit TIME of the newest packaged-extension change, so
|
||||||
|
# dev and main derive the SAME number for the same extension source. A dev
|
||||||
|
# push that changes the extension signs it; the merge to main then finds the
|
||||||
|
# ext-<version> release already there, hits the cache, and bundles the
|
||||||
|
# byte-identical XPI into `:latest` with no second AMO call. One signature
|
||||||
|
# per extension CHANGE, shared by both channels — that is what makes two
|
||||||
|
# channels affordable, and it is why step 4 (derived version) had to land
|
||||||
|
# first. Ungating this while the version was still the hand-set 1.0.11 would
|
||||||
|
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
|
||||||
|
# `:dev` — a dev channel confidently serving old code.
|
||||||
|
#
|
||||||
|
# Tags stay excluded: the tag path deliberately skips signing and polls for
|
||||||
|
# the release instead (see build-web's race note, 2026-05-27).
|
||||||
sign-extension:
|
sign-extension:
|
||||||
if: github.ref == 'refs/heads/main'
|
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full history is load-bearing, not a convenience: the version this
|
||||||
|
# job signs is derived from the commit TIME of the newest packaged
|
||||||
|
# extension change. A depth-1 clone sees one commit and derives a
|
||||||
|
# wrong, too-low value rather than failing (ci-requirements.md).
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
- name: Resolve extension version
|
# The version is DERIVED, not read from the repo (milestone 271 step 4,
|
||||||
|
# cut over 2026-08-27). `packaging.sh version` returns MAJOR.MINOR from
|
||||||
|
# manifest.json plus a patch component that is the commit TIME of the
|
||||||
|
# newest change to a PACKAGED extension file, in minutes since
|
||||||
|
# 2020-01-01 — family rule 149, never a commit count, which orders by
|
||||||
|
# branch rather than by recency.
|
||||||
|
#
|
||||||
|
# The committed "version" in manifest.json / package.json no longer
|
||||||
|
# decides anything: the stamp step below overwrites it in the working
|
||||||
|
# tree before web-ext ever reads it. It is deliberately NOT committed
|
||||||
|
# back — the commit carrying the bump would itself be a change to the
|
||||||
|
# extension and would move the version again. The repo holds the source;
|
||||||
|
# the build derives the label.
|
||||||
|
- name: Derive extension version
|
||||||
id: extver
|
id: extver
|
||||||
run: |
|
run: |
|
||||||
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
set -eu
|
||||||
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
||||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||||
echo "Resolved extension version: $VERSION"
|
echo "Derived extension version: $VERSION"
|
||||||
|
|
||||||
|
# Firefox refuses a downgrade and AMO never releases a burned version,
|
||||||
|
# so a version that moves BACKWARDS is unrecoverable: it strands every
|
||||||
|
# install that already took the higher one. Two ways it could happen —
|
||||||
|
# a checkout without full history (derives too low), or a rewritten
|
||||||
|
# history that drops the newest packaged commit.
|
||||||
|
#
|
||||||
|
# The test is `derived < highest already signed`, strictly. Equality is
|
||||||
|
# the ORDINARY case, not a fault: an unchanged extension derives the same
|
||||||
|
# version it did last build, which is exactly what lets the ext-<version>
|
||||||
|
# cache hit and holds AMO to one call per extension CHANGE. Only moving
|
||||||
|
# backwards is a failure, so this runs on every path — cache hit
|
||||||
|
# included — rather than only before a sign.
|
||||||
|
# --- shadow mode (milestone 313, step 2) -----------------------------
|
||||||
|
# Informational ONLY. Nothing reads this and it must never fail the
|
||||||
|
# build — no `set -e`, and every derivation falls back to UNAVAILABLE.
|
||||||
|
#
|
||||||
|
# What to watch across pushes, because this is what step 3 will trust:
|
||||||
|
# * a push touching only agent/ moves the agent and leaves web and ml
|
||||||
|
# STILL. If web moves, its path set is too wide.
|
||||||
|
# * a push touching only docs moves nothing.
|
||||||
|
# * a push touching the extension moves the extension AND web, since
|
||||||
|
# web bakes in the XPI. If web does not move, its set is too narrow
|
||||||
|
# — the direction that serves stale bytes on a pin.
|
||||||
|
# * dev and main derive the same values for the same source.
|
||||||
|
- name: Shadow — derived artifact version (informational)
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
A=extension
|
||||||
|
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
|
- name: Guard — the derived version must never go backwards
|
||||||
|
env:
|
||||||
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
|
DERIVED: ${{ steps.extver.outputs.version }}
|
||||||
|
run: |
|
||||||
|
python3 - <<'PY'
|
||||||
|
import json, os, sys, urllib.request
|
||||||
|
|
||||||
|
API = ("https://git.fabledsword.com/api/v1/repos/"
|
||||||
|
"bvandeusen/FabledCurator/releases")
|
||||||
|
headers = {"Authorization": "token " + os.environ["TOKEN"]}
|
||||||
|
|
||||||
|
# Paginated rather than first-page-only: ext-* releases share this
|
||||||
|
# list with the v* release tags, so one page would start missing them
|
||||||
|
# as those accumulate. The bound FAILS rather than silently scanning
|
||||||
|
# part of the list and calling the highest it saw the highest there is.
|
||||||
|
tags = []
|
||||||
|
for page in range(1, 21):
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{API}?limit=50&page={page}", headers=headers)
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
batch = json.load(resp)
|
||||||
|
if not batch:
|
||||||
|
break
|
||||||
|
tags += [r.get("tag_name", "") for r in batch]
|
||||||
|
else:
|
||||||
|
sys.exit("guard: >1000 releases — pagination bound reached")
|
||||||
|
|
||||||
|
def parse(v):
|
||||||
|
try:
|
||||||
|
return tuple(int(part) for part in v.split("."))
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
derived_s = os.environ["DERIVED"]
|
||||||
|
derived = parse(derived_s)
|
||||||
|
if derived is None:
|
||||||
|
sys.exit(f"guard: derived version {derived_s!r} is not numeric")
|
||||||
|
|
||||||
|
signed = sorted(
|
||||||
|
(v, t) for t in tags if t.startswith("ext-")
|
||||||
|
for v in [parse(t[4:])] if v
|
||||||
|
)
|
||||||
|
if not signed:
|
||||||
|
print("guard: no ext-* release yet — nothing to go backwards from")
|
||||||
|
raise SystemExit(0)
|
||||||
|
|
||||||
|
hi, hi_tag = signed[-1]
|
||||||
|
print(f"guard: derived={derived_s} highest already signed={hi_tag}")
|
||||||
|
if derived < hi:
|
||||||
|
sys.exit(
|
||||||
|
f"REFUSING TO SIGN: derived {derived_s} is OLDER than the "
|
||||||
|
f"already-signed {hi_tag}. Firefox would reject it as a "
|
||||||
|
f"downgrade, and AMO will not release the burned version. "
|
||||||
|
f"First thing to check: did this job check out with "
|
||||||
|
f"fetch-depth: 0?"
|
||||||
|
)
|
||||||
|
print("guard: ok")
|
||||||
|
PY
|
||||||
|
|
||||||
- name: Check Forgejo release-asset cache
|
- name: Check Forgejo release-asset cache
|
||||||
id: cache
|
id: cache
|
||||||
@@ -84,6 +220,29 @@ jobs:
|
|||||||
# removal — sign-extension's job is just to ensure the cache
|
# removal — sign-extension's job is just to ensure the cache
|
||||||
# exists on Forgejo; the build-web side reads it independently).
|
# exists on Forgejo; the build-web side reads it independently).
|
||||||
|
|
||||||
|
# web-ext signs whatever manifest.json says, so the derived value has to
|
||||||
|
# reach the tree before signing. package.json is written too: the two are
|
||||||
|
# required to agree (ci.yml's guard), and a local `npm run build` reads
|
||||||
|
# it. Working tree only — never committed, per the note on the derive
|
||||||
|
# step.
|
||||||
|
- name: Stamp the derived version into manifest.json + package.json
|
||||||
|
env:
|
||||||
|
DERIVED: ${{ steps.extver.outputs.version }}
|
||||||
|
run: |
|
||||||
|
python3 - <<'PY'
|
||||||
|
import json, os
|
||||||
|
|
||||||
|
version = os.environ["DERIVED"]
|
||||||
|
for path in ("extension/manifest.json", "extension/package.json"):
|
||||||
|
with open(path) as fh:
|
||||||
|
doc = json.load(fh)
|
||||||
|
doc["version"] = version
|
||||||
|
with open(path, "w") as fh:
|
||||||
|
json.dump(doc, fh, indent=2)
|
||||||
|
fh.write("\n")
|
||||||
|
print(f"{path}: version -> {version}")
|
||||||
|
PY
|
||||||
|
|
||||||
- name: Sign via AMO (cache miss)
|
- name: Sign via AMO (cache miss)
|
||||||
if: steps.cache.outputs.cached != 'true'
|
if: steps.cache.outputs.cached != 'true'
|
||||||
run: |
|
run: |
|
||||||
@@ -110,6 +269,11 @@ jobs:
|
|||||||
# created it so an upload failure below can roll back (don't
|
# created it so an upload failure below can roll back (don't
|
||||||
# leave an empty release tombstone that the next run's
|
# leave an empty release tombstone that the next run's
|
||||||
# cache-check mistakes for a partial-failure state).
|
# cache-check mistakes for a partial-failure state).
|
||||||
|
#
|
||||||
|
# target_commitish is the signing commit, not a branch name: since
|
||||||
|
# step 6 either branch can create this release, and hard-coding
|
||||||
|
# `main` would tag a dev-signed XPI against a main commit that may
|
||||||
|
# not even contain the extension source it was built from.
|
||||||
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
||||||
-H "Authorization: token $TOKEN" \
|
-H "Authorization: token $TOKEN" \
|
||||||
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
||||||
@@ -117,7 +281,7 @@ jobs:
|
|||||||
CREATED_BY_US=false
|
CREATED_BY_US=false
|
||||||
else
|
else
|
||||||
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
||||||
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"main\"}" \
|
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"$GITHUB_SHA\"}" \
|
||||||
-o release.json \
|
-o release.json \
|
||||||
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
|
||||||
CREATED_BY_US=true
|
CREATED_BY_US=true
|
||||||
@@ -160,19 +324,208 @@ jobs:
|
|||||||
|
|
||||||
build-web:
|
build-web:
|
||||||
needs: [sign-extension]
|
needs: [sign-extension]
|
||||||
# sign-extension is main-only; on dev it's skipped, build-web still runs.
|
# sign-extension runs on main and dev, and is skipped on a tag push (which
|
||||||
|
# polls for the release instead). Either is fine to build on; a FAILED sign
|
||||||
|
# is not — this condition lets success and skipped through, so a failure
|
||||||
|
# skips build-web rather than shipping an image without the XPI.
|
||||||
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
|
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full history: this job RE-DERIVES the extension version rather than
|
||||||
|
# being handed it, and a depth-1 clone derives a wrong, too-low value
|
||||||
|
# rather than failing — which would 404 the download of a release
|
||||||
|
# that exists perfectly well under its real name.
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
- name: Download signed XPI from Forgejo release asset (main + tags)
|
# --- derived values, one line (milestone 313) ------------------------
|
||||||
# Fires on main-push AND on tag-push. Tag-push builds re-package the
|
# These stopped being shadow output at step 3: `tag` is published on
|
||||||
# same source code as the preceding main-push build but with an
|
# main and `revision` decides whether the build below runs at all. This
|
||||||
# immutable version tag — they need the XPI too, otherwise the
|
# step prints all three anyway, because the load-bearing steps each
|
||||||
# versioned image ships without the signed extension.
|
# print only the one they use, and on dev the date tag is computed
|
||||||
|
# nowhere else. When a build is skipped or a pin looks wrong, this is
|
||||||
|
# the line that says what the commit derived.
|
||||||
|
#
|
||||||
|
# Still diagnostic, so it still must not fail the build — no `set -e`,
|
||||||
|
# and every derivation falls back to UNAVAILABLE. A broken echo must
|
||||||
|
# never be the reason an image does not ship.
|
||||||
|
#
|
||||||
|
# What it should say:
|
||||||
|
# * a push touching only agent/ moves the agent and leaves web and ml
|
||||||
|
# STILL. If web moves, its path set is too wide.
|
||||||
|
# * a push touching only docs moves nothing.
|
||||||
|
# * a push touching the extension moves the extension AND web, since
|
||||||
|
# web bakes in the XPI. If web does not move, its set is too narrow
|
||||||
|
# — the direction that serves stale bytes on a pin.
|
||||||
|
# * dev and main derive the same values for the same source.
|
||||||
|
- name: Report the derived artifact version
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
A=web
|
||||||
|
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
|
- name: Determine tag
|
||||||
|
id: tag
|
||||||
|
run: |
|
||||||
|
# Three trigger shapes:
|
||||||
|
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
|
||||||
|
# plus `.N` when the day already carries a tag —
|
||||||
|
# family rule 148, amended 2026-08-24 after a
|
||||||
|
# same-day tag was retargeted and a release
|
||||||
|
# deleted to make room, note 2813).
|
||||||
|
# Publish ONLY the immutable version tag;
|
||||||
|
# don't touch :latest (the main-push build
|
||||||
|
# for the merge commit already did that).
|
||||||
|
# refs/heads/main → push to main: publish :main + :latest
|
||||||
|
# (floating) AND :c-<short_sha> (immutable
|
||||||
|
# per-commit rollback substrate, per family
|
||||||
|
# release-posture rule "Tags are milestones,
|
||||||
|
# not gates — commit-SHA images are the
|
||||||
|
# rollback unit"). Rollback to any commit
|
||||||
|
# becomes `docker pull …:c-<sha>` without a
|
||||||
|
# release ceremony.
|
||||||
|
# refs/heads/dev → push to dev: publish :dev, the rolling test
|
||||||
|
# channel (family rule 146). Rolling means it may
|
||||||
|
# carry newer contents than the :c-<sha> of the
|
||||||
|
# same commit; it never writes :c-<sha> itself,
|
||||||
|
# because that is the rollback unit (rule 145).
|
||||||
|
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
||||||
|
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
||||||
|
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
||||||
|
# main-push build failed at this step.
|
||||||
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
|
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
||||||
|
# THIS artifact's shipped files last changed in. Day precision is
|
||||||
|
# deliberate — same-day work is not something worth pinning, so a
|
||||||
|
# second main build the same day replaces the first rather than
|
||||||
|
# accumulating a tag nobody would roll back to.
|
||||||
|
#
|
||||||
|
# Derived per artifact, so an image whose files did not change keeps
|
||||||
|
# the tag it already had: the agent reads 2026.7.17 today while web
|
||||||
|
# reads 2026.8.27 — and the reuse step below turns that into a
|
||||||
|
# skipped build rather than a rebuild of bytes that already exist.
|
||||||
|
# `channel` is baked into the image as FC_CHANNEL and reported by
|
||||||
|
# /api/extension/manifest (milestone 271 step 7). A tag-push counts as
|
||||||
|
# `main`: a vYY.MM.DD tag is cut from main, so that image is a
|
||||||
|
# main-channel artifact wearing an immutable name.
|
||||||
|
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
||||||
|
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
||||||
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
|
CALVER=$(sh scripts/artifacts.sh tag web)
|
||||||
|
# Guarded, and computed only on this path. There is no `set -e` in
|
||||||
|
# this step, so a failed derivation would otherwise leave CALVER
|
||||||
|
# empty and publish the tag `fabledcurator:` — an invalid
|
||||||
|
# name, from a green step. An empty pin must never reach the
|
||||||
|
# registry.
|
||||||
|
if [ -z "$CALVER" ]; then
|
||||||
|
echo "ERROR: could not derive a web version tag" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator:${CALVER}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=dev" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# A shell step, not docker/login-action@v3, because the action's shared
|
||||||
|
# cache races itself (#3118). act_runner caches a remote action under one
|
||||||
|
# /root/.cache/act/<hash> per runner, and build-web, build-ml and
|
||||||
|
# build-agent all start in the same second and all want this same action.
|
||||||
|
# One job re-clones the directory — which empties and repopulates it —
|
||||||
|
# while another is walking it to copy into its container, and the walker
|
||||||
|
# lstat()s a file that has just vanished. It failed twice on 2026-08-27,
|
||||||
|
# naming a DIFFERENT missing file each time (`eslint.config.mjs`, then
|
||||||
|
# `jest.config.ts`), which is what rules out a corrupt cache and points at
|
||||||
|
# a race. The loser dies with MODULE_NOT_FOUND on dist/index.js before the
|
||||||
|
# action runs at all, so the secret is never even reached.
|
||||||
|
#
|
||||||
|
# Nothing is lost by dropping it: logging in is one command, the docker
|
||||||
|
# CLI is already in the CI image (ci-requirements.md), and the same
|
||||||
|
# reasoning as family rule 5 applies — a marketplace action buys nothing
|
||||||
|
# when the tool is baked into the image the workflow already selected.
|
||||||
|
#
|
||||||
|
# Password on stdin, never as an argument: an argument lands in the
|
||||||
|
# process table and draws docker's own deprecation warning.
|
||||||
|
- name: Login to Forgejo registry
|
||||||
|
env:
|
||||||
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
|
ACTOR: ${{ github.actor }}
|
||||||
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
|
|
||||||
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
|
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
||||||
|
# commit its shipped files last changed in, plus the channel for images
|
||||||
|
# that bake one in. If the registry already carries it, the bytes this
|
||||||
|
# job would produce are already published and the build is pure waste:
|
||||||
|
# the channel and date tags get repointed at the existing manifest
|
||||||
|
# instead, registry-side, in seconds.
|
||||||
|
#
|
||||||
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
|
#
|
||||||
|
# The failure direction is deliberate. An inspect that errors for ANY
|
||||||
|
# reason — network, auth, a registry hiccup — reads as a miss and the
|
||||||
|
# build runs. Only a genuine 200 skips one, so there is no path here
|
||||||
|
# that skips a build that was actually needed; the worst case is paying
|
||||||
|
# for a build we could have avoided.
|
||||||
|
#
|
||||||
|
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
|
||||||
|
# whose source stops moving stops picking up base-image updates under
|
||||||
|
# its pinned tag. That is what a pin MEANS — a date tag has to keep
|
||||||
|
# serving the bytes it served (fabledcurator:2026.7.17 still
|
||||||
|
# resolves to July's image), or it is not a pin — and family rule
|
||||||
|
# 145 already says where the refresh goes instead: a rebuild with
|
||||||
|
# different contents publishes only the MOVING tag, never the immutable
|
||||||
|
# one. A scheduled channel-only refresh is tracked separately (#3154);
|
||||||
|
# it does not belong in the push path.
|
||||||
|
- name: Is this content already published?
|
||||||
|
id: reuse
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||||
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
ID=$(sh scripts/artifacts.sh identity web "$CHANNEL")
|
||||||
|
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# A tag-push builds a revision that main already published, so it
|
||||||
|
# must NOT claim the identity: image configs are not bit-reproducible
|
||||||
|
# (embedded timestamps), so re-pushing r-<rev> would point an
|
||||||
|
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
||||||
|
# publishes only its own :v… label and otherwise reuses.
|
||||||
|
if [ "$IS_TAG_PUSH" = "true" ]; then
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
||||||
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
||||||
|
else
|
||||||
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is not published — building"
|
||||||
|
fi
|
||||||
|
|
||||||
|
- name: Download signed XPI from Forgejo release asset
|
||||||
|
# Fires on every trigger shape. dev and main each bundle the XPI their
|
||||||
|
# own sign-extension just published — that is the whole point of the
|
||||||
|
# channel work (milestone 271 step 6): the dev image carries the
|
||||||
|
# extension being developed, rather than requiring a merge to try it.
|
||||||
|
# Tag-push builds re-package the same source as the preceding main-push
|
||||||
|
# build but with an immutable version tag — they need the XPI too,
|
||||||
|
# otherwise the versioned image ships without the signed extension.
|
||||||
#
|
#
|
||||||
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
|
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
|
||||||
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
|
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
|
||||||
@@ -184,12 +537,21 @@ jobs:
|
|||||||
# for up to 10min total) before giving up. Main-push's signing
|
# for up to 10min total) before giving up. Main-push's signing
|
||||||
# eventually wins and tag-push picks the release up on a later
|
# eventually wins and tag-push picks the release up on a later
|
||||||
# iteration.
|
# iteration.
|
||||||
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
|
# Gated on the reuse miss as well: if the image is already published it
|
||||||
|
# already contains its XPI, so this would download (and on a tag-push,
|
||||||
|
# poll up to 10 minutes for) a file nothing then reads.
|
||||||
|
if: steps.reuse.outputs.hit != 'true' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/'))
|
||||||
env:
|
env:
|
||||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
run: |
|
run: |
|
||||||
set -eux
|
set -eux
|
||||||
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
# Re-derived, not read from the repo: sign-extension published
|
||||||
|
# ext-<derived>, and the committed version has been inert since
|
||||||
|
# milestone 271 step 4. Both jobs run `packaging.sh version` over the
|
||||||
|
# same commit, so they agree by construction — and if they ever
|
||||||
|
# didn't, this download 404s and the build fails loudly instead of
|
||||||
|
# shipping a stale XPI.
|
||||||
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
||||||
# Poll for the ext-<version> release. main-push's sign-extension
|
# Poll for the ext-<version> release. main-push's sign-extension
|
||||||
# step (AMO round-trip, 1-5min) needs to finish + upload before
|
# step (AMO round-trip, 1-5min) needs to finish + upload before
|
||||||
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
|
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
|
||||||
@@ -238,54 +600,46 @@ jobs:
|
|||||||
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
|
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
|
||||||
ls -la frontend/public/extension/
|
ls -la frontend/public/extension/
|
||||||
|
|
||||||
- name: Determine tag
|
|
||||||
id: tag
|
|
||||||
run: |
|
|
||||||
# Three trigger shapes:
|
|
||||||
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
|
|
||||||
# no `.N` per family release-posture rule).
|
|
||||||
# Publish ONLY the immutable version tag;
|
|
||||||
# don't touch :latest (the main-push build
|
|
||||||
# for the merge commit already did that).
|
|
||||||
# refs/heads/main → push to main: publish :main + :latest
|
|
||||||
# (floating) AND :c-<short_sha> (immutable
|
|
||||||
# per-commit rollback substrate, per family
|
|
||||||
# release-posture rule "Tags are milestones,
|
|
||||||
# not gates — commit-SHA images are the
|
|
||||||
# rollback unit"). Rollback to any commit
|
|
||||||
# becomes `docker pull …:c-<sha>` without a
|
|
||||||
# release ceremony.
|
|
||||||
# anything else → safety net; shouldn't fire given the `on:`
|
|
||||||
# config above. Tag :dev to surface the
|
|
||||||
# unexpected run in the registry.
|
|
||||||
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
|
||||||
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
|
||||||
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
|
||||||
# main-push build failed at this step.
|
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
|
||||||
else
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Login to Forgejo registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: git.fabledsword.com
|
|
||||||
username: ${{ github.actor }}
|
|
||||||
password: ${{ secrets.RELEASE_TOKEN }}
|
|
||||||
|
|
||||||
- name: Build and push web image
|
- name: Build and push web image
|
||||||
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: .
|
context: .
|
||||||
file: Dockerfile
|
file: Dockerfile
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.tag.outputs.tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
# Only the web image carries a channel: it is the one that serves
|
||||||
|
# /api/extension/manifest. The ml and agent images have nothing to
|
||||||
|
# report it to.
|
||||||
|
build-args: |
|
||||||
|
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
||||||
|
|
||||||
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
|
# identity tag holds, so :latest and the date pin are byte-identical to
|
||||||
|
# what was published rather than a lookalike rebuild.
|
||||||
|
#
|
||||||
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
|
# leaving :dev or :latest pointing at something older than the commit
|
||||||
|
# that was just pushed.
|
||||||
|
- name: Repoint the tags at the published image (reuse)
|
||||||
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||||
|
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
run: |
|
||||||
|
set -euf
|
||||||
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
|
ARGS=""
|
||||||
|
IFS=,
|
||||||
|
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
||||||
|
unset IFS
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
||||||
|
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
||||||
|
|
||||||
build-ml:
|
build-ml:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
@@ -293,6 +647,42 @@ jobs:
|
|||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full history: this job derives its artifact's version from the
|
||||||
|
# commit its shipped files last changed in (milestone 313). A
|
||||||
|
# depth-1 clone cannot see that commit — it either derives a wrong,
|
||||||
|
# too-low value or finds nothing at all, and neither is a failure
|
||||||
|
# the build would otherwise notice.
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
# --- derived values, one line (milestone 313) ------------------------
|
||||||
|
# These stopped being shadow output at step 3: `tag` is published on
|
||||||
|
# main and `revision` decides whether the build below runs at all. This
|
||||||
|
# step prints all three anyway, because the load-bearing steps each
|
||||||
|
# print only the one they use, and on dev the date tag is computed
|
||||||
|
# nowhere else. When a build is skipped or a pin looks wrong, this is
|
||||||
|
# the line that says what the commit derived.
|
||||||
|
#
|
||||||
|
# Still diagnostic, so it still must not fail the build — no `set -e`,
|
||||||
|
# and every derivation falls back to UNAVAILABLE. A broken echo must
|
||||||
|
# never be the reason an image does not ship.
|
||||||
|
#
|
||||||
|
# What it should say:
|
||||||
|
# * a push touching only agent/ moves the agent and leaves web and ml
|
||||||
|
# STILL. If web moves, its path set is too wide.
|
||||||
|
# * a push touching only docs moves nothing.
|
||||||
|
# * a push touching the extension moves the extension AND web, since
|
||||||
|
# web bakes in the XPI. If web does not move, its set is too narrow
|
||||||
|
# — the direction that serves stale bytes on a pin.
|
||||||
|
# * dev and main derive the same values for the same source.
|
||||||
|
- name: Report the derived artifact version
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
A=ml
|
||||||
|
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Determine tag
|
- name: Determine tag
|
||||||
id: tag
|
id: tag
|
||||||
@@ -306,29 +696,138 @@ jobs:
|
|||||||
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
||||||
# main-push build failed at this step.
|
# main-push build failed at this step.
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
|
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
||||||
|
# THIS artifact's shipped files last changed in. Day precision is
|
||||||
|
# deliberate — same-day work is not something worth pinning, so a
|
||||||
|
# second main build the same day replaces the first rather than
|
||||||
|
# accumulating a tag nobody would roll back to.
|
||||||
|
#
|
||||||
|
# Derived per artifact, so an image whose files did not change keeps
|
||||||
|
# the tag it already had: the agent reads 2026.7.17 today while web
|
||||||
|
# reads 2026.8.27 — and the reuse step below turns that into a
|
||||||
|
# skipped build rather than a rebuild of bytes that already exist.
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
CALVER=$(sh scripts/artifacts.sh tag ml)
|
||||||
|
# Guarded, and computed only on this path. There is no `set -e` in
|
||||||
|
# this step, so a failed derivation would otherwise leave CALVER
|
||||||
|
# empty and publish the tag `fabledcurator-ml:` — an invalid
|
||||||
|
# name, from a green step. An empty pin must never reach the
|
||||||
|
# registry.
|
||||||
|
if [ -z "$CALVER" ]; then
|
||||||
|
echo "ERROR: could not derive a ml version tag" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-ml:${CALVER}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
else
|
else
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=dev" >> "$GITHUB_OUTPUT"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Shell step rather than docker/login-action — see build-web's note on
|
||||||
|
# the shared action-cache race (#3118).
|
||||||
- name: Login to Forgejo registry
|
- name: Login to Forgejo registry
|
||||||
uses: docker/login-action@v3
|
env:
|
||||||
with:
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
registry: git.fabledsword.com
|
ACTOR: ${{ github.actor }}
|
||||||
username: ${{ github.actor }}
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
password: ${{ secrets.RELEASE_TOKEN }}
|
|
||||||
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
|
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
||||||
|
# commit its shipped files last changed in, plus the channel for images
|
||||||
|
# that bake one in. If the registry already carries it, the bytes this
|
||||||
|
# job would produce are already published and the build is pure waste:
|
||||||
|
# the channel and date tags get repointed at the existing manifest
|
||||||
|
# instead, registry-side, in seconds.
|
||||||
|
#
|
||||||
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
|
#
|
||||||
|
# The failure direction is deliberate. An inspect that errors for ANY
|
||||||
|
# reason — network, auth, a registry hiccup — reads as a miss and the
|
||||||
|
# build runs. Only a genuine 200 skips one, so there is no path here
|
||||||
|
# that skips a build that was actually needed; the worst case is paying
|
||||||
|
# for a build we could have avoided.
|
||||||
|
#
|
||||||
|
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
|
||||||
|
# whose source stops moving stops picking up base-image updates under
|
||||||
|
# its pinned tag. That is what a pin MEANS — a date tag has to keep
|
||||||
|
# serving the bytes it served (fabledcurator-ml:2026.7.17 still
|
||||||
|
# resolves to July's image), or it is not a pin — and family rule
|
||||||
|
# 145 already says where the refresh goes instead: a rebuild with
|
||||||
|
# different contents publishes only the MOVING tag, never the immutable
|
||||||
|
# one. A scheduled channel-only refresh is tracked separately (#3154);
|
||||||
|
# it does not belong in the push path.
|
||||||
|
- name: Is this content already published?
|
||||||
|
id: reuse
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||||
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
ID=$(sh scripts/artifacts.sh identity ml "$CHANNEL")
|
||||||
|
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# A tag-push builds a revision that main already published, so it
|
||||||
|
# must NOT claim the identity: image configs are not bit-reproducible
|
||||||
|
# (embedded timestamps), so re-pushing r-<rev> would point an
|
||||||
|
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
||||||
|
# publishes only its own :v… label and otherwise reuses.
|
||||||
|
if [ "$IS_TAG_PUSH" = "true" ]; then
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
||||||
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
||||||
|
else
|
||||||
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is not published — building"
|
||||||
|
fi
|
||||||
|
|
||||||
- name: Build and push ml image
|
- name: Build and push ml image
|
||||||
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: .
|
context: .
|
||||||
file: Dockerfile.ml
|
file: Dockerfile.ml
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.tag.outputs.tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
|
||||||
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
|
# identity tag holds, so :latest and the date pin are byte-identical to
|
||||||
|
# what was published rather than a lookalike rebuild.
|
||||||
|
#
|
||||||
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
|
# leaving :dev or :latest pointing at something older than the commit
|
||||||
|
# that was just pushed.
|
||||||
|
- name: Repoint the tags at the published image (reuse)
|
||||||
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||||
|
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
run: |
|
||||||
|
set -euf
|
||||||
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
|
ARGS=""
|
||||||
|
IFS=,
|
||||||
|
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
||||||
|
unset IFS
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
||||||
|
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
||||||
|
|
||||||
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
||||||
# the GPU machine instead of building locally. Independent of web/ml (its own
|
# the GPU machine instead of building locally. Independent of web/ml (its own
|
||||||
@@ -339,31 +838,176 @@ jobs:
|
|||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full history: this job derives its artifact's version from the
|
||||||
|
# commit its shipped files last changed in (milestone 313). A
|
||||||
|
# depth-1 clone cannot see that commit — it either derives a wrong,
|
||||||
|
# too-low value or finds nothing at all, and neither is a failure
|
||||||
|
# the build would otherwise notice.
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
# --- derived values, one line (milestone 313) ------------------------
|
||||||
|
# These stopped being shadow output at step 3: `tag` is published on
|
||||||
|
# main and `revision` decides whether the build below runs at all. This
|
||||||
|
# step prints all three anyway, because the load-bearing steps each
|
||||||
|
# print only the one they use, and on dev the date tag is computed
|
||||||
|
# nowhere else. When a build is skipped or a pin looks wrong, this is
|
||||||
|
# the line that says what the commit derived.
|
||||||
|
#
|
||||||
|
# Still diagnostic, so it still must not fail the build — no `set -e`,
|
||||||
|
# and every derivation falls back to UNAVAILABLE. A broken echo must
|
||||||
|
# never be the reason an image does not ship.
|
||||||
|
#
|
||||||
|
# What it should say:
|
||||||
|
# * a push touching only agent/ moves the agent and leaves web and ml
|
||||||
|
# STILL. If web moves, its path set is too wide.
|
||||||
|
# * a push touching only docs moves nothing.
|
||||||
|
# * a push touching the extension moves the extension AND web, since
|
||||||
|
# web bakes in the XPI. If web does not move, its set is too narrow
|
||||||
|
# — the direction that serves stale bytes on a pin.
|
||||||
|
# * dev and main derive the same values for the same source.
|
||||||
|
- name: Report the derived artifact version
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
A=agent
|
||||||
|
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
|
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Determine tag
|
- name: Determine tag
|
||||||
id: tag
|
id: tag
|
||||||
run: |
|
run: |
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
|
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
||||||
|
# THIS artifact's shipped files last changed in. Day precision is
|
||||||
|
# deliberate — same-day work is not something worth pinning, so a
|
||||||
|
# second main build the same day replaces the first rather than
|
||||||
|
# accumulating a tag nobody would roll back to.
|
||||||
|
#
|
||||||
|
# Derived per artifact, so an image whose files did not change keeps
|
||||||
|
# the tag it already had: the agent reads 2026.7.17 today while web
|
||||||
|
# reads 2026.8.27 — and the reuse step below turns that into a
|
||||||
|
# skipped build rather than a rebuild of bytes that already exist.
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
CALVER=$(sh scripts/artifacts.sh tag agent)
|
||||||
|
# Guarded, and computed only on this path. There is no `set -e` in
|
||||||
|
# this step, so a failed derivation would otherwise leave CALVER
|
||||||
|
# empty and publish the tag `fabledcurator-agent:` — an invalid
|
||||||
|
# name, from a green step. An empty pin must never reach the
|
||||||
|
# registry.
|
||||||
|
if [ -z "$CALVER" ]; then
|
||||||
|
echo "ERROR: could not derive a agent version tag" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-agent:${CALVER}" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
else
|
else
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "channel=dev" >> "$GITHUB_OUTPUT"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Shell step rather than docker/login-action — see build-web's note on
|
||||||
|
# the shared action-cache race (#3118).
|
||||||
- name: Login to Forgejo registry
|
- name: Login to Forgejo registry
|
||||||
uses: docker/login-action@v3
|
env:
|
||||||
with:
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
registry: git.fabledsword.com
|
ACTOR: ${{ github.actor }}
|
||||||
username: ${{ github.actor }}
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
password: ${{ secrets.RELEASE_TOKEN }}
|
|
||||||
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
|
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
||||||
|
# commit its shipped files last changed in, plus the channel for images
|
||||||
|
# that bake one in. If the registry already carries it, the bytes this
|
||||||
|
# job would produce are already published and the build is pure waste:
|
||||||
|
# the channel and date tags get repointed at the existing manifest
|
||||||
|
# instead, registry-side, in seconds.
|
||||||
|
#
|
||||||
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
|
#
|
||||||
|
# The failure direction is deliberate. An inspect that errors for ANY
|
||||||
|
# reason — network, auth, a registry hiccup — reads as a miss and the
|
||||||
|
# build runs. Only a genuine 200 skips one, so there is no path here
|
||||||
|
# that skips a build that was actually needed; the worst case is paying
|
||||||
|
# for a build we could have avoided.
|
||||||
|
#
|
||||||
|
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
|
||||||
|
# whose source stops moving stops picking up base-image updates under
|
||||||
|
# its pinned tag. That is what a pin MEANS — a date tag has to keep
|
||||||
|
# serving the bytes it served (fabledcurator-agent:2026.7.17 still
|
||||||
|
# resolves to July's image), or it is not a pin — and family rule
|
||||||
|
# 145 already says where the refresh goes instead: a rebuild with
|
||||||
|
# different contents publishes only the MOVING tag, never the immutable
|
||||||
|
# one. A scheduled channel-only refresh is tracked separately (#3154);
|
||||||
|
# it does not belong in the push path.
|
||||||
|
- name: Is this content already published?
|
||||||
|
id: reuse
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||||
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
ID=$(sh scripts/artifacts.sh identity agent "$CHANNEL")
|
||||||
|
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# A tag-push builds a revision that main already published, so it
|
||||||
|
# must NOT claim the identity: image configs are not bit-reproducible
|
||||||
|
# (embedded timestamps), so re-pushing r-<rev> would point an
|
||||||
|
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
||||||
|
# publishes only its own :v… label and otherwise reuses.
|
||||||
|
if [ "$IS_TAG_PUSH" = "true" ]; then
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
||||||
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
||||||
|
else
|
||||||
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "reuse: $IMAGE:$ID is not published — building"
|
||||||
|
fi
|
||||||
|
|
||||||
- name: Build and push agent image
|
- name: Build and push agent image
|
||||||
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: agent
|
context: agent
|
||||||
file: agent/Dockerfile
|
file: agent/Dockerfile
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.tag.outputs.tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
|
||||||
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
|
# identity tag holds, so :latest and the date pin are byte-identical to
|
||||||
|
# what was published rather than a lookalike rebuild.
|
||||||
|
#
|
||||||
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
|
# leaving :dev or :latest pointing at something older than the commit
|
||||||
|
# that was just pushed.
|
||||||
|
- name: Repoint the tags at the published image (reuse)
|
||||||
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||||
|
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
||||||
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
|
run: |
|
||||||
|
set -euf
|
||||||
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
|
ARGS=""
|
||||||
|
IFS=,
|
||||||
|
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
||||||
|
unset IFS
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
||||||
|
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ name: CI
|
|||||||
|
|
||||||
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
|
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
|
||||||
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
|
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
|
||||||
|
# - extension-version: the derived version resolves and MAJOR.MINOR agrees.
|
||||||
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
|
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
|
||||||
# - frontend-build: vitest unit + vite build.
|
# - frontend-build: vitest unit + vite build.
|
||||||
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
|
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
|
||||||
@@ -9,10 +10,15 @@ name: CI
|
|||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
branches: [dev, main]
|
branches: [dev, main]
|
||||||
# pull_request trigger intentionally absent — with branches: [dev, main]
|
# Renovate opens PRs from `renovate/*` branches into `dev`. Those branches
|
||||||
# above, every PR commit already fires CI via the push event on dev. Adding
|
# never push to dev/main, so the push trigger above gives them NO pre-merge
|
||||||
# pull_request would duplicate runs on dev→main PRs. FC has no fork PRs
|
# CI — a bump could only be validated after it was already merged. This
|
||||||
# (single-operator Forgejo repo) so push coverage is complete.
|
# pull_request trigger (base `dev` only) validates Renovate PRs before merge.
|
||||||
|
# It deliberately does NOT fire on dev→main PRs (base `main`), which still
|
||||||
|
# rely on the dev push run — so no duplicate runs. FC has no fork PRs
|
||||||
|
# (single-operator Forgejo repo), so secrets-on-PR is not a concern.
|
||||||
|
pull_request:
|
||||||
|
branches: [dev]
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
|
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
|
||||||
@@ -36,6 +42,68 @@ jobs:
|
|||||||
# catching syntax errors before the image build.
|
# catching syntax errors before the image build.
|
||||||
run: python -m compileall -q agent/fc_agent
|
run: python -m compileall -q agent/fc_agent
|
||||||
|
|
||||||
|
# The extension version is DERIVED, not hand-maintained (milestone 271 step
|
||||||
|
# 4): build.yml computes it from the commit TIME of the newest packaged
|
||||||
|
# extension change and stamps it into manifest.json / package.json at build
|
||||||
|
# time. The guard that used to live here — "packaged files changed but nobody
|
||||||
|
# bumped the version" — was therefore checking a fact that had stopped
|
||||||
|
# existing. Worse than useless: it would have failed this lane on every real
|
||||||
|
# extension change, demanding a bump that decides nothing. Retired 2026-08-27
|
||||||
|
# rather than left running beside the new mechanism (rule 22).
|
||||||
|
#
|
||||||
|
# Two things are still worth asserting, and this is the only lane that can:
|
||||||
|
# the extension.yml suite runs on node:24-slim, which is exactly why
|
||||||
|
# version.spec.js sticks to packaging.sh's git-free subcommands.
|
||||||
|
# 1. the derivation actually resolves on this commit
|
||||||
|
# 2. MAJOR.MINOR agrees between the two files — the one part still hand-set,
|
||||||
|
# and packaging.sh reads it from manifest.json ALONE, so a divergence
|
||||||
|
# ships a version package.json disagrees with
|
||||||
|
#
|
||||||
|
# Deliberately NOT checked here: that the derived value beats what has already
|
||||||
|
# been signed. That guard belongs in build.yml, where it compares against the
|
||||||
|
# real ext-* releases. Comparing against origin/main here would be wrong —
|
||||||
|
# dev legitimately derives a LOWER value whenever main is ahead on the
|
||||||
|
# extension, and a lane that fails for being behind is a lane people learn to
|
||||||
|
# ignore.
|
||||||
|
extension-version:
|
||||||
|
runs-on: python-ci
|
||||||
|
container:
|
||||||
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# The derivation needs real history: a depth-1 clone sees one commit
|
||||||
|
# and produces a wrong, too-low value RATHER THAN FAILING. Checking
|
||||||
|
# that here is half the point of the lane.
|
||||||
|
fetch-depth: 0
|
||||||
|
- name: Extension version derives cleanly
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
# busybox sh on the act_runner — no bashisms (family rule).
|
||||||
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
||||||
|
echo "derived: $VERSION"
|
||||||
|
# The shape AMO accepts, and the shape build.yml will stamp.
|
||||||
|
if ! echo "$VERSION" | grep -qE '^[0-9]+(\.[0-9]+)*$'; then
|
||||||
|
echo "ERROR: derived version '$VERSION' is not plain dotted-numeric."
|
||||||
|
echo "AMO would reject it, and build.yml stamps it verbatim."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
mm() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([0-9]+\.[0-9]+).*/\1/'; }
|
||||||
|
MAN=$(mm extension/manifest.json)
|
||||||
|
PKG=$(mm extension/package.json)
|
||||||
|
test -n "$MAN" || { echo "ERROR: no parseable version in extension/manifest.json"; exit 1; }
|
||||||
|
test -n "$PKG" || { echo "ERROR: no parseable version in extension/package.json"; exit 1; }
|
||||||
|
if [ "$MAN" != "$PKG" ]; then
|
||||||
|
echo "ERROR: MAJOR.MINOR disagrees between the two files."
|
||||||
|
echo " extension/manifest.json = $MAN <- packaging.sh reads MAJOR.MINOR from here"
|
||||||
|
echo " extension/package.json = $PKG"
|
||||||
|
echo "Only MAJOR.MINOR is hand-set. The patch component is derived from"
|
||||||
|
echo "commit time and overwritten at build time, so the committed patch"
|
||||||
|
echo "numbers are inert — but MAJOR.MINOR still ships. Set both the same."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "OK: MAJOR.MINOR $MAN, derived version $VERSION"
|
||||||
|
|
||||||
backend-lint-and-test:
|
backend-lint-and-test:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
@@ -47,6 +115,13 @@ jobs:
|
|||||||
SECRET_KEY: ci_unit_test_placeholder
|
SECRET_KEY: ci_unit_test_placeholder
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
# Full history for tests/test_artifact_identity.py, which derives
|
||||||
|
# each artifact's revision to check the identity scheme. On a
|
||||||
|
# depth-1 clone that derivation either fails or returns the tip sha
|
||||||
|
# — so the lane would go green while asserting nothing, which is
|
||||||
|
# the one outcome worse than a red one.
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
# Cache step removed 2026-05-26: act_runner's cache backend has been
|
# Cache step removed 2026-05-26: act_runner's cache backend has been
|
||||||
# broken on this homelab runner since 2026-05-15 (first as request-
|
# broken on this homelab runner since 2026-05-15 (first as request-
|
||||||
@@ -92,10 +167,10 @@ jobs:
|
|||||||
# If we want strict lockfile-based reproducibility later, commit a
|
# If we want strict lockfile-based reproducibility later, commit a
|
||||||
# package-lock.json and flip this back to `npm ci`.
|
# package-lock.json and flip this back to `npm ci`.
|
||||||
- run: npm install --no-audit --no-fund
|
- run: npm install --no-audit --no-fund
|
||||||
# `npm run check` (vue-tsc --noEmit) skipped: the frontend is pure JS
|
# No type-check step: the frontend is pure JS (no .ts files, no JSDoc),
|
||||||
# with no .ts files and no JSDoc annotations, so vue-tsc has nothing
|
# so a type-checker has nothing to do. The vue-tsc devDep + its `check`
|
||||||
# to type-check. Re-enable once we add a tsconfig.json and either
|
# script were dropped 2026-07-11 rather than bumped to v3. If we add
|
||||||
# convert to TS or add JSDoc.
|
# TS/JSDoc later, re-add a tsconfig.json + vue-tsc + a type-check step.
|
||||||
- run: npm run test:unit
|
- run: npm run test:unit
|
||||||
- run: npm run build
|
- run: npm run build
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
name: extension
|
name: extension
|
||||||
# Lint-only workflow. The sign-and-publish dance moved into build.yml's
|
# Lint + unit tests. The sign-and-publish dance moved into build.yml's
|
||||||
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
|
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
|
||||||
# because sign-extension runs as a build-web dependency in the SAME workflow,
|
# because sign-extension runs as a build-web dependency in the SAME workflow,
|
||||||
# eliminating the prior race between build.yml and a separate extension.yml.
|
# eliminating the prior race between build.yml and a separate extension.yml.
|
||||||
@@ -10,20 +10,78 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
- 'extension/**'
|
- 'extension/**'
|
||||||
- '.forgejo/workflows/extension.yml'
|
- '.forgejo/workflows/extension.yml'
|
||||||
|
# test/version.spec.js asserts things ABOUT the other two workflows —
|
||||||
|
# that neither inlines the packaged-file set, and that build.yml derives
|
||||||
|
# the shipped version rather than reading it out of the repo. A
|
||||||
|
# workflow-only edit can therefore break this suite, so it has to trigger
|
||||||
|
# it. build.yml joined the list at milestone 271 step 5, when the spec
|
||||||
|
# started asserting against it.
|
||||||
|
- '.forgejo/workflows/ci.yml'
|
||||||
|
- '.forgejo/workflows/build.yml'
|
||||||
pull_request:
|
pull_request:
|
||||||
branches: [main]
|
branches: [main]
|
||||||
paths:
|
paths:
|
||||||
- 'extension/**'
|
- 'extension/**'
|
||||||
|
- '.forgejo/workflows/ci.yml'
|
||||||
|
- '.forgejo/workflows/build.yml'
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
lint:
|
lint:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: node:22-bookworm-slim
|
image: node:24-bookworm-slim
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
- name: Install web-ext
|
# Not --no-save: vitest and web-ext are both real devDependencies now,
|
||||||
run: cd extension && npm install --no-save --no-audit --no-fund
|
# and the suite needs vitest resolvable from node_modules.
|
||||||
|
- name: Install dev dependencies
|
||||||
|
run: cd extension && npm install --no-audit --no-fund
|
||||||
- name: Lint
|
- name: Lint
|
||||||
run: cd extension && npm run lint
|
run: cd extension && npm run lint
|
||||||
|
# Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
|
||||||
|
# package version-consistency checks. No browser, no network.
|
||||||
|
- name: Unit tests
|
||||||
|
run: cd extension && npm run test:unit
|
||||||
|
|
||||||
|
# Everything else about packaging is asserted against our own declaration
|
||||||
|
# of what ships. This is the only check that asks web-ext what it ACTUALLY
|
||||||
|
# put in the archive. Until now that was an unverified assumption about
|
||||||
|
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
|
||||||
|
# only because callers `set -f` first, so losing that quoting would
|
||||||
|
# silently start shipping dev files with no other signal.
|
||||||
|
- name: Verify XPI contents
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
|
||||||
|
cd extension
|
||||||
|
npm run build
|
||||||
|
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
|
||||||
|
echo "=== packaged entries in $ZIP ==="
|
||||||
|
unzip -Z1 "$ZIP" | sort
|
||||||
|
echo "=== end ==="
|
||||||
|
ENTRIES=$(unzip -Z1 "$ZIP")
|
||||||
|
fail=0
|
||||||
|
# Must NOT ship: repo infrastructure with no business in a user's browser.
|
||||||
|
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
|
||||||
|
if echo "$ENTRIES" | grep -q "^$pat"; then
|
||||||
|
echo "ERROR: '$pat' was packaged into the XPI but must not be"
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
# Must ship: if an exclusion pattern ever over-matches, the extension
|
||||||
|
# breaks at runtime rather than at build time, so assert presence too.
|
||||||
|
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
|
||||||
|
if ! echo "$ENTRIES" | grep -q "^$req$"; then
|
||||||
|
echo "ERROR: '$req' is missing from the XPI"
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
|
||||||
|
if ! echo "$ENTRIES" | grep -q "^$dir"; then
|
||||||
|
echo "ERROR: nothing from '$dir' was packaged"
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[ "$fail" -eq 0 ] || exit 1
|
||||||
|
echo "XPI contents verified."
|
||||||
|
|||||||
+18
-1
@@ -1,6 +1,6 @@
|
|||||||
# syntax=docker/dockerfile:1.25
|
# syntax=docker/dockerfile:1.25
|
||||||
|
|
||||||
FROM node:22-alpine AS frontend-builder
|
FROM node:24-alpine AS frontend-builder
|
||||||
WORKDIR /build
|
WORKDIR /build
|
||||||
COPY frontend/package.json frontend/package-lock.json* ./
|
COPY frontend/package.json frontend/package-lock.json* ./
|
||||||
# No package-lock.json is tracked yet (we don't run npm locally per
|
# No package-lock.json is tracked yet (we don't run npm locally per
|
||||||
@@ -47,6 +47,23 @@ RUN chmod +x entrypoint.sh
|
|||||||
|
|
||||||
COPY --from=frontend-builder /build/dist ./frontend/dist
|
COPY --from=frontend-builder /build/dist ./frontend/dist
|
||||||
|
|
||||||
|
# Which channel this image belongs to — `dev` or `main` (milestone 271 step 7).
|
||||||
|
# build.yml passes it; /api/extension/manifest reports it beside the version so
|
||||||
|
# an operator can tell which channel an install came from without the channel
|
||||||
|
# ever touching the version string.
|
||||||
|
#
|
||||||
|
# Empty by default, deliberately: a locally-built image then reports NO channel
|
||||||
|
# rather than claiming to be one, and the manifest omits the field entirely —
|
||||||
|
# indistinguishable from an image built before the field existed, which is
|
||||||
|
# exactly the shape every reader already has to handle.
|
||||||
|
#
|
||||||
|
# Declared LAST on purpose. An ARG/ENV invalidates every layer below it, and
|
||||||
|
# this is the one value that differs between the dev and main builds of
|
||||||
|
# identical source — put it any earlier and the two channels could never share
|
||||||
|
# a cached pip install.
|
||||||
|
ARG FC_CHANNEL=""
|
||||||
|
ENV FC_CHANNEL=${FC_CHANNEL}
|
||||||
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
|
|
||||||
ENTRYPOINT ["./entrypoint.sh"]
|
ENTRYPOINT ["./entrypoint.sh"]
|
||||||
|
|||||||
@@ -6,7 +6,21 @@ Combines what was [ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo)
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Pre-v1. Not yet functional.
|
In production. `main` is continuously deployed — every merge to `main` builds
|
||||||
|
and publishes `:latest` images, so whatever is on `main` is what is running.
|
||||||
|
Day-to-day work happens on `dev`, which publishes `:dev` images.
|
||||||
|
|
||||||
|
## What's in here
|
||||||
|
|
||||||
|
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
|
||||||
|
|
||||||
|
| Piece | Built from | Image | Role |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
|
||||||
|
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
|
||||||
|
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. |
|
||||||
|
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
|
||||||
|
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -29,22 +43,37 @@ docker compose -f docker-compose.yml up -d
|
|||||||
# (skips the override so containers pull registry images)
|
# (skips the override so containers pull registry images)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The GPU agent is deployed separately, on the machine with the card —
|
||||||
|
`agent/docker-compose.yml`, not this stack.
|
||||||
|
|
||||||
## Deployment posture
|
## Deployment posture
|
||||||
|
|
||||||
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
|
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
|
||||||
|
|
||||||
## CI / Forgejo setup
|
## CI / Forgejo setup
|
||||||
|
|
||||||
The repo's workflows expect:
|
Three workflows: `ci.yml` (lint, extension-version check, backend unit tests,
|
||||||
|
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
|
||||||
|
content verification), and `build.yml` (sign + publish).
|
||||||
|
|
||||||
- **Runner label `python-ci`** — a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Both `ci.yml` and `build.yml` use this label. The runner image (`runner-base:python-ci`) is built from `CI-Runner/CI-python/` in the operator's workspace; `make push` from that directory builds and pushes a new image when toolchain pins change.
|
**The toolchain each job runs in is its `container.image`, not its `runs-on`
|
||||||
- **Repo secret `RELEASE_TOKEN`** — a Forgejo PAT with the following scopes:
|
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
|
||||||
|
then names the image it actually wants. `ci-requirements.md` is the current,
|
||||||
|
authoritative list of images and per-job installs — read that rather than a
|
||||||
|
copy here, so the two can't drift.
|
||||||
|
|
||||||
|
The repo expects one secret:
|
||||||
|
|
||||||
|
- **`RELEASE_TOKEN`** — a Forgejo PAT with:
|
||||||
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
|
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
|
||||||
- `write:release` — for future release-cutting workflows
|
- `write:release` — for the `ext-<version>` releases that cache the signed XPI
|
||||||
- `write:issue` — for future issue-management automation
|
- `write:issue` — for issue-management automation
|
||||||
|
|
||||||
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
|
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
|
||||||
|
|
||||||
|
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on
|
||||||
|
`main` only and is cached per version, since AMO rejects a re-signed version.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
Personal project; use at your own discretion.
|
Personal project; use at your own discretion.
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
|
|||||||
# Bump on every agent change. The page embeds this and /status reports it; the UI
|
# Bump on every agent change. The page embeds this and /status reports it; the UI
|
||||||
# warns to reload when they differ — so a stale browser-cached page can't be
|
# warns to reload when they differ — so a stale browser-cached page can't be
|
||||||
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
|
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
|
||||||
VERSION = "2026-07-02.6 · sleep mode: an empty queue sheds to one downloader and backs the lease poll off to 15 min"
|
VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
|
||||||
|
|
||||||
logbuf.install()
|
logbuf.install()
|
||||||
cfg = Config.from_env()
|
cfg = Config.from_env()
|
||||||
@@ -334,9 +334,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
|
|||||||
waited.textContent=s.transient||0
|
waited.textContent=s.transient||0
|
||||||
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
|
// Instantaneous pool state → demoted to the sub-line, where its jumpiness reads
|
||||||
// as live churn rather than a "broken" headline metric.
|
// as live churn rather than a "broken" headline metric.
|
||||||
|
// '=== false' (not falsy) so a stale page that doesn't send models_loaded shows
|
||||||
|
// nothing; when the idle monitor unloads, the VRAM meter drops alongside this.
|
||||||
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'—')+' · consumers '+(s.consumers!=null?s.consumers:'—')+' · on GPU '+(s.active||0)
|
pipe.textContent='downloaders '+(s.downloaders!=null?s.downloaders:'—')+' · consumers '+(s.consumers!=null?s.consumers:'—')+' · on GPU '+(s.active||0)
|
||||||
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'—')+' MB/s'
|
+' · net '+(s.net_mb_s!=null?s.net_mb_s.toFixed(1):'—')+' MB/s'
|
||||||
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
|
+(s.bandwidth_limit_mb_s>0?(' / cap '+s.bandwidth_limit_mb_s):'')
|
||||||
|
+(s.models_loaded===false?' · GPU models unloaded (idle — reload on next job)':'')
|
||||||
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
|
if(document.activeElement!==bw && s.bandwidth_limit_mb_s!=null) bw.value=s.bandwidth_limit_mb_s
|
||||||
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
|
// Buffer occupancy bar (also driven here so it tracks the /status cadence).
|
||||||
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
|
if(s.buffer!=null && s.buffer_max){ const p=Math.round(100*s.buffer/s.buffer_max)
|
||||||
|
|||||||
@@ -51,6 +51,12 @@ class Config:
|
|||||||
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
|
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
|
||||||
# all downloaders + video streams (0 = unlimited);
|
# all downloaders + video streams (0 = unlimited);
|
||||||
# tunable live from the agent UI
|
# tunable live from the agent UI
|
||||||
|
idle_unload_seconds: float # after this long with the GPU idle (nothing in
|
||||||
|
# flight, queue empty or Stopped), unload the
|
||||||
|
# SigLIP embedder + YOLO proposers to free their
|
||||||
|
# VRAM; they reload lazily on the next job. A
|
||||||
|
# 24/7 agent otherwise squats on ~5GB doing
|
||||||
|
# nothing. 0 disables (keep models warm forever).
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_env(cls) -> Config:
|
def from_env(cls) -> Config:
|
||||||
@@ -87,4 +93,8 @@ class Config:
|
|||||||
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
|
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
|
||||||
# from the agent UI on wired/faster networks.
|
# from the agent UI on wired/faster networks.
|
||||||
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
|
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
|
||||||
|
# 5 min: long enough that a lull between job bursts doesn't thrash the
|
||||||
|
# (few-second) reload, short enough that an agent left running with an
|
||||||
|
# empty queue hands its VRAM back promptly.
|
||||||
|
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -170,6 +170,13 @@ class YoloProposer:
|
|||||||
))
|
))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
def unload(self) -> None:
|
||||||
|
"""Drop the loaded YOLO so its VRAM can be reclaimed; detect() reloads it
|
||||||
|
lazily on the next job. Leaves _ok untouched — a healthy proposer comes
|
||||||
|
back, but one that self-disabled on a fault stays off."""
|
||||||
|
with self._lock:
|
||||||
|
self._model = None
|
||||||
|
|
||||||
|
|
||||||
class Proposers:
|
class Proposers:
|
||||||
"""The agent's proposer set, built from config. Each detector is optional —
|
"""The agent's proposer set, built from config. Each detector is optional —
|
||||||
@@ -216,3 +223,11 @@ class Proposers:
|
|||||||
|
|
||||||
def panels(self, image):
|
def panels(self, image):
|
||||||
return self._top(self._panel, image, self.cfg.max_panels)
|
return self._top(self._panel, image, self.cfg.max_panels)
|
||||||
|
|
||||||
|
def unload(self) -> None:
|
||||||
|
"""Release every loaded proposer's YOLO (idle VRAM reclaim). The worker
|
||||||
|
also drops its reference to this Proposers and rebuilds a fresh one via
|
||||||
|
_proposers_for on the next job, so this is belt-and-braces."""
|
||||||
|
for p in (self._person, self._anatomy, self._panel):
|
||||||
|
if p is not None:
|
||||||
|
p.unload()
|
||||||
|
|||||||
@@ -75,3 +75,18 @@ class CropEmbedder:
|
|||||||
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
|
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
|
||||||
arr = pooled.float().cpu().numpy().astype(np.float32)
|
arr = pooled.float().cpu().numpy().astype(np.float32)
|
||||||
return [row.reshape(-1).tolist() for row in arr]
|
return [row.reshape(-1).tolist() for row in arr]
|
||||||
|
|
||||||
|
def unload(self) -> bool:
|
||||||
|
"""Drop the loaded model so its VRAM can be reclaimed — the idle monitor
|
||||||
|
calls this after a spell with no work so an idle agent doesn't squat on
|
||||||
|
the card; the next embed() reloads it lazily (a few seconds). Held under
|
||||||
|
BOTH the load and inference locks so it can never race a concurrent load
|
||||||
|
or an in-flight forward pass. Returns True if a model was actually
|
||||||
|
released (the caller then runs one empty_cache() to hand the freed blocks
|
||||||
|
back to the driver)."""
|
||||||
|
with self._load_lock, self._infer_lock:
|
||||||
|
if self._model is None:
|
||||||
|
return False
|
||||||
|
self._model = None
|
||||||
|
self._processor = None
|
||||||
|
return True
|
||||||
|
|||||||
@@ -57,6 +57,15 @@ MAX_BACKOFF_SECONDS = 60.0
|
|||||||
# up on their own.
|
# up on their own.
|
||||||
IDLE_POLL_MAX_SECONDS = 900.0
|
IDLE_POLL_MAX_SECONDS = 900.0
|
||||||
|
|
||||||
|
# Idle VRAM reclaim (operator 2026-07-17): the SigLIP embedder + YOLO proposers
|
||||||
|
# load lazily and then stay warm for fast job bursts — but a 24/7 agent with an
|
||||||
|
# empty queue would otherwise squat on that VRAM (~5GB on the operator's card)
|
||||||
|
# indefinitely while doing nothing. So a monitor unloads them after
|
||||||
|
# cfg.idle_unload_seconds with the GPU genuinely idle (nothing in flight, buffer
|
||||||
|
# drained); they reload lazily on the next job. This is just how often the
|
||||||
|
# monitor wakes to check — it bounds how soon past the threshold the unload fires.
|
||||||
|
IDLE_UNLOAD_CHECK_INTERVAL = 30.0
|
||||||
|
|
||||||
# A job whose fetch dies transiently this many times IN ONE SESSION stops being
|
# A job whose fetch dies transiently this many times IN ONE SESSION stops being
|
||||||
# handed back and is failed instead. Transient handbacks (release) burn no
|
# handed back and is failed instead. Transient handbacks (release) burn no
|
||||||
# attempts on the server, so a poisoned transfer — an original that stalls the
|
# attempts on the server, so a poisoned transfer — an original that stalls the
|
||||||
@@ -268,6 +277,11 @@ class Worker:
|
|||||||
self._proposers_sig = None # detector-config signature the current
|
self._proposers_sig = None # detector-config signature the current
|
||||||
# proposers were built for (#134)
|
# proposers were built for (#134)
|
||||||
self._proposers_lock = threading.Lock()
|
self._proposers_lock = threading.Lock()
|
||||||
|
# Monotonic time of the last GPU activity (a consumer finishing a job).
|
||||||
|
# The idle monitor unloads the warm models once this goes stale by
|
||||||
|
# cfg.idle_unload_seconds — see _idle_unload_loop.
|
||||||
|
self._last_gpu_activity = time.monotonic()
|
||||||
|
threading.Thread(target=self._idle_unload_loop, daemon=True).start()
|
||||||
|
|
||||||
# --- held-lease bookkeeping --------------------------------------------
|
# --- held-lease bookkeeping --------------------------------------------
|
||||||
def _hold(self, job_ids) -> None:
|
def _hold(self, job_ids) -> None:
|
||||||
@@ -608,6 +622,9 @@ class Worker:
|
|||||||
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
|
"net_mb_s": round(self._net_mb_s, 1), # observed aggregate rate
|
||||||
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
|
"bw_capped": self._bw_capped, # autoscaler holding at the cap (UI hint)
|
||||||
"idle": self._idle, # queue empty → poll backed off (UI hint)
|
"idle": self._idle, # queue empty → poll backed off (UI hint)
|
||||||
|
# Whether the GPU models are currently resident (False after an idle
|
||||||
|
# unload freed their VRAM) — a plain bool read, UI hint only.
|
||||||
|
"models_loaded": self._embedder is not None or self._proposers is not None,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
|
def _bump(self, *, processed=0, downloaded=0, errors=0, active=0, transient=0):
|
||||||
@@ -788,6 +805,9 @@ class Worker:
|
|||||||
self._bump(processed=1)
|
self._bump(processed=1)
|
||||||
finally:
|
finally:
|
||||||
self._bump(active=-1)
|
self._bump(active=-1)
|
||||||
|
# Mark the GPU busy-until-now so the idle monitor starts its
|
||||||
|
# unload countdown from when work actually stopped, not before.
|
||||||
|
self._last_gpu_activity = time.monotonic()
|
||||||
|
|
||||||
def _ensure_embedder(self, model_name: str):
|
def _ensure_embedder(self, model_name: str):
|
||||||
if self._embedder is not None:
|
if self._embedder is not None:
|
||||||
@@ -845,6 +865,61 @@ class Worker:
|
|||||||
self._proposers_sig = sig
|
self._proposers_sig = sig
|
||||||
return self._proposers
|
return self._proposers
|
||||||
|
|
||||||
|
def _unload_models(self) -> bool:
|
||||||
|
"""Release the GPU-resident models (SigLIP embedder + YOLO proposers) so an
|
||||||
|
idle agent hands their VRAM back instead of squatting on the card. They
|
||||||
|
reload lazily on the next job (_ensure_embedder / _proposers_for) — a
|
||||||
|
few seconds' cost paid only when work actually resumes. Dropping the
|
||||||
|
shared instances under their build locks means a concurrent job either
|
||||||
|
sees the old instance (before) or rebuilds a fresh one (after); the idle
|
||||||
|
monitor only calls this with nothing in flight, so no inference is using
|
||||||
|
them. Returns True if anything was released."""
|
||||||
|
released = False
|
||||||
|
with self._embedder_lock:
|
||||||
|
if self._embedder is not None:
|
||||||
|
self._embedder.unload()
|
||||||
|
self._embedder = None
|
||||||
|
released = True
|
||||||
|
with self._proposers_lock:
|
||||||
|
if self._proposers is not None:
|
||||||
|
self._proposers.unload()
|
||||||
|
self._proposers = None
|
||||||
|
self._proposers_sig = None
|
||||||
|
released = True
|
||||||
|
if released:
|
||||||
|
try:
|
||||||
|
import torch
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
# torch's caching allocator holds freed blocks; hand them back
|
||||||
|
# to the driver so nvidia-smi actually reflects the drop.
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
except Exception: # noqa: BLE001 — torch absent / CPU-only → nothing to free
|
||||||
|
pass
|
||||||
|
return released
|
||||||
|
|
||||||
|
def _idle_unload_loop(self) -> None:
|
||||||
|
"""Unload the warm GPU models after a stretch of inactivity so a 24/7
|
||||||
|
agent with an empty queue doesn't hold ~5GB of VRAM doing nothing. Fires
|
||||||
|
only when nothing is in flight (active == 0 AND the buffer is drained) and
|
||||||
|
no job has completed for cfg.idle_unload_seconds — a window long enough
|
||||||
|
that a brief lull between bursts doesn't thrash reload/unload. Covers BOTH
|
||||||
|
sleep mode (queue empty, pipeline still running) and a full Stop; the
|
||||||
|
models reload lazily on the next job. idle_unload_seconds <= 0 disables it."""
|
||||||
|
idle_after = self.cfg.idle_unload_seconds
|
||||||
|
if idle_after <= 0:
|
||||||
|
return
|
||||||
|
while True:
|
||||||
|
time.sleep(IDLE_UNLOAD_CHECK_INTERVAL)
|
||||||
|
if self._embedder is None and self._proposers is None:
|
||||||
|
continue # nothing loaded → nothing to free
|
||||||
|
if self._active != 0 or not self._buffer.empty():
|
||||||
|
continue # work in flight → keep them warm
|
||||||
|
if time.monotonic() - self._last_gpu_activity < idle_after:
|
||||||
|
continue # not idle long enough yet
|
||||||
|
if self._unload_models():
|
||||||
|
log.info("idle %.0fs — unloaded GPU models, freed VRAM "
|
||||||
|
"(reload on next job)", idle_after)
|
||||||
|
|
||||||
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
|
def _consume(self, job: dict, frames: list, stop_evt: threading.Event) -> bool:
|
||||||
"""Detect + embed the decoded frames and submit the result. Returns True
|
"""Detect + embed the decoded frames and submit the result. Returns True
|
||||||
when the job was completed (→ count it processed), False otherwise: a
|
when the job was completed (→ count it processed), False otherwise: a
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""title-based WIP auto-tagging (task #1458) — ImportSettings toggle
|
||||||
|
|
||||||
|
ImportSettings gains wip_title_tagging_enabled (ON by default): when a freshly
|
||||||
|
imported post's title explicitly declares work-in-progress ("WIP" / "work in
|
||||||
|
progress"), the importer applies the `wip` system tag to its images. No new
|
||||||
|
table — the tag itself is the seeded `wip` system tag (migration 0075) and the
|
||||||
|
application reuses image_tag with source='wip_title'.
|
||||||
|
|
||||||
|
Revision ID: 0085
|
||||||
|
Revises: 0084
|
||||||
|
Create Date: 2026-07-12
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision: str = "0085"
|
||||||
|
down_revision: Union[str, None] = "0084"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"import_settings",
|
||||||
|
sa.Column(
|
||||||
|
"wip_title_tagging_enabled", sa.Boolean(), nullable=False,
|
||||||
|
server_default=sa.text("true"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_column("import_settings", "wip_title_tagging_enabled")
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
"""process auto-apply settings + review mode (#1464) — system-tag refactor
|
||||||
|
|
||||||
|
The system-tag behavior refactor gives `wip` / `editor screenshot` (the PROCESS
|
||||||
|
group) their own provisional auto-apply, parallel to the presentation (chrome)
|
||||||
|
sweep. MLSettings gains three knobs: enabled (OFF by default — a new whole-library
|
||||||
|
auto-tagger is opt-in), the flat apply threshold, and the ring-loud conflict
|
||||||
|
threshold. presentation_review gains a `mode` column so one review surface serves
|
||||||
|
both chrome and process flags (existing rows backfill 'chrome'). server_defaults
|
||||||
|
so the existing rows fill cleanly.
|
||||||
|
|
||||||
|
Revision ID: 0086
|
||||||
|
Revises: 0085
|
||||||
|
Create Date: 2026-07-13
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision: str = "0086"
|
||||||
|
down_revision: Union[str, None] = "0085"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"ml_settings",
|
||||||
|
sa.Column(
|
||||||
|
"process_auto_apply_enabled", sa.Boolean(), nullable=False,
|
||||||
|
server_default=sa.text("false"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"ml_settings",
|
||||||
|
sa.Column(
|
||||||
|
"process_auto_apply_threshold", sa.Float(), nullable=False,
|
||||||
|
server_default="0.90",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"ml_settings",
|
||||||
|
sa.Column(
|
||||||
|
"process_conflict_threshold", sa.Float(), nullable=False,
|
||||||
|
server_default="0.50",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
op.add_column(
|
||||||
|
"presentation_review",
|
||||||
|
sa.Column(
|
||||||
|
"mode", sa.String(16), nullable=False,
|
||||||
|
server_default="chrome",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_column("presentation_review", "mode")
|
||||||
|
op.drop_column("ml_settings", "process_conflict_threshold")
|
||||||
|
op.drop_column("ml_settings", "process_auto_apply_threshold")
|
||||||
|
op.drop_column("ml_settings", "process_auto_apply_enabled")
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
"""soft WIP title tier toggle (#1474) — ImportSettings.wip_soft_title_tagging_enabled
|
||||||
|
|
||||||
|
The soft tier also tags sketch/doodle/scribble titles, but with a provisional source
|
||||||
|
that never trains the head. OFF by default (a lower-precision tier is opt-in).
|
||||||
|
server_default so the existing singleton row (id=1) fills cleanly.
|
||||||
|
|
||||||
|
Revision ID: 0087
|
||||||
|
Revises: 0086
|
||||||
|
Create Date: 2026-07-13
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision: str = "0087"
|
||||||
|
down_revision: Union[str, None] = "0086"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"import_settings",
|
||||||
|
sa.Column(
|
||||||
|
"wip_soft_title_tagging_enabled", sa.Boolean(), nullable=False,
|
||||||
|
server_default=sa.text("false"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_column("import_settings", "wip_soft_title_tagging_enabled")
|
||||||
@@ -459,6 +459,22 @@ async def trigger_prune_missing_files():
|
|||||||
return _queued(async_result)
|
return _queued(async_result)
|
||||||
|
|
||||||
|
|
||||||
|
@admin_bp.route("/maintenance/reclaim-attachments", methods=["POST"])
|
||||||
|
async def trigger_reclaim_attachments():
|
||||||
|
"""Reclaim orphaned attachments (#3068). Body {"dry_run": bool}: dry_run
|
||||||
|
(the DEFAULT here) projects the orphan rows and unreferenced store blobs
|
||||||
|
without touching either; dry_run=false deletes the rows then unlinks every
|
||||||
|
blob no surviving row references. Maintenance queue; operator-triggered
|
||||||
|
only — never an unattended sweep, since the apply unlinks files. Returns the
|
||||||
|
Celery task id — poll /maintenance/task-result/<id> for the summary."""
|
||||||
|
from ..tasks.admin import reclaim_orphaned_attachments_task
|
||||||
|
|
||||||
|
body = await request.get_json(silent=True) or {}
|
||||||
|
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
|
||||||
|
async_result = reclaim_orphaned_attachments_task.delay(dry_run=dry_run)
|
||||||
|
return _queued(async_result)
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
||||||
async def trigger_dedup_videos():
|
async def trigger_dedup_videos():
|
||||||
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import os
|
||||||
import re
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -30,6 +32,12 @@ XPI_DIR = Path("/app/frontend/dist/extension")
|
|||||||
|
|
||||||
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
|
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
|
||||||
|
|
||||||
|
# Which channel this image belongs to — "dev" or "main" — baked in at build
|
||||||
|
# time from the FC_CHANNEL build arg (milestone 271 step 7). Empty for a local
|
||||||
|
# build, or for any image predating the field. Tests override by monkeypatching
|
||||||
|
# this constant, same as XPI_DIR above.
|
||||||
|
FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
|
||||||
|
|
||||||
|
|
||||||
async def _ext_key_required(session) -> bool:
|
async def _ext_key_required(session) -> bool:
|
||||||
"""Unlike /api/credentials (which accepts the browser path with no
|
"""Unlike /api/credentials (which accepts the browser path with no
|
||||||
@@ -41,7 +49,15 @@ async def _ext_key_required(session) -> bool:
|
|||||||
stored = (await session.execute(
|
stored = (await session.execute(
|
||||||
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
|
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
|
||||||
)).scalar_one_or_none()
|
)).scalar_one_or_none()
|
||||||
return stored is not None and supplied == stored
|
if stored is None:
|
||||||
|
return False
|
||||||
|
# compare_digest, not `==`: the stored key is a shared secret, and a
|
||||||
|
# short-circuiting compare leaks its prefix through timing. Costs nothing
|
||||||
|
# here — it is not that this route is exposed (#3072). Compared as BYTES:
|
||||||
|
# compare_digest's str form rejects non-ASCII with TypeError, and this
|
||||||
|
# header is attacker-supplied, so a str compare would turn a junk key into
|
||||||
|
# a 500 instead of a 403.
|
||||||
|
return hmac.compare_digest(supplied.encode("utf-8"), stored.encode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
def _extract_version(xpi_name: str) -> str:
|
def _extract_version(xpi_name: str) -> str:
|
||||||
@@ -124,13 +140,30 @@ def _read_manifest_sync() -> dict | None:
|
|||||||
return None
|
return None
|
||||||
versioned.sort(key=lambda p: p.stat().st_mtime)
|
versioned.sort(key=lambda p: p.stat().st_mtime)
|
||||||
latest = versioned[-1]
|
latest = versioned[-1]
|
||||||
return {
|
info = {
|
||||||
"installed": True,
|
"installed": True,
|
||||||
"version": _extract_version(latest.name),
|
"version": _extract_version(latest.name),
|
||||||
"xpi_url": f"/extension/{latest.name}",
|
"xpi_url": f"/extension/{latest.name}",
|
||||||
"latest_url": "/extension/fabledcurator-latest.xpi",
|
"latest_url": "/extension/fabledcurator-latest.xpi",
|
||||||
"sha256": _sha256(latest),
|
"sha256": _sha256(latest),
|
||||||
}
|
}
|
||||||
|
# The channel goes BESIDE the version, never inside it. A `-dev` suffix is
|
||||||
|
# what silently disabled the dev channel in the sibling project this design
|
||||||
|
# comes from: the comparator returned nothing for a non-integer segment, so
|
||||||
|
# every dev version compared equal and "no update available" became
|
||||||
|
# indistinguishable from "I cannot read this version".
|
||||||
|
#
|
||||||
|
# Omitted rather than defaulted when unset. Absence already has a meaning
|
||||||
|
# every reader must handle — an image built before this field existed says
|
||||||
|
# exactly the same thing by not having the key — so a blank channel reuses
|
||||||
|
# that path instead of inventing a second "unknown" spelling.
|
||||||
|
#
|
||||||
|
# Reported verbatim, not validated against {"dev", "main"}: if an image
|
||||||
|
# declares something else, showing what it actually claims is more useful
|
||||||
|
# to whoever is debugging it than dropping the value on the floor.
|
||||||
|
if FC_CHANNEL:
|
||||||
|
info["channel"] = FC_CHANNEL
|
||||||
|
return info
|
||||||
|
|
||||||
|
|
||||||
@extension_bp.route("/manifest", methods=["GET"])
|
@extension_bp.route("/manifest", methods=["GET"])
|
||||||
|
|||||||
@@ -148,6 +148,17 @@ async def similar():
|
|||||||
# Explore passes exclude_wip=1 to also drop work-in-progress from the
|
# Explore passes exclude_wip=1 to also drop work-in-progress from the
|
||||||
# rabbit-hole; the gallery's own "similar" button omits it (keeps wip, #1274).
|
# rabbit-hole; the gallery's own "similar" button omits it (keeps wip, #1274).
|
||||||
exclude_wip = request.args.get("exclude_wip") in ("1", "true", "True")
|
exclude_wip = request.args.get("exclude_wip") in ("1", "true", "True")
|
||||||
|
# Explore reach (#1476): 0 = nearest (gallery default), →1 reaches into farther
|
||||||
|
# distance bands so the walk can escape a dense cluster. exclude_ids = the
|
||||||
|
# breadcrumb, so already-walked images aren't re-served as neighbours.
|
||||||
|
try:
|
||||||
|
reach = max(0.0, min(1.0, float(request.args.get("reach", "0"))))
|
||||||
|
except ValueError:
|
||||||
|
reach = 0.0
|
||||||
|
exclude_ids = [
|
||||||
|
int(x) for x in request.args.get("exclude_ids", "").split(",")
|
||||||
|
if x.strip().isdigit()
|
||||||
|
] or None
|
||||||
# post_id is the exclusive post-detail view — not a similarity scope.
|
# post_id is the exclusive post-detail view — not a similarity scope.
|
||||||
# include_hidden is a gallery-browse flag; similar() has its OWN presentation
|
# include_hidden is a gallery-browse flag; similar() has its OWN presentation
|
||||||
# exclusion (a similarity-quality concern, #1274), so drop it here (#141).
|
# exclusion (a similarity-quality concern, #1274), so drop it here (#141).
|
||||||
@@ -158,7 +169,8 @@ async def similar():
|
|||||||
svc = GalleryService(session)
|
svc = GalleryService(session)
|
||||||
try:
|
try:
|
||||||
images = await svc.similar(
|
images = await svc.similar(
|
||||||
image_id=similar_to, limit=limit, exclude_wip=exclude_wip, **scope)
|
image_id=similar_to, limit=limit, exclude_wip=exclude_wip,
|
||||||
|
reach=reach, exclude_ids=exclude_ids, **scope)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
return jsonify({"error": str(exc)}), 400
|
return jsonify({"error": str(exc)}), 400
|
||||||
if images is None:
|
if images is None:
|
||||||
@@ -236,8 +248,10 @@ async def jump():
|
|||||||
# content", surfaced in the gallery's Show-hidden review strip. -----------
|
# content", surfaced in the gallery's Show-hidden review strip. -----------
|
||||||
@gallery_bp.route("/hidden-review", methods=["GET"])
|
@gallery_bp.route("/hidden-review", methods=["GET"])
|
||||||
async def hidden_review():
|
async def hidden_review():
|
||||||
"""Unresolved presentation auto-hide flags, most-concerning first (highest
|
"""Unresolved system-tag auto-apply review flags (chrome + process, #1464),
|
||||||
content score) — for the gallery's Hidden-view review strip."""
|
most-concerning first (highest content score) — for the review strip. `mode`
|
||||||
|
tells the client whether the flagged tag hid the image ('chrome') or left it
|
||||||
|
visible ('process'), which decides the resolve labels (un-hide vs remove-tag)."""
|
||||||
ptag = aliased(Tag)
|
ptag = aliased(Tag)
|
||||||
ctag = aliased(Tag)
|
ctag = aliased(Tag)
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
@@ -247,6 +261,7 @@ async def hidden_review():
|
|||||||
PresentationReview.tag_id,
|
PresentationReview.tag_id,
|
||||||
PresentationReview.conflict_tag_id,
|
PresentationReview.conflict_tag_id,
|
||||||
PresentationReview.conflict_score,
|
PresentationReview.conflict_score,
|
||||||
|
PresentationReview.mode,
|
||||||
ImageRecord.path, ImageRecord.thumbnail_path,
|
ImageRecord.path, ImageRecord.thumbnail_path,
|
||||||
ImageRecord.sha256, ImageRecord.mime,
|
ImageRecord.sha256, ImageRecord.mime,
|
||||||
ptag.name.label("tag_name"),
|
ptag.name.label("tag_name"),
|
||||||
@@ -266,6 +281,7 @@ async def hidden_review():
|
|||||||
"conflict_tag_id": r.conflict_tag_id,
|
"conflict_tag_id": r.conflict_tag_id,
|
||||||
"conflict_name": r.conflict_name,
|
"conflict_name": r.conflict_name,
|
||||||
"conflict_score": r.conflict_score,
|
"conflict_score": r.conflict_score,
|
||||||
|
"mode": r.mode,
|
||||||
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
||||||
"image_url": image_url(r.path),
|
"image_url": image_url(r.path),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -256,9 +256,7 @@ async def lease():
|
|||||||
if not await _agent_authed(session):
|
if not await _agent_authed(session):
|
||||||
return jsonify({"error": "unauthorized"}), 401
|
return jsonify({"error": "unauthorized"}), 401
|
||||||
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
|
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
|
||||||
ml = (
|
ml = await MLSettings.load(session)
|
||||||
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
|
||||||
).scalar_one()
|
|
||||||
# image rows for url/mime in one shot
|
# image rows for url/mime in one shot
|
||||||
ids = [j.image_record_id for j in jobs]
|
ids = [j.image_record_id for j in jobs]
|
||||||
imgs = {
|
imgs = {
|
||||||
|
|||||||
+24
-38
@@ -4,6 +4,7 @@ from quart import Blueprint, jsonify, request
|
|||||||
|
|
||||||
from ..extensions import get_session
|
from ..extensions import get_session
|
||||||
from ..models import MLSettings
|
from ..models import MLSettings
|
||||||
|
from ..services.ml.heads import AUTO_APPLY_THRESHOLD_MAX, AUTO_APPLY_THRESHOLD_MIN
|
||||||
|
|
||||||
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
|
ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
|
||||||
|
|
||||||
@@ -42,6 +43,9 @@ _EDITABLE = (
|
|||||||
"presentation_auto_apply_enabled",
|
"presentation_auto_apply_enabled",
|
||||||
"presentation_auto_apply_threshold",
|
"presentation_auto_apply_threshold",
|
||||||
"presentation_conflict_threshold",
|
"presentation_conflict_threshold",
|
||||||
|
"process_auto_apply_enabled",
|
||||||
|
"process_auto_apply_threshold",
|
||||||
|
"process_conflict_threshold",
|
||||||
"embedder_model_name",
|
"embedder_model_name",
|
||||||
"embedder_model_version",
|
"embedder_model_version",
|
||||||
*_DETECTOR_FIELDS,
|
*_DETECTOR_FIELDS,
|
||||||
@@ -80,45 +84,21 @@ async def embedder_models():
|
|||||||
|
|
||||||
@ml_admin_bp.route("/settings", methods=["GET"])
|
@ml_admin_bp.route("/settings", methods=["GET"])
|
||||||
async def get_settings():
|
async def get_settings():
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
s = (
|
s = await MLSettings.load(session)
|
||||||
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
# Table-driven off _EDITABLE (which PATCH also writes) so a new settings field
|
||||||
).scalar_one()
|
# can never be silently absent from GET — the split that historically dropped
|
||||||
return jsonify(
|
# fields. _EDITABLE already includes *_DETECTOR_FIELDS.
|
||||||
{
|
return jsonify({f: getattr(s, f) for f in _EDITABLE})
|
||||||
"cpu_embed_enabled": s.cpu_embed_enabled,
|
|
||||||
"video_frame_interval_seconds": s.video_frame_interval_seconds,
|
|
||||||
"video_max_frames": s.video_max_frames,
|
|
||||||
"embedder_model_version": s.embedder_model_version,
|
|
||||||
"head_min_positives": s.head_min_positives,
|
|
||||||
"head_auto_apply_precision": s.head_auto_apply_precision,
|
|
||||||
"head_auto_apply_enabled": s.head_auto_apply_enabled,
|
|
||||||
"head_auto_apply_min_positives": s.head_auto_apply_min_positives,
|
|
||||||
"ccip_match_threshold": s.ccip_match_threshold,
|
|
||||||
"ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
|
|
||||||
"ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
|
|
||||||
"presentation_auto_apply_enabled": s.presentation_auto_apply_enabled,
|
|
||||||
"presentation_auto_apply_threshold": s.presentation_auto_apply_threshold,
|
|
||||||
"presentation_conflict_threshold": s.presentation_conflict_threshold,
|
|
||||||
"embedder_model_name": s.embedder_model_name,
|
|
||||||
**{f: getattr(s, f) for f in _DETECTOR_FIELDS},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@ml_admin_bp.route("/settings", methods=["PATCH"])
|
@ml_admin_bp.route("/settings", methods=["PATCH"])
|
||||||
async def patch_settings():
|
async def patch_settings():
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
body = await request.get_json()
|
body = await request.get_json()
|
||||||
if not isinstance(body, dict):
|
if not isinstance(body, dict):
|
||||||
return jsonify({"error": "body must be an object"}), 400
|
return jsonify({"error": "body must be an object"}), 400
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
s = (
|
s = await MLSettings.load(session)
|
||||||
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
|
||||||
).scalar_one()
|
|
||||||
|
|
||||||
# Merge the patch over current values, then validate the result as a
|
# Merge the patch over current values, then validate the result as a
|
||||||
# whole — the store-floor invariant couples three fields, so they
|
# whole — the store-floor invariant couples three fields, so they
|
||||||
@@ -148,20 +128,26 @@ def _validate(p: dict) -> str | None:
|
|||||||
# Head training (#114).
|
# Head training (#114).
|
||||||
if int(p["head_min_positives"]) < 1:
|
if int(p["head_min_positives"]) < 1:
|
||||||
return "head_min_positives must be >= 1"
|
return "head_min_positives must be >= 1"
|
||||||
if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999):
|
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["head_auto_apply_precision"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
||||||
return "head_auto_apply_precision must be between 0.5 and 0.999"
|
return f"head_auto_apply_precision must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
||||||
if int(p["head_auto_apply_min_positives"]) < 1:
|
if int(p["head_auto_apply_min_positives"]) < 1:
|
||||||
return "head_auto_apply_min_positives must be >= 1"
|
return "head_auto_apply_min_positives must be >= 1"
|
||||||
if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999):
|
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_match_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
||||||
return "ccip_match_threshold must be between 0.5 and 0.999"
|
return f"ccip_match_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
||||||
if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999):
|
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["ccip_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
||||||
return "ccip_auto_apply_threshold must be between 0.5 and 0.999"
|
return f"ccip_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
||||||
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
|
# Presentation chrome auto-hide (#141). Auto-apply runs high (hiding is
|
||||||
# consequential); the conflict cut is a plain probability [0,1].
|
# consequential); the conflict cut is a plain probability [0,1].
|
||||||
if not (0.5 <= float(p["presentation_auto_apply_threshold"]) <= 0.999):
|
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["presentation_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
||||||
return "presentation_auto_apply_threshold must be between 0.5 and 0.999"
|
return f"presentation_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
||||||
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
|
if not (0.0 <= float(p["presentation_conflict_threshold"]) <= 1.0):
|
||||||
return "presentation_conflict_threshold must be between 0 and 1"
|
return "presentation_conflict_threshold must be between 0 and 1"
|
||||||
|
# Process auto-apply (#1464). wip/editor stay VISIBLE so a false apply is
|
||||||
|
# low-harm (excludes-from-training + a review flag), but keep the same bar.
|
||||||
|
if not (AUTO_APPLY_THRESHOLD_MIN <= float(p["process_auto_apply_threshold"]) <= AUTO_APPLY_THRESHOLD_MAX):
|
||||||
|
return f"process_auto_apply_threshold must be between {AUTO_APPLY_THRESHOLD_MIN} and {AUTO_APPLY_THRESHOLD_MAX}"
|
||||||
|
if not (0.0 <= float(p["process_conflict_threshold"]) <= 1.0):
|
||||||
|
return "process_conflict_threshold must be between 0 and 1"
|
||||||
# Embedder model swap (#1190): both must be non-empty. Changing them means a
|
# Embedder model swap (#1190): both must be non-empty. Changing them means a
|
||||||
# different embedding space — the operator must re-embed + retrain after.
|
# different embedding space — the operator must re-embed + retrain after.
|
||||||
for key in ("embedder_model_name", "embedder_model_version"):
|
for key in ("embedder_model_name", "embedder_model_version"):
|
||||||
|
|||||||
+29
-26
@@ -48,6 +48,8 @@ _EDITABLE_FIELDS = (
|
|||||||
"interpreter_base_url",
|
"interpreter_base_url",
|
||||||
"translation_target_lang",
|
"translation_target_lang",
|
||||||
"translation_min_confidence",
|
"translation_min_confidence",
|
||||||
|
"wip_title_tagging_enabled",
|
||||||
|
"wip_soft_title_tagging_enabled",
|
||||||
)
|
)
|
||||||
|
|
||||||
# Per-host external-download toggles — all plain booleans, validated uniformly.
|
# Per-host external-download toggles — all plain booleans, validated uniformly.
|
||||||
@@ -64,32 +66,9 @@ _EXTDL_TOGGLE_FIELDS = (
|
|||||||
async def get_import_settings():
|
async def get_import_settings():
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
row = await ImportSettings.load(session)
|
row = await ImportSettings.load(session)
|
||||||
return jsonify({
|
# Table-driven off _EDITABLE_FIELDS (which PATCH also writes) so a new field
|
||||||
"min_width": row.min_width,
|
# can't be silently absent from GET.
|
||||||
"min_height": row.min_height,
|
return jsonify({f: getattr(row, f) for f in _EDITABLE_FIELDS})
|
||||||
"skip_transparent": row.skip_transparent,
|
|
||||||
"transparency_threshold": row.transparency_threshold,
|
|
||||||
"skip_single_color": row.skip_single_color,
|
|
||||||
"single_color_threshold": row.single_color_threshold,
|
|
||||||
"single_color_tolerance": row.single_color_tolerance,
|
|
||||||
"phash_threshold": row.phash_threshold,
|
|
||||||
"download_rate_limit_seconds": row.download_rate_limit_seconds,
|
|
||||||
"download_validate_files": row.download_validate_files,
|
|
||||||
"download_schedule_default_seconds": row.download_schedule_default_seconds,
|
|
||||||
"download_event_retention_days": row.download_event_retention_days,
|
|
||||||
"download_failure_warning_threshold": row.download_failure_warning_threshold,
|
|
||||||
"series_suggest_enabled": row.series_suggest_enabled,
|
|
||||||
"series_suggest_threshold": row.series_suggest_threshold,
|
|
||||||
"extdl_mega_enabled": row.extdl_mega_enabled,
|
|
||||||
"extdl_gdrive_enabled": row.extdl_gdrive_enabled,
|
|
||||||
"extdl_mediafire_enabled": row.extdl_mediafire_enabled,
|
|
||||||
"extdl_dropbox_enabled": row.extdl_dropbox_enabled,
|
|
||||||
"extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
|
|
||||||
"translation_enabled": row.translation_enabled,
|
|
||||||
"interpreter_base_url": row.interpreter_base_url,
|
|
||||||
"translation_target_lang": row.translation_target_lang,
|
|
||||||
"translation_min_confidence": row.translation_min_confidence,
|
|
||||||
})
|
|
||||||
|
|
||||||
|
|
||||||
@settings_bp.route("/settings/import", methods=["PATCH"])
|
@settings_bp.route("/settings/import", methods=["PATCH"])
|
||||||
@@ -171,6 +150,18 @@ async def update_import_settings():
|
|||||||
return jsonify(
|
return jsonify(
|
||||||
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
||||||
), 400
|
), 400
|
||||||
|
if "wip_title_tagging_enabled" in body and not isinstance(
|
||||||
|
body["wip_title_tagging_enabled"], bool
|
||||||
|
):
|
||||||
|
return jsonify(
|
||||||
|
{"error": "wip_title_tagging_enabled must be a boolean"}
|
||||||
|
), 400
|
||||||
|
if "wip_soft_title_tagging_enabled" in body and not isinstance(
|
||||||
|
body["wip_soft_title_tagging_enabled"], bool
|
||||||
|
):
|
||||||
|
return jsonify(
|
||||||
|
{"error": "wip_soft_title_tagging_enabled must be a boolean"}
|
||||||
|
), 400
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
row = await ImportSettings.load(session)
|
row = await ImportSettings.load(session)
|
||||||
@@ -182,6 +173,18 @@ async def update_import_settings():
|
|||||||
return await get_import_settings()
|
return await get_import_settings()
|
||||||
|
|
||||||
|
|
||||||
|
@settings_bp.route("/settings/wip-title/scan", methods=["POST"])
|
||||||
|
async def wip_title_scan():
|
||||||
|
"""Enqueue the back-catalogue WIP-title scan (task #1458 Settings button):
|
||||||
|
apply the `wip` system tag to EXISTING posts whose title declares
|
||||||
|
work-in-progress. New imports are tagged live by the importer; this catches
|
||||||
|
the existing library. Returns the Celery task id (202)."""
|
||||||
|
from ..tasks.maintenance import backfill_wip_title_tags
|
||||||
|
|
||||||
|
r = backfill_wip_title_tags.delay()
|
||||||
|
return jsonify({"celery_task_id": r.id}), 202
|
||||||
|
|
||||||
|
|
||||||
@settings_bp.route("/system/stats", methods=["GET"])
|
@settings_bp.route("/system/stats", methods=["GET"])
|
||||||
async def system_stats():
|
async def system_stats():
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
|
|||||||
@@ -171,9 +171,19 @@ def make_celery() -> Celery:
|
|||||||
},
|
},
|
||||||
"presentation-auto-apply-daily": {
|
"presentation-auto-apply-daily": {
|
||||||
"task": "backend.app.tasks.ml.scheduled_presentation_auto_apply",
|
"task": "backend.app.tasks.ml.scheduled_presentation_auto_apply",
|
||||||
"schedule": 86400.0, # auto-hide banner/editor chrome (#141);
|
"schedule": 86400.0, # auto-hide banner chrome (#141);
|
||||||
# no-op unless presentation_auto_apply_enabled
|
# no-op unless presentation_auto_apply_enabled
|
||||||
},
|
},
|
||||||
|
"process-auto-apply-daily": {
|
||||||
|
"task": "backend.app.tasks.ml.scheduled_process_auto_apply",
|
||||||
|
"schedule": 86400.0, # auto-tag wip/editor process art (#1464);
|
||||||
|
# no-op unless process_auto_apply_enabled (opt-in)
|
||||||
|
},
|
||||||
|
"soft-wip-conflict-audit-daily": {
|
||||||
|
"task": "backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
|
||||||
|
"schedule": 86400.0, # flag ring-loud soft-WIP (sketch/doodle) tags
|
||||||
|
# for review (#1474); no-op with no content heads
|
||||||
|
},
|
||||||
"prune-presentation-reviews-daily": {
|
"prune-presentation-reviews-daily": {
|
||||||
"task": "backend.app.tasks.ml.prune_presentation_reviews",
|
"task": "backend.app.tasks.ml.prune_presentation_reviews",
|
||||||
"schedule": 86400.0, # retention: drop resolved review flags >30d
|
"schedule": 86400.0, # retention: drop resolved review flags >30d
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ from .patreon_seen_media import PatreonSeenMedia
|
|||||||
from .pixiv_failed_media import PixivFailedMedia
|
from .pixiv_failed_media import PixivFailedMedia
|
||||||
from .pixiv_seen_media import PixivSeenMedia
|
from .pixiv_seen_media import PixivSeenMedia
|
||||||
from .post import Post
|
from .post import Post
|
||||||
from .post_attachment import PostAttachment
|
from .post_attachment import PostAttachment, attachment_download_url
|
||||||
from .presentation_review import PresentationReview
|
from .presentation_review import PresentationReview
|
||||||
from .series_chapter import SeriesChapter
|
from .series_chapter import SeriesChapter
|
||||||
from .series_page import SeriesPage
|
from .series_page import SeriesPage
|
||||||
@@ -58,6 +58,7 @@ __all__ = [
|
|||||||
"SubscribeStarSeenMedia",
|
"SubscribeStarSeenMedia",
|
||||||
"Post",
|
"Post",
|
||||||
"PostAttachment",
|
"PostAttachment",
|
||||||
|
"attachment_download_url",
|
||||||
"PresentationReview",
|
"PresentationReview",
|
||||||
"SeriesChapter",
|
"SeriesChapter",
|
||||||
"SeriesPage",
|
"SeriesPage",
|
||||||
|
|||||||
@@ -116,6 +116,24 @@ class ImportSettings(Base):
|
|||||||
Float, nullable=False, default=0.9, server_default="0.9",
|
Float, nullable=False, default=0.9, server_default="0.9",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Title-based WIP auto-tagging (task #1458). When a freshly-imported post's
|
||||||
|
# TITLE explicitly declares work-in-progress ("WIP" / "work in progress"),
|
||||||
|
# the importer applies the `wip` system tag to its images — the artist's own
|
||||||
|
# label, used to keep unfinished pieces out of the Explore/gallery browse. ON
|
||||||
|
# by default (rule 26 — the feature works out of the box). Gates only the
|
||||||
|
# LIVE import hook; the existing catalogue is caught by the operator-triggered
|
||||||
|
# "Scan existing posts" backfill (which runs regardless of this flag).
|
||||||
|
wip_title_tagging_enabled: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, nullable=False, default=True, server_default="true",
|
||||||
|
)
|
||||||
|
# Soft WIP title tier (#1474): also tag sketch/doodle/scribble titles, but with
|
||||||
|
# a PROVISIONAL source (`wip_title_soft`) that never trains the head, since these
|
||||||
|
# are lower-precision (a finished "sketch" isn't WIP). OFF by default — a lower-
|
||||||
|
# precision tier is opt-in (the ring-loud audit surfaces false positives).
|
||||||
|
wip_soft_title_tagging_enabled: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, nullable=False, default=False, server_default="false",
|
||||||
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
async def load(cls, session) -> ImportSettings:
|
async def load(cls, session) -> ImportSettings:
|
||||||
"""The singleton settings row (id=1), via an async session."""
|
"""The singleton settings row (id=1), via an async session."""
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ from sqlalchemy import (
|
|||||||
Integer,
|
Integer,
|
||||||
String,
|
String,
|
||||||
func,
|
func,
|
||||||
|
select,
|
||||||
)
|
)
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
@@ -85,12 +86,14 @@ class MLSettings(Base):
|
|||||||
Float, nullable=False, default=0.95
|
Float, nullable=False, default=0.95
|
||||||
)
|
)
|
||||||
# -- Presentation chrome auto-hide (#141) -------------------------------
|
# -- Presentation chrome auto-hide (#141) -------------------------------
|
||||||
# banner / editor screenshot auto-apply on the sweep with their OWN flat
|
# `banner` (chrome — clusters on UI, not content) auto-applies on the sweep
|
||||||
# threshold (decoupled from content-head graduation). Hiding is consequential
|
# with its OWN flat threshold (decoupled from content-head graduation) and is
|
||||||
# so it runs HIGH. `wip` is never auto-applied. When an image would be
|
# HIDDEN from the gallery. Hiding is consequential so it runs HIGH. When an
|
||||||
# auto-hidden but ALSO scores >= presentation_conflict_threshold on a content
|
# image would be auto-hidden but ALSO scores >= presentation_conflict_threshold
|
||||||
# head, it's still hidden but flagged for review (PresentationReview) instead
|
# on a content head, it's still hidden but flagged for review
|
||||||
# of buried silently. ON by default (opt-out); every auto-tag is reversible.
|
# (PresentationReview, mode='chrome') instead of buried silently. ON by default
|
||||||
|
# (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor
|
||||||
|
# screenshot` are no longer chrome — they went to the PROCESS path below.
|
||||||
presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
|
presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||||
Boolean, nullable=False, default=True
|
Boolean, nullable=False, default=True
|
||||||
)
|
)
|
||||||
@@ -100,6 +103,26 @@ class MLSettings(Base):
|
|||||||
presentation_conflict_threshold: Mapped[float] = mapped_column(
|
presentation_conflict_threshold: Mapped[float] = mapped_column(
|
||||||
Float, nullable=False, default=0.50
|
Float, nullable=False, default=0.50
|
||||||
)
|
)
|
||||||
|
# -- Process auto-apply (#1464) ----------------------------------------
|
||||||
|
# `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program
|
||||||
|
# screenshots that must stay OUT of head/CCIP training but, unlike chrome,
|
||||||
|
# remain VISIBLE in the gallery (operator 2026-07-12). They auto-apply on the
|
||||||
|
# sweep with their OWN flat threshold and a PROVISIONAL source (`process_auto`,
|
||||||
|
# in training_data._AUTO_SOURCES) so the head NEVER trains on its own output —
|
||||||
|
# it learns only from title (`wip_title`) + manual labels, which breaks the
|
||||||
|
# runaway loop. When a process tag would be applied but the image ALSO scores
|
||||||
|
# >= process_conflict_threshold on a content head, it's flagged for review
|
||||||
|
# (PresentationReview, mode='process') rather than silently marked. OFF by
|
||||||
|
# default — a new whole-library auto-tagger is opt-in; every auto-tag reversible.
|
||||||
|
process_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, nullable=False, default=False
|
||||||
|
)
|
||||||
|
process_auto_apply_threshold: Mapped[float] = mapped_column(
|
||||||
|
Float, nullable=False, default=0.90
|
||||||
|
)
|
||||||
|
process_conflict_threshold: Mapped[float] = mapped_column(
|
||||||
|
Float, nullable=False, default=0.50
|
||||||
|
)
|
||||||
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
||||||
# existing libraries keep their stored value until the operator re-embeds.
|
# existing libraries keep their stored value until the operator re-embeds.
|
||||||
embedder_model_version: Mapped[str] = mapped_column(
|
embedder_model_version: Mapped[str] = mapped_column(
|
||||||
@@ -190,3 +213,14 @@ class MLSettings(Base):
|
|||||||
updated_at: Mapped[datetime] = mapped_column(
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def load(cls, session) -> MLSettings:
|
||||||
|
"""The singleton settings row (id=1), via an async session. Mirrors
|
||||||
|
ImportSettings.load — the shared singleton-loader pattern."""
|
||||||
|
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def load_sync(cls, session) -> MLSettings:
|
||||||
|
"""The singleton settings row (id=1), via a sync session."""
|
||||||
|
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
|
||||||
|
|||||||
@@ -65,3 +65,15 @@ class PostAttachment(Base):
|
|||||||
captured_at: Mapped[datetime] = mapped_column(
|
captured_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def attachment_download_url(attachment_id: int) -> str:
|
||||||
|
"""The path that streams this attachment's bytes.
|
||||||
|
|
||||||
|
Both serializers that expose an attachment to the frontend
|
||||||
|
(`provenance_service`, `post_feed_service`) built this literal themselves,
|
||||||
|
so changing the route in `api/attachments.py` meant two edits and only one
|
||||||
|
would be remembered (#3072). `test_attachment_download_url` pins it against
|
||||||
|
the app's registered rule, so the drift is caught rather than trusted to.
|
||||||
|
"""
|
||||||
|
return f"/api/attachments/{attachment_id}/download"
|
||||||
|
|||||||
@@ -1,15 +1,17 @@
|
|||||||
"""PresentationReview — an auto-hidden presentation tag that ALSO looked like
|
"""PresentationReview — a system-tag the auto-apply sweep applied that ALSO looked
|
||||||
real content, flagged for operator review (milestone 141).
|
like real content, flagged for operator review (milestone 141 + #1464).
|
||||||
|
|
||||||
When the auto-apply sweep hides an image as chrome (banner / editor screenshot)
|
When a sweep applies a system tag but the image ALSO scores highly on a content
|
||||||
but the image ALSO scores highly on a content head, it still hides it but records
|
head, it still applies the tag but records this row so a review strip can surface
|
||||||
this row so the Hidden view can surface it ("⚠ also looks like <conflict tag>")
|
it ("⚠ also looks like <conflict tag>"). Two modes (#1464): 'chrome' (banner —
|
||||||
for a keep-hidden / un-hide decision. Resolved rows are pruned by retention.
|
image is HIDDEN, review is keep-hidden / un-hide) and 'process' (wip / editor
|
||||||
|
screenshot — image stays VISIBLE, review is confirm / remove-tag). Resolved rows
|
||||||
|
are pruned by retention.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import DateTime, Float, ForeignKey, func
|
from sqlalchemy import DateTime, Float, ForeignKey, String, func
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column
|
||||||
|
|
||||||
from .base import Base
|
from .base import Base
|
||||||
@@ -31,6 +33,12 @@ class PresentationReview(Base):
|
|||||||
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True
|
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True
|
||||||
)
|
)
|
||||||
conflict_score: Mapped[float] = mapped_column(Float, nullable=False)
|
conflict_score: Mapped[float] = mapped_column(Float, nullable=False)
|
||||||
|
# Which sweep flagged this (#1464): 'chrome' (banner, hidden) or 'process'
|
||||||
|
# (wip / editor screenshot, shown). Drives which review strip surfaces it and
|
||||||
|
# what "resolve" means (un-hide vs remove-tag). Existing rows backfill 'chrome'.
|
||||||
|
mode: Mapped[str] = mapped_column(
|
||||||
|
String(16), nullable=False, default="chrome", server_default="chrome"
|
||||||
|
)
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -43,14 +43,19 @@ class TagKind(StrEnum):
|
|||||||
# to keep historic tag rows queryable.
|
# to keep historic tag rows queryable.
|
||||||
|
|
||||||
|
|
||||||
# The seeded system tags (migration 0075). PRESENTATION tags additionally
|
# The seeded system tags (migration 0075). Two behavior groups (#1464):
|
||||||
# hide from whole-image similarity results — they cluster on UI chrome, not
|
# CHROME (banner): clusters on UI chrome, not content → HIDDEN from the default
|
||||||
# content. `wip` is real art: only the training pipelines exclude it.
|
# gallery + from similarity; auto-applied via the sweep's chrome mode.
|
||||||
|
# PROCESS (wip, editor screenshot): real-but-unfinished art / program screenshots
|
||||||
|
# → SHOWN in the gallery (operator 2026-07-12), but excluded from the Explore
|
||||||
|
# rabbit-hole; auto-applied via the sweep's process mode (provisional source,
|
||||||
|
# ring-loud review guard).
|
||||||
|
# ALL three are excluded from OTHER concepts' head/CCIP training (training-hygiene,
|
||||||
|
# keyed on is_system); a system tag's OWN head trains on them — that's what makes
|
||||||
|
# auto-flagging work.
|
||||||
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
|
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
|
||||||
PRESENTATION_SYSTEM_TAGS = ("banner", "editor screenshot")
|
CHROME_SYSTEM_TAGS = ("banner",)
|
||||||
# `wip` marks real-but-unfinished art. It's kept in the gallery's own "similar"
|
PROCESS_SYSTEM_TAGS = ("wip", "editor screenshot")
|
||||||
# results (#1274), but the Explore rabbit-hole opts to hide it (exclude_wip) so a
|
|
||||||
# browse doesn't keep surfacing work-in-progress (operator, 2026-07-08).
|
|
||||||
WIP_SYSTEM_TAG = "wip"
|
WIP_SYSTEM_TAG = "wip"
|
||||||
|
|
||||||
image_tag = Table(
|
image_tag = Table(
|
||||||
|
|||||||
@@ -48,6 +48,47 @@ log = logging.getLogger(__name__)
|
|||||||
_VIDEO_DURATION_UNKNOWN = -1.0
|
_VIDEO_DURATION_UNKNOWN = -1.0
|
||||||
|
|
||||||
|
|
||||||
|
# -- artist-cascade predicates (rule 93: ONE definition, preview + apply) ---
|
||||||
|
# project_artist_cascade (preview) and delete_artist_cascade (apply) both build
|
||||||
|
# their queries from these. The preview used to re-derive its own — which is how
|
||||||
|
# it came to count images and stay silent about posts and attachments while the
|
||||||
|
# apply destroyed both. Same failure shape as the 2026-06-08 fandom-tag
|
||||||
|
# deletion, where a re-implemented delete predicate diverged from the preview's.
|
||||||
|
# Returned as condition LISTS spread into `.where(*conds)`, matching
|
||||||
|
# _unused_tag_conditions / _bare_post_conditions below.
|
||||||
|
|
||||||
|
|
||||||
|
def _artist_images_conditions(artist_id: int) -> list:
|
||||||
|
"""Images the cascade deletes (rows AND their on-disk files)."""
|
||||||
|
return [ImageRecord.artist_id == artist_id]
|
||||||
|
|
||||||
|
|
||||||
|
def _artist_posts_conditions(artist_id: int) -> list:
|
||||||
|
"""Posts the cascade destroys. The apply never names these — post.artist_id
|
||||||
|
is ondelete=CASCADE, so Postgres takes them when the artist row goes — which
|
||||||
|
is exactly why the preview has to name them: an artist whose posts are
|
||||||
|
body-only (no images) otherwise previews as `images: 0` and reads as an
|
||||||
|
empty artist, while every captured body/description/external-link set is
|
||||||
|
destroyed."""
|
||||||
|
return [Post.artist_id == artist_id]
|
||||||
|
|
||||||
|
|
||||||
|
def _artist_attachments_conditions(artist_id: int) -> list:
|
||||||
|
"""Attachments the cascade deletes. Matched by artist_id OR by the owning
|
||||||
|
post's artist: artist_id is nullable (_capture_attachment leaves it NULL
|
||||||
|
when no artist resolved), so neither arm alone covers every row. The
|
||||||
|
sha-addressed blobs are NOT unlinked (one blob backs many rows) — these are
|
||||||
|
row counts, and the bytes are not part of this operation's footprint."""
|
||||||
|
return [
|
||||||
|
or_(
|
||||||
|
PostAttachment.artist_id == artist_id,
|
||||||
|
PostAttachment.post_id.in_(
|
||||||
|
select(Post.id).where(*_artist_posts_conditions(artist_id))
|
||||||
|
),
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||||
"""Read-only projection of what delete_artist_cascade would touch.
|
"""Read-only projection of what delete_artist_cascade would touch.
|
||||||
|
|
||||||
@@ -56,12 +97,17 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
|||||||
"artist": {"id": int, "name": str, "slug": str},
|
"artist": {"id": int, "name": str, "slug": str},
|
||||||
"projected": {
|
"projected": {
|
||||||
"images": int,
|
"images": int,
|
||||||
|
"posts": int, # hard-deleted by the post.artist_id CASCADE
|
||||||
|
"attachments": int, # rows deleted; the sha-addressed blobs stay
|
||||||
"sources": int,
|
"sources": int,
|
||||||
"thumbs": int, # images with a thumbnail_path set
|
"thumbs": int, # images with a thumbnail_path set
|
||||||
"import_tasks": int, # ImportTask rows referencing the artist's images
|
"import_tasks": int, # ImportTask rows referencing the artist's images
|
||||||
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
|
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
Every count is built from the shared `_artist_*_conditions` predicates the
|
||||||
|
apply uses, so the two halves cannot drift (rule 93).
|
||||||
|
|
||||||
Raises LookupError if slug not found. No mutations.
|
Raises LookupError if slug not found. No mutations.
|
||||||
"""
|
"""
|
||||||
from ..models.import_task import ImportTask
|
from ..models.import_task import ImportTask
|
||||||
@@ -73,36 +119,49 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
|||||||
if artist is None:
|
if artist is None:
|
||||||
raise LookupError(f"artist slug not found: {slug!r}")
|
raise LookupError(f"artist slug not found: {slug!r}")
|
||||||
|
|
||||||
|
images_conds = _artist_images_conditions(artist.id)
|
||||||
|
|
||||||
images_count = session.execute(
|
images_count = session.execute(
|
||||||
select(func.count(ImageRecord.id))
|
select(func.count(ImageRecord.id)).where(*images_conds)
|
||||||
.where(ImageRecord.artist_id == artist.id)
|
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
posts_count = session.execute(
|
||||||
|
select(func.count(Post.id))
|
||||||
|
.where(*_artist_posts_conditions(artist.id))
|
||||||
|
).scalar_one()
|
||||||
|
attachments_count = session.execute(
|
||||||
|
select(func.count(PostAttachment.id))
|
||||||
|
.where(*_artist_attachments_conditions(artist.id))
|
||||||
|
).scalar_one()
|
||||||
|
# Sources have no shared predicate: the apply never queries them either, it
|
||||||
|
# gets them from the Artist.sources ORM cascade. Counted directly here.
|
||||||
sources_count = session.execute(
|
sources_count = session.execute(
|
||||||
select(func.count(Source.id))
|
select(func.count(Source.id))
|
||||||
.where(Source.artist_id == artist.id)
|
.where(Source.artist_id == artist.id)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
thumbs_count = session.execute(
|
thumbs_count = session.execute(
|
||||||
select(func.count(ImageRecord.id))
|
select(func.count(ImageRecord.id))
|
||||||
.where(ImageRecord.artist_id == artist.id)
|
.where(*images_conds)
|
||||||
.where(ImageRecord.thumbnail_path.is_not(None))
|
.where(ImageRecord.thumbnail_path.is_not(None))
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
import_tasks_count = session.execute(
|
import_tasks_count = session.execute(
|
||||||
select(func.count(ImportTask.id))
|
select(func.count(ImportTask.id))
|
||||||
.where(
|
.where(
|
||||||
ImportTask.result_image_id.in_(
|
ImportTask.result_image_id.in_(
|
||||||
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
|
select(ImageRecord.id).where(*images_conds)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
bytes_on_disk = session.execute(
|
bytes_on_disk = session.execute(
|
||||||
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
|
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
|
||||||
.where(ImageRecord.artist_id == artist.id)
|
.where(*images_conds)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
||||||
"projected": {
|
"projected": {
|
||||||
"images": images_count,
|
"images": images_count,
|
||||||
|
"posts": posts_count,
|
||||||
|
"attachments": attachments_count,
|
||||||
"sources": sources_count,
|
"sources": sources_count,
|
||||||
"thumbs": thumbs_count,
|
"thumbs": thumbs_count,
|
||||||
"import_tasks": import_tasks_count,
|
"import_tasks": import_tasks_count,
|
||||||
@@ -277,6 +336,10 @@ def delete_artist_cascade(
|
|||||||
series_page / tag_suggestion_rejection from ImageRecord delete,
|
series_page / tag_suggestion_rejection from ImageRecord delete,
|
||||||
and source / post / download_event / etc. from Artist delete
|
and source / post / download_event / etc. from Artist delete
|
||||||
(via Artist.sources cascade="all, delete-orphan").
|
(via Artist.sources cascade="all, delete-orphan").
|
||||||
|
|
||||||
|
The artist's post_attachment rows are cleared EXPLICITLY before the
|
||||||
|
artist row goes — see the comment at that step; leaving them to the
|
||||||
|
cascade aborts the whole delete on a unique violation.
|
||||||
"""
|
"""
|
||||||
artist = session.get(Artist, artist_id)
|
artist = session.get(Artist, artist_id)
|
||||||
if artist is None:
|
if artist is None:
|
||||||
@@ -287,11 +350,22 @@ def delete_artist_cascade(
|
|||||||
"files_deleted": 0,
|
"files_deleted": 0,
|
||||||
"thumbs_deleted": 0,
|
"thumbs_deleted": 0,
|
||||||
"import_tasks_nulled": 0,
|
"import_tasks_nulled": 0,
|
||||||
|
"posts_deleted": 0,
|
||||||
|
"attachments_deleted": 0,
|
||||||
"files_failed": 0,
|
"files_failed": 0,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
|
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
|
||||||
|
|
||||||
|
# Counted BEFORE the delete: Postgres takes these via the post.artist_id
|
||||||
|
# CASCADE when the artist row goes, so afterwards there is nothing left to
|
||||||
|
# count. Reported so the summary can be checked against the preview's
|
||||||
|
# `posts` — the parity rule 93 asks for is only testable if both halves
|
||||||
|
# actually state the number.
|
||||||
|
posts_deleted = session.execute(
|
||||||
|
select(func.count(Post.id)).where(*_artist_posts_conditions(artist.id))
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
images_deleted = 0
|
images_deleted = 0
|
||||||
files_deleted = 0
|
files_deleted = 0
|
||||||
thumbs_deleted = 0
|
thumbs_deleted = 0
|
||||||
@@ -300,7 +374,7 @@ def delete_artist_cascade(
|
|||||||
while True:
|
while True:
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
select(ImageRecord)
|
select(ImageRecord)
|
||||||
.where(ImageRecord.artist_id == artist.id)
|
.where(*_artist_images_conditions(artist.id))
|
||||||
.limit(500)
|
.limit(500)
|
||||||
).scalars().all()
|
).scalars().all()
|
||||||
if not rows:
|
if not rows:
|
||||||
@@ -323,6 +397,28 @@ def delete_artist_cascade(
|
|||||||
# source_path_prefix matching that's out of scope here.
|
# source_path_prefix matching that's out of scope here.
|
||||||
import_tasks_nulled = 0
|
import_tasks_nulled = 0
|
||||||
|
|
||||||
|
# Clear the artist's attachments BEFORE the artist row, or the delete below
|
||||||
|
# aborts. Deleting an artist CASCADEs to Post (post.artist_id is
|
||||||
|
# ondelete=CASCADE), which SET NULLs post_attachment.post_id — and
|
||||||
|
# `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE
|
||||||
|
# WHERE post_id IS NULL, so any two of this artist's attachments sharing a
|
||||||
|
# sha collapse onto one another and raise. That is an ORDINARY shape, not a
|
||||||
|
# corrupt one: _capture_attachment deliberately writes one row per post over
|
||||||
|
# a single sha-addressed blob (a creator who attaches the same pdf to two
|
||||||
|
# posts has two rows), and a pre-existing filesystem-import row with the same
|
||||||
|
# sha and a NULL post_id collides on its own. Migration 0043 reasoned only
|
||||||
|
# about upgrade-time safety and never about this later SET NULL.
|
||||||
|
# _repoint_post_links guards the identical collision class in the reconcile
|
||||||
|
# path; this is its artist-cascade counterpart.
|
||||||
|
#
|
||||||
|
# Which rows count as the artist's — and why the blobs are left on disk —
|
||||||
|
# is _artist_attachments_conditions, shared with the preview.
|
||||||
|
attachments_deleted = session.execute(
|
||||||
|
delete(PostAttachment)
|
||||||
|
.where(*_artist_attachments_conditions(artist.id))
|
||||||
|
).rowcount or 0
|
||||||
|
session.commit()
|
||||||
|
|
||||||
session.delete(artist)
|
session.delete(artist)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
@@ -333,6 +429,8 @@ def delete_artist_cascade(
|
|||||||
"files_deleted": files_deleted,
|
"files_deleted": files_deleted,
|
||||||
"thumbs_deleted": thumbs_deleted,
|
"thumbs_deleted": thumbs_deleted,
|
||||||
"import_tasks_nulled": import_tasks_nulled,
|
"import_tasks_nulled": import_tasks_nulled,
|
||||||
|
"posts_deleted": posts_deleted,
|
||||||
|
"attachments_deleted": attachments_deleted,
|
||||||
"files_failed": files_failed,
|
"files_failed": files_failed,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1494,3 +1592,155 @@ def purge_gated_previews(
|
|||||||
"ledger_cleared": ledger_cleared,
|
"ledger_cleared": ledger_cleared,
|
||||||
"posts_deleted": posts_deleted,
|
"posts_deleted": posts_deleted,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# -- orphaned attachment reclamation ---------------------------------------
|
||||||
|
# PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or
|
||||||
|
# artist leaves the row behind rather than taking it. Nothing ever pruned those
|
||||||
|
# rows, and nothing has ever unlinked a file under the attachment store — so
|
||||||
|
# both rows and bytes accumulated permanently and were invisible to every
|
||||||
|
# existing diagnostic.
|
||||||
|
#
|
||||||
|
# Why this is a DISK->DB reconciliation rather than a row sweep: the store is
|
||||||
|
# sha-addressed and idempotent (attachment_store.store), so ONE blob backs MANY
|
||||||
|
# rows. Deleting a row therefore does not free its blob, and — since the artist
|
||||||
|
# cascade now deletes its attachment rows outright — a freed blob has no DB
|
||||||
|
# pointer left to find it by. Walking the store and asking "does any row still
|
||||||
|
# reference this sha?" catches orphans from every cause, including ones no
|
||||||
|
# future delete path will think to report.
|
||||||
|
|
||||||
|
# A blob is written by attachment_store.store BEFORE its row is inserted and
|
||||||
|
# committed, so a just-stored file legitimately has no referencing row for a
|
||||||
|
# moment. Same guard, same reasoning as ORPHAN_TEMP_MIN_AGE_HOURS in
|
||||||
|
# tasks/maintenance.py: never judge a file younger than this.
|
||||||
|
_ATTACHMENT_ORPHAN_MIN_AGE_HOURS = 6
|
||||||
|
|
||||||
|
# Wall-clock budget for the store walk (rule 89). A library with a large
|
||||||
|
# attachment store shouldn't be able to run this past its soft time limit; on
|
||||||
|
# exhaustion it reports partial=True and the operator re-runs to finish.
|
||||||
|
_ATTACHMENT_RECLAIM_BUDGET_SECONDS = 900
|
||||||
|
|
||||||
|
# The store names files `<sha256><ext>`. Parse the sha as the first 64 chars
|
||||||
|
# rather than via Path.stem: store() takes the extension straight from the
|
||||||
|
# source filename, and a URL-encoded basename yields a multi-dot "suffix"
|
||||||
|
# (see [[reference_url_encoded_basename_suffix]]) that would make stem eat part
|
||||||
|
# of the sha. Validating the 64 chars as hex also skips anything else in the
|
||||||
|
# tree that isn't a stored blob.
|
||||||
|
_SHA256_HEX_LEN = 64
|
||||||
|
|
||||||
|
|
||||||
|
def _orphan_attachment_conditions() -> list:
|
||||||
|
"""PostAttachment rows belonging to nothing: both FKs nulled by a deleted
|
||||||
|
post AND a deleted artist. A row with post_id NULL but an artist_id is the
|
||||||
|
deliberate filesystem-import case (importer._capture_attachment writes it
|
||||||
|
that way) and is NOT an orphan — it is still attributed."""
|
||||||
|
return [
|
||||||
|
PostAttachment.post_id.is_(None),
|
||||||
|
PostAttachment.artist_id.is_(None),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _is_sha_named(name: str) -> bool:
|
||||||
|
"""True when `name` starts with a 64-char lowercase-hex sha256."""
|
||||||
|
if len(name) < _SHA256_HEX_LEN:
|
||||||
|
return False
|
||||||
|
head = name[:_SHA256_HEX_LEN]
|
||||||
|
return all(c in "0123456789abcdef" for c in head)
|
||||||
|
|
||||||
|
|
||||||
|
def reclaim_orphaned_attachments(
|
||||||
|
session: Session, *, images_root: Path, dry_run: bool = False,
|
||||||
|
) -> dict:
|
||||||
|
"""Prune unattributed PostAttachment rows, then unlink store blobs that no
|
||||||
|
surviving row references.
|
||||||
|
|
||||||
|
Returns (same discovery keys either way, so the UI renders one shape):
|
||||||
|
{"rows": int, # orphan rows found / deleted
|
||||||
|
"files": int, # unreferenced blobs found / unlinked
|
||||||
|
"bytes": int, # their total size
|
||||||
|
"scanned": int, # blobs examined
|
||||||
|
"skipped_recent": int, # blobs under the min-age guard
|
||||||
|
"files_failed": int, # unlink raised (apply only)
|
||||||
|
"partial": bool} # walk hit the time budget
|
||||||
|
|
||||||
|
dry_run computes exactly what the apply would do and mutates nothing — the
|
||||||
|
surviving-sha set is derived by NEGATING the same orphan predicate the
|
||||||
|
delete uses, so the preview cannot disagree with the apply (rule 93).
|
||||||
|
"""
|
||||||
|
started = time.monotonic()
|
||||||
|
orphan_conds = _orphan_attachment_conditions()
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
rows = session.execute(
|
||||||
|
select(func.count(PostAttachment.id)).where(*orphan_conds)
|
||||||
|
).scalar_one()
|
||||||
|
else:
|
||||||
|
rows = session.execute(
|
||||||
|
delete(PostAttachment).where(*orphan_conds)
|
||||||
|
).rowcount or 0
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
# Shas that still have a home. In the apply path the orphan rows are already
|
||||||
|
# gone, so `NOT orphan` is redundant but harmless; in the dry-run path it is
|
||||||
|
# what makes the projection honest about blobs the delete would free. One
|
||||||
|
# predicate, one query, both modes.
|
||||||
|
surviving_shas = set(session.execute(
|
||||||
|
select(PostAttachment.sha256).where(~and_(*orphan_conds)).distinct()
|
||||||
|
).scalars())
|
||||||
|
|
||||||
|
root = Path(images_root) / "attachments"
|
||||||
|
cutoff = (
|
||||||
|
datetime.now(UTC).timestamp()
|
||||||
|
- _ATTACHMENT_ORPHAN_MIN_AGE_HOURS * 3600
|
||||||
|
)
|
||||||
|
files = 0
|
||||||
|
freed_bytes = 0
|
||||||
|
scanned = 0
|
||||||
|
skipped_recent = 0
|
||||||
|
files_failed = 0
|
||||||
|
partial = False
|
||||||
|
|
||||||
|
if root.is_dir():
|
||||||
|
for path in root.rglob("*"):
|
||||||
|
if time.monotonic() - started >= _ATTACHMENT_RECLAIM_BUDGET_SECONDS:
|
||||||
|
partial = True
|
||||||
|
break
|
||||||
|
# .partial staging files belong to cleanup_orphaned_temp_files —
|
||||||
|
# leave them alone rather than racing an in-flight store().
|
||||||
|
if path.suffix in (".part", ".partial") or not path.is_file():
|
||||||
|
continue
|
||||||
|
if not _is_sha_named(path.name):
|
||||||
|
continue
|
||||||
|
scanned += 1
|
||||||
|
sha = path.name[:_SHA256_HEX_LEN]
|
||||||
|
if sha in surviving_shas:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
st = path.stat()
|
||||||
|
if st.st_mtime >= cutoff:
|
||||||
|
skipped_recent += 1
|
||||||
|
continue
|
||||||
|
size = st.st_size
|
||||||
|
if not dry_run:
|
||||||
|
path.unlink()
|
||||||
|
files += 1
|
||||||
|
freed_bytes += size
|
||||||
|
except OSError as exc:
|
||||||
|
files_failed += 1
|
||||||
|
log.warning("reclaim_orphaned_attachments: %s: %s", path, exc)
|
||||||
|
|
||||||
|
if not dry_run and (rows or files):
|
||||||
|
log.info(
|
||||||
|
"attachment reclaim: %d orphan row(s) deleted, %d blob(s) unlinked "
|
||||||
|
"(%d bytes), %d failed, partial=%s",
|
||||||
|
rows, files, freed_bytes, files_failed, partial,
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"rows": rows,
|
||||||
|
"files": files,
|
||||||
|
"bytes": freed_bytes,
|
||||||
|
"scanned": scanned,
|
||||||
|
"skipped_recent": skipped_recent,
|
||||||
|
"files_failed": files_failed,
|
||||||
|
"partial": partial,
|
||||||
|
}
|
||||||
|
|||||||
@@ -181,7 +181,7 @@ def _augment_cookies(platform: str, netscape: str) -> str:
|
|||||||
"""Delegate to the platform's `augment_cookies` hook if one is
|
"""Delegate to the platform's `augment_cookies` hook if one is
|
||||||
registered (subscribestar, hentaifoundry, etc. — see
|
registered (subscribestar, hentaifoundry, etc. — see
|
||||||
`services/platforms/<name>.py`). No-op when the platform doesn't
|
`services/platforms/<name>.py`). No-op when the platform doesn't
|
||||||
register a hook (Patreon, DeviantArt). Centralizing the
|
register a hook (Patreon, Discord). Centralizing the
|
||||||
quirks-per-platform in the platforms package means adding a new
|
quirks-per-platform in the platforms package means adding a new
|
||||||
platform's cookie quirks doesn't require touching this file."""
|
platform's cookie quirks doesn't require touching this file."""
|
||||||
info = PLATFORMS.get(platform)
|
info = PLATFORMS.get(platform)
|
||||||
|
|||||||
@@ -31,9 +31,8 @@ from .pixiv_ingester import PixivIngester
|
|||||||
from .subscribestar_ingester import SubscribeStarIngester
|
from .subscribestar_ingester import SubscribeStarIngester
|
||||||
|
|
||||||
# Platforms whose download + verify go through the native ingester rather than
|
# Platforms whose download + verify go through the native ingester rather than
|
||||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord,
|
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
|
||||||
# deviantart — the latter slated for retirement, not migration) until they
|
# they migrate too.
|
||||||
# migrate too.
|
|
||||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
|
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
|
||||||
|
|
||||||
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
||||||
|
|||||||
@@ -35,9 +35,14 @@ class InvalidUrlError(Exception):
|
|||||||
# reviewers catch drift.
|
# reviewers catch drift.
|
||||||
_PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
_PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||||
("patreon", re.compile(
|
("patreon", re.compile(
|
||||||
|
# Three creator URL shapes — bare (patreon.com/Atole), `c/`, and `cw/`
|
||||||
|
# (the "creator workspace" URL served once subscribed, see
|
||||||
|
# patreon_resolver._VANITY_RE). A trailing sub-path is allowed so a
|
||||||
|
# creator's inner page still derives the slug. Nav pages stay excluded.
|
||||||
r"^https?://(?:www\.)?patreon\.com/"
|
r"^https?://(?:www\.)?patreon\.com/"
|
||||||
r"(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c/)"
|
r"(?:cw/|c/)?"
|
||||||
r"(?P<slug>[^/?#]+)/?$",
|
r"(?!(?:home|search|messages|notifications|library|settings|posts)(?:[/?#]|$))"
|
||||||
|
r"(?P<slug>[^/?#]+)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
)),
|
)),
|
||||||
("subscribestar", re.compile(
|
("subscribestar", re.compile(
|
||||||
@@ -50,12 +55,6 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
|||||||
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
|
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
)),
|
)),
|
||||||
("deviantart", re.compile(
|
|
||||||
r"^https?://(?:www\.)?deviantart\.com/"
|
|
||||||
r"(?!home$|watch\b|tag\b|browse\b)"
|
|
||||||
r"(?P<slug>[^/?#]+)/?$",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)),
|
|
||||||
("pixiv", re.compile(
|
("pixiv", re.compile(
|
||||||
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
|
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
|
||||||
re.IGNORECASE,
|
re.IGNORECASE,
|
||||||
|
|||||||
@@ -299,8 +299,9 @@ class GalleryDLService:
|
|||||||
# (services/patreon_ingester.py), not gallery-dl.
|
# (services/patreon_ingester.py), not gallery-dl.
|
||||||
PLATFORM_DEFAULTS = {
|
PLATFORM_DEFAULTS = {
|
||||||
# subscribestar removed — native-ingester platform now (#71); pixiv
|
# subscribestar removed — native-ingester platform now (#71); pixiv
|
||||||
# removed likewise (#129). The remaining entries are the gallery-dl
|
# removed likewise (#129); deviantart removed at #3069 as a dropped
|
||||||
# platforms not yet migrated.
|
# platform, not a migrated one. The remaining entries are the
|
||||||
|
# gallery-dl platforms not yet migrated.
|
||||||
"hentaifoundry": {
|
"hentaifoundry": {
|
||||||
"content_types": ["all"],
|
"content_types": ["all"],
|
||||||
"directory": [],
|
"directory": [],
|
||||||
@@ -316,15 +317,6 @@ class GalleryDLService:
|
|||||||
"reactions": False,
|
"reactions": False,
|
||||||
"threads": True,
|
"threads": True,
|
||||||
},
|
},
|
||||||
"deviantart": {
|
|
||||||
"content_types": ["all"],
|
|
||||||
"directory": [],
|
|
||||||
"filename": "{index:>03}_{title[:50]}.{extension}",
|
|
||||||
"flat": True,
|
|
||||||
"original": True,
|
|
||||||
"mature": True,
|
|
||||||
"metadata": True,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ from ..models import (
|
|||||||
Tag,
|
Tag,
|
||||||
TagPositiveConfirmation,
|
TagPositiveConfirmation,
|
||||||
)
|
)
|
||||||
from ..models.tag import PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG, image_tag
|
from ..models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
|
||||||
from .pagination import decode_cursor, encode_cursor
|
from .pagination import decode_cursor, encode_cursor
|
||||||
from .tag_query import (
|
from .tag_query import (
|
||||||
fandom_join_alias,
|
fandom_join_alias,
|
||||||
@@ -396,6 +396,25 @@ def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
|
|||||||
return [kept[i] for i in order]
|
return [kept[i] for i in order]
|
||||||
|
|
||||||
|
|
||||||
|
def _reach_sample(rows, limit, reach):
|
||||||
|
"""From a distance-sorted candidate pool (nearest first), pick a spread of ranks
|
||||||
|
that MIXES near (tag the current cluster) and mid-far (escape it) BEFORE dedup +
|
||||||
|
MMR — the Explore "reach" dial (#1476).
|
||||||
|
|
||||||
|
reach in (0, 1]: the sampled span grows outward from the anchor (0.25→1.0 of the
|
||||||
|
pool), evenly strided from rank 0 so the nearest are still represented. In a
|
||||||
|
dense signature the nearest ranks are near-identical, so reaching farther is the
|
||||||
|
only way to hand MMR genuinely different content — MMR alone can't escape a pool
|
||||||
|
that's already all-near. reach<=0 or a small pool passes through unchanged."""
|
||||||
|
n = len(rows)
|
||||||
|
want = max(limit * 8, 100)
|
||||||
|
if reach <= 0 or n <= want:
|
||||||
|
return rows
|
||||||
|
span = int(min(1.0, 0.25 + 0.75 * reach) * n)
|
||||||
|
idx = sorted({min(int(i * span / want), n - 1) for i in range(want)})
|
||||||
|
return [rows[i] for i in idx]
|
||||||
|
|
||||||
|
|
||||||
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
||||||
"""Map image_id -> {"name","slug"} via the canonical
|
"""Map image_id -> {"name","slug"} via the canonical
|
||||||
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
||||||
@@ -419,16 +438,17 @@ class GalleryService:
|
|||||||
async def _hidden_tag_ids(
|
async def _hidden_tag_ids(
|
||||||
self, include_hidden, tag_ids, tag_or_groups,
|
self, include_hidden, tag_ids, tag_or_groups,
|
||||||
) -> list[int] | None:
|
) -> list[int] | None:
|
||||||
"""Presentation-chrome tag ids to implicitly exclude from a gallery query,
|
"""Chrome (banner) tag ids to implicitly exclude from a gallery query, or
|
||||||
or None. None when the caller asked to include hidden, when the operator
|
None. None when the caller asked to include hidden, when the operator is
|
||||||
is explicitly filtering FOR a presentation tag (they clearly want to see
|
explicitly filtering FOR a chrome tag (they clearly want to see it), or when
|
||||||
it), or when no presentation tags exist. (milestone 141)"""
|
no chrome tags exist. (milestone 141; #1464: editor screenshot is now PROCESS
|
||||||
|
— shown — so only `banner` hides here.)"""
|
||||||
if include_hidden:
|
if include_hidden:
|
||||||
return None
|
return None
|
||||||
rows = await self.session.execute(
|
rows = await self.session.execute(
|
||||||
select(Tag.id).where(
|
select(Tag.id).where(
|
||||||
Tag.is_system.is_(True),
|
Tag.is_system.is_(True),
|
||||||
Tag.name.in_(PRESENTATION_SYSTEM_TAGS),
|
Tag.name.in_(CHROME_SYSTEM_TAGS),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
pres = [r[0] for r in rows]
|
pres = [r[0] for r in rows]
|
||||||
@@ -716,6 +736,7 @@ class GalleryService:
|
|||||||
untagged: bool = False, no_artist: bool = False,
|
untagged: bool = False, no_artist: bool = False,
|
||||||
date_from: datetime | None = None, date_to: datetime | None = None,
|
date_from: datetime | None = None, date_to: datetime | None = None,
|
||||||
exclude_wip: bool = False,
|
exclude_wip: bool = False,
|
||||||
|
reach: float = 0.0, exclude_ids: list[int] | None = None,
|
||||||
) -> list[GalleryImage] | None:
|
) -> list[GalleryImage] | None:
|
||||||
"""Visual "more like this": images near `image_id`'s SigLIP embedding
|
"""Visual "more like this": images near `image_id`'s SigLIP embedding
|
||||||
(pgvector, HNSW-indexed — alembic 0036), then DIVERSIFIED so the result
|
(pgvector, HNSW-indexed — alembic 0036), then DIVERSIFIED so the result
|
||||||
@@ -744,20 +765,27 @@ class GalleryService:
|
|||||||
# wide pool there's nothing but the near-dupes to choose from. Widened
|
# wide pool there's nothing but the near-dupes to choose from. Widened
|
||||||
# (5×→8×, cap 200→400) so the stronger MMR has genuinely distinct
|
# (5×→8×, cap 200→400) so the stronger MMR has genuinely distinct
|
||||||
# neighbourhoods to reach into for more variance (operator, 2026-07-01).
|
# neighbourhoods to reach into for more variance (operator, 2026-07-01).
|
||||||
|
# Explore's reach>0 (#1476) widens it a LOT more: in a dense signature the
|
||||||
|
# nearest few hundred are all near-identical, so far-enough candidates only
|
||||||
|
# exist deeper in the ranked pool. _reach_sample then strides across them.
|
||||||
|
if reach > 0:
|
||||||
|
pool_n = min(1000, max(limit * 25, 100))
|
||||||
|
else:
|
||||||
pool_n = min(400, max(limit * 8, 100))
|
pool_n = min(400, max(limit * 8, 100))
|
||||||
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
|
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
|
||||||
eff = _effective_date_col()
|
eff = _effective_date_col()
|
||||||
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
||||||
stmt = _outer_join_primary_post(stmt)
|
stmt = _outer_join_primary_post(stmt)
|
||||||
# Presentation images (banner / editor-screenshot system tags, #128)
|
# Chrome (banner, #128) clusters on UI rather than content, so near any one
|
||||||
# cluster on UI chrome rather than content, so near any one of them
|
# of them they'd fill the grid → excluded from CANDIDATES always (the anchor
|
||||||
# they'd fill the grid. Excluded from CANDIDATES only — the anchor
|
# itself may be a banner). PROCESS art (wip / editor screenshot) stays
|
||||||
# itself may be a banner. `wip` stays surfaced here by default (real art;
|
# surfaced here by default (real content; only the training pipelines exclude
|
||||||
# only the training pipelines exclude it), but the Explore rabbit-hole
|
# it), but the Explore rabbit-hole passes exclude_wip to also drop the whole
|
||||||
# passes exclude_wip to also drop work-in-progress (operator, 2026-07-08).
|
# process group so a browse doesn't keep surfacing work-in-progress
|
||||||
excluded_system_tags = PRESENTATION_SYSTEM_TAGS
|
# (operator, 2026-07-08; #1464 — editor now rides with wip here).
|
||||||
|
excluded_system_tags = CHROME_SYSTEM_TAGS
|
||||||
if exclude_wip:
|
if exclude_wip:
|
||||||
excluded_system_tags = (*PRESENTATION_SYSTEM_TAGS, WIP_SYSTEM_TAG)
|
excluded_system_tags = (*CHROME_SYSTEM_TAGS, *PROCESS_SYSTEM_TAGS)
|
||||||
presentation = (
|
presentation = (
|
||||||
select(image_tag.c.image_record_id)
|
select(image_tag.c.image_record_id)
|
||||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||||
@@ -771,6 +799,10 @@ class GalleryService:
|
|||||||
ImageRecord.id != image_id,
|
ImageRecord.id != image_id,
|
||||||
ImageRecord.id.not_in(presentation),
|
ImageRecord.id.not_in(presentation),
|
||||||
)
|
)
|
||||||
|
# Anti-revisit (#1476): the Explore walk passes its breadcrumb so already-
|
||||||
|
# walked images aren't re-served as neighbours — → can't loop you back in.
|
||||||
|
if exclude_ids:
|
||||||
|
stmt = stmt.where(ImageRecord.id.not_in(exclude_ids))
|
||||||
stmt = _apply_scope(
|
stmt = _apply_scope(
|
||||||
stmt, tag_ids=tag_ids, post_id=None,
|
stmt, tag_ids=tag_ids, post_id=None,
|
||||||
artist_id=artist_id, media_type=media_type,
|
artist_id=artist_id, media_type=media_type,
|
||||||
@@ -780,6 +812,10 @@ class GalleryService:
|
|||||||
)
|
)
|
||||||
stmt = stmt.order_by(distance.asc()).limit(pool_n)
|
stmt = stmt.order_by(distance.asc()).limit(pool_n)
|
||||||
rows = (await self.session.execute(stmt)).all()
|
rows = (await self.session.execute(stmt)).all()
|
||||||
|
# Explore reach: stride across an outward-growing distance span so the pool
|
||||||
|
# handed to MMR spans near→mid-far, not just the tight cluster (#1476).
|
||||||
|
if reach > 0:
|
||||||
|
rows = _reach_sample(rows, limit, reach)
|
||||||
rows = _diversify_similar(src, rows, limit)
|
rows = _diversify_similar(src, rows, limit)
|
||||||
artists = await _artists_for(self.session, [r[0].id for r in rows])
|
artists = await _artists_for(self.session, [r[0].id for r in rows])
|
||||||
return _gallery_images(rows, artists)
|
return _gallery_images(rows, artists)
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
"""Bulk, idempotent writes to the ``image_tag`` association table.
|
||||||
|
|
||||||
|
Three writers attach tags to images in bulk: the WIP-title backfill
|
||||||
|
(`wip_title.apply_wip_image_tags`), the concept-head auto-apply sweep and the
|
||||||
|
system-tag auto-apply sweep (both in `ml/heads.py`). The two sweeps used to
|
||||||
|
issue ONE INSERT PER ROW from inside their per-image loop — fine in steady
|
||||||
|
state, but a first pass over a back-catalogue is tens of thousands of
|
||||||
|
individual round-trips (#3072). All three share this one chunked multi-row
|
||||||
|
insert now.
|
||||||
|
|
||||||
|
Sync only: every caller runs on a sync ``Session`` (the Celery task path). No
|
||||||
|
async service writes image_tag in bulk, so there is no async sibling to keep in
|
||||||
|
step — unlike `db_helpers.get_or_create`, which does have one.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from ..models.tag import image_tag
|
||||||
|
|
||||||
|
# 5000 rows x 3 bound params = 15000, comfortably inside Postgres' 65535-param
|
||||||
|
# ceiling for a single statement. Raising this past ~21000 rows would exceed it.
|
||||||
|
INSERT_CHUNK = 5000
|
||||||
|
|
||||||
|
|
||||||
|
def insert_image_tags(
|
||||||
|
session: Session, rows: list[dict], *, chunk: int = INSERT_CHUNK
|
||||||
|
) -> None:
|
||||||
|
"""Attach ``rows`` to their images, skipping any tag already on one.
|
||||||
|
|
||||||
|
Each row is ``{"image_record_id": int, "tag_id": int, "source": str}``.
|
||||||
|
Does NOT commit — the caller owns the transaction.
|
||||||
|
|
||||||
|
ON CONFLICT DO NOTHING against the (image_record_id, tag_id) primary key,
|
||||||
|
so an existing tag keeps its ORIGINAL ``source``: re-running a sweep can
|
||||||
|
never re-stamp a tag the operator applied by hand as machine-applied.
|
||||||
|
|
||||||
|
Returns nothing on purpose. psycopg reports ``rowcount`` -1 for a multi-row
|
||||||
|
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so a count
|
||||||
|
taken from the statement would be a lie rather than an approximation.
|
||||||
|
Callers that need an accurate count derive it themselves — see
|
||||||
|
`wip_title.apply_wip_image_tags`' pre-SELECT, and the sweeps' `skip` sets.
|
||||||
|
"""
|
||||||
|
for start in range(0, len(rows), chunk):
|
||||||
|
session.execute(
|
||||||
|
pg_insert(image_tag)
|
||||||
|
.values(rows[start:start + chunk])
|
||||||
|
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
|
||||||
|
)
|
||||||
@@ -47,9 +47,21 @@ from .attachment_store import AttachmentStore
|
|||||||
from .audits import single_color
|
from .audits import single_color
|
||||||
from .link_extract import extract_external_links
|
from .link_extract import extract_external_links
|
||||||
from .thumbnailer import Thumbnailer
|
from .thumbnailer import Thumbnailer
|
||||||
|
from .wip_title import (
|
||||||
|
WIP_TITLE_SOFT_SOURCE,
|
||||||
|
WIP_TITLE_SOURCE,
|
||||||
|
apply_wip_image_tags,
|
||||||
|
matches_soft_wip_title,
|
||||||
|
matches_wip_title,
|
||||||
|
resolve_wip_tag_id,
|
||||||
|
)
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Sentinel for the lazily-resolved wip tag id (distinguishes "not resolved yet"
|
||||||
|
# from a genuine None = tag absent, so absence is cached and not re-queried).
|
||||||
|
_UNSET = object()
|
||||||
|
|
||||||
|
|
||||||
class SkipReason(StrEnum):
|
class SkipReason(StrEnum):
|
||||||
too_small = "too_small"
|
too_small = "too_small"
|
||||||
@@ -183,6 +195,10 @@ class Importer:
|
|||||||
# invalidated mid-Importer (Importer instances are per-task /
|
# invalidated mid-Importer (Importer instances are per-task /
|
||||||
# per-archive-import so cross-instance staleness is harmless).
|
# per-archive-import so cross-instance staleness is harmless).
|
||||||
self._phash_candidates: list[tuple] | None = None
|
self._phash_candidates: list[tuple] | None = None
|
||||||
|
# Lazily-resolved `wip` system tag id for title-based WIP auto-tagging
|
||||||
|
# (task #1458). Sentinel _UNSET so a genuine None (tag absent) is cached
|
||||||
|
# and not re-queried per media. Importer is per-task, so this can't stale.
|
||||||
|
self._wip_tag_id: int | None = _UNSET
|
||||||
|
|
||||||
def _phash_candidates_cache(self) -> list[tuple]:
|
def _phash_candidates_cache(self) -> list[tuple]:
|
||||||
"""Cached `(phash, width, height, id)` rows from image_record.
|
"""Cached `(phash, width, height, id)` rows from image_record.
|
||||||
@@ -933,6 +949,10 @@ class Importer:
|
|||||||
# Thumbnail is queued separately by the calling task; the importer
|
# Thumbnail is queued separately by the calling task; the importer
|
||||||
# does not generate thumbnails inline so the import queue stays moving.
|
# does not generate thumbnails inline so the import queue stays moving.
|
||||||
|
|
||||||
|
# Title-based WIP auto-tag (task #1458): fresh import only, after the
|
||||||
|
# sidecar has linked the post so record.primary_post_id / its title exist.
|
||||||
|
self._maybe_apply_wip_title(record)
|
||||||
|
|
||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="imported", image_id=record.id)
|
return ImportResult(status="imported", image_id=record.id)
|
||||||
|
|
||||||
@@ -976,6 +996,47 @@ class Importer:
|
|||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="refreshed", image_id=existing.id)
|
return ImportResult(status="refreshed", image_id=existing.id)
|
||||||
|
|
||||||
|
def _maybe_apply_wip_title(self, record: ImageRecord) -> None:
|
||||||
|
"""Auto-apply the `wip` system tag to a FRESHLY-imported image when its
|
||||||
|
primary post's TITLE explicitly declares work-in-progress (task #1458 —
|
||||||
|
the artist's own "WIP" / "work in progress" label).
|
||||||
|
|
||||||
|
Called ONLY from the two new-record paths (never deep-scan / supersede),
|
||||||
|
so a manually-removed WIP tag is never re-applied by a routine re-scan —
|
||||||
|
removal sticks. The existing catalogue is covered separately by the
|
||||||
|
operator-triggered backfill sweep. Gated by the settings toggle, and
|
||||||
|
best-effort: any failure is logged, never allowed to fail the import."""
|
||||||
|
hard_on = self.settings.wip_title_tagging_enabled
|
||||||
|
soft_on = self.settings.wip_soft_title_tagging_enabled
|
||||||
|
if not (hard_on or soft_on):
|
||||||
|
return
|
||||||
|
if record.primary_post_id is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
title = self.session.execute(
|
||||||
|
select(Post.post_title).where(Post.id == record.primary_post_id)
|
||||||
|
).scalar_one_or_none()
|
||||||
|
# HARD tier ("WIP"/"work in progress") wins — higher precision, and it
|
||||||
|
# trains the head; SOFT (sketch/doodle, #1474) is the provisional fallback
|
||||||
|
# that never trains (source wip_title_soft).
|
||||||
|
if hard_on and matches_wip_title(title):
|
||||||
|
source = WIP_TITLE_SOURCE
|
||||||
|
elif soft_on and matches_soft_wip_title(title):
|
||||||
|
source = WIP_TITLE_SOFT_SOURCE
|
||||||
|
else:
|
||||||
|
return
|
||||||
|
if self._wip_tag_id is _UNSET:
|
||||||
|
self._wip_tag_id = resolve_wip_tag_id(self.session)
|
||||||
|
if self._wip_tag_id is None:
|
||||||
|
return
|
||||||
|
apply_wip_image_tags(
|
||||||
|
self.session, [record.id], self._wip_tag_id, source=source
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001 — a tag must never fail an import
|
||||||
|
log.warning(
|
||||||
|
"wip-title auto-tag failed for image %s: %s", record.id, exc
|
||||||
|
)
|
||||||
|
|
||||||
def _apply_post_fields(self, post: Post, sd) -> None:
|
def _apply_post_fields(self, post: Post, sd) -> None:
|
||||||
"""Write a parsed sidecar's post-level fields onto a Post — the SINGLE
|
"""Write a parsed sidecar's post-level fields onto a Post — the SINGLE
|
||||||
predicate shared by BOTH ingest paths: the per-media path (_apply_sidecar)
|
predicate shared by BOTH ingest paths: the per-media path (_apply_sidecar)
|
||||||
@@ -1253,6 +1314,10 @@ class Importer:
|
|||||||
# per-post Source row.
|
# per-post Source row.
|
||||||
self._apply_sidecar(record, path, artist, explicit_source=source)
|
self._apply_sidecar(record, path, artist, explicit_source=source)
|
||||||
|
|
||||||
|
# Title-based WIP auto-tag (task #1458): fresh import only, see the
|
||||||
|
# matching call in _import_media.
|
||||||
|
self._maybe_apply_wip_title(record)
|
||||||
|
|
||||||
self.session.commit()
|
self.session.commit()
|
||||||
return ImportResult(status="imported", image_id=record.id)
|
return ImportResult(status="imported", image_id=record.id)
|
||||||
|
|
||||||
|
|||||||
@@ -150,9 +150,7 @@ def refresh_character_prototypes(
|
|||||||
"""Incrementally refresh the prototype store. `full=True` rebuilds every
|
"""Incrementally refresh the prototype store. `full=True` rebuilds every
|
||||||
character regardless of the gate/fingerprints (nightly reconcile). Returns
|
character regardless of the gate/fingerprints (nightly reconcile). Returns
|
||||||
{skipped, rebuilt, removed}; commits."""
|
{skipped, rebuilt, removed}; commits."""
|
||||||
settings = session.execute(
|
settings = MLSettings.load_sync(session)
|
||||||
select(MLSettings).where(MLSettings.id == 1)
|
|
||||||
).scalar_one()
|
|
||||||
sig = _global_signature(session)
|
sig = _global_signature(session)
|
||||||
if not full and settings.ccip_ref_signature == sig:
|
if not full and settings.ccip_ref_signature == sig:
|
||||||
return {"skipped": True, "rebuilt": 0, "removed": 0}
|
return {"skipped": True, "rebuilt": 0, "removed": 0}
|
||||||
@@ -204,9 +202,7 @@ def retract_auto_applied_ccip(session: Session) -> int:
|
|||||||
n_retracted."""
|
n_retracted."""
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
settings = session.execute(
|
settings = MLSettings.load_sync(session)
|
||||||
select(MLSettings).where(MLSettings.id == 1)
|
|
||||||
).scalar_one()
|
|
||||||
if not settings.ccip_auto_apply_enabled:
|
if not settings.ccip_auto_apply_enabled:
|
||||||
return 0
|
return 0
|
||||||
thr = float(settings.ccip_auto_apply_threshold)
|
thr = float(settings.ccip_auto_apply_threshold)
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ from datetime import UTC, datetime
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy import delete, exists, func, select
|
from sqlalchemy import delete, exists, func, select
|
||||||
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
@@ -39,9 +40,11 @@ from ...models import (
|
|||||||
TagPositiveConfirmation,
|
TagPositiveConfirmation,
|
||||||
TagSuggestionRejection,
|
TagSuggestionRejection,
|
||||||
)
|
)
|
||||||
from ...models.tag import PRESENTATION_SYSTEM_TAGS, image_tag
|
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
|
||||||
|
from ..image_tag_apply import insert_image_tags
|
||||||
from .training_data import (
|
from .training_data import (
|
||||||
_AUTO_SOURCES,
|
_AUTO_SOURCES,
|
||||||
|
_applied_or_rejected,
|
||||||
_auto_apply_point,
|
_auto_apply_point,
|
||||||
_hygiene_excluded_ids,
|
_hygiene_excluded_ids,
|
||||||
_ids_with_tag,
|
_ids_with_tag,
|
||||||
@@ -61,6 +64,14 @@ MIN_POSITIVES_FLOOR = 8 # hard floor; settings.head_min_positives can raise
|
|||||||
_UNLABELED_POOL = 4000
|
_UNLABELED_POOL = 4000
|
||||||
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
|
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
|
||||||
|
|
||||||
|
# Auto-apply / match confidence operating range. Every graduated auto-apply or
|
||||||
|
# CCIP-match threshold the operator can set lives in this band, and the head
|
||||||
|
# precision target is clamped to it: below 0.5 "auto-apply" is meaningless, and
|
||||||
|
# 1.0 is unachievable so 0.999 is the ceiling. One source shared by the service
|
||||||
|
# clamp (_normalize_params) and the API validator (ml_admin._validate).
|
||||||
|
AUTO_APPLY_THRESHOLD_MIN = 0.5
|
||||||
|
AUTO_APPLY_THRESHOLD_MAX = 0.999
|
||||||
|
|
||||||
# Only these tag kinds get heads (the surfaced suggestion categories).
|
# Only these tag kinds get heads (the surfaced suggestion categories).
|
||||||
_HEAD_KINDS = (TagKind.general, TagKind.character)
|
_HEAD_KINDS = (TagKind.general, TagKind.character)
|
||||||
# tag.kind -> the suggestion category the rail groups under.
|
# tag.kind -> the suggestion category the rail groups under.
|
||||||
@@ -78,6 +89,38 @@ _CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
|
|||||||
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65
|
_SYSTEM_TAG_SUGGEST_FLOOR = 0.65
|
||||||
|
|
||||||
|
|
||||||
|
def _sigmoid(z, np):
|
||||||
|
"""Logistic sigmoid 1/(1+e^-z): the head score→probability transform. One home
|
||||||
|
for what was inlined at every scoring site (suggest, both sweeps, retract)."""
|
||||||
|
return 1.0 / (1.0 + np.exp(-z))
|
||||||
|
|
||||||
|
|
||||||
|
def _conflict_scores(Xn, Wc, bc, np):
|
||||||
|
"""The presentation conflict signal (#141): per row, the MAX content-head
|
||||||
|
probability and WHICH head produced it. Shared by the system-tag sweep's guard-2
|
||||||
|
and the soft-wip audit — both ask "does this ALSO look like real content?"."""
|
||||||
|
cprobs = _sigmoid(Xn @ Wc.T + bc, np)
|
||||||
|
return cprobs.max(axis=1), cprobs.argmax(axis=1)
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_presentation_review(
|
||||||
|
session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode,
|
||||||
|
):
|
||||||
|
"""Single-source the ring-loud PresentationReview row shape so the two writers
|
||||||
|
(system-tag sweep guard-2 + soft-wip audit) can't drift on columns or `mode` —
|
||||||
|
they share the (image_record_id, tag_id) composite PK, so a divergent `mode`
|
||||||
|
would be a silent first-writer-wins bug."""
|
||||||
|
session.execute(
|
||||||
|
pg_insert(PresentationReview)
|
||||||
|
.values(
|
||||||
|
image_record_id=image_record_id, tag_id=tag_id,
|
||||||
|
conflict_tag_id=conflict_tag_id, conflict_score=conflict_score,
|
||||||
|
mode=mode,
|
||||||
|
)
|
||||||
|
.on_conflict_do_nothing()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class HeadTrainingAlreadyRunning(Exception):
|
class HeadTrainingAlreadyRunning(Exception):
|
||||||
"""Raised by start_head_training_run when a run is already in flight."""
|
"""Raised by start_head_training_run when a run is already in flight."""
|
||||||
|
|
||||||
@@ -103,9 +146,7 @@ def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _settings(session: Session) -> MLSettings:
|
def _settings(session: Session) -> MLSettings:
|
||||||
return session.execute(
|
return MLSettings.load_sync(session)
|
||||||
select(MLSettings).where(MLSettings.id == 1)
|
|
||||||
).scalar_one()
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
|
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
|
||||||
@@ -124,7 +165,7 @@ def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[s
|
|||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
cv_folds = DEFAULT_CV_FOLDS
|
cv_folds = DEFAULT_CV_FOLDS
|
||||||
try:
|
try:
|
||||||
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999)
|
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), AUTO_APPLY_THRESHOLD_MIN), AUTO_APPLY_THRESHOLD_MAX)
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
precision_target = s.head_auto_apply_precision
|
precision_target = s.head_auto_apply_precision
|
||||||
return {
|
return {
|
||||||
@@ -536,7 +577,7 @@ async def score_image(
|
|||||||
norms[norms == 0] = 1.0
|
norms[norms == 0] = 1.0
|
||||||
Xn = X / norms
|
Xn = X / norms
|
||||||
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
|
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
|
||||||
probs_bag = 1.0 / (1.0 + np.exp(-Z)) # (B, H)
|
probs_bag = _sigmoid(Z, np) # (B, H)
|
||||||
probs = probs_bag.max(axis=0) # (H,) best over the bag
|
probs = probs_bag.max(axis=0) # (H,) best over the bag
|
||||||
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds
|
# ARGMAX beside the max: WHICH bag row won each head → the region that grounds
|
||||||
# the tag (bag_meta[win]); None when the whole-image vector won (#1206).
|
# the tag (bag_meta[win]); None when the whole-image vector won (#1206).
|
||||||
@@ -614,9 +655,7 @@ async def ground_applied_tag(
|
|||||||
|
|
||||||
|
|
||||||
async def _settings_async(session: AsyncSession) -> MLSettings:
|
async def _settings_async(session: AsyncSession) -> MLSettings:
|
||||||
return (
|
return await MLSettings.load(session)
|
||||||
await session.execute(select(MLSettings).where(MLSettings.id == 1))
|
|
||||||
).scalar_one()
|
|
||||||
|
|
||||||
|
|
||||||
# --- Earned auto-apply (sync, ml worker) ---------------------------------
|
# --- Earned auto-apply (sync, ml worker) ---------------------------------
|
||||||
@@ -687,7 +726,6 @@ def auto_apply_sweep(
|
|||||||
embeddings in chunks; commits per chunk on a real run. Returns
|
embeddings in chunks; commits per chunk on a real run. Returns
|
||||||
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
|
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
|
|
||||||
settings = _settings(session)
|
settings = _settings(session)
|
||||||
rows = _auto_apply_heads(
|
rows = _auto_apply_heads(
|
||||||
@@ -704,18 +742,7 @@ def auto_apply_sweep(
|
|||||||
names = [r.name for r in rows]
|
names = [r.name for r in rows]
|
||||||
|
|
||||||
# Skip images that already carry, or have rejected, each tag.
|
# Skip images that already carry, or have rejected, each tag.
|
||||||
skip = {tid: set() for tid in tag_ids}
|
skip = _applied_or_rejected(session, tag_ids)
|
||||||
for tid in tag_ids:
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
|
||||||
):
|
|
||||||
skip[tid].add(iid)
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
select(TagSuggestionRejection.image_record_id).where(
|
|
||||||
TagSuggestionRejection.tag_id == tid
|
|
||||||
)
|
|
||||||
):
|
|
||||||
skip[tid].add(iid)
|
|
||||||
|
|
||||||
applied = [0] * len(rows)
|
applied = [0] * len(rows)
|
||||||
scanned = 0
|
scanned = 0
|
||||||
@@ -729,8 +756,12 @@ def auto_apply_sweep(
|
|||||||
if not cids:
|
if not cids:
|
||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b))) # (N, H)
|
probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
|
||||||
scanned += len(cids)
|
scanned += len(cids)
|
||||||
|
# Collected across every head, then written as ONE insert below. Was an
|
||||||
|
# insert per applied tag from inside this loop, which on a first sweep
|
||||||
|
# over a back-catalogue is tens of thousands of round-trips (#3072).
|
||||||
|
pending: list[dict] = []
|
||||||
for h in range(len(rows)):
|
for h in range(len(rows)):
|
||||||
tid = tag_ids[h]
|
tid = tag_ids[h]
|
||||||
for idx in np.where(probs[:, h] >= thr[h])[0]:
|
for idx in np.where(probs[:, h] >= thr[h])[0]:
|
||||||
@@ -740,12 +771,12 @@ def auto_apply_sweep(
|
|||||||
skip[tid].add(iid)
|
skip[tid].add(iid)
|
||||||
applied[h] += 1
|
applied[h] += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
session.execute(
|
pending.append({
|
||||||
pg_insert(image_tag)
|
"image_record_id": iid, "tag_id": tid,
|
||||||
.values(image_record_id=iid, tag_id=tid, source="head_auto")
|
"source": "head_auto",
|
||||||
.on_conflict_do_nothing()
|
})
|
||||||
)
|
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
|
insert_image_tags(session, pending)
|
||||||
session.commit()
|
session.commit()
|
||||||
run.last_progress_at = datetime.now(UTC)
|
run.last_progress_at = datetime.now(UTC)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -759,18 +790,42 @@ def auto_apply_sweep(
|
|||||||
|
|
||||||
|
|
||||||
_PRESENTATION_SOURCE = "presentation_auto"
|
_PRESENTATION_SOURCE = "presentation_auto"
|
||||||
|
_PROCESS_SOURCE = "process_auto"
|
||||||
|
|
||||||
|
# System-tag auto-apply modes (#1464). Both modes run the identical sweep — apply
|
||||||
|
# a system tag at a flat threshold with a PROVISIONAL source + a ring-loud review
|
||||||
|
# guard — and differ ONLY in which tags, which settings knobs, and which
|
||||||
|
# source/review-mode. 'chrome' (banner) is HIDDEN from the gallery; 'process'
|
||||||
|
# (wip / editor screenshot) stays VISIBLE (the hide is a gallery-query effect of
|
||||||
|
# the tag's group membership, not of this sweep).
|
||||||
|
_SWEEP_MODES = {
|
||||||
|
"chrome": {
|
||||||
|
"names": CHROME_SYSTEM_TAGS,
|
||||||
|
"enabled": "presentation_auto_apply_enabled",
|
||||||
|
"threshold": "presentation_auto_apply_threshold",
|
||||||
|
"conflict": "presentation_conflict_threshold",
|
||||||
|
"source": _PRESENTATION_SOURCE,
|
||||||
|
},
|
||||||
|
"process": {
|
||||||
|
"names": PROCESS_SYSTEM_TAGS,
|
||||||
|
"enabled": "process_auto_apply_enabled",
|
||||||
|
"threshold": "process_auto_apply_threshold",
|
||||||
|
"conflict": "process_conflict_threshold",
|
||||||
|
"source": _PROCESS_SOURCE,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _presentation_heads(session: Session, embedding_version: str):
|
def _system_tag_heads(session: Session, embedding_version: str, names):
|
||||||
"""Trained heads for the presentation chrome tags (banner / editor screenshot).
|
"""Trained heads for a system-tag group (chrome banner / process wip+editor).
|
||||||
They fire at the FLAT presentation threshold regardless of graduation — a head
|
They fire at the group's FLAT threshold regardless of graduation — a head
|
||||||
exists once the operator has labelled enough chrome (head_min_positives)."""
|
exists once the operator has labelled enough (head_min_positives)."""
|
||||||
return session.execute(
|
return session.execute(
|
||||||
select(TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias)
|
select(TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias)
|
||||||
.join(Tag, Tag.id == TagHead.tag_id)
|
.join(Tag, Tag.id == TagHead.tag_id)
|
||||||
.where(TagHead.embedding_version == embedding_version)
|
.where(TagHead.embedding_version == embedding_version)
|
||||||
.where(Tag.is_system.is_(True))
|
.where(Tag.is_system.is_(True))
|
||||||
.where(Tag.name.in_(PRESENTATION_SYSTEM_TAGS))
|
.where(Tag.name.in_(names))
|
||||||
).all()
|
).all()
|
||||||
|
|
||||||
|
|
||||||
@@ -802,27 +857,32 @@ def _valued_image_ids(session: Session) -> set[int]:
|
|||||||
return {r[0] for r in rows}
|
return {r[0] for r in rows}
|
||||||
|
|
||||||
|
|
||||||
def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> dict:
|
def system_tag_auto_apply_sweep(
|
||||||
"""Auto-hide presentation chrome (banner / editor screenshot) at the FLAT
|
session: Session, *, mode: str, dry_run: bool = False
|
||||||
presentation threshold (#141) — NOT the per-head graduated threshold. Two
|
) -> dict:
|
||||||
guards keep it safe: (1) never hide an image carrying a human/confirmed content
|
"""Auto-apply a system-tag group at its FLAT threshold. mode='chrome' (banner,
|
||||||
tag; (2) if an image about to be hidden ALSO scores >= the conflict threshold
|
#141) hides the image; mode='process' (wip / editor screenshot, #1464) keeps it
|
||||||
on a content head, still hide it but flag it (PresentationReview) so the Hidden
|
VISIBLE — the ONLY difference is the tag group's gallery membership, not this
|
||||||
view surfaces "also looks like <X>" for review. No-op unless
|
sweep. Two guards keep it safe: (1) never touch an image carrying a
|
||||||
presentation_auto_apply_enabled. numpy-only (no sklearn). Returns
|
human/confirmed content tag; (2) if the image ALSO scores >= the conflict
|
||||||
{n_applied, n_flagged, concepts}."""
|
threshold on a content head, still apply but flag it (PresentationReview,
|
||||||
|
mode=<mode>) so the review strip surfaces "also looks like <X>". The source is
|
||||||
|
PROVISIONAL so the head never trains on its own output. No-op unless the mode's
|
||||||
|
enabled flag is set. numpy-only (no sklearn). Returns {n_applied, n_flagged,
|
||||||
|
concepts}."""
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
|
|
||||||
|
cfg = _SWEEP_MODES[mode]
|
||||||
settings = _settings(session)
|
settings = _settings(session)
|
||||||
if not dry_run and not settings.presentation_auto_apply_enabled:
|
if not dry_run and not getattr(settings, cfg["enabled"]):
|
||||||
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
||||||
ver = settings.embedder_model_version
|
ver = settings.embedder_model_version
|
||||||
pres = _presentation_heads(session, ver)
|
pres = _system_tag_heads(session, ver, cfg["names"])
|
||||||
if not pres:
|
if not pres:
|
||||||
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
return {"n_applied": 0, "n_flagged": 0, "concepts": []}
|
||||||
thr = float(settings.presentation_auto_apply_threshold)
|
thr = float(getattr(settings, cfg["threshold"]))
|
||||||
conflict_thr = float(settings.presentation_conflict_threshold)
|
conflict_thr = float(getattr(settings, cfg["conflict"]))
|
||||||
|
source = cfg["source"]
|
||||||
|
|
||||||
Wp = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in pres])
|
Wp = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in pres])
|
||||||
bp = np.asarray([r.bias for r in pres], dtype=np.float32)
|
bp = np.asarray([r.bias for r in pres], dtype=np.float32)
|
||||||
@@ -839,18 +899,7 @@ def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> di
|
|||||||
valued = _valued_image_ids(session)
|
valued = _valued_image_ids(session)
|
||||||
|
|
||||||
# Skip images that already carry, or have rejected, each presentation tag.
|
# Skip images that already carry, or have rejected, each presentation tag.
|
||||||
skip = {tid: set() for tid in pres_tag_ids}
|
skip = _applied_or_rejected(session, pres_tag_ids)
|
||||||
for tid in pres_tag_ids:
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
|
||||||
):
|
|
||||||
skip[tid].add(iid)
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
select(TagSuggestionRejection.image_record_id).where(
|
|
||||||
TagSuggestionRejection.tag_id == tid
|
|
||||||
)
|
|
||||||
):
|
|
||||||
skip[tid].add(iid)
|
|
||||||
|
|
||||||
applied = [0] * len(pres)
|
applied = [0] * len(pres)
|
||||||
n_flagged = 0
|
n_flagged = 0
|
||||||
@@ -865,12 +914,15 @@ def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> di
|
|||||||
if not cids:
|
if not cids:
|
||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
probs = 1.0 / (1.0 + np.exp(-(Xn @ Wp.T + bp))) # (N, P)
|
probs = _sigmoid(Xn @ Wp.T + bp, np) # (N, P)
|
||||||
if Wc is not None:
|
if Wc is not None:
|
||||||
cprobs = 1.0 / (1.0 + np.exp(-(Xn @ Wc.T + bc))) # (N, C)
|
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
|
||||||
max_c = cprobs.max(axis=1)
|
|
||||||
arg_c = cprobs.argmax(axis=1)
|
|
||||||
scanned += len(cids)
|
scanned += len(cids)
|
||||||
|
# Same batching as auto_apply_sweep (#3072): collect the chunk's rows
|
||||||
|
# and write them once, below. The PresentationReview rows stay per-row —
|
||||||
|
# they FK to image_record/tag, not to image_tag, so writing the tags
|
||||||
|
# after them is safe, and a flagged conflict is rare by construction.
|
||||||
|
pending: list[dict] = []
|
||||||
for p in range(len(pres)):
|
for p in range(len(pres)):
|
||||||
tid = pres_tag_ids[p]
|
tid = pres_tag_ids[p]
|
||||||
for idx in np.where(probs[:, p] >= thr)[0]:
|
for idx in np.where(probs[:, p] >= thr)[0]:
|
||||||
@@ -880,28 +932,25 @@ def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> di
|
|||||||
skip[tid].add(iid)
|
skip[tid].add(iid)
|
||||||
applied[p] += 1
|
applied[p] += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
session.execute(
|
pending.append({
|
||||||
pg_insert(image_tag)
|
"image_record_id": iid, "tag_id": tid,
|
||||||
.values(
|
"source": source,
|
||||||
image_record_id=iid, tag_id=tid,
|
})
|
||||||
source=_PRESENTATION_SOURCE,
|
# Guard 2: also looks like real content → still apply, but flag it
|
||||||
)
|
# for the review strip instead of silently marking (chrome hides,
|
||||||
.on_conflict_do_nothing()
|
# process stays visible — either way the operator gets a heads-up).
|
||||||
)
|
|
||||||
# Guard 2: also looks like content → hide but flag for review.
|
|
||||||
if Wc is not None and float(max_c[idx]) >= conflict_thr:
|
if Wc is not None and float(max_c[idx]) >= conflict_thr:
|
||||||
n_flagged += 1
|
n_flagged += 1
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
session.execute(
|
_insert_presentation_review(
|
||||||
pg_insert(PresentationReview)
|
session,
|
||||||
.values(
|
|
||||||
image_record_id=iid, tag_id=tid,
|
image_record_id=iid, tag_id=tid,
|
||||||
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
|
conflict_tag_id=conf_tag_ids[int(arg_c[idx])],
|
||||||
conflict_score=float(max_c[idx]),
|
conflict_score=float(max_c[idx]),
|
||||||
)
|
mode=mode,
|
||||||
.on_conflict_do_nothing()
|
|
||||||
)
|
)
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
|
insert_image_tags(session, pending)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
concepts = [
|
concepts = [
|
||||||
@@ -914,6 +963,68 @@ def presentation_auto_apply_sweep(session: Session, dry_run: bool = False) -> di
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
|
||||||
|
"""Ring-loud audit for the SOFT WIP-title cohort (#1474). Images auto-tagged
|
||||||
|
`wip` from a low-precision sketch/doodle title (source='wip_title_soft') that ALSO
|
||||||
|
score >= the process conflict threshold on a content head are probably FINISHED
|
||||||
|
art mis-tagged as process — flag them (PresentationReview, mode='process') so the
|
||||||
|
review strip surfaces them ("also looks like <X>", Keep tag / Remove tag). Does
|
||||||
|
NOT remove the tag; the operator decides. No-op when there are no content heads.
|
||||||
|
numpy-only. Returns {n_scanned, n_flagged}."""
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id
|
||||||
|
|
||||||
|
settings = _settings(session)
|
||||||
|
ver = settings.embedder_model_version
|
||||||
|
conflict_thr = float(settings.process_conflict_threshold)
|
||||||
|
conf = _conflict_heads(session, ver)
|
||||||
|
wip_id = resolve_wip_tag_id(session)
|
||||||
|
if not conf or wip_id is None:
|
||||||
|
return {"n_scanned": 0, "n_flagged": 0}
|
||||||
|
Wc = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in conf])
|
||||||
|
bc = np.asarray([r.bias for r in conf], dtype=np.float32)
|
||||||
|
conf_tag_ids = [r.tag_id for r in conf]
|
||||||
|
|
||||||
|
soft_ids = [iid for (iid,) in session.execute(
|
||||||
|
select(image_tag.c.image_record_id)
|
||||||
|
.where(image_tag.c.tag_id == wip_id)
|
||||||
|
.where(image_tag.c.source == WIP_TITLE_SOFT_SOURCE)
|
||||||
|
)]
|
||||||
|
# Skip images already flagged for this tag (idempotent re-runs).
|
||||||
|
flagged = {iid for (iid,) in session.execute(
|
||||||
|
select(PresentationReview.image_record_id)
|
||||||
|
.where(PresentationReview.tag_id == wip_id)
|
||||||
|
)}
|
||||||
|
soft_ids = [i for i in soft_ids if i not in flagged]
|
||||||
|
|
||||||
|
n_flagged = 0
|
||||||
|
scanned = 0
|
||||||
|
for start in range(0, len(soft_ids), _AUTO_APPLY_CHUNK):
|
||||||
|
chunk = soft_ids[start:start + _AUTO_APPLY_CHUNK]
|
||||||
|
emb = _load_embeddings(session, chunk)
|
||||||
|
cids = [i for i in chunk if i in emb]
|
||||||
|
if not cids:
|
||||||
|
continue
|
||||||
|
scanned += len(cids)
|
||||||
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
|
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np)
|
||||||
|
for k in range(len(cids)):
|
||||||
|
if float(max_c[k]) >= conflict_thr:
|
||||||
|
n_flagged += 1
|
||||||
|
if not dry_run:
|
||||||
|
_insert_presentation_review(
|
||||||
|
session,
|
||||||
|
image_record_id=cids[k], tag_id=wip_id,
|
||||||
|
conflict_tag_id=conf_tag_ids[int(arg_c[k])],
|
||||||
|
conflict_score=float(max_c[k]),
|
||||||
|
mode="process",
|
||||||
|
)
|
||||||
|
if not dry_run:
|
||||||
|
session.commit()
|
||||||
|
return {"n_scanned": scanned, "n_flagged": n_flagged}
|
||||||
|
|
||||||
|
|
||||||
def retract_auto_applied_heads(session: Session) -> int:
|
def retract_auto_applied_heads(session: Session) -> int:
|
||||||
"""Soft auto-apply (milestone 139): re-score every standing source='head_auto'
|
"""Soft auto-apply (milestone 139): re-score every standing source='head_auto'
|
||||||
tag against its CURRENT head and REMOVE the ones now BELOW the head's
|
tag against its CURRENT head and REMOVE the ones now BELOW the head's
|
||||||
@@ -961,7 +1072,7 @@ def retract_auto_applied_heads(session: Session) -> int:
|
|||||||
continue
|
continue
|
||||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||||
w = np.asarray(weights, dtype=np.float32)
|
w = np.asarray(weights, dtype=np.float32)
|
||||||
probs = 1.0 / (1.0 + np.exp(-(Xn @ w + float(bias))))
|
probs = _sigmoid(Xn @ w + float(bias), np)
|
||||||
below = [cids[k] for k in np.where(probs < float(thr))[0]]
|
below = [cids[k] for k in np.where(probs < float(thr))[0]]
|
||||||
for iid in below:
|
for iid in below:
|
||||||
session.execute(
|
session.execute(
|
||||||
|
|||||||
@@ -29,7 +29,15 @@ from ...models.tag import image_tag
|
|||||||
# a CCIP reference) unless the operator confirms them (milestone 139). Keeping
|
# a CCIP reference) unless the operator confirms them (milestone 139). Keeping
|
||||||
# auto-applied predictions out of training is what makes them "soft" — a misfire
|
# auto-applied predictions out of training is what makes them "soft" — a misfire
|
||||||
# can't reinforce itself, so the retraction sweep can actually drop it.
|
# can't reinforce itself, so the retraction sweep can actually drop it.
|
||||||
_AUTO_SOURCES = ("head_auto", "ccip_auto", "ml_auto", "presentation_auto")
|
# `process_auto` (#1464): wip/editor screenshot applied by the process sweep are
|
||||||
|
# ALSO provisional — the head must learn only from title (`wip_title`) + manual
|
||||||
|
# labels, never its own auto-applied output, or it would runaway (operator 2026-07-12).
|
||||||
|
# `wip_title_soft` (#1474): the soft title tier (sketch/doodle) is LOW-precision, so
|
||||||
|
# it's provisional too — a finished piece titled "sketch" must not train the wip head.
|
||||||
|
_AUTO_SOURCES = (
|
||||||
|
"head_auto", "ccip_auto", "ml_auto", "presentation_auto", "process_auto",
|
||||||
|
"wip_title_soft",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _hygiene_excluded_ids(session: Session) -> set[int]:
|
def _hygiene_excluded_ids(session: Session) -> set[int]:
|
||||||
@@ -86,6 +94,24 @@ def _rejected_ids(session: Session, tag_id: int) -> list[int]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _applied_or_rejected(session: Session, tag_ids) -> dict[int, set[int]]:
|
||||||
|
"""Per-tag skip set for the auto-apply sweeps: every image that ALREADY carries
|
||||||
|
the tag (ANY source — not just training positives) OR has rejected it. A sweep
|
||||||
|
never re-applies to these. Shared by auto_apply_sweep + system_tag_auto_apply_sweep
|
||||||
|
(heads.py) and scheduled_ccip_auto_apply (tasks/ml.py). Callers mutate the returned
|
||||||
|
sets in-place to also dedupe within a single run."""
|
||||||
|
skip: dict[int, set[int]] = {}
|
||||||
|
for tid in tag_ids:
|
||||||
|
ids = {
|
||||||
|
r[0] for r in session.execute(
|
||||||
|
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
|
||||||
|
).all()
|
||||||
|
}
|
||||||
|
ids.update(_rejected_ids(session, tid))
|
||||||
|
skip[tid] = ids
|
||||||
|
return skip
|
||||||
|
|
||||||
|
|
||||||
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
|
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
|
||||||
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
|
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
|
||||||
sparse, so an untagged image is almost always a true negative."""
|
sparse, so an untagged image is almost always a true negative."""
|
||||||
|
|||||||
@@ -91,48 +91,46 @@ def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
def _campaigns_api_first(vanity: str, cookies_path: str | None) -> dict | None:
|
||||||
|
"""The first `data` object from Patreon's campaigns API filtered by vanity
|
||||||
|
(`?filter[vanity]=<vanity>&fields[campaign]=name`), or None on any failure
|
||||||
|
(network / non-200 / non-JSON / empty). The single request shape shared by
|
||||||
|
_lookup_via_api (plucks the campaign id) and resolve_display_name (plucks the
|
||||||
|
display name)."""
|
||||||
jar = _load_cookie_jar(cookies_path)
|
jar = _load_cookie_jar(cookies_path)
|
||||||
headers = {
|
|
||||||
"User-Agent": _USER_AGENT,
|
|
||||||
"Accept": "application/vnd.api+json",
|
|
||||||
}
|
|
||||||
params = {
|
|
||||||
"filter[vanity]": vanity,
|
|
||||||
"fields[campaign]": "name",
|
|
||||||
}
|
|
||||||
try:
|
try:
|
||||||
resp = requests.get(
|
resp = requests.get(
|
||||||
_CAMPAIGNS_URL,
|
_CAMPAIGNS_URL,
|
||||||
params=params,
|
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
|
||||||
headers=headers,
|
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
|
||||||
cookies=jar,
|
cookies=jar,
|
||||||
timeout=_TIMEOUT_SECONDS,
|
timeout=_TIMEOUT_SECONDS,
|
||||||
)
|
)
|
||||||
except requests.RequestException as exc:
|
except requests.RequestException as exc:
|
||||||
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
|
log.warning("Patreon campaigns API request failed for vanity=%s: %s", vanity, exc)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
if resp.status_code != 200:
|
if resp.status_code != 200:
|
||||||
log.warning(
|
log.warning(
|
||||||
"Patreon campaigns API returned HTTP %d for vanity=%s",
|
"Patreon campaigns API returned HTTP %d for vanity=%s",
|
||||||
resp.status_code, vanity,
|
resp.status_code, vanity,
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
try:
|
try:
|
||||||
payload = resp.json()
|
payload = resp.json()
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
|
log.warning("Patreon campaigns API returned non-JSON for vanity=%s: %s", vanity, exc)
|
||||||
return None
|
return None
|
||||||
|
data = payload.get("data") if isinstance(payload, dict) else None
|
||||||
|
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
|
||||||
|
return None
|
||||||
|
return data[0]
|
||||||
|
|
||||||
if not isinstance(payload, dict):
|
|
||||||
|
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
||||||
|
first = _campaigns_api_first(vanity, cookies_path)
|
||||||
|
if first is None:
|
||||||
return None
|
return None
|
||||||
data = payload.get("data")
|
campaign_id = first.get("id")
|
||||||
if not isinstance(data, list) or not data:
|
|
||||||
return None
|
|
||||||
first = data[0] if isinstance(data[0], dict) else None
|
|
||||||
campaign_id = first.get("id") if first else None
|
|
||||||
if not isinstance(campaign_id, str) or not campaign_id:
|
if not isinstance(campaign_id, str) or not campaign_id:
|
||||||
return None
|
return None
|
||||||
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
|
log.info("Resolved Patreon vanity=%s → campaign_id=%s", vanity, campaign_id)
|
||||||
@@ -144,24 +142,10 @@ def resolve_display_name(vanity: str, cookies_path: str | None) -> str | None:
|
|||||||
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None
|
(`fields[campaign]=name`), used to name the Artist at add-time (#130). None
|
||||||
on any failure — the caller falls back to the vanity handle. Sync: call from
|
on any failure — the caller falls back to the vanity handle. Sync: call from
|
||||||
an executor."""
|
an executor."""
|
||||||
jar = _load_cookie_jar(cookies_path)
|
first = _campaigns_api_first(vanity, cookies_path)
|
||||||
try:
|
if first is None:
|
||||||
resp = requests.get(
|
|
||||||
_CAMPAIGNS_URL,
|
|
||||||
params={"filter[vanity]": vanity, "fields[campaign]": "name"},
|
|
||||||
headers={"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"},
|
|
||||||
cookies=jar,
|
|
||||||
timeout=_TIMEOUT_SECONDS,
|
|
||||||
)
|
|
||||||
if resp.status_code != 200:
|
|
||||||
return None
|
return None
|
||||||
data = resp.json().get("data")
|
name = (first.get("attributes") or {}).get("name")
|
||||||
except (requests.RequestException, ValueError) as exc:
|
|
||||||
log.warning("Patreon name lookup failed for vanity=%s: %s", vanity, exc)
|
|
||||||
return None
|
|
||||||
if not isinstance(data, list) or not data or not isinstance(data[0], dict):
|
|
||||||
return None
|
|
||||||
name = (data[0].get("attributes") or {}).get("name")
|
|
||||||
return name.strip() if isinstance(name, str) and name.strip() else None
|
return name.strip() if isinstance(name, str) and name.strip() else None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -8,9 +8,10 @@ PLATFORMS below. Sidecar parsing, cookie materialization, and
|
|||||||
|
|
||||||
Lifted from GallerySubscriber's
|
Lifted from GallerySubscriber's
|
||||||
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
|
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
|
||||||
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
|
and ~/.../extension/lib/platforms.js. Five platforms; auth_type and
|
||||||
URL patterns match GS exactly so the existing browser extension
|
URL patterns match GS exactly so the existing browser extension
|
||||||
hits FC unmodified.
|
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
|
||||||
|
FC downloaders are art-dedicated services only.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from .base import (
|
from .base import (
|
||||||
@@ -18,7 +19,6 @@ from .base import (
|
|||||||
DEFAULT_EXTERNAL_POST_ID_KEYS,
|
DEFAULT_EXTERNAL_POST_ID_KEYS,
|
||||||
PlatformInfo,
|
PlatformInfo,
|
||||||
)
|
)
|
||||||
from .deviantart import INFO as _DEVIANTART
|
|
||||||
from .discord import INFO as _DISCORD
|
from .discord import INFO as _DISCORD
|
||||||
from .hentaifoundry import INFO as _HENTAIFOUNDRY
|
from .hentaifoundry import INFO as _HENTAIFOUNDRY
|
||||||
from .patreon import INFO as _PATREON
|
from .patreon import INFO as _PATREON
|
||||||
@@ -33,7 +33,6 @@ PLATFORMS: dict[str, PlatformInfo] = {
|
|||||||
_HENTAIFOUNDRY,
|
_HENTAIFOUNDRY,
|
||||||
_DISCORD,
|
_DISCORD,
|
||||||
_PIXIV,
|
_PIXIV,
|
||||||
_DEVIANTART,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ class PlatformInfo:
|
|||||||
# Synthesize a post permalink from sidecar data. Required when
|
# Synthesize a post permalink from sidecar data. Required when
|
||||||
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
||||||
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
||||||
# `url` field (patreon, deviantart).
|
# `url` field (patreon).
|
||||||
derive_post_url: Callable[[dict], str | None] | None = None
|
derive_post_url: Callable[[dict], str | None] | None = None
|
||||||
|
|
||||||
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
||||||
|
|||||||
@@ -1,23 +0,0 @@
|
|||||||
"""DeviantArt — no exercised quirks yet.
|
|
||||||
|
|
||||||
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
|
|
||||||
audit, so we don't know yet whether DA's gallery-dl sidecars are
|
|
||||||
well-behaved or have their own quirks. When DA gets exercised for the
|
|
||||||
first time, add `derive_post_url` / `augment_cookies` here as needed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from .base import GD_DEFAULTS, PlatformInfo
|
|
||||||
|
|
||||||
INFO = PlatformInfo(
|
|
||||||
key="deviantart",
|
|
||||||
name="DeviantArt",
|
|
||||||
description="Download artwork from DeviantArt artists",
|
|
||||||
auth_type="cookies",
|
|
||||||
requires_auth=False,
|
|
||||||
url_pattern=r"^https?://(www\.)?deviantart\.com/",
|
|
||||||
url_examples=[
|
|
||||||
"https://www.deviantart.com/example-artist",
|
|
||||||
"https://www.deviantart.com/example-artist/gallery",
|
|
||||||
],
|
|
||||||
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
|
|
||||||
)
|
|
||||||
@@ -24,6 +24,7 @@ from ..models import (
|
|||||||
Post,
|
Post,
|
||||||
PostAttachment,
|
PostAttachment,
|
||||||
Source,
|
Source,
|
||||||
|
attachment_download_url,
|
||||||
)
|
)
|
||||||
from ..utils.html_sanitize import (
|
from ..utils.html_sanitize import (
|
||||||
extract_img_srcs,
|
extract_img_srcs,
|
||||||
@@ -360,7 +361,7 @@ class PostFeedService:
|
|||||||
"ext": att.ext,
|
"ext": att.ext,
|
||||||
"mime": att.mime,
|
"mime": att.mime,
|
||||||
"size_bytes": att.size_bytes,
|
"size_bytes": att.size_bytes,
|
||||||
"download_url": f"/api/attachments/{att.id}/download",
|
"download_url": attachment_download_url(att.id),
|
||||||
})
|
})
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ from ..models import (
|
|||||||
Post,
|
Post,
|
||||||
PostAttachment,
|
PostAttachment,
|
||||||
Source,
|
Source,
|
||||||
|
attachment_download_url,
|
||||||
)
|
)
|
||||||
from ..utils.html_sanitize import sanitize_post_html
|
from ..utils.html_sanitize import sanitize_post_html
|
||||||
|
|
||||||
@@ -53,7 +54,7 @@ def _attachment_dict(a: PostAttachment) -> dict:
|
|||||||
"original_filename": a.original_filename,
|
"original_filename": a.original_filename,
|
||||||
"size_bytes": a.size_bytes,
|
"size_bytes": a.size_bytes,
|
||||||
"ext": a.ext,
|
"ext": a.ext,
|
||||||
"download_url": f"/api/attachments/{a.id}/download",
|
"download_url": attachment_download_url(a.id),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,121 @@
|
|||||||
|
"""Title-based WIP auto-tagging (task #1458).
|
||||||
|
|
||||||
|
Deterministic heuristic: when a post's TITLE explicitly declares work-in-progress
|
||||||
|
(the artist's own "WIP" / "work in progress" label), the ``wip`` system tag is
|
||||||
|
applied to that post's images — a cheap, high-precision complement to the
|
||||||
|
image-based ML ``wip`` head. WIP images are excluded from the Explore/gallery
|
||||||
|
browse (see gallery_service ``excluded_system_tags``), so honouring the artist's
|
||||||
|
own label keeps unfinished pieces out of the main browse right at import.
|
||||||
|
|
||||||
|
Precision over recall — a false WIP tag HIDES a finished post — so matching is
|
||||||
|
token-anchored: ``swipe`` / ``wiped`` / ``wiping`` never trip it (a letter on the
|
||||||
|
boundary blocks the match).
|
||||||
|
|
||||||
|
Sync-only: both consumers (the importer and the backfill Celery task) run on a
|
||||||
|
sync Session. Application is idempotent-additive (ON CONFLICT DO NOTHING) and
|
||||||
|
stamps a distinct ``image_tag.source`` so a later pass can tell where a wip tag
|
||||||
|
came from — the "manual" / "head_auto" / "ccip_auto" / "ml_accepted" provenance
|
||||||
|
family gains one member.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from ..models.tag import WIP_SYSTEM_TAG, Tag, image_tag
|
||||||
|
from .image_tag_apply import insert_image_tags
|
||||||
|
|
||||||
|
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other
|
||||||
|
# apply sources so provenance stays legible and a future undo can target only these.
|
||||||
|
# HARD tier ("WIP"/"work in progress") is high-precision → trains the wip head.
|
||||||
|
WIP_TITLE_SOURCE = "wip_title"
|
||||||
|
# SOFT tier (sketch/doodle/scribble, #1474) is LOWER-precision — a finished "sketch"
|
||||||
|
# is often not WIP. This source is PROVISIONAL (in training_data._AUTO_SOURCES) so it
|
||||||
|
# NEVER trains the wip head; a soft-tagged image that also looks like real content is
|
||||||
|
# surfaced by the ring-loud audit for review.
|
||||||
|
WIP_TITLE_SOFT_SOURCE = "wip_title_soft"
|
||||||
|
|
||||||
|
# A standalone "WIP" / "W.I.P" token, or the phrase "work in progress"
|
||||||
|
# (space/underscore/hyphen separated). The letter-boundary lookarounds are what
|
||||||
|
# make this precision-first: `s|wip|e`, `|wip|ed`, `|wip|ing` all have a letter
|
||||||
|
# abutting the token, so they're rejected. A trailing digit is allowed so
|
||||||
|
# "WIP2" (= WIP part 2) still matches.
|
||||||
|
_WIP_RE = re.compile(
|
||||||
|
r"(?<![A-Za-z])(?:w\.?i\.?p\.?|work[\s_-]+in[\s_-]+progress)(?![A-Za-z])",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Soft tier: sketch / doodle / scribble (+ plurals), letter-boundary anchored so
|
||||||
|
# "sketchbook" / "kadoodle" don't trip it. Deliberately conservative — recall is
|
||||||
|
# secondary because the soft source doesn't train the head and the ring-loud audit
|
||||||
|
# catches false positives.
|
||||||
|
_SOFT_WIP_RE = re.compile(
|
||||||
|
r"(?<![A-Za-z])(?:sketch|sketches|doodle|doodles|scribble|scribbles)(?![A-Za-z])",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Coarse SQL prefilters for the backfill sweep — narrow the post scan to rows that
|
||||||
|
# COULD match before the precise regex confirms. Case-insensitive ILIKE patterns.
|
||||||
|
# Each MUST stay a SUPERSET of its regex or the sweep would silently miss posts.
|
||||||
|
WIP_TITLE_SQL_PREFILTER = ("%wip%", "%work%progress%")
|
||||||
|
SOFT_WIP_TITLE_SQL_PREFILTER = ("%sketch%", "%doodle%", "%scribble%")
|
||||||
|
|
||||||
|
# Chunk bulk inserts so a large sweep can't blow past psycopg's 65535-parameter
|
||||||
|
# ceiling (3 params/row → ~21k rows max; 5k stays comfortably under).
|
||||||
|
_INSERT_CHUNK = 5000
|
||||||
|
|
||||||
|
|
||||||
|
def matches_wip_title(title: str | None) -> bool:
|
||||||
|
"""True when a post title explicitly marks it work-in-progress (HARD tier)."""
|
||||||
|
if not title:
|
||||||
|
return False
|
||||||
|
return _WIP_RE.search(title) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def matches_soft_wip_title(title: str | None) -> bool:
|
||||||
|
"""True when a title carries a SOFT WIP cue (sketch/doodle/scribble, #1474)."""
|
||||||
|
if not title:
|
||||||
|
return False
|
||||||
|
return _SOFT_WIP_RE.search(title) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_wip_tag_id(session: Session) -> int | None:
|
||||||
|
"""The seeded ``wip`` system tag's id (migration 0075), or None if absent."""
|
||||||
|
return session.execute(
|
||||||
|
select(Tag.id).where(Tag.name == WIP_SYSTEM_TAG, Tag.is_system.is_(True))
|
||||||
|
).scalar_one_or_none()
|
||||||
|
|
||||||
|
|
||||||
|
def apply_wip_image_tags(
|
||||||
|
session: Session, image_ids, tag_id: int, *, source: str = WIP_TITLE_SOURCE
|
||||||
|
) -> int:
|
||||||
|
"""Attach ``tag_id`` (stamped with ``source``) to each image id, idempotently —
|
||||||
|
never disturbs an existing tag or its source. Returns the number of image_tag
|
||||||
|
rows newly inserted. Does NOT commit.
|
||||||
|
|
||||||
|
The insert count is computed from a pre-SELECT of already-tagged ids rather
|
||||||
|
than the statement's ``rowcount``: psycopg reports -1 for a multi-row
|
||||||
|
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so rowcount
|
||||||
|
is unusable here. The SELECT is accurate within this single transaction (no
|
||||||
|
concurrent writer touches these (image, wip) rows); ON CONFLICT DO NOTHING
|
||||||
|
stays as a race-safety belt so a rare concurrent insert can't error."""
|
||||||
|
ids = list({int(i) for i in image_ids})
|
||||||
|
if not ids:
|
||||||
|
return 0
|
||||||
|
inserted = 0
|
||||||
|
for start in range(0, len(ids), _INSERT_CHUNK):
|
||||||
|
chunk = ids[start:start + _INSERT_CHUNK]
|
||||||
|
already = set(session.execute(
|
||||||
|
select(image_tag.c.image_record_id)
|
||||||
|
.where(image_tag.c.tag_id == tag_id)
|
||||||
|
.where(image_tag.c.image_record_id.in_(chunk))
|
||||||
|
).scalars())
|
||||||
|
to_insert = [iid for iid in chunk if iid not in already]
|
||||||
|
if not to_insert:
|
||||||
|
continue
|
||||||
|
insert_image_tags(session, [
|
||||||
|
{"image_record_id": iid, "tag_id": tag_id, "source": source}
|
||||||
|
for iid in to_insert
|
||||||
|
])
|
||||||
|
inserted += len(to_insert)
|
||||||
|
return inserted
|
||||||
@@ -409,3 +409,31 @@ def rescan_series_suggestions_task(self, after_post_id: int = 0) -> dict:
|
|||||||
)
|
)
|
||||||
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
||||||
return summary
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
@celery.task(
|
||||||
|
name="backend.app.tasks.admin.reclaim_orphaned_attachments_task",
|
||||||
|
bind=True,
|
||||||
|
autoretry_for=(OperationalError, DBAPIError),
|
||||||
|
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
||||||
|
# The service stops walking at its own 900s budget and reports partial, so
|
||||||
|
# these limits are the backstop for a wedged filesystem (NFS stall), not the
|
||||||
|
# expected exit. Comfortably above the budget so a normal run always returns
|
||||||
|
# its summary rather than being killed mid-walk.
|
||||||
|
soft_time_limit=1200, time_limit=1500, # 20 min / 25 min
|
||||||
|
)
|
||||||
|
def reclaim_orphaned_attachments_task(self, dry_run: bool = True) -> dict:
|
||||||
|
"""Reclaim unattributed PostAttachment rows and the store blobs nothing
|
||||||
|
references any more (#3068). dry_run (the default) returns the projection
|
||||||
|
without touching rows or files; apply deletes the orphan rows, then unlinks
|
||||||
|
every blob no surviving row references.
|
||||||
|
|
||||||
|
Defaults to the SAFE preview — unlike the other tasks here, whose apply is
|
||||||
|
reversible-ish or scoped; this one deletes files. Operator-triggered only,
|
||||||
|
never on a beat: an unattended sweep that unlinks blobs is not something to
|
||||||
|
run without someone reading the projection first."""
|
||||||
|
SessionLocal = _sync_session_factory()
|
||||||
|
with SessionLocal() as session:
|
||||||
|
return cleanup_service.reclaim_orphaned_attachments(
|
||||||
|
session, images_root=IMAGES_ROOT, dry_run=dry_run,
|
||||||
|
)
|
||||||
|
|||||||
@@ -76,6 +76,8 @@ DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
|
|||||||
OLD_TASK_DAYS = 7
|
OLD_TASK_DAYS = 7
|
||||||
PHASH_PAGE = 500
|
PHASH_PAGE = 500
|
||||||
VERIFY_PAGE = 200
|
VERIFY_PAGE = 200
|
||||||
|
# Title-based WIP backfill (task #1458): posts scanned per keyset page.
|
||||||
|
WIP_BACKFILL_PAGE = 500
|
||||||
FFPROBE_TIMEOUT_SECONDS = 10
|
FFPROBE_TIMEOUT_SECONDS = 10
|
||||||
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
|
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
|
||||||
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
||||||
@@ -171,6 +173,12 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
|||||||
# task-name override beats the queue threshold whatever queue the row records
|
# task-name override beats the queue threshold whatever queue the row records
|
||||||
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
||||||
"backend.app.tasks.external.fetch_external_link": 65,
|
"backend.app.tasks.external.fetch_external_link": 65,
|
||||||
|
# Attachment reclaim walks the whole sha-addressed store; the service caps
|
||||||
|
# itself at a 900s budget and reports partial, but the task's hard limit is
|
||||||
|
# 25 min for a wedged filesystem (NFS stall). Same phantom-flag class as the
|
||||||
|
# external-fetch entry above — without an override a healthy in-flight walk
|
||||||
|
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
|
||||||
|
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -774,89 +782,62 @@ def recover_stalled_library_audit_runs() -> int:
|
|||||||
return recovered
|
return recovered
|
||||||
|
|
||||||
|
|
||||||
|
def _recover_stalled_runs(model, *, stall_minutes: int, keep_runs: int, label: str) -> int:
|
||||||
|
"""Shared recovery + retention sweep for the head run-tracking tables
|
||||||
|
(HeadTrainingRun / HeadAutoApplyRun, which share the
|
||||||
|
status/last_progress_at/started_at/finished_at/error/id columns): flip 'running'
|
||||||
|
rows with no progress past `stall_minutes` to 'error', then prune to the last
|
||||||
|
`keep_runs` (rule 89). Returns the number recovered. NOTE the two other recover
|
||||||
|
tasks are deliberately NOT folded in — library-audit has no prune tail and
|
||||||
|
backup uses a single started_at cutoff."""
|
||||||
|
SessionLocal = _sync_session_factory()
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
cutoff = now - timedelta(minutes=stall_minutes)
|
||||||
|
with SessionLocal() as session:
|
||||||
|
result = session.execute(
|
||||||
|
update(model)
|
||||||
|
.where(model.status == "running")
|
||||||
|
.where(func.coalesce(model.last_progress_at, model.started_at) < cutoff)
|
||||||
|
.values(
|
||||||
|
status="error", finished_at=now,
|
||||||
|
error=f"stranded by recovery sweep (no progress for {stall_minutes} min)",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
keep = session.execute(
|
||||||
|
select(model.id).order_by(model.id.desc()).limit(keep_runs)
|
||||||
|
).scalars().all()
|
||||||
|
if keep:
|
||||||
|
session.execute(delete(model).where(model.id.not_in(keep)))
|
||||||
|
session.commit()
|
||||||
|
recovered = result.rowcount or 0
|
||||||
|
if recovered:
|
||||||
|
log.info("%s: recovered %d rows", label, recovered)
|
||||||
|
return recovered
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
|
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
|
||||||
def recover_stalled_head_training_runs() -> int:
|
def recover_stalled_head_training_runs() -> int:
|
||||||
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
|
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
|
||||||
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
|
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
|
||||||
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
|
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
|
||||||
SessionLocal = _sync_session_factory()
|
return _recover_stalled_runs(
|
||||||
now = datetime.now(UTC)
|
HeadTrainingRun,
|
||||||
cutoff = now - timedelta(minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES)
|
stall_minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES,
|
||||||
with SessionLocal() as session:
|
keep_runs=HEAD_TRAINING_KEEP_RUNS,
|
||||||
result = session.execute(
|
label="recover_stalled_head_training_runs",
|
||||||
update(HeadTrainingRun)
|
|
||||||
.where(HeadTrainingRun.status == "running")
|
|
||||||
.where(
|
|
||||||
func.coalesce(
|
|
||||||
HeadTrainingRun.last_progress_at, HeadTrainingRun.started_at
|
|
||||||
)
|
)
|
||||||
< cutoff
|
|
||||||
)
|
|
||||||
.values(
|
|
||||||
status="error", finished_at=now,
|
|
||||||
error=(
|
|
||||||
f"stranded by recovery sweep (no progress for "
|
|
||||||
f"{HEAD_TRAINING_STALL_THRESHOLD_MINUTES} min)"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
keep = session.execute(
|
|
||||||
select(HeadTrainingRun.id).order_by(HeadTrainingRun.id.desc())
|
|
||||||
.limit(HEAD_TRAINING_KEEP_RUNS)
|
|
||||||
).scalars().all()
|
|
||||||
if keep:
|
|
||||||
session.execute(
|
|
||||||
delete(HeadTrainingRun).where(HeadTrainingRun.id.not_in(keep))
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
recovered = result.rowcount or 0
|
|
||||||
if recovered:
|
|
||||||
log.info(
|
|
||||||
"recover_stalled_head_training_runs: recovered %d rows", recovered
|
|
||||||
)
|
|
||||||
return recovered
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
|
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
|
||||||
def recover_stalled_head_auto_apply_runs() -> int:
|
def recover_stalled_head_auto_apply_runs() -> int:
|
||||||
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
|
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
|
||||||
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
|
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
|
||||||
SessionLocal = _sync_session_factory()
|
return _recover_stalled_runs(
|
||||||
now = datetime.now(UTC)
|
HeadAutoApplyRun,
|
||||||
cutoff = now - timedelta(minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES)
|
stall_minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES,
|
||||||
with SessionLocal() as session:
|
keep_runs=HEAD_AUTO_APPLY_KEEP_RUNS,
|
||||||
result = session.execute(
|
label="recover_stalled_head_auto_apply_runs",
|
||||||
update(HeadAutoApplyRun)
|
|
||||||
.where(HeadAutoApplyRun.status == "running")
|
|
||||||
.where(
|
|
||||||
func.coalesce(
|
|
||||||
HeadAutoApplyRun.last_progress_at, HeadAutoApplyRun.started_at
|
|
||||||
)
|
)
|
||||||
< cutoff
|
|
||||||
)
|
|
||||||
.values(
|
|
||||||
status="error", finished_at=now,
|
|
||||||
error=(
|
|
||||||
f"stranded by recovery sweep (no progress for "
|
|
||||||
f"{HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES} min)"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
keep = session.execute(
|
|
||||||
select(HeadAutoApplyRun.id).order_by(HeadAutoApplyRun.id.desc())
|
|
||||||
.limit(HEAD_AUTO_APPLY_KEEP_RUNS)
|
|
||||||
).scalars().all()
|
|
||||||
if keep:
|
|
||||||
session.execute(
|
|
||||||
delete(HeadAutoApplyRun).where(HeadAutoApplyRun.id.not_in(keep))
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
recovered = result.rowcount or 0
|
|
||||||
if recovered:
|
|
||||||
log.info(
|
|
||||||
"recover_stalled_head_auto_apply_runs: recovered %d rows", recovered
|
|
||||||
)
|
|
||||||
return recovered
|
|
||||||
|
|
||||||
|
|
||||||
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
|
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
|
||||||
@@ -1045,6 +1026,96 @@ def cleanup_old_download_events() -> int:
|
|||||||
return result.rowcount or 0
|
return result.rowcount or 0
|
||||||
|
|
||||||
|
|
||||||
|
def _backfill_wip_tier(session, tag_id, prefilter, matcher, source) -> int:
|
||||||
|
"""One keyset-paginated pass over posts whose title matches a WIP tier, applying
|
||||||
|
`tag_id` (stamped `source`) to their images. Shared by the hard + soft tiers
|
||||||
|
(#1458 / #1474). Coarse `prefilter` (ILIKE superset) narrows the scan; the precise
|
||||||
|
`matcher` confirms. Idempotent-additive (ON CONFLICT DO NOTHING). Returns the row
|
||||||
|
count newly applied."""
|
||||||
|
from ..models import Post
|
||||||
|
from ..models.image_provenance import ImageProvenance
|
||||||
|
from ..services.wip_title import apply_wip_image_tags
|
||||||
|
|
||||||
|
applied = 0
|
||||||
|
last_id = 0
|
||||||
|
while True:
|
||||||
|
rows = session.execute(
|
||||||
|
select(Post.id, Post.post_title)
|
||||||
|
.where(Post.id > last_id)
|
||||||
|
.where(Post.post_title.is_not(None))
|
||||||
|
.where(or_(*[Post.post_title.ilike(p) for p in prefilter]))
|
||||||
|
.order_by(Post.id.asc())
|
||||||
|
.limit(WIP_BACKFILL_PAGE)
|
||||||
|
).all()
|
||||||
|
if not rows:
|
||||||
|
break
|
||||||
|
last_id = rows[-1][0]
|
||||||
|
match_ids = [pid for pid, title in rows if matcher(title)]
|
||||||
|
if match_ids:
|
||||||
|
image_ids = session.execute(
|
||||||
|
select(ImageProvenance.image_record_id)
|
||||||
|
.where(ImageProvenance.post_id.in_(match_ids))
|
||||||
|
).scalars().all()
|
||||||
|
applied += apply_wip_image_tags(session, image_ids, tag_id, source=source)
|
||||||
|
session.commit()
|
||||||
|
return applied
|
||||||
|
|
||||||
|
|
||||||
|
@celery.task(
|
||||||
|
name="backend.app.tasks.maintenance.backfill_wip_title_tags",
|
||||||
|
# Coarse-prefiltered scan over posts; the candidate set is small on a typical
|
||||||
|
# library, but bound it like the other full-library sweeps.
|
||||||
|
soft_time_limit=1800, time_limit=2100,
|
||||||
|
)
|
||||||
|
def backfill_wip_title_tags() -> int:
|
||||||
|
"""Scan EXISTING posts for WIP titles and apply the `wip` system tag to their
|
||||||
|
images — the operator-triggered back-catalogue catch-up (task #1458 hard tier +
|
||||||
|
#1474 soft tier). New imports are tagged live by the importer; this covers the
|
||||||
|
existing library.
|
||||||
|
|
||||||
|
HARD tier ("WIP"/"work in progress") always runs (the operator triggered the
|
||||||
|
scan); the SOFT tier (sketch/doodle, provisional source) runs only when
|
||||||
|
wip_soft_title_tagging_enabled, AFTER hard so a title matching both keeps the
|
||||||
|
trained hard tag (ON CONFLICT DO NOTHING). Keyset-paginated, restart-safe.
|
||||||
|
|
||||||
|
Deliberately NOT scheduled as a beat: a periodic re-run would re-apply to matching
|
||||||
|
posts and silently undo a manual WIP removal, so it stays an explicit operator
|
||||||
|
action (Settings → "Scan existing posts for WIP titles"). Returns rows applied.
|
||||||
|
"""
|
||||||
|
from ..models import ImportSettings
|
||||||
|
from ..services.wip_title import (
|
||||||
|
SOFT_WIP_TITLE_SQL_PREFILTER,
|
||||||
|
WIP_TITLE_SOFT_SOURCE,
|
||||||
|
WIP_TITLE_SOURCE,
|
||||||
|
WIP_TITLE_SQL_PREFILTER,
|
||||||
|
matches_soft_wip_title,
|
||||||
|
matches_wip_title,
|
||||||
|
resolve_wip_tag_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
SessionLocal = _sync_session_factory()
|
||||||
|
with SessionLocal() as session:
|
||||||
|
tag_id = resolve_wip_tag_id(session)
|
||||||
|
if tag_id is None:
|
||||||
|
log.warning(
|
||||||
|
"backfill_wip_title_tags: no `wip` system tag present; nothing to do"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
settings = ImportSettings.load_sync(session)
|
||||||
|
applied = _backfill_wip_tier(
|
||||||
|
session, tag_id, WIP_TITLE_SQL_PREFILTER, matches_wip_title,
|
||||||
|
WIP_TITLE_SOURCE,
|
||||||
|
)
|
||||||
|
if settings.wip_soft_title_tagging_enabled:
|
||||||
|
applied += _backfill_wip_tier(
|
||||||
|
session, tag_id, SOFT_WIP_TITLE_SQL_PREFILTER, matches_soft_wip_title,
|
||||||
|
WIP_TITLE_SOFT_SOURCE,
|
||||||
|
)
|
||||||
|
if applied:
|
||||||
|
log.info("backfill_wip_title_tags: applied wip to %d image(s)", applied)
|
||||||
|
return applied
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
|
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
|
||||||
def vacuum_analyze() -> dict:
|
def vacuum_analyze() -> dict:
|
||||||
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
|
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
|
||||||
|
|||||||
+52
-36
@@ -105,9 +105,7 @@ def embed_image(self, image_id: int) -> dict:
|
|||||||
record = session.get(ImageRecord, image_id)
|
record = session.get(ImageRecord, image_id)
|
||||||
if record is None:
|
if record is None:
|
||||||
return {"status": "missing", "image_id": image_id}
|
return {"status": "missing", "image_id": image_id}
|
||||||
settings = session.execute(
|
settings = MLSettings.load_sync(session)
|
||||||
select(MLSettings).where(MLSettings.id == 1)
|
|
||||||
).scalar_one()
|
|
||||||
|
|
||||||
src = Path(record.path)
|
src = Path(record.path)
|
||||||
is_vid = _is_video(src)
|
is_vid = _is_video(src)
|
||||||
@@ -488,15 +486,10 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
from sqlalchemy import select as sa_select
|
from sqlalchemy import select as sa_select
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
|
|
||||||
from ..models import ImageRegion, MLSettings, Tag, TagKind, TagSuggestionRejection
|
from ..models import ImageRegion, MLSettings, Tag, TagKind
|
||||||
from ..models.tag import image_tag
|
from ..models.tag import image_tag
|
||||||
|
from ..services.ml.ccip import _FIGURE_KINDS
|
||||||
fig = ("face", "figure")
|
from ..services.ml.training_data import _applied_or_rejected, _l2norm
|
||||||
|
|
||||||
def _l2(m):
|
|
||||||
n = np.linalg.norm(m, axis=1, keepdims=True)
|
|
||||||
n[n == 0] = 1.0
|
|
||||||
return m / n
|
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
@@ -521,7 +514,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
)
|
)
|
||||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
.join(Tag, Tag.id == image_tag.c.tag_id)
|
||||||
.where(Tag.kind == TagKind.character)
|
.where(Tag.kind == TagKind.character)
|
||||||
.where(ImageRegion.kind.in_(fig))
|
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
|
||||||
.where(ImageRegion.ccip_embedding.is_not(None))
|
.where(ImageRegion.ccip_embedding.is_not(None))
|
||||||
.where(ImageRegion.image_record_id.in_(single))
|
.where(ImageRegion.image_record_id.in_(single))
|
||||||
).all()
|
).all()
|
||||||
@@ -532,29 +525,16 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
for tid, vec in ref_rows:
|
for tid, vec in ref_rows:
|
||||||
by_char.setdefault(tid, []).append(vec)
|
by_char.setdefault(tid, []).append(vec)
|
||||||
ref_tags = list(by_char)
|
ref_tags = list(by_char)
|
||||||
mats = [_l2(np.asarray(by_char[t], dtype=np.float32)) for t in ref_tags]
|
mats = [_l2norm(np.asarray(by_char[t], dtype=np.float32), np) for t in ref_tags]
|
||||||
allref = np.vstack(mats) # (total, 768)
|
allref = np.vstack(mats) # (total, 768)
|
||||||
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
|
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
|
||||||
|
|
||||||
# Per character: images that already carry OR rejected the tag — skip.
|
# Per character: images that already carry OR rejected the tag — skip.
|
||||||
skip = {t: set() for t in ref_tags}
|
skip = _applied_or_rejected(session, ref_tags)
|
||||||
for t in ref_tags:
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
sa_select(image_tag.c.image_record_id).where(
|
|
||||||
image_tag.c.tag_id == t
|
|
||||||
)
|
|
||||||
):
|
|
||||||
skip[t].add(iid)
|
|
||||||
for (iid,) in session.execute(
|
|
||||||
sa_select(TagSuggestionRejection.image_record_id).where(
|
|
||||||
TagSuggestionRejection.tag_id == t
|
|
||||||
)
|
|
||||||
):
|
|
||||||
skip[t].add(iid)
|
|
||||||
|
|
||||||
img_ids = list(session.execute(
|
img_ids = list(session.execute(
|
||||||
sa_select(ImageRegion.image_record_id)
|
sa_select(ImageRegion.image_record_id)
|
||||||
.where(ImageRegion.kind.in_(fig), ImageRegion.ccip_embedding.is_not(None))
|
.where(ImageRegion.kind.in_(_FIGURE_KINDS), ImageRegion.ccip_embedding.is_not(None))
|
||||||
.distinct()
|
.distinct()
|
||||||
).scalars())
|
).scalars())
|
||||||
|
|
||||||
@@ -566,7 +546,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
|
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
|
||||||
.where(
|
.where(
|
||||||
ImageRegion.image_record_id.in_(chunk),
|
ImageRegion.image_record_id.in_(chunk),
|
||||||
ImageRegion.kind.in_(fig),
|
ImageRegion.kind.in_(_FIGURE_KINDS),
|
||||||
ImageRegion.ccip_embedding.is_not(None),
|
ImageRegion.ccip_embedding.is_not(None),
|
||||||
)
|
)
|
||||||
).all()
|
).all()
|
||||||
@@ -574,7 +554,7 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
for iid, vec in rows:
|
for iid, vec in rows:
|
||||||
by_img.setdefault(iid, []).append(vec)
|
by_img.setdefault(iid, []).append(vec)
|
||||||
for iid, vecs in by_img.items():
|
for iid, vecs in by_img.items():
|
||||||
q = _l2(np.asarray(vecs, dtype=np.float32)) # (nq, 768)
|
q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768)
|
||||||
colmax = (q @ allref.T).max(axis=0) # (total,)
|
colmax = (q @ allref.T).max(axis=0) # (total,)
|
||||||
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
|
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
|
||||||
for ci in np.where(charmax >= thr)[0]:
|
for ci in np.where(charmax >= thr)[0]:
|
||||||
@@ -599,18 +579,54 @@ def scheduled_ccip_auto_apply() -> str:
|
|||||||
soft_time_limit=1800, time_limit=2100,
|
soft_time_limit=1800, time_limit=2100,
|
||||||
)
|
)
|
||||||
def scheduled_presentation_auto_apply() -> str:
|
def scheduled_presentation_auto_apply() -> str:
|
||||||
"""Auto-hide presentation chrome (banner / editor screenshot) on a daily
|
"""Auto-hide presentation chrome (banner) on a daily passive sweep (#141).
|
||||||
passive sweep (#141). No-op unless presentation_auto_apply_enabled. Idempotent
|
No-op unless presentation_auto_apply_enabled. Idempotent — already-tagged images
|
||||||
— already-hidden images are skipped — so an interrupted run simply re-runs next
|
are skipped — so an interrupted run simply re-runs next cycle (that IS the
|
||||||
cycle (that IS the recovery). Wall-clock bounded by the task time limits."""
|
recovery). Wall-clock bounded by the task time limits."""
|
||||||
from ..services.ml.heads import presentation_auto_apply_sweep
|
from ..services.ml.heads import system_tag_auto_apply_sweep
|
||||||
|
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
result = presentation_auto_apply_sweep(session)
|
result = system_tag_auto_apply_sweep(session, mode="chrome")
|
||||||
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
||||||
|
|
||||||
|
|
||||||
|
@celery.task(
|
||||||
|
name="backend.app.tasks.ml.scheduled_process_auto_apply",
|
||||||
|
soft_time_limit=1800, time_limit=2100,
|
||||||
|
)
|
||||||
|
def scheduled_process_auto_apply() -> str:
|
||||||
|
"""Auto-apply the PROCESS system tags (wip / editor screenshot) on a daily
|
||||||
|
passive sweep (#1464) — provisional source, ring-loud review guard, image stays
|
||||||
|
VISIBLE. No-op unless process_auto_apply_enabled (opt-in). Idempotent —
|
||||||
|
already-tagged/rejected images are skipped — so an interrupted run just re-runs
|
||||||
|
next cycle (the recovery). Wall-clock bounded by the task time limits."""
|
||||||
|
from ..services.ml.heads import system_tag_auto_apply_sweep
|
||||||
|
|
||||||
|
SessionLocal = _sync_session_factory()
|
||||||
|
with SessionLocal() as session:
|
||||||
|
result = system_tag_auto_apply_sweep(session, mode="process")
|
||||||
|
return f"applied={result['n_applied']} flagged={result['n_flagged']}"
|
||||||
|
|
||||||
|
|
||||||
|
@celery.task(
|
||||||
|
name="backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
|
||||||
|
soft_time_limit=1800, time_limit=2100,
|
||||||
|
)
|
||||||
|
def scheduled_soft_wip_conflict_audit() -> str:
|
||||||
|
"""Ring-loud audit over the SOFT WIP-title cohort (#1474) — flag sketch/doodle
|
||||||
|
auto-tags that ALSO look like real content for review. No-op when there are no
|
||||||
|
content heads; idempotent (already-flagged images skipped). Runs regardless of
|
||||||
|
the process-sweep toggle, since soft-title tags come from the importer, not that
|
||||||
|
sweep. Wall-clock bounded by the task time limits."""
|
||||||
|
from ..services.ml.heads import soft_wip_conflict_audit
|
||||||
|
|
||||||
|
SessionLocal = _sync_session_factory()
|
||||||
|
with SessionLocal() as session:
|
||||||
|
result = soft_wip_conflict_audit(session)
|
||||||
|
return f"scanned={result['n_scanned']} flagged={result['n_flagged']}"
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.ml.prune_presentation_reviews")
|
@celery.task(name="backend.app.tasks.ml.prune_presentation_reviews")
|
||||||
def prune_presentation_reviews() -> str:
|
def prune_presentation_reviews() -> str:
|
||||||
"""Retention (rule 89): drop RESOLVED presentation-review flags older than 30
|
"""Retention (rule 89): drop RESOLVED presentation-review flags older than 30
|
||||||
|
|||||||
+56
-3
@@ -11,12 +11,26 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
|
|||||||
- python 3.14
|
- python 3.14
|
||||||
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
|
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
|
||||||
- node (frontend job: `npm install` + vitest + vite build)
|
- node (frontend job: `npm install` + vitest + vite build)
|
||||||
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Forgejo registry push)
|
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
|
||||||
|
|
||||||
|
## Secondary runtime image
|
||||||
|
|
||||||
|
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
|
||||||
|
|
||||||
|
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
|
||||||
|
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
|
||||||
|
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
|
||||||
|
per `docs/process.md`'s "add deps to the image when used by >1 project".
|
||||||
|
|
||||||
## Per-job tool installs
|
## Per-job tool installs
|
||||||
|
|
||||||
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
|
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
|
||||||
- `npm install --no-audit --no-fund` — in `frontend-build` job
|
- `npm install --no-audit --no-fund` — in `frontend-build` job
|
||||||
|
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
|
||||||
|
- `unzip` — in `extension.yml`'s "Verify XPI contents" step, installed via apt
|
||||||
|
only when absent (`node:24-bookworm-slim` may or may not carry it). Debian
|
||||||
|
package, ~2s. Not worth baking into a shared image for a single consumer, per
|
||||||
|
`docs/process.md`'s ">1 project" rule.
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
@@ -26,12 +40,51 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
|
|||||||
"add deps to image when used by >1 project" rule: FC alone is one Python
|
"add deps to image when used by >1 project" rule: FC alone is one Python
|
||||||
project, so the deps live in `requirements.txt` and install per-job.
|
project, so the deps live in `requirements.txt` and install per-job.
|
||||||
Reconsider when a second Fabled-family Python backend lands.
|
Reconsider when a second Fabled-family Python backend lands.
|
||||||
- Integration uses Forgejo Actions `services:` + socket-discovered bridge IPs
|
- Integration uses Fabled-Git Actions `services:` + socket-discovered bridge IPs
|
||||||
because `act_runner` (swarm-runner v0.6+) puts services on the default
|
because `act_runner` (swarm-runner v0.6+) puts services on the default
|
||||||
bridge with no embedded DNS. The pattern is documented in the rulebook's
|
bridge with no embedded DNS. The pattern is documented in the rulebook's
|
||||||
`forgejo.md` "CI philosophy" section and FC's `ci.yml` is the canonical
|
`fabled-git.md` "CI philosophy" section and FC's `ci.yml` is the canonical
|
||||||
example.
|
example.
|
||||||
- No `package-lock.json` is tracked yet (FC's `feedback_no_local_runs`
|
- No `package-lock.json` is tracked yet (FC's `feedback_no_local_runs`
|
||||||
memory bans `npm install` locally). Using `npm install` rather than
|
memory bans `npm install` locally). Using `npm install` rather than
|
||||||
`npm ci` until a lockfile lands.
|
`npm ci` until a lockfile lands.
|
||||||
- No `imagemagick` / `pandoc` per-job installs needed.
|
- No `imagemagick` / `pandoc` per-job installs needed.
|
||||||
|
- `extension/`'s vitest specs load `lib/*.js` by evaluating the real file as a
|
||||||
|
classic script (`test/helpers/loadLib.js`) rather than adding `module.exports`
|
||||||
|
shims to production code — the libs ship as `background.scripts`, not ES
|
||||||
|
modules, so the specs exercise exactly the bytes packaged into the XPI.
|
||||||
|
- **`extension/scripts/packaging.sh` is the single definition of what ships
|
||||||
|
inside the XPI.** Two consumers read from it rather than keeping their own
|
||||||
|
copy: web-ext's `--ignore-files` (`extension/package.json`), and the `git log`
|
||||||
|
pathspec inside the script's own version derivation. It was three until
|
||||||
|
2026-08-27 — `ci.yml`'s `extension-version` guard held the third and went when
|
||||||
|
the manual bump it guarded did (milestone 271 step 5). Hand-kept copies of
|
||||||
|
that one fact is what allowed issue #2397, so `extension/test/version.spec.js`
|
||||||
|
asserts no workflow has reintroduced a literal `:(exclude)extension/…`.
|
||||||
|
- **The shipped extension version is derived, not committed.** It is the commit
|
||||||
|
TIME of the newest packaged-extension change (minutes since 2020-01-01, per
|
||||||
|
family rule 149 — never a commit count, which orders by branch rather than by
|
||||||
|
recency). `build.yml`'s `sign-extension` computes it and stamps it into
|
||||||
|
`extension/manifest.json` + `package.json` in the working tree before signing;
|
||||||
|
the stamp is never committed. Treat the version in the repo as a base: only
|
||||||
|
its MAJOR.MINOR is read, and its patch component is inert.
|
||||||
|
- Every job that calls `packaging.sh version` checks out with `fetch-depth: 0` —
|
||||||
|
`build.yml`'s `sign-extension` and `build-web`, and `ci.yml`'s
|
||||||
|
`extension-version`. A depth-1 clone sees one commit and derives a wrong,
|
||||||
|
too-low value **rather than failing**, so the full-history checkout is
|
||||||
|
load-bearing rather than incidental.
|
||||||
|
- **`FC_CHANNEL` is a build arg, not a runtime setting.** `build.yml` passes
|
||||||
|
`dev` / `main` to the web image only (the ml and agent images have nothing to
|
||||||
|
report it to), and `/api/extension/manifest` reports it beside the version so
|
||||||
|
an install can be traced to a channel. It is declared LAST in the Dockerfile
|
||||||
|
on purpose: an ARG invalidates every layer below it, and this is the one value
|
||||||
|
that differs between the dev and main builds of identical source, so placing
|
||||||
|
it earlier would stop the two channels ever sharing a cached `pip install`.
|
||||||
|
Empty by default — a local build then reports no channel at all rather than
|
||||||
|
claiming one.
|
||||||
|
- Callers MUST `set -f` before substituting the script's output. Without it the
|
||||||
|
shell expands `test/**` against the working tree and silently narrows the
|
||||||
|
pattern to whatever files exist at that moment — a failure that looks like
|
||||||
|
nothing until dev files start appearing in the XPI. `test/version.spec.js`
|
||||||
|
asserts every `--ignore-files` consumer sets it, and that no consumer has
|
||||||
|
quietly reinstated a hardcoded list.
|
||||||
|
|||||||
+62
-9
@@ -1,13 +1,14 @@
|
|||||||
# FabledCurator Firefox Extension
|
# FabledCurator Firefox Extension
|
||||||
|
|
||||||
Self-hosted Firefox extension that pushes session cookies from supported
|
Self-hosted Firefox extension that pushes session cookies from supported
|
||||||
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv,
|
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv)
|
||||||
DeviantArt) into FabledCurator, and lets you add a creator as a Source
|
into FabledCurator, and lets you add a creator as a Source from their
|
||||||
from their page in one click.
|
page in one click.
|
||||||
|
|
||||||
## Install (operator)
|
## Install (operator)
|
||||||
|
|
||||||
The signed XPI is bundled into the FC Docker image. Open FC →
|
The signed XPI is bundled into the FC Docker image — `:dev` and
|
||||||
|
`:latest` each carry their own channel's build. Open FC →
|
||||||
Settings → Maintenance → Browser extension → click "Install Firefox
|
Settings → Maintenance → Browser extension → click "Install Firefox
|
||||||
extension". Firefox shows its native install prompt. After installing,
|
extension". Firefox shows its native install prompt. After installing,
|
||||||
open the extension's options page (about:addons → FabledCurator →
|
open the extension's options page (about:addons → FabledCurator →
|
||||||
@@ -20,6 +21,7 @@ same card.
|
|||||||
cd extension/
|
cd extension/
|
||||||
npm install --no-save # web-ext only
|
npm install --no-save # web-ext only
|
||||||
npm run lint # web-ext lint
|
npm run lint # web-ext lint
|
||||||
|
npm run test:unit # vitest — lib/ logic + packaging/version checks
|
||||||
npm run start # launches Firefox with extension loaded
|
npm run start # launches Firefox with extension loaded
|
||||||
npm run build # unsigned XPI in web-ext-artifacts/
|
npm run build # unsigned XPI in web-ext-artifacts/
|
||||||
```
|
```
|
||||||
@@ -36,10 +38,61 @@ npm run build # unsigned XPI in web-ext-artifacts/
|
|||||||
- [ ] Subscriptions list: popup → "Sources" tab → list renders
|
- [ ] Subscriptions list: popup → "Sources" tab → list renders
|
||||||
- [ ] Check now: click play icon on source row → no error toast
|
- [ ] Check now: click play icon on source row → no error toast
|
||||||
|
|
||||||
|
## Versioning — don't hand-edit the patch number
|
||||||
|
|
||||||
|
The shipped version is **derived**, not committed. `scripts/packaging.sh
|
||||||
|
version` returns `MAJOR.MINOR` from `manifest.json` plus a patch component
|
||||||
|
that is the commit *time* of the newest change to a packaged extension file,
|
||||||
|
in minutes since 2020-01-01. `build.yml` computes it and stamps it into both
|
||||||
|
`manifest.json` and `package.json` at build time. The stamp is never
|
||||||
|
committed — the commit carrying it would itself be a change to the extension,
|
||||||
|
which would move the version again.
|
||||||
|
|
||||||
|
So:
|
||||||
|
|
||||||
|
- **Editing the patch number does nothing.** It is overwritten before web-ext
|
||||||
|
ever reads it. There is no bump to make, and none to forget.
|
||||||
|
- **MAJOR.MINOR is still yours.** It carries the deliberate meaning, it is read
|
||||||
|
from `manifest.json` alone, and CI fails the `extension-version` lane if the
|
||||||
|
two files disagree on it.
|
||||||
|
- `npm run build` locally produces an XPI labelled with the *committed*
|
||||||
|
version, since nothing stamped it. Fine for loading into a test profile; not
|
||||||
|
what ships.
|
||||||
|
|
||||||
|
Why commit time and not a commit count: a count is per-branch, so `dev` and
|
||||||
|
`main` count different histories of the same code and their versions end up
|
||||||
|
ordered by which branch accumulated more commits rather than by which is newer.
|
||||||
|
Commit time gives both branches the same number for the same source — which is
|
||||||
|
exactly what lets one AMO signature serve both channels (family rule 149, FC
|
||||||
|
issue #3092).
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
`dev` and `main` each build and sign their own extension, and an install is
|
||||||
|
tied to whichever FC instance it points at — Firefox's static `update_url`
|
||||||
|
cannot apply here, since every FC install is a different host, so the extension
|
||||||
|
asks its configured backend. **The channel therefore IS the instance.**
|
||||||
|
Switching channel means repointing the FC URL in options and reinstalling from
|
||||||
|
that host; there is no separate channel setting, and adding one would
|
||||||
|
contradict each server build shipping its own extension.
|
||||||
|
|
||||||
|
The channel is reported *beside* the version, never inside it:
|
||||||
|
`/api/extension/manifest` answers `{"version": "...", "channel": "dev"}`. It is
|
||||||
|
optional — an instance that declares none simply omits the key, and the popup,
|
||||||
|
the toolbar tooltip and the Settings card all read exactly as they did before
|
||||||
|
the field existed. Do not be tempted to make it a `-dev` version suffix: the
|
||||||
|
comparator parses each dotted segment with `parseInt`, so a suffixed segment
|
||||||
|
reads as 0 and every dev build compares equal to every other, collapsing "no
|
||||||
|
update available" and "I cannot read this version" into one answer.
|
||||||
|
|
||||||
## Release
|
## Release
|
||||||
|
|
||||||
Bump `manifest.json` + `package.json` SemVer (both files) and commit
|
Nothing to do by hand. Push to `dev`: `build.yml` signs the extension if this
|
||||||
under `extension/**`. The `.forgejo/workflows/extension.yml` workflow
|
change moved the version, caches the signed XPI as a Forgejo `ext-<version>`
|
||||||
runs `web-ext sign` on main, commits the signed XPI to
|
release, and bundles it into `fabledcurator:dev`. Merging to `main` derives the
|
||||||
`frontend/public/extension/`, and the next FC server build bundles it
|
same version, hits that cache, and bundles the byte-identical XPI into
|
||||||
into the Docker image.
|
`:latest` with no second AMO call.
|
||||||
|
|
||||||
|
AMO refuses to re-sign a version it has already issued, so signing is one-shot
|
||||||
|
per version — which is why the cache exists and why the version must never move
|
||||||
|
backwards.
|
||||||
|
|||||||
@@ -31,6 +31,91 @@ browser.runtime.onInstalled.addListener(() => ensureInitialized());
|
|||||||
browser.runtime.onStartup.addListener(() => ensureInitialized());
|
browser.runtime.onStartup.addListener(() => ensureInitialized());
|
||||||
ensureInitialized().catch(e => console.error('init failed:', e));
|
ensureInitialized().catch(e => console.error('init failed:', e));
|
||||||
|
|
||||||
|
// ---- Extension self-update check (#1489) ----
|
||||||
|
// Installed per-instance from the operator's FC host, so Firefox's static
|
||||||
|
// update_url can't apply (each instance has a different host). Instead ask the
|
||||||
|
// configured backend for the latest published version and nudge the operator to
|
||||||
|
// reinstall the freshly-signed XPI — surfaced as a popup banner (on demand) and
|
||||||
|
// a toolbar badge (daily). /api/extension/manifest is public and returns
|
||||||
|
// {version, latest_url, sha256} plus an OPTIONAL {channel} naming which channel
|
||||||
|
// that instance serves ("dev"/"main", #3113); the XPI is served from the web
|
||||||
|
// root (not /api).
|
||||||
|
//
|
||||||
|
// The channel IS the instance: Firefox's static update_url cannot apply here
|
||||||
|
// because every FC install is a different host, so the extension asks its
|
||||||
|
// configured backend — which means switching channel is repointing apiUrl in
|
||||||
|
// options and reinstalling from that host. There is no separate channel
|
||||||
|
// setting to build, and building one would contradict each server build
|
||||||
|
// shipping its own extension.
|
||||||
|
|
||||||
|
function versionIsNewer(candidate, current) {
|
||||||
|
// Dotted numeric compare so 1.0.10 > 1.0.9 (a plain string compare wouldn't).
|
||||||
|
const a = String(candidate).split('.').map(n => parseInt(n, 10) || 0);
|
||||||
|
const b = String(current).split('.').map(n => parseInt(n, 10) || 0);
|
||||||
|
for (let i = 0; i < Math.max(a.length, b.length); i++) {
|
||||||
|
if ((a[i] || 0) !== (b[i] || 0)) return (a[i] || 0) > (b[i] || 0);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkForUpdateInfo() {
|
||||||
|
await ensureInitialized();
|
||||||
|
if (!api.isConfigured()) return { updateAvailable: false, configured: false };
|
||||||
|
let info;
|
||||||
|
try {
|
||||||
|
info = await api.getExtensionManifest();
|
||||||
|
} catch (e) {
|
||||||
|
return { updateAvailable: false, error: e.message };
|
||||||
|
}
|
||||||
|
const currentVersion = browser.runtime.getManifest().version;
|
||||||
|
const latestVersion = info && info.version ? info.version : null;
|
||||||
|
// Which channel the configured instance serves — reported ALONGSIDE the
|
||||||
|
// version, never folded into it. A `-dev` suffix would have to survive
|
||||||
|
// versionIsNewer's parseInt above, and it wouldn't: the segment would read
|
||||||
|
// as 0 and every dev build would compare equal to every other.
|
||||||
|
//
|
||||||
|
// null is a normal answer, not a failure — an instance built before the
|
||||||
|
// field existed, or one built locally with no channel declared. Nothing
|
||||||
|
// below branches on it except the label.
|
||||||
|
const channel = info && info.channel ? info.channel : null;
|
||||||
|
// latest_url is served from the web root, not the JSON API.
|
||||||
|
const base = api.webRoot();
|
||||||
|
return {
|
||||||
|
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
|
||||||
|
currentVersion,
|
||||||
|
latestVersion,
|
||||||
|
channel,
|
||||||
|
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function refreshUpdateBadge() {
|
||||||
|
let r;
|
||||||
|
try { r = await checkForUpdateInfo(); } catch { return; }
|
||||||
|
try {
|
||||||
|
await browser.action.setBadgeText({ text: r.updateAvailable ? '↑' : '' });
|
||||||
|
if (r.updateAvailable) {
|
||||||
|
await browser.action.setBadgeBackgroundColor({ color: '#F4BA7A' });
|
||||||
|
// Channel first, version second, and the channel dropped entirely when
|
||||||
|
// the instance doesn't report one — so the tooltip reads exactly as it
|
||||||
|
// did before the field existed rather than saying "(unknown ...)".
|
||||||
|
const label = r.channel ? `${r.channel} v${r.latestVersion}` : `v${r.latestVersion}`;
|
||||||
|
await browser.action.setTitle({ title: `FabledCurator — update available (${label})` });
|
||||||
|
} else {
|
||||||
|
await browser.action.setTitle({ title: 'FabledCurator' });
|
||||||
|
}
|
||||||
|
} catch { /* action API unavailable — non-fatal */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Daily proactive check (needs the "alarms" permission). create() is idempotent
|
||||||
|
// by name, so re-running it on each event-page load is safe.
|
||||||
|
browser.alarms.create('fc-update-check', { periodInMinutes: 24 * 60, delayInMinutes: 1 });
|
||||||
|
browser.alarms.onAlarm.addListener((alarm) => {
|
||||||
|
if (alarm.name === 'fc-update-check') refreshUpdateBadge();
|
||||||
|
});
|
||||||
|
browser.runtime.onStartup.addListener(() => refreshUpdateBadge());
|
||||||
|
browser.runtime.onInstalled.addListener(() => refreshUpdateBadge());
|
||||||
|
|
||||||
// ---- Discord token capture via webRequest ----
|
// ---- Discord token capture via webRequest ----
|
||||||
|
|
||||||
browser.webRequest.onBeforeSendHeaders.addListener(
|
browser.webRequest.onBeforeSendHeaders.addListener(
|
||||||
@@ -148,6 +233,21 @@ browser.webRequest.onBeforeRedirect.addListener(
|
|||||||
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
|
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Extract → verify → upload one cookie-auth platform. Returns a structured
|
||||||
|
// outcome so the two callers (EXPORT_COOKIES single, EXPORT_ALL_COOKIES) shape
|
||||||
|
// their own response + skip semantics. Verifies the captured cookies are
|
||||||
|
// actually live BEFORE uploading, so a confirmed-stale session doesn't overwrite
|
||||||
|
// good FC-side credentials; platforms with no verify config (v.ok === null) fall
|
||||||
|
// through to upload.
|
||||||
|
async function exportPlatformCookies(key) {
|
||||||
|
const cookies = await extractCookiesForPlatform(key);
|
||||||
|
if (cookies.length === 0) return { status: 'empty' };
|
||||||
|
const v = await verifyCookiesForPlatform(key);
|
||||||
|
if (v.ok === false) return { status: 'stale', reason: v.reason, cookieCount: cookies.length };
|
||||||
|
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
|
||||||
|
return { status: 'ok', cookieCount: cookies.length, verified: v.ok === true };
|
||||||
|
}
|
||||||
|
|
||||||
// ---- Message router ----
|
// ---- Message router ----
|
||||||
|
|
||||||
browser.runtime.onMessage.addListener(async (msg) => {
|
browser.runtime.onMessage.addListener(async (msg) => {
|
||||||
@@ -192,22 +292,14 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
if (!platform) return { error: `Unknown platform: ${key}` };
|
if (!platform) return { error: `Unknown platform: ${key}` };
|
||||||
try {
|
try {
|
||||||
if (platform.authType === 'cookies') {
|
if (platform.authType === 'cookies') {
|
||||||
const cookies = await extractCookiesForPlatform(key);
|
const r = await exportPlatformCookies(key);
|
||||||
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
|
if (r.status === 'empty') return { error: 'No cookies found — log in first.' };
|
||||||
// Verify the captured cookies are actually live BEFORE
|
if (r.status === 'stale') {
|
||||||
// uploading. Skips upload on confirmed-stale sessions so we
|
|
||||||
// don't overwrite FC-side credentials with garbage. Platforms
|
|
||||||
// without a verify config (verify.ok === null) fall through
|
|
||||||
// to upload as before.
|
|
||||||
const v = await verifyCookiesForPlatform(key);
|
|
||||||
if (v.ok === false) {
|
|
||||||
return {
|
return {
|
||||||
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
|
error: `Captured ${r.cookieCount} ${platform.name} cookies but they don't appear authenticated (${r.reason}). Log in again in this browser, then retry.`,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
const data = toNetscapeFormat(cookies);
|
return { success: true, cookieCount: r.cookieCount, verified: r.verified };
|
||||||
await api.uploadCredentials(key, 'cookies', data);
|
|
||||||
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
|
|
||||||
}
|
}
|
||||||
if (key === 'discord') {
|
if (key === 'discord') {
|
||||||
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
|
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
|
||||||
@@ -235,18 +327,10 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
const cookies = await extractCookiesForPlatform(key);
|
const r = await exportPlatformCookies(key);
|
||||||
if (cookies.length === 0) {
|
if (r.status === 'empty') results[key] = { skipped: true, reason: 'no cookies' };
|
||||||
results[key] = { skipped: true, reason: 'no cookies' };
|
else if (r.status === 'stale') results[key] = { error: `verify failed: ${r.reason}` };
|
||||||
continue;
|
else results[key] = { success: true, cookieCount: r.cookieCount, verified: r.verified };
|
||||||
}
|
|
||||||
const v = await verifyCookiesForPlatform(key);
|
|
||||||
if (v.ok === false) {
|
|
||||||
results[key] = { error: `verify failed: ${v.reason}` };
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
|
|
||||||
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
|
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
results[key] = { error: e.message };
|
results[key] = { error: e.message };
|
||||||
}
|
}
|
||||||
@@ -283,11 +367,9 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case 'OPEN_ARTIST_PAGE': {
|
case 'OPEN_ARTIST_PAGE': {
|
||||||
// apiUrl is configured with the /api suffix (see
|
// The SPA artist route (/artist/:slug) is served from the web root, not
|
||||||
// options/options.html placeholder); the SPA artist route is
|
// the JSON API — see api.webRoot().
|
||||||
// /artist/:slug, served from the same origin. Strip /api so the
|
const base = api.webRoot();
|
||||||
// browser-level URL hits the Vue router, not the JSON API.
|
|
||||||
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
|
|
||||||
const slug = encodeURIComponent(msg.slug || '');
|
const slug = encodeURIComponent(msg.slug || '');
|
||||||
if (!base || !slug) return { error: 'apiUrl or slug missing' };
|
if (!base || !slug) return { error: 'apiUrl or slug missing' };
|
||||||
try {
|
try {
|
||||||
@@ -298,6 +380,9 @@ browser.runtime.onMessage.addListener(async (msg) => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
case 'CHECK_UPDATE':
|
||||||
|
return await checkForUpdateInfo();
|
||||||
|
|
||||||
default:
|
default:
|
||||||
return { error: `Unknown message type: ${msg.type}` };
|
return { error: `Unknown message type: ${msg.type}` };
|
||||||
}
|
}
|
||||||
|
|||||||
+23
-1
@@ -11,7 +11,10 @@ class FabledCuratorAPI {
|
|||||||
|
|
||||||
async init() {
|
async init() {
|
||||||
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
|
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
|
||||||
this.baseUrl = cfg.apiUrl || null;
|
// Normalize on READ, not just on save: configs stored before the options
|
||||||
|
// page started normalizing are missing the `/api` suffix, and this heals
|
||||||
|
// them without the operator having to reopen Settings.
|
||||||
|
this.baseUrl = normalizeApiUrl(cfg.apiUrl) || null;
|
||||||
this.apiKey = cfg.apiKey || null;
|
this.apiKey = cfg.apiKey || null;
|
||||||
return this.isConfigured();
|
return this.isConfigured();
|
||||||
}
|
}
|
||||||
@@ -50,6 +53,13 @@ class FabledCuratorAPI {
|
|||||||
} catch {
|
} catch {
|
||||||
message = `HTTP ${response.status}: ${response.statusText}`;
|
message = `HTTP ${response.status}: ${response.statusText}`;
|
||||||
}
|
}
|
||||||
|
// 404/405 from FC almost always means the request never reached the JSON
|
||||||
|
// API — it fell through to the SPA catch-all, which serves HTML on GET
|
||||||
|
// and rejects everything else. Say so, rather than making the operator
|
||||||
|
// decode "Method Not Allowed" on an endpoint that plainly allows POST.
|
||||||
|
if (response.status === 404 || response.status === 405) {
|
||||||
|
message += ` — ${url} isn't the FC API. Check the FC URL in settings.`;
|
||||||
|
}
|
||||||
const err = new Error(message);
|
const err = new Error(message);
|
||||||
err.status = response.status;
|
err.status = response.status;
|
||||||
throw err;
|
throw err;
|
||||||
@@ -89,6 +99,18 @@ class FabledCuratorAPI {
|
|||||||
const qs = new URLSearchParams({ url }).toString();
|
const qs = new URLSearchParams({ url }).toString();
|
||||||
return this.request('GET', `/extension/probe?${qs}`);
|
return this.request('GET', `/extension/probe?${qs}`);
|
||||||
}
|
}
|
||||||
|
// Latest published extension version on this instance — drives the in-app
|
||||||
|
// update prompt. Public endpoint (no key needed, but request() sends it
|
||||||
|
// harmlessly). Returns {version, xpi_url, latest_url, sha256}.
|
||||||
|
getExtensionManifest() {
|
||||||
|
return this.request('GET', '/extension/manifest');
|
||||||
|
}
|
||||||
|
|
||||||
|
// The web/SPA root: where the Vue router (artist pages) and the served XPI
|
||||||
|
// live, NOT the JSON API. Used by OPEN_ARTIST_PAGE + the self-update check.
|
||||||
|
webRoot() {
|
||||||
|
return webRootFromApiUrl(this.baseUrl);
|
||||||
|
}
|
||||||
|
|
||||||
// Connection test = the cheapest read with auth.
|
// Connection test = the cheapest read with auth.
|
||||||
testConnection() {
|
testConnection() {
|
||||||
|
|||||||
+10
-12
@@ -68,16 +68,6 @@ const PLATFORMS = {
|
|||||||
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
|
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
|
||||||
note: 'Click to authenticate via OAuth',
|
note: 'Click to authenticate via OAuth',
|
||||||
},
|
},
|
||||||
deviantart: {
|
|
||||||
name: 'DeviantArt',
|
|
||||||
domains: ['.deviantart.com', 'www.deviantart.com', 'deviantart.com'],
|
|
||||||
authType: 'cookies',
|
|
||||||
color: '#05CC47',
|
|
||||||
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
|
|
||||||
// DA's logged-in-only endpoints sit behind their internal _napi
|
|
||||||
// namespace which shifts; skipping verify until a stable check
|
|
||||||
// surfaces. Same posture as SubscribeStar.
|
|
||||||
},
|
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -86,10 +76,18 @@ const PLATFORMS = {
|
|||||||
* script to decide whether to show the floating "Add as source" button.
|
* script to decide whether to show the floating "Add as source" button.
|
||||||
*/
|
*/
|
||||||
const PLATFORM_ARTIST_PATTERNS = {
|
const PLATFORM_ARTIST_PATTERNS = {
|
||||||
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c\/)[^/?#]+\/?$/i,
|
// Patreon serves the same creator under three URL shapes (see backend
|
||||||
|
// patreon_resolver._VANITY_RE): bare `patreon.com/Atole`, `c/` prefix, and
|
||||||
|
// `cw/` "creator workspace" — the last is the URL you land on once you're
|
||||||
|
// SUBSCRIBED, which is exactly when the button matters. Match all three, and
|
||||||
|
// drop the single-segment end-anchor so a creator's inner page
|
||||||
|
// (…/cw/Atole/posts, …/Atole/membership) also injects the button. Nav pages
|
||||||
|
// (home/search/…/posts permalink) stay excluded. Mirrors extension_service
|
||||||
|
// ._PLATFORM_PATTERNS — keep in sync (operator-flagged 2026-07-13: button
|
||||||
|
// vanished once subscribed because the old pattern only matched the bare root).
|
||||||
|
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?:cw\/|c\/)?(?!(?:home|search|messages|notifications|library|settings|posts)(?:[\/?#]|$))[^/?#]+/i,
|
||||||
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
|
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
|
||||||
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
|
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
|
||||||
deviantart: /^https?:\/\/(www\.)?deviantart\.com\/(?!home$|watch\b|tag\b|browse\b)[^/?#]+\/?$/i,
|
|
||||||
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
|
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
/**
|
||||||
|
* Canonical FC endpoint derivation, shared by the background client and the
|
||||||
|
* options page so a URL entered either way behaves identically.
|
||||||
|
*
|
||||||
|
* FC serves two things on one origin: the JSON API under `/api`, and the Vue
|
||||||
|
* SPA from the root. `api.js` builds requests as `${baseUrl}/credentials`, so
|
||||||
|
* the stored base URL has to carry the `/api` suffix.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Accept what an operator would naturally type — the instance root
|
||||||
|
* (`http://curator.example.com`) or the API root (`.../api`) — and return the
|
||||||
|
* API root either way.
|
||||||
|
*
|
||||||
|
* Worth normalizing rather than validating: a root-form URL doesn't fail
|
||||||
|
* loudly, it lands on the SPA catch-all, which answers `GET /credentials` with
|
||||||
|
* 200 HTML and rejects `POST /credentials` with 405. The operator sees a
|
||||||
|
* working Test Connection and a broken export.
|
||||||
|
*/
|
||||||
|
function normalizeApiUrl(raw) {
|
||||||
|
const trimmed = (raw || '').trim().replace(/\/+$/, '');
|
||||||
|
if (!trimmed) return '';
|
||||||
|
return /\/api$/i.test(trimmed) ? trimmed : `${trimmed}/api`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The SPA root — where the Vue router (artist pages) and the served XPI live,
|
||||||
|
* NOT the JSON API. Accepts either input form, same as normalizeApiUrl.
|
||||||
|
*/
|
||||||
|
function webRootFromApiUrl(raw) {
|
||||||
|
return normalizeApiUrl(raw).replace(/\/api$/i, '');
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"manifest_version": 3,
|
"manifest_version": 3,
|
||||||
"name": "FabledCurator",
|
"name": "FabledCurator",
|
||||||
"version": "1.0.7",
|
"version": "1.0.11",
|
||||||
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
|
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
|
||||||
|
|
||||||
"browser_specific_settings": {
|
"browser_specific_settings": {
|
||||||
@@ -22,7 +22,8 @@
|
|||||||
"tabs",
|
"tabs",
|
||||||
"activeTab",
|
"activeTab",
|
||||||
"webRequest",
|
"webRequest",
|
||||||
"webRequestBlocking"
|
"webRequestBlocking",
|
||||||
|
"alarms"
|
||||||
],
|
],
|
||||||
|
|
||||||
"host_permissions": [
|
"host_permissions": [
|
||||||
@@ -32,7 +33,6 @@
|
|||||||
"*://*.hentai-foundry.com/*",
|
"*://*.hentai-foundry.com/*",
|
||||||
"*://*.discord.com/*",
|
"*://*.discord.com/*",
|
||||||
"*://*.pixiv.net/*",
|
"*://*.pixiv.net/*",
|
||||||
"*://*.deviantart.com/*",
|
|
||||||
"*://app-api.pixiv.net/*",
|
"*://app-api.pixiv.net/*",
|
||||||
"*://oauth.secure.pixiv.net/*",
|
"*://oauth.secure.pixiv.net/*",
|
||||||
"*://*/*"
|
"*://*/*"
|
||||||
@@ -45,7 +45,7 @@
|
|||||||
},
|
},
|
||||||
|
|
||||||
"background": {
|
"background": {
|
||||||
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/api.js", "background/background.js"]
|
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/url.js", "lib/api.js", "background/background.js"]
|
||||||
},
|
},
|
||||||
|
|
||||||
"options_ui": {
|
"options_ui": {
|
||||||
@@ -60,7 +60,6 @@
|
|||||||
"*://*.subscribestar.com/*",
|
"*://*.subscribestar.com/*",
|
||||||
"*://*.subscribestar.adult/*",
|
"*://*.subscribestar.adult/*",
|
||||||
"*://*.hentai-foundry.com/*",
|
"*://*.hentai-foundry.com/*",
|
||||||
"*://*.deviantart.com/*",
|
|
||||||
"*://*.pixiv.net/*"
|
"*://*.pixiv.net/*"
|
||||||
],
|
],
|
||||||
"js": ["lib/platforms.js", "content/content-script.js"],
|
"js": ["lib/platforms.js", "content/content-script.js"],
|
||||||
|
|||||||
@@ -21,9 +21,12 @@
|
|||||||
<body>
|
<body>
|
||||||
<h1>FabledCurator extension</h1>
|
<h1>FabledCurator extension</h1>
|
||||||
|
|
||||||
<label for="api-url">FC base URL</label>
|
<label for="api-url">FC instance URL</label>
|
||||||
<input id="api-url" type="url" placeholder="http://curator.example.com/api" />
|
<input id="api-url" type="url" placeholder="http://curator.example.com" />
|
||||||
<div class="hint">Find this on FC → Settings → Maintenance → Browser extension.</div>
|
<div class="hint">
|
||||||
|
Your FabledCurator address — with or without the trailing <code>/api</code>; both work.
|
||||||
|
Find it on FC → Settings → Maintenance → Browser extension.
|
||||||
|
</div>
|
||||||
|
|
||||||
<label for="api-key">Extension API key</label>
|
<label for="api-key">Extension API key</label>
|
||||||
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
|
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
|
||||||
@@ -36,6 +39,7 @@
|
|||||||
|
|
||||||
<div id="status" class="status" style="display:none;"></div>
|
<div id="status" class="status" style="display:none;"></div>
|
||||||
|
|
||||||
|
<script src="../lib/url.js"></script>
|
||||||
<script src="options.js"></script>
|
<script src="options.js"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ document.addEventListener('DOMContentLoaded', async () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
async function save() {
|
async function save() {
|
||||||
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
|
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
|
||||||
const apiKey = document.getElementById('api-key').value.trim();
|
const apiKey = document.getElementById('api-key').value.trim();
|
||||||
if (!apiUrl || !apiKey) {
|
if (!apiUrl || !apiKey) {
|
||||||
showStatus('Both fields are required.', 'err');
|
showStatus('Both fields are required.', 'err');
|
||||||
@@ -16,11 +16,14 @@ async function save() {
|
|||||||
}
|
}
|
||||||
await browser.storage.local.set({ apiUrl, apiKey });
|
await browser.storage.local.set({ apiUrl, apiKey });
|
||||||
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
|
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
|
||||||
showStatus('Saved.', 'ok');
|
// Show what was actually stored — the operator may have typed the instance
|
||||||
|
// root and it was normalized to the API root.
|
||||||
|
document.getElementById('api-url').value = apiUrl;
|
||||||
|
showStatus(`Saved — using ${apiUrl}`, 'ok');
|
||||||
}
|
}
|
||||||
|
|
||||||
async function test() {
|
async function test() {
|
||||||
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
|
const apiUrl = normalizeApiUrl(document.getElementById('api-url').value);
|
||||||
const apiKey = document.getElementById('api-key').value.trim();
|
const apiKey = document.getElementById('api-key').value.trim();
|
||||||
if (!apiUrl || !apiKey) {
|
if (!apiUrl || !apiKey) {
|
||||||
showStatus('Fill both fields first.', 'err');
|
showStatus('Fill both fields first.', 'err');
|
||||||
@@ -31,8 +34,23 @@ async function test() {
|
|||||||
method: 'GET',
|
method: 'GET',
|
||||||
headers: { 'X-Extension-Key': apiKey },
|
headers: { 'X-Extension-Key': apiKey },
|
||||||
});
|
});
|
||||||
if (r.ok) showStatus(`Connected — HTTP ${r.status}.`, 'ok');
|
if (!r.ok) {
|
||||||
else showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
|
showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// A 200 is NOT sufficient. If the URL resolves to the Vue SPA instead of
|
||||||
|
// the JSON API, the catch-all route returns 200 with an HTML document —
|
||||||
|
// which used to report "Connected" on a config that could not POST at all.
|
||||||
|
const contentType = r.headers.get('content-type') || '';
|
||||||
|
if (!contentType.includes('json')) {
|
||||||
|
showStatus(
|
||||||
|
`${apiUrl} answered with ${contentType || 'no content-type'}, not JSON `
|
||||||
|
+ '— that looks like the FC web UI rather than its API.',
|
||||||
|
'err',
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
showStatus(`Connected to ${apiUrl} — HTTP ${r.status}.`, 'ok');
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
|
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,15 +1,18 @@
|
|||||||
{
|
{
|
||||||
"name": "fabledcurator-extension",
|
"name": "fabledcurator-extension",
|
||||||
"version": "1.0.7",
|
"version": "1.0.11",
|
||||||
"private": true,
|
"private": true,
|
||||||
"description": "Firefox extension for FabledCurator",
|
"description": "Firefox extension for FabledCurator",
|
||||||
|
"comment_ignore_files": "The --ignore-files list comes from scripts/packaging.sh, the single source of truth shared with ci.yml's guard and the derived-version patch count. `set -f` is REQUIRED before the substitution: without it the shell globs `test/**` against the working tree and silently narrows the pattern to whatever files happen to exist.",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore",
|
"lint": "set -f; web-ext lint --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore)",
|
||||||
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --firefox=firefox",
|
"start": "set -f; web-ext run --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --firefox=firefox",
|
||||||
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --overwrite-dest",
|
"build": "set -f; web-ext build --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --overwrite-dest",
|
||||||
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
|
"sign": "set -f; web-ext sign --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
|
||||||
|
"test:unit": "vitest run"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"web-ext": "^8.0.0"
|
"vitest": "^4.0.0",
|
||||||
|
"web-ext": "^10.0.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -72,6 +72,17 @@ body {
|
|||||||
.btn.block { display: block; width: 100%; margin-top: 8px; }
|
.btn.block { display: block; width: 100%; margin-top: 8px; }
|
||||||
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
|
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
|
||||||
.btn.link:hover { color: var(--accent); }
|
.btn.link:hover { color: var(--accent); }
|
||||||
|
.btn.small { padding: 6px 12px; font-size: 13px; }
|
||||||
|
|
||||||
|
/* In-app update prompt (accent-tinted so it reads as an actionable notice). */
|
||||||
|
.update-banner {
|
||||||
|
display: flex; align-items: center; gap: 10px;
|
||||||
|
margin: 10px 10px 0; padding: 10px 12px;
|
||||||
|
background: rgba(244, 186, 122, 0.12);
|
||||||
|
border: 1px solid rgba(244, 186, 122, 0.4);
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
#update-text { flex: 1; font-size: 13px; }
|
||||||
|
|
||||||
.source-row .play {
|
.source-row .play {
|
||||||
background: none; border: none; color: var(--on-surface-variant);
|
background: none; border: none; color: var(--on-surface-variant);
|
||||||
|
|||||||
@@ -20,6 +20,11 @@
|
|||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section id="main-content" class="main hidden">
|
<section id="main-content" class="main hidden">
|
||||||
|
<div id="update-banner" class="update-banner hidden">
|
||||||
|
<span id="update-text"></span>
|
||||||
|
<button id="update-btn" class="btn primary small">Update</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
<nav class="tabs">
|
<nav class="tabs">
|
||||||
<button class="tab active" data-tab="platforms">Platforms</button>
|
<button class="tab active" data-tab="platforms">Platforms</button>
|
||||||
<button class="tab" data-tab="sources">Sources</button>
|
<button class="tab" data-tab="sources">Sources</button>
|
||||||
|
|||||||
+37
-12
@@ -2,6 +2,15 @@ document.addEventListener('DOMContentLoaded', init);
|
|||||||
|
|
||||||
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
|
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
|
||||||
|
|
||||||
|
// A centered muted note div — the loading / empty state shared by the platform
|
||||||
|
// and sources lists.
|
||||||
|
function mutedNote(text) {
|
||||||
|
const d = document.createElement('div');
|
||||||
|
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
||||||
|
d.textContent = text;
|
||||||
|
return d;
|
||||||
|
}
|
||||||
|
|
||||||
async function init() {
|
async function init() {
|
||||||
try {
|
try {
|
||||||
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
|
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
|
||||||
@@ -14,6 +23,7 @@ async function init() {
|
|||||||
setupEventListeners();
|
setupEventListeners();
|
||||||
showPlatformsLoading();
|
showPlatformsLoading();
|
||||||
testConnectionIfNeeded();
|
testConnectionIfNeeded();
|
||||||
|
checkForUpdate();
|
||||||
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
|
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
showSetupRequired();
|
showSetupRequired();
|
||||||
@@ -37,10 +47,7 @@ function showSetupRequired() {
|
|||||||
function showPlatformsLoading() {
|
function showPlatformsLoading() {
|
||||||
const c = document.getElementById('platforms-list');
|
const c = document.getElementById('platforms-list');
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
const d = document.createElement('div');
|
c.appendChild(mutedNote('Loading platforms…'));
|
||||||
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
|
||||||
d.textContent = 'Loading platforms…';
|
|
||||||
c.appendChild(d);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function testConnectionIfNeeded() {
|
async function testConnectionIfNeeded() {
|
||||||
@@ -63,6 +70,30 @@ function updateConnectionDot(connected) {
|
|||||||
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
|
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Nudge to reinstall when the configured instance publishes a newer signed XPI
|
||||||
|
// (the extension is self-hosted, so there's no Firefox auto-update). Never
|
||||||
|
// blocks the popup — a failed check just leaves the banner hidden.
|
||||||
|
async function checkForUpdate() {
|
||||||
|
try {
|
||||||
|
const r = await browser.runtime.sendMessage({ type: 'CHECK_UPDATE' });
|
||||||
|
if (r && r.updateAvailable && r.xpiUrl) showUpdateBanner(r);
|
||||||
|
} catch { /* non-fatal */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
function showUpdateBanner(r) {
|
||||||
|
// The channel names itself beside the version, never inside it (#3113).
|
||||||
|
// Absent when the instance doesn't report one, and the banner then reads
|
||||||
|
// exactly as it did before the field existed.
|
||||||
|
const channel = r.channel ? ` (${r.channel})` : '';
|
||||||
|
document.getElementById('update-text').textContent =
|
||||||
|
`Update available${channel} — v${r.latestVersion} (installed v${r.currentVersion})`;
|
||||||
|
// Opening the signed XPI triggers Firefox's native install prompt.
|
||||||
|
document.getElementById('update-btn').addEventListener('click', () => {
|
||||||
|
browser.tabs.create({ url: r.xpiUrl });
|
||||||
|
});
|
||||||
|
document.getElementById('update-banner').classList.remove('hidden');
|
||||||
|
}
|
||||||
|
|
||||||
async function loadPlatformStatus() {
|
async function loadPlatformStatus() {
|
||||||
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
|
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
|
||||||
const c = document.getElementById('platforms-list');
|
const c = document.getElementById('platforms-list');
|
||||||
@@ -162,10 +193,7 @@ async function exportAllCookies() {
|
|||||||
async function loadSources() {
|
async function loadSources() {
|
||||||
const c = document.getElementById('sources-list');
|
const c = document.getElementById('sources-list');
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
const d = document.createElement('div');
|
c.appendChild(mutedNote('Loading sources…'));
|
||||||
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
|
||||||
d.textContent = 'Loading sources…';
|
|
||||||
c.appendChild(d);
|
|
||||||
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
|
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
|
||||||
c.textContent = '';
|
c.textContent = '';
|
||||||
if (r.error) {
|
if (r.error) {
|
||||||
@@ -176,10 +204,7 @@ async function loadSources() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (!r.sources || r.sources.length === 0) {
|
if (!r.sources || r.sources.length === 0) {
|
||||||
const empty = document.createElement('div');
|
c.appendChild(mutedNote('No sources yet.'));
|
||||||
empty.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
|
|
||||||
empty.textContent = 'No sources yet.';
|
|
||||||
c.appendChild(empty);
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (const src of r.sources) c.appendChild(createSourceRow(src));
|
for (const src of r.sources) c.appendChild(createSourceRow(src));
|
||||||
|
|||||||
Executable
+157
@@ -0,0 +1,157 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Single source of truth for "what ships inside the XPI", plus the version
|
||||||
|
# derived from it.
|
||||||
|
#
|
||||||
|
# Three consumers used to hand-maintain their own copy of this list, and
|
||||||
|
# keeping three copies of one fact in sync by hand is how issue #2397 happened:
|
||||||
|
#
|
||||||
|
# 1. web-ext's --ignore-files (extension/package.json's four scripts)
|
||||||
|
# 2. the :(exclude) pathspec (what moves the version — a WIDER
|
||||||
|
# set than the ignore list; see
|
||||||
|
# NOT_VERSION_RELEVANT)
|
||||||
|
# 3. the git-log pathspec (the derived version, below)
|
||||||
|
#
|
||||||
|
# They now all read from here. POSIX sh only — CI's run shell is busybox.
|
||||||
|
#
|
||||||
|
# -f (no pathname expansion) is set for the whole script and is load-bearing:
|
||||||
|
# the lists below are iterated with deliberate word-splitting, and without -f
|
||||||
|
# the shell would also GLOB them, expanding `test/**` into whatever files
|
||||||
|
# happen to exist and corrupting the output. A caller's own `set -f` does not
|
||||||
|
# help here — this runs as a separate sh process and does not inherit it.
|
||||||
|
# Callers still need their own `set -f` for the substituted result; the two
|
||||||
|
# guards protect different expansions.
|
||||||
|
set -euf
|
||||||
|
|
||||||
|
# Paths under extension/ that are NOT packaged into the XPI.
|
||||||
|
#
|
||||||
|
# Split by whether git tracks them: node_modules and web-ext-artifacts are
|
||||||
|
# build/dependency output that never appears in a commit, so they belong in
|
||||||
|
# web-ext's ignore list but would be meaningless in a git pathspec.
|
||||||
|
#
|
||||||
|
# Directories need BOTH forms. `test/**` matches the files inside, but not the
|
||||||
|
# directory entry itself — web-ext writes an entry for the directory too, so
|
||||||
|
# with only the glob the XPI ends up carrying empty `test/` and `scripts/`
|
||||||
|
# entries (caught by the XPI-content check on 2026-08-03). The bare name alone
|
||||||
|
# is not enough either: minimatch's `test` does not match `test/url.spec.js`,
|
||||||
|
# so dropping the glob would ship the contents. Keep both.
|
||||||
|
NOT_PACKAGED_TRACKED='package.json package-lock.json README.md .gitignore vitest.config.js scripts scripts/** test test/**'
|
||||||
|
NOT_PACKAGED_BUILD='web-ext-artifacts node_modules'
|
||||||
|
|
||||||
|
# Paths under extension/ that cannot change the SHIPPED BYTES, and so must not
|
||||||
|
# move the derived version.
|
||||||
|
#
|
||||||
|
# Deliberately NOT the same list as NOT_PACKAGED_TRACKED, and the whole
|
||||||
|
# difference is `scripts/`. packaging.sh is not packaged into the XPI — but it
|
||||||
|
# DECIDES the version string, and build.yml stamps that string into the
|
||||||
|
# manifest.json that is packaged. A change to how the version is computed is
|
||||||
|
# therefore a change to the shipped bytes.
|
||||||
|
#
|
||||||
|
# Excluding it was harmless only while every push rebuilt the web image.
|
||||||
|
# Milestone 313 step 4 made the rebuild conditional on the derived revision
|
||||||
|
# moving, which turned it into a silent failure: a packaging.sh change gives a
|
||||||
|
# NEW version, so sign-extension misses its ext-<version> cache and signs —
|
||||||
|
# while build-web sees an unmoved revision, reuses the published image, and
|
||||||
|
# ships the OLD XPI. An orphaned AMO signature, and an instance quietly serving
|
||||||
|
# code the registry says is current.
|
||||||
|
#
|
||||||
|
# The two directions are not symmetric, which is why this list is the narrower
|
||||||
|
# one. Too wide costs a re-sign and a rebuild for a change that ships nothing
|
||||||
|
# new. Too narrow serves stale bytes and says nothing.
|
||||||
|
NOT_VERSION_RELEVANT='package.json package-lock.json README.md .gitignore vitest.config.js test test/**'
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: packaging.sh {ignore|pathspec|version|major-minor|patch}" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# web-ext --ignore-files values, space-separated.
|
||||||
|
#
|
||||||
|
# Callers MUST disable pathname expansion first (`set -f`), or the shell will
|
||||||
|
# glob `test/**` against the working tree before web-ext ever sees the pattern
|
||||||
|
# and silently narrow it to whatever happens to exist right now.
|
||||||
|
cmd_ignore() {
|
||||||
|
echo "$NOT_PACKAGED_TRACKED $NOT_PACKAGED_BUILD"
|
||||||
|
}
|
||||||
|
|
||||||
|
# git pathspec excluding the tracked files that cannot change the shipped
|
||||||
|
# bytes, e.g. :(exclude)extension/package.json :(exclude)extension/test/**
|
||||||
|
#
|
||||||
|
# This answers "what moves the version?", NOT "what goes in the XPI?" — see
|
||||||
|
# NOT_VERSION_RELEVANT for why those differ. cmd_ignore answers the other one.
|
||||||
|
# Same `set -f` requirement as above.
|
||||||
|
cmd_pathspec() {
|
||||||
|
for entry in $NOT_VERSION_RELEVANT; do
|
||||||
|
printf ':(exclude)extension/%s ' "$entry"
|
||||||
|
done
|
||||||
|
echo
|
||||||
|
}
|
||||||
|
|
||||||
|
# MAJOR.MINOR stays hand-set in manifest.json — it's the part that carries
|
||||||
|
# deliberate meaning. Only the patch component is derived.
|
||||||
|
cmd_major_minor() {
|
||||||
|
root=$(git rev-parse --show-toplevel)
|
||||||
|
grep -E '"version"' "$root/extension/manifest.json" \
|
||||||
|
| head -1 \
|
||||||
|
| sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([0-9]+)\.([0-9]+).*/\1.\2/'
|
||||||
|
}
|
||||||
|
|
||||||
|
# 2020-01-01T00:00:00Z — the anchor for the derived patch component. Fixed
|
||||||
|
# forever; moving it would renumber every version downwards.
|
||||||
|
VERSION_EPOCH=1577836800
|
||||||
|
|
||||||
|
# Minutes since VERSION_EPOCH of the LATEST commit that touched a PACKAGED
|
||||||
|
# extension file.
|
||||||
|
#
|
||||||
|
# Time-derived, per family rule 149: an artifact's ordering key must never be a
|
||||||
|
# commit count. A count is per-branch — `dev` and `main` count different
|
||||||
|
# histories of the same code — so the moment BOTH channels publish, their
|
||||||
|
# versions order by which branch accumulated more commits rather than by which
|
||||||
|
# is newer. A squash-merge makes that permanent: main gains one commit where dev
|
||||||
|
# gained five, so dev climbs away from main and a dev install can never cross
|
||||||
|
# back. That is Roundtable's 2026-08-24 incident (`versionCode` was the branch's
|
||||||
|
# commit count) in a different repo. Measured here on 2026-08-27: main=23,
|
||||||
|
# dev=24 under the old formula — one apart, which is exactly how the inversion
|
||||||
|
# stays invisible until it strands somebody.
|
||||||
|
#
|
||||||
|
# Why the commit's time and not the build's:
|
||||||
|
# * MONOTONIC — max() over a set that only ever gains members. Verified
|
||||||
|
# across all 24 extension-touching commits: zero non-monotonic steps.
|
||||||
|
# * STABLE while the extension is unchanged, so an unchanged extension keeps
|
||||||
|
# its version, the ext-<version> signature cache still hits, and AMO is
|
||||||
|
# called once per extension CHANGE rather than once per push. Build-time
|
||||||
|
# minutes would re-sign on every push and never let two channels share a
|
||||||
|
# signature.
|
||||||
|
# * SHARED ACROSS CHANNELS — after a merge, `main` sees the same commit and
|
||||||
|
# derives the same number, so `:latest` reuses the signature `:dev` already
|
||||||
|
# produced for byte-identical code. Same code, same version, one signing.
|
||||||
|
# * REPRODUCIBLE — any checkout of a commit yields that commit's version.
|
||||||
|
#
|
||||||
|
# Requires real history: a depth-1 clone sees one commit and will derive a wrong
|
||||||
|
# (too low) value. Every consumer must check out with fetch-depth: 0.
|
||||||
|
cmd_patch() {
|
||||||
|
root=$(git rev-parse --show-toplevel)
|
||||||
|
# Unquoted on purpose: the pathspec must word-split into separate args.
|
||||||
|
# Globbing is already off script-wide (set -euf above).
|
||||||
|
# shellcheck disable=SC2046
|
||||||
|
ts=$(cd "$root" && git log --format=%ct HEAD -- extension/ $(cmd_pathspec) \
|
||||||
|
| sort -n | tail -1)
|
||||||
|
if [ -z "$ts" ]; then
|
||||||
|
echo "packaging.sh: no commit touches a packaged extension file" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo $(( (ts - VERSION_EPOCH) / 60 ))
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_version() {
|
||||||
|
echo "$(cmd_major_minor).$(cmd_patch)"
|
||||||
|
}
|
||||||
|
|
||||||
|
[ $# -ge 1 ] || usage
|
||||||
|
case "$1" in
|
||||||
|
ignore) cmd_ignore ;;
|
||||||
|
pathspec) cmd_pathspec ;;
|
||||||
|
version) cmd_version ;;
|
||||||
|
major-minor) cmd_major_minor ;;
|
||||||
|
patch) cmd_patch ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import { readFileSync } from 'node:fs'
|
||||||
|
import { fileURLToPath } from 'node:url'
|
||||||
|
import path from 'node:path'
|
||||||
|
|
||||||
|
const LIB_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', '..', 'lib')
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load an extension lib and hand back the globals it declares.
|
||||||
|
*
|
||||||
|
* The files under lib/ are CLASSIC scripts, not ES modules: manifest.json
|
||||||
|
* lists them in `background.scripts` and options.html pulls them in with a
|
||||||
|
* plain <script> tag, so they declare bare functions into a shared scope and
|
||||||
|
* export nothing. Rather than bolt a `module.exports` shim onto production
|
||||||
|
* code that would never run in the browser, evaluate the real file the same
|
||||||
|
* way the browser does — as a script body — and pick the declarations back out.
|
||||||
|
*
|
||||||
|
* This means the specs exercise the exact bytes that get packaged into the
|
||||||
|
* XPI. Only usable for libs that touch no browser APIs at load time
|
||||||
|
* (url.js, platforms.js); cookies.js and api.js reference `browser.*` and
|
||||||
|
* would need stubbing, which is why they aren't loaded this way.
|
||||||
|
*
|
||||||
|
* @param {string} filename e.g. 'url.js'
|
||||||
|
* @param {string[]} names declarations to return, e.g. ['normalizeApiUrl']
|
||||||
|
*/
|
||||||
|
export function loadLib(filename, names) {
|
||||||
|
const source = readFileSync(path.join(LIB_DIR, filename), 'utf8')
|
||||||
|
const factory = new Function(`${source}\nreturn { ${names.join(', ')} }`)
|
||||||
|
return factory()
|
||||||
|
}
|
||||||
@@ -0,0 +1,190 @@
|
|||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { readFileSync } from 'node:fs'
|
||||||
|
import { fileURLToPath } from 'node:url'
|
||||||
|
import path from 'node:path'
|
||||||
|
import { loadLib } from './helpers/loadLib.js'
|
||||||
|
|
||||||
|
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
||||||
|
const manifest = JSON.parse(readFileSync(path.join(EXT_DIR, 'manifest.json'), 'utf8'))
|
||||||
|
|
||||||
|
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
|
||||||
|
'platforms.js',
|
||||||
|
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
|
||||||
|
)
|
||||||
|
|
||||||
|
describe('getPlatformFromUrl', () => {
|
||||||
|
it('identifies each platform from a domain URL', () => {
|
||||||
|
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
|
||||||
|
expect(getPlatformFromUrl('https://subscribestar.adult/someone')).toBe('subscribestar')
|
||||||
|
expect(getPlatformFromUrl('https://www.hentai-foundry.com/user/someone')).toBe('hentaifoundry')
|
||||||
|
expect(getPlatformFromUrl('https://discord.com/channels/@me')).toBe('discord')
|
||||||
|
expect(getPlatformFromUrl('https://www.pixiv.net/en/users/123')).toBe('pixiv')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('accepts http as well as https, with or without www', () => {
|
||||||
|
expect(getPlatformFromUrl('http://patreon.com/Atole')).toBe('patreon')
|
||||||
|
expect(getPlatformFromUrl('https://www.patreon.com/Atole')).toBe('patreon')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns null for unrelated hosts', () => {
|
||||||
|
expect(getPlatformFromUrl('https://example.com/patreon.com')).toBe(null)
|
||||||
|
expect(getPlatformFromUrl('https://not-patreon.com/Atole')).toBe(null)
|
||||||
|
expect(getPlatformFromUrl('')).toBe(null)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns null for deviantart, retired at #3069', () => {
|
||||||
|
// The 2026-07-05 product decision (FC downloaders = art-dedicated services
|
||||||
|
// only) left deviantart wired for seven weeks. Asserting the negative is
|
||||||
|
// what keeps a partial retirement from being re-completed by accident.
|
||||||
|
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe(null)
|
||||||
|
expect(PLATFORMS.deviantart).toBeUndefined()
|
||||||
|
expect(PLATFORM_ARTIST_PATTERNS.deviantart).toBeUndefined()
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('isArtistPage', () => {
|
||||||
|
// Regression cases from issue #1485: the Add-to-FC button vanished once the
|
||||||
|
// operator SUBSCRIBED to a creator, because Patreon serves subscribed users
|
||||||
|
// the /cw/ ("creator workspace") URL and the pattern only matched the bare
|
||||||
|
// root. All three creator URL shapes must match, plus inner pages — the
|
||||||
|
// button matters most exactly when you're subscribed.
|
||||||
|
it('matches all three Patreon creator URL shapes', () => {
|
||||||
|
expect(isArtistPage('https://www.patreon.com/Atole', 'patreon')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.patreon.com/c/Atole', 'patreon')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.patreon.com/cw/Atole', 'patreon')).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('matches Patreon creator inner pages', () => {
|
||||||
|
expect(isArtistPage('https://www.patreon.com/cw/Atole/posts', 'patreon')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.patreon.com/Atole/membership', 'patreon')).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('excludes Patreon navigation pages that are not creators', () => {
|
||||||
|
for (const nav of ['home', 'search', 'messages', 'notifications', 'library', 'settings']) {
|
||||||
|
expect(isArtistPage(`https://www.patreon.com/${nav}`, 'patreon')).toBe(false)
|
||||||
|
expect(isArtistPage(`https://www.patreon.com/${nav}/anything`, 'patreon')).toBe(false)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('matches SubscribeStar creator roots on both TLDs but not feed pages', () => {
|
||||||
|
expect(isArtistPage('https://subscribestar.adult/someone', 'subscribestar')).toBe(true)
|
||||||
|
expect(isArtistPage('https://subscribestar.com/someone', 'subscribestar')).toBe(true)
|
||||||
|
expect(isArtistPage('https://subscribestar.adult/feed', 'subscribestar')).toBe(false)
|
||||||
|
expect(isArtistPage('https://subscribestar.adult/messages', 'subscribestar')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('matches Hentai Foundry user pages only', () => {
|
||||||
|
expect(isArtistPage('https://www.hentai-foundry.com/user/someone', 'hentaifoundry')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.hentai-foundry.com/pictures/popular', 'hentaifoundry')).toBe(
|
||||||
|
false
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('matches Pixiv numeric user pages, with or without the /en/ prefix', () => {
|
||||||
|
expect(isArtistPage('https://www.pixiv.net/users/12345', 'pixiv')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.pixiv.net/en/users/12345', 'pixiv')).toBe(true)
|
||||||
|
expect(isArtistPage('https://www.pixiv.net/en/artworks/999', 'pixiv')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns false for a platform with no artist pattern (discord)', () => {
|
||||||
|
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns false for an unknown platform key', () => {
|
||||||
|
expect(isArtistPage('https://www.patreon.com/Atole', 'nope')).toBe(false)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('platform table integrity', () => {
|
||||||
|
it('gives every artist pattern a corresponding platform entry', () => {
|
||||||
|
// A pattern keyed to a platform that no longer exists is dead code that
|
||||||
|
// silently never fires; the reverse (a platform with no pattern) is the
|
||||||
|
// legitimate discord case, so only this direction is an error.
|
||||||
|
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
|
||||||
|
expect(Object.keys(PLATFORMS)).toContain(key)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('gives every platform the fields the popup renders', () => {
|
||||||
|
for (const [key, platform] of Object.entries(PLATFORMS)) {
|
||||||
|
expect(platform.name, `${key}.name`).toBeTruthy()
|
||||||
|
expect(platform.color, `${key}.color`).toMatch(/^#[0-9A-Fa-f]{6}$/)
|
||||||
|
expect(['cookies', 'token'], `${key}.authType`).toContain(platform.authType)
|
||||||
|
expect(platform.urlPattern, `${key}.urlPattern`).toBeInstanceOf(RegExp)
|
||||||
|
expect(Array.isArray(platform.domains), `${key}.domains`).toBe(true)
|
||||||
|
expect(platform.domains.length, `${key}.domains`).toBeGreaterThan(0)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps every artist URL matched by its own platform pattern too', () => {
|
||||||
|
// isArtistPage is only ever consulted after getPlatformFromUrl resolves a
|
||||||
|
// key, so an artist pattern matching a URL its platform's urlPattern
|
||||||
|
// rejects would be unreachable.
|
||||||
|
const samples = {
|
||||||
|
patreon: 'https://www.patreon.com/cw/Atole',
|
||||||
|
subscribestar: 'https://subscribestar.adult/someone',
|
||||||
|
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
|
||||||
|
pixiv: 'https://www.pixiv.net/en/users/12345'
|
||||||
|
}
|
||||||
|
for (const [key, url] of Object.entries(samples)) {
|
||||||
|
expect(isArtistPage(url, key), `${key} artist pattern`).toBe(true)
|
||||||
|
expect(getPlatformFromUrl(url), `${key} urlPattern`).toBe(key)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('manifest.json agrees with the platform table', () => {
|
||||||
|
// #3069: deviantart was dropped from the product in July but survived in
|
||||||
|
// manifest.json until late August, because NOTHING tied the manifest's
|
||||||
|
// domain lists back to PLATFORMS. These two specs are that tie. Both
|
||||||
|
// directions matter: a stale match ships host access the product decided
|
||||||
|
// not to use, and a missing one silently kills the Add-to-FC button.
|
||||||
|
const matches = manifest.content_scripts[0].matches
|
||||||
|
// '*://*.patreon.com/*' -> '.patreon.com', the form PLATFORMS.domains uses.
|
||||||
|
const hostOf = (m) => m.replace(/^\*:\/\/\*/, '').replace(/\/\*$/, '')
|
||||||
|
|
||||||
|
it('injects the content script only on domains a platform claims', () => {
|
||||||
|
for (const m of matches) {
|
||||||
|
const host = hostOf(m)
|
||||||
|
const owner = Object.entries(PLATFORMS).find(
|
||||||
|
([, p]) => p.domains.includes(host)
|
||||||
|
)
|
||||||
|
expect(owner, `no platform claims content-script match "${m}"`).toBeTruthy()
|
||||||
|
// The content script exists to draw the Add-as-source button, so a
|
||||||
|
// platform with no artist pattern (discord) has no business here.
|
||||||
|
expect(
|
||||||
|
PLATFORM_ARTIST_PATTERNS[owner[0]],
|
||||||
|
`"${m}" injects for ${owner[0]}, which has no artist pattern`
|
||||||
|
).toBeTruthy()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('injects on every platform that has an artist pattern', () => {
|
||||||
|
const covered = new Set(
|
||||||
|
matches
|
||||||
|
.map(hostOf)
|
||||||
|
.map((h) => Object.entries(PLATFORMS).find(([, p]) => p.domains.includes(h)))
|
||||||
|
.filter(Boolean)
|
||||||
|
.map(([key]) => key)
|
||||||
|
)
|
||||||
|
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
|
||||||
|
expect(covered, `${key} has an artist pattern but no content-script match`).toContain(key)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('requests no host permission for a domain no platform claims', () => {
|
||||||
|
// '*://*/*' is the deliberate exception: FC is self-hosted at an arbitrary
|
||||||
|
// operator-chosen URL, so the extension cannot enumerate its own backend.
|
||||||
|
// Every OTHER entry is a platform domain and must still have an owner.
|
||||||
|
for (const h of manifest.host_permissions) {
|
||||||
|
if (h === '*://*/*') continue
|
||||||
|
const host = hostOf(h)
|
||||||
|
// pixiv's OAuth/API hosts are pixiv infrastructure, not creator pages,
|
||||||
|
// so they are matched by suffix rather than by the domains list.
|
||||||
|
const claimed = Object.values(PLATFORMS).some(
|
||||||
|
(p) => p.domains.includes(host) || p.domains.some((d) => host.endsWith(d))
|
||||||
|
)
|
||||||
|
expect(claimed, `host permission "${h}" belongs to no platform`).toBe(true)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { loadLib } from './helpers/loadLib.js'
|
||||||
|
|
||||||
|
const { normalizeApiUrl, webRootFromApiUrl } = loadLib('url.js', [
|
||||||
|
'normalizeApiUrl',
|
||||||
|
'webRootFromApiUrl'
|
||||||
|
])
|
||||||
|
|
||||||
|
describe('normalizeApiUrl', () => {
|
||||||
|
// The bug this exists for (issue #2393): the instance root was accepted and
|
||||||
|
// stored verbatim, so every request went to /credentials instead of
|
||||||
|
// /api/credentials. That path is a Vue router route, so the SPA catch-all
|
||||||
|
// answered GET with 200 HTML and rejected POST with 405 — which read as a
|
||||||
|
// backend bug rather than a URL one.
|
||||||
|
it('appends /api to an instance root', () => {
|
||||||
|
expect(normalizeApiUrl('http://curator.traefik.internal')).toBe(
|
||||||
|
'http://curator.traefik.internal/api'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves an API root alone rather than doubling the suffix', () => {
|
||||||
|
expect(normalizeApiUrl('http://curator.traefik.internal/api')).toBe(
|
||||||
|
'http://curator.traefik.internal/api'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('is idempotent', () => {
|
||||||
|
const once = normalizeApiUrl('http://curator.example.com')
|
||||||
|
expect(normalizeApiUrl(once)).toBe(once)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('strips trailing slashes before deciding', () => {
|
||||||
|
expect(normalizeApiUrl('http://curator.example.com/')).toBe('http://curator.example.com/api')
|
||||||
|
expect(normalizeApiUrl('http://curator.example.com///')).toBe('http://curator.example.com/api')
|
||||||
|
expect(normalizeApiUrl('http://curator.example.com/api/')).toBe('http://curator.example.com/api')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('trims surrounding whitespace (paste artifacts)', () => {
|
||||||
|
expect(normalizeApiUrl(' http://curator.example.com ')).toBe(
|
||||||
|
'http://curator.example.com/api'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('matches the /api suffix case-insensitively', () => {
|
||||||
|
expect(normalizeApiUrl('http://curator.example.com/API')).toBe('http://curator.example.com/API')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns empty string for empty/nullish input, never a bare "/api"', () => {
|
||||||
|
// isConfigured() gates on truthiness, so a bogus '/api' here would read as
|
||||||
|
// "configured" and produce a request against the options page's own origin.
|
||||||
|
expect(normalizeApiUrl('')).toBe('')
|
||||||
|
expect(normalizeApiUrl(' ')).toBe('')
|
||||||
|
expect(normalizeApiUrl(null)).toBe('')
|
||||||
|
expect(normalizeApiUrl(undefined)).toBe('')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not treat a path merely containing "api" as the suffix', () => {
|
||||||
|
expect(normalizeApiUrl('http://curator.example.com/apiary')).toBe(
|
||||||
|
'http://curator.example.com/apiary/api'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('preserves a subpath deployment', () => {
|
||||||
|
expect(normalizeApiUrl('http://host.internal/curator')).toBe('http://host.internal/curator/api')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('webRootFromApiUrl', () => {
|
||||||
|
// The SPA root, where the Vue router and the served XPI live. Used by
|
||||||
|
// OPEN_ARTIST_PAGE and the self-update check — NOT the JSON API.
|
||||||
|
it('strips the /api suffix', () => {
|
||||||
|
expect(webRootFromApiUrl('http://curator.example.com/api')).toBe('http://curator.example.com')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('accepts an instance root unchanged', () => {
|
||||||
|
expect(webRootFromApiUrl('http://curator.example.com')).toBe('http://curator.example.com')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('agrees with normalizeApiUrl in both directions', () => {
|
||||||
|
for (const input of ['http://curator.example.com', 'http://curator.example.com/api']) {
|
||||||
|
expect(normalizeApiUrl(webRootFromApiUrl(input))).toBe(normalizeApiUrl(input))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('preserves a subpath deployment', () => {
|
||||||
|
expect(webRootFromApiUrl('http://host.internal/curator/api')).toBe('http://host.internal/curator')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('returns empty string for empty/nullish input', () => {
|
||||||
|
expect(webRootFromApiUrl('')).toBe('')
|
||||||
|
expect(webRootFromApiUrl(null)).toBe('')
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { readFileSync } from 'node:fs'
|
||||||
|
import { execFileSync } from 'node:child_process'
|
||||||
|
import { fileURLToPath } from 'node:url'
|
||||||
|
import path from 'node:path'
|
||||||
|
|
||||||
|
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
||||||
|
const read = (name) => JSON.parse(readFileSync(path.join(EXT_DIR, name), 'utf8'))
|
||||||
|
const readText = (...seg) => readFileSync(path.join(EXT_DIR, ...seg), 'utf8')
|
||||||
|
|
||||||
|
// Only the git-free subcommands are exercised here: `version`/`patch` shell out
|
||||||
|
// to git, and the extension lane runs on node:24-bookworm-slim which may not
|
||||||
|
// ship it. Those two are covered where git is guaranteed — ci.yml and build.yml
|
||||||
|
// run on ci-python.
|
||||||
|
const packaging = (cmd) =>
|
||||||
|
execFileSync('sh', [path.join(EXT_DIR, 'scripts', 'packaging.sh'), cmd], {
|
||||||
|
cwd: EXT_DIR,
|
||||||
|
encoding: 'utf8'
|
||||||
|
})
|
||||||
|
.trim()
|
||||||
|
.split(/\s+/)
|
||||||
|
.filter(Boolean)
|
||||||
|
|
||||||
|
describe('packaging.sh — the single definition of what ships', () => {
|
||||||
|
it('emits an ignore list and a pathspec that agree on the tracked files', () => {
|
||||||
|
const ignore = packaging('ignore')
|
||||||
|
const pathspec = packaging('pathspec').map((e) => e.replace(':(exclude)extension/', ''))
|
||||||
|
|
||||||
|
// Every git-excluded path must also be hidden from web-ext. The reverse is
|
||||||
|
// not required: node_modules and web-ext-artifacts are build output git
|
||||||
|
// never tracks, so they appear only in the ignore list.
|
||||||
|
for (const entry of pathspec) {
|
||||||
|
expect(ignore, `pathspec has "${entry}" but --ignore-files does not`).toContain(entry)
|
||||||
|
}
|
||||||
|
expect(pathspec.length).toBeGreaterThan(0)
|
||||||
|
expect(ignore).toContain('node_modules')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('emits glob patterns literally, never expanded against the working tree', () => {
|
||||||
|
// The script iterates its lists with deliberate word-splitting, so it must
|
||||||
|
// run with pathname expansion off. Without that, invoking it from a cwd
|
||||||
|
// where test/ exists (exactly how ci.yml and vitest call it) expands
|
||||||
|
// `test/**` into the individual spec files, and the pathspec silently stops
|
||||||
|
// covering anything added later.
|
||||||
|
const pathspec = packaging('pathspec')
|
||||||
|
expect(pathspec).toContain(':(exclude)extension/test/**')
|
||||||
|
expect(pathspec.some((e) => e.includes('.spec.js'))).toBe(false)
|
||||||
|
expect(pathspec.some((e) => e.includes('helpers'))).toBe(false)
|
||||||
|
|
||||||
|
const ignore = packaging('ignore')
|
||||||
|
expect(ignore).toContain('test/**')
|
||||||
|
expect(ignore).toContain('scripts/**')
|
||||||
|
expect(ignore.some((e) => e.includes('.spec.js'))).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('lets packaging.sh move the version, though it never ships in the XPI', () => {
|
||||||
|
// The two lists answer different questions and this is the one place they
|
||||||
|
// disagree. scripts/ is ignored by web-ext — it is repo tooling, not addon
|
||||||
|
// code — but packaging.sh DECIDES the version string, and build.yml stamps
|
||||||
|
// that string into the manifest.json that does ship. So changing how the
|
||||||
|
// version is computed changes the shipped bytes.
|
||||||
|
//
|
||||||
|
// Excluding it from the pathspec was invisible while every push rebuilt the
|
||||||
|
// web image. Milestone 313 step 4 made that rebuild conditional on the
|
||||||
|
// derived revision moving, and the omission turned into a silent failure:
|
||||||
|
// a new version means sign-extension misses its ext-<version> cache and
|
||||||
|
// signs, while build-web sees an unmoved revision, reuses the published
|
||||||
|
// image and ships the OLD XPI. An orphaned signature, and an instance
|
||||||
|
// serving code the registry calls current.
|
||||||
|
const pathspec = packaging('pathspec')
|
||||||
|
expect(
|
||||||
|
pathspec.some((e) => e.startsWith(':(exclude)extension/scripts')),
|
||||||
|
'the pathspec excludes scripts/, so a change to how the version is '
|
||||||
|
+ 'derived would not move the version it derives',
|
||||||
|
).toBe(false)
|
||||||
|
|
||||||
|
// ...and it is still kept out of the package itself. Both must hold: the
|
||||||
|
// tempting "fix" for either half is to make the two lists one again.
|
||||||
|
expect(packaging('ignore')).toContain('scripts')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps its own scripts and specs out of the XPI', () => {
|
||||||
|
// Both are repo infrastructure. web-ext packages everything not ignored, so
|
||||||
|
// omitting either would ship dev tooling to users -- and `test/**` in
|
||||||
|
// particular only survives because callers `set -f` before substituting it.
|
||||||
|
const ignore = packaging('ignore')
|
||||||
|
expect(ignore).toContain('vitest.config.js')
|
||||||
|
// Both forms per directory. The glob covers the contents; the bare name
|
||||||
|
// covers the directory ENTRY, which web-ext writes separately — with only
|
||||||
|
// the glob, the XPI carries an empty `test/` and `scripts/`.
|
||||||
|
for (const dir of ['test', 'scripts']) {
|
||||||
|
expect(ignore, `${dir} contents`).toContain(`${dir}/**`)
|
||||||
|
expect(ignore, `${dir} directory entry`).toContain(dir)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('consumers delegate rather than keeping their own copy', () => {
|
||||||
|
// These assertions are the actual anti-regression value: it is easy for a
|
||||||
|
// future edit to "simplify" by inlining a literal list again, which silently
|
||||||
|
// reintroduces the drift that issue #2397 was about.
|
||||||
|
it('package.json derives --ignore-files from the script', () => {
|
||||||
|
for (const [name, script] of Object.entries(read('package.json').scripts)) {
|
||||||
|
if (!script.includes('--ignore-files')) continue
|
||||||
|
expect(script, `${name} should call packaging.sh`).toContain('scripts/packaging.sh ignore')
|
||||||
|
expect(script, `${name} must set -f before the substitution`).toMatch(/set -f;/)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
const WORKFLOWS = ['ci.yml', 'build.yml', 'extension.yml']
|
||||||
|
|
||||||
|
it('no workflow hardcodes the packaged-file set', () => {
|
||||||
|
// ci.yml used to substitute `packaging.sh pathspec` directly, for the
|
||||||
|
// manual-bump guard that milestone 271 step 5 retired. Nothing inlines the
|
||||||
|
// set today, and nothing should start to: a literal :(exclude)extension/...
|
||||||
|
// in a workflow means someone bypassed the shared definition, which is
|
||||||
|
// exactly the drift #2397 was about. Asserted across all three rather than
|
||||||
|
// against one named consumer, so it keeps holding as consumers come and go.
|
||||||
|
for (const wf of WORKFLOWS) {
|
||||||
|
const text = readText('..', '.forgejo', 'workflows', wf)
|
||||||
|
expect(text, `${wf} inlines an :(exclude) literal`).not.toMatch(/:\(exclude\)extension\//)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('build.yml takes the shipped version from the script, not from the repo', () => {
|
||||||
|
// The version is DERIVED from commit time (#3092, milestone 271 step 4).
|
||||||
|
// Going back to reading the committed value is not a style regression, it
|
||||||
|
// is the bug: a hand-set version makes dev and main sign the same number
|
||||||
|
// for different code, and the ext-<version> cache then serves one channel
|
||||||
|
// the other's XPI.
|
||||||
|
const build = readText('..', '.forgejo', 'workflows', 'build.yml')
|
||||||
|
expect(build).toContain('packaging.sh version')
|
||||||
|
expect(build, 'build.yml re-reads the committed version instead of deriving it')
|
||||||
|
.not.toMatch(/grep[^\n]*'"version"'[^\n]*package\.json/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('extension version', () => {
|
||||||
|
const majorMinor = (v) => v.split('.').slice(0, 2).join('.')
|
||||||
|
|
||||||
|
it('keeps the hand-set MAJOR.MINOR in lockstep across both files', () => {
|
||||||
|
// Narrowed from full-string equality at milestone 271 step 5. Since step 4
|
||||||
|
// the patch component is derived from commit time and stamped into both
|
||||||
|
// files at build time, so the committed patch numbers are inert — nothing
|
||||||
|
// reads them and they are not what ships. Asserting on them would fail for
|
||||||
|
// a difference that changes nothing.
|
||||||
|
//
|
||||||
|
// MAJOR.MINOR is the opposite: still hand-set, still shipped, and
|
||||||
|
// packaging.sh reads it from manifest.json ALONE. Let the two diverge and
|
||||||
|
// the extension ships a version package.json disagrees with, with no other
|
||||||
|
// signal.
|
||||||
|
expect(majorMinor(read('manifest.json').version))
|
||||||
|
.toBe(majorMinor(read('package.json').version))
|
||||||
|
})
|
||||||
|
|
||||||
|
it('uses a plain dotted numeric version AMO will accept', () => {
|
||||||
|
// The committed value seeds MAJOR.MINOR, so it still has to parse even
|
||||||
|
// though its patch component never ships. ci.yml asserts the same shape on
|
||||||
|
// the DERIVED value, which is the one AMO actually sees.
|
||||||
|
expect(read('package.json').version).toMatch(/^\d+(\.\d+)*$/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('declares manifest v3', () => {
|
||||||
|
expect(read('manifest.json').manifest_version).toBe(3)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('lists every background script that exists, in dependency order', () => {
|
||||||
|
// url.js must load BEFORE api.js: api.js calls normalizeApiUrl at
|
||||||
|
// init()-time, and these are classic scripts sharing one scope, so a
|
||||||
|
// reordering here is a runtime ReferenceError with no build-time signal.
|
||||||
|
const scripts = read('manifest.json').background.scripts
|
||||||
|
for (const rel of scripts) {
|
||||||
|
expect(() => readFileSync(path.join(EXT_DIR, rel)), `missing ${rel}`).not.toThrow()
|
||||||
|
}
|
||||||
|
expect(scripts.indexOf('lib/url.js')).toBeLessThan(scripts.indexOf('lib/api.js'))
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
import { defineConfig } from 'vitest/config'
|
||||||
|
|
||||||
|
// Mirrors frontend/vitest.config.js, minus the Vue plugin — the extension has
|
||||||
|
// no SFCs and mounts nothing. Pure-logic specs only, so `node` is enough; the
|
||||||
|
// libs under test are deliberately the ones with no browser-API surface (see
|
||||||
|
// test/helpers/loadLib.js).
|
||||||
|
export default defineConfig({
|
||||||
|
test: {
|
||||||
|
environment: 'node',
|
||||||
|
include: ['test/**/*.spec.js'],
|
||||||
|
passWithNoTests: true
|
||||||
|
}
|
||||||
|
})
|
||||||
+11
-13
@@ -4,30 +4,28 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=22"
|
"node": ">=24"
|
||||||
},
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite",
|
"dev": "vite",
|
||||||
"build": "vite build",
|
"build": "vite build",
|
||||||
"preview": "vite preview",
|
"preview": "vite preview",
|
||||||
"test:unit": "vitest run",
|
"test:unit": "vitest run"
|
||||||
"check": "vue-tsc --noEmit"
|
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"vue": "^3.4.0",
|
"vue": "^3.5.0",
|
||||||
"vue-router": "^4.3.0",
|
"vue-router": "^5.0.0",
|
||||||
"pinia": "^2.1.0",
|
"pinia": "^3.0.0",
|
||||||
"vuetify": "^3.5.0",
|
"vuetify": "^4.0.0",
|
||||||
"@mdi/font": "^7.4.0"
|
"@mdi/font": "^7.4.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@vitejs/plugin-vue": "^5.0.0",
|
"@vitejs/plugin-vue": "^6.0.0",
|
||||||
"vite": "^5.2.0",
|
"vite": "^8.0.0",
|
||||||
"vue-tsc": "^2.0.0",
|
"vite-plugin-vuetify": "^2.1.0",
|
||||||
"vite-plugin-vuetify": "^2.0.0",
|
|
||||||
"sass": "^1.71.0",
|
"sass": "^1.71.0",
|
||||||
"vitest": "^2.1.0",
|
"vitest": "^4.0.0",
|
||||||
"@vue/test-utils": "^2.4.0",
|
"@vue/test-utils": "^2.4.0",
|
||||||
"happy-dom": "^15.0.0"
|
"happy-dom": "^20.0.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,11 +18,11 @@ const route = useRoute()
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-content {
|
.fc-content {
|
||||||
min-height: 100vh;
|
min-height: 100vh;
|
||||||
/* Push initial viewport content below the sticky TopNav. Without
|
/* NO padding-top: the TopNav is position:sticky, so it already reserves its
|
||||||
this, some views' first rows / form fields / table headers can
|
own space in the v-app flex column — content flows directly below it. The
|
||||||
end up obscured by the navbar (depending on parent overflow
|
old 64px padding-top was a leftover from a FIXED navbar and double-counted
|
||||||
context interacting with position: sticky). Scrolled-down content
|
that space, leaving a large empty band at the top of EVERY view (and pushing
|
||||||
still slides under the nav — the gradient-fade design is intact. */
|
the full-height calc(100vh - 64px) views down so they overflowed). Removed
|
||||||
padding-top: 64px;
|
2026-07-13. Scrolled content still slides under the sticky nav as before. */
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<template>
|
<template>
|
||||||
<v-snackbar
|
<v-snackbar
|
||||||
v-model="show" :color="color" location="bottom right" timeout="4000"
|
v-model="show" :color="color" location="bottom right" timeout="4000"
|
||||||
multi-line elevation="4"
|
min-height="68" elevation="4"
|
||||||
>
|
>
|
||||||
{{ message }}
|
{{ message }}
|
||||||
<template #actions>
|
<template #actions>
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<template>
|
<template>
|
||||||
<header class="fc-topnav">
|
<header ref="navEl" class="fc-topnav" :class="{ 'fc-topnav--chrome': hasStickyChrome }">
|
||||||
<div class="fc-nav-left">
|
<div class="fc-nav-left">
|
||||||
<RouterLink :to="FRONT_DOOR" class="fc-brand" aria-label="FabledCurator home">
|
<RouterLink :to="FRONT_DOOR" class="fc-brand" aria-label="FabledCurator home">
|
||||||
<img src="/favicon.svg" alt="" class="fc-brand__glyph" width="22" height="22" />
|
<img src="/favicon.svg" alt="" class="fc-brand__glyph" width="22" height="22" />
|
||||||
@@ -64,13 +64,39 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, onMounted } from 'vue'
|
import { computed, onBeforeUnmount, onMounted, ref } from 'vue'
|
||||||
|
import { useRoute } from 'vue-router'
|
||||||
import router, { FRONT_DOOR } from '../router.js'
|
import router, { FRONT_DOOR } from '../router.js'
|
||||||
import { useSystemStore } from '../stores/system.js'
|
import { useSystemStore } from '../stores/system.js'
|
||||||
import PipelineStatusChip from './PipelineStatusChip.vue'
|
import PipelineStatusChip from './PipelineStatusChip.vue'
|
||||||
|
|
||||||
const system = useSystemStore()
|
const system = useSystemStore()
|
||||||
onMounted(() => system.refreshHealth())
|
|
||||||
|
// Publish the nav's REAL height as --fc-nav-h so full-height workspaces
|
||||||
|
// (Explore/Subscriptions) and sticky sub-headers pin to it exactly instead of a
|
||||||
|
// hardcoded 64px that Vuetify 4's MD3 sizing broke — the Explore breadcrumb was
|
||||||
|
// tucking under a taller nav (#1481). ResizeObserver keeps it live as the nav
|
||||||
|
// reflows (per-view teleported actions, mobile breakpoint, chip state changes).
|
||||||
|
const navEl = ref(null)
|
||||||
|
let navRO = null
|
||||||
|
onMounted(() => {
|
||||||
|
system.refreshHealth()
|
||||||
|
if (navEl.value && 'ResizeObserver' in window) {
|
||||||
|
navRO = new ResizeObserver(() => {
|
||||||
|
const h = navEl.value?.offsetHeight
|
||||||
|
if (h) document.documentElement.style.setProperty('--fc-nav-h', `${h}px`)
|
||||||
|
})
|
||||||
|
navRO.observe(navEl.value)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
onBeforeUnmount(() => { navRO?.disconnect() })
|
||||||
|
|
||||||
|
// Views that pin a sticky sub-header (filter bar / tabs) directly under the nav
|
||||||
|
// declare `meta.stickyChrome`. On those, the nav doesn't fade to transparent at
|
||||||
|
// its bottom — it hands off at the shared seam alpha so the sub-header can
|
||||||
|
// continue the SAME fade (see .fc-chrome-continues in app.css). One gradient.
|
||||||
|
const route = useRoute()
|
||||||
|
const hasStickyChrome = computed(() => !!route.meta?.stickyChrome)
|
||||||
|
|
||||||
// Every route with a meta.title is a nav entry. Order by meta.navOrder —
|
// Every route with a meta.title is a nav entry. Order by meta.navOrder —
|
||||||
// router.getRoutes() does NOT guarantee declaration order, so explicit numbers
|
// router.getRoutes() does NOT guarantee declaration order, so explicit numbers
|
||||||
@@ -119,16 +145,35 @@ const health = computed(() => {
|
|||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 1rem;
|
gap: 1rem;
|
||||||
padding: 0.75rem 1rem;
|
padding: 0.75rem 1rem;
|
||||||
/* Obsidian (#14171A = 20,23,26) gradient fade — content scrolls under it. */
|
/* Obsidian (#14171A) fade — content scrolls under it. Holds high (0.92 →
|
||||||
|
0.84) through the top half, then eases to transparent over the bottom
|
||||||
|
quarter so it tails off softly instead of a straight line to a hard edge
|
||||||
|
(operator 2026-07-13). Shared --fc-chrome-rgb keeps it in sync with the
|
||||||
|
sub-header continuation. */
|
||||||
background: linear-gradient(
|
background: linear-gradient(
|
||||||
to bottom,
|
to bottom,
|
||||||
rgba(20, 23, 26, 0.92) 0%,
|
rgba(var(--fc-chrome-rgb), 0.92) 0%,
|
||||||
rgba(20, 23, 26, 0.65) 60%,
|
rgba(var(--fc-chrome-rgb), 0.84) 50%,
|
||||||
rgba(20, 23, 26, 0) 100%
|
rgba(var(--fc-chrome-rgb), 0.55) 75%,
|
||||||
|
rgba(var(--fc-chrome-rgb), 0) 100%
|
||||||
);
|
);
|
||||||
backdrop-filter: blur(2px);
|
backdrop-filter: blur(2px);
|
||||||
-webkit-backdrop-filter: blur(2px);
|
-webkit-backdrop-filter: blur(2px);
|
||||||
}
|
}
|
||||||
|
/* On a view with a sticky sub-header pinned beneath (meta.stickyChrome), the nav
|
||||||
|
stops fading at the shared seam alpha instead of going fully transparent — the
|
||||||
|
sub-header (.fc-chrome-continues) picks the fade up from there, so the two read
|
||||||
|
as one continuous gradient. Compound selector out-specifies .fc-topnav so it
|
||||||
|
wins regardless of Vite's production CSS ordering. --fc-chrome-* come from the
|
||||||
|
global :root in app.css (custom props inherit into scoped styles). */
|
||||||
|
.fc-topnav.fc-topnav--chrome {
|
||||||
|
background: linear-gradient(
|
||||||
|
to bottom,
|
||||||
|
rgba(var(--fc-chrome-rgb), 0.92) 0%,
|
||||||
|
rgba(var(--fc-chrome-rgb), 0.84) 60%,
|
||||||
|
rgba(var(--fc-chrome-rgb), var(--fc-chrome-seam)) 100%
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
.fc-brand {
|
.fc-brand {
|
||||||
display: flex;
|
display: flex;
|
||||||
|
|||||||
@@ -50,14 +50,19 @@ const projected = ref(null)
|
|||||||
|
|
||||||
const projectedCounts = computed(() => projected.value?.projected || null)
|
const projectedCounts = computed(() => projected.value?.projected || null)
|
||||||
|
|
||||||
const modalDescription = computed(
|
// `posts` is named here, not left to the counts grid below it: an artist whose
|
||||||
() => projected.value
|
// posts are body-only previews as `images: 0`, and a summary line that says
|
||||||
? `Artist “${props.artistName}” — `
|
// only "0 images" reads as "this artist is empty" while the apply destroys
|
||||||
+ `${projected.value.projected.images} images, `
|
// every captured post body (#3067). Attachments stay in the grid — the grid
|
||||||
+ `${projected.value.projected.sources} sources, `
|
// renders every key, so this line carries only what changes the read.
|
||||||
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
|
const modalDescription = computed(() => {
|
||||||
: '',
|
const p = projectedCounts.value
|
||||||
)
|
return p
|
||||||
|
? `Artist “${props.artistName}” — ${p.images} images, `
|
||||||
|
+ `${p.posts} posts, ${p.sources} sources, `
|
||||||
|
+ `${Math.round(p.bytes_on_disk / 1_048_576)} MiB on disk`
|
||||||
|
: ''
|
||||||
|
})
|
||||||
|
|
||||||
async function onClick() {
|
async function onClick() {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
filter, applied retroactively to the existing library.
|
filter, applied retroactively to the existing library.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<v-row dense>
|
<v-row density="compact">
|
||||||
<v-col cols="6">
|
<v-col cols="6">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model.number="minW" label="Min width (px)" type="number"
|
v-model.number="minW" label="Min width (px)" type="number"
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
cadence as the transparency audit.
|
cadence as the transparency audit.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<v-row dense>
|
<v-row density="compact">
|
||||||
<v-col cols="6">
|
<v-col cols="6">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model.number="threshold" label="Threshold (0–1)"
|
v-model.number="threshold" label="Threshold (0–1)"
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
<!--
|
||||||
|
Canonical settings number field (DRY pass #161): a compact numeric v-text-field
|
||||||
|
with a built-in clamp to [min,max] on commit. Hand-rolled identically across the
|
||||||
|
ML settings cards (HeadsCard x6, CropProposersCard, VideoEmbeddingCard).
|
||||||
|
|
||||||
|
The clamp is the point: the cards previously sent Number(raw) straight to the
|
||||||
|
API, so an out-of-range value bounced off the API's 400 validator (only
|
||||||
|
TranslationCard clamped). This is now the single home for that clamp.
|
||||||
|
|
||||||
|
Binds `modelValue` (v-model) and emits `change` on blur/enter AFTER clamping, so
|
||||||
|
the parent's save reads the already-clamped value — same as the prior
|
||||||
|
`v-model.number` + `@change=save` pattern.
|
||||||
|
-->
|
||||||
|
<template>
|
||||||
|
<v-text-field
|
||||||
|
:model-value="modelValue"
|
||||||
|
:label="label"
|
||||||
|
type="number"
|
||||||
|
:min="min"
|
||||||
|
:max="max"
|
||||||
|
:step="step"
|
||||||
|
:disabled="disabled"
|
||||||
|
:density="density" hide-details
|
||||||
|
:style="{ maxWidth }"
|
||||||
|
@update:model-value="v => emit('update:modelValue', v)"
|
||||||
|
@change="onCommit"
|
||||||
|
/>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
const props = defineProps({
|
||||||
|
modelValue: { type: [Number, String], default: null },
|
||||||
|
label: { type: String, default: '' },
|
||||||
|
min: { type: [Number, String], default: null },
|
||||||
|
max: { type: [Number, String], default: null },
|
||||||
|
step: { type: [Number, String], default: 1 },
|
||||||
|
maxWidth: { type: String, default: '200px' },
|
||||||
|
density: { type: String, default: 'compact' },
|
||||||
|
disabled: { type: Boolean, default: false },
|
||||||
|
})
|
||||||
|
const emit = defineEmits(['update:modelValue', 'change'])
|
||||||
|
|
||||||
|
function onCommit() {
|
||||||
|
// On blur/enter: coerce to a number and clamp to [min,max] so an out-of-range
|
||||||
|
// value never reaches the API. props.modelValue reflects the latest keystroke
|
||||||
|
// (kept in sync by the passthrough above); re-emit the clamped number, then let
|
||||||
|
// the parent persist.
|
||||||
|
let n = Number(props.modelValue)
|
||||||
|
if (!Number.isNaN(n)) {
|
||||||
|
if (props.min !== null && props.min !== '') n = Math.max(Number(props.min), n)
|
||||||
|
if (props.max !== null && props.max !== '') n = Math.min(Number(props.max), n)
|
||||||
|
if (n !== Number(props.modelValue)) emit('update:modelValue', n)
|
||||||
|
}
|
||||||
|
emit('change')
|
||||||
|
}
|
||||||
|
</script>
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
<!--
|
||||||
|
Canonical settings toggle row (DRY pass #161): an accent icon + an uppercase
|
||||||
|
.fc-section-h label + a right-aligned switch. Hand-rolled identically in the
|
||||||
|
ML settings cards (HeadsCard x3, CropProposersCard, MLBackfillCard).
|
||||||
|
|
||||||
|
Two-way binds `modelValue` (so the parent switch state stays optimistic) AND
|
||||||
|
emits `change` with the new boolean, so the parent can persist + revert on
|
||||||
|
failure — matching the prior `v-model` + `@update:model-value=handler` pattern.
|
||||||
|
-->
|
||||||
|
<template>
|
||||||
|
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
||||||
|
<v-icon v-if="icon" size="18" :color="iconColor">{{ icon }}</v-icon>
|
||||||
|
<span class="fc-section-h">{{ label }}</span>
|
||||||
|
<v-switch
|
||||||
|
:model-value="modelValue"
|
||||||
|
:loading="loading"
|
||||||
|
:disabled="disabled"
|
||||||
|
hide-details density="compact" color="success" class="ml-auto"
|
||||||
|
@update:model-value="onSwitch"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
defineProps({
|
||||||
|
modelValue: { type: Boolean, default: false },
|
||||||
|
label: { type: String, default: '' },
|
||||||
|
icon: { type: String, default: '' },
|
||||||
|
// Icon tint. Default accent; pass null for the theme default (e.g. when a row
|
||||||
|
// is off). null (not undefined) so the default doesn't override it.
|
||||||
|
iconColor: { type: String, default: 'accent' },
|
||||||
|
loading: { type: Boolean, default: false },
|
||||||
|
disabled: { type: Boolean, default: false },
|
||||||
|
})
|
||||||
|
const emit = defineEmits(['update:modelValue', 'change'])
|
||||||
|
|
||||||
|
function onSwitch(v) {
|
||||||
|
const b = !!v
|
||||||
|
emit('update:modelValue', b)
|
||||||
|
emit('change', b)
|
||||||
|
}
|
||||||
|
</script>
|
||||||
@@ -14,10 +14,10 @@
|
|||||||
@update:search="onSearch"
|
@update:search="onSearch"
|
||||||
@update:model-value="onPick"
|
@update:model-value="onPick"
|
||||||
>
|
>
|
||||||
<template #item="{ props: itemProps, item }">
|
<template #item="{ props: itemProps, internalItem }">
|
||||||
<v-list-item v-bind="itemProps" :title="item.raw.name">
|
<v-list-item v-bind="itemProps" :title="internalItem.raw.name">
|
||||||
<template #subtitle>
|
<template #subtitle>
|
||||||
{{ item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind }}
|
{{ internalItem.raw.fandom_name ? `character · ${internalItem.raw.fandom_name}` : internalItem.raw.kind }}
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
</template>
|
</template>
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<template>
|
<template>
|
||||||
<div class="fc-filterbar-wrap">
|
<div class="fc-filterbar-wrap fc-chrome-continues">
|
||||||
<div class="fc-filterbar">
|
<div class="fc-filterbar">
|
||||||
<v-autocomplete
|
<v-autocomplete
|
||||||
v-model="selected"
|
v-model="selected"
|
||||||
@@ -13,14 +13,14 @@
|
|||||||
@update:search="onSearch"
|
@update:search="onSearch"
|
||||||
@update:model-value="onPick"
|
@update:model-value="onPick"
|
||||||
>
|
>
|
||||||
<template #item="{ props: itemProps, item }">
|
<template #item="{ props: itemProps, internalItem }">
|
||||||
<v-list-item v-bind="itemProps" :title="item.raw.name">
|
<v-list-item v-bind="itemProps" :title="internalItem.raw.name">
|
||||||
<template #prepend>
|
<template #prepend>
|
||||||
<v-icon size="small">{{ iconFor(item.raw) }}</v-icon>
|
<v-icon size="small">{{ iconFor(internalItem.raw) }}</v-icon>
|
||||||
</template>
|
</template>
|
||||||
<template #subtitle>
|
<template #subtitle>
|
||||||
{{ item.raw.kind === 'artist' ? 'artist'
|
{{ internalItem.raw.kind === 'artist' ? 'artist'
|
||||||
: (item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind) }}
|
: (internalItem.raw.fandom_name ? `character · ${internalItem.raw.fandom_name}` : internalItem.raw.kind) }}
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
</template>
|
</template>
|
||||||
@@ -306,27 +306,17 @@ function pushFilter(mutate) {
|
|||||||
frosted block pinned directly under the 64px TopNav and continuous with it. */
|
frosted block pinned directly under the 64px TopNav and continuous with it. */
|
||||||
.fc-filterbar-wrap {
|
.fc-filterbar-wrap {
|
||||||
position: sticky;
|
position: sticky;
|
||||||
top: 64px;
|
top: var(--fc-nav-h, 64px); /* pins at the nav's real measured bottom (#1481) */
|
||||||
z-index: 5;
|
z-index: 5;
|
||||||
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
|
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
|
||||||
so the bar sits flush at 64px even at scroll 0 — without this it detaches
|
so the bar sits flush at 64px even at scroll 0 — without this it detaches
|
||||||
and a gap shows through when scrolled to the top. */
|
and a gap shows through when scrolled to the top. */
|
||||||
margin-top: -8px;
|
margin-top: -8px;
|
||||||
margin-bottom: 12px;
|
margin-bottom: 12px;
|
||||||
/* EXACT same gradiated obsidian (#14171A = 20,23,26) frost as the TopNav so
|
/* The frost itself (obsidian fade + blur) is the shared .fc-chrome-continues
|
||||||
the two read as one continuous piece of chrome — images scroll visibly
|
primitive: it CONTINUES the TopNav's fade from the seam alpha to transparent
|
||||||
under both. The nav's gradient fades to transparent at ITS bottom; this
|
rather than re-darkening, so the nav + bar read as one gradient (operator
|
||||||
bar re-darkens at its top, so a faint seam (the page/image showing through
|
2026-07-13). This block only owns the sticky positioning now. */
|
||||||
the nav's transparent edge) separates them when scrolled to the very top,
|
|
||||||
while under-scroll they frost as one. */
|
|
||||||
background: linear-gradient(
|
|
||||||
to bottom,
|
|
||||||
rgba(20, 23, 26, 0.92) 0%,
|
|
||||||
rgba(20, 23, 26, 0.65) 60%,
|
|
||||||
rgba(20, 23, 26, 0) 100%
|
|
||||||
);
|
|
||||||
backdrop-filter: blur(2px);
|
|
||||||
-webkit-backdrop-filter: blur(2px);
|
|
||||||
}
|
}
|
||||||
.fc-filterbar {
|
.fc-filterbar {
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -341,6 +331,20 @@ function pushFilter(mutate) {
|
|||||||
.fc-filterbar-wrap :deep(.v-btn-group) {
|
.fc-filterbar-wrap :deep(.v-btn-group) {
|
||||||
background-color: rgba(20, 23, 26, 0.72);
|
background-color: rgba(20, 23, 26, 0.72);
|
||||||
}
|
}
|
||||||
|
/* Media toggle (All / Images / Videos) as ONE cohesive segmented control.
|
||||||
|
FC's global VBtn { rounded: 'pill' } default made Vuetify 4 pill-round each
|
||||||
|
SEGMENT individually, so the rounded ends collided at the joins — the shapes
|
||||||
|
landed awkwardly on the button edges (operator 2026-07-13). Square the inner
|
||||||
|
segments (over the pill utility's !important) and clip the group to a single
|
||||||
|
8px outline (matches the chips/tiles rounding elsewhere in the app). Radius
|
||||||
|
only — no height change, so the bar height and nav offset are untouched. */
|
||||||
|
.fc-filterbar-wrap :deep(.v-btn-toggle) {
|
||||||
|
border-radius: 8px;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
.fc-filterbar-wrap :deep(.v-btn-toggle .v-btn) {
|
||||||
|
border-radius: 0 !important;
|
||||||
|
}
|
||||||
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
|
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
|
||||||
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
||||||
/* The tag chips' bodies toggle include/exclude — signal they're clickable. */
|
/* The tag chips' bodies toggle include/exclude — signal they're clickable. */
|
||||||
|
|||||||
@@ -139,9 +139,9 @@ function onThumbError() { thumbError.value = true }
|
|||||||
position: absolute; top: 8px; left: 8px;
|
position: absolute; top: 8px; left: 8px;
|
||||||
width: 22px; height: 22px; border-radius: 4px;
|
width: 22px; height: 22px; border-radius: 4px;
|
||||||
border: 2px solid rgba(232, 228, 216, 0.8);
|
border: 2px solid rgba(232, 228, 216, 0.8);
|
||||||
background: rgba(20, 23, 26, 0.45);
|
background: rgba(var(--v-theme-background), 0.45);
|
||||||
display: grid; place-items: center;
|
display: grid; place-items: center;
|
||||||
color: #14171A; z-index: 11;
|
color: rgb(var(--v-theme-background)); z-index: 11;
|
||||||
}
|
}
|
||||||
.fc-gallery-item__checkbox.on {
|
.fc-gallery-item__checkbox.on {
|
||||||
background: rgb(var(--v-theme-accent));
|
background: rgb(var(--v-theme-accent));
|
||||||
@@ -152,7 +152,7 @@ function onThumbError() { thumbError.value = true }
|
|||||||
min-width: 22px; height: 22px; padding: 0 5px;
|
min-width: 22px; height: 22px; padding: 0 5px;
|
||||||
border-radius: 11px;
|
border-radius: 11px;
|
||||||
background: rgb(var(--v-theme-accent));
|
background: rgb(var(--v-theme-accent));
|
||||||
color: #14171A; font-size: 12px; font-weight: 700;
|
color: rgb(var(--v-theme-background)); font-size: 12px; font-weight: 700;
|
||||||
display: grid; place-items: center; z-index: 11;
|
display: grid; place-items: center; z-index: 11;
|
||||||
pointer-events: none;
|
pointer-events: none;
|
||||||
}
|
}
|
||||||
@@ -160,7 +160,8 @@ function onThumbError() { thumbError.value = true }
|
|||||||
position: absolute; left: 0; right: 0; bottom: 0;
|
position: absolute; left: 0; right: 0; bottom: 0;
|
||||||
padding: 14px 8px 6px;
|
padding: 14px 8px 6px;
|
||||||
background: linear-gradient(
|
background: linear-gradient(
|
||||||
to top, rgba(20, 23, 26, 0.78), rgba(20, 23, 26, 0)
|
to top, rgba(var(--v-theme-background), 0.78),
|
||||||
|
rgba(var(--v-theme-background), 0)
|
||||||
);
|
);
|
||||||
font-size: 12px; line-height: 1.2;
|
font-size: 12px; line-height: 1.2;
|
||||||
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||||
|
|||||||
@@ -1,14 +1,15 @@
|
|||||||
<template>
|
<template>
|
||||||
<!-- Auto-hidden chrome that ALSO looked like real content — surfaced PROACTIVELY
|
<!-- System-tag auto-applies (chrome hides / process WIP tags) that ALSO looked
|
||||||
atop the gallery whenever there's something to review (NOT gated on the
|
like real content — surfaced PROACTIVELY atop the gallery whenever there's
|
||||||
Show-hidden toggle, so misfires can't go unnoticed), most-concerning first,
|
something to review (NOT gated on the Show-hidden toggle, so misfires can't
|
||||||
with keep / un-hide (#141). Renders nothing when there's nothing to review. -->
|
go unnoticed), most-concerning first, with keep / remove (#141, #1464).
|
||||||
<section v-if="items.length" class="fc-review" aria-label="Hidden images to review">
|
Renders nothing when there's nothing to review. -->
|
||||||
|
<section v-if="items.length" class="fc-review" aria-label="Auto-tagged images to review">
|
||||||
<div class="fc-review__head">
|
<div class="fc-review__head">
|
||||||
<v-icon size="18" color="warning">mdi-alert-outline</v-icon>
|
<v-icon size="18" color="warning">mdi-alert-outline</v-icon>
|
||||||
<span class="fc-review__title">
|
<span class="fc-review__title">
|
||||||
{{ items.length }} auto-hidden {{ items.length === 1 ? 'image' : 'images' }}
|
{{ items.length }} auto-tagged {{ items.length === 1 ? 'image' : 'images' }}
|
||||||
may be real content — review before they stay hidden
|
may be real content — review
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="fc-review__cards">
|
<div class="fc-review__cards">
|
||||||
@@ -26,16 +27,16 @@
|
|||||||
>
|
>
|
||||||
also looks like <strong>{{ it.conflict_name || 'content' }}</strong>
|
also looks like <strong>{{ it.conflict_name || 'content' }}</strong>
|
||||||
</div>
|
</div>
|
||||||
<div class="fc-review-card__tag">hidden as {{ it.tag_name }}</div>
|
<div class="fc-review-card__tag">{{ tagLine(it) }}</div>
|
||||||
<div class="fc-review-card__acts">
|
<div class="fc-review-card__acts">
|
||||||
<button
|
<button
|
||||||
type="button" class="fc-review-btn fc-review-btn--keep"
|
type="button" class="fc-review-btn fc-review-btn--keep"
|
||||||
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')"
|
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')"
|
||||||
>Keep hidden</button>
|
>{{ keepLabel(it) }}</button>
|
||||||
<button
|
<button
|
||||||
type="button" class="fc-review-btn fc-review-btn--unhide"
|
type="button" class="fc-review-btn fc-review-btn--unhide"
|
||||||
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')"
|
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')"
|
||||||
>Un-hide</button>
|
>{{ removeLabel(it) }}</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -54,6 +55,11 @@ const items = ref([])
|
|||||||
const busy = ref([])
|
const busy = ref([])
|
||||||
|
|
||||||
function keyOf(it) { return `${it.image_id}:${it.tag_id}` }
|
function keyOf(it) { return `${it.image_id}:${it.tag_id}` }
|
||||||
|
// Chrome flags hide the image (keep-hidden / un-hide); process flags leave it
|
||||||
|
// visible and just tagged (keep-tag / remove-tag). Same endpoints, different words.
|
||||||
|
function tagLine(it) { return (it.mode === 'process' ? 'auto-tagged ' : 'hidden as ') + it.tag_name }
|
||||||
|
function keepLabel(it) { return it.mode === 'process' ? 'Keep tag' : 'Keep hidden' }
|
||||||
|
function removeLabel(it) { return it.mode === 'process' ? 'Remove tag' : 'Un-hide' }
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
// Fetched unconditionally on mount — the strip prompts for pending misfires
|
// Fetched unconditionally on mount — the strip prompts for pending misfires
|
||||||
@@ -71,7 +77,8 @@ async function resolve(it, action) {
|
|||||||
await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`)
|
await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`)
|
||||||
items.value = items.value.filter((x) => keyOf(x) !== k)
|
items.value = items.value.filter((x) => keyOf(x) !== k)
|
||||||
if (action === 'unhide') {
|
if (action === 'unhide') {
|
||||||
toast({ text: `Un-hidden — “${it.tag_name}” removed; it'll train the head`, type: 'success' })
|
const verb = it.mode === 'process' ? 'Removed' : 'Un-hidden'
|
||||||
|
toast({ text: `${verb} — “${it.tag_name}” removed; it'll train the head`, type: 'success' })
|
||||||
}
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
toast({
|
toast({
|
||||||
|
|||||||
@@ -0,0 +1,127 @@
|
|||||||
|
<template>
|
||||||
|
<!-- #3068: attachment reclamation. PostAttachment's FKs are both SET NULL, so
|
||||||
|
a deleted post or artist leaves the row behind; and the store is
|
||||||
|
sha-addressed, so one blob backs many rows and deleting a row never freed
|
||||||
|
its file. Nothing swept either. Preview first, then apply (destructive:
|
||||||
|
unlinks files). -->
|
||||||
|
<MaintenanceTile
|
||||||
|
icon="mdi-paperclip-off"
|
||||||
|
title="Reclaim orphaned attachments"
|
||||||
|
blurb="Remove attachment records belonging to nothing, and the files nothing references."
|
||||||
|
destructive
|
||||||
|
:open="applying || previewing"
|
||||||
|
>
|
||||||
|
<p class="text-body-2 mb-3">
|
||||||
|
Attachment records survive the post and artist they belonged to, and the
|
||||||
|
files behind them are shared between records — so a deleted record never
|
||||||
|
freed its file on its own. This finds records attributed to
|
||||||
|
<strong>neither</strong> a post nor an artist, and files in the attachment
|
||||||
|
store that <strong>no remaining record</strong> points at.
|
||||||
|
<strong>Preview</strong> first; <strong>Apply</strong> deletes those
|
||||||
|
records and unlinks those files. Files written in the last few hours are
|
||||||
|
always left alone, so an in-progress download is never caught mid-write.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
|
||||||
|
<v-btn
|
||||||
|
color="primary" variant="tonal" rounded="pill"
|
||||||
|
:loading="previewing" :disabled="applying" @click="preview"
|
||||||
|
>
|
||||||
|
<v-icon start>mdi-magnify</v-icon> Preview
|
||||||
|
</v-btn>
|
||||||
|
<v-btn
|
||||||
|
color="error" rounded="pill"
|
||||||
|
:loading="applying"
|
||||||
|
:disabled="previewing || !canApply"
|
||||||
|
@click="confirmOpen = true"
|
||||||
|
>
|
||||||
|
<v-icon start>mdi-paperclip-off</v-icon> Apply
|
||||||
|
</v-btn>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<v-alert
|
||||||
|
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
|
||||||
|
density="comfortable"
|
||||||
|
>
|
||||||
|
<span v-if="applied">
|
||||||
|
Deleted {{ summary.rows }} orphaned record(s) and unlinked
|
||||||
|
{{ summary.files }} file(s), reclaiming {{ humanBytes(summary.bytes) }}.
|
||||||
|
</span>
|
||||||
|
<span v-else-if="hasWork">
|
||||||
|
{{ summary.rows }} orphaned record(s) and {{ summary.files }}
|
||||||
|
unreferenced file(s) — {{ humanBytes(summary.bytes) }} reclaimable.
|
||||||
|
Click <strong>Apply</strong> to remove them.
|
||||||
|
</span>
|
||||||
|
<span v-else>Nothing to reclaim — every attachment is accounted for.</span>
|
||||||
|
|
||||||
|
<!-- Both of these change what the numbers MEAN, so they are stated
|
||||||
|
whenever they are non-zero rather than hidden in a tooltip. -->
|
||||||
|
<div v-if="summary.files_failed" class="mt-1 text-caption">
|
||||||
|
{{ summary.files_failed }} file(s) could not be read or removed — see
|
||||||
|
the worker log.
|
||||||
|
</div>
|
||||||
|
<div v-if="summary.partial" class="mt-1 text-caption">
|
||||||
|
Stopped early at the time limit; some of the store was not examined.
|
||||||
|
Run it again to continue.
|
||||||
|
</div>
|
||||||
|
</v-alert>
|
||||||
|
|
||||||
|
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance" />
|
||||||
|
|
||||||
|
<v-dialog v-model="confirmOpen" max-width="440">
|
||||||
|
<v-card>
|
||||||
|
<v-card-title>Reclaim orphaned attachments?</v-card-title>
|
||||||
|
<v-card-text class="text-body-2">
|
||||||
|
This permanently deletes
|
||||||
|
<strong>{{ summary?.rows ?? 0 }}</strong> attachment record(s) and
|
||||||
|
unlinks <strong>{{ summary?.files ?? 0 }}</strong> file(s)
|
||||||
|
({{ humanBytes(summary?.bytes) }}). Only files that no remaining
|
||||||
|
record points at are removed, so nothing still attached to a post
|
||||||
|
is affected.
|
||||||
|
</v-card-text>
|
||||||
|
<v-card-actions>
|
||||||
|
<v-spacer />
|
||||||
|
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
|
||||||
|
<v-btn color="error" @click="apply">Reclaim</v-btn>
|
||||||
|
</v-card-actions>
|
||||||
|
</v-card>
|
||||||
|
</v-dialog>
|
||||||
|
</MaintenanceTile>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { computed, ref } from 'vue'
|
||||||
|
|
||||||
|
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||||
|
import { humanBytes } from '../../utils/bytes.js'
|
||||||
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
|
import QueueStatusBar from './QueueStatusBar.vue'
|
||||||
|
|
||||||
|
const confirmOpen = ref(false)
|
||||||
|
|
||||||
|
// Walks the whole attachment store, so it can run for minutes on a large
|
||||||
|
// library — the service caps itself at 900s and reports `partial`. 150 polls
|
||||||
|
// × 2s ≈ 5m of foreground waiting; past that the composable hands off to the
|
||||||
|
// task dashboard rather than spinning forever.
|
||||||
|
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
|
||||||
|
endpoint: '/api/admin/maintenance/reclaim-attachments',
|
||||||
|
storageKey: 'fc.maint.reclaimAttachments',
|
||||||
|
appliedToast: 'Orphaned attachments reclaimed',
|
||||||
|
maxPolls: 150,
|
||||||
|
})
|
||||||
|
|
||||||
|
const hasWork = computed(
|
||||||
|
() => !!summary.value && (summary.value.rows > 0 || summary.value.files > 0),
|
||||||
|
)
|
||||||
|
const canApply = computed(() => hasWork.value && !applied.value)
|
||||||
|
const summaryType = computed(() => {
|
||||||
|
if (applied.value) return 'success'
|
||||||
|
return hasWork.value ? 'info' : 'success'
|
||||||
|
})
|
||||||
|
|
||||||
|
// The confirm dialog gates the destructive apply; close it, then run.
|
||||||
|
function apply () {
|
||||||
|
confirmOpen.value = false
|
||||||
|
applyTask()
|
||||||
|
}
|
||||||
|
</script>
|
||||||
@@ -4,12 +4,22 @@
|
|||||||
<span v-if="manifest?.installed" class="text-caption fc-muted">
|
<span v-if="manifest?.installed" class="text-caption fc-muted">
|
||||||
· Firefox · v{{ manifest.version }}
|
· Firefox · v{{ manifest.version }}
|
||||||
</span>
|
</span>
|
||||||
|
<!-- Which channel this instance serves, so it is visible without
|
||||||
|
installing anything. Rendered only when the image declares one: a
|
||||||
|
locally-built image, or one predating the field, says nothing rather
|
||||||
|
than guessing. Never merged into the version string beside it — see
|
||||||
|
the endpoint's note on why a `-dev` suffix breaks the comparator. -->
|
||||||
|
<v-chip
|
||||||
|
v-if="manifest?.channel"
|
||||||
|
size="x-small" variant="tonal" class="ml-2"
|
||||||
|
:color="manifest.channel === 'dev' ? 'warning' : 'info'"
|
||||||
|
>{{ manifest.channel }}</v-chip>
|
||||||
</CardHeading>
|
</CardHeading>
|
||||||
|
|
||||||
<v-card-text>
|
<v-card-text>
|
||||||
<p class="fc-muted text-body-2">
|
<p class="fc-muted text-body-2">
|
||||||
Pushes session cookies from supported platforms
|
Pushes session cookies from supported platforms
|
||||||
(patreon, subscribestar, hentaifoundry, discord, pixiv, deviantart)
|
(patreon, subscribestar, hentaifoundry, discord, pixiv)
|
||||||
into FabledCurator, and lets you add a creator as a source from
|
into FabledCurator, and lets you add a creator as a source from
|
||||||
their page in one click.
|
their page in one click.
|
||||||
</p>
|
</p>
|
||||||
|
|||||||
@@ -15,28 +15,23 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div v-for="p in proposers" :key="p.key" class="fc-proposer">
|
<div v-for="p in proposers" :key="p.key" class="fc-proposer">
|
||||||
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
<SettingToggleRow
|
||||||
<v-icon size="18" :color="p.on ? 'accent' : undefined">{{ p.icon }}</v-icon>
|
v-model="p.on" :loading="busy" :icon="p.icon"
|
||||||
<span class="fc-section-h">{{ p.label }}</span>
|
:icon-color="p.on ? 'accent' : null" :label="p.label"
|
||||||
<v-switch
|
@change="v => saveToggle(p, v)"
|
||||||
v-model="p.on" :loading="busy" hide-details density="compact"
|
|
||||||
color="success" class="ml-auto"
|
|
||||||
@update:model-value="v => saveToggle(p, v)"
|
|
||||||
/>
|
/>
|
||||||
</div>
|
|
||||||
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
|
<p class="fc-muted text-body-2 mb-2">{{ p.help }}</p>
|
||||||
<div class="d-flex flex-wrap mb-4" style="gap: 12px;">
|
<div class="d-flex flex-wrap mb-4" style="gap: 12px;">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
v-model="p.weights" label="Weights" density="compact" hide-details
|
v-model="p.weights" label="Weights" density="compact" hide-details
|
||||||
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
|
style="min-width: 300px; flex: 1;" :disabled="busy || !p.on"
|
||||||
placeholder="name | URL | hf_repo::file"
|
placeholder="name | URL | hf_repo::file"
|
||||||
@change="save({ [`detector_${p.key}_weights`]: p.weights })"
|
@change="saveField({ [`detector_${p.key}_weights`]: p.weights })"
|
||||||
/>
|
/>
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="p.conf" label="Confidence" type="number"
|
v-model="p.conf" label="Confidence" :min="0" :max="1" :step="0.05"
|
||||||
min="0" max="1" step="0.05" density="compact" hide-details
|
max-width="140px" :disabled="busy || !p.on"
|
||||||
style="max-width: 140px;" :disabled="busy || !p.on"
|
@change="saveField({ [`detector_${p.key}_conf`]: Number(p.conf) })"
|
||||||
@change="save({ [`detector_${p.key}_conf`]: Number(p.conf) })"
|
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -48,12 +43,12 @@
|
|||||||
storage. Dedupe IoU drops near-duplicate crops before embedding.
|
storage. Dedupe IoU drops near-duplicate crops before embedding.
|
||||||
</p>
|
</p>
|
||||||
<div class="d-flex flex-wrap" style="gap: 12px;">
|
<div class="d-flex flex-wrap" style="gap: 12px;">
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-for="c in caps" :key="c.key"
|
v-for="c in caps" :key="c.key"
|
||||||
v-model.number="c.val" :label="c.label" type="number"
|
v-model="c.val" :label="c.label"
|
||||||
:min="c.min" :max="c.max" :step="c.step || 1" density="compact"
|
:min="c.min" :max="c.max" :step="c.step || 1"
|
||||||
hide-details style="max-width: 165px;" :disabled="busy"
|
max-width="165px" :disabled="busy"
|
||||||
@change="save({ [c.key]: Number(c.val) })"
|
@change="saveField({ [c.key]: Number(c.val) })"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -61,14 +56,16 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
import { onMounted, ref } from 'vue'
|
import { onMounted, ref } from 'vue'
|
||||||
|
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
|
import SettingNumberField from '../common/SettingNumberField.vue'
|
||||||
|
import SettingToggleRow from '../common/SettingToggleRow.vue'
|
||||||
|
import { useSettingSave } from '../../composables/useSettingSave.js'
|
||||||
import { useMLStore } from '../../stores/ml.js'
|
import { useMLStore } from '../../stores/ml.js'
|
||||||
|
|
||||||
const mlSettings = useMLStore()
|
const mlSettings = useMLStore()
|
||||||
const busy = ref(false)
|
const { busy, save } = useSettingSave(mlSettings.patchSettings)
|
||||||
const proposers = ref([])
|
const proposers = ref([])
|
||||||
const caps = ref([])
|
const caps = ref([])
|
||||||
|
|
||||||
@@ -111,31 +108,20 @@ onMounted(async () => {
|
|||||||
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
|
caps.value = CAP_DEFS.map(c => ({ ...c, val: s[c.key] ?? 0 }))
|
||||||
})
|
})
|
||||||
|
|
||||||
async function save(patch, revert) {
|
// Field @change → persist with a "Saved" confirmation. SettingNumberField has
|
||||||
busy.value = true
|
// already clamped numeric values to their [min,max] before this fires.
|
||||||
try {
|
function saveField(patch) {
|
||||||
await mlSettings.patchSettings(patch)
|
save(patch, { successMessage: 'Saved' })
|
||||||
toast({ text: 'Saved', type: 'success' })
|
|
||||||
} catch (e) {
|
|
||||||
if (revert) revert()
|
|
||||||
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function saveToggle (p, v) {
|
async function saveToggle(p, v) {
|
||||||
// Revert the switch on failure so it never lies about the persisted state.
|
// Revert the switch on failure so it never lies about the persisted state.
|
||||||
save({ [`detector_${p.key}_enabled`]: !!v }, () => { p.on = !v })
|
const ok = await save({ [`detector_${p.key}_enabled`]: !!v }, { successMessage: 'Saved' })
|
||||||
|
if (!ok) p.on = !v
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-section-h {
|
|
||||||
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
|
|
||||||
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-proposer {
|
.fc-proposer {
|
||||||
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
|
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 14px;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -112,7 +112,6 @@ async function onCommit() {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-code {
|
.fc-code {
|
||||||
background: rgb(var(--v-theme-surface-light));
|
background: rgb(var(--v-theme-surface-light));
|
||||||
border-radius: 4px; padding: 2px 8px;
|
border-radius: 4px; padding: 2px 8px;
|
||||||
|
|||||||
@@ -42,7 +42,7 @@
|
|||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</v-table>
|
</v-table>
|
||||||
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
|
<p v-else class="text-caption mt-3 fc-muted">
|
||||||
No table statistics yet.
|
No table statistics yet.
|
||||||
</p>
|
</p>
|
||||||
</MaintenanceTile>
|
</MaintenanceTile>
|
||||||
|
|||||||
@@ -35,7 +35,7 @@
|
|||||||
All subscription sources healthy.
|
All subscription sources healthy.
|
||||||
</p>
|
</p>
|
||||||
<p v-else class="text-body-2 mb-0">
|
<p v-else class="text-body-2 mb-0">
|
||||||
<b class="fc-bad">{{ failing.length }}</b> failing source(s):
|
<b class="fc-weak">{{ failing.length }}</b> failing source(s):
|
||||||
<span class="fc-muted">{{ failingNames }}</span>
|
<span class="fc-muted">{{ failingNames }}</span>
|
||||||
</p>
|
</p>
|
||||||
</v-card-text>
|
</v-card-text>
|
||||||
@@ -72,6 +72,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-bad { color: rgb(var(--v-theme-error)); }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -102,6 +102,7 @@
|
|||||||
import { computed, ref } from 'vue'
|
import { computed, ref } from 'vue'
|
||||||
|
|
||||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||||
|
import { humanBytes } from '../../utils/bytes.js'
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
import QueueStatusBar from './QueueStatusBar.vue'
|
||||||
|
|
||||||
@@ -122,14 +123,6 @@ const summaryType = computed(() => {
|
|||||||
return summary.value && summary.value.matched > 0 ? 'info' : 'success'
|
return summary.value && summary.value.matched > 0 ? 'info' : 'success'
|
||||||
})
|
})
|
||||||
|
|
||||||
function humanBytes (n) {
|
|
||||||
const b = Number(n || 0)
|
|
||||||
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
|
|
||||||
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
|
|
||||||
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
|
|
||||||
return b + ' B'
|
|
||||||
}
|
|
||||||
|
|
||||||
// The confirm dialog gates the destructive apply; close it, then run.
|
// The confirm dialog gates the destructive apply; close it, then run.
|
||||||
function apply () {
|
function apply () {
|
||||||
confirmOpen.value = false
|
confirmOpen.value = false
|
||||||
|
|||||||
@@ -25,7 +25,7 @@
|
|||||||
<div class="fc-cell__l">done</div>
|
<div class="fc-cell__l">done</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="fc-cell">
|
<div class="fc-cell">
|
||||||
<div class="fc-cell__n" :class="q.error ? 'fc-bad' : ''">{{ q.error }}</div>
|
<div class="fc-cell__n" :class="q.error ? 'fc-weak' : ''">{{ q.error }}</div>
|
||||||
<div class="fc-cell__l">errored</div>
|
<div class="fc-cell__l">errored</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -95,7 +95,6 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-cells { display: flex; gap: 28px; }
|
.fc-cells { display: flex; gap: 28px; }
|
||||||
.fc-cell__n {
|
.fc-cell__n {
|
||||||
font-size: 20px; font-weight: 700; line-height: 1.1;
|
font-size: 20px; font-weight: 700; line-height: 1.1;
|
||||||
@@ -105,6 +104,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
|
|||||||
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
}
|
}
|
||||||
.fc-good { color: rgb(var(--v-theme-success)); }
|
|
||||||
.fc-bad { color: rgb(var(--v-theme-error)); }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -367,11 +367,6 @@ async function onReprocess() {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-section-h {
|
|
||||||
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
|
|
||||||
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-token {
|
.fc-token {
|
||||||
display: flex; align-items: center; gap: 4px;
|
display: flex; align-items: center; gap: 4px;
|
||||||
background: rgb(var(--v-theme-surface-light)); border-radius: 6px;
|
background: rgb(var(--v-theme-surface-light)); border-radius: 6px;
|
||||||
@@ -390,6 +385,4 @@ async function onReprocess() {
|
|||||||
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
}
|
}
|
||||||
.fc-good { color: rgb(var(--v-theme-success)); }
|
|
||||||
.fc-weak { color: rgb(var(--v-theme-error)); }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -155,11 +155,6 @@ async function onRecover(it) {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-section-h {
|
|
||||||
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
|
|
||||||
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-queue { display: flex; gap: 24px; }
|
.fc-queue { display: flex; gap: 24px; }
|
||||||
.fc-q__n {
|
.fc-q__n {
|
||||||
font-size: 20px; font-weight: 700; line-height: 1.1;
|
font-size: 20px; font-weight: 700; line-height: 1.1;
|
||||||
@@ -169,8 +164,6 @@ async function onRecover(it) {
|
|||||||
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
}
|
}
|
||||||
.fc-good { color: rgb(var(--v-theme-success)); }
|
|
||||||
.fc-weak { color: rgb(var(--v-theme-error)); }
|
|
||||||
.fc-defect {
|
.fc-defect {
|
||||||
display: flex; align-items: center; gap: 12px;
|
display: flex; align-items: center; gap: 12px;
|
||||||
background: rgb(var(--v-theme-surface-light)); border-radius: 8px;
|
background: rgb(var(--v-theme-surface-light)); border-radius: 8px;
|
||||||
|
|||||||
@@ -95,14 +95,10 @@
|
|||||||
|
|
||||||
<!-- Earned auto-apply -->
|
<!-- Earned auto-apply -->
|
||||||
<div class="fc-auto mt-6">
|
<div class="fc-auto mt-6">
|
||||||
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
<SettingToggleRow
|
||||||
<v-icon size="18" color="accent">mdi-lightning-bolt</v-icon>
|
v-model="autoEnabled" :loading="settingBusy"
|
||||||
<span class="fc-section-h">Auto-apply</span>
|
icon="mdi-lightning-bolt" label="Auto-apply" @change="onToggleAuto"
|
||||||
<v-switch
|
|
||||||
v-model="autoEnabled" :loading="settingBusy" hide-details density="compact"
|
|
||||||
color="success" class="ml-auto" @update:model-value="onToggleAuto"
|
|
||||||
/>
|
/>
|
||||||
</div>
|
|
||||||
<p class="fc-muted text-body-2 mb-3">
|
<p class="fc-muted text-body-2 mb-3">
|
||||||
Graduated heads (⚡, with ≥ {{ autoMinPosInput }} examples) apply their tag
|
Graduated heads (⚡, with ≥ {{ autoMinPosInput }} examples) apply their tag
|
||||||
on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}%
|
on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}%
|
||||||
@@ -111,17 +107,14 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div class="d-flex mb-3" style="gap: 12px;">
|
<div class="d-flex mb-3" style="gap: 12px;">
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="autoPrecisionInput" label="Precision target"
|
v-model="autoPrecisionInput" label="Precision target"
|
||||||
type="number" min="0.5" max="0.999" step="0.01" density="compact"
|
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
|
||||||
hide-details style="max-width: 200px;" :disabled="settingBusy"
|
|
||||||
@change="onSaveSettings"
|
@change="onSaveSettings"
|
||||||
/>
|
/>
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="autoMinPosInput" label="Min examples to fire"
|
v-model="autoMinPosInput" label="Min examples to fire"
|
||||||
type="number" min="1" density="compact" hide-details
|
:min="1" :disabled="settingBusy" @change="onSaveSettings"
|
||||||
style="max-width: 200px;" :disabled="settingBusy"
|
|
||||||
@change="onSaveSettings"
|
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -161,40 +154,65 @@
|
|||||||
|
|
||||||
<!-- Presentation chrome auto-hide (#141) -->
|
<!-- Presentation chrome auto-hide (#141) -->
|
||||||
<div class="fc-auto mt-6">
|
<div class="fc-auto mt-6">
|
||||||
<div class="d-flex align-center mb-1" style="gap: 10px;">
|
<SettingToggleRow
|
||||||
<v-icon size="18" color="accent">mdi-image-off-outline</v-icon>
|
v-model="presentationEnabled" :loading="settingBusy"
|
||||||
<span class="fc-section-h">Hide presentation chrome</span>
|
icon="mdi-image-off-outline" label="Hide presentation chrome"
|
||||||
<v-switch
|
@change="onTogglePresentation"
|
||||||
v-model="presentationEnabled" :loading="settingBusy" hide-details
|
|
||||||
density="compact" color="success" class="ml-auto"
|
|
||||||
@update:model-value="onTogglePresentation"
|
|
||||||
/>
|
/>
|
||||||
</div>
|
|
||||||
<p class="fc-muted text-body-2 mb-3">
|
<p class="fc-muted text-body-2 mb-3">
|
||||||
Auto-hide banners and editor screenshots from the gallery once a head has
|
Auto-hide <code>banner</code> chrome from the gallery once a head has
|
||||||
learned them (≥ {{ minPositives }} examples) and clears
|
learned it (≥ {{ minPositives }} examples) and clears
|
||||||
{{ Math.round((presentationThresholdInput || 0) * 100) }}% confidence.
|
{{ Math.round((presentationThresholdInput || 0) * 100) }}% confidence.
|
||||||
<code>wip</code> is never auto-hidden. If a hidden image also looks like
|
(<code>wip</code> and <code>editor screenshot</code> are handled by the
|
||||||
real content (≥ {{ Math.round((presentationConflictInput || 0) * 100) }}%
|
process auto-tagger below.) If a hidden image also looks like real content
|
||||||
on a content tag), it's flagged in the Hidden view instead of buried.
|
(≥ {{ Math.round((presentationConflictInput || 0) * 100) }}% on a content
|
||||||
Every auto-hide is reversible.
|
tag), it's flagged for review instead of buried. Every auto-hide is reversible.
|
||||||
</p>
|
</p>
|
||||||
<div class="d-flex mb-3" style="gap: 12px;">
|
<div class="d-flex mb-3" style="gap: 12px;">
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="presentationThresholdInput" label="Hide confidence"
|
v-model="presentationThresholdInput" label="Hide confidence"
|
||||||
type="number" min="0.5" max="0.999" step="0.01" density="compact"
|
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
|
||||||
hide-details style="max-width: 200px;" :disabled="settingBusy"
|
|
||||||
@change="onSavePresentation"
|
@change="onSavePresentation"
|
||||||
/>
|
/>
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="presentationConflictInput" label="Flag if content ≥"
|
v-model="presentationConflictInput" label="Flag if content ≥"
|
||||||
type="number" min="0" max="1" step="0.05" density="compact"
|
:min="0" :max="1" :step="0.05" :disabled="settingBusy"
|
||||||
hide-details style="max-width: 200px;" :disabled="settingBusy"
|
|
||||||
@change="onSavePresentation"
|
@change="onSavePresentation"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- Process auto-tagging (#1464): wip / editor screenshot -->
|
||||||
|
<div class="fc-auto mt-6">
|
||||||
|
<SettingToggleRow
|
||||||
|
v-model="processEnabled" :loading="settingBusy"
|
||||||
|
icon="mdi-progress-wrench" label="Auto-tag work-in-progress"
|
||||||
|
@change="onToggleProcess"
|
||||||
|
/>
|
||||||
|
<p class="fc-muted text-body-2 mb-3">
|
||||||
|
Auto-tag <code>wip</code> and <code>editor screenshot</code> process art
|
||||||
|
once a head has learned them (≥ {{ minPositives }} examples) and clears
|
||||||
|
{{ Math.round((processThresholdInput || 0) * 100) }}% confidence. These stay
|
||||||
|
<strong>visible</strong> in the gallery — the tag just keeps them out of
|
||||||
|
training and the Explore rabbit-hole. Off by default. If a tagged image also
|
||||||
|
looks like real content (≥ {{ Math.round((processConflictInput || 0) * 100) }}%
|
||||||
|
on a content tag), it's flagged for review. Learns only from your titles +
|
||||||
|
manual tags, never its own guesses — so it can't run away. Every tag reversible.
|
||||||
|
</p>
|
||||||
|
<div class="d-flex mb-3" style="gap: 12px;">
|
||||||
|
<SettingNumberField
|
||||||
|
v-model="processThresholdInput" label="Tag confidence"
|
||||||
|
:min="0.5" :max="0.999" :step="0.01" :disabled="settingBusy"
|
||||||
|
@change="onSaveProcess"
|
||||||
|
/>
|
||||||
|
<SettingNumberField
|
||||||
|
v-model="processConflictInput" label="Flag if content ≥"
|
||||||
|
:min="0" :max="1" :step="0.05" :disabled="settingBusy"
|
||||||
|
@change="onSaveProcess"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<!-- Performance / tuning -->
|
<!-- Performance / tuning -->
|
||||||
<div v-if="metricsConcepts.length" class="mt-5">
|
<div v-if="metricsConcepts.length" class="mt-5">
|
||||||
<div class="fc-section-h mb-1">How auto-apply is landing</div>
|
<div class="fc-section-h mb-1">How auto-apply is landing</div>
|
||||||
@@ -219,7 +237,7 @@
|
|||||||
<td class="fc-r fc-mono">{{ c.n_auto_applied }}</td>
|
<td class="fc-r fc-mono">{{ c.n_auto_applied }}</td>
|
||||||
<td class="fc-r fc-mono">{{ c.n_misfires }}</td>
|
<td class="fc-r fc-mono">{{ c.n_misfires }}</td>
|
||||||
<td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)">
|
<td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)">
|
||||||
{{ ratePct(c.misfire_rate) }}
|
{{ pct(c.misfire_rate) }}
|
||||||
</td>
|
</td>
|
||||||
<td class="fc-r fc-mono">{{ c.n_underfires }}</td>
|
<td class="fc-r fc-mono">{{ c.n_underfires }}</td>
|
||||||
</tr>
|
</tr>
|
||||||
@@ -235,6 +253,9 @@ import { toast } from '../../utils/toast.js'
|
|||||||
import { computed, onMounted, onUnmounted, ref } from 'vue'
|
import { computed, onMounted, onUnmounted, ref } from 'vue'
|
||||||
|
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
|
import SettingNumberField from '../common/SettingNumberField.vue'
|
||||||
|
import SettingToggleRow from '../common/SettingToggleRow.vue'
|
||||||
|
import { useSettingSave } from '../../composables/useSettingSave.js'
|
||||||
import { useHeadsStore } from '../../stores/heads.js'
|
import { useHeadsStore } from '../../stores/heads.js'
|
||||||
import { useMLStore } from '../../stores/ml.js'
|
import { useMLStore } from '../../stores/ml.js'
|
||||||
|
|
||||||
@@ -248,7 +269,9 @@ let pollTimer = null
|
|||||||
const autoEnabled = ref(false)
|
const autoEnabled = ref(false)
|
||||||
const autoPrecisionInput = ref(0.97)
|
const autoPrecisionInput = ref(0.97)
|
||||||
const autoMinPosInput = ref(30)
|
const autoMinPosInput = ref(30)
|
||||||
const settingBusy = ref(false)
|
// Shared settings-save flow (busy + toast + revert); `settingBusy` gates the
|
||||||
|
// toggles/fields, `save` returns ok/false for the optimistic-switch revert.
|
||||||
|
const { busy: settingBusy, save } = useSettingSave(mlSettings.patchSettings)
|
||||||
const autoBusy = ref(false)
|
const autoBusy = ref(false)
|
||||||
const autoStatus = ref(null)
|
const autoStatus = ref(null)
|
||||||
const metricsData = ref(null)
|
const metricsData = ref(null)
|
||||||
@@ -258,6 +281,9 @@ let autoTimer = null
|
|||||||
const presentationEnabled = ref(true)
|
const presentationEnabled = ref(true)
|
||||||
const presentationThresholdInput = ref(0.90)
|
const presentationThresholdInput = ref(0.90)
|
||||||
const presentationConflictInput = ref(0.50)
|
const presentationConflictInput = ref(0.50)
|
||||||
|
const processEnabled = ref(false)
|
||||||
|
const processThresholdInput = ref(0.90)
|
||||||
|
const processConflictInput = ref(0.50)
|
||||||
|
|
||||||
const autoRunning = computed(() => autoStatus.value?.running_id != null)
|
const autoRunning = computed(() => autoStatus.value?.running_id != null)
|
||||||
const lastSweep = computed(() =>
|
const lastSweep = computed(() =>
|
||||||
@@ -292,6 +318,9 @@ onMounted(async () => {
|
|||||||
presentationEnabled.value = s.presentation_auto_apply_enabled ?? true
|
presentationEnabled.value = s.presentation_auto_apply_enabled ?? true
|
||||||
presentationThresholdInput.value = s.presentation_auto_apply_threshold ?? 0.90
|
presentationThresholdInput.value = s.presentation_auto_apply_threshold ?? 0.90
|
||||||
presentationConflictInput.value = s.presentation_conflict_threshold ?? 0.50
|
presentationConflictInput.value = s.presentation_conflict_threshold ?? 0.50
|
||||||
|
processEnabled.value = s.process_auto_apply_enabled ?? false
|
||||||
|
processThresholdInput.value = s.process_auto_apply_threshold ?? 0.90
|
||||||
|
processConflictInput.value = s.process_conflict_threshold ?? 0.50
|
||||||
} catch { /* non-fatal */ }
|
} catch { /* non-fatal */ }
|
||||||
await refresh()
|
await refresh()
|
||||||
if (running.value) startPoll()
|
if (running.value) startPoll()
|
||||||
@@ -352,55 +381,39 @@ function startAutoPoll() {
|
|||||||
function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } }
|
function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } }
|
||||||
|
|
||||||
async function onToggleAuto(val) {
|
async function onToggleAuto(val) {
|
||||||
settingBusy.value = true
|
const ok = await save({ head_auto_apply_enabled: !!val },
|
||||||
try {
|
{ successMessage: val ? 'Auto-apply on' : 'Auto-apply off', errorPrefix: 'Could not update' })
|
||||||
await mlSettings.patchSettings({ head_auto_apply_enabled: !!val })
|
if (!ok) autoEnabled.value = !val // revert the switch
|
||||||
toast({ text: val ? 'Auto-apply on' : 'Auto-apply off', type: 'success' })
|
|
||||||
} catch (e) {
|
|
||||||
autoEnabled.value = !val // revert the switch
|
|
||||||
toast({ text: `Could not update: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
settingBusy.value = false
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
async function onSaveSettings() {
|
async function onSaveSettings() {
|
||||||
settingBusy.value = true
|
await save({
|
||||||
try {
|
|
||||||
await mlSettings.patchSettings({
|
|
||||||
head_auto_apply_precision: Number(autoPrecisionInput.value),
|
head_auto_apply_precision: Number(autoPrecisionInput.value),
|
||||||
head_auto_apply_min_positives: Number(autoMinPosInput.value),
|
head_auto_apply_min_positives: Number(autoMinPosInput.value),
|
||||||
})
|
})
|
||||||
} catch (e) {
|
|
||||||
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
settingBusy.value = false
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function onTogglePresentation(val) {
|
async function onTogglePresentation(val) {
|
||||||
settingBusy.value = true
|
const ok = await save({ presentation_auto_apply_enabled: !!val },
|
||||||
try {
|
{ successMessage: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', errorPrefix: 'Could not update' })
|
||||||
await mlSettings.patchSettings({ presentation_auto_apply_enabled: !!val })
|
if (!ok) presentationEnabled.value = !val // revert the switch
|
||||||
toast({ text: val ? 'Chrome auto-hide on' : 'Chrome auto-hide off', type: 'success' })
|
|
||||||
} catch (e) {
|
|
||||||
presentationEnabled.value = !val // revert the switch
|
|
||||||
toast({ text: `Could not update: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
settingBusy.value = false
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
async function onSavePresentation() {
|
async function onSavePresentation() {
|
||||||
settingBusy.value = true
|
await save({
|
||||||
try {
|
|
||||||
await mlSettings.patchSettings({
|
|
||||||
presentation_auto_apply_threshold: Number(presentationThresholdInput.value),
|
presentation_auto_apply_threshold: Number(presentationThresholdInput.value),
|
||||||
presentation_conflict_threshold: Number(presentationConflictInput.value),
|
presentation_conflict_threshold: Number(presentationConflictInput.value),
|
||||||
})
|
})
|
||||||
} catch (e) {
|
}
|
||||||
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
async function onToggleProcess(val) {
|
||||||
settingBusy.value = false
|
const ok = await save({ process_auto_apply_enabled: !!val },
|
||||||
}
|
{ successMessage: val ? 'WIP auto-tag on' : 'WIP auto-tag off', errorPrefix: 'Could not update' })
|
||||||
|
if (!ok) processEnabled.value = !val // revert the switch
|
||||||
|
}
|
||||||
|
async function onSaveProcess() {
|
||||||
|
await save({
|
||||||
|
process_auto_apply_threshold: Number(processThresholdInput.value),
|
||||||
|
process_conflict_threshold: Number(processConflictInput.value),
|
||||||
|
})
|
||||||
}
|
}
|
||||||
function onPreview() { startSweep(true) }
|
function onPreview() { startSweep(true) }
|
||||||
function onApplyNow() { startSweep(false) }
|
function onApplyNow() { startSweep(false) }
|
||||||
@@ -426,7 +439,6 @@ function sweepConcepts(run) {
|
|||||||
.sort((a, b) => b.applied - a.applied)
|
.sort((a, b) => b.applied - a.applied)
|
||||||
}
|
}
|
||||||
function sweepTotal(run) { return run?.n_applied ?? 0 }
|
function sweepTotal(run) { return run?.n_applied ?? 0 }
|
||||||
function ratePct(x) { return x == null ? '—' : `${Math.round(x * 100)}%` }
|
|
||||||
function rateClass(x) {
|
function rateClass(x) {
|
||||||
if (x == null) return ''
|
if (x == null) return ''
|
||||||
if (x <= 0.03) return 'fc-good'
|
if (x <= 0.03) return 'fc-good'
|
||||||
@@ -457,12 +469,6 @@ function relTime(iso) {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
|
|
||||||
.fc-section-h {
|
|
||||||
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
|
|
||||||
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-auto {
|
.fc-auto {
|
||||||
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px;
|
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px;
|
||||||
}
|
}
|
||||||
@@ -519,7 +525,5 @@ function relTime(iso) {
|
|||||||
background: rgb(var(--v-theme-surface-light));
|
background: rgb(var(--v-theme-surface-light));
|
||||||
padding: 1px 6px; border-radius: 999px;
|
padding: 1px 6px; border-radius: 999px;
|
||||||
}
|
}
|
||||||
.fc-good { color: rgb(var(--v-theme-success)); }
|
|
||||||
.fc-ok { color: rgb(var(--v-theme-on-surface)); }
|
.fc-ok { color: rgb(var(--v-theme-on-surface)); }
|
||||||
.fc-weak { color: rgb(var(--v-theme-error)); }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -15,7 +15,7 @@
|
|||||||
being dropped as duplicates;</strong> raise it to collapse more
|
being dropped as duplicates;</strong> raise it to collapse more
|
||||||
look-alikes. Applies to new imports.
|
look-alikes. Applies to new imports.
|
||||||
</div>
|
</div>
|
||||||
<v-row align="center" no-gutters>
|
<v-row no-gutters class="align-center">
|
||||||
<v-col cols="12" sm="9">
|
<v-col cols="12" sm="9">
|
||||||
<v-slider
|
<v-slider
|
||||||
v-model="local.phash_threshold"
|
v-model="local.phash_threshold"
|
||||||
@@ -79,6 +79,46 @@
|
|||||||
</v-col>
|
</v-col>
|
||||||
</v-row>
|
</v-row>
|
||||||
|
|
||||||
|
<v-divider class="my-5" />
|
||||||
|
|
||||||
|
<!-- Title-based WIP auto-tagging (task #1458). The switch gates the LIVE
|
||||||
|
import hook; the button runs the one-off back-catalogue scan (an
|
||||||
|
explicit action — it is deliberately not a scheduled sweep so it can't
|
||||||
|
silently re-apply a WIP tag you removed by hand). -->
|
||||||
|
<div class="fc-wip">
|
||||||
|
<div class="fc-wip__title">WIP auto-tagging</div>
|
||||||
|
<v-switch
|
||||||
|
v-model="local.wip_title_tagging_enabled"
|
||||||
|
label="Tag work-in-progress from post titles"
|
||||||
|
density="compact" hide-details color="primary" @change="save"
|
||||||
|
/>
|
||||||
|
<div class="fc-help mb-3">
|
||||||
|
When a post's title says <strong>“WIP”</strong> or
|
||||||
|
<strong>“work in progress”</strong>, new imports get the
|
||||||
|
<code>wip</code> tag automatically — keeping unfinished pieces out of
|
||||||
|
the Explore browse. Applies to new imports; run the scan below to catch
|
||||||
|
posts already in your library.
|
||||||
|
</div>
|
||||||
|
<v-switch
|
||||||
|
v-model="local.wip_soft_title_tagging_enabled"
|
||||||
|
label="Also tag “sketch” / “doodle” titles (lower precision)"
|
||||||
|
density="compact" hide-details color="primary" @change="save"
|
||||||
|
/>
|
||||||
|
<div class="fc-help mb-3">
|
||||||
|
Extends the above to softer cues (<code>sketch</code>, <code>doodle</code>,
|
||||||
|
<code>scribble</code>). These stay <strong>visible</strong> and never train
|
||||||
|
the tagging model — a daily audit flags any that actually look like finished
|
||||||
|
art for review. Off by default.
|
||||||
|
</div>
|
||||||
|
<v-btn
|
||||||
|
variant="tonal" color="primary" size="small"
|
||||||
|
:loading="store.wipScanBusy" prepend-icon="mdi-magnify"
|
||||||
|
@click="store.scanWipTitles()"
|
||||||
|
>
|
||||||
|
Scan existing posts for WIP titles
|
||||||
|
</v-btn>
|
||||||
|
</div>
|
||||||
|
|
||||||
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
|
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
|
||||||
{{ store.settingsError }}
|
{{ store.settingsError }}
|
||||||
</v-alert>
|
</v-alert>
|
||||||
@@ -109,6 +149,8 @@ const local = reactive({
|
|||||||
skip_transparent: false, transparency_threshold: 0.9,
|
skip_transparent: false, transparency_threshold: 0.9,
|
||||||
skip_single_color: false, single_color_threshold: 0.95,
|
skip_single_color: false, single_color_threshold: 0.95,
|
||||||
phash_threshold: 10,
|
phash_threshold: 10,
|
||||||
|
wip_title_tagging_enabled: true,
|
||||||
|
wip_soft_title_tagging_enabled: false,
|
||||||
})
|
})
|
||||||
|
|
||||||
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
|
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
|
||||||
@@ -124,11 +166,18 @@ async function save() {
|
|||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
margin-top: 2px;
|
margin-top: 2px;
|
||||||
}
|
}
|
||||||
.fc-phash__title {
|
.fc-phash__title,
|
||||||
|
.fc-wip__title {
|
||||||
font-size: 0.95rem;
|
font-size: 0.95rem;
|
||||||
font-weight: 600;
|
font-weight: 600;
|
||||||
color: rgb(var(--v-theme-on-surface));
|
color: rgb(var(--v-theme-on-surface));
|
||||||
}
|
}
|
||||||
|
.fc-wip code {
|
||||||
|
font-size: 0.85em;
|
||||||
|
padding: 1px 4px;
|
||||||
|
border-radius: 3px;
|
||||||
|
background: rgb(var(--v-theme-surface-variant));
|
||||||
|
}
|
||||||
/* Headroom so the tick labels (Exact/Strict/Default/Loose) aren't clipped. */
|
/* Headroom so the tick labels (Exact/Strict/Default/Loose) aren't clipped. */
|
||||||
.fc-phash__slider { margin-bottom: 18px; }
|
.fc-phash__slider { margin-bottom: 18px; }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -39,13 +39,14 @@
|
|||||||
import { toast } from '../../utils/toast.js'
|
import { toast } from '../../utils/toast.js'
|
||||||
import { onMounted, ref } from 'vue'
|
import { onMounted, ref } from 'vue'
|
||||||
import { useMLStore } from '../../stores/ml.js'
|
import { useMLStore } from '../../stores/ml.js'
|
||||||
|
import { useSettingSave } from '../../composables/useSettingSave.js'
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
import QueueStatusBar from './QueueStatusBar.vue'
|
||||||
const store = useMLStore()
|
const store = useMLStore()
|
||||||
|
const { busy: saving, save } = useSettingSave(store.patchSettings)
|
||||||
const busy = ref(false)
|
const busy = ref(false)
|
||||||
const done = ref(false)
|
const done = ref(false)
|
||||||
const enabled = ref(true)
|
const enabled = ref(true)
|
||||||
const saving = ref(false)
|
|
||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
try {
|
try {
|
||||||
await store.loadSettings()
|
await store.loadSettings()
|
||||||
@@ -55,21 +56,12 @@ onMounted(async () => {
|
|||||||
} catch { /* non-fatal */ }
|
} catch { /* non-fatal */ }
|
||||||
})
|
})
|
||||||
async function onToggle() {
|
async function onToggle() {
|
||||||
saving.value = true
|
const ok = await save({ cpu_embed_enabled: enabled.value }, {
|
||||||
try {
|
successMessage: enabled.value
|
||||||
await store.patchSettings({ cpu_embed_enabled: enabled.value })
|
|
||||||
toast({
|
|
||||||
text: enabled.value
|
|
||||||
? 'CPU embedding on — imports queue embeds for the ml-worker'
|
? 'CPU embedding on — imports queue embeds for the ml-worker'
|
||||||
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
|
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
|
||||||
type: 'success',
|
|
||||||
})
|
})
|
||||||
} catch (e) {
|
if (!ok) enabled.value = !enabled.value
|
||||||
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
|
||||||
enabled.value = !enabled.value
|
|
||||||
} finally {
|
|
||||||
saving.value = false
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
async function run() {
|
async function run() {
|
||||||
busy.value = true
|
busy.value = true
|
||||||
@@ -80,5 +72,4 @@ async function run() {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -24,6 +24,7 @@
|
|||||||
the CPU fallback.
|
the CPU fallback.
|
||||||
</p>
|
</p>
|
||||||
<div class="fc-tile-stack">
|
<div class="fc-tile-stack">
|
||||||
|
<VideoEmbeddingCard />
|
||||||
<GpuAgentCard />
|
<GpuAgentCard />
|
||||||
<GpuTriageCard />
|
<GpuTriageCard />
|
||||||
<MLBackfillCard />
|
<MLBackfillCard />
|
||||||
@@ -36,7 +37,6 @@
|
|||||||
Suggestion thresholds, trained heads and tag aliases.
|
Suggestion thresholds, trained heads and tag aliases.
|
||||||
</p>
|
</p>
|
||||||
<div class="fc-tile-stack">
|
<div class="fc-tile-stack">
|
||||||
<MLThresholdSliders />
|
|
||||||
<CropProposersCard />
|
<CropProposersCard />
|
||||||
<HeadsCard />
|
<HeadsCard />
|
||||||
<AliasTable />
|
<AliasTable />
|
||||||
@@ -77,7 +77,7 @@ import ArchiveReextractCard from './ArchiveReextractCard.vue'
|
|||||||
import MissingFileRepairCard from './MissingFileRepairCard.vue'
|
import MissingFileRepairCard from './MissingFileRepairCard.vue'
|
||||||
import GpuTriageCard from './GpuTriageCard.vue'
|
import GpuTriageCard from './GpuTriageCard.vue'
|
||||||
import DbMaintenanceCard from './DbMaintenanceCard.vue'
|
import DbMaintenanceCard from './DbMaintenanceCard.vue'
|
||||||
import MLThresholdSliders from './MLThresholdSliders.vue'
|
import VideoEmbeddingCard from './VideoEmbeddingCard.vue'
|
||||||
import CropProposersCard from './CropProposersCard.vue'
|
import CropProposersCard from './CropProposersCard.vue'
|
||||||
import HeadsCard from './HeadsCard.vue'
|
import HeadsCard from './HeadsCard.vue'
|
||||||
import GpuAgentCard from './GpuAgentCard.vue'
|
import GpuAgentCard from './GpuAgentCard.vue'
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<template>
|
<template>
|
||||||
<v-row dense>
|
<v-row density="compact">
|
||||||
<v-col v-for="card in cards" :key="card.label" cols="12" sm="6" md="4" lg="3" xl="2">
|
<v-col v-for="card in cards" :key="card.label" cols="12" sm="6" md="4" lg="3" xl="2">
|
||||||
<v-card class="fc-stat">
|
<v-card class="fc-stat">
|
||||||
<v-card-text>
|
<v-card-text>
|
||||||
|
|||||||
@@ -78,6 +78,7 @@
|
|||||||
import { computed, ref } from 'vue'
|
import { computed, ref } from 'vue'
|
||||||
|
|
||||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||||
|
import { humanBytes } from '../../utils/bytes.js'
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
import QueueStatusBar from './QueueStatusBar.vue'
|
||||||
|
|
||||||
@@ -98,14 +99,6 @@ const summaryType = computed(() => {
|
|||||||
return summary.value && summary.value.redundant > 0 ? 'info' : 'success'
|
return summary.value && summary.value.redundant > 0 ? 'info' : 'success'
|
||||||
})
|
})
|
||||||
|
|
||||||
function humanBytes (n) {
|
|
||||||
const b = Number(n || 0)
|
|
||||||
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
|
|
||||||
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
|
|
||||||
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
|
|
||||||
return b + ' B'
|
|
||||||
}
|
|
||||||
|
|
||||||
// The confirm dialog gates the destructive apply; close it, then run.
|
// The confirm dialog gates the destructive apply; close it, then run.
|
||||||
function apply () {
|
function apply () {
|
||||||
confirmOpen.value = false
|
confirmOpen.value = false
|
||||||
|
|||||||
+18
-16
@@ -12,17 +12,17 @@
|
|||||||
</div>
|
</div>
|
||||||
<v-row>
|
<v-row>
|
||||||
<v-col cols="12" sm="6">
|
<v-col cols="12" sm="6">
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="local.video_frame_interval_seconds"
|
v-model="local.video_frame_interval_seconds"
|
||||||
label="Frame interval (s)" type="number" min="0.5" step="0.5"
|
label="Frame interval (s)" :min="0.5" :step="0.5"
|
||||||
density="comfortable" hide-details @change="save"
|
density="comfortable" max-width="none" @change="onSave"
|
||||||
/>
|
/>
|
||||||
</v-col>
|
</v-col>
|
||||||
<v-col cols="12" sm="6">
|
<v-col cols="12" sm="6">
|
||||||
<v-text-field
|
<SettingNumberField
|
||||||
v-model.number="local.video_max_frames"
|
v-model="local.video_max_frames"
|
||||||
label="Max frames" type="number" min="1" step="1"
|
label="Max frames" :min="1" :step="1"
|
||||||
density="comfortable" hide-details @change="save"
|
density="comfortable" max-width="none" @change="onSave"
|
||||||
/>
|
/>
|
||||||
</v-col>
|
</v-col>
|
||||||
</v-row>
|
</v-row>
|
||||||
@@ -32,21 +32,23 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
import { reactive, watch } from 'vue'
|
import { reactive, watch } from 'vue'
|
||||||
import { useMLStore } from '../../stores/ml.js'
|
import { useMLStore } from '../../stores/ml.js'
|
||||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||||
|
import SettingNumberField from '../common/SettingNumberField.vue'
|
||||||
|
import { useSettingSave } from '../../composables/useSettingSave.js'
|
||||||
|
|
||||||
const store = useMLStore()
|
const store = useMLStore()
|
||||||
|
const { save } = useSettingSave(store.patchSettings)
|
||||||
const local = reactive({})
|
const local = reactive({})
|
||||||
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
|
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
|
||||||
|
|
||||||
async function save() {
|
// SettingNumberField clamps interval to ≥0.5 and max-frames to ≥1 before this
|
||||||
const patch = {
|
// fires, so an out-of-range value never reaches the API.
|
||||||
video_frame_interval_seconds: local.video_frame_interval_seconds,
|
function onSave() {
|
||||||
video_max_frames: local.video_max_frames
|
save({
|
||||||
}
|
video_frame_interval_seconds: Number(local.video_frame_interval_seconds),
|
||||||
try { await store.patchSettings(patch) }
|
video_max_frames: Number(local.video_max_frames),
|
||||||
catch (e) { toast({ text: e.message, type: 'error' }) }
|
})
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
import { ref } from 'vue'
|
||||||
|
import { toast } from '../utils/toast.js'
|
||||||
|
|
||||||
|
// The shared "persist a settings patch" flow for the ML settings cards. Flips a
|
||||||
|
// busy flag, calls the store's patch (which rethrows on failure), toasts
|
||||||
|
// success/error, and returns true/false so a toggle handler can revert its
|
||||||
|
// optimistic switch on failure. Centralises the try/catch/toast the cards each
|
||||||
|
// hand-rolled (HeadsCard x6, CropProposersCard, MLBackfillCard) — and where the
|
||||||
|
// threshold-clamp drifted; the clamp now lives in <SettingNumberField>.
|
||||||
|
//
|
||||||
|
// Pass the store's patch fn, e.g. useSettingSave(ml.patchSettings).
|
||||||
|
export function useSettingSave(patchFn) {
|
||||||
|
const busy = ref(false)
|
||||||
|
|
||||||
|
// opts.successMessage — toast on success (toggles announce their new state;
|
||||||
|
// silent field-saves omit it). opts.errorPrefix — the failure toast prefix
|
||||||
|
// ("Could not save" default; toggles used "Could not update").
|
||||||
|
async function save(patch, { successMessage = '', errorPrefix = 'Could not save' } = {}) {
|
||||||
|
busy.value = true
|
||||||
|
try {
|
||||||
|
await patchFn(patch)
|
||||||
|
if (successMessage) toast({ text: successMessage, type: 'success' })
|
||||||
|
return true
|
||||||
|
} catch (e) {
|
||||||
|
toast({ text: `${errorPrefix}: ${e.message}`, type: 'error' })
|
||||||
|
return false
|
||||||
|
} finally {
|
||||||
|
busy.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { busy, save }
|
||||||
|
}
|
||||||
@@ -20,7 +20,7 @@ const routes = [
|
|||||||
|
|
||||||
// FC-2: image backbone
|
// FC-2: image backbone
|
||||||
{ path: '/showcase', name: 'showcase', component: ShowcaseView, meta: { title: 'Showcase', navOrder: 10 } },
|
{ path: '/showcase', name: 'showcase', component: ShowcaseView, meta: { title: 'Showcase', navOrder: 10 } },
|
||||||
{ path: '/gallery', name: 'gallery', component: GalleryView, meta: { title: 'Gallery', navOrder: 20 } },
|
{ path: '/gallery', name: 'gallery', component: GalleryView, meta: { title: 'Gallery', navOrder: 20, stickyChrome: true } },
|
||||||
// Explore: a 3-pane tagging workspace — walk an image's visual neighbours
|
// Explore: a 3-pane tagging workspace — walk an image's visual neighbours
|
||||||
// (left) while tagging the focused image (center viewer + modal-parity tag
|
// (left) while tagging the focused image (center viewer + modal-parity tag
|
||||||
// rail). Optional anchor param — the bare /explore nav entry SEEDS a random
|
// rail). Optional anchor param — the bare /explore nav entry SEEDS a random
|
||||||
@@ -29,11 +29,11 @@ const routes = [
|
|||||||
// Browse hub (operator-asked 2026-06-09): Posts / Artists / Tags as tabs —
|
// Browse hub (operator-asked 2026-06-09): Posts / Artists / Tags as tabs —
|
||||||
// the three "browse the library by an axis" surfaces. One nav entry; the old
|
// the three "browse the library by an axis" surfaces. One nav entry; the old
|
||||||
// standalone paths redirect into the matching tab (below).
|
// standalone paths redirect into the matching tab (below).
|
||||||
{ path: '/browse', name: 'browse', component: BrowseView, meta: { title: 'Browse', navOrder: 30 } },
|
{ path: '/browse', name: 'browse', component: BrowseView, meta: { title: 'Browse', navOrder: 30, stickyChrome: true } },
|
||||||
// Artist detail — no meta.title (reached by clicking an artist, not nav).
|
// Artist detail — no meta.title (reached by clicking an artist, not nav).
|
||||||
{ path: '/artist/:slug', name: 'artist', component: ArtistView },
|
{ path: '/artist/:slug', name: 'artist', component: ArtistView },
|
||||||
// Series browse — a nav entry (meta.title).
|
// Series browse — a nav entry (meta.title).
|
||||||
{ path: '/series', name: 'series', component: SeriesView, meta: { title: 'Series', navOrder: 40 } },
|
{ path: '/series', name: 'series', component: SeriesView, meta: { title: 'Series', navOrder: 40, stickyChrome: true } },
|
||||||
// Series management — no meta.title (reached from a series card/tag).
|
// Series management — no meta.title (reached from a series card/tag).
|
||||||
{ path: '/series/:tagId', name: 'series-manage', component: SeriesManageView },
|
{ path: '/series/:tagId', name: 'series-manage', component: SeriesManageView },
|
||||||
// Series reader — immersive (no top nav, no meta.title).
|
// Series reader — immersive (no top nav, no meta.title).
|
||||||
@@ -41,10 +41,10 @@ const routes = [
|
|||||||
|
|
||||||
// FC-3: subscription backbone — purely management (sources/downloads),
|
// FC-3: subscription backbone — purely management (sources/downloads),
|
||||||
// distinct from the Browse hub.
|
// distinct from the Browse hub.
|
||||||
{ path: '/subscriptions', name: 'subscriptions', component: SubscriptionsView, meta: { title: 'Subscriptions', navOrder: 50 } },
|
{ path: '/subscriptions', name: 'subscriptions', component: SubscriptionsView, meta: { title: 'Subscriptions', navOrder: 50, stickyChrome: true } },
|
||||||
|
|
||||||
// Settings — config, pinned to the right of the nav (TopNav special-cases it).
|
// Settings — config, pinned to the right of the nav (TopNav special-cases it).
|
||||||
{ path: '/settings', name: 'settings', component: SettingsView, meta: { title: 'Settings' } },
|
{ path: '/settings', name: 'settings', component: SettingsView, meta: { title: 'Settings', stickyChrome: true } },
|
||||||
|
|
||||||
// The old standalone paths now redirect into the Browse hub, preserving any
|
// The old standalone paths now redirect into the Browse hub, preserving any
|
||||||
// deep-link query (e.g. /posts?post_id=N → /browse?tab=posts&post_id=N). The
|
// deep-link query (e.g. /posts?post_id=N → /browse?tab=posts&post_id=N). The
|
||||||
|
|||||||
@@ -25,6 +25,10 @@ export const useExploreStore = defineStore('explore', () => {
|
|||||||
const cursor = ref(-1)
|
const cursor = ref(-1)
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
const error = ref(null)
|
const error = ref(null)
|
||||||
|
// Reach (#1476): how far the walk reaches past the anchor's immediate cluster.
|
||||||
|
// 0 = nearest (can get stuck in a dense signature); ~0.4 default mixes in
|
||||||
|
// mid-far escape routes so the walk diversifies without hitting "Random image".
|
||||||
|
const reach = ref(0.4)
|
||||||
|
|
||||||
const inflight = useInflightToken()
|
const inflight = useInflightToken()
|
||||||
|
|
||||||
@@ -46,7 +50,13 @@ export const useExploreStore = defineStore('explore', () => {
|
|||||||
const body = await api.get('/api/gallery/similar', {
|
const body = await api.get('/api/gallery/similar', {
|
||||||
// exclude_wip: keep work-in-progress out of the Explore rabbit-hole
|
// exclude_wip: keep work-in-progress out of the Explore rabbit-hole
|
||||||
// (the gallery's own "similar" button still shows it) — operator 2026-07-08.
|
// (the gallery's own "similar" button still shows it) — operator 2026-07-08.
|
||||||
params: { similar_to: numId, limit: NEIGHBOR_LIMIT, exclude_wip: 1 },
|
// reach + exclude_ids (#1476): reach past the dense cluster + never re-serve
|
||||||
|
// an already-walked image, so the walk keeps moving instead of getting stuck.
|
||||||
|
params: {
|
||||||
|
similar_to: numId, limit: NEIGHBOR_LIMIT, exclude_wip: 1,
|
||||||
|
reach: reach.value,
|
||||||
|
exclude_ids: breadcrumb.value.map((c) => c.id).join(','),
|
||||||
|
},
|
||||||
})
|
})
|
||||||
if (!t.isCurrent()) return
|
if (!t.isCurrent()) return
|
||||||
neighbors.value = body.images || []
|
neighbors.value = body.images || []
|
||||||
@@ -113,6 +123,14 @@ export const useExploreStore = defineStore('explore', () => {
|
|||||||
loading.value = false
|
loading.value = false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Change how far the walk reaches and re-fetch the current anchor's neighbours
|
||||||
|
// with the new setting (the anchor + trail are unchanged — only the grid varies).
|
||||||
|
function setReach (v) {
|
||||||
|
reach.value = Math.max(0, Math.min(1, Number(v)))
|
||||||
|
const id = anchor.value?.id
|
||||||
|
if (id != null) anchorOn(id)
|
||||||
|
}
|
||||||
|
|
||||||
// --- TagPanel "host" surface ---------------------------------------------
|
// --- TagPanel "host" surface ---------------------------------------------
|
||||||
// The anchor IS the current image (same /api/gallery/image/<id> payload the
|
// The anchor IS the current image (same /api/gallery/image/<id> payload the
|
||||||
// modal uses), so these mirror the modal store's tag-CRUD, targeting the
|
// modal uses), so these mirror the modal store's tag-CRUD, targeting the
|
||||||
@@ -189,8 +207,8 @@ export const useExploreStore = defineStore('explore', () => {
|
|||||||
function close () {}
|
function close () {}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
anchor, neighbors, breadcrumb, cursor, loading, error, NEIGHBOR_LIMIT,
|
anchor, neighbors, breadcrumb, cursor, loading, error, NEIGHBOR_LIMIT, reach,
|
||||||
anchorOn, reset, backTarget, forwardTarget,
|
anchorOn, reset, backTarget, forwardTarget, setReach,
|
||||||
// host surface
|
// host surface
|
||||||
current, currentImageId,
|
current, currentImageId,
|
||||||
reloadTags, addExistingTag, removeTag, createAndAdd, close,
|
reloadTags, addExistingTag, removeTag, createAndAdd, close,
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ export const useImportStore = defineStore('import', () => {
|
|||||||
const settings = ref(null)
|
const settings = ref(null)
|
||||||
const settingsLoading = ref(false)
|
const settingsLoading = ref(false)
|
||||||
const settingsError = ref(null)
|
const settingsError = ref(null)
|
||||||
|
const wipScanBusy = ref(false)
|
||||||
|
|
||||||
async function loadSettings() {
|
async function loadSettings() {
|
||||||
settingsLoading.value = true
|
settingsLoading.value = true
|
||||||
@@ -40,8 +41,25 @@ export const useImportStore = defineStore('import', () => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Enqueue the back-catalogue WIP-title scan (task #1458). New imports are
|
||||||
|
// tagged live; this catches posts already in the library. Fire-and-forget —
|
||||||
|
// the sweep runs on the maintenance worker and its run shows in Activity.
|
||||||
|
async function scanWipTitles() {
|
||||||
|
wipScanBusy.value = true
|
||||||
|
try {
|
||||||
|
const r = await api.post('/api/settings/wip-title/scan')
|
||||||
|
toast({ text: 'Scanning existing posts for WIP titles…', type: 'success' })
|
||||||
|
return r
|
||||||
|
} catch (e) {
|
||||||
|
toast({ text: `WIP scan failed: ${e.message}`, type: 'error' })
|
||||||
|
throw e
|
||||||
|
} finally {
|
||||||
|
wipScanBusy.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
settings, settingsLoading, settingsError,
|
settings, settingsLoading, settingsError, wipScanBusy,
|
||||||
loadSettings, patchSettings,
|
loadSettings, patchSettings, scanWipTitles,
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user