Compare commits
69
Commits
ext-1.0.10
...
725bf15f88
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
725bf15f88 | ||
|
|
bc4eba636d | ||
|
|
dbc4e8b0c6 | ||
|
|
b14818303c | ||
|
|
08418d54a3 | ||
|
|
b1bd2531ad | ||
|
|
389afe2f7b | ||
|
|
b979062dd7 | ||
|
|
573228b9da | ||
|
|
d044e93bdb | ||
|
|
ed2b1adc2e | ||
|
|
5e1996e77f | ||
|
|
98b56330d0 | ||
|
|
6959e1220c | ||
|
|
2529b516e6 | ||
|
|
8f1ac0c96a | ||
|
|
5fd171a544 | ||
|
|
62583791d8 | ||
|
|
0a5bbe81dc | ||
|
|
6663e06aa6 | ||
|
|
63e0a423d7 | ||
|
|
499720d87e | ||
|
|
5e72076298 | ||
|
|
e21c9fdd34 | ||
|
|
6b3ec98fa8 | ||
|
|
bd24f4e876 | ||
|
|
1a941e900b | ||
|
|
2e01242381 | ||
|
|
41f2bec3af | ||
|
|
0e15c44c51 | ||
|
|
d38585ed94 | ||
|
|
a3071a7549 | ||
|
|
b6b9fd8287 | ||
|
|
bce894ba24 | ||
|
|
5771fd5770 | ||
|
|
6d98dfc0ec | ||
|
|
b3989d0224 | ||
|
|
cd0b0ff04a | ||
|
|
454eb3f973 | ||
|
|
7e065fed70 | ||
|
|
dee93faa37 | ||
|
|
d9aa5aa832 | ||
|
|
6c76f08b69 | ||
|
|
fb2c4d5b80 | ||
|
|
609bc82acc | ||
|
|
7a20c55441 | ||
|
|
0c43fa3eb2 | ||
|
|
cf06c81db9 | ||
|
|
7b1019ba82 | ||
|
|
0db38cc111 | ||
|
|
a7e626a67a | ||
|
|
fe48e77821 | ||
|
|
9eb946b21b | ||
|
|
5447a40e97 | ||
|
|
239b1ed8d9 | ||
|
|
cd5444e3ae | ||
|
|
5a0e1bbd03 | ||
|
|
1ac448d881 | ||
|
|
bfc5135f19 | ||
|
|
89155478a8 | ||
|
|
516521e7b0 | ||
|
|
ddf896078c | ||
|
|
2e0f8f8c61 | ||
|
|
2ce467e347 | ||
|
|
39cf81aea6 | ||
|
|
0d204e6837 | ||
|
|
11dd324f89 | ||
|
|
1c6452e10e | ||
|
|
597b91d29b |
@@ -0,0 +1,370 @@
|
||||
|
||||
# TEMPORARY — milestone 328 steps 1-2. Delete once the baseline is stamped.
|
||||
#
|
||||
# Squashing 87 alembic revisions into one baseline has exactly one dangerous
|
||||
# failure: the generated baseline does not reproduce the schema the chain
|
||||
# produced, `alembic stamp` writes a version string anyway (it validates
|
||||
# NOTHING), and the divergence surfaces on the next real migration against the
|
||||
# operator's live data.
|
||||
#
|
||||
# So this workflow does the comparison in CI, where a pgvector Postgres already
|
||||
# gets built from the chain on every integration run, and nothing is at risk.
|
||||
# It answers one question: does `upgrade head` on the collapsed chain produce a
|
||||
# byte-identical schema to `upgrade head` on the 87-revision chain?
|
||||
#
|
||||
# The chain is read from git rather than from the working tree, so this keeps
|
||||
# working AFTER the old revisions are deleted — `chain_ref` names a commit that
|
||||
# still has them. That is what makes this the proof for step 1 and the
|
||||
# pre-flight for step 2, rather than a one-shot script.
|
||||
#
|
||||
# While the chain is still present it also autogenerates a candidate baseline
|
||||
# from the models and prints it. That is a starting point, NOT the answer:
|
||||
# autogenerate reads SQLAlchemy metadata, and three things here do not live
|
||||
# there —
|
||||
# * CREATE EXTENSION vector (0001)
|
||||
# * CREATE EXTENSION tsm_system_rows (0004)
|
||||
# * the HNSW index on image_record.siglip_embedding, which is raw SQL
|
||||
# because alembic's create_index cannot express `USING hnsw (...)` (0036)
|
||||
# plus any CHECK constraint or server_default that a migration added without
|
||||
# the model declaring it. Those must be hand-added, and the diff below is what
|
||||
# proves none were missed.
|
||||
name: Alembic baseline
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
chain_ref:
|
||||
description: 'Commit/tag carrying the full chain; blank = this ref (use a pinned commit only AFTER the collapse)'
|
||||
type: string
|
||||
default: ''
|
||||
mode:
|
||||
description: 'chain = compare against this tree''s migrations; models = compare against a schema built from the MODELS'
|
||||
type: string
|
||||
default: 'chain'
|
||||
|
||||
jobs:
|
||||
compare:
|
||||
runs-on: python-ci
|
||||
container:
|
||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||
env:
|
||||
DB_USER: fabledcurator
|
||||
DB_PASSWORD: ci_integration
|
||||
DB_PORT: "5432"
|
||||
DB_NAME: fabledcurator_test
|
||||
SECRET_KEY: ci_integration_placeholder
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
env:
|
||||
POSTGRES_USER: fabledcurator
|
||||
POSTGRES_PASSWORD: ci_integration
|
||||
POSTGRES_DB: fabledcurator_test
|
||||
options: >-
|
||||
--health-cmd "pg_isready -U fabledcurator"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 10
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# Full history is the point: `chain_ref` is read out of git, so a
|
||||
# shallow clone would not have the revisions to compare against.
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Resolve the Postgres service and install deps
|
||||
run: |
|
||||
set -eux
|
||||
# Same service-IP dance as ci.yml's integration job; see the long
|
||||
# comment there for why the job name must stay separator-free.
|
||||
PG=$(docker ps --filter "name=compare" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
||||
test -n "$PG"
|
||||
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
||||
test -n "$PG_IP"
|
||||
echo "PG_CONTAINER=$PG" >> "$GITHUB_ENV"
|
||||
echo "DB_HOST=$PG_IP" >> "$GITHUB_ENV"
|
||||
for i in $(seq 1 60); do
|
||||
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
uv pip install --system -r requirements.txt
|
||||
else
|
||||
pip install -r requirements.txt
|
||||
fi
|
||||
|
||||
# DB 1: the 87-revision chain, read out of git at `chain_ref`.
|
||||
#
|
||||
# A git worktree rather than a checkout, so the current tree — which is
|
||||
# what we are testing — is left completely alone.
|
||||
- name: Build the schema the OLD chain produces
|
||||
env:
|
||||
CHAIN_REF: ${{ github.event.inputs.chain_ref }}
|
||||
THIS_SHA: ${{ github.sha }}
|
||||
run: |
|
||||
set -eux
|
||||
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_chain
|
||||
# Blank means "the chain in this ref", which is what you want while
|
||||
# the chain is still intact — comparing the models against a PINNED
|
||||
# older commit reports every migration written since as a difference.
|
||||
# Pin it only after the collapse, when the tree no longer has them.
|
||||
git worktree add /tmp/chain "${CHAIN_REF:-$THIS_SHA}"
|
||||
ls /tmp/chain/alembic/versions/*.py | wc -l
|
||||
cd /tmp/chain
|
||||
DB_NAME=fc_chain alembic upgrade head
|
||||
cd -
|
||||
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
|
||||
--no-owner --no-privileges -d fc_chain > chain.sql
|
||||
wc -l chain.sql
|
||||
# Emit the dump itself, checksummed, for local analysis. Reconciling
|
||||
# the models against the deployed schema (#3275) needs the ACTUAL
|
||||
# schema, not an inference from a diff — parsing table context out of
|
||||
# unified-diff hunks drops every table whose CREATE TABLE line falls
|
||||
# outside a hunk, which silently under-reports.
|
||||
#
|
||||
# base64 + sha256 for the same reason as the candidate: a plain cat
|
||||
# of a file this size was truncated mid-line by the runner with the
|
||||
# step still green (run 4964).
|
||||
set +x
|
||||
B64=$(base64 -w 120 chain.sql)
|
||||
echo "===== BEGIN CHAIN SCHEMA (base64) ====="
|
||||
echo "$B64"
|
||||
echo "===== END CHAIN SCHEMA ====="
|
||||
echo "chain-sha256: $(sha256sum chain.sql | cut -d' ' -f1)"
|
||||
echo "chain-bytes: $(wc -c < chain.sql)"
|
||||
set -x
|
||||
|
||||
# A candidate baseline, autogenerated from the models against an EMPTY
|
||||
# database so every table shows up as a create. Printed for a human to
|
||||
# finish — it will be missing the three raw-SQL items named at the top.
|
||||
#
|
||||
# Gated on the TREE, not on a workflow input. A `type: boolean` input
|
||||
# read back as `github.event.inputs.generate == 'true'` silently
|
||||
# evaluated false on this runner (run 4960 skipped this step entirely
|
||||
# with no diagnostic) — the same `github.event.inputs` typing quirk
|
||||
# build.yml already works around. The file count is the real question
|
||||
# anyway: there is nothing to generate once the chain is collapsed.
|
||||
- name: Autogenerate a candidate baseline
|
||||
run: |
|
||||
set -eux
|
||||
if [ "$(ls alembic/versions/*.py | wc -l)" -le 1 ]; then
|
||||
echo "already collapsed — nothing to generate"
|
||||
exit 0
|
||||
fi
|
||||
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_gen
|
||||
# Hide the existing revisions so alembic sees an empty history and
|
||||
# emits the whole schema rather than a delta.
|
||||
mkdir -p /tmp/versions_held
|
||||
mv alembic/versions/*.py /tmp/versions_held/ 2>/dev/null || true
|
||||
DB_NAME=fc_gen alembic revision --autogenerate -m "baseline" || true
|
||||
# Printed rather than uploaded: ci-requirements.md records that this
|
||||
# runner cannot do actions/upload-artifact@v4+, and the repo dropped
|
||||
# the action entirely in 2026-05, so the job log is the retrieval
|
||||
# channel actually proven here.
|
||||
#
|
||||
# base64, not the raw file. A plain `cat` of the ~33KB candidate was
|
||||
# TRUNCATED MID-LINE by the runner on run 4964 — it stopped inside
|
||||
# `sa.Column('mime', sa.String(length=128)` and carried straight on
|
||||
# to the next traced command, with the step still green. A silent
|
||||
# cut in the middle of a schema definition is the worst possible
|
||||
# failure here, because the truncated text still looks like a
|
||||
# plausible file.
|
||||
#
|
||||
# base64 at a fixed narrow width gives many short lines instead of
|
||||
# few long ones, and — the actual point — a checksum and a line
|
||||
# count that make truncation DETECTABLE rather than invisible.
|
||||
set +x
|
||||
F=$(ls alembic/versions/*.py | head -1)
|
||||
B64=$(base64 -w 120 "$F")
|
||||
echo "===== BEGIN CANDIDATE BASELINE (base64) ====="
|
||||
echo "$B64"
|
||||
echo "===== END CANDIDATE BASELINE ====="
|
||||
echo "candidate-sha256: $(sha256sum "$F" | cut -d' ' -f1)"
|
||||
echo "candidate-bytes: $(wc -c < "$F")"
|
||||
echo "candidate-b64-lines: $(echo "$B64" | wc -l)"
|
||||
set -x
|
||||
mkdir -p /tmp/candidate
|
||||
cp alembic/versions/*.py /tmp/candidate/
|
||||
# Put the tree back exactly as it was; this job never mutates state.
|
||||
rm -f alembic/versions/*.py
|
||||
mv /tmp/versions_held/*.py alembic/versions/ 2>/dev/null || true
|
||||
|
||||
# DB 2: what the CURRENT tree produces.
|
||||
#
|
||||
# `mode: models` applies the candidate autogenerated from the MODELS
|
||||
# instead, which is what answers "do the models describe the schema?" —
|
||||
# the question #3275 exists because nobody had ever asked it. Under that
|
||||
# mode a clean diff means autogenerate is trustworthy again.
|
||||
#
|
||||
# The two extensions are created by hand first. They are database
|
||||
# objects, not table metadata, so no model can carry them and their
|
||||
# absence is not a model defect — it is simply outside what this
|
||||
# comparison is asking about.
|
||||
- name: Build the schema the CURRENT tree produces
|
||||
env:
|
||||
MODE: ${{ github.event.inputs.mode }}
|
||||
run: |
|
||||
set -eux
|
||||
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_base
|
||||
if [ "${MODE:-chain}" = "models" ]; then
|
||||
docker exec "$PG_CONTAINER" psql -U fabledcurator -d fc_base \
|
||||
-c "CREATE EXTENSION IF NOT EXISTS vector" \
|
||||
-c "CREATE EXTENSION IF NOT EXISTS tsm_system_rows"
|
||||
mkdir -p /tmp/held
|
||||
mv alembic/versions/*.py /tmp/held/
|
||||
cp /tmp/candidate/*.py alembic/versions/
|
||||
# Autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR(...) without
|
||||
# importing it, so the file it writes cannot run:
|
||||
# NameError: name 'pgvector' is not defined
|
||||
# Observed on run 4988, which is the proof rather than the theory.
|
||||
# This is a defect in the GENERATOR, not in the models, so it is
|
||||
# repaired here rather than counted as a schema difference — the
|
||||
# comparison is about whether the models describe the schema.
|
||||
sed -i '0,/^import sqlalchemy as sa$/s//import sqlalchemy as sa\nimport pgvector.sqlalchemy.vector/' alembic/versions/*.py
|
||||
grep -n 'import pgvector' alembic/versions/*.py
|
||||
# Second generator defect, same class as the missing import.
|
||||
#
|
||||
# base.py's naming convention includes %(constraint_name)s for ck,
|
||||
# which — unlike uq/fk/ix — means the convention is applied even to
|
||||
# a CheckConstraint that HAS a name. So a model declaring
|
||||
# name="singleton" correctly becomes ck_ml_settings_singleton in
|
||||
# the metadata. Autogenerate then writes that RENDERED name into
|
||||
# the migration, and running the migration applies the convention a
|
||||
# SECOND time: ck_ml_settings_ck_ml_settings_singleton.
|
||||
#
|
||||
# That is round-tripping damage done by the generator, not a claim
|
||||
# the models make, so it is repaired here rather than counted as a
|
||||
# schema difference. Undone by removing the ck_<table>_ prefix the
|
||||
# convention will re-add — the exact inverse, and it only fires on
|
||||
# a name that actually carries its own table's prefix.
|
||||
python3 - alembic/versions/*.py <<'PYEOF'
|
||||
import re, sys
|
||||
|
||||
table = None
|
||||
for path in sys.argv[1:]:
|
||||
out = []
|
||||
for line in open(path):
|
||||
m = re.search(r"op\.create_table\(\s*[\"']([A-Za-z0-9_]+)[\"']", line)
|
||||
if m:
|
||||
table = m.group(1)
|
||||
if table and "CheckConstraint" in line:
|
||||
prefix = f"ck_{table}_"
|
||||
line = re.sub(
|
||||
r"(name=[\"'])" + re.escape(prefix),
|
||||
r"\1",
|
||||
line,
|
||||
)
|
||||
out.append(line)
|
||||
open(path, "w").writelines(out)
|
||||
PYEOF
|
||||
grep -n 'CheckConstraint' alembic/versions/*.py || true
|
||||
ls alembic/versions/*.py
|
||||
DB_NAME=fc_base alembic upgrade head
|
||||
rm -f alembic/versions/*.py
|
||||
mv /tmp/held/*.py alembic/versions/
|
||||
else
|
||||
ls alembic/versions/*.py | wc -l
|
||||
DB_NAME=fc_base alembic upgrade head
|
||||
fi
|
||||
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
|
||||
--no-owner --no-privileges -d fc_base > baseline.sql
|
||||
wc -l baseline.sql
|
||||
|
||||
# The verdict.
|
||||
#
|
||||
# pg_dump orders dumpable objects by name within type, not by creation
|
||||
# order, so two schemas built by different routes are directly
|
||||
# comparable. Normalisation is deliberately minimal, because a filter
|
||||
# that hides a real difference is the one way this check passes when it
|
||||
# should fail — blank lines, SQL comments, trailing whitespace, and:
|
||||
#
|
||||
# \restrict / \unrestrict — a per-invocation RANDOM NONCE that newer
|
||||
# pg_dump emits to fence the dump against injection during restore. It
|
||||
# differs on every run by construction, so it is noise by definition,
|
||||
# not a schema difference. Measured on run 4960, the control: two dumps
|
||||
# of the SAME schema came back 1123 lines each and differed on exactly
|
||||
# these two lines and nothing else. That control is what licenses this
|
||||
# filter — it was observed to be the only false positive, rather than
|
||||
# assumed to be one.
|
||||
# Column ORDER inside a CREATE TABLE is compared separately from column
|
||||
# CONTENT, and only content is fatal.
|
||||
#
|
||||
# A table built by 87 migrations has its columns in ADD COLUMN order; the
|
||||
# same table built in one shot has them in declaration order. That is a
|
||||
# real and permanent difference which no baseline can erase — the
|
||||
# operator's existing database keeps chain order forever, a fresh install
|
||||
# gets model order — so a check that fails on it would never pass and
|
||||
# would teach nothing. FC reaches every column through the ORM by name,
|
||||
# and `SELECT *` ordering is not depended on anywhere.
|
||||
#
|
||||
# So the second pass SORTS the column lines within each CREATE TABLE
|
||||
# rather than DELETING them. That distinction is the whole point: sorting
|
||||
# cannot hide a column that exists on one side only, or one whose type,
|
||||
# nullability or default differs — those still land in the diff. A filter
|
||||
# could have hidden all three.
|
||||
#
|
||||
# Both diffs are reported. The ordered one is informational; the
|
||||
# order-insensitive one is the verdict.
|
||||
- name: Diff
|
||||
run: |
|
||||
set -eu
|
||||
norm() {
|
||||
grep -vE '^\s*(--|$)' "$1" \
|
||||
| grep -vE '^\\(un)?restrict ' \
|
||||
| sed 's/[[:space:]]*$//'
|
||||
}
|
||||
norm chain.sql > a.txt
|
||||
norm baseline.sql > b.txt
|
||||
echo "normalised: chain=$(wc -l < a.txt) lines, current=$(wc -l < b.txt) lines"
|
||||
|
||||
sort_table_columns() {
|
||||
python3 - "$1" <<'PYEOF'
|
||||
import re, sys
|
||||
|
||||
lines = open(sys.argv[1]).read().splitlines()
|
||||
out, block = [], None
|
||||
for line in lines:
|
||||
if block is not None:
|
||||
# ');' on its own closes the CREATE TABLE body.
|
||||
if line.strip() == ");":
|
||||
out.extend(sorted(block))
|
||||
out.append(line)
|
||||
block = None
|
||||
else:
|
||||
# Drop the list comma before sorting. Only the LAST
|
||||
# column lacks one, so keeping it would make every
|
||||
# reordering look like a content change as well — the
|
||||
# comma is punctuation, and carries no schema meaning.
|
||||
block.append(line.rstrip().rstrip(","))
|
||||
continue
|
||||
out.append(line)
|
||||
if re.match(r"CREATE TABLE .*\($", line):
|
||||
block = []
|
||||
if block is not None: # unterminated body: emit it rather than drop it
|
||||
out.extend(block)
|
||||
print("\n".join(out))
|
||||
PYEOF
|
||||
}
|
||||
sort_table_columns a.txt > a.sorted.txt
|
||||
sort_table_columns b.txt > b.sorted.txt
|
||||
test "$(wc -l < a.sorted.txt)" = "$(wc -l < a.txt)"
|
||||
test "$(wc -l < b.sorted.txt)" = "$(wc -l < b.txt)"
|
||||
|
||||
if diff -u a.txt b.txt > schema.diff; then
|
||||
echo "ORDERED DIFF: identical, column order included."
|
||||
else
|
||||
echo "ORDERED DIFF: $(grep -cE '^[+-]' schema.diff) changed lines (informational):"
|
||||
cat schema.diff
|
||||
fi
|
||||
echo
|
||||
echo "================================================================"
|
||||
echo
|
||||
if diff -u a.sorted.txt b.sorted.txt > sorted.diff; then
|
||||
echo "SCHEMAS MATCH — every difference above is column ORDER alone."
|
||||
else
|
||||
echo "SCHEMAS DIFFER — $(grep -cE '^[+-]' sorted.diff) changed lines that are NOT ordering:"
|
||||
cat sorted.diff
|
||||
echo
|
||||
echo "The baseline is wrong, not the database. Do not stamp."
|
||||
exit 1
|
||||
fi
|
||||
+1434
-125
File diff suppressed because it is too large
Load Diff
+70
-99
@@ -2,7 +2,7 @@ name: CI
|
||||
|
||||
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
|
||||
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
|
||||
# - extension-version: guards the extension publish path (see the job).
|
||||
# - extension-version: the derived version resolves and is a shape AMO takes.
|
||||
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
|
||||
# - frontend-build: vitest unit + vite build.
|
||||
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
|
||||
@@ -35,25 +35,42 @@ jobs:
|
||||
- name: Ruff lint
|
||||
# agent/ included so the GPU-agent is linted before its image is built
|
||||
# (build.yml only `docker build`s it — this is where it gets checked).
|
||||
run: ruff check backend/ tests/ alembic/ agent/
|
||||
# scripts/ likewise: release_notes.py runs only on a tag push, so a
|
||||
# syntax or import error there would otherwise surface at the one
|
||||
# moment nobody wants to debug a workflow.
|
||||
run: ruff check backend/ tests/ alembic/ agent/ scripts/
|
||||
- name: Agent syntax check
|
||||
# The agent's runtime deps (torch/transformers/ultralytics) aren't in the
|
||||
# CI image, so we can't import it — but compileall parses every module,
|
||||
# catching syntax errors before the image build.
|
||||
run: python -m compileall -q agent/fc_agent
|
||||
|
||||
# Guards the extension publish path, which has no self-correcting behavior.
|
||||
# The extension version is DERIVED, not hand-maintained (milestone 271 step
|
||||
# 4): build.yml computes it from the commit TIME of the newest packaged
|
||||
# extension change and stamps it into manifest.json / package.json at build
|
||||
# time. The guard that used to live here — "packaged files changed but nobody
|
||||
# bumped the version" — was therefore checking a fact that had stopped
|
||||
# existing. Worse than useless: it would have failed this lane on every real
|
||||
# extension change, demanding a bump that decides nothing. Retired 2026-08-27
|
||||
# rather than left running beside the new mechanism (rule 22).
|
||||
#
|
||||
# build.yml's sign-extension job keys its AMO-signing cache purely on the
|
||||
# version string in extension/package.json: if an `ext-<version>` Forgejo
|
||||
# release already carries an XPI, signing is SKIPPED and that old signed XPI
|
||||
# is what build-web bakes into `:latest`. Nothing in that path inspects
|
||||
# whether extension/ actually changed — so a forgotten version bump ships a
|
||||
# stale extension on a fully green build, silently. (AMO can't help: it 409s
|
||||
# on re-signing a version, which is exactly why the cache exists.)
|
||||
# Two things are still worth asserting, and this is the only lane that can:
|
||||
# the extension.yml suite runs on node:24-slim, which is exactly why
|
||||
# version.spec.js sticks to packaging.sh's git-free subcommands.
|
||||
# 1. the derivation actually resolves on this commit
|
||||
# 2. the derived string is one AMO will accept, checked against Mozilla's
|
||||
# own published grammar rather than a loose "digits and dots"
|
||||
#
|
||||
# This job makes that case loud, on the dev push, instead of invisible at
|
||||
# merge-to-main. It is pure git + text work — no deps, no services.
|
||||
# The MAJOR.MINOR-agreement check that used to be (2) is gone with milestone
|
||||
# 318 step 8: the committed version no longer seeds anything, so there is no
|
||||
# hand-set part left for the two files to disagree about.
|
||||
#
|
||||
# Deliberately NOT checked here: that the derived value beats what has already
|
||||
# been signed. That guard belongs in build.yml, where it compares against the
|
||||
# real ext-* releases. Comparing against origin/main here would be wrong —
|
||||
# dev legitimately derives a LOWER value whenever main is ahead on the
|
||||
# extension, and a lane that fails for being behind is a lane people learn to
|
||||
# ignore.
|
||||
extension-version:
|
||||
runs-on: python-ci
|
||||
container:
|
||||
@@ -61,101 +78,48 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# Full history: the check diffs against the push's `before` SHA (or
|
||||
# the PR base), which a depth-1 clone wouldn't contain.
|
||||
# The derivation needs real history: a depth-1 clone sees one commit
|
||||
# and produces a wrong, too-low value RATHER THAN FAILING. Checking
|
||||
# that here is half the point of the lane.
|
||||
fetch-depth: 0
|
||||
- name: Extension version guard
|
||||
env:
|
||||
BEFORE: ${{ github.event.before }}
|
||||
PR_BASE: ${{ github.event.pull_request.base.sha }}
|
||||
- name: Extension version derives cleanly
|
||||
run: |
|
||||
set -eu
|
||||
# busybox sh on the act_runner — no bashisms (family rule).
|
||||
ver() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/'; }
|
||||
PKG=$(ver extension/package.json)
|
||||
MAN=$(ver extension/manifest.json)
|
||||
test -n "$PKG" || { echo "ERROR: no version found in extension/package.json"; exit 1; }
|
||||
test -n "$MAN" || { echo "ERROR: no version found in extension/manifest.json"; exit 1; }
|
||||
VERSION=$(sh extension/scripts/packaging.sh version)
|
||||
echo "derived: $VERSION"
|
||||
|
||||
# (1) Unconditional: the two version strings must agree. `web-ext sign`
|
||||
# reads manifest.json (package.json sits in --ignore-files and isn't
|
||||
# even inside the XPI), so AMO signs MAN and Firefox installs MAN.
|
||||
# build.yml keys its cache, release tag, XPI filename — and therefore
|
||||
# the version /api/extension/manifest reports to the update prompt —
|
||||
# on PKG. Divergence either hard-fails at AMO or ships a mislabelled
|
||||
# XPI whose update prompt lies about what's installed.
|
||||
if [ "$MAN" != "$PKG" ]; then
|
||||
echo "ERROR: extension version mismatch."
|
||||
echo " extension/manifest.json = $MAN <- what AMO signs / Firefox installs"
|
||||
echo " extension/package.json = $PKG <- what CI caches, names, and reports"
|
||||
echo "Set both to the same value."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# (2) If the SHIPPED extension changed, the version must have moved.
|
||||
# Mozilla's published grammar for AMO, transcribed verbatim from
|
||||
# MDN's manifest.json/version page:
|
||||
#
|
||||
# Compare against MAIN, not against the previous push. The publish
|
||||
# decision is made at merge-to-main against whatever ext-<version>
|
||||
# already exists, so "differs from main" is the question that matters.
|
||||
# Diffing against the previous dev push instead would demand a fresh
|
||||
# bump on every iteration — push, tweak the extension again, and CI
|
||||
# would insist on a second bump that buys nothing, inflating the
|
||||
# version for no reason. On a main push there is no "main to compare
|
||||
# to" yet, so fall back to that push's own before-SHA.
|
||||
if [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||
BASE="${BEFORE:-}"
|
||||
else
|
||||
BASE=$(git rev-parse --verify -q origin/main 2>/dev/null || git rev-parse --verify -q main 2>/dev/null || echo "")
|
||||
# PR base is the fallback when main isn't in the clone at all.
|
||||
[ -n "$BASE" ] || BASE="${PR_BASE:-}"
|
||||
fi
|
||||
case "$BASE" in
|
||||
''|0000000000000000000000000000000000000000)
|
||||
echo "No usable base ref (no main in clone / first push) — skipping the bump check."
|
||||
echo "OK: extension version $PKG"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
if ! git cat-file -e "$BASE^{commit}" 2>/dev/null; then
|
||||
echo "Base commit $BASE not in this clone — skipping the bump check."
|
||||
echo "OK: extension version $PKG"
|
||||
exit 0
|
||||
fi
|
||||
# Exclusions mirror --ignore-files in extension/package.json's web-ext
|
||||
# scripts: these files are not packaged into the XPI, so touching them
|
||||
# (e.g. Renovate bumping the web-ext devDep, or editing a spec)
|
||||
# changes nothing shipped and must not demand a version bump.
|
||||
# KEEP IN SYNC with --ignore-files — a file packaged into the XPI but
|
||||
# excluded here is exactly the silent-stale-ship this job exists to
|
||||
# prevent. test/version.spec.js pins the two lists' shared intent.
|
||||
CHANGED=$(git diff --name-only "$BASE" HEAD -- extension/ \
|
||||
':(exclude)extension/package.json' \
|
||||
':(exclude)extension/package-lock.json' \
|
||||
':(exclude)extension/README.md' \
|
||||
':(exclude)extension/.gitignore' \
|
||||
':(exclude)extension/vitest.config.js' \
|
||||
':(exclude)extension/test/**')
|
||||
if [ -z "$CHANGED" ]; then
|
||||
echo "No packaged extension files changed since $BASE — nothing to guard."
|
||||
echo "OK: extension version $PKG"
|
||||
exit 0
|
||||
fi
|
||||
echo "Packaged extension files changed since $BASE:"
|
||||
echo "$CHANGED" | sed 's/^/ /'
|
||||
PKG_OLD=$(git show "$BASE:extension/package.json" 2>/dev/null | grep -E '"version"' | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
|
||||
if [ -z "$PKG_OLD" ]; then
|
||||
echo "Could not read the base version — skipping the bump check."
|
||||
echo "OK: extension version $PKG"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$PKG_OLD" = "$PKG" ]; then
|
||||
echo "ERROR: packaged extension files changed but the version is still $PKG."
|
||||
echo "build.yml would find the existing ext-$PKG release, skip AMO signing,"
|
||||
echo "and bake the OLD signed XPI into :latest — a green build shipping stale code."
|
||||
echo "Bump the version in BOTH extension/package.json and extension/manifest.json."
|
||||
# ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
|
||||
#
|
||||
# Not the looser `^[0-9]+(\.[0-9]+)*$` this lane used to carry. That
|
||||
# one passes `2026.08.29.0201`, which AMO REJECTS — a segment must be
|
||||
# the single digit 0 or start 1-9 — and it also passes five segments,
|
||||
# where AMO allows four. Both would surface as a failed sign with the
|
||||
# version already burned: AMO 409s on re-signing, so a rejected value
|
||||
# cannot be reclaimed and cannot be reused. This lane is the cheap
|
||||
# place to find out. (#3138, milestone 318 step 8.)
|
||||
if ! echo "$VERSION" | grep -qE '^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$'; then
|
||||
echo "ERROR: derived version '$VERSION' is not a version AMO accepts."
|
||||
echo "AMO's grammar: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$"
|
||||
echo "Most likely cause: a zero-padded segment (08, 0201). The rest"
|
||||
echo "of the family pads; the extension must not — see packaging.sh."
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: extension version $PKG_OLD -> $PKG"
|
||||
|
||||
# ...and the shape this project actually derives. AMO would happily
|
||||
# take `1.0.3500147` too, so the grammar check alone would not notice
|
||||
# a regression to the pre-318 shape — which orders BELOW everything
|
||||
# signed since, and is unrecoverable once Firefox has the higher one.
|
||||
if ! echo "$VERSION" | grep -qE '^20[0-9][0-9]\.[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,4}$'; then
|
||||
echo "ERROR: derived version '$VERSION' is not YYYY.M.D.HHMM."
|
||||
echo "Rule 148's CalVer is what build.yml signs; the old"
|
||||
echo "1.0.<minutes> shape would order below every ext-2026.* release."
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: derived version $VERSION"
|
||||
|
||||
backend-lint-and-test:
|
||||
runs-on: python-ci
|
||||
@@ -168,6 +132,13 @@ jobs:
|
||||
SECRET_KEY: ci_unit_test_placeholder
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# Full history for tests/test_artifact_identity.py, which derives
|
||||
# each artifact's revision to check the identity scheme. On a
|
||||
# depth-1 clone that derivation either fails or returns the tip sha
|
||||
# — so the lane would go green while asserting nothing, which is
|
||||
# the one outcome worse than a red one.
|
||||
fetch-depth: 0
|
||||
|
||||
# Cache step removed 2026-05-26: act_runner's cache backend has been
|
||||
# broken on this homelab runner since 2026-05-15 (first as request-
|
||||
|
||||
@@ -10,15 +10,20 @@ on:
|
||||
paths:
|
||||
- 'extension/**'
|
||||
- '.forgejo/workflows/extension.yml'
|
||||
# test/version.spec.js asserts ci.yml's extension-version guard never
|
||||
# ignores a file web-ext actually packages, so a ci.yml-only edit can
|
||||
# break this suite and must trigger it.
|
||||
# test/version.spec.js asserts things ABOUT the other two workflows —
|
||||
# that neither inlines the packaged-file set, and that build.yml derives
|
||||
# the shipped version rather than reading it out of the repo. A
|
||||
# workflow-only edit can therefore break this suite, so it has to trigger
|
||||
# it. build.yml joined the list at milestone 271 step 5, when the spec
|
||||
# started asserting against it.
|
||||
- '.forgejo/workflows/ci.yml'
|
||||
- '.forgejo/workflows/build.yml'
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'extension/**'
|
||||
- '.forgejo/workflows/ci.yml'
|
||||
- '.forgejo/workflows/build.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
@@ -38,3 +43,45 @@ jobs:
|
||||
# package version-consistency checks. No browser, no network.
|
||||
- name: Unit tests
|
||||
run: cd extension && npm run test:unit
|
||||
|
||||
# Everything else about packaging is asserted against our own declaration
|
||||
# of what ships. This is the only check that asks web-ext what it ACTUALLY
|
||||
# put in the archive. Until now that was an unverified assumption about
|
||||
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
|
||||
# only because callers `set -f` first, so losing that quoting would
|
||||
# silently start shipping dev files with no other signal.
|
||||
- name: Verify XPI contents
|
||||
run: |
|
||||
set -eu
|
||||
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
|
||||
cd extension
|
||||
npm run build
|
||||
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
|
||||
echo "=== packaged entries in $ZIP ==="
|
||||
unzip -Z1 "$ZIP" | sort
|
||||
echo "=== end ==="
|
||||
ENTRIES=$(unzip -Z1 "$ZIP")
|
||||
fail=0
|
||||
# Must NOT ship: repo infrastructure with no business in a user's browser.
|
||||
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
|
||||
if echo "$ENTRIES" | grep -q "^$pat"; then
|
||||
echo "ERROR: '$pat' was packaged into the XPI but must not be"
|
||||
fail=1
|
||||
fi
|
||||
done
|
||||
# Must ship: if an exclusion pattern ever over-matches, the extension
|
||||
# breaks at runtime rather than at build time, so assert presence too.
|
||||
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
|
||||
if ! echo "$ENTRIES" | grep -q "^$req$"; then
|
||||
echo "ERROR: '$req' is missing from the XPI"
|
||||
fail=1
|
||||
fi
|
||||
done
|
||||
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
|
||||
if ! echo "$ENTRIES" | grep -q "^$dir"; then
|
||||
echo "ERROR: nothing from '$dir' was packaged"
|
||||
fail=1
|
||||
fi
|
||||
done
|
||||
[ "$fail" -eq 0 ] || exit 1
|
||||
echo "XPI contents verified."
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
name: Release
|
||||
|
||||
# A `v*` tag publishes a changelog. It does NOT build anything.
|
||||
#
|
||||
# Milestone 318 step 2 removed the tag trigger from build.yml: by the time
|
||||
# anyone tags a commit, `main` has already built and published it, and a
|
||||
# rebuild would re-push `:c-<sha>` — which rule 145 forbids even when the
|
||||
# source matches, since image configs carry timestamps and "same source" does
|
||||
# not mean "same manifest". That left the tag with no consequence at all.
|
||||
#
|
||||
# This is the consequence it has instead. Step 6 put the derived version in the
|
||||
# Settings footer, so an operator can say WHICH build they are running; this
|
||||
# says what is IN it that was not in the one they ran last month. Both halves
|
||||
# of one question (note #3127 §5).
|
||||
#
|
||||
# Nothing here runs on a schedule and nothing auto-tags on merge. Release tags
|
||||
# are bookmarks — cut one when you will want to point at that day by name,
|
||||
# otherwise don't (note #3127 §0). FC went twelve weeks between v26.06.04.0 and
|
||||
# the next one and nothing was wrong. A schedule would turn an optional
|
||||
# bookmark back into ceremony, which is the thing this milestone is removing.
|
||||
#
|
||||
# Cutting the tag is an explicit operator action under rule 2 ("`main` — never
|
||||
# without explicit request", which since 2026-08-28 covers PR, merge and tag
|
||||
# alike). This lane only decides what happens once they do.
|
||||
#
|
||||
# Requires repo secret RELEASE_TOKEN with the `write:release` scope — the same
|
||||
# PAT build.yml uses for the ext-<version> XPI asset cache.
|
||||
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
# So a release body can be regenerated after the fact — the publisher PATCHes
|
||||
# an existing release rather than falling through on a conflict, so re-running
|
||||
# this on a tag rewrites the body instead of silently keeping the first one
|
||||
# (note #3127 §6.7).
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: 'Tag to (re)publish notes for'
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
changelog:
|
||||
runs-on: python-ci
|
||||
container:
|
||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# Load-bearing twice over: the previous release is found by walking
|
||||
# ancestry back through the tag graph, and the cross-check against
|
||||
# the derived web version calls artifacts.sh, which reads commit
|
||||
# times. A shallow clone would find no previous tag and emit the
|
||||
# entire history as the changelog — plausible-looking and wrong.
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.event.inputs.tag || github.ref }}
|
||||
|
||||
# The `:c-<sha>` rollback refs are only real if `main` built this commit.
|
||||
# The script checks that against origin/main and downgrades the claim to
|
||||
# "unverified" when it cannot resolve one; fetching it here means that
|
||||
# downgrade stays an actual signal instead of firing on every release.
|
||||
- name: Make main's history resolvable
|
||||
run: git fetch --no-tags --quiet origin +main:refs/remotes/origin/main || true
|
||||
|
||||
# TAG goes through the environment, not through `${{ }}` inside the
|
||||
# run block. The value is operator-supplied, and an expression expanded
|
||||
# into a shell line is expanded BEFORE the shell sees it — there is no
|
||||
# quoting that makes that safe. On a tag push it is empty and the script
|
||||
# falls back to GITHUB_REF.
|
||||
- name: Publish the derived changelog
|
||||
env:
|
||||
RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||
TAG: ${{ github.event.inputs.tag }}
|
||||
run: |
|
||||
set -eu
|
||||
if [ -n "${TAG:-}" ]; then
|
||||
python3 scripts/release_notes.py "$TAG"
|
||||
else
|
||||
python3 scripts/release_notes.py
|
||||
fi
|
||||
@@ -0,0 +1,77 @@
|
||||
# Contributing
|
||||
|
||||
FabledCurator is developed by a single maintainer for their own use, and
|
||||
published because it may be useful to others. That shapes what contribution
|
||||
looks like here.
|
||||
|
||||
**Issues are welcome** — bug reports, and questions about running it, are
|
||||
genuinely useful and often the fastest way to find out that something is
|
||||
broken outside the one environment it was built in.
|
||||
|
||||
**Open an issue before writing a pull request.** Not as a formality: the
|
||||
project has opinions that are not obvious from the code, and it is unpleasant
|
||||
for everyone when a finished patch turns out to conflict with one. A short
|
||||
issue first costs you nothing and may save you an evening.
|
||||
|
||||
**Contributions are licensed under the AGPL-3.0**, like the rest of the
|
||||
project. By submitting one you agree it ships under that licence. There is no
|
||||
CLA and no copyright assignment.
|
||||
|
||||
## Running it for development
|
||||
|
||||
```bash
|
||||
docker compose up -d # UI on http://localhost:8080
|
||||
```
|
||||
|
||||
The dev override (`docker-compose.override.yml`) is auto-merged and builds the
|
||||
app images locally from source, so this needs no `.env` and no registry
|
||||
access. Postgres and Redis ports are exposed on the host.
|
||||
|
||||
## What CI checks
|
||||
|
||||
Every push runs these, and they are the definition of done for a change:
|
||||
|
||||
```bash
|
||||
ruff check backend/ tests/ alembic/ agent/ scripts/ # lint (and import order)
|
||||
pytest tests/ -m "not integration" # backend unit tests
|
||||
pytest tests/ -m integration # needs pgvector + redis
|
||||
cd frontend && npm run test:unit && npm run build # frontend
|
||||
```
|
||||
|
||||
The integration lane builds its schema by running the real migrations
|
||||
(`alembic upgrade head`), never from ORM metadata — so a migration that does
|
||||
not apply cleanly fails CI rather than being discovered later.
|
||||
|
||||
Note for the linter: ruff's isort runs with `order-by-type`, which sorts
|
||||
ALL-CAPS names ahead of CamelCase. `from sqlalchemy import JSON, DateTime, ...`
|
||||
is correct; putting `JSON` alphabetically between `Integer` and `String` is
|
||||
not. This catches people out.
|
||||
|
||||
## Database changes
|
||||
|
||||
The ORM models and the migration chain must agree. This is enforced, and it is
|
||||
enforced because they silently diverged for a long time and nobody noticed
|
||||
until they were compared: the models were missing indexes, defaults and
|
||||
uniqueness guarantees that only ever existed inside a migration, which made
|
||||
`alembic revision --autogenerate` actively unsafe to run.
|
||||
|
||||
So: if you change a model, write the migration; if you write a migration,
|
||||
change the model to match. Both, in the same commit.
|
||||
|
||||
Adding a value to a CHECK-constrained column means swapping the constraint in
|
||||
the same change — the constraint is not documentation, and a new value without
|
||||
it fails at insert time.
|
||||
|
||||
## Branch model
|
||||
|
||||
`dev` is where work happens. `main` is production and is only reached by a
|
||||
merge from `dev`, never pushed to directly. If you are sending a pull request,
|
||||
target `dev`.
|
||||
|
||||
## Style
|
||||
|
||||
Match the surrounding code. The one convention worth stating explicitly is
|
||||
that comments here explain *why*, especially where a choice looks wrong at a
|
||||
glance — a comment recording which migration a constraint came from, or why a
|
||||
default is a `text()` rather than a string, is the kind that has repeatedly
|
||||
turned out to be worth its space.
|
||||
+23
@@ -47,6 +47,29 @@ RUN chmod +x entrypoint.sh
|
||||
|
||||
COPY --from=frontend-builder /build/dist ./frontend/dist
|
||||
|
||||
# Which channel this image belongs to — `dev` or `main` (milestone 271 step 7).
|
||||
# build.yml passes it; /api/extension/manifest reports it beside the version so
|
||||
# an operator can tell which channel an install came from without the channel
|
||||
# ever touching the version string.
|
||||
#
|
||||
# Empty by default, deliberately: a locally-built image then reports NO channel
|
||||
# rather than claiming to be one, and the manifest omits the field entirely —
|
||||
# indistinguishable from an image built before the field existed, which is
|
||||
# exactly the shape every reader already has to handle.
|
||||
#
|
||||
# Declared LAST on purpose. An ARG/ENV invalidates every layer below it, and
|
||||
# these are the values that differ between builds of otherwise identical
|
||||
# source — put them any earlier and the two channels could never share a
|
||||
# cached pip install.
|
||||
#
|
||||
# FC_VERSION is what the instance reports about itself in the UI. Since
|
||||
# milestone 318 stopped publishing version image tags, that self-report is
|
||||
# the only answer to "which build is this?" — nothing else names it.
|
||||
ARG FC_CHANNEL=""
|
||||
ENV FC_CHANNEL=${FC_CHANNEL}
|
||||
ARG FC_VERSION=""
|
||||
ENV FC_VERSION=${FC_VERSION}
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
|
||||
@@ -0,0 +1,661 @@
|
||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
@@ -6,7 +6,50 @@ Combines what was [ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo)
|
||||
|
||||
## Status
|
||||
|
||||
Pre-v1. Not yet functional.
|
||||
In production. `main` is continuously deployed — every merge to `main` builds
|
||||
and publishes `:latest` images, so whatever is on `main` is what is running.
|
||||
Day-to-day work happens on `dev`, which publishes `:dev` images.
|
||||
|
||||
## Versions and tags
|
||||
|
||||
Three image tags exist, and no others:
|
||||
|
||||
| Tag | Branch | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `:latest` | `main` | Production. Moves on every merge. |
|
||||
| `:c-<sha>` | `main` | Immutable — the rollback unit, all three images together. |
|
||||
| `:dev` | `dev` | The rolling test channel. Moves on every push. |
|
||||
|
||||
There are deliberately **no version tags**. Nothing pins one, and a per-build
|
||||
name nobody reads is upkeep for a model FC does not run (family rule 145; the
|
||||
reasoning is note #3127 §5). Rolling back is `docker pull …:c-<sha>`.
|
||||
|
||||
Each artifact still has a version, derived rather than chosen: the commit time
|
||||
of the newest change to that artifact's *own* shipped files, as
|
||||
`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions —
|
||||
a push touching only `agent/` re-versions the agent and leaves web and ml
|
||||
alone, and CI skips the builds whose content did not move.
|
||||
|
||||
Because no registry name carries it, the running instance's own report is the
|
||||
only answer to "which build is this?". The foot of Settings shows
|
||||
`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two
|
||||
fields.
|
||||
|
||||
Release tags are optional bookmarks — FC went twelve weeks without one and
|
||||
nothing was wrong. Pushing `v<version>` publishes a Forgejo release listing the
|
||||
commits since the previous tag; it builds no image.
|
||||
|
||||
## What's in here
|
||||
|
||||
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
|
||||
|
||||
| Piece | Built from | Image | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
|
||||
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
|
||||
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. |
|
||||
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
|
||||
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -26,25 +69,69 @@ or a `.env` file (see `.env.example` for the variable names) and use:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml up -d
|
||||
# (skips the override so containers pull registry images)
|
||||
# (skips the dev override, so containers pull published :latest images)
|
||||
```
|
||||
|
||||
`-f` is doing real work there: it tells Compose to use *only* that file, which
|
||||
skips `docker-compose.override.yml` and its local builds. What you get is the
|
||||
`:latest` images — the stable channel, built from `main`. This is the install
|
||||
path, and it is the one to use if you are running FabledCurator rather than
|
||||
working on it.
|
||||
|
||||
`:dev` is the other channel: rebuilt from the `dev` branch several times a day,
|
||||
bleeding edge, no stability promise. Nothing in this repo points an installer at
|
||||
it, and nothing should.
|
||||
|
||||
The GPU agent is deployed separately, on the machine with the card —
|
||||
`agent/docker-compose.yml`, not this stack.
|
||||
|
||||
## Deployment posture
|
||||
|
||||
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
|
||||
|
||||
## CI / Forgejo setup
|
||||
|
||||
The repo's workflows expect:
|
||||
Four workflows: `ci.yml` (lint, extension-version check, backend unit tests,
|
||||
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
|
||||
content verification), `build.yml` (sign + publish), and `release.yml`, which
|
||||
runs only on a `v*` tag and publishes a changelog without building anything.
|
||||
|
||||
- **Runner label `python-ci`** — a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Both `ci.yml` and `build.yml` use this label. The runner image (`runner-base:python-ci`) is built from `CI-Runner/CI-python/` in the operator's workspace; `make push` from that directory builds and pushes a new image when toolchain pins change.
|
||||
- **Repo secret `RELEASE_TOKEN`** — a Forgejo PAT with the following scopes:
|
||||
**The toolchain each job runs in is its `container.image`, not its `runs-on`
|
||||
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
|
||||
then names the image it actually wants. `ci-requirements.md` is the current,
|
||||
authoritative list of images and per-job installs — read that rather than a
|
||||
copy here, so the two can't drift.
|
||||
|
||||
The repo expects one secret:
|
||||
|
||||
- **`RELEASE_TOKEN`** — a Forgejo PAT with:
|
||||
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
|
||||
- `write:release` — for future release-cutting workflows
|
||||
- `write:issue` — for future issue-management automation
|
||||
- `write:release` — for the `ext-<version>` releases that cache the signed XPI
|
||||
- `write:issue` — for issue-management automation
|
||||
|
||||
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
|
||||
|
||||
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`.
|
||||
It runs on **both** channels and is cached per version: because the version is
|
||||
derived from commit time, `dev` and `main` derive the same number for the same
|
||||
source, so `main` finds `dev`'s signature already cached and makes no second AMO
|
||||
call. That cache is why signing must be one-shot — AMO rejects a re-signed
|
||||
version.
|
||||
|
||||
## License
|
||||
|
||||
Personal project; use at your own discretion.
|
||||
**GNU Affero General Public License v3.0** — see [LICENSE](LICENSE).
|
||||
|
||||
You may run, study, modify and redistribute this software. The condition is
|
||||
reciprocity: if you distribute a modified version, or **run one as a network
|
||||
service that other people use**, you must offer those users the corresponding
|
||||
source under the same licence. That second clause (AGPL §13) is the reason this
|
||||
licence rather than the GPL — for a self-hosted web application, "distribution"
|
||||
otherwise never happens, and the obligation would never bite.
|
||||
|
||||
Running an unmodified copy for yourself, your household or your organisation
|
||||
carries no obligation at all. Neither does modifying it privately. The licence
|
||||
asks something of you only when you hand your modified version to others.
|
||||
|
||||
Contributions ship under the same licence — see [CONTRIBUTING](CONTRIBUTING.md).
|
||||
Security reports: [SECURITY.md](SECURITY.md).
|
||||
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
# Security Policy
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
**Please do not put vulnerability details in a public issue.**
|
||||
|
||||
This project has no private disclosure channel yet. Until it does, open an
|
||||
issue on the repository that says only that you have a security report — no
|
||||
reproduction steps, no affected endpoint, no payload — and a maintainer will
|
||||
reply with a private contact to send the details to.
|
||||
|
||||
That is a deliberately awkward first step, and it exists because the
|
||||
alternative is worse: an issue tracker is public the moment it is written to,
|
||||
and every self-hosted instance stays vulnerable until its operator has had a
|
||||
chance to update.
|
||||
|
||||
Please include, once you have a private channel:
|
||||
|
||||
- what an attacker can do, and what access they need to start
|
||||
- the version or commit you tested
|
||||
- reproduction steps
|
||||
|
||||
## Scope — what this software actually handles
|
||||
|
||||
FabledCurator is self-hosted and holds things worth stating plainly, because
|
||||
they shape what counts as a serious bug here:
|
||||
|
||||
- **Platform credentials.** The app captures and stores session cookies for
|
||||
third-party subscription sites (Patreon, SubscribeStar, Pixiv) so it can
|
||||
download on the operator's behalf. These are live credentials for accounts
|
||||
that usually carry a payment method. Anything that discloses them, decrypts
|
||||
them, or lets one user of a shared instance read another's is high severity.
|
||||
- **An extension API key.** The Firefox extension authenticates to the backend
|
||||
with a shared key. Anything that leaks it or lets it be bypassed is a way in.
|
||||
- **A multi-user sharing ACL.** Instances can be shared. A bug that lets one
|
||||
account see content another has not shared is an access-control failure, not
|
||||
a cosmetic one.
|
||||
- **Arbitrary media from the internet.** Downloaded files are decoded, hashed,
|
||||
thumbnailed and fed to ML models. Anything that turns a hostile file into
|
||||
code execution is in scope.
|
||||
|
||||
## Deployment posture — read this before reporting
|
||||
|
||||
FabledCurator is designed to run **inside a private network, over plain HTTP**.
|
||||
It does not terminate TLS, redirect to HTTPS, or set HSTS; if you want
|
||||
transport security, terminate it at your reverse proxy. This is a documented
|
||||
design decision, not an oversight.
|
||||
|
||||
Reports that reduce to "the application is served over HTTP" or "there is no
|
||||
HSTS header" describe that decision rather than a vulnerability. Reports that
|
||||
an authenticated operator can cause the software to do something destructive
|
||||
are usually also by design — the operator is the administrator of their own
|
||||
instance.
|
||||
|
||||
What remains in scope is everything that crosses a boundary the software is
|
||||
supposed to hold: between one user and another, between an unauthenticated
|
||||
visitor and any of it, and between untrusted downloaded content and the host.
|
||||
|
||||
## Supported versions
|
||||
|
||||
Fixes land on the `main` branch and reach the `:latest` image. There are no
|
||||
maintained release branches — the supported version is the current one, and
|
||||
the remedy for a security issue is to update.
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Reconcile the database with what the models have always claimed (#3275).
|
||||
|
||||
Milestone 328 discovered ~130 places where the ORM models and the deployed
|
||||
schema disagreed. Almost all of them were the MODEL being wrong — missing
|
||||
`server_default`s, indexes and CHECK constraints that only ever existed in a
|
||||
migration — and those are fixed in the model files with no DDL at all, because
|
||||
the database already had them.
|
||||
|
||||
This migration carries the remainder — the two places where DDL is actually
|
||||
needed, because the database is what is wrong.
|
||||
|
||||
`tag.fandom_id` is declared `index=True` on the model, but no migration ever
|
||||
created that index. Every autogenerate run since would have proposed adding
|
||||
it; nobody ran one, so the model and the database simply drifted apart and
|
||||
stayed that way.
|
||||
|
||||
Deliberately NOT in this migration: anything about `image_record.sha256`. An
|
||||
earlier draft of this file claimed sha256 was not unique in the database and
|
||||
that duplicate rows were therefore possible. That was WRONG, and it was wrong
|
||||
because it was read off `op.create_index("ix_image_record_sha256", ...)` at
|
||||
0001 line 151 without reading line 149 two lines above it:
|
||||
|
||||
sa.UniqueConstraint("sha256", name="uq_image_record_sha256"),
|
||||
|
||||
Uniqueness has been enforced since the initial schema. The database simply
|
||||
expresses it as a CONSTRAINT plus a separate non-unique lookup index, where
|
||||
the model expressed it as one `unique=True, index=True` column — the same
|
||||
guarantee built from different objects, which is why the two schemas did not
|
||||
line up. The model now declares the constraint and the plain index separately,
|
||||
so it describes what is actually there. No DDL is needed for it.
|
||||
|
||||
Also here: six CHECK constraints whose names carry their table prefix TWICE.
|
||||
|
||||
`base.py`'s naming convention is `ck_%(table_name)s_%(constraint_name)s`, and
|
||||
unlike the uq/fk/ix entries it applies even to a constraint that already has a
|
||||
name. Six migrations passed an already-prefixed name, so the convention
|
||||
prefixed it again:
|
||||
|
||||
ck_external_link_ck_external_link_host
|
||||
ck_external_link_ck_external_link_status
|
||||
ck_import_settings_ck_import_settings_singleton
|
||||
ck_ml_settings_ck_ml_settings_singleton
|
||||
ck_post_ck_post_translation_override
|
||||
ck_tag_ck_tag_fandom_requires_character
|
||||
|
||||
Nothing reads a CHECK constraint by name, so this has never done any harm —
|
||||
but it is exactly the development-era residue the collapsed baseline exists to
|
||||
leave behind, and a public schema should not ship it. The models now declare
|
||||
bare names, which the convention renders into the single-prefix form; this
|
||||
renames the deployed constraints to match.
|
||||
|
||||
RENAME CONSTRAINT is a catalog-only operation: no table scan, no rewrite, no
|
||||
validation of existing rows. It takes a brief ACCESS EXCLUSIVE lock and
|
||||
returns. That is why this is safe to do on `post` and `tag`, which are the two
|
||||
large tables in the schema.
|
||||
|
||||
Revision ID: 0088
|
||||
Revises: 0087
|
||||
Create Date: 2026-08-30
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0088"
|
||||
down_revision: Union[str, None] = "0087"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
# (table, doubled name, single-prefix name)
|
||||
#
|
||||
# Six, not the four a first read of the migrations turned up. The list that
|
||||
# settles it is the one extracted from the chain's pg_dump by matching
|
||||
# `ck_(\w+?)_ck_\1_` — reading the migrations by eye missed external_link
|
||||
# twice over, in the same way an earlier pass missed a UNIQUE constraint two
|
||||
# lines above the index it was looking at (see the sha256 note above).
|
||||
DOUBLED_CHECKS = (
|
||||
("external_link", "ck_external_link_ck_external_link_host",
|
||||
"ck_external_link_host"),
|
||||
("external_link", "ck_external_link_ck_external_link_status",
|
||||
"ck_external_link_status"),
|
||||
("import_settings", "ck_import_settings_ck_import_settings_singleton",
|
||||
"ck_import_settings_singleton"),
|
||||
("ml_settings", "ck_ml_settings_ck_ml_settings_singleton",
|
||||
"ck_ml_settings_singleton"),
|
||||
("post", "ck_post_ck_post_translation_override",
|
||||
"ck_post_translation_override"),
|
||||
("tag", "ck_tag_ck_tag_fandom_requires_character",
|
||||
"ck_tag_fandom_requires_character"),
|
||||
)
|
||||
|
||||
|
||||
def _rename_check(table: str, old: str, new: str) -> None:
|
||||
# Guarded on pg_constraint rather than run bare: a database built from the
|
||||
# models (a fresh install, or the CI integration schema) already has the
|
||||
# single-prefix name, and this migration must be a no-op there rather than
|
||||
# an error. Same reasoning as the CREATE INDEX IF NOT EXISTS below.
|
||||
op.execute(
|
||||
f"""
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conname = '{old}' AND conrelid = '{table}'::regclass
|
||||
) THEN
|
||||
ALTER TABLE {table} RENAME CONSTRAINT {old} TO {new};
|
||||
END IF;
|
||||
END $$;
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# IF NOT EXISTS because the index is what the model already asks for: any
|
||||
# database built from metadata rather than from this chain will have it,
|
||||
# and this migration must be a no-op there rather than an error.
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_tag_fandom_id ON tag (fandom_id)")
|
||||
|
||||
for table, old, new in DOUBLED_CHECKS:
|
||||
_rename_check(table, old, new)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for table, old, new in DOUBLED_CHECKS:
|
||||
_rename_check(table, new, old)
|
||||
|
||||
op.execute("DROP INDEX IF EXISTS ix_tag_fandom_id")
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Index the seven unindexed FKs; drop the seven redundant indexes (#3300, #3301).
|
||||
|
||||
Found by a structural sweep of the deployed schema done AFTER 0088 brought the
|
||||
models and the migration chain into exact agreement. That agreement is what
|
||||
0088 achieved, and it is worth being precise about what it does NOT prove: a
|
||||
models-vs-chain diff shows the two describe the same schema. It says nothing
|
||||
about whether that schema is right. Everything here was wrong in BOTH, which is
|
||||
exactly the class of problem the reconciliation could not see.
|
||||
|
||||
## Added: seven FK indexes
|
||||
|
||||
`image_tag.tag_id` is the one that matters. The table's only index is
|
||||
PRIMARY KEY (image_record_id, tag_id), which leads with the wrong column for
|
||||
the two hottest things done with it:
|
||||
|
||||
* the gallery's tag filter — services/tag_query.py builds
|
||||
`image_tag.c.tag_id == tid` (and `.in_(tids)`) on every tag-scoped browse;
|
||||
* ON DELETE CASCADE from `tag` — deleting or merging a tag makes Postgres
|
||||
find that tag's rows before it can remove them.
|
||||
|
||||
Both had to scan the largest table in the schema. The other six are the same
|
||||
shape on much smaller tables; `presentation_review.tag_id` is the notable one,
|
||||
since it also CASCADEs.
|
||||
|
||||
## Dropped: seven redundant indexes
|
||||
|
||||
`ix_image_record_sha256` was an exact duplicate. A UNIQUE constraint builds its
|
||||
own index, so `uq_image_record_sha256` already covered the column and
|
||||
`image_record` carried two btrees on `sha256` — on the highest-insert-rate
|
||||
table in the system.
|
||||
|
||||
The other six are single-column indexes that a later composite superseded
|
||||
without the narrow one being retired. A btree on (a, b) already serves lookups
|
||||
on `a`, so each was pure write amplification. `task_run` and `backup_run` are
|
||||
append-heavy operational logs, which is where that cost lands hardest.
|
||||
|
||||
Note for anyone reading 0088 next to this: 0088 deliberately taught the models
|
||||
to declare BOTH sha256 indexes, so they would describe reality. That was right.
|
||||
This migration changes the reality instead, and the models change with it.
|
||||
|
||||
## CONCURRENTLY, and why this migration has no transaction
|
||||
|
||||
`CREATE INDEX` takes an ACCESS EXCLUSIVE lock for the whole build, which on
|
||||
`image_tag` means stalling every write for as long as it takes. CONCURRENTLY
|
||||
builds without blocking writers, at the cost of two table passes and an
|
||||
inability to run inside a transaction — hence `autocommit_block()`.
|
||||
|
||||
The consequence to know about: this migration is NOT atomic. If it fails
|
||||
partway, the work already done stays done. Every statement is therefore written
|
||||
IF NOT EXISTS / IF EXISTS so that re-running it after a failure is safe rather
|
||||
than an error.
|
||||
|
||||
A failed CONCURRENTLY build also leaves an INVALID index behind — it is not
|
||||
used by the planner and not repaired automatically. Find them with:
|
||||
|
||||
SELECT c.relname FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
WHERE NOT i.indisvalid;
|
||||
|
||||
Drop what that returns and re-run; nothing else is needed.
|
||||
|
||||
Revision ID: 0089
|
||||
Revises: 0088
|
||||
Create Date: 2026-08-31
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0089"
|
||||
down_revision: Union[str, None] = "0088"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
# (index name, table, column) — names match what the models render under
|
||||
# base.py's naming convention, so autogenerate stays quiet after this.
|
||||
MISSING_FK_INDEXES = (
|
||||
("ix_image_tag_tag_id", "image_tag", "tag_id"),
|
||||
("ix_presentation_review_tag_id", "presentation_review", "tag_id"),
|
||||
("ix_presentation_review_conflict_tag_id", "presentation_review", "conflict_tag_id"),
|
||||
("ix_import_task_result_image_id", "import_task", "result_image_id"),
|
||||
("ix_external_link_attachment_id", "external_link", "attachment_id"),
|
||||
("ix_character_prototype_region_id", "character_prototype", "region_id"),
|
||||
("ix_backup_run_restored_from_id", "backup_run", "restored_from_id"),
|
||||
)
|
||||
|
||||
# (index name, table, column) — redundant; the second element of each pair in
|
||||
# the docstring is what still covers the column after the drop.
|
||||
REDUNDANT_INDEXES = (
|
||||
("ix_image_record_sha256", "image_record", "sha256"),
|
||||
("ix_backup_run_kind", "backup_run", "kind"),
|
||||
("ix_backup_run_status", "backup_run", "status"),
|
||||
("ix_task_run_queue", "task_run", "queue"),
|
||||
("ix_task_run_status", "task_run", "status"),
|
||||
("ix_task_run_task_name", "task_run", "task_name"),
|
||||
("ix_external_link_post_id", "external_link", "post_id"),
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, table, column in MISSING_FK_INDEXES:
|
||||
op.execute(
|
||||
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} "
|
||||
f"ON {table} ({column})"
|
||||
)
|
||||
for name, _table, _column in REDUNDANT_INDEXES:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.get_context().autocommit_block():
|
||||
for name, table, column in REDUNDANT_INDEXES:
|
||||
op.execute(
|
||||
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} "
|
||||
f"ON {table} ({column})"
|
||||
)
|
||||
for name, _table, _column in MISSING_FK_INDEXES:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
|
||||
@@ -459,6 +459,22 @@ async def trigger_prune_missing_files():
|
||||
return _queued(async_result)
|
||||
|
||||
|
||||
@admin_bp.route("/maintenance/reclaim-attachments", methods=["POST"])
|
||||
async def trigger_reclaim_attachments():
|
||||
"""Reclaim orphaned attachments (#3068). Body {"dry_run": bool}: dry_run
|
||||
(the DEFAULT here) projects the orphan rows and unreferenced store blobs
|
||||
without touching either; dry_run=false deletes the rows then unlinks every
|
||||
blob no surviving row references. Maintenance queue; operator-triggered
|
||||
only — never an unattended sweep, since the apply unlinks files. Returns the
|
||||
Celery task id — poll /maintenance/task-result/<id> for the summary."""
|
||||
from ..tasks.admin import reclaim_orphaned_attachments_task
|
||||
|
||||
body = await request.get_json(silent=True) or {}
|
||||
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
|
||||
async_result = reclaim_orphaned_attachments_task.delay(dry_run=dry_run)
|
||||
return _queued(async_result)
|
||||
|
||||
|
||||
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
||||
async def trigger_dedup_videos():
|
||||
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
||||
|
||||
@@ -6,12 +6,14 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import hmac
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from quart import Blueprint, jsonify, request
|
||||
from sqlalchemy import select
|
||||
|
||||
from ..build_info import FC_CHANNEL as _FC_CHANNEL
|
||||
from ..extensions import get_session
|
||||
from ..models import AppSetting
|
||||
from ..services.extension_service import (
|
||||
@@ -30,6 +32,15 @@ XPI_DIR = Path("/app/frontend/dist/extension")
|
||||
|
||||
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
|
||||
|
||||
# Which channel this image belongs to — "dev" or "main" — baked in at build
|
||||
# time (milestone 271 step 7). Read from build_info rather than the environment
|
||||
# a second time: /api/health reports the same value, and two independent
|
||||
# `os.environ.get` calls are two things that can drift.
|
||||
#
|
||||
# Still bound as a module-level name here, so tests monkeypatch
|
||||
# `extension.FC_CHANNEL` exactly as they did before, same as XPI_DIR above.
|
||||
FC_CHANNEL = _FC_CHANNEL
|
||||
|
||||
|
||||
async def _ext_key_required(session) -> bool:
|
||||
"""Unlike /api/credentials (which accepts the browser path with no
|
||||
@@ -41,7 +52,15 @@ async def _ext_key_required(session) -> bool:
|
||||
stored = (await session.execute(
|
||||
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
|
||||
)).scalar_one_or_none()
|
||||
return stored is not None and supplied == stored
|
||||
if stored is None:
|
||||
return False
|
||||
# compare_digest, not `==`: the stored key is a shared secret, and a
|
||||
# short-circuiting compare leaks its prefix through timing. Costs nothing
|
||||
# here — it is not that this route is exposed (#3072). Compared as BYTES:
|
||||
# compare_digest's str form rejects non-ASCII with TypeError, and this
|
||||
# header is attacker-supplied, so a str compare would turn a junk key into
|
||||
# a 500 instead of a 403.
|
||||
return hmac.compare_digest(supplied.encode("utf-8"), stored.encode("utf-8"))
|
||||
|
||||
|
||||
def _extract_version(xpi_name: str) -> str:
|
||||
@@ -124,13 +143,30 @@ def _read_manifest_sync() -> dict | None:
|
||||
return None
|
||||
versioned.sort(key=lambda p: p.stat().st_mtime)
|
||||
latest = versioned[-1]
|
||||
return {
|
||||
info = {
|
||||
"installed": True,
|
||||
"version": _extract_version(latest.name),
|
||||
"xpi_url": f"/extension/{latest.name}",
|
||||
"latest_url": "/extension/fabledcurator-latest.xpi",
|
||||
"sha256": _sha256(latest),
|
||||
}
|
||||
# The channel goes BESIDE the version, never inside it. A `-dev` suffix is
|
||||
# what silently disabled the dev channel in the sibling project this design
|
||||
# comes from: the comparator returned nothing for a non-integer segment, so
|
||||
# every dev version compared equal and "no update available" became
|
||||
# indistinguishable from "I cannot read this version".
|
||||
#
|
||||
# Omitted rather than defaulted when unset. Absence already has a meaning
|
||||
# every reader must handle — an image built before this field existed says
|
||||
# exactly the same thing by not having the key — so a blank channel reuses
|
||||
# that path instead of inventing a second "unknown" spelling.
|
||||
#
|
||||
# Reported verbatim, not validated against {"dev", "main"}: if an image
|
||||
# declares something else, showing what it actually claims is more useful
|
||||
# to whoever is debugging it than dropping the value on the floor.
|
||||
if FC_CHANNEL:
|
||||
info["channel"] = FC_CHANNEL
|
||||
return info
|
||||
|
||||
|
||||
@extension_bp.route("/manifest", methods=["GET"])
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
"""Health endpoint — no DB or Redis touch; just liveness."""
|
||||
"""Health endpoint — no DB or Redis touch; liveness, plus the build's identity.
|
||||
|
||||
The identity rides here rather than on a route of its own because it answers
|
||||
at the same cost: two module constants, no I/O, nothing that can be slow or
|
||||
fail. It is also already fetched app-wide — TopNav calls `refreshHealth` on
|
||||
mount — so a separate endpoint would mean a second request for two strings.
|
||||
|
||||
Both fields are OMITTED when unset rather than sent empty. See build_info.
|
||||
"""
|
||||
|
||||
from ..build_info import FC_CHANNEL, FC_VERSION
|
||||
|
||||
|
||||
async def get_health():
|
||||
return {"status": "ok"}, 200
|
||||
body = {"status": "ok"}
|
||||
if FC_VERSION:
|
||||
body["version"] = FC_VERSION
|
||||
if FC_CHANNEL:
|
||||
body["channel"] = FC_CHANNEL
|
||||
return body, 200
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""What this build IS — stamped at image build time, not configurable.
|
||||
|
||||
Deliberately separate from `config.py`. Those are operator settings, read from
|
||||
the environment and meant to be changed. These describe the artifact itself and
|
||||
are baked in by CI (the `FC_VERSION` / `FC_CHANNEL` build args); an operator
|
||||
setting them by hand is not a supported thing to do, it is just how a value
|
||||
gets from the build into the running process.
|
||||
|
||||
**Absent rather than empty when unknown.** A locally-built image has no version,
|
||||
and neither did any image predating the field — one spelling of "cannot say",
|
||||
which every reader already has to handle, instead of a second one to
|
||||
special-case (note #3127 §7).
|
||||
|
||||
**Why this matters more than it used to.** Milestone 318 stopped publishing
|
||||
version image tags, so a running instance's self-report is now the *only*
|
||||
answer to "which build is this?" — there is no registry name left to check it
|
||||
against. A wrong value here has nothing to contradict it. That is why the UI
|
||||
renders `unknown` rather than a blank or a plausible default: an empty footer
|
||||
reads as "no version", which is a different and false claim.
|
||||
|
||||
The channel lives BESIDE the version and is never folded into it (rule 149).
|
||||
A `-dev` suffix would be parsed by the extension's comparator as a segment
|
||||
worth 0, making every dev build compare equal to every other — issue #2993's
|
||||
exact failure.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
FC_VERSION = os.environ.get("FC_VERSION", "").strip()
|
||||
FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
|
||||
@@ -27,7 +27,7 @@ from .patreon_seen_media import PatreonSeenMedia
|
||||
from .pixiv_failed_media import PixivFailedMedia
|
||||
from .pixiv_seen_media import PixivSeenMedia
|
||||
from .post import Post
|
||||
from .post_attachment import PostAttachment
|
||||
from .post_attachment import PostAttachment, attachment_download_url
|
||||
from .presentation_review import PresentationReview
|
||||
from .series_chapter import SeriesChapter
|
||||
from .series_page import SeriesPage
|
||||
@@ -58,6 +58,7 @@ __all__ = [
|
||||
"SubscribeStarSeenMedia",
|
||||
"Post",
|
||||
"PostAttachment",
|
||||
"attachment_download_url",
|
||||
"PresentationReview",
|
||||
"SeriesChapter",
|
||||
"SeriesPage",
|
||||
|
||||
@@ -27,10 +27,10 @@ class Artist(Base):
|
||||
notes: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
# True once a Source is attached; flips false if all sources removed.
|
||||
is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
# Per-artist scheduling overrides; null means "use global default".
|
||||
auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
check_interval_seconds: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
|
||||
@@ -20,7 +20,7 @@ feedback_check_existing_enums):
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Index, Integer, String, Text, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -29,10 +29,21 @@ from .base import Base
|
||||
class BackupRun(Base):
|
||||
__tablename__ = "backup_run"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0017: reporting indexes, never declared on the model (#3275).
|
||||
Index("ix_backup_run_kind_started", "kind", text("started_at DESC")),
|
||||
Index("ix_backup_run_status_finished", "status", text("finished_at DESC")),
|
||||
Index("ix_backup_run_tag_partial", "tag", postgresql_where=text("tag IS NOT NULL")),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True)
|
||||
# No index=True: ix_backup_run_kind_started (above) already leads with
|
||||
# `kind`, so a single-column index on it was pure write cost (#3301).
|
||||
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="pending", index=True,
|
||||
# No index=True — ix_backup_run_status_finished leads with `status`.
|
||||
String(16), nullable=False, default="pending",
|
||||
server_default="pending",
|
||||
)
|
||||
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
|
||||
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
@@ -49,7 +60,9 @@ class BackupRun(Base):
|
||||
manifest: Mapped[dict] = mapped_column(
|
||||
JSON, nullable=False, default=dict, server_default="{}",
|
||||
)
|
||||
# Self-referential FK, unindexed until 0089 (#3300): SET NULL has to find
|
||||
# the rows pointing at a deleted run before it can null them.
|
||||
restored_from_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("backup_run.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
nullable=True, index=True,
|
||||
)
|
||||
|
||||
@@ -40,8 +40,10 @@ class CharacterPrototype(Base):
|
||||
)
|
||||
# Provenance: the region this vector was copied from. SET NULL so pruning a
|
||||
# region doesn't delete the prototype mid-cycle (the next refresh reconciles).
|
||||
# index=True added in 0089 — the FK was unindexed (#3300).
|
||||
region_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("image_region.id", ondelete="SET NULL"), nullable=True
|
||||
ForeignKey("image_region.id", ondelete="SET NULL"), nullable=True,
|
||||
index=True,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -25,8 +25,8 @@ class DownloadEvent(Base):
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
|
||||
files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default="0")
|
||||
files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
metadata_: Mapped[dict] = mapped_column(
|
||||
"metadata", JSONB, nullable=False, default=dict,
|
||||
|
||||
@@ -16,6 +16,7 @@ doesn't delete the link record).
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import (
|
||||
CheckConstraint,
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
@@ -38,15 +39,33 @@ STATUSES = ("pending", "downloading", "downloaded", "failed", "skipped", "dead")
|
||||
class ExternalLink(Base):
|
||||
__tablename__ = "external_link"
|
||||
__table_args__ = (
|
||||
# alembic 0028 enum CHECKs. Rule 36 territory: a new host or status value
|
||||
# needs its constraint swapped in the same migration (#3275).
|
||||
CheckConstraint(
|
||||
"host IN ('mega', 'gdrive', 'mediafire', 'dropbox', 'pixeldrain')",
|
||||
# Bare name: Base.metadata's naming convention prepends
|
||||
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
|
||||
# alembic 0088, which renames the four constraints that shipped
|
||||
# that way (#3275).
|
||||
name="host",
|
||||
),
|
||||
CheckConstraint(
|
||||
"status IN ('pending', 'downloading', 'downloaded', 'failed', 'skipped', 'dead')",
|
||||
name="status",
|
||||
),
|
||||
# One row per (post, url). The full url (incl. #fragment) is the identity
|
||||
# — the same file linked twice in a post collapses to one row.
|
||||
Index("uq_external_link_post_url", "post_id", "url", unique=True),
|
||||
Index("ix_external_link_status", "status"),
|
||||
# Unindexed FK (#3300).
|
||||
Index("ix_external_link_attachment_id", "attachment_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# No index=True: uq_external_link_post_url (post_id, url) already leads
|
||||
# with post_id (#3301).
|
||||
post_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
artist_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
|
||||
@@ -50,7 +50,8 @@ class GpuJob(Base):
|
||||
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
|
||||
task: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="pending", index=True
|
||||
String(16), nullable=False, default="pending", index=True,
|
||||
server_default="pending",
|
||||
)
|
||||
# pending | leased | done | error
|
||||
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
@@ -60,7 +61,7 @@ class GpuJob(Base):
|
||||
lease_expires_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# Triage verdict for an ERRORED job (#125): NULL = not yet probed;
|
||||
# 'defect' = the integrity probe says the FILE itself is bad (surfaced for
|
||||
|
||||
@@ -24,10 +24,11 @@ class HeadAutoApplyRun(Base):
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing
|
||||
# (preview/apply parity, rule 93).
|
||||
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="running", index=True
|
||||
String(16), nullable=False, default="running", index=True,
|
||||
server_default="running",
|
||||
)
|
||||
# running | ready | error
|
||||
started_at: Mapped[datetime] = mapped_column(
|
||||
|
||||
@@ -24,9 +24,9 @@ class HeadMetric(Base):
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
|
||||
)
|
||||
# An auto-applied (source='head_auto') tag the operator later REMOVED.
|
||||
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
# A tag with a head that the operator added by HAND (the head missed it).
|
||||
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
|
||||
@@ -19,8 +19,14 @@ class HeadMetricsSnapshot(Base):
|
||||
__tablename__ = "head_metrics_snapshot"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
tag_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), index=True
|
||||
# Nullable, matching alembic 0060, which declared this column without
|
||||
# `nullable=False`. The model had it as `Mapped[int]` — NOT NULL — which
|
||||
# was simply never true of the database (#3275). Left nullable rather than
|
||||
# tightened: a snapshot of a tag that is later hard-deleted is a row worth
|
||||
# keeping, and the FK is ON DELETE CASCADE, so tightening it would only
|
||||
# change behaviour, not correct a bug.
|
||||
tag_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=True, index=True
|
||||
)
|
||||
# Denormalized so a snapshot stays readable even if the tag is later renamed.
|
||||
name: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
@@ -28,9 +34,9 @@ class HeadMetricsSnapshot(Base):
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
|
||||
)
|
||||
# Current count of source='head_auto' applications still standing.
|
||||
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
# The head's measured quality at snapshot time (null if no head exists).
|
||||
ap: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
|
||||
@@ -24,7 +24,8 @@ class HeadTrainingRun(Base):
|
||||
# Training parameters: {min_positives, neg_ratio, precision_target, ...}.
|
||||
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="running", index=True
|
||||
String(16), nullable=False, default="running", index=True,
|
||||
server_default="running",
|
||||
)
|
||||
# running | ready | error
|
||||
started_at: Mapped[datetime] = mapped_column(
|
||||
|
||||
@@ -47,8 +47,15 @@ class ImageProvenance(Base):
|
||||
# attachment on the post. NULL for loose downloads and pre-backfill rows.
|
||||
# SET NULL so deleting the archive attachment never destroys the (image,
|
||||
# post) edge — it just forgets which archive it came from.
|
||||
# FK named explicitly: the convention renders this
|
||||
# `fk_image_provenance_from_attachment_id_post_attachment`, but alembic
|
||||
# 0055 created it as `fk_image_provenance_from_attachment` (#3275).
|
||||
from_attachment_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("post_attachment.id", ondelete="SET NULL"),
|
||||
ForeignKey(
|
||||
"post_attachment.id",
|
||||
ondelete="SET NULL",
|
||||
name="fk_image_provenance_from_attachment",
|
||||
),
|
||||
nullable=True, index=True,
|
||||
)
|
||||
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
|
||||
@@ -14,10 +14,13 @@ from sqlalchemy import (
|
||||
Enum,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -29,11 +32,38 @@ ORIGIN_CHOICES = ("downloaded", "imported_filesystem", "uploaded")
|
||||
class ImageRecord(Base):
|
||||
__tablename__ = "image_record"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0001. The database enforces sha256 uniqueness with a
|
||||
# CONSTRAINT and carries a SEPARATE non-unique btree index; the model
|
||||
# said `unique=True, index=True`, which collapses both into a single
|
||||
# UNIQUE index under a different name. Same guarantee either way, but
|
||||
# not the same objects, so autogenerate saw a drop and an add (#3275).
|
||||
UniqueConstraint("sha256", name="uq_image_record_sha256"),
|
||||
# alembic 0036, and the last thing in this schema that lived only in a
|
||||
# migration. SQLAlchemy CAN express an hnsw index with an operator
|
||||
# class, so there is no reason for it to be invisible to the models —
|
||||
# and its absence was the quietest failure of the lot: everything
|
||||
# works, similarity search just silently stops using an index.
|
||||
Index(
|
||||
"ix_image_record_siglip_hnsw",
|
||||
"siglip_embedding",
|
||||
postgresql_using="hnsw",
|
||||
postgresql_ops={"siglip_embedding": "vector_cosine_ops"},
|
||||
),
|
||||
# alembic 0035/0071: the date-ordered browse indexes (#3275).
|
||||
Index("ix_image_record_effective_date", text("effective_date DESC"), text("id DESC")),
|
||||
Index("ix_image_record_earliest_post_date", text("earliest_post_date DESC"), text("id DESC")),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
|
||||
# On-disk identity
|
||||
path: Mapped[str] = mapped_column(Text, nullable=False, unique=True)
|
||||
sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True, index=True)
|
||||
# Neither unique= nor index=: uq_image_record_sha256 in __table_args__
|
||||
# above creates its own index, and the separate ix_image_record_sha256
|
||||
# that 0001 also built was an exact duplicate of it — dropped in 0089
|
||||
# (#3301). Lookups by sha256 use the constraint's index.
|
||||
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
|
||||
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||||
mime: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
@@ -47,7 +77,8 @@ class ImageRecord(Base):
|
||||
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
|
||||
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
|
||||
integrity_status: Mapped[str] = mapped_column(
|
||||
String(24), nullable=False, default="unknown", index=True
|
||||
String(24), nullable=False, default="unknown", index=True,
|
||||
server_default="unknown",
|
||||
)
|
||||
|
||||
# Thumbnail (populated by FC-2)
|
||||
@@ -72,8 +103,15 @@ class ImageRecord(Base):
|
||||
)
|
||||
# FC-2d-vii-c: canonical per-image artist (the single source of truth
|
||||
# for attribution; provenance posts remain lineage detail).
|
||||
# FK named explicitly: the naming convention renders this
|
||||
# `fk_image_record_artist_id_artist`, but alembic 0008 created it as
|
||||
# `fk_image_record_artist_id` (#3275).
|
||||
artist_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
ForeignKey(
|
||||
"artist.id", ondelete="SET NULL", name="fk_image_record_artist_id"
|
||||
),
|
||||
nullable=True,
|
||||
index=True,
|
||||
)
|
||||
|
||||
# ML fields (populated by the ml-worker / GPU agent). 1152 = SigLIP-so400m
|
||||
|
||||
@@ -21,17 +21,17 @@ class ImportBatch(Base):
|
||||
)
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
# Deep-scan only: count of already-imported files whose sidecar metadata
|
||||
# got re-applied this run (post/source/provenance upsert). Stays 0 on
|
||||
# quick-scan batches. See `Importer.import_one(deep_scan=True)`.
|
||||
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True)
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True, server_default="running")
|
||||
# running | complete | cancelled
|
||||
|
||||
tasks = relationship("ImportTask", back_populates="batch", cascade="all, delete-orphan")
|
||||
|
||||
@@ -4,7 +4,15 @@ Enforced as a single row via a CHECK (id = 1) constraint. The application
|
||||
always SELECTs id=1 and never inserts/deletes after the initial migration.
|
||||
"""
|
||||
|
||||
from sqlalchemy import Boolean, CheckConstraint, Float, Integer, Text, select
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
CheckConstraint,
|
||||
Float,
|
||||
Integer,
|
||||
Text,
|
||||
select,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -14,63 +22,79 @@ class ImportSettings(Base):
|
||||
__tablename__ = "import_settings"
|
||||
# Bare constraint name — Base.metadata's naming convention applies the
|
||||
# ck_<table>_<name> prefix, producing the final ck_import_settings_singleton.
|
||||
# Bare name — Base.metadata's naming convention prepends ck_<table>_,
|
||||
# producing ck_import_settings_singleton. The chain shipped the DOUBLED
|
||||
# ck_import_settings_ck_import_settings_singleton, because the migration
|
||||
# pre-prefixed the name and the convention prefixed it again; alembic
|
||||
# 0088 renames it to what this line has always produced (#3275).
|
||||
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import")
|
||||
import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import", server_default="/import")
|
||||
|
||||
min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
|
||||
skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9)
|
||||
skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9, server_default="0.9")
|
||||
|
||||
skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95)
|
||||
single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30)
|
||||
skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
|
||||
single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
|
||||
|
||||
phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10)
|
||||
phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
|
||||
|
||||
# FC-3c downloader knobs
|
||||
download_rate_limit_seconds: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=3.0
|
||||
Float, nullable=False, default=3.0,
|
||||
server_default="3",
|
||||
)
|
||||
download_validate_files: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
|
||||
# FC-3d scheduling knobs
|
||||
download_schedule_default_seconds: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=28800
|
||||
Integer, nullable=False, default=28800,
|
||||
server_default="28800",
|
||||
)
|
||||
download_event_retention_days: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=90
|
||||
Integer, nullable=False, default=90,
|
||||
server_default="90",
|
||||
)
|
||||
download_failure_warning_threshold: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=5
|
||||
Integer, nullable=False, default=5,
|
||||
server_default="5",
|
||||
)
|
||||
|
||||
# FC-3h backup knobs.
|
||||
backup_db_nightly_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=False,
|
||||
server_default="false",
|
||||
)
|
||||
backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=3,
|
||||
server_default="3",
|
||||
)
|
||||
backup_db_keep_last_n: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=14,
|
||||
server_default="14",
|
||||
)
|
||||
backup_images_keep_last_n: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=3,
|
||||
server_default="3",
|
||||
)
|
||||
|
||||
# FC-6.3 series continuation matcher. enabled gates the rescan; threshold is
|
||||
# the weighted-score cut-off (0..1) above which a pending suggestion is made.
|
||||
series_suggest_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
series_suggest_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.5,
|
||||
server_default="0.5",
|
||||
)
|
||||
|
||||
# #830 off-platform file-host downloads — per-host enable lever (default on,
|
||||
@@ -113,7 +137,9 @@ class ImportSettings(Base):
|
||||
# English (e.g. "… WIP Part 1") as a European language at ~0.86. CJK stays
|
||||
# trusted regardless (script-detected). Per-post overrides handle the misses.
|
||||
translation_min_confidence: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.9, server_default="0.9",
|
||||
# text() because alembic 0084 used sa.text(); see ml_settings for why
|
||||
# the form matters and why it is per-column (#3275).
|
||||
Float, nullable=False, default=0.9, server_default=text("0.9"),
|
||||
)
|
||||
|
||||
# Title-based WIP auto-tagging (task #1458). When a freshly-imported post's
|
||||
|
||||
@@ -13,10 +13,12 @@ from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
@@ -26,6 +28,12 @@ from .base import Base
|
||||
class ImportTask(Base):
|
||||
__tablename__ = "import_task"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_import_task_created_at_desc", text("created_at DESC")),
|
||||
# Unindexed FK (#3300).
|
||||
Index("ix_import_task_result_image_id", "result_image_id"),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
batch_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
@@ -33,14 +41,14 @@ class ImportTask(Base):
|
||||
|
||||
source_path: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True)
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True, server_default="pending")
|
||||
|
||||
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks
|
||||
# how many times the stuck-task sweep has re-queued this row; after
|
||||
# the cap it's failed with a diagnostic instead of looping. refetched
|
||||
# bounds the one-shot re-download remediation to a single attempt.
|
||||
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
result_image_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
|
||||
|
||||
@@ -8,7 +8,7 @@ reads it and routes through cleanup_service.delete_images.
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import DateTime, Integer, String, Text, func
|
||||
from sqlalchemy import DateTime, Integer, String, Text, func, text
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -23,6 +23,7 @@ class LibraryAuditRun(Base):
|
||||
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="running", index=True,
|
||||
server_default="running",
|
||||
)
|
||||
# running | ready | applied | cancelled | error
|
||||
started_at: Mapped[datetime] = mapped_column(
|
||||
@@ -31,14 +32,16 @@ class LibraryAuditRun(Base):
|
||||
finished_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True,
|
||||
)
|
||||
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
|
||||
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
matched_ids: Mapped[list[int]] = mapped_column(
|
||||
JSONB, nullable=False, default=list, server_default=text("'[]'::jsonb")
|
||||
)
|
||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes
|
||||
# from, and the last time a chunk made progress (so the recovery sweep can
|
||||
# tell a progressing multi-chunk audit from a stuck one).
|
||||
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
last_progress_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True,
|
||||
)
|
||||
|
||||
@@ -11,6 +11,7 @@ from sqlalchemy import (
|
||||
String,
|
||||
func,
|
||||
select,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -20,7 +21,10 @@ from .base import Base
|
||||
class MLSettings(Base):
|
||||
__tablename__ = "ml_settings"
|
||||
# Bare name — Base.metadata's naming convention prepends ck_<table>_,
|
||||
# producing the final ck_ml_settings_singleton (matches migration 0003).
|
||||
# producing ck_ml_settings_singleton. The chain shipped the DOUBLED
|
||||
# ck_ml_settings_ck_ml_settings_singleton, because the migration
|
||||
# pre-prefixed the name and the convention prefixed it again; alembic
|
||||
# 0088 renames it to what this line has always produced (#3275).
|
||||
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
@@ -31,17 +35,20 @@ class MLSettings(Base):
|
||||
# queueing embed work nothing will consume (the daily GPU 'embed' backfill
|
||||
# covers those images instead).
|
||||
cpu_embed_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
# Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not
|
||||
# a fixed count) so coverage reflects real screen time regardless of length;
|
||||
# cap the total so a long video can't explode into hundreds of embeds. The
|
||||
# per-frame SigLIP embeddings are mean-pooled. Operator-tunable.
|
||||
video_frame_interval_seconds: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=4.0
|
||||
Float, nullable=False, default=4.0,
|
||||
server_default="4",
|
||||
)
|
||||
video_max_frames: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=64
|
||||
Integer, nullable=False, default=64,
|
||||
server_default="64",
|
||||
)
|
||||
# Tagging-v2 head training (#114). The head is the suggestion source that
|
||||
# LEARNS from the operator's tags (replacing Camie + centroid). A concept
|
||||
@@ -49,10 +56,12 @@ class MLSettings(Base):
|
||||
# head_auto_apply_precision is the precision bar a head must clear (at some
|
||||
# operating point) to "graduate" into earned auto-apply. Operator-tunable.
|
||||
head_min_positives: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=8
|
||||
Integer, nullable=False, default=8,
|
||||
server_default="8",
|
||||
)
|
||||
head_auto_apply_precision: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.97
|
||||
Float, nullable=False, default=0.97,
|
||||
server_default="0.97",
|
||||
)
|
||||
# Earned auto-apply (#114). A graduated head fires (tags images without a
|
||||
# human) when this master switch is on AND the head has at least
|
||||
@@ -61,29 +70,34 @@ class MLSettings(Base):
|
||||
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
|
||||
# measured-precision gates keep it safe, and every auto-tag is reversible.
|
||||
head_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
head_auto_apply_min_positives: Mapped[int] = mapped_column(
|
||||
# Support floor raised 30→50 (operator-asked 2026-07-06): a head needs
|
||||
# more human labels before it may fire without a human.
|
||||
Integer, nullable=False, default=50
|
||||
Integer, nullable=False, default=50,
|
||||
server_default="30",
|
||||
)
|
||||
# CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
|
||||
# over-fired (high-reference characters matched a scatter of images); 0.85
|
||||
# keeps the confident single-character matches. Tunable from the agent card.
|
||||
ccip_match_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.85
|
||||
Float, nullable=False, default=0.85,
|
||||
server_default="0.85",
|
||||
)
|
||||
# CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
|
||||
# above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
|
||||
# single-character references + the high bar keep it safe, every tag reversible.
|
||||
ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
ccip_auto_apply_threshold: Mapped[float] = mapped_column(
|
||||
# Raised 0.92→0.95 (operator-asked 2026-07-06) so only very confident
|
||||
# character matches auto-tag.
|
||||
Float, nullable=False, default=0.95
|
||||
Float, nullable=False, default=0.95,
|
||||
server_default="0.92",
|
||||
)
|
||||
# -- Presentation chrome auto-hide (#141) -------------------------------
|
||||
# `banner` (chrome — clusters on UI, not content) auto-applies on the sweep
|
||||
@@ -95,13 +109,21 @@ class MLSettings(Base):
|
||||
# (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor
|
||||
# screenshot` are no longer chrome — they went to the PROCESS path below.
|
||||
presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
presentation_auto_apply_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.90
|
||||
Float, nullable=False, default=0.90,
|
||||
# text(), not a string, because alembic 0082 used sa.text(): a bare
|
||||
# string renders DEFAULT '0.90'::double precision while text() renders
|
||||
# DEFAULT 0.90, and the chain is MIXED — some migrations used one,
|
||||
# some the other. Same value, different stored expression, so each
|
||||
# column here mirrors whichever form its own migration used (#3275).
|
||||
server_default=text("0.90"),
|
||||
)
|
||||
presentation_conflict_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.50
|
||||
Float, nullable=False, default=0.50,
|
||||
server_default=text("0.50"),
|
||||
)
|
||||
# -- Process auto-apply (#1464) ----------------------------------------
|
||||
# `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program
|
||||
@@ -115,24 +137,29 @@ class MLSettings(Base):
|
||||
# (PresentationReview, mode='process') rather than silently marked. OFF by
|
||||
# default — a new whole-library auto-tagger is opt-in; every auto-tag reversible.
|
||||
process_auto_apply_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=False
|
||||
Boolean, nullable=False, default=False,
|
||||
server_default="false",
|
||||
)
|
||||
process_auto_apply_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.90
|
||||
Float, nullable=False, default=0.90,
|
||||
server_default="0.90",
|
||||
)
|
||||
process_conflict_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.50
|
||||
Float, nullable=False, default=0.50,
|
||||
server_default="0.50",
|
||||
)
|
||||
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
|
||||
# existing libraries keep their stored value until the operator re-embeds.
|
||||
embedder_model_version: Mapped[str] = mapped_column(
|
||||
String(128), nullable=False, default="siglip2-so400m-patch16-512"
|
||||
String(128), nullable=False, default="siglip2-so400m-patch16-512",
|
||||
server_default="siglip2-so400m-patch16-512",
|
||||
)
|
||||
# The HF model NAME the embedder loads (server CPU embed + announced to the
|
||||
# GPU agent in the lease). Operator-settable so the embedder is a choice, not
|
||||
# a hardcode (#1190): set name + version together, then re-embed + retrain.
|
||||
embedder_model_name: Mapped[str] = mapped_column(
|
||||
String(128), nullable=False, default="google/siglip2-so400m-patch16-512"
|
||||
String(128), nullable=False, default="google/siglip2-so400m-patch16-512",
|
||||
server_default="google/siglip2-so400m-patch16-512",
|
||||
)
|
||||
# -- Crop proposers / detectors (#1202, #134) --------------------------
|
||||
# WHERE-to-crop YOLO detectors feeding the crop→SigLIP bag + CCIP. Config
|
||||
@@ -145,20 +172,24 @@ class MLSettings(Base):
|
||||
# person: general COCO figure detector for Western/realistic art the anime
|
||||
# person-detector misses → NMS-merged with imgutils → CCIP + concept.
|
||||
detector_person_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
detector_person_weights: Mapped[str] = mapped_column(
|
||||
String(512), nullable=False, default="yolo11n.pt"
|
||||
String(512), nullable=False, default="yolo11n.pt",
|
||||
server_default="yolo11n.pt",
|
||||
)
|
||||
detector_person_conf: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.35
|
||||
Float, nullable=False, default=0.35,
|
||||
server_default=text("0.35"),
|
||||
)
|
||||
# anatomy: booru_yolo anime/furry/NSFW torso components → concept crops.
|
||||
# Default = yolov11m_aa22 (26 classes, best mAP50-95 0.96), committed in the
|
||||
# upstream repo so the URL resolves. License UNSTATED — fine for a private
|
||||
# homelab (operator accepted #1202).
|
||||
detector_anatomy_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
detector_anatomy_weights: Mapped[str] = mapped_column(
|
||||
String(512), nullable=False,
|
||||
@@ -166,37 +197,47 @@ class MLSettings(Base):
|
||||
"https://github.com/aperveyev/booru_yolo/raw/main/models/"
|
||||
"yolov11m_aa22.pt"
|
||||
),
|
||||
server_default="https://github.com/aperveyev/booru_yolo/raw/main/models/yolov11m_aa22.pt",
|
||||
)
|
||||
detector_anatomy_conf: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.30
|
||||
Float, nullable=False, default=0.30,
|
||||
server_default=text("0.30"),
|
||||
)
|
||||
# panel: comic page → panel regions → concept crops (Apache-2.0, YOLOv12x).
|
||||
detector_panel_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
detector_panel_weights: Mapped[str] = mapped_column(
|
||||
String(512), nullable=False,
|
||||
default="mosesb/best-comic-panel-detection::best.pt",
|
||||
server_default="mosesb/best-comic-panel-detection::best.pt",
|
||||
)
|
||||
detector_panel_conf: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.30
|
||||
Float, nullable=False, default=0.30,
|
||||
server_default=text("0.30"),
|
||||
)
|
||||
# Per-frame caps bound the crop→embed explosion; max_regions is the hard
|
||||
# per-job backstop; dedupe_iou drops near-duplicate crops before the embed.
|
||||
detector_max_figures: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=8
|
||||
Integer, nullable=False, default=8,
|
||||
server_default="8",
|
||||
)
|
||||
detector_max_components: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=8
|
||||
Integer, nullable=False, default=8,
|
||||
server_default="8",
|
||||
)
|
||||
detector_max_panels: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=8
|
||||
Integer, nullable=False, default=8,
|
||||
server_default="8",
|
||||
)
|
||||
detector_max_regions: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=128
|
||||
Integer, nullable=False, default=128,
|
||||
server_default="128",
|
||||
)
|
||||
detector_dedupe_iou: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.85
|
||||
Float, nullable=False, default=0.85,
|
||||
server_default=text("0.85"),
|
||||
)
|
||||
# -- CCIP character prototypes (#1317) ---------------------------------
|
||||
# The per-character reference set is precomputed + refreshed INCREMENTALLY
|
||||
@@ -208,7 +249,8 @@ class MLSettings(Base):
|
||||
String(128), nullable=True
|
||||
)
|
||||
ccip_prototype_cap: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, default=64
|
||||
Integer, nullable=False, default=64,
|
||||
server_default="64",
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -35,7 +35,7 @@ class PatreonFailedMedia(Base):
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
first_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -35,7 +35,7 @@ class PixivFailedMedia(Base):
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
first_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -13,11 +13,13 @@ from sqlalchemy import (
|
||||
CheckConstraint,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
@@ -27,6 +29,10 @@ from .base import Base
|
||||
class Post(Base):
|
||||
__tablename__ = "post"
|
||||
__table_args__ = (
|
||||
# alembic 0030. The comment above described this index; nothing declared
|
||||
# it, so autogenerate proposed dropping it (#3275).
|
||||
Index("uq_post_artist_external_id_null_source", "artist_id", "external_post_id",
|
||||
unique=True, postgresql_where=text("source_id IS NULL")),
|
||||
# Source-bound dedup. Postgres treats NULL != NULL so rows
|
||||
# with source_id IS NULL aren't deduped by this constraint;
|
||||
# the partial unique index `uq_post_artist_external_id_null_source`
|
||||
@@ -35,7 +41,11 @@ class Post(Base):
|
||||
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
||||
CheckConstraint(
|
||||
"translation_override IN ('auto', 'force', 'original')",
|
||||
name="ck_post_translation_override",
|
||||
# Bare name: Base.metadata's naming convention prepends
|
||||
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
|
||||
# alembic 0088, which renames the four constraints that shipped
|
||||
# that way (#3275).
|
||||
name="translation_override",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -65,3 +65,15 @@ class PostAttachment(Base):
|
||||
captured_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
|
||||
|
||||
def attachment_download_url(attachment_id: int) -> str:
|
||||
"""The path that streams this attachment's bytes.
|
||||
|
||||
Both serializers that expose an attachment to the frontend
|
||||
(`provenance_service`, `post_feed_service`) built this literal themselves,
|
||||
so changing the route in `api/attachments.py` meant two edits and only one
|
||||
would be remembered (#3072). `test_attachment_download_url` pins it against
|
||||
the app's registered rule, so the drift is caught rather than trusted to.
|
||||
"""
|
||||
return f"/api/attachments/{attachment_id}/download"
|
||||
|
||||
@@ -11,7 +11,7 @@ are pruned by retention.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, Float, ForeignKey, String, func
|
||||
from sqlalchemy import DateTime, Float, ForeignKey, Index, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -20,6 +20,14 @@ from .base import Base
|
||||
class PresentationReview(Base):
|
||||
__tablename__ = "presentation_review"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_presentation_review_resolved_at", "resolved_at"),
|
||||
# Both FKs to tag were unindexed (#3300); tag_id CASCADEs, so a tag
|
||||
# delete had to scan this table to find its rows.
|
||||
Index("ix_presentation_review_tag_id", "tag_id"),
|
||||
Index("ix_presentation_review_conflict_tag_id", "conflict_tag_id"),
|
||||
)
|
||||
image_record_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
|
||||
)
|
||||
|
||||
@@ -16,7 +16,14 @@ title is the optional chapter name; stated_part is the optional operator-facing
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, Integer, Text, func
|
||||
from sqlalchemy import (
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -25,14 +32,26 @@ from .base import Base
|
||||
class SeriesChapter(Base):
|
||||
__tablename__ = "series_chapter"
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0047 named the UNIQUE `uq_series_chapter_anchor_page`, not
|
||||
# the `uq_series_chapter_anchor_page_id` a bare `unique=True` would
|
||||
# render (#3275).
|
||||
UniqueConstraint("anchor_page_id", name="uq_series_chapter_anchor_page"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
series_tag_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
# Both the UNIQUE (above) and the FK carry the names 0047 gave them; the
|
||||
# convention would render the FK `fk_series_chapter_anchor_page_id_series_page`.
|
||||
anchor_page_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("series_page.id", ondelete="CASCADE"),
|
||||
ForeignKey(
|
||||
"series_page.id",
|
||||
ondelete="CASCADE",
|
||||
name="fk_series_chapter_anchor_page",
|
||||
),
|
||||
nullable=False,
|
||||
unique=True,
|
||||
)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
stated_part: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
|
||||
@@ -14,7 +14,14 @@ number parsed from the source post, nullable when unknown.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, Integer, String, func
|
||||
from sqlalchemy import (
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -23,14 +30,22 @@ from .base import Base
|
||||
class SeriesPage(Base):
|
||||
__tablename__ = "series_page"
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0005 named this `uq_series_page_image`; a bare `unique=True`
|
||||
# on the column renders `uq_series_page_image_id` under the naming
|
||||
# convention, which is a different object from the one the database
|
||||
# has (#3275).
|
||||
UniqueConstraint("image_id", name="uq_series_page_image"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
series_tag_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
# UNIQUE lives in __table_args__ above, under the name 0005 gave it.
|
||||
image_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("image_record.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
unique=True,
|
||||
)
|
||||
# 'placed' = in the series-global run (page_number set); 'pending' = staged
|
||||
# from a post awaiting the operator's sort (page_number NULL). (#789 P2)
|
||||
|
||||
@@ -5,7 +5,16 @@ Multiple sources per artist support creators with cross-platform presence.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from .base import Base
|
||||
@@ -14,13 +23,27 @@ from .base import Base
|
||||
class Source(Base):
|
||||
__tablename__ = "source"
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0010. One row per (artist, platform, url): re-adding a source
|
||||
# the artist already has is an update, not a second row. The model had
|
||||
# never declared it (#3275), so autogenerate would have proposed
|
||||
# DROPPING it — the guarantee existed only in the migration chain.
|
||||
#
|
||||
# Named explicitly because the naming convention would render this
|
||||
# `uq_source_artist_id` (uq keys off column_0_name), which is both
|
||||
# wrong about the shape and not what the database actually has.
|
||||
UniqueConstraint(
|
||||
"artist_id", "platform", "url", name="uq_source_artist_platform_url"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
artist_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("artist.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
platform: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
url: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
|
||||
|
||||
config_overrides: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
|
||||
@@ -32,7 +55,7 @@ class Source(Base):
|
||||
# by _update_source_health alongside last_error; cleared on 'ok'.
|
||||
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
|
||||
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
||||
|
||||
# alembic 0031: sticky deep-scan budget. When > 0, the next N download
|
||||
# runs use gallery-dl's full-walk config (skip: True + 1800s timeout);
|
||||
|
||||
@@ -34,7 +34,7 @@ class SubscribeStarFailedMedia(Base):
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
first_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -15,11 +15,13 @@ from sqlalchemy import (
|
||||
Column,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Table,
|
||||
false,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy import (
|
||||
Enum as SQLEnum,
|
||||
@@ -67,17 +69,31 @@ image_tag = Table(
|
||||
primary_key=True,
|
||||
),
|
||||
Column("tag_id", ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True),
|
||||
Column("source", String(32), nullable=False, default="manual"),
|
||||
Column("source", String(32), nullable=False, default="manual", server_default="manual"),
|
||||
Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()),
|
||||
# The PK is (image_record_id, tag_id), which leads with the WRONG column
|
||||
# for the two things that matter most here (#3300): the gallery's tag
|
||||
# filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the
|
||||
# ON DELETE CASCADE from tag, which has to find a tag's rows to remove
|
||||
# them. Without this index both scan the largest table in the schema.
|
||||
Index("ix_image_tag_tag_id", "tag_id"),
|
||||
)
|
||||
|
||||
|
||||
class Tag(Base):
|
||||
__tablename__ = "tag"
|
||||
__table_args__ = (
|
||||
# alembic 0002. An EXPRESSION index — COALESCE cannot be expressed as a
|
||||
# UniqueConstraint, which is why it only ever existed in a migration (#3275).
|
||||
Index("uq_tag_name_kind_fandom", "name", "kind", text("COALESCE(fandom_id, 0)"),
|
||||
unique=True),
|
||||
CheckConstraint(
|
||||
"(fandom_id IS NULL) OR (kind = 'character')",
|
||||
name="ck_tag_fandom_requires_character",
|
||||
# Bare name: Base.metadata's naming convention prepends
|
||||
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
|
||||
# alembic 0088, which renames the four constraints that shipped
|
||||
# that way (#3275).
|
||||
name="fandom_requires_character",
|
||||
),
|
||||
)
|
||||
|
||||
@@ -87,6 +103,7 @@ class Tag(Base):
|
||||
SQLEnum(TagKind, name="tag_kind", values_callable=lambda e: [m.value for m in e]),
|
||||
nullable=False,
|
||||
default=TagKind.general,
|
||||
server_default="general",
|
||||
)
|
||||
fandom_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True, index=True
|
||||
|
||||
@@ -5,7 +5,7 @@ in image_prediction stay unmolested.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, String, func
|
||||
from sqlalchemy import DateTime, ForeignKey, Index, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -14,10 +14,17 @@ from .base import Base
|
||||
class TagAlias(Base):
|
||||
__tablename__ = "tag_alias"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
# Named explicitly: the database calls this ix_tag_alias_canonical, while
|
||||
# a bare index=True on the column would generate ix_tag_alias_canonical_tag_id
|
||||
# and silently propose a drop+create on the next autogenerate (#3275).
|
||||
Index("ix_tag_alias_canonical", "canonical_tag_id"),
|
||||
)
|
||||
alias_string: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
alias_category: Mapped[str] = mapped_column(String(32), primary_key=True)
|
||||
canonical_tag_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -5,7 +5,7 @@ Prevents re-suggestion AND prevents allowlist auto-apply on that image.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, func
|
||||
from sqlalchemy import DateTime, ForeignKey, Index, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -14,11 +14,24 @@ from .base import Base
|
||||
class TagSuggestionRejection(Base):
|
||||
__tablename__ = "tag_suggestion_rejection"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
# Named explicitly; see tag_alias for why (#3275).
|
||||
Index("ix_tag_suggestion_rejection_tag", "tag_id"),
|
||||
)
|
||||
# Both FKs named explicitly. alembic 0003 used a hand-shortened `tsr`
|
||||
# prefix; the convention would render the full table name (#3275).
|
||||
image_record_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
|
||||
ForeignKey(
|
||||
"image_record.id",
|
||||
ondelete="CASCADE",
|
||||
name="fk_tsr_image_record_id_image_record",
|
||||
),
|
||||
primary_key=True,
|
||||
)
|
||||
tag_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True
|
||||
ForeignKey("tag.id", ondelete="CASCADE", name="fk_tsr_tag_id_tag"),
|
||||
primary_key=True,
|
||||
)
|
||||
rejected_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
|
||||
@@ -15,7 +15,7 @@ backend.app.tasks.maintenance.recover_stalled_task_runs (Beat 5 min).
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, Integer, String, Text
|
||||
from sqlalchemy import DateTime, Index, Integer, String, Text, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
@@ -24,12 +24,21 @@ from .base import Base
|
||||
class TaskRun(Base):
|
||||
__tablename__ = "task_run"
|
||||
|
||||
|
||||
__table_args__ = (
|
||||
# alembic 0016: the three task-history indexes (#3275).
|
||||
Index("ix_task_run_name_started", "task_name", text("started_at DESC")),
|
||||
Index("ix_task_run_queue_started", "queue", text("started_at DESC")),
|
||||
Index("ix_task_run_status_started", "status", text("started_at DESC")),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
celery_task_id: Mapped[str] = mapped_column(
|
||||
String(64), nullable=False, index=True,
|
||||
)
|
||||
queue: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
|
||||
task_name: Mapped[str] = mapped_column(String(128), nullable=False, index=True)
|
||||
# Neither carries index=True: ix_task_run_queue_started and
|
||||
# ix_task_run_name_started already lead with these columns (#3301).
|
||||
queue: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
task_name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
target_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
started_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True,
|
||||
@@ -39,7 +48,9 @@ class TaskRun(Base):
|
||||
)
|
||||
duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default="running", index=True,
|
||||
# No index=True — ix_task_run_status_started leads with `status`.
|
||||
String(16), nullable=False, default="running",
|
||||
server_default="running",
|
||||
)
|
||||
error_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||||
error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
@@ -48,6 +48,47 @@ log = logging.getLogger(__name__)
|
||||
_VIDEO_DURATION_UNKNOWN = -1.0
|
||||
|
||||
|
||||
# -- artist-cascade predicates (rule 93: ONE definition, preview + apply) ---
|
||||
# project_artist_cascade (preview) and delete_artist_cascade (apply) both build
|
||||
# their queries from these. The preview used to re-derive its own — which is how
|
||||
# it came to count images and stay silent about posts and attachments while the
|
||||
# apply destroyed both. Same failure shape as the 2026-06-08 fandom-tag
|
||||
# deletion, where a re-implemented delete predicate diverged from the preview's.
|
||||
# Returned as condition LISTS spread into `.where(*conds)`, matching
|
||||
# _unused_tag_conditions / _bare_post_conditions below.
|
||||
|
||||
|
||||
def _artist_images_conditions(artist_id: int) -> list:
|
||||
"""Images the cascade deletes (rows AND their on-disk files)."""
|
||||
return [ImageRecord.artist_id == artist_id]
|
||||
|
||||
|
||||
def _artist_posts_conditions(artist_id: int) -> list:
|
||||
"""Posts the cascade destroys. The apply never names these — post.artist_id
|
||||
is ondelete=CASCADE, so Postgres takes them when the artist row goes — which
|
||||
is exactly why the preview has to name them: an artist whose posts are
|
||||
body-only (no images) otherwise previews as `images: 0` and reads as an
|
||||
empty artist, while every captured body/description/external-link set is
|
||||
destroyed."""
|
||||
return [Post.artist_id == artist_id]
|
||||
|
||||
|
||||
def _artist_attachments_conditions(artist_id: int) -> list:
|
||||
"""Attachments the cascade deletes. Matched by artist_id OR by the owning
|
||||
post's artist: artist_id is nullable (_capture_attachment leaves it NULL
|
||||
when no artist resolved), so neither arm alone covers every row. The
|
||||
sha-addressed blobs are NOT unlinked (one blob backs many rows) — these are
|
||||
row counts, and the bytes are not part of this operation's footprint."""
|
||||
return [
|
||||
or_(
|
||||
PostAttachment.artist_id == artist_id,
|
||||
PostAttachment.post_id.in_(
|
||||
select(Post.id).where(*_artist_posts_conditions(artist_id))
|
||||
),
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||
"""Read-only projection of what delete_artist_cascade would touch.
|
||||
|
||||
@@ -56,12 +97,17 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||
"artist": {"id": int, "name": str, "slug": str},
|
||||
"projected": {
|
||||
"images": int,
|
||||
"posts": int, # hard-deleted by the post.artist_id CASCADE
|
||||
"attachments": int, # rows deleted; the sha-addressed blobs stay
|
||||
"sources": int,
|
||||
"thumbs": int, # images with a thumbnail_path set
|
||||
"import_tasks": int, # ImportTask rows referencing the artist's images
|
||||
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
|
||||
},
|
||||
}
|
||||
Every count is built from the shared `_artist_*_conditions` predicates the
|
||||
apply uses, so the two halves cannot drift (rule 93).
|
||||
|
||||
Raises LookupError if slug not found. No mutations.
|
||||
"""
|
||||
from ..models.import_task import ImportTask
|
||||
@@ -73,36 +119,49 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||
if artist is None:
|
||||
raise LookupError(f"artist slug not found: {slug!r}")
|
||||
|
||||
images_conds = _artist_images_conditions(artist.id)
|
||||
|
||||
images_count = session.execute(
|
||||
select(func.count(ImageRecord.id))
|
||||
.where(ImageRecord.artist_id == artist.id)
|
||||
select(func.count(ImageRecord.id)).where(*images_conds)
|
||||
).scalar_one()
|
||||
posts_count = session.execute(
|
||||
select(func.count(Post.id))
|
||||
.where(*_artist_posts_conditions(artist.id))
|
||||
).scalar_one()
|
||||
attachments_count = session.execute(
|
||||
select(func.count(PostAttachment.id))
|
||||
.where(*_artist_attachments_conditions(artist.id))
|
||||
).scalar_one()
|
||||
# Sources have no shared predicate: the apply never queries them either, it
|
||||
# gets them from the Artist.sources ORM cascade. Counted directly here.
|
||||
sources_count = session.execute(
|
||||
select(func.count(Source.id))
|
||||
.where(Source.artist_id == artist.id)
|
||||
).scalar_one()
|
||||
thumbs_count = session.execute(
|
||||
select(func.count(ImageRecord.id))
|
||||
.where(ImageRecord.artist_id == artist.id)
|
||||
.where(*images_conds)
|
||||
.where(ImageRecord.thumbnail_path.is_not(None))
|
||||
).scalar_one()
|
||||
import_tasks_count = session.execute(
|
||||
select(func.count(ImportTask.id))
|
||||
.where(
|
||||
ImportTask.result_image_id.in_(
|
||||
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
|
||||
select(ImageRecord.id).where(*images_conds)
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
bytes_on_disk = session.execute(
|
||||
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
|
||||
.where(ImageRecord.artist_id == artist.id)
|
||||
.where(*images_conds)
|
||||
).scalar_one()
|
||||
|
||||
return {
|
||||
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
||||
"projected": {
|
||||
"images": images_count,
|
||||
"posts": posts_count,
|
||||
"attachments": attachments_count,
|
||||
"sources": sources_count,
|
||||
"thumbs": thumbs_count,
|
||||
"import_tasks": import_tasks_count,
|
||||
@@ -277,6 +336,10 @@ def delete_artist_cascade(
|
||||
series_page / tag_suggestion_rejection from ImageRecord delete,
|
||||
and source / post / download_event / etc. from Artist delete
|
||||
(via Artist.sources cascade="all, delete-orphan").
|
||||
|
||||
The artist's post_attachment rows are cleared EXPLICITLY before the
|
||||
artist row goes — see the comment at that step; leaving them to the
|
||||
cascade aborts the whole delete on a unique violation.
|
||||
"""
|
||||
artist = session.get(Artist, artist_id)
|
||||
if artist is None:
|
||||
@@ -287,11 +350,22 @@ def delete_artist_cascade(
|
||||
"files_deleted": 0,
|
||||
"thumbs_deleted": 0,
|
||||
"import_tasks_nulled": 0,
|
||||
"posts_deleted": 0,
|
||||
"attachments_deleted": 0,
|
||||
"files_failed": 0,
|
||||
},
|
||||
}
|
||||
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
|
||||
|
||||
# Counted BEFORE the delete: Postgres takes these via the post.artist_id
|
||||
# CASCADE when the artist row goes, so afterwards there is nothing left to
|
||||
# count. Reported so the summary can be checked against the preview's
|
||||
# `posts` — the parity rule 93 asks for is only testable if both halves
|
||||
# actually state the number.
|
||||
posts_deleted = session.execute(
|
||||
select(func.count(Post.id)).where(*_artist_posts_conditions(artist.id))
|
||||
).scalar_one()
|
||||
|
||||
images_deleted = 0
|
||||
files_deleted = 0
|
||||
thumbs_deleted = 0
|
||||
@@ -300,7 +374,7 @@ def delete_artist_cascade(
|
||||
while True:
|
||||
rows = session.execute(
|
||||
select(ImageRecord)
|
||||
.where(ImageRecord.artist_id == artist.id)
|
||||
.where(*_artist_images_conditions(artist.id))
|
||||
.limit(500)
|
||||
).scalars().all()
|
||||
if not rows:
|
||||
@@ -323,6 +397,28 @@ def delete_artist_cascade(
|
||||
# source_path_prefix matching that's out of scope here.
|
||||
import_tasks_nulled = 0
|
||||
|
||||
# Clear the artist's attachments BEFORE the artist row, or the delete below
|
||||
# aborts. Deleting an artist CASCADEs to Post (post.artist_id is
|
||||
# ondelete=CASCADE), which SET NULLs post_attachment.post_id — and
|
||||
# `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE
|
||||
# WHERE post_id IS NULL, so any two of this artist's attachments sharing a
|
||||
# sha collapse onto one another and raise. That is an ORDINARY shape, not a
|
||||
# corrupt one: _capture_attachment deliberately writes one row per post over
|
||||
# a single sha-addressed blob (a creator who attaches the same pdf to two
|
||||
# posts has two rows), and a pre-existing filesystem-import row with the same
|
||||
# sha and a NULL post_id collides on its own. Migration 0043 reasoned only
|
||||
# about upgrade-time safety and never about this later SET NULL.
|
||||
# _repoint_post_links guards the identical collision class in the reconcile
|
||||
# path; this is its artist-cascade counterpart.
|
||||
#
|
||||
# Which rows count as the artist's — and why the blobs are left on disk —
|
||||
# is _artist_attachments_conditions, shared with the preview.
|
||||
attachments_deleted = session.execute(
|
||||
delete(PostAttachment)
|
||||
.where(*_artist_attachments_conditions(artist.id))
|
||||
).rowcount or 0
|
||||
session.commit()
|
||||
|
||||
session.delete(artist)
|
||||
session.commit()
|
||||
|
||||
@@ -333,6 +429,8 @@ def delete_artist_cascade(
|
||||
"files_deleted": files_deleted,
|
||||
"thumbs_deleted": thumbs_deleted,
|
||||
"import_tasks_nulled": import_tasks_nulled,
|
||||
"posts_deleted": posts_deleted,
|
||||
"attachments_deleted": attachments_deleted,
|
||||
"files_failed": files_failed,
|
||||
},
|
||||
}
|
||||
@@ -1494,3 +1592,155 @@ def purge_gated_previews(
|
||||
"ledger_cleared": ledger_cleared,
|
||||
"posts_deleted": posts_deleted,
|
||||
}
|
||||
|
||||
|
||||
# -- orphaned attachment reclamation ---------------------------------------
|
||||
# PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or
|
||||
# artist leaves the row behind rather than taking it. Nothing ever pruned those
|
||||
# rows, and nothing has ever unlinked a file under the attachment store — so
|
||||
# both rows and bytes accumulated permanently and were invisible to every
|
||||
# existing diagnostic.
|
||||
#
|
||||
# Why this is a DISK->DB reconciliation rather than a row sweep: the store is
|
||||
# sha-addressed and idempotent (attachment_store.store), so ONE blob backs MANY
|
||||
# rows. Deleting a row therefore does not free its blob, and — since the artist
|
||||
# cascade now deletes its attachment rows outright — a freed blob has no DB
|
||||
# pointer left to find it by. Walking the store and asking "does any row still
|
||||
# reference this sha?" catches orphans from every cause, including ones no
|
||||
# future delete path will think to report.
|
||||
|
||||
# A blob is written by attachment_store.store BEFORE its row is inserted and
|
||||
# committed, so a just-stored file legitimately has no referencing row for a
|
||||
# moment. Same guard, same reasoning as ORPHAN_TEMP_MIN_AGE_HOURS in
|
||||
# tasks/maintenance.py: never judge a file younger than this.
|
||||
_ATTACHMENT_ORPHAN_MIN_AGE_HOURS = 6
|
||||
|
||||
# Wall-clock budget for the store walk (rule 89). A library with a large
|
||||
# attachment store shouldn't be able to run this past its soft time limit; on
|
||||
# exhaustion it reports partial=True and the operator re-runs to finish.
|
||||
_ATTACHMENT_RECLAIM_BUDGET_SECONDS = 900
|
||||
|
||||
# The store names files `<sha256><ext>`. Parse the sha as the first 64 chars
|
||||
# rather than via Path.stem: store() takes the extension straight from the
|
||||
# source filename, and a URL-encoded basename yields a multi-dot "suffix"
|
||||
# (see [[reference_url_encoded_basename_suffix]]) that would make stem eat part
|
||||
# of the sha. Validating the 64 chars as hex also skips anything else in the
|
||||
# tree that isn't a stored blob.
|
||||
_SHA256_HEX_LEN = 64
|
||||
|
||||
|
||||
def _orphan_attachment_conditions() -> list:
|
||||
"""PostAttachment rows belonging to nothing: both FKs nulled by a deleted
|
||||
post AND a deleted artist. A row with post_id NULL but an artist_id is the
|
||||
deliberate filesystem-import case (importer._capture_attachment writes it
|
||||
that way) and is NOT an orphan — it is still attributed."""
|
||||
return [
|
||||
PostAttachment.post_id.is_(None),
|
||||
PostAttachment.artist_id.is_(None),
|
||||
]
|
||||
|
||||
|
||||
def _is_sha_named(name: str) -> bool:
|
||||
"""True when `name` starts with a 64-char lowercase-hex sha256."""
|
||||
if len(name) < _SHA256_HEX_LEN:
|
||||
return False
|
||||
head = name[:_SHA256_HEX_LEN]
|
||||
return all(c in "0123456789abcdef" for c in head)
|
||||
|
||||
|
||||
def reclaim_orphaned_attachments(
|
||||
session: Session, *, images_root: Path, dry_run: bool = False,
|
||||
) -> dict:
|
||||
"""Prune unattributed PostAttachment rows, then unlink store blobs that no
|
||||
surviving row references.
|
||||
|
||||
Returns (same discovery keys either way, so the UI renders one shape):
|
||||
{"rows": int, # orphan rows found / deleted
|
||||
"files": int, # unreferenced blobs found / unlinked
|
||||
"bytes": int, # their total size
|
||||
"scanned": int, # blobs examined
|
||||
"skipped_recent": int, # blobs under the min-age guard
|
||||
"files_failed": int, # unlink raised (apply only)
|
||||
"partial": bool} # walk hit the time budget
|
||||
|
||||
dry_run computes exactly what the apply would do and mutates nothing — the
|
||||
surviving-sha set is derived by NEGATING the same orphan predicate the
|
||||
delete uses, so the preview cannot disagree with the apply (rule 93).
|
||||
"""
|
||||
started = time.monotonic()
|
||||
orphan_conds = _orphan_attachment_conditions()
|
||||
|
||||
if dry_run:
|
||||
rows = session.execute(
|
||||
select(func.count(PostAttachment.id)).where(*orphan_conds)
|
||||
).scalar_one()
|
||||
else:
|
||||
rows = session.execute(
|
||||
delete(PostAttachment).where(*orphan_conds)
|
||||
).rowcount or 0
|
||||
session.commit()
|
||||
|
||||
# Shas that still have a home. In the apply path the orphan rows are already
|
||||
# gone, so `NOT orphan` is redundant but harmless; in the dry-run path it is
|
||||
# what makes the projection honest about blobs the delete would free. One
|
||||
# predicate, one query, both modes.
|
||||
surviving_shas = set(session.execute(
|
||||
select(PostAttachment.sha256).where(~and_(*orphan_conds)).distinct()
|
||||
).scalars())
|
||||
|
||||
root = Path(images_root) / "attachments"
|
||||
cutoff = (
|
||||
datetime.now(UTC).timestamp()
|
||||
- _ATTACHMENT_ORPHAN_MIN_AGE_HOURS * 3600
|
||||
)
|
||||
files = 0
|
||||
freed_bytes = 0
|
||||
scanned = 0
|
||||
skipped_recent = 0
|
||||
files_failed = 0
|
||||
partial = False
|
||||
|
||||
if root.is_dir():
|
||||
for path in root.rglob("*"):
|
||||
if time.monotonic() - started >= _ATTACHMENT_RECLAIM_BUDGET_SECONDS:
|
||||
partial = True
|
||||
break
|
||||
# .partial staging files belong to cleanup_orphaned_temp_files —
|
||||
# leave them alone rather than racing an in-flight store().
|
||||
if path.suffix in (".part", ".partial") or not path.is_file():
|
||||
continue
|
||||
if not _is_sha_named(path.name):
|
||||
continue
|
||||
scanned += 1
|
||||
sha = path.name[:_SHA256_HEX_LEN]
|
||||
if sha in surviving_shas:
|
||||
continue
|
||||
try:
|
||||
st = path.stat()
|
||||
if st.st_mtime >= cutoff:
|
||||
skipped_recent += 1
|
||||
continue
|
||||
size = st.st_size
|
||||
if not dry_run:
|
||||
path.unlink()
|
||||
files += 1
|
||||
freed_bytes += size
|
||||
except OSError as exc:
|
||||
files_failed += 1
|
||||
log.warning("reclaim_orphaned_attachments: %s: %s", path, exc)
|
||||
|
||||
if not dry_run and (rows or files):
|
||||
log.info(
|
||||
"attachment reclaim: %d orphan row(s) deleted, %d blob(s) unlinked "
|
||||
"(%d bytes), %d failed, partial=%s",
|
||||
rows, files, freed_bytes, files_failed, partial,
|
||||
)
|
||||
return {
|
||||
"rows": rows,
|
||||
"files": files,
|
||||
"bytes": freed_bytes,
|
||||
"scanned": scanned,
|
||||
"skipped_recent": skipped_recent,
|
||||
"files_failed": files_failed,
|
||||
"partial": partial,
|
||||
}
|
||||
|
||||
@@ -181,7 +181,7 @@ def _augment_cookies(platform: str, netscape: str) -> str:
|
||||
"""Delegate to the platform's `augment_cookies` hook if one is
|
||||
registered (subscribestar, hentaifoundry, etc. — see
|
||||
`services/platforms/<name>.py`). No-op when the platform doesn't
|
||||
register a hook (Patreon, DeviantArt). Centralizing the
|
||||
register a hook (Patreon, Discord). Centralizing the
|
||||
quirks-per-platform in the platforms package means adding a new
|
||||
platform's cookie quirks doesn't require touching this file."""
|
||||
info = PLATFORMS.get(platform)
|
||||
|
||||
@@ -31,9 +31,8 @@ from .pixiv_ingester import PixivIngester
|
||||
from .subscribestar_ingester import SubscribeStarIngester
|
||||
|
||||
# Platforms whose download + verify go through the native ingester rather than
|
||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord,
|
||||
# deviantart — the latter slated for retirement, not migration) until they
|
||||
# migrate too.
|
||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
|
||||
# they migrate too.
|
||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
|
||||
|
||||
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
||||
|
||||
@@ -55,12 +55,6 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
|
||||
re.IGNORECASE,
|
||||
)),
|
||||
("deviantart", re.compile(
|
||||
r"^https?://(?:www\.)?deviantart\.com/"
|
||||
r"(?!home$|watch\b|tag\b|browse\b)"
|
||||
r"(?P<slug>[^/?#]+)/?$",
|
||||
re.IGNORECASE,
|
||||
)),
|
||||
("pixiv", re.compile(
|
||||
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
|
||||
re.IGNORECASE,
|
||||
|
||||
@@ -299,8 +299,9 @@ class GalleryDLService:
|
||||
# (services/patreon_ingester.py), not gallery-dl.
|
||||
PLATFORM_DEFAULTS = {
|
||||
# subscribestar removed — native-ingester platform now (#71); pixiv
|
||||
# removed likewise (#129). The remaining entries are the gallery-dl
|
||||
# platforms not yet migrated.
|
||||
# removed likewise (#129); deviantart removed at #3069 as a dropped
|
||||
# platform, not a migrated one. The remaining entries are the
|
||||
# gallery-dl platforms not yet migrated.
|
||||
"hentaifoundry": {
|
||||
"content_types": ["all"],
|
||||
"directory": [],
|
||||
@@ -316,15 +317,6 @@ class GalleryDLService:
|
||||
"reactions": False,
|
||||
"threads": True,
|
||||
},
|
||||
"deviantart": {
|
||||
"content_types": ["all"],
|
||||
"directory": [],
|
||||
"filename": "{index:>03}_{title[:50]}.{extension}",
|
||||
"flat": True,
|
||||
"original": True,
|
||||
"mature": True,
|
||||
"metadata": True,
|
||||
},
|
||||
}
|
||||
|
||||
def __init__(
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Bulk, idempotent writes to the ``image_tag`` association table.
|
||||
|
||||
Three writers attach tags to images in bulk: the WIP-title backfill
|
||||
(`wip_title.apply_wip_image_tags`), the concept-head auto-apply sweep and the
|
||||
system-tag auto-apply sweep (both in `ml/heads.py`). The two sweeps used to
|
||||
issue ONE INSERT PER ROW from inside their per-image loop — fine in steady
|
||||
state, but a first pass over a back-catalogue is tens of thousands of
|
||||
individual round-trips (#3072). All three share this one chunked multi-row
|
||||
insert now.
|
||||
|
||||
Sync only: every caller runs on a sync ``Session`` (the Celery task path). No
|
||||
async service writes image_tag in bulk, so there is no async sibling to keep in
|
||||
step — unlike `db_helpers.get_or_create`, which does have one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models.tag import image_tag
|
||||
|
||||
# 5000 rows x 3 bound params = 15000, comfortably inside Postgres' 65535-param
|
||||
# ceiling for a single statement. Raising this past ~21000 rows would exceed it.
|
||||
INSERT_CHUNK = 5000
|
||||
|
||||
|
||||
def insert_image_tags(
|
||||
session: Session, rows: list[dict], *, chunk: int = INSERT_CHUNK
|
||||
) -> None:
|
||||
"""Attach ``rows`` to their images, skipping any tag already on one.
|
||||
|
||||
Each row is ``{"image_record_id": int, "tag_id": int, "source": str}``.
|
||||
Does NOT commit — the caller owns the transaction.
|
||||
|
||||
ON CONFLICT DO NOTHING against the (image_record_id, tag_id) primary key,
|
||||
so an existing tag keeps its ORIGINAL ``source``: re-running a sweep can
|
||||
never re-stamp a tag the operator applied by hand as machine-applied.
|
||||
|
||||
Returns nothing on purpose. psycopg reports ``rowcount`` -1 for a multi-row
|
||||
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so a count
|
||||
taken from the statement would be a lie rather than an approximation.
|
||||
Callers that need an accurate count derive it themselves — see
|
||||
`wip_title.apply_wip_image_tags`' pre-SELECT, and the sweeps' `skip` sets.
|
||||
"""
|
||||
for start in range(0, len(rows), chunk):
|
||||
session.execute(
|
||||
pg_insert(image_tag)
|
||||
.values(rows[start:start + chunk])
|
||||
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
|
||||
)
|
||||
@@ -41,6 +41,7 @@ from ...models import (
|
||||
TagSuggestionRejection,
|
||||
)
|
||||
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
|
||||
from ..image_tag_apply import insert_image_tags
|
||||
from .training_data import (
|
||||
_AUTO_SOURCES,
|
||||
_applied_or_rejected,
|
||||
@@ -757,6 +758,10 @@ def auto_apply_sweep(
|
||||
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
|
||||
probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
|
||||
scanned += len(cids)
|
||||
# Collected across every head, then written as ONE insert below. Was an
|
||||
# insert per applied tag from inside this loop, which on a first sweep
|
||||
# over a back-catalogue is tens of thousands of round-trips (#3072).
|
||||
pending: list[dict] = []
|
||||
for h in range(len(rows)):
|
||||
tid = tag_ids[h]
|
||||
for idx in np.where(probs[:, h] >= thr[h])[0]:
|
||||
@@ -766,12 +771,12 @@ def auto_apply_sweep(
|
||||
skip[tid].add(iid)
|
||||
applied[h] += 1
|
||||
if not dry_run:
|
||||
session.execute(
|
||||
pg_insert(image_tag)
|
||||
.values(image_record_id=iid, tag_id=tid, source="head_auto")
|
||||
.on_conflict_do_nothing()
|
||||
)
|
||||
pending.append({
|
||||
"image_record_id": iid, "tag_id": tid,
|
||||
"source": "head_auto",
|
||||
})
|
||||
if not dry_run:
|
||||
insert_image_tags(session, pending)
|
||||
session.commit()
|
||||
run.last_progress_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
@@ -913,6 +918,11 @@ def system_tag_auto_apply_sweep(
|
||||
if Wc is not None:
|
||||
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
|
||||
scanned += len(cids)
|
||||
# Same batching as auto_apply_sweep (#3072): collect the chunk's rows
|
||||
# and write them once, below. The PresentationReview rows stay per-row —
|
||||
# they FK to image_record/tag, not to image_tag, so writing the tags
|
||||
# after them is safe, and a flagged conflict is rare by construction.
|
||||
pending: list[dict] = []
|
||||
for p in range(len(pres)):
|
||||
tid = pres_tag_ids[p]
|
||||
for idx in np.where(probs[:, p] >= thr)[0]:
|
||||
@@ -922,14 +932,10 @@ def system_tag_auto_apply_sweep(
|
||||
skip[tid].add(iid)
|
||||
applied[p] += 1
|
||||
if not dry_run:
|
||||
session.execute(
|
||||
pg_insert(image_tag)
|
||||
.values(
|
||||
image_record_id=iid, tag_id=tid,
|
||||
source=source,
|
||||
)
|
||||
.on_conflict_do_nothing()
|
||||
)
|
||||
pending.append({
|
||||
"image_record_id": iid, "tag_id": tid,
|
||||
"source": source,
|
||||
})
|
||||
# Guard 2: also looks like real content → still apply, but flag it
|
||||
# for the review strip instead of silently marking (chrome hides,
|
||||
# process stays visible — either way the operator gets a heads-up).
|
||||
@@ -944,6 +950,7 @@ def system_tag_auto_apply_sweep(
|
||||
mode=mode,
|
||||
)
|
||||
if not dry_run:
|
||||
insert_image_tags(session, pending)
|
||||
session.commit()
|
||||
|
||||
concepts = [
|
||||
|
||||
@@ -8,9 +8,10 @@ PLATFORMS below. Sidecar parsing, cookie materialization, and
|
||||
|
||||
Lifted from GallerySubscriber's
|
||||
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
|
||||
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
|
||||
and ~/.../extension/lib/platforms.js. Five platforms; auth_type and
|
||||
URL patterns match GS exactly so the existing browser extension
|
||||
hits FC unmodified.
|
||||
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
|
||||
FC downloaders are art-dedicated services only.
|
||||
"""
|
||||
|
||||
from .base import (
|
||||
@@ -18,7 +19,6 @@ from .base import (
|
||||
DEFAULT_EXTERNAL_POST_ID_KEYS,
|
||||
PlatformInfo,
|
||||
)
|
||||
from .deviantart import INFO as _DEVIANTART
|
||||
from .discord import INFO as _DISCORD
|
||||
from .hentaifoundry import INFO as _HENTAIFOUNDRY
|
||||
from .patreon import INFO as _PATREON
|
||||
@@ -33,7 +33,6 @@ PLATFORMS: dict[str, PlatformInfo] = {
|
||||
_HENTAIFOUNDRY,
|
||||
_DISCORD,
|
||||
_PIXIV,
|
||||
_DEVIANTART,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -63,7 +63,7 @@ class PlatformInfo:
|
||||
# Synthesize a post permalink from sidecar data. Required when
|
||||
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
||||
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
||||
# `url` field (patreon, deviantart).
|
||||
# `url` field (patreon).
|
||||
derive_post_url: Callable[[dict], str | None] | None = None
|
||||
|
||||
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
"""DeviantArt — no exercised quirks yet.
|
||||
|
||||
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
|
||||
audit, so we don't know yet whether DA's gallery-dl sidecars are
|
||||
well-behaved or have their own quirks. When DA gets exercised for the
|
||||
first time, add `derive_post_url` / `augment_cookies` here as needed.
|
||||
"""
|
||||
|
||||
from .base import GD_DEFAULTS, PlatformInfo
|
||||
|
||||
INFO = PlatformInfo(
|
||||
key="deviantart",
|
||||
name="DeviantArt",
|
||||
description="Download artwork from DeviantArt artists",
|
||||
auth_type="cookies",
|
||||
requires_auth=False,
|
||||
url_pattern=r"^https?://(www\.)?deviantart\.com/",
|
||||
url_examples=[
|
||||
"https://www.deviantart.com/example-artist",
|
||||
"https://www.deviantart.com/example-artist/gallery",
|
||||
],
|
||||
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
|
||||
)
|
||||
@@ -24,6 +24,7 @@ from ..models import (
|
||||
Post,
|
||||
PostAttachment,
|
||||
Source,
|
||||
attachment_download_url,
|
||||
)
|
||||
from ..utils.html_sanitize import (
|
||||
extract_img_srcs,
|
||||
@@ -360,7 +361,7 @@ class PostFeedService:
|
||||
"ext": att.ext,
|
||||
"mime": att.mime,
|
||||
"size_bytes": att.size_bytes,
|
||||
"download_url": f"/api/attachments/{att.id}/download",
|
||||
"download_url": attachment_download_url(att.id),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ from ..models import (
|
||||
Post,
|
||||
PostAttachment,
|
||||
Source,
|
||||
attachment_download_url,
|
||||
)
|
||||
from ..utils.html_sanitize import sanitize_post_html
|
||||
|
||||
@@ -53,7 +54,7 @@ def _attachment_dict(a: PostAttachment) -> dict:
|
||||
"original_filename": a.original_filename,
|
||||
"size_bytes": a.size_bytes,
|
||||
"ext": a.ext,
|
||||
"download_url": f"/api/attachments/{a.id}/download",
|
||||
"download_url": attachment_download_url(a.id),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -20,10 +20,10 @@ family gains one member.
|
||||
import re
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models.tag import WIP_SYSTEM_TAG, Tag, image_tag
|
||||
from .image_tag_apply import insert_image_tags
|
||||
|
||||
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other
|
||||
# apply sources so provenance stays legible and a future undo can target only these.
|
||||
@@ -113,13 +113,9 @@ def apply_wip_image_tags(
|
||||
to_insert = [iid for iid in chunk if iid not in already]
|
||||
if not to_insert:
|
||||
continue
|
||||
session.execute(
|
||||
pg_insert(image_tag)
|
||||
.values([
|
||||
{"image_record_id": iid, "tag_id": tag_id, "source": source}
|
||||
for iid in to_insert
|
||||
])
|
||||
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
|
||||
)
|
||||
insert_image_tags(session, [
|
||||
{"image_record_id": iid, "tag_id": tag_id, "source": source}
|
||||
for iid in to_insert
|
||||
])
|
||||
inserted += len(to_insert)
|
||||
return inserted
|
||||
|
||||
@@ -409,3 +409,31 @@ def rescan_series_suggestions_task(self, after_post_id: int = 0) -> dict:
|
||||
)
|
||||
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
||||
return summary
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.admin.reclaim_orphaned_attachments_task",
|
||||
bind=True,
|
||||
autoretry_for=(OperationalError, DBAPIError),
|
||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
||||
# The service stops walking at its own 900s budget and reports partial, so
|
||||
# these limits are the backstop for a wedged filesystem (NFS stall), not the
|
||||
# expected exit. Comfortably above the budget so a normal run always returns
|
||||
# its summary rather than being killed mid-walk.
|
||||
soft_time_limit=1200, time_limit=1500, # 20 min / 25 min
|
||||
)
|
||||
def reclaim_orphaned_attachments_task(self, dry_run: bool = True) -> dict:
|
||||
"""Reclaim unattributed PostAttachment rows and the store blobs nothing
|
||||
references any more (#3068). dry_run (the default) returns the projection
|
||||
without touching rows or files; apply deletes the orphan rows, then unlinks
|
||||
every blob no surviving row references.
|
||||
|
||||
Defaults to the SAFE preview — unlike the other tasks here, whose apply is
|
||||
reversible-ish or scoped; this one deletes files. Operator-triggered only,
|
||||
never on a beat: an unattended sweep that unlinks blobs is not something to
|
||||
run without someone reading the projection first."""
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
return cleanup_service.reclaim_orphaned_attachments(
|
||||
session, images_root=IMAGES_ROOT, dry_run=dry_run,
|
||||
)
|
||||
|
||||
@@ -173,6 +173,12 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
# task-name override beats the queue threshold whatever queue the row records
|
||||
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
|
||||
"backend.app.tasks.external.fetch_external_link": 65,
|
||||
# Attachment reclaim walks the whole sha-addressed store; the service caps
|
||||
# itself at a 900s budget and reports partial, but the task's hard limit is
|
||||
# 25 min for a wedged filesystem (NFS stall). Same phantom-flag class as the
|
||||
# external-fetch entry above — without an override a healthy in-flight walk
|
||||
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
|
||||
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
|
||||
}
|
||||
|
||||
|
||||
|
||||
+186
-6
@@ -9,14 +9,17 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||
## Image deps used
|
||||
|
||||
- python 3.14
|
||||
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
|
||||
- ruff (analyzer for `backend/`, `tests/`, `alembic/`, `agent/`, `scripts/`)
|
||||
- node (frontend job: `npm install` + vitest + vite build)
|
||||
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
|
||||
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml, build-agent — Fabled-Git registry push, and `imagetools inspect`/`create` for the reuse path)
|
||||
|
||||
## Secondary runtime image
|
||||
|
||||
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
|
||||
|
||||
`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else
|
||||
and installs nothing: it needs git and stdlib python, and builds no image.
|
||||
|
||||
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
|
||||
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
|
||||
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
|
||||
@@ -27,6 +30,10 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
|
||||
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
|
||||
- `npm install --no-audit --no-fund` — in `frontend-build` job
|
||||
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
|
||||
- `unzip` — in `extension.yml`'s "Verify XPI contents" step, installed via apt
|
||||
only when absent (`node:24-bookworm-slim` may or may not carry it). Debian
|
||||
package, ~2s. Not worth baking into a shared image for a single consumer, per
|
||||
`docs/process.md`'s ">1 project" rule.
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -49,7 +56,180 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
|
||||
classic script (`test/helpers/loadLib.js`) rather than adding `module.exports`
|
||||
shims to production code — the libs ship as `background.scripts`, not ES
|
||||
modules, so the specs exercise exactly the bytes packaged into the XPI.
|
||||
- Extension test files are excluded from the XPI via `--ignore-files` in
|
||||
`extension/package.json`, and the same paths are excluded from `ci.yml`'s
|
||||
`extension-version` guard. Those two lists must agree — `test/version.spec.js`
|
||||
asserts the guard never ignores a file web-ext actually packages.
|
||||
- **`extension/scripts/packaging.sh` is the single definition of what ships
|
||||
inside the XPI.** Three consumers read from it rather than keeping their own
|
||||
copy: web-ext's `--ignore-files` (`extension/package.json`), the `git log`
|
||||
pathspec inside the script's own version derivation, and `scripts/artifacts.sh`,
|
||||
which appends the extension's set to web's because the web image bundles the
|
||||
signed XPI. Hand-kept copies of that one fact is what allowed issue #2397, so
|
||||
`extension/test/version.spec.js` asserts no workflow has reintroduced a
|
||||
literal `:(exclude)extension/…`.
|
||||
- **Packaged and version-relevant are two different sets** (#3156). `scripts/`
|
||||
is excluded from the XPI and is NOT excluded from the version derivation,
|
||||
because `packaging.sh` decides the version string stamped into the packaged
|
||||
`manifest.json`. The membership test is *"can changing this file change the
|
||||
published bytes?"*, not *"is this file copied in?"* — which is why the script
|
||||
keeps two lists rather than one.
|
||||
- **The shipped extension version is derived, not committed.** It is the commit
|
||||
TIME of the newest packaged-extension change, rendered `YYYY.M.D.HHMM` UTC
|
||||
(family rules 148/149 — never a commit count, which orders by branch rather
|
||||
than by recency). `build.yml`'s `sign-extension` computes it and stamps it
|
||||
into `extension/manifest.json` + `package.json` in the working tree before
|
||||
signing; the stamp is never committed. The version in the repo is **wholly
|
||||
inert** — since milestone 318 step 8 there is no hand-set MAJOR.MINOR either.
|
||||
- **The extension is the one artifact that does not zero-pad, and that is not a
|
||||
drift** (#3138). Mozilla's grammar for AMO is
|
||||
`^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$` — a segment is the single
|
||||
digit `0` or starts 1-9, and there are at most four. `2026.08.29.0201` is
|
||||
rejected; `2026.8.29.201` is the same value one character narrower per
|
||||
segment, and rule 148 defines comparison as numeric per segment, so nothing is
|
||||
reordered. `ci.yml`'s `extension-version` lane asserts the derived string
|
||||
against that exact regex, plus a `YYYY.M.D.HHMM` shape check that would catch
|
||||
a regression to the pre-318 `1.0.<minutes>` — which AMO would accept and which
|
||||
orders below everything already signed. Checking here is the whole point: AMO
|
||||
409s on re-signing, so a version it rejects is burned and cannot be reused.
|
||||
`scripts/artifacts.sh version extension` **delegates** to `packaging.sh` so
|
||||
the two cannot answer differently.
|
||||
- Every job that derives anything checks out with `fetch-depth: 0` — all four
|
||||
`build.yml` jobs, `ci.yml`'s `extension-version` and `backend-lint-and-test`
|
||||
(for `tests/test_artifact_paths.py` and `test_artifact_identity.py`), and
|
||||
`release.yml`, which additionally walks the tag graph. A depth-1 clone sees
|
||||
one commit and derives a wrong, too-low value **rather than failing**, so the
|
||||
full-history checkout is load-bearing rather than incidental.
|
||||
- **`scripts/artifacts.sh` is the same shape one level up: one definition per
|
||||
artifact of what it is built from, and the two values derived from it.**
|
||||
`revision` (12 hex of the newest commit touching that set) and `version`
|
||||
(`YYYY.MM.DD.HHMM` UTC, rule 148). Four artifacts, four independent answers,
|
||||
so a push touching only `agent/` leaves web and ml alone.
|
||||
`tests/test_artifact_paths.py` reads each Dockerfile and asserts every COPY
|
||||
source is covered, so adding a COPY without updating the script fails CI.
|
||||
- **A file that DECIDES an artifact's identity belongs in its set even though it
|
||||
is copied into nothing** — `packaging.sh` for the extension and web (#3156),
|
||||
and `artifacts.sh` itself for web (#3202), which decides the `FC_VERSION`
|
||||
baked into that image. Only web needs the second entry: every artifact stamps
|
||||
a revision, but a revision has a backstop (a changed derivation stops matching
|
||||
the published label and forces a rebuild) and a version has none, because
|
||||
nothing compares it to anything. `tests/test_artifact_paths.py`'s `DERIVERS`
|
||||
table is the guard.
|
||||
- **Builds are skipped when the content is already published.** Each image
|
||||
carries its revision as an `fc.revision` LABEL, and `build.yml` reads that
|
||||
label back off the moving channel tag (`imagetools inspect --format`). Equal
|
||||
to the derived revision means the bytes are already published, so the job
|
||||
repoints the remaining tags at the existing manifest instead of rebuilding.
|
||||
Two things this depends on: an inspect that errors for ANY reason reads as a
|
||||
MISS so no needed build is ever skipped, and the repoint must EXCLUDE the
|
||||
source tag — `imagetools create` wraps its source in a manifest index, and
|
||||
config labels do not resolve through an index, so writing the channel tag
|
||||
from itself destroys the label the next run reads (#3183).
|
||||
- **The build pushes exactly ONE tag — the channel's — and every other tag is
|
||||
written registry-side afterwards** (#3190). buildx on this runner pushes the
|
||||
first tag to the registry and then re-pushes the rest through the docker
|
||||
driver, out of a local image store that a registry-direct build never fills;
|
||||
it fails intermittently with `tag does not exist`. On `dev` that only reddens
|
||||
a job, but on `main` it silently skips `:c-<sha>` while `:latest` publishes
|
||||
fine — a missing rollback tag has no consumer that fails, so nothing but the
|
||||
red job would notice until somebody needs to roll back. `imagetools create`
|
||||
has no local store to be absent from, and it is the code the reuse path
|
||||
already ran, so both paths now share one proven route. The cost: `:c-<sha>`
|
||||
is an index rather than a plain image, so `fc.revision` does not resolve
|
||||
through it — nothing reads it there, and the index names the same manifest.
|
||||
- **The image builds run on a `docker-container` buildx builder, and
|
||||
`provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder
|
||||
is what makes a registry layer cache possible at all — the default `docker`
|
||||
driver cannot export one (#3114) — and it is #3190's leading suspect, since
|
||||
it is the driver that resolves image metadata against a local store a
|
||||
registry-direct push never fills. **The attestation flags are load-bearing,
|
||||
not tidiness:** on the container driver `build-push-action@v5` defaults
|
||||
`provenance` to true when pushing, an attestation manifest makes the pushed
|
||||
tag a manifest INDEX, and config labels do not resolve through an index — so
|
||||
leaving them on would make every push read `fc.revision=<none>`, miss, and
|
||||
rebuild forever with every lane green. Same failure as #3183, different door.
|
||||
These jobs run inside a container against a mounted docker socket, so the
|
||||
buildkit container is a sibling rather than a child.
|
||||
- **All three images import and export a registry layer cache**
|
||||
(`<image>:buildcache`, `mode=max`). This is not an optimisation bolted onto
|
||||
the driver change — it is the other half of it. The `docker-container`
|
||||
driver gets a fresh buildkit instance per job and therefore has **no local
|
||||
layer store at all**, where the old `docker` driver at least reused whatever
|
||||
the runner's dockerd happened to hold. Measured on run 4896, the first builds
|
||||
after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent
|
||||
11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every
|
||||
build that runs, is one moving ref per image, holds cache blobs rather than a
|
||||
shippable artifact, and is overwritten in place, so it is not a return of the
|
||||
per-version tags milestone 318 withdrew (#3114).
|
||||
- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which
|
||||
bypasses the reuse check for all three images. It exists because
|
||||
skip-if-exists makes its own build path untestable: `agent/` has not changed
|
||||
since 2026-07-17, so the agent build has not run in six weeks and cannot be
|
||||
exercised on demand — and #3190 lives on exactly that path. Editing
|
||||
`build.yml` does not force a build either, deliberately: the workflow is not
|
||||
shipped bytes and is in no artifact's path set. The flag is read through
|
||||
`github.event.inputs` into an env var rather than interpolated into a run
|
||||
block, and it is checked inside the reuse step so that one decision drives
|
||||
both the build and the repoint.
|
||||
- **A weekly `schedule` rebuilds all three images against fresh base layers**
|
||||
(Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
|
||||
OUR source, so an artifact whose source stops moving stops picking up base
|
||||
updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
|
||||
that day's `nvidia/cuda` layers forever. Four things make it work:
|
||||
- It **builds `main`, not the branch that triggered it.** Forgejo registers a
|
||||
cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
|
||||
`github.ref` on dev. The ref is decided once in a top-level `env:
|
||||
BUILD_REF` that every checkout in the file takes, rather than per job —
|
||||
otherwise `sign-extension` would derive dev's extension version while
|
||||
`build-web` bundled main's, and the release download would 404 on a version
|
||||
that exists perfectly well. Every job then ASSERTS its checkout is `main`
|
||||
before doing anything, because `env` inside `with:` is not a context this
|
||||
runner is known to evaluate — if it silently resolved to empty, checkout
|
||||
would fall back to the triggering ref and the refresh would publish dev's
|
||||
source to `:latest` with every lane green.
|
||||
- It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
|
||||
the bytes that commit built; re-pushing it over refreshed layers would
|
||||
break the one tag rule 145 makes immutable, and it is the rollback unit.
|
||||
The repoint step needs no schedule case for this — the tag list is the
|
||||
channel tag alone, so SOURCE is the only entry, it is excluded as always,
|
||||
and the step correctly does nothing.
|
||||
- **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
|
||||
`main` push, by design.** They re-converge on that push: it hits reuse (a
|
||||
refresh does not move `fc.revision`, because it does not touch the source),
|
||||
and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
|
||||
push path needed no change for this, because the repoint already excluded
|
||||
the source tag — the same rule that keeps the label readable also keeps a
|
||||
refresh from being undone.
|
||||
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
|
||||
tag changes the `FROM` layer's cache key and everything above it rebuilds.
|
||||
**It does not currently make the unmoved case free.** Measured on the first
|
||||
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
|
||||
the bases resolved to unchanged digests, yet all three `:latest` tags got a
|
||||
NEW manifest digest, because buildkit mints a fresh image config per run and
|
||||
republishes identical layers under it. So `:latest` is rewritten weekly
|
||||
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
|
||||
diverge from on the same cadence — a digest change stops meaning anything.
|
||||
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
|
||||
Separately not caught: a Debian package update inside the `apt-get install`
|
||||
layer while the base tag stands still — a lag rather than a hole, since the
|
||||
official python/cuda images rebuild with those updates baked in.
|
||||
- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
|
||||
`build.yml` passes them to the web image only — the ml and agent images have
|
||||
nothing to report them to. `/api/health` returns both, the foot of Settings
|
||||
renders them, and `/api/extension/manifest` reports the channel beside the
|
||||
extension version so an install can be traced to a channel. With no version image tags, that
|
||||
self-report is the ONLY answer to "which build is this?" — which is why a
|
||||
missing version renders `unknown` rather than a blank: an empty footer reads
|
||||
as "no version", a different and false claim.
|
||||
Both are declared LAST in the Dockerfile on purpose: an ARG invalidates every
|
||||
layer below it, and these are the values that differ between the dev and main
|
||||
builds of identical source, so placing them earlier would stop the two
|
||||
channels ever sharing a cached `pip install`. Empty by default — a local build
|
||||
then reports nothing rather than claiming a channel it is not on.
|
||||
- **The channel is never folded into the version.** A `-dev` suffix makes the
|
||||
extension's per-segment `parseInt` comparator read that segment as 0, so every
|
||||
dev build compares equal to every other — issue #2993 exactly (rule 149).
|
||||
`frontend/test/systemBuild.spec.js` pins the rendered version to the bare
|
||||
number.
|
||||
- Callers MUST `set -f` before substituting the script's output. Without it the
|
||||
shell expands `test/**` against the working tree and silently narrows the
|
||||
pattern to whatever files exist at that moment — a failure that looks like
|
||||
nothing until dev files start appearing in the XPI. `test/version.spec.js`
|
||||
asserts every `--ignore-files` consumer sets it, and that no consumer has
|
||||
quietly reinstated a hardcoded list.
|
||||
|
||||
+23
-5
@@ -74,7 +74,25 @@ services:
|
||||
retries: 5
|
||||
|
||||
web:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
# :latest, NOT :dev — this file IS the install path.
|
||||
#
|
||||
# `docker compose up -d` merges docker-compose.override.yml, which sets
|
||||
# build: for all five app services, and a build: wins over image:. So a
|
||||
# contributor never pulls this tag and is unaffected by what it says.
|
||||
#
|
||||
# The tag is consulted only on `docker compose -f docker-compose.yml up -d`
|
||||
# — the documented production path, which skips the override. That is a
|
||||
# stranger installing the product, and they must land on the stable channel.
|
||||
#
|
||||
# :latest is main, which IS production (rule 147). :dev is the rolling
|
||||
# bleeding-edge channel we work out of, republished several times a day with
|
||||
# no stability promise. This file pinned :dev on all five services until
|
||||
# 2026-08-31 (#3270), so the documented install shipped development builds.
|
||||
# It went unnoticed because nobody who works on the project takes this path:
|
||||
# the operator deploys from a swarm stack file, contributors get the
|
||||
# override. Do not "fix" this back to :dev while debugging — use the
|
||||
# override, or -f with an explicit tag on the command line.
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:latest
|
||||
command: ["web"]
|
||||
# Graceful shutdown: give the container time to drain in-flight work on a
|
||||
# deploy (docker SIGTERMs, then SIGKILLs after this window — default is only
|
||||
@@ -122,7 +140,7 @@ services:
|
||||
redis: { condition: service_healthy }
|
||||
|
||||
worker:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:latest
|
||||
command: ["worker"]
|
||||
# Drain in-flight import/thumbnail/download tasks before SIGKILL on deploy.
|
||||
stop_grace_period: 90s
|
||||
@@ -142,7 +160,7 @@ services:
|
||||
redis: { condition: service_healthy }
|
||||
|
||||
scheduler:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:latest
|
||||
command: ["scheduler"]
|
||||
# Quick maintenance/scan lane + beat — short tasks, modest drain window.
|
||||
stop_grace_period: 60s
|
||||
@@ -163,7 +181,7 @@ services:
|
||||
# 30-min backup or a multi-chunk audit can never starve the 5-min recovery
|
||||
# sweeps / vacuum (operator-flagged 2026-06-07). One slot — these are heavy.
|
||||
maintenance-long:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:latest
|
||||
command: ["worker"]
|
||||
# Longest lane (DB backups, library audits, translation backfill) — give it
|
||||
# the most room to finish a chunk gracefully. Chunked + idempotent, so a job
|
||||
@@ -184,7 +202,7 @@ services:
|
||||
redis: { condition: service_healthy }
|
||||
|
||||
ml-worker:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator-ml:dev
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator-ml:latest
|
||||
command: ["ml-worker"]
|
||||
# A single GPU inference pass can run tens of seconds — let it finish.
|
||||
stop_grace_period: 120s
|
||||
|
||||
+77
-9
@@ -1,13 +1,14 @@
|
||||
# FabledCurator Firefox Extension
|
||||
|
||||
Self-hosted Firefox extension that pushes session cookies from supported
|
||||
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv,
|
||||
DeviantArt) into FabledCurator, and lets you add a creator as a Source
|
||||
from their page in one click.
|
||||
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv)
|
||||
into FabledCurator, and lets you add a creator as a Source from their
|
||||
page in one click.
|
||||
|
||||
## Install (operator)
|
||||
|
||||
The signed XPI is bundled into the FC Docker image. Open FC →
|
||||
The signed XPI is bundled into the FC Docker image — `:dev` and
|
||||
`:latest` each carry their own channel's build. Open FC →
|
||||
Settings → Maintenance → Browser extension → click "Install Firefox
|
||||
extension". Firefox shows its native install prompt. After installing,
|
||||
open the extension's options page (about:addons → FabledCurator →
|
||||
@@ -20,6 +21,7 @@ same card.
|
||||
cd extension/
|
||||
npm install --no-save # web-ext only
|
||||
npm run lint # web-ext lint
|
||||
npm run test:unit # vitest — lib/ logic + packaging/version checks
|
||||
npm run start # launches Firefox with extension loaded
|
||||
npm run build # unsigned XPI in web-ext-artifacts/
|
||||
```
|
||||
@@ -36,10 +38,76 @@ npm run build # unsigned XPI in web-ext-artifacts/
|
||||
- [ ] Subscriptions list: popup → "Sources" tab → list renders
|
||||
- [ ] Check now: click play icon on source row → no error toast
|
||||
|
||||
## Versioning — the committed number decides nothing
|
||||
|
||||
The shipped version is **derived**, not committed. `scripts/packaging.sh
|
||||
version` returns `YYYY.M.D.HHMM` in UTC: the commit *time* of the newest change
|
||||
to a packaged extension file. `build.yml` computes it and stamps it into both
|
||||
`manifest.json` and `package.json` at build time. The stamp is never
|
||||
committed — the commit carrying it would itself be a change to the extension,
|
||||
which would move the version again.
|
||||
|
||||
So:
|
||||
|
||||
- **Editing the version does nothing.** All of it is overwritten before web-ext
|
||||
ever reads it. There is no bump to make, and none to forget. There is no
|
||||
hand-set part left either: MAJOR.MINOR went away with milestone 318 step 8.
|
||||
- `npm run build` locally produces an XPI labelled with the *committed*
|
||||
version, since nothing stamped it. Fine for loading into a test profile; not
|
||||
what ships.
|
||||
|
||||
**Why the extension is the one artifact that does not zero-pad.** Every other
|
||||
FC artifact emits rule 148's `YYYY.MM.DD.HHMM`. AMO will not take it: Mozilla's
|
||||
grammar for addons.mozilla.org is
|
||||
|
||||
```
|
||||
^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
|
||||
```
|
||||
|
||||
— each segment is the single digit `0` or starts 1-9, so `08` and `0201` are
|
||||
rejected, and at most four segments are allowed. The extension therefore emits
|
||||
**the same numbers unpadded**: `2026.8.29.201` where the rest of the family
|
||||
says `2026.08.29.0201`. Rule 148 already defines comparison as numeric per
|
||||
segment, under which the two are equal, so nothing is reordered by the choice
|
||||
and left-padding each segment recovers the family string exactly. `ci.yml`'s
|
||||
`extension-version` lane checks the derived string against that regex on every
|
||||
push — the cheap place to find out, because AMO 409s on re-signing and a
|
||||
rejected version is burned for good.
|
||||
|
||||
Why commit time and not a commit count: a count is per-branch, so `dev` and
|
||||
`main` count different histories of the same code and their versions end up
|
||||
ordered by which branch accumulated more commits rather than by which is newer.
|
||||
Commit time gives both branches the same number for the same source — which is
|
||||
exactly what lets one AMO signature serve both channels (family rule 149, FC
|
||||
issue #3092).
|
||||
|
||||
## Channels
|
||||
|
||||
`dev` and `main` each build and sign their own extension, and an install is
|
||||
tied to whichever FC instance it points at — Firefox's static `update_url`
|
||||
cannot apply here, since every FC install is a different host, so the extension
|
||||
asks its configured backend. **The channel therefore IS the instance.**
|
||||
Switching channel means repointing the FC URL in options and reinstalling from
|
||||
that host; there is no separate channel setting, and adding one would
|
||||
contradict each server build shipping its own extension.
|
||||
|
||||
The channel is reported *beside* the version, never inside it:
|
||||
`/api/extension/manifest` answers `{"version": "...", "channel": "dev"}`. It is
|
||||
optional — an instance that declares none simply omits the key, and the popup,
|
||||
the toolbar tooltip and the Settings card all read exactly as they did before
|
||||
the field existed. Do not be tempted to make it a `-dev` version suffix: the
|
||||
comparator parses each dotted segment with `parseInt`, so a suffixed segment
|
||||
reads as 0 and every dev build compares equal to every other, collapsing "no
|
||||
update available" and "I cannot read this version" into one answer.
|
||||
|
||||
## Release
|
||||
|
||||
Bump `manifest.json` + `package.json` SemVer (both files) and commit
|
||||
under `extension/**`. The `.forgejo/workflows/extension.yml` workflow
|
||||
runs `web-ext sign` on main, commits the signed XPI to
|
||||
`frontend/public/extension/`, and the next FC server build bundles it
|
||||
into the Docker image.
|
||||
Nothing to do by hand. Push to `dev`: `build.yml` signs the extension if this
|
||||
change moved the version, caches the signed XPI as a Forgejo `ext-<version>`
|
||||
release, and bundles it into `fabledcurator:dev`. Merging to `main` derives the
|
||||
same version, hits that cache, and bundles the byte-identical XPI into
|
||||
`:latest` with no second AMO call.
|
||||
|
||||
AMO refuses to re-sign a version it has already issued, so signing is one-shot
|
||||
per version — which is why the cache exists and why the version must never move
|
||||
backwards.
|
||||
|
||||
@@ -37,7 +37,16 @@ ensureInitialized().catch(e => console.error('init failed:', e));
|
||||
// configured backend for the latest published version and nudge the operator to
|
||||
// reinstall the freshly-signed XPI — surfaced as a popup banner (on demand) and
|
||||
// a toolbar badge (daily). /api/extension/manifest is public and returns
|
||||
// {version, latest_url, sha256}; the XPI is served from the web root (not /api).
|
||||
// {version, latest_url, sha256} plus an OPTIONAL {channel} naming which channel
|
||||
// that instance serves ("dev"/"main", #3113); the XPI is served from the web
|
||||
// root (not /api).
|
||||
//
|
||||
// The channel IS the instance: Firefox's static update_url cannot apply here
|
||||
// because every FC install is a different host, so the extension asks its
|
||||
// configured backend — which means switching channel is repointing apiUrl in
|
||||
// options and reinstalling from that host. There is no separate channel
|
||||
// setting to build, and building one would contradict each server build
|
||||
// shipping its own extension.
|
||||
|
||||
function versionIsNewer(candidate, current) {
|
||||
// Dotted numeric compare so 1.0.10 > 1.0.9 (a plain string compare wouldn't).
|
||||
@@ -60,12 +69,22 @@ async function checkForUpdateInfo() {
|
||||
}
|
||||
const currentVersion = browser.runtime.getManifest().version;
|
||||
const latestVersion = info && info.version ? info.version : null;
|
||||
// Which channel the configured instance serves — reported ALONGSIDE the
|
||||
// version, never folded into it. A `-dev` suffix would have to survive
|
||||
// versionIsNewer's parseInt above, and it wouldn't: the segment would read
|
||||
// as 0 and every dev build would compare equal to every other.
|
||||
//
|
||||
// null is a normal answer, not a failure — an instance built before the
|
||||
// field existed, or one built locally with no channel declared. Nothing
|
||||
// below branches on it except the label.
|
||||
const channel = info && info.channel ? info.channel : null;
|
||||
// latest_url is served from the web root, not the JSON API.
|
||||
const base = api.webRoot();
|
||||
return {
|
||||
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
|
||||
currentVersion,
|
||||
latestVersion,
|
||||
channel,
|
||||
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null,
|
||||
};
|
||||
}
|
||||
@@ -77,7 +96,11 @@ async function refreshUpdateBadge() {
|
||||
await browser.action.setBadgeText({ text: r.updateAvailable ? '↑' : '' });
|
||||
if (r.updateAvailable) {
|
||||
await browser.action.setBadgeBackgroundColor({ color: '#F4BA7A' });
|
||||
await browser.action.setTitle({ title: `FabledCurator — update available (v${r.latestVersion})` });
|
||||
// Channel first, version second, and the channel dropped entirely when
|
||||
// the instance doesn't report one — so the tooltip reads exactly as it
|
||||
// did before the field existed rather than saying "(unknown ...)".
|
||||
const label = r.channel ? `${r.channel} v${r.latestVersion}` : `v${r.latestVersion}`;
|
||||
await browser.action.setTitle({ title: `FabledCurator — update available (${label})` });
|
||||
} else {
|
||||
await browser.action.setTitle({ title: 'FabledCurator' });
|
||||
}
|
||||
|
||||
@@ -68,16 +68,6 @@ const PLATFORMS = {
|
||||
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
|
||||
note: 'Click to authenticate via OAuth',
|
||||
},
|
||||
deviantart: {
|
||||
name: 'DeviantArt',
|
||||
domains: ['.deviantart.com', 'www.deviantart.com', 'deviantart.com'],
|
||||
authType: 'cookies',
|
||||
color: '#05CC47',
|
||||
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
|
||||
// DA's logged-in-only endpoints sit behind their internal _napi
|
||||
// namespace which shifts; skipping verify until a stable check
|
||||
// surfaces. Same posture as SubscribeStar.
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -98,7 +88,6 @@ const PLATFORM_ARTIST_PATTERNS = {
|
||||
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?:cw\/|c\/)?(?!(?:home|search|messages|notifications|library|settings|posts)(?:[\/?#]|$))[^/?#]+/i,
|
||||
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
|
||||
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
|
||||
deviantart: /^https?:\/\/(www\.)?deviantart\.com\/(?!home$|watch\b|tag\b|browse\b)[^/?#]+\/?$/i,
|
||||
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
|
||||
};
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"manifest_version": 3,
|
||||
"name": "FabledCurator",
|
||||
"version": "1.0.10",
|
||||
"version": "1.0.11",
|
||||
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
|
||||
|
||||
"browser_specific_settings": {
|
||||
@@ -33,7 +33,6 @@
|
||||
"*://*.hentai-foundry.com/*",
|
||||
"*://*.discord.com/*",
|
||||
"*://*.pixiv.net/*",
|
||||
"*://*.deviantart.com/*",
|
||||
"*://app-api.pixiv.net/*",
|
||||
"*://oauth.secure.pixiv.net/*",
|
||||
"*://*/*"
|
||||
@@ -61,7 +60,6 @@
|
||||
"*://*.subscribestar.com/*",
|
||||
"*://*.subscribestar.adult/*",
|
||||
"*://*.hentai-foundry.com/*",
|
||||
"*://*.deviantart.com/*",
|
||||
"*://*.pixiv.net/*"
|
||||
],
|
||||
"js": ["lib/platforms.js", "content/content-script.js"],
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
{
|
||||
"name": "fabledcurator-extension",
|
||||
"version": "1.0.10",
|
||||
"version": "1.0.11",
|
||||
"private": true,
|
||||
"description": "Firefox extension for FabledCurator",
|
||||
"comment_ignore_files": "The --ignore-files list comes from scripts/packaging.sh, the single source of truth shared with ci.yml's guard and the derived-version patch count. `set -f` is REQUIRED before the substitution: without it the shell globs `test/**` against the working tree and silently narrows the pattern to whatever files happen to exist.",
|
||||
"scripts": {
|
||||
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\"",
|
||||
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --firefox=firefox",
|
||||
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --overwrite-dest",
|
||||
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
|
||||
"lint": "set -f; web-ext lint --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore)",
|
||||
"start": "set -f; web-ext run --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --firefox=firefox",
|
||||
"build": "set -f; web-ext build --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --overwrite-dest",
|
||||
"sign": "set -f; web-ext sign --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
|
||||
"test:unit": "vitest run"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@@ -81,8 +81,12 @@ async function checkForUpdate() {
|
||||
}
|
||||
|
||||
function showUpdateBanner(r) {
|
||||
// The channel names itself beside the version, never inside it (#3113).
|
||||
// Absent when the instance doesn't report one, and the banner then reads
|
||||
// exactly as it did before the field existed.
|
||||
const channel = r.channel ? ` (${r.channel})` : '';
|
||||
document.getElementById('update-text').textContent =
|
||||
`Update available — v${r.latestVersion} (installed v${r.currentVersion})`;
|
||||
`Update available${channel} — v${r.latestVersion} (installed v${r.currentVersion})`;
|
||||
// Opening the signed XPI triggers Firefox's native install prompt.
|
||||
document.getElementById('update-btn').addEventListener('click', () => {
|
||||
browser.tabs.create({ url: r.xpiUrl });
|
||||
|
||||
Executable
+177
@@ -0,0 +1,177 @@
|
||||
#!/bin/sh
|
||||
# Single source of truth for "what ships inside the XPI", plus the version
|
||||
# derived from it.
|
||||
#
|
||||
# Three consumers used to hand-maintain their own copy of this list, and
|
||||
# keeping three copies of one fact in sync by hand is how issue #2397 happened:
|
||||
#
|
||||
# 1. web-ext's --ignore-files (extension/package.json's four scripts)
|
||||
# 2. the :(exclude) pathspec (what moves the version — a WIDER
|
||||
# set than the ignore list; see
|
||||
# NOT_VERSION_RELEVANT)
|
||||
# 3. the git-log pathspec (the derived version, below)
|
||||
#
|
||||
# They now all read from here. POSIX sh only — CI's run shell is busybox.
|
||||
#
|
||||
# -f (no pathname expansion) is set for the whole script and is load-bearing:
|
||||
# the lists below are iterated with deliberate word-splitting, and without -f
|
||||
# the shell would also GLOB them, expanding `test/**` into whatever files
|
||||
# happen to exist and corrupting the output. A caller's own `set -f` does not
|
||||
# help here — this runs as a separate sh process and does not inherit it.
|
||||
# Callers still need their own `set -f` for the substituted result; the two
|
||||
# guards protect different expansions.
|
||||
set -euf
|
||||
|
||||
# Paths under extension/ that are NOT packaged into the XPI.
|
||||
#
|
||||
# Split by whether git tracks them: node_modules and web-ext-artifacts are
|
||||
# build/dependency output that never appears in a commit, so they belong in
|
||||
# web-ext's ignore list but would be meaningless in a git pathspec.
|
||||
#
|
||||
# Directories need BOTH forms. `test/**` matches the files inside, but not the
|
||||
# directory entry itself — web-ext writes an entry for the directory too, so
|
||||
# with only the glob the XPI ends up carrying empty `test/` and `scripts/`
|
||||
# entries (caught by the XPI-content check on 2026-08-03). The bare name alone
|
||||
# is not enough either: minimatch's `test` does not match `test/url.spec.js`,
|
||||
# so dropping the glob would ship the contents. Keep both.
|
||||
NOT_PACKAGED_TRACKED='package.json package-lock.json README.md .gitignore vitest.config.js scripts scripts/** test test/**'
|
||||
NOT_PACKAGED_BUILD='web-ext-artifacts node_modules'
|
||||
|
||||
# Paths under extension/ that cannot change the SHIPPED BYTES, and so must not
|
||||
# move the derived version.
|
||||
#
|
||||
# Deliberately NOT the same list as NOT_PACKAGED_TRACKED, and the whole
|
||||
# difference is `scripts/`. packaging.sh is not packaged into the XPI — but it
|
||||
# DECIDES the version string, and build.yml stamps that string into the
|
||||
# manifest.json that is packaged. A change to how the version is computed is
|
||||
# therefore a change to the shipped bytes.
|
||||
#
|
||||
# Excluding it was harmless only while every push rebuilt the web image.
|
||||
# Milestone 313 step 4 made the rebuild conditional on the derived revision
|
||||
# moving, which turned it into a silent failure: a packaging.sh change gives a
|
||||
# NEW version, so sign-extension misses its ext-<version> cache and signs —
|
||||
# while build-web sees an unmoved revision, reuses the published image, and
|
||||
# ships the OLD XPI. An orphaned AMO signature, and an instance quietly serving
|
||||
# code the registry says is current.
|
||||
#
|
||||
# The two directions are not symmetric, which is why this list is the narrower
|
||||
# one. Too wide costs a re-sign and a rebuild for a change that ships nothing
|
||||
# new. Too narrow serves stale bytes and says nothing.
|
||||
NOT_VERSION_RELEVANT='package.json package-lock.json README.md .gitignore vitest.config.js test test/**'
|
||||
|
||||
usage() {
|
||||
echo "usage: packaging.sh {ignore|pathspec|version}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# web-ext --ignore-files values, space-separated.
|
||||
#
|
||||
# Callers MUST disable pathname expansion first (`set -f`), or the shell will
|
||||
# glob `test/**` against the working tree before web-ext ever sees the pattern
|
||||
# and silently narrow it to whatever happens to exist right now.
|
||||
cmd_ignore() {
|
||||
echo "$NOT_PACKAGED_TRACKED $NOT_PACKAGED_BUILD"
|
||||
}
|
||||
|
||||
# git pathspec excluding the tracked files that cannot change the shipped
|
||||
# bytes, e.g. :(exclude)extension/package.json :(exclude)extension/test/**
|
||||
#
|
||||
# This answers "what moves the version?", NOT "what goes in the XPI?" — see
|
||||
# NOT_VERSION_RELEVANT for why those differ. cmd_ignore answers the other one.
|
||||
# Same `set -f` requirement as above.
|
||||
cmd_pathspec() {
|
||||
for entry in $NOT_VERSION_RELEVANT; do
|
||||
printf ':(exclude)extension/%s ' "$entry"
|
||||
done
|
||||
echo
|
||||
}
|
||||
|
||||
# Strip leading zeros from one segment, leaving at least one digit.
|
||||
#
|
||||
# This exists for AMO and nothing else. Mozilla's version grammar for
|
||||
# addons.mozilla.org is documented as
|
||||
#
|
||||
# ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
|
||||
#
|
||||
# — each segment is either the single digit `0` or starts 1-9, so `08` and
|
||||
# `0201` are rejected outright, while `0` itself is fine. MDN states it in
|
||||
# prose too: "Non-zero numbers must not include a leading zero."
|
||||
#
|
||||
# POSIX sh has no trim-loop, hence the while.
|
||||
unpad() {
|
||||
s=$1
|
||||
while [ "${#s}" -gt 1 ]; do
|
||||
case "$s" in
|
||||
0*) s=${s#0} ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
printf '%s' "$s"
|
||||
}
|
||||
|
||||
# The extension's version: `YYYY.M.D.HHMM`, UTC, derived from the commit TIME
|
||||
# of the newest change to a PACKAGED extension file.
|
||||
#
|
||||
# THE ONE DELIBERATE DEPARTURE FROM THE FAMILY SHAPE, and it is a rendering
|
||||
# difference only. Rule 148 says `YYYY.MM.DD.HHMM` zero-padded, and every other
|
||||
# FC artifact emits exactly that. AMO's grammar (see unpad) forbids the padding,
|
||||
# and AMO is not negotiable: a rejected version is burned, since AMO 409s on
|
||||
# re-signing a version it has already seen. So the extension emits THE SAME
|
||||
# NUMBERS unpadded — 2026.08.29.0201 and 2026.8.29.201 are one value in two
|
||||
# renderings, and rule 148 already specifies comparison as numeric per segment,
|
||||
# under which they are equal. Nothing published is reordered by the choice, and
|
||||
# left-padding each segment recovers the family string exactly.
|
||||
#
|
||||
# HHMM is one segment, not two, because AMO allows at most FOUR. Unpadded that
|
||||
# reads oddly (00:14 -> `14`, midnight -> `0`) but stays strictly increasing
|
||||
# within a day, which is all the ordering needs.
|
||||
#
|
||||
# Why the commit's time and not the build's:
|
||||
# * MONOTONIC — max() over a set that only ever gains members.
|
||||
# * STABLE while the extension is unchanged, so an unchanged extension keeps
|
||||
# its version, the ext-<version> signature cache still hits, and AMO is
|
||||
# called once per extension CHANGE rather than once per push. Build-time
|
||||
# minutes would re-sign on every push and never let two channels share a
|
||||
# signature.
|
||||
# * SHARED ACROSS CHANNELS — after a merge, `main` sees the same commit and
|
||||
# derives the same number, so `:latest` reuses the signature `:dev` already
|
||||
# produced for byte-identical code. Same code, same version, one signing.
|
||||
# * REPRODUCIBLE — any checkout of a commit yields that commit's version.
|
||||
#
|
||||
# Never a commit count (family rule 149): a count is per-branch, so `dev` and
|
||||
# `main` count different histories of the same code and order by which branch
|
||||
# accumulated more commits rather than by which is newer. A squash-merge makes
|
||||
# that permanent. Roundtable's 2026-08-24 incident, in a different repo.
|
||||
#
|
||||
# Requires real history: a depth-1 clone sees one commit and will derive a wrong
|
||||
# (too low) value. Every consumer must check out with fetch-depth: 0.
|
||||
#
|
||||
# Formatted through git rather than date(1): busybox date does not reliably
|
||||
# accept `-d @<epoch>`, and git's --date=format-local is available wherever git
|
||||
# is. TZ=UTC so the value does not depend on the runner's timezone.
|
||||
cmd_version() {
|
||||
root=$(git rev-parse --show-toplevel)
|
||||
# Unquoted on purpose: the pathspec must word-split into separate args.
|
||||
# Globbing is already off script-wide (set -euf above).
|
||||
# shellcheck disable=SC2046
|
||||
sha=$(cd "$root" && git log --format='%ct %H' HEAD -- extension/ $(cmd_pathspec) \
|
||||
| sort -n | tail -1 | cut -d' ' -f2)
|
||||
if [ -z "$sha" ]; then
|
||||
echo "packaging.sh: no commit touches a packaged extension file" >&2
|
||||
exit 1
|
||||
fi
|
||||
padded=$(cd "$root" && TZ=UTC git show -s --format=%cd \
|
||||
--date='format-local:%Y.%m.%d.%H%M' "$sha")
|
||||
# Rebinding the function's own positional params, which are unused here.
|
||||
# shellcheck disable=SC2046
|
||||
set -- $(echo "$padded" | tr '.' ' ')
|
||||
echo "$(unpad "$1").$(unpad "$2").$(unpad "$3").$(unpad "$4")"
|
||||
}
|
||||
|
||||
[ $# -ge 1 ] || usage
|
||||
case "$1" in
|
||||
ignore) cmd_ignore ;;
|
||||
pathspec) cmd_pathspec ;;
|
||||
version) cmd_version ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
@@ -1,6 +1,12 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import path from 'node:path'
|
||||
import { loadLib } from './helpers/loadLib.js'
|
||||
|
||||
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
||||
const manifest = JSON.parse(readFileSync(path.join(EXT_DIR, 'manifest.json'), 'utf8'))
|
||||
|
||||
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
|
||||
'platforms.js',
|
||||
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
|
||||
@@ -13,7 +19,6 @@ describe('getPlatformFromUrl', () => {
|
||||
expect(getPlatformFromUrl('https://www.hentai-foundry.com/user/someone')).toBe('hentaifoundry')
|
||||
expect(getPlatformFromUrl('https://discord.com/channels/@me')).toBe('discord')
|
||||
expect(getPlatformFromUrl('https://www.pixiv.net/en/users/123')).toBe('pixiv')
|
||||
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe('deviantart')
|
||||
})
|
||||
|
||||
it('accepts http as well as https, with or without www', () => {
|
||||
@@ -26,6 +31,15 @@ describe('getPlatformFromUrl', () => {
|
||||
expect(getPlatformFromUrl('https://not-patreon.com/Atole')).toBe(null)
|
||||
expect(getPlatformFromUrl('')).toBe(null)
|
||||
})
|
||||
|
||||
it('returns null for deviantart, retired at #3069', () => {
|
||||
// The 2026-07-05 product decision (FC downloaders = art-dedicated services
|
||||
// only) left deviantart wired for seven weeks. Asserting the negative is
|
||||
// what keeps a partial retirement from being re-completed by accident.
|
||||
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe(null)
|
||||
expect(PLATFORMS.deviantart).toBeUndefined()
|
||||
expect(PLATFORM_ARTIST_PATTERNS.deviantart).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('isArtistPage', () => {
|
||||
@@ -72,12 +86,6 @@ describe('isArtistPage', () => {
|
||||
expect(isArtistPage('https://www.pixiv.net/en/artworks/999', 'pixiv')).toBe(false)
|
||||
})
|
||||
|
||||
it('excludes DeviantArt navigation roots', () => {
|
||||
expect(isArtistPage('https://www.deviantart.com/someone', 'deviantart')).toBe(true)
|
||||
expect(isArtistPage('https://www.deviantart.com/home', 'deviantart')).toBe(false)
|
||||
expect(isArtistPage('https://www.deviantart.com/watch', 'deviantart')).toBe(false)
|
||||
})
|
||||
|
||||
it('returns false for a platform with no artist pattern (discord)', () => {
|
||||
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
|
||||
})
|
||||
@@ -116,7 +124,6 @@ describe('platform table integrity', () => {
|
||||
patreon: 'https://www.patreon.com/cw/Atole',
|
||||
subscribestar: 'https://subscribestar.adult/someone',
|
||||
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
|
||||
deviantart: 'https://www.deviantart.com/someone',
|
||||
pixiv: 'https://www.pixiv.net/en/users/12345'
|
||||
}
|
||||
for (const [key, url] of Object.entries(samples)) {
|
||||
@@ -125,3 +132,59 @@ describe('platform table integrity', () => {
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('manifest.json agrees with the platform table', () => {
|
||||
// #3069: deviantart was dropped from the product in July but survived in
|
||||
// manifest.json until late August, because NOTHING tied the manifest's
|
||||
// domain lists back to PLATFORMS. These two specs are that tie. Both
|
||||
// directions matter: a stale match ships host access the product decided
|
||||
// not to use, and a missing one silently kills the Add-to-FC button.
|
||||
const matches = manifest.content_scripts[0].matches
|
||||
// '*://*.patreon.com/*' -> '.patreon.com', the form PLATFORMS.domains uses.
|
||||
const hostOf = (m) => m.replace(/^\*:\/\/\*/, '').replace(/\/\*$/, '')
|
||||
|
||||
it('injects the content script only on domains a platform claims', () => {
|
||||
for (const m of matches) {
|
||||
const host = hostOf(m)
|
||||
const owner = Object.entries(PLATFORMS).find(
|
||||
([, p]) => p.domains.includes(host)
|
||||
)
|
||||
expect(owner, `no platform claims content-script match "${m}"`).toBeTruthy()
|
||||
// The content script exists to draw the Add-as-source button, so a
|
||||
// platform with no artist pattern (discord) has no business here.
|
||||
expect(
|
||||
PLATFORM_ARTIST_PATTERNS[owner[0]],
|
||||
`"${m}" injects for ${owner[0]}, which has no artist pattern`
|
||||
).toBeTruthy()
|
||||
}
|
||||
})
|
||||
|
||||
it('injects on every platform that has an artist pattern', () => {
|
||||
const covered = new Set(
|
||||
matches
|
||||
.map(hostOf)
|
||||
.map((h) => Object.entries(PLATFORMS).find(([, p]) => p.domains.includes(h)))
|
||||
.filter(Boolean)
|
||||
.map(([key]) => key)
|
||||
)
|
||||
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
|
||||
expect(covered, `${key} has an artist pattern but no content-script match`).toContain(key)
|
||||
}
|
||||
})
|
||||
|
||||
it('requests no host permission for a domain no platform claims', () => {
|
||||
// '*://*/*' is the deliberate exception: FC is self-hosted at an arbitrary
|
||||
// operator-chosen URL, so the extension cannot enumerate its own backend.
|
||||
// Every OTHER entry is a platform domain and must still have an owner.
|
||||
for (const h of manifest.host_permissions) {
|
||||
if (h === '*://*/*') continue
|
||||
const host = hostOf(h)
|
||||
// pixiv's OAuth/API hosts are pixiv infrastructure, not creator pages,
|
||||
// so they are matched by suffix rather than by the domains list.
|
||||
const claimed = Object.values(PLATFORMS).some(
|
||||
(p) => p.domains.includes(host) || p.domains.some((d) => host.endsWith(d))
|
||||
)
|
||||
expect(claimed, `host permission "${h}" belongs to no platform`).toBe(true)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
+165
-45
@@ -1,63 +1,183 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { readdirSync, readFileSync } from 'node:fs'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import path from 'node:path'
|
||||
|
||||
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
|
||||
const read = (name) => JSON.parse(readFileSync(path.join(EXT_DIR, name), 'utf8'))
|
||||
const readText = (...seg) => readFileSync(path.join(EXT_DIR, ...seg), 'utf8')
|
||||
|
||||
describe('extension version consistency', () => {
|
||||
// Duplicates check (1) of ci.yml's extension-version job, deliberately.
|
||||
// That job is the gate that can't be bypassed; this spec is the one that
|
||||
// fails in a second on the developer's own CI lane with a readable diff.
|
||||
// The two version strings feed different systems and nothing else reconciles
|
||||
// them:
|
||||
// manifest.json -> what `web-ext sign` signs, so what Firefox installs
|
||||
// (package.json is in --ignore-files, not in the XPI)
|
||||
// package.json -> build.yml's AMO cache key, the ext-<version> release
|
||||
// tag, the XPI filename, and therefore the version
|
||||
// /api/extension/manifest reports to the update prompt
|
||||
it('keeps manifest.json and package.json in lockstep', () => {
|
||||
const manifest = read('manifest.json')
|
||||
const pkg = read('package.json')
|
||||
expect(manifest.version).toBe(pkg.version)
|
||||
// Only the git-free subcommands are exercised here: `version` shells out to
|
||||
// git, and the extension lane runs on node:24-bookworm-slim which may not ship
|
||||
// it. That one is covered where git is guaranteed — ci.yml's extension-version
|
||||
// lane and build.yml both run on ci-python.
|
||||
const packaging = (cmd) =>
|
||||
execFileSync('sh', [path.join(EXT_DIR, 'scripts', 'packaging.sh'), cmd], {
|
||||
cwd: EXT_DIR,
|
||||
encoding: 'utf8'
|
||||
})
|
||||
.trim()
|
||||
.split(/\s+/)
|
||||
.filter(Boolean)
|
||||
|
||||
describe('packaging.sh — the single definition of what ships', () => {
|
||||
it('emits an ignore list and a pathspec that agree on the tracked files', () => {
|
||||
const ignore = packaging('ignore')
|
||||
const pathspec = packaging('pathspec').map((e) => e.replace(':(exclude)extension/', ''))
|
||||
|
||||
// Every git-excluded path must also be hidden from web-ext. The reverse is
|
||||
// not required: node_modules and web-ext-artifacts are build output git
|
||||
// never tracks, so they appear only in the ignore list.
|
||||
for (const entry of pathspec) {
|
||||
expect(ignore, `pathspec has "${entry}" but --ignore-files does not`).toContain(entry)
|
||||
}
|
||||
expect(pathspec.length).toBeGreaterThan(0)
|
||||
expect(ignore).toContain('node_modules')
|
||||
})
|
||||
|
||||
it('uses a plain dotted numeric version AMO will accept', () => {
|
||||
// AMO rejects exotic version strings, and build.yml embeds this value in a
|
||||
// release tag and a filename — so anything needing escaping breaks the
|
||||
// publish path rather than the extension.
|
||||
expect(read('package.json').version).toMatch(/^\d+(\.\d+)*$/)
|
||||
it('emits glob patterns literally, never expanded against the working tree', () => {
|
||||
// The script iterates its lists with deliberate word-splitting, so it must
|
||||
// run with pathname expansion off. Without that, invoking it from a cwd
|
||||
// where test/ exists (exactly how ci.yml and vitest call it) expands
|
||||
// `test/**` into the individual spec files, and the pathspec silently stops
|
||||
// covering anything added later.
|
||||
const pathspec = packaging('pathspec')
|
||||
expect(pathspec).toContain(':(exclude)extension/test/**')
|
||||
expect(pathspec.some((e) => e.includes('.spec.js'))).toBe(false)
|
||||
expect(pathspec.some((e) => e.includes('helpers'))).toBe(false)
|
||||
|
||||
const ignore = packaging('ignore')
|
||||
expect(ignore).toContain('test/**')
|
||||
expect(ignore).toContain('scripts/**')
|
||||
expect(ignore.some((e) => e.includes('.spec.js'))).toBe(false)
|
||||
})
|
||||
|
||||
it('lets packaging.sh move the version, though it never ships in the XPI', () => {
|
||||
// The two lists answer different questions and this is the one place they
|
||||
// disagree. scripts/ is ignored by web-ext — it is repo tooling, not addon
|
||||
// code — but packaging.sh DECIDES the version string, and build.yml stamps
|
||||
// that string into the manifest.json that does ship. So changing how the
|
||||
// version is computed changes the shipped bytes.
|
||||
//
|
||||
// Excluding it from the pathspec was invisible while every push rebuilt the
|
||||
// web image. Milestone 313 step 4 made that rebuild conditional on the
|
||||
// derived revision moving, and the omission turned into a silent failure:
|
||||
// a new version means sign-extension misses its ext-<version> cache and
|
||||
// signs, while build-web sees an unmoved revision, reuses the published
|
||||
// image and ships the OLD XPI. An orphaned signature, and an instance
|
||||
// serving code the registry calls current.
|
||||
const pathspec = packaging('pathspec')
|
||||
expect(
|
||||
pathspec.some((e) => e.startsWith(':(exclude)extension/scripts')),
|
||||
'the pathspec excludes scripts/, so a change to how the version is '
|
||||
+ 'derived would not move the version it derives',
|
||||
).toBe(false)
|
||||
|
||||
// ...and it is still kept out of the package itself. Both must hold: the
|
||||
// tempting "fix" for either half is to make the two lists one again.
|
||||
expect(packaging('ignore')).toContain('scripts')
|
||||
})
|
||||
|
||||
it('keeps its own scripts and specs out of the XPI', () => {
|
||||
// Both are repo infrastructure. web-ext packages everything not ignored, so
|
||||
// omitting either would ship dev tooling to users -- and `test/**` in
|
||||
// particular only survives because callers `set -f` before substituting it.
|
||||
const ignore = packaging('ignore')
|
||||
expect(ignore).toContain('vitest.config.js')
|
||||
// Both forms per directory. The glob covers the contents; the bare name
|
||||
// covers the directory ENTRY, which web-ext writes separately — with only
|
||||
// the glob, the XPI carries an empty `test/` and `scripts/`.
|
||||
for (const dir of ['test', 'scripts']) {
|
||||
expect(ignore, `${dir} contents`).toContain(`${dir}/**`)
|
||||
expect(ignore, `${dir} directory entry`).toContain(dir)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('consumers delegate rather than keeping their own copy', () => {
|
||||
// These assertions are the actual anti-regression value: it is easy for a
|
||||
// future edit to "simplify" by inlining a literal list again, which silently
|
||||
// reintroduces the drift that issue #2397 was about.
|
||||
it('package.json derives --ignore-files from the script', () => {
|
||||
for (const [name, script] of Object.entries(read('package.json').scripts)) {
|
||||
if (!script.includes('--ignore-files')) continue
|
||||
expect(script, `${name} should call packaging.sh`).toContain('scripts/packaging.sh ignore')
|
||||
expect(script, `${name} must set -f before the substitution`).toMatch(/set -f;/)
|
||||
}
|
||||
})
|
||||
|
||||
// Read from disk rather than listed by hand. The point of this assertion is
|
||||
// that it survives consumers coming and going, and a hardcoded list is the
|
||||
// one part of it that cannot — release.yml (milestone 318 step 7) would have
|
||||
// joined the directory without joining the check.
|
||||
const WORKFLOWS = readdirSync(path.join(EXT_DIR, '..', '.forgejo', 'workflows')).filter((f) =>
|
||||
f.endsWith('.yml')
|
||||
)
|
||||
|
||||
it('no workflow hardcodes the packaged-file set', () => {
|
||||
// ci.yml used to substitute `packaging.sh pathspec` directly, for the
|
||||
// manual-bump guard that milestone 271 step 5 retired. Nothing inlines the
|
||||
// set today, and nothing should start to: a literal :(exclude)extension/...
|
||||
// in a workflow means someone bypassed the shared definition, which is
|
||||
// exactly the drift #2397 was about.
|
||||
expect(
|
||||
WORKFLOWS.length,
|
||||
'no workflows found — the glob is not looking where it thinks'
|
||||
).toBeGreaterThan(2)
|
||||
for (const wf of WORKFLOWS) {
|
||||
const text = readText('..', '.forgejo', 'workflows', wf)
|
||||
expect(text, `${wf} inlines an :(exclude) literal`).not.toMatch(/:\(exclude\)extension\//)
|
||||
}
|
||||
})
|
||||
|
||||
it('build.yml takes the shipped version from the script, not from the repo', () => {
|
||||
// The version is DERIVED from commit time (#3092, milestone 271 step 4).
|
||||
// Going back to reading the committed value is not a style regression, it
|
||||
// is the bug: a hand-set version makes dev and main sign the same number
|
||||
// for different code, and the ext-<version> cache then serves one channel
|
||||
// the other's XPI.
|
||||
const build = readText('..', '.forgejo', 'workflows', 'build.yml')
|
||||
expect(build).toContain('packaging.sh version')
|
||||
expect(build, 'build.yml re-reads the committed version instead of deriving it')
|
||||
.not.toMatch(/grep[^\n]*'"version"'[^\n]*package\.json/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('extension version', () => {
|
||||
// Mozilla's published grammar for addons.mozilla.org, transcribed from MDN's
|
||||
// manifest.json/version page. Each segment is the single digit 0 or starts
|
||||
// 1-9 — so no leading zeros — and there are at most four of them.
|
||||
const AMO = /^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$/
|
||||
|
||||
it('keeps a committed version AMO would accept, though it ships nothing', () => {
|
||||
// The committed value is wholly inert since milestone 318 step 8: there is
|
||||
// no hand-set MAJOR.MINOR left for packaging.sh to read, and build.yml
|
||||
// stamps the derived string over both files before web-ext sees them.
|
||||
//
|
||||
// It is still asserted, for one reason: `npm run build` locally packages
|
||||
// whatever is committed, so a value AMO would reject turns a local build
|
||||
// into a confusing failure with no CI signal ahead of it. ci.yml checks
|
||||
// the same grammar against the DERIVED value, which is the one AMO sees.
|
||||
for (const file of ['manifest.json', 'package.json']) {
|
||||
expect(read(file).version, `${file} version is not AMO-shaped`).toMatch(AMO)
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects the zero-padded family shape, which is why the extension unpads', () => {
|
||||
// Guards the reason for the exception, not just its result. If this ever
|
||||
// starts passing, someone has loosened the pattern and the next sign burns
|
||||
// an AMO version to find out. (#3138.)
|
||||
expect('2026.08.29.0201').not.toMatch(AMO)
|
||||
expect('2026.8.29.201').toMatch(AMO)
|
||||
// Five segments: AMO allows four.
|
||||
expect('2026.8.29.2.1').not.toMatch(AMO)
|
||||
})
|
||||
|
||||
it('declares manifest v3', () => {
|
||||
expect(read('manifest.json').manifest_version).toBe(3)
|
||||
})
|
||||
|
||||
it('never lets the CI guard ignore a file that actually ships', () => {
|
||||
// ci.yml's extension-version job skips its bump check for paths it deems
|
||||
// non-shipping. If it excludes something web-ext DOES package, a real
|
||||
// change to shipped code passes the guard unnoticed — precisely the
|
||||
// silent-stale-ship the guard exists to stop. The reverse drift (guard
|
||||
// stricter than web-ext) only costs a needless bump, so it isn't asserted.
|
||||
const ci = readFileSync(path.join(EXT_DIR, '..', '.forgejo', 'workflows', 'ci.yml'), 'utf8')
|
||||
const lint = read('package.json').scripts.lint
|
||||
const after = lint.split('--ignore-files')[1] ?? ''
|
||||
const ignored = new Set(
|
||||
after
|
||||
.split(/\s+/)
|
||||
.filter((tok) => tok && !tok.startsWith('--'))
|
||||
.map((tok) => tok.replace(/^["']|["']$/g, ''))
|
||||
)
|
||||
expect(ignored.size, 'parsed --ignore-files from the lint script').toBeGreaterThan(0)
|
||||
|
||||
const guarded = [...ci.matchAll(/:\(exclude\)extension\/(\S+?)'/g)].map((m) => m[1])
|
||||
expect(guarded.length, 'parsed :(exclude) entries from ci.yml').toBeGreaterThan(0)
|
||||
for (const entry of guarded) {
|
||||
expect(ignored, `ci.yml excludes "${entry}" but web-ext packages it`).toContain(entry)
|
||||
}
|
||||
})
|
||||
|
||||
it('lists every background script that exists, in dependency order', () => {
|
||||
// url.js must load BEFORE api.js: api.js calls normalizeApiUrl at
|
||||
// init()-time, and these are classic scripts sharing one scope, so a
|
||||
|
||||
@@ -50,14 +50,19 @@ const projected = ref(null)
|
||||
|
||||
const projectedCounts = computed(() => projected.value?.projected || null)
|
||||
|
||||
const modalDescription = computed(
|
||||
() => projected.value
|
||||
? `Artist “${props.artistName}” — `
|
||||
+ `${projected.value.projected.images} images, `
|
||||
+ `${projected.value.projected.sources} sources, `
|
||||
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
|
||||
: '',
|
||||
)
|
||||
// `posts` is named here, not left to the counts grid below it: an artist whose
|
||||
// posts are body-only previews as `images: 0`, and a summary line that says
|
||||
// only "0 images" reads as "this artist is empty" while the apply destroys
|
||||
// every captured post body (#3067). Attachments stay in the grid — the grid
|
||||
// renders every key, so this line carries only what changes the read.
|
||||
const modalDescription = computed(() => {
|
||||
const p = projectedCounts.value
|
||||
return p
|
||||
? `Artist “${props.artistName}” — ${p.images} images, `
|
||||
+ `${p.posts} posts, ${p.sources} sources, `
|
||||
+ `${Math.round(p.bytes_on_disk / 1_048_576)} MiB on disk`
|
||||
: ''
|
||||
})
|
||||
|
||||
async function onClick() {
|
||||
loading.value = true
|
||||
|
||||
@@ -139,9 +139,9 @@ function onThumbError() { thumbError.value = true }
|
||||
position: absolute; top: 8px; left: 8px;
|
||||
width: 22px; height: 22px; border-radius: 4px;
|
||||
border: 2px solid rgba(232, 228, 216, 0.8);
|
||||
background: rgba(20, 23, 26, 0.45);
|
||||
background: rgba(var(--v-theme-background), 0.45);
|
||||
display: grid; place-items: center;
|
||||
color: #14171A; z-index: 11;
|
||||
color: rgb(var(--v-theme-background)); z-index: 11;
|
||||
}
|
||||
.fc-gallery-item__checkbox.on {
|
||||
background: rgb(var(--v-theme-accent));
|
||||
@@ -152,7 +152,7 @@ function onThumbError() { thumbError.value = true }
|
||||
min-width: 22px; height: 22px; padding: 0 5px;
|
||||
border-radius: 11px;
|
||||
background: rgb(var(--v-theme-accent));
|
||||
color: #14171A; font-size: 12px; font-weight: 700;
|
||||
color: rgb(var(--v-theme-background)); font-size: 12px; font-weight: 700;
|
||||
display: grid; place-items: center; z-index: 11;
|
||||
pointer-events: none;
|
||||
}
|
||||
@@ -160,7 +160,8 @@ function onThumbError() { thumbError.value = true }
|
||||
position: absolute; left: 0; right: 0; bottom: 0;
|
||||
padding: 14px 8px 6px;
|
||||
background: linear-gradient(
|
||||
to top, rgba(20, 23, 26, 0.78), rgba(20, 23, 26, 0)
|
||||
to top, rgba(var(--v-theme-background), 0.78),
|
||||
rgba(var(--v-theme-background), 0)
|
||||
);
|
||||
font-size: 12px; line-height: 1.2;
|
||||
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
<template>
|
||||
<!-- #3068: attachment reclamation. PostAttachment's FKs are both SET NULL, so
|
||||
a deleted post or artist leaves the row behind; and the store is
|
||||
sha-addressed, so one blob backs many rows and deleting a row never freed
|
||||
its file. Nothing swept either. Preview first, then apply (destructive:
|
||||
unlinks files). -->
|
||||
<MaintenanceTile
|
||||
icon="mdi-paperclip-off"
|
||||
title="Reclaim orphaned attachments"
|
||||
blurb="Remove attachment records belonging to nothing, and the files nothing references."
|
||||
destructive
|
||||
:open="applying || previewing"
|
||||
>
|
||||
<p class="text-body-2 mb-3">
|
||||
Attachment records survive the post and artist they belonged to, and the
|
||||
files behind them are shared between records — so a deleted record never
|
||||
freed its file on its own. This finds records attributed to
|
||||
<strong>neither</strong> a post nor an artist, and files in the attachment
|
||||
store that <strong>no remaining record</strong> points at.
|
||||
<strong>Preview</strong> first; <strong>Apply</strong> deletes those
|
||||
records and unlinks those files. Files written in the last few hours are
|
||||
always left alone, so an in-progress download is never caught mid-write.
|
||||
</p>
|
||||
|
||||
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
|
||||
<v-btn
|
||||
color="primary" variant="tonal" rounded="pill"
|
||||
:loading="previewing" :disabled="applying" @click="preview"
|
||||
>
|
||||
<v-icon start>mdi-magnify</v-icon> Preview
|
||||
</v-btn>
|
||||
<v-btn
|
||||
color="error" rounded="pill"
|
||||
:loading="applying"
|
||||
:disabled="previewing || !canApply"
|
||||
@click="confirmOpen = true"
|
||||
>
|
||||
<v-icon start>mdi-paperclip-off</v-icon> Apply
|
||||
</v-btn>
|
||||
</div>
|
||||
|
||||
<v-alert
|
||||
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
|
||||
density="comfortable"
|
||||
>
|
||||
<span v-if="applied">
|
||||
Deleted {{ summary.rows }} orphaned record(s) and unlinked
|
||||
{{ summary.files }} file(s), reclaiming {{ humanBytes(summary.bytes) }}.
|
||||
</span>
|
||||
<span v-else-if="hasWork">
|
||||
{{ summary.rows }} orphaned record(s) and {{ summary.files }}
|
||||
unreferenced file(s) — {{ humanBytes(summary.bytes) }} reclaimable.
|
||||
Click <strong>Apply</strong> to remove them.
|
||||
</span>
|
||||
<span v-else>Nothing to reclaim — every attachment is accounted for.</span>
|
||||
|
||||
<!-- Both of these change what the numbers MEAN, so they are stated
|
||||
whenever they are non-zero rather than hidden in a tooltip. -->
|
||||
<div v-if="summary.files_failed" class="mt-1 text-caption">
|
||||
{{ summary.files_failed }} file(s) could not be read or removed — see
|
||||
the worker log.
|
||||
</div>
|
||||
<div v-if="summary.partial" class="mt-1 text-caption">
|
||||
Stopped early at the time limit; some of the store was not examined.
|
||||
Run it again to continue.
|
||||
</div>
|
||||
</v-alert>
|
||||
|
||||
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance" />
|
||||
|
||||
<v-dialog v-model="confirmOpen" max-width="440">
|
||||
<v-card>
|
||||
<v-card-title>Reclaim orphaned attachments?</v-card-title>
|
||||
<v-card-text class="text-body-2">
|
||||
This permanently deletes
|
||||
<strong>{{ summary?.rows ?? 0 }}</strong> attachment record(s) and
|
||||
unlinks <strong>{{ summary?.files ?? 0 }}</strong> file(s)
|
||||
({{ humanBytes(summary?.bytes) }}). Only files that no remaining
|
||||
record points at are removed, so nothing still attached to a post
|
||||
is affected.
|
||||
</v-card-text>
|
||||
<v-card-actions>
|
||||
<v-spacer />
|
||||
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
|
||||
<v-btn color="error" @click="apply">Reclaim</v-btn>
|
||||
</v-card-actions>
|
||||
</v-card>
|
||||
</v-dialog>
|
||||
</MaintenanceTile>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed, ref } from 'vue'
|
||||
|
||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||
import { humanBytes } from '../../utils/bytes.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
const confirmOpen = ref(false)
|
||||
|
||||
// Walks the whole attachment store, so it can run for minutes on a large
|
||||
// library — the service caps itself at 900s and reports `partial`. 150 polls
|
||||
// × 2s ≈ 5m of foreground waiting; past that the composable hands off to the
|
||||
// task dashboard rather than spinning forever.
|
||||
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
|
||||
endpoint: '/api/admin/maintenance/reclaim-attachments',
|
||||
storageKey: 'fc.maint.reclaimAttachments',
|
||||
appliedToast: 'Orphaned attachments reclaimed',
|
||||
maxPolls: 150,
|
||||
})
|
||||
|
||||
const hasWork = computed(
|
||||
() => !!summary.value && (summary.value.rows > 0 || summary.value.files > 0),
|
||||
)
|
||||
const canApply = computed(() => hasWork.value && !applied.value)
|
||||
const summaryType = computed(() => {
|
||||
if (applied.value) return 'success'
|
||||
return hasWork.value ? 'info' : 'success'
|
||||
})
|
||||
|
||||
// The confirm dialog gates the destructive apply; close it, then run.
|
||||
function apply () {
|
||||
confirmOpen.value = false
|
||||
applyTask()
|
||||
}
|
||||
</script>
|
||||
@@ -4,12 +4,22 @@
|
||||
<span v-if="manifest?.installed" class="text-caption fc-muted">
|
||||
· Firefox · v{{ manifest.version }}
|
||||
</span>
|
||||
<!-- Which channel this instance serves, so it is visible without
|
||||
installing anything. Rendered only when the image declares one: a
|
||||
locally-built image, or one predating the field, says nothing rather
|
||||
than guessing. Never merged into the version string beside it — see
|
||||
the endpoint's note on why a `-dev` suffix breaks the comparator. -->
|
||||
<v-chip
|
||||
v-if="manifest?.channel"
|
||||
size="x-small" variant="tonal" class="ml-2"
|
||||
:color="manifest.channel === 'dev' ? 'warning' : 'info'"
|
||||
>{{ manifest.channel }}</v-chip>
|
||||
</CardHeading>
|
||||
|
||||
<v-card-text>
|
||||
<p class="fc-muted text-body-2">
|
||||
Pushes session cookies from supported platforms
|
||||
(patreon, subscribestar, hentaifoundry, discord, pixiv, deviantart)
|
||||
(patreon, subscribestar, hentaifoundry, discord, pixiv)
|
||||
into FabledCurator, and lets you add a creator as a source from
|
||||
their page in one click.
|
||||
</p>
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
All subscription sources healthy.
|
||||
</p>
|
||||
<p v-else class="text-body-2 mb-0">
|
||||
<b class="fc-bad">{{ failing.length }}</b> failing source(s):
|
||||
<b class="fc-weak">{{ failing.length }}</b> failing source(s):
|
||||
<span class="fc-muted">{{ failingNames }}</span>
|
||||
</p>
|
||||
</v-card-text>
|
||||
@@ -72,5 +72,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.fc-bad { color: rgb(var(--v-theme-error)); }
|
||||
</style>
|
||||
|
||||
@@ -102,6 +102,7 @@
|
||||
import { computed, ref } from 'vue'
|
||||
|
||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||
import { humanBytes } from '../../utils/bytes.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
@@ -122,14 +123,6 @@ const summaryType = computed(() => {
|
||||
return summary.value && summary.value.matched > 0 ? 'info' : 'success'
|
||||
})
|
||||
|
||||
function humanBytes (n) {
|
||||
const b = Number(n || 0)
|
||||
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
|
||||
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
|
||||
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
|
||||
return b + ' B'
|
||||
}
|
||||
|
||||
// The confirm dialog gates the destructive apply; close it, then run.
|
||||
function apply () {
|
||||
confirmOpen.value = false
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
<div class="fc-cell__l">done</div>
|
||||
</div>
|
||||
<div class="fc-cell">
|
||||
<div class="fc-cell__n" :class="q.error ? 'fc-bad' : ''">{{ q.error }}</div>
|
||||
<div class="fc-cell__n" :class="q.error ? 'fc-weak' : ''">{{ q.error }}</div>
|
||||
<div class="fc-cell__l">errored</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -104,5 +104,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
|
||||
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
|
||||
color: rgb(var(--v-theme-on-surface-variant));
|
||||
}
|
||||
.fc-bad { color: rgb(var(--v-theme-error)); }
|
||||
</style>
|
||||
|
||||
@@ -78,6 +78,7 @@
|
||||
import { computed, ref } from 'vue'
|
||||
|
||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||
import { humanBytes } from '../../utils/bytes.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
@@ -98,14 +99,6 @@ const summaryType = computed(() => {
|
||||
return summary.value && summary.value.redundant > 0 ? 'info' : 'success'
|
||||
})
|
||||
|
||||
function humanBytes (n) {
|
||||
const b = Number(n || 0)
|
||||
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
|
||||
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
|
||||
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
|
||||
return b + ' B'
|
||||
}
|
||||
|
||||
// The confirm dialog gates the destructive apply; close it, then run.
|
||||
function apply () {
|
||||
confirmOpen.value = false
|
||||
|
||||
@@ -5,6 +5,18 @@ import { useApi } from '../composables/useApi.js'
|
||||
export const useSystemStore = defineStore('system', () => {
|
||||
const api = useApi()
|
||||
const healthy = ref(null) // null=unknown, true=ok, false=down
|
||||
// What the instance says it is. Since milestone 318 stopped publishing
|
||||
// version image tags, this is the only answer to "which build is this?" —
|
||||
// there is no registry name left to check it against.
|
||||
//
|
||||
// Three states, and collapsing any two of them would lie:
|
||||
// buildLoaded=false we have not asked yet -> render nothing
|
||||
// buildLoaded=true, version='' the build cannot say -> render "unknown"
|
||||
// buildLoaded=true, version=x this build is x
|
||||
// A blank footer would read as "no version", which is a different claim.
|
||||
const buildVersion = ref('')
|
||||
const buildChannel = ref('')
|
||||
const buildLoaded = ref(false)
|
||||
const stats = ref(null)
|
||||
const statsLoading = ref(false)
|
||||
|
||||
@@ -12,8 +24,17 @@ export const useSystemStore = defineStore('system', () => {
|
||||
try {
|
||||
const body = await api.get('/api/health')
|
||||
healthy.value = body.status === 'ok'
|
||||
// Absent means "cannot say" — the server omits these rather than
|
||||
// sending empty strings, so `?? ''` preserves that rather than
|
||||
// inventing a value for it.
|
||||
buildVersion.value = body.version ?? ''
|
||||
buildChannel.value = body.channel ?? ''
|
||||
buildLoaded.value = true
|
||||
} catch {
|
||||
healthy.value = false
|
||||
// Deliberately NOT setting buildLoaded: a failed health call tells us
|
||||
// nothing about the build, and claiming "unknown" would present a
|
||||
// network blip as a defective image.
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,5 +47,8 @@ export const useSystemStore = defineStore('system', () => {
|
||||
}
|
||||
}
|
||||
|
||||
return { healthy, stats, statsLoading, refreshHealth, refreshStats }
|
||||
return {
|
||||
healthy, stats, statsLoading, refreshHealth, refreshStats,
|
||||
buildVersion, buildChannel, buildLoaded,
|
||||
}
|
||||
})
|
||||
|
||||
@@ -50,7 +50,12 @@
|
||||
|
||||
/* Status text colours (DRY pass #161): fc-good = success, fc-weak = error,
|
||||
consolidated from the GPU / heads cards. fc-ok is intentionally NOT global —
|
||||
it means on-surface in HeadsCard but success in QueuesTable. */
|
||||
it means on-surface in HeadsCard but success in QueuesTable.
|
||||
|
||||
No `.fc-bad` (#3072): it was defined locally and identically in the Downloads
|
||||
and GPU activity panels, and it is fc-weak under a second name — GpuAgentCard
|
||||
and GpuActivityPanel were colouring the same "errored" count with different
|
||||
class names. Both now use fc-weak. Reach for fc-weak, not a new synonym. */
|
||||
.fc-good { color: rgb(var(--v-theme-success)); }
|
||||
.fc-weak { color: rgb(var(--v-theme-error)); }
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
// Human-readable byte sizes for maintenance summaries ("2.4 GB reclaimable").
|
||||
//
|
||||
// Promoted out of the cleanup cards, which had grown byte-identical private
|
||||
// copies (VideoDedupCard, GatedPurgeCard) and were about to grow a third for
|
||||
// the attachment reclaim. Binary units (1 KB = 1024 B) — these numbers come
|
||||
// from st_size / SUM(size_bytes), so they describe disk, not marketing.
|
||||
//
|
||||
// NOT the same shape as the `formatBytes` helpers in SystemStatsCards,
|
||||
// BackupRunsTable and PostCard — those differ in units, precision and
|
||||
// zero-handling. Left alone deliberately rather than force-fitted here.
|
||||
export function humanBytes (n) {
|
||||
const b = Number(n || 0)
|
||||
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
|
||||
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
|
||||
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
|
||||
return b + ' B'
|
||||
}
|
||||
@@ -1,8 +1,10 @@
|
||||
// Single source of truth for platform → color + icon mapping. Used by
|
||||
// PlatformChip and any other GS-style platform-tagged surface. The six
|
||||
// PlatformChip and any other GS-style platform-tagged surface. The five
|
||||
// platforms FC supports map 1:1 to the GS palette; unknown platforms fall
|
||||
// back to grey + mdi-web. Operator-confirmed scope 2026-05-27. The ICONS key
|
||||
// set is pinned against backend known_platform_keys() by
|
||||
// back to grey + mdi-web — which is deliberately what a retired platform
|
||||
// hits: a pre-#3069 deviantart source row still renders, as its raw key on
|
||||
// a grey chip. Operator-confirmed scope 2026-05-27. The ICONS key set is
|
||||
// pinned against backend known_platform_keys() by
|
||||
// tests/test_fe_be_contract.py.
|
||||
|
||||
const ICONS = {
|
||||
@@ -11,7 +13,6 @@ const ICONS = {
|
||||
hentaifoundry: 'mdi-palette',
|
||||
discord: 'mdi-discord',
|
||||
pixiv: 'mdi-alpha-p-box',
|
||||
deviantart: 'mdi-deviantart',
|
||||
}
|
||||
|
||||
const COLORS = {
|
||||
@@ -20,7 +21,6 @@ const COLORS = {
|
||||
hentaifoundry: 'purple',
|
||||
discord: 'indigo',
|
||||
pixiv: 'blue',
|
||||
deviantart: 'green',
|
||||
}
|
||||
|
||||
const LABELS = {
|
||||
@@ -29,7 +29,6 @@ const LABELS = {
|
||||
hentaifoundry: 'HentaiFoundry',
|
||||
discord: 'Discord',
|
||||
pixiv: 'Pixiv',
|
||||
deviantart: 'DeviantArt',
|
||||
}
|
||||
|
||||
export function platformIcon(platform) {
|
||||
|
||||
@@ -19,14 +19,16 @@
|
||||
</section>
|
||||
|
||||
<section class="fc-section">
|
||||
<h3 class="fc-section__title">Duplicates & posts</h3>
|
||||
<h3 class="fc-section__title">Duplicates & leftovers</h3>
|
||||
<p class="fc-section__hint">
|
||||
Tidy post records, duplicates and locked-preview leftovers.
|
||||
Tidy post records, duplicates, locked-preview leftovers and attachments
|
||||
that outlived what they belonged to.
|
||||
</p>
|
||||
<div class="fc-tile-grid">
|
||||
<PostMaintenanceCard />
|
||||
<VideoDedupCard />
|
||||
<GatedPurgeCard />
|
||||
<AttachmentReclaimCard />
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -60,6 +62,7 @@ import SingleColorAuditCard from '../components/cleanup/SingleColorAuditCard.vue
|
||||
import PostMaintenanceCard from '../components/settings/PostMaintenanceCard.vue'
|
||||
import VideoDedupCard from '../components/settings/VideoDedupCard.vue'
|
||||
import GatedPurgeCard from '../components/settings/GatedPurgeCard.vue'
|
||||
import AttachmentReclaimCard from '../components/settings/AttachmentReclaimCard.vue'
|
||||
import TagMaintenanceCard from '../components/settings/TagMaintenanceCard.vue'
|
||||
import DangerZoneCard from '../components/settings/DangerZoneCard.vue'
|
||||
</script>
|
||||
|
||||
@@ -54,6 +54,21 @@
|
||||
<MaintenancePanel />
|
||||
</v-window-item>
|
||||
</v-window>
|
||||
|
||||
<!-- Which build is this? With no version image tags (milestone 318) the
|
||||
instance's own report is the only answer, so it is shown rather than
|
||||
hidden. The instinct to treat it as information disclosure does not
|
||||
survive contact: the JS bundle and asset hashes fingerprint the build
|
||||
anyway, and "I'm on 2026.08.28.1249" is the single most useful line in
|
||||
a bug report.
|
||||
|
||||
Channel sits BESIDE the version, never inside it (rule 149) — a
|
||||
`-dev` suffix would read as a 0 segment to the extension's comparator
|
||||
and make every dev build compare equal (#2993). -->
|
||||
<div v-if="system.buildLoaded" class="text-caption text-medium-emphasis text-center mt-8">
|
||||
FabledCurator {{ system.buildVersion || 'unknown' }}
|
||||
<span v-if="system.buildChannel"> · {{ system.buildChannel }}</span>
|
||||
</div>
|
||||
</v-container>
|
||||
</template>
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { nextTick } from 'vue'
|
||||
|
||||
import BrowserExtensionCard from '../../src/components/settings/BrowserExtensionCard.vue'
|
||||
import { freshPinia, mountComponent } from '../support/mountComponent.js'
|
||||
|
||||
// useApi is a thin fetch wrapper, so the seam is fetch itself (same shape as
|
||||
// showcase.spec.js) rather than a module mock.
|
||||
function stubApi(manifest) {
|
||||
globalThis.fetch = vi.fn(async (url) => {
|
||||
const payload = String(url).includes('/api/extension/manifest')
|
||||
? manifest
|
||||
: { key: 'test-key' }
|
||||
return {
|
||||
ok: true, status: 200, statusText: '200',
|
||||
text: async () => JSON.stringify(payload),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
async function mountCard(manifest) {
|
||||
stubApi(manifest)
|
||||
const w = mountComponent(BrowserExtensionCard, { pinia: freshPinia() })
|
||||
// onMounted fires two fetches (manifest + key) and each resolves through a
|
||||
// chain of microtasks. Yielding to a macrotask drains the whole queue, which
|
||||
// a fixed number of nextTicks would only do by luck.
|
||||
await new Promise((resolve) => setTimeout(resolve, 0))
|
||||
await nextTick()
|
||||
return w
|
||||
}
|
||||
|
||||
const INSTALLED = {
|
||||
installed: true,
|
||||
version: '1.0.3499884',
|
||||
xpi_url: '/extension/fabledcurator-1.0.3499884.xpi',
|
||||
latest_url: '/extension/fabledcurator-latest.xpi',
|
||||
sha256: 'abc',
|
||||
}
|
||||
|
||||
describe('BrowserExtensionCard — channel', () => {
|
||||
beforeEach(() => { vi.restoreAllMocks() })
|
||||
afterEach(() => { delete globalThis.fetch })
|
||||
|
||||
it('names the channel the instance reports', async () => {
|
||||
// The point of the whole channel scheme: an operator can tell a dev
|
||||
// instance from a main one without installing anything.
|
||||
const w = await mountCard({ ...INSTALLED, channel: 'dev' })
|
||||
expect(w.text()).toContain('dev')
|
||||
})
|
||||
|
||||
it('shows the version and the channel as SEPARATE text, never merged', async () => {
|
||||
// Regression guard with teeth: the tempting shortcut is a `-dev` version
|
||||
// suffix, and that is precisely what breaks the extension's comparator —
|
||||
// it parses each dotted segment with parseInt, so a suffixed segment reads
|
||||
// as 0 and every dev build compares equal to every other. If someone ever
|
||||
// "simplifies" by folding the channel into the version, the version text
|
||||
// stops being the bare derived number and this fails.
|
||||
const w = await mountCard({ ...INSTALLED, channel: 'dev' })
|
||||
expect(w.text()).toContain('v1.0.3499884')
|
||||
expect(w.text()).not.toContain('1.0.3499884-dev')
|
||||
})
|
||||
|
||||
it('renders no channel when the instance declares none', async () => {
|
||||
// A locally-built image, or one predating the field. The card must read
|
||||
// exactly as it did before the channel existed rather than inventing an
|
||||
// "unknown" badge — absence is a normal answer here, not a fault.
|
||||
const w = await mountCard(INSTALLED)
|
||||
expect(w.text()).toContain('v1.0.3499884')
|
||||
expect(w.findAll('v-chip')).toHaveLength(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,96 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { setActivePinia, createPinia } from 'pinia'
|
||||
import { useSystemStore } from '../src/stores/system.js'
|
||||
|
||||
// Which build am I running? Milestone 318 stopped publishing version image
|
||||
// tags, so the instance's own report is the ONLY answer — there is no registry
|
||||
// name left to check it against. That promotes this from a convenience to the
|
||||
// mechanism, and it means the three states below have to stay distinct: a
|
||||
// wrong answer here has nothing to contradict it.
|
||||
//
|
||||
// not asked yet -> render nothing
|
||||
// asked, no version -> render "unknown"
|
||||
// asked, has a version -> render it
|
||||
//
|
||||
// Collapsing the first two would show "unknown" during every page load, and
|
||||
// collapsing either into a blank would read as "no version", which is a
|
||||
// different and false claim.
|
||||
|
||||
function stubHealth(body, { fail = false } = {}) {
|
||||
globalThis.fetch = vi.fn(async () => {
|
||||
if (fail) throw new Error('network down')
|
||||
return {
|
||||
ok: true, status: 200, statusText: '200',
|
||||
text: async () => JSON.stringify(body),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
describe('system store — build identity', () => {
|
||||
beforeEach(() => setActivePinia(createPinia()))
|
||||
afterEach(() => { vi.restoreAllMocks(); delete globalThis.fetch })
|
||||
|
||||
it('starts having asked nothing, so the footer renders nothing', () => {
|
||||
const s = useSystemStore()
|
||||
expect(s.buildLoaded).toBe(false)
|
||||
})
|
||||
|
||||
it('reports the version and channel the instance claims', async () => {
|
||||
stubHealth({ status: 'ok', version: '2026.08.28.1249', channel: 'dev' })
|
||||
const s = useSystemStore()
|
||||
await s.refreshHealth()
|
||||
|
||||
expect(s.buildLoaded).toBe(true)
|
||||
expect(s.buildVersion).toBe('2026.08.28.1249')
|
||||
expect(s.buildChannel).toBe('dev')
|
||||
})
|
||||
|
||||
it('keeps the channel OUT of the version string', async () => {
|
||||
// The tempting shortcut is a `-dev` suffix. The extension's comparator
|
||||
// parses each dotted segment with parseInt, so a suffixed segment reads as
|
||||
// 0 and every dev build compares equal to every other — #2993 exactly
|
||||
// (rule 149). If anyone ever "simplifies" by folding them together, the
|
||||
// version stops being the bare derived number and this fails.
|
||||
stubHealth({ status: 'ok', version: '2026.08.28.1249', channel: 'dev' })
|
||||
const s = useSystemStore()
|
||||
await s.refreshHealth()
|
||||
|
||||
expect(s.buildVersion).toBe('2026.08.28.1249')
|
||||
expect(s.buildVersion).not.toContain('dev')
|
||||
})
|
||||
|
||||
it('treats an absent version as "cannot say", not as a value', async () => {
|
||||
// A locally-built image, or one predating the field. The server omits the
|
||||
// key rather than sending an empty string; `?? ''` must preserve that
|
||||
// rather than inventing something. The view renders "unknown" from it.
|
||||
stubHealth({ status: 'ok' })
|
||||
const s = useSystemStore()
|
||||
await s.refreshHealth()
|
||||
|
||||
expect(s.buildLoaded).toBe(true)
|
||||
expect(s.buildVersion).toBe('')
|
||||
expect(s.buildChannel).toBe('')
|
||||
})
|
||||
|
||||
it('reports a version with no channel without inventing one', async () => {
|
||||
stubHealth({ status: 'ok', version: '2026.08.28.1249' })
|
||||
const s = useSystemStore()
|
||||
await s.refreshHealth()
|
||||
|
||||
expect(s.buildVersion).toBe('2026.08.28.1249')
|
||||
expect(s.buildChannel).toBe('')
|
||||
})
|
||||
|
||||
it('does not claim "unknown" when the health call itself failed', async () => {
|
||||
// A network blip says nothing about the build. Marking it loaded here
|
||||
// would present a transient failure as a defective image — and since
|
||||
// nothing else names the build, there would be no second source to
|
||||
// correct the impression.
|
||||
stubHealth(null, { fail: true })
|
||||
const s = useSystemStore()
|
||||
await s.refreshHealth()
|
||||
|
||||
expect(s.healthy).toBe(false)
|
||||
expect(s.buildLoaded).toBe(false)
|
||||
})
|
||||
})
|
||||
Executable
+201
@@ -0,0 +1,201 @@
|
||||
#!/bin/sh
|
||||
# Single definition of WHAT EACH PUBLISHED ARTIFACT IS BUILT FROM, and the
|
||||
# version derived from it. Milestone 313; generalises the shape
|
||||
# extension/scripts/packaging.sh established for the extension alone.
|
||||
#
|
||||
# "Built from" is deliberately wider than "copied into". A file that DECIDES an
|
||||
# artifact's identity is part of what that artifact is built from even though it
|
||||
# never reaches the image — see DERIVER below, and #3156 for the same finding
|
||||
# about packaging.sh.
|
||||
#
|
||||
# Four artifacts, four independent versions. An artifact whose shipped files
|
||||
# did not change keeps its version and does not rebuild — that is the whole
|
||||
# point, and it is why each path set must match its Dockerfile rather than
|
||||
# being a plausible guess. Getting a set wrong is quiet in BOTH directions:
|
||||
#
|
||||
# too narrow -> a pin serves stale bytes, because the version did not move
|
||||
# when the content did. This is the dangerous one.
|
||||
# too wide -> the artifact re-versions and rebuilds for a change it does
|
||||
# not ship. Merely wasteful.
|
||||
#
|
||||
# tests/test_artifact_paths.py asserts every COPY source in each Dockerfile is
|
||||
# covered here, so adding a COPY without updating this file fails CI.
|
||||
#
|
||||
# POSIX sh only — CI's run shell is busybox on some paths.
|
||||
#
|
||||
# -f (no pathname expansion) is load-bearing for the whole script: the lists
|
||||
# below are iterated with deliberate word-splitting, and without it the shell
|
||||
# would glob `frontend/test/**` against the working tree and silently narrow
|
||||
# the pattern. Callers substituting the output need their own `set -f` too;
|
||||
# the two guards protect different expansions.
|
||||
set -euf
|
||||
|
||||
ROOT=$(git rev-parse --show-toplevel)
|
||||
|
||||
# --- what each artifact ships ------------------------------------------------
|
||||
#
|
||||
# Each set includes its own Dockerfile and requirements: changing a base image
|
||||
# or a pin changes the artifact just as surely as changing a source file.
|
||||
#
|
||||
# web (Dockerfile, context `.`) — the runtime stage copies backend/, alembic/,
|
||||
# alembic.ini, entrypoint.sh and requirements.txt; the frontend-builder stage
|
||||
# copies frontend/ and the runtime takes its `dist` output.
|
||||
#
|
||||
# frontend/test is excluded: `npm run build` is vite, which builds from src/,
|
||||
# index.html and public/ and never reads test/. It lands in the builder layer
|
||||
# but not in `dist`, so it cannot reach the shipped image.
|
||||
#
|
||||
# The web image ALSO bundles the signed XPI (build.yml downloads it into
|
||||
# frontend/public/extension/ before the docker build), so an extension change
|
||||
# changes the web image. The extension's packaged set is appended in cmd_paths
|
||||
# rather than restated — one definition, per #2397.
|
||||
WEB_PATHS='Dockerfile requirements.txt backend alembic alembic.ini entrypoint.sh frontend :(exclude)frontend/test :(exclude)frontend/test/**'
|
||||
|
||||
# ml (Dockerfile.ml, context `.`) — no frontend, no extension. Note it copies
|
||||
# BOTH requirements-ml.txt and requirements.txt.
|
||||
ML_PATHS='Dockerfile.ml requirements-ml.txt requirements.txt backend alembic alembic.ini entrypoint.sh'
|
||||
|
||||
# agent (agent/Dockerfile, context `agent`) — copies requirements.txt and
|
||||
# fc_agent only. agent/README.md, agent/docker-compose.yml and agent/ruff.toml
|
||||
# live in the directory but never reach the image, so they must not re-version
|
||||
# it: this is deliberately NOT `agent/`.
|
||||
AGENT_PATHS='agent/Dockerfile agent/requirements.txt agent/fc_agent'
|
||||
|
||||
# This file. It is copied into no image and it is still part of what the web
|
||||
# image is built from, because it DECIDES the FC_VERSION baked into that image
|
||||
# (#3202). Same finding as #3156 about packaging.sh, one level up.
|
||||
#
|
||||
# Why web and nothing else. Every artifact stamps `fc.revision`, but only web
|
||||
# also stamps a version (build.yml line ~488 feeds `version web` to the
|
||||
# FC_VERSION build arg; ml and agent ask for `revision` alone, and the
|
||||
# extension takes its version from packaging.sh). For a revision-only artifact
|
||||
# this file needs no entry: any change to how the revision is COMPUTED changes
|
||||
# the derived value, which then disagrees with the label on the published image
|
||||
# and forces a rebuild. That mechanism is self-correcting because it compares
|
||||
# against a string stamped into a real artifact.
|
||||
#
|
||||
# The version is compared against nothing, so it has no such backstop. Before
|
||||
# this entry, a change to cmd_version alone left every artifact's revision
|
||||
# untouched, the reuse check hit, the build was skipped, and the published
|
||||
# image went on reporting the OLD version format — silently, until some
|
||||
# unrelated commit happened to force a rebuild. Milestone 318 step 5 is the
|
||||
# worked instance: b3989d0 and 5771fd5 share revision fb2c4d5b80be while the
|
||||
# version moved 2026.8.28.1249 -> 2026.08.28.1249. It cost nothing only because
|
||||
# FC_VERSION did not exist until one commit later.
|
||||
#
|
||||
# Named as a file, not as `scripts`: release_notes.py lives beside it and only
|
||||
# READS derived values, so it decides nothing and must not re-version anything.
|
||||
# A future script that derives an identity belongs here explicitly.
|
||||
DERIVER='scripts/artifacts.sh'
|
||||
|
||||
|
||||
usage() {
|
||||
echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# The extension's packaged set, read from its own definition rather than
|
||||
# copied. packaging.sh emits `:(exclude)extension/...` entries, so the bare
|
||||
# `extension` include has to come with them.
|
||||
ext_paths() {
|
||||
echo "extension $(sh "$ROOT/extension/scripts/packaging.sh" pathspec)"
|
||||
}
|
||||
|
||||
cmd_paths() {
|
||||
case "$1" in
|
||||
web) echo "$WEB_PATHS $DERIVER $(ext_paths)" ;;
|
||||
ml) echo "$ML_PATHS" ;;
|
||||
agent) echo "$AGENT_PATHS" ;;
|
||||
extension) ext_paths ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# "<unix ts> <sha>" of the newest commit touching this artifact's shipped set.
|
||||
# Unquoted on purpose: the pathspec must word-split into separate args.
|
||||
# Globbing is already off script-wide.
|
||||
newest() {
|
||||
# shellcheck disable=SC2046
|
||||
set -- "$(cd "$ROOT" && git log --format='%ct %H' HEAD -- $(cmd_paths "$1") \
|
||||
| sort -n | tail -1)"
|
||||
if [ -z "$1" ]; then
|
||||
echo "artifacts.sh: no commit touches this artifact's shipped files" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "$1"
|
||||
}
|
||||
|
||||
# Formatted through git rather than date(1): busybox date does not reliably
|
||||
# accept `-d @<epoch>`, and git's own --date=format-local is available wherever
|
||||
# git is. TZ=UTC so the value does not depend on the runner's timezone.
|
||||
fmt() {
|
||||
(cd "$ROOT" && TZ=UTC git show -s --format=%cd --date="format-local:$2" "$1")
|
||||
}
|
||||
|
||||
# The IDENTITY of an artifact's content: the commit its shipped files last
|
||||
# changed in. This is what decides whether a build can be skipped.
|
||||
#
|
||||
# It is published as the `fc.revision` LABEL on the image itself, and read
|
||||
# back off the moving channel tag — not as a tag of its own (milestone 318
|
||||
# step 3). A tag would be a name minted per build that only one thing reads,
|
||||
# which is what rule 145 narrowed against; it would also be prunable under the
|
||||
# registry's keep_pattern (#3157), so the cache would silently expire.
|
||||
#
|
||||
# A published image with no such label reads as a MISS and rebuilds. That is
|
||||
# the migration path, not a fault: `imagetools create` copies a manifest and
|
||||
# config labels are not manifest annotations, so the reuse path cannot stamp
|
||||
# one and there is nothing to backfill. Each artifact pays one rebuild, once.
|
||||
cmd_revision() {
|
||||
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
||||
}
|
||||
|
||||
# The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
|
||||
# family (note #3127 §1, rule 148) — the number an instance reports about
|
||||
# itself, and, with a `v` in front, the release tag naming the same build.
|
||||
#
|
||||
# Zero-padded since 2026-08-28. This stripped leading zeros until then, on the
|
||||
# reasoning that every segment should read as a plain integer — which never
|
||||
# held, since comparison strips them on parse anyway. Padding costs nothing,
|
||||
# sorts lexically as well as numerically, and keeps this project emitting the
|
||||
# same string as its siblings: unpadded, a `2026.8.28.1432` here sits beside a
|
||||
# `2026.08.28.1432` there, two shapes one character apart. Two obviously
|
||||
# different formats are safer than two nearly identical ones.
|
||||
#
|
||||
# Comparison is numeric per dot-segment, so `08` and `8` are equal and nothing
|
||||
# already published is reordered by the change.
|
||||
#
|
||||
# HHMM is not decoration: it is what makes the value unique per build with no
|
||||
# lookup. A date alone collides on the second build of a day, and resolving
|
||||
# that needs a `.N` suffix, which needs asking the registry what already
|
||||
# exists — at which point two lanes derive different answers for one source
|
||||
# and the shared-signature property is lost.
|
||||
cmd_version() {
|
||||
# The extension is the one artifact this script does not FORMAT, only route.
|
||||
# AMO's version grammar forbids leading zeros, so the extension emits the
|
||||
# same numbers unpadded (#3138) — a rendering exception, documented in
|
||||
# packaging.sh beside the signing step that has to obey it. Delegating keeps
|
||||
# one answer per artifact: `artifacts.sh version extension` and
|
||||
# `packaging.sh version` cannot drift into two.
|
||||
#
|
||||
# The direction is deliberate. artifacts.sh already asks packaging.sh for the
|
||||
# extension's PATH SET (ext_paths above), so the version has to flow the same
|
||||
# way; reversing it would have packaging.sh call back into this script, which
|
||||
# would call packaging.sh for the paths again.
|
||||
if [ "$1" = extension ]; then
|
||||
sh "$ROOT/extension/scripts/packaging.sh" version
|
||||
return
|
||||
fi
|
||||
sha=$(echo "$(newest "$1")" | cut -d' ' -f2)
|
||||
# One git call for the whole string rather than four and a sed. git's
|
||||
# format-local takes the complete format, and doing it in pieces was only
|
||||
# ever there to strip the padding between them.
|
||||
fmt "$sha" '%Y.%m.%d.%H%M'
|
||||
}
|
||||
|
||||
[ $# -ge 2 ] || usage
|
||||
case "$1" in
|
||||
paths) cmd_paths "$2" ;;
|
||||
revision) cmd_revision "$2" ;;
|
||||
version) cmd_version "$2" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Publish a Forgejo release whose body is derived from git, not written by hand.
|
||||
|
||||
Milestone 318 step 2 took the build consequence away from a `v*` tag: `main`
|
||||
has already built and published the commit by the time anyone tags it, and
|
||||
rebuilding would re-push `:c-<sha>`, which rule 145 forbids even when the bytes
|
||||
match. That left the tag with nothing to do. This gives it the job it has left.
|
||||
|
||||
**The half of the question a version string cannot answer.** Step 6 puts
|
||||
`2026.08.28.2208` in the Settings footer, so an operator can say which build
|
||||
they are running. They still cannot say what is in it that was not in the one
|
||||
they ran last month. A dated release carrying the commits since the previous
|
||||
one is the object that interprets the identifier (note #3127 §5).
|
||||
|
||||
**Derived, so it cannot drift.** The alternative is a hand-maintained
|
||||
`CHANGELOG.md`, which goes aspirational the first time someone forgets — and
|
||||
nothing ever catches it, because there is no second source to disagree with.
|
||||
Every line below comes out of `git log` at publish time.
|
||||
|
||||
**Optional by construction.** Release tags are bookmarks: cut one when you will
|
||||
want to point at that day by name, otherwise don't. FC went twelve weeks
|
||||
without one and nothing was wrong (note #3127 §0). This runs on a tag push and
|
||||
on nothing else — deliberately no schedule and no auto-tag on merge, either of
|
||||
which would turn an optional bookmark back into ceremony.
|
||||
|
||||
## Finding the previous release
|
||||
|
||||
`git describe --exclude <this tag>`, which walks ANCESTRY, not a sorted list.
|
||||
That is not fussiness: this repo's existing tags are the old `v26.05.22.0`
|
||||
shape and the next one will be rule 148's `v2026.08.28.2208`. Lexicographically
|
||||
`v2026...` sorts BEFORE `v26...` — every release from here on would report its
|
||||
predecessor as itself-or-nothing and emit a changelog covering the entire
|
||||
history. Ancestry is immune to the shape change, and it is also the more honest
|
||||
question: "what is in this that was not in the last one" IS a reachability
|
||||
question.
|
||||
|
||||
## Re-runs update, they do not fall through
|
||||
|
||||
Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
|
||||
rewrites the body, so a re-run silently keeps the first version. Harmless for a
|
||||
`v*` tag created once — and wrong the moment anything re-points. This one GETs
|
||||
first and PATCHes when the release exists, so it is correct either way rather
|
||||
than correct by luck (ThoughtSync #2182 is the same bug).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
API = "https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator"
|
||||
|
||||
IMAGES = (
|
||||
"git.fabledsword.com/bvandeusen/fabledcurator",
|
||||
"git.fabledsword.com/bvandeusen/fabledcurator-ml",
|
||||
"git.fabledsword.com/bvandeusen/fabledcurator-agent",
|
||||
)
|
||||
|
||||
# Rule 148: `v` + the artifact's own version, zero-padded, no `.N`, no lookup.
|
||||
RULE_148 = re.compile(r"^v\d{4}\.\d{2}\.\d{2}\.\d{4}$")
|
||||
|
||||
# Past this, the list has stopped being something anyone reads. It is reached
|
||||
# in exactly one situation — no previous tag is reachable, so the span is the
|
||||
# whole history — which happens on a genuine first release and on a tag cut
|
||||
# somewhere `main`'s tags cannot be seen from. Truncating says so; emitting
|
||||
# 1100 lines would bury the note explaining why there are 1100 of them.
|
||||
MAX_COMMITS = 200
|
||||
|
||||
|
||||
def git(*args: str) -> str:
|
||||
return subprocess.run(
|
||||
["git", *args], capture_output=True, text=True, check=True
|
||||
).stdout.strip()
|
||||
|
||||
|
||||
def git_ok(*args: str) -> str | None:
|
||||
"""Run git, returning None instead of raising when it fails.
|
||||
|
||||
Used for the questions that legitimately have no answer — no previous tag,
|
||||
no local `main` — where the absence is information rather than a fault.
|
||||
"""
|
||||
try:
|
||||
return git(*args)
|
||||
except subprocess.CalledProcessError:
|
||||
return None
|
||||
|
||||
|
||||
def previous_tag(ref: str, tag: str | None) -> str | None:
|
||||
"""The most recent `v*` tag reachable from `ref`, excluding `tag` itself.
|
||||
|
||||
`--exclude` rather than `<ref>^` so this is the same call whether or not
|
||||
`ref` is the tag being released — and so it does not blow up on a root
|
||||
commit that has no parent to walk to.
|
||||
"""
|
||||
args = ["describe", "--tags", "--abbrev=0", "--match", "v*"]
|
||||
if tag:
|
||||
args += ["--exclude", tag]
|
||||
return git_ok(*args, ref)
|
||||
|
||||
|
||||
def commits(previous: str | None, ref: str) -> list[str]:
|
||||
"""The subjects between the previous release and this one.
|
||||
|
||||
`--no-merges` because rule 153 merges `dev` into `main` with a plain merge
|
||||
commit, so `main`'s first-parent view is a list of "Merge pull request #N"
|
||||
and nothing else. The work is in the commits under those merges.
|
||||
"""
|
||||
span = f"{previous}..{ref}" if previous else ref
|
||||
out = git("log", "--no-merges", "--format=%s (%h)", span)
|
||||
return [line for line in out.split("\n") if line.strip()]
|
||||
|
||||
|
||||
def truncate(log: list[str]) -> tuple[list[str], str | None]:
|
||||
if len(log) <= MAX_COMMITS:
|
||||
return log, None
|
||||
return log[:MAX_COMMITS], (
|
||||
f"{len(log)} commits in this span — more than a changelog is for. "
|
||||
f"Listing the newest {MAX_COMMITS}. This usually means no previous "
|
||||
f"`v*` tag was reachable from here."
|
||||
)
|
||||
|
||||
|
||||
def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str:
|
||||
short = sha[:7]
|
||||
parts = []
|
||||
|
||||
if notes:
|
||||
# Anything the derivation could not stand behind goes at the TOP, not
|
||||
# in a footnote. A release that quietly names a build nobody can find
|
||||
# is the failure this whole milestone is about.
|
||||
parts.append("\n".join(f"> **Note:** {n}" for n in notes))
|
||||
|
||||
parts.append(
|
||||
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
|
||||
f"(rule 145) — these three move together:\n\n```\n"
|
||||
+ "\n".join(f"{image}:c-{short}" for image in IMAGES)
|
||||
+ "\n```"
|
||||
)
|
||||
|
||||
heading = f"## Changes since {previous}" if previous else "## Changes"
|
||||
if log:
|
||||
parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
|
||||
else:
|
||||
parts.append(
|
||||
heading
|
||||
+ "\n\n_No non-merge commits since the previous release. This tag "
|
||||
"names the same source under a new name._"
|
||||
)
|
||||
|
||||
span = f"{previous}..{tag}" if previous else tag
|
||||
parts.append(
|
||||
f"---\n\n_Derived at publish time from `git log --no-merges {span}`. "
|
||||
f"Nothing here is hand-maintained._"
|
||||
)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def cross_checks(tag: str, sha: str) -> list[str]:
|
||||
"""Everything the derivation knows that would make the release a lie.
|
||||
|
||||
Reported rather than enforced. The tag is already pushed by the time this
|
||||
runs, so failing here would leave the operator with a tag and no release
|
||||
and nothing but a red lane to explain it — while the release itself is
|
||||
still the useful object. Say what is wrong, on the release, and publish.
|
||||
"""
|
||||
notes = []
|
||||
|
||||
if not RULE_148.match(tag):
|
||||
notes.append(
|
||||
f"`{tag}` is not rule 148's `vYYYY.MM.DD.HHMM` shape. Published "
|
||||
f"anyway — the old `v26.*` tags predate the rule."
|
||||
)
|
||||
else:
|
||||
derived = artifact_version("web")
|
||||
if derived and derived != tag[1:]:
|
||||
notes.append(
|
||||
f"This tag names `{tag[1:]}`, but the web image built from "
|
||||
f"`{sha[:7]}` reports `{derived}`. The Settings footer will not "
|
||||
f"match this release's name."
|
||||
)
|
||||
|
||||
# `:c-<sha>` only exists if `main` built this commit. Checking costs one
|
||||
# git call; claiming it without checking costs a rollback that 404s at the
|
||||
# moment someone needs it.
|
||||
main = git_ok("rev-parse", "--verify", "-q", "refs/remotes/origin/main")
|
||||
if main is None:
|
||||
notes.append(
|
||||
"Could not resolve `origin/main` here, so the `:c-` tags above are "
|
||||
"unverified — they exist only if `main` built this commit."
|
||||
)
|
||||
elif subprocess.run(
|
||||
["git", "merge-base", "--is-ancestor", sha, main], capture_output=True
|
||||
).returncode != 0:
|
||||
notes.append(
|
||||
f"`{sha[:7]}` is not on `main`, so no `:c-{sha[:7]}` images were "
|
||||
f"ever published. The refs above will not pull."
|
||||
)
|
||||
|
||||
return notes
|
||||
|
||||
|
||||
def artifact_version(artifact: str) -> str | None:
|
||||
"""What `artifacts.sh` derives for one artifact in the CURRENT checkout.
|
||||
|
||||
It takes no ref because `artifacts.sh` takes none — it walks history from
|
||||
HEAD. That is right here only because a tag push checks out the tagged
|
||||
commit; calling this after `--dry-run some-other-ref` would compare the
|
||||
tag against the working tree, which is why the mismatch note below is
|
||||
reported and not enforced.
|
||||
|
||||
Returns None rather than raising if the script is missing or unhappy: a
|
||||
cross-check that cannot run should not take the release down with it.
|
||||
"""
|
||||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
return subprocess.run(
|
||||
["sh", os.path.join(root, "scripts", "artifacts.sh"), "version", artifact],
|
||||
capture_output=True, text=True, check=True, cwd=root,
|
||||
).stdout.strip()
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def api(method: str, path: str, token: str, payload: dict | None = None) -> dict | None:
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
req = urllib.request.Request(
|
||||
API + path, data=body, method=method,
|
||||
headers={
|
||||
"Authorization": "token " + token,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
return json.load(resp)
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
return None
|
||||
sys.exit(f"release: {method} {path} failed with HTTP {exc.code}: {exc.read()!r}")
|
||||
|
||||
|
||||
def publish(tag: str, name: str, body: str, token: str) -> None:
|
||||
existing = api("GET", f"/releases/tags/{tag}", token)
|
||||
if existing:
|
||||
api("PATCH", f"/releases/{existing['id']}", token, {"name": name, "body": body})
|
||||
print(f"release: updated existing release {existing['id']} for {tag}")
|
||||
else:
|
||||
api("POST", "/releases", token, {"tag_name": tag, "name": name, "body": body})
|
||||
print(f"release: created release for {tag}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument(
|
||||
"ref", nargs="?", default=None,
|
||||
help="tag or commit to release. Defaults to GITHUB_REF's tag, else HEAD.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--dry-run", action="store_true",
|
||||
help="render the body to stdout and publish nothing. Needs no token, "
|
||||
"so it also works as a preview before you decide to cut the tag.",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
github_ref = os.environ.get("GITHUB_REF", "")
|
||||
if args.ref:
|
||||
ref = args.ref
|
||||
elif github_ref.startswith("refs/tags/"):
|
||||
ref = github_ref[len("refs/tags/"):]
|
||||
else:
|
||||
ref = "HEAD"
|
||||
|
||||
# A tag only if git knows it as one — `HEAD` and a raw sha are refs to
|
||||
# release FROM, never the name to exclude or to publish under.
|
||||
tag = ref if git_ok("rev-parse", "--verify", "-q", f"refs/tags/{ref}") else None
|
||||
sha = git("rev-parse", ref)
|
||||
previous = previous_tag(ref, tag)
|
||||
|
||||
print(f"release: ref={ref} sha={sha[:12]} previous={previous or '<none>'}")
|
||||
|
||||
notes = cross_checks(tag, sha) if tag else [
|
||||
f"Rendered for `{ref}`, which is not a tag. Nothing was published."
|
||||
]
|
||||
for note in notes:
|
||||
print(f"release: NOTE {note}")
|
||||
|
||||
log = commits(previous, ref)
|
||||
print(f"release: {len(log)} non-merge commits in the span")
|
||||
log, overflow = truncate(log)
|
||||
if overflow:
|
||||
print(f"release: NOTE {overflow}")
|
||||
notes.append(overflow)
|
||||
body = render(tag or ref, sha, previous, log, notes)
|
||||
|
||||
if args.dry_run or not tag:
|
||||
print("--- body ---")
|
||||
print(body)
|
||||
return
|
||||
|
||||
token = os.environ.get("RELEASE_TOKEN") or os.environ.get("TOKEN")
|
||||
if not token:
|
||||
sys.exit("release: no RELEASE_TOKEN in the environment")
|
||||
publish(tag, f"FabledCurator {tag[1:]}", body, token)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -677,3 +677,34 @@ async def test_reset_content_tagging_apply_requires_confirm_token(client, db):
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert (await resp.get_json())["deleted"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_reclaim_attachments_defaults_to_preview(client, monkeypatch):
|
||||
"""Unlike the other maintenance triggers, this one's apply unlinks FILES —
|
||||
so an empty body must mean preview, not apply."""
|
||||
from backend.app.tasks import admin as admin_tasks
|
||||
|
||||
calls = []
|
||||
monkeypatch.setattr(
|
||||
admin_tasks.reclaim_orphaned_attachments_task, "delay", _fake_delay(calls)
|
||||
)
|
||||
resp = await client.post("/api/admin/maintenance/reclaim-attachments", json={})
|
||||
assert resp.status_code == 202
|
||||
assert (await resp.get_json())["task_id"] == "task-xyz"
|
||||
assert calls[0][1] == {"dry_run": True}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_reclaim_attachments_threads_apply(client, monkeypatch):
|
||||
from backend.app.tasks import admin as admin_tasks
|
||||
|
||||
calls = []
|
||||
monkeypatch.setattr(
|
||||
admin_tasks.reclaim_orphaned_attachments_task, "delay", _fake_delay(calls)
|
||||
)
|
||||
resp = await client.post(
|
||||
"/api/admin/maintenance/reclaim-attachments", json={"dry_run": False},
|
||||
)
|
||||
assert resp.status_code == 202
|
||||
assert calls[0][1] == {"dry_run": False}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import pytest
|
||||
|
||||
from backend.app.models import Artist, PostAttachment
|
||||
from backend.app.models import Artist, PostAttachment, attachment_download_url
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
@@ -33,3 +33,16 @@ async def test_download_streams_with_disposition(client, db, tmp_path):
|
||||
async def test_download_404(client):
|
||||
resp = await client.get("/api/attachments/999999/download")
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_attachment_download_url_routes_to_the_download_endpoint(app):
|
||||
"""The two serializers no longer hand-format this path (#3072) — but a
|
||||
single definition is only worth having if it still matches the route. Pin
|
||||
it by MATCHING against the real URL map rather than comparing to a literal:
|
||||
a string equality test would pass just as happily after someone renamed the
|
||||
route, which is the exact drift the helper exists to prevent."""
|
||||
built = attachment_download_url(4242)
|
||||
endpoint, args = app.url_map.bind("localhost").match(built)
|
||||
assert endpoint == "attachments.download"
|
||||
assert args == {"attachment_id": 4242}
|
||||
|
||||
@@ -129,7 +129,6 @@ async def test_resolve_artist_name_dispatches_per_platform(db, monkeypatch):
|
||||
("https://www.subscribestar.com/foobar", "subscribestar", "foobar"),
|
||||
("https://subscribestar.adult/foobar", "subscribestar", "foobar"),
|
||||
("https://www.hentai-foundry.com/user/Foo/profile", "hentaifoundry", "Foo"),
|
||||
("https://www.deviantart.com/baz", "deviantart", "baz"),
|
||||
("https://www.pixiv.net/users/12345", "pixiv", "12345"),
|
||||
("https://www.pixiv.net/en/users/12345", "pixiv", "12345"),
|
||||
])
|
||||
@@ -160,6 +159,23 @@ async def test_quick_add_source_unknown_url_400(client, ext_key):
|
||||
assert "known" in body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quick_add_source_rejects_retired_deviantart(client, ext_key):
|
||||
"""#3069: a DeviantArt creator URL used to derive cleanly. Now that the
|
||||
platform is retired, the extension's own gate should never offer the
|
||||
button — but a stale content script on an un-updated browser still can,
|
||||
so the backend has to refuse it rather than create an unusable source."""
|
||||
resp = await client.post(
|
||||
"/api/extension/quick-add-source",
|
||||
json={"url": "https://www.deviantart.com/baz"},
|
||||
headers={"X-Extension-Key": ext_key},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
body = await resp.get_json()
|
||||
assert body["error"] == "unknown_platform"
|
||||
assert "deviantart" not in body["known"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quick_add_source_invalid_url_400(client, ext_key):
|
||||
resp = await client.post(
|
||||
@@ -367,6 +383,53 @@ async def test_extension_manifest_returns_metadata_when_xpi_present(client, monk
|
||||
assert body["sha256"] == hashlib.sha256(b"fake-xpi-content").hexdigest()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extension_manifest_reports_the_channel_the_image_declares(
|
||||
client, monkeypatch, tmp_path
|
||||
):
|
||||
"""The channel travels BESIDE the version, never inside it.
|
||||
|
||||
Folding it in as a `1.0.3499884-dev` suffix is the failure this design
|
||||
exists to avoid: the extension's comparator parses each dotted segment as
|
||||
an integer, so a suffixed segment collapses to 0 and every dev build
|
||||
compares equal to every other — "no update available" and "I cannot read
|
||||
this version" stop being distinguishable. Asserting the two are separate
|
||||
keys is what keeps a future edit from merging them.
|
||||
"""
|
||||
(tmp_path / "fabledcurator-1.2.3.xpi").write_bytes(b"x")
|
||||
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
|
||||
monkeypatch.setattr(extension_module, "FC_CHANNEL", "dev")
|
||||
resp = await client.get("/api/extension/manifest")
|
||||
assert resp.status_code == 200
|
||||
body = await resp.get_json()
|
||||
assert body["channel"] == "dev"
|
||||
assert body["version"] == "1.2.3"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extension_manifest_omits_the_channel_when_the_image_declares_none(
|
||||
client, monkeypatch, tmp_path
|
||||
):
|
||||
"""A local build, or any image from before the field existed.
|
||||
|
||||
The key must be ABSENT rather than present-and-empty: absence is the state
|
||||
every consumer already handles (an older image conveys it by not having the
|
||||
key at all), so a blank channel reuses that path instead of introducing a
|
||||
second spelling of "unknown" for each reader to special-case.
|
||||
"""
|
||||
(tmp_path / "fabledcurator-1.2.3.xpi").write_bytes(b"x")
|
||||
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
|
||||
monkeypatch.setattr(extension_module, "FC_CHANNEL", "")
|
||||
resp = await client.get("/api/extension/manifest")
|
||||
assert resp.status_code == 200
|
||||
body = await resp.get_json()
|
||||
assert "channel" not in body
|
||||
# Everything else still answers — an image with no channel is not a
|
||||
# degraded one, it just cannot say which channel it came from.
|
||||
assert body["installed"] is True
|
||||
assert body["latest_url"] == "/extension/fabledcurator-latest.xpi"
|
||||
|
||||
|
||||
# --- /extension/<filename> -----------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -6,16 +6,17 @@ pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_platforms_returns_gs_six(client):
|
||||
async def test_platforms_returns_gs_five(client):
|
||||
resp = await client.get("/api/platforms")
|
||||
assert resp.status_code == 200
|
||||
body = await resp.get_json()
|
||||
platforms = body["platforms"]
|
||||
assert set(platforms.keys()) == {
|
||||
"patreon", "subscribestar", "hentaifoundry",
|
||||
"discord", "pixiv", "deviantart",
|
||||
"discord", "pixiv",
|
||||
}
|
||||
assert "fanbox" not in platforms
|
||||
assert "deviantart" not in platforms # retired at #3069
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
"""The two values `artifacts.sh` derives, and what each of them promises.
|
||||
|
||||
`revision` decides whether a build gets skipped; `version` is what an instance
|
||||
reports about itself and what a release tag is named after. Neither has a
|
||||
consumer that would notice it going subtly wrong.
|
||||
|
||||
## revision
|
||||
|
||||
Milestone 318 step 3: each image carries its revision as an `fc.revision`
|
||||
label, and build.yml reads that label back off the moving channel tag. Equal
|
||||
to the derived revision means the bytes this push would produce are already
|
||||
published, so the build is skipped.
|
||||
|
||||
That makes the revision load-bearing in a way a version string is not — it is
|
||||
compared for equality against a value stamped into a real published artifact.
|
||||
Both ways of getting it wrong are silent:
|
||||
|
||||
* **it does not identify the content** — a revision that moves when the source
|
||||
did not (a HEAD-derived value, say) never matches, nothing is ever skipped,
|
||||
and the mechanism quietly buys nothing while every lane stays green.
|
||||
* **it identifies the wrong content** — a revision that holds still when the
|
||||
source DID change matches a stale label, the build is skipped, and the
|
||||
channel serves bytes that do not correspond to the commit. This is the
|
||||
dangerous direction, and it is what `test_artifact_paths.py` guards from the
|
||||
other side by pinning the path sets.
|
||||
|
||||
This module owns the narrower claim: whatever the path sets say, the revision
|
||||
is genuinely the commit those paths last changed in.
|
||||
|
||||
## version
|
||||
|
||||
`YYYY.MM.DD.HHMM`, zero-padded, UTC — one shape across the family (note #3127
|
||||
§1, rule 148), so the string this project emits is the same string its siblings
|
||||
emit. Two nearly-identical formats are more dangerous than two obviously
|
||||
different ones, and the only thing keeping them identical is a test.
|
||||
|
||||
**The extension is the one exception, and it is a rendering exception only.**
|
||||
AMO's version grammar forbids a leading zero, so the extension emits the same
|
||||
numbers unpadded — `2026.8.29.201` where the family says `2026.08.29.0201`
|
||||
(#3138, milestone 318 step 8). Rule 148 defines comparison as numeric per
|
||||
dot-segment, under which the two are equal, so this is pinned in both
|
||||
directions below: the extension must satisfy AMO's grammar, and every artifact
|
||||
must derive the same NUMBERS its own commit stamps. An exception left as "the
|
||||
extension is different" would drift into being differently different.
|
||||
|
||||
The identity-TAG tests this file used to hold are gone with the tag. There is
|
||||
no longer a `CHANNELLED` list to drift (the channel is which tag you inspect),
|
||||
and no `identity` subcommand to refuse an unqualified call.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
ARTIFACTS = ("web", "ml", "agent", "extension")
|
||||
|
||||
# 12 hex chars — the prefix build.yml stamps and compares.
|
||||
_REVISION = re.compile(r"^[0-9a-f]{12}$")
|
||||
|
||||
# YYYY.MM.DD.HHMM, every segment zero-padded to its full width.
|
||||
_VERSION = re.compile(r"^\d{4}\.\d{2}\.\d{2}\.\d{4}$")
|
||||
|
||||
# The artifacts that cannot use the padded rendering. Exactly one, and the
|
||||
# reason is external: `packaging.sh` derives the extension's version and AMO
|
||||
# refuses to sign a padded one.
|
||||
AMO_UNPADDED = frozenset({"extension"})
|
||||
|
||||
# Mozilla's published grammar for addons.mozilla.org, transcribed from MDN's
|
||||
# manifest.json/version page. A segment is the single digit `0` or starts 1-9,
|
||||
# and there are at most four. This is the constraint the exception exists for,
|
||||
# so it is what the exception is tested against — `2026.08.29.0201` fails it.
|
||||
_AMO = re.compile(r"^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$")
|
||||
|
||||
# YYYY.M.D.HHMM — four segments, none of them zero-padded.
|
||||
_UNPADDED = re.compile(r"^\d{4}(\.(0|[1-9]\d*)){3}$")
|
||||
|
||||
|
||||
def segments(value: str) -> tuple[int, ...]:
|
||||
"""A version as the numbers it denotes, which is how rule 148 says to
|
||||
compare one. `2026.08.29.0201` and `2026.8.29.201` are one value here."""
|
||||
return tuple(int(part) for part in value.split("."))
|
||||
|
||||
|
||||
# Everything here goes through artifacts.sh rather than importing a sibling
|
||||
# test module. That is the interface build.yml actually calls, so the tests
|
||||
# exercise the contract instead of a Python re-implementation of it — and no
|
||||
# other test module in this repo imports another, so a cross-test import would
|
||||
# be a new convention introduced for no gain.
|
||||
def artifacts(*args: str) -> str:
|
||||
return subprocess.run(
|
||||
["sh", str(ROOT / "scripts" / "artifacts.sh"), *args],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
).stdout
|
||||
|
||||
|
||||
def revision(artifact: str) -> str:
|
||||
return artifacts("revision", artifact).strip()
|
||||
|
||||
|
||||
def newest_by_commit_time(artifact: str) -> str:
|
||||
"""The full SHA of the newest commit touching this artifact's shipped set.
|
||||
|
||||
Ordered by committer TIME, matching what artifacts.sh means. Deliberately
|
||||
not `git log -1`: git's default order is reverse-chronological only within
|
||||
topological constraints, so on a merged history it can name a different
|
||||
commit than the newest timestamp does. They agree on this repo today, and
|
||||
a test that silently depends on them continuing to agree would be a flake
|
||||
waiting for the branch shape that separates them.
|
||||
"""
|
||||
paths = artifacts("paths", artifact).split()
|
||||
log = subprocess.run(
|
||||
["git", "log", "--format=%ct %H", "HEAD", "--", *paths],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
).stdout.split("\n")
|
||||
commits = [line.split(" ", 1) for line in log if line.strip()]
|
||||
assert commits, (
|
||||
f"no commit in this history touches the {artifact} path set — the "
|
||||
f"derivation has nothing to stand on"
|
||||
)
|
||||
return max(commits, key=lambda c: int(c[0]))[1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||
def test_revision_is_the_commit_its_own_shipped_files_last_changed_in(artifact):
|
||||
"""The claim the whole skip decision rests on.
|
||||
|
||||
Computed from git rather than asked of the script, so it fails if the
|
||||
derivation ever stops meaning what it says — switching to HEAD, to a build
|
||||
clock, or to a path set it did not actually use. Each of those still
|
||||
produces a plausible 12-hex value, which is why this is worth asserting
|
||||
rather than eyeballing.
|
||||
"""
|
||||
expected = newest_by_commit_time(artifact)
|
||||
got = revision(artifact)
|
||||
assert expected.startswith(got), (
|
||||
f"{artifact} derives {got!r}, but the newest commit touching its "
|
||||
f"shipped files is {expected[:12]!r}. The label stamped into the image "
|
||||
f"would not identify its own content."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||
def test_revision_is_a_legal_label_value_and_is_stable(artifact):
|
||||
"""It is stamped as a docker label and compared for string equality, so a
|
||||
stray newline or a varying value breaks the comparison rather than the
|
||||
build — the mechanism would simply stop hitting, silently."""
|
||||
first = revision(artifact)
|
||||
assert _REVISION.match(first), f"{first!r} is not a 12-char hex revision"
|
||||
assert first == revision(artifact), "revision is not stable across calls"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", sorted(set(ARTIFACTS) - AMO_UNPADDED))
|
||||
def test_version_is_zero_padded_calver(artifact):
|
||||
"""The family shape, pinned.
|
||||
|
||||
Padding was stripped until 2026-08-28 on the reasoning that each segment
|
||||
should read as a plain integer — which never held, since comparison strips
|
||||
leading zeros on parse anyway. What it did do was make this project emit
|
||||
`2026.8.28.1432` while a sibling emitted `2026.08.28.1432`: two shapes one
|
||||
character apart, which is the hard kind of difference to notice.
|
||||
|
||||
Also catches the midnight case. A `%H%M` of `0322` must survive as `0322`;
|
||||
the old strip-leading-zeros helper turned it into `322`, silently changing
|
||||
a four-digit field into three.
|
||||
"""
|
||||
value = artifacts("version", artifact).strip()
|
||||
assert _VERSION.match(value), (
|
||||
f"{artifact} derives {value!r}, which is not zero-padded "
|
||||
f"YYYY.MM.DD.HHMM. Note #3127 §1 and rule 148 both specify the padded "
|
||||
f"form, and a release tag is this string with a `v` in front."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", sorted(AMO_UNPADDED))
|
||||
def test_the_unpadded_artifacts_derive_something_amo_will_sign(artifact):
|
||||
"""The other half of the family shape: the documented exception, tested
|
||||
against the constraint that justifies it rather than against itself.
|
||||
|
||||
A padded value passes `_UNPADDED` on any date with no leading zeros, so
|
||||
that pattern alone would let a regression sit unnoticed until the first
|
||||
single-digit month — at which point the failure is a burned AMO version,
|
||||
not a red lane. AMO's grammar is the assertion that fires immediately.
|
||||
"""
|
||||
value = artifacts("version", artifact).strip()
|
||||
assert _AMO.match(value), (
|
||||
f"{artifact} derives {value!r}, which AMO refuses: a segment must be "
|
||||
f"the single digit `0` or start 1-9, and there are at most four. "
|
||||
f"Almost certainly a zero-padded segment — the family pads and this "
|
||||
f"artifact must not (#3138). AMO 409s on re-signing, so a version it "
|
||||
f"rejects is burned."
|
||||
)
|
||||
assert _UNPADDED.match(value), (
|
||||
f"{artifact} derives {value!r}, which is not YYYY.M.D.HHMM. AMO would "
|
||||
f"also accept the pre-318 `1.0.<minutes>`, and that orders below every "
|
||||
f"ext-2026.* release already signed."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||
def test_version_and_revision_describe_the_same_commit(artifact):
|
||||
"""They are derived independently and must not be able to disagree.
|
||||
|
||||
A build reports the version and skips on the revision, so a divergence
|
||||
would mean an instance naming one commit while carrying another's bytes —
|
||||
unfalsifiable from outside, since both values look perfectly well-formed.
|
||||
"""
|
||||
sha = newest_by_commit_time(artifact)
|
||||
stamped = subprocess.run(
|
||||
["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
|
||||
).stdout.strip()
|
||||
derived = artifacts("version", artifact).strip()
|
||||
|
||||
# Compared as NUMBERS, which is how rule 148 defines comparison and the
|
||||
# only way one assertion can cover both renderings. This is what makes the
|
||||
# extension's exception cosmetic rather than semantic: it must denote
|
||||
# exactly the value its own commit stamps, whatever the padding.
|
||||
assert segments(derived) == segments(stamped), (
|
||||
f"{artifact} derives {derived!r}, but its newest shipped commit "
|
||||
f"{sha[:12]} is {stamped!r}. The instance would name one commit while "
|
||||
f"carrying another's bytes."
|
||||
)
|
||||
if artifact not in AMO_UNPADDED:
|
||||
assert derived == stamped, (
|
||||
f"{artifact} derives {derived!r} where the family shape is "
|
||||
f"{stamped!r} — same numbers, wrong rendering. Only the artifacts "
|
||||
f"in AMO_UNPADDED may differ here."
|
||||
)
|
||||
assert sha.startswith(revision(artifact))
|
||||
@@ -0,0 +1,199 @@
|
||||
"""`scripts/artifacts.sh` path sets must match what the Dockerfiles copy.
|
||||
|
||||
Each published artifact's version derives from the newest commit touching its
|
||||
own shipped file set (milestone 313). The whole scheme rests on those sets
|
||||
being right, and both ways of being wrong are silent:
|
||||
|
||||
* **too narrow** — a file ships but is not in the set, so the version does not
|
||||
move when the content does, and a pin serves stale bytes. This is the
|
||||
dangerous direction and the one this module exists for.
|
||||
* **too wide** — a file is in the set but never reaches the image, so the
|
||||
artifact re-versions and rebuilds for a change it does not ship.
|
||||
|
||||
Nothing else notices either. The version still derives, CI still goes green,
|
||||
and the mismatch only surfaces as "I pinned that build and got the wrong
|
||||
bytes". So the Dockerfiles are read here and compared against the declaration.
|
||||
|
||||
The COPY list is not the whole answer, though. A file that DECIDES what an
|
||||
artifact reports belongs in its set even though it is copied into nothing —
|
||||
see DERIVERS below, where the same finding is recorded twice (#3156, #3202).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
# artifact -> (dockerfile, build context relative to the repo root)
|
||||
ARTIFACTS = {
|
||||
"web": ("Dockerfile", ""),
|
||||
"ml": ("Dockerfile.ml", ""),
|
||||
"agent": ("agent/Dockerfile", "agent"),
|
||||
}
|
||||
|
||||
# COPY --from=<stage> copies from an earlier build stage, not from the build
|
||||
# context, so its source is not a repo path and cannot be in a path set.
|
||||
_COPY = re.compile(r"^\s*COPY\s+(?!--from=)(?P<args>.+)$", re.MULTILINE)
|
||||
|
||||
|
||||
def declared_paths(artifact: str) -> list[str]:
|
||||
out = subprocess.run(
|
||||
["sh", str(ROOT / "scripts" / "artifacts.sh"), "paths", artifact],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
).stdout
|
||||
return out.split()
|
||||
|
||||
|
||||
def includes(artifact: str) -> list[str]:
|
||||
"""The set minus its `:(exclude)…` entries."""
|
||||
return [p for p in declared_paths(artifact) if not p.startswith(":(exclude)")]
|
||||
|
||||
|
||||
def copy_sources(dockerfile: str, context: str) -> list[str]:
|
||||
"""Repo-relative sources of every context COPY in a Dockerfile."""
|
||||
text = (ROOT / dockerfile).read_text()
|
||||
sources: list[str] = []
|
||||
for m in _COPY.finditer(text):
|
||||
args = m.group("args").split()
|
||||
# Last arg is the destination; everything before it is a source.
|
||||
for src in args[:-1]:
|
||||
# `frontend/package-lock.json*` — the glob is an optional-file
|
||||
# idiom; the directory it sits in is what matters for coverage.
|
||||
src = src.rstrip("*")
|
||||
sources.append(f"{context}/{src}" if context else src)
|
||||
return sources
|
||||
|
||||
|
||||
def covered_by(path: str, include: str) -> bool:
|
||||
"""`path` ships if an include names it or one of its ancestors."""
|
||||
path = path.rstrip("/").lstrip("./")
|
||||
include = include.rstrip("/")
|
||||
return path == include or path.startswith(include + "/")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", sorted(ARTIFACTS))
|
||||
def test_every_copied_path_is_in_the_artifacts_path_set(artifact):
|
||||
"""The too-narrow direction — the one that serves stale bytes on a pin."""
|
||||
dockerfile, context = ARTIFACTS[artifact]
|
||||
inc = includes(artifact)
|
||||
for src in copy_sources(dockerfile, context):
|
||||
assert any(covered_by(src, i) for i in inc), (
|
||||
f"{dockerfile} copies {src!r} into the {artifact} image, but no "
|
||||
f"include in scripts/artifacts.sh covers it. The {artifact} "
|
||||
f"version will not move when that file changes, so a pinned build "
|
||||
f"will serve stale bytes. Add it to the path set.\n"
|
||||
f" declared includes: {inc}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", sorted(ARTIFACTS))
|
||||
def test_the_dockerfile_itself_is_in_the_path_set(artifact):
|
||||
"""Changing a base image or a RUN changes the artifact as surely as
|
||||
changing a source file, so each set must include its own Dockerfile."""
|
||||
dockerfile, _ = ARTIFACTS[artifact]
|
||||
assert any(covered_by(dockerfile, i) for i in includes(artifact)), (
|
||||
f"{dockerfile} is not in the {artifact} path set — a base-image bump "
|
||||
f"would not move the version."
|
||||
)
|
||||
|
||||
|
||||
def test_the_web_image_versions_on_an_extension_change():
|
||||
"""The web image bundles the signed XPI, so the extension's packaged files
|
||||
are part of what it ships. Miss this and `:latest` serves a NEW extension
|
||||
under an unchanged web version — a pin that quietly disagrees with itself.
|
||||
"""
|
||||
inc = includes("web")
|
||||
assert any(covered_by("extension/background/background.js", i) for i in inc), (
|
||||
"the web path set does not cover the extension's packaged files, but "
|
||||
"build.yml downloads the signed XPI into frontend/public/extension/ "
|
||||
"before the docker build"
|
||||
)
|
||||
|
||||
|
||||
# A file that DECIDES an artifact's identity is part of what that artifact is
|
||||
# built from, even though it is copied into no image. Both entries here are the
|
||||
# same finding twice — #3156 for packaging.sh, #3202 for artifacts.sh — and
|
||||
# both were latent for the same reason: the version has no backstop.
|
||||
#
|
||||
# The revision does. Change how a REVISION is computed and the derived value
|
||||
# stops matching the label on the published image, which forces a rebuild; the
|
||||
# mechanism self-corrects because it compares against a string stamped into a
|
||||
# real artifact. Nothing compares a version to anything, so a version-only
|
||||
# derivation change is invisible unless the deriver is in the set.
|
||||
DERIVERS = [
|
||||
# packaging.sh decides the version build.yml stamps into the packaged
|
||||
# manifest.json, so changing it changes the shipped bytes. Left out,
|
||||
# milestone 313 step 4 turns silent: the new version misses the
|
||||
# ext-<version> cache and gets signed, while web's revision has not moved,
|
||||
# so the reuse path republishes the old image and the fresh signature is
|
||||
# orphaned. Guarded for web too, since web bundles what the extension makes.
|
||||
("extension/scripts/packaging.sh", ("extension", "web")),
|
||||
# artifacts.sh decides the FC_VERSION baked into the web image (#3202).
|
||||
# Web only, and deliberately: ml and agent ask this script for `revision`
|
||||
# alone, so they are covered by the self-correcting path above, and the
|
||||
# extension takes its version from packaging.sh. Milestone 318 step 5 is
|
||||
# the worked instance — b3989d0 and 5771fd5 share revision fb2c4d5b80be
|
||||
# while the version moved 2026.8.28.1249 -> 2026.08.28.1249. It was
|
||||
# harmless only because FC_VERSION did not exist until one commit later.
|
||||
("scripts/artifacts.sh", ("web",)),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path, artifacts", DERIVERS, ids=lambda v: str(v))
|
||||
def test_a_version_deriver_is_in_the_set_of_what_it_decides(path, artifacts):
|
||||
for artifact in artifacts:
|
||||
inc = includes(artifact)
|
||||
excluded = [
|
||||
p[len(":(exclude)"):] for p in declared_paths(artifact)
|
||||
if p.startswith(":(exclude)")
|
||||
]
|
||||
assert any(covered_by(path, i) for i in inc), (
|
||||
f"{path} decides the version {artifact} reports, but is not in the "
|
||||
f"{artifact} path set. A change to the derivation would leave the "
|
||||
f"revision untouched, the build skipped, and the published image "
|
||||
f"reporting the old version — with nothing to disagree with it."
|
||||
)
|
||||
assert not any(
|
||||
covered_by(path, e.rstrip("*").rstrip("/")) for e in excluded
|
||||
), (
|
||||
f"{path} is excluded from the {artifact} path set, so a change to "
|
||||
f"how the version is derived would not move the version — and "
|
||||
f"step 4 would reuse the image that carries the old one"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"artifact, path",
|
||||
[
|
||||
# Deliberate exclusions — the too-wide direction. Each of these lives
|
||||
# beside shipped code but never reaches an image, and including it
|
||||
# would re-version the artifact for a change it does not carry.
|
||||
#
|
||||
# "Never reaches an image" is the test, not "is not source": DERIVERS
|
||||
# above are also copied into nothing and DO belong in their sets,
|
||||
# because they decide what the image reports. The line between the two
|
||||
# lists is whether the file has a say in the artifact's identity.
|
||||
("agent", "agent/README.md"),
|
||||
("agent", "agent/ruff.toml"),
|
||||
("agent", "agent/docker-compose.yml"),
|
||||
# vite builds from src/, index.html and public/; it never reads test/,
|
||||
# so a frontend test change cannot reach `dist`.
|
||||
("web", "frontend/test/gallery.spec.js"),
|
||||
],
|
||||
)
|
||||
def test_files_that_never_reach_an_image_do_not_version_it(artifact, path):
|
||||
paths = declared_paths(artifact)
|
||||
excluded = [p[len(":(exclude)"):] for p in paths if p.startswith(":(exclude)")]
|
||||
inc = [p for p in paths if not p.startswith(":(exclude)")]
|
||||
|
||||
included = any(covered_by(path, i) for i in inc)
|
||||
exempted = any(covered_by(path, e.rstrip("*").rstrip("/")) for e in excluded)
|
||||
assert not included or exempted, (
|
||||
f"{path} is in the {artifact} path set but is not copied into the "
|
||||
f"image — it would re-version and rebuild {artifact} for a change it "
|
||||
f"does not ship."
|
||||
)
|
||||
@@ -154,7 +154,7 @@ async def test_list_platform_filter_excludes_no_source(db):
|
||||
@pytest.mark.asyncio
|
||||
async def test_list_platform_filter_excludes_wrong_platform(db):
|
||||
a = await _seed_artist(db, "alice-wplat")
|
||||
await _seed_source(db, a.id, "deviantart", "https://d/alice-wp")
|
||||
await _seed_source(db, a.id, "discord", "https://d/alice-wp")
|
||||
await db.commit()
|
||||
|
||||
page = await ArtistDirectoryService(db).list_artists(
|
||||
|
||||
@@ -5,6 +5,7 @@ side effects use tmp_path. Assertions on mutated rows use COLUMN
|
||||
SELECTS per reference_async_coredml_test_assertions — never
|
||||
re-read ORM attributes after a service mutates and re-fetches.
|
||||
"""
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
@@ -68,6 +69,8 @@ def test_project_artist_cascade_returns_zeroes_for_empty_artist(db_sync):
|
||||
assert result["artist"]["slug"] == "empty"
|
||||
assert result["projected"] == {
|
||||
"images": 0,
|
||||
"posts": 0,
|
||||
"attachments": 0,
|
||||
"sources": 0,
|
||||
"thumbs": 0,
|
||||
"import_tasks": 0,
|
||||
@@ -92,6 +95,87 @@ def test_project_artist_cascade_counts_images_and_thumbs_and_bytes(db_sync, tmp_
|
||||
assert result["projected"]["bytes_on_disk"] == 3500
|
||||
|
||||
|
||||
def test_project_artist_cascade_counts_posts_and_attachments(db_sync, tmp_path):
|
||||
"""The body-only artist: zero images, but posts and attachments that the
|
||||
apply destroys. Previewing this as `images: 0` alone is what made a
|
||||
content-only artist read as an empty one (#3067)."""
|
||||
a = _make_artist(db_sync, slug="bodyonly")
|
||||
p1 = Post(artist_id=a.id, external_post_id="bo-1", description="a body")
|
||||
p2 = Post(artist_id=a.id, external_post_id="bo-2", description="another")
|
||||
db_sync.add_all([p1, p2])
|
||||
db_sync.flush()
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p1.id, artist_id=a.id, sha256="b0d1".ljust(64, "0"),
|
||||
path="/store/b0d1/a.pdf", original_filename="a.pdf",
|
||||
ext=".pdf", size_bytes=5,
|
||||
))
|
||||
# artist_id NULL, reachable only through its post — the second arm of
|
||||
# _artist_attachments_conditions.
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p2.id, artist_id=None, sha256="b0d2".ljust(64, "0"),
|
||||
path="/store/b0d2/b.pdf", original_filename="b.pdf",
|
||||
ext=".pdf", size_bytes=5,
|
||||
))
|
||||
db_sync.commit()
|
||||
|
||||
projected = cleanup_service.project_artist_cascade(
|
||||
db_sync, slug="bodyonly",
|
||||
)["projected"]
|
||||
assert projected["images"] == 0
|
||||
assert projected["posts"] == 2
|
||||
assert projected["attachments"] == 2
|
||||
|
||||
|
||||
def test_artist_cascade_preview_matches_apply(db_sync, tmp_path):
|
||||
"""Rule 93: the preview's numbers must be what the apply actually does.
|
||||
|
||||
Guards the drift directly rather than trusting that both halves happen to
|
||||
use the same predicate — the preview and apply are asserted against each
|
||||
other on one artist carrying all three row kinds.
|
||||
"""
|
||||
a = _make_artist(db_sync, slug="parity")
|
||||
for i in range(3):
|
||||
f = tmp_path / f"par{i}.jpg"
|
||||
f.write_bytes(b"x")
|
||||
_make_image(
|
||||
db_sync, artist=a, path=str(f), sha256=f"{i:064x}", size=10,
|
||||
)
|
||||
posts = [
|
||||
Post(artist_id=a.id, external_post_id=f"par-{i}") for i in range(4)
|
||||
]
|
||||
db_sync.add_all(posts)
|
||||
db_sync.flush()
|
||||
for i, p in enumerate(posts[:2]):
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p.id, artist_id=a.id, sha256=f"par{i}".ljust(64, "0"),
|
||||
path=f"/store/par{i}/f.zip", original_filename="f.zip",
|
||||
ext=".zip", size_bytes=9,
|
||||
))
|
||||
db_sync.commit()
|
||||
artist_id = a.id
|
||||
|
||||
projected = cleanup_service.project_artist_cascade(
|
||||
db_sync, slug="parity",
|
||||
)["projected"]
|
||||
summary = cleanup_service.delete_artist_cascade(
|
||||
db_sync, artist_id=artist_id, images_root=tmp_path,
|
||||
)["summary"]
|
||||
|
||||
assert projected["images"] == summary["images_deleted"] == 3
|
||||
assert projected["posts"] == summary["posts_deleted"] == 4
|
||||
assert projected["attachments"] == summary["attachments_deleted"] == 2
|
||||
|
||||
# And the apply really did remove them — a matching pair of numbers is
|
||||
# worth nothing if neither half touched the DB.
|
||||
assert db_sync.execute(
|
||||
select(func.count(Post.id)).where(Post.artist_id == artist_id)
|
||||
).scalar_one() == 0
|
||||
assert db_sync.execute(
|
||||
select(func.count(PostAttachment.id))
|
||||
.where(PostAttachment.artist_id == artist_id)
|
||||
).scalar_one() == 0
|
||||
|
||||
|
||||
def test_project_artist_cascade_raises_on_unknown_slug(db_sync):
|
||||
with pytest.raises(LookupError):
|
||||
cleanup_service.project_artist_cascade(db_sync, slug="nope")
|
||||
@@ -294,6 +378,88 @@ def test_delete_artist_cascade_idempotent_on_missing(db_sync, tmp_path):
|
||||
assert result["summary"]["images_deleted"] == 0
|
||||
|
||||
|
||||
def test_delete_artist_cascade_survives_same_sha_on_two_posts(db_sync, tmp_path):
|
||||
"""Same file attached to two of the artist's posts must not abort the delete.
|
||||
|
||||
Left to the cascade this raises: artist delete CASCADEs to Post, which SET
|
||||
NULLs post_attachment.post_id, and `uq_post_attachment_null_post_sha`
|
||||
(sha256 alone, WHERE post_id IS NULL) then rejects the second row. That's an
|
||||
ordinary shape — _capture_attachment writes one row per post over one
|
||||
sha-addressed blob by design. Also covers the NULL-artist_id arm of the
|
||||
delete predicate: the second row has no artist_id, only a post that does.
|
||||
"""
|
||||
a = _make_artist(db_sync, slug="casatt")
|
||||
p1 = Post(artist_id=a.id, external_post_id="att-p1")
|
||||
p2 = Post(artist_id=a.id, external_post_id="att-p2")
|
||||
db_sync.add_all([p1, p2])
|
||||
db_sync.flush()
|
||||
|
||||
shared_sha = "ca5a".ljust(64, "0")
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p1.id, artist_id=a.id, sha256=shared_sha,
|
||||
path="/store/ca5a/bundle.zip", original_filename="bundle.zip",
|
||||
ext=".zip", size_bytes=7,
|
||||
))
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p2.id, artist_id=None, sha256=shared_sha,
|
||||
path="/store/ca5a/bundle.zip", original_filename="bundle.zip",
|
||||
ext=".zip", size_bytes=7,
|
||||
))
|
||||
db_sync.commit()
|
||||
artist_id = a.id
|
||||
|
||||
result = cleanup_service.delete_artist_cascade(
|
||||
db_sync, artist_id=artist_id, images_root=tmp_path,
|
||||
)
|
||||
|
||||
assert result["summary"]["attachments_deleted"] == 2
|
||||
assert db_sync.execute(
|
||||
select(func.count(Artist.id)).where(Artist.id == artist_id)
|
||||
).scalar_one() == 0
|
||||
assert db_sync.execute(
|
||||
select(func.count(PostAttachment.id))
|
||||
.where(PostAttachment.sha256 == shared_sha)
|
||||
).scalar_one() == 0
|
||||
|
||||
|
||||
def test_delete_artist_cascade_keeps_unrelated_null_post_attachment(
|
||||
db_sync, tmp_path,
|
||||
):
|
||||
"""A filesystem-import row (post_id NULL) sharing the sha is the other way
|
||||
this collides — and it must SURVIVE: it belongs to no artist, so the
|
||||
cascade has no claim on it."""
|
||||
a = _make_artist(db_sync, slug="casorph")
|
||||
p = Post(artist_id=a.id, external_post_id="orph-p1")
|
||||
db_sync.add(p)
|
||||
db_sync.flush()
|
||||
|
||||
sha = "0rfa".ljust(64, "0")
|
||||
standalone = PostAttachment(
|
||||
post_id=None, artist_id=None, sha256=sha,
|
||||
path="/store/0rfa/manual.pdf", original_filename="manual.pdf",
|
||||
ext=".pdf", size_bytes=3,
|
||||
)
|
||||
db_sync.add(standalone)
|
||||
db_sync.add(PostAttachment(
|
||||
post_id=p.id, artist_id=a.id, sha256=sha,
|
||||
path="/store/0rfa/manual.pdf", original_filename="manual.pdf",
|
||||
ext=".pdf", size_bytes=3,
|
||||
))
|
||||
db_sync.commit()
|
||||
artist_id, standalone_id = a.id, standalone.id
|
||||
|
||||
result = cleanup_service.delete_artist_cascade(
|
||||
db_sync, artist_id=artist_id, images_root=tmp_path,
|
||||
)
|
||||
|
||||
assert result["summary"]["attachments_deleted"] == 1
|
||||
surviving = db_sync.execute(
|
||||
select(PostAttachment.id, PostAttachment.post_id)
|
||||
.where(PostAttachment.sha256 == sha)
|
||||
).all()
|
||||
assert surviving == [(standalone_id, None)]
|
||||
|
||||
|
||||
# --- delete_images --------------------------------------------------
|
||||
|
||||
|
||||
@@ -916,3 +1082,176 @@ def test_reconcile_preserves_from_attachment_on_provenance_collision(db_sync, tm
|
||||
.where(ImageProvenance.image_record_id == img_id)
|
||||
).all()
|
||||
assert rows == [(native_id, att_id)]
|
||||
|
||||
|
||||
# --- reclaim_orphaned_attachments -----------------------------------
|
||||
|
||||
|
||||
def _store_blob(root, sha, *, ext=".pdf", age_hours=48, data=b"blob"):
|
||||
"""Write a file into the sha-addressed attachment store, aged past the
|
||||
min-age guard by default."""
|
||||
d = root / "attachments" / sha[:3]
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
p = d / f"{sha}{ext}"
|
||||
p.write_bytes(data)
|
||||
old = datetime.now(UTC).timestamp() - age_hours * 3600
|
||||
os.utime(p, (old, old))
|
||||
return p
|
||||
|
||||
|
||||
def _attachment(db_sync, *, sha, post=None, artist=None):
|
||||
att = PostAttachment(
|
||||
post_id=post.id if post else None,
|
||||
artist_id=artist.id if artist else None,
|
||||
sha256=sha, path=f"/store/{sha[:3]}/f.pdf",
|
||||
original_filename="f.pdf", ext=".pdf", size_bytes=4,
|
||||
)
|
||||
db_sync.add(att)
|
||||
db_sync.flush()
|
||||
return att
|
||||
|
||||
|
||||
def test_reclaim_attachments_dry_run_projects_without_mutating(db_sync, tmp_path):
|
||||
a = _make_artist(db_sync, slug="recl-dry")
|
||||
p = Post(artist_id=a.id, external_post_id="rd-1")
|
||||
db_sync.add(p)
|
||||
db_sync.flush()
|
||||
kept_sha, orphan_sha = "aa11".ljust(64, "0"), "bb22".ljust(64, "0")
|
||||
_attachment(db_sync, sha=kept_sha, post=p, artist=a)
|
||||
_attachment(db_sync, sha=orphan_sha) # both FKs NULL → orphan
|
||||
db_sync.commit()
|
||||
kept_blob = _store_blob(tmp_path, kept_sha)
|
||||
orphan_blob = _store_blob(tmp_path, orphan_sha)
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=True,
|
||||
)
|
||||
assert result["rows"] == 1
|
||||
assert result["files"] == 1
|
||||
assert result["bytes"] == orphan_blob.stat().st_size
|
||||
|
||||
# Nothing actually happened.
|
||||
assert kept_blob.exists() and orphan_blob.exists()
|
||||
assert db_sync.execute(
|
||||
select(func.count(PostAttachment.id))
|
||||
).scalar_one() == 2
|
||||
|
||||
|
||||
def test_reclaim_attachments_apply_deletes_rows_and_unlinks_blobs(db_sync, tmp_path):
|
||||
a = _make_artist(db_sync, slug="recl-apply")
|
||||
p = Post(artist_id=a.id, external_post_id="ra-1")
|
||||
db_sync.add(p)
|
||||
db_sync.flush()
|
||||
kept_sha, orphan_sha = "cc33".ljust(64, "0"), "dd44".ljust(64, "0")
|
||||
_attachment(db_sync, sha=kept_sha, post=p, artist=a)
|
||||
_attachment(db_sync, sha=orphan_sha)
|
||||
db_sync.commit()
|
||||
kept_blob = _store_blob(tmp_path, kept_sha)
|
||||
orphan_blob = _store_blob(tmp_path, orphan_sha)
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
assert result["rows"] == 1
|
||||
assert result["files"] == 1
|
||||
|
||||
assert kept_blob.exists() # still referenced
|
||||
assert not orphan_blob.exists() # nothing points at it any more
|
||||
surviving = db_sync.execute(select(PostAttachment.sha256)).scalars().all()
|
||||
assert surviving == [kept_sha]
|
||||
|
||||
|
||||
def test_reclaim_attachments_preview_matches_apply(db_sync, tmp_path):
|
||||
"""Rule 93 — the dry-run's numbers are what the apply does. The projection
|
||||
has to negate the orphan predicate to be honest about blobs the delete is
|
||||
about to free, so this is the assertion that catches getting that backwards.
|
||||
"""
|
||||
orphan_sha = "ee55".ljust(64, "0")
|
||||
_attachment(db_sync, sha=orphan_sha)
|
||||
db_sync.commit()
|
||||
_store_blob(tmp_path, orphan_sha)
|
||||
|
||||
projected = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=True,
|
||||
)
|
||||
applied = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
for key in ("rows", "files", "bytes"):
|
||||
assert projected[key] == applied[key], key
|
||||
assert applied["rows"] == 1 and applied["files"] == 1
|
||||
|
||||
|
||||
def test_reclaim_attachments_keeps_shared_blob_while_any_row_remains(db_sync, tmp_path):
|
||||
"""The refcount case this whole sweep exists for: one sha-addressed blob
|
||||
backs several rows, so deleting SOME of them must not free the file."""
|
||||
a = _make_artist(db_sync, slug="recl-shared")
|
||||
p = Post(artist_id=a.id, external_post_id="rs-1")
|
||||
db_sync.add(p)
|
||||
db_sync.flush()
|
||||
sha = "ff66".ljust(64, "0")
|
||||
_attachment(db_sync, sha=sha, post=p, artist=a) # attributed — survives
|
||||
_attachment(db_sync, sha=sha) # orphan — deleted
|
||||
db_sync.commit()
|
||||
blob = _store_blob(tmp_path, sha)
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
assert result["rows"] == 1 # the orphan row went
|
||||
assert result["files"] == 0 # the blob did NOT
|
||||
assert blob.exists()
|
||||
|
||||
|
||||
def test_reclaim_attachments_spares_filesystem_import_rows(db_sync, tmp_path):
|
||||
"""post_id NULL with an artist_id is the deliberate filesystem-import shape
|
||||
(importer._capture_attachment), not an orphan — it is still attributed."""
|
||||
a = _make_artist(db_sync, slug="recl-fsimport")
|
||||
sha = "1177".ljust(64, "0")
|
||||
_attachment(db_sync, sha=sha, artist=a) # post NULL, artist set
|
||||
db_sync.commit()
|
||||
blob = _store_blob(tmp_path, sha)
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
assert result["rows"] == 0
|
||||
assert result["files"] == 0
|
||||
assert blob.exists()
|
||||
assert db_sync.execute(
|
||||
select(func.count(PostAttachment.id))
|
||||
).scalar_one() == 1
|
||||
|
||||
|
||||
def test_reclaim_attachments_skips_recent_and_staging_files(db_sync, tmp_path):
|
||||
"""A blob is written BEFORE its row commits, so a just-stored file with no
|
||||
row is in-flight, not orphaned. `.partial` staging files belong to
|
||||
cleanup_orphaned_temp_files and must be left alone either way."""
|
||||
fresh_sha, staged_sha = "2288".ljust(64, "0"), "3399".ljust(64, "0")
|
||||
fresh = _store_blob(tmp_path, fresh_sha, age_hours=0)
|
||||
staged = _store_blob(tmp_path, staged_sha, ext=".pdf.partial")
|
||||
db_sync.commit()
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
assert result["files"] == 0
|
||||
assert result["skipped_recent"] == 1
|
||||
assert fresh.exists() and staged.exists()
|
||||
|
||||
|
||||
def test_reclaim_attachments_ignores_non_sha_named_files(db_sync, tmp_path):
|
||||
"""The walk must only judge files it can identify as store blobs — anything
|
||||
else under the root is none of its business."""
|
||||
d = tmp_path / "attachments" / "zzz"
|
||||
d.mkdir(parents=True)
|
||||
stray = d / "notes.txt"
|
||||
stray.write_text("not a blob")
|
||||
old = datetime.now(UTC).timestamp() - 48 * 3600
|
||||
os.utime(stray, (old, old))
|
||||
|
||||
result = cleanup_service.reclaim_orphaned_attachments(
|
||||
db_sync, images_root=tmp_path, dry_run=False,
|
||||
)
|
||||
assert result["files"] == 0
|
||||
assert stray.exists()
|
||||
|
||||
@@ -19,7 +19,7 @@ def test_native_platforms():
|
||||
def test_gallery_dl_platforms_are_not_native():
|
||||
# The platforms still served by gallery-dl must NOT route to the native
|
||||
# ingester — guards an accidental over-broad migration.
|
||||
for platform in ("hentaifoundry", "discord", "deviantart"):
|
||||
for platform in ("hentaifoundry", "discord"):
|
||||
assert uses_native_ingester(platform) is False
|
||||
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user