Compare commits
57
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0533807669 | ||
|
|
279dff3fb6 | ||
|
|
37e66cddc4 | ||
|
|
9cf6b2d363 | ||
|
|
6ef0fed41f | ||
|
|
89b48f8f35 | ||
|
|
d60e0b9494 | ||
|
|
9c27a2d3c7 | ||
|
|
93e37681b7 | ||
|
|
64ca858574 | ||
|
|
9d0c0b7da8 | ||
|
|
8e4d252ae4 | ||
|
|
fdd3e01f56 | ||
|
|
c82fb308b6 | ||
|
|
8cf8d2ca4d | ||
|
|
b1d58bc3b8 | ||
|
|
65386f02a0 | ||
|
|
667b05f14e | ||
|
|
856e9104b4 | ||
|
|
0397642b21 | ||
|
|
237575447d | ||
|
|
ed358757dc | ||
|
|
d181f4afb8 | ||
|
|
2886fa4997 | ||
|
|
f256f587ee | ||
|
|
384d8d5e50 | ||
|
|
319e8c1d18 | ||
|
|
9075d8eadd | ||
|
|
88e53e5b86 | ||
|
|
37e8b796a1 | ||
|
|
4e82208926 | ||
|
|
52fff00353 | ||
|
|
c14338cbce | ||
|
|
8c36dd28b0 | ||
|
|
88cfb3dd02 | ||
|
|
5d4f223b71 | ||
|
|
05090c6e85 | ||
|
|
3a577d5ade | ||
|
|
f4fe02e346 | ||
|
|
e766197d99 | ||
|
|
3872e1dda9 | ||
|
|
9814f3dbaf | ||
|
|
b214460fdb | ||
|
|
ac55d0e8d8 | ||
|
|
89a89e0ded | ||
|
|
4e9aac2c05 | ||
|
|
2879ac6f2b | ||
|
|
b8dce6c483 | ||
|
|
d1c0b82a22 | ||
|
|
5526b8dc78 | ||
|
|
16eb7075c4 | ||
|
|
885dcf64f3 | ||
|
|
f2f6b6d25e | ||
|
|
0822240fde | ||
|
|
27f7f3fd01 | ||
|
|
c5bf564f53 | ||
|
|
602c7d275d |
+149
-30
@@ -92,25 +92,28 @@ jobs:
|
|||||||
- run: npm run test:unit
|
- run: npm run test:unit
|
||||||
- run: npm run build
|
- run: npm run build
|
||||||
|
|
||||||
# Single integration job — collapsed from a 3-way shard split on 2026-06-04.
|
# Integration suite split into THREE parallel shards (2026-05-25, runner
|
||||||
# The shards existed to parallelize ~8.5min of integration tests; once the
|
# capacity bumped 2→6). Each shard gets its own Postgres + Redis service
|
||||||
# throwaway Postgres runs with fsync OFF (the durability step below) the whole
|
# set and runs alembic + a disjoint subset of integration tests. Shards
|
||||||
# suite runs in ~45s, so the split only triplicated the ~2min fixed overhead
|
# share no DB state, so the autouse TRUNCATE fixture in tests/conftest.py
|
||||||
# (container + `uv pip install` + `alembic upgrade head`) and burned 3 of 6
|
# stays single-threaded per shard but multiple shards run in parallel
|
||||||
# runner slots for no wall-clock gain. One job now: spin up once, install
|
# wall-clock. Approximate split — rebalance once --durations=15 output
|
||||||
# once, migrate once, run every integration test.
|
# reveals which shard is the long pole.
|
||||||
#
|
#
|
||||||
# The docker-ps filter scopes to THIS job's own Postgres/Redis service
|
# Each shard's docker-ps filter uses its own unique job name to scope
|
||||||
# containers by job name. act_runner strips underscores from job names when
|
# service-container resolution. act_runner appears to strip underscores
|
||||||
# labelling containers (`int_api` matched nothing on 2026-05-25), so the name
|
# from job names when building container labels — `int_api` yielded
|
||||||
# stays separator-free (`integration`). The step prints `docker ps -a` first
|
# zero matches on 2026-05-25 — so shards use no-separator names
|
||||||
# so a future naming-convention shift surfaces in the log without a
|
# (`intapi`, `intimp`, `intcore`) instead. Each step prints
|
||||||
# guess-and-push cycle.
|
# `docker ps -a` first so a future naming-convention shift surfaces in
|
||||||
|
# the log without another guess-and-push cycle.
|
||||||
#
|
#
|
||||||
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT done —
|
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT
|
||||||
# per ci-requirements.md, FC is the only Python consumer of that image and the
|
# done — per ci-requirements.md, FC is the only Python consumer of that
|
||||||
# CI-Runner "add deps to image when used by >1 project" rule keeps it per-job.
|
# image and the CI-Runner project's "add deps to image when used by >1
|
||||||
integration:
|
# project" rule keeps the install per-job.
|
||||||
|
|
||||||
|
intapi:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
@@ -141,14 +144,14 @@ jobs:
|
|||||||
--health-retries 10
|
--health-retries 10
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
- name: Integration suite (resolve service IPs, migrate, test)
|
- name: API integration shard (resolve service IPs, migrate, test)
|
||||||
run: |
|
run: |
|
||||||
set -eux
|
set -eux
|
||||||
echo "=== container landscape (diagnostic for filter scoping) ==="
|
echo "=== container landscape (diagnostic for filter scoping) ==="
|
||||||
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
|
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
|
||||||
echo "=== end landscape ==="
|
echo "=== end landscape ==="
|
||||||
PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
PG=$(docker ps --filter "name=intapi" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
||||||
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
RD=$(docker ps --filter "name=intapi" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
||||||
test -n "$PG" && test -n "$RD"
|
test -n "$PG" && test -n "$RD"
|
||||||
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
||||||
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
||||||
@@ -165,14 +168,130 @@ jobs:
|
|||||||
else
|
else
|
||||||
pip install -r requirements.txt pytest pytest-asyncio
|
pip install -r requirements.txt pytest pytest-asyncio
|
||||||
fi
|
fi
|
||||||
# Relax durability on the throwaway CI Postgres so the per-test
|
|
||||||
# TRUNCATE's commit-fsync — the integration teardown's dominant cost
|
|
||||||
# (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
|
|
||||||
# skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
|
|
||||||
# is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
|
|
||||||
# NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
|
|
||||||
# surprise can't red the job; fabledcurator is the postgres image's
|
|
||||||
# bootstrap superuser.
|
|
||||||
python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
|
|
||||||
alembic upgrade head
|
alembic upgrade head
|
||||||
pytest tests/ -v -m integration --durations=15
|
pytest tests/test_api_*.py -v -m integration --durations=15
|
||||||
|
|
||||||
|
intimp:
|
||||||
|
runs-on: python-ci
|
||||||
|
container:
|
||||||
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
|
env:
|
||||||
|
DB_USER: fabledcurator
|
||||||
|
DB_PASSWORD: ci_integration
|
||||||
|
DB_PORT: "5432"
|
||||||
|
DB_NAME: fabledcurator_test
|
||||||
|
SECRET_KEY: ci_integration_placeholder
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: pgvector/pgvector:pg16
|
||||||
|
env:
|
||||||
|
POSTGRES_USER: fabledcurator
|
||||||
|
POSTGRES_PASSWORD: ci_integration
|
||||||
|
POSTGRES_DB: fabledcurator_test
|
||||||
|
options: >-
|
||||||
|
--health-cmd "pg_isready -U fabledcurator"
|
||||||
|
--health-interval 10s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
options: >-
|
||||||
|
--health-cmd "redis-cli ping"
|
||||||
|
--health-interval 10s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Importer integration shard (resolve service IPs, migrate, test)
|
||||||
|
run: |
|
||||||
|
set -eux
|
||||||
|
echo "=== container landscape (diagnostic for filter scoping) ==="
|
||||||
|
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
|
||||||
|
echo "=== end landscape ==="
|
||||||
|
PG=$(docker ps --filter "name=intimp" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
||||||
|
RD=$(docker ps --filter "name=intimp" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
||||||
|
test -n "$PG" && test -n "$RD"
|
||||||
|
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
||||||
|
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
||||||
|
test -n "$PG_IP" && test -n "$RD_IP"
|
||||||
|
export DB_HOST="$PG_IP"
|
||||||
|
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
|
||||||
|
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
if command -v uv >/dev/null 2>&1; then
|
||||||
|
uv pip install --system -r requirements.txt pytest pytest-asyncio
|
||||||
|
else
|
||||||
|
pip install -r requirements.txt pytest pytest-asyncio
|
||||||
|
fi
|
||||||
|
alembic upgrade head
|
||||||
|
pytest tests/test_importer*.py tests/test_import_*.py tests/test_migration_*.py tests/test_phash_*.py tests/test_sidecar_*.py tests/test_scan_*.py tests/test_archive_extractor.py tests/test_backfill_phash.py -v -m integration --durations=15
|
||||||
|
|
||||||
|
intcore:
|
||||||
|
runs-on: python-ci
|
||||||
|
container:
|
||||||
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
|
env:
|
||||||
|
DB_USER: fabledcurator
|
||||||
|
DB_PASSWORD: ci_integration
|
||||||
|
DB_PORT: "5432"
|
||||||
|
DB_NAME: fabledcurator_test
|
||||||
|
SECRET_KEY: ci_integration_placeholder
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: pgvector/pgvector:pg16
|
||||||
|
env:
|
||||||
|
POSTGRES_USER: fabledcurator
|
||||||
|
POSTGRES_PASSWORD: ci_integration
|
||||||
|
POSTGRES_DB: fabledcurator_test
|
||||||
|
options: >-
|
||||||
|
--health-cmd "pg_isready -U fabledcurator"
|
||||||
|
--health-interval 10s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
options: >-
|
||||||
|
--health-cmd "redis-cli ping"
|
||||||
|
--health-interval 10s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
|
||||||
|
run: |
|
||||||
|
set -eux
|
||||||
|
echo "=== container landscape (diagnostic for filter scoping) ==="
|
||||||
|
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
|
||||||
|
echo "=== end landscape ==="
|
||||||
|
PG=$(docker ps --filter "name=intcore" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
||||||
|
RD=$(docker ps --filter "name=intcore" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
||||||
|
test -n "$PG" && test -n "$RD"
|
||||||
|
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
||||||
|
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
||||||
|
test -n "$PG_IP" && test -n "$RD_IP"
|
||||||
|
export DB_HOST="$PG_IP"
|
||||||
|
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
|
||||||
|
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
if command -v uv >/dev/null 2>&1; then
|
||||||
|
uv pip install --system -r requirements.txt pytest pytest-asyncio
|
||||||
|
else
|
||||||
|
pip install -r requirements.txt pytest pytest-asyncio
|
||||||
|
fi
|
||||||
|
alembic upgrade head
|
||||||
|
pytest tests/ -v -m integration --durations=15 \
|
||||||
|
--ignore-glob='tests/test_api_*.py' \
|
||||||
|
--ignore-glob='tests/test_importer*.py' \
|
||||||
|
--ignore-glob='tests/test_import_*.py' \
|
||||||
|
--ignore-glob='tests/test_migration_*.py' \
|
||||||
|
--ignore-glob='tests/test_phash_*.py' \
|
||||||
|
--ignore-glob='tests/test_sidecar_*.py' \
|
||||||
|
--ignore-glob='tests/test_scan_*.py' \
|
||||||
|
--ignore-glob='tests/test_archive_extractor.py' \
|
||||||
|
--ignore-glob='tests/test_backfill_phash.py'
|
||||||
|
|||||||
@@ -61,9 +61,6 @@ Thumbs.db
|
|||||||
|
|
||||||
# Claude Code per-user local overrides (shared .claude/settings.json is OK to commit)
|
# Claude Code per-user local overrides (shared .claude/settings.json is OK to commit)
|
||||||
.claude/settings.local.json
|
.claude/settings.local.json
|
||||||
# Transient scheduler lock/state (committed by accident in 3f30327)
|
|
||||||
.claude/scheduled_tasks.lock
|
|
||||||
.claude/scheduled_tasks*.json
|
|
||||||
|
|
||||||
# Alembic / DB scratch
|
# Alembic / DB scratch
|
||||||
alembic/versions/__pycache__/
|
alembic/versions/__pycache__/
|
||||||
|
|||||||
+1
-22
@@ -2,21 +2,12 @@
|
|||||||
|
|
||||||
from logging.config import fileConfig
|
from logging.config import fileConfig
|
||||||
|
|
||||||
from sqlalchemy import engine_from_config, pool, text
|
from sqlalchemy import engine_from_config, pool
|
||||||
|
|
||||||
from alembic import context
|
from alembic import context
|
||||||
from backend.app.config import get_config
|
from backend.app.config import get_config
|
||||||
from backend.app.models import Base
|
from backend.app.models import Base
|
||||||
|
|
||||||
# Arbitrary fixed 64-bit key for the session/transaction advisory lock that
|
|
||||||
# serializes concurrent `alembic upgrade head` runs. Every `web` replica runs
|
|
||||||
# migrations in its entrypoint, so under `docker stack deploy` two replicas can
|
|
||||||
# boot at once and race the same DDL — duplicate CREATE TABLE, then a crashed
|
|
||||||
# replica (operator-flagged 2026-06-07: 0040 raced; one backend died with
|
|
||||||
# AdminShutdown). The first replica to reach the lock migrates; the rest block,
|
|
||||||
# then find the version table already at head and apply nothing.
|
|
||||||
_MIGRATION_LOCK_KEY = 0xFCA1E35C
|
|
||||||
|
|
||||||
config = context.config
|
config = context.config
|
||||||
|
|
||||||
if config.config_file_name is not None:
|
if config.config_file_name is not None:
|
||||||
@@ -53,18 +44,6 @@ def run_migrations_online() -> None:
|
|||||||
compare_type=True,
|
compare_type=True,
|
||||||
)
|
)
|
||||||
with context.begin_transaction():
|
with context.begin_transaction():
|
||||||
# Serialize concurrent migrators (see _MIGRATION_LOCK_KEY). A
|
|
||||||
# transaction-scoped advisory lock: the first replica to get here
|
|
||||||
# holds it for the whole upgrade and is auto-released when this
|
|
||||||
# transaction ends. A sibling replica blocks on this line, and only
|
|
||||||
# once the leader commits does it proceed to read the version table
|
|
||||||
# — now at head — so it runs zero migrations instead of re-applying
|
|
||||||
# the same DDL. The lock is acquired BEFORE run_migrations() reads
|
|
||||||
# the current revision, which is what makes the no-op correct.
|
|
||||||
connection.execute(
|
|
||||||
text("SELECT pg_advisory_xact_lock(:k)"),
|
|
||||||
{"k": _MIGRATION_LOCK_KEY},
|
|
||||||
)
|
|
||||||
context.run_migrations()
|
context.run_migrations()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,53 +0,0 @@
|
|||||||
"""artist_visit: per-artist last-viewed timestamp for the "+N new" badge
|
|
||||||
|
|
||||||
Revision ID: 0034
|
|
||||||
Revises: 0033
|
|
||||||
Create Date: 2026-06-03
|
|
||||||
|
|
||||||
Powers the artists-directory "+N new since last visit" badge + ArtistView
|
|
||||||
banner. Single row per artist (no user_id yet — rule #47 multi-user ACL
|
|
||||||
is aspirational; widens to (user_id, artist_id) PK when User lands).
|
|
||||||
|
|
||||||
Seed every existing artist with `last_viewed_at = NOW()` so the badge
|
|
||||||
starts at 0 across the board — no noisy "you have 5000 unseen images"
|
|
||||||
on first deploy. New artists auto-get a row via
|
|
||||||
`ArtistService.find_or_create`.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0034"
|
|
||||||
down_revision: Union[str, None] = "0033"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.create_table(
|
|
||||||
"artist_visit",
|
|
||||||
sa.Column(
|
|
||||||
"artist_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("artist.id", ondelete="CASCADE"),
|
|
||||||
primary_key=True,
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"last_viewed_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("NOW()"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
# Seed: every existing artist starts "fully caught up". Without this,
|
|
||||||
# every operator with N artists would see N badges (worth of every
|
|
||||||
# image ever imported) on first deploy.
|
|
||||||
op.execute(
|
|
||||||
"INSERT INTO artist_visit (artist_id, last_viewed_at) "
|
|
||||||
"SELECT id, NOW() FROM artist"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_table("artist_visit")
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
"""image_record.effective_date: materialized gallery sort key + index
|
|
||||||
|
|
||||||
Revision ID: 0035
|
|
||||||
Revises: 0034
|
|
||||||
Create Date: 2026-06-04
|
|
||||||
|
|
||||||
The gallery ordered/cursored on COALESCE(post.post_date,
|
|
||||||
image_record.created_at) across the Post outer join. That expression spans
|
|
||||||
two tables, so no index can serve it — every /scroll sorted a large slice
|
|
||||||
of the library, and the frontend fired ten of them serially per initial
|
|
||||||
load. Materialize the value into image_record.effective_date and index
|
|
||||||
(effective_date DESC, id DESC) so the cursor scroll is an index range scan.
|
|
||||||
|
|
||||||
Backfill = COALESCE(primary post's post_date, created_at) so existing rows
|
|
||||||
keep their exact ordering. New rows get the created_at-equivalent server
|
|
||||||
default; services/importer.py overrides it with the post's date when a
|
|
||||||
primary post with a date is linked.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0035"
|
|
||||||
down_revision: Union[str, None] = "0034"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
# Add nullable first so the backfill can populate before NOT NULL.
|
|
||||||
op.add_column(
|
|
||||||
"image_record",
|
|
||||||
sa.Column("effective_date", sa.DateTime(timezone=True), nullable=True),
|
|
||||||
)
|
|
||||||
# Pure set-based UPDATEs (no per-row params) — immune to the 65535
|
|
||||||
# bind-parameter ceiling regardless of library size.
|
|
||||||
op.execute(
|
|
||||||
"""
|
|
||||||
UPDATE image_record AS ir
|
|
||||||
SET effective_date = COALESCE(p.post_date, ir.created_at)
|
|
||||||
FROM post AS p
|
|
||||||
WHERE ir.primary_post_id = p.id
|
|
||||||
"""
|
|
||||||
)
|
|
||||||
op.execute(
|
|
||||||
"""
|
|
||||||
UPDATE image_record
|
|
||||||
SET effective_date = created_at
|
|
||||||
WHERE effective_date IS NULL
|
|
||||||
"""
|
|
||||||
)
|
|
||||||
op.alter_column(
|
|
||||||
"image_record",
|
|
||||||
"effective_date",
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("now()"),
|
|
||||||
)
|
|
||||||
# DESC/DESC matches the gallery's ORDER BY effective_date DESC, id DESC
|
|
||||||
# so the scroll is a forward index scan; raw SQL because alembic's
|
|
||||||
# column list doesn't express per-column DESC cleanly.
|
|
||||||
op.execute(
|
|
||||||
"CREATE INDEX ix_image_record_effective_date "
|
|
||||||
"ON image_record (effective_date DESC, id DESC)"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_index("ix_image_record_effective_date", table_name="image_record")
|
|
||||||
op.drop_column("image_record", "effective_date")
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
"""image_record.siglip_embedding: HNSW cosine index for "more like this"
|
|
||||||
|
|
||||||
Revision ID: 0036
|
|
||||||
Revises: 0035
|
|
||||||
Create Date: 2026-06-04
|
|
||||||
|
|
||||||
Gallery Phase 3 (visual similarity search) ranks images by
|
|
||||||
`siglip_embedding.cosine_distance(source_embedding)`. Without an index that's
|
|
||||||
a sequential scan computing a 1152-dim distance for every row — fine at small
|
|
||||||
scale, but it grows linearly with the library. Add an HNSW index with
|
|
||||||
`vector_cosine_ops` so the top-N nearest search is sub-50ms ANN.
|
|
||||||
|
|
||||||
1152 dims is under pgvector's 2000-dim HNSW limit, so HNSW (no training,
|
|
||||||
better recall than IVFFlat) is the right choice. ONE-TIME COST: building the
|
|
||||||
index over the existing embeddings (~57k vectors on the operator's library)
|
|
||||||
locks image_record for ~30-60s during this migration on deploy — acceptable
|
|
||||||
for a single-operator homelab. NULL embeddings (videos / not-yet-embedded
|
|
||||||
rows) are simply not indexed.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0036"
|
|
||||||
down_revision: Union[str, None] = "0035"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
# Raw SQL: alembic's create_index doesn't express the `USING hnsw (...
|
|
||||||
# vector_cosine_ops)` access-method + opclass cleanly. Must match the
|
|
||||||
# query's cosine_distance operator class to be usable by the planner.
|
|
||||||
op.execute(
|
|
||||||
"CREATE INDEX ix_image_record_siglip_hnsw "
|
|
||||||
"ON image_record USING hnsw (siglip_embedding vector_cosine_ops)"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_index("ix_image_record_siglip_hnsw", table_name="image_record")
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
"""patreon_seen_media: per-source ledger of already-ingested Patreon media
|
|
||||||
|
|
||||||
Revision ID: 0037
|
|
||||||
Revises: 0036
|
|
||||||
Create Date: 2026-06-05
|
|
||||||
|
|
||||||
Native Patreon ingester (build step 2a). Replaces gallery-dl's
|
|
||||||
archive.sqlite3 with our own queryable table. The downloader upserts one
|
|
||||||
row per (source, media) so routine walks skip media we've already
|
|
||||||
processed; a future "recovery" mode bypasses the ledger to re-walk.
|
|
||||||
|
|
||||||
`filehash` is a 32-hex Patreon CDN MD5, OR a video sentinel of the form
|
|
||||||
``video:<post_id>:<media_id>`` — hence String(128). The unique
|
|
||||||
constraint on (source_id, filehash) is the dedup upsert key.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0037"
|
|
||||||
down_revision: Union[str, None] = "0036"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.create_table(
|
|
||||||
"patreon_seen_media",
|
|
||||||
sa.Column("id", sa.Integer, primary_key=True),
|
|
||||||
sa.Column(
|
|
||||||
"source_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("source.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
index=True,
|
|
||||||
),
|
|
||||||
sa.Column("filehash", sa.String(128), nullable=False),
|
|
||||||
sa.Column("post_id", sa.String(64), nullable=True),
|
|
||||||
sa.Column(
|
|
||||||
"seen_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("NOW()"),
|
|
||||||
),
|
|
||||||
sa.UniqueConstraint(
|
|
||||||
"source_id", "filehash", name="uq_patreon_seen_media_source_id"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_table("patreon_seen_media")
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
"""patreon_failed_media: per-source dead-letter ledger for failing Patreon media
|
|
||||||
|
|
||||||
Revision ID: 0038
|
|
||||||
Revises: 0037
|
|
||||||
Create Date: 2026-06-06
|
|
||||||
|
|
||||||
Plan #705 (#7). Media that keeps failing to download/validate (404'd CDN,
|
|
||||||
deleted post, geo-blocked Mux, persistently-corrupt bytes) gets recorded here
|
|
||||||
with an attempt counter; once it crosses the dead-letter threshold the ingester
|
|
||||||
skips it on routine walks (recovery still re-attempts). A clean download clears
|
|
||||||
the row. UNIQUE (source_id, filehash) is the upsert key (same media key the
|
|
||||||
seen-ledger uses).
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0038"
|
|
||||||
down_revision: Union[str, None] = "0037"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.create_table(
|
|
||||||
"patreon_failed_media",
|
|
||||||
sa.Column("id", sa.Integer, primary_key=True),
|
|
||||||
sa.Column(
|
|
||||||
"source_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("source.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
index=True,
|
|
||||||
),
|
|
||||||
sa.Column("filehash", sa.String(128), nullable=False),
|
|
||||||
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
|
|
||||||
sa.Column("last_error", sa.Text, nullable=True),
|
|
||||||
sa.Column(
|
|
||||||
"first_failed_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("NOW()"),
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"last_failed_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("NOW()"),
|
|
||||||
),
|
|
||||||
sa.UniqueConstraint(
|
|
||||||
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_table("patreon_failed_media")
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
"""library_audit_run: resume cursor + progress timestamp for chunked scans
|
|
||||||
|
|
||||||
Revision ID: 0039
|
|
||||||
Revises: 0038
|
|
||||||
Create Date: 2026-06-07
|
|
||||||
|
|
||||||
scan_library_for_rule used to run one 2h pass that timed out on large libraries
|
|
||||||
and monopolized the concurrency-1 maintenance queue (operator-flagged). It now
|
|
||||||
runs short time-boxed chunks that re-enqueue: `resume_after_id` persists the
|
|
||||||
keyset cursor so the next chunk continues where it left off, and
|
|
||||||
`last_progress_at` lets the recovery sweep tell a progressing multi-chunk audit
|
|
||||||
from a genuinely stuck one.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0039"
|
|
||||||
down_revision: Union[str, None] = "0038"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.add_column(
|
|
||||||
"library_audit_run",
|
|
||||||
sa.Column(
|
|
||||||
"resume_after_id", sa.Integer, nullable=False, server_default="0"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"library_audit_run",
|
|
||||||
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("library_audit_run", "last_progress_at")
|
|
||||||
op.drop_column("library_audit_run", "resume_after_id")
|
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
"""series chapters: chapter layer over series_page (FC-6.1)
|
|
||||||
|
|
||||||
Revision ID: 0040
|
|
||||||
Revises: 0039
|
|
||||||
Create Date: 2026-06-07
|
|
||||||
|
|
||||||
A series (Tag kind='series') gains an ordered chapter layer. Reading order
|
|
||||||
becomes (series_chapter.chapter_number, series_page.page_number). Every existing
|
|
||||||
series is backfilled into a single auto-chapter (chapter_number=1) holding its
|
|
||||||
current flat pages, so no data is lost and the old flat ordering is preserved.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0040"
|
|
||||||
down_revision: Union[str, None] = "0039"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.create_table(
|
|
||||||
"series_chapter",
|
|
||||||
sa.Column("id", sa.Integer, primary_key=True),
|
|
||||||
sa.Column(
|
|
||||||
"series_tag_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("tag.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
),
|
|
||||||
sa.Column("chapter_number", sa.Integer, nullable=False),
|
|
||||||
sa.Column("title", sa.Text, nullable=True),
|
|
||||||
sa.Column(
|
|
||||||
"is_placeholder", sa.Boolean, nullable=False, server_default="false"
|
|
||||||
),
|
|
||||||
sa.Column("stated_page_start", sa.Integer, nullable=True),
|
|
||||||
sa.Column("stated_page_end", sa.Integer, nullable=True),
|
|
||||||
sa.Column(
|
|
||||||
"created_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("now()"),
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"updated_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("now()"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.create_index(
|
|
||||||
"ix_series_chapter_series_tag_id", "series_chapter", ["series_tag_id"]
|
|
||||||
)
|
|
||||||
|
|
||||||
# New columns on series_page; chapter_id starts nullable so we can backfill.
|
|
||||||
op.add_column(
|
|
||||||
"series_page", sa.Column("chapter_id", sa.Integer, nullable=True)
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"series_page", sa.Column("stated_page", sa.Integer, nullable=True)
|
|
||||||
)
|
|
||||||
|
|
||||||
conn = op.get_bind()
|
|
||||||
# One auto-chapter per existing series (any series_tag_id present in pages).
|
|
||||||
conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"INSERT INTO series_chapter "
|
|
||||||
"(series_tag_id, chapter_number, is_placeholder, created_at, updated_at) "
|
|
||||||
"SELECT DISTINCT series_tag_id, 1, false, now(), now() "
|
|
||||||
"FROM series_page"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
# Point every existing page at its series' auto-chapter.
|
|
||||||
conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"UPDATE series_page sp "
|
|
||||||
"SET chapter_id = sc.id "
|
|
||||||
"FROM series_chapter sc "
|
|
||||||
"WHERE sc.series_tag_id = sp.series_tag_id"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Now lock chapter_id down: NOT NULL + FK (cascade) + index.
|
|
||||||
op.alter_column("series_page", "chapter_id", nullable=False)
|
|
||||||
op.create_foreign_key(
|
|
||||||
"fk_series_page_chapter_id",
|
|
||||||
"series_page",
|
|
||||||
"series_chapter",
|
|
||||||
["chapter_id"],
|
|
||||||
["id"],
|
|
||||||
ondelete="CASCADE",
|
|
||||||
)
|
|
||||||
op.create_index(
|
|
||||||
"ix_series_page_chapter_id", "series_page", ["chapter_id"]
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_index("ix_series_page_chapter_id", table_name="series_page")
|
|
||||||
op.drop_constraint(
|
|
||||||
"fk_series_page_chapter_id", "series_page", type_="foreignkey"
|
|
||||||
)
|
|
||||||
op.drop_column("series_page", "stated_page")
|
|
||||||
op.drop_column("series_page", "chapter_id")
|
|
||||||
op.drop_index("ix_series_chapter_series_tag_id", table_name="series_chapter")
|
|
||||||
op.drop_table("series_chapter")
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
"""series suggestions: assisted-continuation matcher (FC-6.3)
|
|
||||||
|
|
||||||
Revision ID: 0041
|
|
||||||
Revises: 0040
|
|
||||||
Create Date: 2026-06-07
|
|
||||||
|
|
||||||
A confirm-only queue of "this post may continue this series" hints, plus two
|
|
||||||
import_settings knobs (enable + score threshold) for the matcher.
|
|
||||||
"""
|
|
||||||
from typing import Sequence, Union
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
revision: str = "0041"
|
|
||||||
down_revision: Union[str, None] = "0040"
|
|
||||||
branch_labels: Union[str, Sequence[str], None] = None
|
|
||||||
depends_on: Union[str, Sequence[str], None] = None
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
op.create_table(
|
|
||||||
"series_suggestion",
|
|
||||||
sa.Column("id", sa.Integer, primary_key=True),
|
|
||||||
sa.Column(
|
|
||||||
"post_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("post.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"series_tag_id",
|
|
||||||
sa.Integer,
|
|
||||||
sa.ForeignKey("tag.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
),
|
|
||||||
sa.Column("score", sa.Float, nullable=False),
|
|
||||||
sa.Column("signals", sa.JSON, nullable=True),
|
|
||||||
sa.Column(
|
|
||||||
"status", sa.String(16), nullable=False, server_default="pending"
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"created_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("now()"),
|
|
||||||
),
|
|
||||||
sa.Column(
|
|
||||||
"updated_at",
|
|
||||||
sa.DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.text("now()"),
|
|
||||||
),
|
|
||||||
sa.UniqueConstraint(
|
|
||||||
"post_id", "series_tag_id", name="uq_series_suggestion_post_series"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.create_index(
|
|
||||||
"ix_series_suggestion_post_id", "series_suggestion", ["post_id"]
|
|
||||||
)
|
|
||||||
op.create_index(
|
|
||||||
"ix_series_suggestion_series_tag_id",
|
|
||||||
"series_suggestion",
|
|
||||||
["series_tag_id"],
|
|
||||||
)
|
|
||||||
op.create_index(
|
|
||||||
"ix_series_suggestion_status", "series_suggestion", ["status"]
|
|
||||||
)
|
|
||||||
|
|
||||||
op.add_column(
|
|
||||||
"import_settings",
|
|
||||||
sa.Column(
|
|
||||||
"series_suggest_enabled",
|
|
||||||
sa.Boolean,
|
|
||||||
nullable=False,
|
|
||||||
server_default=sa.true(),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
op.add_column(
|
|
||||||
"import_settings",
|
|
||||||
sa.Column(
|
|
||||||
"series_suggest_threshold",
|
|
||||||
sa.Float,
|
|
||||||
nullable=False,
|
|
||||||
server_default="0.5",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
op.drop_column("import_settings", "series_suggest_threshold")
|
|
||||||
op.drop_column("import_settings", "series_suggest_enabled")
|
|
||||||
op.drop_index("ix_series_suggestion_status", table_name="series_suggestion")
|
|
||||||
op.drop_index(
|
|
||||||
"ix_series_suggestion_series_tag_id", table_name="series_suggestion"
|
|
||||||
)
|
|
||||||
op.drop_index("ix_series_suggestion_post_id", table_name="series_suggestion")
|
|
||||||
op.drop_table("series_suggestion")
|
|
||||||
+1
-105
@@ -19,7 +19,7 @@ from __future__ import annotations
|
|||||||
import hashlib
|
import hashlib
|
||||||
|
|
||||||
from quart import Blueprint, jsonify, request
|
from quart import Blueprint, jsonify, request
|
||||||
from sqlalchemy import select, text
|
from sqlalchemy import select
|
||||||
|
|
||||||
from ..extensions import get_session
|
from ..extensions import get_session
|
||||||
from ..models import Artist
|
from ..models import Artist
|
||||||
@@ -222,107 +222,3 @@ async def tags_purge_legacy():
|
|||||||
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
|
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
|
||||||
)
|
)
|
||||||
return jsonify(result)
|
return jsonify(result)
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/tags/reset-content", methods=["POST"])
|
|
||||||
async def tags_reset_content():
|
|
||||||
"""Tier-A: delete ALL general + character tags (the Camie-suggestable
|
|
||||||
content vocabulary) so the operator can re-tag from scratch via
|
|
||||||
auto-suggest. fandom + series tags + series_page ordering are preserved,
|
|
||||||
and image tagger_predictions are untouched so suggestions repopulate.
|
|
||||||
dry-run preview returns per-kind counts + applications + a sample so the
|
|
||||||
UI shows exactly what'll go before the operator confirms (dry_run=false).
|
|
||||||
Irreversible except via DB backup restore."""
|
|
||||||
from ..services.cleanup_service import reset_content_tagging
|
|
||||||
|
|
||||||
body = await request.get_json(silent=True) or {}
|
|
||||||
dry_run = bool(body.get("dry_run", False))
|
|
||||||
|
|
||||||
async with get_session() as session:
|
|
||||||
result = await session.run_sync(
|
|
||||||
lambda sync_sess: reset_content_tagging(sync_sess, dry_run=dry_run)
|
|
||||||
)
|
|
||||||
return jsonify(result)
|
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/tags/normalize", methods=["POST"])
|
|
||||||
async def tags_normalize():
|
|
||||||
"""#714: retro-normalize existing tags to the #701 canonical form (Title
|
|
||||||
Case + collapsed whitespace) and merge case/whitespace-variant duplicates.
|
|
||||||
|
|
||||||
dry_run=true (default) returns a projection inline — group/collision/rename
|
|
||||||
counts + a sample of the changes — so the UI shows exactly what'll happen.
|
|
||||||
dry_run=false dispatches the long-running maintenance task (the merge FK
|
|
||||||
repoints can touch many tags); the UI tails the activity dashboard for the
|
|
||||||
summary. Idempotent; back up first (the merges are irreversible)."""
|
|
||||||
from ..services.tag_service import normalize_existing_tags
|
|
||||||
|
|
||||||
body = await request.get_json(silent=True) or {}
|
|
||||||
dry_run = bool(body.get("dry_run", True))
|
|
||||||
|
|
||||||
if dry_run:
|
|
||||||
async with get_session() as session:
|
|
||||||
result = await normalize_existing_tags(session, dry_run=True)
|
|
||||||
return jsonify(result)
|
|
||||||
|
|
||||||
from ..tasks.admin import normalize_tags_task
|
|
||||||
|
|
||||||
async_result = normalize_tags_task.delay()
|
|
||||||
return jsonify({"task_id": async_result.id, "status": "queued"}), 202
|
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/db-stats", methods=["GET"])
|
|
||||||
async def db_stats():
|
|
||||||
"""Per-table bloat readout (pg_stat_user_tables) for the high-churn tables
|
|
||||||
so the operator can see when a VACUUM is worth running."""
|
|
||||||
from ..tasks.maintenance import VACUUM_TABLES
|
|
||||||
|
|
||||||
wanted = set(VACUUM_TABLES)
|
|
||||||
async with get_session() as session:
|
|
||||||
rows = (await session.execute(text(
|
|
||||||
"SELECT relname, n_live_tup, n_dead_tup, last_vacuum, "
|
|
||||||
"last_autovacuum, last_analyze FROM pg_stat_user_tables"
|
|
||||||
))).all()
|
|
||||||
|
|
||||||
def _iso(v):
|
|
||||||
return v.isoformat() if v is not None else None
|
|
||||||
|
|
||||||
out = []
|
|
||||||
for r in rows:
|
|
||||||
if r.relname not in wanted:
|
|
||||||
continue
|
|
||||||
live = r.n_live_tup or 0
|
|
||||||
dead = r.n_dead_tup or 0
|
|
||||||
total = live + dead
|
|
||||||
out.append({
|
|
||||||
"table": r.relname,
|
|
||||||
"live": live,
|
|
||||||
"dead": dead,
|
|
||||||
"dead_pct": round(100 * dead / total, 1) if total else 0.0,
|
|
||||||
"last_vacuum": _iso(r.last_vacuum),
|
|
||||||
"last_autovacuum": _iso(r.last_autovacuum),
|
|
||||||
"last_analyze": _iso(r.last_analyze),
|
|
||||||
})
|
|
||||||
out.sort(key=lambda t: t["dead"], reverse=True)
|
|
||||||
return jsonify({"tables": out})
|
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/vacuum", methods=["POST"])
|
|
||||||
async def trigger_vacuum():
|
|
||||||
"""Operator-triggered VACUUM (ANALYZE) over the high-churn tables — the
|
|
||||||
same maintenance-queue task the weekly Beat schedule runs."""
|
|
||||||
from ..tasks.maintenance import vacuum_analyze
|
|
||||||
|
|
||||||
vacuum_analyze.delay()
|
|
||||||
return jsonify({"status": "queued"}), 202
|
|
||||||
|
|
||||||
|
|
||||||
@admin_bp.route("/maintenance/reextract-archives", methods=["POST"])
|
|
||||||
async def trigger_reextract_archives():
|
|
||||||
"""Operator-triggered re-extract (#713): PostAttachments that are actually
|
|
||||||
archives but were filed opaquely (pre magic-byte gate) get extracted and
|
|
||||||
their members linked to the post. Idempotent; runs on the maintenance queue."""
|
|
||||||
from ..tasks.admin import reextract_archive_attachments_task
|
|
||||||
|
|
||||||
async_result = reextract_archive_attachments_task.delay()
|
|
||||||
return jsonify({"task_id": async_result.id, "status": "queued"}), 202
|
|
||||||
|
|||||||
@@ -154,15 +154,12 @@ async def audit_history():
|
|||||||
limit = min(int(request.args.get("limit", "20")), 100)
|
limit = min(int(request.args.get("limit", "20")), 100)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return _bad("invalid_limit")
|
return _bad("invalid_limit")
|
||||||
# Optional rule filter so a card can reconnect to ITS latest run on mount
|
|
||||||
# (?rule=transparency&limit=1) — the audit survives navigation; the UI
|
|
||||||
# rehydrates from this rather than losing the in-flight scan.
|
|
||||||
rule = request.args.get("rule") or None
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
stmt = select(LibraryAuditRun).order_by(LibraryAuditRun.id.desc())
|
rows = (await session.execute(
|
||||||
if rule is not None:
|
select(LibraryAuditRun)
|
||||||
stmt = stmt.where(LibraryAuditRun.rule == rule)
|
.order_by(LibraryAuditRun.id.desc())
|
||||||
rows = (await session.execute(stmt.limit(limit))).scalars().all()
|
.limit(limit)
|
||||||
|
)).scalars().all()
|
||||||
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
|
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -121,15 +121,13 @@ async def delete_credential(platform: str):
|
|||||||
|
|
||||||
@credentials_bp.route("/<platform>/verify", methods=["POST"])
|
@credentials_bp.route("/<platform>/verify", methods=["POST"])
|
||||||
async def verify_credential(platform: str):
|
async def verify_credential(platform: str):
|
||||||
"""Test the stored credential against one of the platform's enabled sources,
|
"""Test the stored credential by running gallery-dl --simulate
|
||||||
WITHOUT downloading. Routes through the platform's backend
|
against one of the platform's enabled sources. On success stamps
|
||||||
(download_backends.verify_credential) — native ingester for Patreon, an
|
last_verified. Returns {valid: bool|null, reason, last_verified?}.
|
||||||
authenticated API page; gallery-dl --simulate for the rest. On success
|
valid=null means "couldn't test" (no credential, or no enabled
|
||||||
stamps last_verified. Returns {valid: bool|null, reason, last_verified?};
|
source to point at)."""
|
||||||
valid=null means "couldn't test" (no credential, no enabled source, or an
|
|
||||||
inconclusive network/drift result)."""
|
|
||||||
from ..models import Artist, Source
|
from ..models import Artist, Source
|
||||||
from ..services.download_backends import verify_source_credential
|
from ..services.gallery_dl import GalleryDLService, SourceConfig
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
if not await _ext_key_ok(session):
|
if not await _ext_key_ok(session):
|
||||||
@@ -156,14 +154,14 @@ async def verify_credential(platform: str):
|
|||||||
cookies_path = await svc.get_cookies_path(platform)
|
cookies_path = await svc.get_cookies_path(platform)
|
||||||
auth_token = await svc.get_token(platform)
|
auth_token = await svc.get_token(platform)
|
||||||
|
|
||||||
ok, message = await verify_source_credential(
|
gdl = GalleryDLService(images_root=Path("/images"))
|
||||||
platform=platform,
|
ok, message = await gdl.verify(
|
||||||
url=source.url,
|
url=source.url,
|
||||||
artist_slug=artist.slug,
|
artist_slug=artist.slug,
|
||||||
config_overrides=source.config_overrides or {},
|
platform=platform,
|
||||||
|
source_config=SourceConfig.from_dict(source.config_overrides or {}),
|
||||||
cookies_path=str(cookies_path) if cookies_path else None,
|
cookies_path=str(cookies_path) if cookies_path else None,
|
||||||
auth_token=auth_token,
|
auth_token=auth_token,
|
||||||
images_root=Path("/images"),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
last_verified = None
|
last_verified = None
|
||||||
|
|||||||
@@ -44,9 +44,6 @@ def _list_record(event: DownloadEvent, source: Source | None, artist: Artist | N
|
|||||||
"bytes_downloaded": event.bytes_downloaded,
|
"bytes_downloaded": event.bytes_downloaded,
|
||||||
"error": event.error,
|
"error": event.error,
|
||||||
"summary": _summary_from_metadata(event.metadata_),
|
"summary": _summary_from_metadata(event.metadata_),
|
||||||
# plan #709: mid-walk live counts for a RUNNING native-ingester event
|
|
||||||
# (None otherwise; phase 3 overwrites metadata with run_stats on finish).
|
|
||||||
"live": (event.metadata_ or {}).get("live"),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+48
-136
@@ -1,6 +1,4 @@
|
|||||||
"""Gallery API: cursor scroll, timeline, jump, image detail, facets."""
|
"""Gallery API: cursor scroll, timeline, jump, image detail."""
|
||||||
|
|
||||||
from datetime import UTC, datetime, timedelta
|
|
||||||
|
|
||||||
from quart import Blueprint, jsonify, request
|
from quart import Blueprint, jsonify, request
|
||||||
|
|
||||||
@@ -10,9 +8,34 @@ from ..services.gallery_service import GalleryService
|
|||||||
gallery_bp = Blueprint("gallery", __name__, url_prefix="/api/gallery")
|
gallery_bp = Blueprint("gallery", __name__, url_prefix="/api/gallery")
|
||||||
|
|
||||||
|
|
||||||
def _image_json(i):
|
@gallery_bp.route("/scroll", methods=["GET"])
|
||||||
"""Serialize a GalleryImage for the scroll/similar list responses."""
|
async def scroll():
|
||||||
return {
|
cursor = request.args.get("cursor") or None
|
||||||
|
try:
|
||||||
|
limit = int(request.args.get("limit", "50"))
|
||||||
|
except ValueError:
|
||||||
|
return jsonify({"error": "limit must be an integer"}), 400
|
||||||
|
tag_id_raw = request.args.get("tag_id")
|
||||||
|
tag_id = int(tag_id_raw) if tag_id_raw else None
|
||||||
|
post_id_raw = request.args.get("post_id")
|
||||||
|
post_id = int(post_id_raw) if post_id_raw else None
|
||||||
|
artist_id_raw = request.args.get("artist_id")
|
||||||
|
artist_id = int(artist_id_raw) if artist_id_raw else None
|
||||||
|
|
||||||
|
async with get_session() as session:
|
||||||
|
svc = GalleryService(session)
|
||||||
|
try:
|
||||||
|
page = await svc.scroll(
|
||||||
|
cursor=cursor, limit=limit, tag_id=tag_id,
|
||||||
|
post_id=post_id, artist_id=artist_id,
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
return jsonify({"error": str(exc)}), 400
|
||||||
|
|
||||||
|
return jsonify(
|
||||||
|
{
|
||||||
|
"images": [
|
||||||
|
{
|
||||||
"id": i.id,
|
"id": i.id,
|
||||||
"sha256": i.sha256,
|
"sha256": i.sha256,
|
||||||
"mime": i.mime,
|
"mime": i.mime,
|
||||||
@@ -23,75 +46,8 @@ def _image_json(i):
|
|||||||
"thumbnail_url": i.thumbnail_url,
|
"thumbnail_url": i.thumbnail_url,
|
||||||
"artist": i.artist,
|
"artist": i.artist,
|
||||||
}
|
}
|
||||||
|
for i in page.images
|
||||||
|
],
|
||||||
def _parse_date(raw):
|
|
||||||
"""Parse a YYYY-MM-DD query value to a UTC midnight datetime, or None.
|
|
||||||
Raises ValueError (→ 400) on a malformed value."""
|
|
||||||
if not raw:
|
|
||||||
return None
|
|
||||||
return datetime.strptime(raw, "%Y-%m-%d").replace(tzinfo=UTC)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_filters():
|
|
||||||
"""Parse the composable gallery filters from query args, returning
|
|
||||||
``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates.
|
|
||||||
|
|
||||||
`tag_id` accepts a single id or a comma-separated list (AND); `media` is
|
|
||||||
image|video; `sort` is newest|oldest; `platform` selects one platform
|
|
||||||
(or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are boolean
|
|
||||||
flags; `date_from`/`date_to` are inclusive calendar-day bounds (date_to is
|
|
||||||
widened by a day so the whole day is covered by the service's half-open
|
|
||||||
`< date_to`)."""
|
|
||||||
tag_raw = request.args.get("tag_id")
|
|
||||||
tag_ids = (
|
|
||||||
[int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None
|
|
||||||
) or None
|
|
||||||
post_id_raw = request.args.get("post_id")
|
|
||||||
post_id = int(post_id_raw) if post_id_raw else None
|
|
||||||
artist_id_raw = request.args.get("artist_id")
|
|
||||||
artist_id = int(artist_id_raw) if artist_id_raw else None
|
|
||||||
media = request.args.get("media")
|
|
||||||
media_type = media if media in ("image", "video") else None
|
|
||||||
sort = request.args.get("sort")
|
|
||||||
sort = sort if sort in ("newest", "oldest") else "newest"
|
|
||||||
platform = request.args.get("platform") or None
|
|
||||||
untagged = request.args.get("untagged") in ("1", "true", "yes")
|
|
||||||
no_artist = request.args.get("no_artist") in ("1", "true", "yes")
|
|
||||||
date_from = _parse_date(request.args.get("date_from"))
|
|
||||||
date_to = _parse_date(request.args.get("date_to"))
|
|
||||||
if date_to is not None:
|
|
||||||
date_to += timedelta(days=1) # inclusive of the date_to calendar day
|
|
||||||
filters = {
|
|
||||||
"tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id,
|
|
||||||
"media_type": media_type, "platform": platform,
|
|
||||||
"untagged": untagged, "no_artist": no_artist,
|
|
||||||
"date_from": date_from, "date_to": date_to,
|
|
||||||
}
|
|
||||||
return filters, sort
|
|
||||||
|
|
||||||
|
|
||||||
@gallery_bp.route("/scroll", methods=["GET"])
|
|
||||||
async def scroll():
|
|
||||||
cursor = request.args.get("cursor") or None
|
|
||||||
try:
|
|
||||||
limit = int(request.args.get("limit", "50"))
|
|
||||||
filters, sort = _parse_filters()
|
|
||||||
except ValueError:
|
|
||||||
return jsonify({"error": "invalid filter or limit parameter"}), 400
|
|
||||||
|
|
||||||
async with get_session() as session:
|
|
||||||
svc = GalleryService(session)
|
|
||||||
try:
|
|
||||||
page = await svc.scroll(
|
|
||||||
cursor=cursor, limit=limit, sort=sort, **filters,
|
|
||||||
)
|
|
||||||
except ValueError as exc:
|
|
||||||
return jsonify({"error": str(exc)}), 400
|
|
||||||
|
|
||||||
return jsonify(
|
|
||||||
{
|
|
||||||
"images": [_image_json(i) for i in page.images],
|
|
||||||
"next_cursor": page.next_cursor,
|
"next_cursor": page.next_cursor,
|
||||||
"date_groups": [
|
"date_groups": [
|
||||||
{"year": y, "month": m, "image_ids": ids} for y, m, ids in page.date_groups
|
{"year": y, "month": m, "image_ids": ids} for y, m, ids in page.date_groups
|
||||||
@@ -100,46 +56,20 @@ async def scroll():
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@gallery_bp.route("/similar", methods=["GET"])
|
|
||||||
async def similar():
|
|
||||||
"""Visual "more like this": images ranked by cosine distance to the
|
|
||||||
`similar_to` image's embedding. Composes with the scope filters (AND) but
|
|
||||||
ignores post_id and sort. Bounded top-N, no cursor."""
|
|
||||||
try:
|
|
||||||
similar_to = int(request.args["similar_to"])
|
|
||||||
limit = int(request.args.get("limit", "100"))
|
|
||||||
filters, _sort = _parse_filters()
|
|
||||||
except (KeyError, ValueError):
|
|
||||||
return jsonify({"error": "similar_to query param required"}), 400
|
|
||||||
# post_id is the exclusive post-detail view — not a similarity scope.
|
|
||||||
scope = {k: v for k, v in filters.items() if k != "post_id"}
|
|
||||||
async with get_session() as session:
|
|
||||||
svc = GalleryService(session)
|
|
||||||
try:
|
|
||||||
images = await svc.similar(image_id=similar_to, limit=limit, **scope)
|
|
||||||
except ValueError as exc:
|
|
||||||
return jsonify({"error": str(exc)}), 400
|
|
||||||
if images is None:
|
|
||||||
return jsonify({"error": "not found"}), 404
|
|
||||||
return jsonify(
|
|
||||||
{
|
|
||||||
"images": [_image_json(i) for i in images],
|
|
||||||
"next_cursor": None,
|
|
||||||
"date_groups": [],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@gallery_bp.route("/timeline", methods=["GET"])
|
@gallery_bp.route("/timeline", methods=["GET"])
|
||||||
async def timeline():
|
async def timeline():
|
||||||
try:
|
tag_id_raw = request.args.get("tag_id")
|
||||||
filters, _sort = _parse_filters()
|
tag_id = int(tag_id_raw) if tag_id_raw else None
|
||||||
except ValueError:
|
post_id_raw = request.args.get("post_id")
|
||||||
return jsonify({"error": "invalid filter parameter"}), 400
|
post_id = int(post_id_raw) if post_id_raw else None
|
||||||
|
artist_id_raw = request.args.get("artist_id")
|
||||||
|
artist_id = int(artist_id_raw) if artist_id_raw else None
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
svc = GalleryService(session)
|
svc = GalleryService(session)
|
||||||
try:
|
try:
|
||||||
buckets = await svc.timeline(**filters)
|
buckets = await svc.timeline(
|
||||||
|
tag_id=tag_id, post_id=post_id, artist_id=artist_id
|
||||||
|
)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
return jsonify({"error": str(exc)}), 400
|
return jsonify({"error": str(exc)}), 400
|
||||||
return jsonify(
|
return jsonify(
|
||||||
@@ -147,43 +77,25 @@ async def timeline():
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@gallery_bp.route("/facets", methods=["GET"])
|
|
||||||
async def facets():
|
|
||||||
try:
|
|
||||||
filters, _sort = _parse_filters()
|
|
||||||
except ValueError:
|
|
||||||
return jsonify({"error": "invalid filter parameter"}), 400
|
|
||||||
async with get_session() as session:
|
|
||||||
svc = GalleryService(session)
|
|
||||||
try:
|
|
||||||
f = await svc.facets(**filters)
|
|
||||||
except ValueError as exc:
|
|
||||||
return jsonify({"error": str(exc)}), 400
|
|
||||||
return jsonify(
|
|
||||||
{
|
|
||||||
"total": f.total,
|
|
||||||
"platforms": f.platforms,
|
|
||||||
"untagged": f.untagged,
|
|
||||||
"no_artist": f.no_artist,
|
|
||||||
"date_min": f.date_min.isoformat() if f.date_min else None,
|
|
||||||
"date_max": f.date_max.isoformat() if f.date_max else None,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@gallery_bp.route("/jump", methods=["GET"])
|
@gallery_bp.route("/jump", methods=["GET"])
|
||||||
async def jump():
|
async def jump():
|
||||||
try:
|
try:
|
||||||
year = int(request.args["year"])
|
year = int(request.args["year"])
|
||||||
month = int(request.args["month"])
|
month = int(request.args["month"])
|
||||||
filters, sort = _parse_filters()
|
|
||||||
except (KeyError, ValueError):
|
except (KeyError, ValueError):
|
||||||
return jsonify({"error": "year and month query params required"}), 400
|
return jsonify({"error": "year and month query params required"}), 400
|
||||||
|
tag_id_raw = request.args.get("tag_id")
|
||||||
|
tag_id = int(tag_id_raw) if tag_id_raw else None
|
||||||
|
post_id_raw = request.args.get("post_id")
|
||||||
|
post_id = int(post_id_raw) if post_id_raw else None
|
||||||
|
artist_id_raw = request.args.get("artist_id")
|
||||||
|
artist_id = int(artist_id_raw) if artist_id_raw else None
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
svc = GalleryService(session)
|
svc = GalleryService(session)
|
||||||
try:
|
try:
|
||||||
cursor = await svc.jump_cursor(
|
cursor = await svc.jump_cursor(
|
||||||
year=year, month=month, sort=sort, **filters,
|
year=year, month=month, tag_id=tag_id,
|
||||||
|
post_id=post_id, artist_id=artist_id,
|
||||||
)
|
)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
return jsonify({"error": str(exc)}), 400
|
return jsonify({"error": str(exc)}), 400
|
||||||
|
|||||||
@@ -25,8 +25,6 @@ _EDITABLE_FIELDS = (
|
|||||||
"download_schedule_default_seconds",
|
"download_schedule_default_seconds",
|
||||||
"download_event_retention_days",
|
"download_event_retention_days",
|
||||||
"download_failure_warning_threshold",
|
"download_failure_warning_threshold",
|
||||||
"series_suggest_enabled",
|
|
||||||
"series_suggest_threshold",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -48,8 +46,6 @@ async def get_import_settings():
|
|||||||
"download_schedule_default_seconds": row.download_schedule_default_seconds,
|
"download_schedule_default_seconds": row.download_schedule_default_seconds,
|
||||||
"download_event_retention_days": row.download_event_retention_days,
|
"download_event_retention_days": row.download_event_retention_days,
|
||||||
"download_failure_warning_threshold": row.download_failure_warning_threshold,
|
"download_failure_warning_threshold": row.download_failure_warning_threshold,
|
||||||
"series_suggest_enabled": row.series_suggest_enabled,
|
|
||||||
"series_suggest_threshold": row.series_suggest_threshold,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
||||||
@@ -100,19 +96,6 @@ async def update_import_settings():
|
|||||||
if not isinstance(v, int) or isinstance(v, bool) or v < 1 or v > 100:
|
if not isinstance(v, int) or isinstance(v, bool) or v < 1 or v > 100:
|
||||||
return _bad_int("download_failure_warning_threshold", 1, 100)
|
return _bad_int("download_failure_warning_threshold", 1, 100)
|
||||||
|
|
||||||
if "series_suggest_enabled" in body and not isinstance(
|
|
||||||
body["series_suggest_enabled"], bool
|
|
||||||
):
|
|
||||||
return jsonify(
|
|
||||||
{"error": "series_suggest_enabled must be a boolean"}
|
|
||||||
), 400
|
|
||||||
if "series_suggest_threshold" in body:
|
|
||||||
v = body["series_suggest_threshold"]
|
|
||||||
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
|
|
||||||
return jsonify(
|
|
||||||
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
|
||||||
), 400
|
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
row = await ImportSettings.load(session)
|
row = await ImportSettings.load(session)
|
||||||
for field in _EDITABLE_FIELDS:
|
for field in _EDITABLE_FIELDS:
|
||||||
|
|||||||
+15
-121
@@ -85,22 +85,6 @@ async def create_source():
|
|||||||
return _bad("empty_url", detail=str(exc))
|
return _bad("empty_url", detail=str(exc))
|
||||||
except DuplicateSourceError as exc:
|
except DuplicateSourceError as exc:
|
||||||
return _bad("duplicate", status=409, existing_id=exc.existing_id)
|
return _bad("duplicate", status=409, existing_id=exc.existing_id)
|
||||||
|
|
||||||
# Immediate kickoff: a new enabled source is armed for backfill (#693)
|
|
||||||
# but would otherwise sit idle until the next scheduler tick (~60s).
|
|
||||||
# Enqueue the first walk now, skipping only if the platform is in a
|
|
||||||
# rate-limit cooldown (the scheduler picks it up when that clears).
|
|
||||||
dispatch_id = None
|
|
||||||
if record.enabled:
|
|
||||||
cooldowns = await active_platform_cooldowns(session)
|
|
||||||
if record.platform not in cooldowns:
|
|
||||||
session.add(DownloadEvent(source_id=record.id, status="pending"))
|
|
||||||
await session.commit()
|
|
||||||
dispatch_id = record.id
|
|
||||||
|
|
||||||
if dispatch_id is not None:
|
|
||||||
from ..tasks.download import download_source
|
|
||||||
download_source.delay(dispatch_id)
|
|
||||||
return jsonify(record.to_dict()), 201
|
return jsonify(record.to_dict()), 201
|
||||||
|
|
||||||
|
|
||||||
@@ -138,119 +122,29 @@ async def delete_source(source_id: int):
|
|||||||
|
|
||||||
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
|
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
|
||||||
async def set_backfill(source_id: int):
|
async def set_backfill(source_id: int):
|
||||||
"""Plan #693/#697: start/stop a run-until-done backfill, or start a recovery.
|
"""Plan #544: arm a source for backfill mode for the next N download
|
||||||
Body: `{"action": "start" | "stop" | "recover"}` (default "start"). 'start'
|
runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
|
||||||
walks the full post history in time-boxed chunks until it reaches the bottom
|
source dict. While backfill_runs_remaining > 0, downloads use
|
||||||
(then the source shows 'complete'); 'recover' is the same walk but bypasses
|
gallery-dl's full-walk config (skip: True + 30-min timeout) instead
|
||||||
the Patreon seen-ledger to re-fetch dropped-and-deleted near-dups under the
|
of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
|
||||||
current pHash threshold; 'stop' cancels either back to tick mode. Returns the
|
|
||||||
updated source dict (incl. backfill_state / backfill_chunks /
|
|
||||||
backfill_bypass_seen)."""
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from ..services.credential_service import CredentialService
|
|
||||||
from ..services.download_backends import (
|
|
||||||
uses_native_ingester,
|
|
||||||
verify_source_credential,
|
|
||||||
)
|
|
||||||
from .credentials import _get_crypto
|
|
||||||
|
|
||||||
payload = await request.get_json(silent=True) or {}
|
payload = await request.get_json(silent=True) or {}
|
||||||
action = payload.get("action", "start")
|
runs = payload.get("runs", 3)
|
||||||
if action not in ("start", "stop", "recover"):
|
try:
|
||||||
return _bad(
|
runs = int(runs)
|
||||||
"invalid_action",
|
except (TypeError, ValueError):
|
||||||
detail="action must be 'start', 'stop', or 'recover'",
|
return _bad("invalid_runs", detail="runs must be an integer")
|
||||||
)
|
|
||||||
|
|
||||||
# Pre-flight (plan #703 #2): before arming a deep walk on a native-ingester
|
|
||||||
# platform (where verify is one cheap API page), refuse if the credential is
|
|
||||||
# DEFINITIVELY rejected — don't burn chunks against expired cookies. Proceed
|
|
||||||
# on valid OR inconclusive (a network blip shouldn't block). Gated to native
|
|
||||||
# platforms: gallery-dl verify is a slow --simulate subprocess, too heavy for
|
|
||||||
# an arm action. The credential read happens in a session that's CLOSED
|
|
||||||
# before the verify network call (don't hold a DB conn across the request).
|
|
||||||
if action in ("start", "recover"):
|
|
||||||
async with get_session() as session:
|
|
||||||
rec = await SourceService(session).get(source_id)
|
|
||||||
if rec is None:
|
|
||||||
return _bad("not_found", status=404)
|
|
||||||
native = uses_native_ingester(rec.platform)
|
|
||||||
if native:
|
|
||||||
cred = CredentialService(session, _get_crypto())
|
|
||||||
cookies_path = await cred.get_cookies_path(rec.platform)
|
|
||||||
auth_token = await cred.get_token(rec.platform)
|
|
||||||
if native:
|
|
||||||
ok, message = await verify_source_credential(
|
|
||||||
platform=rec.platform,
|
|
||||||
url=rec.url,
|
|
||||||
artist_slug=rec.artist_slug,
|
|
||||||
config_overrides=rec.config_overrides or {},
|
|
||||||
cookies_path=str(cookies_path) if cookies_path else None,
|
|
||||||
auth_token=auth_token,
|
|
||||||
images_root=Path("/images"),
|
|
||||||
)
|
|
||||||
if ok is False:
|
|
||||||
return _bad("credential_rejected", detail=message, status=409)
|
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
try:
|
try:
|
||||||
svc = SourceService(session)
|
record = await SourceService(session).set_backfill_runs(
|
||||||
if action == "start":
|
source_id, runs,
|
||||||
record = await svc.start_backfill(source_id)
|
)
|
||||||
elif action == "recover":
|
|
||||||
record = await svc.start_recovery(source_id)
|
|
||||||
else:
|
|
||||||
record = await svc.stop_backfill(source_id)
|
|
||||||
except LookupError:
|
except LookupError:
|
||||||
return _bad("not_found", status=404)
|
return _bad("not_found", status=404)
|
||||||
|
except ValueError as exc:
|
||||||
|
return _bad("invalid_runs", detail=str(exc))
|
||||||
return jsonify(record.to_dict())
|
return jsonify(record.to_dict())
|
||||||
|
|
||||||
|
|
||||||
@sources_bp.route("/<int:source_id>/preview", methods=["POST"])
|
|
||||||
async def preview_source_endpoint(source_id: int):
|
|
||||||
"""Plan #708 B4: dry-run — count what a backfill WOULD download for a native
|
|
||||||
platform (Patreon today), without downloading. Walks the first few feed pages
|
|
||||||
and counts media not already in the seen/dead ledgers. Returns
|
|
||||||
{total_new, posts_scanned, pages_scanned, has_more, sample[]} or 409 + reason
|
|
||||||
(unresolvable campaign id / auth / drift). 400 for gallery-dl platforms (no
|
|
||||||
cheap dry-run — their verify is a slow --simulate)."""
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from ..services.credential_service import CredentialService
|
|
||||||
from ..services.download_backends import preview_source, uses_native_ingester
|
|
||||||
from ..tasks._sync_engine import sync_session_factory
|
|
||||||
from .credentials import _get_crypto
|
|
||||||
|
|
||||||
async with get_session() as session:
|
|
||||||
rec = await SourceService(session).get(source_id)
|
|
||||||
if rec is None:
|
|
||||||
return _bad("not_found", status=404)
|
|
||||||
if not uses_native_ingester(rec.platform):
|
|
||||||
return _bad(
|
|
||||||
"unsupported",
|
|
||||||
detail="Preview is only available for native-ingester platforms.",
|
|
||||||
status=400,
|
|
||||||
)
|
|
||||||
cred = CredentialService(session, _get_crypto())
|
|
||||||
cookies_path = await cred.get_cookies_path(rec.platform)
|
|
||||||
|
|
||||||
# The walk + ledger reads are sync (run off the request loop); the process
|
|
||||||
# sync engine is the same one the download task uses.
|
|
||||||
result = await preview_source(
|
|
||||||
platform=rec.platform,
|
|
||||||
url=rec.url,
|
|
||||||
source_id=source_id,
|
|
||||||
config_overrides=rec.config_overrides or {},
|
|
||||||
cookies_path=str(cookies_path) if cookies_path else None,
|
|
||||||
images_root=Path("/images"),
|
|
||||||
sync_session_factory=sync_session_factory(),
|
|
||||||
)
|
|
||||||
if "error" in result:
|
|
||||||
return _bad("preview_failed", detail=result["error"], status=409)
|
|
||||||
return jsonify(result)
|
|
||||||
|
|
||||||
|
|
||||||
@sources_bp.route("/<int:source_id>/check", methods=["POST"])
|
@sources_bp.route("/<int:source_id>/check", methods=["POST"])
|
||||||
async def check_source(source_id: int):
|
async def check_source(source_id: int):
|
||||||
"""FC-3c: enqueue a download for this source.
|
"""FC-3c: enqueue a download for this source.
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ system_activity_bp = Blueprint(
|
|||||||
# absent.
|
# absent.
|
||||||
_QUEUE_NAMES = (
|
_QUEUE_NAMES = (
|
||||||
"default", "import", "thumbnail", "ml",
|
"default", "import", "thumbnail", "ml",
|
||||||
"download", "scan", "maintenance", "maintenance_long",
|
"download", "scan", "maintenance",
|
||||||
)
|
)
|
||||||
|
|
||||||
# Cache module-level so all requests share the cache between polls.
|
# Cache module-level so all requests share the cache between polls.
|
||||||
|
|||||||
+6
-303
@@ -8,14 +8,12 @@ from ..extensions import get_session
|
|||||||
from ..models import Tag, TagKind
|
from ..models import Tag, TagKind
|
||||||
from ..models.tag_allowlist import TagAllowlist
|
from ..models.tag_allowlist import TagAllowlist
|
||||||
from ..services.bulk_tag_service import BulkTagService
|
from ..services.bulk_tag_service import BulkTagService
|
||||||
from ..services.series_match_service import SeriesMatchService
|
|
||||||
from ..services.series_service import SeriesError, SeriesService
|
from ..services.series_service import SeriesError, SeriesService
|
||||||
from ..services.tag_directory_service import TagDirectoryService
|
from ..services.tag_directory_service import TagDirectoryService
|
||||||
from ..services.tag_service import (
|
from ..services.tag_service import (
|
||||||
TagMergeConflict,
|
TagMergeConflict,
|
||||||
TagService,
|
TagService,
|
||||||
TagValidationError,
|
TagValidationError,
|
||||||
normalize_tag_name,
|
|
||||||
)
|
)
|
||||||
from ..utils.tag_prefix import parse_kind_prefix
|
from ..utils.tag_prefix import parse_kind_prefix
|
||||||
|
|
||||||
@@ -143,11 +141,6 @@ async def create_tag():
|
|||||||
|
|
||||||
fandom_id = body.get("fandom_id")
|
fandom_id = body.get("fandom_id")
|
||||||
|
|
||||||
# #701: Title-Case operator-entered tags. Only here (the explicit create
|
|
||||||
# endpoint), NOT in the shared find_or_create — the ML tagger uses that path
|
|
||||||
# and must keep the booru vocabulary's casing for allowlist matching.
|
|
||||||
name = normalize_tag_name(name)
|
|
||||||
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
svc = TagService(session)
|
svc = TagService(session)
|
||||||
try:
|
try:
|
||||||
@@ -201,46 +194,15 @@ async def remove_tag_from_image(image_id: int, tag_id: int):
|
|||||||
return "", 204
|
return "", 204
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/tags/<int:tag_id>", methods=["GET"])
|
|
||||||
async def get_tag(tag_id: int):
|
|
||||||
"""Resolve a single tag (used by the gallery to label its active
|
|
||||||
tag-filter chip)."""
|
|
||||||
async with get_session() as session:
|
|
||||||
tag = await session.get(Tag, tag_id)
|
|
||||||
if tag is None:
|
|
||||||
return jsonify({"error": "tag not found"}), 404
|
|
||||||
return jsonify(
|
|
||||||
{
|
|
||||||
"id": tag.id,
|
|
||||||
"name": tag.name,
|
|
||||||
"kind": tag.kind.value,
|
|
||||||
"fandom_id": tag.fandom_id,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"])
|
@tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"])
|
||||||
async def update_tag(tag_id: int):
|
async def rename_tag(tag_id: int):
|
||||||
"""Rename and/or re-fandom a tag. Body may carry `name` and/or
|
body = await request.get_json()
|
||||||
`fandom_id` (a fandom tag id, or null to clear — character tags only).
|
if not body or "name" not in body:
|
||||||
`merge: true` resolves a collision by merging into the existing tag.
|
return jsonify({"error": "name required"}), 400
|
||||||
"""
|
|
||||||
body = await request.get_json() or {}
|
|
||||||
has_name = "name" in body
|
|
||||||
has_fandom = "fandom_id" in body
|
|
||||||
if not has_name and not has_fandom:
|
|
||||||
return jsonify({"error": "name or fandom_id required"}), 400
|
|
||||||
do_merge = bool(body.get("merge"))
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
svc = TagService(session)
|
svc = TagService(session)
|
||||||
try:
|
try:
|
||||||
tag = None
|
|
||||||
if has_name:
|
|
||||||
tag = await svc.rename(tag_id, body["name"])
|
tag = await svc.rename(tag_id, body["name"])
|
||||||
if has_fandom:
|
|
||||||
tag = await svc.set_fandom(
|
|
||||||
tag_id, body["fandom_id"], merge=do_merge
|
|
||||||
)
|
|
||||||
except TagMergeConflict as exc:
|
except TagMergeConflict as exc:
|
||||||
return jsonify(
|
return jsonify(
|
||||||
{
|
{
|
||||||
@@ -257,12 +219,7 @@ async def update_tag(tag_id: int):
|
|||||||
return jsonify({"error": str(exc)}), 400
|
return jsonify({"error": str(exc)}), 400
|
||||||
await session.commit()
|
await session.commit()
|
||||||
return jsonify(
|
return jsonify(
|
||||||
{
|
{"id": tag.id, "name": tag.name, "kind": tag.kind.value}
|
||||||
"id": tag.id,
|
|
||||||
"name": tag.name,
|
|
||||||
"kind": tag.kind.value,
|
|
||||||
"fandom_id": tag.fandom_id,
|
|
||||||
}
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -369,31 +326,6 @@ def _series_err(exc: SeriesError):
|
|||||||
return jsonify({"error": msg}), status
|
return jsonify({"error": msg}), status
|
||||||
|
|
||||||
|
|
||||||
def _opt_int(body, key: str):
|
|
||||||
"""(value, error) — value is None when absent, error is (json, status)."""
|
|
||||||
if not body or body.get(key) is None:
|
|
||||||
return None, None
|
|
||||||
try:
|
|
||||||
return int(body[key]), None
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None, (jsonify({"error": f"{key} must be an integer"}), 400)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_int_list(body, key: str, *, max_ids: int = 500):
|
|
||||||
"""(list, error) for a required list of ints under `key`."""
|
|
||||||
if not body or key not in body:
|
|
||||||
return None, (jsonify({"error": f"{key} required"}), 400)
|
|
||||||
raw = body[key]
|
|
||||||
if not isinstance(raw, list) or not raw:
|
|
||||||
return None, (jsonify({"error": f"{key} must be a non-empty list"}), 400)
|
|
||||||
if len(raw) > max_ids:
|
|
||||||
return None, (jsonify({"error": f"too many ids (max {max_ids})"}), 400)
|
|
||||||
try:
|
|
||||||
return [int(x) for x in raw], None
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None, (jsonify({"error": f"{key} must be integers"}), 400)
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/<int:tag_id>/pages", methods=["GET"])
|
@tags_bp.route("/series/<int:tag_id>/pages", methods=["GET"])
|
||||||
async def series_pages(tag_id: int):
|
async def series_pages(tag_id: int):
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
@@ -410,14 +342,9 @@ async def series_add(tag_id: int):
|
|||||||
ids, err = _parse_bulk_ids(body, max_ids=500)
|
ids, err = _parse_bulk_ids(body, max_ids=500)
|
||||||
if err:
|
if err:
|
||||||
return err
|
return err
|
||||||
chapter_id, cerr = _opt_int(body, "chapter_id")
|
|
||||||
if cerr:
|
|
||||||
return cerr
|
|
||||||
async with get_session() as session:
|
async with get_session() as session:
|
||||||
try:
|
try:
|
||||||
n = await SeriesService(session).add_images(
|
n = await SeriesService(session).add_images(tag_id, ids)
|
||||||
tag_id, ids, chapter_id=chapter_id
|
|
||||||
)
|
|
||||||
except SeriesError as exc:
|
except SeriesError as exc:
|
||||||
return _series_err(exc)
|
return _series_err(exc)
|
||||||
await session.commit()
|
await session.commit()
|
||||||
@@ -470,227 +397,3 @@ async def series_cover(tag_id: int):
|
|||||||
return _series_err(exc)
|
return _series_err(exc)
|
||||||
await session.commit()
|
await session.commit()
|
||||||
return jsonify({"ok": True})
|
return jsonify({"ok": True})
|
||||||
|
|
||||||
|
|
||||||
# ---- chapters (FC-6.1) ----------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/<int:tag_id>/chapters", methods=["POST"])
|
|
||||||
async def series_chapter_create(tag_id: int):
|
|
||||||
body = await request.get_json() or {}
|
|
||||||
title = body.get("title")
|
|
||||||
if title is not None and not isinstance(title, str):
|
|
||||||
return jsonify({"error": "title must be a string"}), 400
|
|
||||||
is_placeholder = bool(body.get("is_placeholder", False))
|
|
||||||
start, serr = _opt_int(body, "stated_page_start")
|
|
||||||
if serr:
|
|
||||||
return serr
|
|
||||||
end, eerr = _opt_int(body, "stated_page_end")
|
|
||||||
if eerr:
|
|
||||||
return eerr
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
ch = await SeriesService(session).create_chapter(
|
|
||||||
tag_id,
|
|
||||||
title=title,
|
|
||||||
is_placeholder=is_placeholder,
|
|
||||||
stated_page_start=start,
|
|
||||||
stated_page_end=end,
|
|
||||||
)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify(ch)
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/<int:tag_id>/chapters/reorder", methods=["POST"])
|
|
||||||
async def series_chapter_reorder(tag_id: int):
|
|
||||||
body = await request.get_json()
|
|
||||||
ids, err = _parse_int_list(body, "chapter_ids")
|
|
||||||
if err:
|
|
||||||
return err
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
await SeriesService(session).reorder_chapters(tag_id, ids)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route(
|
|
||||||
"/series/<int:tag_id>/chapters/<int:chapter_id>", methods=["PATCH"]
|
|
||||||
)
|
|
||||||
async def series_chapter_update(tag_id: int, chapter_id: int):
|
|
||||||
body = await request.get_json() or {}
|
|
||||||
kwargs: dict = {}
|
|
||||||
if "title" in body:
|
|
||||||
if body["title"] is not None and not isinstance(body["title"], str):
|
|
||||||
return jsonify({"error": "title must be a string"}), 400
|
|
||||||
kwargs.update(set_title=True, title=body["title"])
|
|
||||||
if "stated_page_start" in body:
|
|
||||||
start, serr = _opt_int(body, "stated_page_start")
|
|
||||||
if serr:
|
|
||||||
return serr
|
|
||||||
kwargs.update(set_start=True, stated_page_start=start)
|
|
||||||
if "stated_page_end" in body:
|
|
||||||
end, eerr = _opt_int(body, "stated_page_end")
|
|
||||||
if eerr:
|
|
||||||
return eerr
|
|
||||||
kwargs.update(set_end=True, stated_page_end=end)
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
await SeriesService(session).update_chapter(
|
|
||||||
tag_id, chapter_id, **kwargs
|
|
||||||
)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route(
|
|
||||||
"/series/<int:tag_id>/chapters/<int:chapter_id>", methods=["DELETE"]
|
|
||||||
)
|
|
||||||
async def series_chapter_delete(tag_id: int, chapter_id: int):
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
await SeriesService(session).delete_chapter(tag_id, chapter_id)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route(
|
|
||||||
"/series/<int:tag_id>/chapters/<int:chapter_id>/merge", methods=["POST"]
|
|
||||||
)
|
|
||||||
async def series_chapter_merge(tag_id: int, chapter_id: int):
|
|
||||||
body = await request.get_json()
|
|
||||||
target, terr = _opt_int(body, "target_chapter_id")
|
|
||||||
if terr:
|
|
||||||
return terr
|
|
||||||
if target is None:
|
|
||||||
return jsonify({"error": "target_chapter_id required"}), 400
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
moved = await SeriesService(session).merge_chapter(
|
|
||||||
tag_id, chapter_id, target
|
|
||||||
)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"moved_count": moved})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route(
|
|
||||||
"/series/<int:tag_id>/chapters/<int:chapter_id>/reorder", methods=["POST"]
|
|
||||||
)
|
|
||||||
async def series_chapter_reorder_pages(tag_id: int, chapter_id: int):
|
|
||||||
body = await request.get_json()
|
|
||||||
ids, err = _parse_bulk_ids(body, max_ids=500)
|
|
||||||
if err:
|
|
||||||
return err
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
await SeriesService(session).reorder_pages(tag_id, chapter_id, ids)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
# ---- browse list + post→series flows (FC-6.2) -----------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series", methods=["GET"])
|
|
||||||
async def series_list():
|
|
||||||
args = request.args
|
|
||||||
sort = args.get("sort", "recent")
|
|
||||||
if sort not in ("recent", "name", "size"):
|
|
||||||
return jsonify({"error": "sort must be recent|name|size"}), 400
|
|
||||||
artist_id = None
|
|
||||||
if args.get("artist_id") is not None:
|
|
||||||
try:
|
|
||||||
artist_id = int(args["artist_id"])
|
|
||||||
except ValueError:
|
|
||||||
return jsonify({"error": "artist_id must be an integer"}), 400
|
|
||||||
async with get_session() as session:
|
|
||||||
rows = await SeriesService(session).list_series(
|
|
||||||
sort=sort, artist_id=artist_id
|
|
||||||
)
|
|
||||||
return jsonify({"series": rows})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/from-post", methods=["POST"])
|
|
||||||
async def series_from_post():
|
|
||||||
body = await request.get_json()
|
|
||||||
post_id, err = _opt_int(body, "post_id")
|
|
||||||
if err:
|
|
||||||
return err
|
|
||||||
if post_id is None:
|
|
||||||
return jsonify({"error": "post_id required"}), 400
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
out = await SeriesService(session).promote_post_to_series(post_id)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify(out)
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/<int:tag_id>/add-post", methods=["POST"])
|
|
||||||
async def series_add_post(tag_id: int):
|
|
||||||
body = await request.get_json()
|
|
||||||
post_id, err = _opt_int(body, "post_id")
|
|
||||||
if err:
|
|
||||||
return err
|
|
||||||
if post_id is None:
|
|
||||||
return jsonify({"error": "post_id required"}), 400
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
out = await SeriesService(session).add_post_as_chapter(tag_id, post_id)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify(out)
|
|
||||||
|
|
||||||
|
|
||||||
# ---- suggestion queue (FC-6.3) --------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/suggestions", methods=["GET"])
|
|
||||||
async def series_suggestions_list():
|
|
||||||
async with get_session() as session:
|
|
||||||
rows = await SeriesMatchService(session).list_pending()
|
|
||||||
return jsonify({"suggestions": rows})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/suggestions/<int:sid>/accept", methods=["POST"])
|
|
||||||
async def series_suggestion_accept(sid: int):
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
out = await SeriesMatchService(session).accept(sid)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify(out)
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/suggestions/<int:sid>/dismiss", methods=["POST"])
|
|
||||||
async def series_suggestion_dismiss(sid: int):
|
|
||||||
async with get_session() as session:
|
|
||||||
try:
|
|
||||||
await SeriesMatchService(session).dismiss(sid)
|
|
||||||
except SeriesError as exc:
|
|
||||||
return _series_err(exc)
|
|
||||||
await session.commit()
|
|
||||||
return jsonify({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
@tags_bp.route("/series/suggestions/rescan", methods=["POST"])
|
|
||||||
async def series_suggestions_rescan():
|
|
||||||
from ..tasks.admin import rescan_series_suggestions_task
|
|
||||||
|
|
||||||
res = rescan_series_suggestions_task.delay()
|
|
||||||
return jsonify({"task_id": res.id})
|
|
||||||
|
|||||||
@@ -43,16 +43,10 @@ def make_celery() -> Celery:
|
|||||||
"backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
|
"backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
|
||||||
"backend.app.tasks.download.*": {"queue": "download"},
|
"backend.app.tasks.download.*": {"queue": "download"},
|
||||||
"backend.app.tasks.scan.*": {"queue": "scan"},
|
"backend.app.tasks.scan.*": {"queue": "scan"},
|
||||||
# `maintenance` is the QUICK lane — recovery sweeps, vacuum, cleanup
|
|
||||||
# (concurrency-1 on the scheduler). The long one-shots (DB backups,
|
|
||||||
# library audits, admin maintenance: normalize/re-extract/cascade-
|
|
||||||
# delete) run on a SEPARATE `maintenance_long` lane + worker so they
|
|
||||||
# can never starve the quick self-healing sweeps (operator-flagged
|
|
||||||
# 2026-06-07: a 2h audit blocked vacuum/backup/normalize for hours).
|
|
||||||
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
|
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
|
||||||
"backend.app.tasks.backup.*": {"queue": "maintenance_long"},
|
"backend.app.tasks.backup.*": {"queue": "maintenance"},
|
||||||
"backend.app.tasks.admin.*": {"queue": "maintenance_long"},
|
"backend.app.tasks.admin.*": {"queue": "maintenance"},
|
||||||
"backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
|
"backend.app.tasks.library_audit.*": {"queue": "maintenance"},
|
||||||
},
|
},
|
||||||
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
|
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
|
||||||
task_acks_late=True,
|
task_acks_late=True,
|
||||||
@@ -103,10 +97,6 @@ def make_celery() -> Celery:
|
|||||||
"task": "backend.app.tasks.maintenance.prune_task_runs",
|
"task": "backend.app.tasks.maintenance.prune_task_runs",
|
||||||
"schedule": 86400.0, # daily
|
"schedule": 86400.0, # daily
|
||||||
},
|
},
|
||||||
"vacuum-analyze": {
|
|
||||||
"task": "backend.app.tasks.maintenance.vacuum_analyze",
|
|
||||||
"schedule": 604800.0, # weekly — reclaim dead-tuple bloat + refresh stats
|
|
||||||
},
|
|
||||||
"fc3h-backup-db-nightly": {
|
"fc3h-backup-db-nightly": {
|
||||||
"task": "backend.app.tasks.backup.backup_db_nightly",
|
"task": "backend.app.tasks.backup.backup_db_nightly",
|
||||||
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
|
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
|
||||||
|
|||||||
@@ -2,7 +2,6 @@
|
|||||||
|
|
||||||
from .app_setting import AppSetting
|
from .app_setting import AppSetting
|
||||||
from .artist import Artist
|
from .artist import Artist
|
||||||
from .artist_visit import ArtistVisit
|
|
||||||
from .backup_run import BackupRun
|
from .backup_run import BackupRun
|
||||||
from .base import Base
|
from .base import Base
|
||||||
from .credential import Credential
|
from .credential import Credential
|
||||||
@@ -14,13 +13,9 @@ from .import_settings import ImportSettings
|
|||||||
from .import_task import ImportTask
|
from .import_task import ImportTask
|
||||||
from .library_audit_run import LibraryAuditRun
|
from .library_audit_run import LibraryAuditRun
|
||||||
from .ml_settings import MLSettings
|
from .ml_settings import MLSettings
|
||||||
from .patreon_failed_media import PatreonFailedMedia
|
|
||||||
from .patreon_seen_media import PatreonSeenMedia
|
|
||||||
from .post import Post
|
from .post import Post
|
||||||
from .post_attachment import PostAttachment
|
from .post_attachment import PostAttachment
|
||||||
from .series_chapter import SeriesChapter
|
|
||||||
from .series_page import SeriesPage
|
from .series_page import SeriesPage
|
||||||
from .series_suggestion import SeriesSuggestion
|
|
||||||
from .source import Source
|
from .source import Source
|
||||||
from .tag import Tag, TagKind, image_tag
|
from .tag import Tag, TagKind, image_tag
|
||||||
from .tag_alias import TagAlias
|
from .tag_alias import TagAlias
|
||||||
@@ -33,17 +28,12 @@ __all__ = [
|
|||||||
"Base",
|
"Base",
|
||||||
"AppSetting",
|
"AppSetting",
|
||||||
"Artist",
|
"Artist",
|
||||||
"ArtistVisit",
|
|
||||||
"BackupRun",
|
"BackupRun",
|
||||||
"Source",
|
"Source",
|
||||||
"Credential",
|
"Credential",
|
||||||
"PatreonFailedMedia",
|
|
||||||
"PatreonSeenMedia",
|
|
||||||
"Post",
|
"Post",
|
||||||
"PostAttachment",
|
"PostAttachment",
|
||||||
"SeriesChapter",
|
|
||||||
"SeriesPage",
|
"SeriesPage",
|
||||||
"SeriesSuggestion",
|
|
||||||
"ImageRecord",
|
"ImageRecord",
|
||||||
"ImageProvenance",
|
"ImageProvenance",
|
||||||
"Tag",
|
"Tag",
|
||||||
|
|||||||
@@ -1,36 +0,0 @@
|
|||||||
"""ArtistVisit — per-artist 'last viewed' timestamp.
|
|
||||||
|
|
||||||
Powers the "+N new since last visit" badge on the artists directory and
|
|
||||||
the matching banner on `ArtistView`. One row per artist, single global
|
|
||||||
operator. When the multi-user model lands, the PK widens to
|
|
||||||
`(user_id, artist_id)` — currently aspirational only (no User model,
|
|
||||||
no services/access.py); operator approved skipping `user_id` for now
|
|
||||||
under rule #22 (breaking changes welcome).
|
|
||||||
|
|
||||||
Seed at migration time: every existing artist gets `last_viewed_at = NOW()`
|
|
||||||
so the badge starts at 0 across the board (no noisy "5000 unseen" on
|
|
||||||
first deploy). New artists also auto-get a row via
|
|
||||||
`ArtistService.find_or_create`.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from sqlalchemy import DateTime, ForeignKey, Integer, func
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
|
||||||
|
|
||||||
from .base import Base
|
|
||||||
|
|
||||||
|
|
||||||
class ArtistVisit(Base):
|
|
||||||
__tablename__ = "artist_visit"
|
|
||||||
|
|
||||||
artist_id: Mapped[int] = mapped_column(
|
|
||||||
Integer,
|
|
||||||
ForeignKey("artist.id", ondelete="CASCADE"),
|
|
||||||
primary_key=True,
|
|
||||||
)
|
|
||||||
last_viewed_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=func.now(),
|
|
||||||
)
|
|
||||||
@@ -74,17 +74,6 @@ class ImageRecord(Base):
|
|||||||
created_at: Mapped[datetime] = mapped_column(
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
# Denormalized gallery sort key = COALESCE(primary post's post_date,
|
|
||||||
# created_at) (alembic 0035). The gallery used to compute this as a
|
|
||||||
# COALESCE across the Post outer join on every /scroll, which can't use
|
|
||||||
# an index and re-sorted a large slice of the library per page (×10 with
|
|
||||||
# the old serial batching). Materializing it lets the cursor scroll read
|
|
||||||
# ix_image_record_effective_date directly. Maintained by the importer
|
|
||||||
# (services/importer.py _apply_sidecar) when a primary post with a date
|
|
||||||
# is linked; plain inserts keep the created_at-equivalent server default.
|
|
||||||
effective_date: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
updated_at: Mapped[datetime] = mapped_column(
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True),
|
DateTime(timezone=True),
|
||||||
nullable=False,
|
nullable=False,
|
||||||
|
|||||||
@@ -64,15 +64,6 @@ class ImportSettings(Base):
|
|||||||
Integer, nullable=False, default=3,
|
Integer, nullable=False, default=3,
|
||||||
)
|
)
|
||||||
|
|
||||||
# FC-6.3 series continuation matcher. enabled gates the rescan; threshold is
|
|
||||||
# the weighted-score cut-off (0..1) above which a pending suggestion is made.
|
|
||||||
series_suggest_enabled: Mapped[bool] = mapped_column(
|
|
||||||
Boolean, nullable=False, default=True,
|
|
||||||
)
|
|
||||||
series_suggest_threshold: Mapped[float] = mapped_column(
|
|
||||||
Float, nullable=False, default=0.5,
|
|
||||||
)
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
async def load(cls, session) -> ImportSettings:
|
async def load(cls, session) -> ImportSettings:
|
||||||
"""The singleton settings row (id=1), via an async session."""
|
"""The singleton settings row (id=1), via an async session."""
|
||||||
|
|||||||
@@ -35,10 +35,3 @@ class LibraryAuditRun(Base):
|
|||||||
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||||
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
|
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
|
||||||
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
# Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes
|
|
||||||
# from, and the last time a chunk made progress (so the recovery sweep can
|
|
||||||
# tell a progressing multi-chunk audit from a stuck one).
|
|
||||||
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
||||||
last_progress_at: Mapped[datetime | None] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=True,
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -1,45 +0,0 @@
|
|||||||
"""PatreonFailedMedia — per-source dead-letter ledger of Patreon media that
|
|
||||||
keeps failing to download/validate.
|
|
||||||
|
|
||||||
Plan #705 (#7). A media that fails every walk (404'd CDN URL, deleted post,
|
|
||||||
geo-blocked Mux stream, persistently-corrupt bytes) would otherwise re-error
|
|
||||||
forever and re-burn backfill chunks. After ``attempts`` reaches the dead-letter
|
|
||||||
threshold the ingester skips it on routine tick/backfill walks (recovery still
|
|
||||||
re-attempts it — the operator's "try everything again"). A later clean download
|
|
||||||
clears the row (the media recovered).
|
|
||||||
|
|
||||||
`filehash` is the same per-media key the seen-ledger uses (32-hex CDN MD5, or a
|
|
||||||
``video:`` / ``post:filename`` synthesized key) — hence String(128). UNIQUE
|
|
||||||
(source_id, filehash) is the upsert key.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
|
||||||
from sqlalchemy.types import DateTime
|
|
||||||
|
|
||||||
from .base import Base
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonFailedMedia(Base):
|
|
||||||
__tablename__ = "patreon_failed_media"
|
|
||||||
__table_args__ = (
|
|
||||||
UniqueConstraint(
|
|
||||||
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
source_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
||||||
)
|
|
||||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
||||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
|
||||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
||||||
first_failed_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
last_failed_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
"""PatreonSeenMedia — per-source ledger of Patreon media already
|
|
||||||
downloaded+processed.
|
|
||||||
|
|
||||||
Replaces gallery-dl's archive.sqlite3 with our own queryable table so
|
|
||||||
routine walks can skip media we've already ingested (and a future
|
|
||||||
"recovery" mode can deliberately bypass the ledger to re-walk).
|
|
||||||
|
|
||||||
`filehash` is normally a Patreon CDN MD5 (32 hex chars), but videos —
|
|
||||||
which have no stable content hash at discovery time — use a sentinel of
|
|
||||||
the form ``video:<post_id>:<media_id>``, hence String(128) rather than 32.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
|
||||||
from sqlalchemy.types import DateTime
|
|
||||||
|
|
||||||
from .base import Base
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonSeenMedia(Base):
|
|
||||||
__tablename__ = "patreon_seen_media"
|
|
||||||
__table_args__ = (
|
|
||||||
# Dedup key the downloader upserts against: one ledger row per
|
|
||||||
# (source, media). A second sighting of the same media is a no-op.
|
|
||||||
UniqueConstraint("source_id", "filehash", name="uq_patreon_seen_media_source_id"),
|
|
||||||
)
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
source_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
||||||
)
|
|
||||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
||||||
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
|
||||||
seen_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
"""SeriesChapter — an ordered chapter/part within a series.
|
|
||||||
|
|
||||||
A series IS a Tag(kind='series'); a chapter groups ordered SeriesPages under it.
|
|
||||||
Reading order is (chapter.chapter_number, series_page.page_number): chapter_number
|
|
||||||
sets the order of chapters, page_number orders pages within a chapter.
|
|
||||||
|
|
||||||
chapter_number is an ordering key only (not unique) — reorder rewrites 1..N
|
|
||||||
wholesale, mirroring series_page.page_number, so a reorder can't transiently
|
|
||||||
collide on a unique index.
|
|
||||||
|
|
||||||
A chapter may be a placeholder (is_placeholder=True) — a reserved empty slot for
|
|
||||||
a section the operator doesn't have yet; it holds no pages and shows as a gap in
|
|
||||||
the reader. stated_page_start/end carry the page range parsed from the source
|
|
||||||
post (FC-6.2), used to flag missing-page gaps; both are nullable when unknown.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, Text, func
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
|
||||||
|
|
||||||
from .base import Base
|
|
||||||
|
|
||||||
|
|
||||||
class SeriesChapter(Base):
|
|
||||||
__tablename__ = "series_chapter"
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
series_tag_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
|
||||||
)
|
|
||||||
chapter_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
||||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
||||||
is_placeholder: Mapped[bool] = mapped_column(
|
|
||||||
Boolean, nullable=False, server_default="false"
|
|
||||||
)
|
|
||||||
stated_page_start: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
||||||
stated_page_end: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
updated_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=func.now(),
|
|
||||||
onupdate=func.now(),
|
|
||||||
)
|
|
||||||
@@ -1,12 +1,9 @@
|
|||||||
"""SeriesPage — ordered image membership for a series-kind Tag.
|
"""SeriesPage — ordered image membership for a series-kind Tag.
|
||||||
|
|
||||||
A series IS a Tag with kind='series'; series_page gives it ordered pages,
|
A series IS a Tag with kind='series'; series_page gives it ordered pages.
|
||||||
grouped into chapters (FC-6). An image belongs to at most one series
|
An image belongs to at most one series (UNIQUE image_id). Cover = the
|
||||||
(UNIQUE image_id) ⇒ at most one chapter. Reading order is
|
lowest page_number. page_number is an ordering key only (not unique) —
|
||||||
(chapter.chapter_number, series_page.page_number): page_number orders pages
|
reorder rewrites 1..N wholesale.
|
||||||
WITHIN a chapter and is an ordering key only (not unique) — reorder rewrites
|
|
||||||
1..N wholesale. stated_page carries the page number parsed from the source
|
|
||||||
post (FC-6.2), nullable when unknown.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -24,18 +21,12 @@ class SeriesPage(Base):
|
|||||||
series_tag_id: Mapped[int] = mapped_column(
|
series_tag_id: Mapped[int] = mapped_column(
|
||||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
||||||
)
|
)
|
||||||
chapter_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("series_chapter.id", ondelete="CASCADE"),
|
|
||||||
nullable=False,
|
|
||||||
index=True,
|
|
||||||
)
|
|
||||||
image_id: Mapped[int] = mapped_column(
|
image_id: Mapped[int] = mapped_column(
|
||||||
ForeignKey("image_record.id", ondelete="CASCADE"),
|
ForeignKey("image_record.id", ondelete="CASCADE"),
|
||||||
nullable=False,
|
nullable=False,
|
||||||
unique=True,
|
unique=True,
|
||||||
)
|
)
|
||||||
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
page_number: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||||
stated_page: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,55 +0,0 @@
|
|||||||
"""SeriesSuggestion — a confirm-only "this post may continue this series" hint.
|
|
||||||
|
|
||||||
The matcher (FC-6.3) scores a (post, candidate series) pair from several weighted
|
|
||||||
signals and, above the configured threshold, records a pending suggestion. The
|
|
||||||
operator confirms (→ the post is added as a chapter) or dismisses it; FC never
|
|
||||||
files a post into a series on its own. status is a plain string (no Postgres
|
|
||||||
ENUM — see the check-existing-enums lesson): pending | added | dismissed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from sqlalchemy import (
|
|
||||||
JSON,
|
|
||||||
DateTime,
|
|
||||||
Float,
|
|
||||||
ForeignKey,
|
|
||||||
Integer,
|
|
||||||
String,
|
|
||||||
UniqueConstraint,
|
|
||||||
func,
|
|
||||||
)
|
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
|
||||||
|
|
||||||
from .base import Base
|
|
||||||
|
|
||||||
|
|
||||||
class SeriesSuggestion(Base):
|
|
||||||
__tablename__ = "series_suggestion"
|
|
||||||
__table_args__ = (
|
|
||||||
UniqueConstraint(
|
|
||||||
"post_id", "series_tag_id", name="uq_series_suggestion_post_series"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
post_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
|
||||||
)
|
|
||||||
series_tag_id: Mapped[int] = mapped_column(
|
|
||||||
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
|
|
||||||
)
|
|
||||||
score: Mapped[float] = mapped_column(Float, nullable=False)
|
|
||||||
signals: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
|
||||||
status: Mapped[str] = mapped_column(
|
|
||||||
String(16), nullable=False, server_default="pending", index=True
|
|
||||||
)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
||||||
)
|
|
||||||
updated_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True),
|
|
||||||
nullable=False,
|
|
||||||
server_default=func.now(),
|
|
||||||
onupdate=func.now(),
|
|
||||||
)
|
|
||||||
@@ -16,45 +16,9 @@ log = logging.getLogger(__name__)
|
|||||||
|
|
||||||
ARCHIVE_EXTS = {".zip", ".cbz", ".rar", ".7z"}
|
ARCHIVE_EXTS = {".zip", ".cbz", ".rar", ".7z"}
|
||||||
|
|
||||||
# Magic-byte signatures, so an archive with a mangled / extension-less filename
|
|
||||||
# is still recognised. Patreon attachment download URLs sanitize to names like
|
|
||||||
# `01_https___www.patreon.com_media-u_v3_131083093`, whose `Path.suffix` is junk
|
|
||||||
# (`.com_media-u_v3_131083093`), never `.zip` — an extension-only gate filed
|
|
||||||
# those as opaque PostAttachments and NEVER extracted them (operator-flagged
|
|
||||||
# 2026-06-06). Detection is by extension first (cheap), then header sniff.
|
|
||||||
_RAR_MAGIC = b"Rar!\x1a\x07"
|
|
||||||
_7Z_MAGIC = b"7z\xbc\xaf\x27\x1c"
|
|
||||||
|
|
||||||
|
|
||||||
def detect_archive_format(path: Path) -> str | None:
|
|
||||||
"""Return "zip" | "rar" | "7z" for an archive, else None.
|
|
||||||
|
|
||||||
Trusts a known extension first, then falls back to magic-byte sniffing so a
|
|
||||||
mis-named or extension-less archive is still handled. (zip covers .cbz too.)
|
|
||||||
"""
|
|
||||||
ext = Path(path).suffix.lower()
|
|
||||||
if ext in (".zip", ".cbz"):
|
|
||||||
return "zip"
|
|
||||||
if ext == ".rar":
|
|
||||||
return "rar"
|
|
||||||
if ext == ".7z":
|
|
||||||
return "7z"
|
|
||||||
try:
|
|
||||||
if zipfile.is_zipfile(path):
|
|
||||||
return "zip"
|
|
||||||
with open(path, "rb") as fh:
|
|
||||||
head = fh.read(8)
|
|
||||||
except OSError:
|
|
||||||
return None
|
|
||||||
if head.startswith(_RAR_MAGIC):
|
|
||||||
return "rar"
|
|
||||||
if head.startswith(_7Z_MAGIC):
|
|
||||||
return "7z"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def is_archive(path: Path) -> bool:
|
def is_archive(path: Path) -> bool:
|
||||||
return detect_archive_format(path) is not None
|
return Path(path).suffix.lower() in ARCHIVE_EXTS
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
@@ -68,16 +32,16 @@ def extract_archive(path: Path):
|
|||||||
members: list[tuple[str, Path]] = []
|
members: list[tuple[str, Path]] = []
|
||||||
try:
|
try:
|
||||||
try:
|
try:
|
||||||
fmt = detect_archive_format(path)
|
ext = Path(path).suffix.lower()
|
||||||
if fmt == "zip":
|
if ext in (".zip", ".cbz"):
|
||||||
with zipfile.ZipFile(path) as zf:
|
with zipfile.ZipFile(path) as zf:
|
||||||
zf.extractall(base)
|
zf.extractall(base)
|
||||||
elif fmt == "rar":
|
elif ext == ".rar":
|
||||||
import rarfile
|
import rarfile
|
||||||
|
|
||||||
with rarfile.RarFile(path) as rf:
|
with rarfile.RarFile(path) as rf:
|
||||||
rf.extractall(base)
|
rf.extractall(base)
|
||||||
elif fmt == "7z":
|
elif ext == ".7z":
|
||||||
import py7zr
|
import py7zr
|
||||||
|
|
||||||
with py7zr.SevenZipFile(path, "r") as zf:
|
with py7zr.SevenZipFile(path, "r") as zf:
|
||||||
|
|||||||
@@ -13,10 +13,10 @@ from __future__ import annotations
|
|||||||
import base64
|
import base64
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
from sqlalchemy import and_, case, exists, func, or_, select
|
from sqlalchemy import and_, exists, func, or_, select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from ..models import Artist, ArtistVisit, ImageRecord, Source
|
from ..models import Artist, ImageRecord, Source
|
||||||
from .gallery_service import thumbnail_url
|
from .gallery_service import thumbnail_url
|
||||||
|
|
||||||
_SEP = "|"
|
_SEP = "|"
|
||||||
@@ -58,27 +58,9 @@ class ArtistDirectoryService:
|
|||||||
raise ValueError("limit must be between 1 and 200")
|
raise ValueError("limit must be between 1 and 200")
|
||||||
|
|
||||||
count_col = func.count(ImageRecord.id).label("image_count")
|
count_col = func.count(ImageRecord.id).label("image_count")
|
||||||
# Unseen = images imported since the artist's last_viewed_at.
|
|
||||||
# NULL last_viewed_at (artist created before alembic 0034 seed
|
|
||||||
# or before find_or_create autoseed) defensively counts as
|
|
||||||
# "never visited" → all images unseen. Single grouped query, no
|
|
||||||
# N+1.
|
|
||||||
unseen_col = func.count(
|
|
||||||
case(
|
|
||||||
(
|
|
||||||
or_(
|
|
||||||
ArtistVisit.last_viewed_at.is_(None),
|
|
||||||
ImageRecord.created_at > ArtistVisit.last_viewed_at,
|
|
||||||
),
|
|
||||||
ImageRecord.id,
|
|
||||||
),
|
|
||||||
else_=None,
|
|
||||||
)
|
|
||||||
).label("unseen_count")
|
|
||||||
stmt = (
|
stmt = (
|
||||||
select(Artist, count_col, unseen_col)
|
select(Artist, count_col)
|
||||||
.outerjoin(ImageRecord, ImageRecord.artist_id == Artist.id)
|
.outerjoin(ImageRecord, ImageRecord.artist_id == Artist.id)
|
||||||
.outerjoin(ArtistVisit, ArtistVisit.artist_id == Artist.id)
|
|
||||||
.group_by(Artist.id)
|
.group_by(Artist.id)
|
||||||
)
|
)
|
||||||
if q:
|
if q:
|
||||||
@@ -112,7 +94,7 @@ class ArtistDirectoryService:
|
|||||||
next_cursor = _encode(last_artist.name, last_artist.id)
|
next_cursor = _encode(last_artist.name, last_artist.id)
|
||||||
rows = rows[:limit]
|
rows = rows[:limit]
|
||||||
|
|
||||||
artist_ids = [a.id for a, _, _ in rows]
|
artist_ids = [a.id for a, _ in rows]
|
||||||
previews = await self._previews(artist_ids)
|
previews = await self._previews(artist_ids)
|
||||||
|
|
||||||
cards = [
|
cards = [
|
||||||
@@ -122,10 +104,9 @@ class ArtistDirectoryService:
|
|||||||
"slug": artist.slug,
|
"slug": artist.slug,
|
||||||
"is_subscription": bool(artist.is_subscription),
|
"is_subscription": bool(artist.is_subscription),
|
||||||
"image_count": int(image_count),
|
"image_count": int(image_count),
|
||||||
"unseen_count": int(unseen_count),
|
|
||||||
"preview_thumbnails": previews.get(artist.id, []),
|
"preview_thumbnails": previews.get(artist.id, []),
|
||||||
}
|
}
|
||||||
for artist, image_count, unseen_count in rows
|
for artist, image_count in rows
|
||||||
]
|
]
|
||||||
return DirectoryPage(cards=cards, next_cursor=next_cursor)
|
return DirectoryPage(cards=cards, next_cursor=next_cursor)
|
||||||
|
|
||||||
|
|||||||
@@ -9,13 +9,11 @@ Dates come from Post.post_date via ImageProvenance.post_id.
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
from sqlalchemy import and_, case, func, or_, select
|
from sqlalchemy import and_, case, func, or_, select
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from ..models import (
|
from ..models import (
|
||||||
Artist,
|
Artist,
|
||||||
ArtistVisit,
|
|
||||||
ImageProvenance,
|
ImageProvenance,
|
||||||
ImageRecord,
|
ImageRecord,
|
||||||
Post,
|
Post,
|
||||||
@@ -124,12 +122,6 @@ class ArtistService:
|
|||||||
)
|
)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
|
||||||
# Mark this artist as "visited now"; the returned count is what
|
|
||||||
# the operator should see in the banner ("N new since last
|
|
||||||
# visit"). Done LAST so the read aggregates above all see the
|
|
||||||
# pre-visit state (cosmetic — none depend on visit data).
|
|
||||||
unseen_at_visit = await self._mark_visited_returning_unseen(aid)
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"id": artist.id,
|
"id": artist.id,
|
||||||
"name": artist.name,
|
"name": artist.name,
|
||||||
@@ -137,7 +129,6 @@ class ArtistService:
|
|||||||
"is_subscription": bool(artist.is_subscription),
|
"is_subscription": bool(artist.is_subscription),
|
||||||
"image_count": int(image_count),
|
"image_count": int(image_count),
|
||||||
"post_count": int(post_count),
|
"post_count": int(post_count),
|
||||||
"unseen_count_at_visit": unseen_at_visit,
|
|
||||||
"date_range": {
|
"date_range": {
|
||||||
"min": dmin.isoformat() if dmin else None,
|
"min": dmin.isoformat() if dmin else None,
|
||||||
"max": dmax.isoformat() if dmax else None,
|
"max": dmax.isoformat() if dmax else None,
|
||||||
@@ -166,39 +157,6 @@ class ArtistService:
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
async def _mark_visited_returning_unseen(self, artist_id: int) -> int:
|
|
||||||
"""Read pre-visit `last_viewed_at`, count images added since,
|
|
||||||
then upsert `last_viewed_at = NOW()`. Returns the count BEFORE
|
|
||||||
the upsert so the banner has data to render.
|
|
||||||
|
|
||||||
Postgres UPSERT (`ON CONFLICT DO UPDATE`) keeps the write
|
|
||||||
atomic — no SELECT-then-INSERT race per
|
|
||||||
`reference_scalar_one_or_none_duplicates`.
|
|
||||||
"""
|
|
||||||
prev = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(ArtistVisit.last_viewed_at).where(
|
|
||||||
ArtistVisit.artist_id == artist_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
|
|
||||||
count_stmt = select(func.count(ImageRecord.id)).where(
|
|
||||||
ImageRecord.artist_id == artist_id
|
|
||||||
)
|
|
||||||
if prev is not None:
|
|
||||||
count_stmt = count_stmt.where(ImageRecord.created_at > prev)
|
|
||||||
unseen = (await self.session.execute(count_stmt)).scalar_one()
|
|
||||||
|
|
||||||
upsert = pg_insert(ArtistVisit.__table__).values(artist_id=artist_id)
|
|
||||||
upsert = upsert.on_conflict_do_update(
|
|
||||||
index_elements=["artist_id"],
|
|
||||||
set_={"last_viewed_at": func.now()},
|
|
||||||
)
|
|
||||||
await self.session.execute(upsert)
|
|
||||||
await self.session.commit()
|
|
||||||
return int(unseen)
|
|
||||||
|
|
||||||
async def images(
|
async def images(
|
||||||
self, slug: str, cursor: str | None, limit: int = 60
|
self, slug: str, cursor: str | None, limit: int = 60
|
||||||
) -> ArtistImagesPage | None:
|
) -> ArtistImagesPage | None:
|
||||||
@@ -272,13 +230,6 @@ class ArtistService:
|
|||||||
artist = Artist(name=cleaned, slug=slug)
|
artist = Artist(name=cleaned, slug=slug)
|
||||||
self.session.add(artist)
|
self.session.add(artist)
|
||||||
await self.session.flush()
|
await self.session.flush()
|
||||||
# New artist starts "caught up" — seed ArtistVisit so the
|
|
||||||
# directory's `+N new` badge stays at 0 until real new
|
|
||||||
# content arrives. Without this, the unseen-count query
|
|
||||||
# treats NULL last_viewed_at as "never visited" and would
|
|
||||||
# count every image imported in the same session.
|
|
||||||
self.session.add(ArtistVisit(artist_id=artist.id))
|
|
||||||
await self.session.flush()
|
|
||||||
await sp.commit()
|
await sp.commit()
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
await sp.rollback()
|
await sp.rollback()
|
||||||
|
|||||||
@@ -15,10 +15,7 @@ lifecycle + soft/hard time limits + retention bookkeeping.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -29,35 +26,6 @@ _BACKUPS_DIRNAME = "_backups"
|
|||||||
# blocking syscall ignores that signal. These bound the worst case.
|
# blocking syscall ignores that signal. These bound the worst case.
|
||||||
_DB_SUBPROCESS_TIMEOUT_S = 12 * 60 # 12 min (Celery soft is 10 min)
|
_DB_SUBPROCESS_TIMEOUT_S = 12 * 60 # 12 min (Celery soft is 10 min)
|
||||||
_IMAGES_SUBPROCESS_TIMEOUT_S = 7 * 60 * 60 # 7 hr (Celery soft is 6 hr)
|
_IMAGES_SUBPROCESS_TIMEOUT_S = 7 * 60 * 60 # 7 hr (Celery soft is 6 hr)
|
||||||
# Grace after SIGKILL to reap the child. If it can't be reaped in this window
|
|
||||||
# (an uninterruptible NFS D-state — the failure mode that wedged the
|
|
||||||
# concurrency-1 maintenance lane for hours, operator-flagged 2026-06-07), we
|
|
||||||
# STOP waiting and fail fast, freeing the worker slot. The orphan is reaped by
|
|
||||||
# the OS once its blocking syscall clears.
|
|
||||||
_KILL_REAP_GRACE_S = 10
|
|
||||||
|
|
||||||
|
|
||||||
def _run_bounded(cmd: list[str], timeout: int) -> None:
|
|
||||||
"""subprocess.run(check=True, timeout) whose reaper can't itself hang.
|
|
||||||
|
|
||||||
subprocess.run's timeout path SIGKILLs the child then blocks in wait() to
|
|
||||||
reap it — but a process stuck in uninterruptible I/O (NFS) can't be reaped,
|
|
||||||
so wait() blocks for hours. Here we bound the post-kill reap and re-raise
|
|
||||||
TimeoutExpired regardless, so the caller fails fast instead of wedging."""
|
|
||||||
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
|
||||||
try:
|
|
||||||
out, err = proc.communicate(timeout=timeout)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
proc.kill()
|
|
||||||
try:
|
|
||||||
proc.communicate(timeout=_KILL_REAP_GRACE_S)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass # unkillable (D-state) — abandon the reap, fail fast
|
|
||||||
raise
|
|
||||||
if proc.returncode != 0:
|
|
||||||
raise subprocess.CalledProcessError(
|
|
||||||
proc.returncode, cmd, output=out, stderr=err
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _libpq_url(sa_url: str) -> str:
|
def _libpq_url(sa_url: str) -> str:
|
||||||
@@ -116,25 +84,14 @@ def backup_db(
|
|||||||
ts = _now_ts()
|
ts = _now_ts()
|
||||||
out_dir = _backups_dir(images_root)
|
out_dir = _backups_dir(images_root)
|
||||||
sql_path = out_dir / f"fc_db_{ts}.sql"
|
sql_path = out_dir / f"fc_db_{ts}.sql"
|
||||||
# Dump to LOCAL disk first, then move the finished file to the (NFS) backups
|
subprocess.run(
|
||||||
# dir. pg_dump's long phase is then a DB-socket wait + local writes — both
|
|
||||||
# killable — instead of an NFS write that can hang uninterruptibly. Only the
|
|
||||||
# final move touches NFS, and it's a bounded single-file step.
|
|
||||||
fd, tmp_name = tempfile.mkstemp(prefix="fc_db_", suffix=".sql")
|
|
||||||
os.close(fd)
|
|
||||||
tmp_path = Path(tmp_name)
|
|
||||||
try:
|
|
||||||
_run_bounded(
|
|
||||||
[
|
[
|
||||||
"pg_dump", "--no-owner", "--no-acl",
|
"pg_dump", "--no-owner", "--no-acl",
|
||||||
"-f", str(tmp_path), _libpq_url(db_url),
|
"-f", str(sql_path), _libpq_url(db_url),
|
||||||
],
|
],
|
||||||
_DB_SUBPROCESS_TIMEOUT_S,
|
capture_output=True, check=True,
|
||||||
|
timeout=_DB_SUBPROCESS_TIMEOUT_S,
|
||||||
)
|
)
|
||||||
shutil.move(str(tmp_path), str(sql_path))
|
|
||||||
finally:
|
|
||||||
if tmp_path.exists():
|
|
||||||
tmp_path.unlink(missing_ok=True)
|
|
||||||
manifest_path = _write_manifest(
|
manifest_path = _write_manifest(
|
||||||
out_dir, kind="db", ts=ts, tag=tag, triggered_by=triggered_by,
|
out_dir, kind="db", ts=ts, tag=tag, triggered_by=triggered_by,
|
||||||
artifact_path=sql_path,
|
artifact_path=sql_path,
|
||||||
@@ -157,17 +114,15 @@ def backup_images(
|
|||||||
ts = _now_ts()
|
ts = _now_ts()
|
||||||
out_dir = _backups_dir(images_root)
|
out_dir = _backups_dir(images_root)
|
||||||
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
|
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
|
||||||
# No local-temp here (the archive is hundreds of GB — it can't stage in
|
subprocess.run(
|
||||||
# /tmp), but bounded-kill still applies so a tar wedged on NFS fails fast
|
|
||||||
# rather than holding the lane for hours.
|
|
||||||
_run_bounded(
|
|
||||||
[
|
[
|
||||||
"tar", "--zstd", "-cf", str(tar_path),
|
"tar", "--zstd", "-cf", str(tar_path),
|
||||||
"-C", str(images_root.parent), images_root.name,
|
"-C", str(images_root.parent), images_root.name,
|
||||||
f"--exclude={images_root.name}/_backups",
|
f"--exclude={images_root.name}/_backups",
|
||||||
f"--exclude={images_root.name}/_quarantine",
|
f"--exclude={images_root.name}/_quarantine",
|
||||||
],
|
],
|
||||||
_IMAGES_SUBPROCESS_TIMEOUT_S,
|
capture_output=True, check=True,
|
||||||
|
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
|
||||||
)
|
)
|
||||||
manifest_path = _write_manifest(
|
manifest_path = _write_manifest(
|
||||||
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
|
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
|
||||||
|
|||||||
@@ -11,8 +11,6 @@ the one-and-done GS/IR migration tooling.)
|
|||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
|
||||||
import time
|
|
||||||
from datetime import UTC, datetime, timedelta
|
from datetime import UTC, datetime, timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -24,8 +22,6 @@ from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
|
|||||||
from ..models.series_page import SeriesPage
|
from ..models.series_page import SeriesPage
|
||||||
from ..models.tag import image_tag
|
from ..models.tag import image_tag
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
def project_artist_cascade(session: Session, *, slug: str) -> dict:
|
||||||
"""Read-only projection of what delete_artist_cascade would touch.
|
"""Read-only projection of what delete_artist_cascade would touch.
|
||||||
@@ -459,62 +455,6 @@ def purge_legacy_tags(session: Session, *, dry_run: bool = False) -> dict:
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
# The Camie-suggestable CONTENT vocabulary. "Reset content tagging" wipes
|
|
||||||
# these so the operator can re-tag from scratch via auto-suggest. fandom +
|
|
||||||
# series (and series_page ordering) are deliberately NOT here — they're kept.
|
|
||||||
RESETTABLE_TAG_KINDS = ("general", "character")
|
|
||||||
|
|
||||||
|
|
||||||
def reset_content_tagging(session: Session, *, dry_run: bool = False) -> dict:
|
|
||||||
"""Count (dry_run) or DELETE every general + character tag so the operator
|
|
||||||
can re-tag from scratch via the Camie auto-suggest.
|
|
||||||
|
|
||||||
PRESERVED: fandom + series tags and their series_page ordering, plus every
|
|
||||||
image's image_record.tagger_predictions (untouched) so suggestions
|
|
||||||
repopulate immediately. CASCADE on image_tag / tag_alias / tag_allowlist /
|
|
||||||
tag_reference_embedding / tag_suggestion_rejection clears each deleted
|
|
||||||
tag's applications + metadata. Tag.fandom_id is SET NULL, so deleting
|
|
||||||
character tags never touches the fandom rows. Irreversible except via DB
|
|
||||||
backup restore.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
{"by_kind": {"general": N, "character": M},
|
|
||||||
"count": total tags,
|
|
||||||
"applications": image_tag rows that will be / were removed,
|
|
||||||
"sample_names": [first 50],
|
|
||||||
and on live runs "deleted": total}
|
|
||||||
"""
|
|
||||||
predicate = Tag.kind.in_(RESETTABLE_TAG_KINDS)
|
|
||||||
rows = session.execute(
|
|
||||||
select(Tag.id, Tag.name, Tag.kind).where(predicate)
|
|
||||||
).all()
|
|
||||||
by_kind: dict[str, int] = {}
|
|
||||||
for _id, _name, kind in rows:
|
|
||||||
key = kind.value if hasattr(kind, "value") else str(kind)
|
|
||||||
by_kind[key] = by_kind.get(key, 0) + 1
|
|
||||||
# Headline impact: applications (image_tag rows) that vanish via cascade.
|
|
||||||
applications = session.execute(
|
|
||||||
select(func.count())
|
|
||||||
.select_from(image_tag)
|
|
||||||
.where(image_tag.c.tag_id.in_(select(Tag.id).where(predicate)))
|
|
||||||
).scalar_one()
|
|
||||||
sample = [name for _id, name, _kind in rows[:50]]
|
|
||||||
total = len(rows)
|
|
||||||
result = {
|
|
||||||
"by_kind": by_kind,
|
|
||||||
"count": total,
|
|
||||||
"applications": applications,
|
|
||||||
"sample_names": sample,
|
|
||||||
}
|
|
||||||
if dry_run:
|
|
||||||
return result
|
|
||||||
if total:
|
|
||||||
session.execute(Tag.__table__.delete().where(predicate))
|
|
||||||
session.commit()
|
|
||||||
result["deleted"] = total
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
|
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -669,170 +609,3 @@ def cancel_audit_run(session: Session, *, audit_id: int) -> None:
|
|||||||
.where(LibraryAuditRun.status == "running")
|
.where(LibraryAuditRun.status == "running")
|
||||||
.values(status="cancelled", finished_at=datetime.now(UTC))
|
.values(status="cancelled", finished_at=datetime.now(UTC))
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# -- archive-attachment re-extraction (#713 part 2) ------------------------
|
|
||||||
|
|
||||||
_ARCHIVE_EXT_FOR_FORMAT = {"zip": ".zip", "rar": ".rar", "7z": ".7z"}
|
|
||||||
|
|
||||||
|
|
||||||
def _reextract_archive_to_post(
|
|
||||||
importer, archive_path: Path, post, source_row, artist, images_root: Path,
|
|
||||||
) -> list[int]:
|
|
||||||
"""Extract one stored archive and link its members to `post`.
|
|
||||||
|
|
||||||
The stored attachment has no adjacent sidecar (it lives in the sha-addressed
|
|
||||||
attachment store). Stage a copy + a reconstructed sidecar UNDER the artist's
|
|
||||||
library dir (`images_root/<slug>/<platform>/<post>/`) — the importer
|
|
||||||
re-derives the artist from the path AND copies members relative to it, so the
|
|
||||||
members land in the real library and resolve to the right artist — then re-run
|
|
||||||
`attach_in_place`: the archive extracts and `find_or_create_post` re-attaches
|
|
||||||
the members to the SAME Post (source_id + external_post_id). Removes only the
|
|
||||||
staged archive + sidecar afterward; the imported member files stay. Returns
|
|
||||||
the new member image ids.
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
|
|
||||||
from .archive_extractor import detect_archive_format
|
|
||||||
|
|
||||||
fmt = detect_archive_format(archive_path)
|
|
||||||
ext = _ARCHIVE_EXT_FOR_FORMAT.get(fmt or "", ".zip")
|
|
||||||
platform = source_row.platform if source_row is not None else "imported"
|
|
||||||
sidecar = {
|
|
||||||
"category": source_row.platform if source_row is not None
|
|
||||||
else (post.raw_metadata or {}).get("category"),
|
|
||||||
"id": post.external_post_id,
|
|
||||||
"title": post.post_title or "",
|
|
||||||
"content": post.description or "",
|
|
||||||
"published_at": post.post_date.isoformat() if post.post_date else None,
|
|
||||||
"url": post.post_url,
|
|
||||||
}
|
|
||||||
work = images_root / artist.slug / platform / str(post.external_post_id)
|
|
||||||
work.mkdir(parents=True, exist_ok=True)
|
|
||||||
staged = work / f"archive{ext}" # clean ext → is_archive + find_sidecar
|
|
||||||
sidecar_path = staged.with_suffix(".json")
|
|
||||||
try:
|
|
||||||
shutil.copy2(archive_path, staged)
|
|
||||||
sidecar_path.write_text(json.dumps(sidecar))
|
|
||||||
res = importer.attach_in_place(staged, artist=artist, source=source_row)
|
|
||||||
return list(res.member_image_ids or [])
|
|
||||||
finally:
|
|
||||||
# Drop only the staged archive + sidecar; the extracted member files
|
|
||||||
# were copied into the library alongside them and must stay.
|
|
||||||
staged.unlink(missing_ok=True)
|
|
||||||
sidecar_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
def reextract_archive_attachments(
|
|
||||||
session: Session,
|
|
||||||
*,
|
|
||||||
images_root: Path,
|
|
||||||
time_budget_seconds: float | None = None,
|
|
||||||
after_id: int = 0,
|
|
||||||
) -> dict:
|
|
||||||
"""Re-process existing PostAttachments that are ACTUALLY archives but were
|
|
||||||
filed opaquely before #713 part 1 (extension-only is_archive missed mangled /
|
|
||||||
extension-less Patreon attachment names). For each: extract the members,
|
|
||||||
import them, and link them to the attachment's post.
|
|
||||||
|
|
||||||
Idempotent — members dedupe by sha256, the archive dedupes by sha — so it's
|
|
||||||
safe to run repeatedly. Returns a summary dict for task_run.metadata.
|
|
||||||
|
|
||||||
Time-boxed + resumable: scans PostAttachments in ascending id order starting
|
|
||||||
after ``after_id``. When ``time_budget_seconds`` elapses, stops and reports
|
|
||||||
``partial=True`` + ``resume_after_id`` (the last scanned id) so the task can
|
|
||||||
re-enqueue itself and continue — a large archive back-catalog can't run the
|
|
||||||
task into the Celery time limit or hog the maintenance lane. A bare re-run
|
|
||||||
(after_id=0) would never advance because an already-extracted archive is
|
|
||||||
still an archive on disk, so the cursor is what guarantees forward progress.
|
|
||||||
"""
|
|
||||||
from ..models import ImportSettings, Post, PostAttachment, Source
|
|
||||||
from ..tasks.ml import tag_and_embed
|
|
||||||
from ..tasks.thumbnail import generate_thumbnail
|
|
||||||
from .archive_extractor import is_archive
|
|
||||||
from .importer import Importer
|
|
||||||
from .thumbnailer import Thumbnailer
|
|
||||||
|
|
||||||
summary = {
|
|
||||||
"scanned": 0, "archives": 0, "members_imported": 0,
|
|
||||||
"posts_touched": 0, "skipped_no_post": 0, "skipped_no_artist": 0,
|
|
||||||
"errors": 0, "partial": False, "resume_after_id": after_id,
|
|
||||||
}
|
|
||||||
settings = ImportSettings.load_sync(session)
|
|
||||||
importer = Importer(
|
|
||||||
session=session, images_root=images_root, import_root=images_root,
|
|
||||||
thumbnailer=Thumbnailer(images_root=images_root), settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
attachments = session.execute(
|
|
||||||
select(PostAttachment)
|
|
||||||
.where(PostAttachment.id > after_id)
|
|
||||||
.order_by(PostAttachment.id)
|
|
||||||
).scalars().all()
|
|
||||||
enqueue_ids: list[int] = []
|
|
||||||
start = time.monotonic()
|
|
||||||
for att in attachments:
|
|
||||||
summary["scanned"] += 1
|
|
||||||
summary["resume_after_id"] = att.id
|
|
||||||
stored = Path(att.path)
|
|
||||||
try:
|
|
||||||
if not stored.is_file() or not is_archive(stored):
|
|
||||||
continue
|
|
||||||
except OSError:
|
|
||||||
continue
|
|
||||||
summary["archives"] += 1
|
|
||||||
if att.post_id is None:
|
|
||||||
summary["skipped_no_post"] += 1
|
|
||||||
continue
|
|
||||||
post = session.get(Post, att.post_id)
|
|
||||||
if post is None:
|
|
||||||
summary["skipped_no_post"] += 1
|
|
||||||
continue
|
|
||||||
artist = session.get(Artist, att.artist_id) if att.artist_id else None
|
|
||||||
if artist is None and post.artist_id:
|
|
||||||
artist = session.get(Artist, post.artist_id)
|
|
||||||
if artist is None or not artist.slug:
|
|
||||||
# The importer re-derives the artist from the staged path, so we need
|
|
||||||
# a real artist+slug to anchor under. (Shouldn't happen for
|
|
||||||
# subscription posts; skip rather than orphan the members.)
|
|
||||||
summary["skipped_no_artist"] += 1
|
|
||||||
continue
|
|
||||||
source_row = session.get(Source, post.source_id) if post.source_id else None
|
|
||||||
try:
|
|
||||||
ids = _reextract_archive_to_post(
|
|
||||||
importer, stored, post, source_row, artist, images_root,
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
except Exception as exc: # one bad archive must not strand the rest
|
|
||||||
session.rollback()
|
|
||||||
summary["errors"] += 1
|
|
||||||
log.warning("re-extract failed for attachment %s: %s", att.id, exc)
|
|
||||||
continue
|
|
||||||
if ids:
|
|
||||||
summary["members_imported"] += len(ids)
|
|
||||||
summary["posts_touched"] += 1
|
|
||||||
enqueue_ids.extend(ids)
|
|
||||||
|
|
||||||
# Time-box the chunk. resume_after_id already points at this attachment,
|
|
||||||
# so the next run starts strictly after it. Checked after the commit so a
|
|
||||||
# half-extracted archive never straddles the boundary.
|
|
||||||
if (
|
|
||||||
time_budget_seconds is not None
|
|
||||||
and time.monotonic() - start >= time_budget_seconds
|
|
||||||
):
|
|
||||||
summary["partial"] = True
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
# Loop ran to exhaustion — nothing left to resume.
|
|
||||||
summary["partial"] = False
|
|
||||||
|
|
||||||
# Thumbnails + ML for the newly-imported members (best-effort; off the
|
|
||||||
# critical path — a Redis hiccup must not fail the whole re-extract).
|
|
||||||
for img_id in enqueue_ids:
|
|
||||||
try:
|
|
||||||
generate_thumbnail.delay(img_id)
|
|
||||||
tag_and_embed.delay(img_id)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("re-extract enqueue failed for image %s: %s", img_id, exc)
|
|
||||||
return summary
|
|
||||||
|
|||||||
@@ -1,241 +0,0 @@
|
|||||||
"""Platform → download-backend dispatch (one place that knows which platforms
|
|
||||||
are served by the native FC ingester vs. the gallery-dl subprocess).
|
|
||||||
|
|
||||||
gallery-dl wasn't built to be driven by an automated scheduler — no native
|
|
||||||
checkpoint/resume, no structured logs, per-file HEADs that dominate wall-clock.
|
|
||||||
The native ingester (services/patreon_ingester.py, plan #697) replaces it for
|
|
||||||
Patreon and is the path we grow as more platforms migrate. To keep that
|
|
||||||
migration DRY, every caller that has to behave differently per backend —
|
|
||||||
download routing, the credential-verify probe, cursor handling — asks THIS
|
|
||||||
module instead of testing ``platform == "patreon"`` inline. When a platform gets
|
|
||||||
a native ingester, it moves into ``NATIVE_INGESTER_PLATFORMS`` here and both the
|
|
||||||
download path and verify switch over together.
|
|
||||||
|
|
||||||
The backend surfaces share a UNIFORM signature so a caller invokes the same
|
|
||||||
function regardless of platform:
|
|
||||||
- verify_credential(...) → (ok: bool|None, message: str)
|
|
||||||
- (download stays in download_service for now; uses_native_ingester() is the
|
|
||||||
shared predicate it routes on, so the decision lives here too.)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from .gallery_dl import DownloadResult, ErrorType
|
|
||||||
from .patreon_ingester import PatreonIngester
|
|
||||||
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
|
|
||||||
|
|
||||||
# Platforms whose download + verify go through the native ingester rather than
|
|
||||||
# gallery-dl. gallery-dl still serves every other platform (subscribestar,
|
|
||||||
# hentaifoundry, discord, pixiv, deviantart) unchanged.
|
|
||||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon"})
|
|
||||||
|
|
||||||
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
|
|
||||||
# messages so the operator sees the exact lookup endpoint that was hit.
|
|
||||||
_CAMPAIGNS_API = "https://www.patreon.com/api/campaigns"
|
|
||||||
|
|
||||||
|
|
||||||
def uses_native_ingester(platform: str) -> bool:
|
|
||||||
"""True when `platform` is served by the native ingester (not gallery-dl).
|
|
||||||
The single predicate the download path and verify both route on."""
|
|
||||||
return platform in NATIVE_INGESTER_PLATFORMS
|
|
||||||
|
|
||||||
|
|
||||||
async def run_download(
|
|
||||||
*,
|
|
||||||
ctx: dict,
|
|
||||||
source_config,
|
|
||||||
skip_value: bool | str,
|
|
||||||
mode: str | None,
|
|
||||||
gdl,
|
|
||||||
sync_session_factory,
|
|
||||||
) -> tuple[DownloadResult, str | None]:
|
|
||||||
"""Uniform download across backends — the download counterpart to
|
|
||||||
`verify_source_credential`, so this module is the ONE place that knows how
|
|
||||||
each platform both downloads AND verifies (the seam that makes adding a
|
|
||||||
platform a bounded job).
|
|
||||||
|
|
||||||
Returns `(DownloadResult, resolved_campaign_id)`; `resolved_campaign_id` is
|
|
||||||
non-None only when a native vanity lookup ran this call (so phase 3 caches
|
|
||||||
it). Native platforms route through their ingester in `mode`
|
|
||||||
(tick/backfill/recovery); gallery-dl platforms run the subprocess. The caller
|
|
||||||
(download_service) prepares `source_config`/`skip_value`/`mode` from the
|
|
||||||
backfill state machine and owns phase 3.
|
|
||||||
"""
|
|
||||||
platform = ctx["platform"]
|
|
||||||
if uses_native_ingester(platform):
|
|
||||||
return await _run_native_ingester(
|
|
||||||
ctx, source_config, mode, gdl, sync_session_factory
|
|
||||||
)
|
|
||||||
result = await gdl.download(
|
|
||||||
url=ctx["url"],
|
|
||||||
artist_slug=ctx["artist_slug"],
|
|
||||||
platform=platform,
|
|
||||||
source_config=source_config,
|
|
||||||
cookies_path=ctx["cookies_path"],
|
|
||||||
auth_token=ctx["auth_token"],
|
|
||||||
skip_value=skip_value,
|
|
||||||
)
|
|
||||||
return result, None
|
|
||||||
|
|
||||||
|
|
||||||
async def _run_native_ingester(
|
|
||||||
ctx: dict, source_config, mode: str | None, gdl, sync_session_factory,
|
|
||||||
) -> tuple[DownloadResult, str | None]:
|
|
||||||
"""Patreon (today the only native platform): resolve the campaign id, then run
|
|
||||||
the native ingester in a worker thread (it is sync requests/subprocess).
|
|
||||||
|
|
||||||
`resolved_campaign_id` is non-None only when we had to look it up from the
|
|
||||||
vanity URL this run, so phase 3 caches it the way the old gallery-dl retry
|
|
||||||
did. A campaign id we cannot resolve is a loud NOT_FOUND — never a silent
|
|
||||||
empty success.
|
|
||||||
"""
|
|
||||||
overrides = ctx["config_overrides"] or {}
|
|
||||||
campaign_id, resolved_campaign_id = await resolve_campaign_id_for_source(
|
|
||||||
ctx["url"], ctx["cookies_path"], overrides
|
|
||||||
)
|
|
||||||
if not campaign_id:
|
|
||||||
url = ctx["url"]
|
|
||||||
vanity = extract_vanity(url)
|
|
||||||
return (
|
|
||||||
DownloadResult(
|
|
||||||
success=False,
|
|
||||||
url=url,
|
|
||||||
artist_slug=ctx["artist_slug"],
|
|
||||||
platform="patreon",
|
|
||||||
error_type=ErrorType.NOT_FOUND,
|
|
||||||
error_message=(
|
|
||||||
f"Could not resolve Patreon campaign id. source_url={url!r}; "
|
|
||||||
f"vanity={vanity!r}; "
|
|
||||||
f"lookup=GET {_CAMPAIGNS_API}?filter[vanity]={vanity or ''} "
|
|
||||||
"(vanity lookup failed — cookies expired or creator moved?)"
|
|
||||||
),
|
|
||||||
),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Honor the operator's existing rate-limit knobs on the native path (plan
|
|
||||||
# #703): the global download_rate_limit_seconds (gallery-dl's `rate_limit`,
|
|
||||||
# here on the gdl service) paces media downloads; page fetches use the
|
|
||||||
# per-source sleep_request override, else `max(0.5, rate_limit/4)` — the same
|
|
||||||
# API-pacing default gallery-dl applied as its `sleep-request`.
|
|
||||||
rate_limit = gdl._rate_limit
|
|
||||||
request_sleep = (
|
|
||||||
source_config.sleep_request
|
|
||||||
if source_config.sleep_request is not None
|
|
||||||
else max(0.5, rate_limit / 4)
|
|
||||||
)
|
|
||||||
ingester = PatreonIngester(
|
|
||||||
images_root=gdl.images_root,
|
|
||||||
cookies_path=ctx["cookies_path"],
|
|
||||||
session_factory=sync_session_factory,
|
|
||||||
validate=gdl._validate_files,
|
|
||||||
rate_limit=rate_limit,
|
|
||||||
request_sleep=request_sleep,
|
|
||||||
)
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
dl_result = await loop.run_in_executor(
|
|
||||||
None,
|
|
||||||
lambda: ingester.run(
|
|
||||||
source_id=ctx["source_id"],
|
|
||||||
campaign_id=campaign_id,
|
|
||||||
artist_slug=ctx["artist_slug"],
|
|
||||||
url=ctx["url"],
|
|
||||||
mode=mode,
|
|
||||||
resume_cursor=source_config.resume_cursor,
|
|
||||||
time_budget_seconds=source_config.timeout,
|
|
||||||
posts_base=int(overrides.get("_backfill_posts", 0)),
|
|
||||||
# plan #709: live progress writes to this running event mid-walk.
|
|
||||||
event_id=ctx.get("event_id"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return dl_result, resolved_campaign_id
|
|
||||||
|
|
||||||
|
|
||||||
async def preview_source(
|
|
||||||
*,
|
|
||||||
platform: str,
|
|
||||||
url: str,
|
|
||||||
source_id: int,
|
|
||||||
config_overrides: dict | None,
|
|
||||||
cookies_path: str | None,
|
|
||||||
images_root: Path,
|
|
||||||
sync_session_factory,
|
|
||||||
page_limit: int = 3,
|
|
||||||
) -> dict:
|
|
||||||
"""Dry-run preview for a native platform (plan #708 B4): resolve the campaign
|
|
||||||
id, then walk a few pages counting media not already seen/dead — no download.
|
|
||||||
|
|
||||||
Returns the preview dict (total_new / posts_scanned / pages_scanned /
|
|
||||||
has_more / sample), or `{"error": msg}` on a resolve / auth / drift failure.
|
|
||||||
Native-only — the caller gates on `uses_native_ingester`.
|
|
||||||
"""
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
from .patreon_client import PatreonAPIError
|
|
||||||
|
|
||||||
campaign_id, _ = await resolve_campaign_id_for_source(
|
|
||||||
url, cookies_path, config_overrides or {}
|
|
||||||
)
|
|
||||||
if not campaign_id:
|
|
||||||
vanity = extract_vanity(url)
|
|
||||||
return {
|
|
||||||
"error": (
|
|
||||||
f"Couldn't resolve the campaign id. source_url={url!r}; "
|
|
||||||
f"vanity={vanity!r}; lookup=GET {_CAMPAIGNS_API}?filter[vanity]={vanity or ''} "
|
|
||||||
"(cookies expired, or the creator moved/renamed?)."
|
|
||||||
)
|
|
||||||
}
|
|
||||||
ingester = PatreonIngester(
|
|
||||||
images_root=images_root,
|
|
||||||
cookies_path=cookies_path,
|
|
||||||
session_factory=sync_session_factory,
|
|
||||||
)
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
try:
|
|
||||||
result = await loop.run_in_executor(
|
|
||||||
None,
|
|
||||||
lambda: ingester.preview(source_id, campaign_id, page_limit=page_limit),
|
|
||||||
)
|
|
||||||
except PatreonAPIError as exc:
|
|
||||||
return {"error": f"Couldn't preview: {exc}"}
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
async def verify_source_credential(
|
|
||||||
*,
|
|
||||||
platform: str,
|
|
||||||
url: str,
|
|
||||||
artist_slug: str,
|
|
||||||
config_overrides: dict | None,
|
|
||||||
cookies_path: str | None,
|
|
||||||
auth_token: str | None,
|
|
||||||
images_root: Path,
|
|
||||||
) -> tuple[bool | None, str]:
|
|
||||||
"""Uniform credential probe across backends. Returns `(ok, message)`:
|
|
||||||
True = authenticated, False = rejected, None = inconclusive (drift /
|
|
||||||
network / nothing to test). Callers don't branch on platform — they call
|
|
||||||
this and render the result.
|
|
||||||
"""
|
|
||||||
if uses_native_ingester(platform):
|
|
||||||
# Native ingester platforms verify via their own lightweight auth probe
|
|
||||||
# (resolve campaign id + one authenticated API page). Patreon today.
|
|
||||||
from .patreon_ingester import verify_patreon_credential
|
|
||||||
|
|
||||||
return await verify_patreon_credential(url, cookies_path, config_overrides)
|
|
||||||
|
|
||||||
# gallery-dl platforms: --simulate one item; the extractor errors before it
|
|
||||||
# can list if auth is bad.
|
|
||||||
from .gallery_dl import GalleryDLService, SourceConfig
|
|
||||||
|
|
||||||
gdl = GalleryDLService(images_root=images_root)
|
|
||||||
return await gdl.verify(
|
|
||||||
url=url,
|
|
||||||
artist_slug=artist_slug,
|
|
||||||
platform=platform,
|
|
||||||
source_config=SourceConfig.from_dict(config_overrides or {}),
|
|
||||||
cookies_path=cookies_path,
|
|
||||||
auth_token=auth_token,
|
|
||||||
)
|
|
||||||
@@ -13,6 +13,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
|
import re
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -24,25 +25,44 @@ from sqlalchemy.orm import joinedload
|
|||||||
|
|
||||||
from ..models import Artist, DownloadEvent, Source
|
from ..models import Artist, DownloadEvent, Source
|
||||||
from .credential_service import CredentialService
|
from .credential_service import CredentialService
|
||||||
from .download_backends import run_download, uses_native_ingester
|
|
||||||
from .gallery_dl import (
|
from .gallery_dl import (
|
||||||
BACKFILL_CHUNK_SECONDS,
|
|
||||||
BACKFILL_SKIP_VALUE,
|
BACKFILL_SKIP_VALUE,
|
||||||
|
BACKFILL_TIMEOUT_SECONDS,
|
||||||
TICK_SKIP_VALUE,
|
TICK_SKIP_VALUE,
|
||||||
DownloadResult,
|
|
||||||
ErrorType,
|
ErrorType,
|
||||||
GalleryDLService,
|
GalleryDLService,
|
||||||
SourceConfig,
|
SourceConfig,
|
||||||
extract_errors_warnings,
|
|
||||||
truncate_log,
|
|
||||||
)
|
)
|
||||||
from .importer import Importer
|
from .importer import Importer
|
||||||
|
from .patreon_resolver import resolve_campaign_id
|
||||||
from .platforms import auth_type_for
|
from .platforms import auth_type_for
|
||||||
from .scheduler_service import set_platform_cooldown
|
from .scheduler_service import set_platform_cooldown
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
_PATREON_VANITY_RE = re.compile(
|
||||||
|
r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
_CAMPAIGN_ID_FAILURE_PATTERN = "failed to extract campaign id"
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_patreon_vanity(url: str) -> str | None:
|
||||||
|
m = _PATREON_VANITY_RE.match(url)
|
||||||
|
return m.group(1) if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def _looks_like_campaign_id_failure(stdout: str, stderr: str) -> bool:
|
||||||
|
return _CAMPAIGN_ID_FAILURE_PATTERN in f"{stdout}\n{stderr}".lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _effective_url(platform: str, source_url: str, overrides: dict) -> str:
|
||||||
|
if platform == "patreon" and overrides.get("patreon_campaign_id"):
|
||||||
|
return f"https://www.patreon.com/id:{overrides['patreon_campaign_id']}"
|
||||||
|
return source_url
|
||||||
|
|
||||||
|
|
||||||
class DownloadService:
|
class DownloadService:
|
||||||
"""Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
|
"""Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
|
||||||
|
|
||||||
@@ -57,18 +77,12 @@ class DownloadService:
|
|||||||
gdl: GalleryDLService,
|
gdl: GalleryDLService,
|
||||||
importer: Importer,
|
importer: Importer,
|
||||||
cred_service: CredentialService,
|
cred_service: CredentialService,
|
||||||
sync_session_factory=None,
|
|
||||||
):
|
):
|
||||||
self.async_session = async_session
|
self.async_session = async_session
|
||||||
self.sync_session = sync_session
|
self.sync_session = sync_session
|
||||||
self.gdl = gdl
|
self.gdl = gdl
|
||||||
self.importer = importer
|
self.importer = importer
|
||||||
self.cred_service = cred_service
|
self.cred_service = cred_service
|
||||||
# Sync sessionmaker the native Patreon ingester opens SHORT-LIVED
|
|
||||||
# sessions from for its seen-ledger reads/writes (never one held across
|
|
||||||
# the multi-minute walk — see PatreonIngester). Only the patreon branch
|
|
||||||
# of phase 2 uses it; gallery-dl sources leave it None.
|
|
||||||
self.sync_session_factory = sync_session_factory
|
|
||||||
|
|
||||||
async def download_source(self, source_id: int) -> int:
|
async def download_source(self, source_id: int) -> int:
|
||||||
"""Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
|
"""Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
|
||||||
@@ -77,101 +91,64 @@ class DownloadService:
|
|||||||
return setup["event_id"]
|
return setup["event_id"]
|
||||||
ctx = setup
|
ctx = setup
|
||||||
|
|
||||||
# Release the phase-1 DB connections before the (up to ~19.5-min in
|
|
||||||
# backfill) gallery-dl subprocess. Held checked-out across that idle
|
|
||||||
# window, the asyncpg/psycopg connections get reaped by the server,
|
|
||||||
# and phase 3's first query then hits a dead socket
|
|
||||||
# (asyncpg ConnectionDoesNotExistError) → download_source autoretry →
|
|
||||||
# _phase1_setup's in-flight guard no-ops the retry → the event
|
|
||||||
# strands empty for the recovery sweep (Anduo #40014, 2026-06-04).
|
|
||||||
# pool_pre_ping can't help a *held* connection — it only validates on
|
|
||||||
# pool checkout. Closing returns them to the pool so phase 3 re-
|
|
||||||
# acquires a live one (the async task engine uses NullPool, the sync
|
|
||||||
# engine pre_ping + pool_recycle=300). This is what makes the
|
|
||||||
# "Phase 2 — no DB connection" contract in the class docstring true.
|
|
||||||
await self.async_session.close()
|
|
||||||
self.sync_session.close()
|
|
||||||
|
|
||||||
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
|
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
|
||||||
# Backfill mode (plan #693): the source's `_backfill_state == "running"`
|
# alembic 0031 / plan #544: derive skip_value + timeout from the
|
||||||
# selects a time-boxed deep-walk chunk — skip: True (walk full history)
|
# source's backfill_runs_remaining counter. When > 0, walk the full
|
||||||
# + the BACKFILL_CHUNK_SECONDS budget, resuming from the cursor
|
# post history (skip: True + 1800s); when 0, exit gallery-dl after
|
||||||
# checkpoint (plan #689). State is the single source of truth; it stays
|
# 20 contiguous archived items (skip: "exit:20" + the default
|
||||||
# "running" across ticks until the walk reaches the bottom (phase 3
|
# 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
|
||||||
# flips it to "complete"). Otherwise tick mode: exit gallery-dl after
|
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
|
||||||
# 20 contiguous archived items (skip: "exit:20" + the default 870s).
|
if backfill_remaining > 0:
|
||||||
# Operator drives this via POST /api/sources/{id}/backfill {action}.
|
|
||||||
overrides = ctx["config_overrides"] or {}
|
|
||||||
in_backfill = overrides.get("_backfill_state") == "running"
|
|
||||||
# Recovery (plan #697) reuses the entire #693 backfill state machine —
|
|
||||||
# cursor checkpoint, time-boxed chunks, complete/stall lifecycle — and
|
|
||||||
# differs only in bypassing the tier-1 seen-ledger so dropped-and-deleted
|
|
||||||
# near-dups get re-fetched and re-evaluated under the CURRENT pHash
|
|
||||||
# threshold (tier-2 disk still spares files we kept). The
|
|
||||||
# `_backfill_bypass_seen` flag rides alongside the running backfill state;
|
|
||||||
# download mode is "recovery" when both are set.
|
|
||||||
bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
|
|
||||||
if in_backfill:
|
|
||||||
skip_value: bool | str = BACKFILL_SKIP_VALUE
|
skip_value: bool | str = BACKFILL_SKIP_VALUE
|
||||||
source_config.timeout = BACKFILL_CHUNK_SECONDS
|
source_config.timeout = BACKFILL_TIMEOUT_SECONDS
|
||||||
pending_cursor = overrides.get("_backfill_cursor")
|
|
||||||
if uses_native_ingester(ctx["platform"]) and pending_cursor:
|
|
||||||
source_config.resume_cursor = pending_cursor
|
|
||||||
else:
|
else:
|
||||||
skip_value = TICK_SKIP_VALUE
|
skip_value = TICK_SKIP_VALUE
|
||||||
|
|
||||||
# Phase 2 dispatch is uniform across backends (download_backends.
|
effective_url = _effective_url(
|
||||||
# run_download — the download counterpart to verify_source_credential):
|
ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
|
||||||
# native platforms run their ingester in `mode` (zero per-file HEADs,
|
|
||||||
# native cursor/resume, loud drift detection); gallery-dl platforms run
|
|
||||||
# the subprocess. Either returns a DownloadResult-shaped object so phase 3
|
|
||||||
# is untouched. `mode` is None for gallery-dl (run_download ignores it).
|
|
||||||
mode: str | None = None
|
|
||||||
if uses_native_ingester(ctx["platform"]):
|
|
||||||
if in_backfill and bypass_seen:
|
|
||||||
mode = "recovery"
|
|
||||||
elif in_backfill:
|
|
||||||
mode = "backfill"
|
|
||||||
else:
|
|
||||||
mode = "tick"
|
|
||||||
dl_result, resolved_campaign_id = await self._run_download(
|
|
||||||
ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# A backfill chunk that hit its time-box but made forward progress is
|
dl_result = await self.gdl.download(
|
||||||
# NORMAL, not a failure — reclassify TIMEOUT → PARTIAL so it reads as
|
url=effective_url,
|
||||||
# "ok/progress" (PARTIAL maps to status "ok"), not a red error, and the
|
artist_slug=ctx["artist_slug"],
|
||||||
# next chunk just resumes from the new cursor (plan #693). A chunk that
|
platform=ctx["platform"],
|
||||||
# timed out with NO progress stays TIMEOUT and feeds phase 3's
|
source_config=source_config,
|
||||||
# stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires.
|
cookies_path=ctx["cookies_path"],
|
||||||
if in_backfill and dl_result.error_type == ErrorType.TIMEOUT:
|
auth_token=ctx["auth_token"],
|
||||||
new_cursor = dl_result.cursor # plan #704: structured, not scraped
|
skip_value=skip_value,
|
||||||
advanced = bool(
|
|
||||||
(new_cursor and new_cursor != overrides.get("_backfill_cursor"))
|
|
||||||
or dl_result.files_downloaded > 0
|
|
||||||
)
|
)
|
||||||
if advanced:
|
|
||||||
dl_result.error_type = ErrorType.PARTIAL
|
resolved_campaign_id: str | None = None
|
||||||
dl_result.error_message = (
|
if (
|
||||||
f"Backfill chunk: {dl_result.files_downloaded} file(s) — continuing"
|
ctx["platform"] == "patreon"
|
||||||
|
and not dl_result.success
|
||||||
|
and "patreon_campaign_id" not in (ctx["config_overrides"] or {})
|
||||||
|
and _looks_like_campaign_id_failure(dl_result.stdout, dl_result.stderr)
|
||||||
|
):
|
||||||
|
vanity = _extract_patreon_vanity(ctx["url"])
|
||||||
|
if vanity:
|
||||||
|
log.info(
|
||||||
|
"Attempting campaign-ID resolution for %s (%s)",
|
||||||
|
ctx["artist_slug"], vanity,
|
||||||
|
)
|
||||||
|
resolved_campaign_id = await resolve_campaign_id(
|
||||||
|
vanity, ctx["cookies_path"]
|
||||||
|
)
|
||||||
|
if resolved_campaign_id:
|
||||||
|
dl_result = await self.gdl.download(
|
||||||
|
url=f"https://www.patreon.com/id:{resolved_campaign_id}",
|
||||||
|
artist_slug=ctx["artist_slug"],
|
||||||
|
platform=ctx["platform"],
|
||||||
|
source_config=source_config,
|
||||||
|
cookies_path=ctx["cookies_path"],
|
||||||
|
auth_token=ctx["auth_token"],
|
||||||
|
skip_value=skip_value,
|
||||||
)
|
)
|
||||||
|
|
||||||
return await self._phase3_persist(
|
return await self._phase3_persist(
|
||||||
ctx["event_id"], ctx, dl_result, resolved_campaign_id,
|
ctx["event_id"], ctx, dl_result, resolved_campaign_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _run_download(
|
|
||||||
self, *, ctx: dict, source_config, skip_value, mode: str | None,
|
|
||||||
) -> tuple[DownloadResult, str | None]:
|
|
||||||
"""Phase-2 dispatch → `download_backends.run_download`, passing this
|
|
||||||
service's gdl + sync sessionmaker. Kept as a thin instance method so tests
|
|
||||||
can stub the whole phase-2 dispatch on the service, and so the per-backend
|
|
||||||
construction lives in download_backends (the backend registry)."""
|
|
||||||
return await run_download(
|
|
||||||
ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
|
|
||||||
gdl=self.gdl, sync_session_factory=self.sync_session_factory,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
|
async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
|
||||||
source = (await self.async_session.execute(
|
source = (await self.async_session.execute(
|
||||||
select(Source).options(joinedload(Source.artist)).where(Source.id == source_id)
|
select(Source).options(joinedload(Source.artist)).where(Source.id == source_id)
|
||||||
@@ -269,11 +246,6 @@ class DownloadService:
|
|||||||
source_row = self.sync_session.get(Source, ctx["source_id"])
|
source_row = self.sync_session.get(Source, ctx["source_id"])
|
||||||
|
|
||||||
import_summary = {"attached": 0, "skipped": 0, "errors": 0}
|
import_summary = {"attached": 0, "skipped": 0, "errors": 0}
|
||||||
# Archives detected but captured WITHOUT extracting any image (probe
|
|
||||||
# rejected / corrupt / missing extractor backend). Surfaced on the event
|
|
||||||
# so a post showing "no images" beside a zip is diagnosable (plan
|
|
||||||
# follow-up 2026-06-06 — the recurring archive-association report).
|
|
||||||
unextracted_archives: list[dict] = []
|
|
||||||
bytes_downloaded = 0
|
bytes_downloaded = 0
|
||||||
|
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
@@ -327,17 +299,6 @@ class DownloadService:
|
|||||||
# mirror duplicate_hash cleanup so we don't keep two copies.
|
# mirror duplicate_hash cleanup so we don't keep two copies.
|
||||||
# Operator-flagged 2026-06-02 (Lustria OST zip).
|
# Operator-flagged 2026-06-02 (Lustria OST zip).
|
||||||
import_summary["attached"] += 1
|
import_summary["attached"] += 1
|
||||||
# An archive captured WITHOUT extracting any image carries a
|
|
||||||
# reason on result.error — record it so the event explains the
|
|
||||||
# "no images beside a zip" symptom instead of staying silent.
|
|
||||||
if result.error:
|
|
||||||
unextracted_archives.append(
|
|
||||||
{"file": path.name, "reason": result.error}
|
|
||||||
)
|
|
||||||
log.warning(
|
|
||||||
"archive captured unextracted (%s): %s",
|
|
||||||
path.name, result.error,
|
|
||||||
)
|
|
||||||
try:
|
try:
|
||||||
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
|
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
|
||||||
except OSError:
|
except OSError:
|
||||||
@@ -384,16 +345,11 @@ class DownloadService:
|
|||||||
select(DownloadEvent).where(DownloadEvent.id == event_id)
|
select(DownloadEvent).where(DownloadEvent.id == event_id)
|
||||||
)).scalar_one()
|
)).scalar_one()
|
||||||
|
|
||||||
# plan #704: the native ingester returns structured run_stats; only the
|
|
||||||
# gallery-dl path needs the regex-over-stdout reconstruction.
|
|
||||||
if dl_result.run_stats is not None:
|
|
||||||
run_stats = dict(dl_result.run_stats)
|
|
||||||
else:
|
|
||||||
run_stats = self.gdl._compute_run_stats(
|
run_stats = self.gdl._compute_run_stats(
|
||||||
dl_result.return_code, dl_result.stdout, dl_result.stderr
|
dl_result.return_code, dl_result.stdout, dl_result.stderr
|
||||||
)
|
)
|
||||||
run_stats["quarantined_count"] = dl_result.files_quarantined
|
run_stats["quarantined_count"] = dl_result.files_quarantined
|
||||||
stderr_summary = extract_errors_warnings(dl_result.stderr)
|
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
|
||||||
|
|
||||||
# Plan #544: PARTIAL means the run downloaded ≥1 file but the
|
# Plan #544: PARTIAL means the run downloaded ≥1 file but the
|
||||||
# subprocess didn't finish in budget (typically wall-clock timeout
|
# subprocess didn't finish in budget (typically wall-clock timeout
|
||||||
@@ -413,116 +369,50 @@ class DownloadService:
|
|||||||
ev.metadata_ = {
|
ev.metadata_ = {
|
||||||
"run_stats": run_stats,
|
"run_stats": run_stats,
|
||||||
"error_type": dl_result.error_type.value if dl_result.error_type else None,
|
"error_type": dl_result.error_type.value if dl_result.error_type else None,
|
||||||
"stdout": truncate_log(dl_result.stdout) or None,
|
"stdout": self.gdl._truncate_log(dl_result.stdout) or None,
|
||||||
"stderr": truncate_log(dl_result.stderr) or None,
|
"stderr": self.gdl._truncate_log(dl_result.stderr) or None,
|
||||||
"stderr_errors_warnings": stderr_summary or None,
|
"stderr_errors_warnings": stderr_summary or None,
|
||||||
"duration_seconds": dl_result.duration_seconds,
|
"duration_seconds": dl_result.duration_seconds,
|
||||||
"quarantined_paths": dl_result.quarantined_paths or None,
|
"quarantined_paths": dl_result.quarantined_paths or None,
|
||||||
"import_summary": import_summary,
|
"import_summary": import_summary,
|
||||||
# Archives detected but captured without extracting an image — the
|
|
||||||
# recurring "post shows a zip but no images" report. Each entry is
|
|
||||||
# {file, reason}; None when every archive extracted cleanly.
|
|
||||||
"unextracted_archives": unextracted_archives or None,
|
|
||||||
}
|
}
|
||||||
await self._update_source_health(
|
await self._update_source_health(
|
||||||
source_id=ctx["source_id"], status=status, error_message=ev.error,
|
source_id=ctx["source_id"], status=status, error_message=ev.error,
|
||||||
error_type=dl_result.error_type.value if dl_result.error_type else None,
|
error_type=dl_result.error_type.value if dl_result.error_type else None,
|
||||||
retry_after_seconds=getattr(dl_result, "retry_after_seconds", None),
|
|
||||||
)
|
)
|
||||||
await self._apply_backfill_lifecycle(ctx, dl_result)
|
# Plan #544: backfill lifecycle — auto-complete when a clean
|
||||||
await self.async_session.commit()
|
# backfill run drained the queue (gallery-dl exited 0 + zero files
|
||||||
return event_id
|
# downloaded means there was nothing to fetch); otherwise decrement
|
||||||
|
# the counter. Next tick falls back to tick mode once it hits 0.
|
||||||
async def _apply_backfill_lifecycle(self, ctx: dict, dl_result) -> None:
|
#
|
||||||
"""Backfill state machine (plan #693, building on the cursor of #689).
|
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
|
||||||
|
# subprocess with return_code=0 and files_downloaded=0 (every
|
||||||
A backfill runs in time-boxed chunks while
|
# file was quarantined), which used to match the auto-complete
|
||||||
`config_overrides["_backfill_state"] == "running"`. Each chunk:
|
# predicate exactly — zeroing the operator's armed budget on
|
||||||
- COMPLETES the walk → clean rc=0 (gallery-dl with skip:True exits 0
|
# the FIRST quarantine run instead of decrementing. Require
|
||||||
only after exhausting the newest→oldest walk; a chunk cut short by
|
# dl_result.success + no error_type so only genuinely-empty
|
||||||
its time-box returns success=False / rc<0 via TimeoutExpired). On
|
# successful runs drain the counter.
|
||||||
completion: state="complete", clear the cursor, return to tick mode.
|
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
|
||||||
- made PROGRESS (cursor advanced and/or files written) → stay
|
if backfill_remaining > 0:
|
||||||
"running", checkpoint the new cursor, bump the chunk counter, and
|
|
||||||
spend one of the safety-cap chunks (backfill_runs_remaining). If the
|
|
||||||
cap is exhausted without finishing → state="stalled".
|
|
||||||
- made NO progress → increment the stall counter; two strikes →
|
|
||||||
state="stalled", clear the cursor (a wedged walk can't loop).
|
|
||||||
|
|
||||||
State is the single source of truth; the cursor lives only inside a
|
|
||||||
running backfill. config_overrides is reassigned (not mutated in place)
|
|
||||||
for SQLAlchemy JSON change detection.
|
|
||||||
"""
|
|
||||||
overrides = ctx["config_overrides"] or {}
|
|
||||||
if overrides.get("_backfill_state") != "running":
|
|
||||||
return # not backfilling — tick mode, nothing to do
|
|
||||||
|
|
||||||
old_cursor = overrides.get("_backfill_cursor")
|
|
||||||
cap_remaining = ctx.get("backfill_runs_remaining", 0) or 0
|
|
||||||
|
|
||||||
src = (await self.async_session.execute(
|
src = (await self.async_session.execute(
|
||||||
select(Source).where(Source.id == ctx["source_id"])
|
select(Source).where(Source.id == ctx["source_id"])
|
||||||
)).scalar_one()
|
)).scalar_one()
|
||||||
new_overrides = dict(src.config_overrides or {})
|
queue_drained = (
|
||||||
chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
|
|
||||||
new_overrides["_backfill_chunks"] = chunks
|
|
||||||
# plan #704 (#5): _backfill_posts (the live progress badge) is OWNED by the
|
|
||||||
# ingester now — it writes a monotonic absolute mid-walk at each page
|
|
||||||
# boundary (ingest_core._checkpoint_posts), so the badge climbs DURING a
|
|
||||||
# chunk instead of jumping once per chunk here, and the re-walked resume
|
|
||||||
# page is no longer double-counted. new_overrides (read fresh above)
|
|
||||||
# carries the ingester's committed value forward untouched.
|
|
||||||
|
|
||||||
completed = (
|
|
||||||
dl_result.success
|
dl_result.success
|
||||||
and dl_result.error_type is None
|
and dl_result.error_type is None
|
||||||
and dl_result.return_code == 0
|
and dl_result.return_code == 0
|
||||||
|
and dl_result.files_downloaded == 0
|
||||||
)
|
)
|
||||||
if completed:
|
if queue_drained:
|
||||||
new_overrides["_backfill_state"] = "complete"
|
|
||||||
new_overrides.pop("_backfill_cursor", None)
|
|
||||||
new_overrides.pop("_backfill_cursor_stalls", None)
|
|
||||||
# plan #697: a recovery walk shares this lifecycle; clear its bypass
|
|
||||||
# flag on completion so the next routine tick honors the seen-ledger.
|
|
||||||
new_overrides.pop("_backfill_bypass_seen", None)
|
|
||||||
src.config_overrides = new_overrides
|
|
||||||
src.backfill_runs_remaining = 0
|
src.backfill_runs_remaining = 0
|
||||||
return
|
|
||||||
|
|
||||||
# Did not finish. The native ingester checkpoints + resumes via cursor
|
|
||||||
# (carried structurally on the result, plan #704); gallery-dl platforms
|
|
||||||
# have no resumable cursor (every chunk re-walks from the top), so they
|
|
||||||
# advance only by the download archive growing.
|
|
||||||
new_cursor = (
|
|
||||||
dl_result.cursor if uses_native_ingester(ctx["platform"]) else None
|
|
||||||
)
|
|
||||||
advanced = bool(
|
|
||||||
(new_cursor and new_cursor != old_cursor)
|
|
||||||
or dl_result.files_downloaded > 0
|
|
||||||
)
|
|
||||||
if advanced:
|
|
||||||
if new_cursor:
|
|
||||||
new_overrides["_backfill_cursor"] = new_cursor
|
|
||||||
new_overrides.pop("_backfill_cursor_stalls", None)
|
|
||||||
cap_remaining = max(0, cap_remaining - 1)
|
|
||||||
src.backfill_runs_remaining = cap_remaining
|
|
||||||
if cap_remaining == 0:
|
|
||||||
# Safety cap hit before reaching the bottom — pause, don't loop.
|
|
||||||
new_overrides["_backfill_state"] = "stalled"
|
|
||||||
else:
|
else:
|
||||||
stalls = int(new_overrides.get("_backfill_cursor_stalls", 0)) + 1
|
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
|
||||||
if stalls >= 2:
|
await self.async_session.commit()
|
||||||
new_overrides["_backfill_state"] = "stalled"
|
return event_id
|
||||||
new_overrides.pop("_backfill_cursor", None)
|
|
||||||
new_overrides.pop("_backfill_cursor_stalls", None)
|
|
||||||
else:
|
|
||||||
new_overrides["_backfill_cursor_stalls"] = stalls
|
|
||||||
|
|
||||||
src.config_overrides = new_overrides
|
|
||||||
|
|
||||||
async def _update_source_health(
|
async def _update_source_health(
|
||||||
self, *, source_id: int, status: str, error_message: str | None,
|
self, *, source_id: int, status: str, error_message: str | None,
|
||||||
error_type: str | None = None, retry_after_seconds: float | None = None,
|
error_type: str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""FC-3d: update Source.{consecutive_failures, last_error, last_checked_at}.
|
"""FC-3d: update Source.{consecutive_failures, last_error, last_checked_at}.
|
||||||
|
|
||||||
@@ -554,16 +444,6 @@ class DownloadService:
|
|||||||
# by error class without opening Logs per row.
|
# by error class without opening Logs per row.
|
||||||
source.error_type = error_type
|
source.error_type = error_type
|
||||||
if error_type == "rate_limited":
|
if error_type == "rate_limited":
|
||||||
# plan #708 B1: honor the server's Retry-After when the native
|
|
||||||
# client surfaced one (clamped to a sane [60, 3600] window so a
|
|
||||||
# tiny hint can't leave the platform effectively un-cooled and a
|
|
||||||
# huge one can't strand it for hours); else the flat default.
|
|
||||||
if retry_after_seconds is not None:
|
|
||||||
seconds = int(min(max(retry_after_seconds, 60), 3600))
|
|
||||||
await set_platform_cooldown(
|
|
||||||
self.async_session, source.platform, seconds=seconds,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
await set_platform_cooldown(self.async_session, source.platform)
|
await set_platform_cooldown(self.async_session, source.platform)
|
||||||
elif status == "skipped":
|
elif status == "skipped":
|
||||||
source.last_error = None
|
source.last_error = None
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||||||
|
|
||||||
from ..models import Artist, Source
|
from ..models import Artist, Source
|
||||||
from ..utils.slug import slugify
|
from ..utils.slug import slugify
|
||||||
from .source_service import BACKFILL_MAX_CHUNKS
|
from .source_service import NEW_SOURCE_BACKFILL_RUNS
|
||||||
|
|
||||||
|
|
||||||
class UnknownPlatformError(Exception):
|
class UnknownPlatformError(Exception):
|
||||||
@@ -205,17 +205,16 @@ class ExtensionService:
|
|||||||
return existing, False
|
return existing, False
|
||||||
sp = await self.session.begin_nested()
|
sp = await self.session.begin_nested()
|
||||||
try:
|
try:
|
||||||
# New subscription sources arm run-until-done backfill (plan #693)
|
# New subscription sources arm a few backfill runs so the
|
||||||
# so the first ticks walk the full history (otherwise gallery-dl's
|
# first ticks walk the full history (otherwise gallery-dl's
|
||||||
# exit:20 short-circuits before the archive is built). Mirrors
|
# exit:20 short-circuits before the archive is built).
|
||||||
# SourceService.create — without it, Firefox quick-add on a creator
|
# Mirrors SourceService.create — without it, Firefox quick-
|
||||||
# with >20 unsynced posts would surface as "check failed" with no
|
# add on a creator with >20 unsynced posts would surface
|
||||||
# diagnosis. Audit 2026-06-02.
|
# as "check failed" with no diagnosis. Audit 2026-06-02.
|
||||||
src = Source(
|
src = Source(
|
||||||
artist_id=artist_id, platform=platform,
|
artist_id=artist_id, platform=platform,
|
||||||
url=url, enabled=True,
|
url=url, enabled=True,
|
||||||
config_overrides={"_backfill_state": "running"},
|
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
|
||||||
backfill_runs_remaining=BACKFILL_MAX_CHUNKS,
|
|
||||||
)
|
)
|
||||||
self.session.add(src)
|
self.session.add(src)
|
||||||
await self.session.flush()
|
await self.session.flush()
|
||||||
|
|||||||
@@ -11,15 +11,9 @@ expected to write.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import shutil
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import UTC, datetime
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
JPEG_HEAD = b"\xff\xd8\xff"
|
JPEG_HEAD = b"\xff\xd8\xff"
|
||||||
JPEG_TAIL = b"\xff\xd9"
|
JPEG_TAIL = b"\xff\xd9"
|
||||||
|
|
||||||
@@ -114,52 +108,3 @@ def validate_file(path: Path) -> ValidationResult:
|
|||||||
return ValidationResult(ok=True, format="webp", size=size)
|
return ValidationResult(ok=True, format="webp", size=size)
|
||||||
|
|
||||||
return ValidationResult(ok=True, format=None, size=size)
|
return ValidationResult(ok=True, format=None, size=size)
|
||||||
|
|
||||||
|
|
||||||
def quarantine_file(
|
|
||||||
images_root: Path,
|
|
||||||
path: Path,
|
|
||||||
artist_slug: str,
|
|
||||||
platform: str,
|
|
||||||
*,
|
|
||||||
url: str | None,
|
|
||||||
result: ValidationResult,
|
|
||||||
) -> Path | None:
|
|
||||||
"""Move a validation-failed file to `_quarantine/<slug>/<platform>` and write
|
|
||||||
a `.quarantine.json` provenance sidecar next to it.
|
|
||||||
|
|
||||||
Returns the destination path, or None if the move itself failed (file left
|
|
||||||
in place — the caller decides what to report). The caller has already run
|
|
||||||
`validate_file` and seen `result.ok is False`. ONE implementation for both
|
|
||||||
download backends — gallery-dl's batch post-process and the native ingester's
|
|
||||||
per-media path — so the quarantine layout + provenance sidecar can't drift.
|
|
||||||
"""
|
|
||||||
quarantine_root = images_root / "_quarantine" / artist_slug / platform
|
|
||||||
try:
|
|
||||||
quarantine_root.mkdir(parents=True, exist_ok=True)
|
|
||||||
dest = quarantine_root / path.name
|
|
||||||
counter = 1
|
|
||||||
while dest.exists():
|
|
||||||
dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
|
|
||||||
counter += 1
|
|
||||||
shutil.move(str(path), str(dest))
|
|
||||||
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
|
|
||||||
sidecar.write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"original_path": str(path),
|
|
||||||
"source_url": url,
|
|
||||||
"artist_slug": artist_slug,
|
|
||||||
"platform": platform,
|
|
||||||
"format": result.format,
|
|
||||||
"reason": result.reason,
|
|
||||||
"size": result.size,
|
|
||||||
"quarantined_at": datetime.now(UTC).isoformat(),
|
|
||||||
},
|
|
||||||
indent=2,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
except OSError as exc:
|
|
||||||
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
|
|
||||||
return None
|
|
||||||
return dest
|
|
||||||
|
|||||||
+123
-164
@@ -22,7 +22,7 @@ from datetime import UTC, datetime
|
|||||||
from enum import StrEnum
|
from enum import StrEnum
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from .file_validator import is_validatable, quarantine_file, validate_file
|
from .file_validator import is_validatable, validate_file
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -39,12 +39,6 @@ class ErrorType(StrEnum):
|
|||||||
HTTP_ERROR = "http_error"
|
HTTP_ERROR = "http_error"
|
||||||
UNSUPPORTED_URL = "unsupported_url"
|
UNSUPPORTED_URL = "unsupported_url"
|
||||||
VALIDATION_FAILED = "validation_failed"
|
VALIDATION_FAILED = "validation_failed"
|
||||||
# Native Patreon ingester only (plan #697): a response parsed as JSON but
|
|
||||||
# didn't match the JSON:API shape the ingester depends on (missing `data`,
|
|
||||||
# media lacking `file_name`/`url`, etc.). Distinct from AUTH_ERROR — the fix
|
|
||||||
# is updating the ingester's field-set/parser, not rotating credentials. The
|
|
||||||
# contract test guards against silently shipping this.
|
|
||||||
API_DRIFT = "api_drift"
|
|
||||||
# Run made real progress (downloaded ≥1 file) but did not finish in the
|
# Run made real progress (downloaded ≥1 file) but did not finish in the
|
||||||
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
|
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
|
||||||
# mapping classifies this as "ok" because the next tick continues.
|
# mapping classifies this as "ok" because the next tick continues.
|
||||||
@@ -61,36 +55,33 @@ class ErrorType(StrEnum):
|
|||||||
# 20 contiguous HEADs is still negligible.
|
# 20 contiguous HEADs is still negligible.
|
||||||
TICK_SKIP_VALUE = "exit:20"
|
TICK_SKIP_VALUE = "exit:20"
|
||||||
|
|
||||||
# Backfill mode (operator-triggered deep scan): walk the full history,
|
# Backfill mode (operator-triggered deep scan): walk the full history.
|
||||||
# one TIME-BOXED CHUNK per run (plan #693). config_overrides["_backfill_state"]
|
# Source.backfill_runs_remaining > 0 selects this mode; the longer
|
||||||
# == "running" selects this mode; the cursor checkpoint (plan #689) lets each
|
# timeout below absorbs creators with thousands of posts.
|
||||||
# chunk resume where the last stopped, so the walk advances across chunks
|
|
||||||
# until gallery-dl exits cleanly (= reached the bottom).
|
|
||||||
#
|
#
|
||||||
# The chunk budget is deliberately FAR below download_source's Celery
|
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
|
||||||
# soft_time_limit (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py), not
|
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
|
||||||
# "just under" it. Hitting this budget is the NORMAL chunk boundary, not a
|
# before Celery SIGKILLs the worker — same rationale as the tick
|
||||||
# failure: subprocess.run raises TimeoutExpired (which captures partial
|
# default at line 74. The audit (2026-06-02) caught this at 1800,
|
||||||
# stdout/stderr + the last emitted cursor), and download_service reclassifies
|
# guaranteeing SIGKILL on any backfill that ran to its subprocess
|
||||||
# a chunk that made progress as PARTIAL (status "ok"), not an error. The huge
|
# budget: stdout/stderr lost, backfill_runs_remaining never
|
||||||
# headroom means a stuck file or a slow chunk can never let Celery's
|
# decrements, recovery sweep stamps generic "stranded" 30 min later.
|
||||||
# SoftTimeLimitExceeded preempt TimeoutExpired (the failure mode behind Knuxy
|
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
|
||||||
# #38275 / Anduo #39912/#40411). Earlier we ran one ~1170s run-to-the-wall
|
# was added to prevent. backfill_runs_remaining=3 still gives ~58
|
||||||
# per arming; that died as a timeout error every time on large catalogs.
|
# minutes of cumulative walk across three runs for prolific creators.
|
||||||
BACKFILL_SKIP_VALUE = True
|
BACKFILL_SKIP_VALUE = True
|
||||||
BACKFILL_CHUNK_SECONDS = 600
|
BACKFILL_TIMEOUT_SECONDS = 1170
|
||||||
|
|
||||||
|
|
||||||
# Sits well below download_source's Celery soft_time_limit
|
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
|
||||||
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py). subprocess.run MUST
|
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
|
||||||
# raise TimeoutExpired before Celery raises SoftTimeLimitExceeded —
|
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
|
||||||
# otherwise Celery wins the race, SIGKILLs the worker, in-memory
|
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
|
||||||
# stdout/stderr is lost, and the DownloadEvent ends up empty-logged with
|
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
|
||||||
# "stranded by recovery sweep" (operator-flagged 2026-05-31, Knuxy event
|
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
|
||||||
# #38275; recurred in backfill mode as Anduo #39912). Per-source bumps
|
# The 30s buffer absorbs scheduler jitter / GC pauses without making
|
||||||
# still live in source.config_overrides for legitimately long syncs —
|
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
|
||||||
# keep any override below the soft limit, or the soft-limit salvage path
|
# still live in source.config_overrides for legitimately long syncs.
|
||||||
# in tasks/download.py (_finalize_soft_limited) is the only safety net.
|
|
||||||
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
|
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
|
||||||
|
|
||||||
|
|
||||||
@@ -103,17 +94,6 @@ class SourceConfig:
|
|||||||
Source.backfill_runs_remaining column at the download_service layer
|
Source.backfill_runs_remaining column at the download_service layer
|
||||||
and is passed to _build_config_for_source as `skip_value`. Same for
|
and is passed to _build_config_for_source as `skip_value`. Same for
|
||||||
the per-run subprocess timeout.
|
the per-run subprocess timeout.
|
||||||
|
|
||||||
`resume_cursor` is RUNTIME state, not persisted operator config: the
|
|
||||||
cursor-paged backfill checkpoint (see parse_last_cursor + the
|
|
||||||
download_service backfill lifecycle). It is consumed only by the native
|
|
||||||
Patreon ingester (the sole cursor-paged platform; gallery-dl's Patreon path
|
|
||||||
was removed at the #697 cutover): on a backfill/recovery run the ingester
|
|
||||||
resumes its newest→oldest walk from that pagination cursor instead of
|
|
||||||
restarting from the top. download_service threads it from
|
|
||||||
config_overrides["_backfill_cursor"]; SourceConfig deliberately does NOT read
|
|
||||||
it in from_dict (config_overrides also holds operator config, and
|
|
||||||
resume_cursor must only apply in backfill/recovery mode).
|
|
||||||
"""
|
"""
|
||||||
content_types: list[str] = field(default_factory=lambda: ["all"])
|
content_types: list[str] = field(default_factory=lambda: ["all"])
|
||||||
sleep: float | None = None
|
sleep: float | None = None
|
||||||
@@ -122,7 +102,6 @@ class SourceConfig:
|
|||||||
filename_pattern: str | None = None
|
filename_pattern: str | None = None
|
||||||
save_metadata: bool = True
|
save_metadata: bool = True
|
||||||
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
|
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
|
||||||
resume_cursor: str | None = None
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, data: dict) -> SourceConfig:
|
def from_dict(cls, data: dict) -> SourceConfig:
|
||||||
@@ -155,51 +134,6 @@ class DownloadResult:
|
|||||||
duration_seconds: float = 0.0
|
duration_seconds: float = 0.0
|
||||||
started_at: str | None = None
|
started_at: str | None = None
|
||||||
completed_at: str | None = None
|
completed_at: str | None = None
|
||||||
# Plan #704 — structured fields the NATIVE ingester populates directly (None
|
|
||||||
# on the gallery-dl path, which keeps the regex-over-stdout route). When set,
|
|
||||||
# phase 3 reads these instead of scraping the text it would otherwise have to
|
|
||||||
# reconstruct: `run_stats` mirrors _compute_run_stats' shape; `cursor` is the
|
|
||||||
# backfill checkpoint the ingester knows exactly (no parse_last_cursor).
|
|
||||||
run_stats: dict | None = None
|
|
||||||
cursor: str | None = None
|
|
||||||
posts_processed: int = 0
|
|
||||||
# Plan #708 B1 — the server's Retry-After seconds on a RATE_LIMITED result, so
|
|
||||||
# the platform cooldown matches the hint instead of a flat default. None when
|
|
||||||
# unknown (no header, or not a rate-limit failure).
|
|
||||||
retry_after_seconds: float | None = None
|
|
||||||
|
|
||||||
|
|
||||||
def extract_errors_warnings(stderr: str) -> str:
|
|
||||||
"""Keep only the `[error]`/`[warning]` lines from a captured stderr blob.
|
|
||||||
|
|
||||||
A generic download-log helper (module-level so the native-ingester result
|
|
||||||
path can shape its logs without reaching through a GalleryDLService instance).
|
|
||||||
"""
|
|
||||||
if not stderr:
|
|
||||||
return ""
|
|
||||||
kept = [
|
|
||||||
line for line in stderr.splitlines()
|
|
||||||
if "][error]" in line.lower() or "][warning]" in line.lower()
|
|
||||||
]
|
|
||||||
return "\n".join(kept)
|
|
||||||
|
|
||||||
|
|
||||||
def truncate_log(text: str, max_bytes: int = 500_000) -> str:
|
|
||||||
"""Cap a captured log to `max_bytes`, eliding the middle (head + tail kept)."""
|
|
||||||
if not text:
|
|
||||||
return text
|
|
||||||
encoded = text.encode("utf-8")
|
|
||||||
if len(encoded) <= max_bytes:
|
|
||||||
return text
|
|
||||||
half = max_bytes // 2
|
|
||||||
head = encoded[:half].decode("utf-8", errors="ignore")
|
|
||||||
tail = encoded[-half:].decode("utf-8", errors="ignore")
|
|
||||||
head_lines = head.count("\n")
|
|
||||||
tail_lines = tail.count("\n")
|
|
||||||
total_lines = text.count("\n")
|
|
||||||
elided = max(0, total_lines - head_lines - tail_lines)
|
|
||||||
marker = f"\n\n... [{elided} lines elided, {len(encoded) - max_bytes} bytes] ...\n\n"
|
|
||||||
return head + marker + tail
|
|
||||||
|
|
||||||
|
|
||||||
def _summarize_validation_failures(failures: list[dict]) -> str:
|
def _summarize_validation_failures(failures: list[dict]) -> str:
|
||||||
@@ -216,40 +150,6 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
|
|||||||
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
|
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
|
||||||
|
|
||||||
|
|
||||||
# (parse_last_cursor was removed in plan #704: the native ingester now carries
|
|
||||||
# its checkpoint cursor as a structured DownloadResult.cursor field, so there is
|
|
||||||
# no log text to scrape — and gallery-dl platforms never had a cursor.)
|
|
||||||
|
|
||||||
|
|
||||||
def make_run_stats(
|
|
||||||
*,
|
|
||||||
exit_code: int = 0,
|
|
||||||
downloaded_count: int = 0,
|
|
||||||
skipped_count: int = 0,
|
|
||||||
per_item_failures: int = 0,
|
|
||||||
warning_count: int = 0,
|
|
||||||
tier_gated_count: int = 0,
|
|
||||||
quarantined_count: int = 0,
|
|
||||||
dead_lettered_count: int = 0,
|
|
||||||
) -> dict:
|
|
||||||
"""The canonical `run_stats` dict shape, in ONE place.
|
|
||||||
|
|
||||||
Both result producers — gallery-dl's `_compute_run_stats` (log scrape) and the
|
|
||||||
native ingester's per-outcome tally (ingest_core) — build through this so the
|
|
||||||
key set can't drift between backends. Phase 3 + the Logs UI read these keys.
|
|
||||||
"""
|
|
||||||
return {
|
|
||||||
"exit_code": exit_code,
|
|
||||||
"downloaded_count": downloaded_count,
|
|
||||||
"skipped_count": skipped_count,
|
|
||||||
"per_item_failures": per_item_failures,
|
|
||||||
"warning_count": warning_count,
|
|
||||||
"tier_gated_count": tier_gated_count,
|
|
||||||
"quarantined_count": quarantined_count,
|
|
||||||
"dead_lettered_count": dead_lettered_count,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class GalleryDLService:
|
class GalleryDLService:
|
||||||
"""Service for executing gallery-dl downloads."""
|
"""Service for executing gallery-dl downloads."""
|
||||||
|
|
||||||
@@ -283,10 +183,16 @@ class GalleryDLService:
|
|||||||
"permission denied", "tier required", "pledge required",
|
"permission denied", "tier required", "pledge required",
|
||||||
]
|
]
|
||||||
|
|
||||||
# Per-platform defaults for the gallery-dl-backed platforms. Patreon was
|
# Per-platform defaults. Lifted from GS — same six platforms FC supports.
|
||||||
# removed at the plan-#697 cutover — it now uses the native ingester
|
|
||||||
# (services/patreon_ingester.py), not gallery-dl.
|
|
||||||
PLATFORM_DEFAULTS = {
|
PLATFORM_DEFAULTS = {
|
||||||
|
"patreon": {
|
||||||
|
"content_types": ["images", "image_large", "attachments", "postfile", "content"],
|
||||||
|
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
|
||||||
|
"filename": "{num:>02}_{filename}.{extension}",
|
||||||
|
"videos": True,
|
||||||
|
"embeds": True,
|
||||||
|
"cursor": True,
|
||||||
|
},
|
||||||
"subscribestar": {
|
"subscribestar": {
|
||||||
"content_types": ["all"],
|
"content_types": ["all"],
|
||||||
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
|
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
|
||||||
@@ -345,12 +251,7 @@ class GalleryDLService:
|
|||||||
"skip": True,
|
"skip": True,
|
||||||
"sleep": self._rate_limit,
|
"sleep": self._rate_limit,
|
||||||
"sleep-request": max(0.5, self._rate_limit / 4),
|
"sleep-request": max(0.5, self._rate_limit / 4),
|
||||||
# 2 (not 3) retries — a stuck CDN host shouldn't burn a whole
|
"retries": 3,
|
||||||
# backfill chunk on one file. Anduo #40838 spent ~600s on a
|
|
||||||
# single image (4 extractor HEAD retries × 30s + 4 downloader
|
|
||||||
# GET retries × 120s); halving retries + the downloader
|
|
||||||
# timeout below caps a wedged file at ~1-2 min instead.
|
|
||||||
"retries": 2,
|
|
||||||
"timeout": 30.0,
|
"timeout": 30.0,
|
||||||
"verify": True,
|
"verify": True,
|
||||||
"postprocessors": [
|
"postprocessors": [
|
||||||
@@ -365,17 +266,27 @@ class GalleryDLService:
|
|||||||
"downloader": {
|
"downloader": {
|
||||||
"part": True,
|
"part": True,
|
||||||
"part-directory": str(self._config_dir / "temp"),
|
"part-directory": str(self._config_dir / "temp"),
|
||||||
# See the extractor retries note above (Anduo #40838): 2
|
"retries": 3,
|
||||||
# retries + a 60s read-timeout (was 120) so a stalled
|
"timeout": 120.0,
|
||||||
# connection fails fast. 60s is a per-read timeout, not a
|
# Forward Patreon as Referer/Origin to yt-dlp when it
|
||||||
# transfer cap — large GIFs that keep streaming are unaffected.
|
# fetches video manifests. Operator-flagged 2026-06-01
|
||||||
"retries": 2,
|
# (DaferQ patreon): video posts hosted on Mux carry a JWT
|
||||||
"timeout": 60.0,
|
# playback restriction that checks Referer/Origin on every
|
||||||
# NOTE: the Patreon/Mux yt-dlp Referer/Origin forwarding lived
|
# request — not just the token signature. gallery-dl's
|
||||||
# here until the plan-#697 cutover. It was Patreon-specific (and
|
# HEAD probe to stream.mux.com returns 200 (the token is
|
||||||
# would have wrongly tagged the other platforms' yt-dlp fetches);
|
# valid), but yt-dlp's actual GET-with-Range to fetch the
|
||||||
# Patreon video is now handled by the native ingester's
|
# m3u8 manifest 403s because yt-dlp sends its own default
|
||||||
# downloader (patreon_downloader._VIDEO_HEADERS), so it's gone.
|
# Referer, which Mux's policy rejects. Forcing the right
|
||||||
|
# headers fixes the headers-only case; Mux IP-range
|
||||||
|
# restrictions are unfixable from here.
|
||||||
|
"ytdl": {
|
||||||
|
"raw-options": {
|
||||||
|
"http_headers": {
|
||||||
|
"Referer": "https://www.patreon.com/",
|
||||||
|
"Origin": "https://www.patreon.com",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
},
|
},
|
||||||
"output": {"progress": True},
|
"output": {"progress": True},
|
||||||
}
|
}
|
||||||
@@ -427,7 +338,14 @@ class GalleryDLService:
|
|||||||
|
|
||||||
platform_section = config["extractor"].setdefault(platform, {})
|
platform_section = config["extractor"].setdefault(platform, {})
|
||||||
|
|
||||||
if platform == "hentaifoundry":
|
if platform == "patreon":
|
||||||
|
if "all" in source_config.content_types:
|
||||||
|
platform_section["files"] = [
|
||||||
|
"images", "image_large", "attachments", "postfile", "content",
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
platform_section["files"] = source_config.content_types
|
||||||
|
elif platform == "hentaifoundry":
|
||||||
if "pictures" in source_config.content_types or "all" in source_config.content_types:
|
if "pictures" in source_config.content_types or "all" in source_config.content_types:
|
||||||
platform_section["include"] = "all"
|
platform_section["include"] = "all"
|
||||||
|
|
||||||
@@ -587,6 +505,10 @@ class GalleryDLService:
|
|||||||
if not written_paths:
|
if not written_paths:
|
||||||
return quarantined_relpaths, failures
|
return quarantined_relpaths, failures
|
||||||
|
|
||||||
|
quarantine_root = (
|
||||||
|
self.images_root / "_quarantine" / artist_slug / platform
|
||||||
|
)
|
||||||
|
|
||||||
for path in written_paths:
|
for path in written_paths:
|
||||||
if not is_validatable(path):
|
if not is_validatable(path):
|
||||||
continue
|
continue
|
||||||
@@ -597,14 +519,34 @@ class GalleryDLService:
|
|||||||
continue
|
continue
|
||||||
if result.ok:
|
if result.ok:
|
||||||
continue
|
continue
|
||||||
# Shared move+sidecar (file_validator.quarantine_file) — same impl the
|
try:
|
||||||
# native ingester uses, so the layout + provenance sidecar can't drift.
|
quarantine_root.mkdir(parents=True, exist_ok=True)
|
||||||
dest = quarantine_file(
|
dest = quarantine_root / path.name
|
||||||
self.images_root, path, artist_slug, platform,
|
counter = 1
|
||||||
url=url, result=result,
|
while dest.exists():
|
||||||
|
dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
|
||||||
|
counter += 1
|
||||||
|
path.rename(dest)
|
||||||
|
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
|
||||||
|
sidecar.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"original_path": str(path),
|
||||||
|
"source_url": url,
|
||||||
|
"artist_slug": artist_slug,
|
||||||
|
"platform": platform,
|
||||||
|
"format": result.format,
|
||||||
|
"reason": result.reason,
|
||||||
|
"size": result.size,
|
||||||
|
"quarantined_at": datetime.now(UTC).isoformat(),
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
)
|
)
|
||||||
if dest is None:
|
)
|
||||||
continue # move failed → left in place, not counted
|
except OSError as exc:
|
||||||
|
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
|
||||||
|
continue
|
||||||
|
|
||||||
log.warning(
|
log.warning(
|
||||||
"Quarantined corrupt file: %s → %s (%s: %s)",
|
"Quarantined corrupt file: %s → %s (%s: %s)",
|
||||||
path, dest, result.format, result.reason,
|
path, dest, result.format, result.reason,
|
||||||
@@ -642,24 +584,41 @@ class GalleryDLService:
|
|||||||
if "][warning]" in line.lower() and "not allowed to view post" in line.lower()
|
if "][warning]" in line.lower() and "not allowed to view post" in line.lower()
|
||||||
)
|
)
|
||||||
|
|
||||||
return make_run_stats(
|
return {
|
||||||
exit_code=return_code,
|
"exit_code": return_code,
|
||||||
downloaded_count=self._count_downloaded_files(stdout),
|
"downloaded_count": self._count_downloaded_files(stdout),
|
||||||
skipped_count=skipped_stdout + skipped_stderr,
|
"skipped_count": skipped_stdout + skipped_stderr,
|
||||||
per_item_failures=per_item_failures,
|
"per_item_failures": per_item_failures,
|
||||||
warning_count=warning_count,
|
"warning_count": warning_count,
|
||||||
tier_gated_count=tier_gated_count,
|
"tier_gated_count": tier_gated_count,
|
||||||
)
|
}
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _extract_errors_warnings(stderr: str) -> str:
|
def _extract_errors_warnings(stderr: str) -> str:
|
||||||
# Thin delegator to the module-level helper (kept for existing callers).
|
if not stderr:
|
||||||
return extract_errors_warnings(stderr)
|
return ""
|
||||||
|
kept = [
|
||||||
|
line for line in stderr.splitlines()
|
||||||
|
if "][error]" in line.lower() or "][warning]" in line.lower()
|
||||||
|
]
|
||||||
|
return "\n".join(kept)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _truncate_log(text: str, max_bytes: int = 500_000) -> str:
|
def _truncate_log(text: str, max_bytes: int = 500_000) -> str:
|
||||||
# Thin delegator to the module-level helper (kept for existing callers).
|
if not text:
|
||||||
return truncate_log(text, max_bytes)
|
return text
|
||||||
|
encoded = text.encode("utf-8")
|
||||||
|
if len(encoded) <= max_bytes:
|
||||||
|
return text
|
||||||
|
half = max_bytes // 2
|
||||||
|
head = encoded[:half].decode("utf-8", errors="ignore")
|
||||||
|
tail = encoded[-half:].decode("utf-8", errors="ignore")
|
||||||
|
head_lines = head.count("\n")
|
||||||
|
tail_lines = tail.count("\n")
|
||||||
|
total_lines = text.count("\n")
|
||||||
|
elided = max(0, total_lines - head_lines - tail_lines)
|
||||||
|
marker = f"\n\n... [{elided} lines elided, {len(encoded) - max_bytes} bytes] ...\n\n"
|
||||||
|
return head + marker + tail
|
||||||
|
|
||||||
async def download(
|
async def download(
|
||||||
self,
|
self,
|
||||||
|
|||||||
@@ -18,22 +18,15 @@ import base64
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from sqlalchemy import Select, and_, distinct, exists, func, or_, select
|
from sqlalchemy import Select, and_, exists, func, or_, select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
from sqlalchemy.orm import aliased
|
from sqlalchemy.orm import aliased
|
||||||
|
|
||||||
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
|
from ..models import Artist, ImageProvenance, ImageRecord, Post, Tag
|
||||||
from ..models.tag import image_tag
|
from ..models.tag import image_tag
|
||||||
|
|
||||||
CURSOR_SEPARATOR = "|"
|
CURSOR_SEPARATOR = "|"
|
||||||
|
|
||||||
# Reserved `platform` filter value selecting images with NO platformed
|
|
||||||
# provenance (filesystem imports). Returned by facets() as a null-valued
|
|
||||||
# bucket; the frontend maps that null back to this sentinel in the URL so the
|
|
||||||
# bucket is selectable. Underscore-wrapped so it can't collide with a real
|
|
||||||
# gallery-dl platform name (patreon/pixiv/...).
|
|
||||||
UNSOURCED_PLATFORM = "__unsourced__"
|
|
||||||
|
|
||||||
|
|
||||||
def encode_cursor(effective_date: datetime, image_id: int) -> str:
|
def encode_cursor(effective_date: datetime, image_id: int) -> str:
|
||||||
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
|
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
|
||||||
@@ -50,17 +43,16 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
|
|||||||
|
|
||||||
|
|
||||||
def _effective_date_col():
|
def _effective_date_col():
|
||||||
"""The materialized gallery sort key: image_record.effective_date
|
"""SQL expression: COALESCE(post.post_date, image_record.created_at).
|
||||||
(alembic 0035) = COALESCE(primary post's post_date, created_at),
|
|
||||||
maintained at write time by the importer.
|
|
||||||
|
|
||||||
Canonical sort/group/filter key across the gallery so images attached
|
Used as the canonical sort/group/filter key across the gallery so
|
||||||
to a post surface at their original publish date, not their FC import
|
images backfilled with primary_post_id (e.g. via tag_apply phase 4)
|
||||||
date — and, now that it's a single indexed column rather than a
|
surface at their original publish date, not their FC import date.
|
||||||
COALESCE across the Post outer join, the cursor scroll is an index
|
Images without a Post (or with Post.post_date NULL) fall back to
|
||||||
range scan instead of a full re-sort per page.
|
image_record.created_at and still order coherently against
|
||||||
|
post-attached ones.
|
||||||
"""
|
"""
|
||||||
return ImageRecord.effective_date
|
return func.coalesce(Post.post_date, ImageRecord.created_at)
|
||||||
|
|
||||||
|
|
||||||
def _outer_join_primary_post(stmt: Select) -> Select:
|
def _outer_join_primary_post(stmt: Select) -> Select:
|
||||||
@@ -99,16 +91,6 @@ class TimelineBucket:
|
|||||||
count: int
|
count: int
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class GalleryFacets:
|
|
||||||
total: int # images matching the FULL active filter
|
|
||||||
platforms: list[dict] # [{"value": str|None, "count": int}], null = unsourced
|
|
||||||
untagged: int # how many the Untagged flag would isolate
|
|
||||||
no_artist: int # how many the No-artist flag would isolate
|
|
||||||
date_min: datetime | None
|
|
||||||
date_max: datetime | None
|
|
||||||
|
|
||||||
|
|
||||||
def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str:
|
def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str:
|
||||||
"""Return the URL to fetch a thumbnail.
|
"""Return the URL to fetch a thumbnail.
|
||||||
|
|
||||||
@@ -135,82 +117,10 @@ def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str
|
|||||||
return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
|
return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
|
||||||
|
|
||||||
|
|
||||||
def _require_single_filter(tag_ids, post_id, artist_id) -> None:
|
def _require_single_filter(tag_id, post_id, artist_id) -> None:
|
||||||
"""post_id is the post-detail view — it can't combine with the
|
if sum(x is not None for x in (tag_id, post_id, artist_id)) > 1:
|
||||||
composable filters. tag_ids + artist_id (+ media_type) compose freely
|
|
||||||
(AND)."""
|
|
||||||
if post_id is not None and (tag_ids or artist_id is not None):
|
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"post_id cannot be combined with tag or artist filters"
|
"tag_id, post_id, artist_id are mutually exclusive"
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_scope(
|
|
||||||
stmt, *, tag_ids, post_id, artist_id, media_type,
|
|
||||||
platform=None, untagged=False, no_artist=False,
|
|
||||||
date_from=None, date_to=None,
|
|
||||||
):
|
|
||||||
"""Apply the composable gallery filters to a statement.
|
|
||||||
|
|
||||||
All clauses are correlated EXISTS / scalar predicates on ImageRecord, so
|
|
||||||
they AND together without row-multiplication and don't require any join to
|
|
||||||
be present on `stmt` (the artist/platform paths alias Post/Source inside
|
|
||||||
their own EXISTS).
|
|
||||||
|
|
||||||
- tag_ids: image must carry ALL of them — one correlated EXISTS per tag.
|
|
||||||
- post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded
|
|
||||||
by _require_single_filter).
|
|
||||||
- media_type: 'image' | 'video' narrows by mime prefix.
|
|
||||||
- platform: EXISTS a provenance→source with that platform; the
|
|
||||||
UNSOURCED_PLATFORM sentinel inverts it (NO platformed provenance).
|
|
||||||
- untagged: NOT EXISTS any image_tag row.
|
|
||||||
- no_artist: ImageRecord.artist_id IS NULL.
|
|
||||||
- date_from / date_to: half-open [from, to) bounds on effective_date.
|
|
||||||
"""
|
|
||||||
for tid in tag_ids or []:
|
|
||||||
stmt = stmt.where(
|
|
||||||
exists().where(
|
|
||||||
image_tag.c.image_record_id == ImageRecord.id,
|
|
||||||
image_tag.c.tag_id == tid,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
prov = _provenance_clause(post_id, artist_id)
|
|
||||||
if prov is not None:
|
|
||||||
stmt = stmt.where(prov)
|
|
||||||
if media_type == "image":
|
|
||||||
stmt = stmt.where(ImageRecord.mime.like("image/%"))
|
|
||||||
elif media_type == "video":
|
|
||||||
stmt = stmt.where(ImageRecord.mime.like("video/%"))
|
|
||||||
if platform is not None:
|
|
||||||
stmt = stmt.where(_platform_clause(platform))
|
|
||||||
if untagged:
|
|
||||||
stmt = stmt.where(
|
|
||||||
~exists().where(image_tag.c.image_record_id == ImageRecord.id)
|
|
||||||
)
|
|
||||||
if no_artist:
|
|
||||||
stmt = stmt.where(ImageRecord.artist_id.is_(None))
|
|
||||||
eff = _effective_date_col()
|
|
||||||
if date_from is not None:
|
|
||||||
stmt = stmt.where(eff >= date_from)
|
|
||||||
if date_to is not None:
|
|
||||||
stmt = stmt.where(eff < date_to)
|
|
||||||
return stmt
|
|
||||||
|
|
||||||
|
|
||||||
def _platform_clause(platform):
|
|
||||||
"""Correlated EXISTS on a provenance row whose Source carries `platform`.
|
|
||||||
The UNSOURCED_PLATFORM sentinel inverts to NOT EXISTS(any sourced
|
|
||||||
provenance) — i.e. filesystem-imported content with no platform."""
|
|
||||||
src = aliased(Source)
|
|
||||||
if platform == UNSOURCED_PLATFORM:
|
|
||||||
return ~exists().where(
|
|
||||||
ImageProvenance.image_record_id == ImageRecord.id,
|
|
||||||
ImageProvenance.source_id == src.id,
|
|
||||||
)
|
|
||||||
return exists().where(
|
|
||||||
ImageProvenance.image_record_id == ImageRecord.id,
|
|
||||||
ImageProvenance.source_id == src.id,
|
|
||||||
src.platform == platform,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -245,27 +155,6 @@ def _provenance_clause(post_id, artist_id):
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _gallery_images(rows, artists: dict[int, dict]) -> list[GalleryImage]:
|
|
||||||
"""Build GalleryImage list from (record, posted_at, eff_date) rows + the
|
|
||||||
artist hydration map. Shared by scroll() and similar()."""
|
|
||||||
return [
|
|
||||||
GalleryImage(
|
|
||||||
id=record.id,
|
|
||||||
path=record.path,
|
|
||||||
sha256=record.sha256,
|
|
||||||
mime=record.mime,
|
|
||||||
width=record.width,
|
|
||||||
height=record.height,
|
|
||||||
created_at=record.created_at,
|
|
||||||
effective_date=eff_date,
|
|
||||||
posted_at=posted_at,
|
|
||||||
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
|
|
||||||
artist=artists.get(record.id),
|
|
||||||
)
|
|
||||||
for record, posted_at, eff_date in rows
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
|
||||||
"""Map image_id -> {"name","slug"} via the canonical
|
"""Map image_id -> {"name","slug"} via the canonical
|
||||||
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
|
||||||
@@ -290,50 +179,35 @@ class GalleryService:
|
|||||||
self,
|
self,
|
||||||
cursor: str | None,
|
cursor: str | None,
|
||||||
limit: int = 50,
|
limit: int = 50,
|
||||||
tag_ids: list[int] | None = None,
|
tag_id: int | None = None,
|
||||||
post_id: int | None = None,
|
post_id: int | None = None,
|
||||||
artist_id: int | None = None,
|
artist_id: int | None = None,
|
||||||
media_type: str | None = None,
|
|
||||||
sort: str = "newest",
|
|
||||||
platform: str | None = None,
|
|
||||||
untagged: bool = False,
|
|
||||||
no_artist: bool = False,
|
|
||||||
date_from: datetime | None = None,
|
|
||||||
date_to: datetime | None = None,
|
|
||||||
) -> GalleryPage:
|
) -> GalleryPage:
|
||||||
if limit < 1 or limit > 200:
|
if limit < 1 or limit > 200:
|
||||||
raise ValueError("limit must be between 1 and 200")
|
raise ValueError("limit must be between 1 and 200")
|
||||||
_require_single_filter(tag_ids, post_id, artist_id)
|
_require_single_filter(tag_id, post_id, artist_id)
|
||||||
|
|
||||||
eff = _effective_date_col()
|
eff = _effective_date_col()
|
||||||
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
||||||
stmt = _outer_join_primary_post(stmt)
|
stmt = _outer_join_primary_post(stmt)
|
||||||
stmt = _apply_scope(
|
if tag_id is not None:
|
||||||
stmt, tag_ids=tag_ids, post_id=post_id,
|
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
|
||||||
artist_id=artist_id, media_type=media_type,
|
image_tag.c.tag_id == tag_id
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to,
|
|
||||||
)
|
)
|
||||||
|
prov = _provenance_clause(post_id, artist_id)
|
||||||
|
if prov is not None:
|
||||||
|
stmt = stmt.where(prov)
|
||||||
|
|
||||||
descending = sort != "oldest"
|
|
||||||
if cursor:
|
if cursor:
|
||||||
cur_ts, cur_id = decode_cursor(cursor)
|
cur_ts, cur_id = decode_cursor(cursor)
|
||||||
# The cursor is just (last eff, last id); the request's sort
|
|
||||||
# decides which side of it the next page lies on.
|
|
||||||
if descending:
|
|
||||||
stmt = stmt.where(
|
stmt = stmt.where(
|
||||||
or_(eff < cur_ts, and_(eff == cur_ts, ImageRecord.id < cur_id))
|
or_(
|
||||||
|
eff < cur_ts,
|
||||||
|
and_(eff == cur_ts, ImageRecord.id < cur_id),
|
||||||
)
|
)
|
||||||
else:
|
|
||||||
stmt = stmt.where(
|
|
||||||
or_(eff > cur_ts, and_(eff == cur_ts, ImageRecord.id > cur_id))
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if descending:
|
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
|
||||||
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc())
|
|
||||||
else:
|
|
||||||
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
|
|
||||||
stmt = stmt.limit(limit + 1)
|
|
||||||
rows = (await self.session.execute(stmt)).all()
|
rows = (await self.session.execute(stmt)).all()
|
||||||
|
|
||||||
next_cursor = None
|
next_cursor = None
|
||||||
@@ -345,7 +219,22 @@ class GalleryService:
|
|||||||
artists = await _artists_for(
|
artists = await _artists_for(
|
||||||
self.session, [r[0].id for r in rows]
|
self.session, [r[0].id for r in rows]
|
||||||
)
|
)
|
||||||
images = _gallery_images(rows, artists)
|
images = [
|
||||||
|
GalleryImage(
|
||||||
|
id=record.id,
|
||||||
|
path=record.path,
|
||||||
|
sha256=record.sha256,
|
||||||
|
mime=record.mime,
|
||||||
|
width=record.width,
|
||||||
|
height=record.height,
|
||||||
|
created_at=record.created_at,
|
||||||
|
effective_date=eff_date,
|
||||||
|
posted_at=posted_at,
|
||||||
|
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
|
||||||
|
artist=artists.get(record.id),
|
||||||
|
)
|
||||||
|
for record, posted_at, eff_date in rows
|
||||||
|
]
|
||||||
return GalleryPage(
|
return GalleryPage(
|
||||||
images=images,
|
images=images,
|
||||||
next_cursor=next_cursor,
|
next_cursor=next_cursor,
|
||||||
@@ -354,15 +243,9 @@ class GalleryService:
|
|||||||
|
|
||||||
async def timeline(
|
async def timeline(
|
||||||
self,
|
self,
|
||||||
tag_ids: list[int] | None = None,
|
tag_id: int | None = None,
|
||||||
post_id: int | None = None,
|
post_id: int | None = None,
|
||||||
artist_id: int | None = None,
|
artist_id: int | None = None,
|
||||||
media_type: str | None = None,
|
|
||||||
platform: str | None = None,
|
|
||||||
untagged: bool = False,
|
|
||||||
no_artist: bool = False,
|
|
||||||
date_from: datetime | None = None,
|
|
||||||
date_to: datetime | None = None,
|
|
||||||
) -> list[TimelineBucket]:
|
) -> list[TimelineBucket]:
|
||||||
eff = _effective_date_col()
|
eff = _effective_date_col()
|
||||||
year_col = func.date_part("year", eff).label("yr")
|
year_col = func.date_part("year", eff).label("yr")
|
||||||
@@ -371,28 +254,25 @@ class GalleryService:
|
|||||||
year_col, month_col, func.count(ImageRecord.id).label("cnt")
|
year_col, month_col, func.count(ImageRecord.id).label("cnt")
|
||||||
)
|
)
|
||||||
stmt = _outer_join_primary_post(stmt)
|
stmt = _outer_join_primary_post(stmt)
|
||||||
_require_single_filter(tag_ids, post_id, artist_id)
|
_require_single_filter(tag_id, post_id, artist_id)
|
||||||
stmt = _apply_scope(
|
if tag_id is not None:
|
||||||
stmt, tag_ids=tag_ids, post_id=post_id,
|
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
|
||||||
artist_id=artist_id, media_type=media_type,
|
image_tag.c.tag_id == tag_id
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to,
|
|
||||||
)
|
)
|
||||||
|
prov = _provenance_clause(post_id, artist_id)
|
||||||
|
if prov is not None:
|
||||||
|
stmt = stmt.where(prov)
|
||||||
stmt = stmt.group_by(year_col, month_col).order_by(year_col.desc(), month_col.desc())
|
stmt = stmt.group_by(year_col, month_col).order_by(year_col.desc(), month_col.desc())
|
||||||
rows = (await self.session.execute(stmt)).all()
|
rows = (await self.session.execute(stmt)).all()
|
||||||
return [TimelineBucket(year=int(r.yr), month=int(r.mo), count=int(r.cnt)) for r in rows]
|
return [TimelineBucket(year=int(r.yr), month=int(r.mo), count=int(r.cnt)) for r in rows]
|
||||||
|
|
||||||
async def jump_cursor(
|
async def jump_cursor(
|
||||||
self, year: int, month: int, tag_ids: list[int] | None = None,
|
self, year: int, month: int, tag_id: int | None = None,
|
||||||
post_id: int | None = None, artist_id: int | None = None,
|
post_id: int | None = None, artist_id: int | None = None,
|
||||||
media_type: str | None = None, sort: str = "newest",
|
|
||||||
platform: str | None = None, untagged: bool = False,
|
|
||||||
no_artist: bool = False, date_from: datetime | None = None,
|
|
||||||
date_to: datetime | None = None,
|
|
||||||
) -> str | None:
|
) -> str | None:
|
||||||
"""Returns a cursor that, when passed to scroll() with the same sort,
|
"""Returns a cursor that, when passed to scroll(), positions at the
|
||||||
positions at the first image of the given year-month. None if the
|
first image of the given year-month (by effective_date, not
|
||||||
bucket is empty.
|
created_at). None if the bucket is empty.
|
||||||
"""
|
"""
|
||||||
from sqlalchemy import extract
|
from sqlalchemy import extract
|
||||||
|
|
||||||
@@ -402,157 +282,22 @@ class GalleryService:
|
|||||||
extract("month", eff) == month,
|
extract("month", eff) == month,
|
||||||
)
|
)
|
||||||
stmt = _outer_join_primary_post(stmt)
|
stmt = _outer_join_primary_post(stmt)
|
||||||
_require_single_filter(tag_ids, post_id, artist_id)
|
_require_single_filter(tag_id, post_id, artist_id)
|
||||||
stmt = _apply_scope(
|
if tag_id is not None:
|
||||||
stmt, tag_ids=tag_ids, post_id=post_id,
|
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
|
||||||
artist_id=artist_id, media_type=media_type,
|
image_tag.c.tag_id == tag_id
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to,
|
|
||||||
)
|
)
|
||||||
descending = sort != "oldest"
|
prov = _provenance_clause(post_id, artist_id)
|
||||||
if descending:
|
if prov is not None:
|
||||||
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc())
|
stmt = stmt.where(prov)
|
||||||
else:
|
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
|
||||||
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
|
first = (await self.session.execute(stmt)).first()
|
||||||
first = (await self.session.execute(stmt.limit(1))).first()
|
|
||||||
if first is None:
|
if first is None:
|
||||||
return None
|
return None
|
||||||
record, eff_date = first
|
record, eff_date = first
|
||||||
# Cursor is exclusive; nudge the id one past the boundary row (in the
|
# Cursor is exclusive; we encode a cursor with id+1 so the row itself
|
||||||
# scan direction) so the row itself is the first result of scroll().
|
# is the first result in the next scroll().
|
||||||
boundary = record.id + 1 if descending else record.id - 1
|
return encode_cursor(eff_date, record.id + 1)
|
||||||
return encode_cursor(eff_date, boundary)
|
|
||||||
|
|
||||||
async def facets(
|
|
||||||
self, *, tag_ids: list[int] | None = None,
|
|
||||||
post_id: int | None = None, artist_id: int | None = None,
|
|
||||||
media_type: str | None = None, platform: str | None = None,
|
|
||||||
untagged: bool = False, no_artist: bool = False,
|
|
||||||
date_from: datetime | None = None, date_to: datetime | None = None,
|
|
||||||
) -> GalleryFacets:
|
|
||||||
"""Live facet counts scoped to the current filter. Each facet GROUP is
|
|
||||||
computed with all OTHER active filters applied but its OWN selection
|
|
||||||
ignored ("minus-self"), so sibling options stay visible/switchable.
|
|
||||||
No outer join is needed — every clause is a correlated EXISTS or a
|
|
||||||
column predicate on ImageRecord.
|
|
||||||
"""
|
|
||||||
_require_single_filter(tag_ids, post_id, artist_id)
|
|
||||||
common = {
|
|
||||||
"tag_ids": tag_ids, "post_id": post_id,
|
|
||||||
"artist_id": artist_id, "media_type": media_type,
|
|
||||||
}
|
|
||||||
|
|
||||||
# total — the full active filter (the headline result count).
|
|
||||||
total = (await self.session.execute(
|
|
||||||
_apply_scope(
|
|
||||||
select(func.count(ImageRecord.id)), **common,
|
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to,
|
|
||||||
)
|
|
||||||
)).scalar_one()
|
|
||||||
|
|
||||||
# platforms — scope minus the platform selection. Inner-join
|
|
||||||
# provenance→source and COUNT(DISTINCT image) per platform (a
|
|
||||||
# cross-posted image counts under each of its platforms).
|
|
||||||
plat_scope = {
|
|
||||||
**common, "untagged": untagged, "no_artist": no_artist,
|
|
||||||
"date_from": date_from, "date_to": date_to,
|
|
||||||
}
|
|
||||||
src = aliased(Source)
|
|
||||||
plat_stmt = (
|
|
||||||
select(src.platform, func.count(distinct(ImageRecord.id)))
|
|
||||||
.select_from(ImageRecord)
|
|
||||||
.join(ImageProvenance, ImageProvenance.image_record_id == ImageRecord.id)
|
|
||||||
.join(src, src.id == ImageProvenance.source_id)
|
|
||||||
)
|
|
||||||
plat_stmt = _apply_scope(plat_stmt, **plat_scope).group_by(src.platform)
|
|
||||||
platforms = [
|
|
||||||
{"value": p, "count": c}
|
|
||||||
for p, c in (await self.session.execute(plat_stmt)).all()
|
|
||||||
]
|
|
||||||
# Unsourced (filesystem) bucket — same minus-platform scope.
|
|
||||||
unsourced = (await self.session.execute(
|
|
||||||
_apply_scope(
|
|
||||||
select(func.count(ImageRecord.id)), **plat_scope,
|
|
||||||
platform=UNSOURCED_PLATFORM,
|
|
||||||
)
|
|
||||||
)).scalar_one()
|
|
||||||
if unsourced:
|
|
||||||
platforms.append({"value": None, "count": unsourced})
|
|
||||||
|
|
||||||
# curation flags — each minus its OWN flag.
|
|
||||||
untagged_count = (await self.session.execute(
|
|
||||||
_apply_scope(
|
|
||||||
select(func.count(ImageRecord.id)), **common,
|
|
||||||
platform=platform, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to, untagged=True,
|
|
||||||
)
|
|
||||||
)).scalar_one()
|
|
||||||
no_artist_count = (await self.session.execute(
|
|
||||||
_apply_scope(
|
|
||||||
select(func.count(ImageRecord.id)), **common,
|
|
||||||
platform=platform, untagged=untagged,
|
|
||||||
date_from=date_from, date_to=date_to, no_artist=True,
|
|
||||||
)
|
|
||||||
)).scalar_one()
|
|
||||||
|
|
||||||
# date bounds — scope minus the date params (those drive the picker).
|
|
||||||
eff = _effective_date_col()
|
|
||||||
dmin, dmax = (await self.session.execute(
|
|
||||||
_apply_scope(
|
|
||||||
select(func.min(eff), func.max(eff)), **common,
|
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
)
|
|
||||||
)).one()
|
|
||||||
|
|
||||||
return GalleryFacets(
|
|
||||||
total=total, platforms=platforms,
|
|
||||||
untagged=untagged_count, no_artist=no_artist_count,
|
|
||||||
date_min=dmin, date_max=dmax,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def similar(
|
|
||||||
self, image_id: int, limit: int = 100, *,
|
|
||||||
tag_ids: list[int] | None = None, artist_id: int | None = None,
|
|
||||||
media_type: str | None = None, platform: str | None = None,
|
|
||||||
untagged: bool = False, no_artist: bool = False,
|
|
||||||
date_from: datetime | None = None, date_to: datetime | None = None,
|
|
||||||
) -> list[GalleryImage] | None:
|
|
||||||
"""Visual "more like this": images ranked by cosine distance to
|
|
||||||
`image_id`'s SigLIP embedding (pgvector, HNSW-indexed — alembic 0036).
|
|
||||||
No ML inference here; the embedding was computed at import.
|
|
||||||
|
|
||||||
Returns None if the source image doesn't exist (→ 404), [] if it has
|
|
||||||
no embedding (a video / not-yet-embedded). Composes with the Phase-1/2
|
|
||||||
scope filters (AND) but REPLACES the date sort — always nearest-first,
|
|
||||||
bounded to `limit` (no cursor; distance-ranking has no date cursor).
|
|
||||||
"""
|
|
||||||
if limit < 1 or limit > 200:
|
|
||||||
raise ValueError("limit must be between 1 and 200")
|
|
||||||
src = await self.session.get(ImageRecord, image_id)
|
|
||||||
if src is None:
|
|
||||||
return None
|
|
||||||
if src.siglip_embedding is None:
|
|
||||||
return []
|
|
||||||
|
|
||||||
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
|
|
||||||
eff = _effective_date_col()
|
|
||||||
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
|
|
||||||
stmt = _outer_join_primary_post(stmt)
|
|
||||||
stmt = stmt.where(
|
|
||||||
ImageRecord.siglip_embedding.is_not(None),
|
|
||||||
ImageRecord.id != image_id,
|
|
||||||
)
|
|
||||||
stmt = _apply_scope(
|
|
||||||
stmt, tag_ids=tag_ids, post_id=None,
|
|
||||||
artist_id=artist_id, media_type=media_type,
|
|
||||||
platform=platform, untagged=untagged, no_artist=no_artist,
|
|
||||||
date_from=date_from, date_to=date_to,
|
|
||||||
)
|
|
||||||
stmt = stmt.order_by(distance.asc()).limit(limit)
|
|
||||||
rows = (await self.session.execute(stmt)).all()
|
|
||||||
artists = await _artists_for(self.session, [r[0].id for r in rows])
|
|
||||||
return _gallery_images(rows, artists)
|
|
||||||
|
|
||||||
async def get_image_with_tags(self, image_id: int) -> dict | None:
|
async def get_image_with_tags(self, image_id: int) -> dict | None:
|
||||||
record = await self.session.get(ImageRecord, image_id)
|
record = await self.session.get(ImageRecord, image_id)
|
||||||
@@ -591,9 +336,6 @@ class GalleryService:
|
|||||||
"height": record.height,
|
"height": record.height,
|
||||||
"size_bytes": record.size_bytes,
|
"size_bytes": record.size_bytes,
|
||||||
"integrity_status": record.integrity_status,
|
"integrity_status": record.integrity_status,
|
||||||
# Phase 3: lets the modal hide the "Related"/find-similar surface
|
|
||||||
# for images that have no embedding yet (videos / pending ML).
|
|
||||||
"has_embedding": record.siglip_embedding is not None,
|
|
||||||
"created_at": record.created_at.isoformat(),
|
"created_at": record.created_at.isoformat(),
|
||||||
"posted_at": posted_at.isoformat() if posted_at else None,
|
"posted_at": posted_at.isoformat() if posted_at else None,
|
||||||
"thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
|
"thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
|
||||||
@@ -615,10 +357,17 @@ class GalleryService:
|
|||||||
}
|
}
|
||||||
|
|
||||||
async def _neighbors(self, record: ImageRecord) -> dict:
|
async def _neighbors(self, record: ImageRecord) -> dict:
|
||||||
# The boundary image's sort key is materialized on the row now
|
# Compute the boundary image's effective_date in Python (one query
|
||||||
# (alembic 0035) — read it directly instead of re-deriving COALESCE
|
# below + the SELECT we already have on `record`) and use it for
|
||||||
# via an extra Post lookup.
|
# the neighbor comparison. Cheaper than re-deriving in SQL via
|
||||||
boundary_eff = record.effective_date
|
# correlated subquery.
|
||||||
|
boundary_eff = record.created_at
|
||||||
|
if record.primary_post_id is not None:
|
||||||
|
post_date = (await self.session.execute(
|
||||||
|
select(Post.post_date).where(Post.id == record.primary_post_id)
|
||||||
|
)).scalar_one_or_none()
|
||||||
|
if post_date is not None:
|
||||||
|
boundary_eff = post_date
|
||||||
|
|
||||||
eff = _effective_date_col()
|
eff = _effective_date_col()
|
||||||
prev_stmt = _outer_join_primary_post(
|
prev_stmt = _outer_join_primary_post(
|
||||||
|
|||||||
@@ -31,12 +31,7 @@ from ..models import (
|
|||||||
Source,
|
Source,
|
||||||
)
|
)
|
||||||
from ..utils import safe_probe
|
from ..utils import safe_probe
|
||||||
from ..utils.paths import (
|
from ..utils.paths import derive_subdir, derive_top_level_artist, hash_suffixed_name
|
||||||
derive_subdir,
|
|
||||||
derive_top_level_artist,
|
|
||||||
hash_suffixed_name,
|
|
||||||
safe_ext,
|
|
||||||
)
|
|
||||||
from ..utils.phash import compute_phash, find_similar
|
from ..utils.phash import compute_phash, find_similar
|
||||||
from ..utils.sidecar import find_sidecar, parse_sidecar
|
from ..utils.sidecar import find_sidecar, parse_sidecar
|
||||||
from ..utils.slug import slugify
|
from ..utils.slug import slugify
|
||||||
@@ -87,9 +82,27 @@ def is_video(path: Path) -> bool:
|
|||||||
|
|
||||||
def _safe_ext(path: Path) -> str:
|
def _safe_ext(path: Path) -> str:
|
||||||
"""Conservatively extract a file extension for PostAttachment.ext
|
"""Conservatively extract a file extension for PostAttachment.ext
|
||||||
(varchar(32)). Thin wrapper over the shared `utils.paths.safe_ext` (kept for
|
(varchar(32)).
|
||||||
the Path-typed call sites + the [[path_suffix_sanitize]] memory pointer)."""
|
|
||||||
return safe_ext(path)
|
gallery-dl produces some filenames with URL-encoded query-string
|
||||||
|
artifacts embedded into the basename (e.g.
|
||||||
|
`79507046_media_..._https___www.patreon.com_media-u_Z0FBQUFBQm5q...`).
|
||||||
|
`Path.suffix` finds the LAST dot and returns everything after, which
|
||||||
|
in those cases yields a 50+ char "extension" of mostly base64-ish
|
||||||
|
junk. That blows the column. Operator-flagged 2026-05-25.
|
||||||
|
|
||||||
|
Real extensions are short and alphanumeric. We accept anything ≤ 16
|
||||||
|
chars where every post-dot character is alphanumeric; anything else
|
||||||
|
means the input wasn't a real extension and we return the empty
|
||||||
|
string. ext is nullable-ish (empty string still satisfies NOT NULL)
|
||||||
|
and consumers should treat "" as "no known extension".
|
||||||
|
"""
|
||||||
|
suffix = path.suffix.lower()
|
||||||
|
if not suffix or len(suffix) > 16:
|
||||||
|
return ""
|
||||||
|
if not all(c.isalnum() for c in suffix[1:]):
|
||||||
|
return ""
|
||||||
|
return suffix
|
||||||
|
|
||||||
|
|
||||||
def _mime_for(path: Path) -> str:
|
def _mime_for(path: Path) -> str:
|
||||||
@@ -430,17 +443,13 @@ class Importer:
|
|||||||
self._capture_attachment(
|
self._capture_attachment(
|
||||||
source, post=post, artist=artist_use, resolved=True,
|
source, post=post, artist=artist_use, resolved=True,
|
||||||
)
|
)
|
||||||
reason = f"archive probe rejected, captured unextracted: {probe.reason}"
|
return ImportResult(status="attached")
|
||||||
log.warning("%s: %s", source.name, reason)
|
|
||||||
return ImportResult(status="attached", error=reason)
|
|
||||||
|
|
||||||
artist_use = artist if artist is not None else self._resolve_artist(source)
|
artist_use = artist if artist is not None else self._resolve_artist(source)
|
||||||
post = self._post_for_sidecar(source, artist_use)
|
post = self._post_for_sidecar(source, artist_use)
|
||||||
member_ids: list[int] = []
|
member_ids: list[int] = []
|
||||||
member_total = 0
|
|
||||||
with extract_archive(source) as members:
|
with extract_archive(source) as members:
|
||||||
for _name, member_path in members:
|
for _name, member_path in members:
|
||||||
member_total += 1
|
|
||||||
if not is_supported(member_path):
|
if not is_supported(member_path):
|
||||||
continue # non-media preserved via the stored archive
|
continue # non-media preserved via the stored archive
|
||||||
res = self._import_media(
|
res = self._import_media(
|
||||||
@@ -457,19 +466,7 @@ class Importer:
|
|||||||
status="imported", image_id=member_ids[0],
|
status="imported", image_id=member_ids[0],
|
||||||
member_image_ids=member_ids,
|
member_image_ids=member_ids,
|
||||||
)
|
)
|
||||||
# No images landed — surface WHY so a post showing "no images" beside an
|
return ImportResult(status="attached")
|
||||||
# archive is diagnosable instead of silent. Zero members usually means
|
|
||||||
# the extractor backend is missing/failed (unar for rar, py7zr for 7z)
|
|
||||||
# or the file is corrupt; non-zero-but-no-images means it held only
|
|
||||||
# non-media files.
|
|
||||||
reason = (
|
|
||||||
"archive extracted but held no supported image/video members"
|
|
||||||
if member_total
|
|
||||||
else "archive yielded no members (unsupported/corrupt, or the "
|
|
||||||
"extractor backend failed)"
|
|
||||||
)
|
|
||||||
log.warning("%s: %s", source.name, reason)
|
|
||||||
return ImportResult(status="attached", error=reason)
|
|
||||||
|
|
||||||
def _import_media(
|
def _import_media(
|
||||||
self, source: Path, attribution_path: Path,
|
self, source: Path, attribution_path: Path,
|
||||||
@@ -963,14 +960,6 @@ class Importer:
|
|||||||
sp.rollback()
|
sp.rollback()
|
||||||
if record.primary_post_id is None:
|
if record.primary_post_id is None:
|
||||||
record.primary_post_id = post.id
|
record.primary_post_id = post.id
|
||||||
# Keep the denormalized gallery sort key (alembic 0035) aligned with
|
|
||||||
# the primary post's publish date so /scroll orders off
|
|
||||||
# ix_image_record_effective_date instead of COALESCE-ing across the
|
|
||||||
# post join. Only override when THIS post is the primary AND carries
|
|
||||||
# a date; otherwise the column keeps its created_at-equivalent server
|
|
||||||
# default (matches the old COALESCE(post_date, created_at) fallback).
|
|
||||||
if record.primary_post_id == post.id and post.post_date is not None:
|
|
||||||
record.effective_date = post.post_date
|
|
||||||
self.session.flush()
|
self.session.flush()
|
||||||
|
|
||||||
def _copy_to_library(
|
def _copy_to_library(
|
||||||
|
|||||||
@@ -1,647 +0,0 @@
|
|||||||
"""Platform-agnostic native-ingest core (plan #706, build on #697/#703/#704/#705).
|
|
||||||
|
|
||||||
The orchestration that drives a native subscription walk — page a feed →
|
|
||||||
extract media → tiered skip (seen-ledger / on-disk / dead-letter) → download →
|
|
||||||
mark-seen / record-failures / checkpoint-cursor → return a gallery-dl-shaped
|
|
||||||
`DownloadResult`, across tick/backfill/recovery modes — is identical for every
|
|
||||||
platform. Only four things are platform-specific, and they're INJECTED at
|
|
||||||
construction by a thin adapter (e.g. `PatreonIngester`):
|
|
||||||
|
|
||||||
- `client` — `.iter_posts(feed_id, cursor)` yielding `(post, included,
|
|
||||||
page_cursor)` + `.extract_media(post, included) -> [media]`.
|
|
||||||
- `downloader`— `.download_post(post, media, artist_slug, is_seen,
|
|
||||||
should_stop) -> [MediaOutcome]` (status in downloaded/
|
|
||||||
skipped_seen/skipped_disk/quarantined/error;
|
|
||||||
`.path`/`.error`/`.post_id`). `should_stop()` is polled
|
|
||||||
between media so the time-box is honoured mid-post.
|
|
||||||
- ledger — `seen_model` + `failed_model` SQLAlchemy models (+ their
|
|
||||||
on-conflict UNIQUE constraint names) and a `ledger_key(media)`.
|
|
||||||
- failure map — the adapter overrides `_failure_result` (platform exception
|
|
||||||
→ DownloadResult.error_type) and supplies `error_base` (the
|
|
||||||
exception type the walk catches) + `platform` (result label).
|
|
||||||
|
|
||||||
Everything DB touches a SHORT-LIVED sync session from the injected sessionmaker —
|
|
||||||
never held across a network fetch ([[db-connection-held-across-subprocess]]).
|
|
||||||
Plain-HTTP homelab: no secure-context Web API.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import time
|
|
||||||
from collections.abc import Callable
|
|
||||||
|
|
||||||
from sqlalchemy import delete, func, select, text
|
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
|
|
||||||
from .gallery_dl import DownloadResult, ErrorType, make_run_stats
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Stop a tick after this many CONTIGUOUS already-have-it media (seen-ledger or
|
|
||||||
# on-disk) — the cheap native equivalent of gallery-dl's `exit:20`, now free of
|
|
||||||
# per-file HEADs. Headroom against paywalled/undownloadable items interleaving.
|
|
||||||
_TICK_SEEN_THRESHOLD = 20
|
|
||||||
|
|
||||||
# plan #705 #7: after this many failed download/validate attempts a media is
|
|
||||||
# "dead-lettered" and skipped on routine tick/backfill walks (recovery still
|
|
||||||
# re-attempts it). Stops a permanently-broken media re-erroring forever.
|
|
||||||
DEAD_LETTER_THRESHOLD = 3
|
|
||||||
# last_error is Text but bound it so a giant traceback doesn't bloat the row.
|
|
||||||
_ERROR_MAX = 1000
|
|
||||||
|
|
||||||
# plan #709: throttle the live-progress write to the running DownloadEvent to one
|
|
||||||
# every ~5s — a steady cadence for the Downloads view regardless of how big/slow a
|
|
||||||
# page is (page boundaries can be minutes apart on image-dense backfills, so a
|
|
||||||
# page-tied update would lurch). Trivial churn (~one single-row UPDATE / 5s).
|
|
||||||
_LIVE_PROGRESS_INTERVAL = 5.0
|
|
||||||
|
|
||||||
|
|
||||||
class Ingester:
|
|
||||||
"""Generic native-ingest orchestration. Subclass with a platform adapter
|
|
||||||
(see the module docstring) — or construct directly with the keyword seams."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
client,
|
|
||||||
downloader,
|
|
||||||
session_factory: Callable[[], object],
|
|
||||||
seen_model,
|
|
||||||
failed_model,
|
|
||||||
seen_constraint: str,
|
|
||||||
failed_constraint: str,
|
|
||||||
ledger_key: Callable[[object], str],
|
|
||||||
platform: str,
|
|
||||||
error_base: type[Exception],
|
|
||||||
):
|
|
||||||
self.client = client
|
|
||||||
self.downloader = downloader
|
|
||||||
self.session_factory = session_factory
|
|
||||||
self._seen_model = seen_model
|
|
||||||
self._failed_model = failed_model
|
|
||||||
self._seen_constraint = seen_constraint
|
|
||||||
self._failed_constraint = failed_constraint
|
|
||||||
self._ledger_key = ledger_key
|
|
||||||
self._platform = platform
|
|
||||||
self._error_base = error_base
|
|
||||||
|
|
||||||
# -- public ------------------------------------------------------------
|
|
||||||
|
|
||||||
def run(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
source_id: int,
|
|
||||||
campaign_id: str,
|
|
||||||
artist_slug: str,
|
|
||||||
url: str,
|
|
||||||
mode: str,
|
|
||||||
resume_cursor: str | None = None,
|
|
||||||
time_budget_seconds: float = 870.0,
|
|
||||||
seen_threshold: int = _TICK_SEEN_THRESHOLD,
|
|
||||||
posts_base: int = 0,
|
|
||||||
event_id: int | None = None,
|
|
||||||
) -> DownloadResult:
|
|
||||||
"""Walk + download for one source, returning a gallery-dl-shaped result.
|
|
||||||
|
|
||||||
`mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
|
|
||||||
seen-ledger AND the dead-letter ledger (tier-2 disk still skips kept
|
|
||||||
files). The walk stops on:
|
|
||||||
- budget exhaustion (time_budget_seconds) → TIMEOUT / PARTIAL
|
|
||||||
- tick early-out (seen_threshold contiguous seen) → success
|
|
||||||
- reaching the bottom of the feed → success (rc 0)
|
|
||||||
A client-level failure (drift / auth / network) fails the whole run loud.
|
|
||||||
"""
|
|
||||||
bypass_seen = mode == "recovery"
|
|
||||||
# Only deep walks checkpoint their cursor mid-flight (plan #705 #6); a
|
|
||||||
# tick has no resumable backfill state.
|
|
||||||
checkpoint = mode in ("backfill", "recovery")
|
|
||||||
ledger_key = self._ledger_key
|
|
||||||
start = time.monotonic()
|
|
||||||
last_live = start # plan #709: last live-progress write timestamp
|
|
||||||
log_lines: list[str] = []
|
|
||||||
written: list[str] = []
|
|
||||||
quarantined_paths: list[str] = []
|
|
||||||
downloaded = 0
|
|
||||||
errors = 0
|
|
||||||
quarantined = 0
|
|
||||||
dead_lettered = 0
|
|
||||||
skipped_count = 0
|
|
||||||
posts_processed = 0
|
|
||||||
# Net-new posts THIS chunk for the live progress badge (plan #704 #5);
|
|
||||||
# excludes the re-walked resume page so _backfill_posts stays a monotonic
|
|
||||||
# absolute across chunks instead of an inflating sum. posts_processed
|
|
||||||
# stays the gross per-chunk count used for the run summary.
|
|
||||||
chunk_new_posts = 0
|
|
||||||
consecutive_seen = 0
|
|
||||||
emitted_cursor: str | None = None
|
|
||||||
reached_bottom = False
|
|
||||||
budget_hit = False
|
|
||||||
early_out = False
|
|
||||||
stopped = False # plan #708 B4: operator hit Stop mid-walk
|
|
||||||
cancel_armed = False # latched once we observe a live "running" state
|
|
||||||
|
|
||||||
def _result(
|
|
||||||
*, success: bool, return_code: int,
|
|
||||||
error_type: ErrorType | None, error_message: str | None,
|
|
||||||
) -> DownloadResult:
|
|
||||||
# plan #704: return STRUCTURED data — phase 3 reads run_stats/cursor
|
|
||||||
# directly instead of regex-scraping a reconstructed stdout. stdout
|
|
||||||
# stays a human-readable summary (no fake `Cursor:` lines).
|
|
||||||
return DownloadResult(
|
|
||||||
success=success,
|
|
||||||
url=url,
|
|
||||||
artist_slug=artist_slug,
|
|
||||||
platform=self._platform,
|
|
||||||
files_downloaded=downloaded,
|
|
||||||
files_quarantined=quarantined,
|
|
||||||
quarantined_paths=list(quarantined_paths),
|
|
||||||
written_paths=written,
|
|
||||||
stdout="\n".join(log_lines),
|
|
||||||
stderr="",
|
|
||||||
return_code=return_code,
|
|
||||||
error_type=error_type,
|
|
||||||
error_message=error_message,
|
|
||||||
duration_seconds=time.monotonic() - start,
|
|
||||||
cursor=emitted_cursor,
|
|
||||||
posts_processed=posts_processed,
|
|
||||||
run_stats=make_run_stats(
|
|
||||||
exit_code=return_code,
|
|
||||||
downloaded_count=downloaded,
|
|
||||||
skipped_count=skipped_count,
|
|
||||||
per_item_failures=errors,
|
|
||||||
quarantined_count=quarantined,
|
|
||||||
dead_lettered_count=dead_lettered,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
for post, included, page_cursor in self.client.iter_posts(
|
|
||||||
campaign_id, cursor=resume_cursor
|
|
||||||
):
|
|
||||||
# Checkpoint the cursor that FETCHED this page the moment we
|
|
||||||
# START it — so a chunk cut mid-page resumes the page, not the one
|
|
||||||
# after it. Carried as DownloadResult.cursor (plan #704).
|
|
||||||
if page_cursor and page_cursor != emitted_cursor:
|
|
||||||
emitted_cursor = page_cursor
|
|
||||||
# plan #705 #6: persist the cursor at each page boundary so a
|
|
||||||
# worker SIGKILL mid-chunk resumes near the crash, not the
|
|
||||||
# chunk start. (phase 3 still writes the final cursor — same
|
|
||||||
# value; this is the crash-safety net.) plan #704 #5: persist
|
|
||||||
# the live posts count alongside it so the badge climbs DURING
|
|
||||||
# the chunk, not only when it ends.
|
|
||||||
if checkpoint:
|
|
||||||
# plan #708 B4: an operator Stop pops `_backfill_state` —
|
|
||||||
# bail at the page boundary (progress already checkpointed)
|
|
||||||
# before more network work, so the live chunk halts
|
|
||||||
# promptly instead of running to its time-box. LATCH on the
|
|
||||||
# first observed "running" state, so a run invoked WITHOUT a
|
|
||||||
# running state (a unit test, or a stale call) never
|
|
||||||
# spuriously self-cancels. A short SELECT, never held.
|
|
||||||
if self._still_running(source_id):
|
|
||||||
cancel_armed = True
|
|
||||||
elif cancel_armed:
|
|
||||||
stopped = True
|
|
||||||
break
|
|
||||||
self._checkpoint_cursor(source_id, emitted_cursor)
|
|
||||||
self._checkpoint_posts(source_id, posts_base + chunk_new_posts)
|
|
||||||
|
|
||||||
# Time-box check at the post boundary (coarse, like a gallery-dl
|
|
||||||
# chunk). Backfill/recovery resume from emitted_cursor next chunk.
|
|
||||||
if time.monotonic() - start >= time_budget_seconds:
|
|
||||||
budget_hit = True
|
|
||||||
break
|
|
||||||
|
|
||||||
posts_processed += 1
|
|
||||||
# The resume page (its cursor == resume_cursor) was already
|
|
||||||
# counted by the chunk that checkpointed it — don't re-count it
|
|
||||||
# into the persisted badge (plan #704 #5). First chunk has
|
|
||||||
# resume_cursor None, so everything counts.
|
|
||||||
if not (resume_cursor and page_cursor == resume_cursor):
|
|
||||||
chunk_new_posts += 1
|
|
||||||
media = self.client.extract_media(post, included)
|
|
||||||
if not media:
|
|
||||||
continue
|
|
||||||
|
|
||||||
keys = [ledger_key(m) for m in media]
|
|
||||||
# Recovery bypasses BOTH the seen-ledger AND the dead-letter
|
|
||||||
# ledger (the operator's "try everything again"); routine walks
|
|
||||||
# skip seen + dead media (tier-1 + tier-1.5, plan #705 #7).
|
|
||||||
dead = set() if bypass_seen else self._dead_keys(source_id, keys)
|
|
||||||
seen = (
|
|
||||||
set()
|
|
||||||
if bypass_seen
|
|
||||||
else self._seen_keys(source_id, keys)
|
|
||||||
)
|
|
||||||
skip = seen | dead
|
|
||||||
|
|
||||||
def _is_skip(m, _skip=skip) -> bool:
|
|
||||||
return ledger_key(m) in _skip
|
|
||||||
|
|
||||||
# Honour the time-box DURING a media-dense post too, not only at
|
|
||||||
# the per-post boundary below — else one heavy post can blow the
|
|
||||||
# chunk budget out to the Celery soft limit (Pocketacer, 2026-06-07).
|
|
||||||
outcomes = self.downloader.download_post(
|
|
||||||
post, media, artist_slug, is_seen=_is_skip,
|
|
||||||
should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
|
|
||||||
)
|
|
||||||
|
|
||||||
to_mark: list[tuple[str, str]] = []
|
|
||||||
to_clear: list[str] = [] # recovered → drop any dead-letter row
|
|
||||||
to_fail: list[tuple[str, str, str]] = [] # (key, post_id, error)
|
|
||||||
for media_item, outcome in zip(media, outcomes, strict=False):
|
|
||||||
key = ledger_key(media_item)
|
|
||||||
if key in dead:
|
|
||||||
dead_lettered += 1 # skipped because previously dead
|
|
||||||
if outcome.status == "downloaded":
|
|
||||||
downloaded += 1
|
|
||||||
if outcome.path is not None:
|
|
||||||
written.append(str(outcome.path))
|
|
||||||
to_mark.append((key, media_item.post_id))
|
|
||||||
to_clear.append(key)
|
|
||||||
consecutive_seen = 0
|
|
||||||
elif outcome.status == "skipped_disk":
|
|
||||||
# Already on disk (a prior run). Reconcile the ledger so a
|
|
||||||
# later tick skips it at tier-1 without a disk stat, but
|
|
||||||
# do NOT re-feed it to phase 3 — attach_in_place would see
|
|
||||||
# the duplicate sha256 and unlink the on-disk copy.
|
|
||||||
to_mark.append((key, media_item.post_id))
|
|
||||||
to_clear.append(key)
|
|
||||||
skipped_count += 1
|
|
||||||
consecutive_seen += 1
|
|
||||||
elif outcome.status == "skipped_seen":
|
|
||||||
skipped_count += 1
|
|
||||||
consecutive_seen += 1
|
|
||||||
elif outcome.status == "quarantined":
|
|
||||||
# New content that failed validation (corrupt) — counted
|
|
||||||
# distinctly so the run surfaces a real quarantined total.
|
|
||||||
# Not marked seen (a later walk may re-fetch a fixed file);
|
|
||||||
# it IS new content, so it breaks the run-of-seen. Counts
|
|
||||||
# toward the dead-letter ledger (plan #705 #7).
|
|
||||||
quarantined += 1
|
|
||||||
if outcome.path is not None:
|
|
||||||
quarantined_paths.append(str(outcome.path))
|
|
||||||
to_fail.append((key, media_item.post_id, outcome.error or "quarantined"))
|
|
||||||
consecutive_seen = 0
|
|
||||||
elif outcome.status == "error":
|
|
||||||
errors += 1
|
|
||||||
to_fail.append((key, media_item.post_id, outcome.error or "error"))
|
|
||||||
# An error neither advances nor resets the run-of-seen.
|
|
||||||
|
|
||||||
if mode == "tick" and consecutive_seen >= seen_threshold:
|
|
||||||
early_out = True
|
|
||||||
break
|
|
||||||
|
|
||||||
# Persist ledger changes AFTER the network fetch, on short
|
|
||||||
# sessions: mark downloaded/on-disk seen, clear any dead-letter
|
|
||||||
# for recovered media, and record failures (plan #705 #7).
|
|
||||||
if to_mark:
|
|
||||||
self._mark_seen(source_id, to_mark)
|
|
||||||
if to_clear:
|
|
||||||
self._clear_failures(source_id, to_clear)
|
|
||||||
if to_fail:
|
|
||||||
self._record_failures(source_id, to_fail)
|
|
||||||
|
|
||||||
# plan #709: time-throttled live progress to the running event so
|
|
||||||
# the Downloads view ticks ~every 5s, independent of page size.
|
|
||||||
now = time.monotonic()
|
|
||||||
if event_id is not None and (now - last_live) >= _LIVE_PROGRESS_INTERVAL:
|
|
||||||
last_live = now
|
|
||||||
self._write_live_progress(event_id, {
|
|
||||||
"downloaded": downloaded,
|
|
||||||
"skipped": skipped_count,
|
|
||||||
"errors": errors,
|
|
||||||
"quarantined": quarantined,
|
|
||||||
"posts": posts_processed,
|
|
||||||
})
|
|
||||||
|
|
||||||
if early_out:
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
reached_bottom = True
|
|
||||||
except self._error_base as exc:
|
|
||||||
# The platform's client-error base — _failure_result (adapter)
|
|
||||||
# maps it to a typed error.
|
|
||||||
return self._failure_result(exc, _result)
|
|
||||||
|
|
||||||
# plan #708 B4: a Stop already popped the backfill state (incl. cursor +
|
|
||||||
# posts), so don't re-write them — return PARTIAL (reads as "ok/progress",
|
|
||||||
# the lifecycle no-ops since state is gone) instead of a false "complete".
|
|
||||||
if stopped:
|
|
||||||
return _result(
|
|
||||||
success=False, return_code=-1,
|
|
||||||
error_type=ErrorType.PARTIAL,
|
|
||||||
error_message=f"Stopped by operator: {downloaded} file(s) this chunk",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Final authoritative posts count for the badge — captures the last page
|
|
||||||
# after the last boundary write and the time-box break (plan #704 #5).
|
|
||||||
if checkpoint:
|
|
||||||
self._checkpoint_posts(source_id, posts_base + chunk_new_posts)
|
|
||||||
|
|
||||||
if errors:
|
|
||||||
log_lines.append(f"{errors} media item(s) failed")
|
|
||||||
if quarantined:
|
|
||||||
log_lines.append(f"{quarantined} media item(s) quarantined (invalid)")
|
|
||||||
if dead_lettered:
|
|
||||||
log_lines.append(f"{dead_lettered} media item(s) skipped (dead-lettered)")
|
|
||||||
log_lines.append(
|
|
||||||
f"{self._platform} ingest ({mode}): {downloaded} downloaded, "
|
|
||||||
f"{skipped_count} skipped, {quarantined} quarantined, "
|
|
||||||
f"{dead_lettered} dead-lettered, {errors} error(s), "
|
|
||||||
f"{posts_processed} post(s)"
|
|
||||||
+ (", reached end" if reached_bottom else "")
|
|
||||||
+ (", time-boxed" if budget_hit else "")
|
|
||||||
)
|
|
||||||
|
|
||||||
if budget_hit:
|
|
||||||
# A chunk that hit its time-box but made forward progress is a
|
|
||||||
# NORMAL chunk boundary, not a failure (PARTIAL → status "ok"); the
|
|
||||||
# next chunk resumes from the emitted cursor. No progress → TIMEOUT,
|
|
||||||
# which feeds download_service's backfill stall-guard. rc<0 mirrors
|
|
||||||
# subprocess TimeoutExpired so completion detection stays false.
|
|
||||||
made_progress = downloaded > 0 or emitted_cursor != resume_cursor
|
|
||||||
if made_progress:
|
|
||||||
return _result(
|
|
||||||
success=False, return_code=-1,
|
|
||||||
error_type=ErrorType.PARTIAL,
|
|
||||||
error_message=(
|
|
||||||
f"Backfill chunk: {downloaded} file(s) — continuing"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return _result(
|
|
||||||
success=False, return_code=-1,
|
|
||||||
error_type=ErrorType.TIMEOUT,
|
|
||||||
error_message="Chunk timed out with no progress",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Normal success: reached the bottom, or a tick that early-outed. rc 0 +
|
|
||||||
# error_type None is REQUIRED for a backfill/recovery walk that reached
|
|
||||||
# the bottom to be marked COMPLETE by
|
|
||||||
# download_service._apply_backfill_lifecycle — so we return None even
|
|
||||||
# when downloaded == 0 (a re-confirming walk that found nothing new still
|
|
||||||
# completed). success=True maps to status "ok" regardless. A tick that
|
|
||||||
# early-outed also returns here; ticks never set backfill state so the
|
|
||||||
# lifecycle is a no-op for them.
|
|
||||||
return _result(
|
|
||||||
success=True, return_code=0,
|
|
||||||
error_type=None, error_message=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
# -- preview (dry-run) -------------------------------------------------
|
|
||||||
|
|
||||||
def preview(
|
|
||||||
self,
|
|
||||||
source_id: int,
|
|
||||||
campaign_id: str,
|
|
||||||
*,
|
|
||||||
page_limit: int = 3,
|
|
||||||
sample_size: int = 10,
|
|
||||||
) -> dict:
|
|
||||||
"""Dry-run (plan #708 B4): walk up to `page_limit` pages and count media
|
|
||||||
NOT already in the seen/dead ledgers, WITHOUT downloading anything.
|
|
||||||
|
|
||||||
Read-only — only the seen/dead SELECTs touch the DB (short sessions). Lets
|
|
||||||
an operator gauge "is this source worth a backfill?" cheaply. Returns:
|
|
||||||
{total_new, posts_scanned, pages_scanned, has_more,
|
|
||||||
sample: [{title, date, new}, ...]} # sample = posts with new media
|
|
||||||
A client-level failure (auth/drift) propagates to the caller.
|
|
||||||
"""
|
|
||||||
total_new = 0
|
|
||||||
posts_scanned = 0
|
|
||||||
pages_scanned = 0
|
|
||||||
has_more = False
|
|
||||||
sample: list[dict] = []
|
|
||||||
unset = object()
|
|
||||||
last_page: object = unset
|
|
||||||
for post, included, page_cursor in self.client.iter_posts(
|
|
||||||
campaign_id, cursor=None
|
|
||||||
):
|
|
||||||
if page_cursor != last_page:
|
|
||||||
last_page = page_cursor
|
|
||||||
pages_scanned += 1
|
|
||||||
if pages_scanned > page_limit:
|
|
||||||
has_more = True
|
|
||||||
pages_scanned = page_limit
|
|
||||||
break
|
|
||||||
posts_scanned += 1
|
|
||||||
media = self.client.extract_media(post, included)
|
|
||||||
if not media:
|
|
||||||
continue
|
|
||||||
keys = [self._ledger_key(m) for m in media]
|
|
||||||
skip = self._seen_keys(source_id, keys) | self._dead_keys(source_id, keys)
|
|
||||||
new_count = sum(1 for m in media if self._ledger_key(m) not in skip)
|
|
||||||
total_new += new_count
|
|
||||||
if new_count > 0 and len(sample) < sample_size:
|
|
||||||
meta = self.client.post_meta(post)
|
|
||||||
sample.append(
|
|
||||||
{
|
|
||||||
"title": meta.get("title") or "(untitled)",
|
|
||||||
"date": meta.get("date"),
|
|
||||||
"new": new_count,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"total_new": total_new,
|
|
||||||
"posts_scanned": posts_scanned,
|
|
||||||
"pages_scanned": pages_scanned,
|
|
||||||
"has_more": has_more,
|
|
||||||
"sample": sample,
|
|
||||||
}
|
|
||||||
|
|
||||||
# -- failure mapping (adapter overrides) -------------------------------
|
|
||||||
|
|
||||||
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
|
|
||||||
"""Map a platform client-error to a typed failed DownloadResult. The base
|
|
||||||
gives a safe default; adapters override with their exception taxonomy."""
|
|
||||||
log.warning("%s ingest failed: %s", self._platform, exc)
|
|
||||||
return _result(
|
|
||||||
success=False, return_code=1,
|
|
||||||
error_type=ErrorType.UNKNOWN_ERROR, error_message=str(exc),
|
|
||||||
)
|
|
||||||
|
|
||||||
# -- seen-ledger (short-lived sessions) --------------------------------
|
|
||||||
|
|
||||||
def _seen_keys(self, source_id: int, keys: list[str]) -> set[str]:
|
|
||||||
"""Which of `keys` are already in the seen-ledger for this source.
|
|
||||||
|
|
||||||
One short SELECT on its own session — opened and closed without any
|
|
||||||
network in between (the GETs happen after, in download_post).
|
|
||||||
"""
|
|
||||||
if not keys:
|
|
||||||
return set()
|
|
||||||
with self.session_factory() as session:
|
|
||||||
rows = session.execute(
|
|
||||||
select(self._seen_model.filehash).where(
|
|
||||||
self._seen_model.source_id == source_id,
|
|
||||||
self._seen_model.filehash.in_(keys),
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return set(rows)
|
|
||||||
|
|
||||||
def _checkpoint_cursor(self, source_id: int, cursor: str) -> None:
|
|
||||||
"""Persist the in-progress backfill cursor mid-walk (plan #705 #6).
|
|
||||||
|
|
||||||
ATOMIC, single-key UPDATE: cast the JSON column to jsonb, set just
|
|
||||||
`_backfill_cursor`, cast back — so it never clobbers operator config or
|
|
||||||
the other backfill keys (no read-modify-write race). The in-flight guard
|
|
||||||
means only this source's one download runs at a time; a concurrent
|
|
||||||
operator stop is benign (a stray cursor with no `_backfill_state` is
|
|
||||||
ignored by tick mode and cleared on the next start).
|
|
||||||
"""
|
|
||||||
with self.session_factory() as session:
|
|
||||||
session.execute(
|
|
||||||
text(
|
|
||||||
"UPDATE source SET config_overrides = jsonb_set("
|
|
||||||
" coalesce(config_overrides::jsonb, '{}'::jsonb),"
|
|
||||||
" '{_backfill_cursor}', to_jsonb(cast(:cur AS text))"
|
|
||||||
")::json WHERE id = :sid"
|
|
||||||
),
|
|
||||||
{"cur": cursor, "sid": source_id},
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _write_live_progress(self, event_id: int, counts: dict) -> None:
|
|
||||||
"""Throttled mid-walk write of live counts to the RUNNING download_event
|
|
||||||
(plan #709) so the Downloads view shows progress before the chunk
|
|
||||||
finishes. A short session (never held across the walk); the `status =
|
|
||||||
'running'` guard avoids clobbering an event phase 3 already finalized.
|
|
||||||
`metadata` is JSONB — jsonb_set sets just the `live` key, leaving the rest
|
|
||||||
for phase 3 to overwrite with the final run_stats."""
|
|
||||||
with self.session_factory() as session:
|
|
||||||
session.execute(
|
|
||||||
text(
|
|
||||||
"UPDATE download_event SET metadata = jsonb_set("
|
|
||||||
" coalesce(metadata, '{}'::jsonb), '{live}',"
|
|
||||||
" cast(:live AS jsonb)) "
|
|
||||||
"WHERE id = :eid AND status = 'running'"
|
|
||||||
),
|
|
||||||
{"live": json.dumps(counts), "eid": event_id},
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _still_running(self, source_id: int) -> bool:
|
|
||||||
"""True while the source is armed for a deep walk (plan #708 B4).
|
|
||||||
|
|
||||||
An operator Stop (`source_service.stop_backfill`) pops `_backfill_state`,
|
|
||||||
so a False here means "cancel this chunk now". One short SELECT on its own
|
|
||||||
session — never held across the walk
|
|
||||||
([[db-connection-held-across-subprocess]])."""
|
|
||||||
with self.session_factory() as session:
|
|
||||||
state = session.execute(
|
|
||||||
text(
|
|
||||||
"SELECT config_overrides::jsonb ->> '_backfill_state' "
|
|
||||||
"FROM source WHERE id = :sid"
|
|
||||||
),
|
|
||||||
{"sid": source_id},
|
|
||||||
).scalar_one_or_none()
|
|
||||||
return state == "running"
|
|
||||||
|
|
||||||
def _checkpoint_posts(self, source_id: int, posts: int) -> None:
|
|
||||||
"""Persist the live backfill posts-processed count mid-walk (plan #704 #5).
|
|
||||||
|
|
||||||
Same atomic single-key jsonb_set dance as _checkpoint_cursor, on the
|
|
||||||
`_backfill_posts` key (cast to a JSON number) — so the progress badge
|
|
||||||
climbs DURING a chunk without clobbering operator config or the cursor.
|
|
||||||
The ingester OWNS this key now; download_service no longer accumulates it
|
|
||||||
post-chunk (which lagged a whole chunk and over-counted the resume page).
|
|
||||||
"""
|
|
||||||
with self.session_factory() as session:
|
|
||||||
session.execute(
|
|
||||||
text(
|
|
||||||
"UPDATE source SET config_overrides = jsonb_set("
|
|
||||||
" coalesce(config_overrides::jsonb, '{}'::jsonb),"
|
|
||||||
" '{_backfill_posts}', to_jsonb(cast(:posts AS int))"
|
|
||||||
")::json WHERE id = :sid"
|
|
||||||
),
|
|
||||||
{"posts": posts, "sid": source_id},
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None:
|
|
||||||
"""Idempotent upsert of (filehash, post_id) seen-ledger rows for a page.
|
|
||||||
|
|
||||||
ON CONFLICT DO NOTHING against the (source_id, filehash) UNIQUE so a
|
|
||||||
re-sighting — or a concurrent walk — is a harmless no-op
|
|
||||||
([[scalar_one_or_none-duplicates]]: never check-then-insert without the
|
|
||||||
DB constraint backing it). De-dup the batch locally first so a single
|
|
||||||
page can't present the same key twice to one INSERT.
|
|
||||||
"""
|
|
||||||
seen_local: set[str] = set()
|
|
||||||
values = []
|
|
||||||
for key, post_id in items:
|
|
||||||
if key in seen_local:
|
|
||||||
continue
|
|
||||||
seen_local.add(key)
|
|
||||||
values.append(
|
|
||||||
{"source_id": source_id, "filehash": key, "post_id": post_id}
|
|
||||||
)
|
|
||||||
if not values:
|
|
||||||
return
|
|
||||||
with self.session_factory() as session:
|
|
||||||
stmt = pg_insert(self._seen_model).values(values)
|
|
||||||
stmt = stmt.on_conflict_do_nothing(constraint=self._seen_constraint)
|
|
||||||
session.execute(stmt)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
# -- dead-letter ledger (plan #705 #7) ---------------------------------
|
|
||||||
|
|
||||||
def _dead_keys(self, source_id: int, keys: list[str]) -> set[str]:
|
|
||||||
"""Which of `keys` have failed >= DEAD_LETTER_THRESHOLD times (dead).
|
|
||||||
One short SELECT; recovery never calls this (it re-attempts dead media)."""
|
|
||||||
if not keys:
|
|
||||||
return set()
|
|
||||||
with self.session_factory() as session:
|
|
||||||
rows = session.execute(
|
|
||||||
select(self._failed_model.filehash).where(
|
|
||||||
self._failed_model.source_id == source_id,
|
|
||||||
self._failed_model.filehash.in_(keys),
|
|
||||||
self._failed_model.attempts >= DEAD_LETTER_THRESHOLD,
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return set(rows)
|
|
||||||
|
|
||||||
def _record_failures(
|
|
||||||
self, source_id: int, items: list[tuple[str, str, str]]
|
|
||||||
) -> None:
|
|
||||||
"""Upsert-increment the dead-letter ledger for failed media. On conflict
|
|
||||||
bump `attempts` and refresh last_error/last_failed_at (UNIQUE backs the
|
|
||||||
upsert — no check-then-insert). De-dup the batch (one row/key, last error
|
|
||||||
wins)."""
|
|
||||||
by_key: dict[str, str] = {}
|
|
||||||
for key, _post_id, err in items:
|
|
||||||
by_key[key] = (err or "")[:_ERROR_MAX]
|
|
||||||
if not by_key:
|
|
||||||
return
|
|
||||||
values = [
|
|
||||||
{"source_id": source_id, "filehash": k, "attempts": 1, "last_error": e}
|
|
||||||
for k, e in by_key.items()
|
|
||||||
]
|
|
||||||
with self.session_factory() as session:
|
|
||||||
stmt = pg_insert(self._failed_model).values(values)
|
|
||||||
stmt = stmt.on_conflict_do_update(
|
|
||||||
constraint=self._failed_constraint,
|
|
||||||
set_={
|
|
||||||
"attempts": self._failed_model.attempts + 1,
|
|
||||||
"last_error": stmt.excluded.last_error,
|
|
||||||
"last_failed_at": func.now(),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
session.execute(stmt)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _clear_failures(self, source_id: int, keys: list[str]) -> None:
|
|
||||||
"""Drop dead-letter rows for media that just downloaded cleanly — they
|
|
||||||
recovered. A no-op DELETE for keys that were never failing."""
|
|
||||||
unique = list(dict.fromkeys(keys))
|
|
||||||
if not unique:
|
|
||||||
return
|
|
||||||
with self.session_factory() as session:
|
|
||||||
session.execute(
|
|
||||||
delete(self._failed_model).where(
|
|
||||||
self._failed_model.source_id == source_id,
|
|
||||||
self._failed_model.filehash.in_(unique),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
"""Parse a stated page number/range out of a post's title/description (FC-6.2).
|
|
||||||
|
|
||||||
Artists often state where an installment sits in a series — "pages 9-12",
|
|
||||||
"Page 5", "[3/8]". We use that to order chapters and flag missing-page gaps.
|
|
||||||
This is best-effort: a confident match wins, otherwise we return None and the
|
|
||||||
caller falls back to capture/post-date order. Keep it conservative — a wrong
|
|
||||||
page number is worse than no page number — so matches require an explicit
|
|
||||||
page keyword (page/pg/pp) or a bracketed N/M fraction, never a bare number.
|
|
||||||
|
|
||||||
Supported forms (case-insensitive):
|
|
||||||
range "pages 9-12", "pg 9–12", "pp. 9 - 12" -> (9, 12)
|
|
||||||
fraction "page 3 of 8", "pg 3/8", "[3/8]", "(3/8)" -> (3, 3)
|
|
||||||
single "page 5", "pg 5", "pp 5" -> (5, 5)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
|
|
||||||
# Page keyword: page/pages/pg/pgs/pp/pp. (NOT a bare "p" — too many false hits.)
|
|
||||||
_KW = r"(?:pages?|pgs?|pp\.?)"
|
|
||||||
_DASH = r"[-–—]"
|
|
||||||
|
|
||||||
_RANGE = re.compile(rf"\b{_KW}\s*(\d{{1,4}})\s*{_DASH}\s*(\d{{1,4}})", re.I)
|
|
||||||
_OF = re.compile(rf"\b{_KW}\s*(\d{{1,4}})\s*(?:of|/)\s*\d{{1,4}}\b", re.I)
|
|
||||||
_BRACKET = re.compile(r"[\[(]\s*(\d{1,4})\s*/\s*\d{1,4}\s*[\])]")
|
|
||||||
_SINGLE = re.compile(rf"\b{_KW}\s*(\d{{1,4}})\b", re.I)
|
|
||||||
|
|
||||||
|
|
||||||
def parse_page_range(text: str | None) -> tuple[int, int] | None:
|
|
||||||
"""Return (start, end) or None. start <= end; a single page yields (n, n)."""
|
|
||||||
if not text:
|
|
||||||
return None
|
|
||||||
m = _RANGE.search(text)
|
|
||||||
if m:
|
|
||||||
a, b = int(m.group(1)), int(m.group(2))
|
|
||||||
return (a, b) if a <= b else (b, a)
|
|
||||||
for rx in (_OF, _BRACKET, _SINGLE):
|
|
||||||
m = rx.search(text)
|
|
||||||
if m:
|
|
||||||
n = int(m.group(1))
|
|
||||||
return (n, n)
|
|
||||||
return None
|
|
||||||
@@ -1,604 +0,0 @@
|
|||||||
"""Native Patreon JSON:API client (build step 1 of the native ingester).
|
|
||||||
|
|
||||||
Clean-room reimplementation of the Patreon `/api/posts` read path. This is a
|
|
||||||
plain, synchronous client over `requests` — the orchestrator already wraps
|
|
||||||
sync importer calls, so nothing here needs to be async (mirrors
|
|
||||||
patreon_resolver.py, which wraps its sync lookup in run_in_executor at the
|
|
||||||
call site).
|
|
||||||
|
|
||||||
Scope (build step 1): fetch + page + parse only. This module is NOT wired into
|
|
||||||
download_service yet — that is a later step. The public surface here exists so
|
|
||||||
the later step can drive it:
|
|
||||||
- PatreonClient(cookies_path).iter_posts(campaign_id)
|
|
||||||
→ (post, included_index, page_cursor)
|
|
||||||
- extract_media(post, included_index) → list[MediaItem]
|
|
||||||
- parse_cursor_from_url(url) → cursor
|
|
||||||
|
|
||||||
Drift detection is loud on purpose: Patreon ships JSON:API and the shapes we
|
|
||||||
depend on (top-level `data`, media resources carrying `file_name`/`url`) are
|
|
||||||
the contract. If a response comes back as an HTML login page or a media
|
|
||||||
resource is missing the fields we resolve against, we raise PatreonDriftError
|
|
||||||
rather than silently yielding empty media — so the later import step surfaces
|
|
||||||
"Patreon changed something" instead of "creator has no posts".
|
|
||||||
|
|
||||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import http.cookiejar
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import time
|
|
||||||
from collections.abc import Iterator
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from html import unescape
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import parse_qs, urlsplit
|
|
||||||
|
|
||||||
import requests
|
|
||||||
|
|
||||||
from ..utils.paths import safe_ext
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
_POSTS_URL = "https://www.patreon.com/api/posts"
|
|
||||||
_USER_AGENT = (
|
|
||||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
|
||||||
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
|
|
||||||
)
|
|
||||||
_TIMEOUT_SECONDS = 30.0
|
|
||||||
|
|
||||||
# 429 backoff (plan #703): ride out a transient API rate-limit instead of
|
|
||||||
# failing the whole walk (which would stamp RATE_LIMITED → platform-wide
|
|
||||||
# cooldown → every Patreon source dark). Honor the server's `Retry-After`;
|
|
||||||
# otherwise exponential base·2^(n-1), capped. Only after the retries are
|
|
||||||
# exhausted does the 429 propagate as terminal RATE_LIMITED.
|
|
||||||
_MAX_429_RETRIES = 3
|
|
||||||
_BACKOFF_BASE_SECONDS = 2.0
|
|
||||||
_BACKOFF_CAP_SECONDS = 30.0
|
|
||||||
|
|
||||||
|
|
||||||
def _retry_after_seconds(
|
|
||||||
resp: requests.Response,
|
|
||||||
attempt: int,
|
|
||||||
*,
|
|
||||||
base: float = _BACKOFF_BASE_SECONDS,
|
|
||||||
cap: float = _BACKOFF_CAP_SECONDS,
|
|
||||||
) -> float:
|
|
||||||
"""Backoff delay for a 429: the `Retry-After` seconds header if present and
|
|
||||||
numeric, else exponential `base·2^(attempt-1)`, both capped. (HTTP-date form
|
|
||||||
of Retry-After is rare here and falls through to exponential.)"""
|
|
||||||
header = resp.headers.get("Retry-After")
|
|
||||||
if header:
|
|
||||||
try:
|
|
||||||
return min(float(header), cap)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
return min(base * (2 ** max(0, attempt - 1)), cap)
|
|
||||||
|
|
||||||
# JSON:API request contract (observed from real traffic — see module plan).
|
|
||||||
_INCLUDE = (
|
|
||||||
"campaign,access_rules,attachments,attachments_media,audio,images,media,"
|
|
||||||
"native_video_insights,user,user_defined_tags,ti_checks"
|
|
||||||
)
|
|
||||||
_FIELDS_POST = (
|
|
||||||
"content,post_file,image,post_type,published_at,title,url,patreon_url,"
|
|
||||||
"current_user_can_view"
|
|
||||||
)
|
|
||||||
_FIELDS_MEDIA = "id,image_urls,download_url,metadata,file_name"
|
|
||||||
_FIELDS_CAMPAIGN = "name,url"
|
|
||||||
|
|
||||||
# A CDN download URL embeds a 32-char hex (MD5) path segment; that segment is
|
|
||||||
# Patreon's stable per-file identity and is what we dedup + ledger against.
|
|
||||||
# Same role gallery-dl's _filehash plays. Match the FIRST 32-hex run anywhere
|
|
||||||
# in the URL (path or query); real Patreon CDN URLs carry exactly one.
|
|
||||||
_FILEHASH_RE = re.compile(r"([0-9a-fA-F]{32})")
|
|
||||||
|
|
||||||
# Inline post `content` is HTML; images are emitted as <img ... src="...">.
|
|
||||||
# Pull every src; downstream dedup collapses any that duplicate a gallery item
|
|
||||||
# by filehash. Tolerant of attribute ordering and single/double quotes.
|
|
||||||
_CONTENT_IMG_RE = re.compile(r"<img\b[^>]*?\bsrc=[\"']([^\"']+)[\"']", re.IGNORECASE)
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonAPIError(Exception):
|
|
||||||
"""Base for native Patreon client failures.
|
|
||||||
|
|
||||||
`status_code` carries the HTTP status when the failure was an HTTP response
|
|
||||||
(None for transport-level / parse failures), so the ingester can map it to a
|
|
||||||
DownloadResult.error_type (429 → rate_limited, 404 → not_found, …).
|
|
||||||
`retry_after` carries the server's `Retry-After` seconds on a terminal 429, so
|
|
||||||
the platform cooldown can match the server's hint instead of a flat default
|
|
||||||
(plan #708 B1).
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
message: str,
|
|
||||||
*,
|
|
||||||
status_code: int | None = None,
|
|
||||||
retry_after: float | None = None,
|
|
||||||
):
|
|
||||||
super().__init__(message)
|
|
||||||
self.status_code = status_code
|
|
||||||
self.retry_after = retry_after
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonAuthError(PatreonAPIError):
|
|
||||||
"""Authentication / authorization failure — missing or expired session
|
|
||||||
cookies, an insufficient pledge tier, or an HTML login/challenge page served
|
|
||||||
where JSON was expected. DISTINCT from drift: the fix is rotating the
|
|
||||||
credential, not updating the ingester. Maps to error_type 'auth_error'.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonDriftError(PatreonAPIError):
|
|
||||||
"""A JSON response did not match the JSON:API shape we depend on.
|
|
||||||
|
|
||||||
Raised for: a missing top-level `data` list, `data` not a list, or a media
|
|
||||||
resource lacking the `file_name`/`url` fields we resolve against. Fail loud
|
|
||||||
so the import step flags API drift (ingester needs update) instead of
|
|
||||||
silently importing nothing. An HTML-login / non-JSON body is auth, not
|
|
||||||
drift — that raises PatreonAuthError.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class MediaItem:
|
|
||||||
"""One resolved downloadable item belonging to a post.
|
|
||||||
|
|
||||||
Fields:
|
|
||||||
url — the CDN/download URL to fetch.
|
|
||||||
filename — media `file_name` when present; otherwise the URL basename
|
|
||||||
(NEVER a network call). Bounded/sane extension.
|
|
||||||
kind — one of: "images", "image_large", "attachments", "postfile",
|
|
||||||
"content". Mirrors gallery-dl's `files` content-type names so
|
|
||||||
the later step can honor the same per-source content_types.
|
|
||||||
filehash — the 32-char hex (MD5) segment from the CDN URL, or None if
|
|
||||||
the URL carries no such segment. Used for in-post dedup and
|
|
||||||
the cross-run seen-ledger.
|
|
||||||
post_id — the owning post's id (so a flattened media list stays
|
|
||||||
traceable to its post).
|
|
||||||
"""
|
|
||||||
|
|
||||||
url: str
|
|
||||||
filename: str
|
|
||||||
kind: str
|
|
||||||
filehash: str | None
|
|
||||||
post_id: str
|
|
||||||
|
|
||||||
|
|
||||||
def _load_session(cookies_path: str | Path | None) -> requests.Session:
|
|
||||||
session = requests.Session()
|
|
||||||
session.headers.update(
|
|
||||||
{
|
|
||||||
"User-Agent": _USER_AGENT,
|
|
||||||
"Accept": "application/vnd.api+json",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
if cookies_path and os.path.isfile(str(cookies_path)):
|
|
||||||
try:
|
|
||||||
jar = http.cookiejar.MozillaCookieJar(str(cookies_path))
|
|
||||||
jar.load(ignore_discard=True, ignore_expires=True)
|
|
||||||
session.cookies = jar # type: ignore[assignment]
|
|
||||||
except (OSError, http.cookiejar.LoadError) as exc:
|
|
||||||
log.warning("Could not load Patreon cookies from %s: %s", cookies_path, exc)
|
|
||||||
return session
|
|
||||||
|
|
||||||
|
|
||||||
def _filehash(url: str) -> str | None:
|
|
||||||
if not url:
|
|
||||||
return None
|
|
||||||
match = _FILEHASH_RE.search(url)
|
|
||||||
return match.group(1).lower() if match else None
|
|
||||||
|
|
||||||
|
|
||||||
def _basename_from_url(url: str) -> str:
|
|
||||||
"""Derive a sane filename from a URL when the media has no file_name.
|
|
||||||
|
|
||||||
Strips query/fragment, takes the path basename, and drops a junk
|
|
||||||
extension (the importer._safe_ext gotcha) so we never write base64 noise
|
|
||||||
as a name. Falls back to the filehash, then to "file".
|
|
||||||
"""
|
|
||||||
path = urlsplit(url).path
|
|
||||||
base = os.path.basename(path)
|
|
||||||
if base:
|
|
||||||
ext = safe_ext(base)
|
|
||||||
stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
|
|
||||||
# Keep the stem bounded; URL-encoded stems can be enormous.
|
|
||||||
stem = stem[:120] or "file"
|
|
||||||
return f"{stem}{ext}"
|
|
||||||
fh = _filehash(url)
|
|
||||||
return fh or "file"
|
|
||||||
|
|
||||||
|
|
||||||
def parse_cursor_from_url(url: str | None) -> str | None:
|
|
||||||
"""Extract the `page[cursor]` query param from a links.next URL."""
|
|
||||||
if not url:
|
|
||||||
return None
|
|
||||||
query = urlsplit(url).query
|
|
||||||
values = parse_qs(query).get("page[cursor]")
|
|
||||||
if values and values[0]:
|
|
||||||
return values[0]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonClient:
|
|
||||||
"""Synchronous Patreon JSON:API read client.
|
|
||||||
|
|
||||||
Construct with a path to a Netscape cookies.txt (the same file
|
|
||||||
CredentialService.get_cookies_path materializes). Cookies are loaded into a
|
|
||||||
requests.Session; no secure-context APIs are used.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
cookies_path: str | Path | None,
|
|
||||||
*,
|
|
||||||
request_sleep: float = 0.0,
|
|
||||||
max_retries: int = _MAX_429_RETRIES,
|
|
||||||
):
|
|
||||||
self.cookies_path = str(cookies_path) if cookies_path else None
|
|
||||||
self._session = _load_session(cookies_path)
|
|
||||||
# Politeness: seconds to sleep before each /api/posts page fetch (paces
|
|
||||||
# the rate-limited API endpoint). 0 = no pacing. plan #703.
|
|
||||||
self._request_sleep = request_sleep or 0.0
|
|
||||||
self._max_retries = max_retries
|
|
||||||
|
|
||||||
# -- request -----------------------------------------------------------
|
|
||||||
|
|
||||||
def _params(self, campaign_id: str, cursor: str | None) -> dict[str, str]:
|
|
||||||
params = {
|
|
||||||
"include": _INCLUDE,
|
|
||||||
"fields[post]": _FIELDS_POST,
|
|
||||||
"fields[media]": _FIELDS_MEDIA,
|
|
||||||
"fields[campaign]": _FIELDS_CAMPAIGN,
|
|
||||||
"filter[campaign_id]": campaign_id,
|
|
||||||
"filter[contains_exclusive_posts]": "true",
|
|
||||||
"filter[is_draft]": "false",
|
|
||||||
"sort": "-published_at",
|
|
||||||
"json-api-version": "1.0",
|
|
||||||
}
|
|
||||||
if cursor:
|
|
||||||
params["page[cursor]"] = cursor
|
|
||||||
return params
|
|
||||||
|
|
||||||
def _fetch(self, campaign_id: str, cursor: str | None) -> dict:
|
|
||||||
if self._request_sleep > 0:
|
|
||||||
time.sleep(self._request_sleep) # pace the API endpoint
|
|
||||||
attempt = 0
|
|
||||||
while True:
|
|
||||||
try:
|
|
||||||
resp = self._session.get(
|
|
||||||
_POSTS_URL,
|
|
||||||
params=self._params(campaign_id, cursor),
|
|
||||||
timeout=_TIMEOUT_SECONDS,
|
|
||||||
)
|
|
||||||
except requests.RequestException as exc:
|
|
||||||
raise PatreonAPIError(
|
|
||||||
f"Patreon posts request failed (campaign_id={campaign_id}): {exc}"
|
|
||||||
) from exc
|
|
||||||
|
|
||||||
# Transient rate-limit: back off and retry rather than failing the
|
|
||||||
# whole walk. Only a PERSISTENT 429 (retries exhausted) falls
|
|
||||||
# through to the terminal RATE_LIMITED raise below.
|
|
||||||
if resp.status_code == 429 and attempt < self._max_retries:
|
|
||||||
attempt += 1
|
|
||||||
delay = _retry_after_seconds(resp, attempt)
|
|
||||||
log.warning(
|
|
||||||
"Patreon 429 (campaign_id=%s) — backing off %.1fs (retry %d/%d)",
|
|
||||||
campaign_id, delay, attempt, self._max_retries,
|
|
||||||
)
|
|
||||||
time.sleep(delay)
|
|
||||||
continue
|
|
||||||
break
|
|
||||||
|
|
||||||
if resp.status_code in (401, 403):
|
|
||||||
# Auth rejected — expired/missing cookies or an insufficient tier.
|
|
||||||
# Actionable as "rotate credentials", so it's auth, not drift/http.
|
|
||||||
raise PatreonAuthError(
|
|
||||||
f"Patreon posts API returned HTTP {resp.status_code} — auth "
|
|
||||||
f"rejected (cookies expired or tier insufficient; "
|
|
||||||
f"campaign_id={campaign_id})",
|
|
||||||
status_code=resp.status_code,
|
|
||||||
)
|
|
||||||
if resp.status_code != 200:
|
|
||||||
# A persistent 429 (retries exhausted) is terminal RATE_LIMITED — carry
|
|
||||||
# the server's raw Retry-After seconds so the cooldown matches its hint
|
|
||||||
# (plan #708 B1). Header is uncapped here; the cooldown clamps it.
|
|
||||||
retry_after = None
|
|
||||||
if resp.status_code == 429:
|
|
||||||
hdr = resp.headers.get("Retry-After")
|
|
||||||
if hdr:
|
|
||||||
try:
|
|
||||||
retry_after = float(hdr)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
retry_after = None
|
|
||||||
raise PatreonAPIError(
|
|
||||||
f"Patreon posts API returned HTTP {resp.status_code} "
|
|
||||||
f"(campaign_id={campaign_id})",
|
|
||||||
status_code=resp.status_code,
|
|
||||||
retry_after=retry_after,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
payload = resp.json()
|
|
||||||
except ValueError as exc:
|
|
||||||
# A non-JSON body here is almost always the HTML login/challenge
|
|
||||||
# page served when cookies are missing/expired — that is an AUTH
|
|
||||||
# failure (rotate cookies), not API drift (update the ingester) and
|
|
||||||
# not a transient network error.
|
|
||||||
raise PatreonAuthError(
|
|
||||||
"Patreon posts API returned a non-JSON response (likely an "
|
|
||||||
f"HTML login/challenge page — session expired; "
|
|
||||||
f"campaign_id={campaign_id}): {exc}"
|
|
||||||
) from exc
|
|
||||||
return payload
|
|
||||||
|
|
||||||
# -- parsing -----------------------------------------------------------
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _transform(response: dict) -> dict:
|
|
||||||
"""Flatten the JSON:API `included` array for relationship resolution.
|
|
||||||
|
|
||||||
Returns a dict keyed by `(type, id)` → that resource's `attributes`
|
|
||||||
(so a post's relationships can be resolved in O(1)). Missing/oddly
|
|
||||||
shaped `included` entries are skipped rather than fatal — drift
|
|
||||||
detection for the top-level shape lives in _validate_response.
|
|
||||||
"""
|
|
||||||
index: dict[tuple[str, str], dict] = {}
|
|
||||||
for inc in response.get("included") or []:
|
|
||||||
if not isinstance(inc, dict):
|
|
||||||
continue
|
|
||||||
rtype = inc.get("type")
|
|
||||||
rid = inc.get("id")
|
|
||||||
if rtype is None or rid is None:
|
|
||||||
continue
|
|
||||||
index[(str(rtype), str(rid))] = inc.get("attributes") or {}
|
|
||||||
return index
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _validate_response(response: dict) -> None:
|
|
||||||
if not isinstance(response, dict):
|
|
||||||
raise PatreonDriftError("Patreon response was not a JSON object")
|
|
||||||
if "data" not in response:
|
|
||||||
raise PatreonDriftError("Patreon response missing top-level 'data' key")
|
|
||||||
data = response.get("data")
|
|
||||||
if not isinstance(data, list):
|
|
||||||
raise PatreonDriftError("Patreon response 'data' was not a list")
|
|
||||||
|
|
||||||
def _related_ids(self, post: dict, rel_name: str) -> list[str]:
|
|
||||||
rels = post.get("relationships") or {}
|
|
||||||
rel = rels.get(rel_name) or {}
|
|
||||||
data = rel.get("data")
|
|
||||||
if isinstance(data, dict): # to-one relationship
|
|
||||||
data = [data]
|
|
||||||
if not isinstance(data, list):
|
|
||||||
return []
|
|
||||||
ids: list[str] = []
|
|
||||||
for ref in data:
|
|
||||||
if isinstance(ref, dict) and ref.get("id") is not None:
|
|
||||||
ids.append(str(ref["id"]))
|
|
||||||
return ids
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _media_url(attrs: dict) -> str | None:
|
|
||||||
"""Pick the best fetchable URL for a media resource.
|
|
||||||
|
|
||||||
Prefer the full-size `download_url`; fall back to the largest
|
|
||||||
`image_urls` size. gallery-dl prefers download_url too, only dipping
|
|
||||||
into image_urls when a smaller configured size is requested — FC
|
|
||||||
always wants the original, so download_url first.
|
|
||||||
"""
|
|
||||||
download_url = attrs.get("download_url")
|
|
||||||
if isinstance(download_url, str) and download_url:
|
|
||||||
return download_url
|
|
||||||
image_urls = attrs.get("image_urls")
|
|
||||||
if isinstance(image_urls, dict):
|
|
||||||
for key in ("original", "full", "large", "default"):
|
|
||||||
candidate = image_urls.get(key)
|
|
||||||
if isinstance(candidate, str) and candidate:
|
|
||||||
return candidate
|
|
||||||
# Otherwise take any non-empty string value.
|
|
||||||
for candidate in image_urls.values():
|
|
||||||
if isinstance(candidate, str) and candidate:
|
|
||||||
return candidate
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _media_item(self, attrs: dict, kind: str, post_id: str) -> MediaItem:
|
|
||||||
url = self._media_url(attrs)
|
|
||||||
if not url:
|
|
||||||
raise PatreonDriftError(
|
|
||||||
f"Patreon media (post {post_id}, kind={kind}) had no resolvable URL "
|
|
||||||
f"(no download_url / image_urls)"
|
|
||||||
)
|
|
||||||
# file_name is OPTIONAL: Patreon legitimately serves some gallery images
|
|
||||||
# without it (operator-flagged 2026-06-07, BlenderKnight post 73665615),
|
|
||||||
# and the URL basename is a fine fallback — the same thing gallery-dl
|
|
||||||
# uses. A genuine schema change shows up as no URL (above) or a media id
|
|
||||||
# absent from `included` (caller), not a missing name.
|
|
||||||
file_name = attrs.get("file_name")
|
|
||||||
filename = file_name if isinstance(file_name, str) and file_name else _basename_from_url(url)
|
|
||||||
return MediaItem(
|
|
||||||
url=url,
|
|
||||||
filename=filename,
|
|
||||||
kind=kind,
|
|
||||||
filehash=_filehash(url),
|
|
||||||
post_id=post_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
|
|
||||||
"""Resolve all downloadable media for one post.
|
|
||||||
|
|
||||||
Walks the kinds in the same order gallery-dl does — images,
|
|
||||||
image_large (post cover), attachments, postfile, content (inline
|
|
||||||
<img>) — and dedups within the post by filehash (first wins). The
|
|
||||||
image_large cover commonly duplicates a gallery image; deduping by
|
|
||||||
filehash collapses them to the gallery item (encountered first).
|
|
||||||
"""
|
|
||||||
post_id = str(post.get("id") or "")
|
|
||||||
attrs = post.get("attributes") or {}
|
|
||||||
items: list[MediaItem] = []
|
|
||||||
|
|
||||||
def _resolve_rel(rel_name: str, kind: str) -> None:
|
|
||||||
for mid in self._related_ids(post, rel_name):
|
|
||||||
media_attrs = included_index.get(("media", mid))
|
|
||||||
if media_attrs is None:
|
|
||||||
# Referenced but not in `included`: a media id with no
|
|
||||||
# resource is drift (we asked for include=media).
|
|
||||||
raise PatreonDriftError(
|
|
||||||
f"Patreon post {post_id} references media {mid} "
|
|
||||||
f"({rel_name}) not present in 'included'"
|
|
||||||
)
|
|
||||||
items.append(self._media_item(media_attrs, kind, post_id))
|
|
||||||
|
|
||||||
# 1. gallery images
|
|
||||||
_resolve_rel("images", "images")
|
|
||||||
|
|
||||||
# 2. image_large — the post-level cover (`image.large_url`). Not a
|
|
||||||
# media relationship; it lives on the post attributes.
|
|
||||||
image = attrs.get("image")
|
|
||||||
if isinstance(image, dict):
|
|
||||||
large_url = image.get("large_url") or image.get("url")
|
|
||||||
if isinstance(large_url, str) and large_url:
|
|
||||||
items.append(
|
|
||||||
MediaItem(
|
|
||||||
url=large_url,
|
|
||||||
filename=_basename_from_url(large_url),
|
|
||||||
kind="image_large",
|
|
||||||
filehash=_filehash(large_url),
|
|
||||||
post_id=post_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# 3. attachments
|
|
||||||
_resolve_rel("attachments_media", "attachments")
|
|
||||||
|
|
||||||
# 4. postfile — the post's primary attached file (`post_file`).
|
|
||||||
post_file = attrs.get("post_file")
|
|
||||||
if isinstance(post_file, dict):
|
|
||||||
pf_url = post_file.get("url") or post_file.get("download_url")
|
|
||||||
if isinstance(pf_url, str) and pf_url:
|
|
||||||
pf_name = post_file.get("name")
|
|
||||||
filename = (
|
|
||||||
pf_name
|
|
||||||
if isinstance(pf_name, str) and pf_name
|
|
||||||
else _basename_from_url(pf_url)
|
|
||||||
)
|
|
||||||
items.append(
|
|
||||||
MediaItem(
|
|
||||||
url=pf_url,
|
|
||||||
filename=filename,
|
|
||||||
kind="postfile",
|
|
||||||
filehash=_filehash(pf_url),
|
|
||||||
post_id=post_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# 5. content — inline <img> in the post HTML body.
|
|
||||||
content = attrs.get("content")
|
|
||||||
if isinstance(content, str) and content:
|
|
||||||
for raw_src in _CONTENT_IMG_RE.findall(content):
|
|
||||||
src = unescape(raw_src)
|
|
||||||
if not src:
|
|
||||||
continue
|
|
||||||
items.append(
|
|
||||||
MediaItem(
|
|
||||||
url=src,
|
|
||||||
filename=_basename_from_url(src),
|
|
||||||
kind="content",
|
|
||||||
filehash=_filehash(src),
|
|
||||||
post_id=post_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return _dedup_by_filehash(items)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def post_meta(post: dict) -> dict:
|
|
||||||
"""Title + published date for a post — for the preview sample (plan #708
|
|
||||||
B4). Part of the client contract `ingest_core.Ingester.preview` calls."""
|
|
||||||
attrs = post.get("attributes") or {}
|
|
||||||
title = attrs.get("title")
|
|
||||||
published = attrs.get("published_at")
|
|
||||||
return {
|
|
||||||
"title": title if isinstance(title, str) else None,
|
|
||||||
"date": published if isinstance(published, str) else None,
|
|
||||||
}
|
|
||||||
|
|
||||||
# -- iteration ---------------------------------------------------------
|
|
||||||
|
|
||||||
def iter_posts(
|
|
||||||
self, campaign_id: str, cursor: str | None = None
|
|
||||||
) -> Iterator[tuple[dict, dict, str | None]]:
|
|
||||||
"""Yield (post, included_index, page_cursor) for every post in the feed.
|
|
||||||
|
|
||||||
Pages newest→oldest via `links.next`, validating each response for
|
|
||||||
drift before yielding. The triple gives a caller everything it needs to
|
|
||||||
resolve and checkpoint without re-fetching:
|
|
||||||
- post — the raw post resource.
|
|
||||||
- included_index — the page's flattened `included` (the same object
|
|
||||||
for every post on a page), to pass straight to
|
|
||||||
extract_media(post, included_index).
|
|
||||||
- page_cursor — the cursor that FETCHED this post's page (None for
|
|
||||||
the first page). The caller checkpoints THIS value,
|
|
||||||
matching the existing backfill cursor logic where
|
|
||||||
the saved cursor re-fetches the page being
|
|
||||||
processed (so a chunk cut mid-page resumes the page,
|
|
||||||
not the one after it).
|
|
||||||
"""
|
|
||||||
current_cursor = cursor
|
|
||||||
while True:
|
|
||||||
response = self._fetch(campaign_id, current_cursor)
|
|
||||||
self._validate_response(response)
|
|
||||||
page_cursor = current_cursor
|
|
||||||
included_index = self._transform(response)
|
|
||||||
for post in response.get("data") or []:
|
|
||||||
if isinstance(post, dict):
|
|
||||||
yield post, included_index, page_cursor
|
|
||||||
next_url = (response.get("links") or {}).get("next")
|
|
||||||
next_cursor = parse_cursor_from_url(next_url)
|
|
||||||
if not next_cursor:
|
|
||||||
return
|
|
||||||
current_cursor = next_cursor
|
|
||||||
|
|
||||||
# -- verify ------------------------------------------------------------
|
|
||||||
|
|
||||||
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
|
|
||||||
"""Cheap auth probe: fetch the first `/api/posts` page and report whether
|
|
||||||
the credential authenticated, WITHOUT downloading anything.
|
|
||||||
|
|
||||||
Returns `(ok, message)` matching the credential-verify contract:
|
|
||||||
- True — authenticated (the feed returned a valid JSON:API page).
|
|
||||||
- False — the credential was rejected (PatreonAuthError: 401/403, or an
|
|
||||||
HTML login page → cookies expired / tier insufficient).
|
|
||||||
- None — inconclusive: API drift (our parser is stale, not a cred
|
|
||||||
problem) or a transient network/HTTP error.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
response = self._fetch(campaign_id, None)
|
|
||||||
self._validate_response(response)
|
|
||||||
except PatreonAuthError as exc:
|
|
||||||
return False, f"Patreon rejected the credential — {exc}"
|
|
||||||
except PatreonDriftError as exc:
|
|
||||||
return None, f"Couldn't verify — Patreon's API shape changed: {exc}"
|
|
||||||
except PatreonAPIError as exc:
|
|
||||||
return None, f"Couldn't verify (network/HTTP issue): {exc}"
|
|
||||||
return True, "Credentials valid — the Patreon feed authenticated."
|
|
||||||
|
|
||||||
|
|
||||||
def _dedup_by_filehash(items: list[MediaItem]) -> list[MediaItem]:
|
|
||||||
"""Drop later items sharing a filehash with an earlier one (first wins).
|
|
||||||
|
|
||||||
Items with no filehash (None) are never deduped against each other — we
|
|
||||||
can't prove they're the same file, so keep them all.
|
|
||||||
"""
|
|
||||||
seen: set[str] = set()
|
|
||||||
out: list[MediaItem] = []
|
|
||||||
for item in items:
|
|
||||||
if item.filehash is not None:
|
|
||||||
if item.filehash in seen:
|
|
||||||
continue
|
|
||||||
seen.add(item.filehash)
|
|
||||||
out.append(item)
|
|
||||||
return out
|
|
||||||
@@ -1,510 +0,0 @@
|
|||||||
"""Native Patreon media downloader (build step 2b of the native ingester).
|
|
||||||
|
|
||||||
Given a Patreon post and its already-resolved `MediaItem`s (from
|
|
||||||
patreon_client.extract_media), download the media to the EXACT on-disk layout
|
|
||||||
gallery-dl produces, write a sidecar JSON the existing importer consumes, and
|
|
||||||
report per-media outcomes.
|
|
||||||
|
|
||||||
This module is PURE: no DB. The cross-run seen-ledger and the iter_posts
|
|
||||||
orchestration are a LATER step. The tier-1 (seen) skip is an INJECTED predicate
|
|
||||||
(`is_seen`), so this module needs no DB and is unit-testable without network.
|
|
||||||
|
|
||||||
On-disk layout (matches gallery-dl):
|
|
||||||
<images_root>/<artist_slug>/patreon/<DIR>/<NN>_<filename>
|
|
||||||
where <DIR> = "<YYYY-MM-DD>_<post_id>_<title40>" (date prefix omitted when the
|
|
||||||
post's published_at is unparseable), <NN> is the 1-based index of the item in
|
|
||||||
the post zero-padded to 2, and <filename> is MediaItem.filename. The sidecar
|
|
||||||
is written next to the media as <NN>_<stem>.json (media_path.with_suffix).
|
|
||||||
|
|
||||||
Video: a MediaItem whose URL is a Mux/HLS stream (host stream.mux.com or path
|
|
||||||
endswith .m3u8) is fetched with yt-dlp (subprocess), passing the same
|
|
||||||
Referer/Origin headers gallery-dl forwards for Mux playback (see gallery_dl.py
|
|
||||||
_get_default_config). yt-dlp may remux to a container of its own choosing, so we
|
|
||||||
accept the actual output extension and record the real path.
|
|
||||||
|
|
||||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import contextlib
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
from collections.abc import Callable
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from datetime import datetime
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import urlsplit
|
|
||||||
|
|
||||||
import requests
|
|
||||||
|
|
||||||
from .file_validator import is_validatable, quarantine_file, validate_file
|
|
||||||
from .patreon_client import (
|
|
||||||
_BACKOFF_CAP_SECONDS,
|
|
||||||
_load_session,
|
|
||||||
_retry_after_seconds,
|
|
||||||
)
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
_TITLE_MAX = 40
|
|
||||||
_TIMEOUT_SECONDS = 120.0
|
|
||||||
_CHUNK = 1 << 16
|
|
||||||
# Retry a media GET that hits a TRANSIENT failure within the same pass (plan
|
|
||||||
# #705 #8): a transport blip (connection reset / timeout / truncated stream), a
|
|
||||||
# 429, or a 5xx. PERMANENT failures (404 gone, 403 forbidden) fail fast straight
|
|
||||||
# to the error/dead-letter path — no point re-fetching them. Keeps a momentary
|
|
||||||
# network hiccup from becoming a per-item error that waits for the next walk.
|
|
||||||
_MAX_MEDIA_RETRIES = 3
|
|
||||||
# requests transport errors worth retrying (vs. an HTTPError, which is a real
|
|
||||||
# server response and is classified by status code).
|
|
||||||
_TRANSIENT_TRANSPORT_EXC = (
|
|
||||||
requests.ConnectionError,
|
|
||||||
requests.Timeout,
|
|
||||||
requests.exceptions.ChunkedEncodingError,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Referer/Origin yt-dlp must send for Mux-hosted Patreon video. Mux's JWT
|
|
||||||
# playback policy checks Referer/Origin on every request, so yt-dlp must send
|
|
||||||
# Patreon's, not its own default. (gallery-dl forwarded the same headers before
|
|
||||||
# the #697 cutover removed its Patreon path; this is now the only place they
|
|
||||||
# live.)
|
|
||||||
_VIDEO_HEADERS = {
|
|
||||||
"Referer": "https://www.patreon.com/",
|
|
||||||
"Origin": "https://www.patreon.com",
|
|
||||||
}
|
|
||||||
|
|
||||||
# Characters Windows/gallery-dl path-restrict forbids, plus path separators.
|
|
||||||
_FORBIDDEN = set('<>:"/\\|?*')
|
|
||||||
|
|
||||||
|
|
||||||
def _sanitize(name: str) -> str:
|
|
||||||
"""Make `name` safe for a single filesystem path segment.
|
|
||||||
|
|
||||||
Replaces path separators, the Windows-forbidden set <>:"/\\|?* and control
|
|
||||||
characters with `_`, then strips trailing dots/spaces (gallery-dl
|
|
||||||
path-restrict behavior). Never returns empty (falls back to "_").
|
|
||||||
"""
|
|
||||||
out = []
|
|
||||||
for ch in name:
|
|
||||||
if ch in _FORBIDDEN or ord(ch) < 32:
|
|
||||||
out.append("_")
|
|
||||||
else:
|
|
||||||
out.append(ch)
|
|
||||||
cleaned = "".join(out).rstrip(". ")
|
|
||||||
return cleaned or "_"
|
|
||||||
|
|
||||||
|
|
||||||
def _is_video_url(url: str) -> bool:
|
|
||||||
parts = urlsplit(url)
|
|
||||||
if parts.hostname and parts.hostname.lower() == "stream.mux.com":
|
|
||||||
return True
|
|
||||||
return parts.path.lower().endswith(".m3u8")
|
|
||||||
|
|
||||||
|
|
||||||
def _post_dir_name(post: dict) -> str:
|
|
||||||
"""Build the post directory name matching gallery-dl's layout."""
|
|
||||||
post_id = str(post.get("id") or "")
|
|
||||||
attrs = post.get("attributes") or {}
|
|
||||||
title = attrs.get("title")
|
|
||||||
title = title if isinstance(title, str) else ""
|
|
||||||
title40 = title[:_TITLE_MAX]
|
|
||||||
|
|
||||||
published = attrs.get("published_at")
|
|
||||||
date_prefix = None
|
|
||||||
if isinstance(published, str) and published:
|
|
||||||
s = published.strip()
|
|
||||||
if s.endswith("Z"):
|
|
||||||
s = s[:-1] + "+00:00"
|
|
||||||
try:
|
|
||||||
dt = datetime.fromisoformat(s)
|
|
||||||
except ValueError:
|
|
||||||
dt = None
|
|
||||||
if dt is not None:
|
|
||||||
date_prefix = f"{dt:%Y-%m-%d}"
|
|
||||||
|
|
||||||
if date_prefix:
|
|
||||||
raw = f"{date_prefix}_{post_id}_{title40}"
|
|
||||||
else:
|
|
||||||
raw = f"{post_id}_{title40}"
|
|
||||||
return _sanitize(raw)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class MediaOutcome:
|
|
||||||
"""Per-media result of a download_post pass.
|
|
||||||
|
|
||||||
status is one of: "downloaded", "skipped_seen", "skipped_disk",
|
|
||||||
"quarantined", "error". `path` is the final on-disk path for "downloaded"
|
|
||||||
(the actual yt-dlp output for video), the path that already existed for
|
|
||||||
"skipped_disk", or the _quarantine destination for "quarantined"; None for
|
|
||||||
"skipped_seen" and (usually) "error". `error` carries the failure/validation
|
|
||||||
reason for "error"/"quarantined", else None.
|
|
||||||
"""
|
|
||||||
|
|
||||||
media: object # MediaItem (avoid importing the name for a bare annotation)
|
|
||||||
status: str
|
|
||||||
path: Path | None
|
|
||||||
error: str | None
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonDownloader:
|
|
||||||
"""Download resolved Patreon media to gallery-dl's on-disk layout.
|
|
||||||
|
|
||||||
PURE: no DB. The HTTP session and the yt-dlp invocation are injectable seams
|
|
||||||
so tests run without network or a real subprocess:
|
|
||||||
- pass `session=` to stub `session.get`, or monkeypatch `_fetch_to_file`.
|
|
||||||
- monkeypatch `_run_ytdlp` to avoid spawning yt-dlp.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
images_root: Path,
|
|
||||||
cookies_path: str | None = None,
|
|
||||||
*,
|
|
||||||
validate: bool = True,
|
|
||||||
rate_limit: float = 0.0,
|
|
||||||
session: requests.Session | None = None,
|
|
||||||
):
|
|
||||||
self.images_root = Path(images_root)
|
|
||||||
self.cookies_path = str(cookies_path) if cookies_path else None
|
|
||||||
self._validate = validate
|
|
||||||
# Politeness: seconds to sleep before each actual media download (paces
|
|
||||||
# the CDN; honors ImportSettings.download_rate_limit_seconds, the same
|
|
||||||
# value gallery-dl used as its between-downloads `sleep`). 0 = no pacing.
|
|
||||||
# Applied only to real downloads, not to seen/disk skips. plan #703.
|
|
||||||
self._rate_limit = rate_limit or 0.0
|
|
||||||
# Build a cookie-loaded session the same way patreon_client does, so the
|
|
||||||
# CDN GETs carry the creator's auth.
|
|
||||||
self.session = session if session is not None else _load_session(cookies_path)
|
|
||||||
|
|
||||||
# -- public ------------------------------------------------------------
|
|
||||||
|
|
||||||
def download_post(
|
|
||||||
self,
|
|
||||||
post: dict,
|
|
||||||
media_items: list,
|
|
||||||
artist_slug: str,
|
|
||||||
*,
|
|
||||||
is_seen: Callable[[object], bool] = lambda m: False,
|
|
||||||
should_stop: Callable[[], bool] = lambda: False,
|
|
||||||
) -> list[MediaOutcome]:
|
|
||||||
"""Download every media item of one post; return per-item outcomes.
|
|
||||||
|
|
||||||
Builds the post directory, iterates media (1-based NN), applies the
|
|
||||||
two-tier skip (injected is_seen, then disk), downloads (plain GET or
|
|
||||||
yt-dlp for video), writes the sidecar for each freshly-downloaded item,
|
|
||||||
validates, and returns outcomes. Resilient: one media's failure yields
|
|
||||||
an "error" outcome for that item; the rest proceed.
|
|
||||||
|
|
||||||
`should_stop()` is polled BEFORE each media item: a media-dense post can
|
|
||||||
otherwise run a backfill chunk far past its time-box (the engine only
|
|
||||||
re-checks the budget between posts), so we honour the deadline mid-post
|
|
||||||
and return the items done so far — the rest re-fetch next chunk (they
|
|
||||||
were never marked seen). Bounds chunk overrun to one media download.
|
|
||||||
"""
|
|
||||||
post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
|
|
||||||
outcomes: list[MediaOutcome] = []
|
|
||||||
|
|
||||||
for i, media in enumerate(media_items, start=1):
|
|
||||||
if should_stop():
|
|
||||||
break
|
|
||||||
try:
|
|
||||||
outcomes.append(
|
|
||||||
self._download_one(post, media, post_dir, artist_slug, i, is_seen)
|
|
||||||
)
|
|
||||||
except Exception as exc: # resilient: isolate one item's failure
|
|
||||||
log.warning(
|
|
||||||
"Patreon media failed (post %s, item %d): %s",
|
|
||||||
post.get("id"), i, exc,
|
|
||||||
)
|
|
||||||
outcomes.append(
|
|
||||||
MediaOutcome(media=media, status="error", path=None, error=str(exc))
|
|
||||||
)
|
|
||||||
return outcomes
|
|
||||||
|
|
||||||
# -- per-item ----------------------------------------------------------
|
|
||||||
|
|
||||||
def _download_one(
|
|
||||||
self,
|
|
||||||
post: dict,
|
|
||||||
media,
|
|
||||||
post_dir: Path,
|
|
||||||
artist_slug: str,
|
|
||||||
index: int,
|
|
||||||
is_seen: Callable[[object], bool],
|
|
||||||
) -> MediaOutcome:
|
|
||||||
# tier-1: seen ledger (injected; no DB here).
|
|
||||||
if is_seen(media):
|
|
||||||
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
|
|
||||||
|
|
||||||
nn = f"{index:02d}"
|
|
||||||
final_name = _sanitize(f"{nn}_{media.filename}")
|
|
||||||
media_path = post_dir / final_name
|
|
||||||
|
|
||||||
# tier-2: already on disk.
|
|
||||||
if media_path.exists():
|
|
||||||
return MediaOutcome(
|
|
||||||
media=media, status="skipped_disk", path=media_path, error=None
|
|
||||||
)
|
|
||||||
# Video may land at a different extension; honor a pre-existing remux.
|
|
||||||
if _is_video_url(media.url):
|
|
||||||
existing = self._existing_video_output(media_path)
|
|
||||||
if existing is not None:
|
|
||||||
return MediaOutcome(
|
|
||||||
media=media, status="skipped_disk", path=existing, error=None
|
|
||||||
)
|
|
||||||
|
|
||||||
post_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
# Pace real downloads only (the skips above already returned). plan #703.
|
|
||||||
if self._rate_limit > 0:
|
|
||||||
time.sleep(self._rate_limit)
|
|
||||||
|
|
||||||
if _is_video_url(media.url):
|
|
||||||
out_path = self._run_ytdlp(media.url, media_path, _VIDEO_HEADERS)
|
|
||||||
if out_path is None or not Path(out_path).exists():
|
|
||||||
return MediaOutcome(
|
|
||||||
media=media,
|
|
||||||
status="error",
|
|
||||||
path=None,
|
|
||||||
error="yt-dlp produced no output",
|
|
||||||
)
|
|
||||||
out_path = Path(out_path)
|
|
||||||
else:
|
|
||||||
out_path = self._fetch_get(media.url, media_path)
|
|
||||||
reason, quarantine_dest = self._validate_path(
|
|
||||||
out_path, artist_slug, media.url
|
|
||||||
)
|
|
||||||
if reason is not None:
|
|
||||||
# Quarantined (corrupt/invalid) — distinct from a download error
|
|
||||||
# so the run can report a real files_quarantined count + paths.
|
|
||||||
return MediaOutcome(
|
|
||||||
media=media, status="quarantined",
|
|
||||||
path=quarantine_dest, error=reason,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._write_sidecar(post, out_path)
|
|
||||||
return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
|
|
||||||
|
|
||||||
# -- download seams ----------------------------------------------------
|
|
||||||
|
|
||||||
def _fetch_get(self, url: str, dest: Path) -> Path:
|
|
||||||
"""Stream `url` to a .part file then atomic-rename to `dest`.
|
|
||||||
|
|
||||||
Thin wrapper over `_fetch_to_file` so tests can stub either the whole
|
|
||||||
GET path (`_fetch_to_file`) or just `session.get`.
|
|
||||||
"""
|
|
||||||
part = dest.with_name(dest.name + ".part")
|
|
||||||
try:
|
|
||||||
self._fetch_to_file(url, part)
|
|
||||||
except Exception:
|
|
||||||
with contextlib.suppress(OSError):
|
|
||||||
part.unlink()
|
|
||||||
raise
|
|
||||||
os.replace(part, dest)
|
|
||||||
return dest
|
|
||||||
|
|
||||||
def _fetch_to_file(self, url: str, dest: Path) -> None:
|
|
||||||
"""Stream a non-video URL to `dest` via the (stubbable) session, retrying
|
|
||||||
TRANSIENT failures within the same pass (plan #705 #8) and RESUMING from
|
|
||||||
the bytes already on disk via a Range request when a retry follows a
|
|
||||||
mid-download cut (plan #708 B5).
|
|
||||||
|
|
||||||
Retried (backoff): transport blips (connection reset / timeout /
|
|
||||||
truncated stream — incl. mid-download), HTTP 429 (honoring Retry-After),
|
|
||||||
and 5xx. Failed fast (no retry → HTTPError → per-item error → dead-letter
|
|
||||||
path): 4xx other than 429 (404 gone, 403 forbidden) — re-fetching a
|
|
||||||
permanent failure is pointless.
|
|
||||||
|
|
||||||
Resume: on a retry, if bytes already landed in `dest`, ask for the rest
|
|
||||||
with `Range: bytes=<have>-`. A 206 means the server honored it → append; a
|
|
||||||
200 means it ignored it (served the whole file) → start clean. The caller
|
|
||||||
(_fetch_get) stages into a `.part`, so a non-range server never corrupts
|
|
||||||
the output — the worst case is re-downloading from zero, as before.
|
|
||||||
"""
|
|
||||||
attempt = 0
|
|
||||||
while True:
|
|
||||||
have = dest.stat().st_size if dest.exists() else 0
|
|
||||||
headers = {"Range": f"bytes={have}-"} if have > 0 else None
|
|
||||||
try:
|
|
||||||
resp = self.session.get(
|
|
||||||
url, stream=True, timeout=_TIMEOUT_SECONDS, headers=headers,
|
|
||||||
)
|
|
||||||
if (resp.status_code == 429 or resp.status_code >= 500) \
|
|
||||||
and attempt < _MAX_MEDIA_RETRIES:
|
|
||||||
attempt += 1
|
|
||||||
delay = _retry_after_seconds(resp, attempt)
|
|
||||||
log.warning(
|
|
||||||
"Patreon media transient HTTP %d (%s) — backing off "
|
|
||||||
"%.1fs (retry %d/%d)",
|
|
||||||
resp.status_code, url, delay, attempt, _MAX_MEDIA_RETRIES,
|
|
||||||
)
|
|
||||||
time.sleep(delay)
|
|
||||||
continue
|
|
||||||
# A Range that starts at/past EOF (we already have the whole file)
|
|
||||||
# comes back 416 — the bytes we kept ARE the file.
|
|
||||||
if have > 0 and resp.status_code == 416:
|
|
||||||
return
|
|
||||||
# 2xx → ok; 4xx-non-429 (or an exhausted 429/5xx) → HTTPError
|
|
||||||
# (permanent for this pass) → not caught below → per-item error.
|
|
||||||
resp.raise_for_status()
|
|
||||||
# 206 → server honored the Range; append after the kept bytes.
|
|
||||||
# Anything else (200) → it served the whole file → start clean.
|
|
||||||
mode = "ab" if (have > 0 and resp.status_code == 206) else "wb"
|
|
||||||
with open(dest, mode) as fh:
|
|
||||||
for chunk in resp.iter_content(chunk_size=_CHUNK):
|
|
||||||
if chunk:
|
|
||||||
fh.write(chunk)
|
|
||||||
return
|
|
||||||
except _TRANSIENT_TRANSPORT_EXC as exc:
|
|
||||||
if attempt >= _MAX_MEDIA_RETRIES:
|
|
||||||
raise # exhausted → terminal error outcome
|
|
||||||
attempt += 1
|
|
||||||
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
|
|
||||||
log.warning(
|
|
||||||
"Patreon media transport error (%s) — backing off %.1fs "
|
|
||||||
"(retry %d/%d): %s",
|
|
||||||
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
|
|
||||||
)
|
|
||||||
time.sleep(delay)
|
|
||||||
|
|
||||||
def _run_ytdlp(self, url: str, dest: Path, headers: dict) -> Path | None:
|
|
||||||
"""Invoke yt-dlp to fetch a Mux/HLS stream to (around) `dest`.
|
|
||||||
|
|
||||||
Returns the actual output path (yt-dlp may remux to a different
|
|
||||||
container, so we resolve the real file afterward). Overridden/
|
|
||||||
monkeypatched in tests to avoid spawning a real subprocess.
|
|
||||||
|
|
||||||
The output template uses `dest` without its extension; yt-dlp appends
|
|
||||||
the chosen container extension. We pass Referer/Origin (Mux JWT policy)
|
|
||||||
and the cookies file.
|
|
||||||
|
|
||||||
Mirrors the GET path's transient/permanent split (plan #705 #8): a hung
|
|
||||||
fetch (TimeoutExpired) or a spawn failure (OSError) is TRANSIENT — back
|
|
||||||
off and retry. A non-zero yt-dlp exit (CalledProcessError) is treated as
|
|
||||||
PERMANENT for this pass — yt-dlp already does its OWN internal network
|
|
||||||
retries, so a non-zero exit is effectively a real failure (private/gone/
|
|
||||||
geo-blocked), like a 4xx on the GET path: fail fast to the per-item error
|
|
||||||
→ dead-letter path.
|
|
||||||
"""
|
|
||||||
dest = Path(dest)
|
|
||||||
out_template = str(dest.with_suffix("")) + ".%(ext)s"
|
|
||||||
cmd = ["yt-dlp", "--no-progress", "-o", out_template]
|
|
||||||
for key, value in headers.items():
|
|
||||||
cmd += ["--add-header", f"{key}:{value}"]
|
|
||||||
if self.cookies_path and os.path.isfile(self.cookies_path):
|
|
||||||
cmd += ["--cookies", self.cookies_path]
|
|
||||||
cmd.append(url)
|
|
||||||
attempt = 0
|
|
||||||
while True:
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
cmd,
|
|
||||||
check=True,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=_TIMEOUT_SECONDS,
|
|
||||||
)
|
|
||||||
break
|
|
||||||
except subprocess.CalledProcessError as exc:
|
|
||||||
# Permanent for this pass — fail fast (no retry).
|
|
||||||
log.warning(
|
|
||||||
"yt-dlp failed (exit %s) for %s: %s",
|
|
||||||
exc.returncode, url, (exc.stderr or "").strip() or exc,
|
|
||||||
)
|
|
||||||
return None
|
|
||||||
except (OSError, subprocess.TimeoutExpired) as exc:
|
|
||||||
# Transient — back off and retry, like a transport blip on a GET.
|
|
||||||
if attempt >= _MAX_MEDIA_RETRIES:
|
|
||||||
log.warning(
|
|
||||||
"yt-dlp transient failure exhausted for %s: %s", url, exc
|
|
||||||
)
|
|
||||||
return None
|
|
||||||
attempt += 1
|
|
||||||
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
|
|
||||||
log.warning(
|
|
||||||
"yt-dlp transient failure (%s) — backing off %.1fs "
|
|
||||||
"(retry %d/%d): %s",
|
|
||||||
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
|
|
||||||
)
|
|
||||||
time.sleep(delay)
|
|
||||||
return self._existing_video_output(dest)
|
|
||||||
|
|
||||||
def _existing_video_output(self, dest: Path) -> Path | None:
|
|
||||||
"""Find a yt-dlp output for `dest` regardless of chosen extension.
|
|
||||||
|
|
||||||
Returns `dest` itself if present, else any sibling sharing the same
|
|
||||||
stem (the remuxed container). None if nothing matched.
|
|
||||||
"""
|
|
||||||
dest = Path(dest)
|
|
||||||
if dest.exists():
|
|
||||||
return dest
|
|
||||||
stem = dest.with_suffix("").name
|
|
||||||
parent = dest.parent
|
|
||||||
if not parent.is_dir():
|
|
||||||
return None
|
|
||||||
for cand in sorted(parent.iterdir()):
|
|
||||||
if cand.is_file() and cand.stem == stem and cand.name != dest.name:
|
|
||||||
return cand
|
|
||||||
return None
|
|
||||||
|
|
||||||
# -- validation --------------------------------------------------------
|
|
||||||
|
|
||||||
def _validate_path(
|
|
||||||
self, path: Path, artist_slug: str, source_url: str | None = None
|
|
||||||
) -> tuple[str | None, Path | None]:
|
|
||||||
"""Validate a freshly-written file; quarantine if bad.
|
|
||||||
|
|
||||||
Uses the shared `file_validator.quarantine_file` — same move + provenance
|
|
||||||
sidecar gallery-dl writes (the native path used to skip the sidecar; that
|
|
||||||
parity gap is closed here). Returns `(reason, quarantine_dest)` when
|
|
||||||
quarantined (dest is the original path if the move itself failed), else
|
|
||||||
`(None, None)` (ok / not validatable / disabled). plan #704: the dest is
|
|
||||||
surfaced so the run reports a real quarantined-paths list.
|
|
||||||
"""
|
|
||||||
if not self._validate or not is_validatable(path):
|
|
||||||
return None, None
|
|
||||||
try:
|
|
||||||
result = validate_file(path)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("Validator raised on %s: %s", path, exc)
|
|
||||||
return None, None
|
|
||||||
if result.ok:
|
|
||||||
return None, None
|
|
||||||
dest = quarantine_file(
|
|
||||||
self.images_root, path, artist_slug, "patreon",
|
|
||||||
url=source_url, result=result,
|
|
||||||
)
|
|
||||||
return (result.reason or "validation failed"), (dest or path)
|
|
||||||
|
|
||||||
# -- sidecar -----------------------------------------------------------
|
|
||||||
|
|
||||||
def _write_sidecar(self, post: dict, media_path: Path) -> Path:
|
|
||||||
"""Write the importer-consumed sidecar next to `media_path`.
|
|
||||||
|
|
||||||
Patreon uses base-default sidecar keys (parse_sidecar maps
|
|
||||||
category->platform, id->external_post_id, title->post_title,
|
|
||||||
content->description, published_at->post_date, url->post_url). Patreon
|
|
||||||
registers no derive_post_url, so `url` is trusted as the permalink — we
|
|
||||||
pass the post's attributes.url.
|
|
||||||
"""
|
|
||||||
attrs = post.get("attributes") or {}
|
|
||||||
title = attrs.get("title")
|
|
||||||
content = attrs.get("content")
|
|
||||||
published = attrs.get("published_at")
|
|
||||||
url = attrs.get("url")
|
|
||||||
data = {
|
|
||||||
"category": "patreon",
|
|
||||||
"id": str(post.get("id") or ""),
|
|
||||||
"title": title if isinstance(title, str) else "",
|
|
||||||
"content": content if isinstance(content, str) else "",
|
|
||||||
"published_at": published if isinstance(published, str) else None,
|
|
||||||
"url": url if isinstance(url, str) else None,
|
|
||||||
}
|
|
||||||
sidecar_path = media_path.with_suffix(".json")
|
|
||||||
sidecar_path.write_text(json.dumps(data, indent=2))
|
|
||||||
return sidecar_path
|
|
||||||
@@ -1,197 +0,0 @@
|
|||||||
"""Native Patreon ingester — the Patreon ADAPTER over the platform-agnostic core.
|
|
||||||
|
|
||||||
The orchestration that drives a native subscription walk (page a feed → extract
|
|
||||||
media → tiered skip → download → mark-seen / record-failures / checkpoint-cursor
|
|
||||||
→ return a gallery-dl-shaped `DownloadResult`, across tick/backfill/recovery)
|
|
||||||
now lives in `ingest_core.Ingester` — it's identical for every platform. This
|
|
||||||
module is the thin Patreon adapter: it wires the Patreon `client`/`downloader`/
|
|
||||||
ledger models/constraints/key into the core and supplies the Patreon-specific
|
|
||||||
failure mapping. `download_service.download_source` calls `PatreonIngester.run`
|
|
||||||
exactly as before; the public surface (this class, `_ledger_key`,
|
|
||||||
`DEAD_LETTER_THRESHOLD`, `verify_patreon_credential`) is unchanged.
|
|
||||||
|
|
||||||
Three modes (selected by `download_service` from `config_overrides` state):
|
|
||||||
- tick — newest→oldest, skip seen (tier-1 ledger + tier-2 disk), early-out
|
|
||||||
after N contiguous already-have-it items (the cheap native
|
|
||||||
equivalent of gallery-dl's `exit:20`, now free of per-file HEADs).
|
|
||||||
- backfill — full-history walk in a time-boxed chunk, resuming from the
|
|
||||||
pagination cursor checkpoint; reaches the bottom → "complete".
|
|
||||||
- recovery — like backfill but BYPASSES the tier-1 seen-ledger AND the
|
|
||||||
dead-letter ledger, so deliberately-dropped-and-deleted near-dups
|
|
||||||
get re-fetched and re-evaluated under the current pHash threshold
|
|
||||||
(tier-2 disk skip still spares files we kept).
|
|
||||||
|
|
||||||
The seen/dead-letter ledgers live in Postgres (`patreon_seen_media` /
|
|
||||||
`patreon_failed_media`); the core opens SHORT-LIVED sync sessions per page batch
|
|
||||||
— never held across a network fetch ([[db-connection-held-across-subprocess]]).
|
|
||||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import logging
|
|
||||||
from collections.abc import Callable
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from ..models import PatreonFailedMedia, PatreonSeenMedia
|
|
||||||
from .gallery_dl import DownloadResult, ErrorType
|
|
||||||
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
|
|
||||||
from .patreon_client import (
|
|
||||||
MediaItem,
|
|
||||||
PatreonAPIError,
|
|
||||||
PatreonAuthError,
|
|
||||||
PatreonClient,
|
|
||||||
PatreonDriftError,
|
|
||||||
)
|
|
||||||
from .patreon_downloader import PatreonDownloader
|
|
||||||
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"DEAD_LETTER_THRESHOLD",
|
|
||||||
"PatreonIngester",
|
|
||||||
"_ledger_key",
|
|
||||||
"verify_patreon_credential",
|
|
||||||
]
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Ledger keys are stored in patreon_seen_media.filehash VARCHAR(128); bound any
|
|
||||||
# synthesized key so a pathologically long file_name can't overflow the column.
|
|
||||||
_LEDGER_KEY_MAX = 128
|
|
||||||
|
|
||||||
|
|
||||||
def _ledger_key(media: MediaItem) -> str:
|
|
||||||
"""Stable per-media identity for the cross-run seen-ledger.
|
|
||||||
|
|
||||||
A Patreon CDN URL carries a 32-char MD5 (`media.filehash`) — that is the
|
|
||||||
natural key. Some media have none: Mux/HLS video (`stream.mux.com`, no
|
|
||||||
content hash at discovery) and the odd inline-content `<img>` pointing at a
|
|
||||||
hashless URL. The plan calls the video case the ``video:<post_id>:<media_id>``
|
|
||||||
sentinel; `MediaItem` carries no media_id, so the post-scoped filename is the
|
|
||||||
stable proxy. Bounded to the column width.
|
|
||||||
"""
|
|
||||||
if media.filehash:
|
|
||||||
return media.filehash
|
|
||||||
return f"{media.post_id}:{media.filename}"[:_LEDGER_KEY_MAX]
|
|
||||||
|
|
||||||
|
|
||||||
class PatreonIngester(Ingester):
|
|
||||||
"""Walk a Patreon campaign's posts, download unseen media, return a
|
|
||||||
`DownloadResult`. A thin adapter over `ingest_core.Ingester`.
|
|
||||||
|
|
||||||
Construct with the per-source `cookies_path` and a sync sessionmaker for the
|
|
||||||
ledgers. `client` / `downloader` are injectable seams so unit tests run
|
|
||||||
without network, subprocess, or a real CDN.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
images_root: Path,
|
|
||||||
cookies_path: str | None,
|
|
||||||
session_factory: Callable[[], object],
|
|
||||||
*,
|
|
||||||
validate: bool = True,
|
|
||||||
rate_limit: float = 0.0,
|
|
||||||
request_sleep: float = 0.0,
|
|
||||||
client: PatreonClient | None = None,
|
|
||||||
downloader: PatreonDownloader | None = None,
|
|
||||||
):
|
|
||||||
self.images_root = Path(images_root)
|
|
||||||
self.cookies_path = str(cookies_path) if cookies_path else None
|
|
||||||
# Pacing (plan #703): request_sleep paces the API page fetches,
|
|
||||||
# rate_limit paces the media downloads. Injected client/downloader (in
|
|
||||||
# tests) already carry their own pacing, so these only apply to the
|
|
||||||
# default-constructed ones.
|
|
||||||
resolved_client = (
|
|
||||||
client
|
|
||||||
if client is not None
|
|
||||||
else PatreonClient(cookies_path, request_sleep=request_sleep)
|
|
||||||
)
|
|
||||||
resolved_downloader = (
|
|
||||||
downloader
|
|
||||||
if downloader is not None
|
|
||||||
else PatreonDownloader(
|
|
||||||
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit
|
|
||||||
)
|
|
||||||
)
|
|
||||||
super().__init__(
|
|
||||||
client=resolved_client,
|
|
||||||
downloader=resolved_downloader,
|
|
||||||
session_factory=session_factory,
|
|
||||||
seen_model=PatreonSeenMedia,
|
|
||||||
failed_model=PatreonFailedMedia,
|
|
||||||
seen_constraint="uq_patreon_seen_media_source_id",
|
|
||||||
failed_constraint="uq_patreon_failed_media_source_id",
|
|
||||||
ledger_key=_ledger_key,
|
|
||||||
platform="patreon",
|
|
||||||
error_base=PatreonAPIError,
|
|
||||||
)
|
|
||||||
|
|
||||||
# -- failure mapping (Patreon exception taxonomy) ----------------------
|
|
||||||
|
|
||||||
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
|
|
||||||
"""Map a client-level exception to a loud, typed failed DownloadResult.
|
|
||||||
|
|
||||||
We NEVER return a silent zero-download "success" — the whole point of the
|
|
||||||
native ingester is to fail RED when Patreon's API shape or our auth
|
|
||||||
changes. The typed mapping lets FailingSourcesCard render the right chip
|
|
||||||
and tells the operator what to do:
|
|
||||||
- PatreonAuthError → AUTH_ERROR (rotate cookies)
|
|
||||||
- PatreonDriftError → API_DRIFT (ingester field-set/parser needs update)
|
|
||||||
- HTTP 429 / 404 → RATE_LIMITED / NOT_FOUND
|
|
||||||
- other HTTP status → HTTP_ERROR; transport failure → NETWORK_ERROR
|
|
||||||
|
|
||||||
PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
|
|
||||||
they must be matched before the generic HTTP/transport fallthrough.
|
|
||||||
"""
|
|
||||||
message = str(exc)
|
|
||||||
if isinstance(exc, PatreonAuthError):
|
|
||||||
error_type = ErrorType.AUTH_ERROR
|
|
||||||
elif isinstance(exc, PatreonDriftError):
|
|
||||||
error_type = ErrorType.API_DRIFT
|
|
||||||
message = f"Patreon API changed — ingester needs update: {message}"
|
|
||||||
else: # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
|
|
||||||
status = getattr(exc, "status_code", None)
|
|
||||||
if status == 429:
|
|
||||||
error_type = ErrorType.RATE_LIMITED
|
|
||||||
elif status == 404:
|
|
||||||
error_type = ErrorType.NOT_FOUND
|
|
||||||
elif status is not None:
|
|
||||||
error_type = ErrorType.HTTP_ERROR
|
|
||||||
else:
|
|
||||||
error_type = ErrorType.NETWORK_ERROR
|
|
||||||
log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
|
|
||||||
result = _result(
|
|
||||||
success=False, return_code=1,
|
|
||||||
error_type=error_type, error_message=message,
|
|
||||||
)
|
|
||||||
# plan #708 B1: carry the server's Retry-After up to the cooldown.
|
|
||||||
if error_type == ErrorType.RATE_LIMITED:
|
|
||||||
result.retry_after_seconds = getattr(exc, "retry_after", None)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
async def verify_patreon_credential(
|
|
||||||
url: str,
|
|
||||||
cookies_path: str | None,
|
|
||||||
overrides: dict | None,
|
|
||||||
) -> tuple[bool | None, str]:
|
|
||||||
"""Native Patreon credential probe — the verify counterpart to the ingester's
|
|
||||||
download path, sharing its campaign-id resolution. Resolves the campaign id
|
|
||||||
(override / id: URL / vanity) then does ONE authenticated `/api/posts` fetch
|
|
||||||
via PatreonClient.verify_auth. Returns the uniform `(ok, message)` contract
|
|
||||||
(True / False / None) so download_backends.verify_credential can treat it
|
|
||||||
interchangeably with the gallery-dl probe. No download, no DB.
|
|
||||||
"""
|
|
||||||
campaign_id, _ = await resolve_campaign_id_for_source(url, cookies_path, overrides)
|
|
||||||
if not campaign_id:
|
|
||||||
vanity = extract_vanity(url)
|
|
||||||
return None, (
|
|
||||||
f"Couldn't resolve the Patreon campaign id — can't verify. "
|
|
||||||
f"source_url={url!r}; vanity={vanity!r} "
|
|
||||||
"(cookies expired, or the creator moved/renamed?)."
|
|
||||||
)
|
|
||||||
client = PatreonClient(cookies_path)
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
return await loop.run_in_executor(None, client.verify_auth, campaign_id)
|
|
||||||
@@ -19,56 +19,18 @@ import asyncio
|
|||||||
import http.cookiejar
|
import http.cookiejar
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import re
|
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
_CAMPAIGNS_URL = "https://www.patreon.com/api/campaigns"
|
_CAMPAIGNS_URL = "https://www.patreon.com/api/campaigns"
|
||||||
_POSTS_API = "https://www.patreon.com/api/posts"
|
|
||||||
|
|
||||||
# A source URL of the form `.../id:<digits>` already carries the campaign id
|
|
||||||
# (no lookup needed). The vanity regex deliberately EXCLUDES the id: form so the
|
|
||||||
# two paths don't overlap.
|
|
||||||
#
|
|
||||||
# Patreon serves the creator vanity under several path prefixes — bare
|
|
||||||
# (`patreon.com/Atole`), `c/` (`patreon.com/c/Atole`), and `cw/`
|
|
||||||
# (`patreon.com/cw/Atole`, its current "creator workspace" URL). The optional
|
|
||||||
# prefix group must list `cw/` BEFORE `c/` so the longer prefix wins — otherwise
|
|
||||||
# `cw/Atole` matches the bare branch and yields vanity="cw" (operator-flagged
|
|
||||||
# 2026-06-07: every `/cw/` source failed resolution on vanity="cw").
|
|
||||||
_ID_URL_RE = re.compile(r"/id:(\d+)")
|
|
||||||
# A single-post permalink — patreon.com/posts/<slug>-<post_id> (or bare
|
|
||||||
# /posts/<post_id>). The trailing digits are the post id; the creator's
|
|
||||||
# campaign is resolved from the post itself (operator-flagged 2026-06-07: a
|
|
||||||
# /posts/ source resolved vanity="posts"). `posts/` is excluded from the vanity
|
|
||||||
# regex so it never masquerades as a creator slug.
|
|
||||||
_POST_URL_RE = re.compile(r"/posts/(?:[^/?#]*-)?(\d+)(?:[/?#]|$)")
|
|
||||||
_VANITY_PREFIXES = ("cw/", "c/")
|
|
||||||
_VANITY_RE = re.compile(
|
|
||||||
r"^https?://(?:www\.)?patreon\.com/(?:cw/|c/)?(?!(?:id:|posts/))([^/?#]+)",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
_USER_AGENT = (
|
_USER_AGENT = (
|
||||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||||||
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
|
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
|
||||||
)
|
)
|
||||||
_TIMEOUT_SECONDS = 10.0
|
_TIMEOUT_SECONDS = 10.0
|
||||||
|
|
||||||
# Fallback resolution: Patreon's `/api/campaigns?filter[vanity]=` lookup has
|
|
||||||
# proven unreliable (returns empty `data` for creators that clearly exist —
|
|
||||||
# operator-flagged 2026-06-06). gallery-dl never used that endpoint; it scrapes
|
|
||||||
# the campaign id out of the creator page's bootstrap JSON. We do the same as a
|
|
||||||
# fallback: fetch the creator page HTML and pull the first campaign id out of
|
|
||||||
# any of these embeddings (ordered most- to least-specific).
|
|
||||||
_PAGE_CAMPAIGN_ID_PATTERNS = (
|
|
||||||
re.compile(r'"id":\s*"(\d+)",\s*"type":\s*"campaign"'),
|
|
||||||
re.compile(r'"campaign":\s*\{\s*"data":\s*\{\s*"id":\s*"(\d+)"'),
|
|
||||||
re.compile(r"/api/campaigns/(\d+)"),
|
|
||||||
re.compile(r'"campaign_id":\s*"?(\d+)'),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJar | None:
|
def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJar | None:
|
||||||
if not cookies_path or not os.path.isfile(cookies_path):
|
if not cookies_path or not os.path.isfile(cookies_path):
|
||||||
@@ -83,15 +45,6 @@ def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJa
|
|||||||
|
|
||||||
|
|
||||||
def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
|
def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
|
||||||
"""Resolve a vanity to a campaign id: try the campaigns API first (cheap,
|
|
||||||
structured), then fall back to scraping the creator page (robust against the
|
|
||||||
API's empty-data failures). Returns None only when both miss."""
|
|
||||||
return _lookup_via_api(vanity, cookies_path) or _lookup_via_page(
|
|
||||||
vanity, cookies_path
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
|
||||||
jar = _load_cookie_jar(cookies_path)
|
jar = _load_cookie_jar(cookies_path)
|
||||||
headers = {
|
headers = {
|
||||||
"User-Agent": _USER_AGENT,
|
"User-Agent": _USER_AGENT,
|
||||||
@@ -139,87 +92,6 @@ def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
|
|||||||
return campaign_id
|
return campaign_id
|
||||||
|
|
||||||
|
|
||||||
def _scrape_campaign_id(html: str) -> str | None:
|
|
||||||
"""First campaign id found in creator-page HTML via the known embeddings."""
|
|
||||||
if not isinstance(html, str):
|
|
||||||
return None
|
|
||||||
for pat in _PAGE_CAMPAIGN_ID_PATTERNS:
|
|
||||||
m = pat.search(html)
|
|
||||||
if m:
|
|
||||||
return m.group(1)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _lookup_via_page(vanity: str, cookies_path: str | None) -> str | None:
|
|
||||||
"""Fallback: GET the creator page and scrape the campaign id from the page
|
|
||||||
bootstrap (gallery-dl's method). Tries both the bare and `/c/` vanity paths
|
|
||||||
Patreon redirects between. Never raises."""
|
|
||||||
jar = _load_cookie_jar(cookies_path)
|
|
||||||
headers = {"User-Agent": _USER_AGENT, "Accept": "text/html"}
|
|
||||||
# Try the bare vanity and every known creator-path prefix Patreon
|
|
||||||
# redirects between (c/, cw/) — the one the source used isn't known here
|
|
||||||
# (extract_vanity already stripped it).
|
|
||||||
page_urls = [f"https://www.patreon.com/{vanity}"]
|
|
||||||
page_urls += [f"https://www.patreon.com/{p}{vanity}" for p in _VANITY_PREFIXES]
|
|
||||||
for page_url in page_urls:
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
page_url,
|
|
||||||
headers=headers,
|
|
||||||
cookies=jar,
|
|
||||||
timeout=_TIMEOUT_SECONDS,
|
|
||||||
allow_redirects=True,
|
|
||||||
)
|
|
||||||
except requests.RequestException as exc:
|
|
||||||
log.warning("Patreon creator-page fetch failed for %s: %s", page_url, exc)
|
|
||||||
continue
|
|
||||||
if resp.status_code != 200:
|
|
||||||
continue
|
|
||||||
campaign_id = _scrape_campaign_id(resp.text)
|
|
||||||
if campaign_id:
|
|
||||||
log.info(
|
|
||||||
"Resolved Patreon vanity=%s → campaign_id=%s via creator page",
|
|
||||||
vanity, campaign_id,
|
|
||||||
)
|
|
||||||
return campaign_id
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _lookup_campaign_from_post(post_id: str, cookies_path: str | None) -> str | None:
|
|
||||||
"""Resolve the owning campaign id from a single post id via the Patreon
|
|
||||||
post API (`/api/posts/<id>?include=campaign`). A `/posts/` source URL points
|
|
||||||
at one post, but a subscription walks the whole creator — so we follow the
|
|
||||||
post to its campaign. Never raises."""
|
|
||||||
jar = _load_cookie_jar(cookies_path)
|
|
||||||
headers = {"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"}
|
|
||||||
params = {"include": "campaign", "fields[campaign]": "name"}
|
|
||||||
url = f"{_POSTS_API}/{post_id}"
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
url, params=params, headers=headers, cookies=jar,
|
|
||||||
timeout=_TIMEOUT_SECONDS,
|
|
||||||
)
|
|
||||||
except requests.RequestException as exc:
|
|
||||||
log.warning("Patreon post lookup failed for post=%s: %s", post_id, exc)
|
|
||||||
return None
|
|
||||||
if resp.status_code != 200:
|
|
||||||
log.warning("Patreon post API returned HTTP %d for post=%s", resp.status_code, post_id)
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
payload = resp.json()
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
data = payload.get("data") if isinstance(payload, dict) else None
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
return None
|
|
||||||
campaign = ((data.get("relationships") or {}).get("campaign") or {}).get("data") or {}
|
|
||||||
campaign_id = campaign.get("id")
|
|
||||||
if isinstance(campaign_id, str) and campaign_id:
|
|
||||||
log.info("Resolved Patreon post=%s → campaign_id=%s", post_id, campaign_id)
|
|
||||||
return campaign_id
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
async def resolve_campaign_id(
|
async def resolve_campaign_id(
|
||||||
vanity: str,
|
vanity: str,
|
||||||
cookies_path: str | None,
|
cookies_path: str | None,
|
||||||
@@ -228,55 +100,3 @@ async def resolve_campaign_id(
|
|||||||
Never raises."""
|
Never raises."""
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
return await loop.run_in_executor(None, _sync_lookup, vanity, cookies_path)
|
return await loop.run_in_executor(None, _sync_lookup, vanity, cookies_path)
|
||||||
|
|
||||||
|
|
||||||
async def resolve_campaign_from_post(
|
|
||||||
post_id: str, cookies_path: str | None,
|
|
||||||
) -> str | None:
|
|
||||||
"""Async wrapper around _lookup_campaign_from_post. Never raises."""
|
|
||||||
loop = asyncio.get_running_loop()
|
|
||||||
return await loop.run_in_executor(
|
|
||||||
None, _lookup_campaign_from_post, post_id, cookies_path
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def extract_vanity(url: str) -> str | None:
|
|
||||||
"""The vanity slug from a Patreon creator URL, or None for an `id:` URL."""
|
|
||||||
m = _VANITY_RE.match(url or "")
|
|
||||||
return m.group(1) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
async def resolve_campaign_id_for_source(
|
|
||||||
url: str,
|
|
||||||
cookies_path: str | None,
|
|
||||||
overrides: dict | None,
|
|
||||||
) -> tuple[str | None, str | None]:
|
|
||||||
"""Resolve a Patreon source to its campaign id — the single resolution path
|
|
||||||
shared by the download ingester and the credential-verify probe.
|
|
||||||
|
|
||||||
Order: cached `patreon_campaign_id` override → an `id:<digits>` URL → a
|
|
||||||
`/posts/<id>` permalink (resolve the owning campaign from the post) → a
|
|
||||||
vanity lookup against the campaigns API. Returns
|
|
||||||
`(campaign_id, newly_resolved_id)`: `newly_resolved_id` is non-None whenever
|
|
||||||
a lookup actually ran, so the caller caches it on the source (the
|
|
||||||
override/id: paths needed no lookup). `(None, None)` when unresolvable.
|
|
||||||
Never raises.
|
|
||||||
"""
|
|
||||||
overrides = overrides or {}
|
|
||||||
cached = overrides.get("patreon_campaign_id")
|
|
||||||
if cached:
|
|
||||||
return cached, None
|
|
||||||
id_match = _ID_URL_RE.search(url or "")
|
|
||||||
if id_match:
|
|
||||||
return id_match.group(1), None
|
|
||||||
# A single-post URL → follow the post to its creator's campaign so the
|
|
||||||
# source subscribes to the whole feed (a subscription isn't one post).
|
|
||||||
post_match = _POST_URL_RE.search(url or "")
|
|
||||||
if post_match:
|
|
||||||
resolved = await resolve_campaign_from_post(post_match.group(1), cookies_path)
|
|
||||||
return resolved, resolved
|
|
||||||
vanity = extract_vanity(url)
|
|
||||||
if vanity:
|
|
||||||
resolved = await resolve_campaign_id(vanity, cookies_path)
|
|
||||||
return resolved, resolved
|
|
||||||
return None, None
|
|
||||||
|
|||||||
@@ -1,59 +0,0 @@
|
|||||||
"""Per-platform download concurrency cap.
|
|
||||||
|
|
||||||
Some platforms (Patreon) are API-rate-sensitive enough that two *simultaneous*
|
|
||||||
walks can trip the server's rate limit even with each source pacing its own
|
|
||||||
requests. The platform-cooldown handles the AFTERMATH of a 429; this is the
|
|
||||||
preventive half — it serializes downloads PER PLATFORM to one at a time.
|
|
||||||
|
|
||||||
Different platforms still run concurrently up to the worker's concurrency; only
|
|
||||||
a second walk on the SAME serialized platform waits. The lock lives in Redis
|
|
||||||
(the Celery broker) with a TTL, so a SIGKILL'd worker can't wedge a platform —
|
|
||||||
the lock auto-expires shortly after the download hard time limit.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
|
|
||||||
import redis
|
|
||||||
|
|
||||||
from ..config import get_config
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
|
|
||||||
# here: each runs as a self-pacing subprocess and they're lower-volume. Add a
|
|
||||||
# platform here to cap it to a single concurrent walk.
|
|
||||||
SERIALIZED_PLATFORMS = frozenset({"patreon"})
|
|
||||||
|
|
||||||
_LOCK_PREFIX = "fc:download_lock:"
|
|
||||||
|
|
||||||
_client: redis.Redis | None = None
|
|
||||||
|
|
||||||
|
|
||||||
def _redis() -> redis.Redis:
|
|
||||||
# One client per worker process (Celery prefork forks before tasks run, so
|
|
||||||
# each process lazily builds its own). redis-py pools connections.
|
|
||||||
global _client
|
|
||||||
if _client is None:
|
|
||||||
_client = redis.from_url(get_config().celery_broker_url)
|
|
||||||
return _client
|
|
||||||
|
|
||||||
|
|
||||||
def platform_lock(platform: str, *, ttl_seconds: int):
|
|
||||||
"""A non-blocking Redis lock for `platform`, or None when the platform is
|
|
||||||
not serialized. Caller does `.acquire(blocking=False)` / `.release()`.
|
|
||||||
|
|
||||||
Returns None (rather than raising) on any Redis error so a broker hiccup
|
|
||||||
degrades to the prior behaviour (uncapped) instead of stalling downloads.
|
|
||||||
"""
|
|
||||||
if platform not in SERIALIZED_PLATFORMS:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return _redis().lock(
|
|
||||||
f"{_LOCK_PREFIX}{platform}",
|
|
||||||
timeout=ttl_seconds,
|
|
||||||
blocking=False,
|
|
||||||
)
|
|
||||||
except redis.RedisError as exc: # pragma: no cover - broker outage
|
|
||||||
log.warning("platform_lock unavailable for %s: %s", platform, exc)
|
|
||||||
return None
|
|
||||||
@@ -1,337 +0,0 @@
|
|||||||
"""FC-6.3 — assisted continuation matcher.
|
|
||||||
|
|
||||||
Scores a (post, candidate series) pair from several weighted signals; above the
|
|
||||||
configured threshold it records a *pending* SeriesSuggestion. Confirm-only — the
|
|
||||||
operator accepts (post becomes a chapter) or dismisses. No single signal gates;
|
|
||||||
the score is an additive weighted sum, each signal a 0..1 strength.
|
|
||||||
|
|
||||||
The learned "title pattern" isn't persisted — it's derived on the fly from the
|
|
||||||
post titles already in a series, so it sharpens automatically as more posts are
|
|
||||||
confirmed into the series.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
import time
|
|
||||||
|
|
||||||
from sqlalchemy import and_, func, select
|
|
||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
|
||||||
|
|
||||||
from ..models import ImageRecord, Post, Tag, TagKind
|
|
||||||
from ..models.series_chapter import SeriesChapter
|
|
||||||
from ..models.series_page import SeriesPage
|
|
||||||
from ..models.series_suggestion import SeriesSuggestion
|
|
||||||
from ..models.tag import image_tag
|
|
||||||
from .page_number_parser import parse_page_range
|
|
||||||
from .series_service import SeriesError, SeriesService
|
|
||||||
|
|
||||||
# Additive signal weights (sum to 1.0 → max score 1.0). Kept as constants;
|
|
||||||
# only the on/off + threshold are operator-tunable (sensitivity is the knob
|
|
||||||
# that matters; per-signal weights are an over-tune for v1).
|
|
||||||
WEIGHTS = {"title": 0.40, "artist": 0.20, "pages": 0.25, "tags": 0.15}
|
|
||||||
|
|
||||||
_DISTINCTIVE_KINDS = (TagKind.character, TagKind.series, TagKind.fandom)
|
|
||||||
_MAX_CANDIDATES = 50
|
|
||||||
|
|
||||||
# Strip installment markers so titles collapse to their stable stem.
|
|
||||||
_PAGE_TOKEN = re.compile(r"\b(?:pages?|pgs?|pp\.?)\s*\d+(?:\s*[-–—/]\s*\d+)?", re.I)
|
|
||||||
_BRACKET_NUM = re.compile(r"[\[(]\s*\d+\s*(?:/\s*\d+)?\s*[\])]")
|
|
||||||
_TRAILING_NUM = re.compile(r"[\s\-_#]*\d+\s*$")
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_title(title: str | None) -> str:
|
|
||||||
if not title:
|
|
||||||
return ""
|
|
||||||
t = _PAGE_TOKEN.sub("", title)
|
|
||||||
t = _BRACKET_NUM.sub("", t)
|
|
||||||
t = _TRAILING_NUM.sub("", t)
|
|
||||||
return " ".join(t.split()).strip().lower()
|
|
||||||
|
|
||||||
|
|
||||||
def _common_prefix(strings: list[str]) -> str:
|
|
||||||
strings = [s for s in strings if s]
|
|
||||||
if not strings:
|
|
||||||
return ""
|
|
||||||
pre = strings[0]
|
|
||||||
for s in strings[1:]:
|
|
||||||
i = 0
|
|
||||||
while i < len(pre) and i < len(s) and pre[i] == s[i]:
|
|
||||||
i += 1
|
|
||||||
pre = pre[:i]
|
|
||||||
if not pre:
|
|
||||||
break
|
|
||||||
return pre.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def title_signal(series_titles: list[str], post_title: str | None) -> float:
|
|
||||||
"""Overlap of the post title against the series' title stem (the longest
|
|
||||||
common prefix of its known titles, page/installment markers removed)."""
|
|
||||||
norm = [normalize_title(t) for t in series_titles]
|
|
||||||
norm = [t for t in norm if t]
|
|
||||||
pt = normalize_title(post_title)
|
|
||||||
if not norm or not pt:
|
|
||||||
return 0.0
|
|
||||||
stem = _common_prefix(norm)
|
|
||||||
if len(stem) < 3:
|
|
||||||
stem = max(norm, key=len)
|
|
||||||
i = 0
|
|
||||||
while i < len(stem) and i < len(pt) and stem[i] == pt[i]:
|
|
||||||
i += 1
|
|
||||||
return min(1.0, i / max(len(stem), 4))
|
|
||||||
|
|
||||||
|
|
||||||
def pages_signal(series_max_stated_end: int | None, post_start: int | None) -> float:
|
|
||||||
"""1.0 when the post's first page continues right after the series' last
|
|
||||||
stated page; partial for a near-continuation; 0 otherwise."""
|
|
||||||
if series_max_stated_end is None or post_start is None:
|
|
||||||
return 0.0
|
|
||||||
diff = post_start - series_max_stated_end
|
|
||||||
if diff == 1:
|
|
||||||
return 1.0
|
|
||||||
if 2 <= diff <= 3:
|
|
||||||
return 0.5
|
|
||||||
return 0.0
|
|
||||||
|
|
||||||
|
|
||||||
def tags_signal(shared_distinctive: int) -> float:
|
|
||||||
if shared_distinctive <= 0:
|
|
||||||
return 0.0
|
|
||||||
return min(1.0, shared_distinctive / 3.0)
|
|
||||||
|
|
||||||
|
|
||||||
def weighted_score(signals: dict) -> float:
|
|
||||||
return round(sum(WEIGHTS[k] * signals.get(k, 0.0) for k in WEIGHTS), 4)
|
|
||||||
|
|
||||||
|
|
||||||
class SeriesMatchService:
|
|
||||||
def __init__(self, session: AsyncSession):
|
|
||||||
self.session = session
|
|
||||||
|
|
||||||
async def _post_image_ids(self, post_id: int) -> list[int]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(ImageRecord.id).where(ImageRecord.primary_post_id == post_id)
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return list(rows)
|
|
||||||
|
|
||||||
async def _series_image_ids(self, series_tag_id: int) -> set[int]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesPage.image_id).where(
|
|
||||||
SeriesPage.series_tag_id == series_tag_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return set(rows)
|
|
||||||
|
|
||||||
async def _series_post_titles(self, series_tag_id: int) -> list[str]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(Post.post_title)
|
|
||||||
.select_from(SeriesPage)
|
|
||||||
.join(ImageRecord, ImageRecord.id == SeriesPage.image_id)
|
|
||||||
.join(Post, Post.id == ImageRecord.primary_post_id)
|
|
||||||
.where(
|
|
||||||
and_(
|
|
||||||
SeriesPage.series_tag_id == series_tag_id,
|
|
||||||
Post.post_title.isnot(None),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.distinct()
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return [t for t in rows if t]
|
|
||||||
|
|
||||||
async def _series_max_stated_end(self, series_tag_id: int) -> int | None:
|
|
||||||
return await self.session.scalar(
|
|
||||||
select(func.max(SeriesChapter.stated_page_end)).where(
|
|
||||||
SeriesChapter.series_tag_id == series_tag_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _distinctive_tags(self, image_ids: list[int] | set[int]) -> set[int]:
|
|
||||||
ids = list(image_ids)
|
|
||||||
if not ids:
|
|
||||||
return set()
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(image_tag.c.tag_id)
|
|
||||||
.join(Tag, Tag.id == image_tag.c.tag_id)
|
|
||||||
.where(
|
|
||||||
and_(
|
|
||||||
image_tag.c.image_record_id.in_(ids),
|
|
||||||
Tag.kind.in_(_DISTINCTIVE_KINDS),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.distinct()
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return set(rows)
|
|
||||||
|
|
||||||
async def _candidate_series(self, artist_id: int) -> list[int]:
|
|
||||||
"""Series that already contain a page by this artist — cross-artist
|
|
||||||
series are rare, so same-artist is the candidate bound."""
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesPage.series_tag_id)
|
|
||||||
.join(ImageRecord, ImageRecord.id == SeriesPage.image_id)
|
|
||||||
.where(ImageRecord.artist_id == artist_id)
|
|
||||||
.distinct()
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return list(rows)
|
|
||||||
|
|
||||||
async def _decided_series(self, post_id: int) -> set[int]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesSuggestion.series_tag_id).where(
|
|
||||||
and_(
|
|
||||||
SeriesSuggestion.post_id == post_id,
|
|
||||||
SeriesSuggestion.status.in_(["added", "dismissed"]),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return set(rows)
|
|
||||||
|
|
||||||
async def match_post(self, post_id: int, *, threshold: float) -> int:
|
|
||||||
"""Score a post against its artist's series; upsert pending suggestions
|
|
||||||
for those at/above threshold. Returns the number written."""
|
|
||||||
post = await self.session.get(Post, post_id)
|
|
||||||
if post is None or post.artist_id is None:
|
|
||||||
return 0
|
|
||||||
post_images = await self._post_image_ids(post_id)
|
|
||||||
if not post_images:
|
|
||||||
return 0
|
|
||||||
post_image_set = set(post_images)
|
|
||||||
rng = parse_page_range(f"{post.post_title or ''} {post.description or ''}")
|
|
||||||
post_start = rng[0] if rng else None
|
|
||||||
post_dtags = await self._distinctive_tags(post_images)
|
|
||||||
|
|
||||||
decided = await self._decided_series(post_id)
|
|
||||||
candidates = await self._candidate_series(post.artist_id)
|
|
||||||
made = 0
|
|
||||||
for sid in candidates[:_MAX_CANDIDATES]:
|
|
||||||
if sid in decided:
|
|
||||||
continue
|
|
||||||
s_images = await self._series_image_ids(sid)
|
|
||||||
if post_image_set & s_images:
|
|
||||||
continue # the post is already (partly) in this series
|
|
||||||
signals = {
|
|
||||||
"title": title_signal(
|
|
||||||
await self._series_post_titles(sid), post.post_title
|
|
||||||
),
|
|
||||||
"artist": 1.0, # candidates are same-artist by construction
|
|
||||||
"pages": pages_signal(
|
|
||||||
await self._series_max_stated_end(sid), post_start
|
|
||||||
),
|
|
||||||
"tags": tags_signal(
|
|
||||||
len(post_dtags & await self._distinctive_tags(s_images))
|
|
||||||
),
|
|
||||||
}
|
|
||||||
score = weighted_score(signals)
|
|
||||||
if score < threshold:
|
|
||||||
continue
|
|
||||||
await self.session.execute(
|
|
||||||
pg_insert(SeriesSuggestion)
|
|
||||||
.values(
|
|
||||||
post_id=post_id, series_tag_id=sid,
|
|
||||||
score=score, signals=signals, status="pending",
|
|
||||||
)
|
|
||||||
.on_conflict_do_update(
|
|
||||||
constraint="uq_series_suggestion_post_series",
|
|
||||||
set_={"score": score, "signals": signals, "status": "pending"},
|
|
||||||
where=SeriesSuggestion.status == "pending",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
made += 1
|
|
||||||
return made
|
|
||||||
|
|
||||||
# ---- queue ops --------------------------------------------------------
|
|
||||||
|
|
||||||
async def list_pending(self) -> list[dict]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(
|
|
||||||
SeriesSuggestion.id,
|
|
||||||
SeriesSuggestion.post_id,
|
|
||||||
SeriesSuggestion.series_tag_id,
|
|
||||||
SeriesSuggestion.score,
|
|
||||||
SeriesSuggestion.signals,
|
|
||||||
Post.post_title,
|
|
||||||
Post.external_post_id,
|
|
||||||
Tag.name.label("series_name"),
|
|
||||||
)
|
|
||||||
.join(Post, Post.id == SeriesSuggestion.post_id)
|
|
||||||
.join(Tag, Tag.id == SeriesSuggestion.series_tag_id)
|
|
||||||
.where(SeriesSuggestion.status == "pending")
|
|
||||||
.order_by(SeriesSuggestion.score.desc())
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
return [
|
|
||||||
{
|
|
||||||
"id": r.id,
|
|
||||||
"post_id": r.post_id,
|
|
||||||
"series_tag_id": r.series_tag_id,
|
|
||||||
"series_name": r.series_name,
|
|
||||||
"post_title": r.post_title or f"Post {r.external_post_id}",
|
|
||||||
"score": r.score,
|
|
||||||
"signals": r.signals or {},
|
|
||||||
}
|
|
||||||
for r in rows
|
|
||||||
]
|
|
||||||
|
|
||||||
async def accept(self, suggestion_id: int) -> dict:
|
|
||||||
s = await self.session.get(SeriesSuggestion, suggestion_id)
|
|
||||||
if s is None:
|
|
||||||
raise SeriesError(f"suggestion {suggestion_id} not found")
|
|
||||||
if s.status != "pending":
|
|
||||||
raise SeriesError(f"suggestion {suggestion_id} is already {s.status}")
|
|
||||||
out = await SeriesService(self.session).add_post_as_chapter(
|
|
||||||
s.series_tag_id, s.post_id
|
|
||||||
)
|
|
||||||
s.status = "added"
|
|
||||||
self.session.add(s)
|
|
||||||
return out
|
|
||||||
|
|
||||||
async def dismiss(self, suggestion_id: int) -> None:
|
|
||||||
s = await self.session.get(SeriesSuggestion, suggestion_id)
|
|
||||||
if s is None:
|
|
||||||
raise SeriesError(f"suggestion {suggestion_id} not found")
|
|
||||||
if s.status == "pending":
|
|
||||||
s.status = "dismissed"
|
|
||||||
self.session.add(s)
|
|
||||||
|
|
||||||
async def rescan(
|
|
||||||
self, *, threshold: float, time_budget_seconds: float | None = None,
|
|
||||||
after_post_id: int = 0,
|
|
||||||
) -> dict:
|
|
||||||
"""Score every post (id > after_post_id) against its artist's series,
|
|
||||||
time-boxed + resumable like the other long maintenance sweeps."""
|
|
||||||
post_ids = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(Post.id)
|
|
||||||
.where(Post.id > after_post_id)
|
|
||||||
.order_by(Post.id.asc())
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
summary = {
|
|
||||||
"scanned": 0, "suggested": 0,
|
|
||||||
"partial": False, "resume_after_id": after_post_id,
|
|
||||||
}
|
|
||||||
start = time.monotonic()
|
|
||||||
for pid in post_ids:
|
|
||||||
summary["scanned"] += 1
|
|
||||||
summary["resume_after_id"] = pid
|
|
||||||
summary["suggested"] += await self.match_post(pid, threshold=threshold)
|
|
||||||
await self.session.commit() # commit per post so progress survives
|
|
||||||
if (
|
|
||||||
time_budget_seconds is not None
|
|
||||||
and time.monotonic() - start >= time_budget_seconds
|
|
||||||
):
|
|
||||||
summary["partial"] = True
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
summary["partial"] = False
|
|
||||||
return summary
|
|
||||||
@@ -1,8 +1,4 @@
|
|||||||
"""Series = a series-kind Tag + ordered chapters, each holding ordered pages.
|
"""Series = a series-kind Tag + ordered series_page membership.
|
||||||
|
|
||||||
Reading order is (series_chapter.chapter_number, series_page.page_number). A
|
|
||||||
chapter may be a placeholder (no pages) reserving a slot. An image lives in at
|
|
||||||
most one series ⇒ one chapter (series_page.image_id is UNIQUE).
|
|
||||||
|
|
||||||
All mutations are Core/set-based and run in the request transaction.
|
All mutations are Core/set-based and run in the request transaction.
|
||||||
"""
|
"""
|
||||||
@@ -11,11 +7,9 @@ from sqlalchemy import and_, func, select, update
|
|||||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from ..models import Artist, ImageRecord, Post, Tag, TagKind
|
from ..models import ImageRecord, Tag, TagKind
|
||||||
from ..models.series_chapter import SeriesChapter
|
|
||||||
from ..models.series_page import SeriesPage
|
from ..models.series_page import SeriesPage
|
||||||
from .gallery_service import thumbnail_url
|
from .gallery_service import thumbnail_url
|
||||||
from .page_number_parser import parse_page_range
|
|
||||||
|
|
||||||
|
|
||||||
class SeriesError(ValueError):
|
class SeriesError(ValueError):
|
||||||
@@ -27,8 +21,6 @@ class SeriesService:
|
|||||||
def __init__(self, session: AsyncSession):
|
def __init__(self, session: AsyncSession):
|
||||||
self.session = session
|
self.session = session
|
||||||
|
|
||||||
# ---- guards / helpers -------------------------------------------------
|
|
||||||
|
|
||||||
async def _require_series(self, series_tag_id: int) -> Tag:
|
async def _require_series(self, series_tag_id: int) -> Tag:
|
||||||
tag = await self.session.get(Tag, series_tag_id)
|
tag = await self.session.get(Tag, series_tag_id)
|
||||||
if tag is None:
|
if tag is None:
|
||||||
@@ -40,7 +32,7 @@ class SeriesService:
|
|||||||
@staticmethod
|
@staticmethod
|
||||||
def _clean_ids(ids: list[int]) -> list[int]:
|
def _clean_ids(ids: list[int]) -> list[int]:
|
||||||
if not ids:
|
if not ids:
|
||||||
raise SeriesError("ids must be a non-empty list")
|
raise SeriesError("image_ids must be a non-empty list")
|
||||||
if len(ids) > 500:
|
if len(ids) > 500:
|
||||||
raise SeriesError("selection too large (max 500)")
|
raise SeriesError("selection too large (max 500)")
|
||||||
seen: set[int] = set()
|
seen: set[int] = set()
|
||||||
@@ -51,172 +43,51 @@ class SeriesService:
|
|||||||
out.append(int(x))
|
out.append(int(x))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
async def _require_chapter(self, series_tag_id: int, chapter_id: int):
|
async def _member_order(self, series_tag_id: int) -> list[int]:
|
||||||
row = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesChapter).where(
|
|
||||||
and_(
|
|
||||||
SeriesChapter.id == chapter_id,
|
|
||||||
SeriesChapter.series_tag_id == series_tag_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
if row is None:
|
|
||||||
raise SeriesError(
|
|
||||||
f"chapter {chapter_id} is not in series {series_tag_id}"
|
|
||||||
)
|
|
||||||
return row
|
|
||||||
|
|
||||||
async def _ensure_default_chapter(self, series_tag_id: int) -> int:
|
|
||||||
"""Lowest-numbered chapter of the series, creating a first chapter if
|
|
||||||
the series has none (a fresh series, or the legacy single-chapter path)."""
|
|
||||||
existing = await self.session.scalar(
|
|
||||||
select(SeriesChapter.id)
|
|
||||||
.where(SeriesChapter.series_tag_id == series_tag_id)
|
|
||||||
.order_by(SeriesChapter.chapter_number.asc())
|
|
||||||
.limit(1)
|
|
||||||
)
|
|
||||||
if existing is not None:
|
|
||||||
return existing
|
|
||||||
res = await self.session.execute(
|
|
||||||
pg_insert(SeriesChapter)
|
|
||||||
.values(series_tag_id=series_tag_id, chapter_number=1)
|
|
||||||
.returning(SeriesChapter.id)
|
|
||||||
)
|
|
||||||
return res.scalar_one()
|
|
||||||
|
|
||||||
async def _chapter_page_order(self, chapter_id: int) -> list[int]:
|
|
||||||
rows = (
|
rows = (
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
select(SeriesPage.image_id)
|
select(SeriesPage.image_id)
|
||||||
.where(SeriesPage.chapter_id == chapter_id)
|
.where(SeriesPage.series_tag_id == series_tag_id)
|
||||||
.order_by(SeriesPage.page_number.asc())
|
.order_by(SeriesPage.page_number.asc())
|
||||||
)
|
)
|
||||||
).scalars().all()
|
).scalars().all()
|
||||||
return list(rows)
|
return list(rows)
|
||||||
|
|
||||||
async def _chapter_order(self, series_tag_id: int) -> list[int]:
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesChapter.id)
|
|
||||||
.where(SeriesChapter.series_tag_id == series_tag_id)
|
|
||||||
.order_by(SeriesChapter.chapter_number.asc())
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return list(rows)
|
|
||||||
|
|
||||||
# ---- read -------------------------------------------------------------
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _gaps(chapters: list[dict]) -> list[dict]:
|
|
||||||
"""Missing-page gaps between consecutive chapters with stated ranges."""
|
|
||||||
out: list[dict] = []
|
|
||||||
prev = None
|
|
||||||
for ch in chapters:
|
|
||||||
start = ch["stated_page_start"]
|
|
||||||
if (
|
|
||||||
prev is not None
|
|
||||||
and prev["stated_page_end"] is not None
|
|
||||||
and start is not None
|
|
||||||
and start > prev["stated_page_end"] + 1
|
|
||||||
):
|
|
||||||
out.append(
|
|
||||||
{
|
|
||||||
"after_chapter_id": prev["id"],
|
|
||||||
"start": prev["stated_page_end"] + 1,
|
|
||||||
"end": start - 1,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
prev = ch
|
|
||||||
return out
|
|
||||||
|
|
||||||
async def list_pages(self, series_tag_id: int) -> dict:
|
async def list_pages(self, series_tag_id: int) -> dict:
|
||||||
tag = await self._require_series(series_tag_id)
|
tag = await self._require_series(series_tag_id)
|
||||||
rows = (
|
rows = (
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
select(
|
select(
|
||||||
SeriesChapter.id.label("chapter_id"),
|
|
||||||
SeriesChapter.chapter_number,
|
|
||||||
SeriesChapter.title,
|
|
||||||
SeriesChapter.is_placeholder,
|
|
||||||
SeriesChapter.stated_page_start,
|
|
||||||
SeriesChapter.stated_page_end,
|
|
||||||
SeriesPage.image_id,
|
SeriesPage.image_id,
|
||||||
SeriesPage.page_number,
|
SeriesPage.page_number,
|
||||||
SeriesPage.stated_page,
|
|
||||||
ImageRecord.sha256,
|
ImageRecord.sha256,
|
||||||
ImageRecord.mime,
|
ImageRecord.mime,
|
||||||
ImageRecord.path,
|
ImageRecord.path,
|
||||||
ImageRecord.thumbnail_path,
|
ImageRecord.thumbnail_path,
|
||||||
)
|
)
|
||||||
.select_from(SeriesChapter)
|
.join(ImageRecord, ImageRecord.id == SeriesPage.image_id)
|
||||||
.outerjoin(SeriesPage, SeriesPage.chapter_id == SeriesChapter.id)
|
.where(SeriesPage.series_tag_id == series_tag_id)
|
||||||
.outerjoin(ImageRecord, ImageRecord.id == SeriesPage.image_id)
|
.order_by(SeriesPage.page_number.asc())
|
||||||
.where(SeriesChapter.series_tag_id == series_tag_id)
|
|
||||||
.order_by(
|
|
||||||
SeriesChapter.chapter_number.asc(),
|
|
||||||
SeriesPage.page_number.asc(),
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
).all()
|
).all()
|
||||||
|
return {
|
||||||
chapters: list[dict] = []
|
"series": {"id": tag.id, "name": tag.name},
|
||||||
flat: list[dict] = []
|
"pages": [
|
||||||
by_id: dict[int, dict] = {}
|
{
|
||||||
for r in rows:
|
|
||||||
ch = by_id.get(r.chapter_id)
|
|
||||||
if ch is None:
|
|
||||||
ch = {
|
|
||||||
"id": r.chapter_id,
|
|
||||||
"chapter_number": r.chapter_number,
|
|
||||||
"title": r.title,
|
|
||||||
"is_placeholder": r.is_placeholder,
|
|
||||||
"stated_page_start": r.stated_page_start,
|
|
||||||
"stated_page_end": r.stated_page_end,
|
|
||||||
"pages": [],
|
|
||||||
}
|
|
||||||
by_id[r.chapter_id] = ch
|
|
||||||
chapters.append(ch)
|
|
||||||
if r.image_id is None:
|
|
||||||
continue # placeholder / empty chapter
|
|
||||||
page = {
|
|
||||||
"image_id": r.image_id,
|
"image_id": r.image_id,
|
||||||
"chapter_id": r.chapter_id,
|
|
||||||
"page_number": r.page_number,
|
"page_number": r.page_number,
|
||||||
"stated_page": r.stated_page,
|
|
||||||
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
|
||||||
"image_url": f"/images/{r.path.split('/images/', 1)[-1]}",
|
"image_url": f"/images/{r.path.split('/images/', 1)[-1]}",
|
||||||
}
|
}
|
||||||
ch["pages"].append(page)
|
for r in rows
|
||||||
flat.append(page)
|
],
|
||||||
|
|
||||||
return {
|
|
||||||
"series": {"id": tag.id, "name": tag.name},
|
|
||||||
"chapters": chapters,
|
|
||||||
"pages": flat, # back-compat: flat reading order across chapters
|
|
||||||
"gaps": self._gaps(chapters),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---- pages ------------------------------------------------------------
|
|
||||||
|
|
||||||
async def add_images(
|
async def add_images(
|
||||||
self,
|
self, series_tag_id: int, image_ids: list[int]
|
||||||
series_tag_id: int,
|
|
||||||
image_ids: list[int],
|
|
||||||
chapter_id: int | None = None,
|
|
||||||
stated_pages: dict[int, int] | None = None,
|
|
||||||
) -> int:
|
) -> int:
|
||||||
"""Append images as pages of a chapter (the series' default chapter when
|
|
||||||
chapter_id is None). Images already in THIS series are left untouched;
|
|
||||||
images in another series are moved here (image_id is UNIQUE)."""
|
|
||||||
await self._require_series(series_tag_id)
|
await self._require_series(series_tag_id)
|
||||||
ids = self._clean_ids(image_ids)
|
ids = self._clean_ids(image_ids)
|
||||||
if chapter_id is None:
|
|
||||||
chapter_id = await self._ensure_default_chapter(series_tag_id)
|
|
||||||
else:
|
|
||||||
await self._require_chapter(series_tag_id, chapter_id)
|
|
||||||
|
|
||||||
existing = dict(
|
existing = dict(
|
||||||
(
|
(
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
@@ -225,30 +96,29 @@ class SeriesService:
|
|||||||
)
|
)
|
||||||
).all()
|
).all()
|
||||||
)
|
)
|
||||||
|
# Already in THIS series → leave untouched (no churn).
|
||||||
to_add = [i for i in ids if existing.get(i) != series_tag_id]
|
to_add = [i for i in ids if existing.get(i) != series_tag_id]
|
||||||
if not to_add:
|
if not to_add:
|
||||||
return 0
|
return 0
|
||||||
# Move: drop any prior membership for these images (UNIQUE image_id).
|
# Remove any prior membership for the to_add images (the "move").
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
SeriesPage.__table__.delete().where(SeriesPage.image_id.in_(to_add))
|
SeriesPage.__table__.delete().where(
|
||||||
|
SeriesPage.image_id.in_(to_add)
|
||||||
|
)
|
||||||
)
|
)
|
||||||
max_pn = (
|
max_pn = (
|
||||||
await self.session.scalar(
|
await self.session.scalar(
|
||||||
select(func.coalesce(func.max(SeriesPage.page_number), 0)).where(
|
select(func.coalesce(func.max(SeriesPage.page_number), 0))
|
||||||
SeriesPage.chapter_id == chapter_id
|
.where(SeriesPage.series_tag_id == series_tag_id)
|
||||||
)
|
|
||||||
)
|
)
|
||||||
) or 0
|
) or 0
|
||||||
sp = stated_pages or {}
|
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
pg_insert(SeriesPage).values(
|
pg_insert(SeriesPage).values(
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"series_tag_id": series_tag_id,
|
"series_tag_id": series_tag_id,
|
||||||
"chapter_id": chapter_id,
|
|
||||||
"image_id": iid,
|
"image_id": iid,
|
||||||
"page_number": max_pn + offset,
|
"page_number": max_pn + offset,
|
||||||
"stated_page": sp.get(iid),
|
|
||||||
}
|
}
|
||||||
for offset, iid in enumerate(to_add, start=1)
|
for offset, iid in enumerate(to_add, start=1)
|
||||||
]
|
]
|
||||||
@@ -271,465 +141,32 @@ class SeriesService:
|
|||||||
)
|
)
|
||||||
return res.rowcount or 0
|
return res.rowcount or 0
|
||||||
|
|
||||||
async def reorder_pages(
|
async def reorder(
|
||||||
self, series_tag_id: int, chapter_id: int, ordered_image_ids: list[int]
|
self, series_tag_id: int, ordered_image_ids: list[int]
|
||||||
) -> None:
|
) -> None:
|
||||||
await self._require_series(series_tag_id)
|
await self._require_series(series_tag_id)
|
||||||
await self._require_chapter(series_tag_id, chapter_id)
|
|
||||||
ordered = self._clean_ids(ordered_image_ids)
|
ordered = self._clean_ids(ordered_image_ids)
|
||||||
current = set(await self._chapter_page_order(chapter_id))
|
current = set(await self._member_order(series_tag_id))
|
||||||
if set(ordered) != current or len(ordered) != len(current):
|
if set(ordered) != current or len(ordered) != len(current):
|
||||||
raise SeriesError(
|
raise SeriesError(
|
||||||
"ordered image_ids must exactly match the chapter's pages"
|
"ordered image_ids must exactly match series membership"
|
||||||
)
|
)
|
||||||
for idx, iid in enumerate(ordered, start=1):
|
for idx, iid in enumerate(ordered, start=1):
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
update(SeriesPage)
|
update(SeriesPage)
|
||||||
.where(
|
.where(
|
||||||
and_(
|
and_(
|
||||||
SeriesPage.chapter_id == chapter_id,
|
SeriesPage.series_tag_id == series_tag_id,
|
||||||
SeriesPage.image_id == iid,
|
SeriesPage.image_id == iid,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
.values(page_number=idx)
|
.values(page_number=idx)
|
||||||
)
|
)
|
||||||
|
|
||||||
async def reorder(
|
|
||||||
self, series_tag_id: int, ordered_image_ids: list[int]
|
|
||||||
) -> None:
|
|
||||||
"""Legacy series-wide reorder — reorders the default chapter. Valid only
|
|
||||||
for single-chapter series (the only shape the pre-chapter UI produced);
|
|
||||||
multi-chapter series must use reorder_pages with an explicit chapter."""
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
chapters = await self._chapter_order(series_tag_id)
|
|
||||||
if len(chapters) > 1:
|
|
||||||
raise SeriesError(
|
|
||||||
"series has multiple chapters; reorder a specific chapter"
|
|
||||||
)
|
|
||||||
chapter_id = (
|
|
||||||
chapters[0]
|
|
||||||
if chapters
|
|
||||||
else await self._ensure_default_chapter(series_tag_id)
|
|
||||||
)
|
|
||||||
await self.reorder_pages(series_tag_id, chapter_id, ordered_image_ids)
|
|
||||||
|
|
||||||
# ---- chapters ---------------------------------------------------------
|
|
||||||
|
|
||||||
async def create_chapter(
|
|
||||||
self,
|
|
||||||
series_tag_id: int,
|
|
||||||
*,
|
|
||||||
title: str | None = None,
|
|
||||||
is_placeholder: bool = False,
|
|
||||||
stated_page_start: int | None = None,
|
|
||||||
stated_page_end: int | None = None,
|
|
||||||
) -> dict:
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
max_cn = (
|
|
||||||
await self.session.scalar(
|
|
||||||
select(
|
|
||||||
func.coalesce(func.max(SeriesChapter.chapter_number), 0)
|
|
||||||
).where(SeriesChapter.series_tag_id == series_tag_id)
|
|
||||||
)
|
|
||||||
) or 0
|
|
||||||
res = await self.session.execute(
|
|
||||||
pg_insert(SeriesChapter)
|
|
||||||
.values(
|
|
||||||
series_tag_id=series_tag_id,
|
|
||||||
chapter_number=max_cn + 1,
|
|
||||||
title=title,
|
|
||||||
is_placeholder=is_placeholder,
|
|
||||||
stated_page_start=stated_page_start,
|
|
||||||
stated_page_end=stated_page_end,
|
|
||||||
)
|
|
||||||
.returning(SeriesChapter.id, SeriesChapter.chapter_number)
|
|
||||||
)
|
|
||||||
row = res.one()
|
|
||||||
return {"id": row.id, "chapter_number": row.chapter_number}
|
|
||||||
|
|
||||||
async def update_chapter(
|
|
||||||
self,
|
|
||||||
series_tag_id: int,
|
|
||||||
chapter_id: int,
|
|
||||||
*,
|
|
||||||
title: str | None = None,
|
|
||||||
stated_page_start: int | None = None,
|
|
||||||
stated_page_end: int | None = None,
|
|
||||||
set_title: bool = False,
|
|
||||||
set_start: bool = False,
|
|
||||||
set_end: bool = False,
|
|
||||||
) -> None:
|
|
||||||
"""Partial chapter edit. The set_* flags say which fields to write (so
|
|
||||||
None can be written explicitly, e.g. clearing a stated page)."""
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
await self._require_chapter(series_tag_id, chapter_id)
|
|
||||||
values: dict = {}
|
|
||||||
if set_title:
|
|
||||||
values["title"] = title
|
|
||||||
if set_start:
|
|
||||||
values["stated_page_start"] = stated_page_start
|
|
||||||
if set_end:
|
|
||||||
values["stated_page_end"] = stated_page_end
|
|
||||||
if not values:
|
|
||||||
return
|
|
||||||
await self.session.execute(
|
|
||||||
update(SeriesChapter)
|
|
||||||
.where(SeriesChapter.id == chapter_id)
|
|
||||||
.values(**values)
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _renumber_chapters(self, series_tag_id: int) -> None:
|
|
||||||
for idx, cid in enumerate(
|
|
||||||
await self._chapter_order(series_tag_id), start=1
|
|
||||||
):
|
|
||||||
await self.session.execute(
|
|
||||||
update(SeriesChapter)
|
|
||||||
.where(SeriesChapter.id == cid)
|
|
||||||
.values(chapter_number=idx)
|
|
||||||
)
|
|
||||||
|
|
||||||
async def reorder_chapters(
|
|
||||||
self, series_tag_id: int, ordered_chapter_ids: list[int]
|
|
||||||
) -> None:
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
ordered = self._clean_ids(ordered_chapter_ids)
|
|
||||||
current = set(await self._chapter_order(series_tag_id))
|
|
||||||
if set(ordered) != current or len(ordered) != len(current):
|
|
||||||
raise SeriesError(
|
|
||||||
"ordered chapter_ids must exactly match the series' chapters"
|
|
||||||
)
|
|
||||||
for idx, cid in enumerate(ordered, start=1):
|
|
||||||
await self.session.execute(
|
|
||||||
update(SeriesChapter)
|
|
||||||
.where(SeriesChapter.id == cid)
|
|
||||||
.values(chapter_number=idx)
|
|
||||||
)
|
|
||||||
|
|
||||||
async def delete_chapter(self, series_tag_id: int, chapter_id: int) -> None:
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
await self._require_chapter(series_tag_id, chapter_id)
|
|
||||||
# Pages cascade-delete with the chapter (FK ondelete=CASCADE).
|
|
||||||
await self.session.execute(
|
|
||||||
SeriesChapter.__table__.delete().where(
|
|
||||||
SeriesChapter.id == chapter_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await self._renumber_chapters(series_tag_id)
|
|
||||||
|
|
||||||
async def merge_chapter(
|
|
||||||
self, series_tag_id: int, source_chapter_id: int, target_chapter_id: int
|
|
||||||
) -> int:
|
|
||||||
"""Move source chapter's pages onto the end of the target chapter, then
|
|
||||||
delete the now-empty source chapter. Returns the number of pages moved."""
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
if source_chapter_id == target_chapter_id:
|
|
||||||
raise SeriesError("cannot merge a chapter into itself")
|
|
||||||
await self._require_chapter(series_tag_id, source_chapter_id)
|
|
||||||
await self._require_chapter(series_tag_id, target_chapter_id)
|
|
||||||
moving = await self._chapter_page_order(source_chapter_id)
|
|
||||||
max_pn = (
|
|
||||||
await self.session.scalar(
|
|
||||||
select(func.coalesce(func.max(SeriesPage.page_number), 0)).where(
|
|
||||||
SeriesPage.chapter_id == target_chapter_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
) or 0
|
|
||||||
for offset, iid in enumerate(moving, start=1):
|
|
||||||
await self.session.execute(
|
|
||||||
update(SeriesPage)
|
|
||||||
.where(
|
|
||||||
and_(
|
|
||||||
SeriesPage.chapter_id == source_chapter_id,
|
|
||||||
SeriesPage.image_id == iid,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.values(chapter_id=target_chapter_id, page_number=max_pn + offset)
|
|
||||||
)
|
|
||||||
await self.session.execute(
|
|
||||||
SeriesChapter.__table__.delete().where(
|
|
||||||
SeriesChapter.id == source_chapter_id
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await self._renumber_chapters(series_tag_id)
|
|
||||||
return len(moving)
|
|
||||||
|
|
||||||
async def set_cover(self, series_tag_id: int, image_id: int) -> None:
|
async def set_cover(self, series_tag_id: int, image_id: int) -> None:
|
||||||
"""Cover = the (chapter_number=1, page_number=1) image. Bring the image's
|
|
||||||
chapter to the front and the image to the front of its chapter."""
|
|
||||||
await self._require_series(series_tag_id)
|
await self._require_series(series_tag_id)
|
||||||
row = (
|
order = await self._member_order(series_tag_id)
|
||||||
await self.session.execute(
|
if image_id not in order:
|
||||||
select(SeriesPage.chapter_id)
|
|
||||||
.where(
|
|
||||||
and_(
|
|
||||||
SeriesPage.series_tag_id == series_tag_id,
|
|
||||||
SeriesPage.image_id == image_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
if row is None:
|
|
||||||
raise SeriesError(f"image {image_id} is not in this series")
|
raise SeriesError(f"image {image_id} is not in this series")
|
||||||
chapter_id = row
|
new_order = [image_id] + [i for i in order if i != image_id]
|
||||||
chapters = await self._chapter_order(series_tag_id)
|
await self.reorder(series_tag_id, new_order)
|
||||||
if chapters and chapters[0] != chapter_id:
|
|
||||||
new_chapters = [chapter_id] + [c for c in chapters if c != chapter_id]
|
|
||||||
await self.reorder_chapters(series_tag_id, new_chapters)
|
|
||||||
pages = await self._chapter_page_order(chapter_id)
|
|
||||||
if pages and pages[0] != image_id:
|
|
||||||
new_pages = [image_id] + [p for p in pages if p != image_id]
|
|
||||||
await self.reorder_pages(series_tag_id, chapter_id, new_pages)
|
|
||||||
|
|
||||||
# ---- post → series (FC-6.2) ------------------------------------------
|
|
||||||
|
|
||||||
async def _post_images_ordered(self, post_id: int) -> list[int]:
|
|
||||||
"""A post's images in capture order (ImageRecord.primary_post_id), the
|
|
||||||
same ordering the posts feed renders thumbnails in."""
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(ImageRecord.id)
|
|
||||||
.where(ImageRecord.primary_post_id == post_id)
|
|
||||||
.order_by(ImageRecord.id.asc())
|
|
||||||
)
|
|
||||||
).scalars().all()
|
|
||||||
return list(rows)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _stated_map(image_ids: list[int], start: int | None) -> dict | None:
|
|
||||||
"""Per-image stated_page = start, start+1, ... when the post stated a
|
|
||||||
starting page; None when it didn't (fall back to capture order)."""
|
|
||||||
if start is None:
|
|
||||||
return None
|
|
||||||
return {iid: start + i for i, iid in enumerate(image_ids)}
|
|
||||||
|
|
||||||
async def promote_post_to_series(self, post_id: int) -> dict:
|
|
||||||
"""Case 1: a self-contained multi-image post becomes its own series.
|
|
||||||
Creates a series tag named after the post, one chapter, the post's
|
|
||||||
images as its pages (stated pages parsed from the post when present)."""
|
|
||||||
from .tag_service import TagService
|
|
||||||
|
|
||||||
post = await self.session.get(Post, post_id)
|
|
||||||
if post is None:
|
|
||||||
raise SeriesError(f"post {post_id} not found")
|
|
||||||
image_ids = await self._post_images_ordered(post_id)
|
|
||||||
if not image_ids:
|
|
||||||
raise SeriesError("post has no images to make a series from")
|
|
||||||
name = (post.post_title or f"Series from post {post_id}").strip()[:200]
|
|
||||||
tag = await TagService(self.session).find_or_create(name, TagKind.series)
|
|
||||||
rng = parse_page_range(f"{post.post_title or ''} {post.description or ''}")
|
|
||||||
start, end = rng if rng else (None, None)
|
|
||||||
ch = await self.create_chapter(
|
|
||||||
tag.id, stated_page_start=start, stated_page_end=end,
|
|
||||||
)
|
|
||||||
added = await self.add_images(
|
|
||||||
tag.id, image_ids, chapter_id=ch["id"],
|
|
||||||
stated_pages=self._stated_map(image_ids, start),
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"series_tag_id": tag.id, "name": tag.name,
|
|
||||||
"chapter_id": ch["id"], "added": added,
|
|
||||||
}
|
|
||||||
|
|
||||||
async def add_post_as_chapter(
|
|
||||||
self, series_tag_id: int, post_id: int
|
|
||||||
) -> dict:
|
|
||||||
"""Case 2: append a post as the next chapter of an existing series. The
|
|
||||||
chapter is titled after the post and slotted by parsed page number when
|
|
||||||
present (else appended at the end)."""
|
|
||||||
await self._require_series(series_tag_id)
|
|
||||||
post = await self.session.get(Post, post_id)
|
|
||||||
if post is None:
|
|
||||||
raise SeriesError(f"post {post_id} not found")
|
|
||||||
image_ids = await self._post_images_ordered(post_id)
|
|
||||||
if not image_ids:
|
|
||||||
raise SeriesError("post has no images to add")
|
|
||||||
rng = parse_page_range(f"{post.post_title or ''} {post.description or ''}")
|
|
||||||
start, end = rng if rng else (None, None)
|
|
||||||
ch = await self.create_chapter(
|
|
||||||
series_tag_id, title=post.post_title or None,
|
|
||||||
stated_page_start=start, stated_page_end=end,
|
|
||||||
)
|
|
||||||
added = await self.add_images(
|
|
||||||
series_tag_id, image_ids, chapter_id=ch["id"],
|
|
||||||
stated_pages=self._stated_map(image_ids, start),
|
|
||||||
)
|
|
||||||
if start is not None:
|
|
||||||
await self._place_chapter_by_stated(series_tag_id, ch["id"], start)
|
|
||||||
return {
|
|
||||||
"series_tag_id": series_tag_id, "chapter_id": ch["id"],
|
|
||||||
"added": added,
|
|
||||||
}
|
|
||||||
|
|
||||||
async def _place_chapter_by_stated(
|
|
||||||
self, series_tag_id: int, chapter_id: int, start: int
|
|
||||||
) -> None:
|
|
||||||
"""Move `chapter_id` to sit before the first chapter whose stated start
|
|
||||||
is higher — so a post stating pages 5-8 lands ahead of the 9-12 chapter."""
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(SeriesChapter.id, SeriesChapter.stated_page_start)
|
|
||||||
.where(SeriesChapter.series_tag_id == series_tag_id)
|
|
||||||
.order_by(SeriesChapter.chapter_number.asc())
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
current = [r.id for r in rows]
|
|
||||||
others = [r for r in rows if r.id != chapter_id]
|
|
||||||
insert_at = len(others)
|
|
||||||
for idx, r in enumerate(others):
|
|
||||||
if r.stated_page_start is not None and r.stated_page_start > start:
|
|
||||||
insert_at = idx
|
|
||||||
break
|
|
||||||
new_order = [r.id for r in others]
|
|
||||||
new_order.insert(insert_at, chapter_id)
|
|
||||||
if new_order != current:
|
|
||||||
await self.reorder_chapters(series_tag_id, new_order)
|
|
||||||
|
|
||||||
# ---- browse list (FC-6.2) --------------------------------------------
|
|
||||||
|
|
||||||
async def list_series(
|
|
||||||
self, *, sort: str = "recent", artist_id: int | None = None
|
|
||||||
) -> list[dict]:
|
|
||||||
"""Series cards for the browse view: cover thumb, name, artist, chapter
|
|
||||||
+ page counts, a gap flag, and last-updated. sort ∈ recent|name|size."""
|
|
||||||
# Page counts + most-recent activity per series.
|
|
||||||
page_rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(
|
|
||||||
SeriesPage.series_tag_id,
|
|
||||||
func.count().label("pages"),
|
|
||||||
).group_by(SeriesPage.series_tag_id)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
page_count = {r.series_tag_id: r.pages for r in page_rows}
|
|
||||||
|
|
||||||
ch_rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(
|
|
||||||
SeriesChapter.series_tag_id,
|
|
||||||
SeriesChapter.id,
|
|
||||||
SeriesChapter.chapter_number,
|
|
||||||
SeriesChapter.stated_page_start,
|
|
||||||
SeriesChapter.stated_page_end,
|
|
||||||
SeriesChapter.updated_at,
|
|
||||||
)
|
|
||||||
.order_by(
|
|
||||||
SeriesChapter.series_tag_id,
|
|
||||||
SeriesChapter.chapter_number.asc(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
chapters_by_series: dict[int, list] = {}
|
|
||||||
updated_by_series: dict[int, object] = {}
|
|
||||||
for r in ch_rows:
|
|
||||||
chapters_by_series.setdefault(r.series_tag_id, []).append(r)
|
|
||||||
prev = updated_by_series.get(r.series_tag_id)
|
|
||||||
if prev is None or (r.updated_at and r.updated_at > prev):
|
|
||||||
updated_by_series[r.series_tag_id] = r.updated_at
|
|
||||||
|
|
||||||
# Cover = the (chapter_number, page_number)-minimum page per series.
|
|
||||||
cover_q = (
|
|
||||||
select(
|
|
||||||
SeriesPage.series_tag_id,
|
|
||||||
ImageRecord.sha256,
|
|
||||||
ImageRecord.mime,
|
|
||||||
ImageRecord.thumbnail_path,
|
|
||||||
ImageRecord.artist_id,
|
|
||||||
func.row_number()
|
|
||||||
.over(
|
|
||||||
partition_by=SeriesPage.series_tag_id,
|
|
||||||
order_by=(
|
|
||||||
SeriesChapter.chapter_number.asc(),
|
|
||||||
SeriesPage.page_number.asc(),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
.label("rn"),
|
|
||||||
)
|
|
||||||
.join(SeriesChapter, SeriesChapter.id == SeriesPage.chapter_id)
|
|
||||||
.join(ImageRecord, ImageRecord.id == SeriesPage.image_id)
|
|
||||||
.subquery()
|
|
||||||
)
|
|
||||||
cover_rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(
|
|
||||||
cover_q.c.series_tag_id,
|
|
||||||
cover_q.c.sha256,
|
|
||||||
cover_q.c.mime,
|
|
||||||
cover_q.c.thumbnail_path,
|
|
||||||
cover_q.c.artist_id,
|
|
||||||
).where(cover_q.c.rn == 1)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
cover_by_series = {r.series_tag_id: r for r in cover_rows}
|
|
||||||
|
|
||||||
# Artist names for the covers.
|
|
||||||
artist_ids = {
|
|
||||||
r.artist_id for r in cover_rows if r.artist_id is not None
|
|
||||||
}
|
|
||||||
artist_by_id: dict[int, object] = {}
|
|
||||||
if artist_ids:
|
|
||||||
arows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(Artist.id, Artist.name, Artist.slug)
|
|
||||||
.where(Artist.id.in_(artist_ids))
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
artist_by_id = {a.id: a for a in arows}
|
|
||||||
|
|
||||||
tags = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(Tag.id, Tag.name).where(Tag.kind == TagKind.series)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
|
|
||||||
out: list[dict] = []
|
|
||||||
for t in tags:
|
|
||||||
cover = cover_by_series.get(t.id)
|
|
||||||
if artist_id is not None and (
|
|
||||||
cover is None or cover.artist_id != artist_id
|
|
||||||
):
|
|
||||||
continue
|
|
||||||
chs = chapters_by_series.get(t.id, [])
|
|
||||||
gap = bool(
|
|
||||||
self._gaps(
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"id": c.id,
|
|
||||||
"stated_page_start": c.stated_page_start,
|
|
||||||
"stated_page_end": c.stated_page_end,
|
|
||||||
}
|
|
||||||
for c in chs
|
|
||||||
]
|
|
||||||
)
|
|
||||||
)
|
|
||||||
artist = artist_by_id.get(cover.artist_id) if cover else None
|
|
||||||
out.append(
|
|
||||||
{
|
|
||||||
"id": t.id,
|
|
||||||
"name": t.name,
|
|
||||||
"cover_thumbnail_url": (
|
|
||||||
thumbnail_url(
|
|
||||||
cover.thumbnail_path, cover.sha256, cover.mime
|
|
||||||
)
|
|
||||||
if cover
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
"artist_name": artist.name if artist else None,
|
|
||||||
"artist_slug": artist.slug if artist else None,
|
|
||||||
"chapter_count": len(chs),
|
|
||||||
"page_count": page_count.get(t.id, 0),
|
|
||||||
"has_gap": gap,
|
|
||||||
"updated_at": (
|
|
||||||
updated_by_series.get(t.id).isoformat()
|
|
||||||
if updated_by_series.get(t.id)
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
if sort == "name":
|
|
||||||
out.sort(key=lambda s: s["name"].lower())
|
|
||||||
elif sort == "size":
|
|
||||||
out.sort(key=lambda s: s["page_count"], reverse=True)
|
|
||||||
else: # recent
|
|
||||||
out.sort(key=lambda s: s["updated_at"] or "", reverse=True)
|
|
||||||
return out
|
|
||||||
|
|||||||
@@ -20,20 +20,11 @@ class ShowcaseService:
|
|||||||
async def random_sample(self, limit: int = 60) -> list[dict]:
|
async def random_sample(self, limit: int = 60) -> list[dict]:
|
||||||
if limit < 1 or limit > 200:
|
if limit < 1 or limit > 200:
|
||||||
raise ValueError("limit must be between 1 and 200")
|
raise ValueError("limit must be between 1 and 200")
|
||||||
# Over-sample then random-order (#699): SYSTEM_ROWS reads CONTIGUOUS rows
|
|
||||||
# from each sampled page, so sequentially-imported near-duplicates
|
|
||||||
# (multi-image posts, variant sets) come back adjacent and cluster in the
|
|
||||||
# showcase ("three near-identical in a row"). Sampling a multiple of
|
|
||||||
# `limit` spans more pages, and ORDER BY random() before taking `limit`
|
|
||||||
# breaks the physical adjacency — far better spread, still cheap
|
|
||||||
# (random() over a few hundred rows, not the whole table).
|
|
||||||
oversample = min(limit * 5, 1000)
|
|
||||||
stmt = select(ImageRecord).from_statement(
|
stmt = select(ImageRecord).from_statement(
|
||||||
text(
|
text(
|
||||||
"SELECT * FROM ("
|
"SELECT * FROM image_record "
|
||||||
" SELECT * FROM image_record TABLESAMPLE SYSTEM_ROWS(:o)"
|
"TABLESAMPLE SYSTEM_ROWS(:n)"
|
||||||
") sub ORDER BY random() LIMIT :n"
|
).bindparams(n=limit)
|
||||||
).bindparams(o=oversample, n=limit)
|
|
||||||
)
|
)
|
||||||
rows = (await self.session.execute(stmt)).scalars().all()
|
rows = (await self.session.execute(stmt)).scalars().all()
|
||||||
return [
|
return [
|
||||||
|
|||||||
@@ -64,15 +64,6 @@ class SourceRecord:
|
|||||||
consecutive_failures: int
|
consecutive_failures: int
|
||||||
next_check_at: str | None
|
next_check_at: str | None
|
||||||
backfill_runs_remaining: int
|
backfill_runs_remaining: int
|
||||||
# plan #693: derived from config_overrides for the UI badge.
|
|
||||||
backfill_state: str | None # "running" | "complete" | "stalled" | None (idle)
|
|
||||||
backfill_chunks: int
|
|
||||||
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
|
|
||||||
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
|
|
||||||
backfill_bypass_seen: bool
|
|
||||||
# plan #704: cumulative posts processed across the walk's chunks — live
|
|
||||||
# progress for the badge.
|
|
||||||
backfill_posts: int
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
def to_dict(self) -> dict:
|
||||||
return {
|
return {
|
||||||
@@ -91,10 +82,6 @@ class SourceRecord:
|
|||||||
"consecutive_failures": self.consecutive_failures,
|
"consecutive_failures": self.consecutive_failures,
|
||||||
"next_check_at": self.next_check_at,
|
"next_check_at": self.next_check_at,
|
||||||
"backfill_runs_remaining": self.backfill_runs_remaining,
|
"backfill_runs_remaining": self.backfill_runs_remaining,
|
||||||
"backfill_state": self.backfill_state,
|
|
||||||
"backfill_chunks": self.backfill_chunks,
|
|
||||||
"backfill_bypass_seen": self.backfill_bypass_seen,
|
|
||||||
"backfill_posts": self.backfill_posts,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -102,13 +89,10 @@ class SourceRecord:
|
|||||||
|
|
||||||
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
|
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
|
||||||
|
|
||||||
# Plan #693: backfill safety cap. "Start backfill" (and a newly created
|
# Plan #544 follow-up: newly created enabled sources pre-arm backfill so
|
||||||
# enabled source) arms a run-until-done walk; this caps how many time-boxed
|
# their first N polls walk gallery-dl's full post history with the longer
|
||||||
# chunks it may spend before pausing as "stalled", so a pathological walk that
|
# timeout (matches the manual "Deep scan" button's default).
|
||||||
# never reaches the bottom can't run forever. Generous on purpose — at
|
NEW_SOURCE_BACKFILL_RUNS = 3
|
||||||
# BACKFILL_CHUNK_SECONDS (600s) per chunk this is ~33h of cumulative walk, far
|
|
||||||
# beyond any real catalog; the cursor stall-guard is the real terminator.
|
|
||||||
BACKFILL_MAX_CHUNKS = 200
|
|
||||||
|
|
||||||
|
|
||||||
class SourceService:
|
class SourceService:
|
||||||
@@ -151,7 +135,6 @@ class SourceService:
|
|||||||
self, source: Source, artist: Artist, settings: ImportSettings,
|
self, source: Source, artist: Artist, settings: ImportSettings,
|
||||||
) -> SourceRecord:
|
) -> SourceRecord:
|
||||||
nxt = compute_next_check_at(source, artist, settings)
|
nxt = compute_next_check_at(source, artist, settings)
|
||||||
co = source.config_overrides or {}
|
|
||||||
return SourceRecord(
|
return SourceRecord(
|
||||||
id=source.id,
|
id=source.id,
|
||||||
artist_id=source.artist_id,
|
artist_id=source.artist_id,
|
||||||
@@ -168,10 +151,6 @@ class SourceService:
|
|||||||
consecutive_failures=source.consecutive_failures or 0,
|
consecutive_failures=source.consecutive_failures or 0,
|
||||||
next_check_at=nxt.isoformat() if nxt else None,
|
next_check_at=nxt.isoformat() if nxt else None,
|
||||||
backfill_runs_remaining=source.backfill_runs_remaining or 0,
|
backfill_runs_remaining=source.backfill_runs_remaining or 0,
|
||||||
backfill_state=co.get("_backfill_state"),
|
|
||||||
backfill_chunks=int(co.get("_backfill_chunks", 0)),
|
|
||||||
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
|
|
||||||
backfill_posts=int(co.get("_backfill_posts", 0)),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _row_to_record(self, source: Source) -> SourceRecord:
|
async def _row_to_record(self, source: Source) -> SourceRecord:
|
||||||
@@ -233,17 +212,16 @@ class SourceService:
|
|||||||
select(func.count(Source.id)).where(Source.artist_id == artist_id)
|
select(func.count(Source.id)).where(Source.artist_id == artist_id)
|
||||||
)).scalar_one()
|
)).scalar_one()
|
||||||
|
|
||||||
# Plan #693: a freshly added subscription has no archive yet, so it
|
# Plan #544 follow-up: a freshly added subscription has no archive
|
||||||
# should walk its full post history once. Arm run-until-done backfill
|
# yet, so the first few polls would walk the full post history in
|
||||||
# (state="running" + the chunk cap); the time-boxed chunks march to the
|
# tick mode and trip exit:20 after ~20 contiguous archive hits —
|
||||||
# bottom across ticks, then flip to "complete" and tick mode takes over.
|
# except there are none yet, so tick mode would walk forever and
|
||||||
# Disabled sources (incl. sidecar synthetics, url='sidecar:...') are
|
# blow the wall-clock cap. Pre-arm backfill so the initial syncs
|
||||||
# never polled, so leave them idle.
|
# use the longer timeout + skip:True walk. Tick mode resumes once
|
||||||
if enabled:
|
# the budget is spent or the queue drains.
|
||||||
config_overrides = {**(config_overrides or {}), "_backfill_state": "running"}
|
# Disabled sources (incl. sidecar synthetics, url='sidecar:...')
|
||||||
backfill_runs = BACKFILL_MAX_CHUNKS
|
# are never polled, so leave their counter at 0.
|
||||||
else:
|
backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
|
||||||
backfill_runs = 0
|
|
||||||
source = Source(
|
source = Source(
|
||||||
artist_id=artist_id, platform=platform, url=url,
|
artist_id=artist_id, platform=platform, url=url,
|
||||||
enabled=enabled, config_overrides=config_overrides,
|
enabled=enabled, config_overrides=config_overrides,
|
||||||
@@ -308,71 +286,22 @@ class SourceService:
|
|||||||
await self.session.commit()
|
await self.session.commit()
|
||||||
return await self._row_to_record(source)
|
return await self._row_to_record(source)
|
||||||
|
|
||||||
async def start_backfill(self, source_id: int) -> SourceRecord:
|
async def set_backfill_runs(
|
||||||
"""Plan #693: arm a run-until-done backfill. Sets state="running" and
|
self, source_id: int, runs: int,
|
||||||
the chunk cap; download runs then walk the full post history in
|
) -> SourceRecord:
|
||||||
time-boxed chunks (skip:True + BACKFILL_CHUNK_SECONDS), resuming from
|
"""Plan #544: arm a source for backfill mode. The next `runs`
|
||||||
the cursor each chunk, until gallery-dl reaches the bottom (→ state
|
download runs will use gallery-dl's full-walk config (skip: True
|
||||||
"complete") or the cap/stall-guard pauses it (→ "stalled"). Clears any
|
+ 30-min timeout) instead of the catch-up default. Runs must be
|
||||||
prior cursor/chunk/stall state so a re-start walks fresh from the top."""
|
1..10 — bigger is rejected to keep the operator from accidentally
|
||||||
|
setting a runaway budget."""
|
||||||
|
if not isinstance(runs, int) or runs < 1 or runs > 10:
|
||||||
|
raise ValueError("runs must be an integer in [1, 10]")
|
||||||
source = (await self.session.execute(
|
source = (await self.session.execute(
|
||||||
select(Source).where(Source.id == source_id)
|
select(Source).where(Source.id == source_id)
|
||||||
)).scalar_one_or_none()
|
)).scalar_one_or_none()
|
||||||
if source is None:
|
if source is None:
|
||||||
raise LookupError(f"source id={source_id} not found")
|
raise LookupError(f"source id={source_id} not found")
|
||||||
co = dict(source.config_overrides or {})
|
source.backfill_runs_remaining = runs
|
||||||
co["_backfill_state"] = "running"
|
|
||||||
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
|
|
||||||
"_backfill_posts"):
|
|
||||||
co.pop(k, None)
|
|
||||||
source.config_overrides = co
|
|
||||||
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
|
|
||||||
await self.session.commit()
|
|
||||||
return await self._row_to_record(source)
|
|
||||||
|
|
||||||
async def start_recovery(self, source_id: int) -> SourceRecord:
|
|
||||||
"""Plan #697: arm a RECOVERY walk — a backfill that bypasses the Patreon
|
|
||||||
seen-ledger so deliberately-dropped-and-deleted near-dups get re-fetched
|
|
||||||
and re-evaluated under the CURRENT pHash threshold (tier-2 disk still
|
|
||||||
spares files we kept). Reuses the entire #693 backfill state machine
|
|
||||||
(time-boxed chunks, cursor checkpoint, complete/stall lifecycle) plus the
|
|
||||||
`_backfill_bypass_seen` flag that flips download mode to recovery. Clears
|
|
||||||
any prior cursor/chunk/stall state so it walks fresh from the top. The
|
|
||||||
flag is cleared on completion (download_service) and on stop.
|
|
||||||
|
|
||||||
Recovery is Patreon-only (the seen-ledger is Patreon's); for other
|
|
||||||
platforms the flag is inert (download_service ignores it) and the walk
|
|
||||||
runs as a plain backfill. The UI gates the action to Patreon sources."""
|
|
||||||
source = (await self.session.execute(
|
|
||||||
select(Source).where(Source.id == source_id)
|
|
||||||
)).scalar_one_or_none()
|
|
||||||
if source is None:
|
|
||||||
raise LookupError(f"source id={source_id} not found")
|
|
||||||
co = dict(source.config_overrides or {})
|
|
||||||
co["_backfill_state"] = "running"
|
|
||||||
co["_backfill_bypass_seen"] = True
|
|
||||||
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
|
|
||||||
"_backfill_posts"):
|
|
||||||
co.pop(k, None)
|
|
||||||
source.config_overrides = co
|
|
||||||
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
|
|
||||||
await self.session.commit()
|
|
||||||
return await self._row_to_record(source)
|
|
||||||
|
|
||||||
async def stop_backfill(self, source_id: int) -> SourceRecord:
|
|
||||||
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
|
|
||||||
Clears the running state + cursor/chunk/stall bookkeeping."""
|
|
||||||
source = (await self.session.execute(
|
|
||||||
select(Source).where(Source.id == source_id)
|
|
||||||
)).scalar_one_or_none()
|
|
||||||
if source is None:
|
|
||||||
raise LookupError(f"source id={source_id} not found")
|
|
||||||
co = dict(source.config_overrides or {})
|
|
||||||
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
|
|
||||||
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_posts"):
|
|
||||||
co.pop(k, None)
|
|
||||||
source.config_overrides = co
|
|
||||||
source.backfill_runs_remaining = 0
|
|
||||||
await self.session.commit()
|
await self.session.commit()
|
||||||
return await self._row_to_record(source)
|
return await self._row_to_record(source)
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,5 @@
|
|||||||
"""Tag CRUD + autocomplete + image-tag association."""
|
"""Tag CRUD + autocomplete + image-tag association."""
|
||||||
|
|
||||||
import logging
|
|
||||||
import time
|
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
@@ -14,25 +12,6 @@ from ..models import Tag, TagKind, image_tag
|
|||||||
from ..models.tag_allowlist import TagAllowlist
|
from ..models.tag_allowlist import TagAllowlist
|
||||||
from ..models.tag_reference_embedding import TagReferenceEmbedding
|
from ..models.tag_reference_embedding import TagReferenceEmbedding
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_tag_name(name: str) -> str:
|
|
||||||
"""Canonical tag form (#701): collapse whitespace + capitalize the first
|
|
||||||
letter of each word, PRESERVING the rest of the word.
|
|
||||||
|
|
||||||
Per-word capitalize (NOT str.title(), which mangles apostrophes:
|
|
||||||
don't → Don'T). The word tail is left untouched so acronyms survive
|
|
||||||
(DC stays DC, NSFW stays NSFW) — operator-revised 2026-06-06: protecting
|
|
||||||
acronyms matters more than folding ALL-CAPS input. This MATCHES
|
|
||||||
ml/tag_name._title_word, so a tag suggested by the Camie tagger keeps the
|
|
||||||
exact casing the suggestion UI showed when it round-trips through the
|
|
||||||
create endpoint on Accept.
|
|
||||||
"""
|
|
||||||
return " ".join(
|
|
||||||
w[:1].upper() + w[1:] for w in (name or "").split()
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TagValidationError(ValueError):
|
class TagValidationError(ValueError):
|
||||||
"""Raised when tag construction breaks the kind/fandom rules."""
|
"""Raised when tag construction breaks the kind/fandom rules."""
|
||||||
@@ -92,10 +71,6 @@ class TagService:
|
|||||||
- if fandom_id is set, the referenced tag must exist and have
|
- if fandom_id is set, the referenced tag must exist and have
|
||||||
kind == TagKind.fandom
|
kind == TagKind.fandom
|
||||||
"""
|
"""
|
||||||
# NOTE: case is NOT normalized here — find_or_create is the shared path
|
|
||||||
# the ML tagger / allowlist also use, and Title-Casing the booru
|
|
||||||
# vocabulary breaks allowlist matching. Display-casing (Title Case) is
|
|
||||||
# applied at the user-entry layer (api/tags create_tag) only (#701).
|
|
||||||
name = name.strip()
|
name = name.strip()
|
||||||
if not name:
|
if not name:
|
||||||
raise TagValidationError("Tag name cannot be empty")
|
raise TagValidationError("Tag name cannot be empty")
|
||||||
@@ -118,17 +93,13 @@ class TagService:
|
|||||||
# inserts; without the savepoint the outer transaction would
|
# inserts; without the savepoint the outer transaction would
|
||||||
# poison and the calling request crashes. Mirrors
|
# poison and the calling request crashes. Mirrors
|
||||||
# importer._get_or_create.
|
# importer._get_or_create.
|
||||||
# Case-insensitive match (#701): a normalized (Title Case) input must
|
|
||||||
# find an existing differently-cased tag instead of forking a duplicate.
|
|
||||||
stmt = (
|
stmt = (
|
||||||
select(Tag)
|
select(Tag)
|
||||||
.where(func.lower(Tag.name) == name.lower())
|
.where(Tag.name == name)
|
||||||
.where(Tag.kind == kind)
|
.where(Tag.kind == kind)
|
||||||
.where(
|
.where(
|
||||||
Tag.fandom_id.is_(None) if fandom_id is None else Tag.fandom_id == fandom_id
|
Tag.fandom_id.is_(None) if fandom_id is None else Tag.fandom_id == fandom_id
|
||||||
)
|
)
|
||||||
.order_by(Tag.id)
|
|
||||||
.limit(1)
|
|
||||||
)
|
)
|
||||||
existing = (await self.session.execute(stmt)).scalar_one_or_none()
|
existing = (await self.session.execute(stmt)).scalar_one_or_none()
|
||||||
if existing:
|
if existing:
|
||||||
@@ -278,11 +249,9 @@ class TagService:
|
|||||||
if tag is None:
|
if tag is None:
|
||||||
raise TagValidationError(f"Tag {tag_id} not found")
|
raise TagValidationError(f"Tag {tag_id} not found")
|
||||||
|
|
||||||
# Case-insensitive clash (#701) — renaming onto a differently-cased tag
|
|
||||||
# is still a merge, not a silent fork.
|
|
||||||
clash_stmt = (
|
clash_stmt = (
|
||||||
select(Tag)
|
select(Tag)
|
||||||
.where(func.lower(Tag.name) == new_name.lower())
|
.where(Tag.name == new_name)
|
||||||
.where(Tag.kind == tag.kind)
|
.where(Tag.kind == tag.kind)
|
||||||
.where(
|
.where(
|
||||||
Tag.fandom_id.is_(None)
|
Tag.fandom_id.is_(None)
|
||||||
@@ -290,8 +259,6 @@ class TagService:
|
|||||||
else Tag.fandom_id == tag.fandom_id
|
else Tag.fandom_id == tag.fandom_id
|
||||||
)
|
)
|
||||||
.where(Tag.id != tag_id)
|
.where(Tag.id != tag_id)
|
||||||
.order_by(Tag.id)
|
|
||||||
.limit(1)
|
|
||||||
)
|
)
|
||||||
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
|
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
|
||||||
if clash is not None:
|
if clash is not None:
|
||||||
@@ -313,69 +280,6 @@ class TagService:
|
|||||||
await self.session.flush()
|
await self.session.flush()
|
||||||
return tag
|
return tag
|
||||||
|
|
||||||
async def set_fandom(
|
|
||||||
self, tag_id: int, fandom_id: int | None, *, merge: bool = False
|
|
||||||
) -> Tag:
|
|
||||||
"""Set / change / clear a character tag's fandom.
|
|
||||||
|
|
||||||
Raises TagValidationError unless the tag is a character and fandom_id
|
|
||||||
(when given) references a fandom tag. If the change would collide with
|
|
||||||
an existing character of the same name in the TARGET fandom, raises
|
|
||||||
TagMergeConflict (the API turns that into a 409 merge hint) — unless
|
|
||||||
merge=True, in which case this tag is merged INTO that existing
|
|
||||||
character (a deliberate cross-fandom merge) and the surviving target
|
|
||||||
is returned. Passing fandom_id=None clears the fandom.
|
|
||||||
"""
|
|
||||||
tag = await self.session.get(Tag, tag_id)
|
|
||||||
if tag is None:
|
|
||||||
raise TagValidationError(f"Tag {tag_id} not found")
|
|
||||||
if tag.kind != TagKind.character:
|
|
||||||
raise TagValidationError("Only character tags can have a fandom")
|
|
||||||
if fandom_id is not None:
|
|
||||||
fandom = await self.session.get(Tag, fandom_id)
|
|
||||||
if fandom is None or fandom.kind != TagKind.fandom:
|
|
||||||
raise TagValidationError(
|
|
||||||
f"fandom_id {fandom_id} does not reference a fandom tag"
|
|
||||||
)
|
|
||||||
if fandom_id == tag.fandom_id:
|
|
||||||
return tag
|
|
||||||
|
|
||||||
# Collision: another character with the same name already lives in the
|
|
||||||
# target fandom. Mirrors rename's (name, kind, fandom_id) uniqueness.
|
|
||||||
clash_stmt = (
|
|
||||||
select(Tag)
|
|
||||||
.where(Tag.name == tag.name)
|
|
||||||
.where(Tag.kind == TagKind.character)
|
|
||||||
.where(
|
|
||||||
Tag.fandom_id.is_(None)
|
|
||||||
if fandom_id is None
|
|
||||||
else Tag.fandom_id == fandom_id
|
|
||||||
)
|
|
||||||
.where(Tag.id != tag_id)
|
|
||||||
)
|
|
||||||
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
|
|
||||||
if clash is not None:
|
|
||||||
if not merge:
|
|
||||||
source_image_count = await self.session.scalar(
|
|
||||||
select(func.count())
|
|
||||||
.select_from(image_tag)
|
|
||||||
.where(image_tag.c.tag_id == tag_id)
|
|
||||||
)
|
|
||||||
will_alias = await self._keep_as_alias(tag_id)
|
|
||||||
raise TagMergeConflict(
|
|
||||||
f"A character named {tag.name!r} already exists in that fandom",
|
|
||||||
target_id=clash.id,
|
|
||||||
target_name=clash.name,
|
|
||||||
source_image_count=int(source_image_count or 0),
|
|
||||||
will_alias=will_alias,
|
|
||||||
)
|
|
||||||
await self._do_merge(tag, clash)
|
|
||||||
return clash
|
|
||||||
|
|
||||||
tag.fandom_id = fandom_id
|
|
||||||
await self.session.flush()
|
|
||||||
return tag
|
|
||||||
|
|
||||||
async def merge(self, source_id: int, target_id: int) -> MergeResult:
|
async def merge(self, source_id: int, target_id: int) -> MergeResult:
|
||||||
"""Transactionally repoint every FK from source→target, optionally
|
"""Transactionally repoint every FK from source→target, optionally
|
||||||
keep source's name as a tagger alias, delete source. Atomic: any
|
keep source's name as a tagger alias, delete source. Atomic: any
|
||||||
@@ -394,14 +298,7 @@ class TagService:
|
|||||||
raise TagValidationError(
|
raise TagValidationError(
|
||||||
"Tags must be the same kind and fandom to merge"
|
"Tags must be the same kind and fandom to merge"
|
||||||
)
|
)
|
||||||
return await self._do_merge(source, target)
|
|
||||||
|
|
||||||
async def _do_merge(self, source: Tag, target: Tag) -> MergeResult:
|
|
||||||
"""Repoint every FK source→target, optionally keep source's name as a
|
|
||||||
tagger alias, delete source. NO kind/fandom validation — callers that
|
|
||||||
need it (public merge()) validate first; set_fandom's collision
|
|
||||||
resolution calls this directly for a deliberate CROSS-fandom merge."""
|
|
||||||
source_id, target_id = source.id, target.id
|
|
||||||
keep_as_alias = await self._keep_as_alias(source_id)
|
keep_as_alias = await self._keep_as_alias(source_id)
|
||||||
source_name = source.name
|
source_name = source.name
|
||||||
source_kind = source.kind
|
source_kind = source.kind
|
||||||
@@ -515,18 +412,8 @@ class TagService:
|
|||||||
)
|
)
|
||||||
|
|
||||||
async def _repoint_series_pages(self, src: int, tgt: int) -> None:
|
async def _repoint_series_pages(self, src: int, tgt: int) -> None:
|
||||||
from ..models.series_chapter import SeriesChapter
|
|
||||||
from ..models.series_page import SeriesPage
|
from ..models.series_page import SeriesPage
|
||||||
|
|
||||||
# Move the chapters first so the pages' chapter_id FK stays valid: a
|
|
||||||
# chapter left pointing at src would cascade-delete (with its pages) when
|
|
||||||
# src is removed. chapter_number may now collide across the merged set —
|
|
||||||
# acceptable (it's an ordering key, not unique).
|
|
||||||
await self.session.execute(
|
|
||||||
update(SeriesChapter)
|
|
||||||
.where(SeriesChapter.series_tag_id == src)
|
|
||||||
.values(series_tag_id=tgt)
|
|
||||||
)
|
|
||||||
# image_id is UNIQUE across series_page and src != tgt, so an
|
# image_id is UNIQUE across series_page and src != tgt, so an
|
||||||
# image in src's series cannot already be in tgt's — no collision.
|
# image in src's series cannot already be in tgt's — no collision.
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
@@ -544,21 +431,6 @@ class TagService:
|
|||||||
update(Tag).where(Tag.fandom_id == src).values(fandom_id=tgt)
|
update(Tag).where(Tag.fandom_id == src).values(fandom_id=tgt)
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _image_assoc_counts(self, tag_ids: list[int]) -> dict[int, int]:
|
|
||||||
"""image_tag row counts keyed by tag_id, for the survivor heuristic
|
|
||||||
(the best-connected tag in a collision group survives → fewest
|
|
||||||
FK repoints). Tags with zero associations are absent from the map."""
|
|
||||||
if not tag_ids:
|
|
||||||
return {}
|
|
||||||
rows = (
|
|
||||||
await self.session.execute(
|
|
||||||
select(image_tag.c.tag_id, func.count())
|
|
||||||
.where(image_tag.c.tag_id.in_(tag_ids))
|
|
||||||
.group_by(image_tag.c.tag_id)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
return {tid: int(n) for tid, n in rows}
|
|
||||||
|
|
||||||
async def _create_protective_aliases(
|
async def _create_protective_aliases(
|
||||||
self, src_name: str, src_kind: TagKind, tgt: int
|
self, src_name: str, src_kind: TagKind, tgt: int
|
||||||
) -> bool:
|
) -> bool:
|
||||||
@@ -606,189 +478,3 @@ class TagService:
|
|||||||
if res.rowcount:
|
if res.rowcount:
|
||||||
created = True
|
created = True
|
||||||
return created
|
return created
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# #714: retro-normalize existing tags to the #701 canonical (Title Case +
|
|
||||||
# collapsed whitespace) and merge case/whitespace-variant duplicates.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
_NORMALIZE_SAMPLE_CAP = 50
|
|
||||||
|
|
||||||
|
|
||||||
def _group_existing_tags(
|
|
||||||
rows,
|
|
||||||
) -> dict[tuple, list[tuple[int, str]]]:
|
|
||||||
"""Group (id, name, kind, fandom_id) rows by their post-normalization
|
|
||||||
identity: (kind, COALESCE(fandom_id, -1), canonical_name). Every tag that
|
|
||||||
would collapse to the same canonical lives in ONE group, so the canonical
|
|
||||||
form is unique within (kind, fandom) once each group is resolved."""
|
|
||||||
groups: dict[tuple, list[tuple[int, str]]] = {}
|
|
||||||
for tag_id, name, kind, fandom_id in rows:
|
|
||||||
canonical = normalize_tag_name(name)
|
|
||||||
key = (kind, fandom_id if fandom_id is not None else -1, canonical)
|
|
||||||
groups.setdefault(key, []).append((tag_id, name))
|
|
||||||
return groups
|
|
||||||
|
|
||||||
|
|
||||||
def _group_needs_change(canonical: str, members: list[tuple[int, str]]) -> bool:
|
|
||||||
"""A group is already canonical iff it's a single member whose name equals
|
|
||||||
the canonical form. Anything else (a collision, or a lone mis-cased tag)
|
|
||||||
needs work."""
|
|
||||||
if len(members) > 1:
|
|
||||||
return True
|
|
||||||
return members[0][1] != canonical
|
|
||||||
|
|
||||||
|
|
||||||
def _best_connected(tag_ids: list[int], counts: dict[int, int]) -> int:
|
|
||||||
"""The tag with the most image associations (→ fewest FK repoints when it
|
|
||||||
survives), tie-broken to the lowest id for determinism. Module-level so the
|
|
||||||
key closes over its parameter, not a loop variable (ruff B023)."""
|
|
||||||
return max(tag_ids, key=lambda tid: (counts.get(tid, 0), -tid))
|
|
||||||
|
|
||||||
|
|
||||||
async def normalize_existing_tags(
|
|
||||||
session: AsyncSession,
|
|
||||||
*,
|
|
||||||
dry_run: bool = False,
|
|
||||||
time_budget_seconds: float | None = None,
|
|
||||||
) -> dict:
|
|
||||||
"""Convert the back-catalog to the #701 canonical tag form.
|
|
||||||
|
|
||||||
For each (kind, fandom, canonical) group: pick a survivor, merge any
|
|
||||||
case/whitespace-variant siblings INTO it via the tested merge path
|
|
||||||
(TagService._do_merge — FK repoints + protective aliases), then rename the
|
|
||||||
survivor to the canonical form. Idempotent: a group that is already a lone
|
|
||||||
canonical tag is a no-op, so re-running is safe.
|
|
||||||
|
|
||||||
A first run over a fresh back-catalog can touch tens of thousands of tags
|
|
||||||
(the whole booru-derived vocabulary needs recasing) and won't finish inside
|
|
||||||
one Celery time limit — it timed out at 40 min (operator-flagged 2026-06-07).
|
|
||||||
`time_budget_seconds` time-boxes the live run: it stops cleanly at the budget
|
|
||||||
and reports `partial`/`remaining` so the caller can re-enqueue and continue.
|
|
||||||
Because it commits per group and is idempotent, the next run just picks up
|
|
||||||
the groups still needing change.
|
|
||||||
|
|
||||||
dry_run=True returns a projection (counts + a sample of the changes) with no
|
|
||||||
mutations. Live runs commit per group and isolate failures per group so one
|
|
||||||
bad group can't strand the rest.
|
|
||||||
|
|
||||||
Returns (dry_run):
|
|
||||||
{"groups": N, "collisions": M, "tags_to_merge": K, "tags_to_rename": R,
|
|
||||||
"total_changes": T, "sample": [{"to", "from": [...], "kind", "merge"}]}
|
|
||||||
Returns (live):
|
|
||||||
{"groups_processed", "merged", "renamed", "aliases_created", "errors",
|
|
||||||
"total_changes", "remaining", "partial", "sample": [...]}
|
|
||||||
"""
|
|
||||||
rows = (
|
|
||||||
await session.execute(
|
|
||||||
select(Tag.id, Tag.name, Tag.kind, Tag.fandom_id)
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
groups = _group_existing_tags(rows)
|
|
||||||
|
|
||||||
# Deterministic sample/ordering: by kind then canonical name.
|
|
||||||
touched = sorted(
|
|
||||||
(
|
|
||||||
(key, members)
|
|
||||||
for key, members in groups.items()
|
|
||||||
if _group_needs_change(key[2], members)
|
|
||||||
),
|
|
||||||
key=lambda km: (
|
|
||||||
km[0][0].value if hasattr(km[0][0], "value") else str(km[0][0]),
|
|
||||||
km[0][2].lower(),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
sample = [
|
|
||||||
{
|
|
||||||
"to": key[2],
|
|
||||||
"from": [name for _id, name in members],
|
|
||||||
"kind": key[0].value if hasattr(key[0], "value") else str(key[0]),
|
|
||||||
"merge": len(members) > 1,
|
|
||||||
}
|
|
||||||
for key, members in touched[:_NORMALIZE_SAMPLE_CAP]
|
|
||||||
]
|
|
||||||
|
|
||||||
if dry_run:
|
|
||||||
collisions = sum(1 for key, m in touched if len(m) > 1)
|
|
||||||
tags_to_merge = sum(len(m) - 1 for key, m in touched if len(m) > 1)
|
|
||||||
# A group renames iff the canonical form isn't already one of its
|
|
||||||
# members' exact names (else that member is picked as survivor → no
|
|
||||||
# rename, the rest merge into it).
|
|
||||||
tags_to_rename = sum(
|
|
||||||
1
|
|
||||||
for key, m in touched
|
|
||||||
if key[2] not in {name for _id, name in m}
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"groups": len(groups),
|
|
||||||
"collisions": collisions,
|
|
||||||
"tags_to_merge": tags_to_merge,
|
|
||||||
"tags_to_rename": tags_to_rename,
|
|
||||||
"total_changes": len(touched),
|
|
||||||
"sample": sample,
|
|
||||||
}
|
|
||||||
|
|
||||||
svc = TagService(session)
|
|
||||||
summary = {
|
|
||||||
"groups_processed": 0,
|
|
||||||
"merged": 0,
|
|
||||||
"renamed": 0,
|
|
||||||
"aliases_created": 0,
|
|
||||||
"errors": 0,
|
|
||||||
"total_changes": len(touched),
|
|
||||||
"remaining": len(touched),
|
|
||||||
"partial": False,
|
|
||||||
"sample": sample,
|
|
||||||
}
|
|
||||||
start = time.monotonic()
|
|
||||||
for done, (key, members) in enumerate(touched):
|
|
||||||
# Time-box: stop cleanly before the Celery limit kills us mid-group and
|
|
||||||
# strands the run as a timeout. The caller re-enqueues to finish the
|
|
||||||
# rest (idempotent — already-canonical groups are skipped next pass).
|
|
||||||
if (
|
|
||||||
time_budget_seconds is not None
|
|
||||||
and time.monotonic() - start >= time_budget_seconds
|
|
||||||
):
|
|
||||||
summary["partial"] = True
|
|
||||||
summary["remaining"] = len(touched) - done
|
|
||||||
break
|
|
||||||
canonical = key[2]
|
|
||||||
names_by_id = dict(members)
|
|
||||||
# Survivor: prefer a member already named canonically (no rename, no
|
|
||||||
# self-alias); else the best-connected (fewest FK repoints); else
|
|
||||||
# lowest id for determinism.
|
|
||||||
survivor_id = next(
|
|
||||||
(tid for tid, name in members if name == canonical), None
|
|
||||||
)
|
|
||||||
if survivor_id is None:
|
|
||||||
counts = await svc._image_assoc_counts(list(names_by_id))
|
|
||||||
survivor_id = _best_connected(list(names_by_id), counts)
|
|
||||||
loser_ids = [tid for tid in names_by_id if tid != survivor_id]
|
|
||||||
try:
|
|
||||||
survivor = await session.get(Tag, survivor_id)
|
|
||||||
for loser_id in loser_ids:
|
|
||||||
loser = await session.get(Tag, loser_id)
|
|
||||||
if loser is None:
|
|
||||||
continue
|
|
||||||
result = await svc._do_merge(loser, survivor)
|
|
||||||
if result.alias_created:
|
|
||||||
summary["aliases_created"] += 1
|
|
||||||
if survivor.name != canonical:
|
|
||||||
survivor.name = canonical
|
|
||||||
await session.flush()
|
|
||||||
summary["renamed"] += 1
|
|
||||||
await session.commit()
|
|
||||||
summary["groups_processed"] += 1
|
|
||||||
summary["merged"] += len(loser_ids)
|
|
||||||
except Exception as exc: # one bad group must not strand the rest
|
|
||||||
await session.rollback()
|
|
||||||
summary["errors"] += 1
|
|
||||||
log.warning(
|
|
||||||
"tag normalize failed for group %r: %s", canonical, exc
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# Loop finished without hitting the time budget — nothing left to do.
|
|
||||||
summary["remaining"] = 0
|
|
||||||
return summary
|
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ disposes it (``await engine.dispose()``) when its loop ends.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||||
from sqlalchemy.pool import NullPool
|
|
||||||
|
|
||||||
from ..config import get_config
|
from ..config import get_config
|
||||||
|
|
||||||
@@ -18,13 +17,5 @@ from ..config import get_config
|
|||||||
def async_session_factory():
|
def async_session_factory():
|
||||||
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
|
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
|
||||||
cfg = get_config()
|
cfg = get_config()
|
||||||
# NullPool: this engine lives for ONE task (created + disposed per
|
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
|
||||||
# asyncio.run loop), so intra-task connection pooling buys nothing and
|
|
||||||
# actively bit us — download_source releases its phase-1 connection
|
|
||||||
# before a multi-minute gallery-dl subprocess, and a *pooled* idle
|
|
||||||
# connection would be reaped by the server and handed back dead to
|
|
||||||
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
|
|
||||||
# opens a fresh real connection on each checkout, so phase 3 always
|
|
||||||
# reconnects clean; pre_ping is then redundant.
|
|
||||||
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
|
|
||||||
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
|
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
|
||||||
|
|||||||
@@ -34,10 +34,3 @@ def sync_session_factory():
|
|||||||
)
|
)
|
||||||
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
|
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
|
||||||
return _SESSIONMAKER
|
return _SESSIONMAKER
|
||||||
|
|
||||||
|
|
||||||
def get_sync_engine():
|
|
||||||
"""The process-wide sync Engine — for raw work that needs a connection
|
|
||||||
directly (e.g. AUTOCOMMIT VACUUM, which can't run inside a transaction)."""
|
|
||||||
sync_session_factory() # ensure _ENGINE is initialized
|
|
||||||
return _ENGINE
|
|
||||||
|
|||||||
@@ -55,146 +55,3 @@ def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
|
|||||||
return cleanup_service.delete_images(
|
return cleanup_service.delete_images(
|
||||||
session, image_ids=image_ids, images_root=IMAGES_ROOT,
|
session, image_ids=image_ids, images_root=IMAGES_ROOT,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# Time-box one chunk well under the soft limit so a large archive back-catalog
|
|
||||||
# can't run the task into the Celery time limit (or hog the maintenance_long
|
|
||||||
# lane). The task re-enqueues itself with the resume cursor until the scan is
|
|
||||||
# exhausted — mirrors normalize_tags_task (operator-asked 2026-06-07: reasonable
|
|
||||||
# timeout, then re-queue so other work keeps flowing).
|
|
||||||
_REEXTRACT_CHUNK_SECONDS = 600
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.admin.reextract_archive_attachments_task",
|
|
||||||
bind=True,
|
|
||||||
autoretry_for=(OperationalError, DBAPIError),
|
|
||||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
|
||||||
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
|
|
||||||
)
|
|
||||||
def reextract_archive_attachments_task(self, after_id: int = 0) -> dict:
|
|
||||||
"""Wraps cleanup_service.reextract_archive_attachments (#713 part 2):
|
|
||||||
re-extract PostAttachments that are actually archives but were filed
|
|
||||||
opaquely before the magic-byte gate, and link their members to the post.
|
|
||||||
|
|
||||||
Time-boxed + self-resuming: scans attachments after ``after_id`` and, on a
|
|
||||||
chunk cut, re-enqueues from where it stopped so a big backlog finishes across
|
|
||||||
chunks instead of dying at the soft limit."""
|
|
||||||
SessionLocal = _sync_session_factory()
|
|
||||||
with SessionLocal() as session:
|
|
||||||
summary = cleanup_service.reextract_archive_attachments(
|
|
||||||
session, images_root=IMAGES_ROOT,
|
|
||||||
time_budget_seconds=_REEXTRACT_CHUNK_SECONDS, after_id=after_id,
|
|
||||||
)
|
|
||||||
# More attachments past this chunk's cursor — continue in the next.
|
|
||||||
if summary.get("partial") and summary.get("resume_after_id", 0) > after_id:
|
|
||||||
log.info(
|
|
||||||
"reextract chunk done (%d scanned, %d archives, resume after id %s) "
|
|
||||||
"— re-enqueuing to continue",
|
|
||||||
summary.get("scanned", 0), summary.get("archives", 0),
|
|
||||||
summary["resume_after_id"],
|
|
||||||
)
|
|
||||||
reextract_archive_attachments_task.delay(summary["resume_after_id"])
|
|
||||||
return summary
|
|
||||||
|
|
||||||
|
|
||||||
# Time-box one chunk well under the soft limit so a large back-catalog (the
|
|
||||||
# first run recases the whole booru vocabulary) can't run the task into the
|
|
||||||
# Celery time limit — it timed out at 40 min, operator-flagged 2026-06-07. The
|
|
||||||
# task re-enqueues itself until nothing remains (idempotent — already-canonical
|
|
||||||
# groups are skipped). 600s keeps each chunk short enough that the recovery
|
|
||||||
# sweep and other maintenance tasks interleave on the concurrency-1 queue.
|
|
||||||
_NORMALIZE_CHUNK_SECONDS = 600
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.admin.normalize_tags_task",
|
|
||||||
bind=True,
|
|
||||||
autoretry_for=(OperationalError, DBAPIError),
|
|
||||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
|
||||||
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
|
|
||||||
)
|
|
||||||
def normalize_tags_task(self) -> dict:
|
|
||||||
"""Wraps tag_service.normalize_existing_tags (#714): Title-Case the
|
|
||||||
back-catalog and merge case/whitespace-variant duplicate tags via the
|
|
||||||
tested async merge path. Time-boxed + self-resuming so a huge first run
|
|
||||||
finishes across chunks instead of timing out. Runs under its own asyncio
|
|
||||||
loop + per-task async engine (NullPool), mirroring download_source."""
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
from ..services.tag_service import normalize_existing_tags
|
|
||||||
from ._async_session import async_session_factory
|
|
||||||
|
|
||||||
async def _run() -> dict:
|
|
||||||
async_factory, async_engine = async_session_factory()
|
|
||||||
try:
|
|
||||||
async with async_factory() as session:
|
|
||||||
# normalize_existing_tags commits per group internally.
|
|
||||||
return await normalize_existing_tags(
|
|
||||||
session, dry_run=False,
|
|
||||||
time_budget_seconds=_NORMALIZE_CHUNK_SECONDS,
|
|
||||||
)
|
|
||||||
finally:
|
|
||||||
await async_engine.dispose()
|
|
||||||
|
|
||||||
summary = asyncio.run(_run())
|
|
||||||
# More groups to canonicalize than fit this chunk — continue in the next.
|
|
||||||
if summary.get("partial") and summary.get("remaining", 0) > 0:
|
|
||||||
log.info(
|
|
||||||
"normalize_tags_task chunk done (%d processed, %d remaining) — "
|
|
||||||
"re-enqueuing to continue",
|
|
||||||
summary.get("groups_processed", 0), summary["remaining"],
|
|
||||||
)
|
|
||||||
normalize_tags_task.delay()
|
|
||||||
return summary
|
|
||||||
|
|
||||||
|
|
||||||
# Time-box one rescan chunk well under the soft limit and re-enqueue from the
|
|
||||||
# cursor — scoring every post against its artist's series is O(posts) and grows
|
|
||||||
# with the library (FC-6.3). Mirrors normalize_tags_task.
|
|
||||||
_SERIES_RESCAN_CHUNK_SECONDS = 600
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
|
||||||
name="backend.app.tasks.admin.rescan_series_suggestions_task",
|
|
||||||
bind=True,
|
|
||||||
autoretry_for=(OperationalError, DBAPIError),
|
|
||||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
|
||||||
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
|
|
||||||
)
|
|
||||||
def rescan_series_suggestions_task(self, after_post_id: int = 0) -> dict:
|
|
||||||
"""Score posts against their artist's series and write pending suggestions
|
|
||||||
(FC-6.3). Settings-gated; time-boxed + self-resuming from a post-id cursor.
|
|
||||||
Per-task async engine (NullPool) under its own asyncio loop, like normalize."""
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
from ..models import ImportSettings
|
|
||||||
from ..services.series_match_service import SeriesMatchService
|
|
||||||
from ._async_session import async_session_factory
|
|
||||||
|
|
||||||
async def _run() -> dict:
|
|
||||||
async_factory, async_engine = async_session_factory()
|
|
||||||
try:
|
|
||||||
async with async_factory() as session:
|
|
||||||
settings = await ImportSettings.load(session)
|
|
||||||
if not settings.series_suggest_enabled:
|
|
||||||
return {"skipped": "series suggestions disabled"}
|
|
||||||
threshold = settings.series_suggest_threshold
|
|
||||||
return await SeriesMatchService(session).rescan(
|
|
||||||
threshold=threshold,
|
|
||||||
time_budget_seconds=_SERIES_RESCAN_CHUNK_SECONDS,
|
|
||||||
after_post_id=after_post_id,
|
|
||||||
)
|
|
||||||
finally:
|
|
||||||
await async_engine.dispose()
|
|
||||||
|
|
||||||
summary = asyncio.run(_run())
|
|
||||||
if summary.get("partial") and summary.get("resume_after_id", 0) > after_post_id:
|
|
||||||
log.info(
|
|
||||||
"rescan_series_suggestions chunk done (%d scanned, %d suggested, "
|
|
||||||
"resume after %s) — re-enqueuing",
|
|
||||||
summary.get("scanned", 0), summary.get("suggested", 0),
|
|
||||||
summary["resume_after_id"],
|
|
||||||
)
|
|
||||||
rescan_series_suggestions_task.delay(summary["resume_after_id"])
|
|
||||||
return summary
|
|
||||||
|
|||||||
@@ -41,11 +41,7 @@ def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
|
|||||||
bind=True,
|
bind=True,
|
||||||
autoretry_for=(OperationalError, DBAPIError),
|
autoretry_for=(OperationalError, DBAPIError),
|
||||||
retry_backoff=10, retry_backoff_max=120, max_retries=2,
|
retry_backoff=10, retry_backoff_max=120, max_retries=2,
|
||||||
# A pg_dump can't be chunked; the 12-min limit timed out once the DB grew
|
soft_time_limit=600, time_limit=720,
|
||||||
# (operator-flagged 2026-06-07). 30/35 min gives real headroom. (A long
|
|
||||||
# backup still briefly holds the concurrency-1 maintenance lane — the
|
|
||||||
# structural fix is a dedicated lane for the long one-shots.)
|
|
||||||
soft_time_limit=1800, time_limit=2100,
|
|
||||||
)
|
)
|
||||||
def backup_db_task(self, *, tag: str | None = None,
|
def backup_db_task(self, *, tag: str | None = None,
|
||||||
triggered_by: str = "manual") -> dict:
|
triggered_by: str = "manual") -> dict:
|
||||||
|
|||||||
@@ -1,17 +1,12 @@
|
|||||||
"""download_source Celery task — runs DownloadService for one source."""
|
"""download_source Celery task — runs DownloadService for one source."""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
|
||||||
from datetime import UTC, datetime
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from celery.exceptions import SoftTimeLimitExceeded
|
|
||||||
from sqlalchemy import select
|
|
||||||
from sqlalchemy.exc import DBAPIError, OperationalError
|
from sqlalchemy.exc import DBAPIError, OperationalError
|
||||||
from sqlalchemy.orm import Session as SyncSession
|
|
||||||
|
|
||||||
from ..celery_app import celery
|
from ..celery_app import celery
|
||||||
from ..models import DownloadEvent, ImportSettings, Source
|
from ..models import ImportSettings
|
||||||
from ..services.credential_crypto import CredentialCrypto
|
from ..services.credential_crypto import CredentialCrypto
|
||||||
from ..services.credential_service import CredentialService
|
from ..services.credential_service import CredentialService
|
||||||
from ..services.download_service import DownloadService
|
from ..services.download_service import DownloadService
|
||||||
@@ -21,99 +16,9 @@ from ..services.thumbnailer import Thumbnailer
|
|||||||
from ._async_session import async_session_factory
|
from ._async_session import async_session_factory
|
||||||
from .import_file import _sync_session_factory
|
from .import_file import _sync_session_factory
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
IMAGES_ROOT = Path("/images")
|
IMAGES_ROOT = Path("/images")
|
||||||
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
|
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
|
||||||
|
|
||||||
# Celery time budget for one download_source run. The ceiling that
|
|
||||||
# governs *clean* teardown is the SOFT limit: it raises a catchable
|
|
||||||
# SoftTimeLimitExceeded in-process, whereas the HARD limit SIGKILLs the
|
|
||||||
# worker (no chance to finalize). Both gallery-dl subprocess budgets
|
|
||||||
# (gallery_dl.py: _DEFAULT_GDL_TIMEOUT_SECONDS=870 tick,
|
|
||||||
# BACKFILL_CHUNK_SECONDS=600 per backfill chunk, plan #693) MUST sit below the soft limit
|
|
||||||
# so subprocess.run raises its own TimeoutExpired first — that path
|
|
||||||
# captures partial stdout/stderr and finalizes the DownloadEvent. soft is
|
|
||||||
# max-subprocess (1170) + ~180s phase-3 persist headroom; hard is soft +
|
|
||||||
# 150s SIGKILL backstop. Audit 2026-06-03 (Anduo #39912): the old
|
|
||||||
# soft=900 sat BELOW the 1170 backfill budget, so SoftTimeLimitExceeded
|
|
||||||
# preempted TimeoutExpired and the event stranded empty. The recovery
|
|
||||||
# sweep's DOWNLOAD_STALL_THRESHOLD_MINUTES (30 min) still trails the new
|
|
||||||
# 25-min hard kill by 5 min, so it stays a true backstop. Invariant
|
|
||||||
# guarded by test_timeout_ladder_keeps_subprocess_budgets_under_soft_limit.
|
|
||||||
DOWNLOAD_SOFT_TIME_LIMIT = 1350
|
|
||||||
DOWNLOAD_HARD_TIME_LIMIT = 1500
|
|
||||||
|
|
||||||
# Per-platform serialization (plan: concurrency cap). When a serialized
|
|
||||||
# platform (Patreon) is already walking, defer this run by re-enqueuing it a
|
|
||||||
# little later rather than holding a worker slot or bowling into the same rate
|
|
||||||
# limit. Bounded so a wedged platform eventually runs anyway. The lock TTL sits
|
|
||||||
# just past the hard kill so a SIGKILL'd worker's lock auto-expires; a backfill
|
|
||||||
# chunk only holds it ~10 min, so the bounded wait stays well under the 30-min
|
|
||||||
# DownloadEvent recovery sweep.
|
|
||||||
_PLATFORM_LOCK_TTL = DOWNLOAD_HARD_TIME_LIMIT + 120
|
|
||||||
_SERIALIZE_COUNTDOWN = 20
|
|
||||||
_MAX_SERIALIZE_WAITS = 45 # ~15 min ceiling, then run uncapped as a safety valve
|
|
||||||
|
|
||||||
|
|
||||||
def _peek_platform(source_id: int) -> str | None:
|
|
||||||
SyncFactory = _sync_session_factory()
|
|
||||||
with SyncFactory() as session:
|
|
||||||
return session.execute(
|
|
||||||
select(Source.platform).where(Source.id == source_id)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
|
|
||||||
|
|
||||||
def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
|
|
||||||
"""Defense in depth for the soft-time-limit kill path.
|
|
||||||
|
|
||||||
A SoftTimeLimitExceeded unwinds download_source before phase 3 can
|
|
||||||
finalize the DownloadEvent, leaving it 'running' until the recovery
|
|
||||||
sweep stamps a context-free "stranded" error 30 min later — AND
|
|
||||||
leaving backfill_runs_remaining undecremented so the source re-runs
|
|
||||||
and re-strands every tick (Anduo #39912, 2026-06-03). Flip the
|
|
||||||
in-flight event to error with a real reason, mirror phase 3's
|
|
||||||
source-health write, and decrement any backfill budget so a
|
|
||||||
chronically-slow source self-heals back to tick mode.
|
|
||||||
|
|
||||||
The caller owns the commit. All mutations are gated on actually
|
|
||||||
finding a running event, so a benign late soft-limit (phase 3 already
|
|
||||||
committed) is a no-op.
|
|
||||||
"""
|
|
||||||
now = datetime.now(UTC)
|
|
||||||
ev = session.execute(
|
|
||||||
select(DownloadEvent)
|
|
||||||
.where(DownloadEvent.source_id == source_id)
|
|
||||||
.where(DownloadEvent.status == "running")
|
|
||||||
.order_by(DownloadEvent.id.desc())
|
|
||||||
.limit(1)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
if ev is None:
|
|
||||||
return
|
|
||||||
ev.status = "error"
|
|
||||||
ev.finished_at = now
|
|
||||||
ev.error = (
|
|
||||||
f"killed by Celery soft time limit ({DOWNLOAD_SOFT_TIME_LIMIT}s) "
|
|
||||||
"before the download finished — the run exceeded its time budget. "
|
|
||||||
"Progress is checkpointed per page, so the next tick resumes near "
|
|
||||||
"the cut; the backfill budget was decremented so it walks less. If "
|
|
||||||
"this recurs, a single post is heavy enough to overrun the chunk "
|
|
||||||
"time-box on its own."
|
|
||||||
)
|
|
||||||
ev.metadata_ = {
|
|
||||||
**(ev.metadata_ or {}),
|
|
||||||
"error_type": "timeout",
|
|
||||||
"soft_time_limited": True,
|
|
||||||
}
|
|
||||||
src = session.get(Source, source_id)
|
|
||||||
if src is not None:
|
|
||||||
src.consecutive_failures = (src.consecutive_failures or 0) + 1
|
|
||||||
src.last_error = "soft time limit exceeded"
|
|
||||||
src.error_type = "timeout"
|
|
||||||
src.last_checked_at = now
|
|
||||||
if (src.backfill_runs_remaining or 0) > 0:
|
|
||||||
src.backfill_runs_remaining = max(0, src.backfill_runs_remaining - 1)
|
|
||||||
|
|
||||||
|
|
||||||
@celery.task(
|
@celery.task(
|
||||||
name="backend.app.tasks.download.download_source",
|
name="backend.app.tasks.download.download_source",
|
||||||
@@ -124,51 +29,12 @@ def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
|
|||||||
retry_backoff_max=120,
|
retry_backoff_max=120,
|
||||||
retry_jitter=True,
|
retry_jitter=True,
|
||||||
max_retries=3,
|
max_retries=3,
|
||||||
soft_time_limit=DOWNLOAD_SOFT_TIME_LIMIT,
|
soft_time_limit=900,
|
||||||
time_limit=DOWNLOAD_HARD_TIME_LIMIT,
|
time_limit=1200,
|
||||||
)
|
)
|
||||||
def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
|
def download_source(self, source_id: int) -> int:
|
||||||
"""Returns the DownloadEvent.id."""
|
"""Returns the DownloadEvent.id."""
|
||||||
|
|
||||||
# Per-platform concurrency cap: only one Patreon walk runs at a time.
|
|
||||||
from ..services.platform_lock import platform_lock
|
|
||||||
|
|
||||||
platform = _peek_platform(source_id)
|
|
||||||
lock = (
|
|
||||||
platform_lock(platform, ttl_seconds=_PLATFORM_LOCK_TTL)
|
|
||||||
if platform is not None
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
if lock is not None:
|
|
||||||
try:
|
|
||||||
got_lock = bool(lock.acquire(blocking=False))
|
|
||||||
except Exception: # noqa: BLE001 — broker hiccup → degrade to uncapped
|
|
||||||
log.warning("platform lock acquire failed for %s; running uncapped", platform)
|
|
||||||
lock = None
|
|
||||||
got_lock = True
|
|
||||||
if lock is not None and not got_lock:
|
|
||||||
if _serialize_waits < _MAX_SERIALIZE_WAITS:
|
|
||||||
# Another walk on this platform holds the lock — re-enqueue
|
|
||||||
# ourselves shortly (the pending DownloadEvent stays; no new
|
|
||||||
# event, no log spam) rather than running concurrently into
|
|
||||||
# the rate limit.
|
|
||||||
download_source.apply_async(
|
|
||||||
(source_id,),
|
|
||||||
{"_serialize_waits": _serialize_waits + 1},
|
|
||||||
countdown=_SERIALIZE_COUNTDOWN,
|
|
||||||
)
|
|
||||||
log.info(
|
|
||||||
"download_source(%s) deferred — %s already walking (wait %d/%d)",
|
|
||||||
source_id, platform, _serialize_waits + 1, _MAX_SERIALIZE_WAITS,
|
|
||||||
)
|
|
||||||
return -1
|
|
||||||
log.warning(
|
|
||||||
"download_source(%s) running WITHOUT the %s lock — max serialize "
|
|
||||||
"waits hit, proceeding uncapped",
|
|
||||||
source_id, platform,
|
|
||||||
)
|
|
||||||
lock = None
|
|
||||||
|
|
||||||
async def _run():
|
async def _run():
|
||||||
async_factory, async_engine = async_session_factory()
|
async_factory, async_engine = async_session_factory()
|
||||||
SyncFactory = _sync_session_factory()
|
SyncFactory = _sync_session_factory()
|
||||||
@@ -202,38 +68,9 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
|
|||||||
gdl=gdl,
|
gdl=gdl,
|
||||||
importer=importer,
|
importer=importer,
|
||||||
cred_service=cred_service,
|
cred_service=cred_service,
|
||||||
# The native Patreon ingester opens its own short-lived
|
|
||||||
# sync sessions for the seen-ledger (never held across
|
|
||||||
# the walk). Same factory the importer's sync session
|
|
||||||
# comes from — a different DB connection per checkout.
|
|
||||||
sync_session_factory=SyncFactory,
|
|
||||||
)
|
)
|
||||||
return await svc.download_source(source_id)
|
return await svc.download_source(source_id)
|
||||||
finally:
|
finally:
|
||||||
await async_engine.dispose()
|
await async_engine.dispose()
|
||||||
|
|
||||||
try:
|
|
||||||
return asyncio.run(_run())
|
return asyncio.run(_run())
|
||||||
except SoftTimeLimitExceeded:
|
|
||||||
# phase 3 never ran — salvage the in-flight event so the operator
|
|
||||||
# sees a real reason instead of the recovery sweep's generic
|
|
||||||
# "stranded" 30 min later (Anduo #39912). Best-effort: a failure
|
|
||||||
# here must not mask the timeout. Re-raise so Celery + the
|
|
||||||
# task_run signal handler still record the kill.
|
|
||||||
try:
|
|
||||||
SyncFactory = _sync_session_factory()
|
|
||||||
with SyncFactory() as session:
|
|
||||||
_finalize_soft_limited(session, source_id)
|
|
||||||
session.commit()
|
|
||||||
except Exception: # noqa: BLE001 — cleanup must not swallow the kill
|
|
||||||
log.exception("soft-limit finalize failed for source %s", source_id)
|
|
||||||
raise
|
|
||||||
finally:
|
|
||||||
# Release the per-platform lock so the next walk can proceed. The TTL is
|
|
||||||
# a backstop for a SIGKILL; here we free it the instant the run ends. A
|
|
||||||
# late release after TTL expiry raises (lock already gone) — ignore it.
|
|
||||||
if lock is not None:
|
|
||||||
try:
|
|
||||||
lock.release()
|
|
||||||
except Exception: # noqa: BLE001 — TTL may have already freed it
|
|
||||||
pass
|
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ State machine:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import time
|
|
||||||
import traceback
|
import traceback
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
@@ -32,12 +31,6 @@ log = logging.getLogger(__name__)
|
|||||||
_BATCH = 500
|
_BATCH = 500
|
||||||
_PROGRESS_TICK = 100
|
_PROGRESS_TICK = 100
|
||||||
_MAX_MATCHED = 50_000
|
_MAX_MATCHED = 50_000
|
||||||
# One chunk's wall-clock budget. Was a single 2h pass that timed out on large
|
|
||||||
# libraries and held the concurrency-1 maintenance queue the whole time
|
|
||||||
# (operator-flagged 2026-06-07). Now: scan ~10 min, persist the keyset cursor +
|
|
||||||
# matches, re-enqueue to continue — so backups/vacuum/normalize chunks can
|
|
||||||
# interleave. soft/hard limits sit just above so the budget fires first.
|
|
||||||
_CHUNK_SECONDS = 600
|
|
||||||
|
|
||||||
_RULES = {
|
_RULES = {
|
||||||
"transparency": transparency.evaluate,
|
"transparency": transparency.evaluate,
|
||||||
@@ -53,16 +46,13 @@ _RULES = {
|
|||||||
retry_backoff_max=60,
|
retry_backoff_max=60,
|
||||||
retry_jitter=True,
|
retry_jitter=True,
|
||||||
max_retries=3,
|
max_retries=3,
|
||||||
soft_time_limit=900,
|
soft_time_limit=7200,
|
||||||
time_limit=1000,
|
time_limit=7500,
|
||||||
)
|
)
|
||||||
def scan_library_for_rule(self, audit_id: int) -> dict:
|
def scan_library_for_rule(self, audit_id: int) -> dict:
|
||||||
"""See module docstring. Time-boxed + self-resuming: one call scans a
|
"""See module docstring. Returns a small summary dict for eager-mode
|
||||||
~10-min chunk, persists the resume cursor + matches, and re-enqueues itself
|
|
||||||
until the library is exhausted. Returns a small summary dict for eager-mode
|
|
||||||
test assertions (real workers ignore the return value)."""
|
test assertions (real workers ignore the return value)."""
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
start = time.monotonic()
|
|
||||||
try:
|
try:
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
audit = session.get(LibraryAuditRun, audit_id)
|
audit = session.get(LibraryAuditRun, audit_id)
|
||||||
@@ -73,10 +63,9 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
_mark_error(session, audit_id, f"unknown rule {audit.rule!r}")
|
_mark_error(session, audit_id, f"unknown rule {audit.rule!r}")
|
||||||
return {"audit_id": audit_id, "status": "error"}
|
return {"audit_id": audit_id, "status": "error"}
|
||||||
params = dict(audit.params or {})
|
params = dict(audit.params or {})
|
||||||
# Resume from the previous chunk's persisted state.
|
matched: list[int] = []
|
||||||
matched: list[int] = list(audit.matched_ids or [])
|
scanned = 0
|
||||||
scanned = audit.scanned_count or 0
|
last_id = 0
|
||||||
last_id = audit.resume_after_id or 0
|
|
||||||
while True:
|
while True:
|
||||||
# Cancellation check between batches.
|
# Cancellation check between batches.
|
||||||
current_status = session.execute(
|
current_status = session.execute(
|
||||||
@@ -85,15 +74,6 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
).scalar_one()
|
).scalar_one()
|
||||||
if current_status == "cancelled":
|
if current_status == "cancelled":
|
||||||
return {"audit_id": audit_id, "status": "cancelled"}
|
return {"audit_id": audit_id, "status": "cancelled"}
|
||||||
# Time-box: persist the cursor + matches and re-enqueue so the
|
|
||||||
# queue is freed between chunks. The next call resumes here.
|
|
||||||
if time.monotonic() - start >= _CHUNK_SECONDS:
|
|
||||||
_persist_chunk(session, audit_id, scanned, matched, last_id)
|
|
||||||
scan_library_for_rule.delay(audit_id)
|
|
||||||
return {
|
|
||||||
"audit_id": audit_id, "status": "running",
|
|
||||||
"partial": True, "scanned": scanned,
|
|
||||||
}
|
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
select(ImageRecord.id, ImageRecord.path)
|
select(ImageRecord.id, ImageRecord.path)
|
||||||
.where(ImageRecord.id > last_id)
|
.where(ImageRecord.id > last_id)
|
||||||
@@ -134,16 +114,10 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
)
|
)
|
||||||
return {"audit_id": audit_id, "status": "error"}
|
return {"audit_id": audit_id, "status": "error"}
|
||||||
if scanned % _PROGRESS_TICK == 0:
|
if scanned % _PROGRESS_TICK == 0:
|
||||||
# Cheap heartbeat: scanned_count + last_progress_at so the
|
|
||||||
# recovery sweep sees the multi-chunk audit is alive. The
|
|
||||||
# cursor + matches are persisted at chunk boundaries.
|
|
||||||
session.execute(
|
session.execute(
|
||||||
update(LibraryAuditRun)
|
update(LibraryAuditRun)
|
||||||
.where(LibraryAuditRun.id == audit_id)
|
.where(LibraryAuditRun.id == audit_id)
|
||||||
.values(
|
.values(scanned_count=scanned)
|
||||||
scanned_count=scanned,
|
|
||||||
last_progress_at=datetime.now(UTC),
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
# Final state.
|
# Final state.
|
||||||
@@ -154,10 +128,8 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
scanned_count=scanned,
|
scanned_count=scanned,
|
||||||
matched_count=len(matched),
|
matched_count=len(matched),
|
||||||
matched_ids=matched,
|
matched_ids=matched,
|
||||||
resume_after_id=last_id,
|
|
||||||
status="ready",
|
status="ready",
|
||||||
finished_at=datetime.now(UTC),
|
finished_at=datetime.now(UTC),
|
||||||
last_progress_at=datetime.now(UTC),
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -168,14 +140,9 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
"matched": len(matched),
|
"matched": len(matched),
|
||||||
}
|
}
|
||||||
except SoftTimeLimitExceeded:
|
except SoftTimeLimitExceeded:
|
||||||
# Backstop (the in-chunk budget should fire first): the audit stays
|
with SessionLocal() as session:
|
||||||
# 'running' with its last committed cursor; re-enqueue to continue from
|
_mark_error(session, audit_id, "soft_time_limit exceeded (>7200s)")
|
||||||
# there rather than marking the whole run an error.
|
raise
|
||||||
log.warning(
|
|
||||||
"audit %s: soft time limit hit — re-enqueuing to resume", audit_id,
|
|
||||||
)
|
|
||||||
scan_library_for_rule.delay(audit_id)
|
|
||||||
return {"audit_id": audit_id, "status": "running", "partial": True}
|
|
||||||
except (OperationalError, DBAPIError):
|
except (OperationalError, DBAPIError):
|
||||||
# Retryable per the decorator; leave row in 'running' and let
|
# Retryable per the decorator; leave row in 'running' and let
|
||||||
# autoretry try again. Recovery sweep catches if all retries fail.
|
# autoretry try again. Recovery sweep catches if all retries fail.
|
||||||
@@ -187,23 +154,6 @@ def scan_library_for_rule(self, audit_id: int) -> dict:
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
|
|
||||||
def _persist_chunk(session, audit_id, scanned, matched, last_id) -> None:
|
|
||||||
"""Persist a chunk boundary: scanned count, matches so far, and the keyset
|
|
||||||
cursor the next chunk resumes from. Keeps status='running'."""
|
|
||||||
session.execute(
|
|
||||||
update(LibraryAuditRun)
|
|
||||||
.where(LibraryAuditRun.id == audit_id)
|
|
||||||
.values(
|
|
||||||
scanned_count=scanned,
|
|
||||||
matched_count=len(matched),
|
|
||||||
matched_ids=list(matched),
|
|
||||||
resume_after_id=last_id,
|
|
||||||
last_progress_at=datetime.now(UTC),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def _mark_error(session, audit_id: int, error_msg: str) -> None:
|
def _mark_error(session, audit_id: int, error_msg: str) -> None:
|
||||||
session.execute(
|
session.execute(
|
||||||
update(LibraryAuditRun)
|
update(LibraryAuditRun)
|
||||||
|
|||||||
@@ -22,27 +22,10 @@ from ..models import (
|
|||||||
TaskRun,
|
TaskRun,
|
||||||
)
|
)
|
||||||
from ..utils.phash import compute_phash
|
from ..utils.phash import compute_phash
|
||||||
from ._sync_engine import get_sync_engine
|
|
||||||
from ._sync_engine import sync_session_factory as _sync_session_factory
|
from ._sync_engine import sync_session_factory as _sync_session_factory
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
# High-churn tables whose dead-tuple bloat matters: the TABLESAMPLE showcase
|
|
||||||
# reads physical blocks (bloat slows it directly), and the periodic
|
|
||||||
# prune/backfill/recovery tasks generate dead tuples faster than autovacuum
|
|
||||||
# always keeps up with. VACUUM reclaims them; ANALYZE refreshes planner stats.
|
|
||||||
# Allowlist ONLY — names are interpolated into VACUUM, so they must never come
|
|
||||||
# from request input.
|
|
||||||
VACUUM_TABLES = (
|
|
||||||
"image_record",
|
|
||||||
"image_provenance",
|
|
||||||
"post_attachment",
|
|
||||||
"download_event",
|
|
||||||
"task_run",
|
|
||||||
"import_task",
|
|
||||||
"import_batch",
|
|
||||||
)
|
|
||||||
|
|
||||||
STUCK_THRESHOLD_MINUTES = 5
|
STUCK_THRESHOLD_MINUTES = 5
|
||||||
# Archive ImportTasks run the per-member pipeline inline for every
|
# Archive ImportTasks run the per-member pipeline inline for every
|
||||||
# member (import_archive_file: soft=30min/hard=35min). The ImportTask
|
# member (import_archive_file: soft=30min/hard=35min). The ImportTask
|
||||||
@@ -63,12 +46,9 @@ MAX_RECOVERY_ATTEMPTS = 3
|
|||||||
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
|
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
|
||||||
|
|
||||||
# DownloadEvent (pending|running) recovery threshold. download_source has
|
# DownloadEvent (pending|running) recovery threshold. download_source has
|
||||||
# time_limit=1500s (25 min, DOWNLOAD_HARD_TIME_LIMIT); 30 min is 5 min past
|
# time_limit=1200s (20 min); 30 min is 10 min past that, so a legitimately-
|
||||||
# that, so a legitimately-running task is hard-killed before the sweep ever
|
# running task is never killed by the sweep. Operator-confirmed 2026-05-29
|
||||||
# touches it — the sweep only catches events whose worker died without
|
# after 43 sources stranded at "last check never" by the in-flight guard.
|
||||||
# finalizing. Operator-confirmed 2026-05-29 after 43 sources stranded at
|
|
||||||
# "last check never" by the in-flight guard; budget bumped 2026-06-03 with
|
|
||||||
# the soft/hard limit raise (Anduo #39912).
|
|
||||||
DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
|
DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
|
||||||
|
|
||||||
OLD_TASK_DAYS = 7
|
OLD_TASK_DAYS = 7
|
||||||
@@ -83,13 +63,8 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
|||||||
# small buffer) so the sweep never flags in-flight work.
|
# small buffer) so the sweep never flags in-flight work.
|
||||||
#
|
#
|
||||||
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
|
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
|
||||||
# with a 30-min buffer.
|
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
|
||||||
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
|
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
|
||||||
# DB backup/restore is seconds-to-minutes (35-min hard limit). It must NOT share
|
|
||||||
# the images' 7h window — a DB backup wedged on NFS would otherwise sit "running"
|
|
||||||
# for 7 hours holding the concurrency-1 maintenance_long lane (operator-flagged
|
|
||||||
# 2026-06-07). 40 min gives a small buffer over the hard limit.
|
|
||||||
BACKUP_DB_STALL_THRESHOLD_MINUTES = 40
|
|
||||||
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
|
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
|
||||||
# 2h15m gives a 10-min buffer.
|
# 2h15m gives a 10-min buffer.
|
||||||
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
|
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
|
||||||
@@ -560,7 +535,7 @@ def recover_stalled_download_events() -> int:
|
|||||||
tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending')
|
tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending')
|
||||||
and fires download_source.delay(). If that task dies before finalizing the
|
and fires download_source.delay(). If that task dies before finalizing the
|
||||||
event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind
|
event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind
|
||||||
on the 1500s hard time_limit — the event stays in-flight forever. The next
|
on the 1200s hard time_limit — the event stays in-flight forever. The next
|
||||||
tick then skips that source because of the in-flight guard (scan.py:168)
|
tick then skips that source because of the in-flight guard (scan.py:168)
|
||||||
and Source.last_checked_at never updates; the operator sees "last check
|
and Source.last_checked_at never updates; the operator sees "last check
|
||||||
never" in the Subscriptions health column, permanently.
|
never" in the Subscriptions health column, permanently.
|
||||||
@@ -624,20 +599,16 @@ def recover_stalled_backup_runs() -> int:
|
|||||||
"""
|
"""
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
db_cutoff = now - timedelta(minutes=BACKUP_DB_STALL_THRESHOLD_MINUTES)
|
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
|
||||||
slow_cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
|
msg = (
|
||||||
msg = "stranded by recovery sweep (no terminal status within the stall window)"
|
f"stranded by recovery sweep (no terminal status after "
|
||||||
|
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
|
||||||
|
)
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
result = session.execute(
|
result = session.execute(
|
||||||
update(BackupRun)
|
update(BackupRun)
|
||||||
.where(BackupRun.status.in_(["running", "restoring"]))
|
.where(BackupRun.status.in_(["running", "restoring"]))
|
||||||
# db backups/restores are fast (40-min window); images run hours (7h).
|
.where(BackupRun.started_at < cutoff)
|
||||||
.where(
|
|
||||||
or_(
|
|
||||||
and_(BackupRun.kind == "db", BackupRun.started_at < db_cutoff),
|
|
||||||
and_(BackupRun.kind != "db", BackupRun.started_at < slow_cutoff),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
.values(status="error", finished_at=now, error=msg)
|
.values(status="error", finished_at=now, error=msg)
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -656,30 +627,19 @@ def recover_stalled_library_audit_runs() -> int:
|
|||||||
guard in start_audit_run — a SIGKILL'd run would block all future
|
guard in start_audit_run — a SIGKILL'd run would block all future
|
||||||
audits until manual DB surgery. (The guard is now age-aware, but
|
audits until manual DB surgery. (The guard is now age-aware, but
|
||||||
this sweep is what makes that work in practice.)
|
this sweep is what makes that work in practice.)
|
||||||
|
|
||||||
Measures staleness from last_progress_at (alembic 0039), NOT started_at:
|
|
||||||
a chunked scan stays 'running' across many re-enqueued chunks and can
|
|
||||||
legitimately run for hours on a big library — only flag one that hasn't
|
|
||||||
made progress in the threshold window (a dead chunk that never re-enqueued).
|
|
||||||
Falls back to started_at for pre-0039 / never-ticked rows.
|
|
||||||
"""
|
"""
|
||||||
SessionLocal = _sync_session_factory()
|
SessionLocal = _sync_session_factory()
|
||||||
now = datetime.now(UTC)
|
now = datetime.now(UTC)
|
||||||
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
|
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
|
||||||
msg = (
|
msg = (
|
||||||
f"stranded by recovery sweep (no progress for "
|
f"stranded by recovery sweep (no terminal status after "
|
||||||
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
|
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
|
||||||
)
|
)
|
||||||
with SessionLocal() as session:
|
with SessionLocal() as session:
|
||||||
result = session.execute(
|
result = session.execute(
|
||||||
update(LibraryAuditRun)
|
update(LibraryAuditRun)
|
||||||
.where(LibraryAuditRun.status == "running")
|
.where(LibraryAuditRun.status == "running")
|
||||||
.where(
|
.where(LibraryAuditRun.started_at < cutoff)
|
||||||
func.coalesce(
|
|
||||||
LibraryAuditRun.last_progress_at,
|
|
||||||
LibraryAuditRun.started_at,
|
|
||||||
) < cutoff
|
|
||||||
)
|
|
||||||
.values(status="error", finished_at=now, error=msg)
|
.values(status="error", finished_at=now, error=msg)
|
||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
@@ -798,20 +758,3 @@ def cleanup_old_download_events() -> int:
|
|||||||
)
|
)
|
||||||
session.commit()
|
session.commit()
|
||||||
return result.rowcount or 0
|
return result.rowcount or 0
|
||||||
|
|
||||||
|
|
||||||
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
|
|
||||||
def vacuum_analyze() -> dict:
|
|
||||||
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
|
|
||||||
reclaim dead-tuple bloat and refresh planner statistics. VACUUM cannot run
|
|
||||||
inside a transaction block, so it runs on an AUTOCOMMIT connection.
|
|
||||||
Scheduled weekly; also operator-triggerable from Settings → Maintenance.
|
|
||||||
"""
|
|
||||||
engine = get_sync_engine()
|
|
||||||
done = []
|
|
||||||
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
|
|
||||||
for table in VACUUM_TABLES:
|
|
||||||
conn.exec_driver_sql(f"VACUUM (ANALYZE) {table}")
|
|
||||||
done.append(table)
|
|
||||||
log.info("vacuum_analyze complete: %s", done)
|
|
||||||
return {"vacuumed": done}
|
|
||||||
|
|||||||
@@ -2,26 +2,6 @@
|
|||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
_MAX_EXT_LEN = 16
|
|
||||||
|
|
||||||
|
|
||||||
def safe_ext(name: str | Path) -> str:
|
|
||||||
"""Conservatively extract a short, alphanumeric file extension.
|
|
||||||
|
|
||||||
gallery-dl and Patreon CDN URLs produce basenames with URL-encoded
|
|
||||||
query-string artifacts, so `Path.suffix` can return 50+ chars of base64-ish
|
|
||||||
junk that blows bounded VARCHAR columns (e.g. PostAttachment.ext varchar(32)).
|
|
||||||
Accept only a suffix ≤16 chars whose post-dot characters are all alphanumeric;
|
|
||||||
otherwise return "" (no known extension). Operator-flagged 2026-05-25 — ONE
|
|
||||||
impl for the importer and the native Patreon client.
|
|
||||||
"""
|
|
||||||
suffix = Path(name).suffix.lower()
|
|
||||||
if not suffix or len(suffix) > _MAX_EXT_LEN:
|
|
||||||
return ""
|
|
||||||
if not all(c.isalnum() for c in suffix[1:]):
|
|
||||||
return ""
|
|
||||||
return suffix
|
|
||||||
|
|
||||||
|
|
||||||
def derive_subdir(source_path: Path, import_root: Path) -> str:
|
def derive_subdir(source_path: Path, import_root: Path) -> str:
|
||||||
"""Returns the relative subdirectory of source_path under import_root.
|
"""Returns the relative subdirectory of source_path under import_root.
|
||||||
|
|||||||
@@ -149,8 +149,9 @@ def _run_probe(path_str: str) -> tuple[str, str | None]:
|
|||||||
call the same code path.
|
call the same code path.
|
||||||
"""
|
"""
|
||||||
path = Path(path_str)
|
path = Path(path_str)
|
||||||
|
ext = path.suffix.lower()
|
||||||
try:
|
try:
|
||||||
total, test_bad = _inspect_archive(path)
|
total, test_bad = _inspect_archive(path, ext)
|
||||||
except Exception as exc: # noqa: BLE001 — clean rejection
|
except Exception as exc: # noqa: BLE001 — clean rejection
|
||||||
return ("error", f"{type(exc).__name__}: {exc}")
|
return ("error", f"{type(exc).__name__}: {exc}")
|
||||||
if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES:
|
if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES:
|
||||||
@@ -161,29 +162,24 @@ def _run_probe(path_str: str) -> tuple[str, str | None]:
|
|||||||
return ("ok", None)
|
return ("ok", None)
|
||||||
|
|
||||||
|
|
||||||
def _inspect_archive(path: Path):
|
def _inspect_archive(path: Path, ext: str):
|
||||||
"""Return (total_uncompressed_bytes | None, first_bad_member | None)
|
"""Return (total_uncompressed_bytes | None, first_bad_member | None)
|
||||||
for the archive. Format detected by extension OR magic bytes (so a
|
for the archive. Format-specific; raises on a structurally-broken
|
||||||
mis-named archive is still bomb-guarded + integrity-tested, matching the
|
container (caught by the child as a clean rejection)."""
|
||||||
extractor's gate); raises on a structurally-broken container (caught by the
|
if ext in (".zip", ".cbz"):
|
||||||
child as a clean rejection)."""
|
|
||||||
from ..services.archive_extractor import detect_archive_format
|
|
||||||
|
|
||||||
fmt = detect_archive_format(path)
|
|
||||||
if fmt == "zip":
|
|
||||||
import zipfile
|
import zipfile
|
||||||
|
|
||||||
with zipfile.ZipFile(path) as zf:
|
with zipfile.ZipFile(path) as zf:
|
||||||
total = sum(zi.file_size for zi in zf.infolist())
|
total = sum(zi.file_size for zi in zf.infolist())
|
||||||
return total, zf.testzip()
|
return total, zf.testzip()
|
||||||
if fmt == "rar":
|
if ext == ".rar":
|
||||||
import rarfile
|
import rarfile
|
||||||
|
|
||||||
with rarfile.RarFile(path) as rf:
|
with rarfile.RarFile(path) as rf:
|
||||||
total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist())
|
total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist())
|
||||||
rf.testrar()
|
rf.testrar()
|
||||||
return total, None
|
return total, None
|
||||||
if fmt == "7z":
|
if ext == ".7z":
|
||||||
import py7zr
|
import py7zr
|
||||||
|
|
||||||
with py7zr.SevenZipFile(path, "r") as zf:
|
with py7zr.SevenZipFile(path, "r") as zf:
|
||||||
@@ -191,5 +187,5 @@ def _inspect_archive(path: Path):
|
|||||||
total = getattr(info, "uncompressed", None)
|
total = getattr(info, "uncompressed", None)
|
||||||
ok = zf.test() # True / None when all members pass
|
ok = zf.test() # True / None when all members pass
|
||||||
return total, (None if ok in (True, None) else "7z test reported corruption")
|
return total, (None if ok in (True, None) else "7z test reported corruption")
|
||||||
# Not a recognised archive — nothing to test; treat as clean.
|
# Unknown extension — nothing to test; treat as clean.
|
||||||
return None, None
|
return None, None
|
||||||
|
|||||||
@@ -35,15 +35,6 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- ./backend:/app/backend
|
- ./backend:/app/backend
|
||||||
|
|
||||||
maintenance-long:
|
|
||||||
build:
|
|
||||||
context: .
|
|
||||||
dockerfile: Dockerfile
|
|
||||||
environment:
|
|
||||||
LOG_LEVEL: DEBUG
|
|
||||||
volumes:
|
|
||||||
- ./backend:/app/backend
|
|
||||||
|
|
||||||
ml-worker:
|
ml-worker:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: .
|
||||||
|
|||||||
+3
-32
@@ -27,17 +27,6 @@ services:
|
|||||||
POSTGRES_DB: ${DB_NAME:-fabledcurator}
|
POSTGRES_DB: ${DB_NAME:-fabledcurator}
|
||||||
volumes:
|
volumes:
|
||||||
- postgres_data:/var/lib/postgresql/data
|
- postgres_data:/var/lib/postgresql/data
|
||||||
# Docker's default /dev/shm is 64MB; VACUUM (ANALYZE) + parallel queries
|
|
||||||
# allocate a POSIX dynamic-shared-memory segment in /dev/shm and fail with
|
|
||||||
# "could not resize shared memory segment ... No space left on device"
|
|
||||||
# (operator-flagged 2026-06-07, vacuum_analyze on import_task needed 67MB).
|
|
||||||
# NOTE: a `shm_size:` key is SILENTLY IGNORED under Docker Swarm
|
|
||||||
# (`docker stack deploy` — the prod deployment here), so size /dev/shm via
|
|
||||||
# a tmpfs mount instead — honored by both Swarm and plain Compose.
|
|
||||||
- type: tmpfs
|
|
||||||
target: /dev/shm
|
|
||||||
tmpfs:
|
|
||||||
size: 536870912 # 512 MB
|
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-fabledcurator}"]
|
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-fabledcurator}"]
|
||||||
interval: 10s
|
interval: 10s
|
||||||
@@ -63,6 +52,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- ./images:/images
|
- ./images:/images
|
||||||
- ./import:/import
|
- ./import:/import
|
||||||
|
- ./downloads:/downloads
|
||||||
# FC-5 legacy migration: bind-mount the host's ImageRepo images dir
|
# FC-5 legacy migration: bind-mount the host's ImageRepo images dir
|
||||||
# under /import (FC's existing filesystem scan picks them up). Read-only
|
# under /import (FC's existing filesystem scan picks them up). Read-only
|
||||||
# is sufficient — FC copies into /images during the scan. The worker +
|
# is sufficient — FC copies into /images during the scan. The worker +
|
||||||
@@ -81,11 +71,10 @@ services:
|
|||||||
<<: *app_env
|
<<: *app_env
|
||||||
CELERY_QUEUES: default,import,thumbnail,download
|
CELERY_QUEUES: default,import,thumbnail,download
|
||||||
CELERY_CONCURRENCY: "2"
|
CELERY_CONCURRENCY: "2"
|
||||||
# /downloads dropped — nothing in the app references it (operator-flagged
|
|
||||||
# 2026-06-07: it wasn't mapped in prod and everything worked).
|
|
||||||
volumes:
|
volumes:
|
||||||
- ./images:/images
|
- ./images:/images
|
||||||
- ./import:/import
|
- ./import:/import
|
||||||
|
- ./downloads:/downloads
|
||||||
depends_on:
|
depends_on:
|
||||||
postgres: { condition: service_healthy }
|
postgres: { condition: service_healthy }
|
||||||
redis: { condition: service_healthy }
|
redis: { condition: service_healthy }
|
||||||
@@ -99,25 +88,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- ./images:/images
|
- ./images:/images
|
||||||
- ./import:/import
|
- ./import:/import
|
||||||
depends_on:
|
- ./downloads:/downloads
|
||||||
postgres: { condition: service_healthy }
|
|
||||||
redis: { condition: service_healthy }
|
|
||||||
|
|
||||||
# Dedicated lane for long one-shot maintenance (DB backups, library audits,
|
|
||||||
# admin maintenance). Kept off the scheduler's quick `maintenance` lane so a
|
|
||||||
# 30-min backup or a multi-chunk audit can never starve the 5-min recovery
|
|
||||||
# sweeps / vacuum (operator-flagged 2026-06-07). One slot — these are heavy.
|
|
||||||
maintenance-long:
|
|
||||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
|
||||||
command: ["worker"]
|
|
||||||
environment:
|
|
||||||
<<: *app_env
|
|
||||||
CELERY_QUEUES: maintenance_long
|
|
||||||
CELERY_CONCURRENCY: "1"
|
|
||||||
# Only /images: backups write to /images/_backups, audits read /images, and
|
|
||||||
# the admin tasks (re-extract/cascade-delete/normalize) operate on /images.
|
|
||||||
volumes:
|
|
||||||
- ./images:/images
|
|
||||||
depends_on:
|
depends_on:
|
||||||
postgres: { condition: service_healthy }
|
postgres: { condition: service_healthy }
|
||||||
redis: { condition: service_healthy }
|
redis: { condition: service_healthy }
|
||||||
|
|||||||
@@ -11,8 +11,6 @@
|
|||||||
<PipelineStatusChip />
|
<PipelineStatusChip />
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Desktop: inline links, centered. Hidden on mobile (see media query),
|
|
||||||
where they fold into the hamburger menu on the right. -->
|
|
||||||
<nav class="fc-links">
|
<nav class="fc-links">
|
||||||
<RouterLink
|
<RouterLink
|
||||||
v-for="r in navRoutes"
|
v-for="r in navRoutes"
|
||||||
@@ -22,31 +20,9 @@
|
|||||||
>{{ r.meta.title }}</RouterLink>
|
>{{ r.meta.title }}</RouterLink>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
<div class="fc-nav-right">
|
|
||||||
<!-- Per-view contextual actions teleport here (Showcase: Shuffle,
|
<!-- Per-view contextual actions teleport here (Showcase: Shuffle,
|
||||||
Gallery: Select). TopNav owns the slot, not its contents. -->
|
Gallery: Select). TopNav owns the slot, not its contents. -->
|
||||||
<div id="fc-nav-actions" class="fc-nav-actions" />
|
<div id="fc-nav-actions" class="fc-nav-actions" />
|
||||||
|
|
||||||
<!-- Mobile nav: the link row collides with brand + actions below ~768px
|
|
||||||
(7+ links in one flex row), so collapse it into a menu. -->
|
|
||||||
<v-menu location="bottom end">
|
|
||||||
<template #activator="{ props: menuProps }">
|
|
||||||
<v-btn
|
|
||||||
v-bind="menuProps"
|
|
||||||
icon="mdi-menu" variant="text" size="small"
|
|
||||||
class="fc-nav-burger" aria-label="Menu"
|
|
||||||
/>
|
|
||||||
</template>
|
|
||||||
<v-list density="compact" min-width="180">
|
|
||||||
<v-list-item
|
|
||||||
v-for="r in navRoutes"
|
|
||||||
:key="r.name"
|
|
||||||
:to="{ name: r.name }"
|
|
||||||
:title="r.meta.title"
|
|
||||||
/>
|
|
||||||
</v-list>
|
|
||||||
</v-menu>
|
|
||||||
</div>
|
|
||||||
</header>
|
</header>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
@@ -161,7 +137,7 @@ const health = computed(() => {
|
|||||||
align-items: center;
|
align-items: center;
|
||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
}
|
}
|
||||||
.fc-nav-right {
|
.fc-nav-actions {
|
||||||
flex: 1 1 0;
|
flex: 1 1 0;
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -169,22 +145,4 @@ const health = computed(() => {
|
|||||||
justify-content: flex-end;
|
justify-content: flex-end;
|
||||||
gap: 0.5rem;
|
gap: 0.5rem;
|
||||||
}
|
}
|
||||||
.fc-nav-actions {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 0.5rem;
|
|
||||||
}
|
|
||||||
/* The hamburger only exists on mobile; the inline links carry desktop. */
|
|
||||||
.fc-nav-burger { display: none; }
|
|
||||||
|
|
||||||
@media (max-width: 768px) {
|
|
||||||
.fc-topnav { gap: 0.5rem; padding: 0.6rem 0.75rem; }
|
|
||||||
/* Fold the link row into the hamburger menu. */
|
|
||||||
.fc-links { display: none; }
|
|
||||||
.fc-nav-burger { display: inline-flex; }
|
|
||||||
}
|
|
||||||
@media (max-width: 480px) {
|
|
||||||
/* Reclaim width on the smallest phones — the glyph alone still brands. */
|
|
||||||
.fc-brand__text { display: none; }
|
|
||||||
}
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -112,16 +112,6 @@ onMounted(async () => {
|
|||||||
await store.loadDefaults()
|
await store.loadDefaults()
|
||||||
threshold.value = store.defaults.single_color_threshold
|
threshold.value = store.defaults.single_color_threshold
|
||||||
tolerance.value = store.defaults.single_color_tolerance
|
tolerance.value = store.defaults.single_color_tolerance
|
||||||
// Reconnect to this rule's latest run so a scan started before navigating
|
|
||||||
// away keeps showing progress / its result on return (the scan itself runs
|
|
||||||
// backend-side regardless).
|
|
||||||
try {
|
|
||||||
const latest = await store.latestAuditForRule('single_color')
|
|
||||||
if (latest) {
|
|
||||||
audit.value = latest
|
|
||||||
if (latest.status === 'running') startPoll(latest.id)
|
|
||||||
}
|
|
||||||
} catch { /* non-fatal — card still works for a fresh scan */ }
|
|
||||||
})
|
})
|
||||||
|
|
||||||
onUnmounted(() => stopPoll())
|
onUnmounted(() => stopPoll())
|
||||||
|
|||||||
@@ -97,16 +97,6 @@ let pollTimer = null
|
|||||||
onMounted(async () => {
|
onMounted(async () => {
|
||||||
await store.loadDefaults()
|
await store.loadDefaults()
|
||||||
threshold.value = store.defaults.transparency_threshold
|
threshold.value = store.defaults.transparency_threshold
|
||||||
// Reconnect to this rule's latest run so a scan started before navigating
|
|
||||||
// away keeps showing progress / its result on return (the scan itself runs
|
|
||||||
// backend-side regardless).
|
|
||||||
try {
|
|
||||||
const latest = await store.latestAuditForRule('transparency')
|
|
||||||
if (latest) {
|
|
||||||
audit.value = latest
|
|
||||||
if (latest.status === 'running') startPoll(latest.id)
|
|
||||||
}
|
|
||||||
} catch { /* non-fatal — card still works for a fresh scan */ }
|
|
||||||
})
|
})
|
||||||
|
|
||||||
onUnmounted(() => stopPoll())
|
onUnmounted(() => stopPoll())
|
||||||
|
|||||||
@@ -8,15 +8,6 @@
|
|||||||
<div v-if="card.preview_thumbnails.length === 0" class="fc-artistcard__noimg">
|
<div v-if="card.preview_thumbnails.length === 0" class="fc-artistcard__noimg">
|
||||||
No preview
|
No preview
|
||||||
</div>
|
</div>
|
||||||
<!-- Accent pill in the corner when this artist has content imported
|
|
||||||
since the operator last opened their detail view. Caps at 99+
|
|
||||||
to keep the layout compact; the actual count appears in the
|
|
||||||
banner inside ArtistView. -->
|
|
||||||
<span
|
|
||||||
v-if="(card.unseen_count || 0) > 0"
|
|
||||||
class="fc-artistcard__unseen"
|
|
||||||
:aria-label="`${card.unseen_count} new since last visit`"
|
|
||||||
>+{{ card.unseen_count > 99 ? '99+' : card.unseen_count }}</span>
|
|
||||||
</div>
|
</div>
|
||||||
<v-card-text class="fc-artistcard__body">
|
<v-card-text class="fc-artistcard__body">
|
||||||
<div class="fc-artistcard__name">{{ card.name }}</div>
|
<div class="fc-artistcard__name">{{ card.name }}</div>
|
||||||
@@ -46,30 +37,14 @@ function onCardClick() {
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-artistcard { cursor: pointer; }
|
.fc-artistcard { cursor: pointer; }
|
||||||
.fc-artistcard__previews {
|
.fc-artistcard__previews {
|
||||||
position: relative;
|
|
||||||
display: grid; grid-template-columns: repeat(3, 1fr);
|
display: grid; grid-template-columns: repeat(3, 1fr);
|
||||||
gap: 2px; aspect-ratio: 3 / 1;
|
gap: 2px; aspect-ratio: 3 / 1;
|
||||||
/* Floor so tall source images can't escape the slot. NO max-height: an
|
/* Explicit floor + ceiling so tall source images can't escape the
|
||||||
aspect-ratio box capped by max-height shrinks its own WIDTH to keep the
|
preview slot even on browsers where aspect-ratio doesn't compute. */
|
||||||
ratio, leaving dead space beside the previews on a wide (single-column)
|
min-height: 150px; max-height: 220px;
|
||||||
card. The grid below keeps columns ≲400px so the strip stays a sane height. */
|
|
||||||
min-height: 120px;
|
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
background: rgb(var(--v-theme-surface-light));
|
background: rgb(var(--v-theme-surface-light));
|
||||||
}
|
}
|
||||||
.fc-artistcard__unseen {
|
|
||||||
position: absolute;
|
|
||||||
top: 6px; right: 6px;
|
|
||||||
display: inline-flex; align-items: center;
|
|
||||||
padding: 2px 8px;
|
|
||||||
font-size: 11px; font-weight: 700; letter-spacing: 0.02em;
|
|
||||||
font-variant-numeric: tabular-nums;
|
|
||||||
color: rgb(var(--v-theme-on-accent, 0, 0, 0));
|
|
||||||
background: rgb(var(--v-theme-accent));
|
|
||||||
border-radius: 999px;
|
|
||||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
|
|
||||||
pointer-events: none;
|
|
||||||
}
|
|
||||||
.fc-artistcard__previews img {
|
.fc-artistcard__previews img {
|
||||||
display: block;
|
display: block;
|
||||||
width: 100%; height: 100%;
|
width: 100%; height: 100%;
|
||||||
|
|||||||
@@ -6,6 +6,7 @@
|
|||||||
v-for="item in col" :key="item.id"
|
v-for="item in col" :key="item.id"
|
||||||
class="fc-masonry__item"
|
class="fc-masonry__item"
|
||||||
:class="{ 'fc-masonry__item--anim': shouldAnimate(item) }"
|
:class="{ 'fc-masonry__item--anim': shouldAnimate(item) }"
|
||||||
|
:style="itemStyle(item)"
|
||||||
type="button"
|
type="button"
|
||||||
@click="$emit('open', item.id)"
|
@click="$emit('open', item.id)"
|
||||||
>
|
>
|
||||||
@@ -65,6 +66,12 @@ function shouldAnimate(item) {
|
|||||||
return idx !== undefined && idx >= props.animateFromIndex
|
return idx !== undefined && idx >= props.animateFromIndex
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function itemStyle(item) {
|
||||||
|
if (!shouldAnimate(item)) return {}
|
||||||
|
const idx = idxById.value.get(item.id) - props.animateFromIndex
|
||||||
|
return { '--stagger-index': idx }
|
||||||
|
}
|
||||||
|
|
||||||
function aspectStyle(item) {
|
function aspectStyle(item) {
|
||||||
const w = Number(item.width)
|
const w = Number(item.width)
|
||||||
const h = Number(item.height)
|
const h = Number(item.height)
|
||||||
@@ -110,15 +117,12 @@ useInfiniteScroll(sentinelEl, () => {
|
|||||||
.fc-masonry__end { text-align: center; padding: 32px 0; }
|
.fc-masonry__end { text-align: center; padding: 32px 0; }
|
||||||
|
|
||||||
/* Cascade entry: each tile flips up out of a backward tilt and settles
|
/* Cascade entry: each tile flips up out of a backward tilt and settles
|
||||||
into place — more pronounced than a plain fade so the showcase reads as an
|
into place, one at a time — more pronounced than a plain fade so the
|
||||||
"experience" (operator-flagged 2026-05-28). The reveal is paced entirely by
|
showcase reads as an "experience" (operator-flagged 2026-05-28). The
|
||||||
the store, which pushes one fully-decoded item at a time (showcase.js); each
|
`both` fill holds the hidden/tilted 0% state until each tile's staggered
|
||||||
tile therefore animates the instant it mounts, with NO per-index CSS delay —
|
turn; the cubic-bezier overshoots slightly past flat then settles.
|
||||||
the old `animation-delay: index×70ms` compounded on top of the insert
|
Honors prefers-reduced-motion. Tunables: tilt (-28deg), stagger (70ms),
|
||||||
cadence and made the cascade drag and desync as it grew (operator-flagged
|
duration (0.6s). */
|
||||||
2026-06-04). The `both` fill holds the hidden/tilted 0% state until mount;
|
|
||||||
the cubic-bezier overshoots slightly past flat then settles. Honors
|
|
||||||
prefers-reduced-motion. Tunables: tilt (-28deg), duration (0.6s). */
|
|
||||||
@keyframes fc-masonry-item-in {
|
@keyframes fc-masonry-item-in {
|
||||||
0% {
|
0% {
|
||||||
opacity: 0;
|
opacity: 0;
|
||||||
@@ -134,6 +138,7 @@ useInfiniteScroll(sentinelEl, () => {
|
|||||||
transform-origin: center top;
|
transform-origin: center top;
|
||||||
backface-visibility: hidden;
|
backface-visibility: hidden;
|
||||||
animation: fc-masonry-item-in 0.6s cubic-bezier(0.34, 1.45, 0.64, 1) both;
|
animation: fc-masonry-item-in 0.6s cubic-bezier(0.34, 1.45, 0.64, 1) both;
|
||||||
|
animation-delay: calc(var(--stagger-index, 0) * 70ms);
|
||||||
}
|
}
|
||||||
@media (prefers-reduced-motion: reduce) {
|
@media (prefers-reduced-motion: reduce) {
|
||||||
.fc-masonry__item--anim {
|
.fc-masonry__item--anim {
|
||||||
|
|||||||
@@ -55,12 +55,6 @@
|
|||||||
/>
|
/>
|
||||||
</template>
|
</template>
|
||||||
<v-list density="compact">
|
<v-list density="compact">
|
||||||
<v-list-item
|
|
||||||
v-if="card.kind === 'character'"
|
|
||||||
title="Set fandom…"
|
|
||||||
prepend-icon="mdi-book-open-page-variant"
|
|
||||||
@click="$emit('set-fandom', card)"
|
|
||||||
/>
|
|
||||||
<v-list-item
|
<v-list-item
|
||||||
title="Merge with…"
|
title="Merge with…"
|
||||||
prepend-icon="mdi-call-merge"
|
prepend-icon="mdi-call-merge"
|
||||||
@@ -84,9 +78,7 @@
|
|||||||
import { ref } from 'vue'
|
import { ref } from 'vue'
|
||||||
|
|
||||||
const props = defineProps({ card: { type: Object, required: true } })
|
const props = defineProps({ card: { type: Object, required: true } })
|
||||||
const emit = defineEmits([
|
const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
|
||||||
'open', 'rename', 'manage', 'read', 'merge-with', 'delete', 'set-fandom',
|
|
||||||
])
|
|
||||||
|
|
||||||
const editing = ref(false)
|
const editing = ref(false)
|
||||||
const draft = ref('')
|
const draft = ref('')
|
||||||
|
|||||||
@@ -186,9 +186,7 @@ async function onDeleteConfirm(token) {
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-bulk-panel {
|
.fc-bulk-panel {
|
||||||
position: fixed; top: 0; right: 0;
|
position: fixed; top: 0; right: 0;
|
||||||
/* min(320px, 90vw): on phones the panel never swallows the whole screen —
|
width: 320px; height: 100vh; z-index: 1100;
|
||||||
leaves a sliver of the gallery visible behind it. */
|
|
||||||
width: min(320px, 90vw); height: 100vh; z-index: 1100;
|
|
||||||
background: rgb(var(--v-theme-surface));
|
background: rgb(var(--v-theme-surface));
|
||||||
border-left: 1px solid rgb(var(--v-theme-surface-light));
|
border-left: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
box-shadow: -2px 0 16px rgba(0, 0, 0, 0.4);
|
box-shadow: -2px 0 16px rgba(0, 0, 0, 0.4);
|
||||||
|
|||||||
@@ -1,153 +0,0 @@
|
|||||||
<template>
|
|
||||||
<div class="fc-facets">
|
|
||||||
<v-progress-linear
|
|
||||||
v-if="store.facetsLoading" indeterminate color="accent"
|
|
||||||
class="fc-facets__bar" height="2"
|
|
||||||
/>
|
|
||||||
|
|
||||||
<div class="fc-facets__group">
|
|
||||||
<span class="fc-facets__label">Platform</span>
|
|
||||||
<div class="fc-facets__chips">
|
|
||||||
<v-chip
|
|
||||||
v-for="p in platformOptions" :key="p.key"
|
|
||||||
size="small" label
|
|
||||||
:variant="p.key === store.filter.platform ? 'flat' : 'tonal'"
|
|
||||||
:color="p.key === store.filter.platform ? 'accent' : undefined"
|
|
||||||
@click="selectPlatform(p.key)"
|
|
||||||
>{{ p.label }}<span class="fc-facets__count">{{ p.count }}</span></v-chip>
|
|
||||||
<span v-if="!platformOptions.length" class="fc-facets__empty">none</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-facets__group">
|
|
||||||
<span class="fc-facets__label">Curation</span>
|
|
||||||
<div class="fc-facets__chips">
|
|
||||||
<v-chip
|
|
||||||
size="small" label
|
|
||||||
:variant="store.filter.untagged ? 'flat' : 'tonal'"
|
|
||||||
:color="store.filter.untagged ? 'accent' : undefined"
|
|
||||||
@click="toggleFlag('untagged')"
|
|
||||||
>Untagged<span class="fc-facets__count">{{ facetCount('untagged') }}</span></v-chip>
|
|
||||||
<v-chip
|
|
||||||
size="small" label
|
|
||||||
:variant="store.filter.no_artist ? 'flat' : 'tonal'"
|
|
||||||
:color="store.filter.no_artist ? 'accent' : undefined"
|
|
||||||
@click="toggleFlag('no_artist')"
|
|
||||||
>No artist<span class="fc-facets__count">{{ facetCount('no_artist') }}</span></v-chip>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-facets__group">
|
|
||||||
<span class="fc-facets__label">Date</span>
|
|
||||||
<v-text-field
|
|
||||||
type="date" density="compact" hide-details variant="outlined"
|
|
||||||
:model-value="store.filter.date_from" :min="dateMin" :max="dateMax"
|
|
||||||
class="fc-facets__date"
|
|
||||||
@update:model-value="setDate('date_from', $event)"
|
|
||||||
/>
|
|
||||||
<span class="fc-facets__dash">–</span>
|
|
||||||
<v-text-field
|
|
||||||
type="date" density="compact" hide-details variant="outlined"
|
|
||||||
:model-value="store.filter.date_to" :min="dateMin" :max="dateMax"
|
|
||||||
class="fc-facets__date"
|
|
||||||
@update:model-value="setDate('date_to', $event)"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed, onBeforeUnmount, onMounted, watch } from 'vue'
|
|
||||||
import { useRouter } from 'vue-router'
|
|
||||||
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
|
|
||||||
|
|
||||||
// Mirrors backend gallery_service.UNSOURCED_PLATFORM — the sentinel that
|
|
||||||
// selects filesystem-imported content (the null/"no platform" bucket).
|
|
||||||
const UNSOURCED = '__unsourced__'
|
|
||||||
|
|
||||||
const store = useGalleryStore()
|
|
||||||
const router = useRouter()
|
|
||||||
|
|
||||||
const platformOptions = computed(() =>
|
|
||||||
(store.facets?.platforms || []).map((p) => ({
|
|
||||||
key: p.value === null ? UNSOURCED : p.value,
|
|
||||||
label: p.value === null ? 'No platform' : p.value,
|
|
||||||
count: p.count,
|
|
||||||
}))
|
|
||||||
)
|
|
||||||
|
|
||||||
function facetCount(flag) {
|
|
||||||
const v = store.facets?.[flag]
|
|
||||||
return v == null ? '–' : v
|
|
||||||
}
|
|
||||||
|
|
||||||
const dateMin = computed(() => (store.facets?.date_min || '').slice(0, 10) || undefined)
|
|
||||||
const dateMax = computed(() => (store.facets?.date_max || '').slice(0, 10) || undefined)
|
|
||||||
|
|
||||||
// Single write path, shared format with the bar: clone → patch → URL. The
|
|
||||||
// route watcher in GalleryView reloads the store (and our filter-watch below
|
|
||||||
// refetches the facet counts for the new scope).
|
|
||||||
function pushPatch(patch) {
|
|
||||||
const n = cloneFilter(store.filter)
|
|
||||||
Object.assign(n, patch)
|
|
||||||
router.push({ name: 'gallery', query: filterToQuery(n) })
|
|
||||||
}
|
|
||||||
|
|
||||||
function selectPlatform(key) {
|
|
||||||
// Single-select, click-active-to-clear (the platform param is one value).
|
|
||||||
pushPatch({ platform: store.filter.platform === key ? null : key })
|
|
||||||
}
|
|
||||||
function toggleFlag(name) {
|
|
||||||
pushPatch({ [name]: !store.filter[name] })
|
|
||||||
}
|
|
||||||
function setDate(field, val) {
|
|
||||||
pushPatch({ [field]: val || null })
|
|
||||||
}
|
|
||||||
|
|
||||||
// Fetch when the panel opens; refetch (debounced) whenever the active filter
|
|
||||||
// changes so the live counts track the current scope. Never fires on scroll —
|
|
||||||
// the panel is the only caller of loadFacets.
|
|
||||||
onMounted(() => store.loadFacets())
|
|
||||||
|
|
||||||
let debounce = null
|
|
||||||
watch(() => store.filter, () => {
|
|
||||||
if (debounce) clearTimeout(debounce)
|
|
||||||
debounce = setTimeout(() => store.loadFacets(), 250)
|
|
||||||
})
|
|
||||||
onBeforeUnmount(() => { if (debounce) clearTimeout(debounce) })
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-facets {
|
|
||||||
position: relative;
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
gap: 16px;
|
|
||||||
padding: 4px 12px 12px;
|
|
||||||
}
|
|
||||||
.fc-facets__bar { position: absolute; top: 0; left: 0; right: 0; }
|
|
||||||
.fc-facets__group { display: flex; align-items: center; gap: 8px; }
|
|
||||||
.fc-facets__label {
|
|
||||||
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
}
|
|
||||||
.fc-facets__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
|
||||||
/* The count rides inside the chip as a dimmer trailing number. */
|
|
||||||
.fc-facets__count {
|
|
||||||
margin-left: 6px;
|
|
||||||
opacity: 0.7;
|
|
||||||
font-variant-numeric: tabular-nums;
|
|
||||||
font-size: 0.78em;
|
|
||||||
}
|
|
||||||
.fc-facets__empty { color: rgb(var(--v-theme-on-surface-variant)); font-size: 0.8rem; }
|
|
||||||
.fc-facets__date { max-width: 160px; }
|
|
||||||
.fc-facets__dash { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
|
|
||||||
/* Phones: let each facet group wrap and the side-by-side date inputs grow to
|
|
||||||
full width so they don't overflow a ~360px viewport. */
|
|
||||||
@media (max-width: 480px) {
|
|
||||||
.fc-facets__group { flex-wrap: wrap; }
|
|
||||||
.fc-facets__date { max-width: none; flex: 1 1 9rem; }
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -1,269 +0,0 @@
|
|||||||
<template>
|
|
||||||
<div class="fc-filterbar-wrap">
|
|
||||||
<div class="fc-filterbar">
|
|
||||||
<v-autocomplete
|
|
||||||
v-model="selected"
|
|
||||||
:items="searchItems"
|
|
||||||
:loading="searchLoading"
|
|
||||||
item-title="name" item-value="value"
|
|
||||||
no-filter clearable hide-details density="compact" variant="outlined"
|
|
||||||
placeholder="Filter by tag or artist…"
|
|
||||||
prepend-inner-icon="mdi-filter-variant"
|
|
||||||
class="fc-filterbar__search"
|
|
||||||
@update:search="onSearch"
|
|
||||||
@update:model-value="onPick"
|
|
||||||
>
|
|
||||||
<template #item="{ props: itemProps, item }">
|
|
||||||
<v-list-item v-bind="itemProps" :title="item.raw.name">
|
|
||||||
<template #prepend>
|
|
||||||
<v-icon size="small">{{ iconFor(item.raw) }}</v-icon>
|
|
||||||
</template>
|
|
||||||
<template #subtitle>
|
|
||||||
{{ item.raw.kind === 'artist' ? 'artist'
|
|
||||||
: (item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind) }}
|
|
||||||
</template>
|
|
||||||
</v-list-item>
|
|
||||||
</template>
|
|
||||||
</v-autocomplete>
|
|
||||||
|
|
||||||
<div class="fc-filterbar__chips">
|
|
||||||
<v-chip
|
|
||||||
v-for="id in store.filter.tag_ids" :key="`t${id}`"
|
|
||||||
size="small" closable :color="chipColor(id)" variant="tonal"
|
|
||||||
@click:close="removeTag(id)"
|
|
||||||
>{{ store.tagLabels[id] || `#${id}` }}</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-if="store.filter.artist_id"
|
|
||||||
size="small" closable color="accent" variant="tonal"
|
|
||||||
prepend-icon="mdi-account"
|
|
||||||
@click:close="clearArtist"
|
|
||||||
>{{ store.artistLabel || `Artist #${store.filter.artist_id}` }}</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-if="store.filter.similar_to"
|
|
||||||
size="small" closable color="accent" variant="tonal"
|
|
||||||
prepend-icon="mdi-image-multiple"
|
|
||||||
@click:close="clearSimilar"
|
|
||||||
>Similar to #{{ store.filter.similar_to }}</v-chip>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-spacer />
|
|
||||||
|
|
||||||
<v-btn-toggle
|
|
||||||
:model-value="store.filter.media_type ?? 'all'"
|
|
||||||
density="compact" mandatory variant="outlined" divided
|
|
||||||
@update:model-value="(v) => setMedia(v === 'all' ? null : v)"
|
|
||||||
>
|
|
||||||
<v-btn value="all" size="small">All</v-btn>
|
|
||||||
<v-btn value="image" size="small">Images</v-btn>
|
|
||||||
<v-btn value="video" size="small">Videos</v-btn>
|
|
||||||
</v-btn-toggle>
|
|
||||||
|
|
||||||
<!-- Sort is meaningless in similar-mode (results are distance-ranked). -->
|
|
||||||
<v-select
|
|
||||||
v-if="!store.filter.similar_to"
|
|
||||||
:model-value="store.filter.sort"
|
|
||||||
:items="SORTS"
|
|
||||||
density="compact" hide-details variant="outlined"
|
|
||||||
class="fc-filterbar__sort"
|
|
||||||
@update:model-value="setSort"
|
|
||||||
/>
|
|
||||||
|
|
||||||
<v-btn
|
|
||||||
:color="refineOpen ? 'accent' : undefined"
|
|
||||||
:variant="refineOpen || hasRefineFilters ? 'tonal' : 'text'"
|
|
||||||
size="small"
|
|
||||||
:append-icon="refineOpen ? 'mdi-chevron-up' : 'mdi-chevron-down'"
|
|
||||||
@click="toggleRefine"
|
|
||||||
>Refine{{ refineCount ? ` (${refineCount})` : '' }}</v-btn>
|
|
||||||
|
|
||||||
<v-btn
|
|
||||||
v-if="hasActiveFilters" variant="text" size="small"
|
|
||||||
prepend-icon="mdi-close" @click="clearAll"
|
|
||||||
>Clear</v-btn>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<GalleryFacetPanel v-if="refineOpen" />
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed, ref, watch } from 'vue'
|
|
||||||
import { useRouter } from 'vue-router'
|
|
||||||
import { useApi } from '../../composables/useApi.js'
|
|
||||||
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
|
|
||||||
import { useTagStore } from '../../stores/tags.js'
|
|
||||||
import GalleryFacetPanel from './GalleryFacetPanel.vue'
|
|
||||||
|
|
||||||
const store = useGalleryStore()
|
|
||||||
const tagStore = useTagStore()
|
|
||||||
const api = useApi()
|
|
||||||
const router = useRouter()
|
|
||||||
|
|
||||||
const SORTS = [
|
|
||||||
{ title: 'Newest first', value: 'newest' },
|
|
||||||
{ title: 'Oldest first', value: 'oldest' },
|
|
||||||
]
|
|
||||||
|
|
||||||
const selected = ref(null)
|
|
||||||
const searchItems = ref([])
|
|
||||||
const searchLoading = ref(false)
|
|
||||||
let debounce = null
|
|
||||||
|
|
||||||
// The faceted-refine sub-filters (platform / curation flags / date range).
|
|
||||||
const refineCount = computed(() => {
|
|
||||||
const f = store.filter
|
|
||||||
return (f.platform ? 1 : 0) + (f.untagged ? 1 : 0) + (f.no_artist ? 1 : 0)
|
|
||||||
+ (f.date_from ? 1 : 0) + (f.date_to ? 1 : 0)
|
|
||||||
})
|
|
||||||
const hasRefineFilters = computed(() => refineCount.value > 0)
|
|
||||||
|
|
||||||
const hasActiveFilters = computed(() =>
|
|
||||||
store.filter.tag_ids.length > 0 ||
|
|
||||||
store.filter.artist_id != null ||
|
|
||||||
store.filter.media_type != null ||
|
|
||||||
store.filter.sort !== 'newest' ||
|
|
||||||
store.filter.similar_to != null ||
|
|
||||||
hasRefineFilters.value
|
|
||||||
)
|
|
||||||
|
|
||||||
// Auto-open the panel when refine filters are present in the URL (deep-link /
|
|
||||||
// back button). The parent applies the query in its onMounted — after this
|
|
||||||
// child has set up — so watch for the transition rather than reading the
|
|
||||||
// initial (still-default) filter state.
|
|
||||||
const refineOpen = ref(false)
|
|
||||||
watch(hasRefineFilters, (v) => { if (v) refineOpen.value = true })
|
|
||||||
function toggleRefine() {
|
|
||||||
// The panel fetches facets itself on mount (and refetches on filter change);
|
|
||||||
// opening it is enough.
|
|
||||||
refineOpen.value = !refineOpen.value
|
|
||||||
}
|
|
||||||
|
|
||||||
function iconFor(raw) {
|
|
||||||
if (raw.kind === 'artist') return 'mdi-account'
|
|
||||||
return { character: 'mdi-account-circle', fandom: 'mdi-book-open-page-variant',
|
|
||||||
series: 'mdi-bookshelf' }[raw.kind] || 'mdi-tag'
|
|
||||||
}
|
|
||||||
function chipColor(id) {
|
|
||||||
// Tag chips use the same per-kind palette as the rest of the app; we only
|
|
||||||
// know the kind from the autocomplete pick, so fall back to a neutral tone.
|
|
||||||
return tagStore.colorFor(pickedKind.value[id] || 'general')
|
|
||||||
}
|
|
||||||
const pickedKind = ref({})
|
|
||||||
|
|
||||||
function onSearch(q) {
|
|
||||||
if (debounce) clearTimeout(debounce)
|
|
||||||
if (!q || !q.trim()) { searchItems.value = []; return }
|
|
||||||
debounce = setTimeout(async () => {
|
|
||||||
searchLoading.value = true
|
|
||||||
try {
|
|
||||||
const [tags, artists] = await Promise.all([
|
|
||||||
api.get('/api/tags/autocomplete', { params: { q, limit: 10 } }),
|
|
||||||
api.get('/api/artists/autocomplete', { params: { q, limit: 10 } }),
|
|
||||||
])
|
|
||||||
searchItems.value = [
|
|
||||||
...(artists || []).map((a) => ({
|
|
||||||
kind: 'artist', id: a.id, name: a.name, value: `artist:${a.id}`,
|
|
||||||
})),
|
|
||||||
...(tags || []).map((t) => ({
|
|
||||||
kind: t.kind, id: t.id, name: t.name, value: `tag:${t.id}`,
|
|
||||||
fandom_name: t.fandom_name,
|
|
||||||
})),
|
|
||||||
]
|
|
||||||
} catch {
|
|
||||||
searchItems.value = []
|
|
||||||
} finally {
|
|
||||||
searchLoading.value = false
|
|
||||||
}
|
|
||||||
}, 250)
|
|
||||||
}
|
|
||||||
|
|
||||||
function onPick(value) {
|
|
||||||
const item = searchItems.value.find((i) => i.value === value)
|
|
||||||
selected.value = null
|
|
||||||
searchItems.value = []
|
|
||||||
if (!item) return
|
|
||||||
if (item.kind === 'artist') {
|
|
||||||
store.noteArtistLabel(item.name)
|
|
||||||
pushFilter((n) => { n.artist_id = item.id })
|
|
||||||
} else {
|
|
||||||
store.noteTagLabel(item.id, item.name)
|
|
||||||
pickedKind.value = { ...pickedKind.value, [item.id]: item.kind }
|
|
||||||
pushFilter((n) => { if (!n.tag_ids.includes(item.id)) n.tag_ids.push(item.id) })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function removeTag(id) {
|
|
||||||
pushFilter((n) => { n.tag_ids = n.tag_ids.filter((t) => t !== id) })
|
|
||||||
}
|
|
||||||
function clearArtist() {
|
|
||||||
store.noteArtistLabel(null)
|
|
||||||
pushFilter((n) => { n.artist_id = null })
|
|
||||||
}
|
|
||||||
function clearSimilar() { pushFilter((n) => { n.similar_to = null }) }
|
|
||||||
function setMedia(m) { pushFilter((n) => { n.media_type = m }) }
|
|
||||||
function setSort(s) { pushFilter((n) => { n.sort = s }) }
|
|
||||||
function clearAll() { router.push({ name: 'gallery', query: {} }) }
|
|
||||||
|
|
||||||
// Single write path: clone the current filter, mutate, serialize to the URL.
|
|
||||||
// The route watcher in GalleryView applies it to the store and reloads.
|
|
||||||
// cloneFilter/filterToQuery are shared with GalleryFacetPanel so the refine
|
|
||||||
// sub-filters survive a bar push and vice versa.
|
|
||||||
function pushFilter(mutate) {
|
|
||||||
const n = cloneFilter(store.filter)
|
|
||||||
mutate(n)
|
|
||||||
router.push({ name: 'gallery', query: filterToQuery(n) })
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
/* The whole chrome (bar row + expandable refine panel) is one sticky,
|
|
||||||
frosted block pinned directly under the 64px TopNav and continuous with it. */
|
|
||||||
.fc-filterbar-wrap {
|
|
||||||
position: sticky;
|
|
||||||
top: 64px;
|
|
||||||
z-index: 5;
|
|
||||||
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
|
|
||||||
so the bar sits flush at 64px even at scroll 0 — without this it detaches
|
|
||||||
and a gap shows through when scrolled to the top. */
|
|
||||||
margin-top: -8px;
|
|
||||||
margin-bottom: 12px;
|
|
||||||
/* EXACT same gradiated obsidian (#14171A = 20,23,26) frost as the TopNav so
|
|
||||||
the two read as one continuous piece of chrome — images scroll visibly
|
|
||||||
under both. The nav's gradient fades to transparent at ITS bottom; this
|
|
||||||
bar re-darkens at its top, so a faint seam (the page/image showing through
|
|
||||||
the nav's transparent edge) separates them when scrolled to the very top,
|
|
||||||
while under-scroll they frost as one. */
|
|
||||||
background: linear-gradient(
|
|
||||||
to bottom,
|
|
||||||
rgba(20, 23, 26, 0.92) 0%,
|
|
||||||
rgba(20, 23, 26, 0.65) 60%,
|
|
||||||
rgba(20, 23, 26, 0) 100%
|
|
||||||
);
|
|
||||||
backdrop-filter: blur(2px);
|
|
||||||
-webkit-backdrop-filter: blur(2px);
|
|
||||||
}
|
|
||||||
.fc-filterbar {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 10px;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
padding: 10px 12px;
|
|
||||||
}
|
|
||||||
/* The inputs/toggles float on the haze: still translucent, but a touch more
|
|
||||||
opaque than the bar/nav so the controls stay legible. */
|
|
||||||
.fc-filterbar-wrap :deep(.v-field),
|
|
||||||
.fc-filterbar-wrap :deep(.v-btn-group) {
|
|
||||||
background-color: rgba(20, 23, 26, 0.72);
|
|
||||||
}
|
|
||||||
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
|
|
||||||
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
|
|
||||||
.fc-filterbar__sort { max-width: 150px; }
|
|
||||||
|
|
||||||
/* Phones: the search's 200px min-width jams the wrapping bar. Give search its
|
|
||||||
own full-width row and let sort grow; everything else wraps under it. */
|
|
||||||
@media (max-width: 600px) {
|
|
||||||
.fc-filterbar { gap: 8px; }
|
|
||||||
.fc-filterbar__search { min-width: 100%; max-width: none; }
|
|
||||||
.fc-filterbar__sort { max-width: none; flex: 1 1 auto; }
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -15,19 +15,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<!-- Similar-mode (and any non-date-grouped result set) returns no date
|
|
||||||
groups — the results are ranked, not chronological. Render them as a
|
|
||||||
single flat list in their given order rather than nothing. -->
|
|
||||||
<div
|
|
||||||
v-if="!store.dateGroups.length && store.images.length"
|
|
||||||
class="fc-gallery-grid__items"
|
|
||||||
>
|
|
||||||
<GalleryItem
|
|
||||||
v-for="img in store.images"
|
|
||||||
:key="img.id" :image="img" @open="$emit('open', img.id)"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div v-if="store.loading" class="fc-gallery-grid__sentinel">
|
<div v-if="store.loading" class="fc-gallery-grid__sentinel">
|
||||||
<v-progress-circular indeterminate color="accent" size="28" />
|
<v-progress-circular indeterminate color="accent" size="28" />
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -9,16 +9,11 @@
|
|||||||
>
|
>
|
||||||
<div class="fc-gallery-item__media">
|
<div class="fc-gallery-item__media">
|
||||||
<img
|
<img
|
||||||
v-if="!isVideo" ref="imgEl" :src="image.thumbnail_url"
|
v-if="!isVideo" :src="image.thumbnail_url" :alt="`Image ${image.id}`"
|
||||||
:alt="`Image ${image.id}`" loading="lazy"
|
loading="lazy" @error="onThumbError"
|
||||||
:class="{ 'is-loaded': loaded }"
|
|
||||||
@load="loaded = true" @error="onThumbError"
|
|
||||||
>
|
>
|
||||||
<div v-else class="fc-gallery-item__video-thumb">
|
<div v-else class="fc-gallery-item__video-thumb">
|
||||||
<img
|
<img :src="image.thumbnail_url" :alt="`Video ${image.id}`" loading="lazy">
|
||||||
ref="imgEl" :src="image.thumbnail_url" :alt="`Video ${image.id}`"
|
|
||||||
loading="lazy" :class="{ 'is-loaded': loaded }" @load="loaded = true"
|
|
||||||
>
|
|
||||||
<v-icon class="fc-gallery-item__video-badge" icon="mdi-play-circle" />
|
<v-icon class="fc-gallery-item__video-badge" icon="mdi-play-circle" />
|
||||||
</div>
|
</div>
|
||||||
<div v-if="thumbError" class="fc-gallery-item__placeholder">
|
<div v-if="thumbError" class="fc-gallery-item__placeholder">
|
||||||
@@ -43,7 +38,7 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, onMounted, ref } from 'vue'
|
import { computed, ref } from 'vue'
|
||||||
import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
|
import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
|
||||||
|
|
||||||
const props = defineProps({ image: { type: Object, required: true } })
|
const props = defineProps({ image: { type: Object, required: true } })
|
||||||
@@ -51,18 +46,6 @@ const emit = defineEmits(['open'])
|
|||||||
|
|
||||||
const sel = useGallerySelectionStore()
|
const sel = useGallerySelectionStore()
|
||||||
const thumbError = ref(false)
|
const thumbError = ref(false)
|
||||||
|
|
||||||
// Reveal each tile when ITS OWN thumbnail finishes loading (`@load`), not
|
|
||||||
// when its metadata batch lands — so tiles fade/flip in individually in
|
|
||||||
// load order instead of popping in together (operator-flagged 2026-06-04).
|
|
||||||
const loaded = ref(false)
|
|
||||||
const imgEl = ref(null)
|
|
||||||
onMounted(() => {
|
|
||||||
// A cached thumbnail can already be complete before @load binds; reflect
|
|
||||||
// that so the tile reveals instead of sitting invisible at opacity 0.
|
|
||||||
const el = imgEl.value
|
|
||||||
if (el && el.complete && el.naturalWidth > 0) loaded.value = true
|
|
||||||
})
|
|
||||||
const isVideo = computed(
|
const isVideo = computed(
|
||||||
() => props.image.mime && props.image.mime.startsWith('video/')
|
() => props.image.mime && props.image.mime.startsWith('video/')
|
||||||
)
|
)
|
||||||
@@ -99,30 +82,6 @@ function onThumbError() { thumbError.value = true }
|
|||||||
}
|
}
|
||||||
.fc-gallery-item__media img {
|
.fc-gallery-item__media img {
|
||||||
width: 100%; height: 100%; object-fit: cover; display: block;
|
width: 100%; height: 100%; object-fit: cover; display: block;
|
||||||
opacity: 0;
|
|
||||||
}
|
|
||||||
/* Entrance: each thumbnail flips up out of a slight backward tilt and
|
|
||||||
settles into place, played WHEN ITS OWN image finishes loading (the
|
|
||||||
`is-loaded` class) rather than on a batch/timer — so tiles cascade in
|
|
||||||
natural load order instead of popping in together. Mirrors the showcase
|
|
||||||
MasonryGrid entrance styling (operator-flagged 2026-06-04). */
|
|
||||||
.fc-gallery-item__media img.is-loaded {
|
|
||||||
animation: fc-gallery-item-in 0.5s cubic-bezier(0.34, 1.45, 0.64, 1) both;
|
|
||||||
}
|
|
||||||
@keyframes fc-gallery-item-in {
|
|
||||||
0% {
|
|
||||||
opacity: 0;
|
|
||||||
transform: perspective(1000px) rotateX(-22deg) translateY(20px) scale(0.96);
|
|
||||||
}
|
|
||||||
55% { opacity: 1; }
|
|
||||||
100% {
|
|
||||||
opacity: 1;
|
|
||||||
transform: perspective(1000px) rotateX(0) translateY(0) scale(1);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@media (prefers-reduced-motion: reduce) {
|
|
||||||
.fc-gallery-item__media img { opacity: 1; }
|
|
||||||
.fc-gallery-item__media img.is-loaded { animation: none; }
|
|
||||||
}
|
}
|
||||||
.fc-gallery-item__video-thumb { position: relative; height: 100%; }
|
.fc-gallery-item__video-thumb { position: relative; height: 100%; }
|
||||||
.fc-gallery-item__video-badge {
|
.fc-gallery-item__video-badge {
|
||||||
|
|||||||
@@ -2,45 +2,21 @@
|
|||||||
<v-card>
|
<v-card>
|
||||||
<v-card-title>Pick a fandom</v-card-title>
|
<v-card-title>Pick a fandom</v-card-title>
|
||||||
<v-card-text>
|
<v-card-text>
|
||||||
<!-- Keyboard flow (operator-specified 2026-06-07):
|
|
||||||
1. Focus lands here (driven by the dialog's @after-enter calling the
|
|
||||||
exposed focusSearch — `autofocus` is unreliable inside a v-dialog
|
|
||||||
because the focus-trap activates after mount and steals it).
|
|
||||||
2. Tab moves to the "new fandom" field below.
|
|
||||||
3. Selecting a fandom (click, or arrow+Enter to pick from the menu)
|
|
||||||
leaves it in the field; a SECOND Enter (menu closed) accepts it.
|
|
||||||
We bind v-model:menu so the Enter handler can tell "pick from the open
|
|
||||||
menu" (let Vuetify handle) apart from "accept the chosen value". -->
|
|
||||||
<v-autocomplete
|
<v-autocomplete
|
||||||
ref="fandomRef"
|
|
||||||
v-model="selectedId"
|
v-model="selectedId"
|
||||||
v-model:menu="menuOpen"
|
|
||||||
:items="store.fandomCache"
|
:items="store.fandomCache"
|
||||||
:item-title="(f) => f.name"
|
:item-title="(f) => f.name"
|
||||||
:item-value="(f) => f.id"
|
:item-value="(f) => f.id"
|
||||||
label="Fandom" clearable density="compact"
|
label="Fandom" clearable density="compact"
|
||||||
@keydown.enter.capture="onSearchEnter"
|
|
||||||
@keydown.tab="onSearchTab"
|
|
||||||
/>
|
/>
|
||||||
<v-divider class="my-3" />
|
<v-divider class="my-3" />
|
||||||
<p class="text-caption mb-2">Or create a new fandom:</p>
|
<p class="text-caption mb-2">Or create a new fandom:</p>
|
||||||
<div class="d-flex" style="gap: 8px;">
|
<div class="d-flex" style="gap: 8px;">
|
||||||
<!-- Enter creates; on success focus returns to the dropdown (onCreate),
|
<v-text-field v-model="newName" placeholder="New fandom name" density="compact" hide-details />
|
||||||
so the operator can immediately Enter-to-accept the new fandom. -->
|
|
||||||
<v-text-field
|
|
||||||
ref="newNameRef"
|
|
||||||
v-model="newName" placeholder="New fandom name"
|
|
||||||
density="compact" hide-details
|
|
||||||
@keydown.enter.prevent="onCreate"
|
|
||||||
/>
|
|
||||||
<v-btn :disabled="!newName.trim()" rounded="pill" @click="onCreate">Create</v-btn>
|
<v-btn :disabled="!newName.trim()" rounded="pill" @click="onCreate">Create</v-btn>
|
||||||
</div>
|
</div>
|
||||||
</v-card-text>
|
</v-card-text>
|
||||||
<v-card-actions>
|
<v-card-actions>
|
||||||
<!-- Not every character belongs to a fandom (original characters,
|
|
||||||
unsorted, etc.). "No fandom" creates the character unassigned;
|
|
||||||
a fandom can still be set later from the chip's kebab menu. -->
|
|
||||||
<v-btn variant="text" @click="onNoFandom">No fandom</v-btn>
|
|
||||||
<v-spacer />
|
<v-spacer />
|
||||||
<v-btn @click="$emit('cancel')">Cancel</v-btn>
|
<v-btn @click="$emit('cancel')">Cancel</v-btn>
|
||||||
<v-btn :disabled="!selectedId" color="primary" rounded="pill" @click="onConfirm">Use this fandom</v-btn>
|
<v-btn :disabled="!selectedId" color="primary" rounded="pill" @click="onConfirm">Use this fandom</v-btn>
|
||||||
@@ -49,7 +25,7 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { nextTick, onMounted, ref } from 'vue'
|
import { onMounted, ref } from 'vue'
|
||||||
import { useTagStore } from '../../stores/tags.js'
|
import { useTagStore } from '../../stores/tags.js'
|
||||||
|
|
||||||
const emit = defineEmits(['confirm', 'cancel'])
|
const emit = defineEmits(['confirm', 'cancel'])
|
||||||
@@ -57,69 +33,16 @@ const store = useTagStore()
|
|||||||
|
|
||||||
const selectedId = ref(null)
|
const selectedId = ref(null)
|
||||||
const newName = ref('')
|
const newName = ref('')
|
||||||
const menuOpen = ref(false)
|
|
||||||
const fandomRef = ref(null)
|
|
||||||
const newNameRef = ref(null)
|
|
||||||
|
|
||||||
// Always refresh on open so the list reflects fandoms created elsewhere (#712).
|
onMounted(() => { if (store.fandomCache.length === 0) store.loadFandoms() })
|
||||||
onMounted(() => store.loadFandoms())
|
|
||||||
|
|
||||||
// Exposed so the parent dialog can focus the search field on @after-enter —
|
async function onCreate() {
|
||||||
// the reliable point to grab focus (the dialog transition + focus-trap are done).
|
const f = await store.createFandom(newName.value.trim())
|
||||||
function focusSearch () {
|
|
||||||
nextTick(() => {
|
|
||||||
fandomRef.value?.focus?.()
|
|
||||||
// Keep the menu closed after a programmatic focus so the very next Enter
|
|
||||||
// accepts the value instead of being swallowed as a menu interaction.
|
|
||||||
menuOpen.value = false
|
|
||||||
})
|
|
||||||
}
|
|
||||||
defineExpose({ focusSearch })
|
|
||||||
|
|
||||||
async function onCreate () {
|
|
||||||
const name = newName.value.trim()
|
|
||||||
if (!name) return
|
|
||||||
const f = await store.createFandom(name)
|
|
||||||
selectedId.value = f.id
|
selectedId.value = f.id
|
||||||
newName.value = ''
|
newName.value = ''
|
||||||
// The created fandom is now the selection; hand focus back to the dropdown
|
|
||||||
// so a single Enter accepts it (operator-specified flow 2026-06-07).
|
|
||||||
focusSearch()
|
|
||||||
}
|
}
|
||||||
|
function onConfirm() {
|
||||||
function onConfirm () {
|
|
||||||
const f = store.fandomCache.find(x => x.id === selectedId.value)
|
const f = store.fandomCache.find(x => x.id === selectedId.value)
|
||||||
if (f) emit('confirm', f)
|
if (f) emit('confirm', f)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Enter on the search field — bound in the CAPTURE phase so this runs BEFORE
|
|
||||||
// Vuetify's own input keydown handler (which opens the menu on Enter). When the
|
|
||||||
// menu is open, let Vuetify pick the highlighted item (it sets selectedId +
|
|
||||||
// closes the menu). When it's closed and a fandom is already chosen, accept it
|
|
||||||
// and stop the event so Vuetify never (re)opens the menu — that re-open was the
|
|
||||||
// bug: Enter popped the dropdown instead of submitting (operator-flagged
|
|
||||||
// 2026-06-07).
|
|
||||||
function onSearchEnter (e) {
|
|
||||||
if (menuOpen.value) return
|
|
||||||
if (selectedId.value != null) {
|
|
||||||
e.preventDefault()
|
|
||||||
e.stopPropagation()
|
|
||||||
onConfirm()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tab from the search field goes to the "new fandom" text field (operator-
|
|
||||||
// specified). Shift+Tab keeps normal reverse traversal.
|
|
||||||
function onSearchTab (e) {
|
|
||||||
if (e.shiftKey) return
|
|
||||||
e.preventDefault()
|
|
||||||
menuOpen.value = false
|
|
||||||
nextTick(() => newNameRef.value?.focus?.())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create the character with no fandom. Emits null so the caller knows this
|
|
||||||
// was a deliberate "unassigned", not a cancel.
|
|
||||||
function onNoFandom () {
|
|
||||||
emit('confirm', null)
|
|
||||||
}
|
|
||||||
</script>
|
</script>
|
||||||
|
|||||||
@@ -1,130 +0,0 @@
|
|||||||
<template>
|
|
||||||
<v-card>
|
|
||||||
<v-card-title class="text-body-1">Fandom for “{{ tag.name }}”</v-card-title>
|
|
||||||
<v-card-text>
|
|
||||||
<template v-if="!collision">
|
|
||||||
<v-autocomplete
|
|
||||||
v-model="selectedId"
|
|
||||||
:items="store.fandomCache"
|
|
||||||
:item-title="(f) => f.name" :item-value="(f) => f.id"
|
|
||||||
label="Fandom" clearable density="compact"
|
|
||||||
autofocus
|
|
||||||
:hint="selectedId == null
|
|
||||||
? 'No fandom — the character will be unassigned.' : ''"
|
|
||||||
persistent-hint
|
|
||||||
/>
|
|
||||||
<v-divider class="my-3" />
|
|
||||||
<p class="text-caption mb-2">Or create a new fandom:</p>
|
|
||||||
<div class="d-flex" style="gap: 8px;">
|
|
||||||
<v-text-field
|
|
||||||
v-model="newName" placeholder="New fandom name"
|
|
||||||
density="compact" hide-details
|
|
||||||
@keydown.enter.prevent="onCreate"
|
|
||||||
/>
|
|
||||||
<v-btn
|
|
||||||
:disabled="!newName.trim() || busy" rounded="pill"
|
|
||||||
@click="onCreate"
|
|
||||||
>Create</v-btn>
|
|
||||||
</div>
|
|
||||||
<v-alert
|
|
||||||
v-if="error" type="error" variant="tonal" density="compact"
|
|
||||||
class="mt-3"
|
|
||||||
>{{ error }}</v-alert>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<template v-else>
|
|
||||||
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
|
|
||||||
A character named “{{ tag.name }}” already exists in that fandom.
|
|
||||||
</v-alert>
|
|
||||||
<p class="text-body-2">
|
|
||||||
Merge this tag into “{{ collision.target.name }}”?
|
|
||||||
{{ collision.source_image_count }} image
|
|
||||||
association{{ collision.source_image_count === 1 ? '' : 's' }}
|
|
||||||
will move over and this tag will be deleted{{
|
|
||||||
collision.will_alias ? ' (its name kept as a tagger alias)' : '' }}.
|
|
||||||
</p>
|
|
||||||
</template>
|
|
||||||
</v-card-text>
|
|
||||||
<v-card-actions>
|
|
||||||
<v-spacer />
|
|
||||||
<template v-if="!collision">
|
|
||||||
<v-btn variant="text" :disabled="busy" @click="$emit('cancel')">
|
|
||||||
Cancel
|
|
||||||
</v-btn>
|
|
||||||
<v-btn
|
|
||||||
color="primary" rounded="pill" :loading="busy"
|
|
||||||
:disabled="selectedId === (tag.fandom_id ?? null)"
|
|
||||||
@click="onSave"
|
|
||||||
>Save</v-btn>
|
|
||||||
</template>
|
|
||||||
<template v-else>
|
|
||||||
<v-btn variant="text" :disabled="busy" @click="collision = null">
|
|
||||||
Back
|
|
||||||
</v-btn>
|
|
||||||
<v-btn
|
|
||||||
color="warning" variant="flat" rounded="pill" :loading="busy"
|
|
||||||
@click="onConfirmMerge"
|
|
||||||
>Merge</v-btn>
|
|
||||||
</template>
|
|
||||||
</v-card-actions>
|
|
||||||
</v-card>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { onMounted, ref } from 'vue'
|
|
||||||
import { useTagStore } from '../../stores/tags.js'
|
|
||||||
|
|
||||||
const props = defineProps({ tag: { type: Object, required: true } })
|
|
||||||
const emit = defineEmits(['updated', 'cancel'])
|
|
||||||
|
|
||||||
const store = useTagStore()
|
|
||||||
|
|
||||||
const selectedId = ref(props.tag.fandom_id ?? null)
|
|
||||||
const newName = ref('')
|
|
||||||
const busy = ref(false)
|
|
||||||
const error = ref(null)
|
|
||||||
const collision = ref(null)
|
|
||||||
|
|
||||||
// Always refresh on open so the list reflects fandoms created elsewhere (#712).
|
|
||||||
onMounted(() => store.loadFandoms())
|
|
||||||
|
|
||||||
async function onCreate() {
|
|
||||||
const name = newName.value.trim()
|
|
||||||
if (!name) return
|
|
||||||
busy.value = true
|
|
||||||
error.value = null
|
|
||||||
try {
|
|
||||||
const f = await store.createFandom(name)
|
|
||||||
selectedId.value = f.id
|
|
||||||
newName.value = ''
|
|
||||||
} catch (e) {
|
|
||||||
error.value = e.message || String(e)
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function save(merge) {
|
|
||||||
busy.value = true
|
|
||||||
error.value = null
|
|
||||||
try {
|
|
||||||
const body = await store.setFandom(props.tag.id, selectedId.value, { merge })
|
|
||||||
emit('updated', body)
|
|
||||||
} catch (e) {
|
|
||||||
// 409 on first attempt → surface the merge confirmation; cross-fandom
|
|
||||||
// collisions can't go through the regular /merge endpoint, so the
|
|
||||||
// resolution is a second setFandom with merge: true.
|
|
||||||
if (!merge && e.status === 409 && e.body && e.body.target) {
|
|
||||||
collision.value = e.body
|
|
||||||
} else {
|
|
||||||
error.value = e.message || String(e)
|
|
||||||
collision.value = null
|
|
||||||
}
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function onSave() { save(false) }
|
|
||||||
function onConfirmMerge() { save(true) }
|
|
||||||
</script>
|
|
||||||
@@ -8,25 +8,6 @@
|
|||||||
<v-icon>mdi-close</v-icon>
|
<v-icon>mdi-close</v-icon>
|
||||||
</button>
|
</button>
|
||||||
|
|
||||||
<!-- C8: keyboard cheatsheet. '?' toggles; the corner hint advertises it. -->
|
|
||||||
<button
|
|
||||||
class="fc-viewer__help-hint" aria-label="Keyboard shortcuts (press ?)"
|
|
||||||
@click="showHelp = !showHelp"
|
|
||||||
>?</button>
|
|
||||||
<div v-if="showHelp" class="fc-viewer__help" @click.self="showHelp = false">
|
|
||||||
<div class="fc-viewer__help-card" role="dialog" aria-label="Keyboard shortcuts">
|
|
||||||
<h3>Keyboard shortcuts</h3>
|
|
||||||
<dl>
|
|
||||||
<div><dt>← / →</dt><dd>Previous / next image</dd></div>
|
|
||||||
<div><dt>T or /</dt><dd>Jump to the tag input</dd></div>
|
|
||||||
<div><dt>↑ / ↓</dt><dd>Move through tag suggestions</dd></div>
|
|
||||||
<div><dt>Enter / Tab</dt><dd>Accept the highlighted tag</dd></div>
|
|
||||||
<div><dt>Esc</dt><dd>Close a dialog, then the viewer</dd></div>
|
|
||||||
<div><dt>?</dt><dd>Toggle this help</dd></div>
|
|
||||||
</dl>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-chip
|
<v-chip
|
||||||
v-if="integrityBadge"
|
v-if="integrityBadge"
|
||||||
:color="integrityBadge.color"
|
:color="integrityBadge.color"
|
||||||
@@ -47,17 +28,12 @@
|
|||||||
<v-icon size="32">mdi-chevron-right</v-icon>
|
<v-icon size="32">mdi-chevron-right</v-icon>
|
||||||
</button>
|
</button>
|
||||||
|
|
||||||
<!-- Large centered loading spinner (operator-flagged 2026-06-07: the old
|
|
||||||
size-36 one sat tiny in the top-left). Dual counter-rotating accent
|
|
||||||
rings, centered over the whole modal. pointer-events:none so the
|
|
||||||
close/nav controls underneath stay clickable. -->
|
|
||||||
<div v-if="modal.loading" class="fc-viewer__loading">
|
|
||||||
<div class="fc-viewer__spinner" />
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-viewer__body">
|
<div class="fc-viewer__body">
|
||||||
<div class="fc-viewer__media">
|
<div class="fc-viewer__media">
|
||||||
<template v-if="modal.current">
|
<v-progress-circular
|
||||||
|
v-if="modal.loading" indeterminate color="accent" size="36"
|
||||||
|
/>
|
||||||
|
<template v-else-if="modal.current">
|
||||||
<ImageCanvas
|
<ImageCanvas
|
||||||
v-if="!isVideo" :src="modal.current.image_url" :alt="`Image ${modal.current.id}`"
|
v-if="!isVideo" :src="modal.current.image_url" :alt="`Image ${modal.current.id}`"
|
||||||
@close-request="$emit('close')"
|
@close-request="$emit('close')"
|
||||||
@@ -75,9 +51,6 @@
|
|||||||
<aside v-if="modal.current" class="fc-viewer__side">
|
<aside v-if="modal.current" class="fc-viewer__side">
|
||||||
<ProvenancePanel />
|
<ProvenancePanel />
|
||||||
<TagPanel />
|
<TagPanel />
|
||||||
<!-- Non-blocking: fetches its own similar set after the modal is up;
|
|
||||||
collapses silently if empty/slow/failed (see RelatedStrip). -->
|
|
||||||
<RelatedStrip />
|
|
||||||
</aside>
|
</aside>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -87,18 +60,15 @@
|
|||||||
<script setup>
|
<script setup>
|
||||||
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
||||||
import { useModalStore } from '../../stores/modal.js'
|
import { useModalStore } from '../../stores/modal.js'
|
||||||
import { arrowNavAllowed, isTextEntry } from '../../utils/textEntry.js'
|
|
||||||
import ImageCanvas from './ImageCanvas.vue'
|
import ImageCanvas from './ImageCanvas.vue'
|
||||||
import VideoCanvas from './VideoCanvas.vue'
|
import VideoCanvas from './VideoCanvas.vue'
|
||||||
import TagPanel from './TagPanel.vue'
|
import TagPanel from './TagPanel.vue'
|
||||||
import ProvenancePanel from './ProvenancePanel.vue'
|
import ProvenancePanel from './ProvenancePanel.vue'
|
||||||
import RelatedStrip from './RelatedStrip.vue'
|
|
||||||
|
|
||||||
const emit = defineEmits(['close'])
|
const emit = defineEmits(['close'])
|
||||||
|
|
||||||
const modal = useModalStore()
|
const modal = useModalStore()
|
||||||
const rootEl = ref(null)
|
const rootEl = ref(null)
|
||||||
const showHelp = ref(false)
|
|
||||||
|
|
||||||
const isVideo = computed(() =>
|
const isVideo = computed(() =>
|
||||||
modal.current?.mime && modal.current.mime.startsWith('video/')
|
modal.current?.mime && modal.current.mime.startsWith('video/')
|
||||||
@@ -121,47 +91,27 @@ let prevBodyOverflow = null
|
|||||||
// that. Filtered via isTextEntry so tag/comment inputs still get their
|
// that. Filtered via isTextEntry so tag/comment inputs still get their
|
||||||
// own keystrokes.
|
// own keystrokes.
|
||||||
function onKeyDown(ev) {
|
function onKeyDown(ev) {
|
||||||
if (ev.key === 'Escape' && showHelp.value) {
|
|
||||||
// Close the cheatsheet first; a second Esc closes the modal.
|
|
||||||
ev.preventDefault()
|
|
||||||
showHelp.value = false
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if (ev.key === 'Escape') {
|
if (ev.key === 'Escape') {
|
||||||
// Escape closes the modal even from inside a text input — the universal
|
// Escape closes the modal even from inside a text input — that's
|
||||||
// "get me out of here" expectation; the autofocused tag field would
|
// the universal "get me out of here" expectation, and the
|
||||||
// otherwise trap focus (operator-flagged 2026-06-01). EXCEPTION: when the
|
// autofocused tag-entry field would otherwise trap focus with no
|
||||||
// keystroke originates INSIDE an open Vuetify overlay's content (a rename/
|
// visible escape (operator-flagged 2026-06-01). EXCEPTION: when a
|
||||||
// fandom/alias dialog, or a kebab menu), let that overlay handle its own
|
// nested Vuetify overlay is open (v-menu autocomplete dropdown,
|
||||||
// Esc and don't close the whole modal.
|
// FandomPicker v-dialog, per-suggestion 3-dot menu), let that
|
||||||
//
|
// overlay's own Esc handling fire instead of closing the whole
|
||||||
// #700 re-fix (2026-06-07): the prior guard queried for ANY active overlay
|
// modal mid-interaction. Vuetify marks open overlays with
|
||||||
// anywhere in the DOM and suppressed the close — so a lingering overlay
|
// `.v-overlay--active`.
|
||||||
// after accepting a suggestion (focus drops to <body>, not into any
|
if (document.querySelector('.v-overlay--active')) return
|
||||||
// overlay) wrongly blocked Esc. Keying off the event's origin instead means
|
|
||||||
// a stray overlay no longer traps the modal: only an Esc pressed from
|
|
||||||
// within overlay content defers to that overlay.
|
|
||||||
if (ev.target?.closest?.('.v-overlay__content')) return
|
|
||||||
ev.preventDefault()
|
ev.preventDefault()
|
||||||
emit('close')
|
emit('close')
|
||||||
} else if (ev.key === 'ArrowLeft') {
|
} else if (ev.key === 'ArrowLeft') {
|
||||||
// Navigate unless the caret is in a non-empty text field (then let it move
|
if (isTextEntry(ev.target)) return
|
||||||
// through the text). An empty tag-entry field still navigates.
|
|
||||||
if (!arrowNavAllowed(ev.target)) return
|
|
||||||
ev.preventDefault()
|
ev.preventDefault()
|
||||||
modal.goPrev()
|
modal.goPrev()
|
||||||
} else if (ev.key === 'ArrowRight') {
|
} else if (ev.key === 'ArrowRight') {
|
||||||
if (!arrowNavAllowed(ev.target)) return
|
if (isTextEntry(ev.target)) return
|
||||||
ev.preventDefault()
|
ev.preventDefault()
|
||||||
modal.goNext()
|
modal.goNext()
|
||||||
} else if ((ev.key === '/' || ev.key === 't') && !isTextEntry(ev.target)) {
|
|
||||||
// C9 (2026-06-07): jump focus to the tag input from anywhere in the modal.
|
|
||||||
const input = document.querySelector('.fc-tag-autocomplete input')
|
|
||||||
if (input) { ev.preventDefault(); input.focus() }
|
|
||||||
} else if (ev.key === '?' && !isTextEntry(ev.target)) {
|
|
||||||
// C8: toggle the keyboard cheatsheet.
|
|
||||||
ev.preventDefault()
|
|
||||||
showHelp.value = !showHelp.value
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -185,14 +135,17 @@ watch(() => modal.currentImageId, async () => {
|
|||||||
function nextFrame() {
|
function nextFrame() {
|
||||||
return new Promise(resolve => requestAnimationFrame(resolve))
|
return new Promise(resolve => requestAnimationFrame(resolve))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function isTextEntry(el) {
|
||||||
|
if (!el) return false
|
||||||
|
const tag = el.tagName
|
||||||
|
return tag === 'INPUT' || tag === 'TEXTAREA' || el.isContentEditable
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.fc-viewer {
|
.fc-viewer {
|
||||||
position: fixed; inset: 0; z-index: 2000;
|
position: fixed; inset: 0; z-index: 2000;
|
||||||
/* Single source of truth for the metadata side-panel width — the next
|
|
||||||
arrow offsets off it so it never overlaps the panel. */
|
|
||||||
--fc-side-w: 320px;
|
|
||||||
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
|
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
|
||||||
mid-opacity + blur so the page behind shows through faintly. */
|
mid-opacity + blur so the page behind shows through faintly. */
|
||||||
background: rgba(20, 23, 26, 0.65);
|
background: rgba(20, 23, 26, 0.65);
|
||||||
@@ -217,81 +170,11 @@ function nextFrame() {
|
|||||||
}
|
}
|
||||||
.fc-viewer__nav:disabled { opacity: 0.3; cursor: not-allowed; }
|
.fc-viewer__nav:disabled { opacity: 0.3; cursor: not-allowed; }
|
||||||
.fc-viewer__close { top: 16px; right: 16px; transform: none; }
|
.fc-viewer__close { top: 16px; right: 16px; transform: none; }
|
||||||
.fc-viewer__help-hint {
|
|
||||||
position: absolute; top: 16px; right: 64px; z-index: 2;
|
|
||||||
width: 32px; height: 32px; border-radius: 50%;
|
|
||||||
background: rgba(255, 255, 255, 0.08);
|
|
||||||
color: rgb(var(--v-theme-on-surface));
|
|
||||||
font-weight: 700; cursor: pointer; opacity: 0.6;
|
|
||||||
}
|
|
||||||
.fc-viewer__help-hint:hover { opacity: 1; }
|
|
||||||
.fc-viewer__help {
|
|
||||||
position: absolute; inset: 0; z-index: 4;
|
|
||||||
display: flex; align-items: center; justify-content: center;
|
|
||||||
background: rgba(0, 0, 0, 0.4);
|
|
||||||
}
|
|
||||||
.fc-viewer__help-card {
|
|
||||||
background: rgb(var(--v-theme-surface));
|
|
||||||
border: 1px solid rgb(var(--v-theme-surface-light));
|
|
||||||
border-radius: 10px; padding: 20px 24px; min-width: 320px;
|
|
||||||
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.5);
|
|
||||||
}
|
|
||||||
.fc-viewer__help-card h3 {
|
|
||||||
font-family: 'Fraunces', Georgia, serif;
|
|
||||||
font-size: 18px; margin-bottom: 12px;
|
|
||||||
color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-viewer__help-card dl { display: flex; flex-direction: column; gap: 8px; }
|
|
||||||
.fc-viewer__help-card dl > div {
|
|
||||||
display: flex; gap: 16px; align-items: baseline;
|
|
||||||
}
|
|
||||||
.fc-viewer__help-card dt {
|
|
||||||
flex: 0 0 96px; text-align: right;
|
|
||||||
font-family: 'JetBrains Mono', monospace; font-size: 12px;
|
|
||||||
color: rgb(var(--v-theme-accent));
|
|
||||||
}
|
|
||||||
.fc-viewer__help-card dd {
|
|
||||||
flex: 1; font-size: 14px;
|
|
||||||
color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
.fc-viewer__integrity {
|
.fc-viewer__integrity {
|
||||||
position: absolute; top: 72px; right: 16px; z-index: 3;
|
position: absolute; top: 72px; right: 16px; z-index: 3;
|
||||||
}
|
}
|
||||||
.fc-viewer__nav--prev { left: 16px; transform: translateY(-50%); }
|
.fc-viewer__nav--prev { left: 16px; transform: translateY(-50%); }
|
||||||
/* Sit just inside the image area, clear of the metadata side panel —
|
.fc-viewer__nav--next { right: 16px; transform: translateY(-50%); }
|
||||||
not floating over it (operator-flagged 2026-06-04). */
|
|
||||||
.fc-viewer__nav--next {
|
|
||||||
right: calc(var(--fc-side-w) + 16px);
|
|
||||||
transform: translateY(-50%);
|
|
||||||
}
|
|
||||||
/* Centered loading overlay — sits over the media band; non-interactive so the
|
|
||||||
close/nav controls underneath keep working. */
|
|
||||||
.fc-viewer__loading {
|
|
||||||
position: absolute; inset: 0; z-index: 2;
|
|
||||||
display: flex; align-items: center; justify-content: center;
|
|
||||||
pointer-events: none;
|
|
||||||
}
|
|
||||||
/* Two concentric accent rings counter-rotating, each lit on a different arc —
|
|
||||||
reads as one orbiting object. ~108px so it's unmistakably the focal point. */
|
|
||||||
.fc-viewer__spinner {
|
|
||||||
width: 108px; height: 108px; border-radius: 50%;
|
|
||||||
border: 5px solid rgb(var(--v-theme-accent), 0.16);
|
|
||||||
border-top-color: rgb(var(--v-theme-accent));
|
|
||||||
animation: fc-viewer-spin 0.95s cubic-bezier(0.5, 0.1, 0.5, 0.9) infinite;
|
|
||||||
position: relative;
|
|
||||||
box-shadow: 0 0 24px rgb(var(--v-theme-accent), 0.18);
|
|
||||||
}
|
|
||||||
.fc-viewer__spinner::after {
|
|
||||||
content: ''; position: absolute; inset: 14px; border-radius: 50%;
|
|
||||||
border: 5px solid rgb(var(--v-theme-accent), 0.10);
|
|
||||||
border-bottom-color: rgb(var(--v-theme-accent));
|
|
||||||
animation: fc-viewer-spin 1.5s cubic-bezier(0.5, 0.1, 0.5, 0.9) infinite reverse;
|
|
||||||
}
|
|
||||||
@keyframes fc-viewer-spin { to { transform: rotate(360deg); } }
|
|
||||||
@media (prefers-reduced-motion: reduce) {
|
|
||||||
.fc-viewer__spinner, .fc-viewer__spinner::after { animation-duration: 3s; }
|
|
||||||
}
|
|
||||||
|
|
||||||
.fc-viewer__body {
|
.fc-viewer__body {
|
||||||
flex: 1; display: flex; min-height: 0;
|
flex: 1; display: flex; min-height: 0;
|
||||||
}
|
}
|
||||||
@@ -304,44 +187,19 @@ function nextFrame() {
|
|||||||
min-width: 0; min-height: 0;
|
min-width: 0; min-height: 0;
|
||||||
}
|
}
|
||||||
.fc-viewer__side {
|
.fc-viewer__side {
|
||||||
width: var(--fc-side-w); flex-shrink: 0;
|
width: 320px; flex-shrink: 0;
|
||||||
background: rgb(var(--v-theme-surface));
|
background: rgb(var(--v-theme-surface));
|
||||||
border-left: 1px solid rgb(var(--v-theme-surface-light));
|
border-left: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
overflow-y: auto;
|
overflow-y: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
@media (max-width: 900px) {
|
@media (max-width: 900px) {
|
||||||
/* Stacked layout (operator-flagged 2026-06-05): pin the image pane and
|
.fc-viewer__body { flex-direction: column; }
|
||||||
its controls at the top while the metadata panel scrolls beneath it.
|
|
||||||
Previously the image + a 40vh panel shared one fixed viewport and the
|
|
||||||
controls could be pushed out of view; now the body scrolls and the
|
|
||||||
media is sticky, so the image + prev/next/close (and the integrity
|
|
||||||
badge) stay visible no matter how far the panel is scrolled. */
|
|
||||||
.fc-viewer__body {
|
|
||||||
flex-direction: column;
|
|
||||||
overflow-y: auto;
|
|
||||||
}
|
|
||||||
.fc-viewer__media {
|
|
||||||
flex: none;
|
|
||||||
height: 55vh;
|
|
||||||
position: sticky;
|
|
||||||
top: 0;
|
|
||||||
z-index: 1;
|
|
||||||
/* Opaque obsidian so the scrolling panel never bleeds through the
|
|
||||||
haze behind the pinned image. */
|
|
||||||
background: rgb(20, 23, 26);
|
|
||||||
}
|
|
||||||
.fc-viewer__side {
|
.fc-viewer__side {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
max-height: none;
|
max-height: 40vh;
|
||||||
border-left: none;
|
border-left: none;
|
||||||
border-top: 1px solid rgb(var(--v-theme-surface-light));
|
border-top: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
}
|
}
|
||||||
/* Re-center the prev/next arrows over the 55vh image band (their base
|
|
||||||
top:50% would land on the scrolling panel); next uses the full width
|
|
||||||
now that the panel is below, not beside. Close + integrity badge keep
|
|
||||||
their top:16px/72px — already within the pinned band. */
|
|
||||||
.fc-viewer__nav { top: 27.5vh; }
|
|
||||||
.fc-viewer__nav--next { right: 16px; }
|
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,133 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- Collapses entirely unless the source has an embedding AND we're loading
|
|
||||||
or have results — so a slow/empty/failed fetch leaves no trace and never
|
|
||||||
affects the modal. -->
|
|
||||||
<div v-if="show" class="fc-related">
|
|
||||||
<div class="fc-related__head">
|
|
||||||
<span class="fc-related__title">Related</span>
|
|
||||||
<v-btn
|
|
||||||
size="x-small" variant="text" color="accent"
|
|
||||||
:disabled="loading || !results.length"
|
|
||||||
@click="seeAll"
|
|
||||||
>See all similar</v-btn>
|
|
||||||
</div>
|
|
||||||
<div class="fc-related__row">
|
|
||||||
<template v-if="loading">
|
|
||||||
<div v-for="n in 6" :key="n" class="fc-related__skel" />
|
|
||||||
</template>
|
|
||||||
<button
|
|
||||||
v-for="img in results" :key="img.id"
|
|
||||||
class="fc-related__item" type="button"
|
|
||||||
@click="openImage(img.id)"
|
|
||||||
>
|
|
||||||
<img :src="img.thumbnail_url" :alt="`image ${img.id}`" loading="lazy" />
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed, onBeforeUnmount, ref, watch } from 'vue'
|
|
||||||
import { useRouter } from 'vue-router'
|
|
||||||
import { useApi } from '../../composables/useApi.js'
|
|
||||||
import { useModalStore } from '../../stores/modal.js'
|
|
||||||
|
|
||||||
// Deferred so the modal's main image gets network/decode priority — the strip
|
|
||||||
// is a nice-to-have and must never block or slow the modal load.
|
|
||||||
const DEFER_MS = 200
|
|
||||||
const STRIP_LIMIT = 12
|
|
||||||
|
|
||||||
const api = useApi()
|
|
||||||
const router = useRouter()
|
|
||||||
const modal = useModalStore()
|
|
||||||
|
|
||||||
const results = ref([])
|
|
||||||
const loading = ref(false)
|
|
||||||
|
|
||||||
const hasEmbedding = computed(() => modal.current?.has_embedding === true)
|
|
||||||
const show = computed(() => hasEmbedding.value && (loading.value || results.value.length > 0))
|
|
||||||
|
|
||||||
let seq = 0
|
|
||||||
let timer = null
|
|
||||||
|
|
||||||
async function fetchSimilar(id) {
|
|
||||||
const mine = ++seq
|
|
||||||
loading.value = true
|
|
||||||
results.value = []
|
|
||||||
try {
|
|
||||||
const body = await api.get('/api/gallery/similar', {
|
|
||||||
params: { similar_to: id, limit: STRIP_LIMIT },
|
|
||||||
})
|
|
||||||
if (mine !== seq) return
|
|
||||||
results.value = body.images || []
|
|
||||||
} catch {
|
|
||||||
if (mine === seq) results.value = [] // quietly collapse on error
|
|
||||||
} finally {
|
|
||||||
if (mine === seq) loading.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Re-fetch whenever the modal lands on a new embedded image. modal.current is
|
|
||||||
// null while the next image loads, so this also clears the strip during
|
|
||||||
// prev/next nav and repopulates once the new payload arrives.
|
|
||||||
watch(
|
|
||||||
() => (hasEmbedding.value ? modal.current?.id : null),
|
|
||||||
(id) => {
|
|
||||||
if (timer) { clearTimeout(timer); timer = null }
|
|
||||||
seq++ // cancel any in-flight fetch
|
|
||||||
results.value = []
|
|
||||||
loading.value = false
|
|
||||||
if (!id) return
|
|
||||||
timer = setTimeout(() => fetchSimilar(id), DEFER_MS)
|
|
||||||
},
|
|
||||||
{ immediate: true },
|
|
||||||
)
|
|
||||||
onBeforeUnmount(() => { if (timer) clearTimeout(timer) })
|
|
||||||
|
|
||||||
function openImage(id) {
|
|
||||||
modal.open(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
function seeAll() {
|
|
||||||
const id = modal.current?.id
|
|
||||||
if (!id) return
|
|
||||||
modal.close()
|
|
||||||
router.push({ name: 'gallery', query: { similar_to: String(id) } })
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-related {
|
|
||||||
padding: 12px;
|
|
||||||
border-top: 1px solid rgb(var(--v-theme-surface-light));
|
|
||||||
}
|
|
||||||
.fc-related__head {
|
|
||||||
display: flex; align-items: center; justify-content: space-between;
|
|
||||||
margin-bottom: 8px;
|
|
||||||
}
|
|
||||||
.fc-related__title {
|
|
||||||
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
}
|
|
||||||
.fc-related__row {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(4, 1fr);
|
|
||||||
gap: 6px;
|
|
||||||
}
|
|
||||||
.fc-related__item {
|
|
||||||
display: block; padding: 0; border: 0; background: none;
|
|
||||||
cursor: pointer; border-radius: 4px; overflow: hidden;
|
|
||||||
aspect-ratio: 1; width: 100%;
|
|
||||||
}
|
|
||||||
.fc-related__item img {
|
|
||||||
width: 100%; height: 100%; object-fit: cover; display: block;
|
|
||||||
background: rgb(var(--v-theme-surface-light));
|
|
||||||
transition: transform 0.2s ease, filter 0.2s ease;
|
|
||||||
}
|
|
||||||
.fc-related__item:hover img { transform: scale(1.05); filter: brightness(1.1); }
|
|
||||||
.fc-related__skel {
|
|
||||||
aspect-ratio: 1; border-radius: 4px;
|
|
||||||
background: rgb(var(--v-theme-surface-light));
|
|
||||||
opacity: 0.5;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -19,26 +19,24 @@
|
|||||||
>
|
>
|
||||||
Accept
|
Accept
|
||||||
</v-btn>
|
</v-btn>
|
||||||
<!-- Operator-flagged 2026-06-04: the kebab still wasn't opening. The
|
<!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
|
||||||
prior `#activator` + `v-bind="props"` path never toggled the menu
|
Wrapping in a <span @click.stop> matches the TagPanel chip
|
||||||
inside this teleported modal, while v-model-driven overlays (the
|
fix — even though there's no parent click capture here today,
|
||||||
dialogs in this modal) work fine. So drive the menu explicitly:
|
the wrap is harmless and keeps both kebabs on the same
|
||||||
the button toggles `menuOpen` with @click.stop (also shields any
|
pattern. Click bubbles from the v-btn → opens menu via
|
||||||
parent), and `activator="parent"` anchors the menu for positioning
|
activator props → bubble continues to span → stopPropagation
|
||||||
only — `:open-on-click="false"` keeps Vuetify's activator-click out
|
halts it. -->
|
||||||
of it, so there's a single, reliable opener. -->
|
<span class="fc-suggestion__menu-wrap" @click.stop>
|
||||||
<span class="fc-suggestion__menu-wrap">
|
<v-menu>
|
||||||
|
<template #activator="{ props }">
|
||||||
<v-btn
|
<v-btn
|
||||||
class="fc-suggestion__menu"
|
class="fc-suggestion__menu"
|
||||||
icon="mdi-dots-vertical" size="small"
|
icon="mdi-dots-vertical" size="small"
|
||||||
variant="outlined" density="compact"
|
variant="outlined" density="compact"
|
||||||
:aria-label="`More actions for ${suggestion.display_name}`"
|
:aria-label="`More actions for ${suggestion.display_name}`"
|
||||||
@click.stop="menuOpen = !menuOpen"
|
v-bind="props"
|
||||||
/>
|
/>
|
||||||
<v-menu
|
</template>
|
||||||
v-model="menuOpen" activator="parent" :open-on-click="false"
|
|
||||||
:z-index="2400"
|
|
||||||
>
|
|
||||||
<v-list density="compact">
|
<v-list density="compact">
|
||||||
<v-list-item @click="$emit('alias', suggestion)">
|
<v-list-item @click="$emit('alias', suggestion)">
|
||||||
<v-list-item-title>Treat as alias for…</v-list-item-title>
|
<v-list-item-title>Treat as alias for…</v-list-item-title>
|
||||||
@@ -53,12 +51,11 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, ref } from 'vue'
|
import { computed } from 'vue'
|
||||||
|
|
||||||
const props = defineProps({ suggestion: { type: Object, required: true } })
|
const props = defineProps({ suggestion: { type: Object, required: true } })
|
||||||
defineEmits(['accept', 'alias', 'dismiss'])
|
defineEmits(['accept', 'alias', 'dismiss'])
|
||||||
|
|
||||||
const menuOpen = ref(false)
|
|
||||||
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
|
|||||||
@@ -8,63 +8,32 @@
|
|||||||
@keydown.down.prevent="moveHighlight(1)"
|
@keydown.down.prevent="moveHighlight(1)"
|
||||||
@keydown.up.prevent="moveHighlight(-1)"
|
@keydown.up.prevent="moveHighlight(-1)"
|
||||||
@keydown.enter.prevent="onEnter"
|
@keydown.enter.prevent="onEnter"
|
||||||
@keydown.tab="onTab"
|
|
||||||
@keydown.esc="$emit('cancel')"
|
@keydown.esc="$emit('cancel')"
|
||||||
/>
|
/>
|
||||||
<v-list
|
<v-list
|
||||||
v-if="rows.length"
|
v-if="hits.length || allowCreate"
|
||||||
ref="listRef"
|
|
||||||
density="compact" class="fc-tag-autocomplete__list"
|
density="compact" class="fc-tag-autocomplete__list"
|
||||||
>
|
>
|
||||||
<template v-for="(row, idx) in rows" :key="row.key">
|
|
||||||
<!-- Existing tag match (server autocomplete). -->
|
|
||||||
<v-list-item
|
<v-list-item
|
||||||
v-if="row.type === 'hit'"
|
v-for="(h, idx) in hits" :key="h.id"
|
||||||
:active="idx === highlight" @click="onPickRow(row)"
|
:active="idx === highlight" @click="onPick(h)"
|
||||||
>
|
>
|
||||||
<template #prepend>
|
<template #prepend>
|
||||||
<v-icon size="small" :color="store.colorFor(row.hit.kind)">
|
<v-icon size="small" :color="store.colorFor(h.kind)">
|
||||||
{{ iconFor(row.hit.kind) }}
|
{{ iconFor(h.kind) }}
|
||||||
</v-icon>
|
</v-icon>
|
||||||
</template>
|
</template>
|
||||||
<v-list-item-title>
|
<v-list-item-title>
|
||||||
{{ row.hit.name }}
|
{{ h.name }}
|
||||||
<span v-if="row.hit.fandom_name" class="text-caption">— {{ row.hit.fandom_name }}</span>
|
<span v-if="h.fandom_name" class="text-caption">— {{ h.fandom_name }}</span>
|
||||||
</v-list-item-title>
|
</v-list-item-title>
|
||||||
<template #append>
|
<template #append>
|
||||||
<span class="text-caption">{{ row.hit.kind }}</span>
|
<span class="text-caption">{{ h.kind }}</span>
|
||||||
</template>
|
</template>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
|
|
||||||
<!-- ML suggestion for THIS image that matches the typed query. Picking
|
|
||||||
it routes through the same accept path as the Suggestions panel, so
|
|
||||||
it's recorded + drops out of the panel (operator-asked 2026-06-07). -->
|
|
||||||
<v-list-item
|
<v-list-item
|
||||||
v-else-if="row.type === 'suggestion'"
|
v-if="allowCreate" :active="highlight === hits.length"
|
||||||
:active="idx === highlight" @click="onPickRow(row)"
|
@click="onCreate"
|
||||||
class="fc-tag-autocomplete__sugg"
|
|
||||||
>
|
|
||||||
<template #prepend>
|
|
||||||
<v-icon size="small" :color="store.colorFor(row.sugg.category)">
|
|
||||||
{{ iconFor(row.sugg.category) }}
|
|
||||||
</v-icon>
|
|
||||||
</template>
|
|
||||||
<v-list-item-title>
|
|
||||||
{{ row.sugg.display_name }}
|
|
||||||
<span v-if="row.sugg.creates_new_tag" class="text-caption">— new</span>
|
|
||||||
</v-list-item-title>
|
|
||||||
<template #append>
|
|
||||||
<span class="fc-tag-autocomplete__sugg-tag">
|
|
||||||
<v-icon size="x-small">mdi-auto-fix</v-icon>
|
|
||||||
{{ scorePct(row.sugg) }}
|
|
||||||
</span>
|
|
||||||
</template>
|
|
||||||
</v-list-item>
|
|
||||||
|
|
||||||
<!-- Create-new row. -->
|
|
||||||
<v-list-item
|
|
||||||
v-else
|
|
||||||
:active="idx === highlight" @click="onPickRow(row)"
|
|
||||||
>
|
>
|
||||||
<template #prepend>
|
<template #prepend>
|
||||||
<v-icon size="small" :color="store.colorFor(parsedKind)">
|
<v-icon size="small" :color="store.colorFor(parsedKind)">
|
||||||
@@ -75,20 +44,10 @@
|
|||||||
Create "{{ parsedName }}" as {{ parsedKind }}
|
Create "{{ parsedName }}" as {{ parsedKind }}
|
||||||
</v-list-item-title>
|
</v-list-item-title>
|
||||||
</v-list-item>
|
</v-list-item>
|
||||||
</template>
|
|
||||||
</v-list>
|
</v-list>
|
||||||
|
|
||||||
<!-- @after-enter is the reliable moment to focus the picker: the dialog's
|
<v-dialog v-model="fandomDialog" max-width="480">
|
||||||
open transition + focus-trap have settled, so focusSearch lands in the
|
<FandomPicker @confirm="onFandomChosen" @cancel="fandomDialog = false" />
|
||||||
fandom field instead of fighting the trap (operator-flagged 2026-06-07). -->
|
|
||||||
<v-dialog
|
|
||||||
v-model="fandomDialog" max-width="480"
|
|
||||||
@after-enter="fandomPickerRef?.focusSearch?.()"
|
|
||||||
>
|
|
||||||
<FandomPicker
|
|
||||||
ref="fandomPickerRef"
|
|
||||||
@confirm="onFandomChosen" @cancel="onFandomCancel"
|
|
||||||
/>
|
|
||||||
</v-dialog>
|
</v-dialog>
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
@@ -96,33 +55,18 @@
|
|||||||
<script setup>
|
<script setup>
|
||||||
import { computed, nextTick, onMounted, ref, watch } from 'vue'
|
import { computed, nextTick, onMounted, ref, watch } from 'vue'
|
||||||
import { useTagStore } from '../../stores/tags.js'
|
import { useTagStore } from '../../stores/tags.js'
|
||||||
import { useSuggestionsStore } from '../../stores/suggestions.js'
|
|
||||||
import FandomPicker from './FandomPicker.vue'
|
import FandomPicker from './FandomPicker.vue'
|
||||||
|
|
||||||
const emit = defineEmits(['pick-existing', 'pick-new', 'accept-suggestion', 'cancel'])
|
const emit = defineEmits(['pick-existing', 'pick-new', 'cancel'])
|
||||||
const store = useTagStore()
|
const store = useTagStore()
|
||||||
// The image's ML (Camie) suggestions, surfaced inline in this dropdown so the
|
|
||||||
// operator can pick a suggestion while typing instead of hunting for it in the
|
|
||||||
// Suggestions panel below (operator-asked 2026-06-07).
|
|
||||||
const suggestions = useSuggestionsStore()
|
|
||||||
|
|
||||||
// Autofocus on modal open so the operator can type the moment the view
|
// Autofocus on modal open so the operator can type the moment the view
|
||||||
// modal renders, no extra click required (operator-asked 2026-06-01).
|
// modal renders, no extra click required (operator-asked 2026-06-01).
|
||||||
// Vuetify's v-text-field exposes .focus() on the component instance;
|
// Vuetify's v-text-field exposes .focus() on the component instance;
|
||||||
// nextTick waits for the modal's mount to finish so the inner <input>
|
// nextTick waits for the modal's mount to finish so the inner <input>
|
||||||
// element exists.
|
// element exists.
|
||||||
//
|
|
||||||
// EXCEPTION — not on mobile (operator-flagged 2026-06-05): focusing the
|
|
||||||
// field pops the soft keyboard, which shrinks the visual viewport and
|
|
||||||
// shoves the pinned image + its nav/close controls out of view. 900px is
|
|
||||||
// the modal's stacked-layout breakpoint (ImageViewer.vue). matchMedia is
|
|
||||||
// safe on plain HTTP (not a secure-context API).
|
|
||||||
const inputRef = ref(null)
|
const inputRef = ref(null)
|
||||||
function focusInput () {
|
onMounted(() => { nextTick(() => inputRef.value?.focus?.()) })
|
||||||
if (window.matchMedia?.('(max-width: 900px)')?.matches) return
|
|
||||||
nextTick(() => inputRef.value?.focus?.())
|
|
||||||
}
|
|
||||||
onMounted(focusInput)
|
|
||||||
|
|
||||||
// Single text input; no kind dropdown. Client-side mirror of the
|
// Single text input; no kind dropdown. Client-side mirror of the
|
||||||
// backend's parse_kind_prefix lives below — kept in sync with
|
// backend's parse_kind_prefix lives below — kept in sync with
|
||||||
@@ -132,9 +76,7 @@ onMounted(focusInput)
|
|||||||
const query = ref('')
|
const query = ref('')
|
||||||
const hits = ref([])
|
const hits = ref([])
|
||||||
const highlight = ref(0)
|
const highlight = ref(0)
|
||||||
const listRef = ref(null)
|
|
||||||
const fandomDialog = ref(false)
|
const fandomDialog = ref(false)
|
||||||
const fandomPickerRef = ref(null)
|
|
||||||
let pendingNewName = null
|
let pendingNewName = null
|
||||||
|
|
||||||
const KNOWN_KINDS = new Set([
|
const KNOWN_KINDS = new Set([
|
||||||
@@ -182,66 +124,13 @@ const allowCreate = computed(() => {
|
|||||||
)
|
)
|
||||||
})
|
})
|
||||||
|
|
||||||
function scorePct (s) { return `${Math.round(s.score * 100)}%` }
|
|
||||||
|
|
||||||
// This image's suggestions that match the typed query, minus any the server
|
|
||||||
// autocomplete already returned (same name+kind) so a tag never shows twice.
|
|
||||||
const suggestionHits = computed(() => {
|
|
||||||
const q = parsedName.value.toLowerCase()
|
|
||||||
if (!q) return []
|
|
||||||
const seen = new Set(hits.value.map(h => `${h.kind}:${h.name.toLowerCase()}`))
|
|
||||||
const out = []
|
|
||||||
for (const list of Object.values(suggestions.byCategory)) {
|
|
||||||
for (const s of list || []) {
|
|
||||||
const key = `${s.category}:${s.display_name.toLowerCase()}`
|
|
||||||
if (!s.display_name.toLowerCase().includes(q)) continue
|
|
||||||
if (seen.has(key)) continue
|
|
||||||
seen.add(key)
|
|
||||||
out.push(s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Best matches first; cap so the dropdown stays scannable.
|
|
||||||
out.sort((a, b) => b.score - a.score)
|
|
||||||
return out.slice(0, 6)
|
|
||||||
})
|
|
||||||
|
|
||||||
// One ordered list backing both the rendered dropdown and keyboard nav, so the
|
|
||||||
// highlight index maps 1:1 to a row regardless of which section it's in.
|
|
||||||
const rows = computed(() => {
|
|
||||||
const r = hits.value.map(h => ({ type: 'hit', key: `h${h.id}`, hit: h }))
|
|
||||||
for (const s of suggestionHits.value) {
|
|
||||||
r.push({ type: 'suggestion', key: `s${s.category}:${s.display_name}`, sugg: s })
|
|
||||||
}
|
|
||||||
if (allowCreate.value) r.push({ type: 'create', key: 'create' })
|
|
||||||
return r
|
|
||||||
})
|
|
||||||
|
|
||||||
function moveHighlight (delta) {
|
function moveHighlight (delta) {
|
||||||
const total = rows.value.length
|
const total = hits.value.length + (allowCreate.value ? 1 : 0)
|
||||||
if (total === 0) return
|
if (total === 0) return
|
||||||
highlight.value = (highlight.value + delta + total) % total
|
highlight.value = (highlight.value + delta + total) % total
|
||||||
// Keep the highlighted row visible — the list is capped at 240px and arrowing
|
|
||||||
// past the fold otherwise left the active item off-screen (operator-flagged
|
|
||||||
// 2026-06-07). block:'nearest' scrolls the minimum needed.
|
|
||||||
nextTick(() => {
|
|
||||||
listRef.value?.$el
|
|
||||||
?.querySelector('.v-list-item--active')
|
|
||||||
?.scrollIntoView({ block: 'nearest' })
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Dispatch a chosen dropdown row by its type.
|
function onPick (hit) { emit('pick-existing', hit); reset() }
|
||||||
function onPickRow (row) {
|
|
||||||
if (row.type === 'hit') { emit('pick-existing', row.hit); reset() }
|
|
||||||
else if (row.type === 'suggestion') { onPickSuggestion(row.sugg) }
|
|
||||||
else { onCreate() }
|
|
||||||
}
|
|
||||||
|
|
||||||
// Picking an ML suggestion hands it to the parent, which runs the same accept
|
|
||||||
// flow as the Suggestions panel (creates the tag if raw, records acceptance,
|
|
||||||
// drops it from the panel). reset() clears the query; the panel-drop makes it
|
|
||||||
// fall out of suggestionHits reactively.
|
|
||||||
function onPickSuggestion (s) { emit('accept-suggestion', s); reset() }
|
|
||||||
|
|
||||||
function onCreate () {
|
function onCreate () {
|
||||||
const name = parsedName.value
|
const name = parsedName.value
|
||||||
@@ -258,42 +147,20 @@ function onCreate () {
|
|||||||
reset()
|
reset()
|
||||||
}
|
}
|
||||||
|
|
||||||
// fandom is null when the user picked "No fandom" — characters don't all
|
|
||||||
// belong to a fandom. The backend already accepts fandom_id: null for the
|
|
||||||
// character kind (tag.kind check + nullable fandom_id), and a fandom can be
|
|
||||||
// assigned later from the chip kebab's "Set fandom…".
|
|
||||||
function onFandomChosen (fandom) {
|
function onFandomChosen (fandom) {
|
||||||
fandomDialog.value = false
|
fandomDialog.value = false
|
||||||
emit('pick-new', {
|
emit('pick-new', {
|
||||||
name: pendingNewName, kind: 'character', fandom_id: fandom ? fandom.id : null,
|
name: pendingNewName, kind: 'character', fandom_id: fandom.id,
|
||||||
})
|
})
|
||||||
pendingNewName = null
|
pendingNewName = null
|
||||||
reset()
|
reset()
|
||||||
// Return focus to the tag input so the keyboard flow continues straight into
|
|
||||||
// the next tag instead of dropping to <body> (operator-flagged 2026-06-07).
|
|
||||||
focusInput()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Cancelling the fandom dialog drops the pending character and hands focus back
|
|
||||||
// to the tag input, same as a successful pick.
|
|
||||||
function onFandomCancel () {
|
|
||||||
fandomDialog.value = false
|
|
||||||
pendingNewName = null
|
|
||||||
focusInput()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function onEnter () {
|
function onEnter () {
|
||||||
const row = rows.value[highlight.value]
|
if (highlight.value < hits.value.length) {
|
||||||
if (row) onPickRow(row)
|
onPick(hits.value[highlight.value])
|
||||||
}
|
} else if (allowCreate.value) {
|
||||||
|
onCreate()
|
||||||
// B5 (2026-06-07): when the dropdown is open, Tab accepts the highlighted row
|
|
||||||
// (standard autocomplete convention) instead of leaving the field. With the
|
|
||||||
// list closed it falls through to normal focus traversal.
|
|
||||||
function onTab (e) {
|
|
||||||
if (rows.value.length) {
|
|
||||||
e.preventDefault()
|
|
||||||
onEnter()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -309,14 +176,4 @@ function reset () { query.value = ''; hits.value = []; highlight.value = 0 }
|
|||||||
border: 1px solid rgb(var(--v-theme-surface-light));
|
border: 1px solid rgb(var(--v-theme-surface-light));
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
}
|
}
|
||||||
/* Mark ML-suggestion rows so they read as distinct from typed/known matches. */
|
|
||||||
.fc-tag-autocomplete__sugg {
|
|
||||||
border-left: 2px solid rgb(var(--v-theme-accent), 0.5);
|
|
||||||
}
|
|
||||||
.fc-tag-autocomplete__sugg-tag {
|
|
||||||
display: inline-flex; align-items: center; gap: 2px;
|
|
||||||
font-size: 11px;
|
|
||||||
font-family: 'JetBrains Mono', monospace;
|
|
||||||
color: rgb(var(--v-theme-accent));
|
|
||||||
}
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- One tag chip + its kebab. The kebab uses the SAME explicit-menu pattern
|
|
||||||
as SuggestionItem (activator="parent" + :open-on-click="false" + a manual
|
|
||||||
v-model), because the #activator / v-bind="props" pattern never toggles a
|
|
||||||
v-menu inside the teleported ImageViewer modal (#711, re-fixed 2026-06-07
|
|
||||||
after the first attempt used the broken pattern). -->
|
|
||||||
<span class="fc-tag-chip">
|
|
||||||
<v-chip
|
|
||||||
size="small" closable
|
|
||||||
:color="store.colorFor(tag.kind)" variant="tonal"
|
|
||||||
@click:close="$emit('remove', tag.id)"
|
|
||||||
>
|
|
||||||
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
|
|
||||||
{{ tag.name }}<span v-if="tag.fandom_id">→</span>
|
|
||||||
</v-chip>
|
|
||||||
<span class="fc-tag-chip__menu-wrap">
|
|
||||||
<v-btn
|
|
||||||
class="fc-tag-chip__kebab"
|
|
||||||
icon="mdi-dots-vertical" size="x-small"
|
|
||||||
variant="text" density="comfortable"
|
|
||||||
:aria-label="`More actions for ${tag.name}`"
|
|
||||||
@click.stop="menuOpen = !menuOpen"
|
|
||||||
/>
|
|
||||||
<!-- :z-index above the modal's 2000 — the teleported menu otherwise lands
|
|
||||||
BEHIND .fc-viewer and gets blurred by its backdrop-filter (operator
|
|
||||||
saw it ghosted behind the sidebar, 2026-06-07). THIS is what made the
|
|
||||||
kebab look dead the whole time: it opened, just underneath. -->
|
|
||||||
<v-menu v-model="menuOpen" activator="parent" :open-on-click="false" :z-index="2400">
|
|
||||||
<v-list density="compact">
|
|
||||||
<v-list-item @click="$emit('rename', tag)">
|
|
||||||
<v-list-item-title>Rename…</v-list-item-title>
|
|
||||||
</v-list-item>
|
|
||||||
<v-list-item v-if="tag.kind === 'character'" @click="$emit('set-fandom', tag)">
|
|
||||||
<v-list-item-title>Set fandom…</v-list-item-title>
|
|
||||||
</v-list-item>
|
|
||||||
</v-list>
|
|
||||||
</v-menu>
|
|
||||||
</span>
|
|
||||||
</span>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { ref } from 'vue'
|
|
||||||
import { useTagStore } from '../../stores/tags.js'
|
|
||||||
|
|
||||||
defineProps({ tag: { type: Object, required: true } })
|
|
||||||
defineEmits(['remove', 'rename', 'set-fandom'])
|
|
||||||
|
|
||||||
const store = useTagStore()
|
|
||||||
const menuOpen = ref(false)
|
|
||||||
|
|
||||||
const KIND_ICONS = {
|
|
||||||
general: 'mdi-tag', character: 'mdi-account-circle',
|
|
||||||
fandom: 'mdi-book-open-page-variant', series: 'mdi-bookshelf',
|
|
||||||
meta: 'mdi-cog-outline', rating: 'mdi-shield-check-outline',
|
|
||||||
}
|
|
||||||
function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-tag-chip { display: inline-flex; align-items: center; gap: 1px; }
|
|
||||||
.fc-tag-chip__menu-wrap { display: inline-flex; align-items: center; }
|
|
||||||
.fc-tag-chip__kebab { opacity: 0.7; }
|
|
||||||
.fc-tag-chip:hover .fc-tag-chip__kebab { opacity: 1; }
|
|
||||||
</style>
|
|
||||||
@@ -2,11 +2,36 @@
|
|||||||
<aside class="fc-tag-panel" aria-label="Tags for this image">
|
<aside class="fc-tag-panel" aria-label="Tags for this image">
|
||||||
<h3 class="fc-tag-panel__title">Tags</h3>
|
<h3 class="fc-tag-panel__title">Tags</h3>
|
||||||
<div class="fc-tag-panel__chips">
|
<div class="fc-tag-panel__chips">
|
||||||
<TagChip
|
<v-chip
|
||||||
v-for="tag in modal.current?.tags || []"
|
v-for="tag in modal.current?.tags || []"
|
||||||
:key="tag.id" :tag="tag"
|
:key="tag.id" size="small" closable
|
||||||
@remove="onRemove" @rename="openRename" @set-fandom="openSetFandom"
|
:color="store.colorFor(tag.kind)" variant="tonal"
|
||||||
|
@click:close="onRemove(tag.id)"
|
||||||
|
>
|
||||||
|
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
|
||||||
|
{{ tag.name }}<span v-if="tag.fandom_id">→</span>
|
||||||
|
<!-- Operator-flagged 2026-06-02: the previous activator had
|
||||||
|
`@click.stop` directly on the v-icon, which silently
|
||||||
|
overrode Vuetify's onClick from `v-bind="mp"` — the menu
|
||||||
|
never opened. Now the v-icon receives the activator
|
||||||
|
onClick cleanly, and the wrapping span absorbs the
|
||||||
|
bubbled click so the chip's close button isn't tripped. -->
|
||||||
|
<span class="kebab-wrap" @click.stop>
|
||||||
|
<v-menu>
|
||||||
|
<template #activator="{ props: mp }">
|
||||||
|
<v-icon
|
||||||
|
v-bind="mp" size="x-small" class="ml-1"
|
||||||
|
icon="mdi-dots-vertical"
|
||||||
/>
|
/>
|
||||||
|
</template>
|
||||||
|
<v-list density="compact">
|
||||||
|
<v-list-item @click="openRename(tag)">
|
||||||
|
<v-list-item-title>Rename…</v-list-item-title>
|
||||||
|
</v-list-item>
|
||||||
|
</v-list>
|
||||||
|
</v-menu>
|
||||||
|
</span>
|
||||||
|
</v-chip>
|
||||||
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
|
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -14,7 +39,6 @@
|
|||||||
|
|
||||||
<TagAutocomplete
|
<TagAutocomplete
|
||||||
@pick-existing="onPickExisting" @pick-new="onPickNew"
|
@pick-existing="onPickExisting" @pick-new="onPickNew"
|
||||||
@accept-suggestion="onAcceptSuggestion"
|
|
||||||
/>
|
/>
|
||||||
|
|
||||||
<v-alert v-if="errorMsg" type="error" variant="tonal" class="mt-2" closable>
|
<v-alert v-if="errorMsg" type="error" variant="tonal" class="mt-2" closable>
|
||||||
@@ -33,30 +57,28 @@
|
|||||||
@renamed="onRenamed" @cancel="renameDialog = false"
|
@renamed="onRenamed" @cancel="renameDialog = false"
|
||||||
/>
|
/>
|
||||||
</v-dialog>
|
</v-dialog>
|
||||||
|
|
||||||
<v-dialog v-model="fandomDialog" max-width="460">
|
|
||||||
<FandomSetDialog
|
|
||||||
v-if="fandomTarget" :tag="fandomTarget"
|
|
||||||
@updated="onFandomUpdated" @cancel="fandomDialog = false"
|
|
||||||
/>
|
|
||||||
</v-dialog>
|
|
||||||
</aside>
|
</aside>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { ref } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { useModalStore } from '../../stores/modal.js'
|
import { useModalStore } from '../../stores/modal.js'
|
||||||
import { useSuggestionsStore } from '../../stores/suggestions.js'
|
import { useTagStore } from '../../stores/tags.js'
|
||||||
import TagChip from './TagChip.vue'
|
|
||||||
import TagAutocomplete from './TagAutocomplete.vue'
|
import TagAutocomplete from './TagAutocomplete.vue'
|
||||||
import SuggestionsPanel from './SuggestionsPanel.vue'
|
import SuggestionsPanel from './SuggestionsPanel.vue'
|
||||||
import TagRenameDialog from './TagRenameDialog.vue'
|
import TagRenameDialog from './TagRenameDialog.vue'
|
||||||
import FandomSetDialog from './FandomSetDialog.vue'
|
|
||||||
|
|
||||||
const modal = useModalStore()
|
const modal = useModalStore()
|
||||||
const suggestions = useSuggestionsStore()
|
const store = useTagStore()
|
||||||
const errorMsg = ref(null)
|
const errorMsg = ref(null)
|
||||||
|
|
||||||
|
const KIND_ICONS = {
|
||||||
|
general: 'mdi-tag', character: 'mdi-account-circle',
|
||||||
|
fandom: 'mdi-book-open-page-variant', series: 'mdi-bookshelf',
|
||||||
|
meta: 'mdi-cog-outline', rating: 'mdi-shield-check-outline'
|
||||||
|
}
|
||||||
|
function iconFor(k) { return KIND_ICONS[k] || 'mdi-tag' }
|
||||||
|
|
||||||
async function onRemove(tagId) {
|
async function onRemove(tagId) {
|
||||||
errorMsg.value = null
|
errorMsg.value = null
|
||||||
try { await modal.removeTag(tagId) }
|
try { await modal.removeTag(tagId) }
|
||||||
@@ -72,16 +94,6 @@ async function onPickNew(payload) {
|
|||||||
try { await modal.createAndAdd(payload) }
|
try { await modal.createAndAdd(payload) }
|
||||||
catch (e) { errorMsg.value = e.message }
|
catch (e) { errorMsg.value = e.message }
|
||||||
}
|
}
|
||||||
// A suggestion picked from the autocomplete dropdown runs the SAME path as the
|
|
||||||
// Suggestions panel's Accept: the store creates the tag if it's raw, records the
|
|
||||||
// acceptance, and drops it from the panel; then we refresh the chip rail.
|
|
||||||
async function onAcceptSuggestion(s) {
|
|
||||||
errorMsg.value = null
|
|
||||||
try {
|
|
||||||
await suggestions.accept(s)
|
|
||||||
await modal.reloadTags()
|
|
||||||
} catch (e) { errorMsg.value = e.message }
|
|
||||||
}
|
|
||||||
|
|
||||||
const renameDialog = ref(false)
|
const renameDialog = ref(false)
|
||||||
const renameTarget = ref(null)
|
const renameTarget = ref(null)
|
||||||
@@ -95,18 +107,6 @@ async function onRenamed() {
|
|||||||
// Reflect the new name in the modal's current tag list without a full reload.
|
// Reflect the new name in the modal's current tag list without a full reload.
|
||||||
await modal.reloadTags()
|
await modal.reloadTags()
|
||||||
}
|
}
|
||||||
|
|
||||||
const fandomDialog = ref(false)
|
|
||||||
const fandomTarget = ref(null)
|
|
||||||
function openSetFandom(tag) {
|
|
||||||
fandomTarget.value = tag
|
|
||||||
fandomDialog.value = true
|
|
||||||
}
|
|
||||||
async function onFandomUpdated() {
|
|
||||||
fandomDialog.value = false
|
|
||||||
// A fandom change can merge the tag away; reload to reflect the new state.
|
|
||||||
await modal.reloadTags()
|
|
||||||
}
|
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
@@ -121,4 +121,5 @@ async function onFandomUpdated() {
|
|||||||
margin-bottom: 12px;
|
margin-bottom: 12px;
|
||||||
}
|
}
|
||||||
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
|
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
|
||||||
|
.kebab-wrap { display: inline-flex; align-items: center; }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,150 +1,172 @@
|
|||||||
<template>
|
<template>
|
||||||
<v-card class="fc-post-card" variant="outlined">
|
<v-card
|
||||||
|
:class="['fc-post-card', expanded && 'fc-post-card--expanded']"
|
||||||
|
variant="outlined"
|
||||||
|
:tabindex="expanded ? -1 : 0"
|
||||||
|
@click="onCardClick"
|
||||||
|
@keydown.enter="onCardClick"
|
||||||
|
>
|
||||||
<div class="fc-post-card__head">
|
<div class="fc-post-card__head">
|
||||||
<!-- Posts with no live subscription have source=null (alembic 0030);
|
<!-- Posts with no live subscription have source=null (alembic
|
||||||
show a "filesystem import" affordance instead of a platform chip. -->
|
0030); show a "filesystem import" affordance instead of a
|
||||||
|
platform chip. -->
|
||||||
<v-chip size="x-small" variant="tonal">
|
<v-chip size="x-small" variant="tonal">
|
||||||
{{ post.source?.platform ?? 'filesystem import' }}
|
{{ post.source?.platform ?? 'filesystem import' }}
|
||||||
</v-chip>
|
</v-chip>
|
||||||
<RouterLink
|
<RouterLink
|
||||||
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
|
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
|
||||||
class="fc-post-card__artist"
|
class="fc-post-card__artist"
|
||||||
|
@click.stop
|
||||||
>{{ post.artist.name }}</RouterLink>
|
>{{ post.artist.name }}</RouterLink>
|
||||||
<span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span>
|
<span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span>
|
||||||
<span v-if="totalImages" class="fc-post-card__meta">
|
<span v-if="expanded && images.length" class="fc-post-card__meta">
|
||||||
· {{ totalImages }} image{{ totalImages === 1 ? '' : 's' }}
|
· {{ images.length }} image{{ images.length === 1 ? '' : 's' }}
|
||||||
|
</span>
|
||||||
|
<span v-if="expanded && attachments.length" class="fc-post-card__meta">
|
||||||
|
· {{ attachments.length }} attachment{{ attachments.length === 1 ? '' : 's' }}
|
||||||
</span>
|
</span>
|
||||||
<v-spacer />
|
<v-spacer />
|
||||||
<PostSeriesMenu :post="post" />
|
|
||||||
<v-btn
|
<v-btn
|
||||||
v-if="post.post_url"
|
v-if="post.post_url"
|
||||||
:href="post.post_url" target="_blank" rel="noopener"
|
:href="post.post_url" target="_blank" rel="noopener"
|
||||||
icon="mdi-open-in-new" size="x-small" variant="text"
|
icon="mdi-open-in-new" size="x-small" variant="text"
|
||||||
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
|
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
|
||||||
|
@click.stop
|
||||||
|
/>
|
||||||
|
<v-btn
|
||||||
|
:icon="expanded ? 'mdi-chevron-up' : 'mdi-chevron-down'"
|
||||||
|
size="x-small" variant="text"
|
||||||
|
:aria-label="expanded ? 'Collapse post' : 'Expand post'"
|
||||||
|
@click.stop="toggleExpanded"
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="fc-post-card__body">
|
<!-- Compact body: collapsed card. Hero + thumb rail + truncated text. -->
|
||||||
|
<div v-if="!expanded" class="fc-post-card__body">
|
||||||
<div class="fc-post-card__media">
|
<div class="fc-post-card__media">
|
||||||
<template v-if="images.length">
|
<template v-if="images.length">
|
||||||
<!-- Images open the post-scoped image modal (look bigger + arrow
|
<div class="fc-post-card__hero">
|
||||||
through ALL the post's images) — the card never expands. -->
|
<img :src="hero.thumbnail_url" :alt="`hero thumbnail`" loading="lazy" />
|
||||||
<button
|
</div>
|
||||||
type="button" class="fc-post-card__hero"
|
<div v-if="rail.length" class="fc-post-card__rail">
|
||||||
aria-label="Open images" @click="openModal(hero.image_id)"
|
<div v-for="t in rail" :key="t.image_id" class="fc-post-card__rail-cell">
|
||||||
>
|
<img :src="t.thumbnail_url" :alt="`thumbnail`" loading="lazy" />
|
||||||
<img :src="hero.thumbnail_url" alt="hero thumbnail" loading="lazy" />
|
</div>
|
||||||
</button>
|
<div v-if="moreCount > 0" class="fc-post-card__rail-more">
|
||||||
<div
|
+{{ moreCount }}
|
||||||
v-if="rail.length || moreCount" class="fc-post-card__rail"
|
</div>
|
||||||
:style="{ '--fc-rail-cols': railCols }"
|
|
||||||
>
|
|
||||||
<button
|
|
||||||
v-for="t in rail" :key="t.image_id" type="button"
|
|
||||||
class="fc-post-card__rail-cell"
|
|
||||||
aria-label="Open image" @click="openModal(t.image_id)"
|
|
||||||
>
|
|
||||||
<img :src="t.thumbnail_url" alt="thumbnail" loading="lazy" />
|
|
||||||
</button>
|
|
||||||
<button
|
|
||||||
v-if="moreCount > 0" type="button"
|
|
||||||
class="fc-post-card__rail-more"
|
|
||||||
:aria-label="`Open ${moreCount} more images`"
|
|
||||||
@click="openModalAtMore"
|
|
||||||
>+{{ moreCount }}</button>
|
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
<PostEmptyThumbs v-else />
|
<PostEmptyThumbs v-else />
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="fc-post-card__text">
|
<div class="fc-post-card__text">
|
||||||
<h3 v-if="plainTitle" class="fc-post-card__title">{{ plainTitle }}</h3>
|
<h3 v-if="plainTitle" class="fc-post-card__title">
|
||||||
|
{{ plainTitle }}
|
||||||
|
</h3>
|
||||||
<h3 v-else class="fc-post-card__title fc-post-card__title--missing">
|
<h3 v-else class="fc-post-card__title fc-post-card__title--missing">
|
||||||
Post {{ post.external_post_id }}
|
Post {{ post.external_post_id }}
|
||||||
</h3>
|
</h3>
|
||||||
|
|
||||||
<p
|
<p v-if="post.description_plain" class="fc-post-card__desc">
|
||||||
v-if="hasDescription" ref="descEl"
|
{{ post.description_plain }}
|
||||||
class="fc-post-card__desc"
|
</p>
|
||||||
:class="{ 'fc-post-card__desc--clamped': !descExpanded }"
|
|
||||||
>{{ descText }}</p>
|
|
||||||
<p v-else class="fc-post-card__desc fc-post-card__desc--missing">
|
<p v-else class="fc-post-card__desc fc-post-card__desc--missing">
|
||||||
(no description)
|
(no description)
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<!-- The ONLY in-place expansion: the post text, and only when it's
|
<div v-if="post.attachments?.length" class="fc-post-card__atts">
|
||||||
actually truncated (server flag or a CSS-clamp overflow). -->
|
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
|
||||||
<button
|
{{ post.attachments.length }} attachment{{ post.attachments.length === 1 ? '' : 's' }}
|
||||||
v-if="canExpand" type="button" class="fc-post-card__more"
|
</div>
|
||||||
@click="toggleDesc"
|
</div>
|
||||||
>{{ descExpanded ? 'Show less' : 'Show more' }}</button>
|
</div>
|
||||||
|
|
||||||
<div v-if="attachments.length" class="fc-post-card__atts">
|
<!-- Expanded body: title, full mosaic, full sanitized HTML description,
|
||||||
|
attachments. Lazy-loaded detail via getPostFull. -->
|
||||||
|
<div v-else class="fc-post-card__expanded">
|
||||||
|
<h2 v-if="plainTitle" class="fc-post-card__title-full">
|
||||||
|
{{ plainTitle }}
|
||||||
|
</h2>
|
||||||
|
<h2 v-else class="fc-post-card__title-full fc-post-card__title--missing">
|
||||||
|
Post {{ post.external_post_id }}
|
||||||
|
</h2>
|
||||||
|
|
||||||
|
<section v-if="images.length" class="fc-post-card__sec">
|
||||||
|
<PostImageGrid :thumbnails="images" />
|
||||||
|
<div v-if="!detailLoaded" class="fc-post-card__loading-hint">
|
||||||
|
Loading full image list…
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section v-if="descriptionHtml" class="fc-post-card__sec">
|
||||||
|
<div class="fc-post-card__desc-full" v-html="descriptionHtml" />
|
||||||
|
</section>
|
||||||
|
<section v-else-if="detailLoaded" class="fc-post-card__sec">
|
||||||
|
<p class="fc-post-card__desc fc-post-card__desc--missing">(no description)</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section v-if="attachments.length" class="fc-post-card__sec">
|
||||||
|
<h3 class="fc-post-card__h3">Attachments</h3>
|
||||||
|
<div class="fc-post-card__atts-full">
|
||||||
<a
|
<a
|
||||||
v-for="att in attachments" :key="att.id"
|
v-for="att in attachments" :key="att.id"
|
||||||
:href="att.download_url" download class="fc-post-card__att"
|
:href="att.download_url" download
|
||||||
|
class="fc-post-card__att"
|
||||||
|
@click.stop
|
||||||
>
|
>
|
||||||
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
|
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
|
||||||
<span>{{ att.original_filename }}</span>
|
<span>{{ att.original_filename }}</span>
|
||||||
<span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span>
|
<span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span>
|
||||||
</a>
|
</a>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</section>
|
||||||
</div>
|
</div>
|
||||||
</v-card>
|
</v-card>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, nextTick, onBeforeUnmount, onMounted, ref } from 'vue'
|
import { computed, ref } from 'vue'
|
||||||
import { RouterLink } from 'vue-router'
|
import { RouterLink } from 'vue-router'
|
||||||
|
|
||||||
import { useModalStore } from '../../stores/modal.js'
|
|
||||||
import { usePostsStore } from '../../stores/posts.js'
|
import { usePostsStore } from '../../stores/posts.js'
|
||||||
import { toPlainText } from '../../utils/htmlSanitize.js'
|
import { sanitizeHtml, toPlainText } from '../../utils/htmlSanitize.js'
|
||||||
import PostEmptyThumbs from './PostEmptyThumbs.vue'
|
import PostEmptyThumbs from './PostEmptyThumbs.vue'
|
||||||
import PostSeriesMenu from './PostSeriesMenu.vue'
|
import PostImageGrid from './PostImageGrid.vue'
|
||||||
|
|
||||||
const props = defineProps({
|
const props = defineProps({
|
||||||
post: { type: Object, required: true },
|
post: { type: Object, required: true },
|
||||||
})
|
})
|
||||||
|
|
||||||
const postsStore = usePostsStore()
|
const postsStore = usePostsStore()
|
||||||
const modal = useModalStore()
|
|
||||||
|
|
||||||
// Full detail (uncapped thumbnails + full description), fetched lazily — only
|
// Per-card expand state. No global modal — each PostCard owns its own
|
||||||
// when opening the modal for a post with >6 images, or expanding a
|
// view-mode and lazy-loaded detail.
|
||||||
// server-truncated description.
|
const expanded = ref(false)
|
||||||
const detail = ref(null)
|
const detail = ref(null)
|
||||||
|
const detailLoaded = ref(false)
|
||||||
|
const detailError = ref(null)
|
||||||
|
|
||||||
const attachments = computed(() => props.post.attachments || [])
|
// When expanded + detail loaded, prefer the uncapped detail thumbnails +
|
||||||
const images = computed(() => props.post.thumbnails || [])
|
// full description. Falls back to feed shape if detail fetch is in flight
|
||||||
const totalImages = computed(() => images.value.length + (props.post.thumbnails_more || 0))
|
// or failed.
|
||||||
|
const merged = computed(() => detail.value || props.post)
|
||||||
|
const images = computed(() => merged.value.thumbnails || [])
|
||||||
|
const attachments = computed(() => merged.value.attachments || [])
|
||||||
|
|
||||||
|
// Titles can arrive as stored HTML (e.g. "<strong>…</strong>"); render as
|
||||||
|
// plain text — the CSS makes the title bold.
|
||||||
const plainTitle = computed(() => toPlainText(props.post.post_title))
|
const plainTitle = computed(() => toPlainText(props.post.post_title))
|
||||||
|
|
||||||
const hero = computed(() => images.value[0])
|
// Compact-view hero+rail derived from the feed-shape (capped 6).
|
||||||
// The thumbnail strip spans the hero's full width (CSS grid, equal columns),
|
const hero = computed(() => props.post.thumbnails?.[0])
|
||||||
// rather than a fixed 3-cell cap. Show up to RAIL_MAX cells; when there are
|
const rail = computed(() => (props.post.thumbnails || []).slice(1, 4))
|
||||||
// more images than fit, the last cell becomes a "+N" overflow tile so the
|
|
||||||
// count stays accurate.
|
|
||||||
const RAIL_MAX = 5
|
|
||||||
const serverMore = computed(() => props.post.thumbnails_more || 0)
|
|
||||||
const afterHero = computed(() => images.value.slice(1))
|
|
||||||
const hasOverflow = computed(
|
|
||||||
() => serverMore.value > 0 || afterHero.value.length > RAIL_MAX,
|
|
||||||
)
|
|
||||||
const rail = computed(() =>
|
|
||||||
hasOverflow.value
|
|
||||||
? afterHero.value.slice(0, RAIL_MAX - 1)
|
|
||||||
: afterHero.value.slice(0, RAIL_MAX),
|
|
||||||
)
|
|
||||||
const visibleCount = computed(() => (images.value.length ? 1 + rail.value.length : 0))
|
|
||||||
const moreCount = computed(() => {
|
const moreCount = computed(() => {
|
||||||
if (!hasOverflow.value) return 0
|
const more = props.post.thumbnails_more || 0
|
||||||
return serverMore.value + Math.max(0, images.value.length - visibleCount.value)
|
const railLen = rail.value.length
|
||||||
|
const extraShown = Math.max(0, (props.post.thumbnails?.length || 0) - 1 - railLen)
|
||||||
|
return more + extraShown
|
||||||
})
|
})
|
||||||
// Columns = thumbnails shown plus the overflow tile, so the strip fills the
|
|
||||||
// hero's full width with equal cells.
|
|
||||||
const railCols = computed(() => rail.value.length + (moreCount.value > 0 ? 1 : 0))
|
|
||||||
|
|
||||||
const sortDateIso = computed(() => props.post.post_date || props.post.downloaded_at)
|
const sortDateIso = computed(() => props.post.post_date || props.post.downloaded_at)
|
||||||
const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString())
|
const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString())
|
||||||
@@ -158,79 +180,51 @@ const relativeDate = computed(() => {
|
|||||||
return new Date(sortDateIso.value).toLocaleDateString()
|
return new Date(sortDateIso.value).toLocaleDateString()
|
||||||
})
|
})
|
||||||
|
|
||||||
// --- images → post-scoped modal ---------------------------------------
|
const descriptionHtml = computed(() => {
|
||||||
async function fullImageIds () {
|
// Detail endpoint returns description_full as plain text (the service
|
||||||
// Feed caps thumbnails at 6; load detail for the complete id list only when
|
// uses html_to_plain on the stored description). Render plain text in
|
||||||
// there are more, so the modal can arrow through ALL of the post's images.
|
// <p> wrappers; sanitize defensively in case the backend ever returns
|
||||||
if ((props.post.thumbnails_more || 0) === 0) {
|
// raw HTML.
|
||||||
return images.value.map((t) => t.image_id)
|
const raw = merged.value.description_full || merged.value.description_plain
|
||||||
}
|
if (!raw) return ''
|
||||||
if (!detail.value) {
|
if (/[<>]/.test(raw)) return sanitizeHtml(raw)
|
||||||
try {
|
const esc = raw
|
||||||
detail.value = await postsStore.getPostFull(props.post.id)
|
.replace(/&/g, '&')
|
||||||
} catch { /* fall back to the capped feed list */ }
|
.replace(/</g, '<')
|
||||||
}
|
.replace(/>/g, '>')
|
||||||
return (detail.value?.thumbnails || images.value).map((t) => t.image_id)
|
return esc
|
||||||
}
|
.split(/\n\s*\n/)
|
||||||
|
.map((p) => `<p>${p.replace(/\n/g, '<br>')}</p>`)
|
||||||
async function openModal (imageId) {
|
.join('')
|
||||||
modal.open(imageId, { postImageIds: await fullImageIds() })
|
|
||||||
}
|
|
||||||
|
|
||||||
async function openModalAtMore () {
|
|
||||||
const ids = await fullImageIds()
|
|
||||||
const first = ids[visibleCount.value] ?? ids[0]
|
|
||||||
if (first != null) modal.open(first, { postImageIds: ids })
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- description "Show more" (text-only, in place, only when truncated) ----
|
|
||||||
const descExpanded = ref(false)
|
|
||||||
const cssOverflow = ref(false)
|
|
||||||
const descEl = ref(null)
|
|
||||||
|
|
||||||
const hasDescription = computed(() => !!props.post.description_plain)
|
|
||||||
const fullDescription = computed(() => detail.value?.description_full || null)
|
|
||||||
const descText = computed(() =>
|
|
||||||
descExpanded.value
|
|
||||||
? (fullDescription.value || props.post.description_plain)
|
|
||||||
: props.post.description_plain,
|
|
||||||
)
|
|
||||||
// Show the toggle iff the server truncated the text OR the clamp is cutting it.
|
|
||||||
const canExpand = computed(
|
|
||||||
() => props.post.description_truncated === true || cssOverflow.value,
|
|
||||||
)
|
|
||||||
|
|
||||||
function measureOverflow () {
|
|
||||||
const el = descEl.value
|
|
||||||
cssOverflow.value = !!el && el.scrollHeight > el.clientHeight + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
let ro = null
|
|
||||||
onMounted(() => {
|
|
||||||
nextTick(measureOverflow)
|
|
||||||
// Re-measure when the card resizes (the container-query clamp differs by
|
|
||||||
// width). Guarded for happy-dom / older runtimes without ResizeObserver.
|
|
||||||
if (typeof ResizeObserver !== 'undefined' && descEl.value) {
|
|
||||||
ro = new ResizeObserver(() => { if (!descExpanded.value) measureOverflow() })
|
|
||||||
ro.observe(descEl.value)
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
onBeforeUnmount(() => { if (ro) { ro.disconnect(); ro = null } })
|
|
||||||
|
|
||||||
async function toggleDesc () {
|
async function loadDetailIfNeeded () {
|
||||||
if (!descExpanded.value) {
|
if (detailLoaded.value || detail.value) return
|
||||||
if (props.post.description_truncated && !fullDescription.value && !detail.value) {
|
|
||||||
try {
|
try {
|
||||||
detail.value = await postsStore.getPostFull(props.post.id)
|
detail.value = await postsStore.getPostFull(props.post.id)
|
||||||
} catch { /* render the truncated text rather than nothing */ }
|
detailLoaded.value = true
|
||||||
}
|
} catch (e) {
|
||||||
descExpanded.value = true
|
detailError.value = e.message
|
||||||
} else {
|
// Leave merged on feed-shape; the card still renders the truncated
|
||||||
descExpanded.value = false
|
// body so the operator isn't staring at a blank panel.
|
||||||
nextTick(measureOverflow)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function toggleExpanded () {
|
||||||
|
expanded.value = !expanded.value
|
||||||
|
if (expanded.value) loadDetailIfNeeded()
|
||||||
|
}
|
||||||
|
|
||||||
|
function onCardClick (e) {
|
||||||
|
// Inner interactive elements use @click.stop so they never reach here.
|
||||||
|
// Whole-card click expands a collapsed card; collapsing is chevron-only
|
||||||
|
// so a mosaic-image click on an expanded card can never accidentally
|
||||||
|
// collapse the surrounding card.
|
||||||
|
if (expanded.value) return
|
||||||
|
expanded.value = true
|
||||||
|
loadDetailIfNeeded()
|
||||||
|
}
|
||||||
|
|
||||||
function formatBytes (n) {
|
function formatBytes (n) {
|
||||||
if (!n) return '0 B'
|
if (!n) return '0 B'
|
||||||
if (n < 1024) return `${n} B`
|
if (n < 1024) return `${n} B`
|
||||||
@@ -245,6 +239,20 @@ function formatBytes (n) {
|
|||||||
padding: 1rem;
|
padding: 1rem;
|
||||||
margin-bottom: 1rem;
|
margin-bottom: 1rem;
|
||||||
container-type: inline-size;
|
container-type: inline-size;
|
||||||
|
transition: border-color 0.15s ease;
|
||||||
|
}
|
||||||
|
.fc-post-card:not(.fc-post-card--expanded) {
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.fc-post-card:not(.fc-post-card--expanded):hover {
|
||||||
|
border-color: rgb(var(--v-theme-accent));
|
||||||
|
}
|
||||||
|
.fc-post-card:focus-visible {
|
||||||
|
outline: 2px solid rgb(var(--v-theme-accent));
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
.fc-post-card--expanded {
|
||||||
|
border-color: rgb(var(--v-theme-accent) / 0.6);
|
||||||
}
|
}
|
||||||
|
|
||||||
.fc-post-card__head {
|
.fc-post-card__head {
|
||||||
@@ -265,6 +273,7 @@ function formatBytes (n) {
|
|||||||
.fc-post-card__date,
|
.fc-post-card__date,
|
||||||
.fc-post-card__meta { white-space: nowrap; }
|
.fc-post-card__meta { white-space: nowrap; }
|
||||||
|
|
||||||
|
/* ---- COMPACT BODY ---- */
|
||||||
.fc-post-card__body {
|
.fc-post-card__body {
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
@@ -279,13 +288,6 @@ function formatBytes (n) {
|
|||||||
.fc-post-card__text { flex: 1 1 0; min-width: 0; }
|
.fc-post-card__text { flex: 1 1 0; min-width: 0; }
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Image tiles are buttons (open the post-scoped modal) — reset button chrome. */
|
|
||||||
.fc-post-card__hero,
|
|
||||||
.fc-post-card__rail-cell,
|
|
||||||
.fc-post-card__rail-more {
|
|
||||||
display: block; padding: 0; border: 0; background: none;
|
|
||||||
cursor: pointer;
|
|
||||||
}
|
|
||||||
.fc-post-card__hero {
|
.fc-post-card__hero {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
aspect-ratio: 16 / 10;
|
aspect-ratio: 16 / 10;
|
||||||
@@ -295,21 +297,13 @@ function formatBytes (n) {
|
|||||||
.fc-post-card__hero img {
|
.fc-post-card__hero img {
|
||||||
width: 100%; height: 100%;
|
width: 100%; height: 100%;
|
||||||
object-fit: cover; display: block;
|
object-fit: cover; display: block;
|
||||||
transition: transform 0.2s ease, filter 0.2s ease;
|
|
||||||
}
|
}
|
||||||
.fc-post-card__hero:hover img,
|
|
||||||
.fc-post-card__rail-cell:hover img { transform: scale(1.03); filter: brightness(1.08); }
|
|
||||||
|
|
||||||
/* Equal columns spanning the hero's full width — cells stretch horizontally
|
|
||||||
(1fr) at a fixed height, so the strip always reaches the edge of the hero
|
|
||||||
regardless of how many thumbnails the post has. */
|
|
||||||
.fc-post-card__rail {
|
.fc-post-card__rail {
|
||||||
display: grid;
|
display: flex; gap: 6px; margin-top: 6px;
|
||||||
grid-template-columns: repeat(var(--fc-rail-cols, 3), 1fr);
|
|
||||||
gap: 6px; margin-top: 6px;
|
|
||||||
}
|
}
|
||||||
.fc-post-card__rail-cell {
|
.fc-post-card__rail-cell {
|
||||||
width: 100%; height: 84px;
|
width: 80px; height: 80px;
|
||||||
overflow: hidden; border-radius: 4px;
|
overflow: hidden; border-radius: 4px;
|
||||||
}
|
}
|
||||||
.fc-post-card__rail-cell img {
|
.fc-post-card__rail-cell img {
|
||||||
@@ -317,17 +311,13 @@ function formatBytes (n) {
|
|||||||
object-fit: cover; display: block;
|
object-fit: cover; display: block;
|
||||||
}
|
}
|
||||||
.fc-post-card__rail-more {
|
.fc-post-card__rail-more {
|
||||||
width: 100%; height: 84px;
|
width: 80px; height: 80px;
|
||||||
display: flex; align-items: center; justify-content: center;
|
display: flex; align-items: center; justify-content: center;
|
||||||
border: 1px dashed rgb(var(--v-theme-on-surface-variant));
|
border: 1px dashed rgb(var(--v-theme-on-surface-variant));
|
||||||
border-radius: 4px;
|
border-radius: 4px;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
font-size: 0.85rem;
|
font-size: 0.85rem;
|
||||||
}
|
}
|
||||||
.fc-post-card__rail-more:hover {
|
|
||||||
border-color: rgb(var(--v-theme-accent));
|
|
||||||
color: rgb(var(--v-theme-accent));
|
|
||||||
}
|
|
||||||
|
|
||||||
.fc-post-card__title {
|
.fc-post-card__title {
|
||||||
font-family: 'Fraunces', Georgia, serif;
|
font-family: 'Fraunces', Georgia, serif;
|
||||||
@@ -356,36 +346,67 @@ function formatBytes (n) {
|
|||||||
font-size: 0.9rem;
|
font-size: 0.9rem;
|
||||||
line-height: 1.5;
|
line-height: 1.5;
|
||||||
color: rgb(var(--v-theme-on-surface));
|
color: rgb(var(--v-theme-on-surface));
|
||||||
margin: 0;
|
margin: 0 0 12px 0;
|
||||||
white-space: pre-wrap;
|
|
||||||
}
|
|
||||||
/* Clamp ONLY while collapsed; expanding drops the clamp to show it all. */
|
|
||||||
.fc-post-card__desc--clamped {
|
|
||||||
display: -webkit-box;
|
display: -webkit-box;
|
||||||
-webkit-line-clamp: 3;
|
-webkit-line-clamp: 3;
|
||||||
-webkit-box-orient: vertical;
|
-webkit-box-orient: vertical;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
}
|
}
|
||||||
@container (min-width: 800px) {
|
|
||||||
.fc-post-card__desc--clamped { -webkit-line-clamp: 5; }
|
|
||||||
}
|
|
||||||
.fc-post-card__desc--missing {
|
.fc-post-card__desc--missing {
|
||||||
font-style: italic;
|
font-style: italic;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
}
|
}
|
||||||
|
@container (min-width: 800px) {
|
||||||
.fc-post-card__more {
|
.fc-post-card__desc { -webkit-line-clamp: 5; }
|
||||||
margin-top: 6px;
|
|
||||||
padding: 0;
|
|
||||||
background: none; border: 0; cursor: pointer;
|
|
||||||
color: rgb(var(--v-theme-accent));
|
|
||||||
font-size: 0.85rem; font-weight: 600;
|
|
||||||
}
|
}
|
||||||
.fc-post-card__more:hover { text-decoration: underline; }
|
|
||||||
|
|
||||||
.fc-post-card__atts {
|
.fc-post-card__atts {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
|
}
|
||||||
|
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
|
||||||
|
|
||||||
|
/* ---- EXPANDED BODY ---- */
|
||||||
|
.fc-post-card__expanded {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 20px;
|
||||||
|
}
|
||||||
|
.fc-post-card__title-full {
|
||||||
|
font-family: 'Fraunces', Georgia, serif;
|
||||||
|
font-size: 22px;
|
||||||
|
font-weight: 700;
|
||||||
|
margin: 0;
|
||||||
|
color: rgb(var(--v-theme-on-surface));
|
||||||
|
}
|
||||||
|
@container (min-width: 800px) {
|
||||||
|
.fc-post-card__title-full { font-size: 26px; }
|
||||||
|
}
|
||||||
|
.fc-post-card__sec { margin: 0; }
|
||||||
|
.fc-post-card__h3 {
|
||||||
|
font-family: 'Fraunces', Georgia, serif;
|
||||||
|
font-size: 16px;
|
||||||
|
font-weight: 500;
|
||||||
|
margin: 0 0 8px 0;
|
||||||
|
color: rgb(var(--v-theme-on-surface));
|
||||||
|
}
|
||||||
|
.fc-post-card__loading-hint {
|
||||||
|
margin-top: 8px;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
|
}
|
||||||
|
.fc-post-card__desc-full {
|
||||||
|
font-size: 0.95rem;
|
||||||
|
line-height: 1.55;
|
||||||
|
color: rgb(var(--v-theme-on-surface));
|
||||||
|
}
|
||||||
|
.fc-post-card__desc-full :deep(p) { margin: 0 0 12px 0; }
|
||||||
|
.fc-post-card__desc-full :deep(a) { color: rgb(var(--v-theme-accent)); }
|
||||||
|
.fc-post-card__atts-full {
|
||||||
display: flex; flex-wrap: wrap; gap: 8px;
|
display: flex; flex-wrap: wrap; gap: 8px;
|
||||||
margin-top: 12px;
|
|
||||||
}
|
}
|
||||||
.fc-post-card__att {
|
.fc-post-card__att {
|
||||||
display: inline-flex;
|
display: inline-flex;
|
||||||
@@ -402,6 +423,5 @@ function formatBytes (n) {
|
|||||||
color: rgb(var(--v-theme-accent));
|
color: rgb(var(--v-theme-accent));
|
||||||
border-color: rgb(var(--v-theme-accent));
|
border-color: rgb(var(--v-theme-accent));
|
||||||
}
|
}
|
||||||
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
|
.fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
<template>
|
||||||
|
<div class="fc-post-grid">
|
||||||
|
<button
|
||||||
|
v-for="(t, idx) in thumbnails"
|
||||||
|
:key="t.image_id"
|
||||||
|
type="button"
|
||||||
|
class="fc-post-grid__cell"
|
||||||
|
:aria-label="`Open image ${idx + 1} of ${thumbnails.length}`"
|
||||||
|
@click="openImage(t.image_id, idx)"
|
||||||
|
>
|
||||||
|
<img
|
||||||
|
:src="t.thumbnail_url"
|
||||||
|
:alt="`thumbnail ${idx + 1}`"
|
||||||
|
loading="lazy"
|
||||||
|
/>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { computed } from 'vue'
|
||||||
|
import { useModalStore } from '../../stores/modal.js'
|
||||||
|
|
||||||
|
const props = defineProps({
|
||||||
|
thumbnails: { type: Array, required: true }, // [{ image_id, thumbnail_url, ... }]
|
||||||
|
})
|
||||||
|
|
||||||
|
const modal = useModalStore()
|
||||||
|
|
||||||
|
const imageIds = computed(() => props.thumbnails.map(t => t.image_id))
|
||||||
|
|
||||||
|
function openImage (id, idx) {
|
||||||
|
modal.open(id, { postImageIds: imageIds.value, initialIndex: idx })
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.fc-post-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
|
||||||
|
gap: 6px;
|
||||||
|
}
|
||||||
|
.fc-post-grid__cell {
|
||||||
|
aspect-ratio: 4 / 3;
|
||||||
|
overflow: hidden;
|
||||||
|
border-radius: 4px;
|
||||||
|
cursor: pointer;
|
||||||
|
border: 0;
|
||||||
|
padding: 0;
|
||||||
|
background: rgb(var(--v-theme-background));
|
||||||
|
transition: transform 0.15s ease, box-shadow 0.15s ease;
|
||||||
|
}
|
||||||
|
.fc-post-grid__cell:hover {
|
||||||
|
transform: scale(1.02);
|
||||||
|
box-shadow: 0 0 0 2px rgb(var(--v-theme-accent));
|
||||||
|
}
|
||||||
|
.fc-post-grid__cell img {
|
||||||
|
width: 100%;
|
||||||
|
height: 100%;
|
||||||
|
object-fit: cover;
|
||||||
|
display: block;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
<template>
|
|
||||||
<span class="fc-post-series">
|
|
||||||
<v-btn
|
|
||||||
size="x-small" variant="text" prepend-icon="mdi-book-plus-outline"
|
|
||||||
append-icon="mdi-menu-down"
|
|
||||||
:aria-label="`Add post ${post.id} to a series`"
|
|
||||||
@click.stop="menuOpen = !menuOpen"
|
|
||||||
>Series</v-btn>
|
|
||||||
<v-menu v-model="menuOpen" activator="parent" :open-on-click="false">
|
|
||||||
<v-list density="compact">
|
|
||||||
<v-list-item :disabled="busy" @click="onPromote">
|
|
||||||
<template #prepend>
|
|
||||||
<v-icon size="small">mdi-book-plus</v-icon>
|
|
||||||
</template>
|
|
||||||
<v-list-item-title>New series from this post</v-list-item-title>
|
|
||||||
</v-list-item>
|
|
||||||
<v-list-item @click="openPicker">
|
|
||||||
<template #prepend>
|
|
||||||
<v-icon size="small">mdi-playlist-plus</v-icon>
|
|
||||||
</template>
|
|
||||||
<v-list-item-title>Add to existing series…</v-list-item-title>
|
|
||||||
</v-list-item>
|
|
||||||
</v-list>
|
|
||||||
</v-menu>
|
|
||||||
|
|
||||||
<v-dialog v-model="pickerOpen" max-width="460">
|
|
||||||
<v-card>
|
|
||||||
<v-card-title>Add to series</v-card-title>
|
|
||||||
<v-card-text>
|
|
||||||
<p class="text-caption mb-2">
|
|
||||||
Appends this post as the next chapter of the chosen series.
|
|
||||||
</p>
|
|
||||||
<v-autocomplete
|
|
||||||
v-model="picked"
|
|
||||||
:items="hits"
|
|
||||||
:item-title="(s) => s.name"
|
|
||||||
:item-value="(s) => s.id"
|
|
||||||
:loading="searching"
|
|
||||||
label="Series" density="compact" autofocus clearable
|
|
||||||
/>
|
|
||||||
</v-card-text>
|
|
||||||
<v-card-actions>
|
|
||||||
<v-spacer />
|
|
||||||
<v-btn @click="pickerOpen = false">Cancel</v-btn>
|
|
||||||
<v-btn
|
|
||||||
color="primary" rounded="pill"
|
|
||||||
:disabled="picked == null || busy" @click="onAddExisting"
|
|
||||||
>Add</v-btn>
|
|
||||||
</v-card-actions>
|
|
||||||
</v-card>
|
|
||||||
</v-dialog>
|
|
||||||
</span>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { ref } from 'vue'
|
|
||||||
import { useRouter } from 'vue-router'
|
|
||||||
import { useApi } from '../../composables/useApi.js'
|
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
|
|
||||||
const props = defineProps({ post: { type: Object, required: true } })
|
|
||||||
|
|
||||||
const api = useApi()
|
|
||||||
const router = useRouter()
|
|
||||||
|
|
||||||
const menuOpen = ref(false)
|
|
||||||
const busy = ref(false)
|
|
||||||
const pickerOpen = ref(false)
|
|
||||||
const picked = ref(null)
|
|
||||||
const hits = ref([])
|
|
||||||
const searching = ref(false)
|
|
||||||
|
|
||||||
async function onPromote() {
|
|
||||||
menuOpen.value = false
|
|
||||||
busy.value = true
|
|
||||||
try {
|
|
||||||
const out = await api.post('/api/series/from-post', {
|
|
||||||
body: { post_id: props.post.id }
|
|
||||||
})
|
|
||||||
toast({ text: `Series created: ${out.name}`, type: 'success' })
|
|
||||||
router.push({ name: 'series-manage', params: { tagId: out.series_tag_id } })
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: `Could not create series: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function openPicker() {
|
|
||||||
menuOpen.value = false
|
|
||||||
picked.value = null
|
|
||||||
hits.value = []
|
|
||||||
pickerOpen.value = true
|
|
||||||
// Load the full series list so the picker is browsable (the tag autocomplete
|
|
||||||
// returns nothing for an empty query — the #712 trap). v-autocomplete filters
|
|
||||||
// client-side by name as the operator types.
|
|
||||||
searching.value = true
|
|
||||||
try {
|
|
||||||
const body = await api.get('/api/series', { params: { sort: 'name' } })
|
|
||||||
hits.value = body.series || []
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: `Could not load series: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
searching.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function onAddExisting() {
|
|
||||||
if (picked.value == null) return
|
|
||||||
busy.value = true
|
|
||||||
try {
|
|
||||||
await api.post(`/api/series/${picked.value}/add-post`, {
|
|
||||||
body: { post_id: props.post.id }
|
|
||||||
})
|
|
||||||
pickerOpen.value = false
|
|
||||||
toast({ text: 'Added to series', type: 'success' })
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: `Could not add to series: ${e.message}`, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
@@ -13,7 +13,7 @@
|
|||||||
clearable
|
clearable
|
||||||
no-filter
|
no-filter
|
||||||
return-object
|
return-object
|
||||||
class="fc-posts-filters__artist"
|
style="min-width: 240px"
|
||||||
@update:search="onArtistSearch"
|
@update:search="onArtistSearch"
|
||||||
@update:model-value="onArtistPicked"
|
@update:model-value="onArtistPicked"
|
||||||
/>
|
/>
|
||||||
@@ -25,7 +25,7 @@
|
|||||||
density="compact"
|
density="compact"
|
||||||
hide-details
|
hide-details
|
||||||
clearable
|
clearable
|
||||||
class="fc-posts-filters__platform"
|
style="min-width: 180px"
|
||||||
@update:model-value="emitFilters"
|
@update:model-value="emitFilters"
|
||||||
/>
|
/>
|
||||||
|
|
||||||
@@ -135,14 +135,6 @@ watch(() => props.platform, (val) => {
|
|||||||
display: flex;
|
display: flex;
|
||||||
gap: 0.75rem;
|
gap: 0.75rem;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
flex-wrap: wrap;
|
|
||||||
padding-bottom: 1rem;
|
padding-bottom: 1rem;
|
||||||
}
|
}
|
||||||
.fc-posts-filters__artist { min-width: 240px; flex: 1 1 240px; max-width: 360px; }
|
|
||||||
.fc-posts-filters__platform { min-width: 180px; flex: 1 1 180px; max-width: 240px; }
|
|
||||||
/* Phones: each field takes a full-width row instead of overflowing. */
|
|
||||||
@media (max-width: 600px) {
|
|
||||||
.fc-posts-filters__artist,
|
|
||||||
.fc-posts-filters__platform { min-width: 100%; max-width: none; flex-basis: 100%; }
|
|
||||||
}
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,46 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- #713: re-extract PostAttachments that are really archives but were filed
|
|
||||||
opaquely before the magic-byte gate, and attach their images to the post. -->
|
|
||||||
<v-card>
|
|
||||||
<v-card-title>Re-extract archive attachments</v-card-title>
|
|
||||||
<v-card-text>
|
|
||||||
<p class="text-body-2 mb-3">
|
|
||||||
Some posts attached an archive (zip) whose images weren't extracted
|
|
||||||
because the downloaded file had no usable extension. This scans existing
|
|
||||||
attachments, extracts any that are really archives, and attaches their
|
|
||||||
images to the post. Idempotent — safe to run more than once.
|
|
||||||
</p>
|
|
||||||
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
|
|
||||||
<v-icon start>mdi-folder-zip-outline</v-icon> Re-extract archives now
|
|
||||||
</v-btn>
|
|
||||||
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
|
|
||||||
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
|
|
||||||
</v-card-text>
|
|
||||||
</v-card>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { ref } from 'vue'
|
|
||||||
|
|
||||||
import { useApi } from '../../composables/useApi.js'
|
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
|
||||||
|
|
||||||
const api = useApi()
|
|
||||||
const busy = ref(false)
|
|
||||||
const queued = ref(false)
|
|
||||||
|
|
||||||
async function run () {
|
|
||||||
busy.value = true
|
|
||||||
queued.value = false
|
|
||||||
try {
|
|
||||||
await api.post('/api/admin/maintenance/reextract-archives')
|
|
||||||
queued.value = true
|
|
||||||
toast({ text: 'Archive re-extract queued', type: 'success' })
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: e?.body?.detail || e?.message || 'Failed to queue', type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
<template>
|
|
||||||
<v-card>
|
|
||||||
<v-card-title>Database maintenance</v-card-title>
|
|
||||||
<v-card-text>
|
|
||||||
<p class="text-body-2 mb-3">
|
|
||||||
VACUUM (ANALYZE) reclaims dead-tuple bloat — which slows the random
|
|
||||||
showcase, since it samples physical blocks — and refreshes the query
|
|
||||||
planner's statistics. Runs automatically each week; trigger a pass
|
|
||||||
here after a large import or cleanup.
|
|
||||||
</p>
|
|
||||||
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
|
|
||||||
<v-icon start>mdi-database-cog</v-icon> Run VACUUM ANALYZE now
|
|
||||||
</v-btn>
|
|
||||||
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
|
|
||||||
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
|
|
||||||
|
|
||||||
<v-table
|
|
||||||
v-if="store.tables.length" density="compact" class="mt-4 fc-dbm__table"
|
|
||||||
>
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Table</th>
|
|
||||||
<th class="text-right">Live rows</th>
|
|
||||||
<th class="text-right">Dead</th>
|
|
||||||
<th class="text-right">Dead %</th>
|
|
||||||
<th>Last vacuum</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr v-for="t in store.tables" :key="t.table">
|
|
||||||
<td>{{ t.table }}</td>
|
|
||||||
<td class="text-right">{{ t.live.toLocaleString() }}</td>
|
|
||||||
<td class="text-right">{{ t.dead.toLocaleString() }}</td>
|
|
||||||
<td
|
|
||||||
class="text-right"
|
|
||||||
:class="{ 'text-warning': t.dead_pct >= 20 }"
|
|
||||||
>{{ t.dead_pct }}%</td>
|
|
||||||
<td class="text-caption">{{ fmt(t.last_vacuum || t.last_autovacuum) }}</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</v-table>
|
|
||||||
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
|
|
||||||
No table statistics yet.
|
|
||||||
</p>
|
|
||||||
</v-card-text>
|
|
||||||
</v-card>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { onMounted, ref } from 'vue'
|
|
||||||
|
|
||||||
import { toast } from '../../utils/toast.js'
|
|
||||||
import { useDbMaintenanceStore } from '../../stores/dbMaintenance.js'
|
|
||||||
import QueueStatusBar from './QueueStatusBar.vue'
|
|
||||||
|
|
||||||
const store = useDbMaintenanceStore()
|
|
||||||
const busy = ref(false)
|
|
||||||
const queued = ref(false)
|
|
||||||
|
|
||||||
onMounted(() => store.loadStats())
|
|
||||||
|
|
||||||
async function run () {
|
|
||||||
busy.value = true
|
|
||||||
queued.value = false
|
|
||||||
try {
|
|
||||||
await store.runVacuum()
|
|
||||||
queued.value = true
|
|
||||||
// pg_stat updates after the vacuum lands — refresh shortly after.
|
|
||||||
setTimeout(() => store.loadStats(), 5000)
|
|
||||||
} catch (e) {
|
|
||||||
toast({ text: e.message, type: 'error' })
|
|
||||||
} finally {
|
|
||||||
busy.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function fmt (iso) {
|
|
||||||
return iso ? new Date(iso).toLocaleString() : '—'
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-dbm__table { background: transparent; }
|
|
||||||
</style>
|
|
||||||
@@ -2,42 +2,6 @@
|
|||||||
<v-card>
|
<v-card>
|
||||||
<v-card-title>Import filters</v-card-title>
|
<v-card-title>Import filters</v-card-title>
|
||||||
<v-card-text v-if="store.settings">
|
<v-card-text v-if="store.settings">
|
||||||
<!-- Near-duplicate dedup sensitivity, hoisted to the top: it's the
|
|
||||||
most-asked knob — too loose and edits/variants of the same image
|
|
||||||
get dropped as duplicates on import. Slider with sane labelled
|
|
||||||
stops for the gist + a number field for precision; both bind the
|
|
||||||
same phash_threshold. -->
|
|
||||||
<div class="fc-phash">
|
|
||||||
<div class="fc-phash__title">Near-duplicate sensitivity</div>
|
|
||||||
<div class="fc-help mb-1">
|
|
||||||
How aggressively imports merge look-alike images (perceptual-hash
|
|
||||||
distance). <strong>Lower it if edits/variants of the same image are
|
|
||||||
being dropped as duplicates;</strong> raise it to collapse more
|
|
||||||
look-alikes. Applies to new imports.
|
|
||||||
</div>
|
|
||||||
<v-row align="center" no-gutters>
|
|
||||||
<v-col cols="12" sm="9">
|
|
||||||
<v-slider
|
|
||||||
v-model="local.phash_threshold"
|
|
||||||
:min="0" :max="16" :step="1"
|
|
||||||
:ticks="PHASH_TICKS" show-ticks="always" tick-size="4"
|
|
||||||
thumb-label color="accent" hide-details
|
|
||||||
class="fc-phash__slider"
|
|
||||||
@end="save"
|
|
||||||
/>
|
|
||||||
</v-col>
|
|
||||||
<v-col cols="8" sm="3" class="ps-sm-4 mt-2 mt-sm-0">
|
|
||||||
<v-text-field
|
|
||||||
v-model.number="local.phash_threshold"
|
|
||||||
label="Distance" type="number" min="0"
|
|
||||||
density="compact" hide-details @blur="save"
|
|
||||||
/>
|
|
||||||
</v-col>
|
|
||||||
</v-row>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-divider class="my-5" />
|
|
||||||
|
|
||||||
<v-row>
|
<v-row>
|
||||||
<v-col cols="12" sm="6">
|
<v-col cols="12" sm="6">
|
||||||
<v-text-field
|
<v-text-field
|
||||||
@@ -77,6 +41,14 @@
|
|||||||
:disabled="!local.skip_single_color" @end="save"
|
:disabled="!local.skip_single_color" @end="save"
|
||||||
/>
|
/>
|
||||||
</v-col>
|
</v-col>
|
||||||
|
<v-col cols="12" sm="6">
|
||||||
|
<v-text-field
|
||||||
|
v-model.number="local.phash_threshold"
|
||||||
|
label="Perceptual-hash threshold" type="number" min="0"
|
||||||
|
density="compact" hide-details @blur="save"
|
||||||
|
/>
|
||||||
|
<div class="fc-help">Higher = looser near-duplicate matching.</div>
|
||||||
|
</v-col>
|
||||||
</v-row>
|
</v-row>
|
||||||
|
|
||||||
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
|
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
|
||||||
@@ -94,9 +66,6 @@ import { reactive, watch } from 'vue'
|
|||||||
import { useImportStore } from '../../stores/import.js'
|
import { useImportStore } from '../../stores/import.js'
|
||||||
|
|
||||||
const store = useImportStore()
|
const store = useImportStore()
|
||||||
// Labelled stops so the less-initiated get the gist without knowing what a
|
|
||||||
// Hamming distance is. 0 = byte-for-byte only; 10 = the shipped default.
|
|
||||||
const PHASH_TICKS = { 0: 'Exact', 4: 'Strict', 10: 'Default', 16: 'Loose' }
|
|
||||||
// Downloader + schedule-defaults fields moved to
|
// Downloader + schedule-defaults fields moved to
|
||||||
// /subscriptions?tab=settings (operator decision 2026-05-27). This form
|
// /subscriptions?tab=settings (operator decision 2026-05-27). This form
|
||||||
// now only owns image-import filters.
|
// now only owns image-import filters.
|
||||||
@@ -120,11 +89,4 @@ async function save() {
|
|||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
margin-top: 2px;
|
margin-top: 2px;
|
||||||
}
|
}
|
||||||
.fc-phash__title {
|
|
||||||
font-size: 0.95rem;
|
|
||||||
font-weight: 600;
|
|
||||||
color: rgb(var(--v-theme-on-surface));
|
|
||||||
}
|
|
||||||
/* Headroom so the tick labels (Exact/Strict/Default/Loose) aren't clipped. */
|
|
||||||
.fc-phash__slider { margin-bottom: 18px; }
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -14,8 +14,6 @@
|
|||||||
<MLThresholdSliders class="mt-4" />
|
<MLThresholdSliders class="mt-4" />
|
||||||
<AllowlistTable class="mt-4" />
|
<AllowlistTable class="mt-4" />
|
||||||
<AliasTable class="mt-4" />
|
<AliasTable class="mt-4" />
|
||||||
<DbMaintenanceCard class="mt-6" />
|
|
||||||
<ArchiveReextractCard class="mt-6" />
|
|
||||||
<BackupCard class="mt-6" />
|
<BackupCard class="mt-6" />
|
||||||
<!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) — it
|
<!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) — it
|
||||||
operates on the existing library which fits the Cleanup-tab
|
operates on the existing library which fits the Cleanup-tab
|
||||||
@@ -33,8 +31,6 @@ import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
|
|||||||
import MLThresholdSliders from './MLThresholdSliders.vue'
|
import MLThresholdSliders from './MLThresholdSliders.vue'
|
||||||
import AllowlistTable from './AllowlistTable.vue'
|
import AllowlistTable from './AllowlistTable.vue'
|
||||||
import AliasTable from './AliasTable.vue'
|
import AliasTable from './AliasTable.vue'
|
||||||
import DbMaintenanceCard from './DbMaintenanceCard.vue'
|
|
||||||
import ArchiveReextractCard from './ArchiveReextractCard.vue'
|
|
||||||
import BackupCard from './BackupCard.vue'
|
import BackupCard from './BackupCard.vue'
|
||||||
import { useSystemActivityStore } from '../../stores/systemActivity.js'
|
import { useSystemActivityStore } from '../../stores/systemActivity.js'
|
||||||
|
|
||||||
|
|||||||
@@ -89,107 +89,6 @@
|
|||||||
@click="onKindCommit"
|
@click="onKindCommit"
|
||||||
>Delete {{ kindPreview.count }} legacy tag(s)</v-btn>
|
>Delete {{ kindPreview.count }} legacy tag(s)</v-btn>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<v-divider class="my-5" />
|
|
||||||
|
|
||||||
<p class="text-body-2 mb-2">
|
|
||||||
<strong class="text-error">Reset content tagging.</strong>
|
|
||||||
Deletes every <code>general</code> and <code>character</code> tag and
|
|
||||||
removes them from every image, so you can re-tag from scratch with the
|
|
||||||
auto-suggest. <strong>Fandoms and series (with their page order) are
|
|
||||||
kept</strong>, and each image's saved predictions are untouched — open
|
|
||||||
an image and its suggestions reappear.
|
|
||||||
</p>
|
|
||||||
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
|
|
||||||
Irreversible — there's no undo except restoring a DB backup.
|
|
||||||
Back one up first (Settings → Maintenance → Backup).
|
|
||||||
</v-alert>
|
|
||||||
|
|
||||||
<v-btn
|
|
||||||
color="accent" variant="flat" rounded="pill"
|
|
||||||
prepend-icon="mdi-magnify"
|
|
||||||
:loading="loadingResetPreview"
|
|
||||||
class="mb-3"
|
|
||||||
@click="onResetPreview"
|
|
||||||
>Preview content-tag reset</v-btn>
|
|
||||||
|
|
||||||
<div v-if="resetPreview">
|
|
||||||
<p class="text-body-2 mb-2">
|
|
||||||
<strong>{{ resetPreview.count }}</strong> content tag(s)
|
|
||||||
<span v-for="(n, k) in resetPreview.by_kind" :key="k" class="fc-muted">
|
|
||||||
({{ k }}: {{ n }})
|
|
||||||
</span>
|
|
||||||
across <strong>{{ resetPreview.applications }}</strong> image
|
|
||||||
application(s).
|
|
||||||
</p>
|
|
||||||
<div v-if="resetPreview.sample_names?.length" class="fc-name-grid mb-3">
|
|
||||||
<span v-for="n in resetPreview.sample_names" :key="n" class="fc-name">
|
|
||||||
{{ n }}
|
|
||||||
</span>
|
|
||||||
</div>
|
|
||||||
<v-btn
|
|
||||||
color="error" variant="flat" rounded="pill"
|
|
||||||
prepend-icon="mdi-delete-alert"
|
|
||||||
:disabled="!resetPreview.count"
|
|
||||||
:loading="resetCommitting"
|
|
||||||
@click="onResetCommit"
|
|
||||||
>Delete {{ resetPreview.count }} content tag(s) +
|
|
||||||
{{ resetPreview.applications }} application(s)</v-btn>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-divider class="my-5" />
|
|
||||||
|
|
||||||
<!-- #714: standardize existing tags to the canonical Title-Case form
|
|
||||||
and merge case/whitespace-variant duplicates. -->
|
|
||||||
<p class="text-body-2 mb-2">
|
|
||||||
<strong>Standardize tag casing.</strong>
|
|
||||||
New tags are saved Title Case, but older tags keep whatever casing they
|
|
||||||
were created with. This renames every existing tag to
|
|
||||||
<code>Title Case</code> (collapsing extra spaces) and
|
|
||||||
<strong>merges</strong> tags that differ only by case or spacing
|
|
||||||
(e.g. <code>hatsune miku</code> + <code>Hatsune Miku</code>)
|
|
||||||
into one — repointing their images, allowlist entries and series pages.
|
|
||||||
</p>
|
|
||||||
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
|
|
||||||
The merges are irreversible — back up first
|
|
||||||
(Settings → Maintenance → Backup). Safe to run more than once.
|
|
||||||
</v-alert>
|
|
||||||
|
|
||||||
<v-btn
|
|
||||||
color="accent" variant="flat" rounded="pill"
|
|
||||||
prepend-icon="mdi-magnify"
|
|
||||||
:loading="loadingNormPreview"
|
|
||||||
class="mb-3"
|
|
||||||
@click="onNormPreview"
|
|
||||||
>Preview tag standardization</v-btn>
|
|
||||||
|
|
||||||
<div v-if="normPreview">
|
|
||||||
<p class="text-body-2 mb-2">
|
|
||||||
<strong>{{ normPreview.total_changes }}</strong> tag group(s) to
|
|
||||||
change — <strong>{{ normPreview.tags_to_rename }}</strong> rename(s),
|
|
||||||
<strong>{{ normPreview.collisions }}</strong> collision(s) merging
|
|
||||||
<strong>{{ normPreview.tags_to_merge }}</strong> tag(s) away.
|
|
||||||
</p>
|
|
||||||
<div v-if="normPreview.sample?.length" class="fc-name-grid mb-3">
|
|
||||||
<span
|
|
||||||
v-for="s in normPreview.sample" :key="s.to + s.kind"
|
|
||||||
class="fc-name"
|
|
||||||
>
|
|
||||||
<template v-if="s.merge">{{ s.from.join(' + ') }} → </template>{{ s.to }}
|
|
||||||
</span>
|
|
||||||
</div>
|
|
||||||
<v-btn
|
|
||||||
color="error" variant="flat" rounded="pill"
|
|
||||||
prepend-icon="mdi-format-letter-case"
|
|
||||||
:disabled="!normPreview.total_changes || normResult === 'queued'"
|
|
||||||
:loading="normCommitting"
|
|
||||||
@click="onNormCommit"
|
|
||||||
>Standardize {{ normPreview.total_changes }} tag group(s)</v-btn>
|
|
||||||
<span v-if="normResult === 'queued'" class="ml-3 text-caption text-success">
|
|
||||||
Queued ✓ — runs in the background (you can leave this page). It
|
|
||||||
processes in chunks, so re-run “Preview” later to confirm it's all done.
|
|
||||||
</span>
|
|
||||||
</div>
|
|
||||||
</v-card-text>
|
</v-card-text>
|
||||||
</v-card>
|
</v-card>
|
||||||
</template>
|
</template>
|
||||||
@@ -206,13 +105,6 @@ const committing = ref(false)
|
|||||||
const kindPreview = ref(null)
|
const kindPreview = ref(null)
|
||||||
const loadingKindPreview = ref(false)
|
const loadingKindPreview = ref(false)
|
||||||
const kindCommitting = ref(false)
|
const kindCommitting = ref(false)
|
||||||
const resetPreview = ref(null)
|
|
||||||
const loadingResetPreview = ref(false)
|
|
||||||
const resetCommitting = ref(false)
|
|
||||||
const normPreview = ref(null)
|
|
||||||
const loadingNormPreview = ref(false)
|
|
||||||
const normCommitting = ref(false)
|
|
||||||
const normResult = ref(null)
|
|
||||||
|
|
||||||
async function onPreview() {
|
async function onPreview() {
|
||||||
loadingPreview.value = true
|
loadingPreview.value = true
|
||||||
@@ -251,55 +143,6 @@ async function onKindCommit() {
|
|||||||
kindCommitting.value = false
|
kindCommitting.value = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function onResetPreview() {
|
|
||||||
loadingResetPreview.value = true
|
|
||||||
try {
|
|
||||||
resetPreview.value = await store.resetContentTagging({ dryRun: true })
|
|
||||||
} finally {
|
|
||||||
loadingResetPreview.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function onResetCommit() {
|
|
||||||
resetCommitting.value = true
|
|
||||||
try {
|
|
||||||
await store.resetContentTagging({ dryRun: false })
|
|
||||||
resetPreview.value = { count: 0, by_kind: {}, applications: 0, sample_names: [] }
|
|
||||||
} finally {
|
|
||||||
resetCommitting.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function onNormPreview() {
|
|
||||||
loadingNormPreview.value = true
|
|
||||||
normResult.value = null
|
|
||||||
try {
|
|
||||||
normPreview.value = await store.normalizeTags({ dryRun: true })
|
|
||||||
} finally {
|
|
||||||
loadingNormPreview.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function onNormCommit() {
|
|
||||||
normCommitting.value = true
|
|
||||||
normResult.value = null
|
|
||||||
try {
|
|
||||||
// Fire-and-forget: the task is time-boxed and self-resuming across chunks
|
|
||||||
// (a large back-catalog can't finish in one run), so we DON'T poll-until-
|
|
||||||
// done — that would falsely report "complete" after the first chunk. Just
|
|
||||||
// confirm it's queued; the operator can re-run Preview later to verify.
|
|
||||||
await store.normalizeTags({ dryRun: false })
|
|
||||||
normResult.value = 'queued'
|
|
||||||
// Keep the preview showing what was QUEUED — do NOT zero it out. Overwriting
|
|
||||||
// it with a zeroed object made the card read "0 tag groups to change" the
|
|
||||||
// instant Standardize was clicked, which looked like nothing happened
|
|
||||||
// (operator-flagged 2026-06-07). The button disables on `queued`; re-running
|
|
||||||
// Preview later reflects the real remaining count as the task applies.
|
|
||||||
} finally {
|
|
||||||
normCommitting.value = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
|
|||||||
@@ -23,18 +23,6 @@
|
|||||||
<span class="fc-active__dot" />
|
<span class="fc-active__dot" />
|
||||||
<PlatformChip :platform="e.platform" size="x-small" />
|
<PlatformChip :platform="e.platform" size="x-small" />
|
||||||
<span class="fc-active__artist">{{ e.artist_name || '—' }}</span>
|
<span class="fc-active__artist">{{ e.artist_name || '—' }}</span>
|
||||||
<!-- #709: live per-file progress for native (Patreon) downloads. -->
|
|
||||||
<span v-if="e.live" class="fc-active__counts">
|
|
||||||
<span class="fc-active__count" title="downloaded">↓ {{ e.live.downloaded }}</span>
|
|
||||||
<span v-if="e.live.skipped" class="fc-active__count" title="skipped">
|
|
||||||
⤼ {{ e.live.skipped }}</span>
|
|
||||||
<span
|
|
||||||
v-if="e.live.errors" class="fc-active__count fc-active__count--err"
|
|
||||||
title="errors"
|
|
||||||
>✕ {{ e.live.errors }}</span>
|
|
||||||
<span class="fc-active__count fc-active__count--posts" title="posts scanned">
|
|
||||||
{{ e.live.posts }} posts</span>
|
|
||||||
</span>
|
|
||||||
<v-spacer />
|
<v-spacer />
|
||||||
<span class="fc-active__timer" :title="`started ${e.started_at}`">
|
<span class="fc-active__timer" :title="`started ${e.started_at}`">
|
||||||
{{ elapsed(e.started_at) }}
|
{{ elapsed(e.started_at) }}
|
||||||
@@ -135,13 +123,6 @@ function elapsed (startedIso) {
|
|||||||
font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;
|
font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
}
|
}
|
||||||
.fc-active__counts {
|
|
||||||
display: inline-flex; align-items: center; gap: 8px;
|
|
||||||
font-variant-numeric: tabular-nums; font-size: 0.78rem;
|
|
||||||
}
|
|
||||||
.fc-active__count { color: rgb(var(--v-theme-on-surface-variant)); }
|
|
||||||
.fc-active__count--err { color: rgb(var(--v-theme-error)); }
|
|
||||||
.fc-active__count--posts { opacity: 0.7; }
|
|
||||||
@keyframes fc-active-pulse {
|
@keyframes fc-active-pulse {
|
||||||
0%, 100% { opacity: 1; transform: scale(1); }
|
0%, 100% { opacity: 1; transform: scale(1); }
|
||||||
50% { opacity: 0.35; transform: scale(0.7); }
|
50% { opacity: 0.35; transform: scale(0.7); }
|
||||||
|
|||||||
@@ -93,7 +93,6 @@ const ERROR_TYPE_COLOR = {
|
|||||||
unsupported_url: 'error',
|
unsupported_url: 'error',
|
||||||
http_error: 'error',
|
http_error: 'error',
|
||||||
unknown_error: 'error',
|
unknown_error: 'error',
|
||||||
api_drift: 'error',
|
|
||||||
partial: 'info',
|
partial: 'info',
|
||||||
tier_limited: 'info',
|
tier_limited: 'info',
|
||||||
no_new_content: 'info',
|
no_new_content: 'info',
|
||||||
@@ -109,7 +108,6 @@ const ERROR_TYPE_HINT = {
|
|||||||
http_error: 'Generic HTTP error — see Logs.',
|
http_error: 'Generic HTTP error — see Logs.',
|
||||||
unsupported_url: 'gallery-dl does not support this URL pattern.',
|
unsupported_url: 'gallery-dl does not support this URL pattern.',
|
||||||
unknown_error: 'Could not classify — see Logs.',
|
unknown_error: 'Could not classify — see Logs.',
|
||||||
api_drift: 'Patreon changed its API shape — the native ingester needs a code update.',
|
|
||||||
}
|
}
|
||||||
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
|
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
|
||||||
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
|
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
|
||||||
|
|||||||
@@ -1,113 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- Plan #708 B4: dry-run preview — what a backfill WOULD download, without
|
|
||||||
downloading. Self-fetches on open; loading / error / empty / result. -->
|
|
||||||
<v-dialog
|
|
||||||
:model-value="modelValue" max-width="520"
|
|
||||||
@update:model-value="$emit('update:modelValue', $event)"
|
|
||||||
>
|
|
||||||
<v-card v-if="source">
|
|
||||||
<v-card-title class="d-flex align-center">
|
|
||||||
<v-icon class="mr-2">mdi-eye-outline</v-icon>
|
|
||||||
Preview · {{ source.artist_name }}
|
|
||||||
</v-card-title>
|
|
||||||
<v-card-subtitle class="fc-preview__url">{{ source.url }}</v-card-subtitle>
|
|
||||||
|
|
||||||
<v-card-text>
|
|
||||||
<div v-if="loading" class="text-center py-8">
|
|
||||||
<v-progress-circular indeterminate color="accent" />
|
|
||||||
<div class="text-caption mt-3 text-medium-emphasis">Walking the feed…</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<v-alert
|
|
||||||
v-else-if="error" type="error" variant="tonal" density="comfortable"
|
|
||||||
>{{ error }}</v-alert>
|
|
||||||
|
|
||||||
<div v-else-if="result">
|
|
||||||
<div class="text-h5">
|
|
||||||
{{ result.total_new }} new item{{ result.total_new === 1 ? '' : 's' }}
|
|
||||||
</div>
|
|
||||||
<div class="text-body-2 text-medium-emphasis mb-3">
|
|
||||||
in the {{ result.posts_scanned }} most recent
|
|
||||||
post{{ result.posts_scanned === 1 ? '' : 's' }}<span
|
|
||||||
v-if="result.has_more"> · more pages not scanned</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div
|
|
||||||
v-if="result.total_new === 0"
|
|
||||||
class="text-body-2 text-medium-emphasis"
|
|
||||||
>You’re caught up — nothing new to download.</div>
|
|
||||||
|
|
||||||
<v-list v-else density="compact" class="fc-preview__list" bg-color="transparent">
|
|
||||||
<v-list-item v-for="(row, i) in result.sample" :key="i" class="px-0">
|
|
||||||
<template #prepend>
|
|
||||||
<v-chip
|
|
||||||
size="x-small" color="info" variant="tonal" label class="mr-3"
|
|
||||||
>+{{ row.new }}</v-chip>
|
|
||||||
</template>
|
|
||||||
<v-list-item-title class="text-body-2">{{ row.title }}</v-list-item-title>
|
|
||||||
<v-list-item-subtitle v-if="row.date" class="text-caption">
|
|
||||||
{{ formatRelative(row.date) }}
|
|
||||||
</v-list-item-subtitle>
|
|
||||||
</v-list-item>
|
|
||||||
</v-list>
|
|
||||||
</div>
|
|
||||||
</v-card-text>
|
|
||||||
|
|
||||||
<v-card-actions>
|
|
||||||
<v-spacer />
|
|
||||||
<v-btn variant="text" @click="$emit('update:modelValue', false)">Close</v-btn>
|
|
||||||
<v-btn
|
|
||||||
v-if="result && result.total_new > 0"
|
|
||||||
color="accent" variant="flat"
|
|
||||||
@click="$emit('backfill', source)"
|
|
||||||
>Start backfill</v-btn>
|
|
||||||
</v-card-actions>
|
|
||||||
</v-card>
|
|
||||||
</v-dialog>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { ref, watch } from 'vue'
|
|
||||||
import { useSourcesStore } from '../../stores/sources.js'
|
|
||||||
import { formatRelative } from '../../utils/date.js'
|
|
||||||
|
|
||||||
const props = defineProps({
|
|
||||||
modelValue: { type: Boolean, default: false },
|
|
||||||
source: { type: Object, default: null },
|
|
||||||
})
|
|
||||||
defineEmits(['update:modelValue', 'backfill'])
|
|
||||||
|
|
||||||
const store = useSourcesStore()
|
|
||||||
const loading = ref(false)
|
|
||||||
const error = ref('')
|
|
||||||
const result = ref(null)
|
|
||||||
|
|
||||||
watch(
|
|
||||||
() => props.modelValue,
|
|
||||||
async (open) => {
|
|
||||||
if (!open || !props.source) return
|
|
||||||
loading.value = true
|
|
||||||
error.value = ''
|
|
||||||
result.value = null
|
|
||||||
try {
|
|
||||||
result.value = await store.previewSource(props.source.id)
|
|
||||||
} catch (e) {
|
|
||||||
error.value = e?.body?.detail || e?.message || 'Preview failed'
|
|
||||||
} finally {
|
|
||||||
loading.value = false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-preview__url {
|
|
||||||
overflow: hidden;
|
|
||||||
text-overflow: ellipsis;
|
|
||||||
white-space: nowrap;
|
|
||||||
}
|
|
||||||
.fc-preview__list {
|
|
||||||
max-height: 320px;
|
|
||||||
overflow-y: auto;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
<template>
|
|
||||||
<!-- Shared per-source action cluster (desktop SourceRow + mobile SourceCard).
|
|
||||||
Frequent actions stay as buttons; low-frequency / destructive ones move
|
|
||||||
into a labelled overflow menu so they can't be mis-clicked, and the
|
|
||||||
labels spell out what each one does (recover vs backfill). -->
|
|
||||||
<div class="fc-src-actions">
|
|
||||||
<v-btn
|
|
||||||
icon size="small" variant="text" :loading="checking"
|
|
||||||
@click.stop="emit('check', source)"
|
|
||||||
>
|
|
||||||
<v-icon>mdi-play</v-icon>
|
|
||||||
<v-tooltip activator="parent" location="top">
|
|
||||||
Check now — fetch posts published since the last check
|
|
||||||
</v-tooltip>
|
|
||||||
</v-btn>
|
|
||||||
<v-btn
|
|
||||||
icon size="small" variant="text"
|
|
||||||
:color="running ? 'warning' : undefined"
|
|
||||||
@click.stop="emit('backfill', source)"
|
|
||||||
>
|
|
||||||
<v-icon>{{ running ? 'mdi-stop' : 'mdi-magnify-scan' }}</v-icon>
|
|
||||||
<v-tooltip activator="parent" location="top">
|
|
||||||
{{ running
|
|
||||||
? (recovering ? 'Stop recovery' : 'Stop backfill')
|
|
||||||
: 'Backfill — one-time walk of the FULL post history until complete' }}
|
|
||||||
</v-tooltip>
|
|
||||||
</v-btn>
|
|
||||||
|
|
||||||
<v-menu location="bottom end">
|
|
||||||
<template #activator="{ props: menuProps }">
|
|
||||||
<v-btn icon size="small" variant="text" v-bind="menuProps" @click.stop>
|
|
||||||
<v-icon>mdi-dots-vertical</v-icon>
|
|
||||||
<v-tooltip activator="parent" location="top">More actions</v-tooltip>
|
|
||||||
</v-btn>
|
|
||||||
</template>
|
|
||||||
<v-list density="compact" min-width="260">
|
|
||||||
<v-list-item
|
|
||||||
v-if="isPatreon && !running"
|
|
||||||
prepend-icon="mdi-backup-restore"
|
|
||||||
@click="emit('recover', source)"
|
|
||||||
>
|
|
||||||
<v-list-item-title>Recover dropped near-duplicates</v-list-item-title>
|
|
||||||
<v-list-item-subtitle>
|
|
||||||
Re-fetch images previously dropped as near-dups and re-judge them
|
|
||||||
under the current similarity threshold
|
|
||||||
</v-list-item-subtitle>
|
|
||||||
</v-list-item>
|
|
||||||
<v-divider v-if="isPatreon && !running" />
|
|
||||||
<v-list-item
|
|
||||||
base-color="error"
|
|
||||||
prepend-icon="mdi-delete-outline"
|
|
||||||
@click="emit('remove', source)"
|
|
||||||
>
|
|
||||||
<v-list-item-title>Remove source</v-list-item-title>
|
|
||||||
</v-list-item>
|
|
||||||
</v-list>
|
|
||||||
</v-menu>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import { computed } from 'vue'
|
|
||||||
|
|
||||||
const props = defineProps({
|
|
||||||
source: { type: Object, required: true },
|
|
||||||
checking: { type: Boolean, default: false },
|
|
||||||
})
|
|
||||||
const emit = defineEmits(['check', 'backfill', 'recover', 'remove'])
|
|
||||||
|
|
||||||
const running = computed(() => props.source.backfill_state === 'running')
|
|
||||||
const recovering = computed(() => !!props.source.backfill_bypass_seen)
|
|
||||||
const isPatreon = computed(() => props.source.platform === 'patreon')
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-src-actions {
|
|
||||||
display: inline-flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 2px;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
<template>
|
|
||||||
<div class="fc-source-card">
|
|
||||||
<div class="fc-source-card__top">
|
|
||||||
<SourceHealthDot :source="source" :warning-threshold="warningThreshold" />
|
|
||||||
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
|
|
||||||
<v-spacer />
|
|
||||||
<v-switch
|
|
||||||
:model-value="source.enabled"
|
|
||||||
density="compact" hide-details color="accent"
|
|
||||||
@click.stop
|
|
||||||
@update:model-value="onToggleEnabled"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-source-card__url-row">
|
|
||||||
<a
|
|
||||||
:href="source.url" target="_blank" rel="noopener"
|
|
||||||
class="fc-source-card__url" @click.stop
|
|
||||||
>{{ source.url }}</a>
|
|
||||||
<v-btn
|
|
||||||
icon="mdi-pencil" size="x-small" variant="text"
|
|
||||||
@click.stop="$emit('edit', source)"
|
|
||||||
>
|
|
||||||
<v-icon>mdi-pencil</v-icon>
|
|
||||||
<v-tooltip activator="parent" location="top">Edit source</v-tooltip>
|
|
||||||
</v-btn>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-source-card__meta">
|
|
||||||
<span>Last {{ formatRelative(source.last_checked_at) }}</span>
|
|
||||||
<span>Next {{ formatRelative(source.next_check_at, { future: true }) }}</span>
|
|
||||||
<v-chip
|
|
||||||
v-if="(source.consecutive_failures || 0) > 0"
|
|
||||||
size="x-small" color="error" variant="tonal" label
|
|
||||||
>{{ source.consecutive_failures }} err
|
|
||||||
<v-tooltip v-if="source.last_error" activator="parent" location="top" max-width="480">
|
|
||||||
<span style="white-space: pre-wrap; word-break: break-word;">{{ source.last_error }}</span>
|
|
||||||
</v-tooltip>
|
|
||||||
</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-else-if="source.backfill_state === 'running'"
|
|
||||||
size="x-small" color="info" variant="tonal" label
|
|
||||||
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling'
|
|
||||||
}}{{ source.backfill_posts
|
|
||||||
? ` · ${source.backfill_posts} posts`
|
|
||||||
: (source.backfill_chunks ? ` (${source.backfill_chunks})` : '') }}</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-else-if="source.backfill_state === 'complete'"
|
|
||||||
size="x-small" color="success" variant="tonal" label
|
|
||||||
>Backfilled</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-else-if="source.backfill_state === 'stalled'"
|
|
||||||
size="x-small" color="warning" variant="tonal" label
|
|
||||||
>Stalled</v-chip>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="fc-source-card__actions">
|
|
||||||
<SourceActions
|
|
||||||
:source="source" :checking="checking"
|
|
||||||
@check="$emit('check', $event)"
|
|
||||||
@backfill="$emit('backfill', $event)"
|
|
||||||
@recover="$emit('recover', $event)"
|
|
||||||
@remove="$emit('remove', $event)"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<script setup>
|
|
||||||
import SourceActions from './SourceActions.vue'
|
|
||||||
import SourceHealthDot from './SourceHealthDot.vue'
|
|
||||||
import { formatRelative } from '../../utils/date.js'
|
|
||||||
|
|
||||||
// Mobile-stacked equivalent of SourceRow (the desktop <tr>) — same data and
|
|
||||||
// emits, but laid out vertically so the wide source columns never force the
|
|
||||||
// lateral scroll the operator flagged on phones.
|
|
||||||
const props = defineProps({
|
|
||||||
source: { type: Object, required: true },
|
|
||||||
checking: { type: Boolean, default: false },
|
|
||||||
warningThreshold: { type: Number, default: 5 },
|
|
||||||
})
|
|
||||||
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover'])
|
|
||||||
|
|
||||||
function onToggleEnabled(value) {
|
|
||||||
emit('toggle', { source: props.source, enabled: value })
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.fc-source-card {
|
|
||||||
padding: 8px 10px;
|
|
||||||
border: 1px solid rgb(var(--v-theme-surface-light));
|
|
||||||
border-radius: 6px;
|
|
||||||
background: rgb(var(--v-theme-surface));
|
|
||||||
display: flex; flex-direction: column; gap: 6px;
|
|
||||||
}
|
|
||||||
.fc-source-card__top { display: flex; align-items: center; gap: 8px; }
|
|
||||||
.fc-source-card__url-row {
|
|
||||||
display: flex; align-items: center; gap: 4px;
|
|
||||||
}
|
|
||||||
.fc-source-card__url {
|
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
text-decoration: none;
|
|
||||||
font-size: 0.8rem;
|
|
||||||
word-break: break-all;
|
|
||||||
flex: 1 1 auto; min-width: 0;
|
|
||||||
}
|
|
||||||
.fc-source-card__url:hover { color: rgb(var(--v-theme-accent)); }
|
|
||||||
.fc-source-card__meta {
|
|
||||||
display: flex; flex-wrap: wrap; align-items: center; gap: 6px 12px;
|
|
||||||
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface-variant));
|
|
||||||
font-variant-numeric: tabular-nums;
|
|
||||||
}
|
|
||||||
.fc-source-card__actions {
|
|
||||||
display: flex; gap: 2px; justify-content: flex-end;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -7,22 +7,10 @@
|
|||||||
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
|
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
|
||||||
</td>
|
</td>
|
||||||
<td class="fc-source-row__url-cell">
|
<td class="fc-source-row__url-cell">
|
||||||
<div class="fc-source-row__url-wrap">
|
|
||||||
<a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url"
|
<a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url"
|
||||||
@click.stop>
|
@click.stop>
|
||||||
{{ source.url }}
|
{{ source.url }}
|
||||||
</a>
|
</a>
|
||||||
<!-- Edit sits next to the source identity (operator-requested), not in
|
|
||||||
the action cluster where it was easy to fat-finger Remove. -->
|
|
||||||
<v-btn
|
|
||||||
icon="mdi-pencil" size="x-small" variant="text"
|
|
||||||
class="fc-source-row__edit"
|
|
||||||
@click.stop="$emit('edit', source)"
|
|
||||||
>
|
|
||||||
<v-icon>mdi-pencil</v-icon>
|
|
||||||
<v-tooltip activator="parent" location="top">Edit source</v-tooltip>
|
|
||||||
</v-btn>
|
|
||||||
</div>
|
|
||||||
</td>
|
</td>
|
||||||
<td>
|
<td>
|
||||||
<v-switch
|
<v-switch
|
||||||
@@ -42,43 +30,53 @@
|
|||||||
<v-chip
|
<v-chip
|
||||||
v-if="(source.consecutive_failures || 0) > 0"
|
v-if="(source.consecutive_failures || 0) > 0"
|
||||||
size="x-small" color="error" variant="tonal" label
|
size="x-small" color="error" variant="tonal" label
|
||||||
>{{ source.consecutive_failures }}
|
>{{ source.consecutive_failures }}</v-chip>
|
||||||
<!-- #1: show the actual failure reason on hover instead of a bare count. -->
|
|
||||||
<v-tooltip v-if="source.last_error" activator="parent" location="top" max-width="480">
|
|
||||||
<span class="fc-source-row__err-text">{{ source.last_error }}</span>
|
|
||||||
</v-tooltip>
|
|
||||||
</v-chip>
|
|
||||||
<v-chip
|
<v-chip
|
||||||
v-else-if="source.backfill_state === 'running'"
|
v-else-if="(source.backfill_runs_remaining || 0) > 0"
|
||||||
size="x-small" color="info" variant="tonal" label
|
size="x-small" color="info" variant="tonal" label
|
||||||
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling'
|
>
|
||||||
}}{{ source.backfill_posts
|
backfill ({{ source.backfill_runs_remaining }}×)
|
||||||
? ` · ${source.backfill_posts} posts`
|
</v-chip>
|
||||||
: (source.backfill_chunks ? ` (${source.backfill_chunks})` : '') }}</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-else-if="source.backfill_state === 'complete'"
|
|
||||||
size="x-small" color="success" variant="tonal" label
|
|
||||||
>Backfilled</v-chip>
|
|
||||||
<v-chip
|
|
||||||
v-else-if="source.backfill_state === 'stalled'"
|
|
||||||
size="x-small" color="warning" variant="tonal" label
|
|
||||||
>Stalled</v-chip>
|
|
||||||
<span v-else class="fc-source-row__zero">0</span>
|
<span v-else class="fc-source-row__zero">0</span>
|
||||||
</td>
|
</td>
|
||||||
<td class="fc-source-row__actions">
|
<td class="fc-source-row__actions">
|
||||||
<SourceActions
|
<v-btn
|
||||||
:source="source" :checking="checking"
|
icon="mdi-play" size="x-small" variant="text"
|
||||||
@check="$emit('check', $event)"
|
:loading="checking"
|
||||||
@backfill="$emit('backfill', $event)"
|
@click.stop="$emit('check', source)"
|
||||||
@recover="$emit('recover', $event)"
|
>
|
||||||
@remove="$emit('remove', $event)"
|
<v-icon>mdi-play</v-icon>
|
||||||
/>
|
<v-tooltip activator="parent" location="top">Check now</v-tooltip>
|
||||||
|
</v-btn>
|
||||||
|
<v-btn
|
||||||
|
icon="mdi-magnify-scan" size="x-small" variant="text"
|
||||||
|
:disabled="(source.backfill_runs_remaining || 0) > 0"
|
||||||
|
@click.stop="$emit('backfill', source)"
|
||||||
|
>
|
||||||
|
<v-icon>mdi-magnify-scan</v-icon>
|
||||||
|
<v-tooltip activator="parent" location="top">
|
||||||
|
Deep scan — walk full history for next few runs
|
||||||
|
</v-tooltip>
|
||||||
|
</v-btn>
|
||||||
|
<v-btn
|
||||||
|
icon="mdi-pencil" size="x-small" variant="text"
|
||||||
|
@click.stop="$emit('edit', source)"
|
||||||
|
>
|
||||||
|
<v-icon>mdi-pencil</v-icon>
|
||||||
|
<v-tooltip activator="parent" location="top">Edit</v-tooltip>
|
||||||
|
</v-btn>
|
||||||
|
<v-btn
|
||||||
|
icon="mdi-close" size="x-small" variant="text" color="error"
|
||||||
|
@click.stop="$emit('remove', source)"
|
||||||
|
>
|
||||||
|
<v-icon>mdi-close</v-icon>
|
||||||
|
<v-tooltip activator="parent" location="top">Remove</v-tooltip>
|
||||||
|
</v-btn>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import SourceActions from './SourceActions.vue'
|
|
||||||
import SourceHealthDot from './SourceHealthDot.vue'
|
import SourceHealthDot from './SourceHealthDot.vue'
|
||||||
import { formatRelative } from '../../utils/date.js'
|
import { formatRelative } from '../../utils/date.js'
|
||||||
|
|
||||||
@@ -87,7 +85,7 @@ const props = defineProps({
|
|||||||
checking: { type: Boolean, default: false },
|
checking: { type: Boolean, default: false },
|
||||||
warningThreshold: { type: Number, default: 5 },
|
warningThreshold: { type: Number, default: 5 },
|
||||||
})
|
})
|
||||||
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover'])
|
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill'])
|
||||||
|
|
||||||
function onToggleEnabled(value) {
|
function onToggleEnabled(value) {
|
||||||
emit('toggle', { source: props.source, enabled: value })
|
emit('toggle', { source: props.source, enabled: value })
|
||||||
@@ -100,27 +98,18 @@ function onToggleEnabled(value) {
|
|||||||
padding-right: 0 !important;
|
padding-right: 0 !important;
|
||||||
}
|
}
|
||||||
.fc-source-row__url-cell {
|
.fc-source-row__url-cell {
|
||||||
max-width: 420px;
|
max-width: 400px;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
}
|
}
|
||||||
.fc-source-row__url-wrap {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 2px;
|
|
||||||
min-width: 0;
|
|
||||||
}
|
|
||||||
.fc-source-row__url {
|
.fc-source-row__url {
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
text-decoration: none;
|
text-decoration: none;
|
||||||
flex: 0 1 auto;
|
display: inline-block;
|
||||||
min-width: 0;
|
max-width: 100%;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
text-overflow: ellipsis;
|
text-overflow: ellipsis;
|
||||||
white-space: nowrap;
|
white-space: nowrap;
|
||||||
}
|
}
|
||||||
/* Keep the edit affordance subtle until the row is hovered. */
|
|
||||||
.fc-source-row__edit { opacity: 0.5; flex: 0 0 auto; }
|
|
||||||
.fc-source-row:hover .fc-source-row__edit { opacity: 1; }
|
|
||||||
.fc-source-row__url:hover { color: rgb(var(--v-theme-accent)); }
|
.fc-source-row__url:hover { color: rgb(var(--v-theme-accent)); }
|
||||||
.fc-source-row__when {
|
.fc-source-row__when {
|
||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
@@ -131,10 +120,6 @@ function onToggleEnabled(value) {
|
|||||||
color: rgb(var(--v-theme-on-surface-variant));
|
color: rgb(var(--v-theme-on-surface-variant));
|
||||||
opacity: 0.6;
|
opacity: 0.6;
|
||||||
}
|
}
|
||||||
.fc-source-row__err-text {
|
|
||||||
white-space: pre-wrap;
|
|
||||||
word-break: break-word;
|
|
||||||
}
|
|
||||||
.fc-source-row__actions {
|
.fc-source-row__actions {
|
||||||
white-space: nowrap;
|
white-space: nowrap;
|
||||||
text-align: right;
|
text-align: right;
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user