Compare commits

..
57 Commits
Author SHA1 Message Date
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-ml (push) Successful in 3m28s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
CI / intimp (push) Successful in 4m26s
CI / lint (push) Successful in 3s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
2026-06-03 13:18:44 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 89b48f8f35 Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 35s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m10s
2026-06-02 16:15:00 -04:00
bvandeusen d60e0b9494 Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 36s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Successful in 7m39s
CI / intcore (push) Successful in 8m10s
2026-06-02 14:49:28 -04:00
bvandeusen 9c27a2d3c7 Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 37s
Build images / build-web (push) Successful in 13s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 8m50s
2026-06-02 14:17:12 -04:00
bvandeusen 93e37681b7 Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 10s
CI / intapi (push) Successful in 7m28s
CI / intcore (push) Successful in 8m3s
CI / intimp (push) Successful in 3m42s
2026-06-02 13:29:17 -04:00
bvandeusen 64ca858574 Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 08:26:58 -04:00
bvandeusen 9d0c0b7da8 Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 18s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m4s
2026-06-01 22:34:38 -04:00
bvandeusen 8e4d252ae4 Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / intimp (push) Successful in 3m40s
CI / lint (push) Successful in 3s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m46s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 26s
2026-06-01 21:52:42 -04:00
bvandeusen fdd3e01f56 Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 10s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Successful in 8m16s
2026-06-01 21:09:28 -04:00
bvandeusen c82fb308b6 Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main
Build images / build-ml (push) Failing after 2s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / intimp (push) Successful in 3m47s
CI / intapi (push) Successful in 7m52s
CI / intcore (push) Successful in 8m16s
Build images / build-web (push) Successful in 12s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 31s
2026-06-01 20:44:23 -04:00
bvandeusen 8cf8d2ca4d Merge pull request 'Modal Esc/overflow polish, artist-scoped post scroll, failing-sources Logs button' (#43) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 2m45s
Build images / build-ml (push) Successful in 3m31s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m47s
2026-06-01 12:10:11 -04:00
bvandeusen b1d58bc3b8 Merge pull request 'fix(ci): POSIX-safe SHORT_SHA in build.yml' (#42) from dev into main
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 20s
Build images / sign-extension (push) Successful in 3s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 3m24s
CI / intimp (push) Successful in 3m22s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m5s
2026-06-01 08:03:17 -04:00
bvandeusen 65386f02a0 Merge pull request 'View modal batch: autofocus, suggestions UX, post-title click, retire copyright/artist' (#41) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 4s
Build images / build-web (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m17s
CI / intcore (push) Successful in 8m2s
2026-06-01 07:01:42 -04:00
bvandeusen 667b05f14e Merge pull request 'Extension probe-and-add (v1.0.6) + per-commit image tags' (#40) from dev into main
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 22s
extension / lint (push) Successful in 8s
Build images / sign-extension (push) Successful in 1m51s
Build images / build-web (push) Failing after 4s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
2026-06-01 01:44:03 -04:00
bvandeusen 856e9104b4 Merge pull request 'Sidecar synthetic anchor cleanup + tier-gated classifier fix' (#39) from dev into main
CI / lint (push) Successful in 5s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m54s
Build images / sign-extension (push) Has been skipped
Build images / build-web (push) Successful in 1m1s
Build images / build-ml (push) Successful in 1m21s
CI / intapi (push) Successful in 7m31s
CI / intcore (push) Successful in 8m0s
2026-06-01 00:16:58 -04:00
bvandeusen 0397642b21 Merge pull request 'Showcase cadence tuning + cooldown-aware bulk retry' (#38) from dev into main 2026-05-30 23:50:36 -04:00
bvandeusen 237575447d Merge pull request 'Thumbnail URL fix + archive daemon fix + batched initial loads' (#37) from dev into main 2026-05-30 22:01:43 -04:00
bvandeusen ed358757dc Merge pull request 'Most-overdue-first scheduling + rich timeout diagnostics' (#36) from dev into main 2026-05-30 14:30:41 -04:00
bvandeusen d181f4afb8 Merge pull request 'Downloads burst-prevention + maintenance-menu fix + gdl timeout' (#35) from dev into main 2026-05-30 11:43:18 -04:00
bvandeusen 2886fa4997 Merge pull request 'Tooltip !important fix — 104cac5 follow-up after Vite CSS reorder' (#34) from dev into main 2026-05-30 00:02:24 -04:00
bvandeusen f256f587ee Merge pull request 'UI batch + I1–I6 service passes + download-event recovery sweep' (#33) from dev into main 2026-05-29 22:46:16 -04:00
bvandeusen 384d8d5e50 Merge pull request 'Dashboard insights + project-wide DRY pass' (#32) from dev into main 2026-05-28 15:38:26 -04:00
bvandeusen 319e8c1d18 Merge pull request 'v26.05.28.0: downloads dashboard + task-resilience overhaul (timeouts, archive split, 3-layer poison-pill defense)' (#31) from dev into main 2026-05-28 00:45:00 -04:00
bvandeusen 9075d8eadd Merge pull request 'v26.05.27.2: subscribestar + HF cookie quirks, platforms package refactor, showcase IR-parity, secure-context audit' (#30) from dev into main 2026-05-27 21:34:02 -04:00
bvandeusen 88e53e5b86 Merge pull request 'v26.05.27.1: subscriptions hub + post-card merge + sidecar audit' (#29) from dev into main 2026-05-27 17:12:48 -04:00
bvandeusen 37e8b796a1 Merge pull request 'v26.05.27.0: PostCard redesign + IR-style tag suffix + drop meta/rating + extension v1.0.4 CSP fix' (#28) from dev into main 2026-05-27 11:31:18 -04:00
bvandeusen 4e82208926 Merge pull request 'v26.05.26.5 — extension CORS unblock + UI gap closes + CI workflow cleanup' (#27) from dev into main 2026-05-26 20:15:07 -04:00
bvandeusen 52fff00353 Merge pull request 'v26.05.26.4 — hotfix: migration 0022 pre-DELETE colliding ImageProvenance before UPDATE' (#26) from dev into main 2026-05-26 18:06:20 -04:00
bvandeusen c14338cbce Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main 2026-05-26 17:52:59 -04:00
bvandeusen 8c36dd28b0 Merge pull request 'v26.05.26.2 — hotfix: alembic 0022 Post-collision pre-merge + ci.yml cache continue-on-error' (#24) from dev into main 2026-05-26 16:50:43 -04:00
bvandeusen 88cfb3dd02 Merge pull request 'v26.05.26.1 — thumb backfill, modal redesign, recovery sweep race-safety, artist view redesign, extension fixes' (#23) from dev into main 2026-05-26 16:32:00 -04:00
bvandeusen 5d4f223b71 Merge pull request 'Release v26.05.25.7 — FC-Cleanup tab + UniqueViolation fix + error modal + extension install fix' (#22) from dev into main 2026-05-26 08:26:46 -04:00
bvandeusen 05090c6e85 Merge pull request 'Release v26.05.25.7 — animated-WebP worker fix + FC-Cleanup backend' (#21) from dev into main 2026-05-26 01:48:13 -04:00
bvandeusen 3a577d5ade Merge pull request 'fix(ext-ci): use browser_download_url + curl -f + ZIP magic check (XPI silently corrupt)' (#20) from dev into main 2026-05-26 00:43:02 -04:00
bvandeusen f4fe02e346 Merge pull request 'fix(ext-ci): drop actions/upload-artifact (Forgejo doesn't support v4+ GHES)' (#19) from dev into main 2026-05-25 23:33:40 -04:00
bvandeusen e766197d99 Merge pull request 'fix(ext-ci): jq→python + bump ext to 1.0.3 + rollback-on-upload-failure' (#18) from dev into main 2026-05-25 23:14:51 -04:00
bvandeusen 3872e1dda9 Merge pull request 'fix(ext-ci): web-ext v8 .cjs config workaround' (#17) from dev into main 2026-05-25 22:49:14 -04:00
bvandeusen 9814f3dbaf Merge pull request 'Release v26.05.25.5 — Extension publish refactor, deep-scan IR-parity, archive-import perf, artist Settings tab' (#16) from dev into main 2026-05-25 22:44:59 -04:00
bvandeusen b214460fdb Merge pull request 'Release v26.05.25.4 — importer ext sanitize fix, CI shard split, BrowserExtensionCard on Overview' (#15) from dev into main 2026-05-25 21:11:50 -04:00
bvandeusen ac55d0e8d8 Merge pull request 'fix(ext-ci): match AMO-renamed signed XPI' (#14) from dev into main 2026-05-25 18:22:50 -04:00
bvandeusen 89a89e0ded Merge pull request 'Release v26.05.25.3 — ML embedder SigLIP fix, import-UX, extension publish' (#13) from dev into main 2026-05-25 17:56:50 -04:00
bvandeusen 4e9aac2c05 Merge pull request 'v26.05.25.2: supersede + sidecar enrichment, scan toast feedback, CI uv + pip cache + durations' (#12) from dev into main 2026-05-25 14:30:25 -04:00
bvandeusen 2879ac6f2b Merge pull request 'v26.05.25.1: maintenance sweep + Camie v2 + corrupt-file handling + post-date gallery + clear-stuck escape hatch' (#11) from dev into main 2026-05-25 12:57:46 -04:00
bvandeusen b8dce6c483 Merge pull request 'FC-3h + FC-3k: backup first-class + admin destructive actions' (#10) from dev into main 2026-05-25 01:41:53 -04:00
bvandeusen d1c0b82a22 Merge pull request 'v26.05.24.3: FC-3i System Activity dashboard + migration backup-gate retired + modal Escape' (#9) from dev into main 2026-05-24 21:47:53 -04:00
bvandeusen 5526b8dc78 Merge pull request 'v26.05.24.2: IR Post/Provenance restore + modal artist fallback' (#8) from dev into main 2026-05-24 14:30:06 -04:00
bvandeusen 16eb7075c4 Merge pull request 'v26.05.24.1: FC-3g Firefox extension + worker resilience + UI/migration fixes' (#7) from dev into main 2026-05-24 12:52:31 -04:00
bvandeusen 885dcf64f3 Merge pull request 'v26.05.24.0: TopNav re-fix (flex 1 1 0 side cells)' (#6) from dev into main 2026-05-23 22:49:29 -04:00
bvandeusen f2f6b6d25e Merge pull request 'v26.05.23.3: dogfood UX polish + accurate active-batch stats' (#5) from dev into main 2026-05-23 22:05:59 -04:00
bvandeusen 0822240fde Merge pull request 'v26.05.23.2: serve /images + artist cleanup migrator' (#4) from dev into main 2026-05-23 12:19:16 -04:00
bvandeusen 27f7f3fd01 Merge pull request 'v26.05.23.1: migration durability + dogfood UX' (#3) from dev into main 2026-05-23 11:21:33 -04:00
bvandeusen c5bf564f53 Merge dev: v26.05.23.0 migration follow-ups (#2)
pg_dump + zstd in runtime image, lift Quart body cap to 1 GiB. See PR #2.
2026-05-22 22:37:06 -04:00
bvandeusen 602c7d275d Merge dev: FC-1 → FC-5 v1 build (#1)
First merge of `dev` into `main` for FabledCurator. Brings FC-1 (Foundation) through FC-5 (Migration tooling) onto `main`. See PR #1 body for the full stage rollup.
2026-05-22 14:15:45 -04:00
144 changed files with 1530 additions and 13266 deletions
+149 -30
View File
@@ -92,25 +92,28 @@ jobs:
- run: npm run test:unit - run: npm run test:unit
- run: npm run build - run: npm run build
# Single integration job — collapsed from a 3-way shard split on 2026-06-04. # Integration suite split into THREE parallel shards (2026-05-25, runner
# The shards existed to parallelize ~8.5min of integration tests; once the # capacity bumped 2→6). Each shard gets its own Postgres + Redis service
# throwaway Postgres runs with fsync OFF (the durability step below) the whole # set and runs alembic + a disjoint subset of integration tests. Shards
# suite runs in ~45s, so the split only triplicated the ~2min fixed overhead # share no DB state, so the autouse TRUNCATE fixture in tests/conftest.py
# (container + `uv pip install` + `alembic upgrade head`) and burned 3 of 6 # stays single-threaded per shard but multiple shards run in parallel
# runner slots for no wall-clock gain. One job now: spin up once, install # wall-clock. Approximate split — rebalance once --durations=15 output
# once, migrate once, run every integration test. # reveals which shard is the long pole.
# #
# The docker-ps filter scopes to THIS job's own Postgres/Redis service # Each shard's docker-ps filter uses its own unique job name to scope
# containers by job name. act_runner strips underscores from job names when # service-container resolution. act_runner appears to strip underscores
# labelling containers (`int_api` matched nothing on 2026-05-25), so the name # from job names when building container labels — `int_api` yielded
# stays separator-free (`integration`). The step prints `docker ps -a` first # zero matches on 2026-05-25 — so shards use no-separator names
# so a future naming-convention shift surfaces in the log without a # (`intapi`, `intimp`, `intcore`) instead. Each step prints
# guess-and-push cycle. # `docker ps -a` first so a future naming-convention shift surfaces in
# the log without another guess-and-push cycle.
# #
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT done — # Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT
# per ci-requirements.md, FC is the only Python consumer of that image and the # done — per ci-requirements.md, FC is the only Python consumer of that
# CI-Runner "add deps to image when used by >1 project" rule keeps it per-job. # image and the CI-Runner project's "add deps to image when used by >1
integration: # project" rule keeps the install per-job.
intapi:
runs-on: python-ci runs-on: python-ci
container: container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14 image: git.fabledsword.com/bvandeusen/ci-python:3.14
@@ -141,14 +144,14 @@ jobs:
--health-retries 10 --health-retries 10
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- name: Integration suite (resolve service IPs, migrate, test) - name: API integration shard (resolve service IPs, migrate, test)
run: | run: |
set -eux set -eux
echo "=== container landscape (diagnostic for filter scoping) ===" echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}' docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ===" echo "=== end landscape ==="
PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1) PG=$(docker ps --filter "name=intapi" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1) RD=$(docker ps --filter "name=intapi" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD" test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG") PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD") RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -165,14 +168,130 @@ jobs:
else else
pip install -r requirements.txt pytest pytest-asyncio pip install -r requirements.txt pytest pytest-asyncio
fi fi
# Relax durability on the throwaway CI Postgres so the per-test
# TRUNCATE's commit-fsync — the integration teardown's dominant cost
# (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
# skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
# is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
# NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
# surprise can't red the job; fabledcurator is the postgres image's
# bootstrap superuser.
python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
alembic upgrade head alembic upgrade head
pytest tests/ -v -m integration --durations=15 pytest tests/test_api_*.py -v -m integration --durations=15
intimp:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Importer integration shard (resolve service IPs, migrate, test)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intimp" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intimp" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/test_importer*.py tests/test_import_*.py tests/test_migration_*.py tests/test_phash_*.py tests/test_sidecar_*.py tests/test_scan_*.py tests/test_archive_extractor.py tests/test_backfill_phash.py -v -m integration --durations=15
intcore:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intcore" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intcore" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration --durations=15 \
--ignore-glob='tests/test_api_*.py' \
--ignore-glob='tests/test_importer*.py' \
--ignore-glob='tests/test_import_*.py' \
--ignore-glob='tests/test_migration_*.py' \
--ignore-glob='tests/test_phash_*.py' \
--ignore-glob='tests/test_sidecar_*.py' \
--ignore-glob='tests/test_scan_*.py' \
--ignore-glob='tests/test_archive_extractor.py' \
--ignore-glob='tests/test_backfill_phash.py'
-3
View File
@@ -61,9 +61,6 @@ Thumbs.db
# Claude Code per-user local overrides (shared .claude/settings.json is OK to commit) # Claude Code per-user local overrides (shared .claude/settings.json is OK to commit)
.claude/settings.local.json .claude/settings.local.json
# Transient scheduler lock/state (committed by accident in 3f30327)
.claude/scheduled_tasks.lock
.claude/scheduled_tasks*.json
# Alembic / DB scratch # Alembic / DB scratch
alembic/versions/__pycache__/ alembic/versions/__pycache__/
-53
View File
@@ -1,53 +0,0 @@
"""artist_visit: per-artist last-viewed timestamp for the "+N new" badge
Revision ID: 0034
Revises: 0033
Create Date: 2026-06-03
Powers the artists-directory "+N new since last visit" badge + ArtistView
banner. Single row per artist (no user_id yet — rule #47 multi-user ACL
is aspirational; widens to (user_id, artist_id) PK when User lands).
Seed every existing artist with `last_viewed_at = NOW()` so the badge
starts at 0 across the board — no noisy "you have 5000 unseen images"
on first deploy. New artists auto-get a row via
`ArtistService.find_or_create`.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0034"
down_revision: Union[str, None] = "0033"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"artist_visit",
sa.Column(
"artist_id",
sa.Integer,
sa.ForeignKey("artist.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column(
"last_viewed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
)
# Seed: every existing artist starts "fully caught up". Without this,
# every operator with N artists would see N badges (worth of every
# image ever imported) on first deploy.
op.execute(
"INSERT INTO artist_visit (artist_id, last_viewed_at) "
"SELECT id, NOW() FROM artist"
)
def downgrade() -> None:
op.drop_table("artist_visit")
@@ -1,70 +0,0 @@
"""image_record.effective_date: materialized gallery sort key + index
Revision ID: 0035
Revises: 0034
Create Date: 2026-06-04
The gallery ordered/cursored on COALESCE(post.post_date,
image_record.created_at) across the Post outer join. That expression spans
two tables, so no index can serve it — every /scroll sorted a large slice
of the library, and the frontend fired ten of them serially per initial
load. Materialize the value into image_record.effective_date and index
(effective_date DESC, id DESC) so the cursor scroll is an index range scan.
Backfill = COALESCE(primary post's post_date, created_at) so existing rows
keep their exact ordering. New rows get the created_at-equivalent server
default; services/importer.py overrides it with the post's date when a
primary post with a date is linked.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0035"
down_revision: Union[str, None] = "0034"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Add nullable first so the backfill can populate before NOT NULL.
op.add_column(
"image_record",
sa.Column("effective_date", sa.DateTime(timezone=True), nullable=True),
)
# Pure set-based UPDATEs (no per-row params) — immune to the 65535
# bind-parameter ceiling regardless of library size.
op.execute(
"""
UPDATE image_record AS ir
SET effective_date = COALESCE(p.post_date, ir.created_at)
FROM post AS p
WHERE ir.primary_post_id = p.id
"""
)
op.execute(
"""
UPDATE image_record
SET effective_date = created_at
WHERE effective_date IS NULL
"""
)
op.alter_column(
"image_record",
"effective_date",
nullable=False,
server_default=sa.text("now()"),
)
# DESC/DESC matches the gallery's ORDER BY effective_date DESC, id DESC
# so the scroll is a forward index scan; raw SQL because alembic's
# column list doesn't express per-column DESC cleanly.
op.execute(
"CREATE INDEX ix_image_record_effective_date "
"ON image_record (effective_date DESC, id DESC)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_effective_date", table_name="image_record")
op.drop_column("image_record", "effective_date")
@@ -1,41 +0,0 @@
"""image_record.siglip_embedding: HNSW cosine index for "more like this"
Revision ID: 0036
Revises: 0035
Create Date: 2026-06-04
Gallery Phase 3 (visual similarity search) ranks images by
`siglip_embedding.cosine_distance(source_embedding)`. Without an index that's
a sequential scan computing a 1152-dim distance for every row — fine at small
scale, but it grows linearly with the library. Add an HNSW index with
`vector_cosine_ops` so the top-N nearest search is sub-50ms ANN.
1152 dims is under pgvector's 2000-dim HNSW limit, so HNSW (no training,
better recall than IVFFlat) is the right choice. ONE-TIME COST: building the
index over the existing embeddings (~57k vectors on the operator's library)
locks image_record for ~30-60s during this migration on deploy — acceptable
for a single-operator homelab. NULL embeddings (videos / not-yet-embedded
rows) are simply not indexed.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0036"
down_revision: Union[str, None] = "0035"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Raw SQL: alembic's create_index doesn't express the `USING hnsw (...
# vector_cosine_ops)` access-method + opclass cleanly. Must match the
# query's cosine_distance operator class to be usable by the planner.
op.execute(
"CREATE INDEX ix_image_record_siglip_hnsw "
"ON image_record USING hnsw (siglip_embedding vector_cosine_ops)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_siglip_hnsw", table_name="image_record")
@@ -1,53 +0,0 @@
"""patreon_seen_media: per-source ledger of already-ingested Patreon media
Revision ID: 0037
Revises: 0036
Create Date: 2026-06-05
Native Patreon ingester (build step 2a). Replaces gallery-dl's
archive.sqlite3 with our own queryable table. The downloader upserts one
row per (source, media) so routine walks skip media we've already
processed; a future "recovery" mode bypasses the ledger to re-walk.
`filehash` is a 32-hex Patreon CDN MD5, OR a video sentinel of the form
``video:<post_id>:<media_id>`` — hence String(128). The unique
constraint on (source_id, filehash) is the dedup upsert key.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0037"
down_revision: Union[str, None] = "0036"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"patreon_seen_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("post_id", sa.String(64), nullable=True),
sa.Column(
"seen_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_patreon_seen_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("patreon_seen_media")
@@ -1,58 +0,0 @@
"""patreon_failed_media: per-source dead-letter ledger for failing Patreon media
Revision ID: 0038
Revises: 0037
Create Date: 2026-06-06
Plan #705 (#7). Media that keeps failing to download/validate (404'd CDN,
deleted post, geo-blocked Mux, persistently-corrupt bytes) gets recorded here
with an attempt counter; once it crosses the dead-letter threshold the ingester
skips it on routine walks (recovery still re-attempts). A clean download clears
the row. UNIQUE (source_id, filehash) is the upsert key (same media key the
seen-ledger uses).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0038"
down_revision: Union[str, None] = "0037"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"patreon_failed_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
sa.Column("last_error", sa.Text, nullable=True),
sa.Column(
"first_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.Column(
"last_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("patreon_failed_media")
+1 -105
View File
@@ -19,7 +19,7 @@ from __future__ import annotations
import hashlib import hashlib
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
from sqlalchemy import select, text from sqlalchemy import select
from ..extensions import get_session from ..extensions import get_session
from ..models import Artist from ..models import Artist
@@ -222,107 +222,3 @@ async def tags_purge_legacy():
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run) lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
) )
return jsonify(result) return jsonify(result)
@admin_bp.route("/tags/reset-content", methods=["POST"])
async def tags_reset_content():
"""Tier-A: delete ALL general + character tags (the Camie-suggestable
content vocabulary) so the operator can re-tag from scratch via
auto-suggest. fandom + series tags + series_page ordering are preserved,
and image tagger_predictions are untouched so suggestions repopulate.
dry-run preview returns per-kind counts + applications + a sample so the
UI shows exactly what'll go before the operator confirms (dry_run=false).
Irreversible except via DB backup restore."""
from ..services.cleanup_service import reset_content_tagging
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: reset_content_tagging(sync_sess, dry_run=dry_run)
)
return jsonify(result)
@admin_bp.route("/tags/normalize", methods=["POST"])
async def tags_normalize():
"""#714: retro-normalize existing tags to the #701 canonical form (Title
Case + collapsed whitespace) and merge case/whitespace-variant duplicates.
dry_run=true (default) returns a projection inline — group/collision/rename
counts + a sample of the changes — so the UI shows exactly what'll happen.
dry_run=false dispatches the long-running maintenance task (the merge FK
repoints can touch many tags); the UI tails the activity dashboard for the
summary. Idempotent; back up first (the merges are irreversible)."""
from ..services.tag_service import normalize_existing_tags
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", True))
if dry_run:
async with get_session() as session:
result = await normalize_existing_tags(session, dry_run=True)
return jsonify(result)
from ..tasks.admin import normalize_tags_task
async_result = normalize_tags_task.delay()
return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/db-stats", methods=["GET"])
async def db_stats():
"""Per-table bloat readout (pg_stat_user_tables) for the high-churn tables
so the operator can see when a VACUUM is worth running."""
from ..tasks.maintenance import VACUUM_TABLES
wanted = set(VACUUM_TABLES)
async with get_session() as session:
rows = (await session.execute(text(
"SELECT relname, n_live_tup, n_dead_tup, last_vacuum, "
"last_autovacuum, last_analyze FROM pg_stat_user_tables"
))).all()
def _iso(v):
return v.isoformat() if v is not None else None
out = []
for r in rows:
if r.relname not in wanted:
continue
live = r.n_live_tup or 0
dead = r.n_dead_tup or 0
total = live + dead
out.append({
"table": r.relname,
"live": live,
"dead": dead,
"dead_pct": round(100 * dead / total, 1) if total else 0.0,
"last_vacuum": _iso(r.last_vacuum),
"last_autovacuum": _iso(r.last_autovacuum),
"last_analyze": _iso(r.last_analyze),
})
out.sort(key=lambda t: t["dead"], reverse=True)
return jsonify({"tables": out})
@admin_bp.route("/maintenance/vacuum", methods=["POST"])
async def trigger_vacuum():
"""Operator-triggered VACUUM (ANALYZE) over the high-churn tables — the
same maintenance-queue task the weekly Beat schedule runs."""
from ..tasks.maintenance import vacuum_analyze
vacuum_analyze.delay()
return jsonify({"status": "queued"}), 202
@admin_bp.route("/maintenance/reextract-archives", methods=["POST"])
async def trigger_reextract_archives():
"""Operator-triggered re-extract (#713): PostAttachments that are actually
archives but were filed opaquely (pre magic-byte gate) get extracted and
their members linked to the post. Idempotent; runs on the maintenance queue."""
from ..tasks.admin import reextract_archive_attachments_task
async_result = reextract_archive_attachments_task.delay()
return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+5 -8
View File
@@ -154,15 +154,12 @@ async def audit_history():
limit = min(int(request.args.get("limit", "20")), 100) limit = min(int(request.args.get("limit", "20")), 100)
except ValueError: except ValueError:
return _bad("invalid_limit") return _bad("invalid_limit")
# Optional rule filter so a card can reconnect to ITS latest run on mount
# (?rule=transparency&limit=1) — the audit survives navigation; the UI
# rehydrates from this rather than losing the in-flight scan.
rule = request.args.get("rule") or None
async with get_session() as session: async with get_session() as session:
stmt = select(LibraryAuditRun).order_by(LibraryAuditRun.id.desc()) rows = (await session.execute(
if rule is not None: select(LibraryAuditRun)
stmt = stmt.where(LibraryAuditRun.rule == rule) .order_by(LibraryAuditRun.id.desc())
rows = (await session.execute(stmt.limit(limit))).scalars().all() .limit(limit)
)).scalars().all()
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]}) return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
+10 -12
View File
@@ -121,15 +121,13 @@ async def delete_credential(platform: str):
@credentials_bp.route("/<platform>/verify", methods=["POST"]) @credentials_bp.route("/<platform>/verify", methods=["POST"])
async def verify_credential(platform: str): async def verify_credential(platform: str):
"""Test the stored credential against one of the platform's enabled sources, """Test the stored credential by running gallery-dl --simulate
WITHOUT downloading. Routes through the platform's backend against one of the platform's enabled sources. On success stamps
(download_backends.verify_credential) — native ingester for Patreon, an last_verified. Returns {valid: bool|null, reason, last_verified?}.
authenticated API page; gallery-dl --simulate for the rest. On success valid=null means "couldn't test" (no credential, or no enabled
stamps last_verified. Returns {valid: bool|null, reason, last_verified?}; source to point at)."""
valid=null means "couldn't test" (no credential, no enabled source, or an
inconclusive network/drift result)."""
from ..models import Artist, Source from ..models import Artist, Source
from ..services.download_backends import verify_source_credential from ..services.gallery_dl import GalleryDLService, SourceConfig
async with get_session() as session: async with get_session() as session:
if not await _ext_key_ok(session): if not await _ext_key_ok(session):
@@ -156,14 +154,14 @@ async def verify_credential(platform: str):
cookies_path = await svc.get_cookies_path(platform) cookies_path = await svc.get_cookies_path(platform)
auth_token = await svc.get_token(platform) auth_token = await svc.get_token(platform)
ok, message = await verify_source_credential( gdl = GalleryDLService(images_root=Path("/images"))
platform=platform, ok, message = await gdl.verify(
url=source.url, url=source.url,
artist_slug=artist.slug, artist_slug=artist.slug,
config_overrides=source.config_overrides or {}, platform=platform,
source_config=SourceConfig.from_dict(source.config_overrides or {}),
cookies_path=str(cookies_path) if cookies_path else None, cookies_path=str(cookies_path) if cookies_path else None,
auth_token=auth_token, auth_token=auth_token,
images_root=Path("/images"),
) )
last_verified = None last_verified = None
-3
View File
@@ -44,9 +44,6 @@ def _list_record(event: DownloadEvent, source: Source | None, artist: Artist | N
"bytes_downloaded": event.bytes_downloaded, "bytes_downloaded": event.bytes_downloaded,
"error": event.error, "error": event.error,
"summary": _summary_from_metadata(event.metadata_), "summary": _summary_from_metadata(event.metadata_),
# plan #709: mid-walk live counts for a RUNNING native-ingester event
# (None otherwise; phase 3 overwrites metadata with run_stats on finish).
"live": (event.metadata_ or {}).get("live"),
} }
+41 -129
View File
@@ -1,6 +1,4 @@
"""Gallery API: cursor scroll, timeline, jump, image detail, facets.""" """Gallery API: cursor scroll, timeline, jump, image detail."""
from datetime import UTC, datetime, timedelta
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
@@ -10,88 +8,46 @@ from ..services.gallery_service import GalleryService
gallery_bp = Blueprint("gallery", __name__, url_prefix="/api/gallery") gallery_bp = Blueprint("gallery", __name__, url_prefix="/api/gallery")
def _image_json(i):
"""Serialize a GalleryImage for the scroll/similar list responses."""
return {
"id": i.id,
"sha256": i.sha256,
"mime": i.mime,
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
def _parse_date(raw):
"""Parse a YYYY-MM-DD query value to a UTC midnight datetime, or None.
Raises ValueError (→ 400) on a malformed value."""
if not raw:
return None
return datetime.strptime(raw, "%Y-%m-%d").replace(tzinfo=UTC)
def _parse_filters():
"""Parse the composable gallery filters from query args, returning
``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates.
`tag_id` accepts a single id or a comma-separated list (AND); `media` is
image|video; `sort` is newest|oldest; `platform` selects one platform
(or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are boolean
flags; `date_from`/`date_to` are inclusive calendar-day bounds (date_to is
widened by a day so the whole day is covered by the service's half-open
`< date_to`)."""
tag_raw = request.args.get("tag_id")
tag_ids = (
[int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None
) or None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
media = request.args.get("media")
media_type = media if media in ("image", "video") else None
sort = request.args.get("sort")
sort = sort if sort in ("newest", "oldest") else "newest"
platform = request.args.get("platform") or None
untagged = request.args.get("untagged") in ("1", "true", "yes")
no_artist = request.args.get("no_artist") in ("1", "true", "yes")
date_from = _parse_date(request.args.get("date_from"))
date_to = _parse_date(request.args.get("date_to"))
if date_to is not None:
date_to += timedelta(days=1) # inclusive of the date_to calendar day
filters = {
"tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id,
"media_type": media_type, "platform": platform,
"untagged": untagged, "no_artist": no_artist,
"date_from": date_from, "date_to": date_to,
}
return filters, sort
@gallery_bp.route("/scroll", methods=["GET"]) @gallery_bp.route("/scroll", methods=["GET"])
async def scroll(): async def scroll():
cursor = request.args.get("cursor") or None cursor = request.args.get("cursor") or None
try: try:
limit = int(request.args.get("limit", "50")) limit = int(request.args.get("limit", "50"))
filters, sort = _parse_filters()
except ValueError: except ValueError:
return jsonify({"error": "invalid filter or limit parameter"}), 400 return jsonify({"error": "limit must be an integer"}), 400
tag_id_raw = request.args.get("tag_id")
tag_id = int(tag_id_raw) if tag_id_raw else None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
async with get_session() as session: async with get_session() as session:
svc = GalleryService(session) svc = GalleryService(session)
try: try:
page = await svc.scroll( page = await svc.scroll(
cursor=cursor, limit=limit, sort=sort, **filters, cursor=cursor, limit=limit, tag_id=tag_id,
post_id=post_id, artist_id=artist_id,
) )
except ValueError as exc: except ValueError as exc:
return jsonify({"error": str(exc)}), 400 return jsonify({"error": str(exc)}), 400
return jsonify( return jsonify(
{ {
"images": [_image_json(i) for i in page.images], "images": [
{
"id": i.id,
"sha256": i.sha256,
"mime": i.mime,
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
for i in page.images
],
"next_cursor": page.next_cursor, "next_cursor": page.next_cursor,
"date_groups": [ "date_groups": [
{"year": y, "month": m, "image_ids": ids} for y, m, ids in page.date_groups {"year": y, "month": m, "image_ids": ids} for y, m, ids in page.date_groups
@@ -100,46 +56,20 @@ async def scroll():
) )
@gallery_bp.route("/similar", methods=["GET"])
async def similar():
"""Visual "more like this": images ranked by cosine distance to the
`similar_to` image's embedding. Composes with the scope filters (AND) but
ignores post_id and sort. Bounded top-N, no cursor."""
try:
similar_to = int(request.args["similar_to"])
limit = int(request.args.get("limit", "100"))
filters, _sort = _parse_filters()
except (KeyError, ValueError):
return jsonify({"error": "similar_to query param required"}), 400
# post_id is the exclusive post-detail view — not a similarity scope.
scope = {k: v for k, v in filters.items() if k != "post_id"}
async with get_session() as session:
svc = GalleryService(session)
try:
images = await svc.similar(image_id=similar_to, limit=limit, **scope)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
if images is None:
return jsonify({"error": "not found"}), 404
return jsonify(
{
"images": [_image_json(i) for i in images],
"next_cursor": None,
"date_groups": [],
}
)
@gallery_bp.route("/timeline", methods=["GET"]) @gallery_bp.route("/timeline", methods=["GET"])
async def timeline(): async def timeline():
try: tag_id_raw = request.args.get("tag_id")
filters, _sort = _parse_filters() tag_id = int(tag_id_raw) if tag_id_raw else None
except ValueError: post_id_raw = request.args.get("post_id")
return jsonify({"error": "invalid filter parameter"}), 400 post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
async with get_session() as session: async with get_session() as session:
svc = GalleryService(session) svc = GalleryService(session)
try: try:
buckets = await svc.timeline(**filters) buckets = await svc.timeline(
tag_id=tag_id, post_id=post_id, artist_id=artist_id
)
except ValueError as exc: except ValueError as exc:
return jsonify({"error": str(exc)}), 400 return jsonify({"error": str(exc)}), 400
return jsonify( return jsonify(
@@ -147,43 +77,25 @@ async def timeline():
) )
@gallery_bp.route("/facets", methods=["GET"])
async def facets():
try:
filters, _sort = _parse_filters()
except ValueError:
return jsonify({"error": "invalid filter parameter"}), 400
async with get_session() as session:
svc = GalleryService(session)
try:
f = await svc.facets(**filters)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(
{
"total": f.total,
"platforms": f.platforms,
"untagged": f.untagged,
"no_artist": f.no_artist,
"date_min": f.date_min.isoformat() if f.date_min else None,
"date_max": f.date_max.isoformat() if f.date_max else None,
}
)
@gallery_bp.route("/jump", methods=["GET"]) @gallery_bp.route("/jump", methods=["GET"])
async def jump(): async def jump():
try: try:
year = int(request.args["year"]) year = int(request.args["year"])
month = int(request.args["month"]) month = int(request.args["month"])
filters, sort = _parse_filters()
except (KeyError, ValueError): except (KeyError, ValueError):
return jsonify({"error": "year and month query params required"}), 400 return jsonify({"error": "year and month query params required"}), 400
tag_id_raw = request.args.get("tag_id")
tag_id = int(tag_id_raw) if tag_id_raw else None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
async with get_session() as session: async with get_session() as session:
svc = GalleryService(session) svc = GalleryService(session)
try: try:
cursor = await svc.jump_cursor( cursor = await svc.jump_cursor(
year=year, month=month, sort=sort, **filters, year=year, month=month, tag_id=tag_id,
post_id=post_id, artist_id=artist_id,
) )
except ValueError as exc: except ValueError as exc:
return jsonify({"error": str(exc)}), 400 return jsonify({"error": str(exc)}), 400
+15 -121
View File
@@ -85,22 +85,6 @@ async def create_source():
return _bad("empty_url", detail=str(exc)) return _bad("empty_url", detail=str(exc))
except DuplicateSourceError as exc: except DuplicateSourceError as exc:
return _bad("duplicate", status=409, existing_id=exc.existing_id) return _bad("duplicate", status=409, existing_id=exc.existing_id)
# Immediate kickoff: a new enabled source is armed for backfill (#693)
# but would otherwise sit idle until the next scheduler tick (~60s).
# Enqueue the first walk now, skipping only if the platform is in a
# rate-limit cooldown (the scheduler picks it up when that clears).
dispatch_id = None
if record.enabled:
cooldowns = await active_platform_cooldowns(session)
if record.platform not in cooldowns:
session.add(DownloadEvent(source_id=record.id, status="pending"))
await session.commit()
dispatch_id = record.id
if dispatch_id is not None:
from ..tasks.download import download_source
download_source.delay(dispatch_id)
return jsonify(record.to_dict()), 201 return jsonify(record.to_dict()), 201
@@ -138,119 +122,29 @@ async def delete_source(source_id: int):
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"]) @sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
async def set_backfill(source_id: int): async def set_backfill(source_id: int):
"""Plan #693/#697: start/stop a run-until-done backfill, or start a recovery. """Plan #544: arm a source for backfill mode for the next N download
Body: `{"action": "start" | "stop" | "recover"}` (default "start"). 'start' runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
walks the full post history in time-boxed chunks until it reaches the bottom source dict. While backfill_runs_remaining > 0, downloads use
(then the source shows 'complete'); 'recover' is the same walk but bypasses gallery-dl's full-walk config (skip: True + 30-min timeout) instead
the Patreon seen-ledger to re-fetch dropped-and-deleted near-dups under the of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
current pHash threshold; 'stop' cancels either back to tick mode. Returns the
updated source dict (incl. backfill_state / backfill_chunks /
backfill_bypass_seen)."""
from pathlib import Path
from ..services.credential_service import CredentialService
from ..services.download_backends import (
uses_native_ingester,
verify_source_credential,
)
from .credentials import _get_crypto
payload = await request.get_json(silent=True) or {} payload = await request.get_json(silent=True) or {}
action = payload.get("action", "start") runs = payload.get("runs", 3)
if action not in ("start", "stop", "recover"): try:
return _bad( runs = int(runs)
"invalid_action", except (TypeError, ValueError):
detail="action must be 'start', 'stop', or 'recover'", return _bad("invalid_runs", detail="runs must be an integer")
)
# Pre-flight (plan #703 #2): before arming a deep walk on a native-ingester
# platform (where verify is one cheap API page), refuse if the credential is
# DEFINITIVELY rejected — don't burn chunks against expired cookies. Proceed
# on valid OR inconclusive (a network blip shouldn't block). Gated to native
# platforms: gallery-dl verify is a slow --simulate subprocess, too heavy for
# an arm action. The credential read happens in a session that's CLOSED
# before the verify network call (don't hold a DB conn across the request).
if action in ("start", "recover"):
async with get_session() as session:
rec = await SourceService(session).get(source_id)
if rec is None:
return _bad("not_found", status=404)
native = uses_native_ingester(rec.platform)
if native:
cred = CredentialService(session, _get_crypto())
cookies_path = await cred.get_cookies_path(rec.platform)
auth_token = await cred.get_token(rec.platform)
if native:
ok, message = await verify_source_credential(
platform=rec.platform,
url=rec.url,
artist_slug=rec.artist_slug,
config_overrides=rec.config_overrides or {},
cookies_path=str(cookies_path) if cookies_path else None,
auth_token=auth_token,
images_root=Path("/images"),
)
if ok is False:
return _bad("credential_rejected", detail=message, status=409)
async with get_session() as session: async with get_session() as session:
try: try:
svc = SourceService(session) record = await SourceService(session).set_backfill_runs(
if action == "start": source_id, runs,
record = await svc.start_backfill(source_id) )
elif action == "recover":
record = await svc.start_recovery(source_id)
else:
record = await svc.stop_backfill(source_id)
except LookupError: except LookupError:
return _bad("not_found", status=404) return _bad("not_found", status=404)
except ValueError as exc:
return _bad("invalid_runs", detail=str(exc))
return jsonify(record.to_dict()) return jsonify(record.to_dict())
@sources_bp.route("/<int:source_id>/preview", methods=["POST"])
async def preview_source_endpoint(source_id: int):
"""Plan #708 B4: dry-run — count what a backfill WOULD download for a native
platform (Patreon today), without downloading. Walks the first few feed pages
and counts media not already in the seen/dead ledgers. Returns
{total_new, posts_scanned, pages_scanned, has_more, sample[]} or 409 + reason
(unresolvable campaign id / auth / drift). 400 for gallery-dl platforms (no
cheap dry-run — their verify is a slow --simulate)."""
from pathlib import Path
from ..services.credential_service import CredentialService
from ..services.download_backends import preview_source, uses_native_ingester
from ..tasks._sync_engine import sync_session_factory
from .credentials import _get_crypto
async with get_session() as session:
rec = await SourceService(session).get(source_id)
if rec is None:
return _bad("not_found", status=404)
if not uses_native_ingester(rec.platform):
return _bad(
"unsupported",
detail="Preview is only available for native-ingester platforms.",
status=400,
)
cred = CredentialService(session, _get_crypto())
cookies_path = await cred.get_cookies_path(rec.platform)
# The walk + ledger reads are sync (run off the request loop); the process
# sync engine is the same one the download task uses.
result = await preview_source(
platform=rec.platform,
url=rec.url,
source_id=source_id,
config_overrides=rec.config_overrides or {},
cookies_path=str(cookies_path) if cookies_path else None,
images_root=Path("/images"),
sync_session_factory=sync_session_factory(),
)
if "error" in result:
return _bad("preview_failed", detail=result["error"], status=409)
return jsonify(result)
@sources_bp.route("/<int:source_id>/check", methods=["POST"]) @sources_bp.route("/<int:source_id>/check", methods=["POST"])
async def check_source(source_id: int): async def check_source(source_id: int):
"""FC-3c: enqueue a download for this source. """FC-3c: enqueue a download for this source.
+6 -48
View File
@@ -14,7 +14,6 @@ from ..services.tag_service import (
TagMergeConflict, TagMergeConflict,
TagService, TagService,
TagValidationError, TagValidationError,
normalize_tag_name,
) )
from ..utils.tag_prefix import parse_kind_prefix from ..utils.tag_prefix import parse_kind_prefix
@@ -142,11 +141,6 @@ async def create_tag():
fandom_id = body.get("fandom_id") fandom_id = body.get("fandom_id")
# #701: Title-Case operator-entered tags. Only here (the explicit create
# endpoint), NOT in the shared find_or_create — the ML tagger uses that path
# and must keep the booru vocabulary's casing for allowlist matching.
name = normalize_tag_name(name)
async with get_session() as session: async with get_session() as session:
svc = TagService(session) svc = TagService(session)
try: try:
@@ -200,46 +194,15 @@ async def remove_tag_from_image(image_id: int, tag_id: int):
return "", 204 return "", 204
@tags_bp.route("/tags/<int:tag_id>", methods=["GET"])
async def get_tag(tag_id: int):
"""Resolve a single tag (used by the gallery to label its active
tag-filter chip)."""
async with get_session() as session:
tag = await session.get(Tag, tag_id)
if tag is None:
return jsonify({"error": "tag not found"}), 404
return jsonify(
{
"id": tag.id,
"name": tag.name,
"kind": tag.kind.value,
"fandom_id": tag.fandom_id,
}
)
@tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"]) @tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"])
async def update_tag(tag_id: int): async def rename_tag(tag_id: int):
"""Rename and/or re-fandom a tag. Body may carry `name` and/or body = await request.get_json()
`fandom_id` (a fandom tag id, or null to clear — character tags only). if not body or "name" not in body:
`merge: true` resolves a collision by merging into the existing tag. return jsonify({"error": "name required"}), 400
"""
body = await request.get_json() or {}
has_name = "name" in body
has_fandom = "fandom_id" in body
if not has_name and not has_fandom:
return jsonify({"error": "name or fandom_id required"}), 400
do_merge = bool(body.get("merge"))
async with get_session() as session: async with get_session() as session:
svc = TagService(session) svc = TagService(session)
try: try:
tag = None tag = await svc.rename(tag_id, body["name"])
if has_name:
tag = await svc.rename(tag_id, body["name"])
if has_fandom:
tag = await svc.set_fandom(
tag_id, body["fandom_id"], merge=do_merge
)
except TagMergeConflict as exc: except TagMergeConflict as exc:
return jsonify( return jsonify(
{ {
@@ -256,12 +219,7 @@ async def update_tag(tag_id: int):
return jsonify({"error": str(exc)}), 400 return jsonify({"error": str(exc)}), 400
await session.commit() await session.commit()
return jsonify( return jsonify(
{ {"id": tag.id, "name": tag.name, "kind": tag.kind.value}
"id": tag.id,
"name": tag.name,
"kind": tag.kind.value,
"fandom_id": tag.fandom_id,
}
) )
-4
View File
@@ -97,10 +97,6 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.prune_task_runs", "task": "backend.app.tasks.maintenance.prune_task_runs",
"schedule": 86400.0, # daily "schedule": 86400.0, # daily
}, },
"vacuum-analyze": {
"task": "backend.app.tasks.maintenance.vacuum_analyze",
"schedule": 604800.0, # weekly — reclaim dead-tuple bloat + refresh stats
},
"fc3h-backup-db-nightly": { "fc3h-backup-db-nightly": {
"task": "backend.app.tasks.backup.backup_db_nightly", "task": "backend.app.tasks.backup.backup_db_nightly",
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour "schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
-6
View File
@@ -2,7 +2,6 @@
from .app_setting import AppSetting from .app_setting import AppSetting
from .artist import Artist from .artist import Artist
from .artist_visit import ArtistVisit
from .backup_run import BackupRun from .backup_run import BackupRun
from .base import Base from .base import Base
from .credential import Credential from .credential import Credential
@@ -14,8 +13,6 @@ from .import_settings import ImportSettings
from .import_task import ImportTask from .import_task import ImportTask
from .library_audit_run import LibraryAuditRun from .library_audit_run import LibraryAuditRun
from .ml_settings import MLSettings from .ml_settings import MLSettings
from .patreon_failed_media import PatreonFailedMedia
from .patreon_seen_media import PatreonSeenMedia
from .post import Post from .post import Post
from .post_attachment import PostAttachment from .post_attachment import PostAttachment
from .series_page import SeriesPage from .series_page import SeriesPage
@@ -31,12 +28,9 @@ __all__ = [
"Base", "Base",
"AppSetting", "AppSetting",
"Artist", "Artist",
"ArtistVisit",
"BackupRun", "BackupRun",
"Source", "Source",
"Credential", "Credential",
"PatreonFailedMedia",
"PatreonSeenMedia",
"Post", "Post",
"PostAttachment", "PostAttachment",
"SeriesPage", "SeriesPage",
-36
View File
@@ -1,36 +0,0 @@
"""ArtistVisit — per-artist 'last viewed' timestamp.
Powers the "+N new since last visit" badge on the artists directory and
the matching banner on `ArtistView`. One row per artist, single global
operator. When the multi-user model lands, the PK widens to
`(user_id, artist_id)` — currently aspirational only (no User model,
no services/access.py); operator approved skipping `user_id` for now
under rule #22 (breaking changes welcome).
Seed at migration time: every existing artist gets `last_viewed_at = NOW()`
so the badge starts at 0 across the board (no noisy "5000 unseen" on
first deploy). New artists also auto-get a row via
`ArtistService.find_or_create`.
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class ArtistVisit(Base):
__tablename__ = "artist_visit"
artist_id: Mapped[int] = mapped_column(
Integer,
ForeignKey("artist.id", ondelete="CASCADE"),
primary_key=True,
)
last_viewed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
-11
View File
@@ -74,17 +74,6 @@ class ImageRecord(Base):
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
# Denormalized gallery sort key = COALESCE(primary post's post_date,
# created_at) (alembic 0035). The gallery used to compute this as a
# COALESCE across the Post outer join on every /scroll, which can't use
# an index and re-sorted a large slice of the library per page (×10 with
# the old serial batching). Materializing it lets the cursor scroll read
# ix_image_record_effective_date directly. Maintained by the importer
# (services/importer.py _apply_sidecar) when a primary post with a date
# is linked; plain inserts keep the created_at-equivalent server default.
effective_date: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), DateTime(timezone=True),
nullable=False, nullable=False,
@@ -1,45 +0,0 @@
"""PatreonFailedMedia — per-source dead-letter ledger of Patreon media that
keeps failing to download/validate.
Plan #705 (#7). A media that fails every walk (404'd CDN URL, deleted post,
geo-blocked Mux stream, persistently-corrupt bytes) would otherwise re-error
forever and re-burn backfill chunks. After ``attempts`` reaches the dead-letter
threshold the ingester skips it on routine tick/backfill walks (recovery still
re-attempts it — the operator's "try everything again"). A later clean download
clears the row (the media recovered).
`filehash` is the same per-media key the seen-ledger uses (32-hex CDN MD5, or a
``video:`` / ``post:filename`` synthesized key) — hence String(128). UNIQUE
(source_id, filehash) is the upsert key.
"""
from datetime import datetime
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.types import DateTime
from .base import Base
class PatreonFailedMedia(Base):
__tablename__ = "patreon_failed_media"
__table_args__ = (
UniqueConstraint(
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
)
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
last_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
-38
View File
@@ -1,38 +0,0 @@
"""PatreonSeenMedia — per-source ledger of Patreon media already
downloaded+processed.
Replaces gallery-dl's archive.sqlite3 with our own queryable table so
routine walks can skip media we've already ingested (and a future
"recovery" mode can deliberately bypass the ledger to re-walk).
`filehash` is normally a Patreon CDN MD5 (32 hex chars), but videos —
which have no stable content hash at discovery time — use a sentinel of
the form ``video:<post_id>:<media_id>``, hence String(128) rather than 32.
"""
from datetime import datetime
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.types import DateTime
from .base import Base
class PatreonSeenMedia(Base):
__tablename__ = "patreon_seen_media"
__table_args__ = (
# Dedup key the downloader upserts against: one ledger row per
# (source, media). A second sighting of the same media is a no-op.
UniqueConstraint("source_id", "filehash", name="uq_patreon_seen_media_source_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
)
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
seen_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+5 -41
View File
@@ -16,45 +16,9 @@ log = logging.getLogger(__name__)
ARCHIVE_EXTS = {".zip", ".cbz", ".rar", ".7z"} ARCHIVE_EXTS = {".zip", ".cbz", ".rar", ".7z"}
# Magic-byte signatures, so an archive with a mangled / extension-less filename
# is still recognised. Patreon attachment download URLs sanitize to names like
# `01_https___www.patreon.com_media-u_v3_131083093`, whose `Path.suffix` is junk
# (`.com_media-u_v3_131083093`), never `.zip` — an extension-only gate filed
# those as opaque PostAttachments and NEVER extracted them (operator-flagged
# 2026-06-06). Detection is by extension first (cheap), then header sniff.
_RAR_MAGIC = b"Rar!\x1a\x07"
_7Z_MAGIC = b"7z\xbc\xaf\x27\x1c"
def detect_archive_format(path: Path) -> str | None:
"""Return "zip" | "rar" | "7z" for an archive, else None.
Trusts a known extension first, then falls back to magic-byte sniffing so a
mis-named or extension-less archive is still handled. (zip covers .cbz too.)
"""
ext = Path(path).suffix.lower()
if ext in (".zip", ".cbz"):
return "zip"
if ext == ".rar":
return "rar"
if ext == ".7z":
return "7z"
try:
if zipfile.is_zipfile(path):
return "zip"
with open(path, "rb") as fh:
head = fh.read(8)
except OSError:
return None
if head.startswith(_RAR_MAGIC):
return "rar"
if head.startswith(_7Z_MAGIC):
return "7z"
return None
def is_archive(path: Path) -> bool: def is_archive(path: Path) -> bool:
return detect_archive_format(path) is not None return Path(path).suffix.lower() in ARCHIVE_EXTS
@contextmanager @contextmanager
@@ -68,16 +32,16 @@ def extract_archive(path: Path):
members: list[tuple[str, Path]] = [] members: list[tuple[str, Path]] = []
try: try:
try: try:
fmt = detect_archive_format(path) ext = Path(path).suffix.lower()
if fmt == "zip": if ext in (".zip", ".cbz"):
with zipfile.ZipFile(path) as zf: with zipfile.ZipFile(path) as zf:
zf.extractall(base) zf.extractall(base)
elif fmt == "rar": elif ext == ".rar":
import rarfile import rarfile
with rarfile.RarFile(path) as rf: with rarfile.RarFile(path) as rf:
rf.extractall(base) rf.extractall(base)
elif fmt == "7z": elif ext == ".7z":
import py7zr import py7zr
with py7zr.SevenZipFile(path, "r") as zf: with py7zr.SevenZipFile(path, "r") as zf:
@@ -13,10 +13,10 @@ from __future__ import annotations
import base64 import base64
from dataclasses import dataclass from dataclasses import dataclass
from sqlalchemy import and_, case, exists, func, or_, select from sqlalchemy import and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ArtistVisit, ImageRecord, Source from ..models import Artist, ImageRecord, Source
from .gallery_service import thumbnail_url from .gallery_service import thumbnail_url
_SEP = "|" _SEP = "|"
@@ -58,27 +58,9 @@ class ArtistDirectoryService:
raise ValueError("limit must be between 1 and 200") raise ValueError("limit must be between 1 and 200")
count_col = func.count(ImageRecord.id).label("image_count") count_col = func.count(ImageRecord.id).label("image_count")
# Unseen = images imported since the artist's last_viewed_at.
# NULL last_viewed_at (artist created before alembic 0034 seed
# or before find_or_create autoseed) defensively counts as
# "never visited" → all images unseen. Single grouped query, no
# N+1.
unseen_col = func.count(
case(
(
or_(
ArtistVisit.last_viewed_at.is_(None),
ImageRecord.created_at > ArtistVisit.last_viewed_at,
),
ImageRecord.id,
),
else_=None,
)
).label("unseen_count")
stmt = ( stmt = (
select(Artist, count_col, unseen_col) select(Artist, count_col)
.outerjoin(ImageRecord, ImageRecord.artist_id == Artist.id) .outerjoin(ImageRecord, ImageRecord.artist_id == Artist.id)
.outerjoin(ArtistVisit, ArtistVisit.artist_id == Artist.id)
.group_by(Artist.id) .group_by(Artist.id)
) )
if q: if q:
@@ -112,7 +94,7 @@ class ArtistDirectoryService:
next_cursor = _encode(last_artist.name, last_artist.id) next_cursor = _encode(last_artist.name, last_artist.id)
rows = rows[:limit] rows = rows[:limit]
artist_ids = [a.id for a, _, _ in rows] artist_ids = [a.id for a, _ in rows]
previews = await self._previews(artist_ids) previews = await self._previews(artist_ids)
cards = [ cards = [
@@ -122,10 +104,9 @@ class ArtistDirectoryService:
"slug": artist.slug, "slug": artist.slug,
"is_subscription": bool(artist.is_subscription), "is_subscription": bool(artist.is_subscription),
"image_count": int(image_count), "image_count": int(image_count),
"unseen_count": int(unseen_count),
"preview_thumbnails": previews.get(artist.id, []), "preview_thumbnails": previews.get(artist.id, []),
} }
for artist, image_count, unseen_count in rows for artist, image_count in rows
] ]
return DirectoryPage(cards=cards, next_cursor=next_cursor) return DirectoryPage(cards=cards, next_cursor=next_cursor)
-49
View File
@@ -9,13 +9,11 @@ Dates come from Post.post_date via ImageProvenance.post_id.
from dataclasses import dataclass from dataclasses import dataclass
from sqlalchemy import and_, case, func, or_, select from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ..models import ( from ..models import (
Artist, Artist,
ArtistVisit,
ImageProvenance, ImageProvenance,
ImageRecord, ImageRecord,
Post, Post,
@@ -124,12 +122,6 @@ class ArtistService:
) )
).scalar_one() ).scalar_one()
# Mark this artist as "visited now"; the returned count is what
# the operator should see in the banner ("N new since last
# visit"). Done LAST so the read aggregates above all see the
# pre-visit state (cosmetic — none depend on visit data).
unseen_at_visit = await self._mark_visited_returning_unseen(aid)
return { return {
"id": artist.id, "id": artist.id,
"name": artist.name, "name": artist.name,
@@ -137,7 +129,6 @@ class ArtistService:
"is_subscription": bool(artist.is_subscription), "is_subscription": bool(artist.is_subscription),
"image_count": int(image_count), "image_count": int(image_count),
"post_count": int(post_count), "post_count": int(post_count),
"unseen_count_at_visit": unseen_at_visit,
"date_range": { "date_range": {
"min": dmin.isoformat() if dmin else None, "min": dmin.isoformat() if dmin else None,
"max": dmax.isoformat() if dmax else None, "max": dmax.isoformat() if dmax else None,
@@ -166,39 +157,6 @@ class ArtistService:
], ],
} }
async def _mark_visited_returning_unseen(self, artist_id: int) -> int:
"""Read pre-visit `last_viewed_at`, count images added since,
then upsert `last_viewed_at = NOW()`. Returns the count BEFORE
the upsert so the banner has data to render.
Postgres UPSERT (`ON CONFLICT DO UPDATE`) keeps the write
atomic — no SELECT-then-INSERT race per
`reference_scalar_one_or_none_duplicates`.
"""
prev = (
await self.session.execute(
select(ArtistVisit.last_viewed_at).where(
ArtistVisit.artist_id == artist_id
)
)
).scalar_one_or_none()
count_stmt = select(func.count(ImageRecord.id)).where(
ImageRecord.artist_id == artist_id
)
if prev is not None:
count_stmt = count_stmt.where(ImageRecord.created_at > prev)
unseen = (await self.session.execute(count_stmt)).scalar_one()
upsert = pg_insert(ArtistVisit.__table__).values(artist_id=artist_id)
upsert = upsert.on_conflict_do_update(
index_elements=["artist_id"],
set_={"last_viewed_at": func.now()},
)
await self.session.execute(upsert)
await self.session.commit()
return int(unseen)
async def images( async def images(
self, slug: str, cursor: str | None, limit: int = 60 self, slug: str, cursor: str | None, limit: int = 60
) -> ArtistImagesPage | None: ) -> ArtistImagesPage | None:
@@ -272,13 +230,6 @@ class ArtistService:
artist = Artist(name=cleaned, slug=slug) artist = Artist(name=cleaned, slug=slug)
self.session.add(artist) self.session.add(artist)
await self.session.flush() await self.session.flush()
# New artist starts "caught up" — seed ArtistVisit so the
# directory's `+N new` badge stays at 0 until real new
# content arrives. Without this, the unseen-count query
# treats NULL last_viewed_at as "never visited" and would
# count every image imported in the same session.
self.session.add(ArtistVisit(artist_id=artist.id))
await self.session.flush()
await sp.commit() await sp.commit()
except IntegrityError: except IntegrityError:
await sp.rollback() await sp.rollback()
-195
View File
@@ -11,7 +11,6 @@ the one-and-done GS/IR migration tooling.)
""" """
from __future__ import annotations from __future__ import annotations
import logging
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@@ -23,8 +22,6 @@ from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
from ..models.series_page import SeriesPage from ..models.series_page import SeriesPage
from ..models.tag import image_tag from ..models.tag import image_tag
log = logging.getLogger(__name__)
def project_artist_cascade(session: Session, *, slug: str) -> dict: def project_artist_cascade(session: Session, *, slug: str) -> dict:
"""Read-only projection of what delete_artist_cascade would touch. """Read-only projection of what delete_artist_cascade would touch.
@@ -458,62 +455,6 @@ def purge_legacy_tags(session: Session, *, dry_run: bool = False) -> dict:
return result return result
# The Camie-suggestable CONTENT vocabulary. "Reset content tagging" wipes
# these so the operator can re-tag from scratch via auto-suggest. fandom +
# series (and series_page ordering) are deliberately NOT here — they're kept.
RESETTABLE_TAG_KINDS = ("general", "character")
def reset_content_tagging(session: Session, *, dry_run: bool = False) -> dict:
"""Count (dry_run) or DELETE every general + character tag so the operator
can re-tag from scratch via the Camie auto-suggest.
PRESERVED: fandom + series tags and their series_page ordering, plus every
image's image_record.tagger_predictions (untouched) so suggestions
repopulate immediately. CASCADE on image_tag / tag_alias / tag_allowlist /
tag_reference_embedding / tag_suggestion_rejection clears each deleted
tag's applications + metadata. Tag.fandom_id is SET NULL, so deleting
character tags never touches the fandom rows. Irreversible except via DB
backup restore.
Returns:
{"by_kind": {"general": N, "character": M},
"count": total tags,
"applications": image_tag rows that will be / were removed,
"sample_names": [first 50],
and on live runs "deleted": total}
"""
predicate = Tag.kind.in_(RESETTABLE_TAG_KINDS)
rows = session.execute(
select(Tag.id, Tag.name, Tag.kind).where(predicate)
).all()
by_kind: dict[str, int] = {}
for _id, _name, kind in rows:
key = kind.value if hasattr(kind, "value") else str(kind)
by_kind[key] = by_kind.get(key, 0) + 1
# Headline impact: applications (image_tag rows) that vanish via cascade.
applications = session.execute(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id.in_(select(Tag.id).where(predicate)))
).scalar_one()
sample = [name for _id, name, _kind in rows[:50]]
total = len(rows)
result = {
"by_kind": by_kind,
"count": total,
"applications": applications,
"sample_names": sample,
}
if dry_run:
return result
if total:
session.execute(Tag.__table__.delete().where(predicate))
session.commit()
result["deleted"] = total
return result
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules. # FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -668,139 +609,3 @@ def cancel_audit_run(session: Session, *, audit_id: int) -> None:
.where(LibraryAuditRun.status == "running") .where(LibraryAuditRun.status == "running")
.values(status="cancelled", finished_at=datetime.now(UTC)) .values(status="cancelled", finished_at=datetime.now(UTC))
) )
# -- archive-attachment re-extraction (#713 part 2) ------------------------
_ARCHIVE_EXT_FOR_FORMAT = {"zip": ".zip", "rar": ".rar", "7z": ".7z"}
def _reextract_archive_to_post(
importer, archive_path: Path, post, source_row, artist, images_root: Path,
) -> list[int]:
"""Extract one stored archive and link its members to `post`.
The stored attachment has no adjacent sidecar (it lives in the sha-addressed
attachment store). Stage a copy + a reconstructed sidecar UNDER the artist's
library dir (`images_root/<slug>/<platform>/<post>/`) — the importer
re-derives the artist from the path AND copies members relative to it, so the
members land in the real library and resolve to the right artist — then re-run
`attach_in_place`: the archive extracts and `find_or_create_post` re-attaches
the members to the SAME Post (source_id + external_post_id). Removes only the
staged archive + sidecar afterward; the imported member files stay. Returns
the new member image ids.
"""
import json
import shutil
from .archive_extractor import detect_archive_format
fmt = detect_archive_format(archive_path)
ext = _ARCHIVE_EXT_FOR_FORMAT.get(fmt or "", ".zip")
platform = source_row.platform if source_row is not None else "imported"
sidecar = {
"category": source_row.platform if source_row is not None
else (post.raw_metadata or {}).get("category"),
"id": post.external_post_id,
"title": post.post_title or "",
"content": post.description or "",
"published_at": post.post_date.isoformat() if post.post_date else None,
"url": post.post_url,
}
work = images_root / artist.slug / platform / str(post.external_post_id)
work.mkdir(parents=True, exist_ok=True)
staged = work / f"archive{ext}" # clean ext → is_archive + find_sidecar
sidecar_path = staged.with_suffix(".json")
try:
shutil.copy2(archive_path, staged)
sidecar_path.write_text(json.dumps(sidecar))
res = importer.attach_in_place(staged, artist=artist, source=source_row)
return list(res.member_image_ids or [])
finally:
# Drop only the staged archive + sidecar; the extracted member files
# were copied into the library alongside them and must stay.
staged.unlink(missing_ok=True)
sidecar_path.unlink(missing_ok=True)
def reextract_archive_attachments(session: Session, *, images_root: Path) -> dict:
"""Re-process existing PostAttachments that are ACTUALLY archives but were
filed opaquely before #713 part 1 (extension-only is_archive missed mangled /
extension-less Patreon attachment names). For each: extract the members,
import them, and link them to the attachment's post.
Idempotent — members dedupe by sha256, the archive dedupes by sha — so it's
safe to run repeatedly. Returns a summary dict for task_run.metadata.
"""
from ..models import ImportSettings, Post, PostAttachment, Source
from ..tasks.ml import tag_and_embed
from ..tasks.thumbnail import generate_thumbnail
from .archive_extractor import is_archive
from .importer import Importer
from .thumbnailer import Thumbnailer
summary = {
"scanned": 0, "archives": 0, "members_imported": 0,
"posts_touched": 0, "skipped_no_post": 0, "skipped_no_artist": 0,
"errors": 0,
}
settings = ImportSettings.load_sync(session)
importer = Importer(
session=session, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=settings,
)
attachments = session.execute(
select(PostAttachment).order_by(PostAttachment.id)
).scalars().all()
enqueue_ids: list[int] = []
for att in attachments:
summary["scanned"] += 1
stored = Path(att.path)
try:
if not stored.is_file() or not is_archive(stored):
continue
except OSError:
continue
summary["archives"] += 1
if att.post_id is None:
summary["skipped_no_post"] += 1
continue
post = session.get(Post, att.post_id)
if post is None:
summary["skipped_no_post"] += 1
continue
artist = session.get(Artist, att.artist_id) if att.artist_id else None
if artist is None and post.artist_id:
artist = session.get(Artist, post.artist_id)
if artist is None or not artist.slug:
# The importer re-derives the artist from the staged path, so we need
# a real artist+slug to anchor under. (Shouldn't happen for
# subscription posts; skip rather than orphan the members.)
summary["skipped_no_artist"] += 1
continue
source_row = session.get(Source, post.source_id) if post.source_id else None
try:
ids = _reextract_archive_to_post(
importer, stored, post, source_row, artist, images_root,
)
session.commit()
except Exception as exc: # one bad archive must not strand the rest
session.rollback()
summary["errors"] += 1
log.warning("re-extract failed for attachment %s: %s", att.id, exc)
continue
if ids:
summary["members_imported"] += len(ids)
summary["posts_touched"] += 1
enqueue_ids.extend(ids)
# Thumbnails + ML for the newly-imported members (best-effort; off the
# critical path — a Redis hiccup must not fail the whole re-extract).
for img_id in enqueue_ids:
try:
generate_thumbnail.delay(img_id)
tag_and_embed.delay(img_id)
except Exception as exc:
log.warning("re-extract enqueue failed for image %s: %s", img_id, exc)
return summary
-241
View File
@@ -1,241 +0,0 @@
"""Platform → download-backend dispatch (one place that knows which platforms
are served by the native FC ingester vs. the gallery-dl subprocess).
gallery-dl wasn't built to be driven by an automated scheduler — no native
checkpoint/resume, no structured logs, per-file HEADs that dominate wall-clock.
The native ingester (services/patreon_ingester.py, plan #697) replaces it for
Patreon and is the path we grow as more platforms migrate. To keep that
migration DRY, every caller that has to behave differently per backend —
download routing, the credential-verify probe, cursor handling — asks THIS
module instead of testing ``platform == "patreon"`` inline. When a platform gets
a native ingester, it moves into ``NATIVE_INGESTER_PLATFORMS`` here and both the
download path and verify switch over together.
The backend surfaces share a UNIFORM signature so a caller invokes the same
function regardless of platform:
- verify_credential(...) → (ok: bool|None, message: str)
- (download stays in download_service for now; uses_native_ingester() is the
shared predicate it routes on, so the decision lives here too.)
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from .gallery_dl import DownloadResult, ErrorType
from .patreon_ingester import PatreonIngester
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
# Platforms whose download + verify go through the native ingester rather than
# gallery-dl. gallery-dl still serves every other platform (subscribestar,
# hentaifoundry, discord, pixiv, deviantart) unchanged.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon"})
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
# messages so the operator sees the exact lookup endpoint that was hit.
_CAMPAIGNS_API = "https://www.patreon.com/api/campaigns"
def uses_native_ingester(platform: str) -> bool:
"""True when `platform` is served by the native ingester (not gallery-dl).
The single predicate the download path and verify both route on."""
return platform in NATIVE_INGESTER_PLATFORMS
async def run_download(
*,
ctx: dict,
source_config,
skip_value: bool | str,
mode: str | None,
gdl,
sync_session_factory,
) -> tuple[DownloadResult, str | None]:
"""Uniform download across backends — the download counterpart to
`verify_source_credential`, so this module is the ONE place that knows how
each platform both downloads AND verifies (the seam that makes adding a
platform a bounded job).
Returns `(DownloadResult, resolved_campaign_id)`; `resolved_campaign_id` is
non-None only when a native vanity lookup ran this call (so phase 3 caches
it). Native platforms route through their ingester in `mode`
(tick/backfill/recovery); gallery-dl platforms run the subprocess. The caller
(download_service) prepares `source_config`/`skip_value`/`mode` from the
backfill state machine and owns phase 3.
"""
platform = ctx["platform"]
if uses_native_ingester(platform):
return await _run_native_ingester(
ctx, source_config, mode, gdl, sync_session_factory
)
result = await gdl.download(
url=ctx["url"],
artist_slug=ctx["artist_slug"],
platform=platform,
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
return result, None
async def _run_native_ingester(
ctx: dict, source_config, mode: str | None, gdl, sync_session_factory,
) -> tuple[DownloadResult, str | None]:
"""Patreon (today the only native platform): resolve the campaign id, then run
the native ingester in a worker thread (it is sync requests/subprocess).
`resolved_campaign_id` is non-None only when we had to look it up from the
vanity URL this run, so phase 3 caches it the way the old gallery-dl retry
did. A campaign id we cannot resolve is a loud NOT_FOUND — never a silent
empty success.
"""
overrides = ctx["config_overrides"] or {}
campaign_id, resolved_campaign_id = await resolve_campaign_id_for_source(
ctx["url"], ctx["cookies_path"], overrides
)
if not campaign_id:
url = ctx["url"]
vanity = extract_vanity(url)
return (
DownloadResult(
success=False,
url=url,
artist_slug=ctx["artist_slug"],
platform="patreon",
error_type=ErrorType.NOT_FOUND,
error_message=(
f"Could not resolve Patreon campaign id. source_url={url!r}; "
f"vanity={vanity!r}; "
f"lookup=GET {_CAMPAIGNS_API}?filter[vanity]={vanity or ''} "
"(vanity lookup failed — cookies expired or creator moved?)"
),
),
None,
)
# Honor the operator's existing rate-limit knobs on the native path (plan
# #703): the global download_rate_limit_seconds (gallery-dl's `rate_limit`,
# here on the gdl service) paces media downloads; page fetches use the
# per-source sleep_request override, else `max(0.5, rate_limit/4)` — the same
# API-pacing default gallery-dl applied as its `sleep-request`.
rate_limit = gdl._rate_limit
request_sleep = (
source_config.sleep_request
if source_config.sleep_request is not None
else max(0.5, rate_limit / 4)
)
ingester = PatreonIngester(
images_root=gdl.images_root,
cookies_path=ctx["cookies_path"],
session_factory=sync_session_factory,
validate=gdl._validate_files,
rate_limit=rate_limit,
request_sleep=request_sleep,
)
loop = asyncio.get_running_loop()
dl_result = await loop.run_in_executor(
None,
lambda: ingester.run(
source_id=ctx["source_id"],
campaign_id=campaign_id,
artist_slug=ctx["artist_slug"],
url=ctx["url"],
mode=mode,
resume_cursor=source_config.resume_cursor,
time_budget_seconds=source_config.timeout,
posts_base=int(overrides.get("_backfill_posts", 0)),
# plan #709: live progress writes to this running event mid-walk.
event_id=ctx.get("event_id"),
),
)
return dl_result, resolved_campaign_id
async def preview_source(
*,
platform: str,
url: str,
source_id: int,
config_overrides: dict | None,
cookies_path: str | None,
images_root: Path,
sync_session_factory,
page_limit: int = 3,
) -> dict:
"""Dry-run preview for a native platform (plan #708 B4): resolve the campaign
id, then walk a few pages counting media not already seen/dead — no download.
Returns the preview dict (total_new / posts_scanned / pages_scanned /
has_more / sample), or `{"error": msg}` on a resolve / auth / drift failure.
Native-only — the caller gates on `uses_native_ingester`.
"""
import asyncio
from .patreon_client import PatreonAPIError
campaign_id, _ = await resolve_campaign_id_for_source(
url, cookies_path, config_overrides or {}
)
if not campaign_id:
vanity = extract_vanity(url)
return {
"error": (
f"Couldn't resolve the campaign id. source_url={url!r}; "
f"vanity={vanity!r}; lookup=GET {_CAMPAIGNS_API}?filter[vanity]={vanity or ''} "
"(cookies expired, or the creator moved/renamed?)."
)
}
ingester = PatreonIngester(
images_root=images_root,
cookies_path=cookies_path,
session_factory=sync_session_factory,
)
loop = asyncio.get_running_loop()
try:
result = await loop.run_in_executor(
None,
lambda: ingester.preview(source_id, campaign_id, page_limit=page_limit),
)
except PatreonAPIError as exc:
return {"error": f"Couldn't preview: {exc}"}
return result
async def verify_source_credential(
*,
platform: str,
url: str,
artist_slug: str,
config_overrides: dict | None,
cookies_path: str | None,
auth_token: str | None,
images_root: Path,
) -> tuple[bool | None, str]:
"""Uniform credential probe across backends. Returns `(ok, message)`:
True = authenticated, False = rejected, None = inconclusive (drift /
network / nothing to test). Callers don't branch on platform — they call
this and render the result.
"""
if uses_native_ingester(platform):
# Native ingester platforms verify via their own lightweight auth probe
# (resolve campaign id + one authenticated API page). Patreon today.
from .patreon_ingester import verify_patreon_credential
return await verify_patreon_credential(url, cookies_path, config_overrides)
# gallery-dl platforms: --simulate one item; the extractor errors before it
# can list if auth is bad.
from .gallery_dl import GalleryDLService, SourceConfig
gdl = GalleryDLService(images_root=images_root)
return await gdl.verify(
url=url,
artist_slug=artist_slug,
platform=platform,
source_config=SourceConfig.from_dict(config_overrides or {}),
cookies_path=cookies_path,
auth_token=auth_token,
)
+105 -225
View File
@@ -13,6 +13,7 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import re
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@@ -24,25 +25,44 @@ from sqlalchemy.orm import joinedload
from ..models import Artist, DownloadEvent, Source from ..models import Artist, DownloadEvent, Source
from .credential_service import CredentialService from .credential_service import CredentialService
from .download_backends import run_download, uses_native_ingester
from .gallery_dl import ( from .gallery_dl import (
BACKFILL_CHUNK_SECONDS,
BACKFILL_SKIP_VALUE, BACKFILL_SKIP_VALUE,
BACKFILL_TIMEOUT_SECONDS,
TICK_SKIP_VALUE, TICK_SKIP_VALUE,
DownloadResult,
ErrorType, ErrorType,
GalleryDLService, GalleryDLService,
SourceConfig, SourceConfig,
extract_errors_warnings,
truncate_log,
) )
from .importer import Importer from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .platforms import auth_type_for from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
_PATREON_VANITY_RE = re.compile(
r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
re.IGNORECASE,
)
_CAMPAIGN_ID_FAILURE_PATTERN = "failed to extract campaign id"
def _extract_patreon_vanity(url: str) -> str | None:
m = _PATREON_VANITY_RE.match(url)
return m.group(1) if m else None
def _looks_like_campaign_id_failure(stdout: str, stderr: str) -> bool:
return _CAMPAIGN_ID_FAILURE_PATTERN in f"{stdout}\n{stderr}".lower()
def _effective_url(platform: str, source_url: str, overrides: dict) -> str:
if platform == "patreon" and overrides.get("patreon_campaign_id"):
return f"https://www.patreon.com/id:{overrides['patreon_campaign_id']}"
return source_url
class DownloadService: class DownloadService:
"""Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`. """Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
@@ -57,18 +77,12 @@ class DownloadService:
gdl: GalleryDLService, gdl: GalleryDLService,
importer: Importer, importer: Importer,
cred_service: CredentialService, cred_service: CredentialService,
sync_session_factory=None,
): ):
self.async_session = async_session self.async_session = async_session
self.sync_session = sync_session self.sync_session = sync_session
self.gdl = gdl self.gdl = gdl
self.importer = importer self.importer = importer
self.cred_service = cred_service self.cred_service = cred_service
# Sync sessionmaker the native Patreon ingester opens SHORT-LIVED
# sessions from for its seen-ledger reads/writes (never one held across
# the multi-minute walk — see PatreonIngester). Only the patreon branch
# of phase 2 uses it; gallery-dl sources leave it None.
self.sync_session_factory = sync_session_factory
async def download_source(self, source_id: int) -> int: async def download_source(self, source_id: int) -> int:
"""Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is.""" """Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
@@ -77,101 +91,64 @@ class DownloadService:
return setup["event_id"] return setup["event_id"]
ctx = setup ctx = setup
# Release the phase-1 DB connections before the (up to ~19.5-min in
# backfill) gallery-dl subprocess. Held checked-out across that idle
# window, the asyncpg/psycopg connections get reaped by the server,
# and phase 3's first query then hits a dead socket
# (asyncpg ConnectionDoesNotExistError) → download_source autoretry →
# _phase1_setup's in-flight guard no-ops the retry → the event
# strands empty for the recovery sweep (Anduo #40014, 2026-06-04).
# pool_pre_ping can't help a *held* connection — it only validates on
# pool checkout. Closing returns them to the pool so phase 3 re-
# acquires a live one (the async task engine uses NullPool, the sync
# engine pre_ping + pool_recycle=300). This is what makes the
# "Phase 2 — no DB connection" contract in the class docstring true.
await self.async_session.close()
self.sync_session.close()
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {}) source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
# Backfill mode (plan #693): the source's `_backfill_state == "running"` # alembic 0031 / plan #544: derive skip_value + timeout from the
# selects a time-boxed deep-walk chunk — skip: True (walk full history) # source's backfill_runs_remaining counter. When > 0, walk the full
# + the BACKFILL_CHUNK_SECONDS budget, resuming from the cursor # post history (skip: True + 1800s); when 0, exit gallery-dl after
# checkpoint (plan #689). State is the single source of truth; it stays # 20 contiguous archived items (skip: "exit:20" + the default
# "running" across ticks until the walk reaches the bottom (phase 3 # 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
# flips it to "complete"). Otherwise tick mode: exit gallery-dl after backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
# 20 contiguous archived items (skip: "exit:20" + the default 870s). if backfill_remaining > 0:
# Operator drives this via POST /api/sources/{id}/backfill {action}.
overrides = ctx["config_overrides"] or {}
in_backfill = overrides.get("_backfill_state") == "running"
# Recovery (plan #697) reuses the entire #693 backfill state machine —
# cursor checkpoint, time-boxed chunks, complete/stall lifecycle — and
# differs only in bypassing the tier-1 seen-ledger so dropped-and-deleted
# near-dups get re-fetched and re-evaluated under the CURRENT pHash
# threshold (tier-2 disk still spares files we kept). The
# `_backfill_bypass_seen` flag rides alongside the running backfill state;
# download mode is "recovery" when both are set.
bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
if in_backfill:
skip_value: bool | str = BACKFILL_SKIP_VALUE skip_value: bool | str = BACKFILL_SKIP_VALUE
source_config.timeout = BACKFILL_CHUNK_SECONDS source_config.timeout = BACKFILL_TIMEOUT_SECONDS
pending_cursor = overrides.get("_backfill_cursor")
if uses_native_ingester(ctx["platform"]) and pending_cursor:
source_config.resume_cursor = pending_cursor
else: else:
skip_value = TICK_SKIP_VALUE skip_value = TICK_SKIP_VALUE
# Phase 2 dispatch is uniform across backends (download_backends. effective_url = _effective_url(
# run_download — the download counterpart to verify_source_credential): ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
# native platforms run their ingester in `mode` (zero per-file HEADs,
# native cursor/resume, loud drift detection); gallery-dl platforms run
# the subprocess. Either returns a DownloadResult-shaped object so phase 3
# is untouched. `mode` is None for gallery-dl (run_download ignores it).
mode: str | None = None
if uses_native_ingester(ctx["platform"]):
if in_backfill and bypass_seen:
mode = "recovery"
elif in_backfill:
mode = "backfill"
else:
mode = "tick"
dl_result, resolved_campaign_id = await self._run_download(
ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
) )
# A backfill chunk that hit its time-box but made forward progress is dl_result = await self.gdl.download(
# NORMAL, not a failure — reclassify TIMEOUT → PARTIAL so it reads as url=effective_url,
# "ok/progress" (PARTIAL maps to status "ok"), not a red error, and the artist_slug=ctx["artist_slug"],
# next chunk just resumes from the new cursor (plan #693). A chunk that platform=ctx["platform"],
# timed out with NO progress stays TIMEOUT and feeds phase 3's source_config=source_config,
# stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires. cookies_path=ctx["cookies_path"],
if in_backfill and dl_result.error_type == ErrorType.TIMEOUT: auth_token=ctx["auth_token"],
new_cursor = dl_result.cursor # plan #704: structured, not scraped skip_value=skip_value,
advanced = bool( )
(new_cursor and new_cursor != overrides.get("_backfill_cursor"))
or dl_result.files_downloaded > 0 resolved_campaign_id: str | None = None
) if (
if advanced: ctx["platform"] == "patreon"
dl_result.error_type = ErrorType.PARTIAL and not dl_result.success
dl_result.error_message = ( and "patreon_campaign_id" not in (ctx["config_overrides"] or {})
f"Backfill chunk: {dl_result.files_downloaded} file(s) — continuing" and _looks_like_campaign_id_failure(dl_result.stdout, dl_result.stderr)
):
vanity = _extract_patreon_vanity(ctx["url"])
if vanity:
log.info(
"Attempting campaign-ID resolution for %s (%s)",
ctx["artist_slug"], vanity,
) )
resolved_campaign_id = await resolve_campaign_id(
vanity, ctx["cookies_path"]
)
if resolved_campaign_id:
dl_result = await self.gdl.download(
url=f"https://www.patreon.com/id:{resolved_campaign_id}",
artist_slug=ctx["artist_slug"],
platform=ctx["platform"],
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
return await self._phase3_persist( return await self._phase3_persist(
ctx["event_id"], ctx, dl_result, resolved_campaign_id, ctx["event_id"], ctx, dl_result, resolved_campaign_id,
) )
async def _run_download(
self, *, ctx: dict, source_config, skip_value, mode: str | None,
) -> tuple[DownloadResult, str | None]:
"""Phase-2 dispatch → `download_backends.run_download`, passing this
service's gdl + sync sessionmaker. Kept as a thin instance method so tests
can stub the whole phase-2 dispatch on the service, and so the per-backend
construction lives in download_backends (the backend registry)."""
return await run_download(
ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
gdl=self.gdl, sync_session_factory=self.sync_session_factory,
)
async def _phase1_setup(self, source_id: int) -> dict[str, Any]: async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
source = (await self.async_session.execute( source = (await self.async_session.execute(
select(Source).options(joinedload(Source.artist)).where(Source.id == source_id) select(Source).options(joinedload(Source.artist)).where(Source.id == source_id)
@@ -269,11 +246,6 @@ class DownloadService:
source_row = self.sync_session.get(Source, ctx["source_id"]) source_row = self.sync_session.get(Source, ctx["source_id"])
import_summary = {"attached": 0, "skipped": 0, "errors": 0} import_summary = {"attached": 0, "skipped": 0, "errors": 0}
# Archives detected but captured WITHOUT extracting any image (probe
# rejected / corrupt / missing extractor backend). Surfaced on the event
# so a post showing "no images" beside a zip is diagnosable (plan
# follow-up 2026-06-06 — the recurring archive-association report).
unextracted_archives: list[dict] = []
bytes_downloaded = 0 bytes_downloaded = 0
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
@@ -327,17 +299,6 @@ class DownloadService:
# mirror duplicate_hash cleanup so we don't keep two copies. # mirror duplicate_hash cleanup so we don't keep two copies.
# Operator-flagged 2026-06-02 (Lustria OST zip). # Operator-flagged 2026-06-02 (Lustria OST zip).
import_summary["attached"] += 1 import_summary["attached"] += 1
# An archive captured WITHOUT extracting any image carries a
# reason on result.error — record it so the event explains the
# "no images beside a zip" symptom instead of staying silent.
if result.error:
unextracted_archives.append(
{"file": path.name, "reason": result.error}
)
log.warning(
"archive captured unextracted (%s): %s",
path.name, result.error,
)
try: try:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240 bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError: except OSError:
@@ -384,16 +345,11 @@ class DownloadService:
select(DownloadEvent).where(DownloadEvent.id == event_id) select(DownloadEvent).where(DownloadEvent.id == event_id)
)).scalar_one() )).scalar_one()
# plan #704: the native ingester returns structured run_stats; only the run_stats = self.gdl._compute_run_stats(
# gallery-dl path needs the regex-over-stdout reconstruction. dl_result.return_code, dl_result.stdout, dl_result.stderr
if dl_result.run_stats is not None: )
run_stats = dict(dl_result.run_stats)
else:
run_stats = self.gdl._compute_run_stats(
dl_result.return_code, dl_result.stdout, dl_result.stderr
)
run_stats["quarantined_count"] = dl_result.files_quarantined run_stats["quarantined_count"] = dl_result.files_quarantined
stderr_summary = extract_errors_warnings(dl_result.stderr) stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
# Plan #544: PARTIAL means the run downloaded ≥1 file but the # Plan #544: PARTIAL means the run downloaded ≥1 file but the
# subprocess didn't finish in budget (typically wall-clock timeout # subprocess didn't finish in budget (typically wall-clock timeout
@@ -413,116 +369,50 @@ class DownloadService:
ev.metadata_ = { ev.metadata_ = {
"run_stats": run_stats, "run_stats": run_stats,
"error_type": dl_result.error_type.value if dl_result.error_type else None, "error_type": dl_result.error_type.value if dl_result.error_type else None,
"stdout": truncate_log(dl_result.stdout) or None, "stdout": self.gdl._truncate_log(dl_result.stdout) or None,
"stderr": truncate_log(dl_result.stderr) or None, "stderr": self.gdl._truncate_log(dl_result.stderr) or None,
"stderr_errors_warnings": stderr_summary or None, "stderr_errors_warnings": stderr_summary or None,
"duration_seconds": dl_result.duration_seconds, "duration_seconds": dl_result.duration_seconds,
"quarantined_paths": dl_result.quarantined_paths or None, "quarantined_paths": dl_result.quarantined_paths or None,
"import_summary": import_summary, "import_summary": import_summary,
# Archives detected but captured without extracting an image — the
# recurring "post shows a zip but no images" report. Each entry is
# {file, reason}; None when every archive extracted cleanly.
"unextracted_archives": unextracted_archives or None,
} }
await self._update_source_health( await self._update_source_health(
source_id=ctx["source_id"], status=status, error_message=ev.error, source_id=ctx["source_id"], status=status, error_message=ev.error,
error_type=dl_result.error_type.value if dl_result.error_type else None, error_type=dl_result.error_type.value if dl_result.error_type else None,
retry_after_seconds=getattr(dl_result, "retry_after_seconds", None),
) )
await self._apply_backfill_lifecycle(ctx, dl_result) # Plan #544: backfill lifecycle — auto-complete when a clean
# backfill run drained the queue (gallery-dl exited 0 + zero files
# downloaded means there was nothing to fetch); otherwise decrement
# the counter. Next tick falls back to tick mode once it hits 0.
#
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
# subprocess with return_code=0 and files_downloaded=0 (every
# file was quarantined), which used to match the auto-complete
# predicate exactly — zeroing the operator's armed budget on
# the FIRST quarantine run instead of decrementing. Require
# dl_result.success + no error_type so only genuinely-empty
# successful runs drain the counter.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
queue_drained = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
and dl_result.files_downloaded == 0
)
if queue_drained:
src.backfill_runs_remaining = 0
else:
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
await self.async_session.commit() await self.async_session.commit()
return event_id return event_id
async def _apply_backfill_lifecycle(self, ctx: dict, dl_result) -> None:
"""Backfill state machine (plan #693, building on the cursor of #689).
A backfill runs in time-boxed chunks while
`config_overrides["_backfill_state"] == "running"`. Each chunk:
- COMPLETES the walk → clean rc=0 (gallery-dl with skip:True exits 0
only after exhausting the newest→oldest walk; a chunk cut short by
its time-box returns success=False / rc<0 via TimeoutExpired). On
completion: state="complete", clear the cursor, return to tick mode.
- made PROGRESS (cursor advanced and/or files written) → stay
"running", checkpoint the new cursor, bump the chunk counter, and
spend one of the safety-cap chunks (backfill_runs_remaining). If the
cap is exhausted without finishing → state="stalled".
- made NO progress → increment the stall counter; two strikes →
state="stalled", clear the cursor (a wedged walk can't loop).
State is the single source of truth; the cursor lives only inside a
running backfill. config_overrides is reassigned (not mutated in place)
for SQLAlchemy JSON change detection.
"""
overrides = ctx["config_overrides"] or {}
if overrides.get("_backfill_state") != "running":
return # not backfilling — tick mode, nothing to do
old_cursor = overrides.get("_backfill_cursor")
cap_remaining = ctx.get("backfill_runs_remaining", 0) or 0
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
new_overrides = dict(src.config_overrides or {})
chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
new_overrides["_backfill_chunks"] = chunks
# plan #704 (#5): _backfill_posts (the live progress badge) is OWNED by the
# ingester now — it writes a monotonic absolute mid-walk at each page
# boundary (ingest_core._checkpoint_posts), so the badge climbs DURING a
# chunk instead of jumping once per chunk here, and the re-walked resume
# page is no longer double-counted. new_overrides (read fresh above)
# carries the ingester's committed value forward untouched.
completed = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
)
if completed:
new_overrides["_backfill_state"] = "complete"
new_overrides.pop("_backfill_cursor", None)
new_overrides.pop("_backfill_cursor_stalls", None)
# plan #697: a recovery walk shares this lifecycle; clear its bypass
# flag on completion so the next routine tick honors the seen-ledger.
new_overrides.pop("_backfill_bypass_seen", None)
src.config_overrides = new_overrides
src.backfill_runs_remaining = 0
return
# Did not finish. The native ingester checkpoints + resumes via cursor
# (carried structurally on the result, plan #704); gallery-dl platforms
# have no resumable cursor (every chunk re-walks from the top), so they
# advance only by the download archive growing.
new_cursor = (
dl_result.cursor if uses_native_ingester(ctx["platform"]) else None
)
advanced = bool(
(new_cursor and new_cursor != old_cursor)
or dl_result.files_downloaded > 0
)
if advanced:
if new_cursor:
new_overrides["_backfill_cursor"] = new_cursor
new_overrides.pop("_backfill_cursor_stalls", None)
cap_remaining = max(0, cap_remaining - 1)
src.backfill_runs_remaining = cap_remaining
if cap_remaining == 0:
# Safety cap hit before reaching the bottom — pause, don't loop.
new_overrides["_backfill_state"] = "stalled"
else:
stalls = int(new_overrides.get("_backfill_cursor_stalls", 0)) + 1
if stalls >= 2:
new_overrides["_backfill_state"] = "stalled"
new_overrides.pop("_backfill_cursor", None)
new_overrides.pop("_backfill_cursor_stalls", None)
else:
new_overrides["_backfill_cursor_stalls"] = stalls
src.config_overrides = new_overrides
async def _update_source_health( async def _update_source_health(
self, *, source_id: int, status: str, error_message: str | None, self, *, source_id: int, status: str, error_message: str | None,
error_type: str | None = None, retry_after_seconds: float | None = None, error_type: str | None = None,
) -> None: ) -> None:
"""FC-3d: update Source.{consecutive_failures, last_error, last_checked_at}. """FC-3d: update Source.{consecutive_failures, last_error, last_checked_at}.
@@ -554,17 +444,7 @@ class DownloadService:
# by error class without opening Logs per row. # by error class without opening Logs per row.
source.error_type = error_type source.error_type = error_type
if error_type == "rate_limited": if error_type == "rate_limited":
# plan #708 B1: honor the server's Retry-After when the native await set_platform_cooldown(self.async_session, source.platform)
# client surfaced one (clamped to a sane [60, 3600] window so a
# tiny hint can't leave the platform effectively un-cooled and a
# huge one can't strand it for hours); else the flat default.
if retry_after_seconds is not None:
seconds = int(min(max(retry_after_seconds, 60), 3600))
await set_platform_cooldown(
self.async_session, source.platform, seconds=seconds,
)
else:
await set_platform_cooldown(self.async_session, source.platform)
elif status == "skipped": elif status == "skipped":
source.last_error = None source.last_error = None
source.last_checked_at = now source.last_checked_at = now
+8 -9
View File
@@ -16,7 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source from ..models import Artist, Source
from ..utils.slug import slugify from ..utils.slug import slugify
from .source_service import BACKFILL_MAX_CHUNKS from .source_service import NEW_SOURCE_BACKFILL_RUNS
class UnknownPlatformError(Exception): class UnknownPlatformError(Exception):
@@ -205,17 +205,16 @@ class ExtensionService:
return existing, False return existing, False
sp = await self.session.begin_nested() sp = await self.session.begin_nested()
try: try:
# New subscription sources arm run-until-done backfill (plan #693) # New subscription sources arm a few backfill runs so the
# so the first ticks walk the full history (otherwise gallery-dl's # first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built). Mirrors # exit:20 short-circuits before the archive is built).
# SourceService.create — without it, Firefox quick-add on a creator # Mirrors SourceService.create — without it, Firefox quick-
# with >20 unsynced posts would surface as "check failed" with no # add on a creator with >20 unsynced posts would surface
# diagnosis. Audit 2026-06-02. # as "check failed" with no diagnosis. Audit 2026-06-02.
src = Source( src = Source(
artist_id=artist_id, platform=platform, artist_id=artist_id, platform=platform,
url=url, enabled=True, url=url, enabled=True,
config_overrides={"_backfill_state": "running"}, backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
backfill_runs_remaining=BACKFILL_MAX_CHUNKS,
) )
self.session.add(src) self.session.add(src)
await self.session.flush() await self.session.flush()
-55
View File
@@ -11,15 +11,9 @@ expected to write.
from __future__ import annotations from __future__ import annotations
import json
import logging
import shutil
from dataclasses import dataclass from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
log = logging.getLogger(__name__)
JPEG_HEAD = b"\xff\xd8\xff" JPEG_HEAD = b"\xff\xd8\xff"
JPEG_TAIL = b"\xff\xd9" JPEG_TAIL = b"\xff\xd9"
@@ -114,52 +108,3 @@ def validate_file(path: Path) -> ValidationResult:
return ValidationResult(ok=True, format="webp", size=size) return ValidationResult(ok=True, format="webp", size=size)
return ValidationResult(ok=True, format=None, size=size) return ValidationResult(ok=True, format=None, size=size)
def quarantine_file(
images_root: Path,
path: Path,
artist_slug: str,
platform: str,
*,
url: str | None,
result: ValidationResult,
) -> Path | None:
"""Move a validation-failed file to `_quarantine/<slug>/<platform>` and write
a `.quarantine.json` provenance sidecar next to it.
Returns the destination path, or None if the move itself failed (file left
in place — the caller decides what to report). The caller has already run
`validate_file` and seen `result.ok is False`. ONE implementation for both
download backends — gallery-dl's batch post-process and the native ingester's
per-media path — so the quarantine layout + provenance sidecar can't drift.
"""
quarantine_root = images_root / "_quarantine" / artist_slug / platform
try:
quarantine_root.mkdir(parents=True, exist_ok=True)
dest = quarantine_root / path.name
counter = 1
while dest.exists():
dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
counter += 1
shutil.move(str(path), str(dest))
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
sidecar.write_text(
json.dumps(
{
"original_path": str(path),
"source_url": url,
"artist_slug": artist_slug,
"platform": platform,
"format": result.format,
"reason": result.reason,
"size": result.size,
"quarantined_at": datetime.now(UTC).isoformat(),
},
indent=2,
)
)
except OSError as exc:
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
return None
return dest
+124 -165
View File
@@ -22,7 +22,7 @@ from datetime import UTC, datetime
from enum import StrEnum from enum import StrEnum
from pathlib import Path from pathlib import Path
from .file_validator import is_validatable, quarantine_file, validate_file from .file_validator import is_validatable, validate_file
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
@@ -39,12 +39,6 @@ class ErrorType(StrEnum):
HTTP_ERROR = "http_error" HTTP_ERROR = "http_error"
UNSUPPORTED_URL = "unsupported_url" UNSUPPORTED_URL = "unsupported_url"
VALIDATION_FAILED = "validation_failed" VALIDATION_FAILED = "validation_failed"
# Native Patreon ingester only (plan #697): a response parsed as JSON but
# didn't match the JSON:API shape the ingester depends on (missing `data`,
# media lacking `file_name`/`url`, etc.). Distinct from AUTH_ERROR — the fix
# is updating the ingester's field-set/parser, not rotating credentials. The
# contract test guards against silently shipping this.
API_DRIFT = "api_drift"
# Run made real progress (downloaded ≥1 file) but did not finish in the # Run made real progress (downloaded ≥1 file) but did not finish in the
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status # subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
# mapping classifies this as "ok" because the next tick continues. # mapping classifies this as "ok" because the next tick continues.
@@ -61,36 +55,33 @@ class ErrorType(StrEnum):
# 20 contiguous HEADs is still negligible. # 20 contiguous HEADs is still negligible.
TICK_SKIP_VALUE = "exit:20" TICK_SKIP_VALUE = "exit:20"
# Backfill mode (operator-triggered deep scan): walk the full history, # Backfill mode (operator-triggered deep scan): walk the full history.
# one TIME-BOXED CHUNK per run (plan #693). config_overrides["_backfill_state"] # Source.backfill_runs_remaining > 0 selects this mode; the longer
# == "running" selects this mode; the cursor checkpoint (plan #689) lets each # timeout below absorbs creators with thousands of posts.
# chunk resume where the last stopped, so the walk advances across chunks
# until gallery-dl exits cleanly (= reached the bottom).
# #
# The chunk budget is deliberately FAR below download_source's Celery # 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# soft_time_limit (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py), not # (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# "just under" it. Hitting this budget is the NORMAL chunk boundary, not a # before Celery SIGKILLs the worker — same rationale as the tick
# failure: subprocess.run raises TimeoutExpired (which captures partial # default at line 74. The audit (2026-06-02) caught this at 1800,
# stdout/stderr + the last emitted cursor), and download_service reclassifies # guaranteeing SIGKILL on any backfill that ran to its subprocess
# a chunk that made progress as PARTIAL (status "ok"), not an error. The huge # budget: stdout/stderr lost, backfill_runs_remaining never
# headroom means a stuck file or a slow chunk can never let Celery's # decrements, recovery sweep stamps generic "stranded" 30 min later.
# SoftTimeLimitExceeded preempt TimeoutExpired (the failure mode behind Knuxy # Recreates the exact Knuxy #38275 failure mode the tick 870s default
# #38275 / Anduo #39912/#40411). Earlier we ran one ~1170s run-to-the-wall # was added to prevent. backfill_runs_remaining=3 still gives ~58
# per arming; that died as a timeout error every time on large catalogs. # minutes of cumulative walk across three runs for prolific creators.
BACKFILL_SKIP_VALUE = True BACKFILL_SKIP_VALUE = True
BACKFILL_CHUNK_SECONDS = 600 BACKFILL_TIMEOUT_SECONDS = 1170
# Sits well below download_source's Celery soft_time_limit # 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py). subprocess.run MUST # tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# raise TimeoutExpired before Celery raises SoftTimeLimitExceeded — # Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
# otherwise Celery wins the race, SIGKILLs the worker, in-memory # SIGKILLs the worker, in-memory stdout/stderr is lost, and the
# stdout/stderr is lost, and the DownloadEvent ends up empty-logged with # DownloadEvent ends up empty-logged with "stranded by recovery sweep"
# "stranded by recovery sweep" (operator-flagged 2026-05-31, Knuxy event # 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
# #38275; recurred in backfill mode as Anduo #39912). Per-source bumps # The 30s buffer absorbs scheduler jitter / GC pauses without making
# still live in source.config_overrides for legitimately long syncs — # legitimately-long-running syncs timeout-friendlier. Per-source bumps
# keep any override below the soft limit, or the soft-limit salvage path # still live in source.config_overrides for legitimately long syncs.
# in tasks/download.py (_finalize_soft_limited) is the only safety net.
_DEFAULT_GDL_TIMEOUT_SECONDS = 870 _DEFAULT_GDL_TIMEOUT_SECONDS = 870
@@ -103,17 +94,6 @@ class SourceConfig:
Source.backfill_runs_remaining column at the download_service layer Source.backfill_runs_remaining column at the download_service layer
and is passed to _build_config_for_source as `skip_value`. Same for and is passed to _build_config_for_source as `skip_value`. Same for
the per-run subprocess timeout. the per-run subprocess timeout.
`resume_cursor` is RUNTIME state, not persisted operator config: the
cursor-paged backfill checkpoint (see parse_last_cursor + the
download_service backfill lifecycle). It is consumed only by the native
Patreon ingester (the sole cursor-paged platform; gallery-dl's Patreon path
was removed at the #697 cutover): on a backfill/recovery run the ingester
resumes its newest→oldest walk from that pagination cursor instead of
restarting from the top. download_service threads it from
config_overrides["_backfill_cursor"]; SourceConfig deliberately does NOT read
it in from_dict (config_overrides also holds operator config, and
resume_cursor must only apply in backfill/recovery mode).
""" """
content_types: list[str] = field(default_factory=lambda: ["all"]) content_types: list[str] = field(default_factory=lambda: ["all"])
sleep: float | None = None sleep: float | None = None
@@ -122,7 +102,6 @@ class SourceConfig:
filename_pattern: str | None = None filename_pattern: str | None = None
save_metadata: bool = True save_metadata: bool = True
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
resume_cursor: str | None = None
@classmethod @classmethod
def from_dict(cls, data: dict) -> SourceConfig: def from_dict(cls, data: dict) -> SourceConfig:
@@ -155,51 +134,6 @@ class DownloadResult:
duration_seconds: float = 0.0 duration_seconds: float = 0.0
started_at: str | None = None started_at: str | None = None
completed_at: str | None = None completed_at: str | None = None
# Plan #704 — structured fields the NATIVE ingester populates directly (None
# on the gallery-dl path, which keeps the regex-over-stdout route). When set,
# phase 3 reads these instead of scraping the text it would otherwise have to
# reconstruct: `run_stats` mirrors _compute_run_stats' shape; `cursor` is the
# backfill checkpoint the ingester knows exactly (no parse_last_cursor).
run_stats: dict | None = None
cursor: str | None = None
posts_processed: int = 0
# Plan #708 B1 — the server's Retry-After seconds on a RATE_LIMITED result, so
# the platform cooldown matches the hint instead of a flat default. None when
# unknown (no header, or not a rate-limit failure).
retry_after_seconds: float | None = None
def extract_errors_warnings(stderr: str) -> str:
"""Keep only the `[error]`/`[warning]` lines from a captured stderr blob.
A generic download-log helper (module-level so the native-ingester result
path can shape its logs without reaching through a GalleryDLService instance).
"""
if not stderr:
return ""
kept = [
line for line in stderr.splitlines()
if "][error]" in line.lower() or "][warning]" in line.lower()
]
return "\n".join(kept)
def truncate_log(text: str, max_bytes: int = 500_000) -> str:
"""Cap a captured log to `max_bytes`, eliding the middle (head + tail kept)."""
if not text:
return text
encoded = text.encode("utf-8")
if len(encoded) <= max_bytes:
return text
half = max_bytes // 2
head = encoded[:half].decode("utf-8", errors="ignore")
tail = encoded[-half:].decode("utf-8", errors="ignore")
head_lines = head.count("\n")
tail_lines = tail.count("\n")
total_lines = text.count("\n")
elided = max(0, total_lines - head_lines - tail_lines)
marker = f"\n\n... [{elided} lines elided, {len(encoded) - max_bytes} bytes] ...\n\n"
return head + marker + tail
def _summarize_validation_failures(failures: list[dict]) -> str: def _summarize_validation_failures(failures: list[dict]) -> str:
@@ -216,40 +150,6 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)" return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
# (parse_last_cursor was removed in plan #704: the native ingester now carries
# its checkpoint cursor as a structured DownloadResult.cursor field, so there is
# no log text to scrape — and gallery-dl platforms never had a cursor.)
def make_run_stats(
*,
exit_code: int = 0,
downloaded_count: int = 0,
skipped_count: int = 0,
per_item_failures: int = 0,
warning_count: int = 0,
tier_gated_count: int = 0,
quarantined_count: int = 0,
dead_lettered_count: int = 0,
) -> dict:
"""The canonical `run_stats` dict shape, in ONE place.
Both result producers — gallery-dl's `_compute_run_stats` (log scrape) and the
native ingester's per-outcome tally (ingest_core) — build through this so the
key set can't drift between backends. Phase 3 + the Logs UI read these keys.
"""
return {
"exit_code": exit_code,
"downloaded_count": downloaded_count,
"skipped_count": skipped_count,
"per_item_failures": per_item_failures,
"warning_count": warning_count,
"tier_gated_count": tier_gated_count,
"quarantined_count": quarantined_count,
"dead_lettered_count": dead_lettered_count,
}
class GalleryDLService: class GalleryDLService:
"""Service for executing gallery-dl downloads.""" """Service for executing gallery-dl downloads."""
@@ -283,10 +183,16 @@ class GalleryDLService:
"permission denied", "tier required", "pledge required", "permission denied", "tier required", "pledge required",
] ]
# Per-platform defaults for the gallery-dl-backed platforms. Patreon was # Per-platform defaults. Lifted from GS — same six platforms FC supports.
# removed at the plan-#697 cutover — it now uses the native ingester
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = { PLATFORM_DEFAULTS = {
"patreon": {
"content_types": ["images", "image_large", "attachments", "postfile", "content"],
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
"filename": "{num:>02}_{filename}.{extension}",
"videos": True,
"embeds": True,
"cursor": True,
},
"subscribestar": { "subscribestar": {
"content_types": ["all"], "content_types": ["all"],
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"], "directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
@@ -345,12 +251,7 @@ class GalleryDLService:
"skip": True, "skip": True,
"sleep": self._rate_limit, "sleep": self._rate_limit,
"sleep-request": max(0.5, self._rate_limit / 4), "sleep-request": max(0.5, self._rate_limit / 4),
# 2 (not 3) retries — a stuck CDN host shouldn't burn a whole "retries": 3,
# backfill chunk on one file. Anduo #40838 spent ~600s on a
# single image (4 extractor HEAD retries × 30s + 4 downloader
# GET retries × 120s); halving retries + the downloader
# timeout below caps a wedged file at ~1-2 min instead.
"retries": 2,
"timeout": 30.0, "timeout": 30.0,
"verify": True, "verify": True,
"postprocessors": [ "postprocessors": [
@@ -365,17 +266,27 @@ class GalleryDLService:
"downloader": { "downloader": {
"part": True, "part": True,
"part-directory": str(self._config_dir / "temp"), "part-directory": str(self._config_dir / "temp"),
# See the extractor retries note above (Anduo #40838): 2 "retries": 3,
# retries + a 60s read-timeout (was 120) so a stalled "timeout": 120.0,
# connection fails fast. 60s is a per-read timeout, not a # Forward Patreon as Referer/Origin to yt-dlp when it
# transfer cap — large GIFs that keep streaming are unaffected. # fetches video manifests. Operator-flagged 2026-06-01
"retries": 2, # (DaferQ patreon): video posts hosted on Mux carry a JWT
"timeout": 60.0, # playback restriction that checks Referer/Origin on every
# NOTE: the Patreon/Mux yt-dlp Referer/Origin forwarding lived # request — not just the token signature. gallery-dl's
# here until the plan-#697 cutover. It was Patreon-specific (and # HEAD probe to stream.mux.com returns 200 (the token is
# would have wrongly tagged the other platforms' yt-dlp fetches); # valid), but yt-dlp's actual GET-with-Range to fetch the
# Patreon video is now handled by the native ingester's # m3u8 manifest 403s because yt-dlp sends its own default
# downloader (patreon_downloader._VIDEO_HEADERS), so it's gone. # Referer, which Mux's policy rejects. Forcing the right
# headers fixes the headers-only case; Mux IP-range
# restrictions are unfixable from here.
"ytdl": {
"raw-options": {
"http_headers": {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
},
},
},
}, },
"output": {"progress": True}, "output": {"progress": True},
} }
@@ -427,7 +338,14 @@ class GalleryDLService:
platform_section = config["extractor"].setdefault(platform, {}) platform_section = config["extractor"].setdefault(platform, {})
if platform == "hentaifoundry": if platform == "patreon":
if "all" in source_config.content_types:
platform_section["files"] = [
"images", "image_large", "attachments", "postfile", "content",
]
else:
platform_section["files"] = source_config.content_types
elif platform == "hentaifoundry":
if "pictures" in source_config.content_types or "all" in source_config.content_types: if "pictures" in source_config.content_types or "all" in source_config.content_types:
platform_section["include"] = "all" platform_section["include"] = "all"
@@ -587,6 +505,10 @@ class GalleryDLService:
if not written_paths: if not written_paths:
return quarantined_relpaths, failures return quarantined_relpaths, failures
quarantine_root = (
self.images_root / "_quarantine" / artist_slug / platform
)
for path in written_paths: for path in written_paths:
if not is_validatable(path): if not is_validatable(path):
continue continue
@@ -597,14 +519,34 @@ class GalleryDLService:
continue continue
if result.ok: if result.ok:
continue continue
# Shared move+sidecar (file_validator.quarantine_file) — same impl the try:
# native ingester uses, so the layout + provenance sidecar can't drift. quarantine_root.mkdir(parents=True, exist_ok=True)
dest = quarantine_file( dest = quarantine_root / path.name
self.images_root, path, artist_slug, platform, counter = 1
url=url, result=result, while dest.exists():
) dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
if dest is None: counter += 1
continue # move failed → left in place, not counted path.rename(dest)
sidecar = dest.with_suffix(dest.suffix + ".quarantine.json")
sidecar.write_text(
json.dumps(
{
"original_path": str(path),
"source_url": url,
"artist_slug": artist_slug,
"platform": platform,
"format": result.format,
"reason": result.reason,
"size": result.size,
"quarantined_at": datetime.now(UTC).isoformat(),
},
indent=2,
)
)
except OSError as exc:
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
continue
log.warning( log.warning(
"Quarantined corrupt file: %s%s (%s: %s)", "Quarantined corrupt file: %s%s (%s: %s)",
path, dest, result.format, result.reason, path, dest, result.format, result.reason,
@@ -642,24 +584,41 @@ class GalleryDLService:
if "][warning]" in line.lower() and "not allowed to view post" in line.lower() if "][warning]" in line.lower() and "not allowed to view post" in line.lower()
) )
return make_run_stats( return {
exit_code=return_code, "exit_code": return_code,
downloaded_count=self._count_downloaded_files(stdout), "downloaded_count": self._count_downloaded_files(stdout),
skipped_count=skipped_stdout + skipped_stderr, "skipped_count": skipped_stdout + skipped_stderr,
per_item_failures=per_item_failures, "per_item_failures": per_item_failures,
warning_count=warning_count, "warning_count": warning_count,
tier_gated_count=tier_gated_count, "tier_gated_count": tier_gated_count,
) }
@staticmethod @staticmethod
def _extract_errors_warnings(stderr: str) -> str: def _extract_errors_warnings(stderr: str) -> str:
# Thin delegator to the module-level helper (kept for existing callers). if not stderr:
return extract_errors_warnings(stderr) return ""
kept = [
line for line in stderr.splitlines()
if "][error]" in line.lower() or "][warning]" in line.lower()
]
return "\n".join(kept)
@staticmethod @staticmethod
def _truncate_log(text: str, max_bytes: int = 500_000) -> str: def _truncate_log(text: str, max_bytes: int = 500_000) -> str:
# Thin delegator to the module-level helper (kept for existing callers). if not text:
return truncate_log(text, max_bytes) return text
encoded = text.encode("utf-8")
if len(encoded) <= max_bytes:
return text
half = max_bytes // 2
head = encoded[:half].decode("utf-8", errors="ignore")
tail = encoded[-half:].decode("utf-8", errors="ignore")
head_lines = head.count("\n")
tail_lines = tail.count("\n")
total_lines = text.count("\n")
elided = max(0, total_lines - head_lines - tail_lines)
marker = f"\n\n... [{elided} lines elided, {len(encoded) - max_bytes} bytes] ...\n\n"
return head + marker + tail
async def download( async def download(
self, self,
+81 -332
View File
@@ -18,22 +18,15 @@ import base64
from dataclasses import dataclass from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from sqlalchemy import Select, and_, distinct, exists, func, or_, select from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import aliased from sqlalchemy.orm import aliased
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag from ..models import Artist, ImageProvenance, ImageRecord, Post, Tag
from ..models.tag import image_tag from ..models.tag import image_tag
CURSOR_SEPARATOR = "|" CURSOR_SEPARATOR = "|"
# Reserved `platform` filter value selecting images with NO platformed
# provenance (filesystem imports). Returned by facets() as a null-valued
# bucket; the frontend maps that null back to this sentinel in the URL so the
# bucket is selectable. Underscore-wrapped so it can't collide with a real
# gallery-dl platform name (patreon/pixiv/...).
UNSOURCED_PLATFORM = "__unsourced__"
def encode_cursor(effective_date: datetime, image_id: int) -> str: def encode_cursor(effective_date: datetime, image_id: int) -> str:
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}" raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
@@ -50,17 +43,16 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
def _effective_date_col(): def _effective_date_col():
"""The materialized gallery sort key: image_record.effective_date """SQL expression: COALESCE(post.post_date, image_record.created_at).
(alembic 0035) = COALESCE(primary post's post_date, created_at),
maintained at write time by the importer.
Canonical sort/group/filter key across the gallery so images attached Used as the canonical sort/group/filter key across the gallery so
to a post surface at their original publish date, not their FC import images backfilled with primary_post_id (e.g. via tag_apply phase 4)
date — and, now that it's a single indexed column rather than a surface at their original publish date, not their FC import date.
COALESCE across the Post outer join, the cursor scroll is an index Images without a Post (or with Post.post_date NULL) fall back to
range scan instead of a full re-sort per page. image_record.created_at and still order coherently against
post-attached ones.
""" """
return ImageRecord.effective_date return func.coalesce(Post.post_date, ImageRecord.created_at)
def _outer_join_primary_post(stmt: Select) -> Select: def _outer_join_primary_post(stmt: Select) -> Select:
@@ -99,16 +91,6 @@ class TimelineBucket:
count: int count: int
@dataclass(frozen=True)
class GalleryFacets:
total: int # images matching the FULL active filter
platforms: list[dict] # [{"value": str|None, "count": int}], null = unsourced
untagged: int # how many the Untagged flag would isolate
no_artist: int # how many the No-artist flag would isolate
date_min: datetime | None
date_max: datetime | None
def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str: def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str:
"""Return the URL to fetch a thumbnail. """Return the URL to fetch a thumbnail.
@@ -135,85 +117,13 @@ def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str
return f"/images/thumbs/{bucket}/{sha256_hex}{ext}" return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
def _require_single_filter(tag_ids, post_id, artist_id) -> None: def _require_single_filter(tag_id, post_id, artist_id) -> None:
"""post_id is the post-detail view — it can't combine with the if sum(x is not None for x in (tag_id, post_id, artist_id)) > 1:
composable filters. tag_ids + artist_id (+ media_type) compose freely
(AND)."""
if post_id is not None and (tag_ids or artist_id is not None):
raise ValueError( raise ValueError(
"post_id cannot be combined with tag or artist filters" "tag_id, post_id, artist_id are mutually exclusive"
) )
def _apply_scope(
stmt, *, tag_ids, post_id, artist_id, media_type,
platform=None, untagged=False, no_artist=False,
date_from=None, date_to=None,
):
"""Apply the composable gallery filters to a statement.
All clauses are correlated EXISTS / scalar predicates on ImageRecord, so
they AND together without row-multiplication and don't require any join to
be present on `stmt` (the artist/platform paths alias Post/Source inside
their own EXISTS).
- tag_ids: image must carry ALL of them — one correlated EXISTS per tag.
- post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded
by _require_single_filter).
- media_type: 'image' | 'video' narrows by mime prefix.
- platform: EXISTS a provenance→source with that platform; the
UNSOURCED_PLATFORM sentinel inverts it (NO platformed provenance).
- untagged: NOT EXISTS any image_tag row.
- no_artist: ImageRecord.artist_id IS NULL.
- date_from / date_to: half-open [from, to) bounds on effective_date.
"""
for tid in tag_ids or []:
stmt = stmt.where(
exists().where(
image_tag.c.image_record_id == ImageRecord.id,
image_tag.c.tag_id == tid,
)
)
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
if media_type == "image":
stmt = stmt.where(ImageRecord.mime.like("image/%"))
elif media_type == "video":
stmt = stmt.where(ImageRecord.mime.like("video/%"))
if platform is not None:
stmt = stmt.where(_platform_clause(platform))
if untagged:
stmt = stmt.where(
~exists().where(image_tag.c.image_record_id == ImageRecord.id)
)
if no_artist:
stmt = stmt.where(ImageRecord.artist_id.is_(None))
eff = _effective_date_col()
if date_from is not None:
stmt = stmt.where(eff >= date_from)
if date_to is not None:
stmt = stmt.where(eff < date_to)
return stmt
def _platform_clause(platform):
"""Correlated EXISTS on a provenance row whose Source carries `platform`.
The UNSOURCED_PLATFORM sentinel inverts to NOT EXISTS(any sourced
provenance) — i.e. filesystem-imported content with no platform."""
src = aliased(Source)
if platform == UNSOURCED_PLATFORM:
return ~exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == src.id,
)
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == src.id,
src.platform == platform,
)
def _provenance_clause(post_id, artist_id): def _provenance_clause(post_id, artist_id):
"""Correlated EXISTS clause (NOT a join) so an image with multiple """Correlated EXISTS clause (NOT a join) so an image with multiple
matching provenance rows is returned exactly once and the matching provenance rows is returned exactly once and the
@@ -245,27 +155,6 @@ def _provenance_clause(post_id, artist_id):
return None return None
def _gallery_images(rows, artists: dict[int, dict]) -> list[GalleryImage]:
"""Build GalleryImage list from (record, posted_at, eff_date) rows + the
artist hydration map. Shared by scroll() and similar()."""
return [
GalleryImage(
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
artist=artists.get(record.id),
)
for record, posted_at, eff_date in rows
]
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]: async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
"""Map image_id -> {"name","slug"} via the canonical """Map image_id -> {"name","slug"} via the canonical
image_record.artist_id (FC-2d-vii-c). Bounded by page size.""" image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
@@ -290,50 +179,35 @@ class GalleryService:
self, self,
cursor: str | None, cursor: str | None,
limit: int = 50, limit: int = 50,
tag_ids: list[int] | None = None, tag_id: int | None = None,
post_id: int | None = None, post_id: int | None = None,
artist_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None,
sort: str = "newest",
platform: str | None = None,
untagged: bool = False,
no_artist: bool = False,
date_from: datetime | None = None,
date_to: datetime | None = None,
) -> GalleryPage: ) -> GalleryPage:
if limit < 1 or limit > 200: if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200") raise ValueError("limit must be between 1 and 200")
_require_single_filter(tag_ids, post_id, artist_id) _require_single_filter(tag_id, post_id, artist_id)
eff = _effective_date_col() eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff")) stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt) stmt = _outer_join_primary_post(stmt)
stmt = _apply_scope( if tag_id is not None:
stmt, tag_ids=tag_ids, post_id=post_id, stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
artist_id=artist_id, media_type=media_type, image_tag.c.tag_id == tag_id
platform=platform, untagged=untagged, no_artist=no_artist, )
date_from=date_from, date_to=date_to, prov = _provenance_clause(post_id, artist_id)
) if prov is not None:
stmt = stmt.where(prov)
descending = sort != "oldest"
if cursor: if cursor:
cur_ts, cur_id = decode_cursor(cursor) cur_ts, cur_id = decode_cursor(cursor)
# The cursor is just (last eff, last id); the request's sort stmt = stmt.where(
# decides which side of it the next page lies on. or_(
if descending: eff < cur_ts,
stmt = stmt.where( and_(eff == cur_ts, ImageRecord.id < cur_id),
or_(eff < cur_ts, and_(eff == cur_ts, ImageRecord.id < cur_id))
)
else:
stmt = stmt.where(
or_(eff > cur_ts, and_(eff == cur_ts, ImageRecord.id > cur_id))
) )
)
if descending: stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc())
else:
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
stmt = stmt.limit(limit + 1)
rows = (await self.session.execute(stmt)).all() rows = (await self.session.execute(stmt)).all()
next_cursor = None next_cursor = None
@@ -345,7 +219,22 @@ class GalleryService:
artists = await _artists_for( artists = await _artists_for(
self.session, [r[0].id for r in rows] self.session, [r[0].id for r in rows]
) )
images = _gallery_images(rows, artists) images = [
GalleryImage(
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
artist=artists.get(record.id),
)
for record, posted_at, eff_date in rows
]
return GalleryPage( return GalleryPage(
images=images, images=images,
next_cursor=next_cursor, next_cursor=next_cursor,
@@ -354,15 +243,9 @@ class GalleryService:
async def timeline( async def timeline(
self, self,
tag_ids: list[int] | None = None, tag_id: int | None = None,
post_id: int | None = None, post_id: int | None = None,
artist_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None,
platform: str | None = None,
untagged: bool = False,
no_artist: bool = False,
date_from: datetime | None = None,
date_to: datetime | None = None,
) -> list[TimelineBucket]: ) -> list[TimelineBucket]:
eff = _effective_date_col() eff = _effective_date_col()
year_col = func.date_part("year", eff).label("yr") year_col = func.date_part("year", eff).label("yr")
@@ -371,28 +254,25 @@ class GalleryService:
year_col, month_col, func.count(ImageRecord.id).label("cnt") year_col, month_col, func.count(ImageRecord.id).label("cnt")
) )
stmt = _outer_join_primary_post(stmt) stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_ids, post_id, artist_id) _require_single_filter(tag_id, post_id, artist_id)
stmt = _apply_scope( if tag_id is not None:
stmt, tag_ids=tag_ids, post_id=post_id, stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
artist_id=artist_id, media_type=media_type, image_tag.c.tag_id == tag_id
platform=platform, untagged=untagged, no_artist=no_artist, )
date_from=date_from, date_to=date_to, prov = _provenance_clause(post_id, artist_id)
) if prov is not None:
stmt = stmt.where(prov)
stmt = stmt.group_by(year_col, month_col).order_by(year_col.desc(), month_col.desc()) stmt = stmt.group_by(year_col, month_col).order_by(year_col.desc(), month_col.desc())
rows = (await self.session.execute(stmt)).all() rows = (await self.session.execute(stmt)).all()
return [TimelineBucket(year=int(r.yr), month=int(r.mo), count=int(r.cnt)) for r in rows] return [TimelineBucket(year=int(r.yr), month=int(r.mo), count=int(r.cnt)) for r in rows]
async def jump_cursor( async def jump_cursor(
self, year: int, month: int, tag_ids: list[int] | None = None, self, year: int, month: int, tag_id: int | None = None,
post_id: int | None = None, artist_id: int | None = None, post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, sort: str = "newest",
platform: str | None = None, untagged: bool = False,
no_artist: bool = False, date_from: datetime | None = None,
date_to: datetime | None = None,
) -> str | None: ) -> str | None:
"""Returns a cursor that, when passed to scroll() with the same sort, """Returns a cursor that, when passed to scroll(), positions at the
positions at the first image of the given year-month. None if the first image of the given year-month (by effective_date, not
bucket is empty. created_at). None if the bucket is empty.
""" """
from sqlalchemy import extract from sqlalchemy import extract
@@ -402,157 +282,22 @@ class GalleryService:
extract("month", eff) == month, extract("month", eff) == month,
) )
stmt = _outer_join_primary_post(stmt) stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_ids, post_id, artist_id) _require_single_filter(tag_id, post_id, artist_id)
stmt = _apply_scope( if tag_id is not None:
stmt, tag_ids=tag_ids, post_id=post_id, stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
artist_id=artist_id, media_type=media_type, image_tag.c.tag_id == tag_id
platform=platform, untagged=untagged, no_artist=no_artist, )
date_from=date_from, date_to=date_to, prov = _provenance_clause(post_id, artist_id)
) if prov is not None:
descending = sort != "oldest" stmt = stmt.where(prov)
if descending: stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()) first = (await self.session.execute(stmt)).first()
else:
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
first = (await self.session.execute(stmt.limit(1))).first()
if first is None: if first is None:
return None return None
record, eff_date = first record, eff_date = first
# Cursor is exclusive; nudge the id one past the boundary row (in the # Cursor is exclusive; we encode a cursor with id+1 so the row itself
# scan direction) so the row itself is the first result of scroll(). # is the first result in the next scroll().
boundary = record.id + 1 if descending else record.id - 1 return encode_cursor(eff_date, record.id + 1)
return encode_cursor(eff_date, boundary)
async def facets(
self, *, tag_ids: list[int] | None = None,
post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, platform: str | None = None,
untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None,
) -> GalleryFacets:
"""Live facet counts scoped to the current filter. Each facet GROUP is
computed with all OTHER active filters applied but its OWN selection
ignored ("minus-self"), so sibling options stay visible/switchable.
No outer join is needed — every clause is a correlated EXISTS or a
column predicate on ImageRecord.
"""
_require_single_filter(tag_ids, post_id, artist_id)
common = {
"tag_ids": tag_ids, "post_id": post_id,
"artist_id": artist_id, "media_type": media_type,
}
# total — the full active filter (the headline result count).
total = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
)).scalar_one()
# platforms — scope minus the platform selection. Inner-join
# provenance→source and COUNT(DISTINCT image) per platform (a
# cross-posted image counts under each of its platforms).
plat_scope = {
**common, "untagged": untagged, "no_artist": no_artist,
"date_from": date_from, "date_to": date_to,
}
src = aliased(Source)
plat_stmt = (
select(src.platform, func.count(distinct(ImageRecord.id)))
.select_from(ImageRecord)
.join(ImageProvenance, ImageProvenance.image_record_id == ImageRecord.id)
.join(src, src.id == ImageProvenance.source_id)
)
plat_stmt = _apply_scope(plat_stmt, **plat_scope).group_by(src.platform)
platforms = [
{"value": p, "count": c}
for p, c in (await self.session.execute(plat_stmt)).all()
]
# Unsourced (filesystem) bucket — same minus-platform scope.
unsourced = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **plat_scope,
platform=UNSOURCED_PLATFORM,
)
)).scalar_one()
if unsourced:
platforms.append({"value": None, "count": unsourced})
# curation flags — each minus its OWN flag.
untagged_count = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, no_artist=no_artist,
date_from=date_from, date_to=date_to, untagged=True,
)
)).scalar_one()
no_artist_count = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, untagged=untagged,
date_from=date_from, date_to=date_to, no_artist=True,
)
)).scalar_one()
# date bounds — scope minus the date params (those drive the picker).
eff = _effective_date_col()
dmin, dmax = (await self.session.execute(
_apply_scope(
select(func.min(eff), func.max(eff)), **common,
platform=platform, untagged=untagged, no_artist=no_artist,
)
)).one()
return GalleryFacets(
total=total, platforms=platforms,
untagged=untagged_count, no_artist=no_artist_count,
date_min=dmin, date_max=dmax,
)
async def similar(
self, image_id: int, limit: int = 100, *,
tag_ids: list[int] | None = None, artist_id: int | None = None,
media_type: str | None = None, platform: str | None = None,
untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None,
) -> list[GalleryImage] | None:
"""Visual "more like this": images ranked by cosine distance to
`image_id`'s SigLIP embedding (pgvector, HNSW-indexed — alembic 0036).
No ML inference here; the embedding was computed at import.
Returns None if the source image doesn't exist (→ 404), [] if it has
no embedding (a video / not-yet-embedded). Composes with the Phase-1/2
scope filters (AND) but REPLACES the date sort — always nearest-first,
bounded to `limit` (no cursor; distance-ranking has no date cursor).
"""
if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200")
src = await self.session.get(ImageRecord, image_id)
if src is None:
return None
if src.siglip_embedding is None:
return []
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
stmt = stmt.where(
ImageRecord.siglip_embedding.is_not(None),
ImageRecord.id != image_id,
)
stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=None,
artist_id=artist_id, media_type=media_type,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
stmt = stmt.order_by(distance.asc()).limit(limit)
rows = (await self.session.execute(stmt)).all()
artists = await _artists_for(self.session, [r[0].id for r in rows])
return _gallery_images(rows, artists)
async def get_image_with_tags(self, image_id: int) -> dict | None: async def get_image_with_tags(self, image_id: int) -> dict | None:
record = await self.session.get(ImageRecord, image_id) record = await self.session.get(ImageRecord, image_id)
@@ -591,9 +336,6 @@ class GalleryService:
"height": record.height, "height": record.height,
"size_bytes": record.size_bytes, "size_bytes": record.size_bytes,
"integrity_status": record.integrity_status, "integrity_status": record.integrity_status,
# Phase 3: lets the modal hide the "Related"/find-similar surface
# for images that have no embedding yet (videos / pending ML).
"has_embedding": record.siglip_embedding is not None,
"created_at": record.created_at.isoformat(), "created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None, "posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime), "thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
@@ -615,10 +357,17 @@ class GalleryService:
} }
async def _neighbors(self, record: ImageRecord) -> dict: async def _neighbors(self, record: ImageRecord) -> dict:
# The boundary image's sort key is materialized on the row now # Compute the boundary image's effective_date in Python (one query
# (alembic 0035) — read it directly instead of re-deriving COALESCE # below + the SELECT we already have on `record`) and use it for
# via an extra Post lookup. # the neighbor comparison. Cheaper than re-deriving in SQL via
boundary_eff = record.effective_date # correlated subquery.
boundary_eff = record.created_at
if record.primary_post_id is not None:
post_date = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
if post_date is not None:
boundary_eff = post_date
eff = _effective_date_col() eff = _effective_date_col()
prev_stmt = _outer_join_primary_post( prev_stmt = _outer_join_primary_post(
+24 -35
View File
@@ -31,12 +31,7 @@ from ..models import (
Source, Source,
) )
from ..utils import safe_probe from ..utils import safe_probe
from ..utils.paths import ( from ..utils.paths import derive_subdir, derive_top_level_artist, hash_suffixed_name
derive_subdir,
derive_top_level_artist,
hash_suffixed_name,
safe_ext,
)
from ..utils.phash import compute_phash, find_similar from ..utils.phash import compute_phash, find_similar
from ..utils.sidecar import find_sidecar, parse_sidecar from ..utils.sidecar import find_sidecar, parse_sidecar
from ..utils.slug import slugify from ..utils.slug import slugify
@@ -87,9 +82,27 @@ def is_video(path: Path) -> bool:
def _safe_ext(path: Path) -> str: def _safe_ext(path: Path) -> str:
"""Conservatively extract a file extension for PostAttachment.ext """Conservatively extract a file extension for PostAttachment.ext
(varchar(32)). Thin wrapper over the shared `utils.paths.safe_ext` (kept for (varchar(32)).
the Path-typed call sites + the [[path_suffix_sanitize]] memory pointer)."""
return safe_ext(path) gallery-dl produces some filenames with URL-encoded query-string
artifacts embedded into the basename (e.g.
`79507046_media_..._https___www.patreon.com_media-u_Z0FBQUFBQm5q...`).
`Path.suffix` finds the LAST dot and returns everything after, which
in those cases yields a 50+ char "extension" of mostly base64-ish
junk. That blows the column. Operator-flagged 2026-05-25.
Real extensions are short and alphanumeric. We accept anything ≤ 16
chars where every post-dot character is alphanumeric; anything else
means the input wasn't a real extension and we return the empty
string. ext is nullable-ish (empty string still satisfies NOT NULL)
and consumers should treat "" as "no known extension".
"""
suffix = path.suffix.lower()
if not suffix or len(suffix) > 16:
return ""
if not all(c.isalnum() for c in suffix[1:]):
return ""
return suffix
def _mime_for(path: Path) -> str: def _mime_for(path: Path) -> str:
@@ -430,17 +443,13 @@ class Importer:
self._capture_attachment( self._capture_attachment(
source, post=post, artist=artist_use, resolved=True, source, post=post, artist=artist_use, resolved=True,
) )
reason = f"archive probe rejected, captured unextracted: {probe.reason}" return ImportResult(status="attached")
log.warning("%s: %s", source.name, reason)
return ImportResult(status="attached", error=reason)
artist_use = artist if artist is not None else self._resolve_artist(source) artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use) post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = [] member_ids: list[int] = []
member_total = 0
with extract_archive(source) as members: with extract_archive(source) as members:
for _name, member_path in members: for _name, member_path in members:
member_total += 1
if not is_supported(member_path): if not is_supported(member_path):
continue # non-media preserved via the stored archive continue # non-media preserved via the stored archive
res = self._import_media( res = self._import_media(
@@ -457,19 +466,7 @@ class Importer:
status="imported", image_id=member_ids[0], status="imported", image_id=member_ids[0],
member_image_ids=member_ids, member_image_ids=member_ids,
) )
# No images landed — surface WHY so a post showing "no images" beside an return ImportResult(status="attached")
# archive is diagnosable instead of silent. Zero members usually means
# the extractor backend is missing/failed (unar for rar, py7zr for 7z)
# or the file is corrupt; non-zero-but-no-images means it held only
# non-media files.
reason = (
"archive extracted but held no supported image/video members"
if member_total
else "archive yielded no members (unsupported/corrupt, or the "
"extractor backend failed)"
)
log.warning("%s: %s", source.name, reason)
return ImportResult(status="attached", error=reason)
def _import_media( def _import_media(
self, source: Path, attribution_path: Path, self, source: Path, attribution_path: Path,
@@ -963,14 +960,6 @@ class Importer:
sp.rollback() sp.rollback()
if record.primary_post_id is None: if record.primary_post_id is None:
record.primary_post_id = post.id record.primary_post_id = post.id
# Keep the denormalized gallery sort key (alembic 0035) aligned with
# the primary post's publish date so /scroll orders off
# ix_image_record_effective_date instead of COALESCE-ing across the
# post join. Only override when THIS post is the primary AND carries
# a date; otherwise the column keeps its created_at-equivalent server
# default (matches the old COALESCE(post_date, created_at) fallback).
if record.primary_post_id == post.id and post.post_date is not None:
record.effective_date = post.post_date
self.session.flush() self.session.flush()
def _copy_to_library( def _copy_to_library(
-647
View File
@@ -1,647 +0,0 @@
"""Platform-agnostic native-ingest core (plan #706, build on #697/#703/#704/#705).
The orchestration that drives a native subscription walk — page a feed →
extract media → tiered skip (seen-ledger / on-disk / dead-letter) → download →
mark-seen / record-failures / checkpoint-cursor → return a gallery-dl-shaped
`DownloadResult`, across tick/backfill/recovery modes — is identical for every
platform. Only four things are platform-specific, and they're INJECTED at
construction by a thin adapter (e.g. `PatreonIngester`):
- `client` — `.iter_posts(feed_id, cursor)` yielding `(post, included,
page_cursor)` + `.extract_media(post, included) -> [media]`.
- `downloader`— `.download_post(post, media, artist_slug, is_seen,
should_stop) -> [MediaOutcome]` (status in downloaded/
skipped_seen/skipped_disk/quarantined/error;
`.path`/`.error`/`.post_id`). `should_stop()` is polled
between media so the time-box is honoured mid-post.
- ledger — `seen_model` + `failed_model` SQLAlchemy models (+ their
on-conflict UNIQUE constraint names) and a `ledger_key(media)`.
- failure map — the adapter overrides `_failure_result` (platform exception
→ DownloadResult.error_type) and supplies `error_base` (the
exception type the walk catches) + `platform` (result label).
Everything DB touches a SHORT-LIVED sync session from the injected sessionmaker —
never held across a network fetch ([[db-connection-held-across-subprocess]]).
Plain-HTTP homelab: no secure-context Web API.
"""
from __future__ import annotations
import json
import logging
import time
from collections.abc import Callable
from sqlalchemy import delete, func, select, text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from .gallery_dl import DownloadResult, ErrorType, make_run_stats
log = logging.getLogger(__name__)
# Stop a tick after this many CONTIGUOUS already-have-it media (seen-ledger or
# on-disk) — the cheap native equivalent of gallery-dl's `exit:20`, now free of
# per-file HEADs. Headroom against paywalled/undownloadable items interleaving.
_TICK_SEEN_THRESHOLD = 20
# plan #705 #7: after this many failed download/validate attempts a media is
# "dead-lettered" and skipped on routine tick/backfill walks (recovery still
# re-attempts it). Stops a permanently-broken media re-erroring forever.
DEAD_LETTER_THRESHOLD = 3
# last_error is Text but bound it so a giant traceback doesn't bloat the row.
_ERROR_MAX = 1000
# plan #709: throttle the live-progress write to the running DownloadEvent to one
# every ~5s — a steady cadence for the Downloads view regardless of how big/slow a
# page is (page boundaries can be minutes apart on image-dense backfills, so a
# page-tied update would lurch). Trivial churn (~one single-row UPDATE / 5s).
_LIVE_PROGRESS_INTERVAL = 5.0
class Ingester:
"""Generic native-ingest orchestration. Subclass with a platform adapter
(see the module docstring) — or construct directly with the keyword seams."""
def __init__(
self,
*,
client,
downloader,
session_factory: Callable[[], object],
seen_model,
failed_model,
seen_constraint: str,
failed_constraint: str,
ledger_key: Callable[[object], str],
platform: str,
error_base: type[Exception],
):
self.client = client
self.downloader = downloader
self.session_factory = session_factory
self._seen_model = seen_model
self._failed_model = failed_model
self._seen_constraint = seen_constraint
self._failed_constraint = failed_constraint
self._ledger_key = ledger_key
self._platform = platform
self._error_base = error_base
# -- public ------------------------------------------------------------
def run(
self,
*,
source_id: int,
campaign_id: str,
artist_slug: str,
url: str,
mode: str,
resume_cursor: str | None = None,
time_budget_seconds: float = 870.0,
seen_threshold: int = _TICK_SEEN_THRESHOLD,
posts_base: int = 0,
event_id: int | None = None,
) -> DownloadResult:
"""Walk + download for one source, returning a gallery-dl-shaped result.
`mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
seen-ledger AND the dead-letter ledger (tier-2 disk still skips kept
files). The walk stops on:
- budget exhaustion (time_budget_seconds) → TIMEOUT / PARTIAL
- tick early-out (seen_threshold contiguous seen) → success
- reaching the bottom of the feed → success (rc 0)
A client-level failure (drift / auth / network) fails the whole run loud.
"""
bypass_seen = mode == "recovery"
# Only deep walks checkpoint their cursor mid-flight (plan #705 #6); a
# tick has no resumable backfill state.
checkpoint = mode in ("backfill", "recovery")
ledger_key = self._ledger_key
start = time.monotonic()
last_live = start # plan #709: last live-progress write timestamp
log_lines: list[str] = []
written: list[str] = []
quarantined_paths: list[str] = []
downloaded = 0
errors = 0
quarantined = 0
dead_lettered = 0
skipped_count = 0
posts_processed = 0
# Net-new posts THIS chunk for the live progress badge (plan #704 #5);
# excludes the re-walked resume page so _backfill_posts stays a monotonic
# absolute across chunks instead of an inflating sum. posts_processed
# stays the gross per-chunk count used for the run summary.
chunk_new_posts = 0
consecutive_seen = 0
emitted_cursor: str | None = None
reached_bottom = False
budget_hit = False
early_out = False
stopped = False # plan #708 B4: operator hit Stop mid-walk
cancel_armed = False # latched once we observe a live "running" state
def _result(
*, success: bool, return_code: int,
error_type: ErrorType | None, error_message: str | None,
) -> DownloadResult:
# plan #704: return STRUCTURED data — phase 3 reads run_stats/cursor
# directly instead of regex-scraping a reconstructed stdout. stdout
# stays a human-readable summary (no fake `Cursor:` lines).
return DownloadResult(
success=success,
url=url,
artist_slug=artist_slug,
platform=self._platform,
files_downloaded=downloaded,
files_quarantined=quarantined,
quarantined_paths=list(quarantined_paths),
written_paths=written,
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
error_type=error_type,
error_message=error_message,
duration_seconds=time.monotonic() - start,
cursor=emitted_cursor,
posts_processed=posts_processed,
run_stats=make_run_stats(
exit_code=return_code,
downloaded_count=downloaded,
skipped_count=skipped_count,
per_item_failures=errors,
quarantined_count=quarantined,
dead_lettered_count=dead_lettered,
),
)
try:
for post, included, page_cursor in self.client.iter_posts(
campaign_id, cursor=resume_cursor
):
# Checkpoint the cursor that FETCHED this page the moment we
# START it — so a chunk cut mid-page resumes the page, not the one
# after it. Carried as DownloadResult.cursor (plan #704).
if page_cursor and page_cursor != emitted_cursor:
emitted_cursor = page_cursor
# plan #705 #6: persist the cursor at each page boundary so a
# worker SIGKILL mid-chunk resumes near the crash, not the
# chunk start. (phase 3 still writes the final cursor — same
# value; this is the crash-safety net.) plan #704 #5: persist
# the live posts count alongside it so the badge climbs DURING
# the chunk, not only when it ends.
if checkpoint:
# plan #708 B4: an operator Stop pops `_backfill_state` —
# bail at the page boundary (progress already checkpointed)
# before more network work, so the live chunk halts
# promptly instead of running to its time-box. LATCH on the
# first observed "running" state, so a run invoked WITHOUT a
# running state (a unit test, or a stale call) never
# spuriously self-cancels. A short SELECT, never held.
if self._still_running(source_id):
cancel_armed = True
elif cancel_armed:
stopped = True
break
self._checkpoint_cursor(source_id, emitted_cursor)
self._checkpoint_posts(source_id, posts_base + chunk_new_posts)
# Time-box check at the post boundary (coarse, like a gallery-dl
# chunk). Backfill/recovery resume from emitted_cursor next chunk.
if time.monotonic() - start >= time_budget_seconds:
budget_hit = True
break
posts_processed += 1
# The resume page (its cursor == resume_cursor) was already
# counted by the chunk that checkpointed it — don't re-count it
# into the persisted badge (plan #704 #5). First chunk has
# resume_cursor None, so everything counts.
if not (resume_cursor and page_cursor == resume_cursor):
chunk_new_posts += 1
media = self.client.extract_media(post, included)
if not media:
continue
keys = [ledger_key(m) for m in media]
# Recovery bypasses BOTH the seen-ledger AND the dead-letter
# ledger (the operator's "try everything again"); routine walks
# skip seen + dead media (tier-1 + tier-1.5, plan #705 #7).
dead = set() if bypass_seen else self._dead_keys(source_id, keys)
seen = (
set()
if bypass_seen
else self._seen_keys(source_id, keys)
)
skip = seen | dead
def _is_skip(m, _skip=skip) -> bool:
return ledger_key(m) in _skip
# Honour the time-box DURING a media-dense post too, not only at
# the per-post boundary below — else one heavy post can blow the
# chunk budget out to the Celery soft limit (Pocketacer, 2026-06-07).
outcomes = self.downloader.download_post(
post, media, artist_slug, is_seen=_is_skip,
should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
)
to_mark: list[tuple[str, str]] = []
to_clear: list[str] = [] # recovered → drop any dead-letter row
to_fail: list[tuple[str, str, str]] = [] # (key, post_id, error)
for media_item, outcome in zip(media, outcomes, strict=False):
key = ledger_key(media_item)
if key in dead:
dead_lettered += 1 # skipped because previously dead
if outcome.status == "downloaded":
downloaded += 1
if outcome.path is not None:
written.append(str(outcome.path))
to_mark.append((key, media_item.post_id))
to_clear.append(key)
consecutive_seen = 0
elif outcome.status == "skipped_disk":
# Already on disk (a prior run). Reconcile the ledger so a
# later tick skips it at tier-1 without a disk stat, but
# do NOT re-feed it to phase 3 — attach_in_place would see
# the duplicate sha256 and unlink the on-disk copy.
to_mark.append((key, media_item.post_id))
to_clear.append(key)
skipped_count += 1
consecutive_seen += 1
elif outcome.status == "skipped_seen":
skipped_count += 1
consecutive_seen += 1
elif outcome.status == "quarantined":
# New content that failed validation (corrupt) — counted
# distinctly so the run surfaces a real quarantined total.
# Not marked seen (a later walk may re-fetch a fixed file);
# it IS new content, so it breaks the run-of-seen. Counts
# toward the dead-letter ledger (plan #705 #7).
quarantined += 1
if outcome.path is not None:
quarantined_paths.append(str(outcome.path))
to_fail.append((key, media_item.post_id, outcome.error or "quarantined"))
consecutive_seen = 0
elif outcome.status == "error":
errors += 1
to_fail.append((key, media_item.post_id, outcome.error or "error"))
# An error neither advances nor resets the run-of-seen.
if mode == "tick" and consecutive_seen >= seen_threshold:
early_out = True
break
# Persist ledger changes AFTER the network fetch, on short
# sessions: mark downloaded/on-disk seen, clear any dead-letter
# for recovered media, and record failures (plan #705 #7).
if to_mark:
self._mark_seen(source_id, to_mark)
if to_clear:
self._clear_failures(source_id, to_clear)
if to_fail:
self._record_failures(source_id, to_fail)
# plan #709: time-throttled live progress to the running event so
# the Downloads view ticks ~every 5s, independent of page size.
now = time.monotonic()
if event_id is not None and (now - last_live) >= _LIVE_PROGRESS_INTERVAL:
last_live = now
self._write_live_progress(event_id, {
"downloaded": downloaded,
"skipped": skipped_count,
"errors": errors,
"quarantined": quarantined,
"posts": posts_processed,
})
if early_out:
break
else:
reached_bottom = True
except self._error_base as exc:
# The platform's client-error base — _failure_result (adapter)
# maps it to a typed error.
return self._failure_result(exc, _result)
# plan #708 B4: a Stop already popped the backfill state (incl. cursor +
# posts), so don't re-write them — return PARTIAL (reads as "ok/progress",
# the lifecycle no-ops since state is gone) instead of a false "complete".
if stopped:
return _result(
success=False, return_code=-1,
error_type=ErrorType.PARTIAL,
error_message=f"Stopped by operator: {downloaded} file(s) this chunk",
)
# Final authoritative posts count for the badge — captures the last page
# after the last boundary write and the time-box break (plan #704 #5).
if checkpoint:
self._checkpoint_posts(source_id, posts_base + chunk_new_posts)
if errors:
log_lines.append(f"{errors} media item(s) failed")
if quarantined:
log_lines.append(f"{quarantined} media item(s) quarantined (invalid)")
if dead_lettered:
log_lines.append(f"{dead_lettered} media item(s) skipped (dead-lettered)")
log_lines.append(
f"{self._platform} ingest ({mode}): {downloaded} downloaded, "
f"{skipped_count} skipped, {quarantined} quarantined, "
f"{dead_lettered} dead-lettered, {errors} error(s), "
f"{posts_processed} post(s)"
+ (", reached end" if reached_bottom else "")
+ (", time-boxed" if budget_hit else "")
)
if budget_hit:
# A chunk that hit its time-box but made forward progress is a
# NORMAL chunk boundary, not a failure (PARTIAL → status "ok"); the
# next chunk resumes from the emitted cursor. No progress → TIMEOUT,
# which feeds download_service's backfill stall-guard. rc<0 mirrors
# subprocess TimeoutExpired so completion detection stays false.
made_progress = downloaded > 0 or emitted_cursor != resume_cursor
if made_progress:
return _result(
success=False, return_code=-1,
error_type=ErrorType.PARTIAL,
error_message=(
f"Backfill chunk: {downloaded} file(s) — continuing"
),
)
return _result(
success=False, return_code=-1,
error_type=ErrorType.TIMEOUT,
error_message="Chunk timed out with no progress",
)
# Normal success: reached the bottom, or a tick that early-outed. rc 0 +
# error_type None is REQUIRED for a backfill/recovery walk that reached
# the bottom to be marked COMPLETE by
# download_service._apply_backfill_lifecycle — so we return None even
# when downloaded == 0 (a re-confirming walk that found nothing new still
# completed). success=True maps to status "ok" regardless. A tick that
# early-outed also returns here; ticks never set backfill state so the
# lifecycle is a no-op for them.
return _result(
success=True, return_code=0,
error_type=None, error_message=None,
)
# -- preview (dry-run) -------------------------------------------------
def preview(
self,
source_id: int,
campaign_id: str,
*,
page_limit: int = 3,
sample_size: int = 10,
) -> dict:
"""Dry-run (plan #708 B4): walk up to `page_limit` pages and count media
NOT already in the seen/dead ledgers, WITHOUT downloading anything.
Read-only — only the seen/dead SELECTs touch the DB (short sessions). Lets
an operator gauge "is this source worth a backfill?" cheaply. Returns:
{total_new, posts_scanned, pages_scanned, has_more,
sample: [{title, date, new}, ...]} # sample = posts with new media
A client-level failure (auth/drift) propagates to the caller.
"""
total_new = 0
posts_scanned = 0
pages_scanned = 0
has_more = False
sample: list[dict] = []
unset = object()
last_page: object = unset
for post, included, page_cursor in self.client.iter_posts(
campaign_id, cursor=None
):
if page_cursor != last_page:
last_page = page_cursor
pages_scanned += 1
if pages_scanned > page_limit:
has_more = True
pages_scanned = page_limit
break
posts_scanned += 1
media = self.client.extract_media(post, included)
if not media:
continue
keys = [self._ledger_key(m) for m in media]
skip = self._seen_keys(source_id, keys) | self._dead_keys(source_id, keys)
new_count = sum(1 for m in media if self._ledger_key(m) not in skip)
total_new += new_count
if new_count > 0 and len(sample) < sample_size:
meta = self.client.post_meta(post)
sample.append(
{
"title": meta.get("title") or "(untitled)",
"date": meta.get("date"),
"new": new_count,
}
)
return {
"total_new": total_new,
"posts_scanned": posts_scanned,
"pages_scanned": pages_scanned,
"has_more": has_more,
"sample": sample,
}
# -- failure mapping (adapter overrides) -------------------------------
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
"""Map a platform client-error to a typed failed DownloadResult. The base
gives a safe default; adapters override with their exception taxonomy."""
log.warning("%s ingest failed: %s", self._platform, exc)
return _result(
success=False, return_code=1,
error_type=ErrorType.UNKNOWN_ERROR, error_message=str(exc),
)
# -- seen-ledger (short-lived sessions) --------------------------------
def _seen_keys(self, source_id: int, keys: list[str]) -> set[str]:
"""Which of `keys` are already in the seen-ledger for this source.
One short SELECT on its own session — opened and closed without any
network in between (the GETs happen after, in download_post).
"""
if not keys:
return set()
with self.session_factory() as session:
rows = session.execute(
select(self._seen_model.filehash).where(
self._seen_model.source_id == source_id,
self._seen_model.filehash.in_(keys),
)
).scalars().all()
return set(rows)
def _checkpoint_cursor(self, source_id: int, cursor: str) -> None:
"""Persist the in-progress backfill cursor mid-walk (plan #705 #6).
ATOMIC, single-key UPDATE: cast the JSON column to jsonb, set just
`_backfill_cursor`, cast back — so it never clobbers operator config or
the other backfill keys (no read-modify-write race). The in-flight guard
means only this source's one download runs at a time; a concurrent
operator stop is benign (a stray cursor with no `_backfill_state` is
ignored by tick mode and cleared on the next start).
"""
with self.session_factory() as session:
session.execute(
text(
"UPDATE source SET config_overrides = jsonb_set("
" coalesce(config_overrides::jsonb, '{}'::jsonb),"
" '{_backfill_cursor}', to_jsonb(cast(:cur AS text))"
")::json WHERE id = :sid"
),
{"cur": cursor, "sid": source_id},
)
session.commit()
def _write_live_progress(self, event_id: int, counts: dict) -> None:
"""Throttled mid-walk write of live counts to the RUNNING download_event
(plan #709) so the Downloads view shows progress before the chunk
finishes. A short session (never held across the walk); the `status =
'running'` guard avoids clobbering an event phase 3 already finalized.
`metadata` is JSONB — jsonb_set sets just the `live` key, leaving the rest
for phase 3 to overwrite with the final run_stats."""
with self.session_factory() as session:
session.execute(
text(
"UPDATE download_event SET metadata = jsonb_set("
" coalesce(metadata, '{}'::jsonb), '{live}',"
" cast(:live AS jsonb)) "
"WHERE id = :eid AND status = 'running'"
),
{"live": json.dumps(counts), "eid": event_id},
)
session.commit()
def _still_running(self, source_id: int) -> bool:
"""True while the source is armed for a deep walk (plan #708 B4).
An operator Stop (`source_service.stop_backfill`) pops `_backfill_state`,
so a False here means "cancel this chunk now". One short SELECT on its own
session — never held across the walk
([[db-connection-held-across-subprocess]])."""
with self.session_factory() as session:
state = session.execute(
text(
"SELECT config_overrides::jsonb ->> '_backfill_state' "
"FROM source WHERE id = :sid"
),
{"sid": source_id},
).scalar_one_or_none()
return state == "running"
def _checkpoint_posts(self, source_id: int, posts: int) -> None:
"""Persist the live backfill posts-processed count mid-walk (plan #704 #5).
Same atomic single-key jsonb_set dance as _checkpoint_cursor, on the
`_backfill_posts` key (cast to a JSON number) — so the progress badge
climbs DURING a chunk without clobbering operator config or the cursor.
The ingester OWNS this key now; download_service no longer accumulates it
post-chunk (which lagged a whole chunk and over-counted the resume page).
"""
with self.session_factory() as session:
session.execute(
text(
"UPDATE source SET config_overrides = jsonb_set("
" coalesce(config_overrides::jsonb, '{}'::jsonb),"
" '{_backfill_posts}', to_jsonb(cast(:posts AS int))"
")::json WHERE id = :sid"
),
{"posts": posts, "sid": source_id},
)
session.commit()
def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None:
"""Idempotent upsert of (filehash, post_id) seen-ledger rows for a page.
ON CONFLICT DO NOTHING against the (source_id, filehash) UNIQUE so a
re-sighting — or a concurrent walk — is a harmless no-op
([[scalar_one_or_none-duplicates]]: never check-then-insert without the
DB constraint backing it). De-dup the batch locally first so a single
page can't present the same key twice to one INSERT.
"""
seen_local: set[str] = set()
values = []
for key, post_id in items:
if key in seen_local:
continue
seen_local.add(key)
values.append(
{"source_id": source_id, "filehash": key, "post_id": post_id}
)
if not values:
return
with self.session_factory() as session:
stmt = pg_insert(self._seen_model).values(values)
stmt = stmt.on_conflict_do_nothing(constraint=self._seen_constraint)
session.execute(stmt)
session.commit()
# -- dead-letter ledger (plan #705 #7) ---------------------------------
def _dead_keys(self, source_id: int, keys: list[str]) -> set[str]:
"""Which of `keys` have failed >= DEAD_LETTER_THRESHOLD times (dead).
One short SELECT; recovery never calls this (it re-attempts dead media)."""
if not keys:
return set()
with self.session_factory() as session:
rows = session.execute(
select(self._failed_model.filehash).where(
self._failed_model.source_id == source_id,
self._failed_model.filehash.in_(keys),
self._failed_model.attempts >= DEAD_LETTER_THRESHOLD,
)
).scalars().all()
return set(rows)
def _record_failures(
self, source_id: int, items: list[tuple[str, str, str]]
) -> None:
"""Upsert-increment the dead-letter ledger for failed media. On conflict
bump `attempts` and refresh last_error/last_failed_at (UNIQUE backs the
upsert — no check-then-insert). De-dup the batch (one row/key, last error
wins)."""
by_key: dict[str, str] = {}
for key, _post_id, err in items:
by_key[key] = (err or "")[:_ERROR_MAX]
if not by_key:
return
values = [
{"source_id": source_id, "filehash": k, "attempts": 1, "last_error": e}
for k, e in by_key.items()
]
with self.session_factory() as session:
stmt = pg_insert(self._failed_model).values(values)
stmt = stmt.on_conflict_do_update(
constraint=self._failed_constraint,
set_={
"attempts": self._failed_model.attempts + 1,
"last_error": stmt.excluded.last_error,
"last_failed_at": func.now(),
},
)
session.execute(stmt)
session.commit()
def _clear_failures(self, source_id: int, keys: list[str]) -> None:
"""Drop dead-letter rows for media that just downloaded cleanly — they
recovered. A no-op DELETE for keys that were never failing."""
unique = list(dict.fromkeys(keys))
if not unique:
return
with self.session_factory() as session:
session.execute(
delete(self._failed_model).where(
self._failed_model.source_id == source_id,
self._failed_model.filehash.in_(unique),
)
)
session.commit()
-604
View File
@@ -1,604 +0,0 @@
"""Native Patreon JSON:API client (build step 1 of the native ingester).
Clean-room reimplementation of the Patreon `/api/posts` read path. This is a
plain, synchronous client over `requests` — the orchestrator already wraps
sync importer calls, so nothing here needs to be async (mirrors
patreon_resolver.py, which wraps its sync lookup in run_in_executor at the
call site).
Scope (build step 1): fetch + page + parse only. This module is NOT wired into
download_service yet — that is a later step. The public surface here exists so
the later step can drive it:
- PatreonClient(cookies_path).iter_posts(campaign_id)
→ (post, included_index, page_cursor)
- extract_media(post, included_index) → list[MediaItem]
- parse_cursor_from_url(url) → cursor
Drift detection is loud on purpose: Patreon ships JSON:API and the shapes we
depend on (top-level `data`, media resources carrying `file_name`/`url`) are
the contract. If a response comes back as an HTML login page or a media
resource is missing the fields we resolve against, we raise PatreonDriftError
rather than silently yielding empty media — so the later import step surfaces
"Patreon changed something" instead of "creator has no posts".
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import http.cookiejar
import logging
import os
import re
import time
from collections.abc import Iterator
from dataclasses import dataclass
from html import unescape
from pathlib import Path
from urllib.parse import parse_qs, urlsplit
import requests
from ..utils.paths import safe_ext
log = logging.getLogger(__name__)
_POSTS_URL = "https://www.patreon.com/api/posts"
_USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
)
_TIMEOUT_SECONDS = 30.0
# 429 backoff (plan #703): ride out a transient API rate-limit instead of
# failing the whole walk (which would stamp RATE_LIMITED → platform-wide
# cooldown → every Patreon source dark). Honor the server's `Retry-After`;
# otherwise exponential base·2^(n-1), capped. Only after the retries are
# exhausted does the 429 propagate as terminal RATE_LIMITED.
_MAX_429_RETRIES = 3
_BACKOFF_BASE_SECONDS = 2.0
_BACKOFF_CAP_SECONDS = 30.0
def _retry_after_seconds(
resp: requests.Response,
attempt: int,
*,
base: float = _BACKOFF_BASE_SECONDS,
cap: float = _BACKOFF_CAP_SECONDS,
) -> float:
"""Backoff delay for a 429: the `Retry-After` seconds header if present and
numeric, else exponential `base·2^(attempt-1)`, both capped. (HTTP-date form
of Retry-After is rare here and falls through to exponential.)"""
header = resp.headers.get("Retry-After")
if header:
try:
return min(float(header), cap)
except (TypeError, ValueError):
pass
return min(base * (2 ** max(0, attempt - 1)), cap)
# JSON:API request contract (observed from real traffic — see module plan).
_INCLUDE = (
"campaign,access_rules,attachments,attachments_media,audio,images,media,"
"native_video_insights,user,user_defined_tags,ti_checks"
)
_FIELDS_POST = (
"content,post_file,image,post_type,published_at,title,url,patreon_url,"
"current_user_can_view"
)
_FIELDS_MEDIA = "id,image_urls,download_url,metadata,file_name"
_FIELDS_CAMPAIGN = "name,url"
# A CDN download URL embeds a 32-char hex (MD5) path segment; that segment is
# Patreon's stable per-file identity and is what we dedup + ledger against.
# Same role gallery-dl's _filehash plays. Match the FIRST 32-hex run anywhere
# in the URL (path or query); real Patreon CDN URLs carry exactly one.
_FILEHASH_RE = re.compile(r"([0-9a-fA-F]{32})")
# Inline post `content` is HTML; images are emitted as <img ... src="...">.
# Pull every src; downstream dedup collapses any that duplicate a gallery item
# by filehash. Tolerant of attribute ordering and single/double quotes.
_CONTENT_IMG_RE = re.compile(r"<img\b[^>]*?\bsrc=[\"']([^\"']+)[\"']", re.IGNORECASE)
class PatreonAPIError(Exception):
"""Base for native Patreon client failures.
`status_code` carries the HTTP status when the failure was an HTTP response
(None for transport-level / parse failures), so the ingester can map it to a
DownloadResult.error_type (429 → rate_limited, 404 → not_found, …).
`retry_after` carries the server's `Retry-After` seconds on a terminal 429, so
the platform cooldown can match the server's hint instead of a flat default
(plan #708 B1).
"""
def __init__(
self,
message: str,
*,
status_code: int | None = None,
retry_after: float | None = None,
):
super().__init__(message)
self.status_code = status_code
self.retry_after = retry_after
class PatreonAuthError(PatreonAPIError):
"""Authentication / authorization failure — missing or expired session
cookies, an insufficient pledge tier, or an HTML login/challenge page served
where JSON was expected. DISTINCT from drift: the fix is rotating the
credential, not updating the ingester. Maps to error_type 'auth_error'.
"""
class PatreonDriftError(PatreonAPIError):
"""A JSON response did not match the JSON:API shape we depend on.
Raised for: a missing top-level `data` list, `data` not a list, or a media
resource lacking the `file_name`/`url` fields we resolve against. Fail loud
so the import step flags API drift (ingester needs update) instead of
silently importing nothing. An HTML-login / non-JSON body is auth, not
drift — that raises PatreonAuthError.
"""
@dataclass
class MediaItem:
"""One resolved downloadable item belonging to a post.
Fields:
url — the CDN/download URL to fetch.
filename — media `file_name` when present; otherwise the URL basename
(NEVER a network call). Bounded/sane extension.
kind — one of: "images", "image_large", "attachments", "postfile",
"content". Mirrors gallery-dl's `files` content-type names so
the later step can honor the same per-source content_types.
filehash — the 32-char hex (MD5) segment from the CDN URL, or None if
the URL carries no such segment. Used for in-post dedup and
the cross-run seen-ledger.
post_id — the owning post's id (so a flattened media list stays
traceable to its post).
"""
url: str
filename: str
kind: str
filehash: str | None
post_id: str
def _load_session(cookies_path: str | Path | None) -> requests.Session:
session = requests.Session()
session.headers.update(
{
"User-Agent": _USER_AGENT,
"Accept": "application/vnd.api+json",
}
)
if cookies_path and os.path.isfile(str(cookies_path)):
try:
jar = http.cookiejar.MozillaCookieJar(str(cookies_path))
jar.load(ignore_discard=True, ignore_expires=True)
session.cookies = jar # type: ignore[assignment]
except (OSError, http.cookiejar.LoadError) as exc:
log.warning("Could not load Patreon cookies from %s: %s", cookies_path, exc)
return session
def _filehash(url: str) -> str | None:
if not url:
return None
match = _FILEHASH_RE.search(url)
return match.group(1).lower() if match else None
def _basename_from_url(url: str) -> str:
"""Derive a sane filename from a URL when the media has no file_name.
Strips query/fragment, takes the path basename, and drops a junk
extension (the importer._safe_ext gotcha) so we never write base64 noise
as a name. Falls back to the filehash, then to "file".
"""
path = urlsplit(url).path
base = os.path.basename(path)
if base:
ext = safe_ext(base)
stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
# Keep the stem bounded; URL-encoded stems can be enormous.
stem = stem[:120] or "file"
return f"{stem}{ext}"
fh = _filehash(url)
return fh or "file"
def parse_cursor_from_url(url: str | None) -> str | None:
"""Extract the `page[cursor]` query param from a links.next URL."""
if not url:
return None
query = urlsplit(url).query
values = parse_qs(query).get("page[cursor]")
if values and values[0]:
return values[0]
return None
class PatreonClient:
"""Synchronous Patreon JSON:API read client.
Construct with a path to a Netscape cookies.txt (the same file
CredentialService.get_cookies_path materializes). Cookies are loaded into a
requests.Session; no secure-context APIs are used.
"""
def __init__(
self,
cookies_path: str | Path | None,
*,
request_sleep: float = 0.0,
max_retries: int = _MAX_429_RETRIES,
):
self.cookies_path = str(cookies_path) if cookies_path else None
self._session = _load_session(cookies_path)
# Politeness: seconds to sleep before each /api/posts page fetch (paces
# the rate-limited API endpoint). 0 = no pacing. plan #703.
self._request_sleep = request_sleep or 0.0
self._max_retries = max_retries
# -- request -----------------------------------------------------------
def _params(self, campaign_id: str, cursor: str | None) -> dict[str, str]:
params = {
"include": _INCLUDE,
"fields[post]": _FIELDS_POST,
"fields[media]": _FIELDS_MEDIA,
"fields[campaign]": _FIELDS_CAMPAIGN,
"filter[campaign_id]": campaign_id,
"filter[contains_exclusive_posts]": "true",
"filter[is_draft]": "false",
"sort": "-published_at",
"json-api-version": "1.0",
}
if cursor:
params["page[cursor]"] = cursor
return params
def _fetch(self, campaign_id: str, cursor: str | None) -> dict:
if self._request_sleep > 0:
time.sleep(self._request_sleep) # pace the API endpoint
attempt = 0
while True:
try:
resp = self._session.get(
_POSTS_URL,
params=self._params(campaign_id, cursor),
timeout=_TIMEOUT_SECONDS,
)
except requests.RequestException as exc:
raise PatreonAPIError(
f"Patreon posts request failed (campaign_id={campaign_id}): {exc}"
) from exc
# Transient rate-limit: back off and retry rather than failing the
# whole walk. Only a PERSISTENT 429 (retries exhausted) falls
# through to the terminal RATE_LIMITED raise below.
if resp.status_code == 429 and attempt < self._max_retries:
attempt += 1
delay = _retry_after_seconds(resp, attempt)
log.warning(
"Patreon 429 (campaign_id=%s) — backing off %.1fs (retry %d/%d)",
campaign_id, delay, attempt, self._max_retries,
)
time.sleep(delay)
continue
break
if resp.status_code in (401, 403):
# Auth rejected — expired/missing cookies or an insufficient tier.
# Actionable as "rotate credentials", so it's auth, not drift/http.
raise PatreonAuthError(
f"Patreon posts API returned HTTP {resp.status_code} — auth "
f"rejected (cookies expired or tier insufficient; "
f"campaign_id={campaign_id})",
status_code=resp.status_code,
)
if resp.status_code != 200:
# A persistent 429 (retries exhausted) is terminal RATE_LIMITED — carry
# the server's raw Retry-After seconds so the cooldown matches its hint
# (plan #708 B1). Header is uncapped here; the cooldown clamps it.
retry_after = None
if resp.status_code == 429:
hdr = resp.headers.get("Retry-After")
if hdr:
try:
retry_after = float(hdr)
except (TypeError, ValueError):
retry_after = None
raise PatreonAPIError(
f"Patreon posts API returned HTTP {resp.status_code} "
f"(campaign_id={campaign_id})",
status_code=resp.status_code,
retry_after=retry_after,
)
try:
payload = resp.json()
except ValueError as exc:
# A non-JSON body here is almost always the HTML login/challenge
# page served when cookies are missing/expired — that is an AUTH
# failure (rotate cookies), not API drift (update the ingester) and
# not a transient network error.
raise PatreonAuthError(
"Patreon posts API returned a non-JSON response (likely an "
f"HTML login/challenge page — session expired; "
f"campaign_id={campaign_id}): {exc}"
) from exc
return payload
# -- parsing -----------------------------------------------------------
@staticmethod
def _transform(response: dict) -> dict:
"""Flatten the JSON:API `included` array for relationship resolution.
Returns a dict keyed by `(type, id)` → that resource's `attributes`
(so a post's relationships can be resolved in O(1)). Missing/oddly
shaped `included` entries are skipped rather than fatal — drift
detection for the top-level shape lives in _validate_response.
"""
index: dict[tuple[str, str], dict] = {}
for inc in response.get("included") or []:
if not isinstance(inc, dict):
continue
rtype = inc.get("type")
rid = inc.get("id")
if rtype is None or rid is None:
continue
index[(str(rtype), str(rid))] = inc.get("attributes") or {}
return index
@staticmethod
def _validate_response(response: dict) -> None:
if not isinstance(response, dict):
raise PatreonDriftError("Patreon response was not a JSON object")
if "data" not in response:
raise PatreonDriftError("Patreon response missing top-level 'data' key")
data = response.get("data")
if not isinstance(data, list):
raise PatreonDriftError("Patreon response 'data' was not a list")
def _related_ids(self, post: dict, rel_name: str) -> list[str]:
rels = post.get("relationships") or {}
rel = rels.get(rel_name) or {}
data = rel.get("data")
if isinstance(data, dict): # to-one relationship
data = [data]
if not isinstance(data, list):
return []
ids: list[str] = []
for ref in data:
if isinstance(ref, dict) and ref.get("id") is not None:
ids.append(str(ref["id"]))
return ids
@staticmethod
def _media_url(attrs: dict) -> str | None:
"""Pick the best fetchable URL for a media resource.
Prefer the full-size `download_url`; fall back to the largest
`image_urls` size. gallery-dl prefers download_url too, only dipping
into image_urls when a smaller configured size is requested — FC
always wants the original, so download_url first.
"""
download_url = attrs.get("download_url")
if isinstance(download_url, str) and download_url:
return download_url
image_urls = attrs.get("image_urls")
if isinstance(image_urls, dict):
for key in ("original", "full", "large", "default"):
candidate = image_urls.get(key)
if isinstance(candidate, str) and candidate:
return candidate
# Otherwise take any non-empty string value.
for candidate in image_urls.values():
if isinstance(candidate, str) and candidate:
return candidate
return None
def _media_item(self, attrs: dict, kind: str, post_id: str) -> MediaItem:
url = self._media_url(attrs)
if not url:
raise PatreonDriftError(
f"Patreon media (post {post_id}, kind={kind}) had no resolvable URL "
f"(no download_url / image_urls)"
)
# file_name is OPTIONAL: Patreon legitimately serves some gallery images
# without it (operator-flagged 2026-06-07, BlenderKnight post 73665615),
# and the URL basename is a fine fallback — the same thing gallery-dl
# uses. A genuine schema change shows up as no URL (above) or a media id
# absent from `included` (caller), not a missing name.
file_name = attrs.get("file_name")
filename = file_name if isinstance(file_name, str) and file_name else _basename_from_url(url)
return MediaItem(
url=url,
filename=filename,
kind=kind,
filehash=_filehash(url),
post_id=post_id,
)
def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
"""Resolve all downloadable media for one post.
Walks the kinds in the same order gallery-dl does — images,
image_large (post cover), attachments, postfile, content (inline
<img>) — and dedups within the post by filehash (first wins). The
image_large cover commonly duplicates a gallery image; deduping by
filehash collapses them to the gallery item (encountered first).
"""
post_id = str(post.get("id") or "")
attrs = post.get("attributes") or {}
items: list[MediaItem] = []
def _resolve_rel(rel_name: str, kind: str) -> None:
for mid in self._related_ids(post, rel_name):
media_attrs = included_index.get(("media", mid))
if media_attrs is None:
# Referenced but not in `included`: a media id with no
# resource is drift (we asked for include=media).
raise PatreonDriftError(
f"Patreon post {post_id} references media {mid} "
f"({rel_name}) not present in 'included'"
)
items.append(self._media_item(media_attrs, kind, post_id))
# 1. gallery images
_resolve_rel("images", "images")
# 2. image_large — the post-level cover (`image.large_url`). Not a
# media relationship; it lives on the post attributes.
image = attrs.get("image")
if isinstance(image, dict):
large_url = image.get("large_url") or image.get("url")
if isinstance(large_url, str) and large_url:
items.append(
MediaItem(
url=large_url,
filename=_basename_from_url(large_url),
kind="image_large",
filehash=_filehash(large_url),
post_id=post_id,
)
)
# 3. attachments
_resolve_rel("attachments_media", "attachments")
# 4. postfile — the post's primary attached file (`post_file`).
post_file = attrs.get("post_file")
if isinstance(post_file, dict):
pf_url = post_file.get("url") or post_file.get("download_url")
if isinstance(pf_url, str) and pf_url:
pf_name = post_file.get("name")
filename = (
pf_name
if isinstance(pf_name, str) and pf_name
else _basename_from_url(pf_url)
)
items.append(
MediaItem(
url=pf_url,
filename=filename,
kind="postfile",
filehash=_filehash(pf_url),
post_id=post_id,
)
)
# 5. content — inline <img> in the post HTML body.
content = attrs.get("content")
if isinstance(content, str) and content:
for raw_src in _CONTENT_IMG_RE.findall(content):
src = unescape(raw_src)
if not src:
continue
items.append(
MediaItem(
url=src,
filename=_basename_from_url(src),
kind="content",
filehash=_filehash(src),
post_id=post_id,
)
)
return _dedup_by_filehash(items)
@staticmethod
def post_meta(post: dict) -> dict:
"""Title + published date for a post — for the preview sample (plan #708
B4). Part of the client contract `ingest_core.Ingester.preview` calls."""
attrs = post.get("attributes") or {}
title = attrs.get("title")
published = attrs.get("published_at")
return {
"title": title if isinstance(title, str) else None,
"date": published if isinstance(published, str) else None,
}
# -- iteration ---------------------------------------------------------
def iter_posts(
self, campaign_id: str, cursor: str | None = None
) -> Iterator[tuple[dict, dict, str | None]]:
"""Yield (post, included_index, page_cursor) for every post in the feed.
Pages newest→oldest via `links.next`, validating each response for
drift before yielding. The triple gives a caller everything it needs to
resolve and checkpoint without re-fetching:
- post — the raw post resource.
- included_index — the page's flattened `included` (the same object
for every post on a page), to pass straight to
extract_media(post, included_index).
- page_cursor — the cursor that FETCHED this post's page (None for
the first page). The caller checkpoints THIS value,
matching the existing backfill cursor logic where
the saved cursor re-fetches the page being
processed (so a chunk cut mid-page resumes the page,
not the one after it).
"""
current_cursor = cursor
while True:
response = self._fetch(campaign_id, current_cursor)
self._validate_response(response)
page_cursor = current_cursor
included_index = self._transform(response)
for post in response.get("data") or []:
if isinstance(post, dict):
yield post, included_index, page_cursor
next_url = (response.get("links") or {}).get("next")
next_cursor = parse_cursor_from_url(next_url)
if not next_cursor:
return
current_cursor = next_cursor
# -- verify ------------------------------------------------------------
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
"""Cheap auth probe: fetch the first `/api/posts` page and report whether
the credential authenticated, WITHOUT downloading anything.
Returns `(ok, message)` matching the credential-verify contract:
- True — authenticated (the feed returned a valid JSON:API page).
- False — the credential was rejected (PatreonAuthError: 401/403, or an
HTML login page → cookies expired / tier insufficient).
- None — inconclusive: API drift (our parser is stale, not a cred
problem) or a transient network/HTTP error.
"""
try:
response = self._fetch(campaign_id, None)
self._validate_response(response)
except PatreonAuthError as exc:
return False, f"Patreon rejected the credential — {exc}"
except PatreonDriftError as exc:
return None, f"Couldn't verify — Patreon's API shape changed: {exc}"
except PatreonAPIError as exc:
return None, f"Couldn't verify (network/HTTP issue): {exc}"
return True, "Credentials valid — the Patreon feed authenticated."
def _dedup_by_filehash(items: list[MediaItem]) -> list[MediaItem]:
"""Drop later items sharing a filehash with an earlier one (first wins).
Items with no filehash (None) are never deduped against each other — we
can't prove they're the same file, so keep them all.
"""
seen: set[str] = set()
out: list[MediaItem] = []
for item in items:
if item.filehash is not None:
if item.filehash in seen:
continue
seen.add(item.filehash)
out.append(item)
return out
-510
View File
@@ -1,510 +0,0 @@
"""Native Patreon media downloader (build step 2b of the native ingester).
Given a Patreon post and its already-resolved `MediaItem`s (from
patreon_client.extract_media), download the media to the EXACT on-disk layout
gallery-dl produces, write a sidecar JSON the existing importer consumes, and
report per-media outcomes.
This module is PURE: no DB. The cross-run seen-ledger and the iter_posts
orchestration are a LATER step. The tier-1 (seen) skip is an INJECTED predicate
(`is_seen`), so this module needs no DB and is unit-testable without network.
On-disk layout (matches gallery-dl):
<images_root>/<artist_slug>/patreon/<DIR>/<NN>_<filename>
where <DIR> = "<YYYY-MM-DD>_<post_id>_<title40>" (date prefix omitted when the
post's published_at is unparseable), <NN> is the 1-based index of the item in
the post zero-padded to 2, and <filename> is MediaItem.filename. The sidecar
is written next to the media as <NN>_<stem>.json (media_path.with_suffix).
Video: a MediaItem whose URL is a Mux/HLS stream (host stream.mux.com or path
endswith .m3u8) is fetched with yt-dlp (subprocess), passing the same
Referer/Origin headers gallery-dl forwards for Mux playback (see gallery_dl.py
_get_default_config). yt-dlp may remux to a container of its own choosing, so we
accept the actual output extension and record the real path.
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import contextlib
import json
import logging
import os
import subprocess
import time
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from urllib.parse import urlsplit
import requests
from .file_validator import is_validatable, quarantine_file, validate_file
from .patreon_client import (
_BACKOFF_CAP_SECONDS,
_load_session,
_retry_after_seconds,
)
log = logging.getLogger(__name__)
_TITLE_MAX = 40
_TIMEOUT_SECONDS = 120.0
_CHUNK = 1 << 16
# Retry a media GET that hits a TRANSIENT failure within the same pass (plan
# #705 #8): a transport blip (connection reset / timeout / truncated stream), a
# 429, or a 5xx. PERMANENT failures (404 gone, 403 forbidden) fail fast straight
# to the error/dead-letter path — no point re-fetching them. Keeps a momentary
# network hiccup from becoming a per-item error that waits for the next walk.
_MAX_MEDIA_RETRIES = 3
# requests transport errors worth retrying (vs. an HTTPError, which is a real
# server response and is classified by status code).
_TRANSIENT_TRANSPORT_EXC = (
requests.ConnectionError,
requests.Timeout,
requests.exceptions.ChunkedEncodingError,
)
# Referer/Origin yt-dlp must send for Mux-hosted Patreon video. Mux's JWT
# playback policy checks Referer/Origin on every request, so yt-dlp must send
# Patreon's, not its own default. (gallery-dl forwarded the same headers before
# the #697 cutover removed its Patreon path; this is now the only place they
# live.)
_VIDEO_HEADERS = {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
}
# Characters Windows/gallery-dl path-restrict forbids, plus path separators.
_FORBIDDEN = set('<>:"/\\|?*')
def _sanitize(name: str) -> str:
"""Make `name` safe for a single filesystem path segment.
Replaces path separators, the Windows-forbidden set <>:"/\\|?* and control
characters with `_`, then strips trailing dots/spaces (gallery-dl
path-restrict behavior). Never returns empty (falls back to "_").
"""
out = []
for ch in name:
if ch in _FORBIDDEN or ord(ch) < 32:
out.append("_")
else:
out.append(ch)
cleaned = "".join(out).rstrip(". ")
return cleaned or "_"
def _is_video_url(url: str) -> bool:
parts = urlsplit(url)
if parts.hostname and parts.hostname.lower() == "stream.mux.com":
return True
return parts.path.lower().endswith(".m3u8")
def _post_dir_name(post: dict) -> str:
"""Build the post directory name matching gallery-dl's layout."""
post_id = str(post.get("id") or "")
attrs = post.get("attributes") or {}
title = attrs.get("title")
title = title if isinstance(title, str) else ""
title40 = title[:_TITLE_MAX]
published = attrs.get("published_at")
date_prefix = None
if isinstance(published, str) and published:
s = published.strip()
if s.endswith("Z"):
s = s[:-1] + "+00:00"
try:
dt = datetime.fromisoformat(s)
except ValueError:
dt = None
if dt is not None:
date_prefix = f"{dt:%Y-%m-%d}"
if date_prefix:
raw = f"{date_prefix}_{post_id}_{title40}"
else:
raw = f"{post_id}_{title40}"
return _sanitize(raw)
@dataclass
class MediaOutcome:
"""Per-media result of a download_post pass.
status is one of: "downloaded", "skipped_seen", "skipped_disk",
"quarantined", "error". `path` is the final on-disk path for "downloaded"
(the actual yt-dlp output for video), the path that already existed for
"skipped_disk", or the _quarantine destination for "quarantined"; None for
"skipped_seen" and (usually) "error". `error` carries the failure/validation
reason for "error"/"quarantined", else None.
"""
media: object # MediaItem (avoid importing the name for a bare annotation)
status: str
path: Path | None
error: str | None
class PatreonDownloader:
"""Download resolved Patreon media to gallery-dl's on-disk layout.
PURE: no DB. The HTTP session and the yt-dlp invocation are injectable seams
so tests run without network or a real subprocess:
- pass `session=` to stub `session.get`, or monkeypatch `_fetch_to_file`.
- monkeypatch `_run_ytdlp` to avoid spawning yt-dlp.
"""
def __init__(
self,
images_root: Path,
cookies_path: str | None = None,
*,
validate: bool = True,
rate_limit: float = 0.0,
session: requests.Session | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
self._validate = validate
# Politeness: seconds to sleep before each actual media download (paces
# the CDN; honors ImportSettings.download_rate_limit_seconds, the same
# value gallery-dl used as its between-downloads `sleep`). 0 = no pacing.
# Applied only to real downloads, not to seen/disk skips. plan #703.
self._rate_limit = rate_limit or 0.0
# Build a cookie-loaded session the same way patreon_client does, so the
# CDN GETs carry the creator's auth.
self.session = session if session is not None else _load_session(cookies_path)
# -- public ------------------------------------------------------------
def download_post(
self,
post: dict,
media_items: list,
artist_slug: str,
*,
is_seen: Callable[[object], bool] = lambda m: False,
should_stop: Callable[[], bool] = lambda: False,
) -> list[MediaOutcome]:
"""Download every media item of one post; return per-item outcomes.
Builds the post directory, iterates media (1-based NN), applies the
two-tier skip (injected is_seen, then disk), downloads (plain GET or
yt-dlp for video), writes the sidecar for each freshly-downloaded item,
validates, and returns outcomes. Resilient: one media's failure yields
an "error" outcome for that item; the rest proceed.
`should_stop()` is polled BEFORE each media item: a media-dense post can
otherwise run a backfill chunk far past its time-box (the engine only
re-checks the budget between posts), so we honour the deadline mid-post
and return the items done so far — the rest re-fetch next chunk (they
were never marked seen). Bounds chunk overrun to one media download.
"""
post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
outcomes: list[MediaOutcome] = []
for i, media in enumerate(media_items, start=1):
if should_stop():
break
try:
outcomes.append(
self._download_one(post, media, post_dir, artist_slug, i, is_seen)
)
except Exception as exc: # resilient: isolate one item's failure
log.warning(
"Patreon media failed (post %s, item %d): %s",
post.get("id"), i, exc,
)
outcomes.append(
MediaOutcome(media=media, status="error", path=None, error=str(exc))
)
return outcomes
# -- per-item ----------------------------------------------------------
def _download_one(
self,
post: dict,
media,
post_dir: Path,
artist_slug: str,
index: int,
is_seen: Callable[[object], bool],
) -> MediaOutcome:
# tier-1: seen ledger (injected; no DB here).
if is_seen(media):
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
nn = f"{index:02d}"
final_name = _sanitize(f"{nn}_{media.filename}")
media_path = post_dir / final_name
# tier-2: already on disk.
if media_path.exists():
return MediaOutcome(
media=media, status="skipped_disk", path=media_path, error=None
)
# Video may land at a different extension; honor a pre-existing remux.
if _is_video_url(media.url):
existing = self._existing_video_output(media_path)
if existing is not None:
return MediaOutcome(
media=media, status="skipped_disk", path=existing, error=None
)
post_dir.mkdir(parents=True, exist_ok=True)
# Pace real downloads only (the skips above already returned). plan #703.
if self._rate_limit > 0:
time.sleep(self._rate_limit)
if _is_video_url(media.url):
out_path = self._run_ytdlp(media.url, media_path, _VIDEO_HEADERS)
if out_path is None or not Path(out_path).exists():
return MediaOutcome(
media=media,
status="error",
path=None,
error="yt-dlp produced no output",
)
out_path = Path(out_path)
else:
out_path = self._fetch_get(media.url, media_path)
reason, quarantine_dest = self._validate_path(
out_path, artist_slug, media.url
)
if reason is not None:
# Quarantined (corrupt/invalid) — distinct from a download error
# so the run can report a real files_quarantined count + paths.
return MediaOutcome(
media=media, status="quarantined",
path=quarantine_dest, error=reason,
)
self._write_sidecar(post, out_path)
return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
# -- download seams ----------------------------------------------------
def _fetch_get(self, url: str, dest: Path) -> Path:
"""Stream `url` to a .part file then atomic-rename to `dest`.
Thin wrapper over `_fetch_to_file` so tests can stub either the whole
GET path (`_fetch_to_file`) or just `session.get`.
"""
part = dest.with_name(dest.name + ".part")
try:
self._fetch_to_file(url, part)
except Exception:
with contextlib.suppress(OSError):
part.unlink()
raise
os.replace(part, dest)
return dest
def _fetch_to_file(self, url: str, dest: Path) -> None:
"""Stream a non-video URL to `dest` via the (stubbable) session, retrying
TRANSIENT failures within the same pass (plan #705 #8) and RESUMING from
the bytes already on disk via a Range request when a retry follows a
mid-download cut (plan #708 B5).
Retried (backoff): transport blips (connection reset / timeout /
truncated stream — incl. mid-download), HTTP 429 (honoring Retry-After),
and 5xx. Failed fast (no retry → HTTPError → per-item error → dead-letter
path): 4xx other than 429 (404 gone, 403 forbidden) — re-fetching a
permanent failure is pointless.
Resume: on a retry, if bytes already landed in `dest`, ask for the rest
with `Range: bytes=<have>-`. A 206 means the server honored it → append; a
200 means it ignored it (served the whole file) → start clean. The caller
(_fetch_get) stages into a `.part`, so a non-range server never corrupts
the output — the worst case is re-downloading from zero, as before.
"""
attempt = 0
while True:
have = dest.stat().st_size if dest.exists() else 0
headers = {"Range": f"bytes={have}-"} if have > 0 else None
try:
resp = self.session.get(
url, stream=True, timeout=_TIMEOUT_SECONDS, headers=headers,
)
if (resp.status_code == 429 or resp.status_code >= 500) \
and attempt < _MAX_MEDIA_RETRIES:
attempt += 1
delay = _retry_after_seconds(resp, attempt)
log.warning(
"Patreon media transient HTTP %d (%s) — backing off "
"%.1fs (retry %d/%d)",
resp.status_code, url, delay, attempt, _MAX_MEDIA_RETRIES,
)
time.sleep(delay)
continue
# A Range that starts at/past EOF (we already have the whole file)
# comes back 416 — the bytes we kept ARE the file.
if have > 0 and resp.status_code == 416:
return
# 2xx → ok; 4xx-non-429 (or an exhausted 429/5xx) → HTTPError
# (permanent for this pass) → not caught below → per-item error.
resp.raise_for_status()
# 206 → server honored the Range; append after the kept bytes.
# Anything else (200) → it served the whole file → start clean.
mode = "ab" if (have > 0 and resp.status_code == 206) else "wb"
with open(dest, mode) as fh:
for chunk in resp.iter_content(chunk_size=_CHUNK):
if chunk:
fh.write(chunk)
return
except _TRANSIENT_TRANSPORT_EXC as exc:
if attempt >= _MAX_MEDIA_RETRIES:
raise # exhausted → terminal error outcome
attempt += 1
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
log.warning(
"Patreon media transport error (%s) — backing off %.1fs "
"(retry %d/%d): %s",
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
)
time.sleep(delay)
def _run_ytdlp(self, url: str, dest: Path, headers: dict) -> Path | None:
"""Invoke yt-dlp to fetch a Mux/HLS stream to (around) `dest`.
Returns the actual output path (yt-dlp may remux to a different
container, so we resolve the real file afterward). Overridden/
monkeypatched in tests to avoid spawning a real subprocess.
The output template uses `dest` without its extension; yt-dlp appends
the chosen container extension. We pass Referer/Origin (Mux JWT policy)
and the cookies file.
Mirrors the GET path's transient/permanent split (plan #705 #8): a hung
fetch (TimeoutExpired) or a spawn failure (OSError) is TRANSIENT — back
off and retry. A non-zero yt-dlp exit (CalledProcessError) is treated as
PERMANENT for this pass — yt-dlp already does its OWN internal network
retries, so a non-zero exit is effectively a real failure (private/gone/
geo-blocked), like a 4xx on the GET path: fail fast to the per-item error
→ dead-letter path.
"""
dest = Path(dest)
out_template = str(dest.with_suffix("")) + ".%(ext)s"
cmd = ["yt-dlp", "--no-progress", "-o", out_template]
for key, value in headers.items():
cmd += ["--add-header", f"{key}:{value}"]
if self.cookies_path and os.path.isfile(self.cookies_path):
cmd += ["--cookies", self.cookies_path]
cmd.append(url)
attempt = 0
while True:
try:
subprocess.run(
cmd,
check=True,
capture_output=True,
text=True,
timeout=_TIMEOUT_SECONDS,
)
break
except subprocess.CalledProcessError as exc:
# Permanent for this pass — fail fast (no retry).
log.warning(
"yt-dlp failed (exit %s) for %s: %s",
exc.returncode, url, (exc.stderr or "").strip() or exc,
)
return None
except (OSError, subprocess.TimeoutExpired) as exc:
# Transient — back off and retry, like a transport blip on a GET.
if attempt >= _MAX_MEDIA_RETRIES:
log.warning(
"yt-dlp transient failure exhausted for %s: %s", url, exc
)
return None
attempt += 1
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
log.warning(
"yt-dlp transient failure (%s) — backing off %.1fs "
"(retry %d/%d): %s",
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
)
time.sleep(delay)
return self._existing_video_output(dest)
def _existing_video_output(self, dest: Path) -> Path | None:
"""Find a yt-dlp output for `dest` regardless of chosen extension.
Returns `dest` itself if present, else any sibling sharing the same
stem (the remuxed container). None if nothing matched.
"""
dest = Path(dest)
if dest.exists():
return dest
stem = dest.with_suffix("").name
parent = dest.parent
if not parent.is_dir():
return None
for cand in sorted(parent.iterdir()):
if cand.is_file() and cand.stem == stem and cand.name != dest.name:
return cand
return None
# -- validation --------------------------------------------------------
def _validate_path(
self, path: Path, artist_slug: str, source_url: str | None = None
) -> tuple[str | None, Path | None]:
"""Validate a freshly-written file; quarantine if bad.
Uses the shared `file_validator.quarantine_file` — same move + provenance
sidecar gallery-dl writes (the native path used to skip the sidecar; that
parity gap is closed here). Returns `(reason, quarantine_dest)` when
quarantined (dest is the original path if the move itself failed), else
`(None, None)` (ok / not validatable / disabled). plan #704: the dest is
surfaced so the run reports a real quarantined-paths list.
"""
if not self._validate or not is_validatable(path):
return None, None
try:
result = validate_file(path)
except Exception as exc:
log.warning("Validator raised on %s: %s", path, exc)
return None, None
if result.ok:
return None, None
dest = quarantine_file(
self.images_root, path, artist_slug, "patreon",
url=source_url, result=result,
)
return (result.reason or "validation failed"), (dest or path)
# -- sidecar -----------------------------------------------------------
def _write_sidecar(self, post: dict, media_path: Path) -> Path:
"""Write the importer-consumed sidecar next to `media_path`.
Patreon uses base-default sidecar keys (parse_sidecar maps
category->platform, id->external_post_id, title->post_title,
content->description, published_at->post_date, url->post_url). Patreon
registers no derive_post_url, so `url` is trusted as the permalink — we
pass the post's attributes.url.
"""
attrs = post.get("attributes") or {}
title = attrs.get("title")
content = attrs.get("content")
published = attrs.get("published_at")
url = attrs.get("url")
data = {
"category": "patreon",
"id": str(post.get("id") or ""),
"title": title if isinstance(title, str) else "",
"content": content if isinstance(content, str) else "",
"published_at": published if isinstance(published, str) else None,
"url": url if isinstance(url, str) else None,
}
sidecar_path = media_path.with_suffix(".json")
sidecar_path.write_text(json.dumps(data, indent=2))
return sidecar_path
-197
View File
@@ -1,197 +0,0 @@
"""Native Patreon ingester — the Patreon ADAPTER over the platform-agnostic core.
The orchestration that drives a native subscription walk (page a feed → extract
media → tiered skip → download → mark-seen / record-failures / checkpoint-cursor
→ return a gallery-dl-shaped `DownloadResult`, across tick/backfill/recovery)
now lives in `ingest_core.Ingester` — it's identical for every platform. This
module is the thin Patreon adapter: it wires the Patreon `client`/`downloader`/
ledger models/constraints/key into the core and supplies the Patreon-specific
failure mapping. `download_service.download_source` calls `PatreonIngester.run`
exactly as before; the public surface (this class, `_ledger_key`,
`DEAD_LETTER_THRESHOLD`, `verify_patreon_credential`) is unchanged.
Three modes (selected by `download_service` from `config_overrides` state):
- tick — newest→oldest, skip seen (tier-1 ledger + tier-2 disk), early-out
after N contiguous already-have-it items (the cheap native
equivalent of gallery-dl's `exit:20`, now free of per-file HEADs).
- backfill — full-history walk in a time-boxed chunk, resuming from the
pagination cursor checkpoint; reaches the bottom → "complete".
- recovery — like backfill but BYPASSES the tier-1 seen-ledger AND the
dead-letter ledger, so deliberately-dropped-and-deleted near-dups
get re-fetched and re-evaluated under the current pHash threshold
(tier-2 disk skip still spares files we kept).
The seen/dead-letter ledgers live in Postgres (`patreon_seen_media` /
`patreon_failed_media`); the core opens SHORT-LIVED sync sessions per page batch
— never held across a network fetch ([[db-connection-held-across-subprocess]]).
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Callable
from pathlib import Path
from ..models import PatreonFailedMedia, PatreonSeenMedia
from .gallery_dl import DownloadResult, ErrorType
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
from .patreon_client import (
MediaItem,
PatreonAPIError,
PatreonAuthError,
PatreonClient,
PatreonDriftError,
)
from .patreon_downloader import PatreonDownloader
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
__all__ = [
"DEAD_LETTER_THRESHOLD",
"PatreonIngester",
"_ledger_key",
"verify_patreon_credential",
]
log = logging.getLogger(__name__)
# Ledger keys are stored in patreon_seen_media.filehash VARCHAR(128); bound any
# synthesized key so a pathologically long file_name can't overflow the column.
_LEDGER_KEY_MAX = 128
def _ledger_key(media: MediaItem) -> str:
"""Stable per-media identity for the cross-run seen-ledger.
A Patreon CDN URL carries a 32-char MD5 (`media.filehash`) — that is the
natural key. Some media have none: Mux/HLS video (`stream.mux.com`, no
content hash at discovery) and the odd inline-content `<img>` pointing at a
hashless URL. The plan calls the video case the ``video:<post_id>:<media_id>``
sentinel; `MediaItem` carries no media_id, so the post-scoped filename is the
stable proxy. Bounded to the column width.
"""
if media.filehash:
return media.filehash
return f"{media.post_id}:{media.filename}"[:_LEDGER_KEY_MAX]
class PatreonIngester(Ingester):
"""Walk a Patreon campaign's posts, download unseen media, return a
`DownloadResult`. A thin adapter over `ingest_core.Ingester`.
Construct with the per-source `cookies_path` and a sync sessionmaker for the
ledgers. `client` / `downloader` are injectable seams so unit tests run
without network, subprocess, or a real CDN.
"""
def __init__(
self,
images_root: Path,
cookies_path: str | None,
session_factory: Callable[[], object],
*,
validate: bool = True,
rate_limit: float = 0.0,
request_sleep: float = 0.0,
client: PatreonClient | None = None,
downloader: PatreonDownloader | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
# Pacing (plan #703): request_sleep paces the API page fetches,
# rate_limit paces the media downloads. Injected client/downloader (in
# tests) already carry their own pacing, so these only apply to the
# default-constructed ones.
resolved_client = (
client
if client is not None
else PatreonClient(cookies_path, request_sleep=request_sleep)
)
resolved_downloader = (
downloader
if downloader is not None
else PatreonDownloader(
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit
)
)
super().__init__(
client=resolved_client,
downloader=resolved_downloader,
session_factory=session_factory,
seen_model=PatreonSeenMedia,
failed_model=PatreonFailedMedia,
seen_constraint="uq_patreon_seen_media_source_id",
failed_constraint="uq_patreon_failed_media_source_id",
ledger_key=_ledger_key,
platform="patreon",
error_base=PatreonAPIError,
)
# -- failure mapping (Patreon exception taxonomy) ----------------------
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
"""Map a client-level exception to a loud, typed failed DownloadResult.
We NEVER return a silent zero-download "success" — the whole point of the
native ingester is to fail RED when Patreon's API shape or our auth
changes. The typed mapping lets FailingSourcesCard render the right chip
and tells the operator what to do:
- PatreonAuthError → AUTH_ERROR (rotate cookies)
- PatreonDriftError → API_DRIFT (ingester field-set/parser needs update)
- HTTP 429 / 404 → RATE_LIMITED / NOT_FOUND
- other HTTP status → HTTP_ERROR; transport failure → NETWORK_ERROR
PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
they must be matched before the generic HTTP/transport fallthrough.
"""
message = str(exc)
if isinstance(exc, PatreonAuthError):
error_type = ErrorType.AUTH_ERROR
elif isinstance(exc, PatreonDriftError):
error_type = ErrorType.API_DRIFT
message = f"Patreon API changed — ingester needs update: {message}"
else: # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
status = getattr(exc, "status_code", None)
if status == 429:
error_type = ErrorType.RATE_LIMITED
elif status == 404:
error_type = ErrorType.NOT_FOUND
elif status is not None:
error_type = ErrorType.HTTP_ERROR
else:
error_type = ErrorType.NETWORK_ERROR
log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
result = _result(
success=False, return_code=1,
error_type=error_type, error_message=message,
)
# plan #708 B1: carry the server's Retry-After up to the cooldown.
if error_type == ErrorType.RATE_LIMITED:
result.retry_after_seconds = getattr(exc, "retry_after", None)
return result
async def verify_patreon_credential(
url: str,
cookies_path: str | None,
overrides: dict | None,
) -> tuple[bool | None, str]:
"""Native Patreon credential probe — the verify counterpart to the ingester's
download path, sharing its campaign-id resolution. Resolves the campaign id
(override / id: URL / vanity) then does ONE authenticated `/api/posts` fetch
via PatreonClient.verify_auth. Returns the uniform `(ok, message)` contract
(True / False / None) so download_backends.verify_credential can treat it
interchangeably with the gallery-dl probe. No download, no DB.
"""
campaign_id, _ = await resolve_campaign_id_for_source(url, cookies_path, overrides)
if not campaign_id:
vanity = extract_vanity(url)
return None, (
f"Couldn't resolve the Patreon campaign id — can't verify. "
f"source_url={url!r}; vanity={vanity!r} "
"(cookies expired, or the creator moved/renamed?)."
)
client = PatreonClient(cookies_path)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, client.verify_auth, campaign_id)
-180
View File
@@ -19,56 +19,18 @@ import asyncio
import http.cookiejar import http.cookiejar
import logging import logging
import os import os
import re
import requests import requests
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
_CAMPAIGNS_URL = "https://www.patreon.com/api/campaigns" _CAMPAIGNS_URL = "https://www.patreon.com/api/campaigns"
_POSTS_API = "https://www.patreon.com/api/posts"
# A source URL of the form `.../id:<digits>` already carries the campaign id
# (no lookup needed). The vanity regex deliberately EXCLUDES the id: form so the
# two paths don't overlap.
#
# Patreon serves the creator vanity under several path prefixes — bare
# (`patreon.com/Atole`), `c/` (`patreon.com/c/Atole`), and `cw/`
# (`patreon.com/cw/Atole`, its current "creator workspace" URL). The optional
# prefix group must list `cw/` BEFORE `c/` so the longer prefix wins — otherwise
# `cw/Atole` matches the bare branch and yields vanity="cw" (operator-flagged
# 2026-06-07: every `/cw/` source failed resolution on vanity="cw").
_ID_URL_RE = re.compile(r"/id:(\d+)")
# A single-post permalink — patreon.com/posts/<slug>-<post_id> (or bare
# /posts/<post_id>). The trailing digits are the post id; the creator's
# campaign is resolved from the post itself (operator-flagged 2026-06-07: a
# /posts/ source resolved vanity="posts"). `posts/` is excluded from the vanity
# regex so it never masquerades as a creator slug.
_POST_URL_RE = re.compile(r"/posts/(?:[^/?#]*-)?(\d+)(?:[/?#]|$)")
_VANITY_PREFIXES = ("cw/", "c/")
_VANITY_RE = re.compile(
r"^https?://(?:www\.)?patreon\.com/(?:cw/|c/)?(?!(?:id:|posts/))([^/?#]+)",
re.IGNORECASE,
)
_USER_AGENT = ( _USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
) )
_TIMEOUT_SECONDS = 10.0 _TIMEOUT_SECONDS = 10.0
# Fallback resolution: Patreon's `/api/campaigns?filter[vanity]=` lookup has
# proven unreliable (returns empty `data` for creators that clearly exist —
# operator-flagged 2026-06-06). gallery-dl never used that endpoint; it scrapes
# the campaign id out of the creator page's bootstrap JSON. We do the same as a
# fallback: fetch the creator page HTML and pull the first campaign id out of
# any of these embeddings (ordered most- to least-specific).
_PAGE_CAMPAIGN_ID_PATTERNS = (
re.compile(r'"id":\s*"(\d+)",\s*"type":\s*"campaign"'),
re.compile(r'"campaign":\s*\{\s*"data":\s*\{\s*"id":\s*"(\d+)"'),
re.compile(r"/api/campaigns/(\d+)"),
re.compile(r'"campaign_id":\s*"?(\d+)'),
)
def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJar | None: def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJar | None:
if not cookies_path or not os.path.isfile(cookies_path): if not cookies_path or not os.path.isfile(cookies_path):
@@ -83,15 +45,6 @@ def _load_cookie_jar(cookies_path: str | None) -> http.cookiejar.MozillaCookieJa
def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None: def _sync_lookup(vanity: str, cookies_path: str | None) -> str | None:
"""Resolve a vanity to a campaign id: try the campaigns API first (cheap,
structured), then fall back to scraping the creator page (robust against the
API's empty-data failures). Returns None only when both miss."""
return _lookup_via_api(vanity, cookies_path) or _lookup_via_page(
vanity, cookies_path
)
def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
jar = _load_cookie_jar(cookies_path) jar = _load_cookie_jar(cookies_path)
headers = { headers = {
"User-Agent": _USER_AGENT, "User-Agent": _USER_AGENT,
@@ -139,87 +92,6 @@ def _lookup_via_api(vanity: str, cookies_path: str | None) -> str | None:
return campaign_id return campaign_id
def _scrape_campaign_id(html: str) -> str | None:
"""First campaign id found in creator-page HTML via the known embeddings."""
if not isinstance(html, str):
return None
for pat in _PAGE_CAMPAIGN_ID_PATTERNS:
m = pat.search(html)
if m:
return m.group(1)
return None
def _lookup_via_page(vanity: str, cookies_path: str | None) -> str | None:
"""Fallback: GET the creator page and scrape the campaign id from the page
bootstrap (gallery-dl's method). Tries both the bare and `/c/` vanity paths
Patreon redirects between. Never raises."""
jar = _load_cookie_jar(cookies_path)
headers = {"User-Agent": _USER_AGENT, "Accept": "text/html"}
# Try the bare vanity and every known creator-path prefix Patreon
# redirects between (c/, cw/) — the one the source used isn't known here
# (extract_vanity already stripped it).
page_urls = [f"https://www.patreon.com/{vanity}"]
page_urls += [f"https://www.patreon.com/{p}{vanity}" for p in _VANITY_PREFIXES]
for page_url in page_urls:
try:
resp = requests.get(
page_url,
headers=headers,
cookies=jar,
timeout=_TIMEOUT_SECONDS,
allow_redirects=True,
)
except requests.RequestException as exc:
log.warning("Patreon creator-page fetch failed for %s: %s", page_url, exc)
continue
if resp.status_code != 200:
continue
campaign_id = _scrape_campaign_id(resp.text)
if campaign_id:
log.info(
"Resolved Patreon vanity=%s → campaign_id=%s via creator page",
vanity, campaign_id,
)
return campaign_id
return None
def _lookup_campaign_from_post(post_id: str, cookies_path: str | None) -> str | None:
"""Resolve the owning campaign id from a single post id via the Patreon
post API (`/api/posts/<id>?include=campaign`). A `/posts/` source URL points
at one post, but a subscription walks the whole creator — so we follow the
post to its campaign. Never raises."""
jar = _load_cookie_jar(cookies_path)
headers = {"User-Agent": _USER_AGENT, "Accept": "application/vnd.api+json"}
params = {"include": "campaign", "fields[campaign]": "name"}
url = f"{_POSTS_API}/{post_id}"
try:
resp = requests.get(
url, params=params, headers=headers, cookies=jar,
timeout=_TIMEOUT_SECONDS,
)
except requests.RequestException as exc:
log.warning("Patreon post lookup failed for post=%s: %s", post_id, exc)
return None
if resp.status_code != 200:
log.warning("Patreon post API returned HTTP %d for post=%s", resp.status_code, post_id)
return None
try:
payload = resp.json()
except ValueError:
return None
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
return None
campaign = ((data.get("relationships") or {}).get("campaign") or {}).get("data") or {}
campaign_id = campaign.get("id")
if isinstance(campaign_id, str) and campaign_id:
log.info("Resolved Patreon post=%s → campaign_id=%s", post_id, campaign_id)
return campaign_id
return None
async def resolve_campaign_id( async def resolve_campaign_id(
vanity: str, vanity: str,
cookies_path: str | None, cookies_path: str | None,
@@ -228,55 +100,3 @@ async def resolve_campaign_id(
Never raises.""" Never raises."""
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, _sync_lookup, vanity, cookies_path) return await loop.run_in_executor(None, _sync_lookup, vanity, cookies_path)
async def resolve_campaign_from_post(
post_id: str, cookies_path: str | None,
) -> str | None:
"""Async wrapper around _lookup_campaign_from_post. Never raises."""
loop = asyncio.get_running_loop()
return await loop.run_in_executor(
None, _lookup_campaign_from_post, post_id, cookies_path
)
def extract_vanity(url: str) -> str | None:
"""The vanity slug from a Patreon creator URL, or None for an `id:` URL."""
m = _VANITY_RE.match(url or "")
return m.group(1) if m else None
async def resolve_campaign_id_for_source(
url: str,
cookies_path: str | None,
overrides: dict | None,
) -> tuple[str | None, str | None]:
"""Resolve a Patreon source to its campaign id — the single resolution path
shared by the download ingester and the credential-verify probe.
Order: cached `patreon_campaign_id` override → an `id:<digits>` URL → a
`/posts/<id>` permalink (resolve the owning campaign from the post) → a
vanity lookup against the campaigns API. Returns
`(campaign_id, newly_resolved_id)`: `newly_resolved_id` is non-None whenever
a lookup actually ran, so the caller caches it on the source (the
override/id: paths needed no lookup). `(None, None)` when unresolvable.
Never raises.
"""
overrides = overrides or {}
cached = overrides.get("patreon_campaign_id")
if cached:
return cached, None
id_match = _ID_URL_RE.search(url or "")
if id_match:
return id_match.group(1), None
# A single-post URL → follow the post to its creator's campaign so the
# source subscribes to the whole feed (a subscription isn't one post).
post_match = _POST_URL_RE.search(url or "")
if post_match:
resolved = await resolve_campaign_from_post(post_match.group(1), cookies_path)
return resolved, resolved
vanity = extract_vanity(url)
if vanity:
resolved = await resolve_campaign_id(vanity, cookies_path)
return resolved, resolved
return None, None
-59
View File
@@ -1,59 +0,0 @@
"""Per-platform download concurrency cap.
Some platforms (Patreon) are API-rate-sensitive enough that two *simultaneous*
walks can trip the server's rate limit even with each source pacing its own
requests. The platform-cooldown handles the AFTERMATH of a 429; this is the
preventive half — it serializes downloads PER PLATFORM to one at a time.
Different platforms still run concurrently up to the worker's concurrency; only
a second walk on the SAME serialized platform waits. The lock lives in Redis
(the Celery broker) with a TTL, so a SIGKILL'd worker can't wedge a platform —
the lock auto-expires shortly after the download hard time limit.
"""
from __future__ import annotations
import logging
import redis
from ..config import get_config
log = logging.getLogger(__name__)
# Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
# here: each runs as a self-pacing subprocess and they're lower-volume. Add a
# platform here to cap it to a single concurrent walk.
SERIALIZED_PLATFORMS = frozenset({"patreon"})
_LOCK_PREFIX = "fc:download_lock:"
_client: redis.Redis | None = None
def _redis() -> redis.Redis:
# One client per worker process (Celery prefork forks before tasks run, so
# each process lazily builds its own). redis-py pools connections.
global _client
if _client is None:
_client = redis.from_url(get_config().celery_broker_url)
return _client
def platform_lock(platform: str, *, ttl_seconds: int):
"""A non-blocking Redis lock for `platform`, or None when the platform is
not serialized. Caller does `.acquire(blocking=False)` / `.release()`.
Returns None (rather than raising) on any Redis error so a broker hiccup
degrades to the prior behaviour (uncapped) instead of stalling downloads.
"""
if platform not in SERIALIZED_PLATFORMS:
return None
try:
return _redis().lock(
f"{_LOCK_PREFIX}{platform}",
timeout=ttl_seconds,
blocking=False,
)
except redis.RedisError as exc: # pragma: no cover - broker outage
log.warning("platform_lock unavailable for %s: %s", platform, exc)
return None
+3 -12
View File
@@ -20,20 +20,11 @@ class ShowcaseService:
async def random_sample(self, limit: int = 60) -> list[dict]: async def random_sample(self, limit: int = 60) -> list[dict]:
if limit < 1 or limit > 200: if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200") raise ValueError("limit must be between 1 and 200")
# Over-sample then random-order (#699): SYSTEM_ROWS reads CONTIGUOUS rows
# from each sampled page, so sequentially-imported near-duplicates
# (multi-image posts, variant sets) come back adjacent and cluster in the
# showcase ("three near-identical in a row"). Sampling a multiple of
# `limit` spans more pages, and ORDER BY random() before taking `limit`
# breaks the physical adjacency — far better spread, still cheap
# (random() over a few hundred rows, not the whole table).
oversample = min(limit * 5, 1000)
stmt = select(ImageRecord).from_statement( stmt = select(ImageRecord).from_statement(
text( text(
"SELECT * FROM (" "SELECT * FROM image_record "
" SELECT * FROM image_record TABLESAMPLE SYSTEM_ROWS(:o)" "TABLESAMPLE SYSTEM_ROWS(:n)"
") sub ORDER BY random() LIMIT :n" ).bindparams(n=limit)
).bindparams(o=oversample, n=limit)
) )
rows = (await self.session.execute(stmt)).scalars().all() rows = (await self.session.execute(stmt)).scalars().all()
return [ return [
+25 -96
View File
@@ -64,15 +64,6 @@ class SourceRecord:
consecutive_failures: int consecutive_failures: int
next_check_at: str | None next_check_at: str | None
backfill_runs_remaining: int backfill_runs_remaining: int
# plan #693: derived from config_overrides for the UI badge.
backfill_state: str | None # "running" | "complete" | "stalled" | None (idle)
backfill_chunks: int
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
backfill_bypass_seen: bool
# plan #704: cumulative posts processed across the walk's chunks — live
# progress for the badge.
backfill_posts: int
def to_dict(self) -> dict: def to_dict(self) -> dict:
return { return {
@@ -91,10 +82,6 @@ class SourceRecord:
"consecutive_failures": self.consecutive_failures, "consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at, "next_check_at": self.next_check_at,
"backfill_runs_remaining": self.backfill_runs_remaining, "backfill_runs_remaining": self.backfill_runs_remaining,
"backfill_state": self.backfill_state,
"backfill_chunks": self.backfill_chunks,
"backfill_bypass_seen": self.backfill_bypass_seen,
"backfill_posts": self.backfill_posts,
} }
@@ -102,13 +89,10 @@ class SourceRecord:
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"} _EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
# Plan #693: backfill safety cap. "Start backfill" (and a newly created # Plan #544 follow-up: newly created enabled sources pre-arm backfill so
# enabled source) arms a run-until-done walk; this caps how many time-boxed # their first N polls walk gallery-dl's full post history with the longer
# chunks it may spend before pausing as "stalled", so a pathological walk that # timeout (matches the manual "Deep scan" button's default).
# never reaches the bottom can't run forever. Generous on purpose — at NEW_SOURCE_BACKFILL_RUNS = 3
# BACKFILL_CHUNK_SECONDS (600s) per chunk this is ~33h of cumulative walk, far
# beyond any real catalog; the cursor stall-guard is the real terminator.
BACKFILL_MAX_CHUNKS = 200
class SourceService: class SourceService:
@@ -151,7 +135,6 @@ class SourceService:
self, source: Source, artist: Artist, settings: ImportSettings, self, source: Source, artist: Artist, settings: ImportSettings,
) -> SourceRecord: ) -> SourceRecord:
nxt = compute_next_check_at(source, artist, settings) nxt = compute_next_check_at(source, artist, settings)
co = source.config_overrides or {}
return SourceRecord( return SourceRecord(
id=source.id, id=source.id,
artist_id=source.artist_id, artist_id=source.artist_id,
@@ -168,10 +151,6 @@ class SourceService:
consecutive_failures=source.consecutive_failures or 0, consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None, next_check_at=nxt.isoformat() if nxt else None,
backfill_runs_remaining=source.backfill_runs_remaining or 0, backfill_runs_remaining=source.backfill_runs_remaining or 0,
backfill_state=co.get("_backfill_state"),
backfill_chunks=int(co.get("_backfill_chunks", 0)),
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
backfill_posts=int(co.get("_backfill_posts", 0)),
) )
async def _row_to_record(self, source: Source) -> SourceRecord: async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -233,17 +212,16 @@ class SourceService:
select(func.count(Source.id)).where(Source.artist_id == artist_id) select(func.count(Source.id)).where(Source.artist_id == artist_id)
)).scalar_one() )).scalar_one()
# Plan #693: a freshly added subscription has no archive yet, so it # Plan #544 follow-up: a freshly added subscription has no archive
# should walk its full post history once. Arm run-until-done backfill # yet, so the first few polls would walk the full post history in
# (state="running" + the chunk cap); the time-boxed chunks march to the # tick mode and trip exit:20 after ~20 contiguous archive hits —
# bottom across ticks, then flip to "complete" and tick mode takes over. # except there are none yet, so tick mode would walk forever and
# Disabled sources (incl. sidecar synthetics, url='sidecar:...') are # blow the wall-clock cap. Pre-arm backfill so the initial syncs
# never polled, so leave them idle. # use the longer timeout + skip:True walk. Tick mode resumes once
if enabled: # the budget is spent or the queue drains.
config_overrides = {**(config_overrides or {}), "_backfill_state": "running"} # Disabled sources (incl. sidecar synthetics, url='sidecar:...')
backfill_runs = BACKFILL_MAX_CHUNKS # are never polled, so leave their counter at 0.
else: backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
backfill_runs = 0
source = Source( source = Source(
artist_id=artist_id, platform=platform, url=url, artist_id=artist_id, platform=platform, url=url,
enabled=enabled, config_overrides=config_overrides, enabled=enabled, config_overrides=config_overrides,
@@ -308,71 +286,22 @@ class SourceService:
await self.session.commit() await self.session.commit()
return await self._row_to_record(source) return await self._row_to_record(source)
async def start_backfill(self, source_id: int) -> SourceRecord: async def set_backfill_runs(
"""Plan #693: arm a run-until-done backfill. Sets state="running" and self, source_id: int, runs: int,
the chunk cap; download runs then walk the full post history in ) -> SourceRecord:
time-boxed chunks (skip:True + BACKFILL_CHUNK_SECONDS), resuming from """Plan #544: arm a source for backfill mode. The next `runs`
the cursor each chunk, until gallery-dl reaches the bottom (→ state download runs will use gallery-dl's full-walk config (skip: True
"complete") or the cap/stall-guard pauses it (→ "stalled"). Clears any + 30-min timeout) instead of the catch-up default. Runs must be
prior cursor/chunk/stall state so a re-start walks fresh from the top.""" 1..10 — bigger is rejected to keep the operator from accidentally
setting a runaway budget."""
if not isinstance(runs, int) or runs < 1 or runs > 10:
raise ValueError("runs must be an integer in [1, 10]")
source = (await self.session.execute( source = (await self.session.execute(
select(Source).where(Source.id == source_id) select(Source).where(Source.id == source_id)
)).scalar_one_or_none() )).scalar_one_or_none()
if source is None: if source is None:
raise LookupError(f"source id={source_id} not found") raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {}) source.backfill_runs_remaining = runs
co["_backfill_state"] = "running"
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
"_backfill_posts"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def start_recovery(self, source_id: int) -> SourceRecord:
"""Plan #697: arm a RECOVERY walk — a backfill that bypasses the Patreon
seen-ledger so deliberately-dropped-and-deleted near-dups get re-fetched
and re-evaluated under the CURRENT pHash threshold (tier-2 disk still
spares files we kept). Reuses the entire #693 backfill state machine
(time-boxed chunks, cursor checkpoint, complete/stall lifecycle) plus the
`_backfill_bypass_seen` flag that flips download mode to recovery. Clears
any prior cursor/chunk/stall state so it walks fresh from the top. The
flag is cleared on completion (download_service) and on stop.
Recovery is Patreon-only (the seen-ledger is Patreon's); for other
platforms the flag is inert (download_service ignores it) and the walk
runs as a plain backfill. The UI gates the action to Patreon sources."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
co["_backfill_state"] = "running"
co["_backfill_bypass_seen"] = True
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
"_backfill_posts"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def stop_backfill(self, source_id: int) -> SourceRecord:
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
Clears the running state + cursor/chunk/stall bookkeeping."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_posts"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = 0
await self.session.commit() await self.session.commit()
return await self._row_to_record(source) return await self._row_to_record(source)
+2 -306
View File
@@ -1,7 +1,5 @@
"""Tag CRUD + autocomplete + image-tag association.""" """Tag CRUD + autocomplete + image-tag association."""
import logging
import time
from collections.abc import Sequence from collections.abc import Sequence
from dataclasses import dataclass from dataclasses import dataclass
@@ -14,25 +12,6 @@ from ..models import Tag, TagKind, image_tag
from ..models.tag_allowlist import TagAllowlist from ..models.tag_allowlist import TagAllowlist
from ..models.tag_reference_embedding import TagReferenceEmbedding from ..models.tag_reference_embedding import TagReferenceEmbedding
log = logging.getLogger(__name__)
def normalize_tag_name(name: str) -> str:
"""Canonical tag form (#701): collapse whitespace + capitalize the first
letter of each word, PRESERVING the rest of the word.
Per-word capitalize (NOT str.title(), which mangles apostrophes:
don't → Don'T). The word tail is left untouched so acronyms survive
(DC stays DC, NSFW stays NSFW) — operator-revised 2026-06-06: protecting
acronyms matters more than folding ALL-CAPS input. This MATCHES
ml/tag_name._title_word, so a tag suggested by the Camie tagger keeps the
exact casing the suggestion UI showed when it round-trips through the
create endpoint on Accept.
"""
return " ".join(
w[:1].upper() + w[1:] for w in (name or "").split()
)
class TagValidationError(ValueError): class TagValidationError(ValueError):
"""Raised when tag construction breaks the kind/fandom rules.""" """Raised when tag construction breaks the kind/fandom rules."""
@@ -92,10 +71,6 @@ class TagService:
- if fandom_id is set, the referenced tag must exist and have - if fandom_id is set, the referenced tag must exist and have
kind == TagKind.fandom kind == TagKind.fandom
""" """
# NOTE: case is NOT normalized here — find_or_create is the shared path
# the ML tagger / allowlist also use, and Title-Casing the booru
# vocabulary breaks allowlist matching. Display-casing (Title Case) is
# applied at the user-entry layer (api/tags create_tag) only (#701).
name = name.strip() name = name.strip()
if not name: if not name:
raise TagValidationError("Tag name cannot be empty") raise TagValidationError("Tag name cannot be empty")
@@ -118,17 +93,13 @@ class TagService:
# inserts; without the savepoint the outer transaction would # inserts; without the savepoint the outer transaction would
# poison and the calling request crashes. Mirrors # poison and the calling request crashes. Mirrors
# importer._get_or_create. # importer._get_or_create.
# Case-insensitive match (#701): a normalized (Title Case) input must
# find an existing differently-cased tag instead of forking a duplicate.
stmt = ( stmt = (
select(Tag) select(Tag)
.where(func.lower(Tag.name) == name.lower()) .where(Tag.name == name)
.where(Tag.kind == kind) .where(Tag.kind == kind)
.where( .where(
Tag.fandom_id.is_(None) if fandom_id is None else Tag.fandom_id == fandom_id Tag.fandom_id.is_(None) if fandom_id is None else Tag.fandom_id == fandom_id
) )
.order_by(Tag.id)
.limit(1)
) )
existing = (await self.session.execute(stmt)).scalar_one_or_none() existing = (await self.session.execute(stmt)).scalar_one_or_none()
if existing: if existing:
@@ -278,11 +249,9 @@ class TagService:
if tag is None: if tag is None:
raise TagValidationError(f"Tag {tag_id} not found") raise TagValidationError(f"Tag {tag_id} not found")
# Case-insensitive clash (#701) — renaming onto a differently-cased tag
# is still a merge, not a silent fork.
clash_stmt = ( clash_stmt = (
select(Tag) select(Tag)
.where(func.lower(Tag.name) == new_name.lower()) .where(Tag.name == new_name)
.where(Tag.kind == tag.kind) .where(Tag.kind == tag.kind)
.where( .where(
Tag.fandom_id.is_(None) Tag.fandom_id.is_(None)
@@ -290,8 +259,6 @@ class TagService:
else Tag.fandom_id == tag.fandom_id else Tag.fandom_id == tag.fandom_id
) )
.where(Tag.id != tag_id) .where(Tag.id != tag_id)
.order_by(Tag.id)
.limit(1)
) )
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none() clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
if clash is not None: if clash is not None:
@@ -313,69 +280,6 @@ class TagService:
await self.session.flush() await self.session.flush()
return tag return tag
async def set_fandom(
self, tag_id: int, fandom_id: int | None, *, merge: bool = False
) -> Tag:
"""Set / change / clear a character tag's fandom.
Raises TagValidationError unless the tag is a character and fandom_id
(when given) references a fandom tag. If the change would collide with
an existing character of the same name in the TARGET fandom, raises
TagMergeConflict (the API turns that into a 409 merge hint) — unless
merge=True, in which case this tag is merged INTO that existing
character (a deliberate cross-fandom merge) and the surviving target
is returned. Passing fandom_id=None clears the fandom.
"""
tag = await self.session.get(Tag, tag_id)
if tag is None:
raise TagValidationError(f"Tag {tag_id} not found")
if tag.kind != TagKind.character:
raise TagValidationError("Only character tags can have a fandom")
if fandom_id is not None:
fandom = await self.session.get(Tag, fandom_id)
if fandom is None or fandom.kind != TagKind.fandom:
raise TagValidationError(
f"fandom_id {fandom_id} does not reference a fandom tag"
)
if fandom_id == tag.fandom_id:
return tag
# Collision: another character with the same name already lives in the
# target fandom. Mirrors rename's (name, kind, fandom_id) uniqueness.
clash_stmt = (
select(Tag)
.where(Tag.name == tag.name)
.where(Tag.kind == TagKind.character)
.where(
Tag.fandom_id.is_(None)
if fandom_id is None
else Tag.fandom_id == fandom_id
)
.where(Tag.id != tag_id)
)
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
if clash is not None:
if not merge:
source_image_count = await self.session.scalar(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id == tag_id)
)
will_alias = await self._keep_as_alias(tag_id)
raise TagMergeConflict(
f"A character named {tag.name!r} already exists in that fandom",
target_id=clash.id,
target_name=clash.name,
source_image_count=int(source_image_count or 0),
will_alias=will_alias,
)
await self._do_merge(tag, clash)
return clash
tag.fandom_id = fandom_id
await self.session.flush()
return tag
async def merge(self, source_id: int, target_id: int) -> MergeResult: async def merge(self, source_id: int, target_id: int) -> MergeResult:
"""Transactionally repoint every FK from source→target, optionally """Transactionally repoint every FK from source→target, optionally
keep source's name as a tagger alias, delete source. Atomic: any keep source's name as a tagger alias, delete source. Atomic: any
@@ -394,14 +298,7 @@ class TagService:
raise TagValidationError( raise TagValidationError(
"Tags must be the same kind and fandom to merge" "Tags must be the same kind and fandom to merge"
) )
return await self._do_merge(source, target)
async def _do_merge(self, source: Tag, target: Tag) -> MergeResult:
"""Repoint every FK source→target, optionally keep source's name as a
tagger alias, delete source. NO kind/fandom validation — callers that
need it (public merge()) validate first; set_fandom's collision
resolution calls this directly for a deliberate CROSS-fandom merge."""
source_id, target_id = source.id, target.id
keep_as_alias = await self._keep_as_alias(source_id) keep_as_alias = await self._keep_as_alias(source_id)
source_name = source.name source_name = source.name
source_kind = source.kind source_kind = source.kind
@@ -534,21 +431,6 @@ class TagService:
update(Tag).where(Tag.fandom_id == src).values(fandom_id=tgt) update(Tag).where(Tag.fandom_id == src).values(fandom_id=tgt)
) )
async def _image_assoc_counts(self, tag_ids: list[int]) -> dict[int, int]:
"""image_tag row counts keyed by tag_id, for the survivor heuristic
(the best-connected tag in a collision group survives → fewest
FK repoints). Tags with zero associations are absent from the map."""
if not tag_ids:
return {}
rows = (
await self.session.execute(
select(image_tag.c.tag_id, func.count())
.where(image_tag.c.tag_id.in_(tag_ids))
.group_by(image_tag.c.tag_id)
)
).all()
return {tid: int(n) for tid, n in rows}
async def _create_protective_aliases( async def _create_protective_aliases(
self, src_name: str, src_kind: TagKind, tgt: int self, src_name: str, src_kind: TagKind, tgt: int
) -> bool: ) -> bool:
@@ -596,189 +478,3 @@ class TagService:
if res.rowcount: if res.rowcount:
created = True created = True
return created return created
# ---------------------------------------------------------------------------
# #714: retro-normalize existing tags to the #701 canonical (Title Case +
# collapsed whitespace) and merge case/whitespace-variant duplicates.
# ---------------------------------------------------------------------------
_NORMALIZE_SAMPLE_CAP = 50
def _group_existing_tags(
rows,
) -> dict[tuple, list[tuple[int, str]]]:
"""Group (id, name, kind, fandom_id) rows by their post-normalization
identity: (kind, COALESCE(fandom_id, -1), canonical_name). Every tag that
would collapse to the same canonical lives in ONE group, so the canonical
form is unique within (kind, fandom) once each group is resolved."""
groups: dict[tuple, list[tuple[int, str]]] = {}
for tag_id, name, kind, fandom_id in rows:
canonical = normalize_tag_name(name)
key = (kind, fandom_id if fandom_id is not None else -1, canonical)
groups.setdefault(key, []).append((tag_id, name))
return groups
def _group_needs_change(canonical: str, members: list[tuple[int, str]]) -> bool:
"""A group is already canonical iff it's a single member whose name equals
the canonical form. Anything else (a collision, or a lone mis-cased tag)
needs work."""
if len(members) > 1:
return True
return members[0][1] != canonical
def _best_connected(tag_ids: list[int], counts: dict[int, int]) -> int:
"""The tag with the most image associations (→ fewest FK repoints when it
survives), tie-broken to the lowest id for determinism. Module-level so the
key closes over its parameter, not a loop variable (ruff B023)."""
return max(tag_ids, key=lambda tid: (counts.get(tid, 0), -tid))
async def normalize_existing_tags(
session: AsyncSession,
*,
dry_run: bool = False,
time_budget_seconds: float | None = None,
) -> dict:
"""Convert the back-catalog to the #701 canonical tag form.
For each (kind, fandom, canonical) group: pick a survivor, merge any
case/whitespace-variant siblings INTO it via the tested merge path
(TagService._do_merge — FK repoints + protective aliases), then rename the
survivor to the canonical form. Idempotent: a group that is already a lone
canonical tag is a no-op, so re-running is safe.
A first run over a fresh back-catalog can touch tens of thousands of tags
(the whole booru-derived vocabulary needs recasing) and won't finish inside
one Celery time limit — it timed out at 40 min (operator-flagged 2026-06-07).
`time_budget_seconds` time-boxes the live run: it stops cleanly at the budget
and reports `partial`/`remaining` so the caller can re-enqueue and continue.
Because it commits per group and is idempotent, the next run just picks up
the groups still needing change.
dry_run=True returns a projection (counts + a sample of the changes) with no
mutations. Live runs commit per group and isolate failures per group so one
bad group can't strand the rest.
Returns (dry_run):
{"groups": N, "collisions": M, "tags_to_merge": K, "tags_to_rename": R,
"total_changes": T, "sample": [{"to", "from": [...], "kind", "merge"}]}
Returns (live):
{"groups_processed", "merged", "renamed", "aliases_created", "errors",
"total_changes", "remaining", "partial", "sample": [...]}
"""
rows = (
await session.execute(
select(Tag.id, Tag.name, Tag.kind, Tag.fandom_id)
)
).all()
groups = _group_existing_tags(rows)
# Deterministic sample/ordering: by kind then canonical name.
touched = sorted(
(
(key, members)
for key, members in groups.items()
if _group_needs_change(key[2], members)
),
key=lambda km: (
km[0][0].value if hasattr(km[0][0], "value") else str(km[0][0]),
km[0][2].lower(),
),
)
sample = [
{
"to": key[2],
"from": [name for _id, name in members],
"kind": key[0].value if hasattr(key[0], "value") else str(key[0]),
"merge": len(members) > 1,
}
for key, members in touched[:_NORMALIZE_SAMPLE_CAP]
]
if dry_run:
collisions = sum(1 for key, m in touched if len(m) > 1)
tags_to_merge = sum(len(m) - 1 for key, m in touched if len(m) > 1)
# A group renames iff the canonical form isn't already one of its
# members' exact names (else that member is picked as survivor → no
# rename, the rest merge into it).
tags_to_rename = sum(
1
for key, m in touched
if key[2] not in {name for _id, name in m}
)
return {
"groups": len(groups),
"collisions": collisions,
"tags_to_merge": tags_to_merge,
"tags_to_rename": tags_to_rename,
"total_changes": len(touched),
"sample": sample,
}
svc = TagService(session)
summary = {
"groups_processed": 0,
"merged": 0,
"renamed": 0,
"aliases_created": 0,
"errors": 0,
"total_changes": len(touched),
"remaining": len(touched),
"partial": False,
"sample": sample,
}
start = time.monotonic()
for done, (key, members) in enumerate(touched):
# Time-box: stop cleanly before the Celery limit kills us mid-group and
# strands the run as a timeout. The caller re-enqueues to finish the
# rest (idempotent — already-canonical groups are skipped next pass).
if (
time_budget_seconds is not None
and time.monotonic() - start >= time_budget_seconds
):
summary["partial"] = True
summary["remaining"] = len(touched) - done
break
canonical = key[2]
names_by_id = dict(members)
# Survivor: prefer a member already named canonically (no rename, no
# self-alias); else the best-connected (fewest FK repoints); else
# lowest id for determinism.
survivor_id = next(
(tid for tid, name in members if name == canonical), None
)
if survivor_id is None:
counts = await svc._image_assoc_counts(list(names_by_id))
survivor_id = _best_connected(list(names_by_id), counts)
loser_ids = [tid for tid in names_by_id if tid != survivor_id]
try:
survivor = await session.get(Tag, survivor_id)
for loser_id in loser_ids:
loser = await session.get(Tag, loser_id)
if loser is None:
continue
result = await svc._do_merge(loser, survivor)
if result.alias_created:
summary["aliases_created"] += 1
if survivor.name != canonical:
survivor.name = canonical
await session.flush()
summary["renamed"] += 1
await session.commit()
summary["groups_processed"] += 1
summary["merged"] += len(loser_ids)
except Exception as exc: # one bad group must not strand the rest
await session.rollback()
summary["errors"] += 1
log.warning(
"tag normalize failed for group %r: %s", canonical, exc
)
else:
# Loop finished without hitting the time budget — nothing left to do.
summary["remaining"] = 0
return summary
+1 -10
View File
@@ -10,7 +10,6 @@ disposes it (``await engine.dispose()``) when its loop ends.
""" """
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.pool import NullPool
from ..config import get_config from ..config import get_config
@@ -18,13 +17,5 @@ from ..config import get_config
def async_session_factory(): def async_session_factory():
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine.""" """Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
cfg = get_config() cfg = get_config()
# NullPool: this engine lives for ONE task (created + disposed per engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
# asyncio.run loop), so intra-task connection pooling buys nothing and
# actively bit us — download_source releases its phase-1 connection
# before a multi-minute gallery-dl subprocess, and a *pooled* idle
# connection would be reaped by the server and handed back dead to
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
# opens a fresh real connection on each checkout, so phase 3 always
# reconnects clean; pre_ping is then redundant.
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
-7
View File
@@ -34,10 +34,3 @@ def sync_session_factory():
) )
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False) _SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
return _SESSIONMAKER return _SESSIONMAKER
def get_sync_engine():
"""The process-wide sync Engine — for raw work that needs a connection
directly (e.g. AUTOCOMMIT VACUUM, which can't run inside a transaction)."""
sync_session_factory() # ensure _ENGINE is initialized
return _ENGINE
-69
View File
@@ -55,72 +55,3 @@ def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
return cleanup_service.delete_images( return cleanup_service.delete_images(
session, image_ids=image_ids, images_root=IMAGES_ROOT, session, image_ids=image_ids, images_root=IMAGES_ROOT,
) )
@celery.task(
name="backend.app.tasks.admin.reextract_archive_attachments_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
def reextract_archive_attachments_task(self) -> dict:
"""Wraps cleanup_service.reextract_archive_attachments (#713 part 2):
re-extract PostAttachments that are actually archives but were filed
opaquely before the magic-byte gate, and link their members to the post."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.reextract_archive_attachments(
session, images_root=IMAGES_ROOT,
)
@celery.task(
name="backend.app.tasks.admin.normalize_tags_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
# Time-box one chunk well under the soft limit so a large back-catalog (the
# first run recases the whole booru vocabulary) can't run the task into the
# Celery time limit — it timed out at 40 min, operator-flagged 2026-06-07. The
# task re-enqueues itself until nothing remains (idempotent — already-canonical
# groups are skipped). 600s keeps each chunk short enough that the recovery
# sweep and other maintenance tasks interleave on the concurrency-1 queue.
_NORMALIZE_CHUNK_SECONDS = 600
def normalize_tags_task(self) -> dict:
"""Wraps tag_service.normalize_existing_tags (#714): Title-Case the
back-catalog and merge case/whitespace-variant duplicate tags via the
tested async merge path. Time-boxed + self-resuming so a huge first run
finishes across chunks instead of timing out. Runs under its own asyncio
loop + per-task async engine (NullPool), mirroring download_source."""
import asyncio
from ..services.tag_service import normalize_existing_tags
from ._async_session import async_session_factory
async def _run() -> dict:
async_factory, async_engine = async_session_factory()
try:
async with async_factory() as session:
# normalize_existing_tags commits per group internally.
return await normalize_existing_tags(
session, dry_run=False,
time_budget_seconds=_NORMALIZE_CHUNK_SECONDS,
)
finally:
await async_engine.dispose()
summary = asyncio.run(_run())
# More groups to canonicalize than fit this chunk — continue in the next.
if summary.get("partial") and summary.get("remaining", 0) > 0:
log.info(
"normalize_tags_task chunk done (%d processed, %d remaining) — "
"re-enqueuing to continue",
summary.get("groups_processed", 0), summary["remaining"],
)
normalize_tags_task.delay()
return summary
+5 -168
View File
@@ -1,17 +1,12 @@
"""download_source Celery task — runs DownloadService for one source.""" """download_source Celery task — runs DownloadService for one source."""
import asyncio import asyncio
import logging
from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError from sqlalchemy.exc import DBAPIError, OperationalError
from sqlalchemy.orm import Session as SyncSession
from ..celery_app import celery from ..celery_app import celery
from ..models import DownloadEvent, ImportSettings, Source from ..models import ImportSettings
from ..services.credential_crypto import CredentialCrypto from ..services.credential_crypto import CredentialCrypto
from ..services.credential_service import CredentialService from ..services.credential_service import CredentialService
from ..services.download_service import DownloadService from ..services.download_service import DownloadService
@@ -21,99 +16,9 @@ from ..services.thumbnailer import Thumbnailer
from ._async_session import async_session_factory from ._async_session import async_session_factory
from .import_file import _sync_session_factory from .import_file import _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images") IMAGES_ROOT = Path("/images")
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64" _KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
# Celery time budget for one download_source run. The ceiling that
# governs *clean* teardown is the SOFT limit: it raises a catchable
# SoftTimeLimitExceeded in-process, whereas the HARD limit SIGKILLs the
# worker (no chance to finalize). Both gallery-dl subprocess budgets
# (gallery_dl.py: _DEFAULT_GDL_TIMEOUT_SECONDS=870 tick,
# BACKFILL_CHUNK_SECONDS=600 per backfill chunk, plan #693) MUST sit below the soft limit
# so subprocess.run raises its own TimeoutExpired first — that path
# captures partial stdout/stderr and finalizes the DownloadEvent. soft is
# max-subprocess (1170) + ~180s phase-3 persist headroom; hard is soft +
# 150s SIGKILL backstop. Audit 2026-06-03 (Anduo #39912): the old
# soft=900 sat BELOW the 1170 backfill budget, so SoftTimeLimitExceeded
# preempted TimeoutExpired and the event stranded empty. The recovery
# sweep's DOWNLOAD_STALL_THRESHOLD_MINUTES (30 min) still trails the new
# 25-min hard kill by 5 min, so it stays a true backstop. Invariant
# guarded by test_timeout_ladder_keeps_subprocess_budgets_under_soft_limit.
DOWNLOAD_SOFT_TIME_LIMIT = 1350
DOWNLOAD_HARD_TIME_LIMIT = 1500
# Per-platform serialization (plan: concurrency cap). When a serialized
# platform (Patreon) is already walking, defer this run by re-enqueuing it a
# little later rather than holding a worker slot or bowling into the same rate
# limit. Bounded so a wedged platform eventually runs anyway. The lock TTL sits
# just past the hard kill so a SIGKILL'd worker's lock auto-expires; a backfill
# chunk only holds it ~10 min, so the bounded wait stays well under the 30-min
# DownloadEvent recovery sweep.
_PLATFORM_LOCK_TTL = DOWNLOAD_HARD_TIME_LIMIT + 120
_SERIALIZE_COUNTDOWN = 20
_MAX_SERIALIZE_WAITS = 45 # ~15 min ceiling, then run uncapped as a safety valve
def _peek_platform(source_id: int) -> str | None:
SyncFactory = _sync_session_factory()
with SyncFactory() as session:
return session.execute(
select(Source.platform).where(Source.id == source_id)
).scalar_one_or_none()
def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
"""Defense in depth for the soft-time-limit kill path.
A SoftTimeLimitExceeded unwinds download_source before phase 3 can
finalize the DownloadEvent, leaving it 'running' until the recovery
sweep stamps a context-free "stranded" error 30 min later — AND
leaving backfill_runs_remaining undecremented so the source re-runs
and re-strands every tick (Anduo #39912, 2026-06-03). Flip the
in-flight event to error with a real reason, mirror phase 3's
source-health write, and decrement any backfill budget so a
chronically-slow source self-heals back to tick mode.
The caller owns the commit. All mutations are gated on actually
finding a running event, so a benign late soft-limit (phase 3 already
committed) is a no-op.
"""
now = datetime.now(UTC)
ev = session.execute(
select(DownloadEvent)
.where(DownloadEvent.source_id == source_id)
.where(DownloadEvent.status == "running")
.order_by(DownloadEvent.id.desc())
.limit(1)
).scalar_one_or_none()
if ev is None:
return
ev.status = "error"
ev.finished_at = now
ev.error = (
f"killed by Celery soft time limit ({DOWNLOAD_SOFT_TIME_LIMIT}s) "
"before the download finished — the run exceeded its time budget. "
"Progress is checkpointed per page, so the next tick resumes near "
"the cut; the backfill budget was decremented so it walks less. If "
"this recurs, a single post is heavy enough to overrun the chunk "
"time-box on its own."
)
ev.metadata_ = {
**(ev.metadata_ or {}),
"error_type": "timeout",
"soft_time_limited": True,
}
src = session.get(Source, source_id)
if src is not None:
src.consecutive_failures = (src.consecutive_failures or 0) + 1
src.last_error = "soft time limit exceeded"
src.error_type = "timeout"
src.last_checked_at = now
if (src.backfill_runs_remaining or 0) > 0:
src.backfill_runs_remaining = max(0, src.backfill_runs_remaining - 1)
@celery.task( @celery.task(
name="backend.app.tasks.download.download_source", name="backend.app.tasks.download.download_source",
@@ -124,51 +29,12 @@ def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
retry_backoff_max=120, retry_backoff_max=120,
retry_jitter=True, retry_jitter=True,
max_retries=3, max_retries=3,
soft_time_limit=DOWNLOAD_SOFT_TIME_LIMIT, soft_time_limit=900,
time_limit=DOWNLOAD_HARD_TIME_LIMIT, time_limit=1200,
) )
def download_source(self, source_id: int, _serialize_waits: int = 0) -> int: def download_source(self, source_id: int) -> int:
"""Returns the DownloadEvent.id.""" """Returns the DownloadEvent.id."""
# Per-platform concurrency cap: only one Patreon walk runs at a time.
from ..services.platform_lock import platform_lock
platform = _peek_platform(source_id)
lock = (
platform_lock(platform, ttl_seconds=_PLATFORM_LOCK_TTL)
if platform is not None
else None
)
if lock is not None:
try:
got_lock = bool(lock.acquire(blocking=False))
except Exception: # noqa: BLE001 — broker hiccup → degrade to uncapped
log.warning("platform lock acquire failed for %s; running uncapped", platform)
lock = None
got_lock = True
if lock is not None and not got_lock:
if _serialize_waits < _MAX_SERIALIZE_WAITS:
# Another walk on this platform holds the lock — re-enqueue
# ourselves shortly (the pending DownloadEvent stays; no new
# event, no log spam) rather than running concurrently into
# the rate limit.
download_source.apply_async(
(source_id,),
{"_serialize_waits": _serialize_waits + 1},
countdown=_SERIALIZE_COUNTDOWN,
)
log.info(
"download_source(%s) deferred — %s already walking (wait %d/%d)",
source_id, platform, _serialize_waits + 1, _MAX_SERIALIZE_WAITS,
)
return -1
log.warning(
"download_source(%s) running WITHOUT the %s lock — max serialize "
"waits hit, proceeding uncapped",
source_id, platform,
)
lock = None
async def _run(): async def _run():
async_factory, async_engine = async_session_factory() async_factory, async_engine = async_session_factory()
SyncFactory = _sync_session_factory() SyncFactory = _sync_session_factory()
@@ -202,38 +68,9 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
gdl=gdl, gdl=gdl,
importer=importer, importer=importer,
cred_service=cred_service, cred_service=cred_service,
# The native Patreon ingester opens its own short-lived
# sync sessions for the seen-ledger (never held across
# the walk). Same factory the importer's sync session
# comes from — a different DB connection per checkout.
sync_session_factory=SyncFactory,
) )
return await svc.download_source(source_id) return await svc.download_source(source_id)
finally: finally:
await async_engine.dispose() await async_engine.dispose()
try: return asyncio.run(_run())
return asyncio.run(_run())
except SoftTimeLimitExceeded:
# phase 3 never ran — salvage the in-flight event so the operator
# sees a real reason instead of the recovery sweep's generic
# "stranded" 30 min later (Anduo #39912). Best-effort: a failure
# here must not mask the timeout. Re-raise so Celery + the
# task_run signal handler still record the kill.
try:
SyncFactory = _sync_session_factory()
with SyncFactory() as session:
_finalize_soft_limited(session, source_id)
session.commit()
except Exception: # noqa: BLE001 — cleanup must not swallow the kill
log.exception("soft-limit finalize failed for source %s", source_id)
raise
finally:
# Release the per-platform lock so the next walk can proceed. The TTL is
# a backstop for a SIGKILL; here we free it the instant the run ends. A
# late release after TTL expiry raises (lock already gone) — ignore it.
if lock is not None:
try:
lock.release()
except Exception: # noqa: BLE001 — TTL may have already freed it
pass
+4 -41
View File
@@ -22,27 +22,10 @@ from ..models import (
TaskRun, TaskRun,
) )
from ..utils.phash import compute_phash from ..utils.phash import compute_phash
from ._sync_engine import get_sync_engine
from ._sync_engine import sync_session_factory as _sync_session_factory from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
# High-churn tables whose dead-tuple bloat matters: the TABLESAMPLE showcase
# reads physical blocks (bloat slows it directly), and the periodic
# prune/backfill/recovery tasks generate dead tuples faster than autovacuum
# always keeps up with. VACUUM reclaims them; ANALYZE refreshes planner stats.
# Allowlist ONLY — names are interpolated into VACUUM, so they must never come
# from request input.
VACUUM_TABLES = (
"image_record",
"image_provenance",
"post_attachment",
"download_event",
"task_run",
"import_task",
"import_batch",
)
STUCK_THRESHOLD_MINUTES = 5 STUCK_THRESHOLD_MINUTES = 5
# Archive ImportTasks run the per-member pipeline inline for every # Archive ImportTasks run the per-member pipeline inline for every
# member (import_archive_file: soft=30min/hard=35min). The ImportTask # member (import_archive_file: soft=30min/hard=35min). The ImportTask
@@ -63,12 +46,9 @@ MAX_RECOVERY_ATTEMPTS = 3
ORPHAN_PENDING_THRESHOLD_MINUTES = 30 ORPHAN_PENDING_THRESHOLD_MINUTES = 30
# DownloadEvent (pending|running) recovery threshold. download_source has # DownloadEvent (pending|running) recovery threshold. download_source has
# time_limit=1500s (25 min, DOWNLOAD_HARD_TIME_LIMIT); 30 min is 5 min past # time_limit=1200s (20 min); 30 min is 10 min past that, so a legitimately-
# that, so a legitimately-running task is hard-killed before the sweep ever # running task is never killed by the sweep. Operator-confirmed 2026-05-29
# touches it — the sweep only catches events whose worker died without # after 43 sources stranded at "last check never" by the in-flight guard.
# finalizing. Operator-confirmed 2026-05-29 after 43 sources stranded at
# "last check never" by the in-flight guard; budget bumped 2026-06-03 with
# the soft/hard limit raise (Anduo #39912).
DOWNLOAD_STALL_THRESHOLD_MINUTES = 30 DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7 OLD_TASK_DAYS = 7
@@ -555,7 +535,7 @@ def recover_stalled_download_events() -> int:
tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending') tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending')
and fires download_source.delay(). If that task dies before finalizing the and fires download_source.delay(). If that task dies before finalizing the
event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind
on the 1500s hard time_limit — the event stays in-flight forever. The next on the 1200s hard time_limit — the event stays in-flight forever. The next
tick then skips that source because of the in-flight guard (scan.py:168) tick then skips that source because of the in-flight guard (scan.py:168)
and Source.last_checked_at never updates; the operator sees "last check and Source.last_checked_at never updates; the operator sees "last check
never" in the Subscriptions health column, permanently. never" in the Subscriptions health column, permanently.
@@ -778,20 +758,3 @@ def cleanup_old_download_events() -> int:
) )
session.commit() session.commit()
return result.rowcount or 0 return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
def vacuum_analyze() -> dict:
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
reclaim dead-tuple bloat and refresh planner statistics. VACUUM cannot run
inside a transaction block, so it runs on an AUTOCOMMIT connection.
Scheduled weekly; also operator-triggerable from Settings → Maintenance.
"""
engine = get_sync_engine()
done = []
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
for table in VACUUM_TABLES:
conn.exec_driver_sql(f"VACUUM (ANALYZE) {table}")
done.append(table)
log.info("vacuum_analyze complete: %s", done)
return {"vacuumed": done}
-20
View File
@@ -2,26 +2,6 @@
from pathlib import Path from pathlib import Path
_MAX_EXT_LEN = 16
def safe_ext(name: str | Path) -> str:
"""Conservatively extract a short, alphanumeric file extension.
gallery-dl and Patreon CDN URLs produce basenames with URL-encoded
query-string artifacts, so `Path.suffix` can return 50+ chars of base64-ish
junk that blows bounded VARCHAR columns (e.g. PostAttachment.ext varchar(32)).
Accept only a suffix ≤16 chars whose post-dot characters are all alphanumeric;
otherwise return "" (no known extension). Operator-flagged 2026-05-25 — ONE
impl for the importer and the native Patreon client.
"""
suffix = Path(name).suffix.lower()
if not suffix or len(suffix) > _MAX_EXT_LEN:
return ""
if not all(c.isalnum() for c in suffix[1:]):
return ""
return suffix
def derive_subdir(source_path: Path, import_root: Path) -> str: def derive_subdir(source_path: Path, import_root: Path) -> str:
"""Returns the relative subdirectory of source_path under import_root. """Returns the relative subdirectory of source_path under import_root.
+9 -13
View File
@@ -149,8 +149,9 @@ def _run_probe(path_str: str) -> tuple[str, str | None]:
call the same code path. call the same code path.
""" """
path = Path(path_str) path = Path(path_str)
ext = path.suffix.lower()
try: try:
total, test_bad = _inspect_archive(path) total, test_bad = _inspect_archive(path, ext)
except Exception as exc: # noqa: BLE001 — clean rejection except Exception as exc: # noqa: BLE001 — clean rejection
return ("error", f"{type(exc).__name__}: {exc}") return ("error", f"{type(exc).__name__}: {exc}")
if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES: if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES:
@@ -161,29 +162,24 @@ def _run_probe(path_str: str) -> tuple[str, str | None]:
return ("ok", None) return ("ok", None)
def _inspect_archive(path: Path): def _inspect_archive(path: Path, ext: str):
"""Return (total_uncompressed_bytes | None, first_bad_member | None) """Return (total_uncompressed_bytes | None, first_bad_member | None)
for the archive. Format detected by extension OR magic bytes (so a for the archive. Format-specific; raises on a structurally-broken
mis-named archive is still bomb-guarded + integrity-tested, matching the container (caught by the child as a clean rejection)."""
extractor's gate); raises on a structurally-broken container (caught by the if ext in (".zip", ".cbz"):
child as a clean rejection)."""
from ..services.archive_extractor import detect_archive_format
fmt = detect_archive_format(path)
if fmt == "zip":
import zipfile import zipfile
with zipfile.ZipFile(path) as zf: with zipfile.ZipFile(path) as zf:
total = sum(zi.file_size for zi in zf.infolist()) total = sum(zi.file_size for zi in zf.infolist())
return total, zf.testzip() return total, zf.testzip()
if fmt == "rar": if ext == ".rar":
import rarfile import rarfile
with rarfile.RarFile(path) as rf: with rarfile.RarFile(path) as rf:
total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist()) total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist())
rf.testrar() rf.testrar()
return total, None return total, None
if fmt == "7z": if ext == ".7z":
import py7zr import py7zr
with py7zr.SevenZipFile(path, "r") as zf: with py7zr.SevenZipFile(path, "r") as zf:
@@ -191,5 +187,5 @@ def _inspect_archive(path: Path):
total = getattr(info, "uncompressed", None) total = getattr(info, "uncompressed", None)
ok = zf.test() # True / None when all members pass ok = zf.test() # True / None when all members pass
return total, (None if ok in (True, None) else "7z test reported corruption") return total, (None if ok in (True, None) else "7z test reported corruption")
# Not a recognised archive — nothing to test; treat as clean. # Unknown extension — nothing to test; treat as clean.
return None, None return None, None
+4 -46
View File
@@ -11,8 +11,6 @@
<PipelineStatusChip /> <PipelineStatusChip />
</div> </div>
<!-- Desktop: inline links, centered. Hidden on mobile (see media query),
where they fold into the hamburger menu on the right. -->
<nav class="fc-links"> <nav class="fc-links">
<RouterLink <RouterLink
v-for="r in navRoutes" v-for="r in navRoutes"
@@ -22,31 +20,9 @@
>{{ r.meta.title }}</RouterLink> >{{ r.meta.title }}</RouterLink>
</nav> </nav>
<div class="fc-nav-right"> <!-- Per-view contextual actions teleport here (Showcase: Shuffle,
<!-- Per-view contextual actions teleport here (Showcase: Shuffle, Gallery: Select). TopNav owns the slot, not its contents. -->
Gallery: Select). TopNav owns the slot, not its contents. --> <div id="fc-nav-actions" class="fc-nav-actions" />
<div id="fc-nav-actions" class="fc-nav-actions" />
<!-- Mobile nav: the link row collides with brand + actions below ~768px
(7+ links in one flex row), so collapse it into a menu. -->
<v-menu location="bottom end">
<template #activator="{ props: menuProps }">
<v-btn
v-bind="menuProps"
icon="mdi-menu" variant="text" size="small"
class="fc-nav-burger" aria-label="Menu"
/>
</template>
<v-list density="compact" min-width="180">
<v-list-item
v-for="r in navRoutes"
:key="r.name"
:to="{ name: r.name }"
:title="r.meta.title"
/>
</v-list>
</v-menu>
</div>
</header> </header>
</template> </template>
@@ -161,7 +137,7 @@ const health = computed(() => {
align-items: center; align-items: center;
flex-shrink: 0; flex-shrink: 0;
} }
.fc-nav-right { .fc-nav-actions {
flex: 1 1 0; flex: 1 1 0;
min-width: 0; min-width: 0;
display: flex; display: flex;
@@ -169,22 +145,4 @@ const health = computed(() => {
justify-content: flex-end; justify-content: flex-end;
gap: 0.5rem; gap: 0.5rem;
} }
.fc-nav-actions {
display: flex;
align-items: center;
gap: 0.5rem;
}
/* The hamburger only exists on mobile; the inline links carry desktop. */
.fc-nav-burger { display: none; }
@media (max-width: 768px) {
.fc-topnav { gap: 0.5rem; padding: 0.6rem 0.75rem; }
/* Fold the link row into the hamburger menu. */
.fc-links { display: none; }
.fc-nav-burger { display: inline-flex; }
}
@media (max-width: 480px) {
/* Reclaim width on the smallest phones — the glyph alone still brands. */
.fc-brand__text { display: none; }
}
</style> </style>
@@ -112,16 +112,6 @@ onMounted(async () => {
await store.loadDefaults() await store.loadDefaults()
threshold.value = store.defaults.single_color_threshold threshold.value = store.defaults.single_color_threshold
tolerance.value = store.defaults.single_color_tolerance tolerance.value = store.defaults.single_color_tolerance
// Reconnect to this rule's latest run so a scan started before navigating
// away keeps showing progress / its result on return (the scan itself runs
// backend-side regardless).
try {
const latest = await store.latestAuditForRule('single_color')
if (latest) {
audit.value = latest
if (latest.status === 'running') startPoll(latest.id)
}
} catch { /* non-fatal — card still works for a fresh scan */ }
}) })
onUnmounted(() => stopPoll()) onUnmounted(() => stopPoll())
@@ -97,16 +97,6 @@ let pollTimer = null
onMounted(async () => { onMounted(async () => {
await store.loadDefaults() await store.loadDefaults()
threshold.value = store.defaults.transparency_threshold threshold.value = store.defaults.transparency_threshold
// Reconnect to this rule's latest run so a scan started before navigating
// away keeps showing progress / its result on return (the scan itself runs
// backend-side regardless).
try {
const latest = await store.latestAuditForRule('transparency')
if (latest) {
audit.value = latest
if (latest.status === 'running') startPoll(latest.id)
}
} catch { /* non-fatal — card still works for a fresh scan */ }
}) })
onUnmounted(() => stopPoll()) onUnmounted(() => stopPoll())
@@ -8,15 +8,6 @@
<div v-if="card.preview_thumbnails.length === 0" class="fc-artistcard__noimg"> <div v-if="card.preview_thumbnails.length === 0" class="fc-artistcard__noimg">
No preview No preview
</div> </div>
<!-- Accent pill in the corner when this artist has content imported
since the operator last opened their detail view. Caps at 99+
to keep the layout compact; the actual count appears in the
banner inside ArtistView. -->
<span
v-if="(card.unseen_count || 0) > 0"
class="fc-artistcard__unseen"
:aria-label="`${card.unseen_count} new since last visit`"
>+{{ card.unseen_count > 99 ? '99+' : card.unseen_count }}</span>
</div> </div>
<v-card-text class="fc-artistcard__body"> <v-card-text class="fc-artistcard__body">
<div class="fc-artistcard__name">{{ card.name }}</div> <div class="fc-artistcard__name">{{ card.name }}</div>
@@ -46,30 +37,14 @@ function onCardClick() {
<style scoped> <style scoped>
.fc-artistcard { cursor: pointer; } .fc-artistcard { cursor: pointer; }
.fc-artistcard__previews { .fc-artistcard__previews {
position: relative;
display: grid; grid-template-columns: repeat(3, 1fr); display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; gap: 2px; aspect-ratio: 3 / 1;
/* Floor so tall source images can't escape the slot. NO max-height: an /* Explicit floor + ceiling so tall source images can't escape the
aspect-ratio box capped by max-height shrinks its own WIDTH to keep the preview slot even on browsers where aspect-ratio doesn't compute. */
ratio, leaving dead space beside the previews on a wide (single-column) min-height: 150px; max-height: 220px;
card. The grid below keeps columns ≲400px so the strip stays a sane height. */
min-height: 120px;
overflow: hidden; overflow: hidden;
background: rgb(var(--v-theme-surface-light)); background: rgb(var(--v-theme-surface-light));
} }
.fc-artistcard__unseen {
position: absolute;
top: 6px; right: 6px;
display: inline-flex; align-items: center;
padding: 2px 8px;
font-size: 11px; font-weight: 700; letter-spacing: 0.02em;
font-variant-numeric: tabular-nums;
color: rgb(var(--v-theme-on-accent, 0, 0, 0));
background: rgb(var(--v-theme-accent));
border-radius: 999px;
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
pointer-events: none;
}
.fc-artistcard__previews img { .fc-artistcard__previews img {
display: block; display: block;
width: 100%; height: 100%; width: 100%; height: 100%;
@@ -6,6 +6,7 @@
v-for="item in col" :key="item.id" v-for="item in col" :key="item.id"
class="fc-masonry__item" class="fc-masonry__item"
:class="{ 'fc-masonry__item--anim': shouldAnimate(item) }" :class="{ 'fc-masonry__item--anim': shouldAnimate(item) }"
:style="itemStyle(item)"
type="button" type="button"
@click="$emit('open', item.id)" @click="$emit('open', item.id)"
> >
@@ -65,6 +66,12 @@ function shouldAnimate(item) {
return idx !== undefined && idx >= props.animateFromIndex return idx !== undefined && idx >= props.animateFromIndex
} }
function itemStyle(item) {
if (!shouldAnimate(item)) return {}
const idx = idxById.value.get(item.id) - props.animateFromIndex
return { '--stagger-index': idx }
}
function aspectStyle(item) { function aspectStyle(item) {
const w = Number(item.width) const w = Number(item.width)
const h = Number(item.height) const h = Number(item.height)
@@ -110,15 +117,12 @@ useInfiniteScroll(sentinelEl, () => {
.fc-masonry__end { text-align: center; padding: 32px 0; } .fc-masonry__end { text-align: center; padding: 32px 0; }
/* Cascade entry: each tile flips up out of a backward tilt and settles /* Cascade entry: each tile flips up out of a backward tilt and settles
into place — more pronounced than a plain fade so the showcase reads as an into place, one at a time — more pronounced than a plain fade so the
"experience" (operator-flagged 2026-05-28). The reveal is paced entirely by showcase reads as an "experience" (operator-flagged 2026-05-28). The
the store, which pushes one fully-decoded item at a time (showcase.js); each `both` fill holds the hidden/tilted 0% state until each tile's staggered
tile therefore animates the instant it mounts, with NO per-index CSS delay — turn; the cubic-bezier overshoots slightly past flat then settles.
the old `animation-delay: index×70ms` compounded on top of the insert Honors prefers-reduced-motion. Tunables: tilt (-28deg), stagger (70ms),
cadence and made the cascade drag and desync as it grew (operator-flagged duration (0.6s). */
2026-06-04). The `both` fill holds the hidden/tilted 0% state until mount;
the cubic-bezier overshoots slightly past flat then settles. Honors
prefers-reduced-motion. Tunables: tilt (-28deg), duration (0.6s). */
@keyframes fc-masonry-item-in { @keyframes fc-masonry-item-in {
0% { 0% {
opacity: 0; opacity: 0;
@@ -134,6 +138,7 @@ useInfiniteScroll(sentinelEl, () => {
transform-origin: center top; transform-origin: center top;
backface-visibility: hidden; backface-visibility: hidden;
animation: fc-masonry-item-in 0.6s cubic-bezier(0.34, 1.45, 0.64, 1) both; animation: fc-masonry-item-in 0.6s cubic-bezier(0.34, 1.45, 0.64, 1) both;
animation-delay: calc(var(--stagger-index, 0) * 70ms);
} }
@media (prefers-reduced-motion: reduce) { @media (prefers-reduced-motion: reduce) {
.fc-masonry__item--anim { .fc-masonry__item--anim {
@@ -55,12 +55,6 @@
/> />
</template> </template>
<v-list density="compact"> <v-list density="compact">
<v-list-item
v-if="card.kind === 'character'"
title="Set fandom…"
prepend-icon="mdi-book-open-page-variant"
@click="$emit('set-fandom', card)"
/>
<v-list-item <v-list-item
title="Merge with…" title="Merge with…"
prepend-icon="mdi-call-merge" prepend-icon="mdi-call-merge"
@@ -84,9 +78,7 @@
import { ref } from 'vue' import { ref } from 'vue'
const props = defineProps({ card: { type: Object, required: true } }) const props = defineProps({ card: { type: Object, required: true } })
const emit = defineEmits([ const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
'open', 'rename', 'manage', 'read', 'merge-with', 'delete', 'set-fandom',
])
const editing = ref(false) const editing = ref(false)
const draft = ref('') const draft = ref('')
@@ -186,9 +186,7 @@ async function onDeleteConfirm(token) {
<style scoped> <style scoped>
.fc-bulk-panel { .fc-bulk-panel {
position: fixed; top: 0; right: 0; position: fixed; top: 0; right: 0;
/* min(320px, 90vw): on phones the panel never swallows the whole screen — width: 320px; height: 100vh; z-index: 1100;
leaves a sliver of the gallery visible behind it. */
width: min(320px, 90vw); height: 100vh; z-index: 1100;
background: rgb(var(--v-theme-surface)); background: rgb(var(--v-theme-surface));
border-left: 1px solid rgb(var(--v-theme-surface-light)); border-left: 1px solid rgb(var(--v-theme-surface-light));
box-shadow: -2px 0 16px rgba(0, 0, 0, 0.4); box-shadow: -2px 0 16px rgba(0, 0, 0, 0.4);
@@ -1,153 +0,0 @@
<template>
<div class="fc-facets">
<v-progress-linear
v-if="store.facetsLoading" indeterminate color="accent"
class="fc-facets__bar" height="2"
/>
<div class="fc-facets__group">
<span class="fc-facets__label">Platform</span>
<div class="fc-facets__chips">
<v-chip
v-for="p in platformOptions" :key="p.key"
size="small" label
:variant="p.key === store.filter.platform ? 'flat' : 'tonal'"
:color="p.key === store.filter.platform ? 'accent' : undefined"
@click="selectPlatform(p.key)"
>{{ p.label }}<span class="fc-facets__count">{{ p.count }}</span></v-chip>
<span v-if="!platformOptions.length" class="fc-facets__empty">none</span>
</div>
</div>
<div class="fc-facets__group">
<span class="fc-facets__label">Curation</span>
<div class="fc-facets__chips">
<v-chip
size="small" label
:variant="store.filter.untagged ? 'flat' : 'tonal'"
:color="store.filter.untagged ? 'accent' : undefined"
@click="toggleFlag('untagged')"
>Untagged<span class="fc-facets__count">{{ facetCount('untagged') }}</span></v-chip>
<v-chip
size="small" label
:variant="store.filter.no_artist ? 'flat' : 'tonal'"
:color="store.filter.no_artist ? 'accent' : undefined"
@click="toggleFlag('no_artist')"
>No artist<span class="fc-facets__count">{{ facetCount('no_artist') }}</span></v-chip>
</div>
</div>
<div class="fc-facets__group">
<span class="fc-facets__label">Date</span>
<v-text-field
type="date" density="compact" hide-details variant="outlined"
:model-value="store.filter.date_from" :min="dateMin" :max="dateMax"
class="fc-facets__date"
@update:model-value="setDate('date_from', $event)"
/>
<span class="fc-facets__dash"></span>
<v-text-field
type="date" density="compact" hide-details variant="outlined"
:model-value="store.filter.date_to" :min="dateMin" :max="dateMax"
class="fc-facets__date"
@update:model-value="setDate('date_to', $event)"
/>
</div>
</div>
</template>
<script setup>
import { computed, onBeforeUnmount, onMounted, watch } from 'vue'
import { useRouter } from 'vue-router'
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
// Mirrors backend gallery_service.UNSOURCED_PLATFORM — the sentinel that
// selects filesystem-imported content (the null/"no platform" bucket).
const UNSOURCED = '__unsourced__'
const store = useGalleryStore()
const router = useRouter()
const platformOptions = computed(() =>
(store.facets?.platforms || []).map((p) => ({
key: p.value === null ? UNSOURCED : p.value,
label: p.value === null ? 'No platform' : p.value,
count: p.count,
}))
)
function facetCount(flag) {
const v = store.facets?.[flag]
return v == null ? '' : v
}
const dateMin = computed(() => (store.facets?.date_min || '').slice(0, 10) || undefined)
const dateMax = computed(() => (store.facets?.date_max || '').slice(0, 10) || undefined)
// Single write path, shared format with the bar: clone → patch → URL. The
// route watcher in GalleryView reloads the store (and our filter-watch below
// refetches the facet counts for the new scope).
function pushPatch(patch) {
const n = cloneFilter(store.filter)
Object.assign(n, patch)
router.push({ name: 'gallery', query: filterToQuery(n) })
}
function selectPlatform(key) {
// Single-select, click-active-to-clear (the platform param is one value).
pushPatch({ platform: store.filter.platform === key ? null : key })
}
function toggleFlag(name) {
pushPatch({ [name]: !store.filter[name] })
}
function setDate(field, val) {
pushPatch({ [field]: val || null })
}
// Fetch when the panel opens; refetch (debounced) whenever the active filter
// changes so the live counts track the current scope. Never fires on scroll —
// the panel is the only caller of loadFacets.
onMounted(() => store.loadFacets())
let debounce = null
watch(() => store.filter, () => {
if (debounce) clearTimeout(debounce)
debounce = setTimeout(() => store.loadFacets(), 250)
})
onBeforeUnmount(() => { if (debounce) clearTimeout(debounce) })
</script>
<style scoped>
.fc-facets {
position: relative;
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 16px;
padding: 4px 12px 12px;
}
.fc-facets__bar { position: absolute; top: 0; left: 0; right: 0; }
.fc-facets__group { display: flex; align-items: center; gap: 8px; }
.fc-facets__label {
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-facets__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
/* The count rides inside the chip as a dimmer trailing number. */
.fc-facets__count {
margin-left: 6px;
opacity: 0.7;
font-variant-numeric: tabular-nums;
font-size: 0.78em;
}
.fc-facets__empty { color: rgb(var(--v-theme-on-surface-variant)); font-size: 0.8rem; }
.fc-facets__date { max-width: 160px; }
.fc-facets__dash { color: rgb(var(--v-theme-on-surface-variant)); }
/* Phones: let each facet group wrap and the side-by-side date inputs grow to
full width so they don't overflow a ~360px viewport. */
@media (max-width: 480px) {
.fc-facets__group { flex-wrap: wrap; }
.fc-facets__date { max-width: none; flex: 1 1 9rem; }
}
</style>
@@ -1,269 +0,0 @@
<template>
<div class="fc-filterbar-wrap">
<div class="fc-filterbar">
<v-autocomplete
v-model="selected"
:items="searchItems"
:loading="searchLoading"
item-title="name" item-value="value"
no-filter clearable hide-details density="compact" variant="outlined"
placeholder="Filter by tag or artist…"
prepend-inner-icon="mdi-filter-variant"
class="fc-filterbar__search"
@update:search="onSearch"
@update:model-value="onPick"
>
<template #item="{ props: itemProps, item }">
<v-list-item v-bind="itemProps" :title="item.raw.name">
<template #prepend>
<v-icon size="small">{{ iconFor(item.raw) }}</v-icon>
</template>
<template #subtitle>
{{ item.raw.kind === 'artist' ? 'artist'
: (item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind) }}
</template>
</v-list-item>
</template>
</v-autocomplete>
<div class="fc-filterbar__chips">
<v-chip
v-for="id in store.filter.tag_ids" :key="`t${id}`"
size="small" closable :color="chipColor(id)" variant="tonal"
@click:close="removeTag(id)"
>{{ store.tagLabels[id] || `#${id}` }}</v-chip>
<v-chip
v-if="store.filter.artist_id"
size="small" closable color="accent" variant="tonal"
prepend-icon="mdi-account"
@click:close="clearArtist"
>{{ store.artistLabel || `Artist #${store.filter.artist_id}` }}</v-chip>
<v-chip
v-if="store.filter.similar_to"
size="small" closable color="accent" variant="tonal"
prepend-icon="mdi-image-multiple"
@click:close="clearSimilar"
>Similar to #{{ store.filter.similar_to }}</v-chip>
</div>
<v-spacer />
<v-btn-toggle
:model-value="store.filter.media_type ?? 'all'"
density="compact" mandatory variant="outlined" divided
@update:model-value="(v) => setMedia(v === 'all' ? null : v)"
>
<v-btn value="all" size="small">All</v-btn>
<v-btn value="image" size="small">Images</v-btn>
<v-btn value="video" size="small">Videos</v-btn>
</v-btn-toggle>
<!-- Sort is meaningless in similar-mode (results are distance-ranked). -->
<v-select
v-if="!store.filter.similar_to"
:model-value="store.filter.sort"
:items="SORTS"
density="compact" hide-details variant="outlined"
class="fc-filterbar__sort"
@update:model-value="setSort"
/>
<v-btn
:color="refineOpen ? 'accent' : undefined"
:variant="refineOpen || hasRefineFilters ? 'tonal' : 'text'"
size="small"
:append-icon="refineOpen ? 'mdi-chevron-up' : 'mdi-chevron-down'"
@click="toggleRefine"
>Refine{{ refineCount ? ` (${refineCount})` : '' }}</v-btn>
<v-btn
v-if="hasActiveFilters" variant="text" size="small"
prepend-icon="mdi-close" @click="clearAll"
>Clear</v-btn>
</div>
<GalleryFacetPanel v-if="refineOpen" />
</div>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { useRouter } from 'vue-router'
import { useApi } from '../../composables/useApi.js'
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
import { useTagStore } from '../../stores/tags.js'
import GalleryFacetPanel from './GalleryFacetPanel.vue'
const store = useGalleryStore()
const tagStore = useTagStore()
const api = useApi()
const router = useRouter()
const SORTS = [
{ title: 'Newest first', value: 'newest' },
{ title: 'Oldest first', value: 'oldest' },
]
const selected = ref(null)
const searchItems = ref([])
const searchLoading = ref(false)
let debounce = null
// The faceted-refine sub-filters (platform / curation flags / date range).
const refineCount = computed(() => {
const f = store.filter
return (f.platform ? 1 : 0) + (f.untagged ? 1 : 0) + (f.no_artist ? 1 : 0)
+ (f.date_from ? 1 : 0) + (f.date_to ? 1 : 0)
})
const hasRefineFilters = computed(() => refineCount.value > 0)
const hasActiveFilters = computed(() =>
store.filter.tag_ids.length > 0 ||
store.filter.artist_id != null ||
store.filter.media_type != null ||
store.filter.sort !== 'newest' ||
store.filter.similar_to != null ||
hasRefineFilters.value
)
// Auto-open the panel when refine filters are present in the URL (deep-link /
// back button). The parent applies the query in its onMounted — after this
// child has set up — so watch for the transition rather than reading the
// initial (still-default) filter state.
const refineOpen = ref(false)
watch(hasRefineFilters, (v) => { if (v) refineOpen.value = true })
function toggleRefine() {
// The panel fetches facets itself on mount (and refetches on filter change);
// opening it is enough.
refineOpen.value = !refineOpen.value
}
function iconFor(raw) {
if (raw.kind === 'artist') return 'mdi-account'
return { character: 'mdi-account-circle', fandom: 'mdi-book-open-page-variant',
series: 'mdi-bookshelf' }[raw.kind] || 'mdi-tag'
}
function chipColor(id) {
// Tag chips use the same per-kind palette as the rest of the app; we only
// know the kind from the autocomplete pick, so fall back to a neutral tone.
return tagStore.colorFor(pickedKind.value[id] || 'general')
}
const pickedKind = ref({})
function onSearch(q) {
if (debounce) clearTimeout(debounce)
if (!q || !q.trim()) { searchItems.value = []; return }
debounce = setTimeout(async () => {
searchLoading.value = true
try {
const [tags, artists] = await Promise.all([
api.get('/api/tags/autocomplete', { params: { q, limit: 10 } }),
api.get('/api/artists/autocomplete', { params: { q, limit: 10 } }),
])
searchItems.value = [
...(artists || []).map((a) => ({
kind: 'artist', id: a.id, name: a.name, value: `artist:${a.id}`,
})),
...(tags || []).map((t) => ({
kind: t.kind, id: t.id, name: t.name, value: `tag:${t.id}`,
fandom_name: t.fandom_name,
})),
]
} catch {
searchItems.value = []
} finally {
searchLoading.value = false
}
}, 250)
}
function onPick(value) {
const item = searchItems.value.find((i) => i.value === value)
selected.value = null
searchItems.value = []
if (!item) return
if (item.kind === 'artist') {
store.noteArtistLabel(item.name)
pushFilter((n) => { n.artist_id = item.id })
} else {
store.noteTagLabel(item.id, item.name)
pickedKind.value = { ...pickedKind.value, [item.id]: item.kind }
pushFilter((n) => { if (!n.tag_ids.includes(item.id)) n.tag_ids.push(item.id) })
}
}
function removeTag(id) {
pushFilter((n) => { n.tag_ids = n.tag_ids.filter((t) => t !== id) })
}
function clearArtist() {
store.noteArtistLabel(null)
pushFilter((n) => { n.artist_id = null })
}
function clearSimilar() { pushFilter((n) => { n.similar_to = null }) }
function setMedia(m) { pushFilter((n) => { n.media_type = m }) }
function setSort(s) { pushFilter((n) => { n.sort = s }) }
function clearAll() { router.push({ name: 'gallery', query: {} }) }
// Single write path: clone the current filter, mutate, serialize to the URL.
// The route watcher in GalleryView applies it to the store and reloads.
// cloneFilter/filterToQuery are shared with GalleryFacetPanel so the refine
// sub-filters survive a bar push and vice versa.
function pushFilter(mutate) {
const n = cloneFilter(store.filter)
mutate(n)
router.push({ name: 'gallery', query: filterToQuery(n) })
}
</script>
<style scoped>
/* The whole chrome (bar row + expandable refine panel) is one sticky,
frosted block pinned directly under the 64px TopNav and continuous with it. */
.fc-filterbar-wrap {
position: sticky;
top: 64px;
z-index: 5;
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
so the bar sits flush at 64px even at scroll 0 — without this it detaches
and a gap shows through when scrolled to the top. */
margin-top: -8px;
margin-bottom: 12px;
/* EXACT same gradiated obsidian (#14171A = 20,23,26) frost as the TopNav so
the two read as one continuous piece of chrome — images scroll visibly
under both. The nav's gradient fades to transparent at ITS bottom; this
bar re-darkens at its top, so a faint seam (the page/image showing through
the nav's transparent edge) separates them when scrolled to the very top,
while under-scroll they frost as one. */
background: linear-gradient(
to bottom,
rgba(20, 23, 26, 0.92) 0%,
rgba(20, 23, 26, 0.65) 60%,
rgba(20, 23, 26, 0) 100%
);
backdrop-filter: blur(2px);
-webkit-backdrop-filter: blur(2px);
}
.fc-filterbar {
display: flex;
align-items: center;
gap: 10px;
flex-wrap: wrap;
padding: 10px 12px;
}
/* The inputs/toggles float on the haze: still translucent, but a touch more
opaque than the bar/nav so the controls stay legible. */
.fc-filterbar-wrap :deep(.v-field),
.fc-filterbar-wrap :deep(.v-btn-group) {
background-color: rgba(20, 23, 26, 0.72);
}
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
.fc-filterbar__sort { max-width: 150px; }
/* Phones: the search's 200px min-width jams the wrapping bar. Give search its
own full-width row and let sort grow; everything else wraps under it. */
@media (max-width: 600px) {
.fc-filterbar { gap: 8px; }
.fc-filterbar__search { min-width: 100%; max-width: none; }
.fc-filterbar__sort { max-width: none; flex: 1 1 auto; }
}
</style>
@@ -15,19 +15,6 @@
</div> </div>
</template> </template>
<!-- Similar-mode (and any non-date-grouped result set) returns no date
groups the results are ranked, not chronological. Render them as a
single flat list in their given order rather than nothing. -->
<div
v-if="!store.dateGroups.length && store.images.length"
class="fc-gallery-grid__items"
>
<GalleryItem
v-for="img in store.images"
:key="img.id" :image="img" @open="$emit('open', img.id)"
/>
</div>
<div v-if="store.loading" class="fc-gallery-grid__sentinel"> <div v-if="store.loading" class="fc-gallery-grid__sentinel">
<v-progress-circular indeterminate color="accent" size="28" /> <v-progress-circular indeterminate color="accent" size="28" />
</div> </div>
@@ -9,16 +9,11 @@
> >
<div class="fc-gallery-item__media"> <div class="fc-gallery-item__media">
<img <img
v-if="!isVideo" ref="imgEl" :src="image.thumbnail_url" v-if="!isVideo" :src="image.thumbnail_url" :alt="`Image ${image.id}`"
:alt="`Image ${image.id}`" loading="lazy" loading="lazy" @error="onThumbError"
:class="{ 'is-loaded': loaded }"
@load="loaded = true" @error="onThumbError"
> >
<div v-else class="fc-gallery-item__video-thumb"> <div v-else class="fc-gallery-item__video-thumb">
<img <img :src="image.thumbnail_url" :alt="`Video ${image.id}`" loading="lazy">
ref="imgEl" :src="image.thumbnail_url" :alt="`Video ${image.id}`"
loading="lazy" :class="{ 'is-loaded': loaded }" @load="loaded = true"
>
<v-icon class="fc-gallery-item__video-badge" icon="mdi-play-circle" /> <v-icon class="fc-gallery-item__video-badge" icon="mdi-play-circle" />
</div> </div>
<div v-if="thumbError" class="fc-gallery-item__placeholder"> <div v-if="thumbError" class="fc-gallery-item__placeholder">
@@ -43,7 +38,7 @@
</template> </template>
<script setup> <script setup>
import { computed, onMounted, ref } from 'vue' import { computed, ref } from 'vue'
import { useGallerySelectionStore } from '../../stores/gallerySelection.js' import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
const props = defineProps({ image: { type: Object, required: true } }) const props = defineProps({ image: { type: Object, required: true } })
@@ -51,18 +46,6 @@ const emit = defineEmits(['open'])
const sel = useGallerySelectionStore() const sel = useGallerySelectionStore()
const thumbError = ref(false) const thumbError = ref(false)
// Reveal each tile when ITS OWN thumbnail finishes loading (`@load`), not
// when its metadata batch lands — so tiles fade/flip in individually in
// load order instead of popping in together (operator-flagged 2026-06-04).
const loaded = ref(false)
const imgEl = ref(null)
onMounted(() => {
// A cached thumbnail can already be complete before @load binds; reflect
// that so the tile reveals instead of sitting invisible at opacity 0.
const el = imgEl.value
if (el && el.complete && el.naturalWidth > 0) loaded.value = true
})
const isVideo = computed( const isVideo = computed(
() => props.image.mime && props.image.mime.startsWith('video/') () => props.image.mime && props.image.mime.startsWith('video/')
) )
@@ -99,30 +82,6 @@ function onThumbError() { thumbError.value = true }
} }
.fc-gallery-item__media img { .fc-gallery-item__media img {
width: 100%; height: 100%; object-fit: cover; display: block; width: 100%; height: 100%; object-fit: cover; display: block;
opacity: 0;
}
/* Entrance: each thumbnail flips up out of a slight backward tilt and
settles into place, played WHEN ITS OWN image finishes loading (the
`is-loaded` class) rather than on a batch/timer — so tiles cascade in
natural load order instead of popping in together. Mirrors the showcase
MasonryGrid entrance styling (operator-flagged 2026-06-04). */
.fc-gallery-item__media img.is-loaded {
animation: fc-gallery-item-in 0.5s cubic-bezier(0.34, 1.45, 0.64, 1) both;
}
@keyframes fc-gallery-item-in {
0% {
opacity: 0;
transform: perspective(1000px) rotateX(-22deg) translateY(20px) scale(0.96);
}
55% { opacity: 1; }
100% {
opacity: 1;
transform: perspective(1000px) rotateX(0) translateY(0) scale(1);
}
}
@media (prefers-reduced-motion: reduce) {
.fc-gallery-item__media img { opacity: 1; }
.fc-gallery-item__media img.is-loaded { animation: none; }
} }
.fc-gallery-item__video-thumb { position: relative; height: 100%; } .fc-gallery-item__video-thumb { position: relative; height: 100%; }
.fc-gallery-item__video-badge { .fc-gallery-item__video-badge {
+1 -22
View File
@@ -2,17 +2,12 @@
<v-card> <v-card>
<v-card-title>Pick a fandom</v-card-title> <v-card-title>Pick a fandom</v-card-title>
<v-card-text> <v-card-text>
<!-- A2/A3 (2026-06-07): autofocus so the operator types immediately, and
picking a fandom (keyboard: type arrow Enter, or a click) confirms
in one step selecting IS the decision in this dialog. -->
<v-autocomplete <v-autocomplete
v-model="selectedId" v-model="selectedId"
:items="store.fandomCache" :items="store.fandomCache"
:item-title="(f) => f.name" :item-title="(f) => f.name"
:item-value="(f) => f.id" :item-value="(f) => f.id"
label="Fandom" clearable density="compact" label="Fandom" clearable density="compact"
autofocus
@update:model-value="onSelect"
/> />
<v-divider class="my-3" /> <v-divider class="my-3" />
<p class="text-caption mb-2">Or create a new fandom:</p> <p class="text-caption mb-2">Or create a new fandom:</p>
@@ -22,10 +17,6 @@
</div> </div>
</v-card-text> </v-card-text>
<v-card-actions> <v-card-actions>
<!-- Not every character belongs to a fandom (original characters,
unsorted, etc.). "No fandom" creates the character unassigned;
a fandom can still be set later from the chip's kebab menu. -->
<v-btn variant="text" @click="onNoFandom">No fandom</v-btn>
<v-spacer /> <v-spacer />
<v-btn @click="$emit('cancel')">Cancel</v-btn> <v-btn @click="$emit('cancel')">Cancel</v-btn>
<v-btn :disabled="!selectedId" color="primary" rounded="pill" @click="onConfirm">Use this fandom</v-btn> <v-btn :disabled="!selectedId" color="primary" rounded="pill" @click="onConfirm">Use this fandom</v-btn>
@@ -43,8 +34,7 @@ const store = useTagStore()
const selectedId = ref(null) const selectedId = ref(null)
const newName = ref('') const newName = ref('')
// Always refresh on open so the list reflects fandoms created elsewhere (#712). onMounted(() => { if (store.fandomCache.length === 0) store.loadFandoms() })
onMounted(() => store.loadFandoms())
async function onCreate() { async function onCreate() {
const f = await store.createFandom(newName.value.trim()) const f = await store.createFandom(newName.value.trim())
@@ -55,15 +45,4 @@ function onConfirm() {
const f = store.fandomCache.find(x => x.id === selectedId.value) const f = store.fandomCache.find(x => x.id === selectedId.value)
if (f) emit('confirm', f) if (f) emit('confirm', f)
} }
// Picking a fandom (keyboard Enter or click) confirms immediately. Ignore the
// clear action (null) so clearing the field doesn't fire a confirm.
function onSelect(id) {
if (id == null) return
onConfirm()
}
// Create the character with no fandom. Emits null so the caller knows this
// was a deliberate "unassigned", not a cancel.
function onNoFandom() {
emit('confirm', null)
}
</script> </script>
@@ -1,130 +0,0 @@
<template>
<v-card>
<v-card-title class="text-body-1">Fandom for {{ tag.name }}</v-card-title>
<v-card-text>
<template v-if="!collision">
<v-autocomplete
v-model="selectedId"
:items="store.fandomCache"
:item-title="(f) => f.name" :item-value="(f) => f.id"
label="Fandom" clearable density="compact"
autofocus
:hint="selectedId == null
? 'No fandom — the character will be unassigned.' : ''"
persistent-hint
/>
<v-divider class="my-3" />
<p class="text-caption mb-2">Or create a new fandom:</p>
<div class="d-flex" style="gap: 8px;">
<v-text-field
v-model="newName" placeholder="New fandom name"
density="compact" hide-details
@keydown.enter.prevent="onCreate"
/>
<v-btn
:disabled="!newName.trim() || busy" rounded="pill"
@click="onCreate"
>Create</v-btn>
</div>
<v-alert
v-if="error" type="error" variant="tonal" density="compact"
class="mt-3"
>{{ error }}</v-alert>
</template>
<template v-else>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
A character named “{{ tag.name }}” already exists in that fandom.
</v-alert>
<p class="text-body-2">
Merge this tag into “{{ collision.target.name }}”?
{{ collision.source_image_count }} image
association{{ collision.source_image_count === 1 ? '' : 's' }}
will move over and this tag will be deleted{{
collision.will_alias ? ' (its name kept as a tagger alias)' : '' }}.
</p>
</template>
</v-card-text>
<v-card-actions>
<v-spacer />
<template v-if="!collision">
<v-btn variant="text" :disabled="busy" @click="$emit('cancel')">
Cancel
</v-btn>
<v-btn
color="primary" rounded="pill" :loading="busy"
:disabled="selectedId === (tag.fandom_id ?? null)"
@click="onSave"
>Save</v-btn>
</template>
<template v-else>
<v-btn variant="text" :disabled="busy" @click="collision = null">
Back
</v-btn>
<v-btn
color="warning" variant="flat" rounded="pill" :loading="busy"
@click="onConfirmMerge"
>Merge</v-btn>
</template>
</v-card-actions>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { useTagStore } from '../../stores/tags.js'
const props = defineProps({ tag: { type: Object, required: true } })
const emit = defineEmits(['updated', 'cancel'])
const store = useTagStore()
const selectedId = ref(props.tag.fandom_id ?? null)
const newName = ref('')
const busy = ref(false)
const error = ref(null)
const collision = ref(null)
// Always refresh on open so the list reflects fandoms created elsewhere (#712).
onMounted(() => store.loadFandoms())
async function onCreate() {
const name = newName.value.trim()
if (!name) return
busy.value = true
error.value = null
try {
const f = await store.createFandom(name)
selectedId.value = f.id
newName.value = ''
} catch (e) {
error.value = e.message || String(e)
} finally {
busy.value = false
}
}
async function save(merge) {
busy.value = true
error.value = null
try {
const body = await store.setFandom(props.tag.id, selectedId.value, { merge })
emit('updated', body)
} catch (e) {
// 409 on first attempt → surface the merge confirmation; cross-fandom
// collisions can't go through the regular /merge endpoint, so the
// resolution is a second setFandom with merge: true.
if (!merge && e.status === 409 && e.body && e.body.target) {
collision.value = e.body
} else {
error.value = e.message || String(e)
collision.value = null
}
} finally {
busy.value = false
}
}
function onSave() { save(false) }
function onConfirmMerge() { save(true) }
</script>
+22 -131
View File
@@ -8,25 +8,6 @@
<v-icon>mdi-close</v-icon> <v-icon>mdi-close</v-icon>
</button> </button>
<!-- C8: keyboard cheatsheet. '?' toggles; the corner hint advertises it. -->
<button
class="fc-viewer__help-hint" aria-label="Keyboard shortcuts (press ?)"
@click="showHelp = !showHelp"
>?</button>
<div v-if="showHelp" class="fc-viewer__help" @click.self="showHelp = false">
<div class="fc-viewer__help-card" role="dialog" aria-label="Keyboard shortcuts">
<h3>Keyboard shortcuts</h3>
<dl>
<div><dt> / </dt><dd>Previous / next image</dd></div>
<div><dt>T or /</dt><dd>Jump to the tag input</dd></div>
<div><dt> / </dt><dd>Move through tag suggestions</dd></div>
<div><dt>Enter / Tab</dt><dd>Accept the highlighted tag</dd></div>
<div><dt>Esc</dt><dd>Close a dialog, then the viewer</dd></div>
<div><dt>?</dt><dd>Toggle this help</dd></div>
</dl>
</div>
</div>
<v-chip <v-chip
v-if="integrityBadge" v-if="integrityBadge"
:color="integrityBadge.color" :color="integrityBadge.color"
@@ -70,9 +51,6 @@
<aside v-if="modal.current" class="fc-viewer__side"> <aside v-if="modal.current" class="fc-viewer__side">
<ProvenancePanel /> <ProvenancePanel />
<TagPanel /> <TagPanel />
<!-- Non-blocking: fetches its own similar set after the modal is up;
collapses silently if empty/slow/failed (see RelatedStrip). -->
<RelatedStrip />
</aside> </aside>
</div> </div>
</div> </div>
@@ -82,18 +60,15 @@
<script setup> <script setup>
import { computed, onMounted, onUnmounted, ref, watch } from 'vue' import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
import { useModalStore } from '../../stores/modal.js' import { useModalStore } from '../../stores/modal.js'
import { arrowNavAllowed, isTextEntry } from '../../utils/textEntry.js'
import ImageCanvas from './ImageCanvas.vue' import ImageCanvas from './ImageCanvas.vue'
import VideoCanvas from './VideoCanvas.vue' import VideoCanvas from './VideoCanvas.vue'
import TagPanel from './TagPanel.vue' import TagPanel from './TagPanel.vue'
import ProvenancePanel from './ProvenancePanel.vue' import ProvenancePanel from './ProvenancePanel.vue'
import RelatedStrip from './RelatedStrip.vue'
const emit = defineEmits(['close']) const emit = defineEmits(['close'])
const modal = useModalStore() const modal = useModalStore()
const rootEl = ref(null) const rootEl = ref(null)
const showHelp = ref(false)
const isVideo = computed(() => const isVideo = computed(() =>
modal.current?.mime && modal.current.mime.startsWith('video/') modal.current?.mime && modal.current.mime.startsWith('video/')
@@ -116,47 +91,27 @@ let prevBodyOverflow = null
// that. Filtered via isTextEntry so tag/comment inputs still get their // that. Filtered via isTextEntry so tag/comment inputs still get their
// own keystrokes. // own keystrokes.
function onKeyDown(ev) { function onKeyDown(ev) {
if (ev.key === 'Escape' && showHelp.value) {
// Close the cheatsheet first; a second Esc closes the modal.
ev.preventDefault()
showHelp.value = false
return
}
if (ev.key === 'Escape') { if (ev.key === 'Escape') {
// Escape closes the modal even from inside a text input — the universal // Escape closes the modal even from inside a text input — that's
// "get me out of here" expectation; the autofocused tag field would // the universal "get me out of here" expectation, and the
// otherwise trap focus (operator-flagged 2026-06-01). EXCEPTION: when the // autofocused tag-entry field would otherwise trap focus with no
// keystroke originates INSIDE an open Vuetify overlay's content (a rename/ // visible escape (operator-flagged 2026-06-01). EXCEPTION: when a
// fandom/alias dialog, or a kebab menu), let that overlay handle its own // nested Vuetify overlay is open (v-menu autocomplete dropdown,
// Esc and don't close the whole modal. // FandomPicker v-dialog, per-suggestion 3-dot menu), let that
// // overlay's own Esc handling fire instead of closing the whole
// #700 re-fix (2026-06-07): the prior guard queried for ANY active overlay // modal mid-interaction. Vuetify marks open overlays with
// anywhere in the DOM and suppressed the close — so a lingering overlay // `.v-overlay--active`.
// after accepting a suggestion (focus drops to <body>, not into any if (document.querySelector('.v-overlay--active')) return
// overlay) wrongly blocked Esc. Keying off the event's origin instead means
// a stray overlay no longer traps the modal: only an Esc pressed from
// within overlay content defers to that overlay.
if (ev.target?.closest?.('.v-overlay__content')) return
ev.preventDefault() ev.preventDefault()
emit('close') emit('close')
} else if (ev.key === 'ArrowLeft') { } else if (ev.key === 'ArrowLeft') {
// Navigate unless the caret is in a non-empty text field (then let it move if (isTextEntry(ev.target)) return
// through the text). An empty tag-entry field still navigates.
if (!arrowNavAllowed(ev.target)) return
ev.preventDefault() ev.preventDefault()
modal.goPrev() modal.goPrev()
} else if (ev.key === 'ArrowRight') { } else if (ev.key === 'ArrowRight') {
if (!arrowNavAllowed(ev.target)) return if (isTextEntry(ev.target)) return
ev.preventDefault() ev.preventDefault()
modal.goNext() modal.goNext()
} else if ((ev.key === '/' || ev.key === 't') && !isTextEntry(ev.target)) {
// C9 (2026-06-07): jump focus to the tag input from anywhere in the modal.
const input = document.querySelector('.fc-tag-autocomplete input')
if (input) { ev.preventDefault(); input.focus() }
} else if (ev.key === '?' && !isTextEntry(ev.target)) {
// C8: toggle the keyboard cheatsheet.
ev.preventDefault()
showHelp.value = !showHelp.value
} }
} }
@@ -180,14 +135,17 @@ watch(() => modal.currentImageId, async () => {
function nextFrame() { function nextFrame() {
return new Promise(resolve => requestAnimationFrame(resolve)) return new Promise(resolve => requestAnimationFrame(resolve))
} }
function isTextEntry(el) {
if (!el) return false
const tag = el.tagName
return tag === 'INPUT' || tag === 'TEXTAREA' || el.isContentEditable
}
</script> </script>
<style scoped> <style scoped>
.fc-viewer { .fc-viewer {
position: fixed; inset: 0; z-index: 2000; position: fixed; inset: 0; z-index: 2000;
/* Single source of truth for the metadata side-panel width — the next
arrow offsets off it so it never overlaps the panel. */
--fc-side-w: 320px;
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav, /* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
mid-opacity + blur so the page behind shows through faintly. */ mid-opacity + blur so the page behind shows through faintly. */
background: rgba(20, 23, 26, 0.65); background: rgba(20, 23, 26, 0.65);
@@ -212,53 +170,11 @@ function nextFrame() {
} }
.fc-viewer__nav:disabled { opacity: 0.3; cursor: not-allowed; } .fc-viewer__nav:disabled { opacity: 0.3; cursor: not-allowed; }
.fc-viewer__close { top: 16px; right: 16px; transform: none; } .fc-viewer__close { top: 16px; right: 16px; transform: none; }
.fc-viewer__help-hint {
position: absolute; top: 16px; right: 64px; z-index: 2;
width: 32px; height: 32px; border-radius: 50%;
background: rgba(255, 255, 255, 0.08);
color: rgb(var(--v-theme-on-surface));
font-weight: 700; cursor: pointer; opacity: 0.6;
}
.fc-viewer__help-hint:hover { opacity: 1; }
.fc-viewer__help {
position: absolute; inset: 0; z-index: 4;
display: flex; align-items: center; justify-content: center;
background: rgba(0, 0, 0, 0.4);
}
.fc-viewer__help-card {
background: rgb(var(--v-theme-surface));
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 10px; padding: 20px 24px; min-width: 320px;
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.5);
}
.fc-viewer__help-card h3 {
font-family: 'Fraunces', Georgia, serif;
font-size: 18px; margin-bottom: 12px;
color: rgb(var(--v-theme-on-surface));
}
.fc-viewer__help-card dl { display: flex; flex-direction: column; gap: 8px; }
.fc-viewer__help-card dl > div {
display: flex; gap: 16px; align-items: baseline;
}
.fc-viewer__help-card dt {
flex: 0 0 96px; text-align: right;
font-family: 'JetBrains Mono', monospace; font-size: 12px;
color: rgb(var(--v-theme-accent));
}
.fc-viewer__help-card dd {
flex: 1; font-size: 14px;
color: rgb(var(--v-theme-on-surface));
}
.fc-viewer__integrity { .fc-viewer__integrity {
position: absolute; top: 72px; right: 16px; z-index: 3; position: absolute; top: 72px; right: 16px; z-index: 3;
} }
.fc-viewer__nav--prev { left: 16px; transform: translateY(-50%); } .fc-viewer__nav--prev { left: 16px; transform: translateY(-50%); }
/* Sit just inside the image area, clear of the metadata side panel — .fc-viewer__nav--next { right: 16px; transform: translateY(-50%); }
not floating over it (operator-flagged 2026-06-04). */
.fc-viewer__nav--next {
right: calc(var(--fc-side-w) + 16px);
transform: translateY(-50%);
}
.fc-viewer__body { .fc-viewer__body {
flex: 1; display: flex; min-height: 0; flex: 1; display: flex; min-height: 0;
} }
@@ -271,44 +187,19 @@ function nextFrame() {
min-width: 0; min-height: 0; min-width: 0; min-height: 0;
} }
.fc-viewer__side { .fc-viewer__side {
width: var(--fc-side-w); flex-shrink: 0; width: 320px; flex-shrink: 0;
background: rgb(var(--v-theme-surface)); background: rgb(var(--v-theme-surface));
border-left: 1px solid rgb(var(--v-theme-surface-light)); border-left: 1px solid rgb(var(--v-theme-surface-light));
overflow-y: auto; overflow-y: auto;
} }
@media (max-width: 900px) { @media (max-width: 900px) {
/* Stacked layout (operator-flagged 2026-06-05): pin the image pane and .fc-viewer__body { flex-direction: column; }
its controls at the top while the metadata panel scrolls beneath it.
Previously the image + a 40vh panel shared one fixed viewport and the
controls could be pushed out of view; now the body scrolls and the
media is sticky, so the image + prev/next/close (and the integrity
badge) stay visible no matter how far the panel is scrolled. */
.fc-viewer__body {
flex-direction: column;
overflow-y: auto;
}
.fc-viewer__media {
flex: none;
height: 55vh;
position: sticky;
top: 0;
z-index: 1;
/* Opaque obsidian so the scrolling panel never bleeds through the
haze behind the pinned image. */
background: rgb(20, 23, 26);
}
.fc-viewer__side { .fc-viewer__side {
width: 100%; width: 100%;
max-height: none; max-height: 40vh;
border-left: none; border-left: none;
border-top: 1px solid rgb(var(--v-theme-surface-light)); border-top: 1px solid rgb(var(--v-theme-surface-light));
} }
/* Re-center the prev/next arrows over the 55vh image band (their base
top:50% would land on the scrolling panel); next uses the full width
now that the panel is below, not beside. Close + integrity badge keep
their top:16px/72px — already within the pinned band. */
.fc-viewer__nav { top: 27.5vh; }
.fc-viewer__nav--next { right: 16px; }
} }
</style> </style>
@@ -1,133 +0,0 @@
<template>
<!-- Collapses entirely unless the source has an embedding AND we're loading
or have results — so a slow/empty/failed fetch leaves no trace and never
affects the modal. -->
<div v-if="show" class="fc-related">
<div class="fc-related__head">
<span class="fc-related__title">Related</span>
<v-btn
size="x-small" variant="text" color="accent"
:disabled="loading || !results.length"
@click="seeAll"
>See all similar</v-btn>
</div>
<div class="fc-related__row">
<template v-if="loading">
<div v-for="n in 6" :key="n" class="fc-related__skel" />
</template>
<button
v-for="img in results" :key="img.id"
class="fc-related__item" type="button"
@click="openImage(img.id)"
>
<img :src="img.thumbnail_url" :alt="`image ${img.id}`" loading="lazy" />
</button>
</div>
</div>
</template>
<script setup>
import { computed, onBeforeUnmount, ref, watch } from 'vue'
import { useRouter } from 'vue-router'
import { useApi } from '../../composables/useApi.js'
import { useModalStore } from '../../stores/modal.js'
// Deferred so the modal's main image gets network/decode priority the strip
// is a nice-to-have and must never block or slow the modal load.
const DEFER_MS = 200
const STRIP_LIMIT = 12
const api = useApi()
const router = useRouter()
const modal = useModalStore()
const results = ref([])
const loading = ref(false)
const hasEmbedding = computed(() => modal.current?.has_embedding === true)
const show = computed(() => hasEmbedding.value && (loading.value || results.value.length > 0))
let seq = 0
let timer = null
async function fetchSimilar(id) {
const mine = ++seq
loading.value = true
results.value = []
try {
const body = await api.get('/api/gallery/similar', {
params: { similar_to: id, limit: STRIP_LIMIT },
})
if (mine !== seq) return
results.value = body.images || []
} catch {
if (mine === seq) results.value = [] // quietly collapse on error
} finally {
if (mine === seq) loading.value = false
}
}
// Re-fetch whenever the modal lands on a new embedded image. modal.current is
// null while the next image loads, so this also clears the strip during
// prev/next nav and repopulates once the new payload arrives.
watch(
() => (hasEmbedding.value ? modal.current?.id : null),
(id) => {
if (timer) { clearTimeout(timer); timer = null }
seq++ // cancel any in-flight fetch
results.value = []
loading.value = false
if (!id) return
timer = setTimeout(() => fetchSimilar(id), DEFER_MS)
},
{ immediate: true },
)
onBeforeUnmount(() => { if (timer) clearTimeout(timer) })
function openImage(id) {
modal.open(id)
}
function seeAll() {
const id = modal.current?.id
if (!id) return
modal.close()
router.push({ name: 'gallery', query: { similar_to: String(id) } })
}
</script>
<style scoped>
.fc-related {
padding: 12px;
border-top: 1px solid rgb(var(--v-theme-surface-light));
}
.fc-related__head {
display: flex; align-items: center; justify-content: space-between;
margin-bottom: 8px;
}
.fc-related__title {
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-related__row {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 6px;
}
.fc-related__item {
display: block; padding: 0; border: 0; background: none;
cursor: pointer; border-radius: 4px; overflow: hidden;
aspect-ratio: 1; width: 100%;
}
.fc-related__item img {
width: 100%; height: 100%; object-fit: cover; display: block;
background: rgb(var(--v-theme-surface-light));
transition: transform 0.2s ease, filter 0.2s ease;
}
.fc-related__item:hover img { transform: scale(1.05); filter: brightness(1.1); }
.fc-related__skel {
aspect-ratio: 1; border-radius: 4px;
background: rgb(var(--v-theme-surface-light));
opacity: 0.5;
}
</style>
@@ -19,25 +19,24 @@
> >
Accept Accept
</v-btn> </v-btn>
<!-- Operator-flagged 2026-06-04: the kebab still wasn't opening. The <!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
prior `#activator` + `v-bind="props"` path never toggled the menu Wrapping in a <span @click.stop> matches the TagPanel chip
inside this teleported modal, while v-model-driven overlays (the fix — even though there's no parent click capture here today,
dialogs in this modal) work fine. So drive the menu explicitly: the wrap is harmless and keeps both kebabs on the same
the button toggles `menuOpen` with @click.stop (also shields any pattern. Click bubbles from the v-btn opens menu via
parent), and `activator="parent"` anchors the menu for positioning activator props bubble continues to span stopPropagation
only — `:open-on-click="false"` keeps Vuetify's activator-click out halts it. -->
of it, so there's a single, reliable opener. --> <span class="fc-suggestion__menu-wrap" @click.stop>
<span class="fc-suggestion__menu-wrap"> <v-menu>
<v-btn <template #activator="{ props }">
class="fc-suggestion__menu" <v-btn
icon="mdi-dots-vertical" size="small" class="fc-suggestion__menu"
variant="outlined" density="compact" icon="mdi-dots-vertical" size="small"
:aria-label="`More actions for ${suggestion.display_name}`" variant="outlined" density="compact"
@click.stop="menuOpen = !menuOpen" :aria-label="`More actions for ${suggestion.display_name}`"
/> v-bind="props"
<v-menu />
v-model="menuOpen" activator="parent" :open-on-click="false" </template>
>
<v-list density="compact"> <v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)"> <v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title> <v-list-item-title>Treat as alias for</v-list-item-title>
@@ -52,12 +51,11 @@
</template> </template>
<script setup> <script setup>
import { computed, ref } from 'vue' import { computed } from 'vue'
const props = defineProps({ suggestion: { type: Object, required: true } }) const props = defineProps({ suggestion: { type: Object, required: true } })
defineEmits(['accept', 'alias', 'dismiss']) defineEmits(['accept', 'alias', 'dismiss'])
const menuOpen = ref(false)
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`) const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
</script> </script>
@@ -8,12 +8,10 @@
@keydown.down.prevent="moveHighlight(1)" @keydown.down.prevent="moveHighlight(1)"
@keydown.up.prevent="moveHighlight(-1)" @keydown.up.prevent="moveHighlight(-1)"
@keydown.enter.prevent="onEnter" @keydown.enter.prevent="onEnter"
@keydown.tab="onTab"
@keydown.esc="$emit('cancel')" @keydown.esc="$emit('cancel')"
/> />
<v-list <v-list
v-if="hits.length || allowCreate" v-if="hits.length || allowCreate"
ref="listRef"
density="compact" class="fc-tag-autocomplete__list" density="compact" class="fc-tag-autocomplete__list"
> >
<v-list-item <v-list-item
@@ -67,17 +65,8 @@ const store = useTagStore()
// Vuetify's v-text-field exposes .focus() on the component instance; // Vuetify's v-text-field exposes .focus() on the component instance;
// nextTick waits for the modal's mount to finish so the inner <input> // nextTick waits for the modal's mount to finish so the inner <input>
// element exists. // element exists.
//
// EXCEPTION — not on mobile (operator-flagged 2026-06-05): focusing the
// field pops the soft keyboard, which shrinks the visual viewport and
// shoves the pinned image + its nav/close controls out of view. 900px is
// the modal's stacked-layout breakpoint (ImageViewer.vue). matchMedia is
// safe on plain HTTP (not a secure-context API).
const inputRef = ref(null) const inputRef = ref(null)
onMounted(() => { onMounted(() => { nextTick(() => inputRef.value?.focus?.()) })
if (window.matchMedia?.('(max-width: 900px)')?.matches) return
nextTick(() => inputRef.value?.focus?.())
})
// Single text input; no kind dropdown. Client-side mirror of the // Single text input; no kind dropdown. Client-side mirror of the
// backend's parse_kind_prefix lives below — kept in sync with // backend's parse_kind_prefix lives below — kept in sync with
@@ -87,7 +76,6 @@ onMounted(() => {
const query = ref('') const query = ref('')
const hits = ref([]) const hits = ref([])
const highlight = ref(0) const highlight = ref(0)
const listRef = ref(null)
const fandomDialog = ref(false) const fandomDialog = ref(false)
let pendingNewName = null let pendingNewName = null
@@ -140,14 +128,6 @@ function moveHighlight (delta) {
const total = hits.value.length + (allowCreate.value ? 1 : 0) const total = hits.value.length + (allowCreate.value ? 1 : 0)
if (total === 0) return if (total === 0) return
highlight.value = (highlight.value + delta + total) % total highlight.value = (highlight.value + delta + total) % total
// Keep the highlighted row visible — the list is capped at 240px and arrowing
// past the fold otherwise left the active item off-screen (operator-flagged
// 2026-06-07). block:'nearest' scrolls the minimum needed.
nextTick(() => {
listRef.value?.$el
?.querySelector('.v-list-item--active')
?.scrollIntoView({ block: 'nearest' })
})
} }
function onPick (hit) { emit('pick-existing', hit); reset() } function onPick (hit) { emit('pick-existing', hit); reset() }
@@ -167,14 +147,10 @@ function onCreate () {
reset() reset()
} }
// fandom is null when the user picked "No fandom" — characters don't all
// belong to a fandom. The backend already accepts fandom_id: null for the
// character kind (tag.kind check + nullable fandom_id), and a fandom can be
// assigned later from the chip kebab's "Set fandom…".
function onFandomChosen (fandom) { function onFandomChosen (fandom) {
fandomDialog.value = false fandomDialog.value = false
emit('pick-new', { emit('pick-new', {
name: pendingNewName, kind: 'character', fandom_id: fandom ? fandom.id : null, name: pendingNewName, kind: 'character', fandom_id: fandom.id,
}) })
pendingNewName = null pendingNewName = null
reset() reset()
@@ -188,16 +164,6 @@ function onEnter () {
} }
} }
// B5 (2026-06-07): when the suggestion list is open, Tab accepts the
// highlighted row (standard autocomplete convention) instead of leaving the
// field. With the list closed it falls through to normal focus traversal.
function onTab (e) {
if (hits.value.length || allowCreate.value) {
e.preventDefault()
onEnter()
}
}
function reset () { query.value = ''; hits.value = []; highlight.value = 0 } function reset () { query.value = ''; hits.value = []; highlight.value = 0 }
</script> </script>
-61
View File
@@ -1,61 +0,0 @@
<template>
<!-- One tag chip + its kebab. The kebab uses the SAME explicit-menu pattern
as SuggestionItem (activator="parent" + :open-on-click="false" + a manual
v-model), because the #activator / v-bind="props" pattern never toggles a
v-menu inside the teleported ImageViewer modal (#711, re-fixed 2026-06-07
after the first attempt used the broken pattern). -->
<span class="fc-tag-chip">
<v-chip
size="small" closable
:color="store.colorFor(tag.kind)" variant="tonal"
@click:close="$emit('remove', tag.id)"
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
</v-chip>
<span class="fc-tag-chip__menu-wrap">
<v-btn
class="fc-tag-chip__kebab"
icon="mdi-dots-vertical" size="x-small"
variant="text" density="comfortable"
:aria-label="`More actions for ${tag.name}`"
@click.stop="menuOpen = !menuOpen"
/>
<v-menu v-model="menuOpen" activator="parent" :open-on-click="false">
<v-list density="compact">
<v-list-item @click="$emit('rename', tag)">
<v-list-item-title>Rename</v-list-item-title>
</v-list-item>
<v-list-item v-if="tag.kind === 'character'" @click="$emit('set-fandom', tag)">
<v-list-item-title>Set fandom</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</span>
</template>
<script setup>
import { ref } from 'vue'
import { useTagStore } from '../../stores/tags.js'
defineProps({ tag: { type: Object, required: true } })
defineEmits(['remove', 'rename', 'set-fandom'])
const store = useTagStore()
const menuOpen = ref(false)
const KIND_ICONS = {
general: 'mdi-tag', character: 'mdi-account-circle',
fandom: 'mdi-book-open-page-variant', series: 'mdi-bookshelf',
meta: 'mdi-cog-outline', rating: 'mdi-shield-check-outline',
}
function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
</script>
<style scoped>
.fc-tag-chip { display: inline-flex; align-items: center; gap: 1px; }
.fc-tag-chip__menu-wrap { display: inline-flex; align-items: center; }
.fc-tag-chip__kebab { opacity: 0.7; }
.fc-tag-chip:hover .fc-tag-chip__kebab { opacity: 1; }
</style>
+39 -25
View File
@@ -2,11 +2,36 @@
<aside class="fc-tag-panel" aria-label="Tags for this image"> <aside class="fc-tag-panel" aria-label="Tags for this image">
<h3 class="fc-tag-panel__title">Tags</h3> <h3 class="fc-tag-panel__title">Tags</h3>
<div class="fc-tag-panel__chips"> <div class="fc-tag-panel__chips">
<TagChip <v-chip
v-for="tag in modal.current?.tags || []" v-for="tag in modal.current?.tags || []"
:key="tag.id" :tag="tag" :key="tag.id" size="small" closable
@remove="onRemove" @rename="openRename" @set-fandom="openSetFandom" :color="store.colorFor(tag.kind)" variant="tonal"
/> @click:close="onRemove(tag.id)"
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
<!-- Operator-flagged 2026-06-02: the previous activator had
`@click.stop` directly on the v-icon, which silently
overrode Vuetify's onClick from `v-bind="mp"` — the menu
never opened. Now the v-icon receives the activator
onClick cleanly, and the wrapping span absorbs the
bubbled click so the chip's close button isn't tripped. -->
<span class="kebab-wrap" @click.stop>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical"
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename…</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</v-chip>
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span> <span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
</div> </div>
@@ -32,28 +57,28 @@
@renamed="onRenamed" @cancel="renameDialog = false" @renamed="onRenamed" @cancel="renameDialog = false"
/> />
</v-dialog> </v-dialog>
<v-dialog v-model="fandomDialog" max-width="460">
<FandomSetDialog
v-if="fandomTarget" :tag="fandomTarget"
@updated="onFandomUpdated" @cancel="fandomDialog = false"
/>
</v-dialog>
</aside> </aside>
</template> </template>
<script setup> <script setup>
import { ref } from 'vue' import { ref } from 'vue'
import { useModalStore } from '../../stores/modal.js' import { useModalStore } from '../../stores/modal.js'
import TagChip from './TagChip.vue' import { useTagStore } from '../../stores/tags.js'
import TagAutocomplete from './TagAutocomplete.vue' import TagAutocomplete from './TagAutocomplete.vue'
import SuggestionsPanel from './SuggestionsPanel.vue' import SuggestionsPanel from './SuggestionsPanel.vue'
import TagRenameDialog from './TagRenameDialog.vue' import TagRenameDialog from './TagRenameDialog.vue'
import FandomSetDialog from './FandomSetDialog.vue'
const modal = useModalStore() const modal = useModalStore()
const store = useTagStore()
const errorMsg = ref(null) const errorMsg = ref(null)
const KIND_ICONS = {
general: 'mdi-tag', character: 'mdi-account-circle',
fandom: 'mdi-book-open-page-variant', series: 'mdi-bookshelf',
meta: 'mdi-cog-outline', rating: 'mdi-shield-check-outline'
}
function iconFor(k) { return KIND_ICONS[k] || 'mdi-tag' }
async function onRemove(tagId) { async function onRemove(tagId) {
errorMsg.value = null errorMsg.value = null
try { await modal.removeTag(tagId) } try { await modal.removeTag(tagId) }
@@ -82,18 +107,6 @@ async function onRenamed() {
// Reflect the new name in the modal's current tag list without a full reload. // Reflect the new name in the modal's current tag list without a full reload.
await modal.reloadTags() await modal.reloadTags()
} }
const fandomDialog = ref(false)
const fandomTarget = ref(null)
function openSetFandom(tag) {
fandomTarget.value = tag
fandomDialog.value = true
}
async function onFandomUpdated() {
fandomDialog.value = false
// A fandom change can merge the tag away; reload to reflect the new state.
await modal.reloadTags()
}
</script> </script>
<style scoped> <style scoped>
@@ -108,4 +121,5 @@ async function onFandomUpdated() {
margin-bottom: 12px; margin-bottom: 12px;
} }
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; } .fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
.kebab-wrap { display: inline-flex; align-items: center; }
</style> </style>
+208 -186
View File
@@ -1,18 +1,29 @@
<template> <template>
<v-card class="fc-post-card" variant="outlined"> <v-card
:class="['fc-post-card', expanded && 'fc-post-card--expanded']"
variant="outlined"
:tabindex="expanded ? -1 : 0"
@click="onCardClick"
@keydown.enter="onCardClick"
>
<div class="fc-post-card__head"> <div class="fc-post-card__head">
<!-- Posts with no live subscription have source=null (alembic 0030); <!-- Posts with no live subscription have source=null (alembic
show a "filesystem import" affordance instead of a platform chip. --> 0030); show a "filesystem import" affordance instead of a
platform chip. -->
<v-chip size="x-small" variant="tonal"> <v-chip size="x-small" variant="tonal">
{{ post.source?.platform ?? 'filesystem import' }} {{ post.source?.platform ?? 'filesystem import' }}
</v-chip> </v-chip>
<RouterLink <RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }" :to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-card__artist" class="fc-post-card__artist"
@click.stop
>{{ post.artist.name }}</RouterLink> >{{ post.artist.name }}</RouterLink>
<span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span> <span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span>
<span v-if="totalImages" class="fc-post-card__meta"> <span v-if="expanded && images.length" class="fc-post-card__meta">
· {{ totalImages }} image{{ totalImages === 1 ? '' : 's' }} · {{ images.length }} image{{ images.length === 1 ? '' : 's' }}
</span>
<span v-if="expanded && attachments.length" class="fc-post-card__meta">
· {{ attachments.length }} attachment{{ attachments.length === 1 ? '' : 's' }}
</span> </span>
<v-spacer /> <v-spacer />
<v-btn <v-btn
@@ -20,129 +31,142 @@
:href="post.post_url" target="_blank" rel="noopener" :href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="x-small" variant="text" icon="mdi-open-in-new" size="x-small" variant="text"
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`" :aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
@click.stop
/>
<v-btn
:icon="expanded ? 'mdi-chevron-up' : 'mdi-chevron-down'"
size="x-small" variant="text"
:aria-label="expanded ? 'Collapse post' : 'Expand post'"
@click.stop="toggleExpanded"
/> />
</div> </div>
<div class="fc-post-card__body"> <!-- Compact body: collapsed card. Hero + thumb rail + truncated text. -->
<div v-if="!expanded" class="fc-post-card__body">
<div class="fc-post-card__media"> <div class="fc-post-card__media">
<template v-if="images.length"> <template v-if="images.length">
<!-- Images open the post-scoped image modal (look bigger + arrow <div class="fc-post-card__hero">
through ALL the post's images) the card never expands. --> <img :src="hero.thumbnail_url" :alt="`hero thumbnail`" loading="lazy" />
<button </div>
type="button" class="fc-post-card__hero" <div v-if="rail.length" class="fc-post-card__rail">
aria-label="Open images" @click="openModal(hero.image_id)" <div v-for="t in rail" :key="t.image_id" class="fc-post-card__rail-cell">
> <img :src="t.thumbnail_url" :alt="`thumbnail`" loading="lazy" />
<img :src="hero.thumbnail_url" alt="hero thumbnail" loading="lazy" /> </div>
</button> <div v-if="moreCount > 0" class="fc-post-card__rail-more">
<div +{{ moreCount }}
v-if="rail.length || moreCount" class="fc-post-card__rail" </div>
:style="{ '--fc-rail-cols': railCols }"
>
<button
v-for="t in rail" :key="t.image_id" type="button"
class="fc-post-card__rail-cell"
aria-label="Open image" @click="openModal(t.image_id)"
>
<img :src="t.thumbnail_url" alt="thumbnail" loading="lazy" />
</button>
<button
v-if="moreCount > 0" type="button"
class="fc-post-card__rail-more"
:aria-label="`Open ${moreCount} more images`"
@click="openModalAtMore"
>+{{ moreCount }}</button>
</div> </div>
</template> </template>
<PostEmptyThumbs v-else /> <PostEmptyThumbs v-else />
</div> </div>
<div class="fc-post-card__text"> <div class="fc-post-card__text">
<h3 v-if="plainTitle" class="fc-post-card__title">{{ plainTitle }}</h3> <h3 v-if="plainTitle" class="fc-post-card__title">
{{ plainTitle }}
</h3>
<h3 v-else class="fc-post-card__title fc-post-card__title--missing"> <h3 v-else class="fc-post-card__title fc-post-card__title--missing">
Post {{ post.external_post_id }} Post {{ post.external_post_id }}
</h3> </h3>
<p <p v-if="post.description_plain" class="fc-post-card__desc">
v-if="hasDescription" ref="descEl" {{ post.description_plain }}
class="fc-post-card__desc" </p>
:class="{ 'fc-post-card__desc--clamped': !descExpanded }"
>{{ descText }}</p>
<p v-else class="fc-post-card__desc fc-post-card__desc--missing"> <p v-else class="fc-post-card__desc fc-post-card__desc--missing">
(no description) (no description)
</p> </p>
<!-- The ONLY in-place expansion: the post text, and only when it's <div v-if="post.attachments?.length" class="fc-post-card__atts">
actually truncated (server flag or a CSS-clamp overflow). --> <v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
<button {{ post.attachments.length }} attachment{{ post.attachments.length === 1 ? '' : 's' }}
v-if="canExpand" type="button" class="fc-post-card__more" </div>
@click="toggleDesc" </div>
>{{ descExpanded ? 'Show less' : 'Show more' }}</button> </div>
<div v-if="attachments.length" class="fc-post-card__atts"> <!-- Expanded body: title, full mosaic, full sanitized HTML description,
attachments. Lazy-loaded detail via getPostFull. -->
<div v-else class="fc-post-card__expanded">
<h2 v-if="plainTitle" class="fc-post-card__title-full">
{{ plainTitle }}
</h2>
<h2 v-else class="fc-post-card__title-full fc-post-card__title--missing">
Post {{ post.external_post_id }}
</h2>
<section v-if="images.length" class="fc-post-card__sec">
<PostImageGrid :thumbnails="images" />
<div v-if="!detailLoaded" class="fc-post-card__loading-hint">
Loading full image list
</div>
</section>
<section v-if="descriptionHtml" class="fc-post-card__sec">
<div class="fc-post-card__desc-full" v-html="descriptionHtml" />
</section>
<section v-else-if="detailLoaded" class="fc-post-card__sec">
<p class="fc-post-card__desc fc-post-card__desc--missing">(no description)</p>
</section>
<section v-if="attachments.length" class="fc-post-card__sec">
<h3 class="fc-post-card__h3">Attachments</h3>
<div class="fc-post-card__atts-full">
<a <a
v-for="att in attachments" :key="att.id" v-for="att in attachments" :key="att.id"
:href="att.download_url" download class="fc-post-card__att" :href="att.download_url" download
class="fc-post-card__att"
@click.stop
> >
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon> <v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
<span>{{ att.original_filename }}</span> <span>{{ att.original_filename }}</span>
<span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span> <span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span>
</a> </a>
</div> </div>
</div> </section>
</div> </div>
</v-card> </v-card>
</template> </template>
<script setup> <script setup>
import { computed, nextTick, onBeforeUnmount, onMounted, ref } from 'vue' import { computed, ref } from 'vue'
import { RouterLink } from 'vue-router' import { RouterLink } from 'vue-router'
import { useModalStore } from '../../stores/modal.js'
import { usePostsStore } from '../../stores/posts.js' import { usePostsStore } from '../../stores/posts.js'
import { toPlainText } from '../../utils/htmlSanitize.js' import { sanitizeHtml, toPlainText } from '../../utils/htmlSanitize.js'
import PostEmptyThumbs from './PostEmptyThumbs.vue' import PostEmptyThumbs from './PostEmptyThumbs.vue'
import PostImageGrid from './PostImageGrid.vue'
const props = defineProps({ const props = defineProps({
post: { type: Object, required: true }, post: { type: Object, required: true },
}) })
const postsStore = usePostsStore() const postsStore = usePostsStore()
const modal = useModalStore()
// Full detail (uncapped thumbnails + full description), fetched lazily — only // Per-card expand state. No global modal — each PostCard owns its own
// when opening the modal for a post with >6 images, or expanding a // view-mode and lazy-loaded detail.
// server-truncated description. const expanded = ref(false)
const detail = ref(null) const detail = ref(null)
const detailLoaded = ref(false)
const detailError = ref(null)
const attachments = computed(() => props.post.attachments || []) // When expanded + detail loaded, prefer the uncapped detail thumbnails +
const images = computed(() => props.post.thumbnails || []) // full description. Falls back to feed shape if detail fetch is in flight
const totalImages = computed(() => images.value.length + (props.post.thumbnails_more || 0)) // or failed.
const merged = computed(() => detail.value || props.post)
const images = computed(() => merged.value.thumbnails || [])
const attachments = computed(() => merged.value.attachments || [])
// Titles can arrive as stored HTML (e.g. "<strong>…</strong>"); render as
// plain text — the CSS makes the title bold.
const plainTitle = computed(() => toPlainText(props.post.post_title)) const plainTitle = computed(() => toPlainText(props.post.post_title))
const hero = computed(() => images.value[0]) // Compact-view hero+rail derived from the feed-shape (capped 6).
// The thumbnail strip spans the hero's full width (CSS grid, equal columns), const hero = computed(() => props.post.thumbnails?.[0])
// rather than a fixed 3-cell cap. Show up to RAIL_MAX cells; when there are const rail = computed(() => (props.post.thumbnails || []).slice(1, 4))
// more images than fit, the last cell becomes a "+N" overflow tile so the
// count stays accurate.
const RAIL_MAX = 5
const serverMore = computed(() => props.post.thumbnails_more || 0)
const afterHero = computed(() => images.value.slice(1))
const hasOverflow = computed(
() => serverMore.value > 0 || afterHero.value.length > RAIL_MAX,
)
const rail = computed(() =>
hasOverflow.value
? afterHero.value.slice(0, RAIL_MAX - 1)
: afterHero.value.slice(0, RAIL_MAX),
)
const visibleCount = computed(() => (images.value.length ? 1 + rail.value.length : 0))
const moreCount = computed(() => { const moreCount = computed(() => {
if (!hasOverflow.value) return 0 const more = props.post.thumbnails_more || 0
return serverMore.value + Math.max(0, images.value.length - visibleCount.value) const railLen = rail.value.length
const extraShown = Math.max(0, (props.post.thumbnails?.length || 0) - 1 - railLen)
return more + extraShown
}) })
// Columns = thumbnails shown plus the overflow tile, so the strip fills the
// hero's full width with equal cells.
const railCols = computed(() => rail.value.length + (moreCount.value > 0 ? 1 : 0))
const sortDateIso = computed(() => props.post.post_date || props.post.downloaded_at) const sortDateIso = computed(() => props.post.post_date || props.post.downloaded_at)
const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString()) const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString())
@@ -156,79 +180,51 @@ const relativeDate = computed(() => {
return new Date(sortDateIso.value).toLocaleDateString() return new Date(sortDateIso.value).toLocaleDateString()
}) })
// --- images → post-scoped modal --------------------------------------- const descriptionHtml = computed(() => {
async function fullImageIds () { // Detail endpoint returns description_full as plain text (the service
// Feed caps thumbnails at 6; load detail for the complete id list only when // uses html_to_plain on the stored description). Render plain text in
// there are more, so the modal can arrow through ALL of the post's images. // <p> wrappers; sanitize defensively in case the backend ever returns
if ((props.post.thumbnails_more || 0) === 0) { // raw HTML.
return images.value.map((t) => t.image_id) const raw = merged.value.description_full || merged.value.description_plain
} if (!raw) return ''
if (!detail.value) { if (/[<>]/.test(raw)) return sanitizeHtml(raw)
try { const esc = raw
detail.value = await postsStore.getPostFull(props.post.id) .replace(/&/g, '&amp;')
} catch { /* fall back to the capped feed list */ } .replace(/</g, '&lt;')
} .replace(/>/g, '&gt;')
return (detail.value?.thumbnails || images.value).map((t) => t.image_id) return esc
} .split(/\n\s*\n/)
.map((p) => `<p>${p.replace(/\n/g, '<br>')}</p>`)
async function openModal (imageId) { .join('')
modal.open(imageId, { postImageIds: await fullImageIds() })
}
async function openModalAtMore () {
const ids = await fullImageIds()
const first = ids[visibleCount.value] ?? ids[0]
if (first != null) modal.open(first, { postImageIds: ids })
}
// --- description "Show more" (text-only, in place, only when truncated) ----
const descExpanded = ref(false)
const cssOverflow = ref(false)
const descEl = ref(null)
const hasDescription = computed(() => !!props.post.description_plain)
const fullDescription = computed(() => detail.value?.description_full || null)
const descText = computed(() =>
descExpanded.value
? (fullDescription.value || props.post.description_plain)
: props.post.description_plain,
)
// Show the toggle iff the server truncated the text OR the clamp is cutting it.
const canExpand = computed(
() => props.post.description_truncated === true || cssOverflow.value,
)
function measureOverflow () {
const el = descEl.value
cssOverflow.value = !!el && el.scrollHeight > el.clientHeight + 1
}
let ro = null
onMounted(() => {
nextTick(measureOverflow)
// Re-measure when the card resizes (the container-query clamp differs by
// width). Guarded for happy-dom / older runtimes without ResizeObserver.
if (typeof ResizeObserver !== 'undefined' && descEl.value) {
ro = new ResizeObserver(() => { if (!descExpanded.value) measureOverflow() })
ro.observe(descEl.value)
}
}) })
onBeforeUnmount(() => { if (ro) { ro.disconnect(); ro = null } })
async function toggleDesc () { async function loadDetailIfNeeded () {
if (!descExpanded.value) { if (detailLoaded.value || detail.value) return
if (props.post.description_truncated && !fullDescription.value && !detail.value) { try {
try { detail.value = await postsStore.getPostFull(props.post.id)
detail.value = await postsStore.getPostFull(props.post.id) detailLoaded.value = true
} catch { /* render the truncated text rather than nothing */ } } catch (e) {
} detailError.value = e.message
descExpanded.value = true // Leave merged on feed-shape; the card still renders the truncated
} else { // body so the operator isn't staring at a blank panel.
descExpanded.value = false
nextTick(measureOverflow)
} }
} }
function toggleExpanded () {
expanded.value = !expanded.value
if (expanded.value) loadDetailIfNeeded()
}
function onCardClick (e) {
// Inner interactive elements use @click.stop so they never reach here.
// Whole-card click expands a collapsed card; collapsing is chevron-only
// so a mosaic-image click on an expanded card can never accidentally
// collapse the surrounding card.
if (expanded.value) return
expanded.value = true
loadDetailIfNeeded()
}
function formatBytes (n) { function formatBytes (n) {
if (!n) return '0 B' if (!n) return '0 B'
if (n < 1024) return `${n} B` if (n < 1024) return `${n} B`
@@ -243,6 +239,20 @@ function formatBytes (n) {
padding: 1rem; padding: 1rem;
margin-bottom: 1rem; margin-bottom: 1rem;
container-type: inline-size; container-type: inline-size;
transition: border-color 0.15s ease;
}
.fc-post-card:not(.fc-post-card--expanded) {
cursor: pointer;
}
.fc-post-card:not(.fc-post-card--expanded):hover {
border-color: rgb(var(--v-theme-accent));
}
.fc-post-card:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 2px;
}
.fc-post-card--expanded {
border-color: rgb(var(--v-theme-accent) / 0.6);
} }
.fc-post-card__head { .fc-post-card__head {
@@ -263,6 +273,7 @@ function formatBytes (n) {
.fc-post-card__date, .fc-post-card__date,
.fc-post-card__meta { white-space: nowrap; } .fc-post-card__meta { white-space: nowrap; }
/* ---- COMPACT BODY ---- */
.fc-post-card__body { .fc-post-card__body {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
@@ -277,13 +288,6 @@ function formatBytes (n) {
.fc-post-card__text { flex: 1 1 0; min-width: 0; } .fc-post-card__text { flex: 1 1 0; min-width: 0; }
} }
/* Image tiles are buttons (open the post-scoped modal) — reset button chrome. */
.fc-post-card__hero,
.fc-post-card__rail-cell,
.fc-post-card__rail-more {
display: block; padding: 0; border: 0; background: none;
cursor: pointer;
}
.fc-post-card__hero { .fc-post-card__hero {
width: 100%; width: 100%;
aspect-ratio: 16 / 10; aspect-ratio: 16 / 10;
@@ -293,21 +297,13 @@ function formatBytes (n) {
.fc-post-card__hero img { .fc-post-card__hero img {
width: 100%; height: 100%; width: 100%; height: 100%;
object-fit: cover; display: block; object-fit: cover; display: block;
transition: transform 0.2s ease, filter 0.2s ease;
} }
.fc-post-card__hero:hover img,
.fc-post-card__rail-cell:hover img { transform: scale(1.03); filter: brightness(1.08); }
/* Equal columns spanning the hero's full width — cells stretch horizontally
(1fr) at a fixed height, so the strip always reaches the edge of the hero
regardless of how many thumbnails the post has. */
.fc-post-card__rail { .fc-post-card__rail {
display: grid; display: flex; gap: 6px; margin-top: 6px;
grid-template-columns: repeat(var(--fc-rail-cols, 3), 1fr);
gap: 6px; margin-top: 6px;
} }
.fc-post-card__rail-cell { .fc-post-card__rail-cell {
width: 100%; height: 84px; width: 80px; height: 80px;
overflow: hidden; border-radius: 4px; overflow: hidden; border-radius: 4px;
} }
.fc-post-card__rail-cell img { .fc-post-card__rail-cell img {
@@ -315,17 +311,13 @@ function formatBytes (n) {
object-fit: cover; display: block; object-fit: cover; display: block;
} }
.fc-post-card__rail-more { .fc-post-card__rail-more {
width: 100%; height: 84px; width: 80px; height: 80px;
display: flex; align-items: center; justify-content: center; display: flex; align-items: center; justify-content: center;
border: 1px dashed rgb(var(--v-theme-on-surface-variant)); border: 1px dashed rgb(var(--v-theme-on-surface-variant));
border-radius: 4px; border-radius: 4px;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem; font-size: 0.85rem;
} }
.fc-post-card__rail-more:hover {
border-color: rgb(var(--v-theme-accent));
color: rgb(var(--v-theme-accent));
}
.fc-post-card__title { .fc-post-card__title {
font-family: 'Fraunces', Georgia, serif; font-family: 'Fraunces', Georgia, serif;
@@ -354,36 +346,67 @@ function formatBytes (n) {
font-size: 0.9rem; font-size: 0.9rem;
line-height: 1.5; line-height: 1.5;
color: rgb(var(--v-theme-on-surface)); color: rgb(var(--v-theme-on-surface));
margin: 0; margin: 0 0 12px 0;
white-space: pre-wrap;
}
/* Clamp ONLY while collapsed; expanding drops the clamp to show it all. */
.fc-post-card__desc--clamped {
display: -webkit-box; display: -webkit-box;
-webkit-line-clamp: 3; -webkit-line-clamp: 3;
-webkit-box-orient: vertical; -webkit-box-orient: vertical;
overflow: hidden; overflow: hidden;
} }
@container (min-width: 800px) {
.fc-post-card__desc--clamped { -webkit-line-clamp: 5; }
}
.fc-post-card__desc--missing { .fc-post-card__desc--missing {
font-style: italic; font-style: italic;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
} }
@container (min-width: 800px) {
.fc-post-card__more { .fc-post-card__desc { -webkit-line-clamp: 5; }
margin-top: 6px;
padding: 0;
background: none; border: 0; cursor: pointer;
color: rgb(var(--v-theme-accent));
font-size: 0.85rem; font-weight: 600;
} }
.fc-post-card__more:hover { text-decoration: underline; }
.fc-post-card__atts { .fc-post-card__atts {
display: inline-flex;
align-items: center;
gap: 6px;
font-size: 0.85rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
/* ---- EXPANDED BODY ---- */
.fc-post-card__expanded {
display: flex;
flex-direction: column;
gap: 20px;
}
.fc-post-card__title-full {
font-family: 'Fraunces', Georgia, serif;
font-size: 22px;
font-weight: 700;
margin: 0;
color: rgb(var(--v-theme-on-surface));
}
@container (min-width: 800px) {
.fc-post-card__title-full { font-size: 26px; }
}
.fc-post-card__sec { margin: 0; }
.fc-post-card__h3 {
font-family: 'Fraunces', Georgia, serif;
font-size: 16px;
font-weight: 500;
margin: 0 0 8px 0;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-card__loading-hint {
margin-top: 8px;
font-size: 0.8rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__desc-full {
font-size: 0.95rem;
line-height: 1.55;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-card__desc-full :deep(p) { margin: 0 0 12px 0; }
.fc-post-card__desc-full :deep(a) { color: rgb(var(--v-theme-accent)); }
.fc-post-card__atts-full {
display: flex; flex-wrap: wrap; gap: 8px; display: flex; flex-wrap: wrap; gap: 8px;
margin-top: 12px;
} }
.fc-post-card__att { .fc-post-card__att {
display: inline-flex; display: inline-flex;
@@ -400,6 +423,5 @@ function formatBytes (n) {
color: rgb(var(--v-theme-accent)); color: rgb(var(--v-theme-accent));
border-color: rgb(var(--v-theme-accent)); border-color: rgb(var(--v-theme-accent));
} }
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); } .fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
</style> </style>
@@ -0,0 +1,63 @@
<template>
<div class="fc-post-grid">
<button
v-for="(t, idx) in thumbnails"
:key="t.image_id"
type="button"
class="fc-post-grid__cell"
:aria-label="`Open image ${idx + 1} of ${thumbnails.length}`"
@click="openImage(t.image_id, idx)"
>
<img
:src="t.thumbnail_url"
:alt="`thumbnail ${idx + 1}`"
loading="lazy"
/>
</button>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { useModalStore } from '../../stores/modal.js'
const props = defineProps({
thumbnails: { type: Array, required: true }, // [{ image_id, thumbnail_url, ... }]
})
const modal = useModalStore()
const imageIds = computed(() => props.thumbnails.map(t => t.image_id))
function openImage (id, idx) {
modal.open(id, { postImageIds: imageIds.value, initialIndex: idx })
}
</script>
<style scoped>
.fc-post-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 6px;
}
.fc-post-grid__cell {
aspect-ratio: 4 / 3;
overflow: hidden;
border-radius: 4px;
cursor: pointer;
border: 0;
padding: 0;
background: rgb(var(--v-theme-background));
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.fc-post-grid__cell:hover {
transform: scale(1.02);
box-shadow: 0 0 0 2px rgb(var(--v-theme-accent));
}
.fc-post-grid__cell img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
</style>
@@ -13,7 +13,7 @@
clearable clearable
no-filter no-filter
return-object return-object
class="fc-posts-filters__artist" style="min-width: 240px"
@update:search="onArtistSearch" @update:search="onArtistSearch"
@update:model-value="onArtistPicked" @update:model-value="onArtistPicked"
/> />
@@ -25,7 +25,7 @@
density="compact" density="compact"
hide-details hide-details
clearable clearable
class="fc-posts-filters__platform" style="min-width: 180px"
@update:model-value="emitFilters" @update:model-value="emitFilters"
/> />
@@ -135,14 +135,6 @@ watch(() => props.platform, (val) => {
display: flex; display: flex;
gap: 0.75rem; gap: 0.75rem;
align-items: center; align-items: center;
flex-wrap: wrap;
padding-bottom: 1rem; padding-bottom: 1rem;
} }
.fc-posts-filters__artist { min-width: 240px; flex: 1 1 240px; max-width: 360px; }
.fc-posts-filters__platform { min-width: 180px; flex: 1 1 180px; max-width: 240px; }
/* Phones: each field takes a full-width row instead of overflowing. */
@media (max-width: 600px) {
.fc-posts-filters__artist,
.fc-posts-filters__platform { min-width: 100%; max-width: none; flex-basis: 100%; }
}
</style> </style>
@@ -1,46 +0,0 @@
<template>
<!-- #713: re-extract PostAttachments that are really archives but were filed
opaquely before the magic-byte gate, and attach their images to the post. -->
<v-card>
<v-card-title>Re-extract archive attachments</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3">
Some posts attached an archive (zip) whose images weren't extracted
because the downloaded file had no usable extension. This scans existing
attachments, extracts any that are really archives, and attaches their
images to the post. Idempotent — safe to run more than once.
</p>
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
<v-icon start>mdi-folder-zip-outline</v-icon> Re-extract archives now
</v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
</v-card-text>
</v-card>
</template>
<script setup>
import { ref } from 'vue'
import { useApi } from '../../composables/useApi.js'
import { toast } from '../../utils/toast.js'
import QueueStatusBar from './QueueStatusBar.vue'
const api = useApi()
const busy = ref(false)
const queued = ref(false)
async function run () {
busy.value = true
queued.value = false
try {
await api.post('/api/admin/maintenance/reextract-archives')
queued.value = true
toast({ text: 'Archive re-extract queued', type: 'success' })
} catch (e) {
toast({ text: e?.body?.detail || e?.message || 'Failed to queue', type: 'error' })
} finally {
busy.value = false
}
}
</script>
@@ -1,84 +0,0 @@
<template>
<v-card>
<v-card-title>Database maintenance</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3">
VACUUM (ANALYZE) reclaims dead-tuple bloat which slows the random
showcase, since it samples physical blocks and refreshes the query
planner's statistics. Runs automatically each week; trigger a pass
here after a large import or cleanup.
</p>
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
<v-icon start>mdi-database-cog</v-icon> Run VACUUM ANALYZE now
</v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
<v-table
v-if="store.tables.length" density="compact" class="mt-4 fc-dbm__table"
>
<thead>
<tr>
<th>Table</th>
<th class="text-right">Live rows</th>
<th class="text-right">Dead</th>
<th class="text-right">Dead %</th>
<th>Last vacuum</th>
</tr>
</thead>
<tbody>
<tr v-for="t in store.tables" :key="t.table">
<td>{{ t.table }}</td>
<td class="text-right">{{ t.live.toLocaleString() }}</td>
<td class="text-right">{{ t.dead.toLocaleString() }}</td>
<td
class="text-right"
:class="{ 'text-warning': t.dead_pct >= 20 }"
>{{ t.dead_pct }}%</td>
<td class="text-caption">{{ fmt(t.last_vacuum || t.last_autovacuum) }}</td>
</tr>
</tbody>
</v-table>
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
No table statistics yet.
</p>
</v-card-text>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { toast } from '../../utils/toast.js'
import { useDbMaintenanceStore } from '../../stores/dbMaintenance.js'
import QueueStatusBar from './QueueStatusBar.vue'
const store = useDbMaintenanceStore()
const busy = ref(false)
const queued = ref(false)
onMounted(() => store.loadStats())
async function run () {
busy.value = true
queued.value = false
try {
await store.runVacuum()
queued.value = true
// pg_stat updates after the vacuum lands — refresh shortly after.
setTimeout(() => store.loadStats(), 5000)
} catch (e) {
toast({ text: e.message, type: 'error' })
} finally {
busy.value = false
}
}
function fmt (iso) {
return iso ? new Date(iso).toLocaleString() : ''
}
</script>
<style scoped>
.fc-dbm__table { background: transparent; }
</style>
@@ -2,42 +2,6 @@
<v-card> <v-card>
<v-card-title>Import filters</v-card-title> <v-card-title>Import filters</v-card-title>
<v-card-text v-if="store.settings"> <v-card-text v-if="store.settings">
<!-- Near-duplicate dedup sensitivity, hoisted to the top: it's the
most-asked knob too loose and edits/variants of the same image
get dropped as duplicates on import. Slider with sane labelled
stops for the gist + a number field for precision; both bind the
same phash_threshold. -->
<div class="fc-phash">
<div class="fc-phash__title">Near-duplicate sensitivity</div>
<div class="fc-help mb-1">
How aggressively imports merge look-alike images (perceptual-hash
distance). <strong>Lower it if edits/variants of the same image are
being dropped as duplicates;</strong> raise it to collapse more
look-alikes. Applies to new imports.
</div>
<v-row align="center" no-gutters>
<v-col cols="12" sm="9">
<v-slider
v-model="local.phash_threshold"
:min="0" :max="16" :step="1"
:ticks="PHASH_TICKS" show-ticks="always" tick-size="4"
thumb-label color="accent" hide-details
class="fc-phash__slider"
@end="save"
/>
</v-col>
<v-col cols="8" sm="3" class="ps-sm-4 mt-2 mt-sm-0">
<v-text-field
v-model.number="local.phash_threshold"
label="Distance" type="number" min="0"
density="compact" hide-details @blur="save"
/>
</v-col>
</v-row>
</div>
<v-divider class="my-5" />
<v-row> <v-row>
<v-col cols="12" sm="6"> <v-col cols="12" sm="6">
<v-text-field <v-text-field
@@ -77,6 +41,14 @@
:disabled="!local.skip_single_color" @end="save" :disabled="!local.skip_single_color" @end="save"
/> />
</v-col> </v-col>
<v-col cols="12" sm="6">
<v-text-field
v-model.number="local.phash_threshold"
label="Perceptual-hash threshold" type="number" min="0"
density="compact" hide-details @blur="save"
/>
<div class="fc-help">Higher = looser near-duplicate matching.</div>
</v-col>
</v-row> </v-row>
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable> <v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
@@ -94,9 +66,6 @@ import { reactive, watch } from 'vue'
import { useImportStore } from '../../stores/import.js' import { useImportStore } from '../../stores/import.js'
const store = useImportStore() const store = useImportStore()
// Labelled stops so the less-initiated get the gist without knowing what a
// Hamming distance is. 0 = byte-for-byte only; 10 = the shipped default.
const PHASH_TICKS = { 0: 'Exact', 4: 'Strict', 10: 'Default', 16: 'Loose' }
// Downloader + schedule-defaults fields moved to // Downloader + schedule-defaults fields moved to
// /subscriptions?tab=settings (operator decision 2026-05-27). This form // /subscriptions?tab=settings (operator decision 2026-05-27). This form
// now only owns image-import filters. // now only owns image-import filters.
@@ -120,11 +89,4 @@ async function save() {
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
margin-top: 2px; margin-top: 2px;
} }
.fc-phash__title {
font-size: 0.95rem;
font-weight: 600;
color: rgb(var(--v-theme-on-surface));
}
/* Headroom so the tick labels (Exact/Strict/Default/Loose) aren't clipped. */
.fc-phash__slider { margin-bottom: 18px; }
</style> </style>
@@ -14,8 +14,6 @@
<MLThresholdSliders class="mt-4" /> <MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" /> <AllowlistTable class="mt-4" />
<AliasTable class="mt-4" /> <AliasTable class="mt-4" />
<DbMaintenanceCard class="mt-6" />
<ArchiveReextractCard class="mt-6" />
<BackupCard class="mt-6" /> <BackupCard class="mt-6" />
<!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) it <!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) it
operates on the existing library which fits the Cleanup-tab operates on the existing library which fits the Cleanup-tab
@@ -33,8 +31,6 @@ import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue' import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue' import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue' import AliasTable from './AliasTable.vue'
import DbMaintenanceCard from './DbMaintenanceCard.vue'
import ArchiveReextractCard from './ArchiveReextractCard.vue'
import BackupCard from './BackupCard.vue' import BackupCard from './BackupCard.vue'
import { useSystemActivityStore } from '../../stores/systemActivity.js' import { useSystemActivityStore } from '../../stores/systemActivity.js'
@@ -89,107 +89,6 @@
@click="onKindCommit" @click="onKindCommit"
>Delete {{ kindPreview.count }} legacy tag(s)</v-btn> >Delete {{ kindPreview.count }} legacy tag(s)</v-btn>
</div> </div>
<v-divider class="my-5" />
<p class="text-body-2 mb-2">
<strong class="text-error">Reset content tagging.</strong>
Deletes every <code>general</code> and <code>character</code> tag and
removes them from every image, so you can re-tag from scratch with the
auto-suggest. <strong>Fandoms and series (with their page order) are
kept</strong>, and each image's saved predictions are untouched open
an image and its suggestions reappear.
</p>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
Irreversible there's no undo except restoring a DB backup.
Back one up first (Settings → Maintenance → Backup).
</v-alert>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="loadingResetPreview"
class="mb-3"
@click="onResetPreview"
>Preview content-tag reset</v-btn>
<div v-if="resetPreview">
<p class="text-body-2 mb-2">
<strong>{{ resetPreview.count }}</strong> content tag(s)
<span v-for="(n, k) in resetPreview.by_kind" :key="k" class="fc-muted">
({{ k }}: {{ n }})&nbsp;
</span>
across <strong>{{ resetPreview.applications }}</strong> image
application(s).
</p>
<div v-if="resetPreview.sample_names?.length" class="fc-name-grid mb-3">
<span v-for="n in resetPreview.sample_names" :key="n" class="fc-name">
{{ n }}
</span>
</div>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-alert"
:disabled="!resetPreview.count"
:loading="resetCommitting"
@click="onResetCommit"
>Delete {{ resetPreview.count }} content tag(s) +
{{ resetPreview.applications }} application(s)</v-btn>
</div>
<v-divider class="my-5" />
<!-- #714: standardize existing tags to the canonical Title-Case form
and merge case/whitespace-variant duplicates. -->
<p class="text-body-2 mb-2">
<strong>Standardize tag casing.</strong>
New tags are saved Title Case, but older tags keep whatever casing they
were created with. This renames every existing tag to
<code>Title Case</code> (collapsing extra spaces) and
<strong>merges</strong> tags that differ only by case or spacing
(e.g. <code>hatsune miku</code> + <code>Hatsune&nbsp;&nbsp;Miku</code>)
into one — repointing their images, allowlist entries and series pages.
</p>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
The merges are irreversible — back up first
(Settings → Maintenance → Backup). Safe to run more than once.
</v-alert>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="loadingNormPreview"
class="mb-3"
@click="onNormPreview"
>Preview tag standardization</v-btn>
<div v-if="normPreview">
<p class="text-body-2 mb-2">
<strong>{{ normPreview.total_changes }}</strong> tag group(s) to
change — <strong>{{ normPreview.tags_to_rename }}</strong> rename(s),
<strong>{{ normPreview.collisions }}</strong> collision(s) merging
<strong>{{ normPreview.tags_to_merge }}</strong> tag(s) away.
</p>
<div v-if="normPreview.sample?.length" class="fc-name-grid mb-3">
<span
v-for="s in normPreview.sample" :key="s.to + s.kind"
class="fc-name"
>
<template v-if="s.merge">{{ s.from.join(' + ') }} → </template>{{ s.to }}
</span>
</div>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-format-letter-case"
:disabled="!normPreview.total_changes"
:loading="normCommitting"
@click="onNormCommit"
>Standardize {{ normPreview.total_changes }} tag group(s)</v-btn>
<span v-if="normResult === 'queued'" class="ml-3 text-caption text-success">
Queued ✓ — runs in the background (you can leave this page). It
processes in chunks, so re-run “Preview” later to confirm it's all done.
</span>
</div>
</v-card-text> </v-card-text>
</v-card> </v-card>
</template> </template>
@@ -206,13 +105,6 @@ const committing = ref(false)
const kindPreview = ref(null) const kindPreview = ref(null)
const loadingKindPreview = ref(false) const loadingKindPreview = ref(false)
const kindCommitting = ref(false) const kindCommitting = ref(false)
const resetPreview = ref(null)
const loadingResetPreview = ref(false)
const resetCommitting = ref(false)
const normPreview = ref(null)
const loadingNormPreview = ref(false)
const normCommitting = ref(false)
const normResult = ref(null)
async function onPreview() { async function onPreview() {
loadingPreview.value = true loadingPreview.value = true
@@ -251,51 +143,6 @@ async function onKindCommit() {
kindCommitting.value = false kindCommitting.value = false
} }
} }
async function onResetPreview() {
loadingResetPreview.value = true
try {
resetPreview.value = await store.resetContentTagging({ dryRun: true })
} finally {
loadingResetPreview.value = false
}
}
async function onResetCommit() {
resetCommitting.value = true
try {
await store.resetContentTagging({ dryRun: false })
resetPreview.value = { count: 0, by_kind: {}, applications: 0, sample_names: [] }
} finally {
resetCommitting.value = false
}
}
async function onNormPreview() {
loadingNormPreview.value = true
normResult.value = null
try {
normPreview.value = await store.normalizeTags({ dryRun: true })
} finally {
loadingNormPreview.value = false
}
}
async function onNormCommit() {
normCommitting.value = true
normResult.value = null
try {
// Fire-and-forget: the task is time-boxed and self-resuming across chunks
// (a large back-catalog can't finish in one run), so we DON'T poll-until-
// done — that would falsely report "complete" after the first chunk. Just
// confirm it's queued; the operator can re-run Preview later to verify.
await store.normalizeTags({ dryRun: false })
normResult.value = 'queued'
normPreview.value = { total_changes: 0, tags_to_rename: 0, collisions: 0, tags_to_merge: 0, sample: [] }
} finally {
normCommitting.value = false
}
}
</script> </script>
<style scoped> <style scoped>
@@ -23,18 +23,6 @@
<span class="fc-active__dot" /> <span class="fc-active__dot" />
<PlatformChip :platform="e.platform" size="x-small" /> <PlatformChip :platform="e.platform" size="x-small" />
<span class="fc-active__artist">{{ e.artist_name || '—' }}</span> <span class="fc-active__artist">{{ e.artist_name || '—' }}</span>
<!-- #709: live per-file progress for native (Patreon) downloads. -->
<span v-if="e.live" class="fc-active__counts">
<span class="fc-active__count" title="downloaded"> {{ e.live.downloaded }}</span>
<span v-if="e.live.skipped" class="fc-active__count" title="skipped">
{{ e.live.skipped }}</span>
<span
v-if="e.live.errors" class="fc-active__count fc-active__count--err"
title="errors"
> {{ e.live.errors }}</span>
<span class="fc-active__count fc-active__count--posts" title="posts scanned">
{{ e.live.posts }} posts</span>
</span>
<v-spacer /> <v-spacer />
<span class="fc-active__timer" :title="`started ${e.started_at}`"> <span class="fc-active__timer" :title="`started ${e.started_at}`">
{{ elapsed(e.started_at) }} {{ elapsed(e.started_at) }}
@@ -135,13 +123,6 @@ function elapsed (startedIso) {
font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em; font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
} }
.fc-active__counts {
display: inline-flex; align-items: center; gap: 8px;
font-variant-numeric: tabular-nums; font-size: 0.78rem;
}
.fc-active__count { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-active__count--err { color: rgb(var(--v-theme-error)); }
.fc-active__count--posts { opacity: 0.7; }
@keyframes fc-active-pulse { @keyframes fc-active-pulse {
0%, 100% { opacity: 1; transform: scale(1); } 0%, 100% { opacity: 1; transform: scale(1); }
50% { opacity: 0.35; transform: scale(0.7); } 50% { opacity: 0.35; transform: scale(0.7); }
@@ -93,7 +93,6 @@ const ERROR_TYPE_COLOR = {
unsupported_url: 'error', unsupported_url: 'error',
http_error: 'error', http_error: 'error',
unknown_error: 'error', unknown_error: 'error',
api_drift: 'error',
partial: 'info', partial: 'info',
tier_limited: 'info', tier_limited: 'info',
no_new_content: 'info', no_new_content: 'info',
@@ -109,7 +108,6 @@ const ERROR_TYPE_HINT = {
http_error: 'Generic HTTP error — see Logs.', http_error: 'Generic HTTP error — see Logs.',
unsupported_url: 'gallery-dl does not support this URL pattern.', unsupported_url: 'gallery-dl does not support this URL pattern.',
unknown_error: 'Could not classify — see Logs.', unknown_error: 'Could not classify — see Logs.',
api_drift: 'Patreon changed its API shape — the native ingester needs a code update.',
} }
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' } function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' } function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
@@ -1,113 +0,0 @@
<template>
<!-- Plan #708 B4: dry-run preview what a backfill WOULD download, without
downloading. Self-fetches on open; loading / error / empty / result. -->
<v-dialog
:model-value="modelValue" max-width="520"
@update:model-value="$emit('update:modelValue', $event)"
>
<v-card v-if="source">
<v-card-title class="d-flex align-center">
<v-icon class="mr-2">mdi-eye-outline</v-icon>
Preview · {{ source.artist_name }}
</v-card-title>
<v-card-subtitle class="fc-preview__url">{{ source.url }}</v-card-subtitle>
<v-card-text>
<div v-if="loading" class="text-center py-8">
<v-progress-circular indeterminate color="accent" />
<div class="text-caption mt-3 text-medium-emphasis">Walking the feed</div>
</div>
<v-alert
v-else-if="error" type="error" variant="tonal" density="comfortable"
>{{ error }}</v-alert>
<div v-else-if="result">
<div class="text-h5">
{{ result.total_new }} new item{{ result.total_new === 1 ? '' : 's' }}
</div>
<div class="text-body-2 text-medium-emphasis mb-3">
in the {{ result.posts_scanned }} most recent
post{{ result.posts_scanned === 1 ? '' : 's' }}<span
v-if="result.has_more"> · more pages not scanned</span>
</div>
<div
v-if="result.total_new === 0"
class="text-body-2 text-medium-emphasis"
>Youre caught up nothing new to download.</div>
<v-list v-else density="compact" class="fc-preview__list" bg-color="transparent">
<v-list-item v-for="(row, i) in result.sample" :key="i" class="px-0">
<template #prepend>
<v-chip
size="x-small" color="info" variant="tonal" label class="mr-3"
>+{{ row.new }}</v-chip>
</template>
<v-list-item-title class="text-body-2">{{ row.title }}</v-list-item-title>
<v-list-item-subtitle v-if="row.date" class="text-caption">
{{ formatRelative(row.date) }}
</v-list-item-subtitle>
</v-list-item>
</v-list>
</div>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="$emit('update:modelValue', false)">Close</v-btn>
<v-btn
v-if="result && result.total_new > 0"
color="accent" variant="flat"
@click="$emit('backfill', source)"
>Start backfill</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { ref, watch } from 'vue'
import { useSourcesStore } from '../../stores/sources.js'
import { formatRelative } from '../../utils/date.js'
const props = defineProps({
modelValue: { type: Boolean, default: false },
source: { type: Object, default: null },
})
defineEmits(['update:modelValue', 'backfill'])
const store = useSourcesStore()
const loading = ref(false)
const error = ref('')
const result = ref(null)
watch(
() => props.modelValue,
async (open) => {
if (!open || !props.source) return
loading.value = true
error.value = ''
result.value = null
try {
result.value = await store.previewSource(props.source.id)
} catch (e) {
error.value = e?.body?.detail || e?.message || 'Preview failed'
} finally {
loading.value = false
}
},
)
</script>
<style scoped>
.fc-preview__url {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.fc-preview__list {
max-height: 320px;
overflow-y: auto;
}
</style>
@@ -1,81 +0,0 @@
<template>
<!-- Shared per-source action cluster (desktop SourceRow + mobile SourceCard).
Frequent actions stay as buttons; low-frequency / destructive ones move
into a labelled overflow menu so they can't be mis-clicked, and the
labels spell out what each one does (recover vs backfill). -->
<div class="fc-src-actions">
<v-btn
icon size="small" variant="text" :loading="checking"
@click.stop="emit('check', source)"
>
<v-icon>mdi-play</v-icon>
<v-tooltip activator="parent" location="top">
Check now — fetch posts published since the last check
</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:color="running ? 'warning' : undefined"
@click.stop="emit('backfill', source)"
>
<v-icon>{{ running ? 'mdi-stop' : 'mdi-magnify-scan' }}</v-icon>
<v-tooltip activator="parent" location="top">
{{ running
? (recovering ? 'Stop recovery' : 'Stop backfill')
: 'Backfill one-time walk of the FULL post history until complete' }}
</v-tooltip>
</v-btn>
<v-menu location="bottom end">
<template #activator="{ props: menuProps }">
<v-btn icon size="small" variant="text" v-bind="menuProps" @click.stop>
<v-icon>mdi-dots-vertical</v-icon>
<v-tooltip activator="parent" location="top">More actions</v-tooltip>
</v-btn>
</template>
<v-list density="compact" min-width="260">
<v-list-item
v-if="isPatreon && !running"
prepend-icon="mdi-backup-restore"
@click="emit('recover', source)"
>
<v-list-item-title>Recover dropped near-duplicates</v-list-item-title>
<v-list-item-subtitle>
Re-fetch images previously dropped as near-dups and re-judge them
under the current similarity threshold
</v-list-item-subtitle>
</v-list-item>
<v-divider v-if="isPatreon && !running" />
<v-list-item
base-color="error"
prepend-icon="mdi-delete-outline"
@click="emit('remove', source)"
>
<v-list-item-title>Remove source</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</div>
</template>
<script setup>
import { computed } from 'vue'
const props = defineProps({
source: { type: Object, required: true },
checking: { type: Boolean, default: false },
})
const emit = defineEmits(['check', 'backfill', 'recover', 'remove'])
const running = computed(() => props.source.backfill_state === 'running')
const recovering = computed(() => !!props.source.backfill_bypass_seen)
const isPatreon = computed(() => props.source.platform === 'patreon')
</script>
<style scoped>
.fc-src-actions {
display: inline-flex;
align-items: center;
gap: 2px;
}
</style>
@@ -1,117 +0,0 @@
<template>
<div class="fc-source-card">
<div class="fc-source-card__top">
<SourceHealthDot :source="source" :warning-threshold="warningThreshold" />
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
<v-spacer />
<v-switch
:model-value="source.enabled"
density="compact" hide-details color="accent"
@click.stop
@update:model-value="onToggleEnabled"
/>
</div>
<div class="fc-source-card__url-row">
<a
:href="source.url" target="_blank" rel="noopener"
class="fc-source-card__url" @click.stop
>{{ source.url }}</a>
<v-btn
icon="mdi-pencil" size="x-small" variant="text"
@click.stop="$emit('edit', source)"
>
<v-icon>mdi-pencil</v-icon>
<v-tooltip activator="parent" location="top">Edit source</v-tooltip>
</v-btn>
</div>
<div class="fc-source-card__meta">
<span>Last {{ formatRelative(source.last_checked_at) }}</span>
<span>Next {{ formatRelative(source.next_check_at, { future: true }) }}</span>
<v-chip
v-if="(source.consecutive_failures || 0) > 0"
size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }} err
<v-tooltip v-if="source.last_error" activator="parent" location="top" max-width="480">
<span style="white-space: pre-wrap; word-break: break-word;">{{ source.last_error }}</span>
</v-tooltip>
</v-chip>
<v-chip
v-else-if="source.backfill_state === 'running'"
size="x-small" color="info" variant="tonal" label
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling'
}}{{ source.backfill_posts
? ` · ${source.backfill_posts} posts`
: (source.backfill_chunks ? ` (${source.backfill_chunks})` : '') }}</v-chip>
<v-chip
v-else-if="source.backfill_state === 'complete'"
size="x-small" color="success" variant="tonal" label
>Backfilled</v-chip>
<v-chip
v-else-if="source.backfill_state === 'stalled'"
size="x-small" color="warning" variant="tonal" label
>Stalled</v-chip>
</div>
<div class="fc-source-card__actions">
<SourceActions
:source="source" :checking="checking"
@check="$emit('check', $event)"
@backfill="$emit('backfill', $event)"
@recover="$emit('recover', $event)"
@remove="$emit('remove', $event)"
/>
</div>
</div>
</template>
<script setup>
import SourceActions from './SourceActions.vue'
import SourceHealthDot from './SourceHealthDot.vue'
import { formatRelative } from '../../utils/date.js'
// Mobile-stacked equivalent of SourceRow (the desktop <tr>) — same data and
// emits, but laid out vertically so the wide source columns never force the
// lateral scroll the operator flagged on phones.
const props = defineProps({
source: { type: Object, required: true },
checking: { type: Boolean, default: false },
warningThreshold: { type: Number, default: 5 },
})
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover'])
function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value })
}
</script>
<style scoped>
.fc-source-card {
padding: 8px 10px;
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 6px;
background: rgb(var(--v-theme-surface));
display: flex; flex-direction: column; gap: 6px;
}
.fc-source-card__top { display: flex; align-items: center; gap: 8px; }
.fc-source-card__url-row {
display: flex; align-items: center; gap: 4px;
}
.fc-source-card__url {
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none;
font-size: 0.8rem;
word-break: break-all;
flex: 1 1 auto; min-width: 0;
}
.fc-source-card__url:hover { color: rgb(var(--v-theme-accent)); }
.fc-source-card__meta {
display: flex; flex-wrap: wrap; align-items: center; gap: 6px 12px;
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
.fc-source-card__actions {
display: flex; gap: 2px; justify-content: flex-end;
}
</style>
@@ -7,22 +7,10 @@
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip> <v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
</td> </td>
<td class="fc-source-row__url-cell"> <td class="fc-source-row__url-cell">
<div class="fc-source-row__url-wrap"> <a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url"
<a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url" @click.stop>
@click.stop> {{ source.url }}
{{ source.url }} </a>
</a>
<!-- Edit sits next to the source identity (operator-requested), not in
the action cluster where it was easy to fat-finger Remove. -->
<v-btn
icon="mdi-pencil" size="x-small" variant="text"
class="fc-source-row__edit"
@click.stop="$emit('edit', source)"
>
<v-icon>mdi-pencil</v-icon>
<v-tooltip activator="parent" location="top">Edit source</v-tooltip>
</v-btn>
</div>
</td> </td>
<td> <td>
<v-switch <v-switch
@@ -42,43 +30,53 @@
<v-chip <v-chip
v-if="(source.consecutive_failures || 0) > 0" v-if="(source.consecutive_failures || 0) > 0"
size="x-small" color="error" variant="tonal" label size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }} >{{ source.consecutive_failures }}</v-chip>
<!-- #1: show the actual failure reason on hover instead of a bare count. -->
<v-tooltip v-if="source.last_error" activator="parent" location="top" max-width="480">
<span class="fc-source-row__err-text">{{ source.last_error }}</span>
</v-tooltip>
</v-chip>
<v-chip <v-chip
v-else-if="source.backfill_state === 'running'" v-else-if="(source.backfill_runs_remaining || 0) > 0"
size="x-small" color="info" variant="tonal" label size="x-small" color="info" variant="tonal" label
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling' >
}}{{ source.backfill_posts backfill ({{ source.backfill_runs_remaining }}×)
? ` · ${source.backfill_posts} posts` </v-chip>
: (source.backfill_chunks ? ` (${source.backfill_chunks})` : '') }}</v-chip>
<v-chip
v-else-if="source.backfill_state === 'complete'"
size="x-small" color="success" variant="tonal" label
>Backfilled</v-chip>
<v-chip
v-else-if="source.backfill_state === 'stalled'"
size="x-small" color="warning" variant="tonal" label
>Stalled</v-chip>
<span v-else class="fc-source-row__zero">0</span> <span v-else class="fc-source-row__zero">0</span>
</td> </td>
<td class="fc-source-row__actions"> <td class="fc-source-row__actions">
<SourceActions <v-btn
:source="source" :checking="checking" icon="mdi-play" size="x-small" variant="text"
@check="$emit('check', $event)" :loading="checking"
@backfill="$emit('backfill', $event)" @click.stop="$emit('check', source)"
@recover="$emit('recover', $event)" >
@remove="$emit('remove', $event)" <v-icon>mdi-play</v-icon>
/> <v-tooltip activator="parent" location="top">Check now</v-tooltip>
</v-btn>
<v-btn
icon="mdi-magnify-scan" size="x-small" variant="text"
:disabled="(source.backfill_runs_remaining || 0) > 0"
@click.stop="$emit('backfill', source)"
>
<v-icon>mdi-magnify-scan</v-icon>
<v-tooltip activator="parent" location="top">
Deep scan walk full history for next few runs
</v-tooltip>
</v-btn>
<v-btn
icon="mdi-pencil" size="x-small" variant="text"
@click.stop="$emit('edit', source)"
>
<v-icon>mdi-pencil</v-icon>
<v-tooltip activator="parent" location="top">Edit</v-tooltip>
</v-btn>
<v-btn
icon="mdi-close" size="x-small" variant="text" color="error"
@click.stop="$emit('remove', source)"
>
<v-icon>mdi-close</v-icon>
<v-tooltip activator="parent" location="top">Remove</v-tooltip>
</v-btn>
</td> </td>
</tr> </tr>
</template> </template>
<script setup> <script setup>
import SourceActions from './SourceActions.vue'
import SourceHealthDot from './SourceHealthDot.vue' import SourceHealthDot from './SourceHealthDot.vue'
import { formatRelative } from '../../utils/date.js' import { formatRelative } from '../../utils/date.js'
@@ -87,7 +85,7 @@ const props = defineProps({
checking: { type: Boolean, default: false }, checking: { type: Boolean, default: false },
warningThreshold: { type: Number, default: 5 }, warningThreshold: { type: Number, default: 5 },
}) })
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover']) const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill'])
function onToggleEnabled(value) { function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value }) emit('toggle', { source: props.source, enabled: value })
@@ -100,27 +98,18 @@ function onToggleEnabled(value) {
padding-right: 0 !important; padding-right: 0 !important;
} }
.fc-source-row__url-cell { .fc-source-row__url-cell {
max-width: 420px; max-width: 400px;
overflow: hidden; overflow: hidden;
} }
.fc-source-row__url-wrap {
display: flex;
align-items: center;
gap: 2px;
min-width: 0;
}
.fc-source-row__url { .fc-source-row__url {
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none; text-decoration: none;
flex: 0 1 auto; display: inline-block;
min-width: 0; max-width: 100%;
overflow: hidden; overflow: hidden;
text-overflow: ellipsis; text-overflow: ellipsis;
white-space: nowrap; white-space: nowrap;
} }
/* Keep the edit affordance subtle until the row is hovered. */
.fc-source-row__edit { opacity: 0.5; flex: 0 0 auto; }
.fc-source-row:hover .fc-source-row__edit { opacity: 1; }
.fc-source-row__url:hover { color: rgb(var(--v-theme-accent)); } .fc-source-row__url:hover { color: rgb(var(--v-theme-accent)); }
.fc-source-row__when { .fc-source-row__when {
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
@@ -131,10 +120,6 @@ function onToggleEnabled(value) {
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.6; opacity: 0.6;
} }
.fc-source-row__err-text {
white-space: pre-wrap;
word-break: break-word;
}
.fc-source-row__actions { .fc-source-row__actions {
white-space: nowrap; white-space: nowrap;
text-align: right; text-align: right;
@@ -26,14 +26,14 @@
:items="STATUS_OPTIONS" :items="STATUS_OPTIONS"
:disabled="needsAttention" :disabled="needsAttention"
density="compact" variant="outlined" hide-details density="compact" variant="outlined" hide-details
class="fc-subs__status" style="max-width: 180px"
/> />
<v-text-field <v-text-field
v-model="search" v-model="search"
density="compact" variant="outlined" hide-details clearable density="compact" variant="outlined" hide-details clearable
prepend-inner-icon="mdi-magnify" prepend-inner-icon="mdi-magnify"
placeholder="Search subscriptions" placeholder="Search subscriptions"
class="fc-subs__search" style="max-width: 320px"
/> />
</div> </div>
@@ -54,9 +54,6 @@
<v-btn size="small" variant="text" prepend-icon="mdi-toggle-switch-off-outline" @click="bulkSetEnabled(false)"> <v-btn size="small" variant="text" prepend-icon="mdi-toggle-switch-off-outline" @click="bulkSetEnabled(false)">
Disable all Disable all
</v-btn> </v-btn>
<v-btn size="small" variant="text" prepend-icon="mdi-magnify-scan" @click="bulkBackfill">
Backfill
</v-btn>
<v-btn size="small" variant="text" color="error" prepend-icon="mdi-delete" @click="bulkDelete"> <v-btn size="small" variant="text" color="error" prepend-icon="mdi-delete" @click="bulkDelete">
Delete Delete
</v-btn> </v-btn>
@@ -78,14 +75,13 @@
<p v-else>No subscriptions match the current filter.</p> <p v-else>No subscriptions match the current filter.</p>
</div> </div>
<v-card v-else-if="!isMobile" class="fc-subs__card" variant="outlined"> <v-card v-else class="fc-subs__card" variant="outlined">
<v-data-table <v-data-table
:headers="headers" :headers="headers"
:items="filteredGroups" :items="filteredGroups"
item-value="key" item-value="key"
v-model="selected" v-model="selected"
v-model:expanded="expanded" v-model:expanded="expanded"
v-model:sort-by="sortBy"
:items-per-page="50" :items-per-page="50"
:items-per-page-options="ITEMS_PER_PAGE_OPTIONS" :items-per-page-options="ITEMS_PER_PAGE_OPTIONS"
density="comfortable" density="comfortable"
@@ -93,13 +89,7 @@
@click:row="onRowClick" @click:row="onRowClick"
> >
<template #item.name="{ item }"> <template #item.name="{ item }">
<div class="fc-subs__name">{{ item.artist.name }}</div> <span class="fc-subs__name">{{ item.artist.name }}</span>
<!-- #2: single-source subscriptions show their URL inline (no expand). -->
<a
v-if="item.singleSource"
:href="item.singleSource.url" target="_blank" rel="noopener"
class="fc-subs__sub-url" @click.stop
>{{ item.singleSource.url }}</a>
</template> </template>
<template #item.platforms="{ item }"> <template #item.platforms="{ item }">
@@ -131,60 +121,32 @@
</template> </template>
<template #item.actions="{ item }"> <template #item.actions="{ item }">
<!-- #2: single source its own actions inline (Check / Backfill / ), <v-btn
plus Edit, so the row is actionable without expanding. --> icon size="small" variant="text"
<template v-if="item.singleSource"> :loading="anyChecking(item.sources)"
<SourceActions @click.stop="checkAll(item)"
:source="item.singleSource" >
:checking="store.checkingIds.has(item.singleSource.id)" <v-icon>mdi-refresh</v-icon>
@check="onCheck" @backfill="onBackfill" <v-tooltip activator="parent" location="top">Check all sources</v-tooltip>
@recover="onRecover" @remove="removeSource" </v-btn>
/> <v-btn icon size="small" variant="text" @click.stop="openAddSource(item.artist)">
<v-btn icon size="small" variant="text" @click.stop="openEditSource(item.singleSource)"> <v-icon>mdi-plus</v-icon>
<v-icon>mdi-pencil</v-icon> <v-tooltip activator="parent" location="top">Add source</v-tooltip>
<v-tooltip activator="parent" location="top">Edit source</v-tooltip> </v-btn>
</v-btn> <v-btn
<v-btn icon size="small" variant="text" @click.stop="openAddSource(item.artist)"> icon size="small" variant="text"
<v-icon>mdi-plus</v-icon> :to="`/posts?artist_id=${item.artist.id}`" @click.stop
<v-tooltip activator="parent" location="top">Add another source</v-tooltip> >
</v-btn> <v-icon>mdi-rss</v-icon>
<v-btn <v-tooltip activator="parent" location="top">View posts</v-tooltip>
icon size="small" variant="text" </v-btn>
:to="`/artist/${item.artist.slug}`" @click.stop <v-btn
> icon size="small" variant="text"
<v-icon>mdi-account</v-icon> :to="`/artist/${item.artist.slug}`" @click.stop
<v-tooltip activator="parent" location="top">Open artist page</v-tooltip> >
</v-btn> <v-icon>mdi-account</v-icon>
</template> <v-tooltip activator="parent" location="top">Open artist page</v-tooltip>
<!-- Multi-source artist-level actions; expand for per-source rows. --> </v-btn>
<template v-else>
<v-btn
icon size="small" variant="text"
:loading="anyChecking(item.sources)"
@click.stop="checkAll(item)"
>
<v-icon>mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Check all sources</v-tooltip>
</v-btn>
<v-btn icon size="small" variant="text" @click.stop="openAddSource(item.artist)">
<v-icon>mdi-plus</v-icon>
<v-tooltip activator="parent" location="top">Add source</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/posts?artist_id=${item.artist.id}`" @click.stop
>
<v-icon>mdi-rss</v-icon>
<v-tooltip activator="parent" location="top">View posts</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/artist/${item.artist.slug}`" @click.stop
>
<v-icon>mdi-account</v-icon>
<v-tooltip activator="parent" location="top">Open artist page</v-tooltip>
</v-btn>
</template>
</template> </template>
<template #expanded-row="{ columns, item }"> <template #expanded-row="{ columns, item }">
@@ -213,7 +175,6 @@
@toggle="toggleSourceEnabled" @toggle="toggleSourceEnabled"
@check="onCheck" @check="onCheck"
@backfill="onBackfill" @backfill="onBackfill"
@recover="onRecover"
/> />
<tr v-if="item.sources.length === 0"> <tr v-if="item.sources.length === 0">
<td colspan="8" class="fc-subs__sources-empty"> <td colspan="8" class="fc-subs__sources-empty">
@@ -228,77 +189,6 @@
</v-data-table> </v-data-table>
</v-card> </v-card>
<!-- Mobile: compact cards (several fit per screen), expanding to STACKED
source cards so the wide source columns never force lateral scroll. -->
<div v-else class="fc-subs__mlist">
<div
v-for="item in filteredGroups" :key="item.key"
class="fc-subs__mcard"
>
<div class="fc-subs__mhead" @click="toggleExpand(item)">
<v-checkbox-btn
:model-value="isSelected(item)" density="compact" hide-details
@click.stop @update:model-value="toggleSelect(item)"
/>
<span class="fc-subs__name">{{ item.artist.name }}</span>
<v-spacer />
<SourceHealthDot
v-if="item.worstSource"
:source="item.worstSource" :warning-threshold="failureThreshold"
/>
<v-icon size="small">
{{ isExpanded(item) ? 'mdi-chevron-up' : 'mdi-chevron-down' }}
</v-icon>
</div>
<div class="fc-subs__mmeta">
<PlatformChip
v-for="p in item.platforms" :key="p" :platform="p" size="x-small"
/>
<span class="fc-subs__mmeta-text">
{{ item.sources.length }} src · {{ formatRelative(item.lastActivity) }}
</span>
</div>
<div v-if="isExpanded(item)" class="fc-subs__mbody">
<div class="fc-subs__mactions">
<v-btn
size="small" variant="text" :loading="anyChecking(item.sources)"
@click="checkAll(item)"
>
<v-icon>mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Check all</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" @click="openAddSource(item.artist)">
<v-icon>mdi-plus</v-icon>
<v-tooltip activator="parent" location="top">Add source</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" :to="`/posts?artist_id=${item.artist.id}`">
<v-icon>mdi-rss</v-icon>
<v-tooltip activator="parent" location="top">View posts</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" :to="`/artist/${item.artist.slug}`">
<v-icon>mdi-account</v-icon>
<v-tooltip activator="parent" location="top">Artist page</v-tooltip>
</v-btn>
</div>
<SourceCard
v-for="s in item.sources" :key="s.id" :source="s"
:checking="store.checkingIds.has(s.id)"
:warning-threshold="failureThreshold"
@edit="openEditSource"
@remove="removeSource"
@toggle="toggleSourceEnabled"
@check="onCheck"
@backfill="onBackfill"
@recover="onRecover"
/>
<div v-if="item.sources.length === 0" class="fc-subs__sources-empty">
No sources yet. Tap + to add one.
</div>
</div>
</div>
</div>
<SourceFormDialog <SourceFormDialog
v-model="showSourceDialog" v-model="showSourceDialog"
:source="editingSource" :source="editingSource"
@@ -312,17 +202,14 @@
<script setup> <script setup>
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref, watch } from 'vue' import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
import { useDisplay } from 'vuetify'
import { useRoute, useRouter } from 'vue-router' import { useRoute, useRouter } from 'vue-router'
import { useSourcesStore } from '../../stores/sources.js' import { useSourcesStore } from '../../stores/sources.js'
import { usePlatformsStore } from '../../stores/platforms.js' import { usePlatformsStore } from '../../stores/platforms.js'
import { useImportStore } from '../../stores/import.js' import { useImportStore } from '../../stores/import.js'
import SourceRow from './SourceRow.vue' import SourceRow from './SourceRow.vue'
import SourceCard from './SourceCard.vue'
import SourceHealthDot from './SourceHealthDot.vue' import SourceHealthDot from './SourceHealthDot.vue'
import SourceFormDialog from './SourceFormDialog.vue' import SourceFormDialog from './SourceFormDialog.vue'
import ArtistCreateDialog from './ArtistCreateDialog.vue' import ArtistCreateDialog from './ArtistCreateDialog.vue'
import SourceActions from './SourceActions.vue'
import PlatformChip from './PlatformChip.vue' import PlatformChip from './PlatformChip.vue'
import SchedulerStatusBar from './SchedulerStatusBar.vue' import SchedulerStatusBar from './SchedulerStatusBar.vue'
import { formatRelative } from '../../utils/date.js' import { formatRelative } from '../../utils/date.js'
@@ -343,10 +230,6 @@ const STATUS_OPTIONS = [
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
const display = useDisplay()
// Match the 600px breakpoint the rest of the hub uses; below it the table is
// replaced by the custom compact-card list.
const isMobile = computed(() => display.width.value < 600)
const store = useSourcesStore() const store = useSourcesStore()
const platformsStore = usePlatformsStore() const platformsStore = usePlatformsStore()
const importStore = useImportStore() const importStore = useImportStore()
@@ -356,25 +239,6 @@ const statusFilter = ref('all')
const needsAttention = ref(false) const needsAttention = ref(false)
const expanded = ref([]) const expanded = ref([])
const selected = ref([]) const selected = ref([])
// #3: default to worst-health first, name as the tiebreak (the table header
// stays clickable to re-sort by any column).
const sortBy = ref([
{ key: 'health', order: 'desc' },
{ key: 'name', order: 'asc' },
])
// Mobile card list drives the same `selected`/`expanded` key arrays the
// desktop v-data-table binds, so selection + bulk actions work identically.
function _toggleKey(arr, key) {
const i = arr.value.indexOf(key)
if (i === -1) arr.value = [...arr.value, key]
else arr.value = arr.value.filter((k) => k !== key)
}
function isSelected(item) { return selected.value.includes(item.key) }
function toggleSelect(item) { _toggleKey(selected, item.key) }
function isExpanded(item) { return expanded.value.includes(item.key) }
function toggleExpand(item) { _toggleKey(expanded, item.key) }
const showSourceDialog = ref(false) const showSourceDialog = ref(false)
const editingSource = ref(null) const editingSource = ref(null)
const editingArtist = ref(null) const editingArtist = ref(null)
@@ -417,7 +281,7 @@ const headers = [
{ title: 'Subscription', key: 'name', sortable: true, align: 'start' }, { title: 'Subscription', key: 'name', sortable: true, align: 'start' },
{ title: 'Platforms', key: 'platforms', sortable: false, align: 'start', width: 240 }, { title: 'Platforms', key: 'platforms', sortable: false, align: 'start', width: 240 },
{ title: 'Sources', key: 'sources_count', sortable: true, align: 'start', width: 90 }, { title: 'Sources', key: 'sources_count', sortable: true, align: 'start', width: 90 },
{ title: 'Health', key: 'health', sortable: true, align: 'start', width: 80 }, { title: 'Health', key: 'health', sortable: false, align: 'start', width: 80 },
{ title: 'Last activity',key: 'last_activity', sortable: true, align: 'start', width: 140 }, { title: 'Last activity',key: 'last_activity', sortable: true, align: 'start', width: 140 },
{ title: 'Actions', key: 'actions', sortable: false, align: 'end', width: 200 }, { title: 'Actions', key: 'actions', sortable: false, align: 'end', width: 200 },
] ]
@@ -438,26 +302,10 @@ const groups = computed(() => {
lastActivity, lastActivity,
name: g.artist.name, name: g.artist.name,
last_activity: lastActivity ?? '', last_activity: lastActivity ?? '',
// #3: numeric health rank so the Health column is sortable worst-first.
health: healthRank(worstSource, failureThreshold.value),
// #2: when a subscription has exactly one source, surface it on the
// artist row directly (URL + actions) so the common case needs no expand.
singleSource: g.sources.length === 1 ? g.sources[0] : null,
} }
}) })
}) })
// 0 never-checked · 1 ok · 2 warning · 3 failing — higher = worse, so a
// descending sort floats the broken subscriptions to the top.
function healthRank(source, threshold) {
if (!source) return 0
if (!source.last_checked_at) return 0
const f = source.consecutive_failures || 0
if (f === 0) return 1
if (f < threshold) return 2
return 3
}
// A group "needs attention" if any of its sources has accumulated // A group "needs attention" if any of its sources has accumulated
// failures OR has never been checked — the ones worth acting on. // failures OR has never been checked — the ones worth acting on.
function groupNeedsAttention(g) { function groupNeedsAttention(g) {
@@ -542,26 +390,13 @@ function openEditSource(source) {
} }
async function removeSource(source) { async function removeSource(source) {
// Confirm — Remove moved into the overflow menu, but a destructive action await store.remove(source.id, source.artist_id)
// still deserves a guard (single-source removal had none before). await refresh()
if (!globalThis.window?.confirm(
`Remove this ${source.platform} source?\n${source.url}\n\nThe artist and already-downloaded images stay.`,
)) return
try {
// store._dropSource patches the cache in place — no full reload, so the
// table + expansion don't reset (operator-flagged lock/reload, 2026-06-07).
await store.remove(source.id, source.artist_id)
} catch (e) {
toast({ text: `Remove failed: ${e?.body?.detail || e?.message || e}`, type: 'error' })
}
} }
async function toggleSourceEnabled({ source, enabled }) { async function toggleSourceEnabled({ source, enabled }) {
try { await store.update(source.id, { enabled }, source.artist_id)
await store.update(source.id, { enabled }, source.artist_id) await refresh()
} catch (e) {
toast({ text: `Update failed: ${e?.body?.detail || e?.message || e}`, type: 'error' })
}
} }
async function onSourceSaved() { async function onSourceSaved() {
@@ -608,39 +443,31 @@ async function onCheck(source) {
} }
} }
// Plan #693: toggle a run-until-done backfill. Starting walks the full post // Plan #544: arm a source for backfill mode (gallery-dl walks the full
// history in time-boxed chunks across ticks until it reaches the bottom (the // post history) for the next N download runs. Default 3 — enough budget
// row badge tracks progress / completion); stopping cancels back to tick mode. // to finish a deep creator without re-prompting the operator across
// timeout boundaries. The chip on the row reflects the remaining count.
async function onBackfill(source) { async function onBackfill(source) {
const running = source.backfill_state === 'running' const raw = globalThis.window?.prompt(
try { `Deep scan "${source.artist_name} (${source.platform})" — walk full history for the next how many download runs? (110, default 3)`,
if (running) { '3',
await store.stopBackfill(source.id, source.artist_id) )
toast({ text: `Backfill stopped for ${source.artist_name}`, type: 'success' }) if (raw == null) return
} else { const runs = parseInt(raw, 10)
await store.startBackfill(source.id, source.artist_id) if (!Number.isFinite(runs) || runs < 1 || runs > 10) {
toast({ text: `Backfill started for ${source.artist_name}`, type: 'success' }) toast({ text: 'Deep scan: runs must be 110', type: 'error' })
} return
// startBackfill/stopBackfill patch the source in place — no full reload.
} catch (e) {
toast({
text: `Backfill ${running ? 'stop' : 'start'} failed: ${e?.body?.detail || e?.message || e}`,
type: 'error',
})
} }
}
// Plan #697: arm a recovery walk (Patreon-only) — re-fetches dropped-and-deleted
// near-dups and re-evaluates them under the current pHash threshold. Reuses the
// backfill lifecycle/badge; stop via the same Stop control (onBackfill).
async function onRecover(source) {
try { try {
await store.recoverSource(source.id, source.artist_id) await store.setBackfill(source.id, runs, source.artist_id)
toast({ text: `Recovery started for ${source.artist_name}`, type: 'success' }) toast({
// recoverSource patches the source in place — no full reload. text: `Deep scan armed for ${runs} run${runs === 1 ? '' : 's'}`,
type: 'success',
})
await store.loadAll()
} catch (e) { } catch (e) {
toast({ toast({
text: `Recovery start failed: ${e?.body?.detail || e?.message || e}`, text: `Deep scan failed: ${e?.detail || e?.message || e}`,
type: 'error', type: 'error',
}) })
} }
@@ -701,28 +528,6 @@ async function bulkSetEnabled(enabled) {
selected.value = [] selected.value = []
} }
// #5: arm a run-until-done backfill on every enabled source in the selected
// subscriptions (skips ones already backfilling). Patches each row in place.
async function bulkBackfill() {
const sel = resolveSelectedGroups()
let armed = 0
for (const g of sel) {
for (const s of g.sources) {
if (!s.enabled || s.backfill_state === 'running') continue
try {
await store.startBackfill(s.id, s.artist_id)
armed += 1
} catch { /* keep going */ }
}
}
toast({
text: armed ? `Backfill started on ${armed} source${armed === 1 ? '' : 's'}`
: 'Nothing to backfill (already running or disabled)',
type: armed ? 'success' : 'info',
})
selected.value = []
}
async function bulkDelete() { async function bulkDelete() {
const groups = resolveSelectedGroups() const groups = resolveSelectedGroups()
const total = groups.reduce((n, g) => n + g.sources.length, 0) const total = groups.reduce((n, g) => n + g.sources.length, 0)
@@ -763,18 +568,6 @@ async function bulkDelete() {
padding: 8px 0 1rem; padding: 8px 0 1rem;
} }
.fc-subs__bar .v-chip { cursor: pointer; } .fc-subs__bar .v-chip { cursor: pointer; }
.fc-subs__status { max-width: 180px; }
.fc-subs__search { max-width: 320px; flex: 1 1 200px; }
/* Phones: status + search each take a full-width row; drop the spacer that
would otherwise eat a row pushing them around. */
@media (max-width: 600px) {
.fc-subs__status, .fc-subs__search { max-width: none; flex-basis: 100%; }
.fc-subs__bar :deep(.v-spacer) { display: none; }
/* The table renders as stacked cards (mobile-breakpoint); reclaim the
desktop indent on the expanded sources detail (it keeps its own
horizontal scroll for the wide source columns). */
.fc-subs__sources-cell { padding-left: 0.5rem !important; }
}
.fc-subs__loading, .fc-subs__empty { .fc-subs__loading, .fc-subs__empty {
display: flex; justify-content: center; padding: 2rem; display: flex; justify-content: center; padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
@@ -782,42 +575,7 @@ async function bulkDelete() {
.fc-subs__card { .fc-subs__card {
background: rgb(var(--v-theme-surface)); background: rgb(var(--v-theme-surface));
} }
/* Mobile compact-card list (replaces the data-table <600px). */
.fc-subs__mlist { display: flex; flex-direction: column; gap: 8px; }
.fc-subs__mcard {
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 8px;
background: rgb(var(--v-theme-surface));
padding: 8px 10px;
}
.fc-subs__mhead { display: flex; align-items: center; gap: 6px; cursor: pointer; }
.fc-subs__mmeta {
display: flex; flex-wrap: wrap; align-items: center; gap: 6px;
margin-top: 4px;
}
.fc-subs__mmeta-text {
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
.fc-subs__mbody {
margin-top: 8px; padding-top: 8px;
border-top: 1px solid rgb(var(--v-theme-surface-light));
display: flex; flex-direction: column; gap: 6px;
}
.fc-subs__mactions { display: flex; gap: 2px; }
.fc-subs__name { font-weight: 600; } .fc-subs__name { font-weight: 600; }
.fc-subs__sub-url {
display: block;
max-width: 460px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-size: 0.76rem;
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none;
}
.fc-subs__sub-url:hover { color: rgb(var(--v-theme-accent)); }
.fc-subs__chips { .fc-subs__chips {
display: flex; flex-wrap: wrap; gap: 4px; display: flex; flex-wrap: wrap; gap: 4px;
} }
-33
View File
@@ -127,37 +127,6 @@ export const useAdminStore = defineStore('admin', () => {
} }
} }
// Destructive: deletes ALL general + character tags so the operator can
// re-tag from scratch via auto-suggest. fandom + series preserved.
async function resetContentTagging({ dryRun = true } = {}) {
lastError.value = null
try {
return await api.post(
'/api/admin/tags/reset-content',
{ body: { dry_run: dryRun } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// #714: Title-Case the back-catalog + merge case/whitespace-variant tags.
// dry-run returns a projection inline; live returns {task_id} (long op —
// FK repoints) the caller polls via pollTaskUntilDone.
async function normalizeTags({ dryRun = true } = {}) {
lastError.value = null
try {
return await api.post(
'/api/admin/tags/normalize',
{ body: { dry_run: dryRun } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Task progress polling (taps FC-3i activity dashboard) -------- // --- Task progress polling (taps FC-3i activity dashboard) --------
/** /**
@@ -193,8 +162,6 @@ export const useAdminStore = defineStore('admin', () => {
tagUsageCount, tagUsageCount,
pruneUnusedTags, pruneUnusedTags,
purgeLegacyTags, purgeLegacyTags,
resetContentTagging,
normalizeTags,
pollTaskUntilDone, pollTaskUntilDone,
} }
}) })
+1 -8
View File
@@ -14,10 +14,6 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
const q = ref('') const q = ref('')
const platform = ref(null) const platform = ref(null)
let started = false let started = false
// Has a load ATTEMPT completed yet? Gates the "No artists match" empty state
// so it can't flash on the very first render (loading is still false before
// onMounted fires the first loadMore) or between a query change and its fetch.
const loaded = ref(false)
// Typed "alice" then "alice bob" used to drop the second fetch // Typed "alice" then "alice bob" used to drop the second fetch
// entirely (loading flag still true from the first), so the UI // entirely (loading flag still true from the first), so the UI
// showed alice results while the input said "alice bob". Inflight // showed alice results while the input said "alice bob". Inflight
@@ -39,7 +35,6 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
nextCursor.value = body.next_cursor nextCursor.value = body.next_cursor
started = true started = true
}) })
if (t.isCurrent()) loaded.value = true
} }
async function reset() { async function reset() {
@@ -47,7 +42,6 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
cards.value = [] cards.value = []
nextCursor.value = null nextCursor.value = null
started = false started = false
loaded.value = false
await loadMore() await loadMore()
} }
@@ -56,8 +50,7 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
const hasMore = computed(() => !started || nextCursor.value !== null) const hasMore = computed(() => !started || nextCursor.value !== null)
const isEmpty = computed( const isEmpty = computed(
() => loaded.value && !loading.value () => !loading.value && cards.value.length === 0 && error.value === null
&& cards.value.length === 0 && error.value === null
) )
return { return {
+1 -9
View File
@@ -55,14 +55,6 @@ export const useCleanupStore = defineStore('cleanup', () => {
return body.runs return body.runs
} }
// The most recent audit run for a given rule, or null. Cards call this on
// mount to reconnect to a scan that's still running (or to show the last
// completed result) after the user navigates away and back.
async function latestAuditForRule(rule) {
const body = await api.get('/api/cleanup/audit', { params: { rule, limit: 1 } })
return (body.runs && body.runs[0]) || null
}
async function applyAudit(id, confirm) { async function applyAudit(id, confirm) {
return await api.post(`/api/cleanup/audit/${id}/apply`, { body: { confirm } }) return await api.post(`/api/cleanup/audit/${id}/apply`, { body: { confirm } })
} }
@@ -75,6 +67,6 @@ export const useCleanupStore = defineStore('cleanup', () => {
defaults, recentRuns, defaults, recentRuns,
loadDefaults, loadDefaults,
previewMinDim, deleteMinDim, previewMinDim, deleteMinDim,
startAudit, getAudit, loadHistory, latestAuditForRule, applyAudit, cancelAudit, startAudit, getAudit, loadHistory, applyAudit, cancelAudit,
} }
}) })
-25
View File
@@ -1,25 +0,0 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useDbMaintenanceStore = defineStore('dbMaintenance', () => {
const api = useApi()
const tables = ref([])
const loading = ref(false)
async function loadStats() {
loading.value = true
try {
const body = await api.get('/api/admin/maintenance/db-stats')
tables.value = body.tables || []
} finally {
loading.value = false
}
}
async function runVacuum() {
return await api.post('/api/admin/maintenance/vacuum')
}
return { tables, loading, loadStats, runVacuum }
})
+30 -186
View File
@@ -3,15 +3,12 @@ import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js' import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js' import { useInflightToken } from '../composables/useInflightToken.js'
// Initial paint is a SINGLE request (INITIAL_LIMIT). The old 10×serial- // Operator-confirmed 2026-05-30: fetch PAGE-sized chunks instead of one
// batch loop (2026-05-30) only staggered METADATA, which isn't the visual // 50-item request so items render as each batch lands. Total initial
// bottleneck — thumbnails load as independent `<img>` requests, and // count is unchanged (PAGE * INITIAL_BATCHES = 50). Infinite-scroll also
// GalleryItem now reveals each tile on its own image `@load`. One fetch is // pulls PAGE per trigger to keep appends progressive.
// far fewer round-trips and faster to first paint; the reveal-on-load is const PAGE = 5
// what makes appearance progressive. Infinite scroll pulls PAGE per trigger. const INITIAL_BATCHES = 10
// Reworked 2026-06-04.
const PAGE = 25
const INITIAL_LIMIT = 50
export const useGalleryStore = defineStore('gallery', () => { export const useGalleryStore = defineStore('gallery', () => {
const api = useApi() const api = useApi()
@@ -21,26 +18,7 @@ export const useGalleryStore = defineStore('gallery', () => {
const nextCursor = ref(null) const nextCursor = ref(null)
const loading = ref(false) const loading = ref(false)
const error = ref(null) const error = ref(null)
const filter = ref({ const filter = ref({ tag_id: null, post_id: null })
tag_ids: [], artist_id: null, media_type: null,
sort: 'newest', post_id: null,
// Phase-2 faceted refine params.
platform: null, untagged: false, no_artist: false,
date_from: null, date_to: null,
// Phase-3 visual similarity: when set, the gallery is in "similar mode" —
// ranked by cosine distance to this image, bounded top-N, no cursor.
similar_to: null,
})
// Live facet counts for the refine panel; fetched on-demand (panel open +
// filter change), never on plain scroll. Null until first load.
const facets = ref(null)
const facetsLoading = ref(false)
const facetsInflight = useInflightToken()
// Display names for the active filter chips — resolved by id on deep-link
// and pre-noted by the filter bar when a user picks from autocomplete.
const tagLabels = ref({}) // tagId -> name
const artistLabel = ref(null)
const timelineBuckets = ref([]) const timelineBuckets = ref([])
const timelineLoading = ref(false) const timelineLoading = ref(false)
@@ -54,16 +32,22 @@ export const useGalleryStore = defineStore('gallery', () => {
images.value = [] images.value = []
dateGroups.value = [] dateGroups.value = []
nextCursor.value = null nextCursor.value = null
await loadMore(INITIAL_LIMIT) // Sequentially fetch INITIAL_BATCHES chunks so items render as each
// batch lands rather than blocking on one big response. Stop early
// when the backend reports no more pages.
for (let i = 0; i < INITIAL_BATCHES; i++) {
if (i > 0 && nextCursor.value === null) break
await loadMore()
}
} }
async function loadMore(limit = PAGE) { async function loadMore() {
if (loading.value) return if (loading.value) return
loading.value = true loading.value = true
error.value = null error.value = null
const t = inflight.claim() const t = inflight.claim()
try { try {
const params = { limit, ...activeFilterParam() } const params = { limit: PAGE, ...activeFilterParam() }
if (nextCursor.value) params.cursor = nextCursor.value if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/gallery/scroll', { params }) const body = await api.get('/api/gallery/scroll', { params })
if (!t.isCurrent()) return if (!t.isCurrent()) return
@@ -87,39 +71,6 @@ export const useGalleryStore = defineStore('gallery', () => {
} }
} }
// Visual "more like this": ranked top-N by cosine distance, scope filters
// composed (AND). No cursor / no timeline — bounded result set.
async function loadSimilar() {
inflight.cancel()
images.value = []
dateGroups.value = []
nextCursor.value = null
timelineBuckets.value = []
loading.value = true
error.value = null
const t = inflight.claim()
try {
const f = filter.value
const params = { similar_to: f.similar_to, limit: 100 }
if (f.tag_ids.length) params.tag_id = f.tag_ids.join(',')
if (f.artist_id) params.artist_id = f.artist_id
if (f.media_type) params.media = f.media_type
if (f.platform) params.platform = f.platform
if (f.untagged) params.untagged = '1'
if (f.no_artist) params.no_artist = '1'
if (f.date_from) params.date_from = f.date_from
if (f.date_to) params.date_to = f.date_to
const body = await api.get('/api/gallery/similar', { params })
if (!t.isCurrent()) return
images.value = body.images
// ranked + bounded → no next page (nextCursor stays null → hasMore false)
} catch (e) {
error.value = e.message
} finally {
if (t.isCurrent()) loading.value = false
}
}
async function jumpTo(year, month) { async function jumpTo(year, month) {
// Rapid timeline-jump clicks need the same race guard as // Rapid timeline-jump clicks need the same race guard as
// loadMore — first jump's late body could clobber the second // loadMore — first jump's late body could clobber the second
@@ -138,101 +89,23 @@ export const useGalleryStore = defineStore('gallery', () => {
} }
function activeFilterParam() { function activeFilterParam() {
// post_id is the exclusive post-detail view. if (filter.value.tag_id) return { tag_id: filter.value.tag_id }
if (filter.value.post_id) return { post_id: filter.value.post_id } if (filter.value.post_id) return { post_id: filter.value.post_id }
const p = {} return {}
if (filter.value.tag_ids.length) p.tag_id = filter.value.tag_ids.join(',')
if (filter.value.artist_id) p.artist_id = filter.value.artist_id
if (filter.value.media_type) p.media = filter.value.media_type
if (filter.value.sort && filter.value.sort !== 'newest') p.sort = filter.value.sort
if (filter.value.platform) p.platform = filter.value.platform
if (filter.value.untagged) p.untagged = '1'
if (filter.value.no_artist) p.no_artist = '1'
if (filter.value.date_from) p.date_from = filter.value.date_from
if (filter.value.date_to) p.date_to = filter.value.date_to
return p
} }
// Live facet counts, scoped to the current filter (panel-gated — callers function setTagFilter(tagId) {
// are the refine panel only). Single-flighted so rapid toggles don't let a filter.value.tag_id = tagId
// stale response clobber a newer one. filter.value.post_id = null
async function loadFacets() { loadInitial()
facetsLoading.value = true loadTimeline()
const t = facetsInflight.claim()
try {
const body = await api.get('/api/gallery/facets', { params: activeFilterParam() })
if (!t.isCurrent()) return
facets.value = body
} catch (e) {
if (t.isCurrent()) error.value = e.message
} finally {
if (t.isCurrent()) facetsLoading.value = false
}
} }
// URL is the source of truth for filters. GalleryView calls this on mount function setPostFilter(postId) {
// and on every route-query change; the filter bar mutates the URL filter.value.post_id = postId
// (router.push) rather than the store directly, so deep-links, the back filter.value.tag_id = null
// button, and bar actions all funnel through one path. loadInitial()
async function applyFilterFromQuery(q) { loadTimeline()
filter.value = {
tag_ids: _parseIds(q.tag_id),
artist_id: _toId(q.artist_id),
media_type: ['image', 'video'].includes(q.media) ? q.media : null,
sort: q.sort === 'oldest' ? 'oldest' : 'newest',
post_id: _toId(q.post_id),
platform: q.platform || null,
untagged: _truthy(q.untagged),
no_artist: _truthy(q.no_artist),
date_from: _parseDate(q.date_from),
date_to: _parseDate(q.date_to),
similar_to: _toId(q.similar_to),
}
if (filter.value.similar_to) {
// Similar mode: ranked, no timeline scroll.
await loadSimilar()
} else {
await loadInitial()
await loadTimeline()
}
_resolveLabels()
}
function _truthy(v) { return v === '1' || v === 'true' || v === true }
function _parseDate(v) {
return typeof v === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(v) ? v : null
}
function _toId(v) {
const n = Number(v)
return Number.isInteger(n) && n > 0 ? n : null
}
function _parseIds(raw) {
if (!raw) return []
return String(raw).split(',').map(Number).filter((n) => Number.isInteger(n) && n > 0)
}
// Pre-seed a label so a freshly-picked chip shows its name without a
// round-trip; the bar calls these before pushing the new URL.
function noteTagLabel(id, name) { tagLabels.value = { ...tagLabels.value, [id]: name } }
function noteArtistLabel(name) { artistLabel.value = name || null }
async function _resolveLabels() {
for (const id of filter.value.tag_ids) {
if (tagLabels.value[id]) continue
try {
const t = await api.get(`/api/tags/${id}`)
tagLabels.value = { ...tagLabels.value, [id]: t.name }
} catch { /* chip falls back to #id */ }
}
if (filter.value.artist_id && !artistLabel.value) {
// The filtered set is this artist — derive the chip label from a tile.
const hit = images.value.find(
(i) => i.artist && i.artist.id === filter.value.artist_id
)
artistLabel.value = hit?.artist?.name || null
}
if (!filter.value.artist_id) artistLabel.value = null
} }
const hasMore = computed(() => nextCursor.value !== null) const hasMore = computed(() => nextCursor.value !== null)
@@ -240,40 +113,11 @@ export const useGalleryStore = defineStore('gallery', () => {
return { return {
images, dateGroups, hasMore, isEmpty, loading, error, images, dateGroups, hasMore, isEmpty, loading, error,
filter, tagLabels, artistLabel, timelineBuckets, timelineLoading, filter, timelineBuckets, timelineLoading,
facets, facetsLoading, loadInitial, loadMore, loadTimeline, jumpTo, setTagFilter, setPostFilter
loadInitial, loadMore, loadTimeline, jumpTo, loadFacets, loadSimilar,
applyFilterFromQuery, noteTagLabel, noteArtistLabel,
} }
}) })
// Shared by GalleryFilterBar and GalleryFacetPanel so both write the URL in
// one format. post_id is intentionally absent — the bar/panel are hidden in
// the exclusive post-detail view.
export function cloneFilter(f) {
return {
tag_ids: [...f.tag_ids], artist_id: f.artist_id, media_type: f.media_type,
sort: f.sort, platform: f.platform, untagged: f.untagged,
no_artist: f.no_artist, date_from: f.date_from, date_to: f.date_to,
similar_to: f.similar_to,
}
}
export function filterToQuery(f) {
const q = {}
if (f.tag_ids?.length) q.tag_id = f.tag_ids.join(',')
if (f.artist_id) q.artist_id = String(f.artist_id)
if (f.media_type) q.media = f.media_type
if (f.sort && f.sort !== 'newest') q.sort = f.sort
if (f.platform) q.platform = f.platform
if (f.untagged) q.untagged = '1'
if (f.no_artist) q.no_artist = '1'
if (f.date_from) q.date_from = f.date_from
if (f.date_to) q.date_to = f.date_to
if (f.similar_to) q.similar_to = String(f.similar_to)
return q
}
function mergeGroups(existing, incoming) { function mergeGroups(existing, incoming) {
// Merge sequential groups with the same (year, month) instead of duplicating. // Merge sequential groups with the same (year, month) instead of duplicating.
const merged = [...existing] const merged = [...existing]
+3 -3
View File
@@ -18,9 +18,9 @@ export const useModalStore = defineStore('modal', () => {
const inflight = useInflightToken() const inflight = useInflightToken()
// Post-scoped cycle. When set, prev/next cycles within this array // Post-scoped cycle. When set, prev/next cycles within this array
// (used by PostCard image clicks — the modal is scoped to that post's // (used by PostCard's expanded-mosaic PostImageGrid clicks). When
// images). When null, prev/next falls back to current.value.neighbors // null, prev/next falls back to current.value.neighbors (the
// (the gallery-store-driven /api/gallery/image/<id> neighbors). // gallery-store-driven /api/gallery/image/<id> neighbors).
const postImageIds = ref(null) const postImageIds = ref(null)
const postImageIndex = ref(0) const postImageIndex = ref(0)
+88 -123
View File
@@ -1,42 +1,37 @@
import { defineStore } from 'pinia' import { defineStore } from 'pinia'
import { ref, computed } from 'vue' import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js' import { useApi } from '../composables/useApi.js'
import { preloadImage } from '../utils/preloadImage.js'
// Buffered producer/consumer so the cascade cadence is decoupled from fetch // Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast
// latency (operator-flagged 2026-06-04). The OLD pipeline trickled each batch // but risked later chunks arriving first — undesirable even when each
// right after its fetch and bet the next round-trip would finish inside the // chunk is a random sample. Switched to a PIPELINE: only one fetch in
// ~240ms trickle window; when a fetch ran long (TABLESAMPLE hits random, // flight at any moment, but the next fetch kicks off as soon as the
// sometimes-cold blocks; RTT jitter) the animation starved and the view // previous one resolves (NOT after its trickle finishes). The next RTT
// "burped" out a clump of images. Now: // overlaps with the current batch's trickle, hiding the per-batch
// - PRODUCER (_fill): races ahead fetching batches into `queue` up to a // round-trip behind the visible animation cadence. Responses arrive in
// target depth, refilling whenever the queue dips below BUFFER_MIN. It // fire-order, so no out-of-order rendering surprises.
// also kicks off the image PRELOAD for each queued item so decoding //
// pipelines ahead of the reveal. // Smaller PAGE (3 vs 5) → first chunk's items appear sooner: a chunk of
// - CONSUMER (_drain): pops ONE item, WAITS for its thumbnail to be fully // 3 trickles in 240 ms, well within one RTT, so by the time chunk 2 is
// decoded, then reveals it — at most one per CADENCE_MS. Because the // in-hand the trickle is just finishing. Total wall-clock is roughly
// producer preloads ahead, the decode-wait is usually already satisfied, // RTT + N × max(trickle_time, RTT); APPEND_DELAY_MS keeps the visible
// so the reveal stays evenly paced without idling the network. // cadence smooth throughout.
// The decode-gate is what makes the showcase's signature cascade land each
// tile fully-loaded (the flip-in animates a real image, never a gray
// placeholder); the single-item reveal keeps it strictly one-at-a-time
// (operator-flagged 2026-06-04). The very first image still waits on the first
// fetch + decode (a cold TABLESAMPLE is a separate, query-side concern);
// everything after it is buffer-smoothed.
const PAGE = 3 const PAGE = 3
const CADENCE_MS = 160 // floor between fully-loaded reveals (doubled const INITIAL_BATCHES = 20
// 2026-06-04 — slower, more deliberate cadence) const APPEND_DELAY_MS = 80 // ≈ the MasonryGrid stagger animation (70 ms)
const PRIME = 6 // items buffered before the drain starts // Operator-flagged 2026-06-01: scrolling the showcase eventually hit a
const BUFFER_TARGET = 30 // producer tops the queue up to this // premature "End." because /api/showcase returns a *random sample* and
const BUFFER_MIN = 12 // ...and refills once the queue dips below this // after enough scrolling the `seen` Set accumulated enough to fully
const INITIAL_COUNT = 60 // cascade length on load / shuffle // collide with a 3-item batch. The showcase is supposed to be endless;
const SCROLL_COUNT = 15 // cascade length per infinite-scroll demand // only a genuinely empty API response (library has zero images) should
// Showcase is endless by design (random sample); an unlucky all-duplicate // mark it exhausted. Retry up to FETCH_RETRY_CAP times on all-dupe
// batch must be retried — only a genuinely empty API response is exhaustion. // batches; only flip `exhausted` when the API returns 0 items OR every
// retry came back dupe-only (graceful fallback for tiny libraries
// where retries will keep returning the same handful of items).
const FETCH_RETRY_CAP = 8 const FETCH_RETRY_CAP = 8
function _sleep(ms) { return new Promise((r) => setTimeout(r, ms)) } function _sleep(ms) { return new Promise(r => setTimeout(r, ms)) }
export const useShowcaseStore = defineStore('showcase', () => { export const useShowcaseStore = defineStore('showcase', () => {
@@ -47,125 +42,95 @@ export const useShowcaseStore = defineStore('showcase', () => {
const exhausted = ref(false) const exhausted = ref(false)
const seen = new Set() const seen = new Set()
// Internal buffer (not reactive — the consumer is what feeds the UI via // Sequence token: every call to loadInitial bumps this. _trickleAppend
// images.value). // bails between items if its captured seq is no longer current — guards
let queue = [] // against a fast shuffle / mount-then-shuffle from interleaving two
// id -> Promise that settles when the thumbnail is paint-ready. Started by // trickles into the same images.value.
// the producer so decoding runs ahead of the reveal; awaited by the consumer
// so no tile is shown before its image is loaded.
let _preloads = new Map()
// Sequence token: shuffle / re-mount bumps it so in-flight producers and
// the drain bail instead of interleaving two runs into one images list.
let _seq = 0 let _seq = 0
let _filling = false
let _draining = false
function _preload(item) { async function _trickleAppend(items, mySeq) {
if (!_preloads.has(item.id)) _preloads.set(item.id, preloadImage(item.thumbnail_url)) for (const item of items) {
return _preloads.get(item.id) if (mySeq !== _seq) return
} if (seen.has(item.id)) continue
seen.add(item.id)
// One batch, retried while the random sample comes back all-duplicates. images.value.push(item)
// Returns the fresh items, or null when the API is genuinely empty. await _sleep(APPEND_DELAY_MS)
async function _fetchFresh(mySeq) {
for (let attempt = 0; attempt < FETCH_RETRY_CAP; attempt++) {
if (mySeq !== _seq) return []
const body = await api
.get('/api/showcase', { params: { limit: PAGE } })
.catch((e) => {
error.value = error.value || (e.message || String(e))
return null
})
if (mySeq !== _seq) return []
const items = (body && body.images) || []
if (items.length === 0) return null
const fresh = items.filter((i) => !seen.has(i.id))
fresh.forEach((f) => seen.add(f.id))
if (fresh.length) return fresh
}
return null // retry cap hit → tiny library, treat as exhausted
}
// Producer: top the buffer up to `target`. Single-flight.
async function _fill(mySeq, target) {
if (_filling) return
_filling = true
try {
while (mySeq === _seq && !exhausted.value && queue.length < target) {
const batch = await _fetchFresh(mySeq)
if (mySeq !== _seq) return
if (batch === null) { exhausted.value = true; return }
queue.push(...batch)
batch.forEach(_preload) // pipeline decoding ahead of the reveal
}
} finally {
_filling = false
} }
} }
// Consumer: show `count` items at a fixed cadence, topping up the buffer as // Single batch — used by infinite-scroll appends. Trickles its items
// it drains. Single-flight so the initial cascade and scroll appends can't // in for the same one-at-a-time cadence as the initial load. Retries
// interleave. // up to FETCH_RETRY_CAP times when the API's random sample comes back
async function _drain(mySeq, count) { // all-duplicates (the showcase is endless by design; only a genuinely
if (_draining) return // empty API response should mark it exhausted, not an unlucky sample).
_draining = true async function fetchPage() {
if (loading.value) return
loading.value = true loading.value = true
error.value = null
try { try {
for (let shown = 0; shown < count && mySeq === _seq; shown++) { for (let attempt = 0; attempt < FETCH_RETRY_CAP; attempt++) {
if (!exhausted.value && queue.length < BUFFER_MIN) { const body = await api.get('/api/showcase', { params: { limit: PAGE } })
_fill(mySeq, BUFFER_TARGET) // topup, no await const items = body.images || []
// API genuinely empty → library is empty / endpoint exhausted.
if (items.length === 0) { exhausted.value = true; return }
const fresh = items.filter(i => !seen.has(i.id))
if (fresh.length > 0) {
await _trickleAppend(fresh, _seq)
return
} }
let guard = 0 // All-dupes batch — keep trying. Showcase is endless by intent.
while (queue.length === 0 && !exhausted.value && mySeq === _seq) {
await _sleep(20)
if (++guard > 500) break // 10s starvation safety net
}
if (mySeq !== _seq) return
if (queue.length === 0) return // exhausted + empty
const item = queue.shift()
await _preload(item) // reveal only once the thumbnail is fully decoded
if (mySeq !== _seq) return
images.value.push(item)
await _sleep(CADENCE_MS)
} }
// Retry cap hit with zero fresh items: library is probably much
// smaller than the running `seen` set, fall back to exhausted so
// the UI stops trying. Operator can shuffle to reset `seen`.
exhausted.value = true
} catch (e) {
error.value = e.message || String(e)
} finally { } finally {
if (mySeq === _seq) { loading.value = false
_draining = false
loading.value = false
}
} }
} }
function _fetchOne() {
return api.get('/api/showcase', { params: { limit: PAGE } }).catch(e => {
error.value = error.value || (e.message || String(e))
return null
})
}
// Reset state and pipeline INITIAL_BATCHES fetches: only one in flight
// at a time, but kick off the next one as soon as the previous resolves
// (NOT after its trickle finishes), so the next RTT runs concurrently
// with the current batch's trickle. Responses arrive in fire-order, so
// items always render in the order they were fetched — no out-of-order
// surprises from parallel races.
async function loadInitial() { async function loadInitial() {
_seq += 1 _seq += 1
const mySeq = _seq const mySeq = _seq
images.value = [] images.value = []
queue = []
_preloads = new Map()
seen.clear() seen.clear()
exhausted.value = false exhausted.value = false
error.value = null error.value = null
_filling = false
_draining = false
loading.value = true loading.value = true
try { try {
// Prime a small buffer before the cascade starts so it doesn't starve let nextFetch = _fetchOne()
// at the front; the drain's own topup grows it to BUFFER_TARGET. for (let i = 0; i < INITIAL_BATCHES; i++) {
await _fill(mySeq, PRIME) if (mySeq !== _seq) return
if (mySeq !== _seq) return const body = await nextFetch
if (queue.length === 0 && exhausted.value) return // empty library // Fire the NEXT fetch immediately so its RTT overlaps the trickle.
await _drain(mySeq, INITIAL_COUNT) if (i + 1 < INITIAL_BATCHES) nextFetch = _fetchOne()
if (!body || !body.images || body.images.length === 0) {
exhausted.value = true
break
}
await _trickleAppend(body.images, mySeq)
}
if (mySeq === _seq && images.value.length === 0) exhausted.value = true
} finally { } finally {
if (mySeq === _seq) loading.value = false if (mySeq === _seq) loading.value = false
} }
} }
async function fetchPage() {
// Infinite-scroll demand — append another cascade of SCROLL_COUNT.
if (_draining || exhausted.value) return
await _drain(_seq, SCROLL_COUNT)
}
async function shuffle() { async function shuffle() {
await loadInitial() await loadInitial()
} }
+9 -56
View File
@@ -32,31 +32,6 @@ export const useSourcesStore = defineStore('sources', () => {
if (artistId != null) byArtist.value.delete(artistId) if (artistId != null) byArtist.value.delete(artistId)
} }
// Patch a single updated source into every cached array IN PLACE, rather than
// invalidating + refetching the whole list. The full refetch was what locked
// the Subscriptions UI and collapsed the expanded row after every inline
// action (check / backfill / recover / toggle) — operator-flagged 2026-06-07.
// Map.set on a Vue-reactive ref(Map) triggers re-render; we hand each cache a
// NEW array so the table diffs just the one changed row.
function _patchSource(updated) {
if (!updated || updated.id == null) return
for (const [key, arr] of byArtist.value) {
const i = arr.findIndex((s) => s.id === updated.id)
if (i === -1) continue
const next = arr.slice()
next[i] = { ...arr[i], ...updated }
byArtist.value.set(key, next)
}
}
function _dropSource(id) {
for (const [key, arr] of byArtist.value) {
if (arr.some((s) => s.id === id)) {
byArtist.value.set(key, arr.filter((s) => s.id !== id))
}
}
}
async function create(payload) { async function create(payload) {
const body = await api.post('/api/sources', { body: payload }) const body = await api.post('/api/sources', { body: payload })
_invalidate(payload.artist_id) _invalidate(payload.artist_id)
@@ -65,13 +40,13 @@ export const useSourcesStore = defineStore('sources', () => {
async function update(id, patch, artistIdHint = null) { async function update(id, patch, artistIdHint = null) {
const body = await api.patch(`/api/sources/${id}`, { body: patch }) const body = await api.patch(`/api/sources/${id}`, { body: patch })
_patchSource(body) _invalidate(artistIdHint ?? body.artist_id)
return body return body
} }
async function remove(id, artistIdHint = null) { async function remove(id, artistIdHint = null) {
await api.delete(`/api/sources/${id}`) await api.delete(`/api/sources/${id}`)
_dropSource(id) _invalidate(artistIdHint)
} }
async function findOrCreateArtist(name) { async function findOrCreateArtist(name) {
@@ -110,33 +85,14 @@ export const useSourcesStore = defineStore('sources', () => {
} }
} }
// Plan #693: start/stop a run-until-done backfill. 'start' walks the full // Plan #544: arm a source for backfill mode. The next `runs` download
// post history in time-boxed chunks until it reaches the bottom (then the // runs (default 3) walk gallery-dl's full post history instead of
// source shows backfill_state 'complete'); 'stop' cancels back to tick mode. // exiting early at the first contiguous archived block.
async function startBackfill(id, artistIdHint = null) { async function setBackfill(id, runs = 3, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'start' } }) const body = await api.post(`/api/sources/${id}/backfill`, { body: { runs } })
_patchSource(body) _invalidate(artistIdHint ?? body.artist_id)
return body return body
} }
async function stopBackfill(id, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'stop' } })
_patchSource(body)
return body
}
// Plan #697: arm a recovery walk — a backfill that bypasses the Patreon
// seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate them
// under the current pHash threshold. Stop with stopBackfill (shared lifecycle).
async function recoverSource(id, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'recover' } })
_patchSource(body)
return body
}
// Plan #708 B4: dry-run preview — count what a backfill WOULD download
// (native platforms), without downloading. Read-only, so no cache invalidate.
async function previewSource(id) {
return await api.post(`/api/sources/${id}/preview`)
}
function sourcesByArtistGrouped() { function sourcesByArtistGrouped() {
// returns [{artist: {id,name,slug}, sources: [...]}, ...] // returns [{artist: {id,name,slug}, sources: [...]}, ...]
@@ -164,10 +120,7 @@ export const useSourcesStore = defineStore('sources', () => {
loadAll, loadForArtist, loadAll, loadForArtist,
create, update, remove, create, update, remove,
checkNow, checkNow,
startBackfill, setBackfill,
stopBackfill,
recoverSource,
previewSource,
findOrCreateArtist, autocompleteArtist, findOrCreateArtist, autocompleteArtist,
loadScheduleStatus, loadScheduleStatus,
sourcesByArtistGrouped, sourcesByArtistGrouped,
+5 -28
View File
@@ -34,20 +34,10 @@ export const useTagStore = defineStore('tags', () => {
} }
async function loadFandoms() { async function loadFandoms() {
// List ALL fandoms via the cursor-paged tag directory. (#712: the old impl fandomCache.value = await api.get('/api/tags/autocomplete', {
// abused /tags/autocomplete with q=' ', which the backend strips to empty params: { q: ' ', kind: 'fandom', limit: 200 }
// and returns [] — so the picker showed no existing fandoms.) })
const all = [] return fandomCache.value
let cursor = null
do {
const params = { kind: 'fandom', limit: 200 }
if (cursor) params.cursor = cursor
const body = await api.get('/api/tags/directory', { params })
all.push(...(body.cards || []))
cursor = body.next_cursor
} while (cursor)
fandomCache.value = all
return all
} }
async function createFandom(name) { async function createFandom(name) {
@@ -59,21 +49,8 @@ export const useTagStore = defineStore('tags', () => {
return fandom return fandom
} }
// Set / change / clear a character tag's fandom. fandomId null clears it.
// Throws ApiError (status 409, body.target) on a name collision in the
// target fandom; pass { merge: true } to resolve it by merging this tag
// into the existing character. Returns the updated/surviving tag.
async function setFandom(tagId, fandomId, { merge = false } = {}) {
const body = { fandom_id: fandomId ?? null }
if (merge) body.merge = true
return await api.patch(`/api/tags/${tagId}`, { body })
}
function kindOptions() { return KIND_OPTIONS } function kindOptions() { return KIND_OPTIONS }
function colorFor(kind) { return KIND_COLOR[kind] || 'on-surface' } function colorFor(kind) { return KIND_COLOR[kind] || 'on-surface' }
return { return { fandomCache, autocomplete, loadFandoms, createFandom, kindOptions, colorFor }
fandomCache, autocomplete, loadFandoms, createFandom, setFandom,
kindOptions, colorFor
}
}) })
-21
View File
@@ -1,21 +0,0 @@
// Resolve once the image at `url` is fully loaded AND decoded (paint-ready),
// or after `timeoutMs` as a safety net so a broken/slow image never stalls a
// cascade. Never rejects — the caller only cares that it's safe to reveal.
//
// Used by the showcase cascade to gate each tile's entry animation on the
// thumbnail actually being ready, so the flip-in plays on a real image rather
// than on a gray placeholder (operator-flagged 2026-06-04).
export function preloadImage (url, timeoutMs = 4000) {
return new Promise((resolve) => {
let done = false
const finish = () => { if (!done) { done = true; resolve() } }
const img = new Image()
img.onload = finish
img.onerror = finish
img.src = url
// decode() resolves at paint-ready (a beat after onload); prefer it when
// available, but onload/onerror/timeout all still settle the promise.
if (typeof img.decode === 'function') img.decode().then(finish).catch(() => {})
setTimeout(finish, timeoutMs)
})
}
-21
View File
@@ -1,21 +0,0 @@
// Helpers for deciding whether a keyboard shortcut should fire or yield to a
// focused text field.
export function isTextEntry (el) {
if (!el) return false
const tag = el.tagName
return tag === 'INPUT' || tag === 'TEXTAREA' || el.isContentEditable
}
export function hasText (el) {
if (!el) return false
if (el.isContentEditable) return (el.textContent || '').length > 0
return (el.value || '').length > 0
}
// Prev/next arrow navigation should fire UNLESS focus is in a text entry that
// already has content — in that case the arrows belong to the caret so it can
// move through the text. An empty (or non-text) target still navigates.
export function arrowNavAllowed (target) {
return !(isTextEntry(target) && hasText(target))
}
+1 -25
View File
@@ -20,22 +20,6 @@
:last-added="store.lastAdded" :last-added="store.lastAdded"
/> />
<v-container fluid class="pt-2 pb-4"> <v-container fluid class="pt-2 pb-4">
<!-- "N new since last visit" banner. Visible only on the initial
load that triggered the visit-mark; dismissable via close
button or by switching tabs. Re-entry only re-shows if more
content has arrived (overview returns 0 immediately after a
previous visit). -->
<v-alert
v-if="unseenBanner"
type="info" variant="tonal" density="compact"
class="mb-3" closable
@click:close="unseenBanner = false"
>
<span class="fc-artist__unseen-msg">
<strong>{{ store.overview.unseen_count_at_visit }}</strong>
new since last visit
</span>
</v-alert>
<v-window v-model="tab"> <v-window v-model="tab">
<v-window-item value="posts"> <v-window-item value="posts">
<ArtistPostsTab <ArtistPostsTab
@@ -55,7 +39,7 @@
</template> </template>
<script setup> <script setup>
import { computed, ref, watch } from 'vue' import { computed, watch } from 'vue'
import { useRoute } from 'vue-router' import { useRoute } from 'vue-router'
import { useArtistStore } from '../stores/artist.js' import { useArtistStore } from '../stores/artist.js'
@@ -76,10 +60,6 @@ const { tab, resolve } = useTabQuery(
() => ((store.postCount ?? 0) > 0 ? 'posts' : 'gallery'), () => ((store.postCount ?? 0) > 0 ? 'posts' : 'gallery'),
) )
// One-shot banner — reset on each new artist-slug load so it re-appears
// when navigating between artists that each have unseen content.
const unseenBanner = ref(false)
watch(slug, async (s) => { watch(slug, async (s) => {
if (!s) return if (!s) return
await store.load(s) await store.load(s)
@@ -87,7 +67,6 @@ watch(slug, async (s) => {
? `${store.overview.name} — FabledCurator` ? `${store.overview.name} — FabledCurator`
: 'FabledCurator' : 'FabledCurator'
tab.value = resolve() tab.value = resolve()
unseenBanner.value = (store.overview?.unseen_count_at_visit || 0) > 0
}, { immediate: true }) }, { immediate: true })
</script> </script>
@@ -95,7 +74,4 @@ watch(slug, async (s) => {
.fc-artist__loading { .fc-artist__loading {
display: flex; justify-content: center; padding: 64px 0; display: flex; justify-content: center; padding: 64px 0;
} }
.fc-artist__unseen-msg {
font-size: 14px;
}
</style> </style>
+1 -5
View File
@@ -76,11 +76,7 @@ onMounted(async () => {
} }
.fc-artists__grid { .fc-artists__grid {
display: grid; display: grid;
/* min(360px, 100%) so a card never exceeds the viewport (phones collapse to grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
one column). 360 keeps a LONE column under ~732px wide, so the 3/1 preview
strip stays ≲244px tall and always fills the card width (no dead space),
while giving 2+ columns on a normal desktop. */
grid-template-columns: repeat(auto-fill, minmax(min(360px, 100%), 1fr));
gap: 12px; gap: 12px;
} }
.fc-artists__sentinel { .fc-artists__sentinel {
+18 -16
View File
@@ -12,14 +12,10 @@
<div class="fc-gallery-layout"> <div class="fc-gallery-layout">
<div class="fc-gallery-layout__main"> <div class="fc-gallery-layout__main">
<PostInfoHeader /> <PostInfoHeader />
<GalleryFilterBar v-if="store.filter.post_id == null" />
<EmptyState v-if="store.isEmpty" /> <EmptyState v-if="store.isEmpty" />
<GalleryGrid v-else @open="openImage" /> <GalleryGrid v-else @open="openImage" />
</div> </div>
<TimelineSidebar <TimelineSidebar v-if="store.images.length > 0" class="fc-gallery-layout__sidebar" />
v-if="store.images.length > 0 && store.filter.similar_to == null"
class="fc-gallery-layout__sidebar"
/>
</div> </div>
<BulkEditorPanel /> <BulkEditorPanel />
@@ -32,7 +28,6 @@ import { useRoute } from 'vue-router'
import { useGalleryStore } from '../stores/gallery.js' import { useGalleryStore } from '../stores/gallery.js'
import { useModalStore } from '../stores/modal.js' import { useModalStore } from '../stores/modal.js'
import GalleryGrid from '../components/gallery/GalleryGrid.vue' import GalleryGrid from '../components/gallery/GalleryGrid.vue'
import GalleryFilterBar from '../components/gallery/GalleryFilterBar.vue'
import TimelineSidebar from '../components/gallery/TimelineSidebar.vue' import TimelineSidebar from '../components/gallery/TimelineSidebar.vue'
import EmptyState from '../components/gallery/EmptyState.vue' import EmptyState from '../components/gallery/EmptyState.vue'
import PostInfoHeader from '../components/gallery/PostInfoHeader.vue' import PostInfoHeader from '../components/gallery/PostInfoHeader.vue'
@@ -44,13 +39,25 @@ const modal = useModalStore()
const sel = useGallerySelectionStore() const sel = useGallerySelectionStore()
const route = useRoute() const route = useRoute()
// The URL query is the single source of truth for filters. Apply it on onMounted(async () => {
// mount and on any query change (filter bar pushes, back button, deep-link). const postId = parseInt(route.query.post_id, 10)
onMounted(() => store.applyFilterFromQuery(route.query)) const tagId = parseInt(route.query.tag_id, 10)
if (!isNaN(postId)) store.setPostFilter(postId)
else if (!isNaN(tagId)) store.setTagFilter(tagId)
await store.loadInitial()
await store.loadTimeline()
})
watch(() => route.query, (q) => { watch(() => route.query.tag_id, (q) => {
sel.clear() // result set changed — selected ids are no longer valid sel.clear() // result set changed — selected ids are no longer valid
store.applyFilterFromQuery(q) const tagId = parseInt(q, 10)
store.setTagFilter(isNaN(tagId) ? null : tagId)
})
watch(() => route.query.post_id, (q) => {
sel.clear() // result set changed — selected ids are no longer valid
const postId = parseInt(q, 10)
store.setPostFilter(isNaN(postId) ? null : postId)
}) })
function openImage(id) { function openImage(id) {
@@ -71,9 +78,4 @@ function openImage(id) {
.fc-gallery-layout { flex-direction: column-reverse; } .fc-gallery-layout { flex-direction: column-reverse; }
.fc-gallery-layout__sidebar { width: 100%; max-height: 200px; } .fc-gallery-layout__sidebar { width: 100%; max-height: 200px; }
} }
/* Phones: the year/month timeline strip eats vertical space and reads poorly
as a horizontal band — drop it; the gallery scroll is the primary nav here. */
@media (max-width: 600px) {
.fc-gallery-layout__sidebar { display: none; }
}
</style> </style>
+5 -6
View File
@@ -44,15 +44,14 @@
</v-window-item> </v-window-item>
<v-window-item value="import"> <v-window-item value="import">
<!-- Order: filters → trigger → recent tasks. Filters hoisted above the <!-- Order: trigger → recent tasks → filters. Tasks sit directly
trigger (operator-flagged 2026-06-04); the task list stays below the trigger so operator sees hit/miss feedback without
directly below the trigger so hit/miss feedback is adjacent to the scrolling past the filter card (operator-flagged 2026-05-25). -->
button that produced it (operator-flagged 2026-05-25). -->
<ImportFiltersForm />
<v-divider class="my-6" />
<ImportTriggerPanel /> <ImportTriggerPanel />
<v-divider class="my-6" /> <v-divider class="my-6" />
<ImportTaskList /> <ImportTaskList />
<v-divider class="my-6" />
<ImportFiltersForm />
</v-window-item> </v-window-item>
<v-window-item value="cleanup"> <v-window-item value="cleanup">
+1 -22
View File
@@ -8,7 +8,7 @@
placeholder="Search tags" clearable style="max-width: 320px;" placeholder="Search tags" clearable style="max-width: 320px;"
/> />
<v-chip-group <v-chip-group
v-model="kind" selected-class="text-accent" :mandatory="false" v-model="kind" selected-class="text-accent" mandatory="false"
> >
<v-chip v-for="k in KINDS" :key="k" :value="k" filter size="small"> <v-chip v-for="k in KINDS" :key="k" :value="k" filter size="small">
{{ k }} {{ k }}
@@ -28,7 +28,6 @@
v-for="c in store.cards" :key="c.id" :card="c" v-for="c in store.cards" :key="c.id" :card="c"
@open="openTag" @rename="onRename" @manage="onManage" @read="onRead" @open="openTag" @rename="onRename" @manage="onManage" @read="onRead"
@merge-with="onMergeWith" @delete="onDeleteTag" @merge-with="onMergeWith" @delete="onDeleteTag"
@set-fandom="onSetFandom"
/> />
</div> </div>
@@ -86,13 +85,6 @@
: ''" : ''"
@confirm="onDeleteTagConfirm" @confirm="onDeleteTagConfirm"
/> />
<v-dialog v-model="fandomDialogOpen" max-width="460">
<FandomSetDialog
v-if="fandomTarget" :tag="fandomTarget"
@updated="onFandomUpdated" @cancel="fandomDialogOpen = false"
/>
</v-dialog>
</v-container> </v-container>
</template> </template>
@@ -106,7 +98,6 @@ import { useInfiniteScroll } from '../composables/useInfiniteScroll.js'
import TagCard from '../components/discovery/TagCard.vue' import TagCard from '../components/discovery/TagCard.vue'
import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue' import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue'
import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue' import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue'
import FandomSetDialog from '../components/modal/FandomSetDialog.vue'
// Must stay a subset of the backend TagKind enum (character, fandom, // Must stay a subset of the backend TagKind enum (character, fandom,
// general, series, archive, post). 'fandom' is this model's // general, series, archive, post). 'fandom' is this model's
@@ -149,18 +140,6 @@ function openTag(tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } }) router.push({ name: 'gallery', query: { tag_id: tagId } })
} }
// Character fandom editing (dots-menu → FandomSetDialog).
const fandomDialogOpen = ref(false)
const fandomTarget = ref(null)
function onSetFandom(card) {
fandomTarget.value = card
fandomDialogOpen.value = true
}
function onFandomUpdated() {
fandomDialogOpen.value = false
store.reset() // reload so the card reflects the new fandom (or its removal)
}
function onManage(id) { function onManage(id) {
router.push({ name: 'series-manage', params: { tagId: id } }) router.push({ name: 'series-manage', params: { tagId: id } })
} }
@@ -1,53 +0,0 @@
// @vitest-environment happy-dom
import { describe, it, expect, vi, afterEach } from 'vitest'
import GalleryFacetPanel from '../../src/components/gallery/GalleryFacetPanel.vue'
import { useGalleryStore } from '../../src/stores/gallery.js'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
const FACETS = {
total: 5,
platforms: [
{ value: 'patreon', count: 3 },
{ value: null, count: 2 }, // unsourced → "No platform" bucket
],
untagged: 4,
no_artist: 1,
date_min: '2020-01-01T00:00:00+00:00',
date_max: '2026-06-01T00:00:00+00:00',
}
function stubFetchOk(body) {
globalThis.fetch = vi.fn(async () => ({
ok: true, status: 200, statusText: 'OK',
text: async () => JSON.stringify(body),
}))
}
describe('GalleryFacetPanel', () => {
afterEach(() => vi.restoreAllMocks())
it('renders platform options with counts and the curation flags', () => {
const pinia = freshPinia()
stubFetchOk(FACETS) // covers the onMounted loadFacets() call
const s = useGalleryStore()
s.facets = FACETS // seed so counts render synchronously
const w = mountComponent(GalleryFacetPanel, { pinia })
const t = w.text()
expect(t).toContain('patreon')
expect(t).toContain('3')
expect(t).toContain('No platform')
expect(t).toContain('Untagged')
expect(t).toContain('4')
expect(t).toContain('No artist')
})
it('shows an en-dash placeholder for a flag count before facets load', () => {
const pinia = freshPinia()
stubFetchOk(FACETS)
useGalleryStore().facets = null // not loaded yet
const w = mountComponent(GalleryFacetPanel, { pinia })
expect(w.text()).toContain('Untagged')
expect(w.text()).toContain('')
})
})
@@ -1,47 +0,0 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import { mount } from '@vue/test-utils'
import GalleryGrid from '../../src/components/gallery/GalleryGrid.vue'
import { useGalleryStore } from '../../src/stores/gallery.js'
import { freshPinia } from '../support/mountComponent.js'
const GIStub = { name: 'GalleryItem', props: ['image'], template: '<div class="gi" />' }
function mountGrid(pinia) {
return mount(GalleryGrid, {
global: {
plugins: [pinia],
stubs: { GalleryItem: GIStub, RouterLink: { template: '<a><slot /></a>' } },
},
})
}
describe('GalleryGrid', () => {
it('renders a flat ranked list when there are no date groups (similar mode)', () => {
const pinia = freshPinia()
const store = useGalleryStore()
// similar-mode shape: images present, date_groups empty, no cursor.
store.images = [
{ id: 1, thumbnail_url: '/a' },
{ id: 2, thumbnail_url: '/b' },
{ id: 3, thumbnail_url: '/c' },
]
store.dateGroups = []
const w = mountGrid(pinia)
expect(w.findAll('.gi').length).toBe(3)
})
it('renders grouped by date when date groups are present', () => {
const pinia = freshPinia()
const store = useGalleryStore()
store.images = [
{ id: 1, thumbnail_url: '/a' },
{ id: 2, thumbnail_url: '/b' },
]
store.dateGroups = [{ year: 2026, month: 6, image_ids: [1, 2] }]
const w = mountGrid(pinia)
expect(w.find('.fc-gallery-grid__date-header').exists()).toBe(true)
expect(w.findAll('.gi').length).toBe(2)
})
})
@@ -1,28 +0,0 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import GalleryItem from '../../src/components/gallery/GalleryItem.vue'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
describe('GalleryItem', () => {
const image = {
id: 7, sha256: 'a'.repeat(64), mime: 'image/jpeg',
width: 100, height: 100,
thumbnail_url: '/images/thumbs/aa/abc.jpg', artist: null,
}
it('reveals the thumbnail only once it has loaded, so tiles fade in individually', async () => {
const pinia = freshPinia()
const w = mountComponent(GalleryItem, { props: { image }, pinia })
const img = w.find('img')
expect(img.exists()).toBe(true)
// Pre-load: the tile must NOT be revealed, so it can fade in when its
// own thumbnail lands rather than pop together with its API batch.
expect(img.classes()).not.toContain('is-loaded')
await img.trigger('load')
expect(img.classes()).toContain('is-loaded')
})
})

Some files were not shown because too many files have changed in this diff Show More