Compare commits

...

42 Commits

Author SHA1 Message Date
bvandeusen 16eb7075c4 Merge pull request 'v26.05.24.1: FC-3g Firefox extension + worker resilience + UI/migration fixes' (#7) from dev into main 2026-05-24 12:52:31 -04:00
bvandeusen 4437488899 fix(tests): unpin test_recover_interrupted_only_old from STUCK_THRESHOLD_MINUTES (was -5min fresh / now -30sec so threshold can move freely)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:40:27 -04:00
bvandeusen 96039fc983 fix(ext): wrap /api/extension/manifest filesystem work in asyncio.to_thread (ruff ASYNC240)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:20:36 -04:00
bvandeusen 9f008af6c8 fix(ci): extension.yml — single-line bot commit message so YAML block scalar doesn't break
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 11:58:39 -04:00
bvandeusen be0f472894 fix(workers): worker-level recovery — autoretry on transient errors + tightened sweep (5min) + import_media_file body-wrap so no path leaves rows stuck in 'processing'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 11:46:36 -04:00
bvandeusen dd4874ae8d fix: 30min wall-clock timeout on backup subprocess + global padding-top on .fc-content so views don't start under the sticky TopNav
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 11:10:04 -04:00
bvandeusen 4ae9815d61 fix(ui): useApi passes FormData through unaltered so multipart uploads work (IR/GS ingest from UI was JSON-stringifying FormData into '{}')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:43:44 -04:00
bvandeusen 4da8538054 fc3g(ci): extension workflow — lint on dev/main, sign+commit XPI on main
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:41:15 -04:00
bvandeusen d85c108cec fc3g(ui): BrowserExtensionCard on Settings → Maintenance — one-click Firefox install + key copy/rotate
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:40:56 -04:00
bvandeusen cd838ec904 fc3g: integration tests for /api/extension/* and /extension/<filename>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:40:00 -04:00
bvandeusen 2e1aaffd93 fc3g: serve /extension/<filename> with x-xpinstall MIME so Firefox installs in one click
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:39:10 -04:00
bvandeusen 2065672a31 fc3g: /api/extension blueprint — quick-add-source + manifest endpoints, registered
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:38:35 -04:00
bvandeusen 46d199450d fc3g: ExtensionService.quick_add_source — derive platform+slug, find-or-create Artist+Source
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:37:39 -04:00
bvandeusen 379445c244 fc3g(ext): content script — floating 'Add to FabledCurator' button on artist pages
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:37:02 -04:00
bvandeusen b067a3eec1 fc3g(ext): options page — FC URL + API key form with test-connection button
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:36:31 -04:00
bvandeusen b7832c941d fc3g(ext): popup UI — platforms + sources tabs, export-all CTA, status indicator
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:35:55 -04:00
bvandeusen df82abe75e fc3g(ext): background script — message router, Discord token capture, Pixiv OAuth
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:34:11 -04:00
bvandeusen c1d3046778 fc3g(ext): cookies extraction + FC backend API client
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:33:26 -04:00
bvandeusen 9b01b19666 fc3g(ext): platform registry + artist-page URL patterns
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:32:47 -04:00
bvandeusen a06c4f009f fc3g(ext): bootstrap extension/ skeleton (manifest, package, icon, README)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 23:32:23 -04:00
bvandeusen 885dcf64f3 Merge pull request 'v26.05.24.0: TopNav re-fix (flex 1 1 0 side cells)' (#6) from dev into main 2026-05-23 22:49:29 -04:00
bvandeusen 05b398c352 fix(ui): TopNav side cells use flex: 1 1 0 (equal weight) so middle link block stays centered regardless of action-slot content
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 22:24:41 -04:00
bvandeusen f2f6b6d25e Merge pull request 'v26.05.23.3: dogfood UX polish + accurate active-batch stats' (#5) from dev into main 2026-05-23 22:05:59 -04:00
bvandeusen 7a5a71471e fix(scan): auto-finalize empty scans + system_stats picks running batch with actual work
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 21:25:57 -04:00
bvandeusen 38a45baad5 feat(ui): click outside the image in the viewer modal closes it (parity with IR)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 21:24:06 -04:00
bvandeusen c9ddcd0f60 fix(ui): viewer modal — haze background through to the image, fit-to-view at zoom 1
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 21:21:02 -04:00
bvandeusen 95bc761a69 fix(ui): TopNav uses 1fr/auto/1fr grid so Gallery/Showcase action buttons don't shift the centered links
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 21:16:37 -04:00
bvandeusen 6acf273267 fix(ui): poll import task list during active batch so Recent Import Tasks doesn't sit stale
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 21:12:17 -04:00
bvandeusen 0822240fde Merge pull request 'v26.05.23.2: serve /images + artist cleanup migrator' (#4) from dev into main 2026-05-23 12:19:16 -04:00
bvandeusen f5efbea053 feat(fc5): /api/migrate/cleanup — delete every image attributed to one artist, files + thumbs + DB
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 12:09:56 -04:00
bvandeusen f653c26680 fix(web): serve /images/<path> from disk so thumbnails+originals render instead of grey placeholders
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 11:51:08 -04:00
bvandeusen 27f7f3fd01 Merge pull request 'v26.05.23.1: migration durability + dogfood UX' (#3) from dev into main 2026-05-23 11:21:33 -04:00
bvandeusen 505dca1b4d fix(ui): mount ImageViewer globally so tile clicks open as overlay, not route-redirect to /gallery
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 11:13:19 -04:00
bvandeusen 061dc9e605 fix(tasks): collapse double blank line between imports and IMAGES_ROOT in import_file.py (ruff I001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 11:05:35 -04:00
bvandeusen 7d8b9c3d90 fix(tasks): share one sync engine per worker process to stop connection-pool leak
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 10:44:16 -04:00
bvandeusen 8f25b27315 fix(import): scan_directory skips paths with existing non-failed ImportTask
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 10:25:27 -04:00
bvandeusen d6eba8e4bd feat(ui): redesign /subscriptions as v-data-table with expandable source rows (GS-style)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 10:17:17 -04:00
bvandeusen 6f68bf5fa7 feat(ui): bump thumbnail sizes 1.5x, make Settings fluid, add subscription_count stat
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 23:18:22 -04:00
bvandeusen ec44c653fe fc5: strip SQLAlchemy +psycopg/+asyncpg driver suffix when passing URL to pg_dump/psql
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 22:54:19 -04:00
bvandeusen c5bf564f53 Merge dev: v26.05.23.0 migration follow-ups (#2)
pg_dump + zstd in runtime image, lift Quart body cap to 1 GiB. See PR #2.
2026-05-22 22:37:06 -04:00
bvandeusen 8188985029 fc5: lift Quart request body cap to 1 GiB so large IR exports don't 413
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 22:23:29 -04:00
bvandeusen 7f26217166 fc5: install postgresql-client + zstd in runtime image so backup/rollback work
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 22:17:18 -04:00
58 changed files with 3077 additions and 323 deletions
+59
View File
@@ -0,0 +1,59 @@
name: extension
on:
push:
branches: [dev, main]
paths: ['extension/**']
workflow_dispatch:
jobs:
lint:
runs-on: python-ci
container:
image: node:22-bookworm-slim
steps:
- uses: actions/checkout@v4
- name: Install web-ext
run: cd extension && npm install --no-save --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
sign-and-publish:
needs: lint
if: github.ref == 'refs/heads/main'
runs-on: python-ci
container:
image: node:22-bookworm-slim
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.RELEASE_TOKEN }}
- name: Install web-ext + git
run: |
apt-get update && apt-get install -y --no-install-recommends git ca-certificates
cd extension && npm install --no-save --no-audit --no-fund
- name: Sign XPI
run: cd extension && npm run sign
env:
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
- name: Commit signed XPI to frontend/public/extension/
run: |
set -e
XPI=$(ls extension/web-ext-artifacts/fabledcurator-*.xpi | head -1)
if [ -z "$XPI" ]; then
echo "No XPI produced by web-ext sign — exiting"
exit 1
fi
mkdir -p frontend/public/extension
cp "$XPI" frontend/public/extension/
# Also copy as -latest.xpi so the FC server can serve a stable URL.
cp "$XPI" "frontend/public/extension/fabledcurator-latest.xpi"
git config user.name "FC extension CI"
git config user.email "noreply@fabledsword.com"
git add frontend/public/extension/
if git diff --cached --quiet; then
echo "No changes to commit"
else
git commit -m "ext: publish signed XPI $(basename $XPI)"
git push origin HEAD:main
fi
+5 -2
View File
@@ -16,12 +16,15 @@ ENV PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
# System deps: ffmpeg (transcode + thumbnails, used in FC-2), unar (archives, FC-2),
# libpq for psycopg, image libs.
# System deps: ffmpeg (transcode + thumbnails, FC-2), unar (archives, FC-2),
# libpq for psycopg, postgresql-client + zstd for FC-5 backup/restore
# (pg_dump + tar --zstd), image libs.
RUN apt-get update && apt-get install -y --no-install-recommends \
ffmpeg \
unar \
libpq5 \
postgresql-client \
zstd \
libjpeg62-turbo \
libwebp7 \
libpng16-16 \
+6
View File
@@ -23,6 +23,12 @@ def create_app() -> Quart:
app = Quart(__name__)
app.secret_key = cfg.secret_key
# FC-5: legacy IR ingest JSON can run to tens of MB (hundreds of
# thousands of image_tag_associations). Werkzeug's default form
# memory cap is 500KB; raise both ceilings so the multipart upload
# for /api/migrate/ir_ingest doesn't 413.
app.config["MAX_CONTENT_LENGTH"] = 1024 * 1024 * 1024 # 1 GB
app.config["MAX_FORM_MEMORY_SIZE"] = 1024 * 1024 * 1024 # 1 GB
for bp in all_blueprints():
app.register_blueprint(bp)
+2
View File
@@ -21,6 +21,7 @@ def all_blueprints() -> list[Blueprint]:
from .attachments import attachments_bp
from .credentials import credentials_bp
from .downloads import downloads_bp
from .extension import extension_bp
from .gallery import gallery_bp
from .import_admin import import_admin_bp
from .migrate import migrate_bp
@@ -53,5 +54,6 @@ def all_blueprints() -> list[Blueprint]:
platforms_bp,
posts_bp,
credentials_bp,
extension_bp,
downloads_bp,
]
+114
View File
@@ -0,0 +1,114 @@
"""FC-3g: /api/extension — quick-add-source for the Firefox extension
+ install-time manifest for the Settings card.
"""
from __future__ import annotations
import asyncio
import hashlib
import re
from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import AppSetting
from ..services.extension_service import (
ExtensionService,
InvalidUrlError,
UnknownPlatformError,
)
from ..services.source_service import KNOWN_PLATFORMS
extension_bp = Blueprint("extension", __name__, url_prefix="/api/extension")
# Default XPI directory; tests override via monkeypatching this module-
# level constant.
XPI_DIR = Path("/app/frontend/dist/extension")
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
async def _ext_key_required(session) -> bool:
"""Unlike /api/credentials (which accepts the browser path with no
header), quick-add-source writes server state and must be explicitly
authenticated."""
supplied = request.headers.get("X-Extension-Key")
if supplied is None:
return False
stored = (await session.execute(
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
)).scalar_one_or_none()
return stored is not None and supplied == stored
def _extract_version(xpi_name: str) -> str:
m = _XPI_VERSION_RE.search(xpi_name)
return m.group("version") if m else "unknown"
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as fp:
for chunk in iter(lambda: fp.read(65536), b""):
h.update(chunk)
return h.hexdigest()
@extension_bp.route("/quick-add-source", methods=["POST"])
async def quick_add_source():
body = await request.get_json(silent=True)
if not isinstance(body, dict):
return _bad("invalid_body", detail="body must be a JSON object")
url = body.get("url")
if not isinstance(url, str) or not url.strip():
return _bad("invalid_body", detail="url is required")
async with get_session() as session:
if not await _ext_key_required(session):
return _bad("unauthorized", status=401)
try:
result = await ExtensionService(session).quick_add_source(url)
except UnknownPlatformError as exc:
return _bad(
"unknown_platform",
detail=str(exc),
known=sorted(KNOWN_PLATFORMS),
)
except InvalidUrlError as exc:
return _bad("invalid_url", detail=str(exc))
return jsonify(result), (201 if result["created_source"] else 200)
def _read_manifest_sync() -> dict | None:
"""All the filesystem-touching work for /api/extension/manifest,
in a sync helper so the async route can dispatch it via
asyncio.to_thread (ASYNC240: no pathlib I/O in async functions)."""
if not XPI_DIR.is_dir():
return None
xpis = sorted(XPI_DIR.glob("fabledcurator-*.xpi"), key=lambda p: p.stat().st_mtime)
if not xpis:
return None
latest = xpis[-1]
return {
"installed": True,
"version": _extract_version(latest.name),
"xpi_url": f"/extension/{latest.name}",
"latest_url": "/extension/fabledcurator-latest.xpi",
"sha256": _sha256(latest),
}
@extension_bp.route("/manifest", methods=["GET"])
async def extension_manifest():
info = await asyncio.to_thread(_read_manifest_sync)
if info is None:
return jsonify({"installed": False}), 404
return jsonify(info)
+2 -2
View File
@@ -21,10 +21,10 @@ migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate")
_VALID_KINDS = frozenset({
"backup", "gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "rollback",
"ml_queue", "verify", "rollback", "cleanup",
})
_INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"})
_APPLY_KINDS = frozenset({"gs_ingest", "ir_ingest", "tag_apply", "rollback"})
_APPLY_KINDS = frozenset({"gs_ingest", "ir_ingest", "tag_apply", "rollback", "cleanup"})
def _bad(error: str, *, status: int = 400, **extra):
+19 -3
View File
@@ -6,7 +6,7 @@ from quart import Blueprint, jsonify, request
from sqlalchemy import func, select
from ..extensions import get_session
from ..models import AppSetting, ImageRecord, ImportBatch, ImportSettings, ImportTask, Tag
from ..models import AppSetting, Artist, ImageRecord, ImportBatch, ImportSettings, ImportTask, Tag
settings_bp = Blueprint("settings", __name__, url_prefix="/api")
@@ -118,6 +118,9 @@ async def system_stats():
storage_bytes = (
(await session.execute(select(func.coalesce(func.sum(ImageRecord.size_bytes), 0)))).scalar_one()
)
subscription_count = (await session.execute(
select(func.count(Artist.id)).where(Artist.is_subscription.is_(True))
)).scalar_one()
# Task counts grouped by status
status_rows = (
@@ -138,11 +141,23 @@ async def system_stats():
).all()
integrity_counts = {row[0]: row[1] for row in integrity_rows}
# Active batch (most recent running)
# Active batch = running batch that still has outstanding work.
# Plain "most recent running" picks a freshly-created scan that
# enqueued zero new files and hides the older batch that's
# actually being processed; the EXISTS clause filters those
# empty batches out.
active_batch_row = (
await session.execute(
select(ImportBatch)
.where(ImportBatch.status == "running")
.where(
ImportBatch.status == "running",
select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists(),
)
.order_by(ImportBatch.started_at.desc())
.limit(1)
)
@@ -163,6 +178,7 @@ async def system_stats():
"total_images": total_images,
"total_tags": total_tags,
"storage_bytes": storage_bytes,
"subscription_count": int(subscription_count),
"tasks": {
"pending": status_counts.get("pending", 0),
"queued": status_counts.get("queued", 0),
+66 -2
View File
@@ -1,14 +1,78 @@
"""Serves the built Vue SPA from frontend/dist/ with history-mode fallback."""
"""Serves the built Vue SPA from frontend/dist/ with history-mode fallback,
the on-disk image library + thumbnails from /images, and the signed
Firefox extension XPI from frontend/dist/extension/.
"""
import re
from pathlib import Path
from quart import Blueprint, send_from_directory
from quart import Blueprint, abort, send_file, send_from_directory
FRONTEND_DIST = Path(__file__).resolve().parent.parent.parent / "frontend" / "dist"
IMAGES_ROOT = Path("/images")
XPI_DIR = FRONTEND_DIST / "extension"
_XPI_NAME_RE = re.compile(r"^fabledcurator-[\w.-]+\.xpi$")
frontend_bp = Blueprint("frontend", __name__)
@frontend_bp.route("/images/<path:subpath>")
async def serve_image(subpath: str):
"""Serve a file from the /images volume (originals + thumbnails).
Without this route the SPA catch-all below would swallow image
requests and return index.html, leaving the browser to render the
aspect-ratio-shaped grey placeholder.
"""
target = (IMAGES_ROOT / subpath).resolve()
# Defend against path-traversal: refuse anything that escapes /images.
try:
target.relative_to(IMAGES_ROOT)
except ValueError:
abort(404)
if not target.is_file():
abort(404)
return await send_from_directory(IMAGES_ROOT, subpath)
@frontend_bp.route("/extension/<filename>")
async def serve_extension(filename: str):
"""Serve the signed FC Firefox extension XPI.
Path whitelist: filename must match fabledcurator-*.xpi. The special
name fabledcurator-latest.xpi serves the most-recently-modified XPI
in the directory.
The application/x-xpinstall MIME tells Firefox to show its native
install prompt instead of downloading the file as a blob.
"""
if not _XPI_NAME_RE.fullmatch(filename):
abort(404)
if not XPI_DIR.is_dir():
abort(404)
if filename == "fabledcurator-latest.xpi":
xpis = sorted(XPI_DIR.glob("fabledcurator-*.xpi"), key=lambda p: p.stat().st_mtime)
if not xpis:
abort(404)
latest = xpis[-1]
return await send_file(
latest, mimetype="application/x-xpinstall",
attachment_filename=latest.name,
)
target = (XPI_DIR / filename).resolve()
try:
target.relative_to(XPI_DIR)
except ValueError:
abort(404)
if not target.is_file():
abort(404)
return await send_file(
target, mimetype="application/x-xpinstall",
attachment_filename=filename,
)
@frontend_bp.route("/")
@frontend_bp.route("/<path:subpath>")
async def serve_spa(subpath: str = ""):
+127
View File
@@ -0,0 +1,127 @@
"""FC-3g: backend support for the Firefox extension.
`ExtensionService.quick_add_source(url)` derives platform + artist
slug from a URL using regex patterns mirrored from
extension/lib/platforms.js, then find-or-creates Artist + Source rows
and returns a JSON-shaped dict for the API layer.
"""
from __future__ import annotations
import re
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
from ..utils.slug import slugify
class UnknownPlatformError(Exception):
"""URL didn't match any platform pattern."""
class InvalidUrlError(Exception):
"""URL was empty or missing a scheme."""
# Mirrored byte-for-byte from extension/lib/platforms.js
# PLATFORM_ARTIST_PATTERNS. Keep these two copies in sync by hand —
# reviewers catch drift.
_PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
("patreon", re.compile(
r"^https?://(?:www\.)?patreon\.com/"
r"(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c/)"
r"(?P<slug>[^/?#]+)/?$",
re.IGNORECASE,
)),
("subscribestar", re.compile(
r"^https?://(?:www\.)?subscribestar\.(?:com|adult)/"
r"(?!feed$|messages$|library$)"
r"(?P<slug>[^/?#]+)/?$",
re.IGNORECASE,
)),
("hentaifoundry", re.compile(
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
re.IGNORECASE,
)),
("deviantart", re.compile(
r"^https?://(?:www\.)?deviantart\.com/"
r"(?!home$|watch\b|tag\b|browse\b)"
r"(?P<slug>[^/?#]+)/?$",
re.IGNORECASE,
)),
("pixiv", re.compile(
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
re.IGNORECASE,
)),
]
class ExtensionService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def quick_add_source(self, url: str) -> dict:
platform, raw_slug = self._derive(url)
artist, created_artist = await self._find_or_create_artist(raw_slug)
source, created_source = await self._find_or_create_source(
artist_id=artist.id, platform=platform, url=url,
)
return {
"source": {
"id": source.id,
"artist_id": source.artist_id,
"platform": source.platform,
"url": source.url,
"enabled": source.enabled,
},
"artist": {
"id": artist.id,
"name": artist.name,
"slug": artist.slug,
},
"created_source": created_source,
"created_artist": created_artist,
}
def _derive(self, url: str) -> tuple[str, str]:
if not isinstance(url, str) or not url.strip():
raise InvalidUrlError("url is empty")
if not url.startswith(("http://", "https://")):
raise InvalidUrlError(f"url must include http:// or https:// scheme: {url!r}")
for platform, pattern in _PLATFORM_PATTERNS:
m = pattern.match(url)
if m:
return platform, m.group("slug")
raise UnknownPlatformError(f"no platform pattern matched {url!r}")
async def _find_or_create_artist(self, raw_name: str) -> tuple[Artist, bool]:
slug = slugify(raw_name)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=raw_name, slug=slug, is_subscription=True)
self.session.add(artist)
await self.session.flush()
return artist, True
async def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> tuple[Source, bool]:
existing = (await self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one_or_none()
if existing is not None:
return existing, False
src = Source(artist_id=artist_id, platform=platform, url=url, enabled=True)
self.session.add(src)
await self.session.flush()
await self.session.commit()
return src, True
+25 -3
View File
@@ -16,6 +16,19 @@ from typing import Any
_BACKUPS_DIRNAME = "_backups"
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy driver suffix so pg_dump/psql accept the URL.
SQLAlchemy uses URLs like `postgresql+psycopg://...` or
`postgresql+asyncpg://...`. libpq tools (pg_dump, psql) only know
the plain `postgresql://` scheme.
"""
for driver in ("postgresql+psycopg", "postgresql+asyncpg", "postgresql+psycopg2"):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path | None = None) -> Path:
# Overridable for tests via monkeypatch.
root = images_root if images_root is not None else Path("/images")
@@ -24,12 +37,21 @@ def _backups_dir(images_root: Path | None = None) -> Path:
return p
_DEFAULT_SUBPROCESS_TIMEOUT_S = 30 * 60 # 30 minutes
def _run_subprocess(cmd: list[str], **kwargs: Any):
# Overridable for tests via monkeypatch.
# Overridable for tests via monkeypatch. Hard wall-clock timeout
# guards against pg_dump / tar / zstd hangs on NFS — without it the
# task pretends to be 'running' forever (operator hit this 2026-05-
# 23 with two backups stuck in MigrationRun). On timeout
# subprocess.run raises TimeoutExpired which the caller surfaces as
# a task error.
return subprocess.run(
cmd,
capture_output=True,
check=True,
timeout=_DEFAULT_SUBPROCESS_TIMEOUT_S,
**{k: v for k, v in kwargs.items() if not k.startswith("_")},
)
@@ -49,7 +71,7 @@ def create_backup(
manifest_path = out_dir / f"fc_{ts}.json"
_run_subprocess(
["pg_dump", "--no-owner", "--no-acl", "-f", str(sql_path), db_url],
["pg_dump", "--no-owner", "--no-acl", "-f", str(sql_path), _libpq_url(db_url)],
_test_ts=ts,
)
_run_subprocess(
@@ -104,7 +126,7 @@ def restore_backup(
tar_path = Path(manifest["tar_path"])
_run_subprocess(
["psql", "-d", db_url, "-f", str(sql_path)],
["psql", "-d", _libpq_url(db_url), "-f", str(sql_path)],
)
# Wipe everything in images_root EXCEPT _backups/ (we'd delete the backup
+182
View File
@@ -0,0 +1,182 @@
"""Targeted cleanup migrator: delete every image attributed to one Artist.
Built for the IR-migration rescue case where the filesystem scan derived
a bogus 'imagerepo' artist from a mismatched bind-mount layout. Every
image attributed to that artist (40k+ rows) needs to be removed — DB
rows, original files under `/images/<bucket>/...`, and thumbnails under
`/images/thumbs/...` — before the operator remounts and re-scans.
CASCADE handles image_tag, image_provenance, series_page, and
tag_suggestion_rejection child rows; import_task.result_image_id is
SET NULL by FK. We also delete ImportTask rows whose source_path starts
with the (still-existing) IR scan prefix so the next scan isn't fooled
by them.
"""
from __future__ import annotations
import logging
from pathlib import Path
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Artist, ImageRecord, ImportBatch, ImportTask
log = logging.getLogger(__name__)
_BATCH_SIZE = 500
def _zero_counts() -> dict:
return {
"rows_processed": 0, "rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0, "conflicts": 0,
}
def _thumb_path(images_root: Path, sha256_hex: str) -> tuple[Path, Path]:
"""Return both possible thumbnail paths (.jpg and .png). We try both
because the extension is chosen at generate-time based on the source
image's mode (alpha → .png, otherwise → .jpg)."""
bucket = sha256_hex[:3]
base = images_root / "thumbs" / bucket / sha256_hex
return base.with_suffix(".jpg"), base.with_suffix(".png")
def _delete_file(path: Path) -> bool:
"""Best-effort unlink; True if the file was actually removed."""
try:
path.unlink(missing_ok=True)
return True
except OSError as exc:
log.warning("cleanup: failed to unlink %s: %s", path, exc)
return False
async def cleanup_artist_async(
db: AsyncSession,
*,
slug: str,
images_root: Path | None = None,
dry_run: bool = False,
source_path_prefix: str | None = None,
) -> dict:
"""Delete every image attributed to the Artist with this slug,
along with the artist row itself and any associated import tasks.
Args:
slug: artist.slug to target (e.g. 'imagerepo').
images_root: defaults to /images.
dry_run: skip filesystem + DB writes; still walk rows for counts.
source_path_prefix: if set, ImportTask rows whose source_path
starts with this string are deleted too (use the IR scan
mount prefix, e.g. '/import/imagerepo').
"""
root = images_root if images_root is not None else Path("/images")
artist = (await db.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
raise ValueError(f"no Artist with slug={slug!r}")
artist_id = artist.id
artist_name = artist.name
total_images = (await db.execute(
select(func.count(ImageRecord.id)).where(ImageRecord.artist_id == artist_id)
)).scalar_one()
counts = _zero_counts()
files_deleted = 0
thumbs_deleted = 0
images_deleted = 0
# Batched delete loop. CASCADE handles image_tag, image_provenance,
# series_page, tag_suggestion_rejection. import_task.result_image_id
# is SET NULL by FK.
while True:
rows = (await db.execute(
select(ImageRecord.id, ImageRecord.path, ImageRecord.sha256)
.where(ImageRecord.artist_id == artist_id)
.limit(_BATCH_SIZE)
)).all()
if not rows:
break
ids = [r.id for r in rows]
counts["rows_processed"] += len(ids)
if not dry_run:
for r in rows:
if r.path:
if _delete_file(Path(r.path)):
files_deleted += 1
if r.sha256:
jpg, png = _thumb_path(root, r.sha256)
if _delete_file(jpg):
thumbs_deleted += 1
if _delete_file(png):
thumbs_deleted += 1
await db.execute(
delete(ImageRecord).where(ImageRecord.id.in_(ids))
)
await db.commit()
images_deleted += len(ids)
if dry_run:
# Nothing was actually deleted from the DB; bail after one
# pass so we don't loop forever.
break
import_tasks_deleted = 0
if source_path_prefix and not dry_run:
# Delete ImportTask rows whose source_path is under the bad mount
# prefix. These are mostly orphaned now (result_image_id was set
# NULL by CASCADE) but their presence still blocks the
# idempotency check in scan_directory if the operator remounts
# the same prefix.
like_pattern = source_path_prefix.rstrip("/") + "/%"
result = await db.execute(
delete(ImportTask).where(ImportTask.source_path.like(like_pattern))
)
import_tasks_deleted = result.rowcount or 0
await db.commit()
# Sweep ImportBatch rows that are now empty.
empty_batches_deleted = 0
if not dry_run:
empty_batch_ids = (await db.execute(
select(ImportBatch.id).where(
~select(ImportTask.id)
.where(ImportTask.batch_id == ImportBatch.id)
.exists()
)
)).scalars().all()
if empty_batch_ids:
result = await db.execute(
delete(ImportBatch).where(ImportBatch.id.in_(empty_batch_ids))
)
empty_batches_deleted = result.rowcount or 0
await db.commit()
# Finally, the artist row.
if not dry_run:
await db.execute(delete(Artist).where(Artist.id == artist_id))
await db.commit()
return {
"counts": counts,
"artist": {"id": artist_id, "name": artist_name, "slug": slug},
"summary": {
"images_targeted": total_images,
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_deleted": import_tasks_deleted,
"empty_batches_deleted": empty_batches_deleted,
"dry_run": dry_run,
},
}
+36
View File
@@ -0,0 +1,36 @@
"""Process-level sync engine for Celery task modules.
Each task module used to call ``create_engine(...)`` on every invocation,
which leaked engines (and Postgres connections) — high-fire-rate tasks
like ``import_media_file`` would exhaust ``max_connections`` within
minutes during a bulk migration.
This module owns one engine per process. Celery prefork forks before any
task runs, so each worker process lazily initializes its own engine on
the first task and reuses it for the rest of its life.
"""
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from ..config import get_config
_ENGINE = None
_SESSIONMAKER = None
def sync_session_factory():
"""Return a process-wide ``sessionmaker`` bound to a single engine."""
global _ENGINE, _SESSIONMAKER
if _SESSIONMAKER is None:
cfg = get_config()
_ENGINE = create_engine(
cfg.database_url_sync,
future=True,
pool_pre_ping=True,
pool_size=5,
max_overflow=5,
pool_recycle=300,
)
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
return _SESSIONMAKER
+13 -1
View File
@@ -4,6 +4,7 @@ import asyncio
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from ..celery_app import celery
@@ -27,7 +28,18 @@ def _async_session_factory():
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
@celery.task(name="backend.app.tasks.download.download_source", bind=True, acks_late=True)
@celery.task(
name="backend.app.tasks.download.download_source",
bind=True,
acks_late=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=10,
retry_backoff_max=120,
retry_jitter=True,
max_retries=3,
soft_time_limit=900,
time_limit=1200,
)
def download_source(self, source_id: int) -> int:
"""Returns the DownloadEvent.id."""
+150 -98
View File
@@ -1,25 +1,27 @@
"""import_media_file task: imports one file via the Importer service and
updates the ImportTask state machine + ImportBatch counters atomically.
Resilience contract (2026-05-24, operator-mandated): once a row has been
flipped to 'processing' inside this task, EVERY exit path MUST flip it
to a terminal state (complete / skipped / failed) or rely on Celery's
autoretry to attempt the work again. No exit path is allowed to leave
the row stuck in 'processing'. The `recover_interrupted_tasks`
maintenance sweep is the safety net (5 min threshold) for the case
where even the failure-marking commit can't be written.
"""
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import create_engine, select, update
from sqlalchemy.orm import sessionmaker
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select, update
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import ImportBatch, ImportSettings, ImportTask
from ..services.importer import Importer
from ..services.thumbnailer import Thumbnailer
def _sync_session_factory():
cfg = get_config()
engine = create_engine(cfg.database_url_sync, future=True, pool_pre_ping=True)
return sessionmaker(engine, expire_on_commit=False)
from ._sync_engine import sync_session_factory as _sync_session_factory
IMAGES_ROOT = Path("/images")
@@ -37,9 +39,54 @@ def _map_result_to_status(result):
return ("failed", False)
@celery.task(name="backend.app.tasks.import_file.import_media_file", bind=True)
def _mark_failed(session, task, error_msg: str) -> None:
"""Best-effort flip of a 'processing' row to 'failed' + batch counter
increment. Wrapped in its own try because if the DB is what just
broke, this commit will also fail — that's why the maintenance sweep
exists as a backstop."""
try:
task.status = "failed"
task.error = error_msg
task.finished_at = datetime.now(UTC)
session.add(task)
session.execute(
update(ImportBatch)
.where(ImportBatch.id == task.batch_id)
.values(failed=ImportBatch.failed + 1)
)
session.commit()
except Exception: # noqa: BLE001 — best-effort, sweep catches the rest
try:
session.rollback()
except Exception: # noqa: BLE001
pass
@celery.task(
name="backend.app.tasks.import_file.import_media_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=360,
)
def import_media_file(self, import_task_id: int) -> dict:
"""Returns a dict so the eager-mode tests can assert without DB."""
"""Returns a dict so the eager-mode tests can assert without DB.
Decorator notes:
- autoretry_for: transient DB / filesystem errors retry with
exponential backoff (5s base, jitter, max 3 attempts). On final
give-up the task raises and acks_late=True (set globally on the
Celery app) does NOT redeliver — the recovery sweep catches the
row instead.
- soft_time_limit (300s) raises SoftTimeLimitExceeded in this
process so the task can mark its row failed before being killed.
- time_limit (360s) is the hard cap; SIGKILL if the soft signal
was swallowed.
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
task = session.get(ImportTask, import_task_id)
@@ -51,99 +98,104 @@ def import_media_file(self, import_task_id: int) -> dict:
session.add(task)
session.commit()
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
import_root = Path(settings.import_scan_path)
batch = session.get(ImportBatch, task.batch_id)
deep = bool(batch and batch.scan_mode == "deep")
importer = Importer(
session=session,
images_root=IMAGES_ROOT,
import_root=import_root,
thumbnailer=Thumbnailer(images_root=IMAGES_ROOT),
settings=settings,
deep=deep,
)
try:
result = importer.import_one(Path(task.source_path))
except Exception as exc: # pragma: no cover — pipeline crash
task.status = "failed"
task.error = f"{type(exc).__name__}: {exc}"
task.finished_at = datetime.now(UTC)
session.execute(
update(ImportBatch)
.where(ImportBatch.id == task.batch_id)
.values(failed=ImportBatch.failed + 1)
)
session.add(task)
session.commit()
return _do_import(session, task, import_task_id)
except SoftTimeLimitExceeded:
_mark_failed(session, task, "soft_time_limit exceeded (>300s)")
raise
except (OperationalError, DBAPIError, OSError):
# Retryable per the decorator; do NOT mark failed (let
# autoretry have a clean go at it). If autoretry exhausts,
# the row stays 'processing' and the maintenance sweep
# flips it within 5 min.
raise
except Exception as exc: # noqa: BLE001 — pipeline crash, mark + re-raise
_mark_failed(session, task, f"{type(exc).__name__}: {exc}")
raise
if result.status in ("imported", "superseded"):
task.status = "complete"
task.result_image_id = result.image_id
counter_col_name = "imported"
counter_col = ImportBatch.imported
elif result.status == "attached":
task.status = "complete"
counter_col_name = "attachments"
counter_col = ImportBatch.attachments
elif result.status == "skipped":
task.status = "skipped"
task.error = (
f"{result.skip_reason.value}: {result.error}"
if result.skip_reason
else result.error
)
task.result_image_id = result.image_id
counter_col_name = "skipped"
counter_col = ImportBatch.skipped
else:
task.status = "failed"
task.error = result.error
counter_col_name = "failed"
counter_col = ImportBatch.failed
task.finished_at = datetime.now(UTC)
def _do_import(session, task, import_task_id: int) -> dict:
"""Actual work, called from inside the resilience wrapper."""
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
import_root = Path(settings.import_scan_path)
batch = session.get(ImportBatch, task.batch_id)
deep = bool(batch and batch.scan_mode == "deep")
importer = Importer(
session=session,
images_root=IMAGES_ROOT,
import_root=import_root,
thumbnailer=Thumbnailer(images_root=IMAGES_ROOT),
settings=settings,
deep=deep,
)
result = importer.import_one(Path(task.source_path))
if result.status in ("imported", "superseded"):
task.status = "complete"
task.result_image_id = result.image_id
counter_col_name = "imported"
counter_col = ImportBatch.imported
elif result.status == "attached":
task.status = "complete"
counter_col_name = "attachments"
counter_col = ImportBatch.attachments
elif result.status == "skipped":
task.status = "skipped"
task.error = (
f"{result.skip_reason.value}: {result.error}"
if result.skip_reason
else result.error
)
task.result_image_id = result.image_id
counter_col_name = "skipped"
counter_col = ImportBatch.skipped
else:
task.status = "failed"
task.error = result.error
counter_col_name = "failed"
counter_col = ImportBatch.failed
task.finished_at = datetime.now(UTC)
session.execute(
update(ImportBatch)
.where(ImportBatch.id == task.batch_id)
.values({counter_col_name: counter_col + 1})
)
session.add(task)
session.commit()
# Enqueue thumbnail + ML for newly imported AND superseded images
# (a superseded row has cleared ML + no thumbnail).
if result.status in ("imported", "superseded"):
from .ml import tag_and_embed
from .thumbnail import generate_thumbnail
ids = list(result.member_image_ids)
if result.image_id is not None and result.image_id not in ids:
ids.append(result.image_id)
for img_id in ids:
generate_thumbnail.delay(img_id)
tag_and_embed.delay(img_id)
# If this was the last task in the batch, mark the batch complete.
remaining = session.execute(
select(ImportTask.id)
.where(ImportTask.batch_id == task.batch_id)
.where(ImportTask.status.in_(["pending", "queued", "processing"]))
.limit(1)
).scalar_one_or_none()
if remaining is None:
session.execute(
update(ImportBatch)
.where(ImportBatch.id == task.batch_id)
.values({counter_col_name: counter_col + 1})
.values(
status="complete",
finished_at=datetime.now(UTC),
)
)
session.add(task)
session.commit()
# Enqueue thumbnail + ML for newly imported AND superseded images
# (a superseded row has cleared ML + no thumbnail).
if result.status in ("imported", "superseded"):
from .ml import tag_and_embed
from .thumbnail import generate_thumbnail
ids = list(result.member_image_ids)
if result.image_id is not None and result.image_id not in ids:
ids.append(result.image_id)
for img_id in ids:
generate_thumbnail.delay(img_id)
tag_and_embed.delay(img_id)
# If this was the last task in the batch, mark the batch complete.
remaining = session.execute(
select(ImportTask.id)
.where(ImportTask.batch_id == task.batch_id)
.where(ImportTask.status.in_(["pending", "queued", "processing"]))
.limit(1)
).scalar_one_or_none()
if remaining is None:
session.execute(
update(ImportBatch)
.where(ImportBatch.id == task.batch_id)
.values(
status="complete",
finished_at=datetime.now(UTC),
)
)
session.commit()
return {"task_id": import_task_id, "status": task.status}
return {"task_id": import_task_id, "status": task.status}
+11 -14
View File
@@ -6,36 +6,33 @@ from datetime import UTC, datetime, timedelta
from pathlib import Path
from PIL import Image
from sqlalchemy import create_engine, delete, select, update
from sqlalchemy.orm import sessionmaker
from sqlalchemy import delete, select, update
from ..celery_app import celery
from ..config import get_config
from ..models import DownloadEvent, ImageRecord, ImportSettings, ImportTask
from ..utils.phash import compute_phash
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
STUCK_THRESHOLD_MINUTES = 30
STUCK_THRESHOLD_MINUTES = 5
OLD_TASK_DAYS = 7
PHASH_PAGE = 500
VERIFY_PAGE = 200
FFPROBE_TIMEOUT_SECONDS = 10
def _sync_session_factory():
cfg = get_config()
engine = create_engine(cfg.database_url_sync, future=True, pool_pre_ping=True)
return sessionmaker(engine, expire_on_commit=False)
@celery.task(name="backend.app.tasks.maintenance.recover_interrupted_tasks")
def recover_interrupted_tasks() -> int:
"""Find ImportTask rows stuck in 'processing' for >30 min and re-queue them.
"""Find ImportTask rows stuck in 'processing' for >5 min and re-queue them.
Why 30 min: large videos can legitimately take many minutes to import;
30 is a safe gate that catches actual crashes (which leave the row stuck
forever) without resetting slow-but-still-running jobs.
Why 5 min: import_media_file is sub-second for the vast majority of
files; even a large-video transcode caps at the per-task soft_time_limit
(5 min) defined on the task itself. Anything still 'processing' after
that window is a confirmed crash (worker died, DB disconnect mid-flush,
OOM) and must be recycled. Was 30 min historically; tightened
2026-05-24 after operator hit a 2224-row zombie pile during the IR
migration scan.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
+22 -1
View File
@@ -4,7 +4,7 @@ Dispatches to the right migrator based on `kind`. Updates MigrationRun
row's status/counts/finished_at as it runs. Failures set status='error'
with the error message preserved.
kinds: backup, gs_ingest, ir_ingest, tag_apply, ml_queue, verify, rollback
kinds: backup, gs_ingest, ir_ingest, tag_apply, ml_queue, verify, rollback, cleanup
"""
from __future__ import annotations
@@ -21,6 +21,7 @@ from ..config import get_config
from ..models import MigrationRun
from ..services.credential_crypto import CredentialCrypto
from ..services.migrators import backup as backup_mod
from ..services.migrators import cleanup as cleanup_mod
from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify
from ..services.migrators import rollback as rollback_mod
@@ -145,6 +146,26 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
)
return {"checks": checks, "sample": sample}
elif kind == "cleanup":
slug = params.get("slug")
if not slug:
raise ValueError("cleanup requires params.slug")
result = await cleanup_mod.cleanup_artist_async(
db, slug=slug, images_root=IMAGES_ROOT,
dry_run=params.get("dry_run", False),
source_path_prefix=params.get("source_path_prefix"),
)
await _update_run(
db, run_id, status="ok",
counts=result["counts"],
finished_at=datetime.now(UTC),
metadata_patch={
"artist": result["artist"],
"summary": result["summary"],
},
)
return result
elif kind == "rollback":
result = rollback_mod.rollback_to_pre_migration(
db_url=get_config().database_url_sync,
+14 -10
View File
@@ -8,28 +8,32 @@ apply_allowlist_tags sweeps which are 'maintenance' lane. Sync sessions
from pathlib import Path
from sqlalchemy import create_engine, select
from sqlalchemy.orm import sessionmaker
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import ImageRecord, MLSettings
from ._sync_engine import sync_session_factory as _sync_session_factory
IMAGES_ROOT = Path("/images")
VIDEO_EXTS = {".mp4", ".mov", ".avi", ".mkv", ".webm", ".m4v", ".wmv", ".flv"}
def _sync_session_factory():
cfg = get_config()
engine = create_engine(cfg.database_url_sync, future=True, pool_pre_ping=True)
return sessionmaker(engine, expire_on_commit=False)
def _is_video(path: Path) -> bool:
return path.suffix.lower() in VIDEO_EXTS
@celery.task(name="backend.app.tasks.ml.tag_and_embed", bind=True)
@celery.task(
name="backend.app.tasks.ml.tag_and_embed",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=420,
)
def tag_and_embed(self, image_id: int) -> dict:
"""Run Camie + SigLIP on one image; store predictions + embedding;
then enqueue per-image allowlist application.
+30 -9
View File
@@ -11,14 +11,14 @@ import asyncio
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import create_engine, select
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.orm import sessionmaker
from ..celery_app import celery
from ..config import get_config
from ..models import DownloadEvent, ImportBatch, ImportSettings, ImportTask
from ..services.scheduler_service import select_due_sources
from ._sync_engine import sync_session_factory as _sync_session_factory
def _iter_import_files(import_root: Path):
@@ -35,12 +35,6 @@ def _iter_import_files(import_root: Path):
yield entry
def _sync_session_factory():
cfg = get_config()
engine = create_engine(cfg.database_url_sync, future=True, pool_pre_ping=True)
return sessionmaker(engine, expire_on_commit=False)
@celery.task(name="backend.app.tasks.scan.scan_directory", bind=True)
def scan_directory(self, triggered_by: str = "manual",
mode: str = "quick") -> int:
@@ -65,16 +59,34 @@ def scan_directory(self, triggered_by: str = "manual",
session.flush()
batch_id = batch.id
# Skip-set: any source_path that already has a non-failed ImportTask
# row. Re-running scan_directory must not re-enqueue files the
# importer has already handled (or is currently handling); doing so
# creates duplicate work and inflates the queue. Failed prior tasks
# are eligible for retry.
non_failed_existing = set(session.execute(
select(ImportTask.source_path).where(
ImportTask.status.in_(
["pending", "queued", "processing", "complete", "skipped"]
),
)
).scalars().all())
# Walk and enumerate.
files_seen = 0
files_skipped_existing = 0
for entry in _iter_import_files(import_root):
entry_str = str(entry)
if entry_str in non_failed_existing:
files_skipped_existing += 1
continue
try:
size = entry.stat().st_size
except OSError:
size = None
task = ImportTask(
batch_id=batch_id,
source_path=str(entry),
source_path=entry_str,
task_type="media",
status="pending",
size_bytes=size,
@@ -83,6 +95,15 @@ def scan_directory(self, triggered_by: str = "manual",
files_seen += 1
batch.total_files = files_seen
# If the walk enqueued nothing (every file was already on a
# non-failed ImportTask from a prior scan), there's no
# import_media_file message that would ever flip this batch to
# 'complete' — finalize it now so the active-batch query in
# /api/system/stats doesn't get stuck reporting all-zero
# counters from an empty scan.
if files_seen == 0:
batch.status = "complete"
batch.finished_at = datetime.now(UTC)
session.commit()
# Now enqueue import_media_file for each pending task.
+13 -10
View File
@@ -7,25 +7,28 @@ so they deserve their own queue lane.
from pathlib import Path
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import ImageRecord
from ..services.importer import is_video
from ..services.thumbnailer import Thumbnailer
from ._sync_engine import sync_session_factory as _sync_session_factory
IMAGES_ROOT = Path("/images")
def _sync_session_factory():
cfg = get_config()
engine = create_engine(cfg.database_url_sync, future=True, pool_pre_ping=True)
return sessionmaker(engine, expire_on_commit=False)
@celery.task(name="backend.app.tasks.thumbnail.generate_thumbnail", bind=True)
@celery.task(
name="backend.app.tasks.thumbnail.generate_thumbnail",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=120,
time_limit=180,
)
def generate_thumbnail(self, image_id: int) -> dict:
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
+4
View File
@@ -0,0 +1,4 @@
node_modules/
web-ext-artifacts/
.web-ext-id
.amo-upload-uuid
+45
View File
@@ -0,0 +1,45 @@
# FabledCurator Firefox Extension
Self-hosted Firefox extension that pushes session cookies from supported
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv,
DeviantArt) into FabledCurator, and lets you add a creator as a Source
from their page in one click.
## Install (operator)
The signed XPI is bundled into the FC Docker image. Open FC →
Settings → Maintenance → Browser extension → click "Install Firefox
extension". Firefox shows its native install prompt. After installing,
open the extension's options page (about:addons → FabledCurator →
Preferences) and paste in the FC URL + extension API key shown on the
same card.
## Develop
```sh
cd extension/
npm install --no-save # web-ext only
npm run lint # web-ext lint
npm run start # launches Firefox with extension loaded
npm run build # unsigned XPI in web-ext-artifacts/
```
## Smoke checklist (after every release that touches `extension/**`)
- [ ] `npm run lint` passes
- [ ] `npm run start` loads the extension in a clean Firefox profile
- [ ] Options page accepts FC URL + key, indicator turns green
- [ ] Cookie export: log into patreon.com, click Patreon card → "X cookies exported"
- [ ] Discord token: open discord.com, click Discord card → "Token captured"
- [ ] Pixiv OAuth: click Pixiv card → login redirects, token stored
- [ ] Add as source: visit patreon.com/<creator>, click floating button → toast
- [ ] Subscriptions list: popup → "Sources" tab → list renders
- [ ] Check now: click play icon on source row → no error toast
## Release
Bump `manifest.json` + `package.json` SemVer (both files) and commit
under `extension/**`. The `.forgejo/workflows/extension.yml` workflow
runs `web-ext sign` on main, commits the signed XPI to
`frontend/public/extension/`, and the next FC server build bundles it
into the Docker image.
+265
View File
@@ -0,0 +1,265 @@
/**
* Background script — message router + Discord token capture
* (webRequest) + Pixiv PKCE OAuth. Direct port of GS background.js;
* api.js client points at FC instead of GS.
*/
let discordToken = null;
let discordTokenCapturedAt = null;
let pixivRefreshToken = null;
let pixivTokenCapturedAt = null;
let pixivOAuthPending = null;
const PIXIV_CLIENT_ID = 'MOBrBDS8blbauoSck0ZfDbtuzpyT';
const PIXIV_CLIENT_SECRET = 'lsACyCD94FhDUtGTXi3QzcFE2uU1hqtDaKeqrdwj';
const PIXIV_OAUTH_URL = 'https://app-api.pixiv.net/web/v1/login';
const PIXIV_TOKEN_URL = 'https://oauth.secure.pixiv.net/auth/token';
const PIXIV_REDIRECT_URI = 'https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback';
let initialized = false;
async function ensureInitialized() {
if (initialized) return;
await api.init();
await loadDiscordToken();
await loadPixivToken();
initialized = true;
}
browser.runtime.onInstalled.addListener(() => ensureInitialized());
browser.runtime.onStartup.addListener(() => ensureInitialized());
ensureInitialized().catch(e => console.error('init failed:', e));
// ---- Discord token capture via webRequest ----
browser.webRequest.onBeforeSendHeaders.addListener(
(details) => {
const auth = details.requestHeaders?.find(h => h.name.toLowerCase() === 'authorization');
if (auth?.value && auth.value !== discordToken) {
saveDiscordToken(auth.value);
}
},
{ urls: ['https://discord.com/api/*'] },
['requestHeaders'],
);
async function loadDiscordToken() {
const s = await browser.storage.local.get(['discordToken', 'discordTokenCapturedAt']);
discordToken = s.discordToken || null;
discordTokenCapturedAt = s.discordTokenCapturedAt || null;
}
async function saveDiscordToken(token) {
discordToken = token;
discordTokenCapturedAt = new Date().toISOString();
await browser.storage.local.set({ discordToken: token, discordTokenCapturedAt });
}
// ---- Pixiv PKCE OAuth ----
async function loadPixivToken() {
const s = await browser.storage.local.get(['pixivRefreshToken', 'pixivTokenCapturedAt']);
pixivRefreshToken = s.pixivRefreshToken || null;
pixivTokenCapturedAt = s.pixivTokenCapturedAt || null;
}
async function savePixivToken(token) {
pixivRefreshToken = token;
pixivTokenCapturedAt = new Date().toISOString();
await browser.storage.local.set({ pixivRefreshToken: token, pixivTokenCapturedAt });
}
function generateCodeVerifier() {
const a = new Uint8Array(32);
crypto.getRandomValues(a);
return base64UrlEncode(a);
}
async function generateCodeChallenge(verifier) {
const data = new TextEncoder().encode(verifier);
const hash = await crypto.subtle.digest('SHA-256', data);
return base64UrlEncode(new Uint8Array(hash));
}
function base64UrlEncode(buf) {
return btoa(String.fromCharCode(...buf)).replace(/\+/g, '-').replace(/\//g, '_').replace(/=/g, '');
}
async function initiatePixivOAuth() {
const codeVerifier = generateCodeVerifier();
const codeChallenge = await generateCodeChallenge(codeVerifier);
const params = new URLSearchParams({
code_challenge: codeChallenge,
code_challenge_method: 'S256',
client: 'pixiv-android',
});
const tab = await browser.tabs.create({ url: `${PIXIV_OAUTH_URL}?${params}` });
return new Promise((resolve, reject) => {
pixivOAuthPending = { codeVerifier, tabId: tab.id, resolve, reject };
setTimeout(() => {
if (pixivOAuthPending && pixivOAuthPending.tabId === tab.id) {
pixivOAuthPending = null;
reject(new Error('Pixiv OAuth timed out (5 min)'));
}
}, 5 * 60 * 1000);
});
}
browser.webRequest.onBeforeRedirect.addListener(
async (details) => {
if (!pixivOAuthPending) return;
if (details.tabId !== pixivOAuthPending.tabId) return;
const url = new URL(details.redirectUrl);
const code = url.searchParams.get('code');
if (!code) return;
const verifier = pixivOAuthPending.codeVerifier;
const resolve = pixivOAuthPending.resolve;
const reject = pixivOAuthPending.reject;
pixivOAuthPending = null;
try {
const tokenResp = await fetch(PIXIV_TOKEN_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
client_id: PIXIV_CLIENT_ID,
client_secret: PIXIV_CLIENT_SECRET,
code,
code_verifier: verifier,
grant_type: 'authorization_code',
include_policy: 'true',
redirect_uri: PIXIV_REDIRECT_URI,
}),
});
const body = await tokenResp.json();
if (!body.refresh_token) {
reject(new Error(`Pixiv token exchange failed: ${JSON.stringify(body)}`));
return;
}
await savePixivToken(body.refresh_token);
try { await browser.tabs.remove(details.tabId); } catch {}
resolve(body.refresh_token);
} catch (e) {
reject(e);
}
},
{ urls: ['https://app-api.pixiv.net/web/v1/users/auth/pixiv/callback*'] },
);
// ---- Message router ----
browser.runtime.onMessage.addListener(async (msg) => {
await ensureInitialized();
switch (msg.type) {
case 'GET_CONFIG':
return { apiUrl: api.baseUrl, apiKey: api.apiKey };
case 'TEST_CONNECTION':
try {
await api.testConnection();
return { connected: true };
} catch (e) {
return { connected: false, error: e.message };
}
case 'GET_PLATFORM_STATUS': {
const status = {};
for (const key of Object.keys(PLATFORMS)) {
if (PLATFORMS[key].authType === 'cookies') {
status[key] = { hasCookies: false, cookieCount: 0 };
try {
const n = await getCookieCount(key);
status[key] = { hasCookies: n > 0, cookieCount: n };
} catch (e) {
status[key] = { error: e.message };
}
} else if (key === 'discord') {
status[key] = { hasToken: !!discordToken, capturedAt: discordTokenCapturedAt };
} else if (key === 'pixiv') {
status[key] = { hasToken: !!pixivRefreshToken, capturedAt: pixivTokenCapturedAt };
} else {
status[key] = {};
}
}
return status;
}
case 'EXPORT_COOKIES': {
const key = msg.platform;
const platform = PLATFORMS[key];
if (!platform) return { error: `Unknown platform: ${key}` };
try {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
await api.uploadCredentials('discord', 'token', discordToken);
return { success: true };
}
if (key === 'pixiv') {
if (!pixivRefreshToken) {
await initiatePixivOAuth();
}
await api.uploadCredentials('pixiv', 'token', pixivRefreshToken);
return { success: true };
}
return { error: 'Unsupported platform.' };
} catch (e) {
return { error: e.message };
}
}
case 'EXPORT_ALL_COOKIES': {
const results = {};
for (const key of Object.keys(PLATFORMS)) {
if (PLATFORMS[key].authType !== 'cookies') {
results[key] = { skipped: true };
continue;
}
try {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length };
} catch (e) {
results[key] = { error: e.message };
}
}
return results;
}
case 'LIST_SOURCES':
try {
return { sources: await api.listSources() };
} catch (e) {
return { error: e.message };
}
case 'CHECK_SOURCE':
try {
return await api.triggerSourceCheck(msg.sourceId);
} catch (e) {
return { error: e.message };
}
case 'ADD_AS_SOURCE':
try {
return await api.quickAddSource(msg.url);
} catch (e) {
return { error: e.message };
}
default:
return { error: `Unknown message type: ${msg.type}` };
}
});
+27
View File
@@ -0,0 +1,27 @@
.fc-add-source-btn {
all: revert;
position: fixed; bottom: 24px; right: 24px; z-index: 2147483647;
padding: 10px 16px; border-radius: 999px; border: none;
background: rgb(20, 23, 26); color: rgb(244, 186, 122);
font: 500 14px/1.2 system-ui, sans-serif;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.4); cursor: pointer;
transition: transform 100ms ease;
}
.fc-add-source-btn:hover { transform: translateY(-1px); }
.fc-add-source-btn:disabled { opacity: 0.6; cursor: wait; }
.fc-toast {
all: revert;
position: fixed; bottom: 84px; right: 24px; z-index: 2147483647;
max-width: 360px; padding: 12px 16px; border-radius: 8px;
background: rgb(20, 23, 26); color: rgb(232, 228, 216);
font: 14px/1.4 system-ui, sans-serif;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.4);
animation: fc-toast-in 200ms ease;
}
.fc-toast--success { border-left: 3px solid rgb(244, 186, 122); }
.fc-toast--error { border-left: 3px solid rgb(220, 80, 80); }
@keyframes fc-toast-in {
from { transform: translateY(20px); opacity: 0; }
to { transform: translateY(0); opacity: 1; }
}
+60
View File
@@ -0,0 +1,60 @@
(function () {
if (window.__fc_addsource_injected) return;
window.__fc_addsource_injected = true;
evaluate();
const reEval = () => evaluate();
window.addEventListener('popstate', reEval);
const origPush = history.pushState;
history.pushState = function () { origPush.apply(this, arguments); reEval(); };
function evaluate() {
const platform = getPlatformFromUrl(window.location.href);
const onArtist = platform && isArtistPage(window.location.href, platform);
let btn = document.getElementById('fc-add-source-btn');
if (onArtist && !btn) injectButton();
else if (!onArtist && btn) btn.remove();
}
function injectButton() {
const btn = document.createElement('button');
btn.id = 'fc-add-source-btn';
btn.className = 'fc-add-source-btn';
btn.textContent = '+ Add to FabledCurator';
btn.addEventListener('click', onClick);
document.body.appendChild(btn);
}
async function onClick() {
const btn = document.getElementById('fc-add-source-btn');
btn.disabled = true;
const original = btn.textContent;
btn.textContent = 'Adding…';
try {
const r = await browser.runtime.sendMessage({
type: 'ADD_AS_SOURCE',
url: window.location.href,
});
if (r.error) {
showToast(`Error: ${r.error}`, 'error');
} else {
const verb = r.created_source ? 'Added' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success');
}
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
} finally {
btn.disabled = false;
btn.textContent = original;
}
}
function showToast(text, kind) {
const t = document.createElement('div');
t.className = `fc-toast fc-toast--${kind}`;
t.textContent = text;
document.body.appendChild(t);
setTimeout(() => t.remove(), 4000);
}
})();
+5
View File
@@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96" width="96" height="96">
<rect width="96" height="96" rx="16" fill="#14171A"/>
<text x="48" y="62" font-family="Georgia, serif" font-size="56" font-weight="500"
fill="#F4BA7A" text-anchor="middle">F</text>
</svg>

After

Width:  |  Height:  |  Size: 284 B

+93
View File
@@ -0,0 +1,93 @@
/**
* FC backend client. Talks to /api/credentials (FC-3b), /api/sources
* (FC-3a), and the new /api/extension/* endpoints (FC-3g).
*/
class FabledCuratorAPI {
constructor() {
this.baseUrl = null;
this.apiKey = null;
}
async init() {
const cfg = await browser.storage.local.get(['apiUrl', 'apiKey']);
this.baseUrl = cfg.apiUrl || null;
this.apiKey = cfg.apiKey || null;
return this.isConfigured();
}
isConfigured() {
return !!(this.baseUrl && this.apiKey);
}
async request(method, endpoint, data = null) {
if (!this.isConfigured()) {
throw new Error('Not configured. Set FC URL + API key in settings.');
}
const url = `${this.baseUrl.replace(/\/+$/, '')}${endpoint}`;
const options = {
method,
headers: {
'Content-Type': 'application/json',
'X-Extension-Key': this.apiKey,
},
};
if (data) options.body = JSON.stringify(data);
let response;
try {
response = await fetch(url, options);
} catch (e) {
throw new Error('Cannot connect to FabledCurator. Check URL.');
}
if (!response.ok) {
let message;
try {
const body = await response.json();
message = body.error || `HTTP ${response.status}`;
if (body.detail) message += `: ${body.detail}`;
} catch {
message = `HTTP ${response.status}: ${response.statusText}`;
}
const err = new Error(message);
err.status = response.status;
throw err;
}
if (response.status === 204) return { success: true };
return response.json();
}
// FC-3b — credentials.
uploadCredentials(platform, credentialType, data) {
return this.request('POST', '/credentials', {
platform,
credential_type: credentialType,
data,
});
}
getCredentials() {
return this.request('GET', '/credentials');
}
// FC-3a — sources.
listSources() {
return this.request('GET', '/sources');
}
triggerSourceCheck(sourceId) {
return this.request('POST', `/sources/${sourceId}/check`);
}
// FC-3g — extension-specific.
quickAddSource(url) {
return this.request('POST', '/extension/quick-add-source', { url });
}
// Connection test = the cheapest read with auth.
testConnection() {
return this.request('GET', '/credentials');
}
}
const api = new FabledCuratorAPI();
+56
View File
@@ -0,0 +1,56 @@
/**
* Cookie extraction + Netscape format conversion.
* Direct port of GS extension/lib/cookies.js.
*/
async function extractCookiesForPlatform(platformKey) {
const platform = PLATFORMS[platformKey];
if (!platform) throw new Error(`Unknown platform: ${platformKey}`);
const all = [];
for (const domain of platform.domains) {
try {
const cookies = await browser.cookies.getAll({ domain });
all.push(...cookies);
} catch (e) {
console.warn(`cookies.getAll failed for ${domain}:`, e);
}
}
return deduplicateCookies(all);
}
function deduplicateCookies(cookies) {
const seen = new Map();
for (const c of cookies) {
const key = `${c.name}|${c.domain}|${c.path}`;
if (!seen.has(key)) {
seen.set(key, c);
} else {
const existing = seen.get(key);
if (c.expirationDate && (!existing.expirationDate || c.expirationDate > existing.expirationDate)) {
seen.set(key, c);
}
}
}
return Array.from(seen.values());
}
function toNetscapeFormat(cookies) {
const lines = ['# Netscape HTTP Cookie File'];
for (const c of cookies) {
let domain = c.domain.replace(/^\.?www\./, '.');
if (!domain.startsWith('.')) domain = '.' + domain;
const secure = c.secure ? 'TRUE' : 'FALSE';
const expiration = c.expirationDate ? Math.floor(c.expirationDate) : 0;
lines.push([domain, 'TRUE', c.path || '/', secure, String(expiration), c.name, c.value].join('\t'));
}
return lines.join('\n');
}
async function getCookieCount(platformKey) {
try {
return (await extractCookiesForPlatform(platformKey)).length;
} catch {
return 0;
}
}
+85
View File
@@ -0,0 +1,85 @@
/**
* Platform definitions — mirrored from backend/app/services/platforms.py.
* Two copies of these regexes (here and in backend extension_service.py)
* is cheaper than building a shared definition file for two short
* tables in different runtimes. Keep in sync by hand; reviewers catch
* drift.
*/
const PLATFORMS = {
patreon: {
name: 'Patreon',
domains: ['.patreon.com', 'www.patreon.com', 'patreon.com'],
authType: 'cookies',
color: '#FF424D',
urlPattern: /^https?:\/\/(www\.)?patreon\.com/,
},
subscribestar: {
name: 'SubscribeStar',
domains: [
'.subscribestar.com',
'.subscribestar.adult',
'www.subscribestar.com',
'subscribestar.com',
'subscribestar.adult',
],
authType: 'cookies',
color: '#FFD700',
urlPattern: /^https?:\/\/(www\.)?subscribestar\.(com|adult)/,
},
hentaifoundry: {
name: 'Hentai Foundry',
domains: ['.hentai-foundry.com', 'www.hentai-foundry.com', 'hentai-foundry.com'],
authType: 'cookies',
color: '#9C27B0',
urlPattern: /^https?:\/\/(www\.)?hentai-foundry\.com/,
},
discord: {
name: 'Discord',
domains: ['.discord.com', 'discord.com'],
authType: 'token',
color: '#5865F2',
urlPattern: /^https?:\/\/(www\.)?discord\.com/,
note: 'Open Discord in browser to capture token',
},
pixiv: {
name: 'Pixiv',
domains: ['.pixiv.net', 'www.pixiv.net', 'pixiv.net'],
authType: 'token',
color: '#0096FA',
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
note: 'Click to authenticate via OAuth',
},
deviantart: {
name: 'DeviantArt',
domains: ['.deviantart.com', 'www.deviantart.com', 'deviantart.com'],
authType: 'cookies',
color: '#05CC47',
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
},
};
/**
* Per-platform artist-page URL patterns — these are STRICTER than
* urlPattern (which matches any URL on the domain). Used by the content
* script to decide whether to show the floating "Add as source" button.
*/
const PLATFORM_ARTIST_PATTERNS = {
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?!home$|search\b|messages\b|notifications\b|library\b|settings\b|posts\b|c\/)[^/?#]+\/?$/i,
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
deviantart: /^https?:\/\/(www\.)?deviantart\.com\/(?!home$|watch\b|tag\b|browse\b)[^/?#]+\/?$/i,
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
};
function getPlatformFromUrl(url) {
for (const [key, platform] of Object.entries(PLATFORMS)) {
if (platform.urlPattern.test(url)) return key;
}
return null;
}
function isArtistPage(url, platformKey) {
const pattern = PLATFORM_ARTIST_PATTERNS[platformKey];
return pattern ? pattern.test(url) : false;
}
+71
View File
@@ -0,0 +1,71 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.0",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
"gecko": {
"id": "fabledcurator@fabledsword.com",
"strict_min_version": "115.0"
}
},
"permissions": [
"cookies",
"storage",
"tabs",
"activeTab",
"webRequest",
"webRequestBlocking"
],
"host_permissions": [
"*://*.patreon.com/*",
"*://*.subscribestar.com/*",
"*://*.subscribestar.adult/*",
"*://*.hentai-foundry.com/*",
"*://*.discord.com/*",
"*://*.pixiv.net/*",
"*://*.deviantart.com/*",
"*://app-api.pixiv.net/*",
"*://oauth.secure.pixiv.net/*",
"*://*/*"
],
"action": {
"default_popup": "popup/popup.html",
"default_icon": "icons/icon.svg",
"default_title": "FabledCurator"
},
"background": {
"scripts": ["lib/platforms.js", "lib/cookies.js", "lib/api.js", "background/background.js"]
},
"options_ui": {
"page": "options/options.html",
"browser_style": true
},
"content_scripts": [
{
"matches": [
"*://*.patreon.com/*",
"*://*.subscribestar.com/*",
"*://*.subscribestar.adult/*",
"*://*.hentai-foundry.com/*",
"*://*.deviantart.com/*",
"*://*.pixiv.net/*"
],
"js": ["lib/platforms.js", "content/content-script.js"],
"css": ["content/content-script.css"],
"run_at": "document_idle"
}
],
"icons": {
"48": "icons/icon.svg",
"96": "icons/icon.svg"
}
}
+41
View File
@@ -0,0 +1,41 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>FabledCurator — Settings</title>
<style>
body { font: 14px/1.4 system-ui, sans-serif; max-width: 520px; margin: 32px auto; padding: 0 16px; }
h1 { font-family: Georgia, serif; font-size: 20px; }
label { display: block; font-weight: 500; margin: 16px 0 6px; }
input { width: 100%; padding: 8px; box-sizing: border-box; font: inherit; }
.hint { color: #666; font-size: 12px; margin-top: 4px; }
.row { display: flex; gap: 8px; margin-top: 16px; }
button { padding: 8px 16px; cursor: pointer; font: inherit; }
button.primary { background: #F4BA7A; color: #14171A; border: none; border-radius: 4px; }
button.test { background: none; border: 1px solid #ccc; border-radius: 4px; }
.status { margin-top: 16px; padding: 8px 12px; border-radius: 4px; }
.status.ok { background: #DFF5DF; color: #2B6A2B; }
.status.err { background: #FBE1E1; color: #862525; }
</style>
</head>
<body>
<h1>FabledCurator extension</h1>
<label for="api-url">FC base URL</label>
<input id="api-url" type="url" placeholder="http://curator.example.com/api" />
<div class="hint">Find this on FC → Settings → Maintenance → Browser extension.</div>
<label for="api-key">Extension API key</label>
<input id="api-key" type="password" placeholder="paste from FC Settings card" />
<div class="hint">Generate or rotate on FC → Settings → Maintenance → Browser extension.</div>
<div class="row">
<button class="primary" id="save-btn">Save</button>
<button class="test" id="test-btn">Test connection</button>
</div>
<div id="status" class="status" style="display:none;"></div>
<script src="options.js"></script>
</body>
</html>
+46
View File
@@ -0,0 +1,46 @@
document.addEventListener('DOMContentLoaded', async () => {
const stored = await browser.storage.local.get(['apiUrl', 'apiKey']);
document.getElementById('api-url').value = stored.apiUrl || '';
document.getElementById('api-key').value = stored.apiKey || '';
document.getElementById('save-btn').addEventListener('click', save);
document.getElementById('test-btn').addEventListener('click', test);
});
async function save() {
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
const apiKey = document.getElementById('api-key').value.trim();
if (!apiUrl || !apiKey) {
showStatus('Both fields are required.', 'err');
return;
}
await browser.storage.local.set({ apiUrl, apiKey });
await browser.storage.local.remove(['lastConnectionTest', 'lastConnectionStatus']);
showStatus('Saved.', 'ok');
}
async function test() {
const apiUrl = document.getElementById('api-url').value.trim().replace(/\/+$/, '');
const apiKey = document.getElementById('api-key').value.trim();
if (!apiUrl || !apiKey) {
showStatus('Fill both fields first.', 'err');
return;
}
try {
const r = await fetch(`${apiUrl}/credentials`, {
method: 'GET',
headers: { 'X-Extension-Key': apiKey },
});
if (r.ok) showStatus(`Connected — HTTP ${r.status}.`, 'ok');
else showStatus(`HTTP ${r.status}: ${r.statusText}`, 'err');
} catch (e) {
showStatus(`Cannot reach ${apiUrl}: ${e.message}`, 'err');
}
}
function showStatus(text, kind) {
const el = document.getElementById('status');
el.textContent = text;
el.className = `status ${kind}`;
el.style.display = 'block';
}
+15
View File
@@ -0,0 +1,15 @@
{
"name": "fabledcurator-extension",
"version": "1.0.0",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
"lint": "web-ext lint --source-dir=.",
"start": "web-ext run --source-dir=. --firefox=firefox",
"build": "web-ext build --source-dir=. --overwrite-dest",
"sign": "web-ext sign --source-dir=. --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
},
"devDependencies": {
"web-ext": "^8.0.0"
}
}
+94
View File
@@ -0,0 +1,94 @@
:root {
--bg: #14171A;
--surface: #1F2428;
--on-surface: #E8E4D8;
--on-surface-variant: rgba(232, 228, 216, 0.65);
--accent: #F4BA7A;
--error: #DC5050;
--success: #4CAF50;
--warning: #FFB300;
}
* { box-sizing: border-box; }
body {
margin: 0;
min-width: 340px;
font: 14px/1.4 system-ui, sans-serif;
background: var(--bg);
color: var(--on-surface);
}
.hidden { display: none !important; }
.topbar {
display: flex; align-items: center; justify-content: space-between;
padding: 10px 14px; background: var(--surface);
}
.brand { font-family: Georgia, serif; font-size: 16px; font-weight: 500; color: var(--accent); }
.dot { width: 10px; height: 10px; border-radius: 50%; background: var(--on-surface-variant); }
.dot.connected { background: var(--success); }
.setup { padding: 20px; text-align: center; }
.alert {
background: var(--surface); padding: 12px; border-radius: 6px;
margin-bottom: 12px; border-left: 3px solid var(--accent);
}
.alert.alert-error { border-left-color: var(--error); }
.alert strong { display: block; margin-bottom: 4px; }
.alert p { margin: 0; color: var(--on-surface-variant); font-size: 13px; }
.tabs { display: flex; border-bottom: 1px solid var(--surface); }
.tab {
flex: 1; padding: 10px 0; background: none; border: none;
color: var(--on-surface-variant); cursor: pointer; font: inherit;
border-bottom: 2px solid transparent;
}
.tab.active { color: var(--accent); border-bottom-color: var(--accent); }
.tab-panel { padding: 10px; }
.platform-card, .source-row {
display: flex; align-items: center; gap: 10px;
padding: 10px; margin-bottom: 6px; background: var(--surface);
border-radius: 6px; cursor: pointer;
}
.platform-card.disabled { opacity: 0.5; cursor: not-allowed; }
.platform-card.loading { opacity: 0.6; cursor: wait; }
.platform-icon {
width: 28px; height: 28px; border-radius: 50%;
display: grid; place-items: center; font-weight: 600; color: white;
}
.info { flex: 1; min-width: 0; }
.info .name { font-weight: 500; }
.info .status { font-size: 12px; color: var(--on-surface-variant); }
.info .status.ready { color: var(--success); }
.info .status.error { color: var(--error); }
.info .status.no-cookies { color: var(--warning); }
.action-icon { color: var(--on-surface-variant); }
.btn {
padding: 10px 14px; border: none; border-radius: 6px; cursor: pointer;
font: 500 14px/1 system-ui, sans-serif;
}
.btn.primary { background: var(--accent); color: var(--bg); }
.btn.primary:hover { filter: brightness(1.05); }
.btn.primary:disabled { opacity: 0.5; cursor: not-allowed; }
.btn.block { display: block; width: 100%; margin-top: 8px; }
.btn.link { background: none; color: var(--on-surface-variant); padding: 4px; }
.btn.link:hover { color: var(--accent); }
.source-row .play {
background: none; border: none; color: var(--on-surface-variant);
cursor: pointer; padding: 4px;
}
.source-row .play:hover { color: var(--accent); }
.source-row .url { font-size: 11px; color: var(--on-surface-variant); word-break: break-all; }
.status-message {
padding: 10px 14px; margin: 8px 10px; border-radius: 6px;
background: var(--surface); border-left: 3px solid;
}
.status-message.success { border-left-color: var(--success); }
.status-message.error { border-left-color: var(--error); }
.status-message.warning { border-left-color: var(--warning); }
.footer {
border-top: 1px solid var(--surface);
padding: 6px 10px; text-align: right;
}
+47
View File
@@ -0,0 +1,47 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="stylesheet" href="popup.css" />
<title>FabledCurator</title>
</head>
<body>
<header class="topbar">
<span class="brand">FabledCurator</span>
<span id="connection-status" class="dot" title="checking…"></span>
</header>
<section id="setup-required" class="setup hidden">
<div class="alert">
<strong>Setup required</strong>
<p>Configure the FC URL and extension API key.</p>
</div>
<button id="open-settings-btn" class="btn primary">Open settings</button>
</section>
<section id="main-content" class="main hidden">
<nav class="tabs">
<button class="tab active" data-tab="platforms">Platforms</button>
<button class="tab" data-tab="sources">Sources</button>
</nav>
<div id="tab-platforms" class="tab-panel">
<div id="platforms-list"></div>
<button id="export-all-btn" class="btn primary block">Export all platforms</button>
</div>
<div id="tab-sources" class="tab-panel hidden">
<div id="sources-list"></div>
</div>
<div id="status-message" class="status-message hidden"></div>
<footer class="footer">
<button id="settings-btn" class="btn link">Settings</button>
</footer>
</section>
<script src="../lib/platforms.js"></script>
<script src="popup.js"></script>
</body>
</html>
+239
View File
@@ -0,0 +1,239 @@
document.addEventListener('DOMContentLoaded', init);
const CONNECTION_TEST_INTERVAL = 2 * 60 * 1000;
async function init() {
try {
const cfg = await browser.runtime.sendMessage({ type: 'GET_CONFIG' });
if (!cfg || !cfg.apiUrl || !cfg.apiKey) {
showSetupRequired();
return;
}
document.getElementById('setup-required').classList.add('hidden');
document.getElementById('main-content').classList.remove('hidden');
setupEventListeners();
showPlatformsLoading();
testConnectionIfNeeded();
loadPlatformStatus().catch(e => showError(`Failed to load platforms: ${e.message}`));
} catch (e) {
showSetupRequired();
const alert = document.querySelector('#setup-required .alert');
alert.textContent = '';
const s = document.createElement('strong'); s.textContent = 'Error';
const p = document.createElement('p'); p.textContent = e.message;
alert.appendChild(s); alert.appendChild(p);
alert.classList.add('alert-error');
}
}
function showSetupRequired() {
document.getElementById('setup-required').classList.remove('hidden');
document.getElementById('main-content').classList.add('hidden');
document.getElementById('open-settings-btn').addEventListener('click', () => {
browser.runtime.openOptionsPage();
});
}
function showPlatformsLoading() {
const c = document.getElementById('platforms-list');
c.textContent = '';
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading platforms…';
c.appendChild(d);
}
async function testConnectionIfNeeded() {
const stored = await browser.storage.local.get(['lastConnectionTest', 'lastConnectionStatus']);
const now = Date.now();
if (now - (stored.lastConnectionTest || 0) < CONNECTION_TEST_INTERVAL && stored.lastConnectionStatus !== undefined) {
updateConnectionDot(stored.lastConnectionStatus);
if (!stored.lastConnectionStatus) showError('Cannot connect to backend (cached).');
return;
}
const r = await browser.runtime.sendMessage({ type: 'TEST_CONNECTION' });
await browser.storage.local.set({ lastConnectionTest: now, lastConnectionStatus: r.connected });
updateConnectionDot(r.connected);
if (!r.connected) showError(`Cannot connect to backend: ${r.error}`);
}
function updateConnectionDot(connected) {
const d = document.getElementById('connection-status');
d.classList.toggle('connected', connected);
d.title = connected ? 'Connected to FabledCurator' : 'Disconnected';
}
async function loadPlatformStatus() {
const status = await browser.runtime.sendMessage({ type: 'GET_PLATFORM_STATUS' });
const c = document.getElementById('platforms-list');
c.textContent = '';
for (const [key, platform] of Object.entries(PLATFORMS)) {
c.appendChild(createPlatformCard(key, platform, status[key] || {}));
}
}
function createPlatformCard(key, platform, status) {
const card = document.createElement('div');
card.className = 'platform-card';
card.dataset.platform = key;
const isTokenOnly = platform.authType === 'token' && !['discord', 'pixiv'].includes(key);
const discordNeedsToken = key === 'discord' && !status.hasToken;
if (isTokenOnly || discordNeedsToken) card.classList.add('disabled');
const icon = document.createElement('div');
icon.className = 'platform-icon';
icon.style.background = platform.color;
icon.textContent = platform.name[0];
const info = document.createElement('div');
info.className = 'info';
const name = document.createElement('div');
name.className = 'name';
name.textContent = platform.name;
const st = document.createElement('div');
st.className = `status ${statusClass(status, platform, key)}`;
st.textContent = statusText(status, platform, key);
info.appendChild(name); info.appendChild(st);
const act = document.createElement('span');
act.className = 'action-icon';
act.textContent = '↥';
card.appendChild(icon); card.appendChild(info); card.appendChild(act);
if (!isTokenOnly && !discordNeedsToken) {
card.addEventListener('click', () => exportPlatformCookies(key, card));
}
return card;
}
function statusText(s, platform, key) {
if (key === 'discord') return s.hasToken ? 'Token captured — ready' : 'Open Discord to capture token';
if (key === 'pixiv') return s.hasToken ? 'Token captured — ready' : 'Click to authenticate via OAuth';
if (platform.authType === 'token') return 'Manual token entry required';
if (s.error) return 'Error checking cookies';
if (!s.hasCookies || !s.cookieCount) return 'No cookies — log in first';
return `${s.cookieCount} cookies ready`;
}
function statusClass(s, platform, key) {
if (key === 'discord') return s.hasToken ? 'ready' : 'no-cookies';
if (key === 'pixiv') return s.hasToken ? 'ready' : 'no-cookies';
if (platform.authType === 'token') return 'no-cookies';
if (s.error) return 'error';
if (!s.hasCookies || !s.cookieCount) return 'no-cookies';
return 'ready';
}
async function exportPlatformCookies(key, card) {
card.classList.add('loading'); hideStatusMessage();
try {
const r = await browser.runtime.sendMessage({ type: 'EXPORT_COOKIES', platform: key });
if (r.error) showError(r.error);
else {
const n = r.cookieCount ?? null;
const msg = n !== null
? `${PLATFORMS[key].name}: ${n} cookies exported`
: `${PLATFORMS[key].name}: token exported`;
showSuccess(msg);
await loadPlatformStatus();
}
} catch (e) { showError(e.message); }
finally { card.classList.remove('loading'); }
}
async function exportAllCookies() {
const btn = document.getElementById('export-all-btn');
btn.disabled = true; btn.textContent = 'Exporting…'; hideStatusMessage();
try {
const r = await browser.runtime.sendMessage({ type: 'EXPORT_ALL_COOKIES' });
const wins = Object.values(r).filter(x => x.success).length;
const fails = Object.values(r).filter(x => !x.success && !x.skipped).length;
if (wins && !fails) showSuccess(`Exported ${wins} platforms`);
else if (wins) showWarning(`${wins} succeeded, ${fails} failed`);
else if (fails) showError('All exports failed. Are you logged in?');
else showWarning('Nothing to export');
await loadPlatformStatus();
} catch (e) { showError(e.message); }
finally { btn.disabled = false; btn.textContent = 'Export all platforms'; }
}
async function loadSources() {
const c = document.getElementById('sources-list');
c.textContent = '';
const d = document.createElement('div');
d.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
d.textContent = 'Loading sources…';
c.appendChild(d);
const r = await browser.runtime.sendMessage({ type: 'LIST_SOURCES' });
c.textContent = '';
if (r.error) {
const e = document.createElement('div');
e.style.cssText = 'padding:12px;color:var(--error);';
e.textContent = r.error;
c.appendChild(e);
return;
}
if (!r.sources || r.sources.length === 0) {
const empty = document.createElement('div');
empty.style.cssText = 'text-align:center;padding:18px;color:var(--on-surface-variant);';
empty.textContent = 'No sources yet.';
c.appendChild(empty);
return;
}
for (const src of r.sources) c.appendChild(createSourceRow(src));
}
function createSourceRow(src) {
const row = document.createElement('div');
row.className = 'source-row';
const info = document.createElement('div');
info.className = 'info';
const name = document.createElement('div');
name.className = 'name';
name.textContent = `${src.platform} · #${src.id}`;
const url = document.createElement('div');
url.className = 'url';
url.textContent = src.url;
info.appendChild(name); info.appendChild(url);
const play = document.createElement('button');
play.className = 'play';
play.textContent = '▶';
play.title = 'Check now';
play.addEventListener('click', async () => {
play.disabled = true;
const r = await browser.runtime.sendMessage({ type: 'CHECK_SOURCE', sourceId: src.id });
play.disabled = false;
if (r.error) showError(r.error);
else showSuccess(`Triggered check for source #${src.id}`);
});
row.appendChild(info); row.appendChild(play);
return row;
}
function setupEventListeners() {
document.getElementById('export-all-btn').addEventListener('click', exportAllCookies);
document.getElementById('settings-btn').addEventListener('click', () => browser.runtime.openOptionsPage());
for (const tab of document.querySelectorAll('.tab')) {
tab.addEventListener('click', () => {
for (const t of document.querySelectorAll('.tab')) t.classList.remove('active');
tab.classList.add('active');
for (const p of document.querySelectorAll('.tab-panel')) p.classList.add('hidden');
document.getElementById(`tab-${tab.dataset.tab}`).classList.remove('hidden');
if (tab.dataset.tab === 'sources') loadSources();
});
}
}
function showSuccess(m) { showStatusMessage(m, 'success'); }
function showError(m) { showStatusMessage(m, 'error'); }
function showWarning(m) { showStatusMessage(m, 'warning'); }
function showStatusMessage(text, kind) {
const el = document.getElementById('status-message');
el.textContent = text;
el.className = `status-message ${kind}`;
el.classList.remove('hidden');
}
function hideStatusMessage() {
document.getElementById('status-message').classList.add('hidden');
}
+13
View File
@@ -0,0 +1,13 @@
module.exports = {
sourceDir: '.',
artifactsDir: './web-ext-artifacts',
ignoreFiles: [
'package.json',
'package-lock.json',
'web-ext-config.cjs',
'web-ext-artifacts',
'node_modules',
'README.md',
'.gitignore',
],
};
+4
View File
@@ -3,6 +3,7 @@
<AppShell>
<RouterView />
</AppShell>
<ImageViewer v-if="modal.isOpen" @close="modal.close()" />
<AppSnackbar ref="snackbar" />
</v-app>
</template>
@@ -11,7 +12,10 @@
import { onMounted, ref } from 'vue'
import AppShell from './components/AppShell.vue'
import AppSnackbar from './components/AppSnackbar.vue'
import ImageViewer from './components/modal/ImageViewer.vue'
import { useModalStore } from './stores/modal.js'
const modal = useModalStore()
const snackbar = ref(null)
onMounted(() => {
+6
View File
@@ -18,5 +18,11 @@ const route = useRoute()
<style scoped>
.fc-content {
min-height: 100vh;
/* Push initial viewport content below the sticky TopNav. Without
this, some views' first rows / form fields / table headers can
end up obscured by the navbar (depending on parent overflow
context interacting with position: sticky). Scrolled-down content
still slides under the nav — the gradient-fade design is intact. */
padding-top: 64px;
}
</style>
+16 -3
View File
@@ -55,6 +55,15 @@ const health = computed(() => {
position: sticky;
top: 0;
z-index: 1000;
/* Both side cells use `flex: 1 1 0` — equal flex weight, basis 0 —
so they grow/shrink at the same rate regardless of which one has
content (brand vs. teleport-slot action button). The middle cell
is `flex: 0 0 auto` (content width), and because the side cells
are symmetric, the middle stays geometrically centered. Earlier
attempts: `flex: 1` defaults to basis 0%, which made the centered
links shift when actions appeared; `grid-template-columns: 1fr
auto 1fr` actually means `minmax(auto, 1fr)` so a wide brand
pushed the link block off-center on narrow viewports. */
display: flex;
align-items: center;
gap: 1rem;
@@ -87,7 +96,7 @@ const health = computed(() => {
}
.fc-links {
flex: 1;
flex: 0 0 auto;
display: flex;
flex-wrap: wrap;
justify-content: center;
@@ -114,10 +123,12 @@ const health = computed(() => {
}
.fc-nav-left {
flex: 1 1 0;
min-width: 0;
display: flex;
align-items: center;
justify-content: flex-start;
gap: 8px;
flex-shrink: 0;
}
.fc-health {
display: flex;
@@ -125,9 +136,11 @@ const health = computed(() => {
flex-shrink: 0;
}
.fc-nav-actions {
flex: 1 1 0;
min-width: 0;
display: flex;
align-items: center;
justify-content: flex-end;
gap: 0.5rem;
flex-shrink: 0;
}
</style>
@@ -86,7 +86,7 @@ function imagesForGroup(group) {
}
.fc-gallery-grid__items {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(160px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
gap: 8px;
}
.fc-gallery-grid__sentinel {
@@ -101,7 +101,7 @@ function imagesForGroup(group) {
}
.fc-gallery-grid__skeleton {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(160px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
gap: 8px;
}
.fc-gallery-grid__skeleton-item {
@@ -124,7 +124,7 @@ function imagesForGroup(group) {
@media (max-width: 600px) {
.fc-gallery-grid__items,
.fc-gallery-grid__skeleton {
grid-template-columns: repeat(auto-fill, minmax(120px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(180px, 1fr));
gap: 4px;
}
}
+35 -3
View File
@@ -1,14 +1,16 @@
<template>
<div
ref="canvasEl"
class="fc-canvas"
:class="{ 'fc-canvas--zoomed': panZoom.state.scale > 1 }"
@wheel="panZoom.handlers.onWheel"
@pointerdown="panZoom.handlers.onPointerDown"
@pointermove="panZoom.handlers.onPointerMove"
@pointerup="panZoom.handlers.onPointerUp"
@click="panZoom.handlers.onClick"
@click="onCanvasClick"
>
<img
ref="imgEl"
:src="src" :alt="alt"
:style="{
transform: `translate(${panZoom.state.x}px, ${panZoom.state.y}px) scale(${panZoom.state.scale})`
@@ -19,21 +21,51 @@
</template>
<script setup>
import { watch } from 'vue'
import { ref, watch } from 'vue'
import { usePanZoom } from '../../composables/usePanZoom.js'
const props = defineProps({ src: String, alt: String })
const emit = defineEmits(['close-request'])
const panZoom = usePanZoom()
const canvasEl = ref(null)
const imgEl = ref(null)
// Reset zoom when the src changes (prev/next nav).
watch(() => props.src, () => panZoom.reset())
// Click on the image → toggle zoom (IR/expected behavior). Click on
// the haze area around the image → request close. We use the image's
// own bounding rect rather than @click.self because the <img> sets
// pointer-events: none for the pan-drag pathway, so the click always
// targets the canvas div regardless of where the cursor was.
function onCanvasClick(ev) {
if (panZoom.state.scale > 1) {
// When zoomed, any click resets zoom — preserve old behavior.
panZoom.handlers.onClick(ev)
return
}
const img = imgEl.value
if (!img) return
const r = img.getBoundingClientRect()
const inside =
ev.clientX >= r.left && ev.clientX <= r.right &&
ev.clientY >= r.top && ev.clientY <= r.bottom
if (inside) {
panZoom.handlers.onClick(ev)
} else {
emit('close-request')
}
}
</script>
<style scoped>
.fc-canvas {
flex: 1;
position: relative;
background: rgb(var(--v-theme-background));
/* Transparent so the viewer's haze shows through to the image area
instead of a solid panel sitting on top of it. */
background: transparent;
overflow: hidden;
cursor: zoom-in;
user-select: none;
+13 -3
View File
@@ -39,9 +39,11 @@
<template v-else-if="modal.current">
<ImageCanvas
v-if="!isVideo" :src="modal.current.image_url" :alt="`Image ${modal.current.id}`"
@close-request="$emit('close')"
/>
<VideoCanvas
v-else :src="modal.current.image_url" :mime="modal.current.mime"
@close-request="$emit('close')"
/>
</template>
<v-alert v-else-if="modal.error" type="error" variant="tonal">
@@ -123,7 +125,11 @@ function isTextEntry(el) {
<style scoped>
.fc-viewer {
position: fixed; inset: 0; z-index: 2000;
background: rgba(20, 23, 26, 0.96);
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
mid-opacity + blur so the page behind shows through faintly. */
background: rgba(20, 23, 26, 0.65);
backdrop-filter: blur(8px);
-webkit-backdrop-filter: blur(8px);
outline: none;
display: flex; flex-direction: column;
}
@@ -152,8 +158,12 @@ function isTextEntry(el) {
flex: 1; display: flex; min-height: 0;
}
.fc-viewer__media {
flex: 1; display: flex; align-items: center; justify-content: center;
min-width: 0;
/* Let the canvas fill us; the canvas does its own centering. Both
min-* are needed so this child can shrink inside its flex parents
(without them, max-height/max-width on the <img> have nothing to
bound against and the image overflows the viewport). */
flex: 1; display: flex;
min-width: 0; min-height: 0;
}
.fc-viewer__side {
width: 320px; flex-shrink: 0;
@@ -1,10 +1,11 @@
<template>
<div class="fc-canvas fc-canvas--video">
<div class="fc-canvas fc-canvas--video" @click.self="$emit('close-request')">
<video
v-if="playable" :src="src" controls playsinline preload="metadata"
class="fc-canvas__video"
@click.stop
/>
<div v-else class="fc-canvas__unsupported">
<div v-else class="fc-canvas__unsupported" @click.stop>
<v-icon size="56" icon="mdi-video-off-outline" />
<h3 class="fc-canvas__unsupported-title">Format not browser-playable</h3>
<p>
@@ -23,6 +24,7 @@
<script setup>
import { computed } from 'vue'
defineEmits(['close-request'])
const props = defineProps({ src: String, mime: String })
const BROWSER_PLAYABLE = new Set([
@@ -34,7 +36,9 @@ const playable = computed(() => BROWSER_PLAYABLE.has(props.mime))
<style scoped>
.fc-canvas {
flex: 1;
background: rgb(var(--v-theme-background));
/* Transparent so the viewer's haze shows through (parallels
ImageCanvas both sit inside ImageViewer's blurred backdrop). */
background: transparent;
display: flex; align-items: center; justify-content: center;
}
.fc-canvas__video { max-width: 100%; max-height: 100%; }
@@ -0,0 +1,189 @@
<template>
<v-card class="fc-ext-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-puzzle" size="small" />
<span>Browser extension</span>
<span v-if="manifest?.installed" class="text-caption fc-muted">
· Firefox · v{{ manifest.version }}
</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2">
Pushes session cookies from supported platforms
(patreon, subscribestar, hentaifoundry, discord, pixiv, deviantart)
into FabledCurator, and lets you add a creator as a source from
their page in one click.
</p>
<v-alert
v-if="manifestError"
type="warning" variant="tonal" density="compact" class="mt-3"
>
Could not load extension manifest: {{ manifestError }}
</v-alert>
<v-alert
v-else-if="manifest && !manifest.installed"
type="warning" variant="tonal" density="compact" class="mt-3"
>
No bundled extension found in this image. Push a release that
runs the extension CI workflow, or grab the XPI from the
FabledCurator Forgejo releases page.
</v-alert>
<template v-else-if="manifest?.installed">
<div class="fc-ext-install mt-3">
<v-btn
v-if="isFirefox"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-firefox"
@click="installXpi"
>Install Firefox extension</v-btn>
<v-btn
variant="outlined" rounded="pill"
:href="manifest.latest_url" download
prepend-icon="mdi-download"
>Download XPI</v-btn>
<v-alert
v-if="!isFirefox"
type="info" variant="tonal" density="compact" class="mt-3"
>
Open this page in Firefox to install in one click, or use
"Download XPI" to install manually.
</v-alert>
</div>
<v-divider class="my-4" />
<div class="fc-muted text-body-2 mb-3">
After installing, open the extension's options page
(about:addons → FabledCurator → Preferences) and paste these:
</div>
<v-text-field
label="FC base URL" :model-value="apiUrl"
readonly density="compact" hide-details
append-inner-icon="mdi-content-copy"
@click:append-inner="copy(apiUrl, 'URL')"
/>
<v-text-field
label="Extension API key"
:model-value="apiKey"
:type="keyShown ? 'text' : 'password'"
readonly density="compact" hide-details class="mt-3"
>
<template #append-inner>
<v-btn
variant="text" density="compact" size="small"
:icon="keyShown ? 'mdi-eye-off' : 'mdi-eye'"
@click="keyShown = !keyShown"
/>
<v-btn
variant="text" density="compact" size="small"
icon="mdi-content-copy"
@click="copy(apiKey, 'API key')"
/>
<v-btn
variant="text" density="compact" size="small"
icon="mdi-refresh" color="warning"
:loading="rotating"
@click="rotateKey"
/>
</template>
</v-text-field>
</template>
</v-card-text>
</v-card>
</template>
<script setup>
import { computed, onMounted, ref } from 'vue'
import { useApi } from '../../composables/useApi.js'
const api = useApi()
const manifest = ref(null)
const manifestError = ref(null)
const apiKey = ref('')
const keyShown = ref(false)
const rotating = ref(false)
const apiUrl = computed(() => `${window.location.origin}/api`)
const isFirefox = computed(() => navigator.userAgent.includes('Firefox'))
onMounted(async () => {
await Promise.all([loadManifest(), loadKey()])
})
async function loadManifest() {
try {
manifest.value = await api.get('/api/extension/manifest')
} catch (e) {
if (e.status === 404) {
// Backend says no XPI is bundled — surface the not-installed
// state, not an error.
manifest.value = { installed: false }
} else {
manifestError.value = e.message
}
}
}
async function loadKey() {
try {
const { key } = await api.get('/api/settings/extension_api_key')
apiKey.value = key
} catch (e) {
apiKey.value = ''
window.__fcToast?.({
text: `Failed to load extension API key: ${e.message}`,
type: 'error',
})
}
}
function installXpi() {
if (!manifest.value?.latest_url) return
window.location.assign(manifest.value.latest_url)
}
async function rotateKey() {
rotating.value = true
try {
const { key } = await api.post('/api/settings/extension_api_key/rotate')
apiKey.value = key
keyShown.value = true
window.__fcToast?.({ text: 'Extension API key rotated.', type: 'success' })
} catch (e) {
window.__fcToast?.({
text: `Rotate failed: ${e.message}`,
type: 'error',
})
} finally {
rotating.value = false
}
}
async function copy(text, label) {
try {
await navigator.clipboard.writeText(text)
window.__fcToast?.({ text: `${label} copied.`, type: 'success' })
} catch (e) {
window.__fcToast?.({ text: `Copy failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-ext-card { border-radius: 8px; }
.fc-ext-install {
display: flex; flex-wrap: wrap; gap: 8px;
}
.fc-muted {
color: rgb(var(--v-theme-on-surface-variant));
}
</style>
@@ -12,6 +12,7 @@
<MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" />
<AliasTable class="mt-4" />
<BrowserExtensionCard class="mt-6" />
<LegacyMigrationCard class="mt-6" />
</div>
</template>
@@ -22,6 +23,7 @@ import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue'
import BrowserExtensionCard from './BrowserExtensionCard.vue'
import LegacyMigrationCard from './LegacyMigrationCard.vue'
</script>
@@ -1,6 +1,6 @@
<template>
<v-row dense>
<v-col v-for="card in cards" :key="card.label" cols="12" sm="6" md="4" lg="2">
<v-col v-for="card in cards" :key="card.label" cols="12" sm="6" md="4" lg="3" xl="2">
<v-card class="fc-stat">
<v-card-text>
<div class="fc-stat__label text-caption">{{ card.label }}</div>
@@ -33,15 +33,17 @@ const cards = computed(() => {
{ label: 'Total images', value: '—' },
{ label: 'Total tags', value: '—' },
{ label: 'Storage used', value: '—' },
{ label: 'Subscriptions', value: '—' },
{ label: 'Pending tasks', value: '—' },
{ label: 'Failed tasks', value: '—' }
]
return [
{ label: 'Total images', value: s.total_images.toLocaleString() },
{ label: 'Total tags', value: s.total_tags.toLocaleString() },
{ label: 'Storage used', value: formatBytes(s.storage_bytes) },
{ label: 'Pending', value: (s.tasks.pending + s.tasks.queued).toLocaleString() },
{ label: 'Failed', value: s.tasks.failed.toLocaleString() }
{ label: 'Total images', value: s.total_images.toLocaleString() },
{ label: 'Total tags', value: s.total_tags.toLocaleString() },
{ label: 'Storage used', value: formatBytes(s.storage_bytes) },
{ label: 'Subscriptions', value: (s.subscription_count ?? 0).toLocaleString() },
{ label: 'Pending', value: (s.tasks.pending + s.tasks.queued).toLocaleString() },
{ label: 'Failed', value: s.tasks.failed.toLocaleString() }
]
})
</script>
@@ -1,64 +0,0 @@
<template>
<section class="fc-artist-section">
<header class="fc-artist-section__head" @click="$emit('toggle')">
<v-icon :icon="open ? 'mdi-chevron-down' : 'mdi-chevron-right'" size="small" />
<span class="fc-artist-section__name">{{ artist.name }}</span>
<span class="fc-artist-section__meta">
{{ sources.length }} source{{ sources.length === 1 ? '' : 's' }}
<template v-if="latestChecked">
· last check {{ latestChecked }}
</template>
</span>
</header>
<div v-if="open" class="fc-artist-section__body">
<SourceRow
v-for="s in sources" :key="s.id" :source="s"
:checking="checkingIds.has(s.id)"
@edit="$emit('edit', $event)"
@remove="$emit('remove', $event)"
@toggle="$emit('toggle-source', $event)"
@check="$emit('check', $event)"
/>
<v-btn
size="small" variant="text" prepend-icon="mdi-plus"
class="fc-artist-section__add"
@click="$emit('add-source', artist)"
>Add source to {{ artist.name }}</v-btn>
</div>
</section>
</template>
<script setup>
import { computed } from 'vue'
import SourceRow from './SourceRow.vue'
const props = defineProps({
artist: { type: Object, required: true },
sources: { type: Array, required: true },
open: { type: Boolean, default: false },
checkingIds: { type: Set, default: () => new Set() },
})
defineEmits(['toggle', 'edit', 'remove', 'toggle-source', 'add-source', 'check'])
const latestChecked = computed(() => {
const dates = props.sources.map(s => s.last_checked_at).filter(Boolean)
if (dates.length === 0) return null
const latest = dates.sort().slice(-1)[0]
return latest.slice(0, 10)
})
</script>
<style scoped>
.fc-artist-section { border-bottom: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.18); }
.fc-artist-section__head {
display: flex; align-items: center; gap: 0.5rem;
padding: 0.75rem 0; cursor: pointer; user-select: none;
}
.fc-artist-section__name { font-weight: 600; }
.fc-artist-section__meta {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem;
}
.fc-artist-section__body { padding: 0 0 0.75rem 1.5rem; }
.fc-artist-section__add { margin-top: 0.25rem; }
</style>
@@ -1,25 +1,63 @@
<template>
<div class="fc-source-row">
<SourceHealthDot :source="source" :warning-threshold="warningThreshold" />
<v-chip size="x-small" variant="tonal" class="fc-source-row__platform">
{{ source.platform }}
</v-chip>
<a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url">
{{ source.url }}
</a>
<v-switch
:model-value="source.enabled"
density="compact" hide-details color="accent"
@update:model-value="onToggleEnabled"
/>
<v-btn
icon="mdi-play" size="x-small" variant="text"
:loading="checking"
@click="$emit('check', source)"
/>
<v-btn icon="mdi-pencil" size="x-small" variant="text" @click="$emit('edit', source)" />
<v-btn icon="mdi-close" size="x-small" variant="text" @click="$emit('remove', source)" />
</div>
<tr class="fc-source-row">
<td class="fc-source-row__health">
<SourceHealthDot :source="source" :warning-threshold="warningThreshold" />
</td>
<td>
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
</td>
<td class="fc-source-row__url-cell">
<a :href="source.url" target="_blank" rel="noopener" class="fc-source-row__url"
@click.stop>
{{ source.url }}
</a>
</td>
<td>
<v-switch
:model-value="source.enabled"
density="compact" hide-details color="accent"
@click.stop
@update:model-value="onToggleEnabled"
/>
</td>
<td class="fc-source-row__when">
{{ formatRelative(source.last_checked_at) }}
</td>
<td class="fc-source-row__when">
{{ formatRelative(source.next_check_at, { future: true }) }}
</td>
<td>
<v-chip
v-if="(source.consecutive_failures || 0) > 0"
size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }}</v-chip>
<span v-else class="fc-source-row__zero">0</span>
</td>
<td class="fc-source-row__actions">
<v-btn
icon="mdi-play" size="x-small" variant="text"
:loading="checking"
@click.stop="$emit('check', source)"
>
<v-icon>mdi-play</v-icon>
<v-tooltip activator="parent" location="top">Check now</v-tooltip>
</v-btn>
<v-btn
icon="mdi-pencil" size="x-small" variant="text"
@click.stop="$emit('edit', source)"
>
<v-icon>mdi-pencil</v-icon>
<v-tooltip activator="parent" location="top">Edit</v-tooltip>
</v-btn>
<v-btn
icon="mdi-close" size="x-small" variant="text" color="error"
@click.stop="$emit('remove', source)"
>
<v-icon>mdi-close</v-icon>
<v-tooltip activator="parent" location="top">Remove</v-tooltip>
</v-btn>
</td>
</tr>
</template>
<script setup>
@@ -31,25 +69,59 @@ const props = defineProps({
warningThreshold: { type: Number, default: 5 },
})
const emit = defineEmits(['edit', 'remove', 'toggle', 'check'])
function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value })
}
function formatRelative(iso, opts = {}) {
if (!iso) return opts.future ? '—' : 'Never'
const then = new Date(iso).getTime()
const now = Date.now()
const diff = (then - now) / 1000 // seconds; positive = future, negative = past
const abs = Math.abs(diff)
let body
if (abs < 60) body = `${Math.floor(abs)}s`
else if (abs < 3600) body = `${Math.floor(abs / 60)}m`
else if (abs < 86400) body = `${Math.floor(abs / 3600)}h`
else body = `${Math.floor(abs / 86400)}d`
if (opts.future) return diff <= 0 ? 'imminent' : `in ${body}`
return `${body} ago`
}
</script>
<style scoped>
.fc-source-row {
display: grid;
grid-template-columns: auto 96px 1fr auto auto auto auto;
gap: 0.75rem;
align-items: center;
padding: 0.4rem 0;
.fc-source-row__health {
width: 24px;
padding-right: 0 !important;
}
.fc-source-row__url-cell {
max-width: 400px;
overflow: hidden;
}
.fc-source-row__url {
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none;
display: inline-block;
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.fc-source-row__url:hover { color: rgb(var(--v-theme-accent)); }
.fc-source-row__when {
color: rgb(var(--v-theme-on-surface-variant));
white-space: nowrap;
font-variant-numeric: tabular-nums;
}
.fc-source-row__zero {
color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.6;
}
.fc-source-row__actions {
white-space: nowrap;
text-align: right;
}
</style>
+10 -2
View File
@@ -24,8 +24,16 @@ async function request(method, url, { body, params, signal } = {}) {
const headers = {}
const init = { method, headers, signal }
if (body !== undefined) {
headers['Content-Type'] = 'application/json'
init.body = JSON.stringify(body)
if (body instanceof FormData) {
// Let the browser set Content-Type with the multipart boundary.
// JSON-stringifying FormData produces "{}" and the backend sees
// an empty multipart payload (this broke FC-5 IR/GS ingest from
// the UI for several releases).
init.body = body
} else {
headers['Content-Type'] = 'application/json'
init.body = JSON.stringify(body)
}
}
const r = await fetch(fullUrl, init)
+3 -2
View File
@@ -29,10 +29,11 @@ export function distributeIntoColumns(items, columnCount) {
}
// Reactive column count from container width. Breakpoints mirror the
// gallery grid intent: ~260px target column width, min 1 column.
// gallery grid intent: ~390px target column width, min 1 column. Bumped
// from 260px on 2026-05-23 per dogfood UX feedback (thumbs were too small).
export function columnCountForWidth(width) {
if (!width || width < 0) return 1
return Math.max(1, Math.floor(width / 260))
return Math.max(1, Math.floor(width / 390))
}
export function usePolyMasonry(containerRef) {
+3 -1
View File
@@ -104,11 +104,13 @@
import { computed, watch } from 'vue'
import { useRoute, useRouter, RouterLink } from 'vue-router'
import { useArtistStore } from '../stores/artist.js'
import { useModalStore } from '../stores/modal.js'
import MasonryGrid from '../components/discovery/MasonryGrid.vue'
const route = useRoute()
const router = useRouter()
const store = useArtistStore()
const modal = useModalStore()
const slug = computed(() => route.params.slug)
@@ -136,7 +138,7 @@ const sparkPoints = computed(() => {
})
function openImage(id) {
router.push({ name: 'gallery', query: { image: id } })
modal.open(id)
}
function openTag(tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } })
-5
View File
@@ -18,10 +18,6 @@
<TimelineSidebar v-if="store.images.length > 0" class="fc-gallery-layout__sidebar" />
</div>
<ImageViewer
v-if="modal.currentImageId !== null"
@close="closeImage"
/>
<BulkEditorPanel />
</v-container>
</template>
@@ -35,7 +31,6 @@ import GalleryGrid from '../components/gallery/GalleryGrid.vue'
import TimelineSidebar from '../components/gallery/TimelineSidebar.vue'
import EmptyState from '../components/gallery/EmptyState.vue'
import PostInfoHeader from '../components/gallery/PostInfoHeader.vue'
import ImageViewer from '../components/modal/ImageViewer.vue'
import BulkEditorPanel from '../components/gallery/BulkEditorPanel.vue'
import { useGallerySelectionStore } from '../stores/gallerySelection.js'
+7 -2
View File
@@ -1,5 +1,5 @@
<template>
<v-container class="py-6">
<v-container fluid class="py-6">
<v-tabs v-model="tab" color="accent" class="mb-4">
<v-tab value="overview">Overview</v-tab>
<v-tab value="import">Import</v-tab>
@@ -57,7 +57,12 @@ function startPolling() {
pollId = setInterval(() => {
if (!document.hidden) {
system.refreshStats()
if (tab.value === 'import') importStore.refreshStatus()
if (tab.value === 'import') {
importStore.refreshStatus()
// Refresh the task list while a batch is in flight so the UI
// doesn't sit stale next to a ticking imported/skipped counter.
if (importStore.activeBatch) importStore.loadTasks(true)
}
}
}, 5000)
}
+3 -3
View File
@@ -26,16 +26,16 @@
<script setup>
import { onMounted } from 'vue'
import { useRouter } from 'vue-router'
import { useShowcaseStore } from '../stores/showcase.js'
import { useModalStore } from '../stores/modal.js'
import MasonryGrid from '../components/discovery/MasonryGrid.vue'
const store = useShowcaseStore()
const router = useRouter()
const modal = useModalStore()
onMounted(() => { if (store.images.length === 0) store.fetchPage() })
function openImage(id) {
router.push({ name: 'gallery', query: { image: id } })
modal.open(id)
}
</script>
+283 -39
View File
@@ -8,9 +8,13 @@
New artist
</v-btn>
<v-spacer />
<v-btn variant="text" size="small" @click="expandAll = !expandAll">
{{ expandAll ? 'Collapse all' : 'Expand all' }}
</v-btn>
<v-text-field
v-model="search"
density="compact" variant="outlined" hide-details clearable
prepend-inner-icon="mdi-magnify"
placeholder="Search subscriptions"
style="max-width: 320px"
/>
</div>
<v-alert v-if="store.error" type="error" variant="tonal" closable class="mt-4">
@@ -21,24 +25,121 @@
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<div v-else-if="groups.length === 0" class="fc-subs__empty">
<p>No subscriptions yet. Add your first artist.</p>
<div v-else-if="filteredGroups.length === 0" class="fc-subs__empty">
<p v-if="groups.length === 0">No subscriptions yet. Add your first artist.</p>
<p v-else>No subscriptions match "{{ search }}".</p>
</div>
<div v-else>
<ArtistSection
v-for="g in groups" :key="g.artist.id"
:artist="g.artist" :sources="g.sources"
:open="isOpen(g.artist.id)"
:checking-ids="store.checkingIds"
@toggle="toggleSection(g.artist.id)"
@edit="openEditSource"
@remove="removeSource"
@toggle-source="toggleSourceEnabled"
@add-source="openAddSource"
@check="onCheck"
/>
</div>
<v-card v-else class="fc-subs__card" variant="outlined">
<v-data-table
:headers="headers"
:items="filteredGroups"
item-value="key"
v-model:expanded="expanded"
:items-per-page="50"
:items-per-page-options="ITEMS_PER_PAGE_OPTIONS"
density="comfortable"
hover
show-expand
@click:row="onRowClick"
>
<template #item.name="{ item }">
<span class="fc-subs__name">{{ item.artist.name }}</span>
</template>
<template #item.sources_count="{ item }">
<v-chip size="x-small" variant="tonal" label>
{{ item.sources.length }} source{{ item.sources.length === 1 ? '' : 's' }}
</v-chip>
</template>
<template #item.health="{ item }">
<SourceHealthDot
v-if="item.worstSource"
:source="item.worstSource"
:warning-threshold="failureThreshold"
/>
<span v-else class="fc-subs__zero"></span>
</template>
<template #item.last_activity="{ item }">
<span class="fc-subs__when">
{{ formatRelative(item.lastActivity) }}
</span>
</template>
<template #item.actions="{ item }">
<v-btn
icon size="small" variant="text"
:loading="anyChecking(item.sources)"
@click.stop="checkAll(item)"
>
<v-icon>mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Check all sources</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
@click.stop="openAddSource(item.artist)"
>
<v-icon>mdi-plus</v-icon>
<v-tooltip activator="parent" location="top">Add source</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/posts?artist_id=${item.artist.id}`"
@click.stop
>
<v-icon>mdi-rss</v-icon>
<v-tooltip activator="parent" location="top">View posts</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/artist/${item.artist.slug}`"
@click.stop
>
<v-icon>mdi-account</v-icon>
<v-tooltip activator="parent" location="top">Open artist page</v-tooltip>
</v-btn>
</template>
<template #expanded-row="{ columns, item }">
<tr class="fc-subs__sources-row">
<td :colspan="columns.length" class="fc-subs__sources-cell">
<v-table density="compact" class="fc-subs__sources-table">
<thead>
<tr>
<th></th>
<th>Platform</th>
<th>URL</th>
<th>Enabled</th>
<th>Last check</th>
<th>Next check</th>
<th>Errors</th>
<th class="text-right">Actions</th>
</tr>
</thead>
<tbody>
<SourceRow
v-for="s in item.sources" :key="s.id" :source="s"
:checking="store.checkingIds.has(s.id)"
:warning-threshold="failureThreshold"
@edit="openEditSource"
@remove="removeSource"
@toggle="toggleSourceEnabled"
@check="onCheck"
/>
<tr v-if="item.sources.length === 0">
<td colspan="8" class="fc-subs__sources-empty">
No sources yet. Click + to add one.
</td>
</tr>
</tbody>
</v-table>
</td>
</tr>
</template>
</v-data-table>
</v-card>
<SourceFormDialog
v-model="showSourceDialog"
@@ -55,52 +156,140 @@ import { computed, onMounted, ref, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useSourcesStore } from '../stores/sources.js'
import { usePlatformsStore } from '../stores/platforms.js'
import ArtistSection from '../components/subscriptions/ArtistSection.vue'
import { useImportStore } from '../stores/import.js'
import SourceRow from '../components/subscriptions/SourceRow.vue'
import SourceHealthDot from '../components/subscriptions/SourceHealthDot.vue'
import SourceFormDialog from '../components/subscriptions/SourceFormDialog.vue'
import ArtistCreateDialog from '../components/subscriptions/ArtistCreateDialog.vue'
const ITEMS_PER_PAGE_OPTIONS = [
{ value: 25, title: '25' },
{ value: 50, title: '50' },
{ value: 100, title: '100' },
{ value: -1, title: 'All' },
]
const route = useRoute()
const router = useRouter()
const store = useSourcesStore()
const platformsStore = usePlatformsStore()
const importStore = useImportStore()
const search = ref('')
const expanded = ref([])
const showSourceDialog = ref(false)
const editingSource = ref(null)
const editingArtist = ref(null)
const showArtistDialog = ref(false)
const artistFilter = computed(() => {
const raw = route.query.artist_id
return raw == null ? null : Number(raw)
})
const expandAll = ref(false)
const openSections = ref(new Set())
const showSourceDialog = ref(false)
const editingSource = ref(null)
const editingArtist = ref(null)
const showArtistDialog = ref(false)
const failureThreshold = computed(() =>
importStore.settings?.download_failure_warning_threshold ?? 5
)
async function refresh() {
await store.loadAll()
await platformsStore.loadAll()
if (artistFilter.value != null) {
openSections.value = new Set([artistFilter.value])
expandAll.value = false
}
if (!importStore.settings) await importStore.loadSettings()
}
onMounted(refresh)
onMounted(() => {
refresh()
if (artistFilter.value != null) {
// Pre-expand the row that the deep-link refers to.
expanded.value = [`artist-${artistFilter.value}`]
}
})
watch(() => route.query.artist_id, refresh)
const headers = [
{ title: 'Subscription', key: 'name', sortable: true, align: 'start' },
{ title: 'Sources', key: 'sources_count', sortable: true, align: 'start', width: 110 },
{ title: 'Health', key: 'health', sortable: false, align: 'start', width: 80 },
{ title: 'Last activity',key: 'last_activity', sortable: true, align: 'start', width: 140 },
{ title: 'Actions', key: 'actions', sortable: false, align: 'end', width: 200 },
]
const groups = computed(() => {
const all = store.sourcesByArtistGrouped()
if (artistFilter.value == null) return all
return all.filter(g => g.artist.id === artistFilter.value)
return all.map(g => {
const worstSource = pickWorstSource(g.sources, failureThreshold.value)
const lastActivity = pickLastActivity(g.sources)
return {
key: `artist-${g.artist.id}`,
artist: g.artist,
sources: g.sources,
sources_count: g.sources.length,
worstSource,
lastActivity,
name: g.artist.name, // for sortable column
last_activity: lastActivity ?? '', // for sortable column
}
})
})
function isOpen(artistId) {
return expandAll.value || openSections.value.has(artistId)
const filteredGroups = computed(() => {
let arr = groups.value
if (artistFilter.value != null) {
arr = arr.filter(g => g.artist.id === artistFilter.value)
}
const q = search.value?.trim().toLowerCase()
if (q) {
arr = arr.filter(g =>
g.artist.name.toLowerCase().includes(q)
|| g.sources.some(s => (s.url || '').toLowerCase().includes(q)
|| (s.platform || '').toLowerCase().includes(q))
)
}
return arr
})
function pickLastActivity(sources) {
let max = null
for (const s of sources) {
if (s.last_checked_at && (!max || s.last_checked_at > max)) max = s.last_checked_at
}
return max
}
function toggleSection(artistId) {
if (openSections.value.has(artistId)) openSections.value.delete(artistId)
else openSections.value.add(artistId)
function pickWorstSource(sources, threshold) {
// Health order (worst → best): critical, warning, healthy, unchecked.
// Picks the source with the worst level so the row's dot reflects the
// worst-case state. Within a level, the first is fine.
if (!sources || sources.length === 0) return null
function level(s) {
if (!s.last_checked_at) return 0 // unchecked
const f = s.consecutive_failures || 0
if (f === 0) return 1 // healthy
if (f < threshold) return 2 // warning
return 3 // critical
}
return sources.reduce((worst, s) =>
level(s) > level(worst) ? s : worst,
sources[0],
)
}
function formatRelative(iso) {
if (!iso) return 'Never'
const then = new Date(iso).getTime()
const diff = (Date.now() - then) / 1000
if (diff < 60) return `${Math.floor(diff)}s ago`
if (diff < 3600) return `${Math.floor(diff / 60)}m ago`
if (diff < 86400) return `${Math.floor(diff / 3600)}h ago`
return `${Math.floor(diff / 86400)}d ago`
}
function onRowClick(_evt, { item, internalItem }) {
// Toggle expansion on row click (in addition to the chevron).
const key = item.key
const idx = expanded.value.indexOf(key)
if (idx === -1) expanded.value = [...expanded.value, key]
else expanded.value = expanded.value.filter(k => k !== key)
}
function openAddSource(artist) {
@@ -132,7 +321,6 @@ async function onSourceSaved() {
function onArtistCreated(artist) {
showArtistDialog.value = false
// Move into Add Source for the new artist immediately.
openAddSource(artist)
}
@@ -158,6 +346,31 @@ async function onCheck(source) {
}
}
}
async function checkAll(group) {
let ok = 0
let conflict = 0
for (const s of group.sources) {
if (!s.enabled) continue
try {
await store.checkNow(s.id)
ok += 1
} catch (e) {
if (e?.body?.download_event_id) conflict += 1
}
}
const parts = []
if (ok) parts.push(`${ok} queued`)
if (conflict) parts.push(`${conflict} already running`)
globalThis.window?.__fcToast?.({
text: parts.join(', ') || 'Nothing to check (no enabled sources)',
type: 'info',
})
}
function anyChecking(sources) {
return sources.some(s => store.checkingIds.has(s.id))
}
</script>
<style scoped>
@@ -169,4 +382,35 @@ async function onCheck(source) {
display: flex; justify-content: center; padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-subs__card {
background: rgb(var(--v-theme-surface));
}
.fc-subs__name {
font-weight: 600;
}
.fc-subs__when {
color: rgb(var(--v-theme-on-surface-variant));
white-space: nowrap;
font-variant-numeric: tabular-nums;
}
.fc-subs__zero {
color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.6;
}
.fc-subs__sources-row td {
padding: 0 !important;
background: rgb(var(--v-theme-surface-light));
}
.fc-subs__sources-cell {
padding-left: 2rem !important;
border-top: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.15);
}
.fc-subs__sources-table {
background: transparent !important;
}
.fc-subs__sources-empty {
color: rgb(var(--v-theme-on-surface-variant));
text-align: center;
padding: 1rem;
}
</style>
+277
View File
@@ -0,0 +1,277 @@
"""FC-3g: /api/extension and /extension/<filename> integration tests."""
import hashlib
import pytest
import pytest_asyncio
from sqlalchemy import func, select
from backend.app import create_app
from backend.app import frontend as frontend_module
from backend.app.api import extension as extension_module
from backend.app.models import AppSetting, Artist, Source
pytestmark = pytest.mark.integration
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
@pytest_asyncio.fixture
async def ext_key(db):
db.add(AppSetting(key="extension_api_key", value="test-ext-key"))
await db.commit()
return "test-ext-key"
# --- /api/extension/quick-add-source ---------------------------------
@pytest.mark.asyncio
async def test_quick_add_source_creates_artist_and_source(client, ext_key, db_sync):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/maewix"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 201
body = await resp.get_json()
assert body["created_artist"] is True
assert body["created_source"] is True
assert body["artist"]["slug"] == "maewix"
assert body["source"]["platform"] == "patreon"
assert body["source"]["enabled"] is True
# Async Core-DML assertion: column select, not ORM attribute access.
artist_count = db_sync.execute(
select(func.count(Artist.id)).where(Artist.slug == "maewix")
).scalar_one()
assert artist_count == 1
source_count = db_sync.execute(
select(func.count(Source.id)).where(
Source.platform == "patreon",
Source.url == "https://www.patreon.com/maewix",
)
).scalar_one()
assert source_count == 1
@pytest.mark.asyncio
async def test_quick_add_source_idempotent(client, ext_key):
body = {"url": "https://www.subscribestar.com/some-creator"}
headers = {"X-Extension-Key": ext_key}
r1 = await client.post("/api/extension/quick-add-source", json=body, headers=headers)
r2 = await client.post("/api/extension/quick-add-source", json=body, headers=headers)
assert r1.status_code == 201
assert r2.status_code == 200
body2 = await r2.get_json()
assert body2["created_source"] is False
assert body2["created_artist"] is False
@pytest.mark.parametrize("url,platform,slug", [
("https://www.patreon.com/maewix", "patreon", "maewix"),
("https://patreon.com/maewix", "patreon", "maewix"),
("https://www.subscribestar.com/foobar", "subscribestar", "foobar"),
("https://subscribestar.adult/foobar", "subscribestar", "foobar"),
("https://www.hentai-foundry.com/user/Foo/profile", "hentaifoundry", "Foo"),
("https://www.deviantart.com/baz", "deviantart", "baz"),
("https://www.pixiv.net/users/12345", "pixiv", "12345"),
("https://www.pixiv.net/en/users/12345", "pixiv", "12345"),
])
@pytest.mark.asyncio
async def test_quick_add_source_url_patterns(client, ext_key, url, platform, slug):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": url},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 201, await resp.get_json()
body = await resp.get_json()
assert body["source"]["platform"] == platform
# slugify lowercases — the Artist slug should reflect that.
assert body["artist"]["slug"] == slug.lower()
@pytest.mark.asyncio
async def test_quick_add_source_unknown_url_400(client, ext_key):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://example.com/foo"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "unknown_platform"
assert "known" in body
@pytest.mark.asyncio
async def test_quick_add_source_invalid_url_400(client, ext_key):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "not-a-url"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_url"
@pytest.mark.asyncio
async def test_quick_add_source_missing_key_401(client):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/maewix"},
)
assert resp.status_code == 401
body = await resp.get_json()
assert body["error"] == "unauthorized"
@pytest.mark.asyncio
async def test_quick_add_source_wrong_key_401(client, ext_key):
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/maewix"},
headers={"X-Extension-Key": "wrong-key"},
)
assert resp.status_code == 401
@pytest.mark.asyncio
async def test_quick_add_source_missing_body_400(client, ext_key):
resp = await client.post(
"/api/extension/quick-add-source",
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_body"
@pytest.mark.asyncio
async def test_quick_add_source_attaches_to_existing_artist(client, ext_key, db, db_sync):
db.add(Artist(name="Maewix Original", slug="maewix", is_subscription=False))
await db.commit()
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/maewix"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 201
body = await resp.get_json()
assert body["created_artist"] is False
assert body["created_source"] is True
artist_count = db_sync.execute(
select(func.count(Artist.id)).where(Artist.slug == "maewix")
).scalar_one()
assert artist_count == 1 # no duplicate created
# --- /api/extension/manifest ---------------------------------------
@pytest.mark.asyncio
async def test_extension_manifest_returns_404_when_dir_missing(client, monkeypatch, tmp_path):
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path / "does-not-exist")
resp = await client.get("/api/extension/manifest")
assert resp.status_code == 404
body = await resp.get_json()
assert body == {"installed": False}
@pytest.mark.asyncio
async def test_extension_manifest_returns_404_when_no_xpi_files(client, monkeypatch, tmp_path):
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
resp = await client.get("/api/extension/manifest")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_extension_manifest_returns_metadata_when_xpi_present(client, monkeypatch, tmp_path):
xpi = tmp_path / "fabledcurator-1.2.3.xpi"
xpi.write_bytes(b"fake-xpi-content")
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
resp = await client.get("/api/extension/manifest")
assert resp.status_code == 200
body = await resp.get_json()
assert body["installed"] is True
assert body["version"] == "1.2.3"
assert body["xpi_url"] == "/extension/fabledcurator-1.2.3.xpi"
assert body["latest_url"] == "/extension/fabledcurator-latest.xpi"
assert body["sha256"] == hashlib.sha256(b"fake-xpi-content").hexdigest()
# --- /extension/<filename> -----------------------------------------
@pytest.mark.asyncio
async def test_serve_extension_rejects_non_xpi_filename(client, monkeypatch, tmp_path):
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/passwd")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_serve_extension_rejects_path_traversal(client, monkeypatch, tmp_path):
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
# Path-traversal attempt — the route regex alone catches this since
# `..` characters aren't in [\w.-]+, but cover the case anyway.
resp = await client.get("/extension/fabledcurator-..%2Fetc%2Fpasswd.xpi")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_serve_extension_404_when_xpi_missing(client, monkeypatch, tmp_path):
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-1.0.0.xpi")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_serve_extension_serves_specific_xpi_with_correct_mime(
client, monkeypatch, tmp_path,
):
xpi = tmp_path / "fabledcurator-1.0.0.xpi"
xpi.write_bytes(b"xpi-bytes")
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-1.0.0.xpi")
assert resp.status_code == 200
assert resp.headers["Content-Type"].startswith("application/x-xpinstall")
@pytest.mark.asyncio
async def test_serve_extension_latest_returns_most_recent_xpi(
client, monkeypatch, tmp_path,
):
import os
import time
older = tmp_path / "fabledcurator-1.0.0.xpi"
newer = tmp_path / "fabledcurator-1.0.1.xpi"
older.write_bytes(b"old")
newer.write_bytes(b"new")
os.utime(older, (time.time() - 10, time.time() - 10))
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-latest.xpi")
assert resp.status_code == 200
data = await resp.get_data()
assert data == b"new"
@pytest.mark.asyncio
async def test_serve_extension_latest_404_when_dir_empty(client, monkeypatch, tmp_path):
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-latest.xpi")
assert resp.status_code == 404
+1 -1
View File
@@ -76,7 +76,7 @@ async def test_system_stats_shape(client):
resp = await client.get("/api/system/stats")
assert resp.status_code == 200
body = await resp.get_json()
for key in ("total_images", "total_tags", "storage_bytes", "tasks", "active_batch"):
for key in ("total_images", "total_tags", "storage_bytes", "subscription_count", "tasks", "active_batch"):
assert key in body
for status in ("pending", "queued", "processing", "complete", "skipped", "failed"):
assert status in body["tasks"]
+6 -1
View File
@@ -30,9 +30,14 @@ def test_recover_interrupted_only_old(db_sync, monkeypatch):
batch_id = _make_batch(db_sync)
now = datetime.now(UTC)
# "Fresh" must sit comfortably under whatever STUCK_THRESHOLD_MINUTES
# currently is (5 min as of 2026-05-24, tightened from 30); 30
# seconds is well below any reasonable threshold. "Stale" stays at
# 2 hours so the test remains valid if the threshold ever moves
# back up.
fresh = ImportTask(
batch_id=batch_id, source_path="/import/a.jpg", task_type="media",
status="processing", started_at=now - timedelta(minutes=5),
status="processing", started_at=now - timedelta(seconds=30),
)
stale = ImportTask(
batch_id=batch_id, source_path="/import/b.jpg", task_type="media",