Compare commits

...
20 Commits
Author SHA1 Message Date
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-ml (push) Successful in 3m28s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
CI / intimp (push) Successful in 4m26s
CI / lint (push) Successful in 3s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen d3245f0c22 feat(ext): verify cookies in-browser before uploading (1.0.7)
CI / intcore (push) Successful in 8m45s
extension / lint (pull_request) Successful in 14s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 40s
extension / lint (push) Successful in 36s
CI / lint (push) Successful in 4s
CI / intimp (push) Successful in 4m4s
CI / intapi (push) Successful in 8m2s
Pre-upload verify request: after capturing the live browser cookies,
hit a known authenticated endpoint with credentials:'include' from the
extension's background context. If the platform reports we're not
logged in, abort the upload so we don't overwrite FC-side credentials
with stale data.

- platforms.js: add `verify` config per cookie-auth platform
  - hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF
    _init_site_filters; same 401 path the operator hit 2026-06-03)
  - patreon: GET /api/current_user (clean 401 when logged out)
  - subscribestar, deviantart: no stable auth endpoint, skip verify
- cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}.
  ok=true/false/null tri-state — null = verify not configured, caller
  treats as "proceed".
- background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the
  upload; failures bubble up with the platform's name + reason.
- popup.js: success message now appends "(verified ✓)" when applicable.
- manifest + package.json: 1.0.6 → 1.0.7.
2026-06-03 14:04:45 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
2026-06-03 13:18:44 -04:00
bvandeusen e450145304 fix(ml): preserve digit-only tag names in normalize (year tags)
CI / frontend-build (push) Successful in 21s
CI / intapi (push) Successful in 7m42s
CI / intcore (push) Successful in 8m12s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / intimp (push) Successful in 3m45s
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
2026-06-03 13:09:35 -04:00
bvandeusen a6e8d4b52e feat(ml): normalize Camie suggestion names to human-readable
CI / lint (push) Successful in 2s
CI / intimp (push) Successful in 3m57s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m22s
CI / backend-lint-and-test (push) Failing after 24s
CI / frontend-build (push) Successful in 28s
Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`,
`#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were
surfacing raw in SuggestionsPanel — and worse, the SAME raw string was
written to tag.name on Accept, polluting the DB with `underscored_lowercase`
names that don't match the operator's "Title Case" tag convention.

Add backend/app/services/ml/tag_name.py with a single normalize()
applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing
_(disambiguator) blocks iteratively, strip wrapping quotes, underscores
to spaces, space after colon, title-case each word's first char,
preserve hyphens/apostrophes/digits, drop entries with no letters).

Wire into SuggestionService.for_image:
- raw Camie key kept for alias_map lookup (alias rows are hand-curated
  against raw keys; don't disturb)
- display_name = normalize(raw); None means drop the candidate
- existing-tag lookup widened to case-insensitive match against BOTH
  raw and normalized forms so legacy underscore-named Tag rows accepted
  before this change still surface as "existing" not "+ new"
2026-06-03 13:00:08 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen f1860866de chore(modal): drop the ?image=N soft-compat from G5.4
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m38s
CI / intcore (push) Successful in 9m31s
Operator confirmed they have no existing ?image=N bookmarks to
preserve, so the soft-compat read on initial mount + router.replace
strip is dead weight. The modal is now purely a Pinia overlay — no
URL involvement on open OR initial mount.

Drops 33 lines plus the now-unused vue-router imports in both
ArtistGalleryTab (entire onMounted gone) and GalleryView (just the
?image=N block; the post_id/tag_id filter handling stays).
2026-06-02 19:24:09 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen b181d779fe fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 8m13s
CI / intcore (push) Successful in 7m6s
2026-06-02 18:49:24 -04:00
bvandeusen 0fbb19dc24 fix(modal): TagPanel kebab — apply the wrapping-span fix that the prior commit missed
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m59s
CI / intapi (push) Failing after 8m56s
CI / intcore (push) Successful in 10m0s
The previous commit (8326e54) updated the CSS but the structural
edit to the v-menu wrapper didn't take. Re-applying so the chip
kebab actually opens.
2026-06-02 18:48:53 -04:00
bvandeusen 8326e5447a fix(modal): kebab menus weren't opening (chip + suggestion rows)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Failing after 8m41s
CI / intcore (push) Successful in 10m0s
Operator-flagged 2026-06-02. Both kebab activators were broken:

- TagPanel chip kebab had `@click.stop` directly on the v-icon
  activator. In Vue 3, an explicit @click on the same element as
  `v-bind="props"` overrides the spread onClick — so Vuetify's
  activator handler never fired. Menu never opened.

- SuggestionItem kebab didn't have @click.stop, but for consistency
  and to make both kebabs follow the same shape, wrap it too.

The fix: each kebab is now wrapped in a `<span @click.stop>`. The
v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu;
the bubbling click then reaches the span where stopPropagation
absorbs it before it can affect a parent (the v-chip's close button
in TagPanel's case).
2026-06-02 18:48:32 -04:00
bvandeusen 1fd594baaf chore(ml): suggestion_threshold default 0.50 → 0.70
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Failing after 8m16s
CI / intcore (push) Successful in 8m43s
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).

Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.

Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
2026-06-02 18:38:12 -04:00
bvandeusen ecac6c4bda fix(audit-g5): centroid version DB-as-truth + modal as overlay
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 24s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 7m57s
CI / intcore (push) Successful in 8m46s
Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4).

G5.1 — Centroid version no longer drifts:

CentroidService now reads MLSettings.embedder_model_version (the DB
row tag_and_embed already writes from) for both the centroid model-
version stamp and the drift-detection comparison. Previously the
centroid sites imported MODEL_VERSION from env, so the version stamped
on centroids could disagree with the version stamped on the embeddings
they were built from. By construction those now match, so list_drifted
won't silently miss the env-vs-DB drift case.

embedder.py keeps MODEL_VERSION as an env-driven constant for the
actual model loader — that's a different concern (which weights are
loaded) from the version-stamp that gets persisted alongside data.

G5.4 — Modal is a Pinia-only overlay:

The previous URL↔modal sync in GalleryView and ArtistGalleryTab
leaked the modal across route changes (RouterLink to /artist/<slug>
left the modal mounted on top of the new route) and re-opened it
on history back/forward with stale ?image=N entries.

Now: openImage() just calls modal.open(id) — no URL push.
GalleryView's dead closeImage helper is deleted. A route.name
watcher in App.vue closes the modal whenever the route changes,
which auto-fixes RouterLink-in-modal and back/forward.

Backward-compat: ?image=N is still honored on initial mount as a
one-shot deep-link opener, then router.replace strips the query so
the URL doesn't re-trigger and no extra history entry is added.
Existing bookmarks / shared URLs keep working; new opens stay
Pinia-only.
2026-06-02 18:28:57 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 9f7261b9c0 fix(audit-g5c): set CURATOR_BOOTSTRAP_NEW_KEY=1 in conftest
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m50s
CI / intapi (push) Successful in 8m33s
CI / intcore (push) Successful in 9m7s
CredentialCrypto's safety check fires on create_app() instantiation
because the test environment has no pre-seeded Fernet key file. Set
the bootstrap env var before any test imports so the auto-create path
is allowed during tests.
2026-06-02 17:55:25 -04:00
bvandeusen f05aaa707b fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Failing after 26s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Failing after 8m10s
CI / intcore (push) Failing after 9m42s
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.

FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
  - warning (yellow) — operator action needed (auth_error)
  - info (blue)      — backend-paced (rate_limited / timeout /
                       network_error / partial / tier_limited)
  - error (red)      — likely terminal without intervention
                       (not_found / access_denied / validation_failed /
                        unsupported_url / http_error / unknown_error)

Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
2026-06-02 17:54:44 -04:00
bvandeusen 4df98171ab fix(audit-g5c): refuse silent Fernet key regeneration on partial restore
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Failing after 25s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Failing after 8m13s
CI / intcore (push) Failing after 8m48s
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.

Two opt-ins now needed for auto-creation:
  1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
  2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)

Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.

Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.

Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
2026-06-02 17:04:27 -04:00
bvandeusen 8d75ade1d5 fix(audit-g5b): race-poisoning in three find-or-create sites
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m29s
Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the
race under concurrent writers / recovery-sweep replays, poisoning the
outer transaction with an unrecoverable IntegrityError and crashing
the calling task. Same shape as the banked 2026-05-26 ImageProvenance
incident (see reference_scalar_one_or_none_duplicates memory).

Mirrors the importer._get_or_create pattern in all three: savepoint
via begin_nested + IntegrityError rollback to that savepoint + re-
SELECT to grab the row the other worker committed first.

- importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store
  is sha-addressed so both workers race to write the same on-disk path
  (shutil.copy2 + rename is idempotent), so no extra cleanup needed.

- TagService.find_or_create: partial uniqueness index on
  (name, kind, COALESCE(fandom_id, -1)). The previous docstring
  claimed "INSERT ... ON CONFLICT" but the implementation was
  SELECT-then-INSERT with no recovery.

- ArtistService.find_or_create: already had IntegrityError handling
  but did session.rollback() (unwinds the WHOLE transaction); now
  uses begin_nested + sp.rollback() so the surrounding request's
  progress isn't lost.
2026-06-02 17:02:12 -04:00
bvandeusen 75c63e1511 fix(audit-g5a): ruff isort — platforms after patreon_resolver
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 16:55:56 -04:00
bvandeusen 98673d4dca fix(audit-g5a): small architectural cleanups bundle
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m23s
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.

- download_service: replace hardcoded ('discord','pixiv') tuple with
  auth_type_for(platform) == 'token'. A 7th token-platform now picks
  up the right credential path without touching this site.

- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
  merge so the target's centroid reflects its new image set
  immediately. Daily list_drifted catches it within 24h, but eager
  recompute closes the suggestion-quality dip in the meantime.

- backfill_thumbnails added to beat_schedule (daily). The task
  docstring claimed periodic Beat but the entry was never registered,
  so the library got no self-healing thumbnail repair; only the
  manual admin-UI button fired it.

- modal.createAndAdd pushes a kind='fandom' tag into
  tagsStore.fandomCache so FandomPicker sees the new fandom on next
  open. Was: cache-gated load (length===0) skipped refetch, new
  fandom invisible until full page reload.

- cleanup cluster:
  - Drop .webp from cleanup_service.unlink — thumbnailer only writes
    .jpg/.png; the third tuple member was dead code.
  - Drop effective_date from /api/gallery/scroll response — no FE
    consumer reads it. Service still computes the attribute for
    timeline ordering; this just trims the JSON.
  - Rename store.recentMinute → store.recentRuns across the
    systemActivity store + three consumers (SystemActivitySummary,
    QueuesTable, SystemActivityTab). The data is the last 200 runs
    (not actually "last minute"), so the name lied.

NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
2026-06-02 16:46:46 -04:00
41 changed files with 679 additions and 163 deletions
@@ -0,0 +1,41 @@
"""source.error_type: surface ErrorType taxonomy in FailingSourcesCard
Revision ID: 0032
Revises: 0031
Create Date: 2026-06-02
Audit 2026-06-02: the backend computes 13 ErrorType categories (auth_error,
rate_limited, not_found, access_denied, validation_failed, etc.) and
stamps each one on DownloadEvent.metadata, but the Source row only carried
the free-text last_error. Operators couldn't bulk-triage failing sources
("all auth_error → rotate cookies, all rate_limited → just wait") without
opening Logs per row.
This column receives the last error_type from _update_source_health
and gets cleared on a successful run. Nullable + indexed so the failing-
sources rollup can filter/group cheaply.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0032"
down_revision: Union[str, None] = "0031"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column("error_type", sa.String(length=32), nullable=True),
)
op.create_index(
"ix_source_error_type", "source", ["error_type"],
)
def downgrade() -> None:
op.drop_index("ix_source_error_type", table_name="source")
op.drop_column("source", "error_type")
@@ -0,0 +1,48 @@
"""suggestion_threshold default 0.50 → 0.70
Revision ID: 0033
Revises: 0032
Create Date: 2026-06-02
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) is
too noisy in practice; raise to 0.70 for both suggestion categories.
Only conditionally updates singletons whose current value is still the
2026-06-01 default (0.50). Operators who deliberately tuned their row
to some other value (0.55, 0.65, 0.80, etc. via the Settings UI) keep
their pick — the migration only catches the unchanged-default case.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0033"
down_revision: Union[str, None] = "0032"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.70 "
"WHERE id = 1 AND suggestion_threshold_character = 0.50"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.70 "
"WHERE id = 1 AND suggestion_threshold_general = 0.50"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.50 "
"WHERE id = 1 AND suggestion_threshold_character = 0.70"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.70"
)
-1
View File
@@ -43,7 +43,6 @@ async def scroll():
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+6
View File
@@ -245,6 +245,12 @@ async def merge_tag(source_id: int):
from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=result.target_id)
# Tag merge invalidates the target's centroid (the merged-in source
# tag's images now contribute to it). Daily list_drifted catches it
# within 24h, but eager recompute closes the suggestion-quality dip
# in the meantime. Audit 2026-06-02.
from ..tasks.ml import recompute_centroid
recompute_centroid.delay(result.target_id)
return jsonify(
{
"target": {
+10
View File
@@ -130,6 +130,16 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.prune_import_batches",
"schedule": 86400.0,
},
# Audit 2026-06-02 — backfill_thumbnails's docstring claimed
# "periodic Beat" but the entry was never registered, so the
# library got no self-healing thumbnail repair; only the
# manual admin-UI button fired it. Daily cadence is gentle
# (the task is idempotent and only enqueues regen for rows
# whose stored thumbnails are missing or corrupt).
"backfill-thumbnails-daily": {
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
"schedule": 86400.0,
},
},
timezone="UTC",
)
+6 -5
View File
@@ -16,13 +16,14 @@ class MLSettings(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True)
suggestion_threshold_character: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
# Default lowered 0.95 → 0.50 on 2026-06-01 — operator-flagged that
# 0.95 hid most general suggestions. Operator-tunable via Settings →
# ML if too noisy.
# Default raised 0.50 → 0.70 on 2026-06-02 — operator-flagged 0.50
# surfaced too many low-confidence picks; 0.70 keeps the rail
# signal-rich while still surfacing more than the original 0.95
# which hid almost everything. Operator-tunable via Settings → ML.
suggestion_threshold_general: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
centroid_similarity_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.55
+5
View File
@@ -26,6 +26,11 @@ class Source(Base):
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
# alembic 0032: last ErrorType category (auth_error, rate_limited,
# not_found, ...). Lets FailingSourcesCard surface the taxonomy as
# a colored chip so operators can bulk-triage by error class. Set
# by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+15 -10
View File
@@ -208,27 +208,32 @@ class ArtistService:
)
async def find_or_create(self, name: str) -> tuple[Artist, bool]:
"""Return (artist, created). Slug-keyed; idempotent under races."""
"""Return (artist, created). Slug-keyed; idempotent under races.
Audit 2026-06-02: switched from session.rollback() to a
begin_nested savepoint + IntegrityError recovery so a lost
race doesn't unwind the calling request's surrounding work.
Mirrors importer._get_or_create.
"""
cleaned = (name or "").strip()
if not cleaned:
raise ValueError("artist name must not be empty")
slug = slugify(cleaned)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
select_existing = select(Artist).where(Artist.slug == slug)
existing = (await self.session.execute(select_existing)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
sp = await self.session.begin_nested()
try:
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
await self.session.flush()
await sp.commit()
except IntegrityError:
await self.session.rollback()
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
await sp.rollback()
existing = (await self.session.execute(select_existing)).scalar_one()
return existing, False
await self.session.commit()
return artist, True
+6 -2
View File
@@ -187,10 +187,14 @@ def unlink_image_files(
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
# Convention thumbs dir — try both extensions thumbnailer writes
# (.jpg for opaque, .png for alpha). `.webp` used to be in this
# tuple but the thumbnailer never writes it (operator-flagged in
# the 2026-06-02 audit) — keep the tuple aligned with what
# actually lands on disk.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
for ext in ("jpg", "png"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
+53 -11
View File
@@ -1,40 +1,82 @@
"""Fernet-based encryption for credential blobs.
The key is a single 32-byte value (urlsafe-base64-encoded; what
Fernet.generate_key produces) stored at a fixed path inside the
images/data root. Created on first boot if absent; mode 0600. No KDF
needed — the file contents are already maximum-entropy random bytes.
Fernet.generate_key produces) stored at /images/secrets/credential_key.b64
(mode 0600, parent dir 0700). The 2026-06-02 audit caught a silent
key-regeneration path: on a partial disaster restore where the DB was
restored but the secrets dir was lost, the old `_load_or_create_key`
would mint a fresh key with no log, producing a working-looking system
where every authenticated download failed AUTH_ERROR until the operator
re-uploaded every credential by hand. Now the constructor refuses to
auto-generate unless either:
Operator backup procedure must include this file alongside the rest
of /images/ — losing it makes existing encrypted_blob rows
undecryptable (recovery = delete the rows and re-upload).
* the caller explicitly passes `bootstrap_ok=True` (tests, scripts), or
* the env var `CURATOR_BOOTSTRAP_NEW_KEY=1` is set (operator opt-in
during first-time setup).
Otherwise it raises `MissingCredentialKey` so the app fails fast at
startup and the operator can restore the key file from backup.
Operator backup procedure must include /images/secrets/ alongside the
rest of /images/ — losing the key file makes existing encrypted_blob
rows undecryptable (recovery = delete the rows and re-upload).
"""
import logging
import os
from pathlib import Path
from cryptography.fernet import Fernet, InvalidToken
log = logging.getLogger(__name__)
_BOOTSTRAP_ENV_VAR = "CURATOR_BOOTSTRAP_NEW_KEY"
class InvalidCredentialBlob(Exception):
"""Raised when decryption fails (wrong key, tampered blob, …)."""
class MissingCredentialKey(Exception):
"""The Fernet key file is missing AND the caller hasn't opted in to
generating a new one. Audit 2026-06-02: prevents silent key
regeneration on partial DB-restored / secrets-lost deployments.
Set CURATOR_BOOTSTRAP_NEW_KEY=1 for first-time setup, or restore the
key file from backup."""
class CredentialCrypto:
"""Fernet encrypt/decrypt with an on-disk key file.
Instantiate with a path; the file is created on first access and
reused thereafter. Tests pass a tmp_path; production calls with
Instantiate with a path; the file is loaded if present, or created
if absent AND the caller has opted in (bootstrap_ok=True or
CURATOR_BOOTSTRAP_NEW_KEY=1 env var). Production sites:
`IMAGES_ROOT / "secrets" / "credential_key.b64"`.
"""
def __init__(self, key_path: Path):
def __init__(self, key_path: Path, *, bootstrap_ok: bool | None = None):
self._key_path = Path(key_path)
self._fernet = Fernet(self._load_or_create_key())
if bootstrap_ok is None:
bootstrap_ok = os.environ.get(_BOOTSTRAP_ENV_VAR) == "1"
self._fernet = Fernet(self._load_or_create_key(bootstrap_ok))
def _load_or_create_key(self) -> bytes:
def _load_or_create_key(self, bootstrap_ok: bool) -> bytes:
if self._key_path.exists():
return self._key_path.read_bytes()
if not bootstrap_ok:
raise MissingCredentialKey(
f"Fernet key file not found at {self._key_path}. "
f"For first-time setup, set {_BOOTSTRAP_ENV_VAR}=1. "
f"If this is a restored instance, restore the key file "
f"from backup — generating a new one would make every "
f"existing Credential row undecryptable."
)
log.warning(
"Generating NEW Fernet credential key at %s. Any existing "
"encrypted_blob rows in the DB will be undecryptable — "
"re-upload each credential after this completes.",
self._key_path,
)
parent = self._key_path.parent
parent.mkdir(parents=True, exist_ok=True)
os.chmod(parent, 0o700)
+13 -1
View File
@@ -35,6 +35,7 @@ from .gallery_dl import (
)
from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__)
@@ -194,7 +195,12 @@ class DownloadService:
event_id = ev.id
artist = source.artist
if source.platform in ("discord", "pixiv"):
# Drive cookies-vs-token selection from the platform registry's
# auth_type so a new 7th token-platform automatically picks the
# right credential path. The hardcoded tuple here used to drift
# out of sync with credential_service's auth_type_for(). Audit
# 2026-06-02.
if auth_type_for(source.platform) == "token":
cookies_path = None
auth_token = await self.cred_service.get_token(source.platform)
else:
@@ -428,9 +434,15 @@ class DownloadService:
if status == "ok":
source.consecutive_failures = 0
source.last_error = None
# alembic 0032 — clear the failure-class chip on success.
source.error_type = None
elif status == "error":
source.consecutive_failures = (source.consecutive_failures or 0) + 1
source.last_error = error_message
# alembic 0032 — stamp the failure-class so FailingSourcesCard
# can render a colored chip and operators can bulk-triage
# by error class without opening Logs per row.
source.error_type = error_type
if error_type == "rate_limited":
await set_platform_cooldown(self.async_session, source.platform)
elif status == "skipped":
+18 -4
View File
@@ -374,10 +374,19 @@ class Importer:
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
sha = _sha256_of(source)
existing = self.session.execute(
select(PostAttachment).where(PostAttachment.sha256 == sha)
).scalar_one_or_none()
if existing is None:
select_existing = select(PostAttachment).where(PostAttachment.sha256 == sha)
existing = self.session.execute(select_existing).scalar_one_or_none()
if existing is not None:
self.session.commit()
return ImportResult(status="attached")
# Savepoint + IntegrityError recovery — PostAttachment.sha256 is
# UNIQUE, so two workers can both pass the SELECT and only the
# second INSERT fails. Without savepoint, the outer transaction
# poisons and the calling task crashes. attachments.store is
# sha-addressed so both workers race to write the same target
# path; shutil.copy2 + rename is idempotent. Audit 2026-06-02.
sp = self.session.begin_nested()
try:
stored = self.attachments.store(source, sha)
self.session.add(PostAttachment(
post_id=post.id if post else None,
@@ -390,6 +399,11 @@ class Importer:
size_bytes=source.stat().st_size,
))
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
# Lost the race — the other worker's row is canonical.
self.session.execute(select_existing).scalar_one()
self.session.commit()
return ImportResult(status="attached")
+20 -4
View File
@@ -19,7 +19,6 @@ from ...models import (
TagReferenceEmbedding,
)
from ...models.tag import image_tag
from .embedder import MODEL_VERSION as SIGLIP_VERSION
ELIGIBLE_KINDS = {
TagKind.character,
@@ -46,6 +45,21 @@ class CentroidService:
)
).scalar_one()
async def _model_version(self) -> str:
"""Audit 2026-06-02: SigLIP model-version stamp comes from the
DB row, not the env constant. tag_and_embed (tasks/ml.py:110)
already reads from MLSettings.embedder_model_version, so by
sourcing centroid stamps + drift checks from the same row, we
eliminate the silent-drift case the audit flagged. env
SIGLIP_MODEL_VERSION still drives which model embedder.py
loads at runtime; the version stamp is purely the operator-
controlled identifier."""
return (
await self.session.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
)
).scalar_one()
async def recompute_for_tag(self, tag_id: int) -> bool:
"""Recompute one tag's centroid. Returns True if a centroid was
written, False if skipped (ineligible kind or too few members)."""
@@ -69,19 +83,20 @@ class CentroidService:
return False
centroid = np.mean(np.stack(embeddings), axis=0).astype(np.float32)
model_version = await self._model_version()
stmt = insert(TagReferenceEmbedding).values(
tag_id=tag_id,
embedding=centroid.tolist(),
reference_count=len(embeddings),
model_version=SIGLIP_VERSION,
model_version=model_version,
)
stmt = stmt.on_conflict_do_update(
index_elements=["tag_id"],
set_={
"embedding": centroid.tolist(),
"reference_count": len(embeddings),
"model_version": SIGLIP_VERSION,
"model_version": model_version,
"updated_at": func.now(),
},
)
@@ -92,6 +107,7 @@ class CentroidService:
"""Tag ids whose centroid is stale: member count != reference_count,
OR no centroid row, OR centroid built on a different SigLIP version.
Only considers eligible-kind tags with embeddings present."""
current_model_version = await self._model_version()
member_counts = (
select(
image_tag.c.tag_id.label("tag_id"),
@@ -116,7 +132,7 @@ class CentroidService:
TagReferenceEmbedding.reference_count
!= member_counts.c.members
)
| (TagReferenceEmbedding.model_version != SIGLIP_VERSION)
| (TagReferenceEmbedding.model_version != current_model_version)
)
)
return list((await self.session.execute(stmt)).scalars().all())
+27 -9
View File
@@ -4,7 +4,7 @@ threshold-filtered, category-grouped, ranked suggestions for one image.
from dataclasses import dataclass, field
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import (
@@ -16,6 +16,7 @@ from ...models import (
from ...models.tag import image_tag
from .aliases import AliasService
from .centroids import CentroidService
from .tag_name import normalize as normalize_tag_name
from .tagger import SURFACED_CATEGORIES
@@ -84,7 +85,12 @@ class SuggestionService:
)
# --- Camie predictions ---
candidates: list[tuple[str, str, float]] = []
# candidates carry (raw_name, display_name, category, confidence).
# raw_name = the booru-formatted vocab key, kept for alias_map
# lookup since alias rows are hand-curated against raw keys.
# display_name = normalize_tag_name(raw_name) — what the operator
# sees AND what gets written to tag.name on Accept.
candidates: list[tuple[str, str, str, float]] = []
for name, p in predictions.items():
category = p.get("category", "general")
if category not in SURFACED_CATEGORIES:
@@ -92,10 +98,14 @@ class SuggestionService:
conf = float(p.get("confidence", 0.0))
if conf < self._threshold_for(settings, category):
continue
candidates.append((name, category, conf))
display = normalize_tag_name(name)
if display is None:
# emoticon / pure-punctuation vocab entry — drop entirely
continue
candidates.append((name, display, category, conf))
alias_map = await self.aliases.resolve_many(
[(n, c) for n, c, _ in candidates]
[(raw, c) for raw, _disp, c, _conf in candidates]
)
merged: dict[object, Suggestion] = {}
@@ -116,8 +126,8 @@ class SuggestionService:
creates_new_tag=existing.creates_new_tag,
)
for name, category, conf in candidates:
canonical = alias_map.get((name, category))
for raw, display, category, conf in candidates:
canonical = alias_map.get((raw, category))
if canonical is not None:
if canonical.id in applied or canonical.id in rejected:
continue
@@ -133,9 +143,17 @@ class SuggestionService:
),
)
else:
# Case-insensitive match on BOTH the raw camie key AND
# the normalized form — covers legacy underscore-named
# Tag rows accepted before normalization shipped, AND
# any tag the operator created with the human form.
existing_tag = (
await self.session.execute(
select(Tag).where(Tag.name == name)
select(Tag).where(
func.lower(Tag.name).in_(
[raw.lower(), display.lower()]
)
)
)
).scalars().first()
if existing_tag is not None:
@@ -157,10 +175,10 @@ class SuggestionService:
)
else:
_merge(
f"raw:{name}:{category}",
f"raw:{display}:{category}",
Suggestion(
canonical_tag_id=None,
display_name=name,
display_name=display,
category=category,
score=conf,
source="tagger",
+62
View File
@@ -0,0 +1,62 @@
"""Camie vocabulary -> human-readable tag-name normalization.
Camie v2's ~57k tag vocabulary is booru-derived and arrives as raw
strings like `uchiha_sasuke_(naruto)`, `#unicus_(idolmaster)`,
`1000-nen_ikiteru_(vocaloid)`, or `:/`. We want the operator to see
"Uchiha Sasuke", "Unicus", "1000-Nen Ikiteru", or to never see the
emoticon at all — and we want the same clean string to be what lands
in `tag.name` when the suggestion is accepted, so Accept matches the
existing-tag convention (`tag_service.find_or_create`).
Rules (operator-approved 2026-06-03):
1. Strip leading junk chars (#, ., +, ;, ~, _, whitespace)
2. Drop trailing `_(disambiguator)` block(s), iteratively
3. Strip wrapping single/double quotes (after disambig removal so
`"foo_em_up"_(series)` -> `"foo_em_up"` -> `foo_em_up`)
4. Replace remaining `_` with space; collapse runs of whitespace
5. Add a space after any `:` (namespace:tag -> namespace: tag)
6. Preserve hyphens (booru hyphens often carry meaning)
7. Title-case each space-separated word (first character only —
apostrophes, digits, hyphens stay)
8. If no letters AND no digits remain, return None (drops emoticons
like `:/` or `^_^`; preserves bare digit tags like `2005`)
9. No surname/givenname swap — no reliable signal in the vocab
"""
import re
_LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
_MULTISPACE = re.compile(r"\s+")
_COLON_NOSPACE = re.compile(r":(?=\S)")
_HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
def _strip_wrapping_quotes(s: str) -> str:
if len(s) >= 2 and s[0] == s[-1] and s[0] in ('"', "'"):
return s[1:-1]
return s
def _title_word(w: str) -> str:
return w[:1].upper() + w[1:] if w else w
def normalize(raw: str) -> str | None:
"""Return the human-readable form of a raw Camie tag, or None if the
string is junk (emoticon, empty after stripping)."""
if not raw:
return None
s = _LEADING_JUNK.sub("", raw)
while True:
new = _TRAILING_DISAMBIG.sub("", s)
if new == s:
break
s = new
s = _strip_wrapping_quotes(s)
s = s.replace("_", " ")
s = _COLON_NOSPACE.sub(": ", s)
s = _MULTISPACE.sub(" ", s).strip()
if not s or not _HAS_ALPHANUMERIC.search(s):
return None
return " ".join(_title_word(w) for w in s.split(" "))
+3
View File
@@ -59,6 +59,7 @@ class SourceRecord:
config_overrides: dict | None
last_checked_at: str | None
last_error: str | None
error_type: str | None
check_interval_override: int | None
consecutive_failures: int
next_check_at: str | None
@@ -76,6 +77,7 @@ class SourceRecord:
"config_overrides": self.config_overrides,
"last_checked_at": self.last_checked_at,
"last_error": self.last_error,
"error_type": self.error_type,
"check_interval_override": self.check_interval_override,
"consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at,
@@ -144,6 +146,7 @@ class SourceService:
config_overrides=source.config_overrides,
last_checked_at=source.last_checked_at.isoformat() if source.last_checked_at else None,
last_error=source.last_error,
error_type=source.error_type,
check_interval_override=source.check_interval_override,
consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None,
+17 -7
View File
@@ -5,6 +5,7 @@ from dataclasses import dataclass
from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Tag, TagKind, image_tag
@@ -86,9 +87,12 @@ class TagService:
f"fandom_id {fandom_id} does not reference a fandom tag"
)
# Upsert via INSERT ... ON CONFLICT DO NOTHING. We can't use the
# uniqueness index name directly (it's a partial coalesce-based
# expression), so we re-select after insert.
# Audit 2026-06-02: race-safe upsert via savepoint +
# IntegrityError recovery. The partial uniqueness index on
# (name, kind, COALESCE(fandom_id, -1)) catches concurrent
# inserts; without the savepoint the outer transaction would
# poison and the calling request crashes. Mirrors
# importer._get_or_create.
stmt = (
select(Tag)
.where(Tag.name == name)
@@ -101,10 +105,16 @@ class TagService:
if existing:
return existing
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
return new_tag
sp = await self.session.begin_nested()
try:
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
await sp.commit()
return new_tag
except IntegrityError:
await sp.rollback()
return (await self.session.execute(stmt)).scalar_one()
async def autocomplete(
self,
+18 -2
View File
@@ -194,9 +194,20 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
};
}
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length };
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -229,8 +240,13 @@ browser.runtime.onMessage.addListener(async (msg) => {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length };
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
} catch (e) {
results[key] = { error: e.message };
}
+35
View File
@@ -76,3 +76,38 @@ async function getCookieCount(platformKey) {
return 0;
}
}
/**
* Verify cookies are live by hitting an authenticated endpoint with the
* browser's current cookie jar. Returns:
* { ok: true, status } — verified
* { ok: false, status, reason } — endpoint said we're not logged in
* { ok: null, reason } — no verify config for this platform; caller
* should treat as "verify not available,
* proceed with upload"
*
* Implementation note: extensions with `host_permissions` for the target
* domain get the user's cookies auto-attached to fetch() — same set
* gallery-dl will later use on the backend.
*/
async function verifyCookiesForPlatform(platformKey) {
const platform = PLATFORMS[platformKey];
if (!platform) return { ok: false, reason: `Unknown platform: ${platformKey}` };
if (!platform.verify) return { ok: null, reason: 'verify-not-configured' };
const { url, method, okStatuses } = platform.verify;
let resp;
try {
resp = await fetch(url, { method, credentials: 'include', cache: 'no-store' });
} catch (e) {
return { ok: false, reason: `Verify request failed: ${e.message}` };
}
if (okStatuses.includes(resp.status)) {
return { ok: true, status: resp.status };
}
return {
ok: false,
status: resp.status,
reason: `${url} returned HTTP ${resp.status} — session looks stale or logged out`,
};
}
+21
View File
@@ -13,6 +13,13 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FF424D',
urlPattern: /^https?:\/\/(www\.)?patreon\.com/,
// Patreon's `/api/current_user` returns 200 + the logged-in user
// when authenticated, 401 otherwise. Cheapest definitive check.
verify: {
url: 'https://www.patreon.com/api/current_user',
method: 'GET',
okStatuses: [200],
},
},
subscribestar: {
name: 'SubscribeStar',
@@ -26,6 +33,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FFD700',
urlPattern: /^https?:\/\/(www\.)?subscribestar\.(com|adult)/,
// No known stable auth-required endpoint that returns a definitive
// status code; skipping verify so we don't false-positive-fail
// good cookies. Operator can add later if a clean endpoint surfaces.
},
hentaifoundry: {
name: 'Hentai Foundry',
@@ -33,6 +43,14 @@ const PLATFORMS = {
authType: 'cookies',
color: '#9C27B0',
urlPattern: /^https?:\/\/(www\.)?hentai-foundry\.com/,
// Mirror gallery-dl's _init_site_filters: HEAD on `?enterAgree=1`.
// Logged in → 200, logged out → 401. Catches the exact failure mode
// the backend extractor would hit later.
verify: {
url: 'https://www.hentai-foundry.com/?enterAgree=1',
method: 'HEAD',
okStatuses: [200],
},
},
discord: {
name: 'Discord',
@@ -56,6 +74,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#05CC47',
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
// DA's logged-in-only endpoints sit behind their internal _napi
// namespace which shifts; skipping verify until a stable check
// surfaces. Same posture as SubscribeStar.
},
};
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.6",
"version": "1.0.7",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "fabledcurator-extension",
"version": "1.0.6",
"version": "1.0.7",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
+2 -1
View File
@@ -132,8 +132,9 @@ async function exportPlatformCookies(key, card) {
if (r.error) showError(r.error);
else {
const n = r.cookieCount ?? null;
const verifiedSuffix = r.verified ? ' (verified ✓)' : '';
const msg = n !== null
? `${PLATFORMS[key].name}: ${n} cookies exported`
? `${PLATFORMS[key].name}: ${n} cookies exported${verifiedSuffix}`
: `${PLATFORMS[key].name}: token exported`;
showSuccess(msg);
await loadPlatformStatus();
+14 -1
View File
@@ -9,7 +9,9 @@
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { onMounted, ref, watch } from 'vue'
import { useRoute } from 'vue-router'
import AppShell from './components/AppShell.vue'
import AppSnackbar from './components/AppSnackbar.vue'
import ImageViewer from './components/modal/ImageViewer.vue'
@@ -17,9 +19,20 @@ import { useModalStore } from './stores/modal.js'
const modal = useModalStore()
const snackbar = ref(null)
const route = useRoute()
onMounted(() => {
// Expose snackbar via a simple global so stores can call it without props.
window.__fcToast = (opts) => snackbar.value?.open(opts)
})
// Audit 2026-06-02: the modal is an overlay, not a page. When the
// route changes (RouterLink inside the modal, history back/forward,
// programmatic push from any view), close the modal so it doesn't
// hover over a different route. Watching route.name (not the path)
// keeps within-route nav like /artist/foo → /artist/bar from
// dismissing the modal mid-browse.
watch(() => route.name, () => {
if (modal.isOpen) modal.close()
})
</script>
@@ -11,9 +11,6 @@
</template>
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../../stores/artist.js'
import { useModalStore } from '../../stores/modal.js'
import MasonryGrid from '../discovery/MasonryGrid.vue'
@@ -24,22 +21,9 @@ const props = defineProps({
const store = useArtistStore()
const modal = useModalStore()
const route = useRoute()
const router = useRouter()
onMounted(() => {
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage (id) {
router.push({ query: { ...route.query, image: id } })
modal.open(id)
}
</script>
@@ -19,25 +19,34 @@
>
Accept
</v-btn>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
<!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
Wrapping in a <span @click.stop> matches the TagPanel chip
fix — even though there's no parent click capture here today,
the wrap is harmless and keeps both kebabs on the same
pattern. Click bubbles from the v-btn opens menu via
activator props bubble continues to span stopPropagation
halts it. -->
<span class="fc-suggestion__menu-wrap" @click.stop>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</div>
</template>
@@ -90,6 +99,11 @@ const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
.fc-suggestion__accept :deep(.v-btn__content) {
font-size: 12px; letter-spacing: 0.02em;
}
.fc-suggestion__menu-wrap {
flex: 0 0 auto;
display: inline-flex;
align-items: center;
}
.fc-suggestion__menu {
flex: 0 0 auto;
}
+22 -13
View File
@@ -10,19 +10,27 @@
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
<!-- Operator-flagged 2026-06-02: the previous activator had
`@click.stop` directly on the v-icon, which silently
overrode Vuetify's onClick from `v-bind="mp"` — the menu
never opened. Now the v-icon receives the activator
onClick cleanly, and the wrapping span absorbs the
bubbled click so the chip's close button isn't tripped. -->
<span class="kebab-wrap" @click.stop>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical"
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename…</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</v-chip>
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
</div>
@@ -113,4 +121,5 @@ async function onRenamed() {
margin-bottom: 12px;
}
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
.kebab-wrap { display: inline-flex; align-items: center; }
</style>
@@ -35,7 +35,7 @@ import { computed } from 'vue'
const props = defineProps({
queues: { type: Object, default: null }, // store.queues
workers: { type: Object, default: null }, // store.workers
recentMinute: { type: Array, default: () => [] }, // store.recentMinute
recentRuns: { type: Array, default: () => [] }, // store.recentRuns
compact: { type: Boolean, default: false },
})
@@ -80,7 +80,7 @@ function activeCount(name) {
const recentByQueue = computed(() => {
const out = {}
for (const r of props.recentMinute) {
for (const r of props.recentRuns) {
if (!out[r.queue]) out[r.queue] = { ok: 0, err: 0 }
if (r.status === 'ok') out[r.queue].ok++
else if (r.status === 'error' || r.status === 'timeout') out[r.queue].err++
@@ -24,7 +24,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
compact
/>
</v-card-text>
@@ -47,7 +47,7 @@ function pollOnce() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
onMounted(() => {
@@ -14,7 +14,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
/>
</v-card-text>
</v-card>
@@ -202,7 +202,7 @@ function pollQueues() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
function pollFailures() {
if (document.hidden) return
@@ -25,6 +25,15 @@
<v-chip size="x-small" color="error" variant="flat" label class="fc-fail__count">
{{ s.consecutive_failures }}× failed
</v-chip>
<v-chip
v-if="s.error_type"
size="x-small" variant="outlined" label
:color="errorTypeColor(s.error_type)"
class="fc-fail__class"
:title="errorTypeHint(s.error_type)"
>
{{ s.error_type }}
</v-chip>
<span class="fc-fail__err" :title="s.last_error || ''">
{{ s.last_error || 'no error message recorded' }}
</span>
@@ -66,6 +75,42 @@ const open = ref(true)
// Per-row loading flag so the spinner lives on the row whose Logs
// button was clicked, not on every row.
const logLoadingIds = ref(new Set())
// Audit 2026-06-02: surface the ErrorType taxonomy as a colored chip
// next to the consecutive-failures count so operators can bulk-triage
// by error class. Color reflects "what to do next":
// warning (yellow) — auth/cookie issue: operator should rotate
// info (blue) — backend-paced (cooldown / rate limit / timeout)
// error (red) — likely terminal without operator intervention
const ERROR_TYPE_COLOR = {
auth_error: 'warning',
rate_limited: 'info',
timeout: 'info',
network_error: 'info',
not_found: 'error',
access_denied: 'error',
validation_failed: 'error',
unsupported_url: 'error',
http_error: 'error',
unknown_error: 'error',
partial: 'info',
tier_limited: 'info',
no_new_content: 'info',
}
const ERROR_TYPE_HINT = {
auth_error: 'Cookies likely expired — re-upload in Credentials.',
rate_limited: 'Platform-wide cooldown active. Will retry after it expires.',
timeout: 'Subprocess exceeded its time budget. Often retries cleanly.',
network_error: 'Transient network issue. Will retry on next tick.',
not_found: 'URL 404 — creator may have renamed or deleted.',
access_denied: 'Subscription tier may not grant this content.',
validation_failed: 'Downloaded files were quarantined by the validator.',
http_error: 'Generic HTTP error — see Logs.',
unsupported_url: 'gallery-dl does not support this URL pattern.',
unknown_error: 'Could not classify — see Logs.',
}
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
async function onViewLogs(s) {
if (logLoadingIds.value.has(s.id)) return
logLoadingIds.value = new Set(logLoadingIds.value).add(s.id)
@@ -115,6 +160,7 @@ async function onViewLogs(s) {
}
.fc-fail__artist { font-weight: 600; white-space: nowrap; }
.fc-fail__count { flex: 0 0 auto; }
.fc-fail__class { flex: 0 0 auto; }
.fc-fail__err {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.8rem;
+13
View File
@@ -155,6 +155,19 @@ export const useModalStore = defineStore('modal', () => {
const imageId = currentImageId.value
if (!imageId) return
const tag = await api.post('/api/tags', { body: { name, kind, fandom_id } })
// Audit 2026-06-02: a kind='fandom' created here used to be
// invisible to FandomPicker until a full page reload — its load
// gates on fandomCache.length, so a non-empty cache skips the
// refetch and the new fandom never appears. Push it into the
// cache directly so the next open sees it.
if (kind === 'fandom') {
const { useTagStore } = await import('./tags.js')
const tagStore = useTagStore()
tagStore.fandomCache.push({
id: tag.id, name: tag.name, kind: 'fandom',
fandom_id: null, fandom_name: null, image_count: 0,
})
}
if (currentImageId.value !== imageId) return // navigated away
await addExistingTag(tag.id)
}
+5 -5
View File
@@ -9,7 +9,7 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
// Live polled state.
const queues = ref(null) // { queues: {name: depth|null}, fetched_at }
const workers = ref(null) // { workers: {hostname: {...}}, fetched_at }
const recentMinute = ref([]) // last-60s rows (for Overview summary)
const recentRuns = ref([]) // last-60s rows (for Overview summary)
const failures = ref(null) // { recent, count_by_type, since }
// Paginated runs (Activity tab "All recent activity" pane).
@@ -45,14 +45,14 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
}
async function loadRecentMinute() {
async function loadRecentRuns() {
// Used by the Overview summary card: pull last 60s of runs to compute
// per-queue ok/err counts. One call covers all queues; UI groups.
try {
const body = await api.get('/api/system/activity/runs', {
params: { limit: 200 },
})
recentMinute.value = body.runs || []
recentRuns.value = body.runs || []
} catch (e) {
lastError.value = e.message
}
@@ -106,10 +106,10 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
return {
queues, workers, recentMinute, failures, summary,
queues, workers, recentRuns, failures, summary,
runs, runsCursor, runsHasMore, runsFilter,
loading, lastError,
loadQueues, loadWorkers, loadRecentMinute,
loadQueues, loadWorkers, loadRecentRuns,
loadRuns, loadFailures, loadSummary, setFilter,
}
})
+2 -17
View File
@@ -24,7 +24,7 @@
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useRoute } from 'vue-router'
import { useGalleryStore } from '../stores/gallery.js'
import { useModalStore } from '../stores/modal.js'
import GalleryGrid from '../components/gallery/GalleryGrid.vue'
@@ -37,7 +37,6 @@ import { useGallerySelectionStore } from '../stores/gallerySelection.js'
const store = useGalleryStore()
const modal = useModalStore()
const sel = useGallerySelectionStore()
const router = useRouter()
const route = useRoute()
onMounted(async () => {
@@ -47,9 +46,6 @@ onMounted(async () => {
else if (!isNaN(tagId)) store.setTagFilter(tagId)
await store.loadInitial()
await store.loadTimeline()
// Open modal if URL has ?image=N
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.tag_id, (q) => {
@@ -64,19 +60,8 @@ watch(() => route.query.post_id, (q) => {
store.setPostFilter(isNaN(postId) ? null : postId)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage(id) {
router.push({ query: { ...route.query, image: id } })
}
function closeImage() {
const q = { ...route.query }
delete q.image
router.push({ query: q })
modal.open(id)
}
</script>
+14 -7
View File
@@ -8,18 +8,25 @@ migration code paths.
import os
import pytest
import pytest_asyncio
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import (
# Audit 2026-06-02: CredentialCrypto now refuses to auto-generate a
# Fernet key without explicit opt-in (production safety against silent
# key regeneration on partial restore). The test environment never has
# a pre-seeded key file, so set the bootstrap flag here before any
# create_app() / CredentialCrypto() import path fires.
os.environ.setdefault("CURATOR_BOOTSTRAP_NEW_KEY", "1")
import pytest # noqa: E402
import pytest_asyncio # noqa: E402
from sqlalchemy import create_engine # noqa: E402
from sqlalchemy.ext.asyncio import ( # noqa: E402
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import sessionmaker # noqa: E402
from backend.app import create_app
from backend.app.models import Base
from backend.app import create_app # noqa: E402
from backend.app.models import Base # noqa: E402
def _async_database_url() -> str:
+4 -3
View File
@@ -19,9 +19,10 @@ async def test_get_and_patch_settings(client):
resp = await client.get("/api/ml/settings")
assert resp.status_code == 200
body = await resp.get_json()
# Default lowered 0.95 → 0.50 on 2026-06-01 (alembic 0029) — 0.95
# hid most general suggestions in the view modal.
assert body["suggestion_threshold_general"] == pytest.approx(0.50)
# Default raised 0.50 → 0.70 on 2026-06-02 (alembic 0033) — 0.50
# was too noisy in practice. The 0.70 default keeps the rail
# signal-rich without hiding everything like the original 0.95.
assert body["suggestion_threshold_general"] == pytest.approx(0.70)
# Retired threshold columns must not appear in the payload.
assert "suggestion_threshold_artist" not in body
assert "suggestion_threshold_copyright" not in body
+25 -6
View File
@@ -14,7 +14,7 @@ pytestmark = pytest.mark.integration
def test_load_or_create_writes_key_with_mode_0600(tmp_path):
key_path = tmp_path / "secrets" / "credential_key.b64"
CredentialCrypto(key_path)
CredentialCrypto(key_path, bootstrap_ok=True)
assert key_path.exists()
mode = stat.S_IMODE(os.stat(key_path).st_mode)
assert mode == 0o600
@@ -25,9 +25,9 @@ def test_load_or_create_writes_key_with_mode_0600(tmp_path):
def test_load_existing_key_is_idempotent(tmp_path):
key_path = tmp_path / "credential_key.b64"
crypto1 = CredentialCrypto(key_path)
crypto1 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_first = key_path.read_bytes()
crypto2 = CredentialCrypto(key_path)
crypto2 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_second = key_path.read_bytes()
assert contents_after_first == contents_after_second
# And both crypto instances decrypt each other's ciphertext
@@ -36,7 +36,7 @@ def test_load_existing_key_is_idempotent(tmp_path):
def test_encrypt_decrypt_round_trip(tmp_path):
crypto = CredentialCrypto(tmp_path / "k")
crypto = CredentialCrypto(tmp_path / "k", bootstrap_ok=True)
plaintext = "domain.com\tTRUE\t/\tTRUE\t1700000000\tname\tvalue"
ct = crypto.encrypt(plaintext)
assert isinstance(ct, bytes)
@@ -45,8 +45,27 @@ def test_encrypt_decrypt_round_trip(tmp_path):
def test_decrypt_with_wrong_key_raises(tmp_path):
crypto_a = CredentialCrypto(tmp_path / "a")
crypto_b = CredentialCrypto(tmp_path / "b")
crypto_a = CredentialCrypto(tmp_path / "a", bootstrap_ok=True)
crypto_b = CredentialCrypto(tmp_path / "b", bootstrap_ok=True)
ct = crypto_a.encrypt("secret")
with pytest.raises(InvalidCredentialBlob):
crypto_b.decrypt(ct)
def test_missing_key_without_bootstrap_raises(tmp_path, monkeypatch):
"""Audit 2026-06-02: without explicit opt-in, a missing key file
is a fatal startup error — silent regeneration on partial restore
would make every existing Credential row undecryptable."""
from backend.app.services.credential_crypto import MissingCredentialKey
monkeypatch.delenv("CURATOR_BOOTSTRAP_NEW_KEY", raising=False)
with pytest.raises(MissingCredentialKey):
CredentialCrypto(tmp_path / "absent.b64")
def test_missing_key_with_env_var_bootstraps(tmp_path, monkeypatch):
"""The env var CURATOR_BOOTSTRAP_NEW_KEY=1 is the operator's
first-time-setup opt-in for auto-creating the key file."""
monkeypatch.setenv("CURATOR_BOOTSTRAP_NEW_KEY", "1")
key_path = tmp_path / "bootstrap.b64"
CredentialCrypto(key_path) # no bootstrap_ok kwarg — relies on env
assert key_path.exists()
+1 -1
View File
@@ -21,7 +21,7 @@ _NETSCAPE = (
@pytest.fixture
def crypto(tmp_path):
return CredentialCrypto(tmp_path / "k")
return CredentialCrypto(tmp_path / "k", bootstrap_ok=True)
@pytest.mark.asyncio
+6 -6
View File
@@ -130,7 +130,7 @@ async def test_download_source_attaches_written_files(
thumbnailer=Thumbnailer(images_root=images_root),
settings=sync_settings,
)
crypto = CredentialCrypto(tmp_path / "key.b64")
crypto = CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True)
cred_service = CredentialService(db, crypto)
svc = DownloadService(
@@ -405,7 +405,7 @@ async def test_backfill_decrements_after_run(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -446,7 +446,7 @@ async def test_backfill_auto_resets_on_clean_zero_files(
thumbnailer=Thumbnailer(images_root=tmp_path / "images"),
settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -484,7 +484,7 @@ async def test_tick_mode_does_not_touch_backfill_counter(
thumbnailer=Thumbnailer(images_root=tmp_path / "images"),
settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -529,7 +529,7 @@ async def test_partial_error_type_maps_to_ok_status(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -582,7 +582,7 @@ async def test_download_enqueues_thumbnail_and_ml_per_attached_image(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
# Capture the IDs that the orchestrator hands off to each Celery task.
# The .delay() shim runs inside DownloadService._phase3_persist (lazy
+6 -3
View File
@@ -39,8 +39,9 @@ async def test_threshold_filters_low_confidence_general(db):
await db.flush()
sl = await SuggestionService(db).for_image(img.id)
names = [s.display_name for s in sl.by_category.get("general", [])]
assert "sword" in names
assert "lowconf" not in names
# display_name is normalized (tag_name.normalize) before surfacing.
assert "Sword" in names
assert "Lowconf" not in names
@pytest.mark.asyncio
@@ -84,7 +85,9 @@ async def test_raw_tag_creates_new(db):
await db.flush()
sl = await SuggestionService(db).for_image(img.id)
chars = sl.by_category["character"]
assert chars[0].display_name == "brand_new_tag"
# display_name is the normalized Camie name (underscores -> spaces,
# title-cased), not the raw vocab key.
assert chars[0].display_name == "Brand New Tag"
assert chars[0].creates_new_tag is True
assert chars[0].canonical_tag_id is None
+53
View File
@@ -0,0 +1,53 @@
import pytest
from backend.app.services.ml.tag_name import normalize
@pytest.mark.parametrize(
"raw, expected",
[
# Rule 4: underscores -> spaces; rule 7: title case
("light_purple_hair", "Light Purple Hair"),
("no_pants", "No Pants"),
("year_2005", "Year 2005"),
# Single-word still title-cased
("sword", "Sword"),
# Rule 3: drop trailing _(disambiguator)
("uchiha_sasuke_(naruto)", "Uchiha Sasuke"),
("apple_(fruit)", "Apple"),
("kirby_(series)", "Kirby"),
# Repeated trailing disambig blocks
("foo_(bar)_(baz)", "Foo"),
# Rule 1: leading junk chars
("#unicus_(idolmaster)", "Unicus"),
(".52_gal_(splatoon)", "52 Gal"),
("+_+_smile_(emote)", "Smile"),
# Rule 5: space after colon
("nier:automata", "Nier: Automata"),
# Already-spaced colon left alone
("nier: automata", "Nier: Automata"),
# Rule 6: hyphens preserved
("1000-nen_ikiteru_(vocaloid)", "1000-nen Ikiteru"),
("well-known_face", "Well-known Face"),
# Rule 2: wrapping quotes
('"pile_em_up"_(genshin_impact)', "Pile Em Up"),
("'foo_bar'", "Foo Bar"),
# Rule 8: emoticons -> None
(":/", None),
(";)", None),
("+_+", None),
("^_^", None),
# Empty / whitespace-only
("", None),
(" ", None),
("___", None),
# Apostrophe inside word — preserved, not title-cased
("it's_okay", "It's Okay"),
# Digit-only still surfaces (year tags)
("2005", "2005"),
# Multi-space collapse
("foo___bar", "Foo Bar"),
],
)
def test_normalize(raw, expected):
assert normalize(raw) == expected