Compare commits

..
29 Commits
Author SHA1 Message Date
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-ml (push) Successful in 3m28s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
CI / intimp (push) Successful in 4m26s
CI / lint (push) Successful in 3s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen d3245f0c22 feat(ext): verify cookies in-browser before uploading (1.0.7)
CI / intcore (push) Successful in 8m45s
extension / lint (pull_request) Successful in 14s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 40s
extension / lint (push) Successful in 36s
CI / lint (push) Successful in 4s
CI / intimp (push) Successful in 4m4s
CI / intapi (push) Successful in 8m2s
Pre-upload verify request: after capturing the live browser cookies,
hit a known authenticated endpoint with credentials:'include' from the
extension's background context. If the platform reports we're not
logged in, abort the upload so we don't overwrite FC-side credentials
with stale data.

- platforms.js: add `verify` config per cookie-auth platform
  - hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF
    _init_site_filters; same 401 path the operator hit 2026-06-03)
  - patreon: GET /api/current_user (clean 401 when logged out)
  - subscribestar, deviantart: no stable auth endpoint, skip verify
- cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}.
  ok=true/false/null tri-state — null = verify not configured, caller
  treats as "proceed".
- background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the
  upload; failures bubble up with the platform's name + reason.
- popup.js: success message now appends "(verified ✓)" when applicable.
- manifest + package.json: 1.0.6 → 1.0.7.
2026-06-03 14:04:45 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
2026-06-03 13:18:44 -04:00
bvandeusen e450145304 fix(ml): preserve digit-only tag names in normalize (year tags)
CI / frontend-build (push) Successful in 21s
CI / intapi (push) Successful in 7m42s
CI / intcore (push) Successful in 8m12s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / intimp (push) Successful in 3m45s
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
2026-06-03 13:09:35 -04:00
bvandeusen a6e8d4b52e feat(ml): normalize Camie suggestion names to human-readable
CI / lint (push) Successful in 2s
CI / intimp (push) Successful in 3m57s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m22s
CI / backend-lint-and-test (push) Failing after 24s
CI / frontend-build (push) Successful in 28s
Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`,
`#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were
surfacing raw in SuggestionsPanel — and worse, the SAME raw string was
written to tag.name on Accept, polluting the DB with `underscored_lowercase`
names that don't match the operator's "Title Case" tag convention.

Add backend/app/services/ml/tag_name.py with a single normalize()
applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing
_(disambiguator) blocks iteratively, strip wrapping quotes, underscores
to spaces, space after colon, title-case each word's first char,
preserve hyphens/apostrophes/digits, drop entries with no letters).

Wire into SuggestionService.for_image:
- raw Camie key kept for alias_map lookup (alias rows are hand-curated
  against raw keys; don't disturb)
- display_name = normalize(raw); None means drop the candidate
- existing-tag lookup widened to case-insensitive match against BOTH
  raw and normalized forms so legacy underscore-named Tag rows accepted
  before this change still surface as "existing" not "+ new"
2026-06-03 13:00:08 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen f1860866de chore(modal): drop the ?image=N soft-compat from G5.4
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m38s
CI / intcore (push) Successful in 9m31s
Operator confirmed they have no existing ?image=N bookmarks to
preserve, so the soft-compat read on initial mount + router.replace
strip is dead weight. The modal is now purely a Pinia overlay — no
URL involvement on open OR initial mount.

Drops 33 lines plus the now-unused vue-router imports in both
ArtistGalleryTab (entire onMounted gone) and GalleryView (just the
?image=N block; the post_id/tag_id filter handling stays).
2026-06-02 19:24:09 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen b181d779fe fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 8m13s
CI / intcore (push) Successful in 7m6s
2026-06-02 18:49:24 -04:00
bvandeusen 0fbb19dc24 fix(modal): TagPanel kebab — apply the wrapping-span fix that the prior commit missed
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m59s
CI / intapi (push) Failing after 8m56s
CI / intcore (push) Successful in 10m0s
The previous commit (8326e54) updated the CSS but the structural
edit to the v-menu wrapper didn't take. Re-applying so the chip
kebab actually opens.
2026-06-02 18:48:53 -04:00
bvandeusen 8326e5447a fix(modal): kebab menus weren't opening (chip + suggestion rows)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Failing after 8m41s
CI / intcore (push) Successful in 10m0s
Operator-flagged 2026-06-02. Both kebab activators were broken:

- TagPanel chip kebab had `@click.stop` directly on the v-icon
  activator. In Vue 3, an explicit @click on the same element as
  `v-bind="props"` overrides the spread onClick — so Vuetify's
  activator handler never fired. Menu never opened.

- SuggestionItem kebab didn't have @click.stop, but for consistency
  and to make both kebabs follow the same shape, wrap it too.

The fix: each kebab is now wrapped in a `<span @click.stop>`. The
v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu;
the bubbling click then reaches the span where stopPropagation
absorbs it before it can affect a parent (the v-chip's close button
in TagPanel's case).
2026-06-02 18:48:32 -04:00
bvandeusen 1fd594baaf chore(ml): suggestion_threshold default 0.50 → 0.70
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Failing after 8m16s
CI / intcore (push) Successful in 8m43s
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).

Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.

Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
2026-06-02 18:38:12 -04:00
bvandeusen ecac6c4bda fix(audit-g5): centroid version DB-as-truth + modal as overlay
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 24s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 7m57s
CI / intcore (push) Successful in 8m46s
Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4).

G5.1 — Centroid version no longer drifts:

CentroidService now reads MLSettings.embedder_model_version (the DB
row tag_and_embed already writes from) for both the centroid model-
version stamp and the drift-detection comparison. Previously the
centroid sites imported MODEL_VERSION from env, so the version stamped
on centroids could disagree with the version stamped on the embeddings
they were built from. By construction those now match, so list_drifted
won't silently miss the env-vs-DB drift case.

embedder.py keeps MODEL_VERSION as an env-driven constant for the
actual model loader — that's a different concern (which weights are
loaded) from the version-stamp that gets persisted alongside data.

G5.4 — Modal is a Pinia-only overlay:

The previous URL↔modal sync in GalleryView and ArtistGalleryTab
leaked the modal across route changes (RouterLink to /artist/<slug>
left the modal mounted on top of the new route) and re-opened it
on history back/forward with stale ?image=N entries.

Now: openImage() just calls modal.open(id) — no URL push.
GalleryView's dead closeImage helper is deleted. A route.name
watcher in App.vue closes the modal whenever the route changes,
which auto-fixes RouterLink-in-modal and back/forward.

Backward-compat: ?image=N is still honored on initial mount as a
one-shot deep-link opener, then router.replace strips the query so
the URL doesn't re-trigger and no extra history entry is added.
Existing bookmarks / shared URLs keep working; new opens stay
Pinia-only.
2026-06-02 18:28:57 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 9f7261b9c0 fix(audit-g5c): set CURATOR_BOOTSTRAP_NEW_KEY=1 in conftest
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m50s
CI / intapi (push) Successful in 8m33s
CI / intcore (push) Successful in 9m7s
CredentialCrypto's safety check fires on create_app() instantiation
because the test environment has no pre-seeded Fernet key file. Set
the bootstrap env var before any test imports so the auto-create path
is allowed during tests.
2026-06-02 17:55:25 -04:00
bvandeusen f05aaa707b fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Failing after 26s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Failing after 8m10s
CI / intcore (push) Failing after 9m42s
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.

FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
  - warning (yellow) — operator action needed (auth_error)
  - info (blue)      — backend-paced (rate_limited / timeout /
                       network_error / partial / tier_limited)
  - error (red)      — likely terminal without intervention
                       (not_found / access_denied / validation_failed /
                        unsupported_url / http_error / unknown_error)

Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
2026-06-02 17:54:44 -04:00
bvandeusen 4df98171ab fix(audit-g5c): refuse silent Fernet key regeneration on partial restore
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Failing after 25s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Failing after 8m13s
CI / intcore (push) Failing after 8m48s
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.

Two opt-ins now needed for auto-creation:
  1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
  2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)

Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.

Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.

Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
2026-06-02 17:04:27 -04:00
bvandeusen 8d75ade1d5 fix(audit-g5b): race-poisoning in three find-or-create sites
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m29s
Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the
race under concurrent writers / recovery-sweep replays, poisoning the
outer transaction with an unrecoverable IntegrityError and crashing
the calling task. Same shape as the banked 2026-05-26 ImageProvenance
incident (see reference_scalar_one_or_none_duplicates memory).

Mirrors the importer._get_or_create pattern in all three: savepoint
via begin_nested + IntegrityError rollback to that savepoint + re-
SELECT to grab the row the other worker committed first.

- importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store
  is sha-addressed so both workers race to write the same on-disk path
  (shutil.copy2 + rename is idempotent), so no extra cleanup needed.

- TagService.find_or_create: partial uniqueness index on
  (name, kind, COALESCE(fandom_id, -1)). The previous docstring
  claimed "INSERT ... ON CONFLICT" but the implementation was
  SELECT-then-INSERT with no recovery.

- ArtistService.find_or_create: already had IntegrityError handling
  but did session.rollback() (unwinds the WHOLE transaction); now
  uses begin_nested + sp.rollback() so the surrounding request's
  progress isn't lost.
2026-06-02 17:02:12 -04:00
bvandeusen 75c63e1511 fix(audit-g5a): ruff isort — platforms after patreon_resolver
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 16:55:56 -04:00
bvandeusen 98673d4dca fix(audit-g5a): small architectural cleanups bundle
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m23s
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.

- download_service: replace hardcoded ('discord','pixiv') tuple with
  auth_type_for(platform) == 'token'. A 7th token-platform now picks
  up the right credential path without touching this site.

- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
  merge so the target's centroid reflects its new image set
  immediately. Daily list_drifted catches it within 24h, but eager
  recompute closes the suggestion-quality dip in the meantime.

- backfill_thumbnails added to beat_schedule (daily). The task
  docstring claimed periodic Beat but the entry was never registered,
  so the library got no self-healing thumbnail repair; only the
  manual admin-UI button fired it.

- modal.createAndAdd pushes a kind='fandom' tag into
  tagsStore.fandomCache so FandomPicker sees the new fandom on next
  open. Was: cache-gated load (length===0) skipped refetch, new
  fandom invisible until full page reload.

- cleanup cluster:
  - Drop .webp from cleanup_service.unlink — thumbnailer only writes
    .jpg/.png; the third tuple member was dead code.
  - Drop effective_date from /api/gallery/scroll response — no FE
    consumer reads it. Service still computes the attribute for
    timeline ordering; this just trims the JSON.
  - Rename store.recentMinute → store.recentRuns across the
    systemActivity store + three consumers (SystemActivitySummary,
    QueuesTable, SystemActivityTab). The data is the last 200 runs
    (not actually "last minute"), so the name lied.

NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
2026-06-02 16:46:46 -04:00
bvandeusen 89b48f8f35 Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 35s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m10s
2026-06-02 16:15:00 -04:00
bvandeusen 4bff1d8558 fix(audit-g4): status-enum miss batch
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m37s
CI / intcore (push) Successful in 8m18s
Five extension-miss findings from the 2026-06-02 audit, where a status
value was added on one side but a downstream consumer didn't pick it up.

- download_service._phase3_persist: explicit branches for
  ImportResult.status in ('failed','refreshed'). For 'failed' (archive
  probe crash from _import_archive), unlink the source file so the
  filesystem scanner doesn't re-import and re-crash on the same
  archive forever. 'refreshed' is currently unreachable from the
  download path (no deep=True) but matches the importer's documented
  contract; treat as 'attached'.

- gallery-dl backfill auto-complete now gates on dl_result.success +
  no error_type, not just return_code==0 + files_downloaded==0.
  VALIDATION_FAILED exits the subprocess with returncode=0 and
  files_downloaded=0 when every file was quarantined, matching the
  prior predicate exactly and zeroing the operator's armed backfill
  budget on the FIRST quarantine run instead of decrementing.

- attach_in_place archive dispatch now threads artist + source_row
  through _import_archive (and _import_media for archive members)
  and _supersede. The path-walk fallback (_resolve_artist) is still
  used by filesystem-import; the download path now binds
  ImageProvenance to the explicit subscription Source instead of
  rediscovering by (artist_id, platform).

- Three FE handlers now recognize status:'deferred' from
  /api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting
  "event #undefined"), SubscriptionsTab.checkAll (was counting
  deferred as queued), DownloadEventRow.onRetry (was saying
  "re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll
  which already had it.

- celery_signals._queue_for now maps backup/admin/library_audit
  prefixes to 'maintenance' (matching task_routes). TaskRun.queue
  was returning 'default' for those rows, so per-queue dashboard
  filters and per-queue threshold overrides (added in G3) silently
  missed them.
2026-06-02 16:04:59 -04:00
bvandeusen d60e0b9494 Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 36s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Successful in 7m39s
CI / intcore (push) Successful in 8m10s
2026-06-02 14:49:28 -04:00
bvandeusen e30f50e6fe fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
CI / lint (push) Successful in 3s
CI / intapi (push) Successful in 7m30s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m30s
CI / intcore (push) Successful in 8m8s
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.

Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
  running/restoring past 7h (covers the 6.5h images-backup hard
  limit) to error. prune_backups docstring corrected — the FC-3i
  TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
  past 135 min (10-min buffer above scan_library_for_rule's 2h5m
  hard limit) to error. Previously a SIGKILL'd row blocked all
  future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
  stuck running >2h whose child tasks are all terminal (orphan case
  where the orchestrator crashed before the closing UPDATE). Uses
  the same EXISTS predicate /api/system/stats already had.

Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
  matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
  the model relationship.

time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).

start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.

/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.

DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
2026-06-02 14:30:46 -04:00
bvandeusen 9c27a2d3c7 Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 37s
Build images / build-web (push) Successful in 13s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 8m50s
2026-06-02 14:17:12 -04:00
bvandeusen e66987f092 fix(audit-g2): async race / state-leak across eight stores
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 34s
CI / intimp (push) Successful in 3m31s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m2s
Extracts gallery.js's hand-rolled inflightId pattern into a new
useInflightToken composable; adopts in every store that previously
had no guard against late-response overwrites or wrong-image URL
interpolation.

Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62)
flagged:

- modal.removeTag rolled back the chip rail unconditionally even
  when only the secondary dismiss POST had failed — UI lied until
  refresh. And all tag-mutation URLs interpolated currentImageId
  AFTER an await, so a fast prev/next could route DELETE/POST to
  the wrong image. Both fixed: split try/catch (dismiss failure
  surfaces a warning, doesn't roll back the delete); imageId
  captured at call-time and used in URLs throughout.

- suggestions.accept dereferenced currentImageId after the awaited
  POST /api/tags, so the subsequent /suggestions/accept could
  apply A's chosen tag to image B AND push it to B's allowlist.
  Fixed by capturing imageId at click-time + inflight guard on
  load().

Same shape across artist / downloads / artistDirectory /
tagDirectory / posts stores: rapid filter/nav changes used to
interleave responses (last-writer-wins). Now the late response is
discarded and the most-recent request wins. Filter-change-during-
search no longer drops the second fetch because the loading flag
was still true from the first.

gallery.js's inflightId removed in favor of the shared composable
so the pattern stays consistent.
2026-06-02 14:07:58 -04:00
bvandeusen 93e37681b7 Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 10s
CI / intapi (push) Successful in 7m28s
CI / intcore (push) Successful in 8m3s
CI / intimp (push) Successful in 3m42s
2026-06-02 13:29:17 -04:00
bvandeusen 80ef9bce48 fix(test): credentials.upload reflects record into cache (not invalidate)
CI / lint (push) Successful in 6s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 35s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 8m12s
Stale assertion pinned the buggy pre-G1.6 behavior (the test name
"upload invalidates the cache" literally describes the bug). The
audit's correction makes upload mirror the returned record into the
store so the card updates immediately — update the assertion to
match the corrected behavior.
2026-06-02 13:11:39 -04:00
bvandeusen 3898ce7be4 fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit
CI / lint (push) Successful in 5s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Failing after 27s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 8m6s
CI / intcore (push) Successful in 9m3s
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
  30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
  TimeoutExpired (matched the tick 870s/900s rationale). Backfill
  runs that hit the cap let the next tick continue via the archive.

- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
  without it cleanup_old_tasks' WHERE finished_at<cutoff never
  reaped orphan-swept rows. recover_stalled_task_runs also now sets
  duration_ms (matches celery_signals.finalize's millisecond math).

- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
  on Source creation, mirroring SourceService.create. Without it,
  Firefox quick-add on a creator with >20 unsynced posts walked the
  full feed until subprocess timeout. Renamed the constant from
  _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.

- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
  NO_NEW_CONTENT — the download path (line 712) already does, and
  TIER_LIMITED proves auth reached the post and was told it was
  tier-gated. Verify endpoint previously showed red on this and
  prompted operators to rotate working cookies.

- prune_unused_tags now runs a single DELETE with the NOT-IN
  predicate find_unused_tags uses, instead of SELECT-ids →
  DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
  would have surfaced on a tag explosion (>65k unused tags).

- credentials.upload() reflects the returned record into the store
  cache (`.set(platform, rec)`) instead of evicting it; previously
  the card briefly rendered "no credential" between upload and
  loadAll().
2026-06-02 11:24:05 -04:00
61 changed files with 1419 additions and 246 deletions
@@ -0,0 +1,41 @@
"""source.error_type: surface ErrorType taxonomy in FailingSourcesCard
Revision ID: 0032
Revises: 0031
Create Date: 2026-06-02
Audit 2026-06-02: the backend computes 13 ErrorType categories (auth_error,
rate_limited, not_found, access_denied, validation_failed, etc.) and
stamps each one on DownloadEvent.metadata, but the Source row only carried
the free-text last_error. Operators couldn't bulk-triage failing sources
("all auth_error → rotate cookies, all rate_limited → just wait") without
opening Logs per row.
This column receives the last error_type from _update_source_health
and gets cleared on a successful run. Nullable + indexed so the failing-
sources rollup can filter/group cheaply.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0032"
down_revision: Union[str, None] = "0031"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column("error_type", sa.String(length=32), nullable=True),
)
op.create_index(
"ix_source_error_type", "source", ["error_type"],
)
def downgrade() -> None:
op.drop_index("ix_source_error_type", table_name="source")
op.drop_column("source", "error_type")
@@ -0,0 +1,48 @@
"""suggestion_threshold default 0.50 → 0.70
Revision ID: 0033
Revises: 0032
Create Date: 2026-06-02
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) is
too noisy in practice; raise to 0.70 for both suggestion categories.
Only conditionally updates singletons whose current value is still the
2026-06-01 default (0.50). Operators who deliberately tuned their row
to some other value (0.55, 0.65, 0.80, etc. via the Settings UI) keep
their pick — the migration only catches the unchanged-default case.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0033"
down_revision: Union[str, None] = "0032"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.70 "
"WHERE id = 1 AND suggestion_threshold_character = 0.50"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.70 "
"WHERE id = 1 AND suggestion_threshold_general = 0.50"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.50 "
"WHERE id = 1 AND suggestion_threshold_character = 0.70"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.70"
)
-1
View File
@@ -43,7 +43,6 @@ async def scroll():
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+17 -1
View File
@@ -35,10 +35,26 @@ async def trigger_scan():
@import_admin_bp.route("/status", methods=["GET"])
async def status():
async with get_session() as session:
# Active batch = running batch that still has outstanding work.
# Plain "most recent running" picks freshly-created scans that
# enqueued zero new files and hides the older batch that's
# actually being processed. Mirrors the EXISTS predicate
# /api/system/stats already uses (api/settings.py:145-160).
# Audit 2026-06-02 — /api/import/status and /api/system/stats
# used to disagree on the active-batch predicate; the UI banner
# said "Scanning…" indefinitely while the stats card said idle.
active = (
await session.execute(
select(ImportBatch)
.where(ImportBatch.status == "running")
.where(
ImportBatch.status == "running",
select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists(),
)
.order_by(ImportBatch.started_at.desc())
.limit(1)
)
+6
View File
@@ -245,6 +245,12 @@ async def merge_tag(source_id: int):
from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=result.target_id)
# Tag merge invalidates the target's centroid (the merged-in source
# tag's images now contribute to it). Daily list_drifted catches it
# within 24h, but eager recompute closes the suggestion-quality dip
# in the meantime. Audit 2026-06-02.
from ..tasks.ml import recompute_centroid
recompute_centroid.delay(result.target_id)
return jsonify(
{
"target": {
+35
View File
@@ -105,6 +105,41 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
# Audit 2026-06-02 — three new per-entity recovery sweeps.
# Each runs every 5 min like the other recover_stalled_*
# sweeps; each is a no-op when nothing is stuck.
"recover-stalled-backup-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_backup_runs",
"schedule": 300.0,
},
"recover-stalled-library-audit-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
"schedule": 300.0,
},
"recover-stalled-import-batches": {
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
"schedule": 300.0,
},
# Audit 2026-06-02 — daily retention for two entities
# whose terminal rows otherwise accumulate forever.
"prune-library-audit-runs": {
"task": "backend.app.tasks.maintenance.prune_library_audit_runs",
"schedule": 86400.0,
},
"prune-import-batches": {
"task": "backend.app.tasks.maintenance.prune_import_batches",
"schedule": 86400.0,
},
# Audit 2026-06-02 — backfill_thumbnails's docstring claimed
# "periodic Beat" but the entry was never registered, so the
# library got no self-healing thumbnail repair; only the
# manual admin-UI button fired it. Daily cadence is gentle
# (the task is idempotent and only enqueues regen for rows
# whose stored thumbnails are missing or corrupt).
"backfill-thumbnails-daily": {
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
"schedule": 86400.0,
},
},
timezone="UTC",
)
+14 -2
View File
@@ -54,7 +54,14 @@ _INT32_MIN = -2_147_483_648
def _queue_for(task) -> str:
"""Reverse the task→queue routing from celery_app.task_routes.
Keep in sync if task_routes is reordered."""
Keep in sync if task_routes is reordered.
Audit 2026-06-02: backup/admin/library_audit prefixes were
missing here even though task_routes sent all three to
'maintenance'. The TaskRun.queue column then lied for those
rows (claimed 'default') so per-queue dashboard filters and
per-queue threshold overrides silently missed them.
"""
name = getattr(task, "name", "") or ""
if name.startswith("backend.app.tasks.import_file."):
return "import"
@@ -66,7 +73,12 @@ def _queue_for(task) -> str:
return "download"
if name.startswith("backend.app.tasks.scan."):
return "scan"
if name.startswith("backend.app.tasks.maintenance."):
if name.startswith((
"backend.app.tasks.maintenance.",
"backend.app.tasks.backup.",
"backend.app.tasks.admin.",
"backend.app.tasks.library_audit.",
)):
return "maintenance"
return "default"
+6 -5
View File
@@ -16,13 +16,14 @@ class MLSettings(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True)
suggestion_threshold_character: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
# Default lowered 0.95 → 0.50 on 2026-06-01 — operator-flagged that
# 0.95 hid most general suggestions. Operator-tunable via Settings →
# ML if too noisy.
# Default raised 0.50 → 0.70 on 2026-06-02 — operator-flagged 0.50
# surfaced too many low-confidence picks; 0.70 keeps the rail
# signal-rich while still surfacing more than the original 0.95
# which hid almost everything. Operator-tunable via Settings → ML.
suggestion_threshold_general: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
centroid_similarity_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.55
+5
View File
@@ -26,6 +26,11 @@ class Source(Base):
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
# alembic 0032: last ErrorType category (auth_error, rate_limited,
# not_found, ...). Lets FailingSourcesCard surface the taxonomy as
# a colored chip so operators can bulk-triage by error class. Set
# by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+15 -10
View File
@@ -208,27 +208,32 @@ class ArtistService:
)
async def find_or_create(self, name: str) -> tuple[Artist, bool]:
"""Return (artist, created). Slug-keyed; idempotent under races."""
"""Return (artist, created). Slug-keyed; idempotent under races.
Audit 2026-06-02: switched from session.rollback() to a
begin_nested savepoint + IntegrityError recovery so a lost
race doesn't unwind the calling request's surrounding work.
Mirrors importer._get_or_create.
"""
cleaned = (name or "").strip()
if not cleaned:
raise ValueError("artist name must not be empty")
slug = slugify(cleaned)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
select_existing = select(Artist).where(Artist.slug == slug)
existing = (await self.session.execute(select_existing)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
sp = await self.session.begin_nested()
try:
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
await self.session.flush()
await sp.commit()
except IntegrityError:
await self.session.rollback()
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
await sp.rollback()
existing = (await self.session.execute(select_existing)).scalar_one()
return existing, False
await self.session.commit()
return artist, True
+49 -15
View File
@@ -11,7 +11,7 @@ the one-and-done GS/IR migration tooling.)
"""
from __future__ import annotations
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -187,10 +187,14 @@ def unlink_image_files(
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
# Convention thumbs dir — try both extensions thumbnailer writes
# (.jpg for opaque, .png for alpha). `.webp` used to be in this
# tuple but the thumbnailer never writes it (operator-flagged in
# the 2026-06-02 audit) — keep the tuple aligned with what
# actually lands on disk.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
for ext in ("jpg", "png"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
@@ -354,18 +358,35 @@ def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
Implementation note: the previous SELECT-ids → DELETE-WHERE-IN
pattern was vulnerable to the psycopg 65535-parameter ceiling on
libraries with tag explosions. The live delete now runs a single
DELETE with the same NOT-IN predicate find_unused_tags uses, so
the row count scales without binding every id as a parameter.
Audit 2026-06-02.
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
sample_rows = find_unused_tags(session, limit=50)
sample = [t.name for t in sample_rows]
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
count = session.execute(
select(func.count())
.select_from(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
).scalar_one()
return {"count": count, "sample_names": sample}
result = session.execute(
Tag.__table__.delete()
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
)
session.commit()
return {"deleted": result.rowcount or 0, "sample_names": sample}
# Legacy tags FC no longer uses, in two shapes:
@@ -500,6 +521,9 @@ class ConfirmTokenMismatch(Exception):
_VALID_RULES = ("transparency", "single_color")
_AUDIT_GUARD_THRESHOLD_MINUTES = 135 # matches LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES
def start_audit_run(
session: Session, *, rule: str, params: dict[str, Any],
) -> int:
@@ -507,11 +531,21 @@ def start_audit_run(
scan_library_for_rule Celery task. Returns the new audit_id.
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
has status='running'. Operator must cancel or wait."""
has status='running' AND started recently. Audit 2026-06-02 made
the guard age-aware: a SIGKILL'd run leaves a row in 'running'
that the recovery sweep flips on its next pass (~5 min), but a
fresh start_audit_run between the SIGKILL and the sweep would
previously block forever. Past the threshold, treat the running
row as stale and let the sweep clean it up — the new run still
gets to start.
"""
if rule not in _VALID_RULES:
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
cutoff = datetime.now(UTC) - timedelta(minutes=_AUDIT_GUARD_THRESHOLD_MINUTES)
existing = session.execute(
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
select(LibraryAuditRun.id)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at >= cutoff)
).scalar_one_or_none()
if existing is not None:
raise AuditAlreadyRunning(existing)
+53 -11
View File
@@ -1,40 +1,82 @@
"""Fernet-based encryption for credential blobs.
The key is a single 32-byte value (urlsafe-base64-encoded; what
Fernet.generate_key produces) stored at a fixed path inside the
images/data root. Created on first boot if absent; mode 0600. No KDF
needed — the file contents are already maximum-entropy random bytes.
Fernet.generate_key produces) stored at /images/secrets/credential_key.b64
(mode 0600, parent dir 0700). The 2026-06-02 audit caught a silent
key-regeneration path: on a partial disaster restore where the DB was
restored but the secrets dir was lost, the old `_load_or_create_key`
would mint a fresh key with no log, producing a working-looking system
where every authenticated download failed AUTH_ERROR until the operator
re-uploaded every credential by hand. Now the constructor refuses to
auto-generate unless either:
Operator backup procedure must include this file alongside the rest
of /images/ — losing it makes existing encrypted_blob rows
undecryptable (recovery = delete the rows and re-upload).
* the caller explicitly passes `bootstrap_ok=True` (tests, scripts), or
* the env var `CURATOR_BOOTSTRAP_NEW_KEY=1` is set (operator opt-in
during first-time setup).
Otherwise it raises `MissingCredentialKey` so the app fails fast at
startup and the operator can restore the key file from backup.
Operator backup procedure must include /images/secrets/ alongside the
rest of /images/ — losing the key file makes existing encrypted_blob
rows undecryptable (recovery = delete the rows and re-upload).
"""
import logging
import os
from pathlib import Path
from cryptography.fernet import Fernet, InvalidToken
log = logging.getLogger(__name__)
_BOOTSTRAP_ENV_VAR = "CURATOR_BOOTSTRAP_NEW_KEY"
class InvalidCredentialBlob(Exception):
"""Raised when decryption fails (wrong key, tampered blob, …)."""
class MissingCredentialKey(Exception):
"""The Fernet key file is missing AND the caller hasn't opted in to
generating a new one. Audit 2026-06-02: prevents silent key
regeneration on partial DB-restored / secrets-lost deployments.
Set CURATOR_BOOTSTRAP_NEW_KEY=1 for first-time setup, or restore the
key file from backup."""
class CredentialCrypto:
"""Fernet encrypt/decrypt with an on-disk key file.
Instantiate with a path; the file is created on first access and
reused thereafter. Tests pass a tmp_path; production calls with
Instantiate with a path; the file is loaded if present, or created
if absent AND the caller has opted in (bootstrap_ok=True or
CURATOR_BOOTSTRAP_NEW_KEY=1 env var). Production sites:
`IMAGES_ROOT / "secrets" / "credential_key.b64"`.
"""
def __init__(self, key_path: Path):
def __init__(self, key_path: Path, *, bootstrap_ok: bool | None = None):
self._key_path = Path(key_path)
self._fernet = Fernet(self._load_or_create_key())
if bootstrap_ok is None:
bootstrap_ok = os.environ.get(_BOOTSTRAP_ENV_VAR) == "1"
self._fernet = Fernet(self._load_or_create_key(bootstrap_ok))
def _load_or_create_key(self) -> bytes:
def _load_or_create_key(self, bootstrap_ok: bool) -> bytes:
if self._key_path.exists():
return self._key_path.read_bytes()
if not bootstrap_ok:
raise MissingCredentialKey(
f"Fernet key file not found at {self._key_path}. "
f"For first-time setup, set {_BOOTSTRAP_ENV_VAR}=1. "
f"If this is a restored instance, restore the key file "
f"from backup — generating a new one would make every "
f"existing Credential row undecryptable."
)
log.warning(
"Generating NEW Fernet credential key at %s. Any existing "
"encrypted_blob rows in the DB will be undecryptable — "
"re-upload each credential after this completes.",
self._key_path,
)
parent = self._key_path.parent
parent.mkdir(parents=True, exist_ok=True)
os.chmod(parent, 0o700)
+52 -2
View File
@@ -35,6 +35,7 @@ from .gallery_dl import (
)
from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__)
@@ -177,6 +178,13 @@ class DownloadService:
return {"status": "in_flight", "event_id": existing.id}
if existing and existing.status == "pending":
existing.status = "running"
# Reset started_at on the pending→running transition so the
# recovery sweep (DOWNLOAD_STALL_THRESHOLD_MINUTES, 30 min)
# measures from real start, not from enqueue. On heavy-queue
# days a freshly-promoted event whose original started_at
# predated the cutoff would otherwise get swept mid-flight,
# racing phase3's commit. Audit 2026-06-02.
existing.started_at = datetime.now(UTC)
await self.async_session.commit()
event_id = existing.id
else:
@@ -187,7 +195,12 @@ class DownloadService:
event_id = ev.id
artist = source.artist
if source.platform in ("discord", "pixiv"):
# Drive cookies-vs-token selection from the platform registry's
# auth_type so a new 7th token-platform automatically picks the
# right credential path. The hardcoded tuple here used to drift
# out of sync with credential_service's auth_type_for(). Audit
# 2026-06-02.
if auth_type_for(source.platform) == "token":
cookies_path = None
auth_token = await self.cred_service.get_token(source.platform)
else:
@@ -308,6 +321,23 @@ class DownloadService:
# failure. Don't flag the run as error; the file stays
# on disk for operator inspection.
import_summary["skipped"] += 1
elif result.status == "failed":
# Hard failure (today only: archive probe crash/timeout).
# The original archive sits in /images/ as an orphan; the
# filesystem scanner would re-import and re-crash on the
# same file, so delete the source file and surface the
# error in import_summary. Audit 2026-06-02.
import_summary["errors"] += 1
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "refreshed":
# Currently unreachable from attach_in_place (the download
# path never runs in deep=True mode), but the importer's
# ImportResult contract enumerates it. Treat the same as
# 'attached' — work happened, no error. Audit 2026-06-02.
import_summary["attached"] += 1
else:
import_summary["errors"] += 1
@@ -354,12 +384,26 @@ class DownloadService:
# backfill run drained the queue (gallery-dl exited 0 + zero files
# downloaded means there was nothing to fetch); otherwise decrement
# the counter. Next tick falls back to tick mode once it hits 0.
#
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
# subprocess with return_code=0 and files_downloaded=0 (every
# file was quarantined), which used to match the auto-complete
# predicate exactly — zeroing the operator's armed budget on
# the FIRST quarantine run instead of decrementing. Require
# dl_result.success + no error_type so only genuinely-empty
# successful runs drain the counter.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
if dl_result.return_code == 0 and dl_result.files_downloaded == 0:
queue_drained = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
and dl_result.files_downloaded == 0
)
if queue_drained:
src.backfill_runs_remaining = 0
else:
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
@@ -390,9 +434,15 @@ class DownloadService:
if status == "ok":
source.consecutive_failures = 0
source.last_error = None
# alembic 0032 — clear the failure-class chip on success.
source.error_type = None
elif status == "error":
source.consecutive_failures = (source.consecutive_failures or 0) + 1
source.last_error = error_message
# alembic 0032 — stamp the failure-class so FailingSourcesCard
# can render a colored chip and operators can bulk-triage
# by error class without opening Logs per row.
source.error_type = error_type
if error_type == "rate_limited":
await set_platform_cooldown(self.async_session, source.platform)
elif status == "skipped":
@@ -16,6 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
from ..utils.slug import slugify
from .source_service import NEW_SOURCE_BACKFILL_RUNS
class UnknownPlatformError(Exception):
@@ -204,9 +205,16 @@ class ExtensionService:
return existing, False
sp = await self.session.begin_nested()
try:
# New subscription sources arm a few backfill runs so the
# first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built).
# Mirrors SourceService.create — without it, Firefox quick-
# add on a creator with >20 unsynced posts would surface
# as "check failed" with no diagnosis. Audit 2026-06-02.
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
)
self.session.add(src)
await self.session.flush()
+19 -2
View File
@@ -58,8 +58,19 @@ TICK_SKIP_VALUE = "exit:20"
# Backfill mode (operator-triggered deep scan): walk the full history.
# Source.backfill_runs_remaining > 0 selects this mode; the longer
# timeout below absorbs creators with thousands of posts.
#
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# before Celery SIGKILLs the worker — same rationale as the tick
# default at line 74. The audit (2026-06-02) caught this at 1800,
# guaranteeing SIGKILL on any backfill that ran to its subprocess
# budget: stdout/stderr lost, backfill_runs_remaining never
# decrements, recovery sweep stamps generic "stranded" 30 min later.
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
# was added to prevent. backfill_runs_remaining=3 still gives ~58
# minutes of cumulative walk across three runs for prolific creators.
BACKFILL_SKIP_VALUE = True
BACKFILL_TIMEOUT_SECONDS = 1800
BACKFILL_TIMEOUT_SECONDS = 1170
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
@@ -844,7 +855,13 @@ class GalleryDLService:
),
)
etype, msg = self._categorize_error(proc.returncode, proc.stdout, proc.stderr)
if proc.returncode == 0 or etype == ErrorType.NO_NEW_CONTENT:
# TIER_LIMITED proves auth worked — gallery-dl reached the
# post, was told it's tier-gated. The download path treats
# this as success (line 712); verify must too, or operators
# rotate working cookies for no reason. Audit 2026-06-02.
if proc.returncode == 0 or etype in (
ErrorType.NO_NEW_CONTENT, ErrorType.TIER_LIMITED,
):
return True, "Credentials valid — the feed authenticated."
if etype == ErrorType.AUTH_ERROR:
return False, msg
+67 -17
View File
@@ -374,10 +374,19 @@ class Importer:
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
sha = _sha256_of(source)
existing = self.session.execute(
select(PostAttachment).where(PostAttachment.sha256 == sha)
).scalar_one_or_none()
if existing is None:
select_existing = select(PostAttachment).where(PostAttachment.sha256 == sha)
existing = self.session.execute(select_existing).scalar_one_or_none()
if existing is not None:
self.session.commit()
return ImportResult(status="attached")
# Savepoint + IntegrityError recovery — PostAttachment.sha256 is
# UNIQUE, so two workers can both pass the SELECT and only the
# second INSERT fails. Without savepoint, the outer transaction
# poisons and the calling task crashes. attachments.store is
# sha-addressed so both workers race to write the same target
# path; shutil.copy2 + rename is idempotent. Audit 2026-06-02.
sp = self.session.begin_nested()
try:
stored = self.attachments.store(source, sha)
self.session.add(PostAttachment(
post_id=post.id if post else None,
@@ -390,10 +399,19 @@ class Importer:
size_bytes=source.stat().st_size,
))
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
# Lost the race — the other worker's row is canonical.
self.session.execute(select_existing).scalar_one()
self.session.commit()
return ImportResult(status="attached")
def _import_archive(self, source: Path) -> ImportResult:
def _import_archive(
self, source: Path, *,
artist: Artist | None = None,
source_row: Source | None = None,
) -> ImportResult:
# Layer-3 isolation: bomb-size guard + integrity test in a
# spawned child BEFORE extracting in this process. A
# decompression bomb or a native-lib crash on a malformed
@@ -401,6 +419,14 @@ class Importer:
# instead of OOMing/segfaulting the import worker. extract_archive
# is already fail-soft for plain exceptions, so this only adds
# the hard-crash protection.
#
# Audit 2026-06-02: optional artist/source_row kwargs let the
# download path thread its explicit subscription context
# through instead of having _resolve_artist re-derive from
# path-walk (which works by coincidence today because gallery-dl
# lays files out under /images/<artist_slug>/...). Filesystem
# import still calls bare _import_archive(source) and falls
# back to the path-walk derivation as before.
probe = safe_probe.probe_archive(source)
if not probe.ok:
if probe.crashed:
@@ -412,24 +438,28 @@ class Importer:
# still preserve the archive file itself as an attachment so
# nothing silently vanishes, matching extract_archive's
# fail-soft contract.
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
self._capture_attachment(source, post=post, artist=artist, resolved=True)
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
self._capture_attachment(
source, post=post, artist=artist_use, resolved=True,
)
return ImportResult(status="attached")
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = []
with extract_archive(source) as members:
for _name, member_path in members:
if not is_supported(member_path):
continue # non-media preserved via the stored archive
res = self._import_media(member_path, source)
res = self._import_media(
member_path, source, explicit_source=source_row,
)
if res.status in ("imported", "superseded") and res.image_id:
member_ids.append(res.image_id)
# Preserve the archive itself (links to the same Post/Artist).
self._capture_attachment(
source, post=post, artist=artist, resolved=True
source, post=post, artist=artist_use, resolved=True
)
if member_ids:
return ImportResult(
@@ -439,7 +469,8 @@ class Importer:
return ImportResult(status="attached")
def _import_media(
self, source: Path, attribution_path: Path
self, source: Path, attribution_path: Path,
*, explicit_source: Source | None = None,
) -> ImportResult:
"""The media import pipeline (filters, dedup, copy, provenance).
@@ -586,7 +617,15 @@ class Importer:
artist = self._attach_artist(record, artist_name)
# Sidecar provenance (best-effort; never fails the import).
self._apply_sidecar(record, attribution_path, artist)
# explicit_source lets the FC-3c download path bind the new
# ImageProvenance row to its subscription Source instead of
# having _apply_sidecar re-derive via _lookup_source_for_sidecar.
# Audit 2026-06-02 — archive members extracted from a
# subscription-downloaded zip previously lost subscription
# linkage if the on-disk layout didn't match assumptions.
self._apply_sidecar(
record, attribution_path, artist, explicit_source=explicit_source,
)
# Thumbnail is queued separately by the calling task; the importer
# does not generate thumbnails inline so the import queue stays moving.
@@ -673,7 +712,9 @@ class Importer:
error="sidecar json is metadata, not content",
)
if is_archive(path):
return self._import_archive(path)
return self._import_archive(
path, artist=artist, source_row=source,
)
if not is_supported(path):
post = self._post_for_sidecar(path, artist) if artist else None
return self._capture_attachment(
@@ -749,7 +790,8 @@ class Importer:
if rel == "smaller_exists":
target = self.session.get(ImageRecord, match_id)
self._supersede(
target, path, sha, phash, width, height, new_path=path
target, path, sha, phash, width, height,
new_path=path, artist=artist, source_row=source,
)
return ImportResult(status="superseded", image_id=match_id)
@@ -944,6 +986,8 @@ class Importer:
self, existing: ImageRecord, source: Path, sha: str,
phash: str, width: int | None, height: int | None,
*, new_path: Path | None = None,
artist: Artist | None = None,
source_row: Source | None = None,
) -> None:
"""Replace `existing`'s file with the larger `source`, keeping the
row id (so tags/series/curation stay attached). ML is cleared so
@@ -995,8 +1039,14 @@ class Importer:
# _apply_sidecar resolves artist from the sidecar itself if the
# existing row has none, and is internally guarded against
# missing-or-malformed sidecars (silent return).
# Audit 2026-06-02: thread artist/source_row from the
# download-path caller (attach_in_place smaller_exists branch)
# so the supersede preserves explicit subscription linkage
# instead of re-deriving via path-walk.
try:
self._apply_sidecar(existing, source, None)
self._apply_sidecar(
existing, source, artist, explicit_source=source_row,
)
except Exception as exc:
# Don't unwind the supersede DB swap if sidecar parsing
# blows up unexpectedly — the file replacement is the
+20 -4
View File
@@ -19,7 +19,6 @@ from ...models import (
TagReferenceEmbedding,
)
from ...models.tag import image_tag
from .embedder import MODEL_VERSION as SIGLIP_VERSION
ELIGIBLE_KINDS = {
TagKind.character,
@@ -46,6 +45,21 @@ class CentroidService:
)
).scalar_one()
async def _model_version(self) -> str:
"""Audit 2026-06-02: SigLIP model-version stamp comes from the
DB row, not the env constant. tag_and_embed (tasks/ml.py:110)
already reads from MLSettings.embedder_model_version, so by
sourcing centroid stamps + drift checks from the same row, we
eliminate the silent-drift case the audit flagged. env
SIGLIP_MODEL_VERSION still drives which model embedder.py
loads at runtime; the version stamp is purely the operator-
controlled identifier."""
return (
await self.session.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
)
).scalar_one()
async def recompute_for_tag(self, tag_id: int) -> bool:
"""Recompute one tag's centroid. Returns True if a centroid was
written, False if skipped (ineligible kind or too few members)."""
@@ -69,19 +83,20 @@ class CentroidService:
return False
centroid = np.mean(np.stack(embeddings), axis=0).astype(np.float32)
model_version = await self._model_version()
stmt = insert(TagReferenceEmbedding).values(
tag_id=tag_id,
embedding=centroid.tolist(),
reference_count=len(embeddings),
model_version=SIGLIP_VERSION,
model_version=model_version,
)
stmt = stmt.on_conflict_do_update(
index_elements=["tag_id"],
set_={
"embedding": centroid.tolist(),
"reference_count": len(embeddings),
"model_version": SIGLIP_VERSION,
"model_version": model_version,
"updated_at": func.now(),
},
)
@@ -92,6 +107,7 @@ class CentroidService:
"""Tag ids whose centroid is stale: member count != reference_count,
OR no centroid row, OR centroid built on a different SigLIP version.
Only considers eligible-kind tags with embeddings present."""
current_model_version = await self._model_version()
member_counts = (
select(
image_tag.c.tag_id.label("tag_id"),
@@ -116,7 +132,7 @@ class CentroidService:
TagReferenceEmbedding.reference_count
!= member_counts.c.members
)
| (TagReferenceEmbedding.model_version != SIGLIP_VERSION)
| (TagReferenceEmbedding.model_version != current_model_version)
)
)
return list((await self.session.execute(stmt)).scalars().all())
+27 -9
View File
@@ -4,7 +4,7 @@ threshold-filtered, category-grouped, ranked suggestions for one image.
from dataclasses import dataclass, field
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import (
@@ -16,6 +16,7 @@ from ...models import (
from ...models.tag import image_tag
from .aliases import AliasService
from .centroids import CentroidService
from .tag_name import normalize as normalize_tag_name
from .tagger import SURFACED_CATEGORIES
@@ -84,7 +85,12 @@ class SuggestionService:
)
# --- Camie predictions ---
candidates: list[tuple[str, str, float]] = []
# candidates carry (raw_name, display_name, category, confidence).
# raw_name = the booru-formatted vocab key, kept for alias_map
# lookup since alias rows are hand-curated against raw keys.
# display_name = normalize_tag_name(raw_name) — what the operator
# sees AND what gets written to tag.name on Accept.
candidates: list[tuple[str, str, str, float]] = []
for name, p in predictions.items():
category = p.get("category", "general")
if category not in SURFACED_CATEGORIES:
@@ -92,10 +98,14 @@ class SuggestionService:
conf = float(p.get("confidence", 0.0))
if conf < self._threshold_for(settings, category):
continue
candidates.append((name, category, conf))
display = normalize_tag_name(name)
if display is None:
# emoticon / pure-punctuation vocab entry — drop entirely
continue
candidates.append((name, display, category, conf))
alias_map = await self.aliases.resolve_many(
[(n, c) for n, c, _ in candidates]
[(raw, c) for raw, _disp, c, _conf in candidates]
)
merged: dict[object, Suggestion] = {}
@@ -116,8 +126,8 @@ class SuggestionService:
creates_new_tag=existing.creates_new_tag,
)
for name, category, conf in candidates:
canonical = alias_map.get((name, category))
for raw, display, category, conf in candidates:
canonical = alias_map.get((raw, category))
if canonical is not None:
if canonical.id in applied or canonical.id in rejected:
continue
@@ -133,9 +143,17 @@ class SuggestionService:
),
)
else:
# Case-insensitive match on BOTH the raw camie key AND
# the normalized form — covers legacy underscore-named
# Tag rows accepted before normalization shipped, AND
# any tag the operator created with the human form.
existing_tag = (
await self.session.execute(
select(Tag).where(Tag.name == name)
select(Tag).where(
func.lower(Tag.name).in_(
[raw.lower(), display.lower()]
)
)
)
).scalars().first()
if existing_tag is not None:
@@ -157,10 +175,10 @@ class SuggestionService:
)
else:
_merge(
f"raw:{name}:{category}",
f"raw:{display}:{category}",
Suggestion(
canonical_tag_id=None,
display_name=name,
display_name=display,
category=category,
score=conf,
source="tagger",
+62
View File
@@ -0,0 +1,62 @@
"""Camie vocabulary -> human-readable tag-name normalization.
Camie v2's ~57k tag vocabulary is booru-derived and arrives as raw
strings like `uchiha_sasuke_(naruto)`, `#unicus_(idolmaster)`,
`1000-nen_ikiteru_(vocaloid)`, or `:/`. We want the operator to see
"Uchiha Sasuke", "Unicus", "1000-Nen Ikiteru", or to never see the
emoticon at all — and we want the same clean string to be what lands
in `tag.name` when the suggestion is accepted, so Accept matches the
existing-tag convention (`tag_service.find_or_create`).
Rules (operator-approved 2026-06-03):
1. Strip leading junk chars (#, ., +, ;, ~, _, whitespace)
2. Drop trailing `_(disambiguator)` block(s), iteratively
3. Strip wrapping single/double quotes (after disambig removal so
`"foo_em_up"_(series)` -> `"foo_em_up"` -> `foo_em_up`)
4. Replace remaining `_` with space; collapse runs of whitespace
5. Add a space after any `:` (namespace:tag -> namespace: tag)
6. Preserve hyphens (booru hyphens often carry meaning)
7. Title-case each space-separated word (first character only —
apostrophes, digits, hyphens stay)
8. If no letters AND no digits remain, return None (drops emoticons
like `:/` or `^_^`; preserves bare digit tags like `2005`)
9. No surname/givenname swap — no reliable signal in the vocab
"""
import re
_LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
_MULTISPACE = re.compile(r"\s+")
_COLON_NOSPACE = re.compile(r":(?=\S)")
_HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
def _strip_wrapping_quotes(s: str) -> str:
if len(s) >= 2 and s[0] == s[-1] and s[0] in ('"', "'"):
return s[1:-1]
return s
def _title_word(w: str) -> str:
return w[:1].upper() + w[1:] if w else w
def normalize(raw: str) -> str | None:
"""Return the human-readable form of a raw Camie tag, or None if the
string is junk (emoticon, empty after stripping)."""
if not raw:
return None
s = _LEADING_JUNK.sub("", raw)
while True:
new = _TRAILING_DISAMBIG.sub("", s)
if new == s:
break
s = new
s = _strip_wrapping_quotes(s)
s = s.replace("_", " ")
s = _COLON_NOSPACE.sub(": ", s)
s = _MULTISPACE.sub(" ", s).strip()
if not s or not _HAS_ALPHANUMERIC.search(s):
return None
return " ".join(_title_word(w) for w in s.split(" "))
+5 -2
View File
@@ -59,6 +59,7 @@ class SourceRecord:
config_overrides: dict | None
last_checked_at: str | None
last_error: str | None
error_type: str | None
check_interval_override: int | None
consecutive_failures: int
next_check_at: str | None
@@ -76,6 +77,7 @@ class SourceRecord:
"config_overrides": self.config_overrides,
"last_checked_at": self.last_checked_at,
"last_error": self.last_error,
"error_type": self.error_type,
"check_interval_override": self.check_interval_override,
"consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at,
@@ -90,7 +92,7 @@ _EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "p
# Plan #544 follow-up: newly created enabled sources pre-arm backfill so
# their first N polls walk gallery-dl's full post history with the longer
# timeout (matches the manual "Deep scan" button's default).
_NEW_SOURCE_BACKFILL_RUNS = 3
NEW_SOURCE_BACKFILL_RUNS = 3
class SourceService:
@@ -144,6 +146,7 @@ class SourceService:
config_overrides=source.config_overrides,
last_checked_at=source.last_checked_at.isoformat() if source.last_checked_at else None,
last_error=source.last_error,
error_type=source.error_type,
check_interval_override=source.check_interval_override,
consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None,
@@ -218,7 +221,7 @@ class SourceService:
# the budget is spent or the queue drains.
# Disabled sources (incl. sidecar synthetics, url='sidecar:...')
# are never polled, so leave their counter at 0.
backfill_runs = _NEW_SOURCE_BACKFILL_RUNS if enabled else 0
backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
source = Source(
artist_id=artist_id, platform=platform, url=url,
enabled=enabled, config_overrides=config_overrides,
+17 -7
View File
@@ -5,6 +5,7 @@ from dataclasses import dataclass
from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Tag, TagKind, image_tag
@@ -86,9 +87,12 @@ class TagService:
f"fandom_id {fandom_id} does not reference a fandom tag"
)
# Upsert via INSERT ... ON CONFLICT DO NOTHING. We can't use the
# uniqueness index name directly (it's a partial coalesce-based
# expression), so we re-select after insert.
# Audit 2026-06-02: race-safe upsert via savepoint +
# IntegrityError recovery. The partial uniqueness index on
# (name, kind, COALESCE(fandom_id, -1)) catches concurrent
# inserts; without the savepoint the outer transaction would
# poison and the calling request crashes. Mirrors
# importer._get_or_create.
stmt = (
select(Tag)
.where(Tag.name == name)
@@ -101,10 +105,16 @@ class TagService:
if existing:
return existing
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
return new_tag
sp = await self.session.begin_nested()
try:
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
await sp.commit()
return new_tag
except IntegrityError:
await sp.rollback()
return (await self.session.execute(stmt)).scalar_one()
async def autocomplete(
self,
+5 -2
View File
@@ -240,8 +240,11 @@ def prune_backups() -> dict:
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
Status='running' / 'restoring' rows are never pruned — the
recover_stalled_backup_runs sweep flips truly-stuck ones to
'error' first. (Earlier docstring claimed the FC-3i TaskRun sweep
handled those, but TaskRun cleanup never touched BackupRun rows.
Audit 2026-06-02 added the dedicated sweep.)
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
+218 -3
View File
@@ -7,14 +7,17 @@ from datetime import UTC, datetime, timedelta
from pathlib import Path
from PIL import Image
from sqlalchemy import and_, delete, or_, select, update
from sqlalchemy import Integer, and_, cast, delete, func, or_, select, update
from ..celery_app import celery
from ..models import (
BackupRun,
DownloadEvent,
ImageRecord,
ImportBatch,
ImportSettings,
ImportTask,
LibraryAuditRun,
Source,
TaskRun,
)
@@ -55,6 +58,29 @@ FFPROBE_TIMEOUT_SECONDS = 10
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# Audit 2026-06-02: per-entity recovery sweep thresholds. Each must be
# > the entity's longest legitimate runtime (its task's time_limit + a
# small buffer) so the sweep never flags in-flight work.
#
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
# 2h15m gives a 10-min buffer.
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
# Import batches finalize only after every child ImportTask hits a
# terminal state. The recovery sweep targets the case where every
# task is done but the batch never got its closing UPDATE
# (orchestrator crashed at the wrong instant). 2h is well past any
# realistic single-batch import.
IMPORT_BATCH_STALL_THRESHOLD_MINUTES = 120
# Retention windows (terminal rows older than these get deleted by
# the daily prune sweeps). 30 days = operator-flagged "useful for
# triage for a few weeks, then noise."
LIBRARY_AUDIT_KEEP_DAYS = 30
IMPORT_BATCH_KEEP_DAYS = 30
# Overrides for recover_stalled_task_runs (the TaskRun 'running' sweep).
# Tasks/queues that legitimately run longer than the default 5-min
# threshold need their own larger value, else the sweep marks in-flight
@@ -69,9 +95,24 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# files); time_limit=2100.
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"ml": 25,
# Audit 2026-06-02 — maintenance/scan queues run tasks that
# legitimately exceed the 5-min default (verify_integrity at 70m
# hard, scan_directory at 70m hard, apply_allowlist_tags /
# recompute_centroids / backfill_phash at 35m hard). 75 min lives
# above the longest of those and the per-task overrides below
# cover the outliers (backups, library audit).
"maintenance": 75,
"scan": 75,
}
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"backend.app.tasks.import_file.import_archive_file": 40,
# Backup images runs hours, not minutes (6.5h hard limit). The
# task-name override beats the queue's 75-min default so a
# legitimately-running backup isn't flagged.
"backend.app.tasks.backup.backup_images_task": 420,
"backend.app.tasks.backup.restore_images_task": 420,
# Library audit scans the full library — 2h hard limit.
"backend.app.tasks.library_audit.scan_library_for_rule": 130,
}
@@ -182,6 +223,11 @@ def recover_interrupted_tasks() -> int:
.where(ImportTask.created_at < orphan_cutoff)
.values(
status="failed",
# Without finished_at, cleanup_old_tasks (`WHERE
# finished_at < cutoff`) never reaps these rows —
# orphan-swept rows would become permanent table
# tenants. Audit 2026-06-02.
finished_at=now,
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
@@ -278,6 +324,14 @@ def recover_stalled_task_runs() -> int:
f"no completion signal received within {minutes} min"
),
finished_at=now,
# Matches celery_signals.finalize's
# int((now - started_at).total_seconds() * 1000)
# — sweep-closed rows now carry duration like
# normally-finalized rows. Audit 2026-06-02.
duration_ms=cast(
func.extract("epoch", now - TaskRun.started_at) * 1000,
Integer,
),
)
)
for w in extra_where:
@@ -347,7 +401,12 @@ def prune_task_runs() -> dict:
return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@celery.task(name="backend.app.tasks.maintenance.backfill_phash")
@celery.task(
name="backend.app.tasks.maintenance.backfill_phash",
# Audit 2026-06-02 — keyset-paginated phash recompute over the whole
# library; legitimately runs >5 min on large libraries.
soft_time_limit=1800, time_limit=2100,
)
def backfill_phash() -> int:
"""Recompute phash for stored images that have none (imported before
FC-2d-i+ii). Keyset-paginated by id (restart-safe), NULL-only fill,
@@ -425,7 +484,13 @@ def _verify_one(path: Path, expected_sha: str, mime: str, sha_fn) -> str:
return "failed_verification"
@celery.task(name="backend.app.tasks.maintenance.verify_integrity")
@celery.task(
name="backend.app.tasks.maintenance.verify_integrity",
# Audit 2026-06-02 — full library sha256 + decode probe; on 100k-image
# libraries this runs an hour or more. Match the maintenance queue's
# recovery threshold (75 min) with 30s buffer below.
soft_time_limit=3600, time_limit=4200,
)
def verify_integrity() -> int:
"""Verify every ImageRecord file: sha256 recompute + decode/probe
(PIL for images; ffprobe for videos). Writes integrity_status
@@ -521,6 +586,156 @@ def recover_stalled_download_events() -> int:
return events_recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_backup_runs")
def recover_stalled_backup_runs() -> int:
"""Flip BackupRun rows stuck in running/restoring past the hard limit
to error. Audit 2026-06-02.
prune_backups (FC-3h) used to claim the FC-3i task_run sweep handled
these — but that sweep only flips TaskRun rows, not the BackupRun
artifact rows. A SIGKILL'd backup left BackupRun stuck forever
(dashboard showed phantom in-flight backups, keep_last_n offset
arithmetic skewed because zombies sat outside the ok/error window).
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
)
with SessionLocal() as session:
result = session.execute(
update(BackupRun)
.where(BackupRun.status.in_(["running", "restoring"]))
.where(BackupRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("recover_stalled_backup_runs: recovered %d rows", recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_library_audit_runs")
def recover_stalled_library_audit_runs() -> int:
"""Flip LibraryAuditRun rows stuck in running past the hard limit
to error. Audit 2026-06-02.
LibraryAuditRun.status='running' was protected by an exclusive
guard in start_audit_run — a SIGKILL'd run would block all future
audits until manual DB surgery. (The guard is now age-aware, but
this sweep is what makes that work in practice.)
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
)
with SessionLocal() as session:
result = session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_library_audit_runs: recovered %d rows", recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
def recover_stalled_import_batches() -> int:
"""Finalize ImportBatch rows stuck in running past the hard limit
when NO outstanding ImportTask remains. Audit 2026-06-02.
A batch row finalizes only after every child task hits a terminal
state. The orphan case: scanner crashed between the last task's
completion and the batch's closing UPDATE. The
`/api/import/status` route then surfaces the batch as 'active'
indefinitely while `/api/system/stats` (which uses the same
EXISTS predicate we apply below) correctly returns null.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=IMPORT_BATCH_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
# Batches still 'running' past the cutoff whose tasks are all
# terminal — there's no outstanding work, so flip the batch
# too. Mirrors the EXISTS predicate the active-batch surfaces use.
result = session.execute(
update(ImportBatch)
.where(ImportBatch.status == "running")
.where(ImportBatch.started_at < cutoff)
.where(
~select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists()
)
.values(status="complete", finished_at=now)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_import_batches: finalized %d zombie batches",
recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.prune_library_audit_runs")
def prune_library_audit_runs() -> int:
"""Daily retention: delete terminal LibraryAuditRun rows older than
LIBRARY_AUDIT_KEEP_DAYS. Never touches 'running'. Audit 2026-06-02.
Audit rows carry matched_ids JSONB blobs that can hold tens of
thousands of ids; without retention these accumulate.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=LIBRARY_AUDIT_KEEP_DAYS)
with SessionLocal() as session:
result = session.execute(
delete(LibraryAuditRun)
.where(LibraryAuditRun.status.in_(["ready", "applied", "cancelled", "error"]))
.where(LibraryAuditRun.finished_at < cutoff)
)
session.commit()
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.prune_import_batches")
def prune_import_batches() -> int:
"""Daily retention: delete terminal ImportBatch rows older than
IMPORT_BATCH_KEEP_DAYS. Cascade-deletes child ImportTask rows via
the model relationship. Never touches 'running'. Audit 2026-06-02.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=IMPORT_BATCH_KEEP_DAYS)
with SessionLocal() as session:
# ORM-level delete here (not Core delete) so the
# ImportBatch->tasks cascade fires; Core delete would skip it.
old_batches = session.execute(
select(ImportBatch)
.where(ImportBatch.status.in_(["complete", "cancelled"]))
.where(ImportBatch.finished_at < cutoff)
).scalars().all()
for batch in old_batches:
session.delete(batch)
session.commit()
return len(old_batches)
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_download_events")
def cleanup_old_download_events() -> int:
"""FC-3d: delete terminal DownloadEvent rows older than the configured
+15 -2
View File
@@ -212,7 +212,14 @@ def backfill(self) -> int:
return enqueued
@celery.task(name="backend.app.tasks.ml.apply_allowlist_tags", bind=True)
@celery.task(
name="backend.app.tasks.ml.apply_allowlist_tags",
bind=True,
# Audit 2026-06-02 — the full-sweep mode (neither tag_id nor image_id)
# is O(images × allowlist) and legitimately runs >5 min on large
# libraries. Cap matches the maintenance queue's recovery threshold.
soft_time_limit=1800, time_limit=2100,
)
def apply_allowlist_tags(self, tag_id: int | None = None,
image_id: int | None = None) -> int:
"""Retroactively apply allowlisted tags.
@@ -341,7 +348,13 @@ def recompute_centroid(self, tag_id: int) -> bool:
return asyncio.run(_run())
@celery.task(name="backend.app.tasks.ml.recompute_centroids", bind=True)
@celery.task(
name="backend.app.tasks.ml.recompute_centroids",
bind=True,
# Audit 2026-06-02 — drifted-centroid rebuild over potentially
# hundreds of tags.
soft_time_limit=1800, time_limit=2100,
)
def recompute_centroids(self) -> int:
"""Daily: find drifted centroids, enqueue recompute_centroid for each."""
import asyncio
+9 -1
View File
@@ -35,7 +35,15 @@ def _iter_import_files(import_root: Path):
yield entry
@celery.task(name="backend.app.tasks.scan.scan_directory", bind=True)
@celery.task(
name="backend.app.tasks.scan.scan_directory",
bind=True,
# Audit 2026-06-02 — large libraries make the scan legitimately long.
# Hard cap at 70 min so the corresponding QUEUE_STUCK_THRESHOLD_MINUTES
# ("scan") of 75 min always wins; soft limit gives the task a clean
# exit window before SIGKILL.
soft_time_limit=3600, time_limit=4200,
)
def scan_directory(self, triggered_by: str = "manual",
mode: str = "quick") -> int:
"""Walks the import root and creates ImportTasks. `mode` is 'quick'
+18 -2
View File
@@ -194,9 +194,20 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
};
}
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length };
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -229,8 +240,13 @@ browser.runtime.onMessage.addListener(async (msg) => {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length };
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
} catch (e) {
results[key] = { error: e.message };
}
+35
View File
@@ -76,3 +76,38 @@ async function getCookieCount(platformKey) {
return 0;
}
}
/**
* Verify cookies are live by hitting an authenticated endpoint with the
* browser's current cookie jar. Returns:
* { ok: true, status } — verified
* { ok: false, status, reason } — endpoint said we're not logged in
* { ok: null, reason } — no verify config for this platform; caller
* should treat as "verify not available,
* proceed with upload"
*
* Implementation note: extensions with `host_permissions` for the target
* domain get the user's cookies auto-attached to fetch() — same set
* gallery-dl will later use on the backend.
*/
async function verifyCookiesForPlatform(platformKey) {
const platform = PLATFORMS[platformKey];
if (!platform) return { ok: false, reason: `Unknown platform: ${platformKey}` };
if (!platform.verify) return { ok: null, reason: 'verify-not-configured' };
const { url, method, okStatuses } = platform.verify;
let resp;
try {
resp = await fetch(url, { method, credentials: 'include', cache: 'no-store' });
} catch (e) {
return { ok: false, reason: `Verify request failed: ${e.message}` };
}
if (okStatuses.includes(resp.status)) {
return { ok: true, status: resp.status };
}
return {
ok: false,
status: resp.status,
reason: `${url} returned HTTP ${resp.status} — session looks stale or logged out`,
};
}
+21
View File
@@ -13,6 +13,13 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FF424D',
urlPattern: /^https?:\/\/(www\.)?patreon\.com/,
// Patreon's `/api/current_user` returns 200 + the logged-in user
// when authenticated, 401 otherwise. Cheapest definitive check.
verify: {
url: 'https://www.patreon.com/api/current_user',
method: 'GET',
okStatuses: [200],
},
},
subscribestar: {
name: 'SubscribeStar',
@@ -26,6 +33,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FFD700',
urlPattern: /^https?:\/\/(www\.)?subscribestar\.(com|adult)/,
// No known stable auth-required endpoint that returns a definitive
// status code; skipping verify so we don't false-positive-fail
// good cookies. Operator can add later if a clean endpoint surfaces.
},
hentaifoundry: {
name: 'Hentai Foundry',
@@ -33,6 +43,14 @@ const PLATFORMS = {
authType: 'cookies',
color: '#9C27B0',
urlPattern: /^https?:\/\/(www\.)?hentai-foundry\.com/,
// Mirror gallery-dl's _init_site_filters: HEAD on `?enterAgree=1`.
// Logged in → 200, logged out → 401. Catches the exact failure mode
// the backend extractor would hit later.
verify: {
url: 'https://www.hentai-foundry.com/?enterAgree=1',
method: 'HEAD',
okStatuses: [200],
},
},
discord: {
name: 'Discord',
@@ -56,6 +74,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#05CC47',
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
// DA's logged-in-only endpoints sit behind their internal _napi
// namespace which shifts; skipping verify until a stable check
// surfaces. Same posture as SubscribeStar.
},
};
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.6",
"version": "1.0.7",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "fabledcurator-extension",
"version": "1.0.6",
"version": "1.0.7",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
+2 -1
View File
@@ -132,8 +132,9 @@ async function exportPlatformCookies(key, card) {
if (r.error) showError(r.error);
else {
const n = r.cookieCount ?? null;
const verifiedSuffix = r.verified ? ' (verified ✓)' : '';
const msg = n !== null
? `${PLATFORMS[key].name}: ${n} cookies exported`
? `${PLATFORMS[key].name}: ${n} cookies exported${verifiedSuffix}`
: `${PLATFORMS[key].name}: token exported`;
showSuccess(msg);
await loadPlatformStatus();
+14 -1
View File
@@ -9,7 +9,9 @@
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { onMounted, ref, watch } from 'vue'
import { useRoute } from 'vue-router'
import AppShell from './components/AppShell.vue'
import AppSnackbar from './components/AppSnackbar.vue'
import ImageViewer from './components/modal/ImageViewer.vue'
@@ -17,9 +19,20 @@ import { useModalStore } from './stores/modal.js'
const modal = useModalStore()
const snackbar = ref(null)
const route = useRoute()
onMounted(() => {
// Expose snackbar via a simple global so stores can call it without props.
window.__fcToast = (opts) => snackbar.value?.open(opts)
})
// Audit 2026-06-02: the modal is an overlay, not a page. When the
// route changes (RouterLink inside the modal, history back/forward,
// programmatic push from any view), close the modal so it doesn't
// hover over a different route. Watching route.name (not the path)
// keeps within-route nav like /artist/foo → /artist/bar from
// dismissing the modal mid-browse.
watch(() => route.name, () => {
if (modal.isOpen) modal.close()
})
</script>
@@ -11,9 +11,6 @@
</template>
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../../stores/artist.js'
import { useModalStore } from '../../stores/modal.js'
import MasonryGrid from '../discovery/MasonryGrid.vue'
@@ -24,22 +21,9 @@ const props = defineProps({
const store = useArtistStore()
const modal = useModalStore()
const route = useRoute()
const router = useRouter()
onMounted(() => {
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage (id) {
router.push({ query: { ...route.query, image: id } })
modal.open(id)
}
</script>
@@ -124,10 +124,16 @@ async function onRetry() {
if (!props.event.source_id) return
retrying.value = true
try {
await sourcesStore.checkNow(props.event.source_id)
toast({
text: `Source check re-queued`, type: 'success',
})
const body = await sourcesStore.checkNow(props.event.source_id)
// Audit 2026-06-02: the previous handler unconditionally toasted
// "re-queued" even when the platform was in cooldown (202 +
// status='deferred'). Operator thought work was in flight when
// nothing was actually enqueued.
if (body?.status === 'deferred') {
toast({ text: 'Retry deferred — platform in cooldown', type: 'info' })
} else {
toast({ text: 'Source check re-queued', type: 'success' })
}
} catch (e) {
const isInFlight = !!e?.body?.download_event_id
toast({
@@ -19,25 +19,34 @@
>
Accept
</v-btn>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
<!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
Wrapping in a <span @click.stop> matches the TagPanel chip
fix — even though there's no parent click capture here today,
the wrap is harmless and keeps both kebabs on the same
pattern. Click bubbles from the v-btn opens menu via
activator props bubble continues to span stopPropagation
halts it. -->
<span class="fc-suggestion__menu-wrap" @click.stop>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</div>
</template>
@@ -90,6 +99,11 @@ const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
.fc-suggestion__accept :deep(.v-btn__content) {
font-size: 12px; letter-spacing: 0.02em;
}
.fc-suggestion__menu-wrap {
flex: 0 0 auto;
display: inline-flex;
align-items: center;
}
.fc-suggestion__menu {
flex: 0 0 auto;
}
+22 -13
View File
@@ -10,19 +10,27 @@
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
<!-- Operator-flagged 2026-06-02: the previous activator had
`@click.stop` directly on the v-icon, which silently
overrode Vuetify's onClick from `v-bind="mp"` — the menu
never opened. Now the v-icon receives the activator
onClick cleanly, and the wrapping span absorbs the
bubbled click so the chip's close button isn't tripped. -->
<span class="kebab-wrap" @click.stop>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical"
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename…</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</v-chip>
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
</div>
@@ -113,4 +121,5 @@ async function onRenamed() {
margin-bottom: 12px;
}
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
.kebab-wrap { display: inline-flex; align-items: center; }
</style>
@@ -35,7 +35,7 @@ import { computed } from 'vue'
const props = defineProps({
queues: { type: Object, default: null }, // store.queues
workers: { type: Object, default: null }, // store.workers
recentMinute: { type: Array, default: () => [] }, // store.recentMinute
recentRuns: { type: Array, default: () => [] }, // store.recentRuns
compact: { type: Boolean, default: false },
})
@@ -80,7 +80,7 @@ function activeCount(name) {
const recentByQueue = computed(() => {
const out = {}
for (const r of props.recentMinute) {
for (const r of props.recentRuns) {
if (!out[r.queue]) out[r.queue] = { ok: 0, err: 0 }
if (r.status === 'ok') out[r.queue].ok++
else if (r.status === 'error' || r.status === 'timeout') out[r.queue].err++
@@ -24,7 +24,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
compact
/>
</v-card-text>
@@ -47,7 +47,7 @@ function pollOnce() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
onMounted(() => {
@@ -14,7 +14,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
/>
</v-card-text>
</v-card>
@@ -202,7 +202,7 @@ function pollQueues() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
function pollFailures() {
if (document.hidden) return
@@ -25,6 +25,15 @@
<v-chip size="x-small" color="error" variant="flat" label class="fc-fail__count">
{{ s.consecutive_failures }}× failed
</v-chip>
<v-chip
v-if="s.error_type"
size="x-small" variant="outlined" label
:color="errorTypeColor(s.error_type)"
class="fc-fail__class"
:title="errorTypeHint(s.error_type)"
>
{{ s.error_type }}
</v-chip>
<span class="fc-fail__err" :title="s.last_error || ''">
{{ s.last_error || 'no error message recorded' }}
</span>
@@ -66,6 +75,42 @@ const open = ref(true)
// Per-row loading flag so the spinner lives on the row whose Logs
// button was clicked, not on every row.
const logLoadingIds = ref(new Set())
// Audit 2026-06-02: surface the ErrorType taxonomy as a colored chip
// next to the consecutive-failures count so operators can bulk-triage
// by error class. Color reflects "what to do next":
// warning (yellow) — auth/cookie issue: operator should rotate
// info (blue) — backend-paced (cooldown / rate limit / timeout)
// error (red) — likely terminal without operator intervention
const ERROR_TYPE_COLOR = {
auth_error: 'warning',
rate_limited: 'info',
timeout: 'info',
network_error: 'info',
not_found: 'error',
access_denied: 'error',
validation_failed: 'error',
unsupported_url: 'error',
http_error: 'error',
unknown_error: 'error',
partial: 'info',
tier_limited: 'info',
no_new_content: 'info',
}
const ERROR_TYPE_HINT = {
auth_error: 'Cookies likely expired — re-upload in Credentials.',
rate_limited: 'Platform-wide cooldown active. Will retry after it expires.',
timeout: 'Subprocess exceeded its time budget. Often retries cleanly.',
network_error: 'Transient network issue. Will retry on next tick.',
not_found: 'URL 404 — creator may have renamed or deleted.',
access_denied: 'Subscription tier may not grant this content.',
validation_failed: 'Downloaded files were quarantined by the validator.',
http_error: 'Generic HTTP error — see Logs.',
unsupported_url: 'gallery-dl does not support this URL pattern.',
unknown_error: 'Could not classify — see Logs.',
}
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
async function onViewLogs(s) {
if (logLoadingIds.value.has(s.id)) return
logLoadingIds.value = new Set(logLoadingIds.value).add(s.id)
@@ -115,6 +160,7 @@ async function onViewLogs(s) {
}
.fc-fail__artist { font-weight: 600; white-space: nowrap; }
.fc-fail__count { flex: 0 0 auto; }
.fc-fail__class { flex: 0 0 auto; }
.fc-fail__err {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.8rem;
@@ -412,6 +412,17 @@ function onArtistCreated(artist) {
async function onCheck(source) {
try {
const body = await store.checkNow(source.id)
// Audit 2026-06-02: /api/sources/<id>/check returns 202 with
// `{status:'deferred', cooldown_until}` when the platform is in
// cooldown — the previous handler treated this as success and
// toasted "event #undefined", masking that nothing was enqueued.
if (body?.status === 'deferred') {
toast({
text: 'Check deferred — platform in cooldown',
type: 'info',
})
return
}
toast({
text: `Check enqueued (event #${body.download_event_id})`,
type: 'success',
@@ -465,17 +476,23 @@ async function onBackfill(source) {
async function checkAll(group) {
let ok = 0
let conflict = 0
let deferred = 0
for (const s of group.sources) {
if (!s.enabled) continue
try {
await store.checkNow(s.id)
ok += 1
const body = await store.checkNow(s.id)
// Audit 2026-06-02: deferred (202 + cooldown_until) used to be
// counted as queued, inflating the success tally and hiding
// that the cooldown actually held the work back.
if (body?.status === 'deferred') deferred += 1
else ok += 1
} catch (e) {
if (e?.body?.download_event_id) conflict += 1
}
}
const parts = []
if (ok) parts.push(`${ok} queued`)
if (deferred) parts.push(`${deferred} deferred (cooldown)`)
if (conflict) parts.push(`${conflict} already running`)
toast({
text: parts.join(', ') || 'Nothing to check (no enabled sources)',
@@ -0,0 +1,52 @@
// Inflight-token guard for stores whose async loads can be re-triggered
// by rapid filter/navigation changes. Without this, late responses
// from a prior load overwrite the store with stale data
// (last-writer-wins, not request-order-wins). gallery.js had a
// hand-rolled `inflightId` of the same shape; this composable
// extracts it so every store can use the same pattern.
//
// Audit 2026-06-02 (workflow wf_bbe3fdb1-e62) found this missing in
// modal/suggestions/artist/downloads/directory/posts. The two most
// operator-impacting consequences: (1) modal tag mutations could
// land DELETE/POST on the wrong image when the user navigated mid-
// flight; (2) suggestions accept could push a tag to the wrong
// image AND add it to the allowlist.
//
// Usage:
// const inflight = useInflightToken()
//
// async function load() {
// const t = inflight.claim()
// const body = await api.get(...)
// if (!t.isCurrent()) return // stale — abort write
// items.value = body.items // safe to commit
// }
//
// function setFilter(f) {
// inflight.cancel() // any in-flight token is now stale
// filter.value = f
// load() // claims a fresh token
// }
//
// For multi-await flows (POST then GET, optimistic mutation then
// reconcile), check isCurrent() after EACH await — any intervening
// claim() or cancel() invalidates the prior token.
export function useInflightToken() {
let _seq = 0
let _current = 0
function claim() {
_current = ++_seq
const id = _current
return {
id,
isCurrent: () => _current === id,
}
}
function cancel() {
_current = ++_seq
}
return { claim, cancel }
}
+16 -3
View File
@@ -1,6 +1,7 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js'
import { usePostsStore } from './posts.js'
const PAGE = 60
@@ -15,12 +16,17 @@ export const useArtistStore = defineStore('artist', () => {
const error = ref(null)
const notFound = ref(false)
let started = false
// Rapid artist-to-artist navigation used to render the previous
// artist's overview/images briefly when the second load resolved
// after the third. Audit 2026-06-02.
const inflight = useInflightToken()
async function load (slug) {
// Cross-artist reset: clear this store AND the posts store so the new
// artist doesn't briefly render with the previous artist's content
// when the user is on the Posts tab. (Gallery tab uses this artist
// store's own images list — cleared above.)
inflight.cancel()
overview.value = null
images.value = []
nextCursor.value = null
@@ -29,14 +35,18 @@ export const useArtistStore = defineStore('artist', () => {
error.value = null
loading.value = true
usePostsStore().$reset?.()
const t = inflight.claim()
try {
overview.value = await api.get(`/api/artist/${encodeURIComponent(slug)}`)
const body = await api.get(`/api/artist/${encodeURIComponent(slug)}`)
if (!t.isCurrent()) return
overview.value = body
await loadMoreImages(slug)
} catch (e) {
if (!t.isCurrent()) return
if (e.status === 404) notFound.value = true
else error.value = e.message
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
@@ -44,19 +54,22 @@ export const useArtistStore = defineStore('artist', () => {
if (imagesLoading.value) return
if (started && nextCursor.value === null) return
imagesLoading.value = true
const t = inflight.claim()
try {
const params = { limit: PAGE }
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get(
`/api/artist/${encodeURIComponent(slug)}/images`, { params }
)
if (!t.isCurrent()) return
images.value.push(...body.images)
nextCursor.value = body.next_cursor
started = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e.message
} finally {
imagesLoading.value = false
if (t.isCurrent()) imagesLoading.value = false
}
}
+10 -1
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
const PAGE = 60
@@ -13,16 +14,23 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
const q = ref('')
const platform = ref(null)
let started = false
// Typed "alice" then "alice bob" used to drop the second fetch
// entirely (loading flag still true from the first), so the UI
// showed alice results while the input said "alice bob". Inflight
// token + reset() cancelling in-flight requests fixes both: the
// first response is discarded, the second is fetched. Audit 2026-06-02.
const inflight = useInflightToken()
async function loadMore() {
if (loading.value) return
if (started && nextCursor.value === null) return
const t = inflight.claim()
await run(async () => {
const params = { limit: PAGE }
if (q.value) params.q = q.value
if (platform.value) params.platform = platform.value
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/artists/directory', { params })
if (!t.isCurrent()) return
cards.value.push(...body.cards)
nextCursor.value = body.next_cursor
started = true
@@ -30,6 +38,7 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
}
async function reset() {
inflight.cancel()
cards.value = []
nextCursor.value = null
started = false
+5 -1
View File
@@ -23,7 +23,11 @@ export const useCredentialsStore = defineStore('credentials', () => {
const rec = await api.post('/api/credentials', {
body: { platform, credential_type, data },
})
byPlatform.value.delete(platform)
// Reflect the returned record immediately — the previous .delete()
// call left the card rendering "no credential" for the gap between
// upload completion and the caller's follow-up loadAll(). Audit
// 2026-06-02.
byPlatform.value.set(platform, rec)
return rec
}
+12
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const useDownloadsStore = defineStore('downloads', () => {
const api = useApi()
@@ -22,6 +23,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
// the "active now" panel always reflects what's happening regardless of
// how the operator has filtered the historical list below.
const activeEvents = ref([])
// Filter changes (applyFilter) and rapid pagination can interleave
// responses; without an inflight guard the late response from a
// prior filter overwrites the current view. Audit 2026-06-02.
const inflight = useInflightToken()
function _params(extra = {}) {
const out = { limit: 50, ...extra }
@@ -32,8 +37,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
}
async function loadFirst() {
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/downloads', { params: _params() })
if (!t.isCurrent()) return
events.value = body
cursor.value = body.length ? body[body.length - 1].id : null
hasMore.value = body.length === 50
@@ -42,8 +49,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
async function loadMore() {
if (!hasMore.value || cursor.value == null) return
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/downloads', { params: _params({ before: cursor.value }) })
if (!t.isCurrent()) return
events.value.push(...body)
cursor.value = body.length ? body[body.length - 1].id : cursor.value
hasMore.value = body.length === 50
@@ -71,6 +80,9 @@ export const useDownloadsStore = defineStore('downloads', () => {
}
async function applyFilter(patch) {
// Drop any in-flight loadFirst/loadMore from the previous filter
// so its late response doesn't overwrite this filter's results.
inflight.cancel()
filter.value = { ...filter.value, ...patch }
await loadFirst()
}
+14 -4
View File
@@ -1,6 +1,7 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js'
// Operator-confirmed 2026-05-30: fetch PAGE-sized chunks instead of one
// 50-item request so items render as each batch lands. Total initial
@@ -22,9 +23,12 @@ export const useGalleryStore = defineStore('gallery', () => {
const timelineBuckets = ref([])
const timelineLoading = ref(false)
let inflightId = 0
// Was a hand-rolled inflightId counter; the audit-2026-06-02 fan-out
// moved this pattern into useInflightToken so every store can share it.
const inflight = useInflightToken()
async function loadInitial() {
inflight.cancel()
images.value = []
dateGroups.value = []
nextCursor.value = null
@@ -41,19 +45,19 @@ export const useGalleryStore = defineStore('gallery', () => {
if (loading.value) return
loading.value = true
error.value = null
const myId = ++inflightId
const t = inflight.claim()
try {
const params = { limit: PAGE, ...activeFilterParam() }
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/gallery/scroll', { params })
if (myId !== inflightId) return // stale response
if (!t.isCurrent()) return
images.value.push(...body.images)
dateGroups.value = mergeGroups(dateGroups.value, body.date_groups)
nextCursor.value = body.next_cursor
} catch (e) {
error.value = e.message
} finally {
if (myId === inflightId) loading.value = false
if (t.isCurrent()) loading.value = false
}
}
@@ -68,8 +72,14 @@ export const useGalleryStore = defineStore('gallery', () => {
}
async function jumpTo(year, month) {
// Rapid timeline-jump clicks need the same race guard as
// loadMore — first jump's late body could clobber the second
// jump's already-applied state.
inflight.cancel()
const t = inflight.claim()
const params = { year, month, ...activeFilterParam() }
const body = await api.get('/api/gallery/jump', { params })
if (!t.isCurrent()) return
if (body.cursor) {
images.value = []
dateGroups.value = []
+75 -12
View File
@@ -3,6 +3,7 @@ import { toast } from '../utils/toast.js'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const useModalStore = defineStore('modal', () => {
const api = useApi()
@@ -10,6 +11,11 @@ export const useModalStore = defineStore('modal', () => {
const currentImageId = ref(null)
const current = ref(null)
const { loading, error, run } = useAsyncAction({ errorAs: 'message' })
// Tag mutations interpolate the image id into the URL after an
// await; without an inflight token, a fast prev/next can route the
// DELETE/POST to the wrong image AND the response to the wrong
// chip rail. Audit 2026-06-02.
const inflight = useInflightToken()
// Post-scoped cycle. When set, prev/next cycles within this array
// (used by PostCard's expanded-mosaic PostImageGrid clicks). When
@@ -19,6 +25,9 @@ export const useModalStore = defineStore('modal', () => {
const postImageIndex = ref(0)
async function open (id, opts = {}) {
// Cancel any in-flight tag mutation or reloadTags from the
// previous image so its late response can't apply to this one.
inflight.cancel()
currentImageId.value = id
current.value = null // cleared upfront so it stays null on error
// Update post-scoped state if caller passed it; otherwise clear so
@@ -31,12 +40,16 @@ export const useModalStore = defineStore('modal', () => {
postImageIds.value = null
postImageIndex.value = 0
}
const t = inflight.claim()
await run(async () => {
current.value = await api.get(`/api/gallery/image/${id}`)
const body = await api.get(`/api/gallery/image/${id}`)
if (!t.isCurrent()) return
current.value = body
})
}
async function close () {
inflight.cancel()
currentImageId.value = null
current.value = null
error.value = null
@@ -75,37 +88,87 @@ export const useModalStore = defineStore('modal', () => {
}
async function reloadTags () {
if (!currentImageId.value) return
const tags = await api.get(`/api/images/${currentImageId.value}/tags`)
if (current.value) current.value.tags = tags
// Capture image id at call-time. After an await, currentImageId
// may have advanced via prev/next navigation; without capture, the
// GET would target the new image and write its tags onto a chip
// rail the user didn't open. Audit 2026-06-02.
const imageId = currentImageId.value
if (!imageId) return
const t = inflight.claim()
const tags = await api.get(`/api/images/${imageId}/tags`)
if (!t.isCurrent()) return
// Only commit if the modal is still showing this image — guards
// against close() / nav clearing current between the await and now.
if (current.value && currentImageId.value === imageId) {
current.value.tags = tags
}
}
async function removeTag (tagId) {
if (!currentImageId.value) return
const imageId = currentImageId.value
if (!imageId) return
const prev = current.value.tags
// Optimistic UI: drop the chip immediately.
current.value.tags = current.value.tags.filter(t => t.id !== tagId)
// Split the two POSTs so a dismiss failure (secondary side-effect)
// doesn't roll back the successful DELETE — previously the catch
// unconditionally restored the chip rail even when only the
// dismiss had failed, so the UI lied until refresh. Audit 2026-06-02.
try {
await api.delete(`/api/images/${currentImageId.value}/tags/${tagId}`)
await api.post(`/api/images/${currentImageId.value}/suggestions/dismiss`, {
await api.delete(`/api/images/${imageId}/tags/${tagId}`)
} catch (e) {
// Real failure: roll back, surface, rethrow.
if (current.value && currentImageId.value === imageId) {
current.value.tags = prev
}
toast({ text: `Failed to remove tag: ${e.message}`, type: 'error' })
throw e
}
// DELETE landed. The dismiss is fire-and-best-effort — log on
// failure but DON'T roll back the chip rail; the tag is gone
// server-side regardless.
try {
await api.post(`/api/images/${imageId}/suggestions/dismiss`, {
body: { tag_id: tagId },
})
} catch (e) {
current.value.tags = prev
toast({ text: `Failed to remove tag: ${e.message}`, type: 'error' })
throw e
toast({
text: `Tag removed, but failed to dismiss suggestion: ${e.message}`,
type: 'warning',
})
}
}
async function addExistingTag (tagId) {
if (!currentImageId.value) return
await api.post(`/api/images/${currentImageId.value}/tags`, {
const imageId = currentImageId.value
if (!imageId) return
await api.post(`/api/images/${imageId}/tags`, {
body: { tag_id: tagId, source: 'manual' },
})
await reloadTags()
}
async function createAndAdd ({ name, kind, fandom_id = null }) {
// Capture imageId so the post-create reloadTags / addExistingTag
// flow stays bound to the image the user clicked on, even if
// they navigated during the /api/tags POST.
const imageId = currentImageId.value
if (!imageId) return
const tag = await api.post('/api/tags', { body: { name, kind, fandom_id } })
// Audit 2026-06-02: a kind='fandom' created here used to be
// invisible to FandomPicker until a full page reload — its load
// gates on fandomCache.length, so a non-empty cache skips the
// refetch and the new fandom never appears. Push it into the
// cache directly so the next open sees it.
if (kind === 'fandom') {
const { useTagStore } = await import('./tags.js')
const tagStore = useTagStore()
tagStore.fandomCache.push({
id: tag.id, name: tag.name, kind: 'fandom',
fandom_id: null, fandom_name: null, image_count: 0,
})
}
if (currentImageId.value !== imageId) return // navigated away
await addExistingTag(tag.id)
}
+23 -3
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const usePostsStore = defineStore('posts', () => {
const api = useApi()
@@ -19,6 +20,12 @@ export const usePostsStore = defineStore('posts', () => {
const doneOlder = ref(false)
const doneNewer = ref(false)
const anchorId = ref(null)
// loadInitial, loadMore, loadAround, loadOlder, loadNewer all share
// one `loading` flag and previously had no inflight guard. A filter
// change (loadInitial) racing a still-in-flight loadMore would
// append the prior filter's items into the new filter's feed.
// Audit 2026-06-02.
const inflight = useInflightToken()
function _qs() {
const q = {}
@@ -36,6 +43,7 @@ export const usePostsStore = defineStore('posts', () => {
}
async function loadInitial(newFilters) {
inflight.cancel()
filters.value = {
artist_id: newFilters?.artist_id ?? null,
platform: newFilters?.platform ?? null,
@@ -46,8 +54,10 @@ export const usePostsStore = defineStore('posts', () => {
async function loadMore() {
if (loading.value || done.value) return
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/posts', { params: _qs() })
if (!t.isCurrent()) return
items.value.push(...body.items)
cursor.value = body.next_cursor
if (body.next_cursor == null) done.value = true
@@ -79,6 +89,7 @@ export const usePostsStore = defineStore('posts', () => {
// posts feed; without this the older/newer scroll loaded unfiltered
// global posts instead of staying in the artist's stream).
async function loadAround(postId, newFilters) {
inflight.cancel()
filters.value = {
artist_id: newFilters?.artist_id ?? null,
platform: newFilters?.platform ?? null,
@@ -86,10 +97,12 @@ export const usePostsStore = defineStore('posts', () => {
loading.value = true
error.value = null
anchorId.value = null
const t = inflight.claim()
try {
const body = await api.get('/api/posts', {
params: _aroundParams({ around: postId }),
})
if (!t.isCurrent()) return
items.value = body.items
cursorOlder.value = body.cursor_older
cursorNewer.value = body.cursor_newer
@@ -97,47 +110,54 @@ export const usePostsStore = defineStore('posts', () => {
doneNewer.value = body.cursor_newer == null
anchorId.value = body.anchor_id
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
async function loadOlder() {
if (loading.value || doneOlder.value || cursorOlder.value == null) return
loading.value = true
const t = inflight.claim()
try {
const body = await api.get('/api/posts', {
params: _aroundParams({
cursor: cursorOlder.value, direction: 'older',
}),
})
if (!t.isCurrent()) return
items.value.push(...body.items)
cursorOlder.value = body.next_cursor
if (body.next_cursor == null) doneOlder.value = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
async function loadNewer() {
if (loading.value || doneNewer.value || cursorNewer.value == null) return
loading.value = true
const t = inflight.claim()
try {
const body = await api.get('/api/posts', {
params: _aroundParams({
cursor: cursorNewer.value, direction: 'newer',
}),
})
if (!t.isCurrent()) return
items.value.unshift(...body.items)
cursorNewer.value = body.next_cursor
if (body.next_cursor == null) doneNewer.value = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
+37 -6
View File
@@ -3,6 +3,7 @@ import { toast } from '../utils/toast.js'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
// Category display order: people first, general last.
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired — only
@@ -18,12 +19,25 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
const byCategory = ref({}) // { category: [suggestion, ...] }
const { loading, error, run } = useAsyncAction({ errorAs: 'message' })
let currentImageId = null
// Audit 2026-06-02: this store had no inflight guard — a late
// /suggestions response from a prior image could overwrite
// byCategory while currentImageId pointed at a new one, and
// accept() dereferenced currentImageId AFTER an awaited POST so
// the subsequent /suggestions/accept could apply A's chosen tag
// to image B (and push it to the allowlist). Both fixed below
// by capturing imageId at call-time and gating writes on the token.
const inflight = useInflightToken()
async function load(imageId) {
// Cancel any in-flight load from the previous image so its late
// response can't overwrite this image's byCategory.
inflight.cancel()
currentImageId = imageId
byCategory.value = {} // cleared upfront so it stays empty on error
const t = inflight.claim()
await run(async () => {
const body = await api.get(`/api/images/${imageId}/suggestions`)
if (!t.isCurrent()) return
byCategory.value = body.by_category || {}
})
}
@@ -35,6 +49,11 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function accept(suggestion) {
// Capture imageId so a mid-flight prev/next can't reroute the
// accept POST to a different image AND push the tag to that
// image's allowlist.
const imageId = currentImageId
if (imageId == null) return
// Raw tags (creates_new_tag) have no canonical_tag_id; the backend's
// accept endpoint needs a tag_id, so for raw tags we create the tag
// first via the existing /api/tags endpoint, then accept by id.
@@ -45,10 +64,14 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
})
tagId = created.id
}
await api.post(`/api/images/${currentImageId}/suggestions/accept`, {
await api.post(`/api/images/${imageId}/suggestions/accept`, {
body: { tag_id: tagId }
})
_drop(suggestion.category, s => s === suggestion)
// Only drop from THIS image's category list — if the user navigated,
// the new image has its own suggestions and this drop would corrupt them.
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
toast({
text: `Tagged: ${suggestion.display_name}`,
type: 'success'
@@ -56,14 +79,18 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function aliasAccept(suggestion, canonicalTagId) {
await api.post(`/api/images/${currentImageId}/suggestions/alias`, {
const imageId = currentImageId
if (imageId == null) return
await api.post(`/api/images/${imageId}/suggestions/alias`, {
body: {
alias_string: suggestion.display_name,
alias_category: suggestion.category,
canonical_tag_id: canonicalTagId
}
})
_drop(suggestion.category, s => s === suggestion)
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
toast({
text: `Aliased & tagged: ${suggestion.display_name}`,
type: 'success'
@@ -71,15 +98,19 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function dismiss(suggestion) {
const imageId = currentImageId
if (imageId == null) return
// Dismiss needs a tag_id; raw tags have none, so dismissing a raw
// suggestion just hides it client-side (nothing to persist a rejection
// against until the tag exists).
if (suggestion.canonical_tag_id != null) {
await api.post(`/api/images/${currentImageId}/suggestions/dismiss`, {
await api.post(`/api/images/${imageId}/suggestions/dismiss`, {
body: { tag_id: suggestion.canonical_tag_id }
})
}
_drop(suggestion.category, s => s === suggestion)
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
}
return {
+5 -5
View File
@@ -9,7 +9,7 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
// Live polled state.
const queues = ref(null) // { queues: {name: depth|null}, fetched_at }
const workers = ref(null) // { workers: {hostname: {...}}, fetched_at }
const recentMinute = ref([]) // last-60s rows (for Overview summary)
const recentRuns = ref([]) // last-60s rows (for Overview summary)
const failures = ref(null) // { recent, count_by_type, since }
// Paginated runs (Activity tab "All recent activity" pane).
@@ -45,14 +45,14 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
}
async function loadRecentMinute() {
async function loadRecentRuns() {
// Used by the Overview summary card: pull last 60s of runs to compute
// per-queue ok/err counts. One call covers all queues; UI groups.
try {
const body = await api.get('/api/system/activity/runs', {
params: { limit: 200 },
})
recentMinute.value = body.runs || []
recentRuns.value = body.runs || []
} catch (e) {
lastError.value = e.message
}
@@ -106,10 +106,10 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
return {
queues, workers, recentMinute, failures, summary,
queues, workers, recentRuns, failures, summary,
runs, runsCursor, runsHasMore, runsFilter,
loading, lastError,
loadQueues, loadWorkers, loadRecentMinute,
loadQueues, loadWorkers, loadRecentRuns,
loadRuns, loadFailures, loadSummary, setFilter,
}
})
+8 -1
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
const PAGE = 60
@@ -13,16 +14,21 @@ export const useTagDirectoryStore = defineStore('tagDirectory', () => {
const kind = ref(null)
const q = ref('')
let started = false
// Same shape as artistDirectory — rapid setQuery/setKind dropped
// the second fetch because loading was still true from the first.
// Audit 2026-06-02.
const inflight = useInflightToken()
async function loadMore() {
if (loading.value) return
if (started && nextCursor.value === null) return
const t = inflight.claim()
await run(async () => {
const params = { limit: PAGE }
if (kind.value) params.kind = kind.value
if (q.value) params.q = q.value
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/tags/directory', { params })
if (!t.isCurrent()) return
cards.value.push(...body.cards)
nextCursor.value = body.next_cursor
started = true
@@ -30,6 +36,7 @@ export const useTagDirectoryStore = defineStore('tagDirectory', () => {
}
async function reset() {
inflight.cancel()
cards.value = []
nextCursor.value = null
started = false
+2 -17
View File
@@ -24,7 +24,7 @@
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useRoute } from 'vue-router'
import { useGalleryStore } from '../stores/gallery.js'
import { useModalStore } from '../stores/modal.js'
import GalleryGrid from '../components/gallery/GalleryGrid.vue'
@@ -37,7 +37,6 @@ import { useGallerySelectionStore } from '../stores/gallerySelection.js'
const store = useGalleryStore()
const modal = useModalStore()
const sel = useGallerySelectionStore()
const router = useRouter()
const route = useRoute()
onMounted(async () => {
@@ -47,9 +46,6 @@ onMounted(async () => {
else if (!isNaN(tagId)) store.setTagFilter(tagId)
await store.loadInitial()
await store.loadTimeline()
// Open modal if URL has ?image=N
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.tag_id, (q) => {
@@ -64,19 +60,8 @@ watch(() => route.query.post_id, (q) => {
store.setPostFilter(isNaN(postId) ? null : postId)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage(id) {
router.push({ query: { ...route.query, image: id } })
}
function closeImage() {
const q = { ...route.query }
delete q.image
router.push({ query: q })
modal.open(id)
}
</script>
+9 -4
View File
@@ -27,12 +27,17 @@ describe('credentials store', () => {
expect(s.byPlatform.get('patreon').credential_type).toBe('cookies')
})
it('upload invalidates the cache', async () => {
it('upload reflects the returned record into the cache', async () => {
// The previous behavior was to delete the cache entry on upload, which
// briefly showed "no credential" until a follow-up loadAll() resolved.
// Audit 2026-06-02 corrected this: upload now stores the returned
// record directly so the card updates immediately.
const s = useCredentialsStore()
s.byPlatform.set('patreon', { platform: 'patreon' })
stubFetch(() => ({ status: 201, body: { platform: 'patreon', credential_type: 'cookies' } }))
s.byPlatform.set('patreon', { platform: 'patreon', credential_type: 'token' })
const fresh = { platform: 'patreon', credential_type: 'cookies' }
stubFetch(() => ({ status: 201, body: fresh }))
await s.upload('patreon', 'cookies', 'NETSCAPE_CONTENT')
expect(s.byPlatform.has('patreon')).toBe(false)
expect(s.byPlatform.get('patreon')).toEqual(fresh)
})
it('remove deletes and invalidates the cache', async () => {
+14 -7
View File
@@ -8,18 +8,25 @@ migration code paths.
import os
import pytest
import pytest_asyncio
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import (
# Audit 2026-06-02: CredentialCrypto now refuses to auto-generate a
# Fernet key without explicit opt-in (production safety against silent
# key regeneration on partial restore). The test environment never has
# a pre-seeded key file, so set the bootstrap flag here before any
# create_app() / CredentialCrypto() import path fires.
os.environ.setdefault("CURATOR_BOOTSTRAP_NEW_KEY", "1")
import pytest # noqa: E402
import pytest_asyncio # noqa: E402
from sqlalchemy import create_engine # noqa: E402
from sqlalchemy.ext.asyncio import ( # noqa: E402
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import sessionmaker # noqa: E402
from backend.app import create_app
from backend.app.models import Base
from backend.app import create_app # noqa: E402
from backend.app.models import Base # noqa: E402
def _async_database_url() -> str:
+4 -3
View File
@@ -19,9 +19,10 @@ async def test_get_and_patch_settings(client):
resp = await client.get("/api/ml/settings")
assert resp.status_code == 200
body = await resp.get_json()
# Default lowered 0.95 → 0.50 on 2026-06-01 (alembic 0029) — 0.95
# hid most general suggestions in the view modal.
assert body["suggestion_threshold_general"] == pytest.approx(0.50)
# Default raised 0.50 → 0.70 on 2026-06-02 (alembic 0033) — 0.50
# was too noisy in practice. The 0.70 default keeps the rail
# signal-rich without hiding everything like the original 0.95.
assert body["suggestion_threshold_general"] == pytest.approx(0.70)
# Retired threshold columns must not appear in the payload.
assert "suggestion_threshold_artist" not in body
assert "suggestion_threshold_copyright" not in body
+25 -6
View File
@@ -14,7 +14,7 @@ pytestmark = pytest.mark.integration
def test_load_or_create_writes_key_with_mode_0600(tmp_path):
key_path = tmp_path / "secrets" / "credential_key.b64"
CredentialCrypto(key_path)
CredentialCrypto(key_path, bootstrap_ok=True)
assert key_path.exists()
mode = stat.S_IMODE(os.stat(key_path).st_mode)
assert mode == 0o600
@@ -25,9 +25,9 @@ def test_load_or_create_writes_key_with_mode_0600(tmp_path):
def test_load_existing_key_is_idempotent(tmp_path):
key_path = tmp_path / "credential_key.b64"
crypto1 = CredentialCrypto(key_path)
crypto1 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_first = key_path.read_bytes()
crypto2 = CredentialCrypto(key_path)
crypto2 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_second = key_path.read_bytes()
assert contents_after_first == contents_after_second
# And both crypto instances decrypt each other's ciphertext
@@ -36,7 +36,7 @@ def test_load_existing_key_is_idempotent(tmp_path):
def test_encrypt_decrypt_round_trip(tmp_path):
crypto = CredentialCrypto(tmp_path / "k")
crypto = CredentialCrypto(tmp_path / "k", bootstrap_ok=True)
plaintext = "domain.com\tTRUE\t/\tTRUE\t1700000000\tname\tvalue"
ct = crypto.encrypt(plaintext)
assert isinstance(ct, bytes)
@@ -45,8 +45,27 @@ def test_encrypt_decrypt_round_trip(tmp_path):
def test_decrypt_with_wrong_key_raises(tmp_path):
crypto_a = CredentialCrypto(tmp_path / "a")
crypto_b = CredentialCrypto(tmp_path / "b")
crypto_a = CredentialCrypto(tmp_path / "a", bootstrap_ok=True)
crypto_b = CredentialCrypto(tmp_path / "b", bootstrap_ok=True)
ct = crypto_a.encrypt("secret")
with pytest.raises(InvalidCredentialBlob):
crypto_b.decrypt(ct)
def test_missing_key_without_bootstrap_raises(tmp_path, monkeypatch):
"""Audit 2026-06-02: without explicit opt-in, a missing key file
is a fatal startup error — silent regeneration on partial restore
would make every existing Credential row undecryptable."""
from backend.app.services.credential_crypto import MissingCredentialKey
monkeypatch.delenv("CURATOR_BOOTSTRAP_NEW_KEY", raising=False)
with pytest.raises(MissingCredentialKey):
CredentialCrypto(tmp_path / "absent.b64")
def test_missing_key_with_env_var_bootstraps(tmp_path, monkeypatch):
"""The env var CURATOR_BOOTSTRAP_NEW_KEY=1 is the operator's
first-time-setup opt-in for auto-creating the key file."""
monkeypatch.setenv("CURATOR_BOOTSTRAP_NEW_KEY", "1")
key_path = tmp_path / "bootstrap.b64"
CredentialCrypto(key_path) # no bootstrap_ok kwarg — relies on env
assert key_path.exists()
+1 -1
View File
@@ -21,7 +21,7 @@ _NETSCAPE = (
@pytest.fixture
def crypto(tmp_path):
return CredentialCrypto(tmp_path / "k")
return CredentialCrypto(tmp_path / "k", bootstrap_ok=True)
@pytest.mark.asyncio
+6 -6
View File
@@ -130,7 +130,7 @@ async def test_download_source_attaches_written_files(
thumbnailer=Thumbnailer(images_root=images_root),
settings=sync_settings,
)
crypto = CredentialCrypto(tmp_path / "key.b64")
crypto = CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True)
cred_service = CredentialService(db, crypto)
svc = DownloadService(
@@ -405,7 +405,7 @@ async def test_backfill_decrements_after_run(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -446,7 +446,7 @@ async def test_backfill_auto_resets_on_clean_zero_files(
thumbnailer=Thumbnailer(images_root=tmp_path / "images"),
settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -484,7 +484,7 @@ async def test_tick_mode_does_not_touch_backfill_counter(
thumbnailer=Thumbnailer(images_root=tmp_path / "images"),
settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -529,7 +529,7 @@ async def test_partial_error_type_maps_to_ok_status(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
svc = DownloadService(
async_session=db, sync_session=db_sync,
gdl=fake_gdl, importer=importer, cred_service=cred_service,
@@ -582,7 +582,7 @@ async def test_download_enqueues_thumbnail_and_ml_per_attached_image(
session=db_sync, images_root=images_root, import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root), settings=sync_settings,
)
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64"))
cred_service = CredentialService(db, CredentialCrypto(tmp_path / "key.b64", bootstrap_ok=True))
# Capture the IDs that the orchestrator hands off to each Celery task.
# The .delay() shim runs inside DownloadService._phase3_persist (lazy
+6 -3
View File
@@ -39,8 +39,9 @@ async def test_threshold_filters_low_confidence_general(db):
await db.flush()
sl = await SuggestionService(db).for_image(img.id)
names = [s.display_name for s in sl.by_category.get("general", [])]
assert "sword" in names
assert "lowconf" not in names
# display_name is normalized (tag_name.normalize) before surfacing.
assert "Sword" in names
assert "Lowconf" not in names
@pytest.mark.asyncio
@@ -84,7 +85,9 @@ async def test_raw_tag_creates_new(db):
await db.flush()
sl = await SuggestionService(db).for_image(img.id)
chars = sl.by_category["character"]
assert chars[0].display_name == "brand_new_tag"
# display_name is the normalized Camie name (underscores -> spaces,
# title-cased), not the raw vocab key.
assert chars[0].display_name == "Brand New Tag"
assert chars[0].creates_new_tag is True
assert chars[0].canonical_tag_id is None
+53
View File
@@ -0,0 +1,53 @@
import pytest
from backend.app.services.ml.tag_name import normalize
@pytest.mark.parametrize(
"raw, expected",
[
# Rule 4: underscores -> spaces; rule 7: title case
("light_purple_hair", "Light Purple Hair"),
("no_pants", "No Pants"),
("year_2005", "Year 2005"),
# Single-word still title-cased
("sword", "Sword"),
# Rule 3: drop trailing _(disambiguator)
("uchiha_sasuke_(naruto)", "Uchiha Sasuke"),
("apple_(fruit)", "Apple"),
("kirby_(series)", "Kirby"),
# Repeated trailing disambig blocks
("foo_(bar)_(baz)", "Foo"),
# Rule 1: leading junk chars
("#unicus_(idolmaster)", "Unicus"),
(".52_gal_(splatoon)", "52 Gal"),
("+_+_smile_(emote)", "Smile"),
# Rule 5: space after colon
("nier:automata", "Nier: Automata"),
# Already-spaced colon left alone
("nier: automata", "Nier: Automata"),
# Rule 6: hyphens preserved
("1000-nen_ikiteru_(vocaloid)", "1000-nen Ikiteru"),
("well-known_face", "Well-known Face"),
# Rule 2: wrapping quotes
('"pile_em_up"_(genshin_impact)', "Pile Em Up"),
("'foo_bar'", "Foo Bar"),
# Rule 8: emoticons -> None
(":/", None),
(";)", None),
("+_+", None),
("^_^", None),
# Empty / whitespace-only
("", None),
(" ", None),
("___", None),
# Apostrophe inside word — preserved, not title-cased
("it's_okay", "It's Okay"),
# Digit-only still surfaces (year tags)
("2005", "2005"),
# Multi-space collapse
("foo___bar", "Foo Bar"),
],
)
def test_normalize(raw, expected):
assert normalize(raw) == expected