Compare commits

...

63 Commits

Author SHA1 Message Date
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
Build images / build-ml (push) Successful in 3m28s
CI / intimp (push) Successful in 4m26s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen d3245f0c22 feat(ext): verify cookies in-browser before uploading (1.0.7)
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 40s
extension / lint (push) Successful in 36s
CI / intimp (push) Successful in 4m4s
CI / intapi (push) Successful in 8m2s
CI / intcore (push) Successful in 8m45s
extension / lint (pull_request) Successful in 14s
Pre-upload verify request: after capturing the live browser cookies,
hit a known authenticated endpoint with credentials:'include' from the
extension's background context. If the platform reports we're not
logged in, abort the upload so we don't overwrite FC-side credentials
with stale data.

- platforms.js: add `verify` config per cookie-auth platform
  - hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF
    _init_site_filters; same 401 path the operator hit 2026-06-03)
  - patreon: GET /api/current_user (clean 401 when logged out)
  - subscribestar, deviantart: no stable auth endpoint, skip verify
- cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}.
  ok=true/false/null tri-state — null = verify not configured, caller
  treats as "proceed".
- background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the
  upload; failures bubble up with the platform's name + reason.
- popup.js: success message now appends "(verified ✓)" when applicable.
- manifest + package.json: 1.0.6 → 1.0.7.
2026-06-03 14:04:45 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
2026-06-03 13:18:44 -04:00
bvandeusen e450145304 fix(ml): preserve digit-only tag names in normalize (year tags)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m42s
CI / intcore (push) Successful in 8m12s
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
2026-06-03 13:09:35 -04:00
bvandeusen a6e8d4b52e feat(ml): normalize Camie suggestion names to human-readable
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Failing after 24s
CI / frontend-build (push) Successful in 28s
CI / intimp (push) Successful in 3m57s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m22s
Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`,
`#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were
surfacing raw in SuggestionsPanel — and worse, the SAME raw string was
written to tag.name on Accept, polluting the DB with `underscored_lowercase`
names that don't match the operator's "Title Case" tag convention.

Add backend/app/services/ml/tag_name.py with a single normalize()
applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing
_(disambiguator) blocks iteratively, strip wrapping quotes, underscores
to spaces, space after colon, title-case each word's first char,
preserve hyphens/apostrophes/digits, drop entries with no letters).

Wire into SuggestionService.for_image:
- raw Camie key kept for alias_map lookup (alias rows are hand-curated
  against raw keys; don't disturb)
- display_name = normalize(raw); None means drop the candidate
- existing-tag lookup widened to case-insensitive match against BOTH
  raw and normalized forms so legacy underscore-named Tag rows accepted
  before this change still surface as "existing" not "+ new"
2026-06-03 13:00:08 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen f1860866de chore(modal): drop the ?image=N soft-compat from G5.4
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m38s
CI / intcore (push) Successful in 9m31s
Operator confirmed they have no existing ?image=N bookmarks to
preserve, so the soft-compat read on initial mount + router.replace
strip is dead weight. The modal is now purely a Pinia overlay — no
URL involvement on open OR initial mount.

Drops 33 lines plus the now-unused vue-router imports in both
ArtistGalleryTab (entire onMounted gone) and GalleryView (just the
?image=N block; the post_id/tag_id filter handling stays).
2026-06-02 19:24:09 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen b181d779fe fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 8m13s
CI / intcore (push) Successful in 7m6s
2026-06-02 18:49:24 -04:00
bvandeusen 0fbb19dc24 fix(modal): TagPanel kebab — apply the wrapping-span fix that the prior commit missed
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m59s
CI / intapi (push) Failing after 8m56s
CI / intcore (push) Successful in 10m0s
The previous commit (8326e54) updated the CSS but the structural
edit to the v-menu wrapper didn't take. Re-applying so the chip
kebab actually opens.
2026-06-02 18:48:53 -04:00
bvandeusen 8326e5447a fix(modal): kebab menus weren't opening (chip + suggestion rows)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Failing after 8m41s
CI / intcore (push) Successful in 10m0s
Operator-flagged 2026-06-02. Both kebab activators were broken:

- TagPanel chip kebab had `@click.stop` directly on the v-icon
  activator. In Vue 3, an explicit @click on the same element as
  `v-bind="props"` overrides the spread onClick — so Vuetify's
  activator handler never fired. Menu never opened.

- SuggestionItem kebab didn't have @click.stop, but for consistency
  and to make both kebabs follow the same shape, wrap it too.

The fix: each kebab is now wrapped in a `<span @click.stop>`. The
v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu;
the bubbling click then reaches the span where stopPropagation
absorbs it before it can affect a parent (the v-chip's close button
in TagPanel's case).
2026-06-02 18:48:32 -04:00
bvandeusen 1fd594baaf chore(ml): suggestion_threshold default 0.50 → 0.70
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Failing after 8m16s
CI / intcore (push) Successful in 8m43s
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).

Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.

Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
2026-06-02 18:38:12 -04:00
bvandeusen ecac6c4bda fix(audit-g5): centroid version DB-as-truth + modal as overlay
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 24s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 7m57s
CI / intcore (push) Successful in 8m46s
Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4).

G5.1 — Centroid version no longer drifts:

CentroidService now reads MLSettings.embedder_model_version (the DB
row tag_and_embed already writes from) for both the centroid model-
version stamp and the drift-detection comparison. Previously the
centroid sites imported MODEL_VERSION from env, so the version stamped
on centroids could disagree with the version stamped on the embeddings
they were built from. By construction those now match, so list_drifted
won't silently miss the env-vs-DB drift case.

embedder.py keeps MODEL_VERSION as an env-driven constant for the
actual model loader — that's a different concern (which weights are
loaded) from the version-stamp that gets persisted alongside data.

G5.4 — Modal is a Pinia-only overlay:

The previous URL↔modal sync in GalleryView and ArtistGalleryTab
leaked the modal across route changes (RouterLink to /artist/<slug>
left the modal mounted on top of the new route) and re-opened it
on history back/forward with stale ?image=N entries.

Now: openImage() just calls modal.open(id) — no URL push.
GalleryView's dead closeImage helper is deleted. A route.name
watcher in App.vue closes the modal whenever the route changes,
which auto-fixes RouterLink-in-modal and back/forward.

Backward-compat: ?image=N is still honored on initial mount as a
one-shot deep-link opener, then router.replace strips the query so
the URL doesn't re-trigger and no extra history entry is added.
Existing bookmarks / shared URLs keep working; new opens stay
Pinia-only.
2026-06-02 18:28:57 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 9f7261b9c0 fix(audit-g5c): set CURATOR_BOOTSTRAP_NEW_KEY=1 in conftest
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m50s
CI / intapi (push) Successful in 8m33s
CI / intcore (push) Successful in 9m7s
CredentialCrypto's safety check fires on create_app() instantiation
because the test environment has no pre-seeded Fernet key file. Set
the bootstrap env var before any test imports so the auto-create path
is allowed during tests.
2026-06-02 17:55:25 -04:00
bvandeusen f05aaa707b fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Failing after 26s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Failing after 8m10s
CI / intcore (push) Failing after 9m42s
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.

FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
  - warning (yellow) — operator action needed (auth_error)
  - info (blue)      — backend-paced (rate_limited / timeout /
                       network_error / partial / tier_limited)
  - error (red)      — likely terminal without intervention
                       (not_found / access_denied / validation_failed /
                        unsupported_url / http_error / unknown_error)

Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
2026-06-02 17:54:44 -04:00
bvandeusen 4df98171ab fix(audit-g5c): refuse silent Fernet key regeneration on partial restore
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Failing after 25s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Failing after 8m13s
CI / intcore (push) Failing after 8m48s
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.

Two opt-ins now needed for auto-creation:
  1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
  2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)

Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.

Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.

Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
2026-06-02 17:04:27 -04:00
bvandeusen 8d75ade1d5 fix(audit-g5b): race-poisoning in three find-or-create sites
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m29s
Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the
race under concurrent writers / recovery-sweep replays, poisoning the
outer transaction with an unrecoverable IntegrityError and crashing
the calling task. Same shape as the banked 2026-05-26 ImageProvenance
incident (see reference_scalar_one_or_none_duplicates memory).

Mirrors the importer._get_or_create pattern in all three: savepoint
via begin_nested + IntegrityError rollback to that savepoint + re-
SELECT to grab the row the other worker committed first.

- importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store
  is sha-addressed so both workers race to write the same on-disk path
  (shutil.copy2 + rename is idempotent), so no extra cleanup needed.

- TagService.find_or_create: partial uniqueness index on
  (name, kind, COALESCE(fandom_id, -1)). The previous docstring
  claimed "INSERT ... ON CONFLICT" but the implementation was
  SELECT-then-INSERT with no recovery.

- ArtistService.find_or_create: already had IntegrityError handling
  but did session.rollback() (unwinds the WHOLE transaction); now
  uses begin_nested + sp.rollback() so the surrounding request's
  progress isn't lost.
2026-06-02 17:02:12 -04:00
bvandeusen 75c63e1511 fix(audit-g5a): ruff isort — platforms after patreon_resolver
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 16:55:56 -04:00
bvandeusen 98673d4dca fix(audit-g5a): small architectural cleanups bundle
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m23s
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.

- download_service: replace hardcoded ('discord','pixiv') tuple with
  auth_type_for(platform) == 'token'. A 7th token-platform now picks
  up the right credential path without touching this site.

- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
  merge so the target's centroid reflects its new image set
  immediately. Daily list_drifted catches it within 24h, but eager
  recompute closes the suggestion-quality dip in the meantime.

- backfill_thumbnails added to beat_schedule (daily). The task
  docstring claimed periodic Beat but the entry was never registered,
  so the library got no self-healing thumbnail repair; only the
  manual admin-UI button fired it.

- modal.createAndAdd pushes a kind='fandom' tag into
  tagsStore.fandomCache so FandomPicker sees the new fandom on next
  open. Was: cache-gated load (length===0) skipped refetch, new
  fandom invisible until full page reload.

- cleanup cluster:
  - Drop .webp from cleanup_service.unlink — thumbnailer only writes
    .jpg/.png; the third tuple member was dead code.
  - Drop effective_date from /api/gallery/scroll response — no FE
    consumer reads it. Service still computes the attribute for
    timeline ordering; this just trims the JSON.
  - Rename store.recentMinute → store.recentRuns across the
    systemActivity store + three consumers (SystemActivitySummary,
    QueuesTable, SystemActivityTab). The data is the last 200 runs
    (not actually "last minute"), so the name lied.

NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
2026-06-02 16:46:46 -04:00
bvandeusen 89b48f8f35 Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 35s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m10s
2026-06-02 16:15:00 -04:00
bvandeusen 4bff1d8558 fix(audit-g4): status-enum miss batch
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m37s
CI / intcore (push) Successful in 8m18s
Five extension-miss findings from the 2026-06-02 audit, where a status
value was added on one side but a downstream consumer didn't pick it up.

- download_service._phase3_persist: explicit branches for
  ImportResult.status in ('failed','refreshed'). For 'failed' (archive
  probe crash from _import_archive), unlink the source file so the
  filesystem scanner doesn't re-import and re-crash on the same
  archive forever. 'refreshed' is currently unreachable from the
  download path (no deep=True) but matches the importer's documented
  contract; treat as 'attached'.

- gallery-dl backfill auto-complete now gates on dl_result.success +
  no error_type, not just return_code==0 + files_downloaded==0.
  VALIDATION_FAILED exits the subprocess with returncode=0 and
  files_downloaded=0 when every file was quarantined, matching the
  prior predicate exactly and zeroing the operator's armed backfill
  budget on the FIRST quarantine run instead of decrementing.

- attach_in_place archive dispatch now threads artist + source_row
  through _import_archive (and _import_media for archive members)
  and _supersede. The path-walk fallback (_resolve_artist) is still
  used by filesystem-import; the download path now binds
  ImageProvenance to the explicit subscription Source instead of
  rediscovering by (artist_id, platform).

- Three FE handlers now recognize status:'deferred' from
  /api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting
  "event #undefined"), SubscriptionsTab.checkAll (was counting
  deferred as queued), DownloadEventRow.onRetry (was saying
  "re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll
  which already had it.

- celery_signals._queue_for now maps backup/admin/library_audit
  prefixes to 'maintenance' (matching task_routes). TaskRun.queue
  was returning 'default' for those rows, so per-queue dashboard
  filters and per-queue threshold overrides (added in G3) silently
  missed them.
2026-06-02 16:04:59 -04:00
bvandeusen d60e0b9494 Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 36s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Successful in 7m39s
CI / intcore (push) Successful in 8m10s
2026-06-02 14:49:28 -04:00
bvandeusen e30f50e6fe fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m30s
CI / intapi (push) Successful in 7m30s
CI / intcore (push) Successful in 8m8s
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.

Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
  running/restoring past 7h (covers the 6.5h images-backup hard
  limit) to error. prune_backups docstring corrected — the FC-3i
  TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
  past 135 min (10-min buffer above scan_library_for_rule's 2h5m
  hard limit) to error. Previously a SIGKILL'd row blocked all
  future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
  stuck running >2h whose child tasks are all terminal (orphan case
  where the orchestrator crashed before the closing UPDATE). Uses
  the same EXISTS predicate /api/system/stats already had.

Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
  matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
  the model relationship.

time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).

start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.

/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.

DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
2026-06-02 14:30:46 -04:00
bvandeusen 9c27a2d3c7 Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 37s
Build images / build-web (push) Successful in 13s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 8m50s
2026-06-02 14:17:12 -04:00
bvandeusen e66987f092 fix(audit-g2): async race / state-leak across eight stores
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 34s
CI / intimp (push) Successful in 3m31s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m2s
Extracts gallery.js's hand-rolled inflightId pattern into a new
useInflightToken composable; adopts in every store that previously
had no guard against late-response overwrites or wrong-image URL
interpolation.

Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62)
flagged:

- modal.removeTag rolled back the chip rail unconditionally even
  when only the secondary dismiss POST had failed — UI lied until
  refresh. And all tag-mutation URLs interpolated currentImageId
  AFTER an await, so a fast prev/next could route DELETE/POST to
  the wrong image. Both fixed: split try/catch (dismiss failure
  surfaces a warning, doesn't roll back the delete); imageId
  captured at call-time and used in URLs throughout.

- suggestions.accept dereferenced currentImageId after the awaited
  POST /api/tags, so the subsequent /suggestions/accept could
  apply A's chosen tag to image B AND push it to B's allowlist.
  Fixed by capturing imageId at click-time + inflight guard on
  load().

Same shape across artist / downloads / artistDirectory /
tagDirectory / posts stores: rapid filter/nav changes used to
interleave responses (last-writer-wins). Now the late response is
discarded and the most-recent request wins. Filter-change-during-
search no longer drops the second fetch because the loading flag
was still true from the first.

gallery.js's inflightId removed in favor of the shared composable
so the pattern stays consistent.
2026-06-02 14:07:58 -04:00
bvandeusen 93e37681b7 Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 10s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m28s
CI / intcore (push) Successful in 8m3s
2026-06-02 13:29:17 -04:00
bvandeusen 80ef9bce48 fix(test): credentials.upload reflects record into cache (not invalidate)
CI / lint (push) Successful in 6s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 35s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 8m12s
Stale assertion pinned the buggy pre-G1.6 behavior (the test name
"upload invalidates the cache" literally describes the bug). The
audit's correction makes upload mirror the returned record into the
store so the card updates immediately — update the assertion to
match the corrected behavior.
2026-06-02 13:11:39 -04:00
bvandeusen 3898ce7be4 fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit
CI / lint (push) Successful in 5s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Failing after 27s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 8m6s
CI / intcore (push) Successful in 9m3s
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
  30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
  TimeoutExpired (matched the tick 870s/900s rationale). Backfill
  runs that hit the cap let the next tick continue via the archive.

- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
  without it cleanup_old_tasks' WHERE finished_at<cutoff never
  reaped orphan-swept rows. recover_stalled_task_runs also now sets
  duration_ms (matches celery_signals.finalize's millisecond math).

- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
  on Source creation, mirroring SourceService.create. Without it,
  Firefox quick-add on a creator with >20 unsynced posts walked the
  full feed until subprocess timeout. Renamed the constant from
  _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.

- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
  NO_NEW_CONTENT — the download path (line 712) already does, and
  TIER_LIMITED proves auth reached the post and was told it was
  tier-gated. Verify endpoint previously showed red on this and
  prompted operators to rotate working cookies.

- prune_unused_tags now runs a single DELETE with the NOT-IN
  predicate find_unused_tags uses, instead of SELECT-ids →
  DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
  would have surfaced on a tag explosion (>65k unused tags).

- credentials.upload() reflects the returned record into the store
  cache (`.set(platform, rec)`) instead of evicting it; previously
  the card briefly rendered "no credential" between upload and
  loadAll().
2026-06-02 11:24:05 -04:00
bvandeusen 64ca858574 Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 08:26:58 -04:00
bvandeusen 91be9df671 fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m43s
CI / intcore (push) Successful in 8m14s
attach_in_place mirrored only the media flow, so gallery-dl-downloaded
zips/PDFs/audio bounced back as `skipped+invalid_image`, which
download_service counted as an ingest error and flipped runs to
status="error" despite N successful image attaches. Lustria patreon
event #38998 (21 images + 1 OST zip) went red for exactly this reason.

Now attach_in_place dispatches the same way as import_one: archives →
_import_archive (extracts media members, captures archive as
PostAttachment), non-media → _capture_attachment. Download_service
accepts the new `attached` result and treats non-duplicate skips as
soft skips, not ingest errors.

Also: ShowcaseView always loadInitial() on mount, not just when the
store is empty — Pinia persists across navigations and operator wants
a fresh shuffle every time the showcase loads.
2026-06-02 08:16:13 -04:00
bvandeusen 412edec028 fix(showcase): don't exhaust on all-dupe batches; retry up to 8x
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 23s
CI / intimp (push) Successful in 3m25s
CI / intapi (push) Successful in 7m41s
CI / intcore (push) Successful in 8m14s
/api/showcase returns a random sample; after enough scrolling, an
unlucky 3-item batch can be entirely in the `seen` Set, which was
flipping `exhausted=true` and surfacing "End." mid-scroll. The
showcase is endless by intent — only a genuinely empty API response
(library has 0 images) should mark it exhausted. Tiny-library
fallback: cap retries at 8 to avoid spinning forever.
2026-06-01 23:37:21 -04:00
bvandeusen 937421485d fix(modal): refresh tag chips after accepting a suggestion
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 25s
CI / intimp (push) Successful in 3m37s
CI / intapi (push) Successful in 7m33s
CI / intcore (push) Successful in 8m13s
Operator-flagged 2026-06-01: clicking Accept on a suggestion added
the tag to the database but the modal's chip rail kept showing the
old set. Suggestion would disappear from the suggestions list
(suggestionsStore drops it from byCategory) and the toast would
confirm "Tagged: …", but visually the modal looked unchanged
until you closed and reopened it.

Root cause: useSuggestionsStore.accept() POSTs to the backend and
updates its own state, but never tells useModalStore that the
backing image's tag set has changed. The addExistingTag and
createAndAdd flows in TagPanel already call modal.reloadTags()
after applying — the suggestions path needed the same refresh.

Two-line fix in SuggestionsPanel: after store.accept() and
store.aliasAccept(), call modal.reloadTags() so TagPanel's
`modal.current?.tags` rebinds.
2026-06-01 23:02:57 -04:00
bvandeusen 9d0c0b7da8 Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 18s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m4s
2026-06-01 22:34:38 -04:00
bvandeusen 43b778aa04 fix(test): include 'scanned' in all backfill result assertions
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m11s
Same 'change shared shape, miss pinned tests' trap as cfa4fb4 — the
prior commit updated only test_backfill_mixed_aggregate; three sibling
tests in the same file also pinned the old {enqueued,ok,regenerated}
shape without 'scanned' and CI bounced.
2026-06-01 22:09:02 -04:00
bvandeusen 9cbdb70e13 fix(thumbnails): surface backfill results + tighten validity check
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 24s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m54s
CI / intcore (push) Failing after 8m18s
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."

1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
   returned just `{celery_task_id}` and the admin card said
   "Enqueued." with no counts. There was no way to tell whether the
   scan found 0 candidates, 5000 candidates, or whether the worker
   even picked up the task. "Found nothing" was indistinguishable
   from a broken queue.

   Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
   shared by the Celery task and the API endpoint. The API now runs
   the scan in an executor and returns `{scanned, enqueued, ok,
   regenerated}`. The actual thumbnail generation work still goes
   to the thumbnail Celery queue per row via
   `generate_thumbnail.delay()` — the scan itself is fast
   (SELECT id+thumbnail_path + a file.stat() per row).

2. **`_thumb_is_valid` accepted header-only corrupt files.** The
   magic-byte check passed for any 8-byte file starting with a JPEG
   or PNG header, including empty/truncated/zero-pad files that
   browsers render as broken. Backfill counted these as `ok` and
   never regenerated.

   Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
   thumbnails are minimum ~2KB even on solid-color sources; header-
   only corrupt files top out around 12 bytes. 256 is well above
   the corrupt floor and well below any legitimate thumbnail.

Plus the admin card now shows the per-run counts instead of
"Enqueued.":
  Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
2026-06-01 21:57:29 -04:00
bvandeusen 8e4d252ae4 Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m46s
2026-06-01 21:52:42 -04:00
bvandeusen bd06794647 fix(downloads): enqueue thumbnail + ML tasks per attached image
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 21s
CI / intimp (push) Successful in 3m43s
CI / intapi (push) Successful in 7m53s
CI / intcore (push) Successful in 8m23s
Operator-flagged 2026-06-01: downloaded images stayed at
thumbnail_path=NULL until a periodic backfill sweep picked them up,
surfacing as broken-thumbnail tiles in the gallery for hours after
the download landed.

Importer.attach_in_place deliberately skips inline thumbnail
generation (importer.py:591-592) so the import queue stays moving —
the CALLING task is responsible for the enqueue. tasks/import_file.py
already did this (line 228-239). tasks/download.py / download_service
did not — every gallery-dl-attached image landed un-thumbnailed.

Fix in download_service._phase3_persist: after each
`attach_in_place` returning status in (imported, superseded), fan out
`generate_thumbnail.delay()` + `tag_and_embed.delay()` for each
image_id. Lazy import avoids circular-import risk between
download_service and the celery task modules that depend on it.

Mirrors the existing pattern verbatim — single source of truth for
"what fires after a successful attach" remains a comment in two
places (filesystem-import task, download orchestrator) rather than
a shared helper, because the contexts differ enough (sync session vs
async orchestrator) that abstracting would obscure more than it'd
share.

Test covers the happy-path with two attached files: both get the
thumbnail enqueue AND the ML enqueue, with image_ids drawn from
ImportResult (so future supersede-on-attach paths stay covered).
2026-06-01 21:39:17 -04:00
bvandeusen fdd3e01f56 Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 10s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Successful in 8m16s
2026-06-01 21:09:28 -04:00
bvandeusen f575cfb93b ux(failing-sources): visible row separators + clearer hover
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m15s
The zebra striping in 717b601 was too subtle — inside the tonal error
card, surface-tint contrast washed out and rows still looked uniform.
Operator-flagged the same issue twice.

Replace the surface-tint approach with hard visual structure:
- Bottom border on each row (1px white/0.08) — clearly delineated rows
  regardless of card background
- Hover darkens to black/0.25 — much more obvious than the previous
  error-tint hover, gives the eye a clear horizontal track
- Slightly more vertical padding (8px vs 6px) for tap-target comfort
- Drop the gap; borders are the separator now
2026-06-01 20:57:41 -04:00
bvandeusen c82fb308b6 Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main
Build images / build-ml (push) Failing after 2s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-web (push) Successful in 12s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m47s
CI / intapi (push) Successful in 7m52s
CI / intcore (push) Successful in 8m16s
2026-06-01 20:44:23 -04:00
bvandeusen 717b601c81 ux(failing-sources): zebra striping + hover highlight on rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 8m29s
CI / intcore (push) Successful in 8m58s
Operator-flagged 2026-06-01: with the failing-sources panel expanded,
all rows have the same flat low-opacity background, no visual track
from the artist name on the left to the Logs/Retry buttons on the
right. Hard to tell which row a button belongs to when scanning down
a list of 17.

Three pure-CSS changes (no DOM, no logic):
- 3px gap between rows (was 2px) — slightly more breathing room
- Even rows get a darker surface tint (zebra striping)
- Hover paints the whole row in a low-opacity error tint, so moving
  the cursor toward Logs/Retry lights up the whole horizontal band
  and the eye traces back to the artist name automatically
2026-06-01 20:14:48 -04:00
bvandeusen cfa4fb4084 fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 28s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m32s
`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial
value (0) when checking that the override works. Now that create() pre-arms
enabled sources to 3, the assertion is stale — the test was already
verifying the override path, the pre-assertion just decorated it.

Drop the pre-assertion; keep the override check. Other tests use raw
Source(...) constructors (default to 0 via the column server_default),
not SourceService.create(), so they're unaffected.
2026-06-01 20:12:55 -04:00
bvandeusen 2aa2002f22 feat(subscriptions): newly added enabled sources start in backfill mode
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Failing after 8m23s
A freshly created subscription has no gallery-dl archive yet, so the
first poll in tick mode would walk forever — exit:20 doesn't trip
until 20 contiguous archive-hits, and there are none. The wall-clock
cap kicks in mid-walk, and the partial-success classifier (plan #544)
gracefully labels it status=ok, but the operator still wonders why
their new subscription isn't grabbing the back-catalog.

Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`.
Same default as the manual "Deep scan" button, same auto-decrement
and auto-reset rules — once the queue drains (clean exit + zero new
files) or the budget runs out, tick mode resumes naturally.

Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'`
that arrive disabled = False, or operator-added sources that start
disabled deliberately) skip the pre-arm — they're never polled, no
budget to waste.
2026-06-01 20:02:19 -04:00
bvandeusen 66ff671f09 fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m58s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m19s
Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.

gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.

Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.

Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.
2026-06-01 19:00:50 -04:00
bvandeusen 19aece1fc4 feat(download): tick/backfill modes + partial-success classifier (plan #544)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m20s
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.

Two coupled changes, both operator-flagged 2026-06-01:

* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
  asks gallery-dl to exit after 20 contiguous archived items. Fresh
  subscriptions + new-content cases still walk normally; established
  subscription with zero new content exits in ~30s of HEAD requests
  instead of pegging the timeout. 20 (not 5) gives headroom against
  paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
  via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
  downloads use `skip: True` + 1800s timeout. Auto-decrements per run
  with early-reset to 0 when a clean run finds zero files (queue
  drained). N defaults to 3 — multiple runs give the system enough
  budget to finish a deep walk across timeout boundaries. New
  `POST /api/sources/{id}/backfill` arms the source; "Deep scan"
  button on each SourceRow (chip shows remaining count) wires it.

Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."

Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).

Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
2026-06-01 18:23:28 -04:00
bvandeusen c9089b1d03 fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 8m18s
CI / intcore (push) Successful in 8m48s
gallery_service._provenance_clause artist branch was correlating its bare
Post reference to the outer query's primary_post_id outer-join, so the
artist filter silently matched zero rows for images with no primary post.
Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner
FROM rather than treating it as a correlated outer table.

Five sidecar/import tests still asserted that a synthetic Source row
appears after a filesystem import. Alembic 0030 retired that behavior;
the Post sits null-source and the artist linkage lives on Post.artist_id.
Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent,
test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar,
and test_apply_sidecar_recovers_from_integrity_error to assert
post.source_id IS NULL + post.artist_id linkage instead.
2026-06-01 14:40:28 -04:00
bvandeusen 644d538bab fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 3m46s
CI / intapi (push) Successful in 8m0s
CI / intcore (push) Failing after 8m45s
2026-06-01 14:23:18 -04:00
bvandeusen ff35da4743 fix(lint): drop unused Source import after Post.artist_id refactor
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 21s
CI / intapi (push) Failing after 2m16s
CI / intimp (push) Failing after 2m16s
CI / intcore (push) Failing after 2m20s
2026-06-01 14:19:37 -04:00
bvandeusen 2f66de2928 feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 2m15s
CI / intapi (push) Failing after 2m18s
CI / intcore (push) Failing after 2m17s
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.
2026-06-01 14:17:52 -04:00
bvandeusen 8cf8d2ca4d Merge pull request 'Modal Esc/overflow polish, artist-scoped post scroll, failing-sources Logs button' (#43) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 2m45s
Build images / build-ml (push) Successful in 3m31s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m47s
2026-06-01 12:10:11 -04:00
bvandeusen 94e7d20792 feat(downloads): Logs button on failing-sources rollup rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m43s
CI / intcore (push) Successful in 8m10s
Operator-asked 2026-06-01: each row in the "X sources are failing"
rollup needs a button to surface the last run's stdout/stderr/error
without leaving the Downloads tab to find the matching event row.

Wired:
- `downloadsStore.loadLastForSource(sourceId)` two-steps via
  `GET /api/downloads?source_id=N&limit=1` (most recent event) then
  the existing `loadOne(id)` for the full detail payload. Returns
  null if no events exist (toast warns).
- `FailingSourcesCard` row: new text button `Logs` with
  `mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`,
  emits `view-logs` with the source record.
- `DownloadsTab.onViewFailingLogs` is the handler — same
  `DownloadDetailModal` instance the event-row clicks use.
2026-06-01 10:04:22 -04:00
bvandeusen fb605af959 fix(posts): anchored view scroll inherits artist_id/platform filters
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
Operator-flagged 2026-06-01: clicking a Provenance post title from the
view modal opens the post in the posts feed scoped to that artist
(`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll
loaded UNFILTERED global posts instead of staying in the artist's
stream. Root cause: the posts store's loadAround/loadOlder/loadNewer
sent only `{around, cursor, direction}` — never the `artist_id` /
`platform` filters. Backend `/api/posts` accepts them on every path
(post_feed_service.scroll filters via `Source.artist_id`); the
frontend just wasn't passing them.

Fix:
- `posts.js` `loadAround(postId, filters)` now takes the filter
  snapshot and stores it. `_aroundParams(extra)` mixes the active
  filters into around/cursor calls.
- `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` +
  `platform` from the route query.
2026-06-01 08:24:43 -04:00
bvandeusen 4c56cf121f fix(modal): Esc closes from tag input; Provenance cards scroll at ~2.5
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 29s
CI / intimp (push) Successful in 3m54s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m16s
Two operator-flagged UX gaps from 2026-06-01:

1. **Esc trapped inside the tag-entry field**
   The autofocused TagAutocomplete input made Esc-to-close unreachable
   because ImageViewer's keydown handler bailed early on
   `isTextEntry(ev.target)` for every key including Escape. Now Escape
   closes the modal from text inputs too — except when a Vuetify
   overlay is open (FandomPicker, autocomplete dropdown, suggestion
   3-dot menu), in which case that overlay's own Esc-handling fires
   instead of closing the whole modal mid-interaction. Detected via
   `.v-overlay--active`. Arrow keys still gate on isTextEntry so the
   tag input handles typing without navigating images.

2. **Provenance cards expanded the side panel unbounded**
   When an image had many ImageProvenance entries the cards stack
   pushed the Tags section below the fold. Wrapped the cards in a
   `.fc-prov__cards` container with max-height 270px (≈2.5 cards) and
   thin overflow scrollbar. Title stays anchored at top; Tags panel
   sits at a consistent position below regardless of provenance count.
2026-06-01 08:17:54 -04:00
bvandeusen b1d58bc3b8 Merge pull request 'fix(ci): POSIX-safe SHORT_SHA in build.yml' (#42) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 3m24s
CI / intimp (push) Successful in 3m22s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m5s
2026-06-01 08:03:17 -04:00
bvandeusen 9564d073b9 fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m26s
CI / intcore (push) Successful in 8m3s
`${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner
executes the step via dash/BusyBox sh and errored with
`Bad substitution` (act_runner workflow shell, observed on the
65386f0 main-push run). Switched to
`SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in
both shells. Both build-web and build-ml tag-determination steps
updated.
2026-06-01 07:53:33 -04:00
bvandeusen 65386f02a0 Merge pull request 'View modal batch: autofocus, suggestions UX, post-title click, retire copyright/artist' (#41) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 4s
Build images / build-web (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m17s
CI / intcore (push) Successful in 8m2s
2026-06-01 07:01:42 -04:00
bvandeusen f87a06a6bd feat(modal): post title is the primary click (artist-scoped) + suggestion rows look like buttons
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 16s
CI / intimp (push) Successful in 4m16s
CI / intapi (push) Successful in 8m16s
CI / intcore (push) Successful in 8m41s
Two operator-asked modal UX changes (Scribe plan #514):

1. **Post title click → artist-scoped posts feed**
   - The bold post title in ProvenancePanel is now the primary click
     target. Click navigates to /posts?post_id=N&artist_id=A; the
     PostsView already filters on `artist_id`, so the user lands in
     that creator's stream, not the global one.
   - The redundant "View post" link is removed. "Show description"
     stays as the only action link below the meta line.
   - Title is styled as a button-link: accent color, hover underline,
     focus ring for keyboard nav (family rule 24 — UI quality bar).

2. **Suggestion rows look like buttons**
   - SuggestionItem becomes a chip-card: visible border, hover/focus
     background, unified container for name + score + actions
     (operator's "nothing visual to unify the buttons to their object"
     complaint).
   - Accept is an explicit tonal pill button labeled "Accept"
     (operator's pick over whole-row-clickable). 3-dot menu retains
     alias/dismiss, now `variant="outlined"` so it reads as a button.
   - Score is a fixed-width monospace pill on the right.
   - "+ new" badge upgraded to a pill chip with accent border so
     it's visibly an annotation, not part of the tag name.
2026-06-01 02:21:59 -04:00
bvandeusen 5d284aae9f fix(test): unpin general-threshold test from old 0.95 default (alembic 0029)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m55s
CI / intcore (push) Successful in 8m46s
2026-06-01 02:18:54 -04:00
bvandeusen af7b5c95e9 feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Failing after 8m18s
Four coupled operator-asked changes to the view modal (Scribe plan #509):

1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
   in onMounted/nextTick so the caret is in the input the moment the
   modal renders. No click needed to start typing.

2. **General suggestions expanded by default** — SuggestionsPanel's
   general-category group now mounts with `:default-open="true"`.
   Operator can collapse if too noisy, but the v1 frame shows them.

3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
   suggestion_threshold_general default matches character. Alembic
   0029 also bumps the existing singleton row's value if it's still
   at the old 0.95. Operator can re-tune from Settings → ML.

4. **Retire `copyright` + `artist` as ML suggestion categories** —
   neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
   really existed as a copyright tag-kind). They were surfaced in the
   suggestions pipeline + threshold settings UI but had no follow-
   through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
   ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
   drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
   SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
   MLThresholdSliders rows.

Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.

Tests:
- test_surfaced_categories asserts {character, general}, excludes
  artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
  artist + copyright keys in the GET payload.
2026-06-01 02:08:10 -04:00
bvandeusen 667b05f14e Merge pull request 'Extension probe-and-add (v1.0.6) + per-commit image tags' (#40) from dev into main
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 22s
extension / lint (push) Successful in 8s
Build images / sign-extension (push) Successful in 1m51s
Build images / build-web (push) Failing after 4s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
2026-06-01 01:44:03 -04:00
bvandeusen 8de7ccd07d build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m15s
extension / lint (pull_request) Successful in 16s
Adds an immutable per-commit docker tag to every main-push build:
`git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`,
alongside the existing floating `:main` + `:latest`. Implements the
new family release-posture rule "Tags are milestones, not gates —
commit-SHA images are the rollback unit" so rollback to any commit
on main is `docker pull …:c-<sha>` with no release ceremony required.

Behavior change summary:
- main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>}
- tag-push (opt-in vYY.MM.DD only, no .N): unchanged
- safety-net dev: unchanged

No code changes; the rule is about how the tag list is constructed.
Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still
fire them when the operator wants a labeled checkpoint.
2026-06-01 01:28:55 -04:00
bvandeusen d65f0b2091 feat(extension): probe shows current state before click; v1.0.6
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 20s
extension / lint (push) Successful in 13s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m41s
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup):
"the add source/subscription button idea to the firefox extension so
it can tell me if a source/artist is added or not and offer an option
to add it if it isn't." Plan tracked in Scribe task #507.

## Backend

- `ExtensionService.probe(url)` — read-only resolution. Reuses
  `_derive` for platform+slug, then 2 SELECTs. Returns one of:
  - `source_match` (exact (artist, platform, url) Source exists)
  - `artist_match` (artist exists, this URL isn't a Source yet;
     collapses the sidecar-synthetic-only case from v26.06.01.0)
  - `new` (neither exists)
  - `unknown_platform` (URL didn't match any artist-page regex)
- `GET /api/extension/probe?url=...` route with `X-Extension-Key`
  auth posture matching `/quick-add-source`. Read-only, side-effect
  free.
- 6 backend tests in tests/test_api_extension.py covering each state
  + auth + invalid URL.

## Extension

- `api.js`: `probeSource(url)` mirroring `quickAddSource` shape.
- `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The
  latter strips the `/api` suffix from configured `apiUrl` (placeholder
  format per options.html) and opens `${base}/artist/{slug}` in a new
  tab via `browser.tabs.create`.
- `content-script.js`: probe-first render — on page-load and SPA
  navigation, asks the backend for the URL's state and renders the
  chip in the matching color/copy on FIRST paint instead of flashing
  generic "Add" and updating after. Click handler branches:
  `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing
  ADD_AS_SOURCE flow (then re-probes so the chip flips green
  immediately, no wait for next nav).
- `content-script.css`: three state-color modifiers
  (--new, --artist-match, --source-match) on the FC parchment-on-slate
  palette. Sage for already-added, amber for artist-exists, accent
  orange for new.

## Versioning

- `extension/manifest.json` + `extension/package.json` → 1.0.6.
  build.yml's sign-extension job will fire on push to main since no
  `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the
  regenerated AMO keys end-to-end.

## Behavior on the sidecar-synthetic case

Filesystem-imported "Dymkens"-style artist with only a sidecar
synthetic Source: probe returns `artist_match` (not `new`), so the
chip reads "+ Add Patreon source to Dymkens" rather than offering to
recreate the artist. Clicking adds the real Source; existing
`_source_for_sidecar` preference logic (v26.06.01.0) routes future
gallery-dl Posts to the real one.
2026-06-01 00:41:17 -04:00
119 changed files with 3554 additions and 614 deletions
+30 -12
View File
@@ -242,20 +242,32 @@ jobs:
id: tag
run: |
# Three trigger shapes:
# refs/tags/v… → tag-push: publish ONLY the immutable version
# tag (e.g. :v26.05.26.5). Don't touch :latest;
# that already got published by the main-push
# build for the merge commit.
# refs/heads/main → push to main (incl. PR merge commits):
# publish :main + :latest (floating).
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
# no `.N` per family release-posture rule).
# Publish ONLY the immutable version tag;
# don't touch :latest (the main-push build
# for the merge commit already did that).
# refs/heads/main → push to main: publish :main + :latest
# (floating) AND :c-<short_sha> (immutable
# per-commit rollback substrate, per family
# release-posture rule "Tags are milestones,
# not gates — commit-SHA images are the
# rollback unit"). Rollback to any commit
# becomes `docker pull …:c-<sha>` without a
# release ceremony.
# anything else → safety net; shouldn't fire given the `on:`
# config above (dev was dropped). Tag :dev to
# surface the unexpected run in the registry.
# config above. Tag :dev to surface the
# unexpected run in the registry.
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
fi
@@ -286,13 +298,19 @@ jobs:
id: tag
run: |
# Mirrors build-web's three-shape logic (tag-push / main-push /
# safety-net dev). The -ml image follows the same release cadence
# as the web image.
# safety-net dev) including the per-commit :c-<short_sha> tag
# on main-push per the family release-posture rule. The -ml
# image follows the same release cadence as the web image.
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
fi
@@ -0,0 +1,71 @@
"""drop artist + copyright ml thresholds; lower general default to 0.50
Revision ID: 0029
Revises: 0028
Create Date: 2026-06-01
Operator-flagged 2026-06-01: the view modal's Suggestions panel hides
most general-category predictions because the default threshold is
0.95. Lowering the default to 0.50 (matches character) so general
suggestions surface more aggressively; the value remains tunable in
Settings → ML.
Same change retires two ML suggestion categories whose Tag.kind
surfaces are unused:
- `artist`: retired in FC-2d-vii-c — artist identity is acquisition-
derived (image_record.artist_id), never ML-inferred. The threshold
column was a leftover from before that retirement.
- `copyright`: retired 2026-06-01 — the app uses `fandom` for the
franchise/copyright concept (per TagsView.vue's doc comment); no
Tag rows of kind=copyright exist, and the threshold column never
fed anything user-visible.
Both columns are dropped from ml_settings; the existing row's
suggestion_threshold_general value is bumped from 0.95 to 0.50 iff
it's still at the old default, so deployed installs pick up the new
UX without overriding any operator tuning.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0029"
down_revision: Union[str, None] = "0028"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Bump the general threshold for installs still at the old default.
op.execute(text(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.95"
))
op.drop_column("ml_settings", "suggestion_threshold_artist")
op.drop_column("ml_settings", "suggestion_threshold_copyright")
def downgrade() -> None:
# Restore the columns with their prior defaults. The bump from
# 0.95 → 0.50 isn't reversible without remembering whether the
# operator had explicitly set 0.95 (unlikely — that was just the
# default) so we leave the current general value as-is.
from sqlalchemy import Column, Float
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_artist",
Float, nullable=False, server_default="0.30",
),
)
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_copyright",
Float, nullable=False, server_default="0.50",
),
)
@@ -0,0 +1,145 @@
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
Revision ID: 0030
Revises: 0029
Create Date: 2026-06-01
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
with enabled=false) was technically correct but misled the operator
into thinking they had phantom subscriptions. The synthetics existed
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
content with no real subscription.
This migration makes the data model honest:
1. **Post gets a denormalized `artist_id` column** so artist filters
work without traversing `Post → Source.artist_id`. Backfilled from
the existing Source linkage, then NOT NULL'd.
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
NULL`. Deleting a Source detaches its Posts instead of destroying
imported content (semantically: subscription ends, archive stays).
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
semantic change.
4. **Sidecar synthetic Sources are deleted** — first NULL out the
FKs from Post + ImageProvenance pointing at them (so the implicit
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
so no events exist for them.
Uniqueness handling: the existing `uq_post_source_external_id`
(source_id, external_post_id) keeps working for source-bound Posts
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
it). A second partial unique index covers the NULL-source case on
(artist_id, external_post_id) so filesystem-imported posts still
dedupe within an artist.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
revision: str = "0030"
down_revision: Union[str, None] = "0029"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Step 1: add Post.artist_id, initially nullable for backfill.
# FK naming follows the Base.metadata naming_convention
# (fk_<table>_<column>_<referred_table>) — alembic 0001 set this up.
op.add_column(
"post",
sa.Column("artist_id", sa.Integer, nullable=True),
)
op.create_foreign_key(
"fk_post_artist_id_artist", "post", "artist",
["artist_id"], ["id"], ondelete="CASCADE",
)
# Step 2: backfill from Source.artist_id (every existing Post has a
# Source today, so every row gets populated).
conn.execute(text("""
UPDATE post p
SET artist_id = s.artist_id
FROM source s
WHERE p.source_id = s.id AND p.artist_id IS NULL
"""))
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
remaining = conn.execute(text(
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
)).scalar_one()
if remaining:
raise RuntimeError(
f"alembic 0030: {remaining} post rows have no resolvable "
f"artist_id after backfill. Investigate before continuing."
)
# Step 3: enforce NOT NULL + add index for artist-filter queries.
op.alter_column("post", "artist_id", nullable=False)
op.create_index("ix_post_artist_id", "post", ["artist_id"])
# Step 4: relax post.source_id + flip FK to SET NULL. The original FK
# name from alembic 0001 is `fk_post_source_id_source` per the
# NAMING_CONVENTION in models/base.py.
op.alter_column("post", "source_id", nullable=True)
op.drop_constraint("fk_post_source_id_source", "post", type_="foreignkey")
op.create_foreign_key(
"fk_post_source_id_source", "post", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
op.alter_column("image_provenance", "source_id", nullable=True)
op.drop_constraint(
"fk_image_provenance_source_id_source", "image_provenance",
type_="foreignkey",
)
op.create_foreign_key(
"fk_image_provenance_source_id_source", "image_provenance", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 6: partial unique index on (artist_id, external_post_id) for
# NULL-source Posts. The existing uq_post_source_external_id keeps
# guarding source-bound rows; NULL-source rows now dedupe within
# an artist.
op.execute(
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
"ON post (artist_id, external_post_id) "
"WHERE source_id IS NULL"
)
# Step 7: retire sidecar synthetic Sources. NULL out the references
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
# being explicit makes the intent clear). Then delete the synthetic
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
# with the source — synthetics have enabled=false so there shouldn't
# be any in practice.
conn.execute(text("""
UPDATE post
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
conn.execute(text("""
UPDATE image_provenance
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
deleted = conn.execute(text(
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
)).rowcount
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
def downgrade() -> None:
# Lossy migration — the deleted sidecar synthetics can't be
# restored from the orphan post.source_id / image_provenance.source_id
# values, and the partial unique index encodes a constraint that
# NULL-source Posts may now exist. No safe downgrade.
pass
@@ -0,0 +1,45 @@
"""source.backfill_runs_remaining: sticky deep-scan mode
Revision ID: 0031
Revises: 0030
Create Date: 2026-06-01
Tick vs backfill mode for subscription downloads. When
`backfill_runs_remaining > 0`, the next N download runs use
`skip: True` + 30-min timeout (walk full history). When 0, runs use
`skip: "exit:20"` + 14.5-min timeout (catch-up mode, exits early once
20 contiguous archived items are seen).
Operator-flagged 2026-06-01 (Knuxy run #38887): a creator with ~550
archived posts saturates the 870s catch-up timeout even when there is
no new content, because gallery-dl's default `skip: True` keeps walking.
Tick mode short-circuits that; backfill mode is the explicit opt-in for
deep history scans.
Default 0 (all existing subscriptions start in tick mode).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0031"
down_revision: Union[str, None] = "0030"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column(
"backfill_runs_remaining",
sa.Integer,
nullable=False,
server_default="0",
),
)
def downgrade() -> None:
op.drop_column("source", "backfill_runs_remaining")
@@ -0,0 +1,41 @@
"""source.error_type: surface ErrorType taxonomy in FailingSourcesCard
Revision ID: 0032
Revises: 0031
Create Date: 2026-06-02
Audit 2026-06-02: the backend computes 13 ErrorType categories (auth_error,
rate_limited, not_found, access_denied, validation_failed, etc.) and
stamps each one on DownloadEvent.metadata, but the Source row only carried
the free-text last_error. Operators couldn't bulk-triage failing sources
("all auth_error → rotate cookies, all rate_limited → just wait") without
opening Logs per row.
This column receives the last error_type from _update_source_health
and gets cleared on a successful run. Nullable + indexed so the failing-
sources rollup can filter/group cheaply.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0032"
down_revision: Union[str, None] = "0031"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column("error_type", sa.String(length=32), nullable=True),
)
op.create_index(
"ix_source_error_type", "source", ["error_type"],
)
def downgrade() -> None:
op.drop_index("ix_source_error_type", table_name="source")
op.drop_column("source", "error_type")
@@ -0,0 +1,48 @@
"""suggestion_threshold default 0.50 → 0.70
Revision ID: 0033
Revises: 0032
Create Date: 2026-06-02
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) is
too noisy in practice; raise to 0.70 for both suggestion categories.
Only conditionally updates singletons whose current value is still the
2026-06-01 default (0.50). Operators who deliberately tuned their row
to some other value (0.55, 0.65, 0.80, etc. via the Settings UI) keep
their pick — the migration only catches the unchanged-default case.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0033"
down_revision: Union[str, None] = "0032"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.70 "
"WHERE id = 1 AND suggestion_threshold_character = 0.50"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.70 "
"WHERE id = 1 AND suggestion_threshold_general = 0.50"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.50 "
"WHERE id = 1 AND suggestion_threshold_character = 0.70"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.70"
)
+18
View File
@@ -57,6 +57,24 @@ def _sha256(path: Path) -> str:
return h.hexdigest()
@extension_bp.route("/probe", methods=["GET"])
async def probe_source():
"""Read-only resolution of a creator-page URL: tells the extension
whether this URL is already a Source, is for an Artist that exists
but with a different URL, is brand new, or doesn't match any known
platform pattern. Drives the content-script chip's color/copy
BEFORE the operator clicks, so the button can show 'already added'
without requiring an add-attempt."""
url = (request.args.get("url") or "").strip()
if not url:
return _bad("invalid_body", detail="url query parameter is required")
async with get_session() as session:
if not await _ext_key_required(session):
return _bad("unauthorized", status=401)
result = await ExtensionService(session).probe(url)
return jsonify(result)
@extension_bp.route("/quick-add-source", methods=["POST"])
async def quick_add_source():
body = await request.get_json(silent=True)
-1
View File
@@ -43,7 +43,6 @@ async def scroll():
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+17 -1
View File
@@ -35,10 +35,26 @@ async def trigger_scan():
@import_admin_bp.route("/status", methods=["GET"])
async def status():
async with get_session() as session:
# Active batch = running batch that still has outstanding work.
# Plain "most recent running" picks freshly-created scans that
# enqueued zero new files and hides the older batch that's
# actually being processed. Mirrors the EXISTS predicate
# /api/system/stats already uses (api/settings.py:145-160).
# Audit 2026-06-02 — /api/import/status and /api/system/stats
# used to disagree on the active-batch predicate; the UI banner
# said "Scanning…" indefinitely while the stats card said idle.
active = (
await session.execute(
select(ImportBatch)
.where(ImportBatch.status == "running")
.where(
ImportBatch.status == "running",
select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists(),
)
.order_by(ImportBatch.started_at.desc())
.limit(1)
)
-4
View File
@@ -9,9 +9,7 @@ ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
_EDITABLE = (
"suggestion_threshold_artist",
"suggestion_threshold_character",
"suggestion_threshold_copyright",
"suggestion_threshold_general",
"centroid_similarity_threshold",
"min_reference_images",
@@ -28,9 +26,7 @@ async def get_settings():
).scalar_one()
return jsonify(
{
"suggestion_threshold_artist": s.suggestion_threshold_artist,
"suggestion_threshold_character": s.suggestion_threshold_character,
"suggestion_threshold_copyright": s.suggestion_threshold_copyright,
"suggestion_threshold_general": s.suggestion_threshold_general,
"centroid_similarity_threshold": s.centroid_similarity_threshold,
"min_reference_images": s.min_reference_images,
+25
View File
@@ -120,6 +120,31 @@ async def delete_source(source_id: int):
return "", 204
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
async def set_backfill(source_id: int):
"""Plan #544: arm a source for backfill mode for the next N download
runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
source dict. While backfill_runs_remaining > 0, downloads use
gallery-dl's full-walk config (skip: True + 30-min timeout) instead
of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
payload = await request.get_json(silent=True) or {}
runs = payload.get("runs", 3)
try:
runs = int(runs)
except (TypeError, ValueError):
return _bad("invalid_runs", detail="runs must be an integer")
async with get_session() as session:
try:
record = await SourceService(session).set_backfill_runs(
source_id, runs,
)
except LookupError:
return _bad("not_found", status=404)
except ValueError as exc:
return _bad("invalid_runs", detail=str(exc))
return jsonify(record.to_dict())
@sources_bp.route("/<int:source_id>/check", methods=["POST"])
async def check_source(source_id: int):
"""FC-3c: enqueue a download for this source.
+6
View File
@@ -245,6 +245,12 @@ async def merge_tag(source_id: int):
from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=result.target_id)
# Tag merge invalidates the target's centroid (the merged-in source
# tag's images now contribute to it). Daily list_drifted catches it
# within 24h, but eager recompute closes the suggestion-quality dip
# in the meantime. Audit 2026-06-02.
from ..tasks.ml import recompute_centroid
recompute_centroid.delay(result.target_id)
return jsonify(
{
"target": {
+18 -3
View File
@@ -1,5 +1,7 @@
"""Thumbnail admin API: backfill trigger."""
import asyncio
from quart import Blueprint, jsonify
thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@@ -7,7 +9,20 @@ thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@thumbnails_bp.route("/backfill", methods=["POST"])
async def trigger_backfill():
from ..tasks.thumbnail import backfill_thumbnails
"""Run the backfill scan synchronously, return the counts. The actual
thumbnail generation work is still off-loaded to the thumbnail Celery
queue via `generate_thumbnail.delay()` per missing row — so this
handler is fast even on a 100k-image library (a scan is just SELECT
id, thumbnail_path + a file.stat() per row, no heavy work).
r = backfill_thumbnails.delay()
return jsonify({"celery_task_id": r.id}), 202
Operator-flagged 2026-06-01: the previous fire-and-forget shape
returned `{celery_task_id}` only, so the admin UI had no idea whether
backfill found 0 or 5000 candidates — \"found nothing\" was
indistinguishable from \"the worker isn't picking up the task.\""""
from ..tasks.thumbnail import _run_backfill_scan
# Sync scan inside an executor so we don't block the event loop.
counts = await asyncio.get_running_loop().run_in_executor(
None, _run_backfill_scan,
)
return jsonify(counts), 200
+35
View File
@@ -105,6 +105,41 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
# Audit 2026-06-02 — three new per-entity recovery sweeps.
# Each runs every 5 min like the other recover_stalled_*
# sweeps; each is a no-op when nothing is stuck.
"recover-stalled-backup-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_backup_runs",
"schedule": 300.0,
},
"recover-stalled-library-audit-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
"schedule": 300.0,
},
"recover-stalled-import-batches": {
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
"schedule": 300.0,
},
# Audit 2026-06-02 — daily retention for two entities
# whose terminal rows otherwise accumulate forever.
"prune-library-audit-runs": {
"task": "backend.app.tasks.maintenance.prune_library_audit_runs",
"schedule": 86400.0,
},
"prune-import-batches": {
"task": "backend.app.tasks.maintenance.prune_import_batches",
"schedule": 86400.0,
},
# Audit 2026-06-02 — backfill_thumbnails's docstring claimed
# "periodic Beat" but the entry was never registered, so the
# library got no self-healing thumbnail repair; only the
# manual admin-UI button fired it. Daily cadence is gentle
# (the task is idempotent and only enqueues regen for rows
# whose stored thumbnails are missing or corrupt).
"backfill-thumbnails-daily": {
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
"schedule": 86400.0,
},
},
timezone="UTC",
)
+14 -2
View File
@@ -54,7 +54,14 @@ _INT32_MIN = -2_147_483_648
def _queue_for(task) -> str:
"""Reverse the task→queue routing from celery_app.task_routes.
Keep in sync if task_routes is reordered."""
Keep in sync if task_routes is reordered.
Audit 2026-06-02: backup/admin/library_audit prefixes were
missing here even though task_routes sent all three to
'maintenance'. The TaskRun.queue column then lied for those
rows (claimed 'default') so per-queue dashboard filters and
per-queue threshold overrides silently missed them.
"""
name = getattr(task, "name", "") or ""
if name.startswith("backend.app.tasks.import_file."):
return "import"
@@ -66,7 +73,12 @@ def _queue_for(task) -> str:
return "download"
if name.startswith("backend.app.tasks.scan."):
return "scan"
if name.startswith("backend.app.tasks.maintenance."):
if name.startswith((
"backend.app.tasks.maintenance.",
"backend.app.tasks.backup.",
"backend.app.tasks.admin.",
"backend.app.tasks.library_audit.",
)):
return "maintenance"
return "default"
+6 -2
View File
@@ -34,8 +34,12 @@ class ImageProvenance(Base):
post_id: Mapped[int] = mapped_column(
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
# Nullable since alembic 0030 — provenance rows for filesystem-imported
# content with no subscription have NULL source_id. FK ondelete SET
# NULL so deleting a Source detaches its provenance rows instead of
# destroying the linkage between image and post.
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
captured_at: Mapped[datetime] = mapped_column(
+6 -8
View File
@@ -15,17 +15,15 @@ class MLSettings(Base):
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
suggestion_threshold_artist: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30
)
suggestion_threshold_character: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
)
suggestion_threshold_copyright: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
# Default raised 0.50 → 0.70 on 2026-06-02 — operator-flagged 0.50
# surfaced too many low-confidence picks; 0.70 keeps the rail
# signal-rich while still surfacing more than the original 0.95
# which hid almost everything. Operator-tunable via Settings → ML.
suggestion_threshold_general: Mapped[float] = mapped_column(
Float, nullable=False, default=0.95
Float, nullable=False, default=0.70
)
centroid_similarity_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.55
+20 -4
View File
@@ -1,6 +1,9 @@
"""Post — provenance anchor for content downloaded from a Source.
"""Post — provenance anchor for one creator post (may contain many images).
A Post is one creator post; it may contain many images/videos.
`source_id` is nullable since alembic 0030 — filesystem-imported posts
with no live subscription have NULL source_id. `artist_id` is the
denormalized always-present link to the creator (added in 0030 so
artist-filter queries don't depend on the Source detour).
"""
from datetime import datetime
@@ -14,12 +17,25 @@ from .base import Base
class Post(Base):
__tablename__ = "post"
__table_args__ = (
# Source-bound dedup. Postgres treats NULL != NULL so rows
# with source_id IS NULL aren't deduped by this constraint;
# the partial unique index `uq_post_artist_external_id_null_source`
# (created in alembic 0030) covers that case via
# (artist_id, external_post_id).
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
# Denormalized; always equals source.artist_id when source_id is set
# (the importer is responsible for keeping them consistent on insert).
# Filter queries (artist detail, artist-scoped posts feed) use this
# directly instead of joining through Source.
artist_id: Mapped[int] = mapped_column(
ForeignKey("artist.id", ondelete="CASCADE"),
nullable=False, index=True,
)
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
+14
View File
@@ -26,7 +26,21 @@ class Source(Base):
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
# alembic 0032: last ErrorType category (auth_error, rate_limited,
# not_found, ...). Lets FailingSourcesCard surface the taxonomy as
# a colored chip so operators can bulk-triage by error class. Set
# by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# alembic 0031: sticky deep-scan budget. When > 0, the next N download
# runs use gallery-dl's full-walk config (skip: True + 1800s timeout);
# when 0, runs use tick mode (skip: "exit:20" + 870s, exits early once
# 20 contiguous archived items are seen). Auto-decrements per run, with
# an auto-reset to 0 on clean exit + zero downloads (queue drained).
backfill_runs_remaining: Mapped[int] = mapped_column(
Integer, nullable=False, default=0, server_default="0",
)
artist = relationship("Artist", back_populates="sources")
+24 -17
View File
@@ -58,13 +58,17 @@ class ArtistService:
)
).scalar_one()
# Posts under this artist that have at least one image attached.
# Use Post.artist_id (alembic 0030) for the artist filter; keep
# the ImageProvenance JOIN so date bounds reflect only image-
# bearing posts (matches the original semantic). NULL-source
# posts now surface too.
date_row = (
await self.session.execute(
select(func.min(Post.post_date), func.max(Post.post_date))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).first()
dmin, dmax = date_row if date_row else (None, None)
@@ -98,14 +102,14 @@ class ArtistService:
)
).all()
# Same Post.artist_id direct filter — counts NULL-source posts too.
month = func.date_trunc("month", Post.post_date).label("m")
activity = (
await self.session.execute(
select(month, func.count(func.distinct(ImageProvenance.image_record_id)))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(and_(Source.artist_id == aid, Post.post_date.isnot(None)))
.where(and_(Post.artist_id == aid, Post.post_date.isnot(None)))
.group_by(month)
.order_by(month)
)
@@ -114,9 +118,7 @@ class ArtistService:
post_count = (
await self.session.execute(
select(func.count(func.distinct(Post.id)))
.select_from(Post)
.join(Source, Source.id == Post.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).scalar_one()
@@ -206,27 +208,32 @@ class ArtistService:
)
async def find_or_create(self, name: str) -> tuple[Artist, bool]:
"""Return (artist, created). Slug-keyed; idempotent under races."""
"""Return (artist, created). Slug-keyed; idempotent under races.
Audit 2026-06-02: switched from session.rollback() to a
begin_nested savepoint + IntegrityError recovery so a lost
race doesn't unwind the calling request's surrounding work.
Mirrors importer._get_or_create.
"""
cleaned = (name or "").strip()
if not cleaned:
raise ValueError("artist name must not be empty")
slug = slugify(cleaned)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
select_existing = select(Artist).where(Artist.slug == slug)
existing = (await self.session.execute(select_existing)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
sp = await self.session.begin_nested()
try:
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
await self.session.flush()
await sp.commit()
except IntegrityError:
await self.session.rollback()
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
await sp.rollback()
existing = (await self.session.execute(select_existing)).scalar_one()
return existing, False
await self.session.commit()
return artist, True
+49 -15
View File
@@ -11,7 +11,7 @@ the one-and-done GS/IR migration tooling.)
"""
from __future__ import annotations
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
@@ -187,10 +187,14 @@ def unlink_image_files(
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
# Convention thumbs dir — try both extensions thumbnailer writes
# (.jpg for opaque, .png for alpha). `.webp` used to be in this
# tuple but the thumbnailer never writes it (operator-flagged in
# the 2026-06-02 audit) — keep the tuple aligned with what
# actually lands on disk.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
for ext in ("jpg", "png"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
@@ -354,18 +358,35 @@ def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
Implementation note: the previous SELECT-ids → DELETE-WHERE-IN
pattern was vulnerable to the psycopg 65535-parameter ceiling on
libraries with tag explosions. The live delete now runs a single
DELETE with the same NOT-IN predicate find_unused_tags uses, so
the row count scales without binding every id as a parameter.
Audit 2026-06-02.
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
sample_rows = find_unused_tags(session, limit=50)
sample = [t.name for t in sample_rows]
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
count = session.execute(
select(func.count())
.select_from(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
).scalar_one()
return {"count": count, "sample_names": sample}
result = session.execute(
Tag.__table__.delete()
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
)
session.commit()
return {"deleted": result.rowcount or 0, "sample_names": sample}
# Legacy tags FC no longer uses, in two shapes:
@@ -500,6 +521,9 @@ class ConfirmTokenMismatch(Exception):
_VALID_RULES = ("transparency", "single_color")
_AUDIT_GUARD_THRESHOLD_MINUTES = 135 # matches LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES
def start_audit_run(
session: Session, *, rule: str, params: dict[str, Any],
) -> int:
@@ -507,11 +531,21 @@ def start_audit_run(
scan_library_for_rule Celery task. Returns the new audit_id.
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
has status='running'. Operator must cancel or wait."""
has status='running' AND started recently. Audit 2026-06-02 made
the guard age-aware: a SIGKILL'd run leaves a row in 'running'
that the recovery sweep flips on its next pass (~5 min), but a
fresh start_audit_run between the SIGKILL and the sweep would
previously block forever. Past the threshold, treat the running
row as stale and let the sweep clean it up — the new run still
gets to start.
"""
if rule not in _VALID_RULES:
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
cutoff = datetime.now(UTC) - timedelta(minutes=_AUDIT_GUARD_THRESHOLD_MINUTES)
existing = session.execute(
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
select(LibraryAuditRun.id)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at >= cutoff)
).scalar_one_or_none()
if existing is not None:
raise AuditAlreadyRunning(existing)
+53 -11
View File
@@ -1,40 +1,82 @@
"""Fernet-based encryption for credential blobs.
The key is a single 32-byte value (urlsafe-base64-encoded; what
Fernet.generate_key produces) stored at a fixed path inside the
images/data root. Created on first boot if absent; mode 0600. No KDF
needed — the file contents are already maximum-entropy random bytes.
Fernet.generate_key produces) stored at /images/secrets/credential_key.b64
(mode 0600, parent dir 0700). The 2026-06-02 audit caught a silent
key-regeneration path: on a partial disaster restore where the DB was
restored but the secrets dir was lost, the old `_load_or_create_key`
would mint a fresh key with no log, producing a working-looking system
where every authenticated download failed AUTH_ERROR until the operator
re-uploaded every credential by hand. Now the constructor refuses to
auto-generate unless either:
Operator backup procedure must include this file alongside the rest
of /images/ — losing it makes existing encrypted_blob rows
undecryptable (recovery = delete the rows and re-upload).
* the caller explicitly passes `bootstrap_ok=True` (tests, scripts), or
* the env var `CURATOR_BOOTSTRAP_NEW_KEY=1` is set (operator opt-in
during first-time setup).
Otherwise it raises `MissingCredentialKey` so the app fails fast at
startup and the operator can restore the key file from backup.
Operator backup procedure must include /images/secrets/ alongside the
rest of /images/ — losing the key file makes existing encrypted_blob
rows undecryptable (recovery = delete the rows and re-upload).
"""
import logging
import os
from pathlib import Path
from cryptography.fernet import Fernet, InvalidToken
log = logging.getLogger(__name__)
_BOOTSTRAP_ENV_VAR = "CURATOR_BOOTSTRAP_NEW_KEY"
class InvalidCredentialBlob(Exception):
"""Raised when decryption fails (wrong key, tampered blob, …)."""
class MissingCredentialKey(Exception):
"""The Fernet key file is missing AND the caller hasn't opted in to
generating a new one. Audit 2026-06-02: prevents silent key
regeneration on partial DB-restored / secrets-lost deployments.
Set CURATOR_BOOTSTRAP_NEW_KEY=1 for first-time setup, or restore the
key file from backup."""
class CredentialCrypto:
"""Fernet encrypt/decrypt with an on-disk key file.
Instantiate with a path; the file is created on first access and
reused thereafter. Tests pass a tmp_path; production calls with
Instantiate with a path; the file is loaded if present, or created
if absent AND the caller has opted in (bootstrap_ok=True or
CURATOR_BOOTSTRAP_NEW_KEY=1 env var). Production sites:
`IMAGES_ROOT / "secrets" / "credential_key.b64"`.
"""
def __init__(self, key_path: Path):
def __init__(self, key_path: Path, *, bootstrap_ok: bool | None = None):
self._key_path = Path(key_path)
self._fernet = Fernet(self._load_or_create_key())
if bootstrap_ok is None:
bootstrap_ok = os.environ.get(_BOOTSTRAP_ENV_VAR) == "1"
self._fernet = Fernet(self._load_or_create_key(bootstrap_ok))
def _load_or_create_key(self) -> bytes:
def _load_or_create_key(self, bootstrap_ok: bool) -> bytes:
if self._key_path.exists():
return self._key_path.read_bytes()
if not bootstrap_ok:
raise MissingCredentialKey(
f"Fernet key file not found at {self._key_path}. "
f"For first-time setup, set {_BOOTSTRAP_ENV_VAR}=1. "
f"If this is a restored instance, restore the key file "
f"from backup — generating a new one would make every "
f"existing Credential row undecryptable."
)
log.warning(
"Generating NEW Fernet credential key at %s. Any existing "
"encrypted_blob rows in the DB will be undecryptable — "
"re-upload each credential after this completes.",
self._key_path,
)
parent = self._key_path.parent
parent.mkdir(parents=True, exist_ok=True)
os.chmod(parent, 0o700)
+138 -4
View File
@@ -25,9 +25,17 @@ from sqlalchemy.orm import joinedload
from ..models import Artist, DownloadEvent, Source
from .credential_service import CredentialService
from .gallery_dl import GalleryDLService, SourceConfig
from .gallery_dl import (
BACKFILL_SKIP_VALUE,
BACKFILL_TIMEOUT_SECONDS,
TICK_SKIP_VALUE,
ErrorType,
GalleryDLService,
SourceConfig,
)
from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__)
@@ -84,6 +92,18 @@ class DownloadService:
ctx = setup
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
# alembic 0031 / plan #544: derive skip_value + timeout from the
# source's backfill_runs_remaining counter. When > 0, walk the full
# post history (skip: True + 1800s); when 0, exit gallery-dl after
# 20 contiguous archived items (skip: "exit:20" + the default
# 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
skip_value: bool | str = BACKFILL_SKIP_VALUE
source_config.timeout = BACKFILL_TIMEOUT_SECONDS
else:
skip_value = TICK_SKIP_VALUE
effective_url = _effective_url(
ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
)
@@ -95,6 +115,7 @@ class DownloadService:
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
resolved_campaign_id: str | None = None
@@ -121,6 +142,7 @@ class DownloadService:
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
return await self._phase3_persist(
@@ -156,6 +178,13 @@ class DownloadService:
return {"status": "in_flight", "event_id": existing.id}
if existing and existing.status == "pending":
existing.status = "running"
# Reset started_at on the pending→running transition so the
# recovery sweep (DOWNLOAD_STALL_THRESHOLD_MINUTES, 30 min)
# measures from real start, not from enqueue. On heavy-queue
# days a freshly-promoted event whose original started_at
# predated the cutoff would otherwise get swept mid-flight,
# racing phase3's commit. Audit 2026-06-02.
existing.started_at = datetime.now(UTC)
await self.async_session.commit()
event_id = existing.id
else:
@@ -166,7 +195,12 @@ class DownloadService:
event_id = ev.id
artist = source.artist
if source.platform in ("discord", "pixiv"):
# Drive cookies-vs-token selection from the platform registry's
# auth_type so a new 7th token-platform automatically picks the
# right credential path. The hardcoded tuple here used to drift
# out of sync with credential_service's auth_type_for(). Audit
# 2026-06-02.
if auth_type_for(source.platform) == "token":
cookies_path = None
auth_token = await self.cred_service.get_token(source.platform)
else:
@@ -185,6 +219,7 @@ class DownloadService:
"config_overrides": dict(source.config_overrides or {}),
"cookies_path": cookies_path,
"auth_token": auth_token,
"backfill_runs_remaining": source.backfill_runs_remaining or 0,
}
async def _phase3_persist(
@@ -238,6 +273,40 @@ class DownloadService:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError:
pass
# Enqueue thumbnail + ML for newly-attached images, matching
# the filesystem-import path (tasks/import_file.py:228-239).
# Importer.attach_in_place deliberately skips inline thumb
# generation to keep the import queue moving; the calling
# task is responsible for the enqueue. Operator-flagged
# 2026-06-01: without this, every downloaded image stayed
# at thumbnail_path=NULL until a periodic backfill swept
# it up, surfacing as broken-thumbnail tiles in the gallery
# for hours after a download landed. Lazy import to avoid
# circular-import risk between this service and the
# tasks/* modules that import it.
from ..tasks.ml import tag_and_embed
from ..tasks.thumbnail import generate_thumbnail
ids = list(result.member_image_ids)
if result.image_id is not None and result.image_id not in ids:
ids.append(result.image_id)
for img_id in ids:
generate_thumbnail.delay(img_id)
tag_and_embed.delay(img_id)
elif result.status == "attached":
# Non-media or extracted archive captured as PostAttachment
# (FC-2d-iii). The canonical copy lives in the attachments
# store; the original download path is now redundant —
# mirror duplicate_hash cleanup so we don't keep two copies.
# Operator-flagged 2026-06-02 (Lustria OST zip).
import_summary["attached"] += 1
try:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError:
pass
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped" and result.skip_reason and result.skip_reason.value in (
"duplicate_hash", "duplicate_phash",
):
@@ -246,6 +315,29 @@ class DownloadService:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped":
# Soft skip (too_small, too_transparent, invalid_image) —
# the file just didn't qualify, not a download/ingest
# failure. Don't flag the run as error; the file stays
# on disk for operator inspection.
import_summary["skipped"] += 1
elif result.status == "failed":
# Hard failure (today only: archive probe crash/timeout).
# The original archive sits in /images/ as an orphan; the
# filesystem scanner would re-import and re-crash on the
# same file, so delete the source file and surface the
# error in import_summary. Audit 2026-06-02.
import_summary["errors"] += 1
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "refreshed":
# Currently unreachable from attach_in_place (the download
# path never runs in deep=True mode), but the importer's
# ImportResult contract enumerates it. Treat the same as
# 'attached' — work happened, no error. Audit 2026-06-02.
import_summary["attached"] += 1
else:
import_summary["errors"] += 1
@@ -259,12 +351,21 @@ class DownloadService:
run_stats["quarantined_count"] = dl_result.files_quarantined
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
status = "ok" if (dl_result.success and import_summary["errors"] == 0) else "error"
# Plan #544: PARTIAL means the run downloaded ≥1 file but the
# subprocess didn't finish in budget (typically wall-clock timeout
# mid-walk). Real work happened; the next tick continues via
# gallery-dl's archive. NOT a failure for status purposes.
if dl_result.success and import_summary["errors"] == 0:
status = "ok"
elif dl_result.error_type == ErrorType.PARTIAL and import_summary["errors"] == 0:
status = "ok"
else:
status = "error"
ev.status = status
ev.finished_at = datetime.now(UTC)
ev.files_count = import_summary["attached"]
ev.bytes_downloaded = bytes_downloaded
ev.error = dl_result.error_message if not dl_result.success else None
ev.error = dl_result.error_message if status == "error" else None
ev.metadata_ = {
"run_stats": run_stats,
"error_type": dl_result.error_type.value if dl_result.error_type else None,
@@ -279,6 +380,33 @@ class DownloadService:
source_id=ctx["source_id"], status=status, error_message=ev.error,
error_type=dl_result.error_type.value if dl_result.error_type else None,
)
# Plan #544: backfill lifecycle — auto-complete when a clean
# backfill run drained the queue (gallery-dl exited 0 + zero files
# downloaded means there was nothing to fetch); otherwise decrement
# the counter. Next tick falls back to tick mode once it hits 0.
#
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
# subprocess with return_code=0 and files_downloaded=0 (every
# file was quarantined), which used to match the auto-complete
# predicate exactly — zeroing the operator's armed budget on
# the FIRST quarantine run instead of decrementing. Require
# dl_result.success + no error_type so only genuinely-empty
# successful runs drain the counter.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
queue_drained = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
and dl_result.files_downloaded == 0
)
if queue_drained:
src.backfill_runs_remaining = 0
else:
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
await self.async_session.commit()
return event_id
@@ -306,9 +434,15 @@ class DownloadService:
if status == "ok":
source.consecutive_failures = 0
source.last_error = None
# alembic 0032 — clear the failure-class chip on success.
source.error_type = None
elif status == "error":
source.consecutive_failures = (source.consecutive_failures or 0) + 1
source.last_error = error_message
# alembic 0032 — stamp the failure-class so FailingSourcesCard
# can render a colored chip and operators can bulk-triage
# by error class without opening Logs per row.
source.error_type = error_type
if error_type == "rate_limited":
await set_platform_cooldown(self.async_session, source.platform)
elif status == "skipped":
+69
View File
@@ -16,6 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
from ..utils.slug import slugify
from .source_service import NEW_SOURCE_BACKFILL_RUNS
class UnknownPlatformError(Exception):
@@ -86,6 +87,67 @@ class ExtensionService:
"created_artist": created_artist,
}
async def probe(self, url: str) -> dict:
"""Read-only resolution of a creator-page URL against the FC DB.
Returns one of:
- {state: 'unknown_platform'} — URL didn't match any
platform's strict artist-page pattern
- {state: 'new', platform, slug} — would create both
artist and source on quick-add
- {state: 'artist_match', platform, slug, artist}
— artist exists, this
exact URL isn't a Source yet (collapses the sidecar-synthetic
case too — the synthetic anchor counts as an existing artist
row but not as a pollable Source for this URL)
- {state: 'source_match', platform, slug, artist, source}
— exact (artist, platform,
url) Source already exists
Side-effect-free: two SELECTs at most.
"""
try:
platform, raw_slug = self._derive(url)
except (UnknownPlatformError, InvalidUrlError):
return {"state": "unknown_platform"}
slug = slugify(raw_slug)
artist = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
return {"state": "new", "platform": platform, "slug": slug}
artist_payload = {"id": artist.id, "name": artist.name, "slug": artist.slug}
source = (await self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one_or_none()
if source is None:
return {
"state": "artist_match",
"platform": platform,
"slug": slug,
"artist": artist_payload,
}
return {
"state": "source_match",
"platform": platform,
"slug": slug,
"artist": artist_payload,
"source": {
"id": source.id,
"artist_id": source.artist_id,
"platform": source.platform,
"url": source.url,
"enabled": source.enabled,
},
}
def _derive(self, url: str) -> tuple[str, str]:
if not isinstance(url, str) or not url.strip():
raise InvalidUrlError("url is empty")
@@ -143,9 +205,16 @@ class ExtensionService:
return existing, False
sp = await self.session.begin_nested()
try:
# New subscription sources arm a few backfill runs so the
# first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built).
# Mirrors SourceService.create — without it, Firefox quick-
# add on a creator with >20 unsynced posts would surface
# as "check failed" with no diagnosis. Audit 2026-06-02.
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
)
self.session.add(src)
await self.session.flush()
+97 -5
View File
@@ -39,9 +39,40 @@ class ErrorType(StrEnum):
HTTP_ERROR = "http_error"
UNSUPPORTED_URL = "unsupported_url"
VALIDATION_FAILED = "validation_failed"
# Run made real progress (downloaded ≥1 file) but did not finish in the
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
# mapping classifies this as "ok" because the next tick continues.
PARTIAL = "partial"
UNKNOWN_ERROR = "unknown_error"
# Tick mode (routine cron polls): skip ≤20 contiguous already-archived
# items, then exit gallery-dl. Established subscription with zero new
# content exits in ~30s of HEAD requests instead of walking to the bottom
# of the post history (which can be hours for prolific creators). 20 (not
# 5) is operator-set headroom against any edge case where paywalled or
# otherwise-non-downloadable items might interleave with archived ones —
# 20 contiguous HEADs is still negligible.
TICK_SKIP_VALUE = "exit:20"
# Backfill mode (operator-triggered deep scan): walk the full history.
# Source.backfill_runs_remaining > 0 selects this mode; the longer
# timeout below absorbs creators with thousands of posts.
#
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# before Celery SIGKILLs the worker — same rationale as the tick
# default at line 74. The audit (2026-06-02) caught this at 1800,
# guaranteeing SIGKILL on any backfill that ran to its subprocess
# budget: stdout/stderr lost, backfill_runs_remaining never
# decrements, recovery sweep stamps generic "stranded" 30 min later.
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
# was added to prevent. backfill_runs_remaining=3 still gives ~58
# minutes of cumulative walk across three runs for prolific creators.
BACKFILL_SKIP_VALUE = True
BACKFILL_TIMEOUT_SECONDS = 1170
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
@@ -56,12 +87,19 @@ _DEFAULT_GDL_TIMEOUT_SECONDS = 870
@dataclass
class SourceConfig:
"""Per-source overrides loaded from Source.config_overrides JSON.
Note: the gallery-dl `skip` value (tick vs backfill, see TICK_SKIP_VALUE /
BACKFILL_SKIP_VALUE) is NOT carried here — it derives from the
Source.backfill_runs_remaining column at the download_service layer
and is passed to _build_config_for_source as `skip_value`. Same for
the per-run subprocess timeout.
"""
content_types: list[str] = field(default_factory=lambda: ["all"])
sleep: float | None = None
sleep_request: float | None = None
directory_pattern: str | None = None
filename_pattern: str | None = None
skip_existing: bool = True
save_metadata: bool = True
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
@@ -73,7 +111,6 @@ class SourceConfig:
sleep_request=data.get("sleep_request"),
directory_pattern=data.get("directory_pattern"),
filename_pattern=data.get("filename_pattern"),
skip_existing=data.get("skip_existing", True),
save_metadata=data.get("save_metadata", True),
timeout=data.get("timeout", _DEFAULT_GDL_TIMEOUT_SECONDS),
)
@@ -231,6 +268,25 @@ class GalleryDLService:
"part-directory": str(self._config_dir / "temp"),
"retries": 3,
"timeout": 120.0,
# Forward Patreon as Referer/Origin to yt-dlp when it
# fetches video manifests. Operator-flagged 2026-06-01
# (DaferQ patreon): video posts hosted on Mux carry a JWT
# playback restriction that checks Referer/Origin on every
# request — not just the token signature. gallery-dl's
# HEAD probe to stream.mux.com returns 200 (the token is
# valid), but yt-dlp's actual GET-with-Range to fetch the
# m3u8 manifest 403s because yt-dlp sends its own default
# Referer, which Mux's policy rejects. Forcing the right
# headers fixes the headers-only case; Mux IP-range
# restrictions are unfixable from here.
"ytdl": {
"raw-options": {
"http_headers": {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
},
},
},
},
"output": {"progress": True},
}
@@ -245,7 +301,18 @@ class GalleryDLService:
platform: str,
source_config: SourceConfig,
artist_slug: str,
skip_value: bool | str = BACKFILL_SKIP_VALUE,
) -> dict:
"""`skip_value` controls gallery-dl's archive-walk behavior:
- True (BACKFILL_SKIP_VALUE): walk full post history, skipping
archived items but continuing past them. Used in backfill mode.
- "exit:20" (TICK_SKIP_VALUE): exit gallery-dl after 20
contiguous archived items. Used in tick (routine catch-up)
mode for fast no-op syncs on creators with deep history.
- False: don't skip — redownload everything (not used in FC).
The caller (download_service) chooses based on
Source.backfill_runs_remaining.
"""
config = json.loads(json.dumps(self._get_default_config())) # deep copy
destination = str(self.images_root / artist_slug / platform)
@@ -255,7 +322,7 @@ class GalleryDLService:
config["extractor"]["sleep"] = source_config.sleep
if source_config.sleep_request is not None:
config["extractor"]["sleep-request"] = source_config.sleep_request
config["extractor"]["skip"] = source_config.skip_existing
config["extractor"]["skip"] = skip_value
if source_config.save_metadata:
config["extractor"]["postprocessors"] = [
@@ -394,6 +461,22 @@ class GalleryDLService:
f"Subscription tier does not grant access to {count} post{'s' if count != 1 else ''}",
)
# Partial-success: the subprocess exited non-zero (typically because
# the wall-clock timeout fired mid-walk), but it had downloaded ≥1
# file by then and no source-level error category fired. The work
# the run DID do is real; gallery-dl's archive will pick up where
# it left off on the next tick. Mapped to status="ok" downstream
# (download_service.py) so this doesn't flag the source as
# "needs attention." Operator-flagged 2026-06-01 after a Knuxy
# patreon run downloaded hundreds of files then ran red on timeout.
files_downloaded = self._count_downloaded_files(stdout)
if not has_actual_error and files_downloaded > 0:
return (
ErrorType.PARTIAL,
f"Downloaded {files_downloaded} file{'s' if files_downloaded != 1 else ''}; "
"run did not complete in budget — next tick will continue",
)
return ErrorType.UNKNOWN_ERROR, f"Unknown error (return code: {return_code})"
def _count_downloaded_files(self, stdout: str) -> int:
@@ -545,6 +628,7 @@ class GalleryDLService:
source_config: SourceConfig | None = None,
cookies_path: str | None = None,
auth_token: str | None = None,
skip_value: bool | str = BACKFILL_SKIP_VALUE,
) -> DownloadResult:
start_time = time.time()
started_at = datetime.now(UTC).isoformat()
@@ -552,7 +636,9 @@ class GalleryDLService:
if source_config is None:
source_config = SourceConfig()
config = self._build_config_for_source(platform, source_config, artist_slug)
config = self._build_config_for_source(
platform, source_config, artist_slug, skip_value=skip_value,
)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
@@ -769,7 +855,13 @@ class GalleryDLService:
),
)
etype, msg = self._categorize_error(proc.returncode, proc.stdout, proc.stderr)
if proc.returncode == 0 or etype == ErrorType.NO_NEW_CONTENT:
# TIER_LIMITED proves auth worked — gallery-dl reached the
# post, was told it's tier-gated. The download path treats
# this as success (line 712); verify must too, or operators
# rotate working cookies for no reason. Audit 2026-06-02.
if proc.returncode == 0 or etype in (
ErrorType.NO_NEW_CONTENT, ErrorType.TIER_LIMITED,
):
return True, "Credentials valid — the feed authenticated."
if etype == ErrorType.AUTH_ERROR:
return False, msg
+17 -3
View File
@@ -20,8 +20,9 @@ from datetime import datetime
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import aliased
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
@@ -133,10 +134,23 @@ def _provenance_clause(post_id, artist_id):
ImageProvenance.post_id == post_id,
)
if artist_id is not None:
# Use Post.artist_id (alembic 0030 denormalized column) instead
# of joining through ImageProvenance.source_id → Source.artist_id.
# The denormalization is the always-present linkage; the source
# path now drops NULL-source provenance rows (filesystem-imported
# content) which would otherwise vanish from artist-filtered
# gallery views.
# ALIAS Post: the gallery query outer-joins Post on
# ImageRecord.primary_post_id (`_outer_join_primary_post`).
# SQLAlchemy would otherwise correlate a bare `Post` reference
# in this EXISTS subquery to that outer Post (which is NULL for
# images with no primary post), and the filter would silently
# match nothing.
post_inner = aliased(Post)
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == Source.id,
Source.artist_id == artist_id,
ImageProvenance.post_id == post_inner.id,
post_inner.artist_id == artist_id,
)
return None
+143 -86
View File
@@ -212,10 +212,10 @@ class Importer:
which would lose the surrounding scan's progress — and re-run `stmt`
(scalar_one) to return the row the other worker created.
Centralizes the pattern shared by _find_or_create_source,
_source_for_sidecar, and _find_or_create_post. The plain
SELECT-then-INSERT version lost races under the 5-min recovery sweep
(operator-flagged 2026-05-26)."""
Centralizes the pattern shared by _find_or_create_source and
_find_or_create_post. The plain SELECT-then-INSERT version lost
races under the 5-min recovery sweep (operator-flagged
2026-05-26)."""
existing = self.session.execute(stmt).scalar_one_or_none()
if existing is not None:
return existing
@@ -258,47 +258,25 @@ class Importer:
lambda: Source(artist_id=artist_id, platform=platform, url=url),
)
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Sidecar-import Source resolver. Used by both filesystem imports
and gallery-dl downloads (both write sidecar JSON, both flow through
_apply_sidecar / _capture_attachment).
def _lookup_source_for_sidecar(
self, *, artist_id: int, platform: str,
) -> Source | None:
"""Find the real subscription Source for (artist, platform), or
None if no subscription exists.
Source represents a subscription feed (one per artist+platform — the
URL polled by the FC-3 downloader). The filesystem importer used to
call _find_or_create_source(url=sd.post_url), creating one Source
row per post URL — 100s of junk Sources per artist, all with
enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26; consolidated via alembic 0022.
Resolution order: prefer a real (non-sidecar) Source over a
synthetic anchor. When alembic 0022 ran, it may have rewritten
per-post Sources into `sidecar:<platform>:<slug>` synthetic
anchors. If the operator later added the real subscription, both
rows now coexist. A naive `ORDER BY id ASC LIMIT 1` lookup would
pick the older synthetic and silently attach every gallery-dl
download to the wrong Source — operator-flagged 2026-05-31 after
the Subscriptions UI surfaced the phantom anchors. Pick the real
one when one exists; fall back to the synthetic; only create a
new synthetic when nothing exists for (artist, platform).
Pre-alembic-0030 this method would CREATE a synthetic
`sidecar:<platform>:<slug>` Source when no real one existed
because `Post.source_id` was NOT NULL and the importer needed
something to attach Posts to. Alembic 0030 relaxed both
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
synthetic anchors are obsolete; the importer now leaves
source_id as None when no subscription exists for the (artist,
platform). Operator-asked 2026-06-01: synthetic Sources had
leaked into the Subscriptions UI as phantom subscriptions and
the operator wanted the data model to truthfully say "this
content has no live subscription."
"""
real_stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
~Source.url.like("sidecar:%"),
)
.order_by(Source.id.asc())
.limit(1)
)
real = self.session.execute(real_stmt).scalar_one_or_none()
if real is not None:
return real
any_stmt = (
stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
@@ -307,29 +285,37 @@ class Importer:
.order_by(Source.id.asc())
.limit(1)
)
return self._get_or_create(
any_stmt,
lambda: Source(
artist_id=artist_id,
platform=platform,
url=f"sidecar:{platform}:{artist_slug}",
enabled=False,
),
)
return self.session.execute(stmt).scalar_one_or_none()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
self, *, source_id: int | None, external_post_id: str,
artist_id: int,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
"""Race-safe find-or-create on `post`. Keyed by
(source_id, external_post_id) when source_id is set — the
`uq_post_source_external_id` constraint guards. For NULL-source
posts the existence check matches on (artist_id, external_post_id),
which the partial unique index `uq_post_artist_external_id_null_source`
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
pattern as the rest of the helpers."""
if source_id is not None:
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
else:
stmt = select(Post).where(
Post.source_id.is_(None),
Post.artist_id == artist_id,
Post.external_post_id == external_post_id,
)
return self._get_or_create(
stmt,
lambda: Post(source_id=source_id, external_post_id=external_post_id),
lambda: Post(
source_id=source_id,
artist_id=artist_id,
external_post_id=external_post_id,
),
)
def import_one(self, source: Path) -> ImportResult:
@@ -370,12 +356,14 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
)
epid = sd.external_post_id or sc.stem
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
def _capture_attachment(
@@ -386,10 +374,19 @@ class Importer:
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
sha = _sha256_of(source)
existing = self.session.execute(
select(PostAttachment).where(PostAttachment.sha256 == sha)
).scalar_one_or_none()
if existing is None:
select_existing = select(PostAttachment).where(PostAttachment.sha256 == sha)
existing = self.session.execute(select_existing).scalar_one_or_none()
if existing is not None:
self.session.commit()
return ImportResult(status="attached")
# Savepoint + IntegrityError recovery — PostAttachment.sha256 is
# UNIQUE, so two workers can both pass the SELECT and only the
# second INSERT fails. Without savepoint, the outer transaction
# poisons and the calling task crashes. attachments.store is
# sha-addressed so both workers race to write the same target
# path; shutil.copy2 + rename is idempotent. Audit 2026-06-02.
sp = self.session.begin_nested()
try:
stored = self.attachments.store(source, sha)
self.session.add(PostAttachment(
post_id=post.id if post else None,
@@ -402,10 +399,19 @@ class Importer:
size_bytes=source.stat().st_size,
))
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
# Lost the race — the other worker's row is canonical.
self.session.execute(select_existing).scalar_one()
self.session.commit()
return ImportResult(status="attached")
def _import_archive(self, source: Path) -> ImportResult:
def _import_archive(
self, source: Path, *,
artist: Artist | None = None,
source_row: Source | None = None,
) -> ImportResult:
# Layer-3 isolation: bomb-size guard + integrity test in a
# spawned child BEFORE extracting in this process. A
# decompression bomb or a native-lib crash on a malformed
@@ -413,6 +419,14 @@ class Importer:
# instead of OOMing/segfaulting the import worker. extract_archive
# is already fail-soft for plain exceptions, so this only adds
# the hard-crash protection.
#
# Audit 2026-06-02: optional artist/source_row kwargs let the
# download path thread its explicit subscription context
# through instead of having _resolve_artist re-derive from
# path-walk (which works by coincidence today because gallery-dl
# lays files out under /images/<artist_slug>/...). Filesystem
# import still calls bare _import_archive(source) and falls
# back to the path-walk derivation as before.
probe = safe_probe.probe_archive(source)
if not probe.ok:
if probe.crashed:
@@ -424,24 +438,28 @@ class Importer:
# still preserve the archive file itself as an attachment so
# nothing silently vanishes, matching extract_archive's
# fail-soft contract.
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
self._capture_attachment(source, post=post, artist=artist, resolved=True)
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
self._capture_attachment(
source, post=post, artist=artist_use, resolved=True,
)
return ImportResult(status="attached")
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = []
with extract_archive(source) as members:
for _name, member_path in members:
if not is_supported(member_path):
continue # non-media preserved via the stored archive
res = self._import_media(member_path, source)
res = self._import_media(
member_path, source, explicit_source=source_row,
)
if res.status in ("imported", "superseded") and res.image_id:
member_ids.append(res.image_id)
# Preserve the archive itself (links to the same Post/Artist).
self._capture_attachment(
source, post=post, artist=artist, resolved=True
source, post=post, artist=artist_use, resolved=True
)
if member_ids:
return ImportResult(
@@ -451,7 +469,8 @@ class Importer:
return ImportResult(status="attached")
def _import_media(
self, source: Path, attribution_path: Path
self, source: Path, attribution_path: Path,
*, explicit_source: Source | None = None,
) -> ImportResult:
"""The media import pipeline (filters, dedup, copy, provenance).
@@ -598,7 +617,15 @@ class Importer:
artist = self._attach_artist(record, artist_name)
# Sidecar provenance (best-effort; never fails the import).
self._apply_sidecar(record, attribution_path, artist)
# explicit_source lets the FC-3c download path bind the new
# ImageProvenance row to its subscription Source instead of
# having _apply_sidecar re-derive via _lookup_source_for_sidecar.
# Audit 2026-06-02 — archive members extracted from a
# subscription-downloaded zip previously lost subscription
# linkage if the on-disk layout didn't match assumptions.
self._apply_sidecar(
record, attribution_path, artist, explicit_source=explicit_source,
)
# Thumbnail is queued separately by the calling task; the importer
# does not generate thumbnails inline so the import queue stays moving.
@@ -662,16 +689,36 @@ class Importer:
them through. The sidecar JSON gallery-dl emits next to each
downloaded file is read by `_apply_sidecar` via `find_sidecar`.
File-type dispatch parity with `import_one` (FC-2d-iii): zips,
PDFs, audio etc. become PostAttachments; archives are extracted.
Without this dispatch, gallery-dl-downloaded non-media bounced
back as `skipped+invalid_image`, which DownloadService counted
as an ingest error and flipped otherwise-successful runs to
status="error". Operator-flagged 2026-06-02 after a Lustria
patreon run with a 94MB OST zip went red despite 21 successful
image attaches.
Caller's responsibilities after this returns:
- duplicate_hash / duplicate_phash skip → delete the on-disk file
- superseded → file stays where it is (now canonical)
- imported → file stays where it is
- attached → the file's been copied into the attachments store;
caller may delete the on-disk original (mirrors duplicate_hash)
- failed → file untouched; caller decides
"""
if not is_supported(path):
if path.suffix.lower() == ".json":
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"unsupported extension {path.suffix}",
error="sidecar json is metadata, not content",
)
if is_archive(path):
return self._import_archive(
path, artist=artist, source_row=source,
)
if not is_supported(path):
post = self._post_for_sidecar(path, artist) if artist else None
return self._capture_attachment(
path, post=post, artist=artist, resolved=True,
)
# Format / dimension / transparency filters (mirror _import_media).
@@ -743,7 +790,8 @@ class Importer:
if rel == "smaller_exists":
target = self.session.get(ImageRecord, match_id)
self._supersede(
target, path, sha, phash, width, height, new_path=path
target, path, sha, phash, width, height,
new_path=path, artist=artist, source_row=source,
)
return ImportResult(status="superseded", image_id=match_id)
@@ -858,14 +906,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
if sd.post_url is not None:
post.post_url = sd.post_url
@@ -901,7 +950,7 @@ class Importer:
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
source_id=src.id if src else None,
captured_metadata=sd.raw,
)
)
@@ -937,6 +986,8 @@ class Importer:
self, existing: ImageRecord, source: Path, sha: str,
phash: str, width: int | None, height: int | None,
*, new_path: Path | None = None,
artist: Artist | None = None,
source_row: Source | None = None,
) -> None:
"""Replace `existing`'s file with the larger `source`, keeping the
row id (so tags/series/curation stay attached). ML is cleared so
@@ -988,8 +1039,14 @@ class Importer:
# _apply_sidecar resolves artist from the sidecar itself if the
# existing row has none, and is internally guarded against
# missing-or-malformed sidecars (silent return).
# Audit 2026-06-02: thread artist/source_row from the
# download-path caller (attach_in_place smaller_exists branch)
# so the supersede preserves explicit subscription linkage
# instead of re-deriving via path-walk.
try:
self._apply_sidecar(existing, source, None)
self._apply_sidecar(
existing, source, artist, explicit_source=source_row,
)
except Exception as exc:
# Don't unwind the supersede DB swap if sidecar parsing
# blows up unexpectedly — the file replacement is the
+20 -4
View File
@@ -19,7 +19,6 @@ from ...models import (
TagReferenceEmbedding,
)
from ...models.tag import image_tag
from .embedder import MODEL_VERSION as SIGLIP_VERSION
ELIGIBLE_KINDS = {
TagKind.character,
@@ -46,6 +45,21 @@ class CentroidService:
)
).scalar_one()
async def _model_version(self) -> str:
"""Audit 2026-06-02: SigLIP model-version stamp comes from the
DB row, not the env constant. tag_and_embed (tasks/ml.py:110)
already reads from MLSettings.embedder_model_version, so by
sourcing centroid stamps + drift checks from the same row, we
eliminate the silent-drift case the audit flagged. env
SIGLIP_MODEL_VERSION still drives which model embedder.py
loads at runtime; the version stamp is purely the operator-
controlled identifier."""
return (
await self.session.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
)
).scalar_one()
async def recompute_for_tag(self, tag_id: int) -> bool:
"""Recompute one tag's centroid. Returns True if a centroid was
written, False if skipped (ineligible kind or too few members)."""
@@ -69,19 +83,20 @@ class CentroidService:
return False
centroid = np.mean(np.stack(embeddings), axis=0).astype(np.float32)
model_version = await self._model_version()
stmt = insert(TagReferenceEmbedding).values(
tag_id=tag_id,
embedding=centroid.tolist(),
reference_count=len(embeddings),
model_version=SIGLIP_VERSION,
model_version=model_version,
)
stmt = stmt.on_conflict_do_update(
index_elements=["tag_id"],
set_={
"embedding": centroid.tolist(),
"reference_count": len(embeddings),
"model_version": SIGLIP_VERSION,
"model_version": model_version,
"updated_at": func.now(),
},
)
@@ -92,6 +107,7 @@ class CentroidService:
"""Tag ids whose centroid is stale: member count != reference_count,
OR no centroid row, OR centroid built on a different SigLIP version.
Only considers eligible-kind tags with embeddings present."""
current_model_version = await self._model_version()
member_counts = (
select(
image_tag.c.tag_id.label("tag_id"),
@@ -116,7 +132,7 @@ class CentroidService:
TagReferenceEmbedding.reference_count
!= member_counts.c.members
)
| (TagReferenceEmbedding.model_version != SIGLIP_VERSION)
| (TagReferenceEmbedding.model_version != current_model_version)
)
)
return list((await self.session.execute(stmt)).scalars().all())
+30 -12
View File
@@ -4,7 +4,7 @@ threshold-filtered, category-grouped, ranked suggestions for one image.
from dataclasses import dataclass, field
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import (
@@ -16,6 +16,7 @@ from ...models import (
from ...models.tag import image_tag
from .aliases import AliasService
from .centroids import CentroidService
from .tag_name import normalize as normalize_tag_name
from .tagger import SURFACED_CATEGORIES
@@ -48,11 +49,11 @@ class SuggestionService:
).scalar_one()
def _threshold_for(self, s: MLSettings, category: str) -> float:
# 'artist' intentionally absent (FC-2d-vii-c) — falls through to
# the 1.01 "never surfaces" default like any unsurfaced category.
# 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired;
# both fall through to the 1.01 "never surfaces" default like any
# unsurfaced category.
return {
"character": s.suggestion_threshold_character,
"copyright": s.suggestion_threshold_copyright,
"general": s.suggestion_threshold_general,
}.get(category, 1.01)
@@ -84,7 +85,12 @@ class SuggestionService:
)
# --- Camie predictions ---
candidates: list[tuple[str, str, float]] = []
# candidates carry (raw_name, display_name, category, confidence).
# raw_name = the booru-formatted vocab key, kept for alias_map
# lookup since alias rows are hand-curated against raw keys.
# display_name = normalize_tag_name(raw_name) — what the operator
# sees AND what gets written to tag.name on Accept.
candidates: list[tuple[str, str, str, float]] = []
for name, p in predictions.items():
category = p.get("category", "general")
if category not in SURFACED_CATEGORIES:
@@ -92,10 +98,14 @@ class SuggestionService:
conf = float(p.get("confidence", 0.0))
if conf < self._threshold_for(settings, category):
continue
candidates.append((name, category, conf))
display = normalize_tag_name(name)
if display is None:
# emoticon / pure-punctuation vocab entry — drop entirely
continue
candidates.append((name, display, category, conf))
alias_map = await self.aliases.resolve_many(
[(n, c) for n, c, _ in candidates]
[(raw, c) for raw, _disp, c, _conf in candidates]
)
merged: dict[object, Suggestion] = {}
@@ -116,8 +126,8 @@ class SuggestionService:
creates_new_tag=existing.creates_new_tag,
)
for name, category, conf in candidates:
canonical = alias_map.get((name, category))
for raw, display, category, conf in candidates:
canonical = alias_map.get((raw, category))
if canonical is not None:
if canonical.id in applied or canonical.id in rejected:
continue
@@ -133,9 +143,17 @@ class SuggestionService:
),
)
else:
# Case-insensitive match on BOTH the raw camie key AND
# the normalized form — covers legacy underscore-named
# Tag rows accepted before normalization shipped, AND
# any tag the operator created with the human form.
existing_tag = (
await self.session.execute(
select(Tag).where(Tag.name == name)
select(Tag).where(
func.lower(Tag.name).in_(
[raw.lower(), display.lower()]
)
)
)
).scalars().first()
if existing_tag is not None:
@@ -157,10 +175,10 @@ class SuggestionService:
)
else:
_merge(
f"raw:{name}:{category}",
f"raw:{display}:{category}",
Suggestion(
canonical_tag_id=None,
display_name=name,
display_name=display,
category=category,
score=conf,
source="tagger",
+62
View File
@@ -0,0 +1,62 @@
"""Camie vocabulary -> human-readable tag-name normalization.
Camie v2's ~57k tag vocabulary is booru-derived and arrives as raw
strings like `uchiha_sasuke_(naruto)`, `#unicus_(idolmaster)`,
`1000-nen_ikiteru_(vocaloid)`, or `:/`. We want the operator to see
"Uchiha Sasuke", "Unicus", "1000-Nen Ikiteru", or to never see the
emoticon at all — and we want the same clean string to be what lands
in `tag.name` when the suggestion is accepted, so Accept matches the
existing-tag convention (`tag_service.find_or_create`).
Rules (operator-approved 2026-06-03):
1. Strip leading junk chars (#, ., +, ;, ~, _, whitespace)
2. Drop trailing `_(disambiguator)` block(s), iteratively
3. Strip wrapping single/double quotes (after disambig removal so
`"foo_em_up"_(series)` -> `"foo_em_up"` -> `foo_em_up`)
4. Replace remaining `_` with space; collapse runs of whitespace
5. Add a space after any `:` (namespace:tag -> namespace: tag)
6. Preserve hyphens (booru hyphens often carry meaning)
7. Title-case each space-separated word (first character only —
apostrophes, digits, hyphens stay)
8. If no letters AND no digits remain, return None (drops emoticons
like `:/` or `^_^`; preserves bare digit tags like `2005`)
9. No surname/givenname swap — no reliable signal in the vocab
"""
import re
_LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
_MULTISPACE = re.compile(r"\s+")
_COLON_NOSPACE = re.compile(r":(?=\S)")
_HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
def _strip_wrapping_quotes(s: str) -> str:
if len(s) >= 2 and s[0] == s[-1] and s[0] in ('"', "'"):
return s[1:-1]
return s
def _title_word(w: str) -> str:
return w[:1].upper() + w[1:] if w else w
def normalize(raw: str) -> str | None:
"""Return the human-readable form of a raw Camie tag, or None if the
string is junk (emoticon, empty after stripping)."""
if not raw:
return None
s = _LEADING_JUNK.sub("", raw)
while True:
new = _TRAILING_DISAMBIG.sub("", s)
if new == s:
break
s = new
s = _strip_wrapping_quotes(s)
s = s.replace("_", " ")
s = _COLON_NOSPACE.sub(": ", s)
s = _MULTISPACE.sub(" ", s).strip()
if not s or not _HAS_ALPHANUMERIC.search(s):
return None
return " ".join(_title_word(w) for w in s.split(" "))
+7 -4
View File
@@ -38,10 +38,13 @@ STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
# The categories FC-2b surfaces in the UI. Others (meta/rating/year) are
# still stored but the suggestion service filters them out.
# FC-2d-vii-c: 'artist' retired — artist identity is acquisition-derived
# (image_record.artist_id), never ML-inferred. Raw predictions are still
# stored at STORE_FLOOR but artist never surfaces.
SURFACED_CATEGORIES = {"character", "copyright", "general"}
# 'artist' retired in FC-2d-vii-c — artist identity is acquisition-derived
# (image_record.artist_id), never ML-inferred. 'copyright' retired
# 2026-06-01 — operator doesn't use the copyright tag-kind; fandom is
# this app's franchise/series concept (per TagsView.vue's doc comment).
# Raw predictions for both categories still get stored at STORE_FLOOR but
# don't surface in suggestions.
SURFACED_CATEGORIES = {"character", "general"}
# ImageNet preprocessing constants (per Camie v2 onnx_inference.py).
_IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
+3 -1
View File
@@ -96,10 +96,12 @@ def str_field(v) -> str | None:
# Shared gallery-dl invocation defaults. Embedded in each platform's
# default_config (with platform-specific overrides) so per-platform
# choices stay explicit.
# Note: the gallery-dl `skip` value (tick "exit:20" vs backfill True) is
# NOT here — it's derived from Source.backfill_runs_remaining at download
# time. See plan #544 / gallery_dl.TICK_SKIP_VALUE,BACKFILL_SKIP_VALUE.
GD_DEFAULTS = {
"sleep": 3.0,
"sleep_request": 1.5,
"skip_existing": True,
"save_metadata": True,
"timeout": 3600,
}
+21 -9
View File
@@ -69,13 +69,19 @@ class PostFeedService:
raise ValueError("direction must be 'older' or 'newer'")
sort_key = _sort_key()
# Artist via the denormalized Post.artist_id (alembic 0030);
# Source via LEFT JOIN since post.source_id can now be NULL for
# filesystem-imported posts with no live subscription. A
# platform= filter implicitly excludes NULL-source posts (they
# have no platform); an artist_id= filter still surfaces them
# because Post.artist_id is always set.
stmt = (
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
)
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
stmt = stmt.where(Post.artist_id == artist_id)
if platform is not None:
stmt = stmt.where(Source.platform == platform)
if cursor:
@@ -135,8 +141,8 @@ class PostFeedService:
cursor for each end. Returns None if the post doesn't exist."""
anchor = (await self.session.execute(
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if anchor is None:
@@ -168,8 +174,8 @@ class PostFeedService:
async def get_post(self, post_id: int) -> dict | None:
row = (await self.session.execute(
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if row is None:
@@ -260,7 +266,7 @@ class PostFeedService:
return out
def _to_dict(
self, post: Post, artist: Artist, source: Source,
self, post: Post, artist: Artist, source: Source | None,
thumbs_map: dict, atts_map: dict,
) -> dict:
plain_full = html_to_plain(post.description) if post.description else None
@@ -269,6 +275,9 @@ class PostFeedService:
else:
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
# `source` is null for filesystem-imported posts with no live
# subscription (alembic 0030). Frontend renders that as a
# "filesystem import" affordance instead of a platform chip.
return {
"id": post.id,
"external_post_id": post.external_post_id,
@@ -279,7 +288,10 @@ class PostFeedService:
"description_plain": description_plain,
"description_truncated": truncated,
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"source": {"id": source.id, "platform": source.platform},
"source": (
{"id": source.id, "platform": source.platform}
if source is not None else None
),
"thumbnails": thumbs_entry["thumbs"],
"thumbnails_more": thumbs_entry["more"],
"attachments": atts_map.get(post.id, []),
+12 -6
View File
@@ -70,11 +70,15 @@ class ProvenanceService:
rec = await self.session.get(ImageRecord, image_id)
if rec is None:
return None
# Artist via Post.artist_id (alembic 0030); Source via LEFT JOIN
# since both Post.source_id and ImageProvenance.source_id can be
# NULL for filesystem-imported content. Frontend renders source=
# null as "filesystem import."
stmt = (
select(ImageProvenance, Post, Source, Artist)
.join(Post, Post.id == ImageProvenance.post_id)
.join(Source, Source.id == ImageProvenance.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == ImageProvenance.source_id)
.where(ImageProvenance.image_record_id == image_id)
.order_by(ImageProvenance.captured_at.asc(),
ImageProvenance.id.asc())
@@ -90,7 +94,7 @@ class ProvenanceService:
"captured_at": ip.captured_at.isoformat()
if ip.captured_at else None,
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
}
for ip, post, src, art in rows
@@ -99,10 +103,12 @@ class ProvenanceService:
}
async def for_post(self, post_id: int) -> dict | None:
# Same LEFT JOIN to Source — get_post must succeed for a
# NULL-source post.
stmt = (
select(Post, Source, Artist)
.join(Source, Source.id == Post.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == Post.source_id)
.where(Post.id == post_id)
)
row = (await self.session.execute(stmt)).first()
@@ -111,7 +117,7 @@ class ProvenanceService:
post, src, art = row
return {
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
"attachments": await self._attachments_for_posts([post.id]),
}
+1 -1
View File
@@ -86,7 +86,7 @@ def attempt_refetch(
if src is None:
return {"status": "no_source"}
# Remove the bad copy so gallery-dl (skip_existing) re-fetches it on
# Remove the bad copy so gallery-dl's archive-skip re-fetches it on
# the source re-check instead of skipping the still-present corrupt
# file.
try:
+42 -1
View File
@@ -59,9 +59,11 @@ class SourceRecord:
config_overrides: dict | None
last_checked_at: str | None
last_error: str | None
error_type: str | None
check_interval_override: int | None
consecutive_failures: int
next_check_at: str | None
backfill_runs_remaining: int
def to_dict(self) -> dict:
return {
@@ -75,9 +77,11 @@ class SourceRecord:
"config_overrides": self.config_overrides,
"last_checked_at": self.last_checked_at,
"last_error": self.last_error,
"error_type": self.error_type,
"check_interval_override": self.check_interval_override,
"consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at,
"backfill_runs_remaining": self.backfill_runs_remaining,
}
@@ -85,6 +89,11 @@ class SourceRecord:
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
# Plan #544 follow-up: newly created enabled sources pre-arm backfill so
# their first N polls walk gallery-dl's full post history with the longer
# timeout (matches the manual "Deep scan" button's default).
NEW_SOURCE_BACKFILL_RUNS = 3
class SourceService:
def __init__(self, session: AsyncSession):
@@ -137,9 +146,11 @@ class SourceService:
config_overrides=source.config_overrides,
last_checked_at=source.last_checked_at.isoformat() if source.last_checked_at else None,
last_error=source.last_error,
error_type=source.error_type,
check_interval_override=source.check_interval_override,
consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None,
backfill_runs_remaining=source.backfill_runs_remaining or 0,
)
async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -157,7 +168,7 @@ class SourceService:
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
if not include_synthetic:
# Filesystem-import sidecar anchors (importer._source_for_sidecar)
# Pre-alembic-0030 sidecar synthetic anchors
# have url='sidecar:<platform>:<slug>' and exist only to give
# imported Posts a NOT-NULL Source FK. They aren't pollable
# feeds; the Subscriptions UI used to render them as phantom
@@ -201,10 +212,21 @@ class SourceService:
select(func.count(Source.id)).where(Source.artist_id == artist_id)
)).scalar_one()
# Plan #544 follow-up: a freshly added subscription has no archive
# yet, so the first few polls would walk the full post history in
# tick mode and trip exit:20 after ~20 contiguous archive hits —
# except there are none yet, so tick mode would walk forever and
# blow the wall-clock cap. Pre-arm backfill so the initial syncs
# use the longer timeout + skip:True walk. Tick mode resumes once
# the budget is spent or the queue drains.
# Disabled sources (incl. sidecar synthetics, url='sidecar:...')
# are never polled, so leave their counter at 0.
backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
source = Source(
artist_id=artist_id, platform=platform, url=url,
enabled=enabled, config_overrides=config_overrides,
check_interval_override=check_interval_override,
backfill_runs_remaining=backfill_runs,
)
self.session.add(source)
try:
@@ -264,6 +286,25 @@ class SourceService:
await self.session.commit()
return await self._row_to_record(source)
async def set_backfill_runs(
self, source_id: int, runs: int,
) -> SourceRecord:
"""Plan #544: arm a source for backfill mode. The next `runs`
download runs will use gallery-dl's full-walk config (skip: True
+ 30-min timeout) instead of the catch-up default. Runs must be
1..10 — bigger is rejected to keep the operator from accidentally
setting a runaway budget."""
if not isinstance(runs, int) or runs < 1 or runs > 10:
raise ValueError("runs must be an integer in [1, 10]")
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
source.backfill_runs_remaining = runs
await self.session.commit()
return await self._row_to_record(source)
async def delete(self, source_id: int) -> None:
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
+17 -7
View File
@@ -5,6 +5,7 @@ from dataclasses import dataclass
from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Tag, TagKind, image_tag
@@ -86,9 +87,12 @@ class TagService:
f"fandom_id {fandom_id} does not reference a fandom tag"
)
# Upsert via INSERT ... ON CONFLICT DO NOTHING. We can't use the
# uniqueness index name directly (it's a partial coalesce-based
# expression), so we re-select after insert.
# Audit 2026-06-02: race-safe upsert via savepoint +
# IntegrityError recovery. The partial uniqueness index on
# (name, kind, COALESCE(fandom_id, -1)) catches concurrent
# inserts; without the savepoint the outer transaction would
# poison and the calling request crashes. Mirrors
# importer._get_or_create.
stmt = (
select(Tag)
.where(Tag.name == name)
@@ -101,10 +105,16 @@ class TagService:
if existing:
return existing
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
return new_tag
sp = await self.session.begin_nested()
try:
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
await sp.commit()
return new_tag
except IntegrityError:
await sp.rollback()
return (await self.session.execute(stmt)).scalar_one()
async def autocomplete(
self,
+5 -2
View File
@@ -240,8 +240,11 @@ def prune_backups() -> dict:
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
Status='running' / 'restoring' rows are never pruned — the
recover_stalled_backup_runs sweep flips truly-stuck ones to
'error' first. (Earlier docstring claimed the FC-3i TaskRun sweep
handled those, but TaskRun cleanup never touched BackupRun rows.
Audit 2026-06-02 added the dedicated sweep.)
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
+218 -3
View File
@@ -7,14 +7,17 @@ from datetime import UTC, datetime, timedelta
from pathlib import Path
from PIL import Image
from sqlalchemy import and_, delete, or_, select, update
from sqlalchemy import Integer, and_, cast, delete, func, or_, select, update
from ..celery_app import celery
from ..models import (
BackupRun,
DownloadEvent,
ImageRecord,
ImportBatch,
ImportSettings,
ImportTask,
LibraryAuditRun,
Source,
TaskRun,
)
@@ -55,6 +58,29 @@ FFPROBE_TIMEOUT_SECONDS = 10
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# Audit 2026-06-02: per-entity recovery sweep thresholds. Each must be
# > the entity's longest legitimate runtime (its task's time_limit + a
# small buffer) so the sweep never flags in-flight work.
#
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
# 2h15m gives a 10-min buffer.
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
# Import batches finalize only after every child ImportTask hits a
# terminal state. The recovery sweep targets the case where every
# task is done but the batch never got its closing UPDATE
# (orchestrator crashed at the wrong instant). 2h is well past any
# realistic single-batch import.
IMPORT_BATCH_STALL_THRESHOLD_MINUTES = 120
# Retention windows (terminal rows older than these get deleted by
# the daily prune sweeps). 30 days = operator-flagged "useful for
# triage for a few weeks, then noise."
LIBRARY_AUDIT_KEEP_DAYS = 30
IMPORT_BATCH_KEEP_DAYS = 30
# Overrides for recover_stalled_task_runs (the TaskRun 'running' sweep).
# Tasks/queues that legitimately run longer than the default 5-min
# threshold need their own larger value, else the sweep marks in-flight
@@ -69,9 +95,24 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# files); time_limit=2100.
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"ml": 25,
# Audit 2026-06-02 — maintenance/scan queues run tasks that
# legitimately exceed the 5-min default (verify_integrity at 70m
# hard, scan_directory at 70m hard, apply_allowlist_tags /
# recompute_centroids / backfill_phash at 35m hard). 75 min lives
# above the longest of those and the per-task overrides below
# cover the outliers (backups, library audit).
"maintenance": 75,
"scan": 75,
}
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"backend.app.tasks.import_file.import_archive_file": 40,
# Backup images runs hours, not minutes (6.5h hard limit). The
# task-name override beats the queue's 75-min default so a
# legitimately-running backup isn't flagged.
"backend.app.tasks.backup.backup_images_task": 420,
"backend.app.tasks.backup.restore_images_task": 420,
# Library audit scans the full library — 2h hard limit.
"backend.app.tasks.library_audit.scan_library_for_rule": 130,
}
@@ -182,6 +223,11 @@ def recover_interrupted_tasks() -> int:
.where(ImportTask.created_at < orphan_cutoff)
.values(
status="failed",
# Without finished_at, cleanup_old_tasks (`WHERE
# finished_at < cutoff`) never reaps these rows —
# orphan-swept rows would become permanent table
# tenants. Audit 2026-06-02.
finished_at=now,
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
@@ -278,6 +324,14 @@ def recover_stalled_task_runs() -> int:
f"no completion signal received within {minutes} min"
),
finished_at=now,
# Matches celery_signals.finalize's
# int((now - started_at).total_seconds() * 1000)
# — sweep-closed rows now carry duration like
# normally-finalized rows. Audit 2026-06-02.
duration_ms=cast(
func.extract("epoch", now - TaskRun.started_at) * 1000,
Integer,
),
)
)
for w in extra_where:
@@ -347,7 +401,12 @@ def prune_task_runs() -> dict:
return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@celery.task(name="backend.app.tasks.maintenance.backfill_phash")
@celery.task(
name="backend.app.tasks.maintenance.backfill_phash",
# Audit 2026-06-02 — keyset-paginated phash recompute over the whole
# library; legitimately runs >5 min on large libraries.
soft_time_limit=1800, time_limit=2100,
)
def backfill_phash() -> int:
"""Recompute phash for stored images that have none (imported before
FC-2d-i+ii). Keyset-paginated by id (restart-safe), NULL-only fill,
@@ -425,7 +484,13 @@ def _verify_one(path: Path, expected_sha: str, mime: str, sha_fn) -> str:
return "failed_verification"
@celery.task(name="backend.app.tasks.maintenance.verify_integrity")
@celery.task(
name="backend.app.tasks.maintenance.verify_integrity",
# Audit 2026-06-02 — full library sha256 + decode probe; on 100k-image
# libraries this runs an hour or more. Match the maintenance queue's
# recovery threshold (75 min) with 30s buffer below.
soft_time_limit=3600, time_limit=4200,
)
def verify_integrity() -> int:
"""Verify every ImageRecord file: sha256 recompute + decode/probe
(PIL for images; ffprobe for videos). Writes integrity_status
@@ -521,6 +586,156 @@ def recover_stalled_download_events() -> int:
return events_recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_backup_runs")
def recover_stalled_backup_runs() -> int:
"""Flip BackupRun rows stuck in running/restoring past the hard limit
to error. Audit 2026-06-02.
prune_backups (FC-3h) used to claim the FC-3i task_run sweep handled
these — but that sweep only flips TaskRun rows, not the BackupRun
artifact rows. A SIGKILL'd backup left BackupRun stuck forever
(dashboard showed phantom in-flight backups, keep_last_n offset
arithmetic skewed because zombies sat outside the ok/error window).
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
)
with SessionLocal() as session:
result = session.execute(
update(BackupRun)
.where(BackupRun.status.in_(["running", "restoring"]))
.where(BackupRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("recover_stalled_backup_runs: recovered %d rows", recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_library_audit_runs")
def recover_stalled_library_audit_runs() -> int:
"""Flip LibraryAuditRun rows stuck in running past the hard limit
to error. Audit 2026-06-02.
LibraryAuditRun.status='running' was protected by an exclusive
guard in start_audit_run — a SIGKILL'd run would block all future
audits until manual DB surgery. (The guard is now age-aware, but
this sweep is what makes that work in practice.)
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
)
with SessionLocal() as session:
result = session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_library_audit_runs: recovered %d rows", recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
def recover_stalled_import_batches() -> int:
"""Finalize ImportBatch rows stuck in running past the hard limit
when NO outstanding ImportTask remains. Audit 2026-06-02.
A batch row finalizes only after every child task hits a terminal
state. The orphan case: scanner crashed between the last task's
completion and the batch's closing UPDATE. The
`/api/import/status` route then surfaces the batch as 'active'
indefinitely while `/api/system/stats` (which uses the same
EXISTS predicate we apply below) correctly returns null.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=IMPORT_BATCH_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
# Batches still 'running' past the cutoff whose tasks are all
# terminal — there's no outstanding work, so flip the batch
# too. Mirrors the EXISTS predicate the active-batch surfaces use.
result = session.execute(
update(ImportBatch)
.where(ImportBatch.status == "running")
.where(ImportBatch.started_at < cutoff)
.where(
~select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists()
)
.values(status="complete", finished_at=now)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_import_batches: finalized %d zombie batches",
recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.prune_library_audit_runs")
def prune_library_audit_runs() -> int:
"""Daily retention: delete terminal LibraryAuditRun rows older than
LIBRARY_AUDIT_KEEP_DAYS. Never touches 'running'. Audit 2026-06-02.
Audit rows carry matched_ids JSONB blobs that can hold tens of
thousands of ids; without retention these accumulate.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=LIBRARY_AUDIT_KEEP_DAYS)
with SessionLocal() as session:
result = session.execute(
delete(LibraryAuditRun)
.where(LibraryAuditRun.status.in_(["ready", "applied", "cancelled", "error"]))
.where(LibraryAuditRun.finished_at < cutoff)
)
session.commit()
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.prune_import_batches")
def prune_import_batches() -> int:
"""Daily retention: delete terminal ImportBatch rows older than
IMPORT_BATCH_KEEP_DAYS. Cascade-deletes child ImportTask rows via
the model relationship. Never touches 'running'. Audit 2026-06-02.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=IMPORT_BATCH_KEEP_DAYS)
with SessionLocal() as session:
# ORM-level delete here (not Core delete) so the
# ImportBatch->tasks cascade fires; Core delete would skip it.
old_batches = session.execute(
select(ImportBatch)
.where(ImportBatch.status.in_(["complete", "cancelled"]))
.where(ImportBatch.finished_at < cutoff)
).scalars().all()
for batch in old_batches:
session.delete(batch)
session.commit()
return len(old_batches)
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_download_events")
def cleanup_old_download_events() -> int:
"""FC-3d: delete terminal DownloadEvent rows older than the configured
+15 -2
View File
@@ -212,7 +212,14 @@ def backfill(self) -> int:
return enqueued
@celery.task(name="backend.app.tasks.ml.apply_allowlist_tags", bind=True)
@celery.task(
name="backend.app.tasks.ml.apply_allowlist_tags",
bind=True,
# Audit 2026-06-02 — the full-sweep mode (neither tag_id nor image_id)
# is O(images × allowlist) and legitimately runs >5 min on large
# libraries. Cap matches the maintenance queue's recovery threshold.
soft_time_limit=1800, time_limit=2100,
)
def apply_allowlist_tags(self, tag_id: int | None = None,
image_id: int | None = None) -> int:
"""Retroactively apply allowlisted tags.
@@ -341,7 +348,13 @@ def recompute_centroid(self, tag_id: int) -> bool:
return asyncio.run(_run())
@celery.task(name="backend.app.tasks.ml.recompute_centroids", bind=True)
@celery.task(
name="backend.app.tasks.ml.recompute_centroids",
bind=True,
# Audit 2026-06-02 — drifted-centroid rebuild over potentially
# hundreds of tags.
soft_time_limit=1800, time_limit=2100,
)
def recompute_centroids(self) -> int:
"""Daily: find drifted centroids, enqueue recompute_centroid for each."""
import asyncio
+9 -1
View File
@@ -35,7 +35,15 @@ def _iter_import_files(import_root: Path):
yield entry
@celery.task(name="backend.app.tasks.scan.scan_directory", bind=True)
@celery.task(
name="backend.app.tasks.scan.scan_directory",
bind=True,
# Audit 2026-06-02 — large libraries make the scan legitimately long.
# Hard cap at 70 min so the corresponding QUEUE_STUCK_THRESHOLD_MINUTES
# ("scan") of 75 min always wins; soft limit gives the task a clean
# exit window before SIGKILL.
soft_time_limit=3600, time_limit=4200,
)
def scan_directory(self, triggered_by: str = "manual",
mode: str = "quick") -> int:
"""Walks the import root and creates ImportTasks. `mode` is 'quick'
+85 -47
View File
@@ -20,14 +20,32 @@ IMAGES_ROOT = Path("/images")
THUMB_MAGIC_JPEG = b"\xff\xd8\xff"
THUMB_MAGIC_PNG = b"\x89PNG\r\n\x1a\n"
# Minimum file size for a thumbnail to count as valid. Anything smaller
# is almost certainly truncated/corrupt — a legitimate 400×400 JPEG@85
# bottoms out around 2KB even on a solid-color image; 400×400 PNG starts
# around 1KB. 256 bytes is well below any real thumbnail and well above
# header-only corrupt files (~8-12 bytes). Operator-flagged 2026-06-01:
# header-only corrupt files were silently passing the magic-byte check
# and backfill counted them as "ok" — so broken-image tiles in the UI
# never got regenerated even after running backfill.
MIN_THUMB_BYTES = 256
def _thumb_is_valid(path: Path) -> bool:
"""Return True iff `path` exists and starts with a JPEG or PNG magic header.
"""Return True iff `path` exists, starts with a JPEG or PNG magic
header, AND is at least MIN_THUMB_BYTES on disk.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG for
opaque sources, PNG for alpha sources. Anything else (missing file, OSError,
truncated, wrong magic) is invalid.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG
for opaque sources, PNG for alpha sources. Anything else (missing
file, OSError, truncated below the size floor, wrong magic) is
invalid and gets re-enqueued.
"""
try:
size = path.stat().st_size
except OSError:
return False
if size < MIN_THUMB_BYTES:
return False
try:
with path.open("rb") as f:
head = f.read(12)
@@ -42,6 +60,59 @@ def _thumb_is_valid(path: Path) -> bool:
return False
def _run_backfill_scan() -> dict:
"""Synchronous scan logic shared by the Celery task and the API
endpoint. Returns {enqueued, ok, regenerated, scanned}.
Operator-flagged 2026-06-01: the original task was fire-and-forget,
so the admin UI couldn't show what backfill actually found —
operator saw \"Enqueued.\" with no counts and assumed nothing was
happening. Now the API runs this synchronously and returns the
real numbers; the periodic Celery task wraps it too."""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
scanned = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
scanned += len(rows)
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {
"scanned": scanned,
"enqueued": enqueued,
"ok": ok,
"regenerated": regenerated,
}
@celery.task(
name="backend.app.tasks.thumbnail.generate_thumbnail",
bind=True,
@@ -84,48 +155,15 @@ def backfill_thumbnails(self) -> dict:
"""Scan ImageRecord and enqueue generate_thumbnail for rows whose
thumbnail is missing, gone from disk, or has wrong magic bytes.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path for
rows that point at a missing or corrupt file before enqueueing — keeps
the DB self-consistent on partial runs and makes re-runs safe.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path
for rows that point at a missing or corrupt file before enqueueing —
keeps the DB self-consistent on partial runs and makes re-runs safe.
Returns {"enqueued": N, "ok": M, "regenerated": K} where:
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL thumbnail_path
cleared (i.e. missing + corrupt)
Returns {scanned, enqueued, ok, regenerated} where:
- scanned = total rows examined
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL
thumbnail_path cleared (i.e. missing + corrupt)
"""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {"enqueued": enqueued, "ok": ok, "regenerated": regenerated}
return _run_backfill_scan()
+41 -2
View File
@@ -194,9 +194,20 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
};
}
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length };
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -229,8 +240,13 @@ browser.runtime.onMessage.addListener(async (msg) => {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length };
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
} catch (e) {
results[key] = { error: e.message };
}
@@ -259,6 +275,29 @@ browser.runtime.onMessage.addListener(async (msg) => {
return { error: e.message };
}
case 'PROBE_SOURCE':
try {
return await api.probeSource(msg.url);
} catch (e) {
return { error: e.message };
}
case 'OPEN_ARTIST_PAGE': {
// apiUrl is configured with the /api suffix (see
// options/options.html placeholder); the SPA artist route is
// /artist/:slug, served from the same origin. Strip /api so the
// browser-level URL hits the Vue router, not the JSON API.
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
const slug = encodeURIComponent(msg.slug || '');
if (!base || !slug) return { error: 'apiUrl or slug missing' };
try {
await browser.tabs.create({ url: `${base}/artist/${slug}` });
return { success: true };
} catch (e) {
return { error: e.message };
}
}
default:
return { error: `Unknown message type: ${msg.type}` };
}
+16 -1
View File
@@ -5,11 +5,26 @@
background: rgb(20, 23, 26); color: rgb(244, 186, 122);
font: 500 14px/1.2 system-ui, sans-serif;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.4); cursor: pointer;
transition: transform 100ms ease;
transition: transform 100ms ease, background 150ms ease, color 150ms ease;
}
.fc-add-source-btn:hover { transform: translateY(-1px); }
.fc-add-source-btn:disabled { opacity: 0.6; cursor: wait; }
/* state colors map to the FC palette: parchment-on-slate base,
accent-orange for new, sage for already-subscribed, amber-warning for
artist-exists-but-source-missing. All readable on the dark base. */
.fc-add-source-btn--new {
background: rgb(20, 23, 26); color: rgb(244, 186, 122);
}
.fc-add-source-btn--artist-match {
background: rgb(28, 23, 16); color: rgb(255, 200, 120);
border: 1px solid rgb(180, 130, 60);
}
.fc-add-source-btn--source-match {
background: rgb(18, 28, 20); color: rgb(140, 220, 160);
border: 1px solid rgb(80, 160, 100);
}
.fc-toast {
all: revert;
position: fixed; bottom: 84px; right: 24px; z-index: 2147483647;
+95 -13
View File
@@ -2,6 +2,10 @@
if (window.__fc_addsource_injected) return;
window.__fc_addsource_injected = true;
// Cached probe result for the current URL so click-handlers know which
// action to dispatch without round-tripping again.
let currentProbe = null;
evaluate();
const reEval = () => evaluate();
@@ -9,38 +13,116 @@
const origPush = history.pushState;
history.pushState = function () { origPush.apply(this, arguments); reEval(); };
function evaluate() {
const platform = getPlatformFromUrl(window.location.href);
const onArtist = platform && isArtistPage(window.location.href, platform);
let btn = document.getElementById('fc-add-source-btn');
if (onArtist && !btn) injectButton();
else if (!onArtist && btn) btn.remove();
async function evaluate() {
const url = window.location.href;
const platform = getPlatformFromUrl(url);
const onArtist = platform && isArtistPage(url, platform);
const btn = document.getElementById('fc-add-source-btn');
if (!onArtist) {
if (btn) btn.remove();
currentProbe = null;
return;
}
// On artist pages, ask the backend what state the URL is in BEFORE
// injecting the button — so the chip can render the right state on
// first paint instead of flashing the generic "Add" copy and
// updating afterwards.
let probe;
try {
probe = await browser.runtime.sendMessage({ type: 'PROBE_SOURCE', url });
} catch (e) {
probe = { error: e?.message || 'probe failed' };
}
currentProbe = probe;
if (probe?.state === 'unknown_platform') {
if (btn) btn.remove();
return;
}
renderButton(probe);
}
function injectButton() {
const btn = document.createElement('button');
btn.id = 'fc-add-source-btn';
function renderButton(probe) {
let btn = document.getElementById('fc-add-source-btn');
if (!btn) {
btn = document.createElement('button');
btn.id = 'fc-add-source-btn';
btn.addEventListener('click', onClick);
document.body.appendChild(btn);
}
// Reset state classes so re-renders (SPA navigation) don't stack.
btn.className = 'fc-add-source-btn';
btn.textContent = '+ Add to FabledCurator';
btn.addEventListener('click', onClick);
document.body.appendChild(btn);
btn.classList.add(`fc-add-source-btn--${stateModifier(probe)}`);
btn.textContent = labelFor(probe);
btn.disabled = false;
}
function stateModifier(probe) {
if (!probe || probe.error) return 'new';
return ({
source_match: 'source-match',
artist_match: 'artist-match',
new: 'new',
})[probe.state] || 'new';
}
function labelFor(probe) {
if (!probe || probe.error) return '+ Add to FabledCurator';
const platformName = platformDisplayName(probe.platform);
const artistName = probe.artist?.name;
switch (probe.state) {
case 'source_match':
return `✓ In FabledCurator · ${platformName}`;
case 'artist_match':
return `+ Add ${platformName} source to ${artistName || 'artist'}`;
case 'new':
default:
return '+ Add to FabledCurator';
}
}
function platformDisplayName(key) {
return PLATFORMS[key]?.name || key || '';
}
async function onClick() {
const btn = document.getElementById('fc-add-source-btn');
if (!btn) return;
btn.disabled = true;
const original = btn.textContent;
const probe = currentProbe;
if (probe?.state === 'source_match') {
btn.textContent = 'Opening…';
try {
const r = await browser.runtime.sendMessage({
type: 'OPEN_ARTIST_PAGE',
slug: probe.artist?.slug,
});
if (r?.error) showToast(`Error: ${r.error}`, 'error');
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
} finally {
btn.disabled = false;
btn.textContent = original;
}
return;
}
btn.textContent = 'Adding…';
try {
const r = await browser.runtime.sendMessage({
type: 'ADD_AS_SOURCE',
url: window.location.href,
});
if (r.error) {
if (r?.error) {
showToast(`Error: ${r.error}`, 'error');
} else {
const verb = r.created_source ? 'Added' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success');
// Re-probe so the chip flips green without waiting for the next
// navigation.
evaluate();
return;
}
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
+6
View File
@@ -83,6 +83,12 @@ class FabledCuratorAPI {
quickAddSource(url) {
return this.request('POST', '/extension/quick-add-source', { url });
}
probeSource(url) {
// Read-only existence check. Drives the content-script chip's
// color/copy BEFORE the operator clicks Add.
const qs = new URLSearchParams({ url }).toString();
return this.request('GET', `/extension/probe?${qs}`);
}
// Connection test = the cheapest read with auth.
testConnection() {
+35
View File
@@ -76,3 +76,38 @@ async function getCookieCount(platformKey) {
return 0;
}
}
/**
* Verify cookies are live by hitting an authenticated endpoint with the
* browser's current cookie jar. Returns:
* { ok: true, status } — verified
* { ok: false, status, reason } — endpoint said we're not logged in
* { ok: null, reason } — no verify config for this platform; caller
* should treat as "verify not available,
* proceed with upload"
*
* Implementation note: extensions with `host_permissions` for the target
* domain get the user's cookies auto-attached to fetch() — same set
* gallery-dl will later use on the backend.
*/
async function verifyCookiesForPlatform(platformKey) {
const platform = PLATFORMS[platformKey];
if (!platform) return { ok: false, reason: `Unknown platform: ${platformKey}` };
if (!platform.verify) return { ok: null, reason: 'verify-not-configured' };
const { url, method, okStatuses } = platform.verify;
let resp;
try {
resp = await fetch(url, { method, credentials: 'include', cache: 'no-store' });
} catch (e) {
return { ok: false, reason: `Verify request failed: ${e.message}` };
}
if (okStatuses.includes(resp.status)) {
return { ok: true, status: resp.status };
}
return {
ok: false,
status: resp.status,
reason: `${url} returned HTTP ${resp.status} — session looks stale or logged out`,
};
}
+21
View File
@@ -13,6 +13,13 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FF424D',
urlPattern: /^https?:\/\/(www\.)?patreon\.com/,
// Patreon's `/api/current_user` returns 200 + the logged-in user
// when authenticated, 401 otherwise. Cheapest definitive check.
verify: {
url: 'https://www.patreon.com/api/current_user',
method: 'GET',
okStatuses: [200],
},
},
subscribestar: {
name: 'SubscribeStar',
@@ -26,6 +33,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#FFD700',
urlPattern: /^https?:\/\/(www\.)?subscribestar\.(com|adult)/,
// No known stable auth-required endpoint that returns a definitive
// status code; skipping verify so we don't false-positive-fail
// good cookies. Operator can add later if a clean endpoint surfaces.
},
hentaifoundry: {
name: 'Hentai Foundry',
@@ -33,6 +43,14 @@ const PLATFORMS = {
authType: 'cookies',
color: '#9C27B0',
urlPattern: /^https?:\/\/(www\.)?hentai-foundry\.com/,
// Mirror gallery-dl's _init_site_filters: HEAD on `?enterAgree=1`.
// Logged in → 200, logged out → 401. Catches the exact failure mode
// the backend extractor would hit later.
verify: {
url: 'https://www.hentai-foundry.com/?enterAgree=1',
method: 'HEAD',
okStatuses: [200],
},
},
discord: {
name: 'Discord',
@@ -56,6 +74,9 @@ const PLATFORMS = {
authType: 'cookies',
color: '#05CC47',
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
// DA's logged-in-only endpoints sit behind their internal _napi
// namespace which shifts; skipping verify until a stable check
// surfaces. Same posture as SubscribeStar.
},
};
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.5",
"version": "1.0.7",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "fabledcurator-extension",
"version": "1.0.5",
"version": "1.0.7",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
+2 -1
View File
@@ -132,8 +132,9 @@ async function exportPlatformCookies(key, card) {
if (r.error) showError(r.error);
else {
const n = r.cookieCount ?? null;
const verifiedSuffix = r.verified ? ' (verified ✓)' : '';
const msg = n !== null
? `${PLATFORMS[key].name}: ${n} cookies exported`
? `${PLATFORMS[key].name}: ${n} cookies exported${verifiedSuffix}`
: `${PLATFORMS[key].name}: token exported`;
showSuccess(msg);
await loadPlatformStatus();
+14 -1
View File
@@ -9,7 +9,9 @@
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { onMounted, ref, watch } from 'vue'
import { useRoute } from 'vue-router'
import AppShell from './components/AppShell.vue'
import AppSnackbar from './components/AppSnackbar.vue'
import ImageViewer from './components/modal/ImageViewer.vue'
@@ -17,9 +19,20 @@ import { useModalStore } from './stores/modal.js'
const modal = useModalStore()
const snackbar = ref(null)
const route = useRoute()
onMounted(() => {
// Expose snackbar via a simple global so stores can call it without props.
window.__fcToast = (opts) => snackbar.value?.open(opts)
})
// Audit 2026-06-02: the modal is an overlay, not a page. When the
// route changes (RouterLink inside the modal, history back/forward,
// programmatic push from any view), close the modal so it doesn't
// hover over a different route. Watching route.name (not the path)
// keeps within-route nav like /artist/foo → /artist/bar from
// dismissing the modal mid-browse.
watch(() => route.name, () => {
if (modal.isOpen) modal.close()
})
</script>
@@ -11,9 +11,6 @@
</template>
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../../stores/artist.js'
import { useModalStore } from '../../stores/modal.js'
import MasonryGrid from '../discovery/MasonryGrid.vue'
@@ -24,22 +21,9 @@ const props = defineProps({
const store = useArtistStore()
const modal = useModalStore()
const route = useRoute()
const router = useRouter()
onMounted(() => {
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage (id) {
router.push({ query: { ...route.query, image: id } })
modal.open(id)
}
</script>
@@ -124,10 +124,16 @@ async function onRetry() {
if (!props.event.source_id) return
retrying.value = true
try {
await sourcesStore.checkNow(props.event.source_id)
toast({
text: `Source check re-queued`, type: 'success',
})
const body = await sourcesStore.checkNow(props.event.source_id)
// Audit 2026-06-02: the previous handler unconditionally toasted
// "re-queued" even when the platform was in cooldown (202 +
// status='deferred'). Operator thought work was in flight when
// nothing was actually enqueued.
if (body?.status === 'deferred') {
toast({ text: 'Retry deferred — platform in cooldown', type: 'info' })
} else {
toast({ text: 'Source check re-queued', type: 'success' })
}
} catch (e) {
const isInFlight = !!e?.body?.download_event_id
toast({
@@ -48,10 +48,11 @@ function onSearch(q) {
if (!query) { results.value = []; return }
loading.value = true
try {
// Scope the autocomplete to the prediction's category where it maps
// to a tag kind. 'copyright' has no tag kind; search unscoped there.
const kind = ['artist', 'character'].includes(props.category)
? props.category : null
// Scope the autocomplete to the prediction's category where it
// maps to a tag kind. Only 'character' surfaces as both a
// suggestion category and a tag kind now ('artist' + 'copyright'
// retired); other categories search unscoped.
const kind = props.category === 'character' ? 'character' : null
const params = { q: query, limit: 20 }
if (kind) params.kind = kind
results.value = await api.get('/api/tags/autocomplete', { params })
+10 -1
View File
@@ -92,7 +92,16 @@ let prevBodyOverflow = null
// own keystrokes.
function onKeyDown(ev) {
if (ev.key === 'Escape') {
if (isTextEntry(ev.target)) return
// Escape closes the modal even from inside a text input — that's
// the universal "get me out of here" expectation, and the
// autofocused tag-entry field would otherwise trap focus with no
// visible escape (operator-flagged 2026-06-01). EXCEPTION: when a
// nested Vuetify overlay is open (v-menu autocomplete dropdown,
// FandomPicker v-dialog, per-suggestion 3-dot menu), let that
// overlay's own Esc handling fire instead of closing the whole
// modal mid-interaction. Vuetify marks open overlays with
// `.v-overlay--active`.
if (document.querySelector('.v-overlay--active')) return
ev.preventDefault()
emit('close')
} else if (ev.key === 'ArrowLeft') {
@@ -10,17 +10,30 @@
density="compact"
>{{ state.error }}</v-alert>
<template v-else>
<!-- Cards scroll independently of the section title + attachments
below them. Cap at ~2.5 cards visible (operator-asked 2026-06-01:
keeps the Tags section anchored below at a consistent position;
the half-visible third card hints there's more). -->
<div v-else class="fc-prov__cards">
<article
v-for="e in state.entries" :key="e.provenance_id" class="fc-prov__card"
>
<div class="fc-prov__head">
<span class="fc-prov__platform">{{ e.source.platform }}</span>
<!-- Posts with no live subscription have source=null (alembic
0030); render an explicit "filesystem import" affordance
instead of a platform chip. -->
<span class="fc-prov__platform">
{{ e.source?.platform ?? 'filesystem import' }}
</span>
<span v-if="postDate(e)" class="fc-prov__date">{{ postDate(e) }}</span>
</div>
<div class="fc-prov__post">
<button
type="button" class="fc-prov__post"
:title="`Open ${postTitle(e)} in the posts feed for ${e.artist.name}`"
@click="openPost(e.post.id, e.artist.id)"
>
{{ postTitle(e) }}
</div>
</button>
<div class="fc-prov__meta">
<RouterLink :to="`/artist/${e.artist.slug}`">
by {{ e.artist.name }}
@@ -29,12 +42,8 @@
· {{ e.post.attachment_count }} files
</span>
</div>
<div class="fc-prov__actions">
<a href="#" @click.prevent="openPost(e.post.id)">
View post
</a>
<div v-if="e.post.description_html" class="fc-prov__actions">
<a
v-if="e.post.description_html"
href="#" @click.prevent="toggleDesc(e.provenance_id)"
>{{ expanded[e.provenance_id] ? 'Hide description ' : 'Show description ' }}</a>
</div>
@@ -58,7 +67,7 @@
</RouterLink>
</div>
</article>
</template>
</div>
<div v-if="attachments.length" class="fc-prov__attach">
<h4 class="fc-prov__attach-title">Attachments</h4>
@@ -137,10 +146,16 @@ function postTitle(e) {
return toPlainText(e.post.title) || `Post ${e.post.external_post_id}`
}
function openPost(postId) {
function openPost(postId, artistId) {
// Land on the post in the posts feed (in context), not the gallery
// image grid. Operator-flagged 2026-05-28.
router.push({ path: '/posts', query: { post_id: postId } })
// image grid. Scope the feed to this artist so the user lands in
// that creator's stream, not the global one — operator-flagged
// 2026-06-01. PostsView reads `artist_id` from the query string
// (PostsView.vue line ~92) and filters via post_feed_service.
router.push({
path: '/posts',
query: { post_id: postId, artist_id: artistId },
})
modal.close()
}
</script>
@@ -153,6 +168,18 @@ function openPost(postId) {
color: rgb(var(--v-theme-on-surface));
margin-bottom: 12px;
}
.fc-prov__cards {
/* 2.5 cards-worth at the typical collapsed card height (~108px each
incl. 10px gap). Slightly under to ensure the third card's bottom
edge is clipped — the visual cue that there's more below. */
max-height: 270px;
overflow-y: auto;
/* Hairline scrollbar that doesn't compete with content. */
scrollbar-width: thin;
scrollbar-color: rgb(var(--v-theme-surface-light)) transparent;
/* Pad-right so the scrollbar gutter doesn't squeeze card borders. */
padding-right: 4px;
}
.fc-prov__card {
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 6px; padding: 10px 12px; margin-bottom: 10px;
@@ -163,8 +190,21 @@ function openPost(postId) {
text-transform: lowercase;
}
.fc-prov__post {
font-weight: 700; margin: 4px 0;
color: rgb(var(--v-theme-on-surface));
/* Clickable title — opens the post in the artist-scoped feed
(operator-flagged 2026-06-01: title IS the primary action, the
prior "View post" link was redundant). Styled as a button-link:
accent color, underline on hover, focus ring for keyboard nav. */
display: block; width: 100%; text-align: left;
background: none; border: none; padding: 0;
font: inherit; font-weight: 700;
margin: 4px 0;
color: rgb(var(--v-theme-accent));
cursor: pointer;
}
.fc-prov__post:hover { text-decoration: underline; }
.fc-prov__post:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 2px; border-radius: 3px;
}
.fc-prov__meta {
font-size: 13px; color: rgb(var(--v-theme-on-surface-variant));
@@ -1,29 +1,52 @@
<template>
<!-- Chip-card row: visible border + hover/focus state unifies the
name, score, and action buttons as one "object" (operator-asked
2026-06-01). The row itself is informational; the explicit
Accept button + 3-dot menu are the action affordances. -->
<div class="fc-suggestion">
<span class="fc-suggestion__name">
{{ suggestion.display_name }}
<span v-if="suggestion.creates_new_tag" class="fc-suggestion__new"
title="No matching tag yet — accepting creates it">new</span>
title="No matching tag yet — accepting creates it">+ new</span>
</span>
<span class="fc-suggestion__score">{{ scorePct }}</span>
<v-btn
icon="mdi-plus" size="x-small" variant="text" color="accent"
class="fc-suggestion__accept"
size="small" variant="tonal" color="accent"
density="compact" rounded="pill"
:aria-label="`Accept ${suggestion.display_name}`"
@click="$emit('accept', suggestion)"
/>
<v-menu>
<template #activator="{ props }">
<v-btn icon="mdi-dots-vertical" size="x-small" variant="text" v-bind="props" />
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
>
Accept
</v-btn>
<!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
Wrapping in a <span @click.stop> matches the TagPanel chip
fix — even though there's no parent click capture here today,
the wrap is harmless and keeps both kebabs on the same
pattern. Click bubbles from the v-btn opens menu via
activator props bubble continues to span stopPropagation
halts it. -->
<span class="fc-suggestion__menu-wrap" @click.stop>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for</v-list-item-title>
</v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</div>
</template>
@@ -38,17 +61,50 @@ const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
<style scoped>
.fc-suggestion {
display: flex; align-items: center; gap: 6px;
padding: 2px 0;
display: flex; align-items: center; gap: 8px;
padding: 6px 10px; margin-bottom: 4px;
background: rgb(var(--v-theme-surface));
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 6px;
transition: background 120ms ease, border-color 120ms ease;
}
.fc-suggestion:hover {
background: rgb(var(--v-theme-surface-light));
border-color: rgb(var(--v-theme-accent), 0.4);
}
.fc-suggestion__name {
flex: 1; min-width: 0;
font-size: 14px;
color: rgb(var(--v-theme-on-surface));
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
}
.fc-suggestion__name { flex: 1; min-width: 0; }
.fc-suggestion__new {
font-size: 10px; color: rgb(var(--v-theme-accent));
margin-left: 4px;
display: inline-block;
font-size: 10px; font-weight: 600;
color: rgb(var(--v-theme-accent));
background: rgba(var(--v-theme-accent), 0.12);
border: 1px solid rgb(var(--v-theme-accent), 0.4);
padding: 1px 6px; border-radius: 999px;
margin-left: 6px;
text-transform: uppercase; letter-spacing: 0.04em;
}
.fc-suggestion__score {
flex: 0 0 auto; min-width: 38px; text-align: right;
font-size: 11px;
color: rgb(var(--v-theme-on-surface-variant, var(--v-theme-on-surface)));
font-family: 'JetBrains Mono', monospace;
}
/* Vuetify's compact density doesn't shrink the tonal button enough
for a tight row; clamp the min-width so Accept stays compact. */
.fc-suggestion__accept :deep(.v-btn__content) {
font-size: 12px; letter-spacing: 0.02em;
}
.fc-suggestion__menu-wrap {
flex: 0 0 auto;
display: inline-flex;
align-items: center;
}
.fc-suggestion__menu {
flex: 0 0 auto;
}
</style>
@@ -22,7 +22,7 @@
<SuggestionsCategoryGroup
v-if="store.byCategory.general && store.byCategory.general.length"
label="General" :items="store.byCategory.general"
collapsible :default-open="false"
collapsible :default-open="true"
@accept="onAccept" @alias="onAlias" @dismiss="store.dismiss"
/>
</template>
@@ -41,13 +41,18 @@
import { toast } from '../../utils/toast.js'
import { computed, ref, watch } from 'vue'
import { useSuggestionsStore, CATEGORY_LABELS } from '../../stores/suggestions.js'
import { useModalStore } from '../../stores/modal.js'
import SuggestionsCategoryGroup from './SuggestionsCategoryGroup.vue'
import AliasPickerDialog from './AliasPickerDialog.vue'
const props = defineProps({ imageId: { type: Number, required: true } })
const store = useSuggestionsStore()
const modal = useModalStore()
const peopleCats = ['artist', 'character', 'copyright']
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as
// suggestion categories. Only 'character' remains as a people-style
// category alongside the general bucket.
const peopleCats = ['character']
function labelFor(c) { return CATEGORY_LABELS[c] || c }
const isEmpty = computed(() =>
@@ -56,9 +61,20 @@ const isEmpty = computed(() =>
watch(() => props.imageId, (id) => { if (id != null) store.load(id) }, { immediate: true })
// After a successful accept/alias-accept, refresh the modal's current
// tag list so TagPanel's chip rail reflects the newly-attached tag.
// Operator-flagged 2026-06-01: the suggestion store dropped the
// suggestion (correct) but didn't propagate the new tag back into the
// modal store, so the chip rail looked unchanged even though the
// backend had recorded the application. Mirrors the addExistingTag /
// createAndAdd flows which already call reloadTags() after applying.
async function onAccept(s) {
try { await store.accept(s) }
catch (e) { toast({ text: `Accept failed: ${e.message}`, type: 'error' }) }
try {
await store.accept(s)
await modal.reloadTags()
} catch (e) {
toast({ text: `Accept failed: ${e.message}`, type: 'error' })
}
}
const aliasDialog = ref(false)
@@ -68,6 +84,7 @@ async function onAliasConfirm(canonicalTagId) {
try {
await store.aliasAccept(aliasTarget.value, canonicalTagId)
aliasDialog.value = false
await modal.reloadTags()
} catch (e) {
toast({ text: `Alias failed: ${e.message}`, type: 'error' })
}
@@ -1,6 +1,7 @@
<template>
<div class="fc-tag-autocomplete">
<v-text-field
ref="inputRef"
v-model="query"
placeholder="Add tag (or kind:name — character/fandom/series)"
density="compact" hide-details
@@ -52,13 +53,21 @@
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { computed, nextTick, onMounted, ref, watch } from 'vue'
import { useTagStore } from '../../stores/tags.js'
import FandomPicker from './FandomPicker.vue'
const emit = defineEmits(['pick-existing', 'pick-new', 'cancel'])
const store = useTagStore()
// Autofocus on modal open so the operator can type the moment the view
// modal renders, no extra click required (operator-asked 2026-06-01).
// Vuetify's v-text-field exposes .focus() on the component instance;
// nextTick waits for the modal's mount to finish so the inner <input>
// element exists.
const inputRef = ref(null)
onMounted(() => { nextTick(() => inputRef.value?.focus?.()) })
// Single text input; no kind dropdown. Client-side mirror of the
// backend's parse_kind_prefix lives below — kept in sync with
// KNOWN_KINDS in backend/app/utils/tag_prefix.py. The backend is the
+22 -13
View File
@@ -10,19 +10,27 @@
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
<!-- Operator-flagged 2026-06-02: the previous activator had
`@click.stop` directly on the v-icon, which silently
overrode Vuetify's onClick from `v-bind="mp"` — the menu
never opened. Now the v-icon receives the activator
onClick cleanly, and the wrapping span absorbs the
bubbled click so the chip's close button isn't tripped. -->
<span class="kebab-wrap" @click.stop>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical"
/>
</template>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename…</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
</v-chip>
<span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
</div>
@@ -113,4 +121,5 @@ async function onRenamed() {
margin-bottom: 12px;
}
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
.kebab-wrap { display: inline-flex; align-items: center; }
</style>
+7 -2
View File
@@ -7,7 +7,12 @@
@keydown.enter="onCardClick"
>
<div class="fc-post-card__head">
<v-chip size="x-small" variant="tonal">{{ post.source.platform }}</v-chip>
<!-- Posts with no live subscription have source=null (alembic
0030); show a "filesystem import" affordance instead of a
platform chip. -->
<v-chip size="x-small" variant="tonal">
{{ post.source?.platform ?? 'filesystem import' }}
</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-card__artist"
@@ -25,7 +30,7 @@
v-if="post.post_url"
:href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="x-small" variant="text"
:aria-label="`open original post on ${post.source.platform}`"
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
@click.stop
/>
<v-btn
@@ -22,10 +22,10 @@ import { reactive, watch } from 'vue'
import { useMLStore } from '../../stores/ml.js'
const store = useMLStore()
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as
// suggestion categories; their threshold rows are gone.
const fields = [
{ key: 'suggestion_threshold_artist', label: 'Artist' },
{ key: 'suggestion_threshold_character', label: 'Character' },
{ key: 'suggestion_threshold_copyright', label: 'Copyright' },
{ key: 'suggestion_threshold_general', label: 'General' },
{ key: 'centroid_similarity_threshold', label: 'Centroid similarity' }
]
@@ -35,7 +35,7 @@ import { computed } from 'vue'
const props = defineProps({
queues: { type: Object, default: null }, // store.queues
workers: { type: Object, default: null }, // store.workers
recentMinute: { type: Array, default: () => [] }, // store.recentMinute
recentRuns: { type: Array, default: () => [] }, // store.recentRuns
compact: { type: Boolean, default: false },
})
@@ -80,7 +80,7 @@ function activeCount(name) {
const recentByQueue = computed(() => {
const out = {}
for (const r of props.recentMinute) {
for (const r of props.recentRuns) {
if (!out[r.queue]) out[r.queue] = { ok: 0, err: 0 }
if (r.status === 'ok') out[r.queue].ok++
else if (r.status === 'error' || r.status === 'timeout') out[r.queue].err++
@@ -24,7 +24,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
compact
/>
</v-card-text>
@@ -47,7 +47,7 @@ function pollOnce() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
onMounted(() => {
@@ -14,7 +14,7 @@
<QueuesTable
:queues="store.queues"
:workers="store.workers"
:recent-minute="store.recentMinute"
:recent-runs="store.recentRuns"
/>
</v-card-text>
</v-card>
@@ -202,7 +202,7 @@ function pollQueues() {
if (document.hidden) return
store.loadQueues()
store.loadWorkers()
store.loadRecentMinute()
store.loadRecentRuns()
}
function pollFailures() {
if (document.hidden) return
@@ -10,7 +10,14 @@
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
<v-icon start>mdi-image-refresh</v-icon> Run backfill now
</v-btn>
<span v-if="done" class="ml-3 text-caption">Enqueued.</span>
<span v-if="result" class="ml-3 text-caption">
Scanned <strong>{{ result.scanned }}</strong> · enqueued
<strong>{{ result.enqueued }}</strong>
<span v-if="result.regenerated > 0">
({{ result.regenerated }} regenerated)
</span>
· {{ result.ok }} ok
</span>
<QueueStatusBar queue="thumbnail" queue-label="Thumbnail" />
</v-card-text>
</v-card>
@@ -23,10 +30,11 @@ import { useThumbnailsStore } from '../../stores/thumbnails.js'
import QueueStatusBar from './QueueStatusBar.vue'
const store = useThumbnailsStore()
const busy = ref(false)
const done = ref(false)
const result = ref(null)
async function run () {
busy.value = true
try { await store.triggerBackfill(); done.value = true }
result.value = null
try { result.value = await store.triggerBackfill() }
catch (e) { toast({ text: e.message, type: 'error' }) }
finally { busy.value = false }
}
@@ -58,6 +58,7 @@
:retrying-all="retryingAll"
@retry="onRetrySource"
@retry-all="onRetryAll"
@view-logs="onViewFailingLogs"
/>
<div v-if="store.loading && store.events.length === 0" class="fc-dl__loading">
@@ -364,6 +365,23 @@ watch(filterModel, async (m) => {
async function openDetail(id) {
await store.loadOne(id)
}
async function onViewFailingLogs(source) {
// Find and open the most recent DownloadEvent for this source.
// Reuses the existing DownloadDetailModal — same stdout/stderr/error
// surface the row-click in the events feed shows.
try {
const ev = await store.loadLastForSource(source.id)
if (!ev) {
toast({
text: `No download events recorded for ${source.artist_name || source.platform} yet.`,
type: 'warning',
})
}
} catch (e) {
toast({ text: `Failed to load logs: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
@@ -25,10 +25,27 @@
<v-chip size="x-small" color="error" variant="flat" label class="fc-fail__count">
{{ s.consecutive_failures }}× failed
</v-chip>
<v-chip
v-if="s.error_type"
size="x-small" variant="outlined" label
:color="errorTypeColor(s.error_type)"
class="fc-fail__class"
:title="errorTypeHint(s.error_type)"
>
{{ s.error_type }}
</v-chip>
<span class="fc-fail__err" :title="s.last_error || ''">
{{ s.last_error || 'no error message recorded' }}
</span>
<v-spacer />
<v-btn
size="x-small" variant="text" prepend-icon="mdi-text-box-search-outline"
:loading="logLoadingIds.has(s.id)"
@click="onViewLogs(s)"
title="Show the most recent download event's stdout/stderr/error"
>
Logs
</v-btn>
<v-btn
size="x-small" variant="text" prepend-icon="mdi-refresh"
:loading="retryingIds.has(s.id)"
@@ -52,9 +69,59 @@ defineProps({
retryingIds: { type: Set, default: () => new Set() },
retryingAll: { type: Boolean, default: false },
})
defineEmits(['retry', 'retry-all'])
const emit = defineEmits(['retry', 'retry-all', 'view-logs'])
const open = ref(true)
// Per-row loading flag so the spinner lives on the row whose Logs
// button was clicked, not on every row.
const logLoadingIds = ref(new Set())
// Audit 2026-06-02: surface the ErrorType taxonomy as a colored chip
// next to the consecutive-failures count so operators can bulk-triage
// by error class. Color reflects "what to do next":
// warning (yellow) — auth/cookie issue: operator should rotate
// info (blue) — backend-paced (cooldown / rate limit / timeout)
// error (red) — likely terminal without operator intervention
const ERROR_TYPE_COLOR = {
auth_error: 'warning',
rate_limited: 'info',
timeout: 'info',
network_error: 'info',
not_found: 'error',
access_denied: 'error',
validation_failed: 'error',
unsupported_url: 'error',
http_error: 'error',
unknown_error: 'error',
partial: 'info',
tier_limited: 'info',
no_new_content: 'info',
}
const ERROR_TYPE_HINT = {
auth_error: 'Cookies likely expired — re-upload in Credentials.',
rate_limited: 'Platform-wide cooldown active. Will retry after it expires.',
timeout: 'Subprocess exceeded its time budget. Often retries cleanly.',
network_error: 'Transient network issue. Will retry on next tick.',
not_found: 'URL 404 — creator may have renamed or deleted.',
access_denied: 'Subscription tier may not grant this content.',
validation_failed: 'Downloaded files were quarantined by the validator.',
http_error: 'Generic HTTP error — see Logs.',
unsupported_url: 'gallery-dl does not support this URL pattern.',
unknown_error: 'Could not classify — see Logs.',
}
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
async function onViewLogs(s) {
if (logLoadingIds.value.has(s.id)) return
logLoadingIds.value = new Set(logLoadingIds.value).add(s.id)
try {
await emit('view-logs', s)
} finally {
const next = new Set(logLoadingIds.value)
next.delete(s.id)
logLoadingIds.value = next
}
}
</script>
<style scoped>
@@ -68,16 +135,32 @@ const open = ref(true)
.fc-fail__title { font-weight: 600; }
.fc-fail__body {
padding: 0 14px 10px;
display: flex; flex-direction: column; gap: 2px;
display: flex; flex-direction: column;
/* Borders, not gap, so the row separators are visible inside the
* tonal error card (where surface tints get washed out and gap is
* just empty space). */
}
.fc-fail__row {
display: flex; align-items: center; gap: 10px;
padding: 6px 8px;
padding: 8px 10px;
border-radius: 4px;
background: rgb(var(--v-theme-surface) / 0.4);
/* Hover-darken gives the eye a horizontal track from the artist
* name on the left to the Logs/Retry buttons on the right.
* Bottom border replaces the previous-too-subtle zebra striping —
* inside the tonal error card, surface-tint contrast was negligible.
* Operator-flagged 2026-06-01 (twice). */
border-bottom: 1px solid rgb(255 255 255 / 0.08);
transition: background 80ms ease;
}
.fc-fail__row:last-child {
border-bottom: none;
}
.fc-fail__row:hover {
background: rgb(0 0 0 / 0.25);
}
.fc-fail__artist { font-weight: 600; white-space: nowrap; }
.fc-fail__count { flex: 0 0 auto; }
.fc-fail__class { flex: 0 0 auto; }
.fc-fail__err {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.8rem;
@@ -31,6 +31,12 @@
v-if="(source.consecutive_failures || 0) > 0"
size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }}</v-chip>
<v-chip
v-else-if="(source.backfill_runs_remaining || 0) > 0"
size="x-small" color="info" variant="tonal" label
>
backfill ({{ source.backfill_runs_remaining }}×)
</v-chip>
<span v-else class="fc-source-row__zero">0</span>
</td>
<td class="fc-source-row__actions">
@@ -42,6 +48,16 @@
<v-icon>mdi-play</v-icon>
<v-tooltip activator="parent" location="top">Check now</v-tooltip>
</v-btn>
<v-btn
icon="mdi-magnify-scan" size="x-small" variant="text"
:disabled="(source.backfill_runs_remaining || 0) > 0"
@click.stop="$emit('backfill', source)"
>
<v-icon>mdi-magnify-scan</v-icon>
<v-tooltip activator="parent" location="top">
Deep scan walk full history for next few runs
</v-tooltip>
</v-btn>
<v-btn
icon="mdi-pencil" size="x-small" variant="text"
@click.stop="$emit('edit', source)"
@@ -69,7 +85,7 @@ const props = defineProps({
checking: { type: Boolean, default: false },
warningThreshold: { type: Number, default: 5 },
})
const emit = defineEmits(['edit', 'remove', 'toggle', 'check'])
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill'])
function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value })
@@ -174,6 +174,7 @@
@remove="removeSource"
@toggle="toggleSourceEnabled"
@check="onCheck"
@backfill="onBackfill"
/>
<tr v-if="item.sources.length === 0">
<td colspan="8" class="fc-subs__sources-empty">
@@ -411,6 +412,17 @@ function onArtistCreated(artist) {
async function onCheck(source) {
try {
const body = await store.checkNow(source.id)
// Audit 2026-06-02: /api/sources/<id>/check returns 202 with
// `{status:'deferred', cooldown_until}` when the platform is in
// cooldown — the previous handler treated this as success and
// toasted "event #undefined", masking that nothing was enqueued.
if (body?.status === 'deferred') {
toast({
text: 'Check deferred — platform in cooldown',
type: 'info',
})
return
}
toast({
text: `Check enqueued (event #${body.download_event_id})`,
type: 'success',
@@ -431,20 +443,56 @@ async function onCheck(source) {
}
}
// Plan #544: arm a source for backfill mode (gallery-dl walks the full
// post history) for the next N download runs. Default 3 — enough budget
// to finish a deep creator without re-prompting the operator across
// timeout boundaries. The chip on the row reflects the remaining count.
async function onBackfill(source) {
const raw = globalThis.window?.prompt(
`Deep scan "${source.artist_name} (${source.platform})" — walk full history for the next how many download runs? (110, default 3)`,
'3',
)
if (raw == null) return
const runs = parseInt(raw, 10)
if (!Number.isFinite(runs) || runs < 1 || runs > 10) {
toast({ text: 'Deep scan: runs must be 110', type: 'error' })
return
}
try {
await store.setBackfill(source.id, runs, source.artist_id)
toast({
text: `Deep scan armed for ${runs} run${runs === 1 ? '' : 's'}`,
type: 'success',
})
await store.loadAll()
} catch (e) {
toast({
text: `Deep scan failed: ${e?.detail || e?.message || e}`,
type: 'error',
})
}
}
async function checkAll(group) {
let ok = 0
let conflict = 0
let deferred = 0
for (const s of group.sources) {
if (!s.enabled) continue
try {
await store.checkNow(s.id)
ok += 1
const body = await store.checkNow(s.id)
// Audit 2026-06-02: deferred (202 + cooldown_until) used to be
// counted as queued, inflating the success tally and hiding
// that the cooldown actually held the work back.
if (body?.status === 'deferred') deferred += 1
else ok += 1
} catch (e) {
if (e?.body?.download_event_id) conflict += 1
}
}
const parts = []
if (ok) parts.push(`${ok} queued`)
if (deferred) parts.push(`${deferred} deferred (cooldown)`)
if (conflict) parts.push(`${conflict} already running`)
toast({
text: parts.join(', ') || 'Nothing to check (no enabled sources)',
@@ -0,0 +1,52 @@
// Inflight-token guard for stores whose async loads can be re-triggered
// by rapid filter/navigation changes. Without this, late responses
// from a prior load overwrite the store with stale data
// (last-writer-wins, not request-order-wins). gallery.js had a
// hand-rolled `inflightId` of the same shape; this composable
// extracts it so every store can use the same pattern.
//
// Audit 2026-06-02 (workflow wf_bbe3fdb1-e62) found this missing in
// modal/suggestions/artist/downloads/directory/posts. The two most
// operator-impacting consequences: (1) modal tag mutations could
// land DELETE/POST on the wrong image when the user navigated mid-
// flight; (2) suggestions accept could push a tag to the wrong
// image AND add it to the allowlist.
//
// Usage:
// const inflight = useInflightToken()
//
// async function load() {
// const t = inflight.claim()
// const body = await api.get(...)
// if (!t.isCurrent()) return // stale — abort write
// items.value = body.items // safe to commit
// }
//
// function setFilter(f) {
// inflight.cancel() // any in-flight token is now stale
// filter.value = f
// load() // claims a fresh token
// }
//
// For multi-await flows (POST then GET, optimistic mutation then
// reconcile), check isCurrent() after EACH await — any intervening
// claim() or cancel() invalidates the prior token.
export function useInflightToken() {
let _seq = 0
let _current = 0
function claim() {
_current = ++_seq
const id = _current
return {
id,
isCurrent: () => _current === id,
}
}
function cancel() {
_current = ++_seq
}
return { claim, cancel }
}
+16 -3
View File
@@ -1,6 +1,7 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js'
import { usePostsStore } from './posts.js'
const PAGE = 60
@@ -15,12 +16,17 @@ export const useArtistStore = defineStore('artist', () => {
const error = ref(null)
const notFound = ref(false)
let started = false
// Rapid artist-to-artist navigation used to render the previous
// artist's overview/images briefly when the second load resolved
// after the third. Audit 2026-06-02.
const inflight = useInflightToken()
async function load (slug) {
// Cross-artist reset: clear this store AND the posts store so the new
// artist doesn't briefly render with the previous artist's content
// when the user is on the Posts tab. (Gallery tab uses this artist
// store's own images list — cleared above.)
inflight.cancel()
overview.value = null
images.value = []
nextCursor.value = null
@@ -29,14 +35,18 @@ export const useArtistStore = defineStore('artist', () => {
error.value = null
loading.value = true
usePostsStore().$reset?.()
const t = inflight.claim()
try {
overview.value = await api.get(`/api/artist/${encodeURIComponent(slug)}`)
const body = await api.get(`/api/artist/${encodeURIComponent(slug)}`)
if (!t.isCurrent()) return
overview.value = body
await loadMoreImages(slug)
} catch (e) {
if (!t.isCurrent()) return
if (e.status === 404) notFound.value = true
else error.value = e.message
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
@@ -44,19 +54,22 @@ export const useArtistStore = defineStore('artist', () => {
if (imagesLoading.value) return
if (started && nextCursor.value === null) return
imagesLoading.value = true
const t = inflight.claim()
try {
const params = { limit: PAGE }
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get(
`/api/artist/${encodeURIComponent(slug)}/images`, { params }
)
if (!t.isCurrent()) return
images.value.push(...body.images)
nextCursor.value = body.next_cursor
started = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e.message
} finally {
imagesLoading.value = false
if (t.isCurrent()) imagesLoading.value = false
}
}
+10 -1
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
const PAGE = 60
@@ -13,16 +14,23 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
const q = ref('')
const platform = ref(null)
let started = false
// Typed "alice" then "alice bob" used to drop the second fetch
// entirely (loading flag still true from the first), so the UI
// showed alice results while the input said "alice bob". Inflight
// token + reset() cancelling in-flight requests fixes both: the
// first response is discarded, the second is fetched. Audit 2026-06-02.
const inflight = useInflightToken()
async function loadMore() {
if (loading.value) return
if (started && nextCursor.value === null) return
const t = inflight.claim()
await run(async () => {
const params = { limit: PAGE }
if (q.value) params.q = q.value
if (platform.value) params.platform = platform.value
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/artists/directory', { params })
if (!t.isCurrent()) return
cards.value.push(...body.cards)
nextCursor.value = body.next_cursor
started = true
@@ -30,6 +38,7 @@ export const useArtistDirectoryStore = defineStore('artistDirectory', () => {
}
async function reset() {
inflight.cancel()
cards.value = []
nextCursor.value = null
started = false
+5 -1
View File
@@ -23,7 +23,11 @@ export const useCredentialsStore = defineStore('credentials', () => {
const rec = await api.post('/api/credentials', {
body: { platform, credential_type, data },
})
byPlatform.value.delete(platform)
// Reflect the returned record immediately — the previous .delete()
// call left the card rendering "no credential" for the gap between
// upload completion and the caller's follow-up loadAll(). Audit
// 2026-06-02.
byPlatform.value.set(platform, rec)
return rec
}
+29 -1
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const useDownloadsStore = defineStore('downloads', () => {
const api = useApi()
@@ -22,6 +23,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
// the "active now" panel always reflects what's happening regardless of
// how the operator has filtered the historical list below.
const activeEvents = ref([])
// Filter changes (applyFilter) and rapid pagination can interleave
// responses; without an inflight guard the late response from a
// prior filter overwrites the current view. Audit 2026-06-02.
const inflight = useInflightToken()
function _params(extra = {}) {
const out = { limit: 50, ...extra }
@@ -32,8 +37,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
}
async function loadFirst() {
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/downloads', { params: _params() })
if (!t.isCurrent()) return
events.value = body
cursor.value = body.length ? body[body.length - 1].id : null
hasMore.value = body.length === 50
@@ -42,8 +49,10 @@ export const useDownloadsStore = defineStore('downloads', () => {
async function loadMore() {
if (!hasMore.value || cursor.value == null) return
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/downloads', { params: _params({ before: cursor.value }) })
if (!t.isCurrent()) return
events.value.push(...body)
cursor.value = body.length ? body[body.length - 1].id : cursor.value
hasMore.value = body.length === 50
@@ -55,7 +64,25 @@ export const useDownloadsStore = defineStore('downloads', () => {
return selected.value
}
// Open the detail modal for the most recent DownloadEvent of a given
// source. Used by the failing-sources rollup's "Logs" button so the
// operator can troubleshoot without leaving the Downloads tab to find
// the row (operator-flagged 2026-06-01).
async function loadLastForSource(sourceId) {
const events = await api.get('/api/downloads', {
params: { source_id: sourceId, limit: 1 },
})
if (!events.length) {
selected.value = null
return null
}
return await loadOne(events[0].id)
}
async function applyFilter(patch) {
// Drop any in-flight loadFirst/loadMore from the previous filter
// so its late response doesn't overwrite this filter's results.
inflight.cancel()
filter.value = { ...filter.value, ...patch }
await loadFirst()
}
@@ -98,7 +125,8 @@ export const useDownloadsStore = defineStore('downloads', () => {
return {
events, cursor, hasMore, filter, selected, loading, error, stats,
activity, failing, activeEvents,
loadFirst, loadMore, loadOne, applyFilter, closeDetail, loadStats,
loadFirst, loadMore, loadOne, loadLastForSource, applyFilter,
closeDetail, loadStats,
loadActivity, loadFailing, loadActive, recoverStalled,
}
})
+14 -4
View File
@@ -1,6 +1,7 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js'
// Operator-confirmed 2026-05-30: fetch PAGE-sized chunks instead of one
// 50-item request so items render as each batch lands. Total initial
@@ -22,9 +23,12 @@ export const useGalleryStore = defineStore('gallery', () => {
const timelineBuckets = ref([])
const timelineLoading = ref(false)
let inflightId = 0
// Was a hand-rolled inflightId counter; the audit-2026-06-02 fan-out
// moved this pattern into useInflightToken so every store can share it.
const inflight = useInflightToken()
async function loadInitial() {
inflight.cancel()
images.value = []
dateGroups.value = []
nextCursor.value = null
@@ -41,19 +45,19 @@ export const useGalleryStore = defineStore('gallery', () => {
if (loading.value) return
loading.value = true
error.value = null
const myId = ++inflightId
const t = inflight.claim()
try {
const params = { limit: PAGE, ...activeFilterParam() }
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/gallery/scroll', { params })
if (myId !== inflightId) return // stale response
if (!t.isCurrent()) return
images.value.push(...body.images)
dateGroups.value = mergeGroups(dateGroups.value, body.date_groups)
nextCursor.value = body.next_cursor
} catch (e) {
error.value = e.message
} finally {
if (myId === inflightId) loading.value = false
if (t.isCurrent()) loading.value = false
}
}
@@ -68,8 +72,14 @@ export const useGalleryStore = defineStore('gallery', () => {
}
async function jumpTo(year, month) {
// Rapid timeline-jump clicks need the same race guard as
// loadMore — first jump's late body could clobber the second
// jump's already-applied state.
inflight.cancel()
const t = inflight.claim()
const params = { year, month, ...activeFilterParam() }
const body = await api.get('/api/gallery/jump', { params })
if (!t.isCurrent()) return
if (body.cursor) {
images.value = []
dateGroups.value = []
+75 -12
View File
@@ -3,6 +3,7 @@ import { toast } from '../utils/toast.js'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const useModalStore = defineStore('modal', () => {
const api = useApi()
@@ -10,6 +11,11 @@ export const useModalStore = defineStore('modal', () => {
const currentImageId = ref(null)
const current = ref(null)
const { loading, error, run } = useAsyncAction({ errorAs: 'message' })
// Tag mutations interpolate the image id into the URL after an
// await; without an inflight token, a fast prev/next can route the
// DELETE/POST to the wrong image AND the response to the wrong
// chip rail. Audit 2026-06-02.
const inflight = useInflightToken()
// Post-scoped cycle. When set, prev/next cycles within this array
// (used by PostCard's expanded-mosaic PostImageGrid clicks). When
@@ -19,6 +25,9 @@ export const useModalStore = defineStore('modal', () => {
const postImageIndex = ref(0)
async function open (id, opts = {}) {
// Cancel any in-flight tag mutation or reloadTags from the
// previous image so its late response can't apply to this one.
inflight.cancel()
currentImageId.value = id
current.value = null // cleared upfront so it stays null on error
// Update post-scoped state if caller passed it; otherwise clear so
@@ -31,12 +40,16 @@ export const useModalStore = defineStore('modal', () => {
postImageIds.value = null
postImageIndex.value = 0
}
const t = inflight.claim()
await run(async () => {
current.value = await api.get(`/api/gallery/image/${id}`)
const body = await api.get(`/api/gallery/image/${id}`)
if (!t.isCurrent()) return
current.value = body
})
}
async function close () {
inflight.cancel()
currentImageId.value = null
current.value = null
error.value = null
@@ -75,37 +88,87 @@ export const useModalStore = defineStore('modal', () => {
}
async function reloadTags () {
if (!currentImageId.value) return
const tags = await api.get(`/api/images/${currentImageId.value}/tags`)
if (current.value) current.value.tags = tags
// Capture image id at call-time. After an await, currentImageId
// may have advanced via prev/next navigation; without capture, the
// GET would target the new image and write its tags onto a chip
// rail the user didn't open. Audit 2026-06-02.
const imageId = currentImageId.value
if (!imageId) return
const t = inflight.claim()
const tags = await api.get(`/api/images/${imageId}/tags`)
if (!t.isCurrent()) return
// Only commit if the modal is still showing this image — guards
// against close() / nav clearing current between the await and now.
if (current.value && currentImageId.value === imageId) {
current.value.tags = tags
}
}
async function removeTag (tagId) {
if (!currentImageId.value) return
const imageId = currentImageId.value
if (!imageId) return
const prev = current.value.tags
// Optimistic UI: drop the chip immediately.
current.value.tags = current.value.tags.filter(t => t.id !== tagId)
// Split the two POSTs so a dismiss failure (secondary side-effect)
// doesn't roll back the successful DELETE — previously the catch
// unconditionally restored the chip rail even when only the
// dismiss had failed, so the UI lied until refresh. Audit 2026-06-02.
try {
await api.delete(`/api/images/${currentImageId.value}/tags/${tagId}`)
await api.post(`/api/images/${currentImageId.value}/suggestions/dismiss`, {
await api.delete(`/api/images/${imageId}/tags/${tagId}`)
} catch (e) {
// Real failure: roll back, surface, rethrow.
if (current.value && currentImageId.value === imageId) {
current.value.tags = prev
}
toast({ text: `Failed to remove tag: ${e.message}`, type: 'error' })
throw e
}
// DELETE landed. The dismiss is fire-and-best-effort — log on
// failure but DON'T roll back the chip rail; the tag is gone
// server-side regardless.
try {
await api.post(`/api/images/${imageId}/suggestions/dismiss`, {
body: { tag_id: tagId },
})
} catch (e) {
current.value.tags = prev
toast({ text: `Failed to remove tag: ${e.message}`, type: 'error' })
throw e
toast({
text: `Tag removed, but failed to dismiss suggestion: ${e.message}`,
type: 'warning',
})
}
}
async function addExistingTag (tagId) {
if (!currentImageId.value) return
await api.post(`/api/images/${currentImageId.value}/tags`, {
const imageId = currentImageId.value
if (!imageId) return
await api.post(`/api/images/${imageId}/tags`, {
body: { tag_id: tagId, source: 'manual' },
})
await reloadTags()
}
async function createAndAdd ({ name, kind, fandom_id = null }) {
// Capture imageId so the post-create reloadTags / addExistingTag
// flow stays bound to the image the user clicked on, even if
// they navigated during the /api/tags POST.
const imageId = currentImageId.value
if (!imageId) return
const tag = await api.post('/api/tags', { body: { name, kind, fandom_id } })
// Audit 2026-06-02: a kind='fandom' created here used to be
// invisible to FandomPicker until a full page reload — its load
// gates on fandomCache.length, so a non-empty cache skips the
// refetch and the new fandom never appears. Push it into the
// cache directly so the next open sees it.
if (kind === 'fandom') {
const { useTagStore } = await import('./tags.js')
const tagStore = useTagStore()
tagStore.fandomCache.push({
id: tag.id, name: tag.name, kind: 'fandom',
fandom_id: null, fandom_name: null, image_count: 0,
})
}
if (currentImageId.value !== imageId) return // navigated away
await addExistingTag(tag.id)
}
+54 -7
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
export const usePostsStore = defineStore('posts', () => {
const api = useApi()
@@ -19,6 +20,12 @@ export const usePostsStore = defineStore('posts', () => {
const doneOlder = ref(false)
const doneNewer = ref(false)
const anchorId = ref(null)
// loadInitial, loadMore, loadAround, loadOlder, loadNewer all share
// one `loading` flag and previously had no inflight guard. A filter
// change (loadInitial) racing a still-in-flight loadMore would
// append the prior filter's items into the new filter's feed.
// Audit 2026-06-02.
const inflight = useInflightToken()
function _qs() {
const q = {}
@@ -36,6 +43,7 @@ export const usePostsStore = defineStore('posts', () => {
}
async function loadInitial(newFilters) {
inflight.cancel()
filters.value = {
artist_id: newFilters?.artist_id ?? null,
platform: newFilters?.platform ?? null,
@@ -46,8 +54,10 @@ export const usePostsStore = defineStore('posts', () => {
async function loadMore() {
if (loading.value || done.value) return
const t = inflight.claim()
await run(async () => {
const body = await api.get('/api/posts', { params: _qs() })
if (!t.isCurrent()) return
items.value.push(...body.items)
cursor.value = body.next_cursor
if (body.next_cursor == null) done.value = true
@@ -59,14 +69,40 @@ export const usePostsStore = defineStore('posts', () => {
return await api.get(`/api/posts/${id}`)
}
// Filter overlay for the around/older/newer (in-context anchored)
// path. Keep this distinct from `filters.value` (the down-only feed)
// so a normal-feed filter change doesn't leak into an active anchored
// view (or vice versa). Caller of loadAround passes the snapshot; the
// subsequent loadOlder/loadNewer use it verbatim.
function _aroundParams(extra) {
const p = { ...extra }
if (filters.value.artist_id != null) p.artist_id = filters.value.artist_id
if (filters.value.platform) p.platform = filters.value.platform
return p
}
// Load a window centered on `postId`: newer posts above, the post, older
// posts below. Sets both directional cursors for subsequent scrolling.
async function loadAround(postId) {
// Accepts the same filter shape as loadInitial so the anchored view
// stays artist/platform-scoped (operator-flagged 2026-06-01: clicking a
// post title from the modal's Provenance card opens the post in the
// posts feed; without this the older/newer scroll loaded unfiltered
// global posts instead of staying in the artist's stream).
async function loadAround(postId, newFilters) {
inflight.cancel()
filters.value = {
artist_id: newFilters?.artist_id ?? null,
platform: newFilters?.platform ?? null,
}
loading.value = true
error.value = null
anchorId.value = null
const t = inflight.claim()
try {
const body = await api.get('/api/posts', { params: { around: postId } })
const body = await api.get('/api/posts', {
params: _aroundParams({ around: postId }),
})
if (!t.isCurrent()) return
items.value = body.items
cursorOlder.value = body.cursor_older
cursorNewer.value = body.cursor_newer
@@ -74,43 +110,54 @@ export const usePostsStore = defineStore('posts', () => {
doneNewer.value = body.cursor_newer == null
anchorId.value = body.anchor_id
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
async function loadOlder() {
if (loading.value || doneOlder.value || cursorOlder.value == null) return
loading.value = true
const t = inflight.claim()
try {
const body = await api.get('/api/posts', {
params: { cursor: cursorOlder.value, direction: 'older' },
params: _aroundParams({
cursor: cursorOlder.value, direction: 'older',
}),
})
if (!t.isCurrent()) return
items.value.push(...body.items)
cursorOlder.value = body.next_cursor
if (body.next_cursor == null) doneOlder.value = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
async function loadNewer() {
if (loading.value || doneNewer.value || cursorNewer.value == null) return
loading.value = true
const t = inflight.claim()
try {
const body = await api.get('/api/posts', {
params: { cursor: cursorNewer.value, direction: 'newer' },
params: _aroundParams({
cursor: cursorNewer.value, direction: 'newer',
}),
})
if (!t.isCurrent()) return
items.value.unshift(...body.items)
cursorNewer.value = body.next_cursor
if (body.next_cursor == null) doneNewer.value = true
} catch (e) {
if (!t.isCurrent()) return
error.value = e
} finally {
loading.value = false
if (t.isCurrent()) loading.value = false
}
}
+31 -6
View File
@@ -19,6 +19,16 @@ import { useApi } from '../composables/useApi.js'
const PAGE = 3
const INITIAL_BATCHES = 20
const APPEND_DELAY_MS = 80 // ≈ the MasonryGrid stagger animation (70 ms)
// Operator-flagged 2026-06-01: scrolling the showcase eventually hit a
// premature "End." because /api/showcase returns a *random sample* and
// after enough scrolling the `seen` Set accumulated enough to fully
// collide with a 3-item batch. The showcase is supposed to be endless;
// only a genuinely empty API response (library has zero images) should
// mark it exhausted. Retry up to FETCH_RETRY_CAP times on all-dupe
// batches; only flip `exhausted` when the API returns 0 items OR every
// retry came back dupe-only (graceful fallback for tiny libraries
// where retries will keep returning the same handful of items).
const FETCH_RETRY_CAP = 8
function _sleep(ms) { return new Promise(r => setTimeout(r, ms)) }
@@ -48,17 +58,32 @@ export const useShowcaseStore = defineStore('showcase', () => {
}
}
// Single batch — used by infinite-scroll appends. Trickles its 5 items
// in for the same one-at-a-time cadence as the initial load.
// Single batch — used by infinite-scroll appends. Trickles its items
// in for the same one-at-a-time cadence as the initial load. Retries
// up to FETCH_RETRY_CAP times when the API's random sample comes back
// all-duplicates (the showcase is endless by design; only a genuinely
// empty API response should mark it exhausted, not an unlucky sample).
async function fetchPage() {
if (loading.value) return
loading.value = true
error.value = null
try {
const body = await api.get('/api/showcase', { params: { limit: PAGE } })
const fresh = (body.images || []).filter(i => !seen.has(i.id))
if (fresh.length === 0) { exhausted.value = true; return }
await _trickleAppend(fresh, _seq)
for (let attempt = 0; attempt < FETCH_RETRY_CAP; attempt++) {
const body = await api.get('/api/showcase', { params: { limit: PAGE } })
const items = body.images || []
// API genuinely empty → library is empty / endpoint exhausted.
if (items.length === 0) { exhausted.value = true; return }
const fresh = items.filter(i => !seen.has(i.id))
if (fresh.length > 0) {
await _trickleAppend(fresh, _seq)
return
}
// All-dupes batch — keep trying. Showcase is endless by intent.
}
// Retry cap hit with zero fresh items: library is probably much
// smaller than the running `seen` set, fall back to exhausted so
// the UI stops trying. Operator can shuffle to reset `seen`.
exhausted.value = true
} catch (e) {
error.value = e.message || String(e)
} finally {
+10
View File
@@ -85,6 +85,15 @@ export const useSourcesStore = defineStore('sources', () => {
}
}
// Plan #544: arm a source for backfill mode. The next `runs` download
// runs (default 3) walk gallery-dl's full post history instead of
// exiting early at the first contiguous archived block.
async function setBackfill(id, runs = 3, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { runs } })
_invalidate(artistIdHint ?? body.artist_id)
return body
}
function sourcesByArtistGrouped() {
// returns [{artist: {id,name,slug}, sources: [...]}, ...]
const arr = byArtist.value.get(null) ?? []
@@ -111,6 +120,7 @@ export const useSourcesStore = defineStore('sources', () => {
loadAll, loadForArtist,
create, update, remove,
checkNow,
setBackfill,
findOrCreateArtist, autocompleteArtist,
loadScheduleStatus,
sourcesByArtistGrouped,
+41 -10
View File
@@ -3,13 +3,14 @@ import { toast } from '../utils/toast.js'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
// Category display order: people/sources first, general last.
export const CATEGORY_ORDER = ['artist', 'character', 'copyright', 'general']
// Category display order: people first, general last.
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired — only
// character and general surface as suggestion categories now.
export const CATEGORY_ORDER = ['character', 'general']
export const CATEGORY_LABELS = {
artist: 'Artist',
character: 'Character',
copyright: 'Copyright',
general: 'General'
}
@@ -18,12 +19,25 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
const byCategory = ref({}) // { category: [suggestion, ...] }
const { loading, error, run } = useAsyncAction({ errorAs: 'message' })
let currentImageId = null
// Audit 2026-06-02: this store had no inflight guard — a late
// /suggestions response from a prior image could overwrite
// byCategory while currentImageId pointed at a new one, and
// accept() dereferenced currentImageId AFTER an awaited POST so
// the subsequent /suggestions/accept could apply A's chosen tag
// to image B (and push it to the allowlist). Both fixed below
// by capturing imageId at call-time and gating writes on the token.
const inflight = useInflightToken()
async function load(imageId) {
// Cancel any in-flight load from the previous image so its late
// response can't overwrite this image's byCategory.
inflight.cancel()
currentImageId = imageId
byCategory.value = {} // cleared upfront so it stays empty on error
const t = inflight.claim()
await run(async () => {
const body = await api.get(`/api/images/${imageId}/suggestions`)
if (!t.isCurrent()) return
byCategory.value = body.by_category || {}
})
}
@@ -35,6 +49,11 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function accept(suggestion) {
// Capture imageId so a mid-flight prev/next can't reroute the
// accept POST to a different image AND push the tag to that
// image's allowlist.
const imageId = currentImageId
if (imageId == null) return
// Raw tags (creates_new_tag) have no canonical_tag_id; the backend's
// accept endpoint needs a tag_id, so for raw tags we create the tag
// first via the existing /api/tags endpoint, then accept by id.
@@ -45,10 +64,14 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
})
tagId = created.id
}
await api.post(`/api/images/${currentImageId}/suggestions/accept`, {
await api.post(`/api/images/${imageId}/suggestions/accept`, {
body: { tag_id: tagId }
})
_drop(suggestion.category, s => s === suggestion)
// Only drop from THIS image's category list — if the user navigated,
// the new image has its own suggestions and this drop would corrupt them.
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
toast({
text: `Tagged: ${suggestion.display_name}`,
type: 'success'
@@ -56,14 +79,18 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function aliasAccept(suggestion, canonicalTagId) {
await api.post(`/api/images/${currentImageId}/suggestions/alias`, {
const imageId = currentImageId
if (imageId == null) return
await api.post(`/api/images/${imageId}/suggestions/alias`, {
body: {
alias_string: suggestion.display_name,
alias_category: suggestion.category,
canonical_tag_id: canonicalTagId
}
})
_drop(suggestion.category, s => s === suggestion)
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
toast({
text: `Aliased & tagged: ${suggestion.display_name}`,
type: 'success'
@@ -71,15 +98,19 @@ export const useSuggestionsStore = defineStore('suggestions', () => {
}
async function dismiss(suggestion) {
const imageId = currentImageId
if (imageId == null) return
// Dismiss needs a tag_id; raw tags have none, so dismissing a raw
// suggestion just hides it client-side (nothing to persist a rejection
// against until the tag exists).
if (suggestion.canonical_tag_id != null) {
await api.post(`/api/images/${currentImageId}/suggestions/dismiss`, {
await api.post(`/api/images/${imageId}/suggestions/dismiss`, {
body: { tag_id: suggestion.canonical_tag_id }
})
}
_drop(suggestion.category, s => s === suggestion)
if (currentImageId === imageId) {
_drop(suggestion.category, s => s === suggestion)
}
}
return {
+5 -5
View File
@@ -9,7 +9,7 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
// Live polled state.
const queues = ref(null) // { queues: {name: depth|null}, fetched_at }
const workers = ref(null) // { workers: {hostname: {...}}, fetched_at }
const recentMinute = ref([]) // last-60s rows (for Overview summary)
const recentRuns = ref([]) // last-60s rows (for Overview summary)
const failures = ref(null) // { recent, count_by_type, since }
// Paginated runs (Activity tab "All recent activity" pane).
@@ -45,14 +45,14 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
}
async function loadRecentMinute() {
async function loadRecentRuns() {
// Used by the Overview summary card: pull last 60s of runs to compute
// per-queue ok/err counts. One call covers all queues; UI groups.
try {
const body = await api.get('/api/system/activity/runs', {
params: { limit: 200 },
})
recentMinute.value = body.runs || []
recentRuns.value = body.runs || []
} catch (e) {
lastError.value = e.message
}
@@ -106,10 +106,10 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
return {
queues, workers, recentMinute, failures, summary,
queues, workers, recentRuns, failures, summary,
runs, runsCursor, runsHasMore, runsFilter,
loading, lastError,
loadQueues, loadWorkers, loadRecentMinute,
loadQueues, loadWorkers, loadRecentRuns,
loadRuns, loadFailures, loadSummary, setFilter,
}
})
+8 -1
View File
@@ -2,6 +2,7 @@ import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useAsyncAction } from '../composables/useAsyncAction.js'
import { useInflightToken } from '../composables/useInflightToken.js'
const PAGE = 60
@@ -13,16 +14,21 @@ export const useTagDirectoryStore = defineStore('tagDirectory', () => {
const kind = ref(null)
const q = ref('')
let started = false
// Same shape as artistDirectory — rapid setQuery/setKind dropped
// the second fetch because loading was still true from the first.
// Audit 2026-06-02.
const inflight = useInflightToken()
async function loadMore() {
if (loading.value) return
if (started && nextCursor.value === null) return
const t = inflight.claim()
await run(async () => {
const params = { limit: PAGE }
if (kind.value) params.kind = kind.value
if (q.value) params.q = q.value
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/tags/directory', { params })
if (!t.isCurrent()) return
cards.value.push(...body.cards)
nextCursor.value = body.next_cursor
started = true
@@ -30,6 +36,7 @@ export const useTagDirectoryStore = defineStore('tagDirectory', () => {
}
async function reset() {
inflight.cancel()
cards.value = []
nextCursor.value = null
started = false
+4 -1
View File
@@ -4,8 +4,11 @@ import { useApi } from '../composables/useApi.js'
export const useThumbnailsStore = defineStore('thumbnails', () => {
const api = useApi()
// Returns { scanned, enqueued, ok, regenerated } — the API now runs
// the scan synchronously so the operator gets immediate feedback
// instead of just a Celery task id with no visible outcome.
async function triggerBackfill () {
await api.post('/api/thumbnails/backfill')
return await api.post('/api/thumbnails/backfill')
}
return { triggerBackfill }
+2 -17
View File
@@ -24,7 +24,7 @@
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useRoute } from 'vue-router'
import { useGalleryStore } from '../stores/gallery.js'
import { useModalStore } from '../stores/modal.js'
import GalleryGrid from '../components/gallery/GalleryGrid.vue'
@@ -37,7 +37,6 @@ import { useGallerySelectionStore } from '../stores/gallerySelection.js'
const store = useGalleryStore()
const modal = useModalStore()
const sel = useGallerySelectionStore()
const router = useRouter()
const route = useRoute()
onMounted(async () => {
@@ -47,9 +46,6 @@ onMounted(async () => {
else if (!isNaN(tagId)) store.setTagFilter(tagId)
await store.loadInitial()
await store.loadTimeline()
// Open modal if URL has ?image=N
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.tag_id, (q) => {
@@ -64,19 +60,8 @@ watch(() => route.query.post_id, (q) => {
store.setPostFilter(isNaN(postId) ? null : postId)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage(id) {
router.push({ query: { ...route.query, image: id } })
}
function closeImage() {
const q = { ...route.query }
delete q.image
router.push({ query: q })
modal.open(id)
}
</script>
+9 -1
View File
@@ -161,7 +161,15 @@ function setupAroundObservers() {
}
async function loadAroundAndAnchor() {
teardownFeed()
await store.loadAround(postIdFilter.value)
// Pass artist_id + platform through so the anchored view stays
// scoped — the older/newer infinite scrolls then read these filters
// back via the store's _aroundParams (operator-flagged 2026-06-01:
// post-title click from the modal landed scoped but the scroll then
// pulled unfiltered global posts).
await store.loadAround(postIdFilter.value, {
artist_id: artistFilter.value,
platform: platformFilter.value,
})
await nextTick()
const el = document.getElementById(`fc-post-${store.anchorId}`)
if (el) el.scrollIntoView({ block: 'center' })
+6 -1
View File
@@ -70,7 +70,12 @@ function onKeydown(e) {
}
onMounted(() => {
if (store.images.length === 0) store.loadInitial()
// Operator-stated 2026-06-02: every load of the showcase view should
// show new images. Pinia persists the store across navigations, so
// returning to /showcase used to render the same set as before. Now
// we always reshuffle on mount — loadInitial() resets `seen`,
// `images`, and `exhausted` and pipelines a fresh batch.
store.loadInitial()
window.addEventListener('keydown', onKeydown)
})
onUnmounted(() => window.removeEventListener('keydown', onKeydown))
+9 -4
View File
@@ -27,12 +27,17 @@ describe('credentials store', () => {
expect(s.byPlatform.get('patreon').credential_type).toBe('cookies')
})
it('upload invalidates the cache', async () => {
it('upload reflects the returned record into the cache', async () => {
// The previous behavior was to delete the cache entry on upload, which
// briefly showed "no credential" until a follow-up loadAll() resolved.
// Audit 2026-06-02 corrected this: upload now stores the returned
// record directly so the card updates immediately.
const s = useCredentialsStore()
s.byPlatform.set('patreon', { platform: 'patreon' })
stubFetch(() => ({ status: 201, body: { platform: 'patreon', credential_type: 'cookies' } }))
s.byPlatform.set('patreon', { platform: 'patreon', credential_type: 'token' })
const fresh = { platform: 'patreon', credential_type: 'cookies' }
stubFetch(() => ({ status: 201, body: fresh }))
await s.upload('patreon', 'cookies', 'NETSCAPE_CONTENT')
expect(s.byPlatform.has('patreon')).toBe(false)
expect(s.byPlatform.get('patreon')).toEqual(fresh)
})
it('remove deletes and invalidates the cache', async () => {
+14 -7
View File
@@ -8,18 +8,25 @@ migration code paths.
import os
import pytest
import pytest_asyncio
from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import (
# Audit 2026-06-02: CredentialCrypto now refuses to auto-generate a
# Fernet key without explicit opt-in (production safety against silent
# key regeneration on partial restore). The test environment never has
# a pre-seeded key file, so set the bootstrap flag here before any
# create_app() / CredentialCrypto() import path fires.
os.environ.setdefault("CURATOR_BOOTSTRAP_NEW_KEY", "1")
import pytest # noqa: E402
import pytest_asyncio # noqa: E402
from sqlalchemy import create_engine # noqa: E402
from sqlalchemy.ext.asyncio import ( # noqa: E402
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import sessionmaker # noqa: E402
from backend.app import create_app
from backend.app.models import Base
from backend.app import create_app # noqa: E402
from backend.app.models import Base # noqa: E402
def _async_database_url() -> str:
+2 -2
View File
@@ -37,8 +37,8 @@ async def test_artist_overview_post_count(client, db):
)
db.add(s)
await db.flush()
db.add(Post(source_id=s.id, external_post_id="p1"))
db.add(Post(source_id=s.id, external_post_id="p2"))
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p1"))
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p2"))
await db.flush()
await db.commit()
resp = await client.get("/api/artist/lyra")
+108
View File
@@ -135,6 +135,114 @@ async def test_quick_add_source_wrong_key_401(client, ext_key):
assert resp.status_code == 401
# --- /api/extension/probe ---------------------------------------------
@pytest.mark.asyncio
async def test_probe_returns_new_when_nothing_exists(client, ext_key):
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/freshcreator"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["state"] == "new"
assert body["platform"] == "patreon"
assert body["slug"] == "freshcreator"
@pytest.mark.asyncio
async def test_probe_returns_source_match_for_already_added(client, ext_key, db):
artist = Artist(name="Alice", slug="alice", is_subscription=True)
db.add(artist)
await db.flush()
src = Source(
artist_id=artist.id, platform="patreon",
url="https://www.patreon.com/alice", enabled=True, config_overrides={},
)
db.add(src)
await db.commit()
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/alice"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["state"] == "source_match"
assert body["artist"]["slug"] == "alice"
assert body["source"]["url"] == "https://www.patreon.com/alice"
assert body["source"]["platform"] == "patreon"
@pytest.mark.asyncio
async def test_probe_returns_artist_match_when_only_synthetic_anchor_exists(
client, ext_key, db,
):
"""Filesystem-imported artist with only a sidecar synthetic Source
for the (artist, platform) — the URL the operator's browsing isn't
yet a real Source. The probe should collapse this into artist_match
so the chip says '+ Add Patreon source to Dymkens' rather than
'+ Add to FabledCurator' (which would re-create the artist)."""
artist = Artist(name="Dymkens", slug="dymkens", is_subscription=False)
db.add(artist)
await db.flush()
synthetic = Source(
artist_id=artist.id, platform="patreon",
url="sidecar:patreon:dymkens", enabled=False, config_overrides={},
)
db.add(synthetic)
await db.commit()
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/dymkens"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["state"] == "artist_match"
assert body["artist"]["slug"] == "dymkens"
assert "source" not in body
@pytest.mark.asyncio
async def test_probe_returns_unknown_platform_for_non_artist_url(client, ext_key):
"""A patreon URL that isn't an artist page (e.g. /home, /posts/N)
shouldn't trigger the button. Sentinel 'unknown_platform' state
tells the content script to skip injection."""
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/posts/12345"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["state"] == "unknown_platform"
@pytest.mark.asyncio
async def test_probe_missing_key_401(client):
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/maewix"},
)
assert resp.status_code == 401
@pytest.mark.asyncio
async def test_probe_missing_url_400(client, ext_key):
resp = await client.get(
"/api/extension/probe",
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_body"
@pytest.mark.asyncio
async def test_quick_add_source_missing_body_400(client, ext_key):
resp = await client.post(
+7 -1
View File
@@ -19,7 +19,13 @@ async def test_get_and_patch_settings(client):
resp = await client.get("/api/ml/settings")
assert resp.status_code == 200
body = await resp.get_json()
assert body["suggestion_threshold_general"] == pytest.approx(0.95)
# Default raised 0.50 → 0.70 on 2026-06-02 (alembic 0033) — 0.50
# was too noisy in practice. The 0.70 default keeps the rail
# signal-rich without hiding everything like the original 0.95.
assert body["suggestion_threshold_general"] == pytest.approx(0.70)
# Retired threshold columns must not appear in the payload.
assert "suggestion_threshold_artist" not in body
assert "suggestion_threshold_copyright" not in body
resp = await client.patch(
"/api/ml/settings", json={"suggestion_threshold_general": 0.90}
+4 -3
View File
@@ -25,7 +25,7 @@ async def seeded_post(db):
db.add(source)
await db.flush()
post = Post(
source_id=source.id, external_post_id="API1",
source_id=source.id, artist_id=artist.id, external_post_id="API1",
post_title="Hello", post_url="https://p/alice-api/1",
post_date=datetime.now(UTC),
description="<p>hi</p>",
@@ -123,7 +123,8 @@ async def post_timeline(db):
posts = []
for i in range(5):
p = Post(
source_id=source.id, external_post_id=f"TL{i}",
source_id=source.id, artist_id=artist.id,
external_post_id=f"TL{i}",
post_title=f"post {i}", post_date=base + timedelta(days=i),
)
db.add(p)
@@ -189,7 +190,7 @@ async def test_detail_returns_uncapped_thumbnails(client, db):
db.add(s)
await db.flush()
p = Post(
source_id=s.id, external_post_id="DETAIL10",
source_id=s.id, artist_id=a.id, external_post_id="DETAIL10",
post_title="big post", description="<p>body</p>",
)
db.add(p)
+2 -1
View File
@@ -26,7 +26,8 @@ async def _seed_full(db):
url="https://patreon.test/alice")
db.add(source)
await db.flush()
post = Post(source_id=source.id, external_post_id="555",
post = Post(source_id=source.id, artist_id=artist.id,
external_post_id="555",
post_url="https://patreon.test/p/555", post_title="Set 1",
post_date=datetime(2023, 8, 1, tzinfo=UTC),
description="<p>hi</p>", attachment_count=2)
+58
View File
@@ -193,3 +193,61 @@ async def test_list_derives_next_check_at_when_last_checked_set(
)
assert target["next_check_at"] is not None
assert "T" in target["next_check_at"] # ISO 8601
# --- Plan #544: POST /api/sources/{id}/backfill ----------------------------
@pytest.mark.asyncio
async def test_backfill_endpoint_arms_source(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill", enabled=True,
)
db.add(src)
await db.commit()
sid = src.id
resp = await client.post(f"/api/sources/{sid}/backfill", json={"runs": 5})
assert resp.status_code == 200
body = await resp.get_json()
assert body["backfill_runs_remaining"] == 5
# GET reflects the new state.
one = await client.get(f"/api/sources/{sid}")
assert (await one.get_json())["backfill_runs_remaining"] == 5
@pytest.mark.asyncio
async def test_backfill_endpoint_defaults_to_three(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill-default", enabled=True,
)
db.add(src)
await db.commit()
resp = await client.post(f"/api/sources/{src.id}/backfill", json={})
body = await resp.get_json()
assert body["backfill_runs_remaining"] == 3
@pytest.mark.asyncio
async def test_backfill_endpoint_rejects_out_of_range(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill-bad", enabled=True,
)
db.add(src)
await db.commit()
bad = await client.post(f"/api/sources/{src.id}/backfill", json={"runs": 0})
assert bad.status_code == 400
too_big = await client.post(
f"/api/sources/{src.id}/backfill", json={"runs": 99}
)
assert too_big.status_code == 400
@pytest.mark.asyncio
async def test_backfill_endpoint_404_when_source_missing(client):
resp = await client.post("/api/sources/999999/backfill", json={"runs": 3})
assert resp.status_code == 404
+11 -3
View File
@@ -13,8 +13,16 @@ def eager():
@pytest.mark.asyncio
async def test_trigger_thumbnail_backfill(client):
async def test_trigger_thumbnail_backfill_returns_counts(client):
"""The API runs the scan synchronously now and returns
{scanned, enqueued, ok, regenerated} — operator-flagged 2026-06-01:
the previous fire-and-forget shape returned only a celery_task_id,
so the admin UI couldn't show whether backfill found 0 or 5000
candidates. \"Found nothing\" was indistinguishable from \"the
worker isn't picking up the task.\""""
r = await client.post("/api/thumbnails/backfill")
assert r.status_code == 202
assert r.status_code == 200
body = await r.get_json()
assert "celery_task_id" in body
assert set(body) == {"scanned", "enqueued", "ok", "regenerated"}
for key in ("scanned", "enqueued", "ok", "regenerated"):
assert isinstance(body[key], int)
+1 -1
View File
@@ -25,7 +25,7 @@ async def _fixture(db):
db.add(src)
await db.flush()
post = Post(
source_id=src.id, external_post_id="p1",
source_id=src.id, artist_id=artist.id, external_post_id="p1",
post_date=datetime(2026, 3, 1, tzinfo=UTC),
)
db.add(post)
+24 -10
View File
@@ -12,13 +12,16 @@ pytestmark = pytest.mark.integration
def test_thumb_is_valid_jpeg(tmp_path):
p = tmp_path / "good.jpg"
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 100)
# Real thumbnails are at least ~2KB; size check (MIN_THUMB_BYTES=256)
# requires the file body be plausible. 300 bytes here clears the
# floor with margin.
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 300)
assert _thumb_is_valid(p) is True
def test_thumb_is_valid_png(tmp_path):
p = tmp_path / "good.png"
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 100)
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 300)
assert _thumb_is_valid(p) is True
@@ -38,6 +41,16 @@ def test_thumb_is_valid_missing_file(tmp_path):
assert _thumb_is_valid(tmp_path / "nope") is False
def test_thumb_is_valid_header_only_below_min_size(tmp_path):
"""Operator-flagged 2026-06-01: header-only corrupt files were
silently passing the magic-byte check and backfill counted them as
`ok`, so the UI's broken-image tiles never got regenerated. Files
smaller than MIN_THUMB_BYTES are now invalid even with valid magic."""
p = tmp_path / "header_only.jpg"
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 50) # 54 bytes total
assert _thumb_is_valid(p) is False
# --- backfill_thumbnails planner tests ------------------------------------
@@ -80,13 +93,14 @@ def _rec(db_sync, path, *, sha, thumb_path=None, mime="image/jpeg"):
def _write_jpeg(p: Path) -> Path:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 100)
# ≥ MIN_THUMB_BYTES (256) so the size floor doesn't reject it.
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 300)
return p
def _write_png(p: Path) -> Path:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 100)
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 300)
return p
@@ -111,7 +125,7 @@ def test_backfill_null_path_enqueued(db_sync, tmp_path, monkeypatch):
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 0}
assert result == {"scanned": 1, "enqueued": 1, "ok": 0, "regenerated": 0}
assert delayed == [rec.id]
@@ -134,7 +148,7 @@ def test_backfill_missing_file_clears_and_enqueues(db_sync, tmp_path, monkeypatc
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 1}
assert result == {"scanned": 1, "enqueued": 1, "ok": 0, "regenerated": 1}
assert delayed == [rec.id]
assert db_sync.get(ImageRecord, rec.id).thumbnail_path is None
@@ -156,7 +170,7 @@ def test_backfill_valid_jpeg_skipped(db_sync, tmp_path, monkeypatch):
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 0, "ok": 1, "regenerated": 0}
assert result == {"scanned": 1, "enqueued": 0, "ok": 1, "regenerated": 0}
assert delayed == []
assert db_sync.get(ImageRecord, rec.id).thumbnail_path == str(thumb)
@@ -177,7 +191,7 @@ def test_backfill_valid_png_skipped(db_sync, tmp_path, monkeypatch):
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 0, "ok": 1, "regenerated": 0}
assert result == {"scanned": 1, "enqueued": 0, "ok": 1, "regenerated": 0}
assert delayed == []
@@ -198,7 +212,7 @@ def test_backfill_corrupt_magic_clears_and_enqueues(db_sync, tmp_path, monkeypat
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 1}
assert result == {"scanned": 1, "enqueued": 1, "ok": 0, "regenerated": 1}
assert delayed == [rec.id]
assert db_sync.get(ImageRecord, rec.id).thumbnail_path is None
@@ -238,5 +252,5 @@ def test_backfill_mixed_aggregate(db_sync, tmp_path, monkeypatch):
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 3, "ok": 2, "regenerated": 2}
assert result == {"scanned": 5, "enqueued": 3, "ok": 2, "regenerated": 2}
assert sorted(delayed) == sorted([r_null.id, r_missing.id, r_bad.id])
+25 -6
View File
@@ -14,7 +14,7 @@ pytestmark = pytest.mark.integration
def test_load_or_create_writes_key_with_mode_0600(tmp_path):
key_path = tmp_path / "secrets" / "credential_key.b64"
CredentialCrypto(key_path)
CredentialCrypto(key_path, bootstrap_ok=True)
assert key_path.exists()
mode = stat.S_IMODE(os.stat(key_path).st_mode)
assert mode == 0o600
@@ -25,9 +25,9 @@ def test_load_or_create_writes_key_with_mode_0600(tmp_path):
def test_load_existing_key_is_idempotent(tmp_path):
key_path = tmp_path / "credential_key.b64"
crypto1 = CredentialCrypto(key_path)
crypto1 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_first = key_path.read_bytes()
crypto2 = CredentialCrypto(key_path)
crypto2 = CredentialCrypto(key_path, bootstrap_ok=True)
contents_after_second = key_path.read_bytes()
assert contents_after_first == contents_after_second
# And both crypto instances decrypt each other's ciphertext
@@ -36,7 +36,7 @@ def test_load_existing_key_is_idempotent(tmp_path):
def test_encrypt_decrypt_round_trip(tmp_path):
crypto = CredentialCrypto(tmp_path / "k")
crypto = CredentialCrypto(tmp_path / "k", bootstrap_ok=True)
plaintext = "domain.com\tTRUE\t/\tTRUE\t1700000000\tname\tvalue"
ct = crypto.encrypt(plaintext)
assert isinstance(ct, bytes)
@@ -45,8 +45,27 @@ def test_encrypt_decrypt_round_trip(tmp_path):
def test_decrypt_with_wrong_key_raises(tmp_path):
crypto_a = CredentialCrypto(tmp_path / "a")
crypto_b = CredentialCrypto(tmp_path / "b")
crypto_a = CredentialCrypto(tmp_path / "a", bootstrap_ok=True)
crypto_b = CredentialCrypto(tmp_path / "b", bootstrap_ok=True)
ct = crypto_a.encrypt("secret")
with pytest.raises(InvalidCredentialBlob):
crypto_b.decrypt(ct)
def test_missing_key_without_bootstrap_raises(tmp_path, monkeypatch):
"""Audit 2026-06-02: without explicit opt-in, a missing key file
is a fatal startup error — silent regeneration on partial restore
would make every existing Credential row undecryptable."""
from backend.app.services.credential_crypto import MissingCredentialKey
monkeypatch.delenv("CURATOR_BOOTSTRAP_NEW_KEY", raising=False)
with pytest.raises(MissingCredentialKey):
CredentialCrypto(tmp_path / "absent.b64")
def test_missing_key_with_env_var_bootstraps(tmp_path, monkeypatch):
"""The env var CURATOR_BOOTSTRAP_NEW_KEY=1 is the operator's
first-time-setup opt-in for auto-creating the key file."""
monkeypatch.setenv("CURATOR_BOOTSTRAP_NEW_KEY", "1")
key_path = tmp_path / "bootstrap.b64"
CredentialCrypto(key_path) # no bootstrap_ok kwarg — relies on env
assert key_path.exists()

Some files were not shown because too many files have changed in this diff Show More