bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 33 Wiki Activity
424 Commits 3 Branches 33 Tags
397021dcbd377d6b335437cdf323d1ad94ee459d
Commit Graph

424 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
bvandeusen 397021dcbd fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped. 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 13:47:55 -04:00
bvandeusen b0bfbc585a fix(import-admin): retry-failed + clear-stuck — same UPDATE…WHERE pattern as the maintenance sweep, so neither endpoint can hit psycopg's 65535-parameter ceiling once accumulated row counts exceed ~65k 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 11:47:40 -04:00
bvandeusen 110c1c0e51 fix(maintenance): recover_interrupted_tasks — fold SELECT into UPDATE…WHERE…RETURNING so the IN-list no longer blows past psycopg's 65535-parameter ceiling (operator-hit 2026-05-26 after deep scan orphan pile) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 11:46:06 -04:00
bvandeusen 6de84d0d60 feat(ui): ErrorDetailModal — context panel (task/queue/target/duration/started/retries/worker/celery-id/args) + contrast fix (background-token bg vs surface-variant pale-on-pale) + copyText helper for Copy button 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 11:16:37 -04:00
bvandeusen 4e1f208a9f fix(ui-copy): copyText utility with execCommand fallback — navigator.clipboard is gated by Secure Context (HTTPS-only) and is undefined on plain-HTTP self-hosted deployments. Apply to ExtensionKeyBar + BrowserExtensionCard. 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 11:16:32 -04:00
bvandeusen 06913eba8e feat(thumb-backfill): MaintenancePanel — wire ThumbnailBackfillCard into grid, broaden intro to cover non-ML backfills 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:37:09 -04:00
bvandeusen b7f693b15e feat(thumb-backfill): ThumbnailBackfillCard — 'Run backfill now' button, mirrors MLBackfillCard pattern 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:36:55 -04:00
bvandeusen ecd0199799 feat(thumb-backfill): Pinia store — triggerBackfill() POSTs /api/thumbnails/backfill 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:36:45 -04:00
bvandeusen 983da9e5b1 feat(thumb-backfill): /api/thumbnails/backfill endpoint — POST → 202 + celery_task_id 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:36:38 -04:00
bvandeusen a41eddae3f feat(thumb-backfill): backfill_thumbnails planner task — keyset-paginates ImageRecord, NULLs bad thumb paths, enqueues generate_thumbnail for NULL/missing/corrupt 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:36:21 -04:00
bvandeusen 7aa7f5a3d6 feat(thumb-backfill): _thumb_is_valid helper — JPEG/PNG magic-byte check on the on-disk thumbnail file 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 10:35:06 -04:00
bvandeusen 2505b197ae feat(fc-cleanup): Pinia store + 3 cards + CleanupView + SettingsView tab + TagMaintenanceCard moved from Maintenance + ruff lint fixes — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:16:46 -04:00
bvandeusen 0d0b236ac3 feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:13:53 -04:00
bvandeusen a06ada4c9b fix(ext-ui): direct :href install button (Firefox needs anchor click, not programmatic navigation) + manifest version detection ignores -latest.xpi alias — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:07:22 -04:00
bvandeusen ebd985990c feat(ui): ErrorDetailModal — click error → flat-text modal with copy button (replaces unusable :title tooltip for multi-line SQLAlchemy tracebacks) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:53:14 -04:00
bvandeusen 4da8d1d774 fix(importer): race-safe savepoint-based find-or-create for Source + Post (uq_source_artist_platform_url UniqueViolation operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:51:12 -04:00
bvandeusen 2d4bfa4375 fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:32:50 -04:00
bvandeusen 6ed2021ad6 feat(fc-cleanup): scan_library_for_rule Celery task + maintenance-queue registration — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:21:37 -04:00
bvandeusen 4f2ceaaf31 feat(fc-cleanup): audit lifecycle service functions (start/apply/cancel) + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:41 -04:00
bvandeusen 8a5b337a53 feat(fc-cleanup): min-dimension service functions + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:04 -04:00
bvandeusen 900d878d27 feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:38 -04:00
bvandeusen fd80d40a34 feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:07 -04:00
bvandeusen 929d3fc092 feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:17:42 -04:00
bvandeusen c0c9e56fb9 fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:45:08 -04:00
bvandeusen 06a2f60c08 fix(ext-ci): use browser_download_url not /releases/assets/<id> + add -f to curl + magic-byte sanity check (operator-flagged 2026-05-26: prior build silently wrote '404 page not found' into the XPI file, Firefox rejected as corrupt) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:42:35 -04:00
bvandeusen 0978fbac66 fix(sidecar): strip gallery-dl 'NN_' numbering prefix when locating sidecars — fixes 'deep scan refresh count high but 0 Posts created' (operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:27:43 -04:00
bvandeusen efb142239d fix(ext-ci): drop actions/upload-artifact (Forgejo Actions doesn't support v4+ GHES) — build-web reads XPI directly from the ext-<version> Forgejo release asset — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:32:58 -04:00
bvandeusen 5587a76606 fix(ext-ci): replace jq with python3 (jq not in ci-python image) + bump ext 1.0.2→1.0.3 (escape AMO 'version already exists' from prior partial-failure run) + add rollback to prevent empty cache-release tombstones — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:14:11 -04:00
bvandeusen 17e19081a2 fix(ext-ci): drop web-ext-config.cjs (v8 mis-parses .cjs configs as if module.exports were a config option) — inline ignore-files on CLI + --no-config-discovery — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:46:58 -04:00
bvandeusen 770bcf3aa6 feat(artist): tab split (Overview/Settings) so DangerZone is reachable without exhausting the infinite-scroll image grid — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:32:07 -04:00
bvandeusen 52d7905c43 perf(importer): cache phash candidates on Importer to fix archive-import soft-timeout (was O(M×N) per-member SELECTs) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:25:32 -04:00
bvandeusen e6ededbe8e feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:16:26 -04:00
bvandeusen c06cbc0abe feat(ci): inline extension sign into build.yml + Forgejo Release Assets as XPI cache (v26.05.25.5) — bump ext to 1.0.2 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 21:30:11 -04:00
bvandeusen ac39509a74 fix(ci): rename shard jobs to no-separator names (intapi/intimp/intcore) + add diagnostic docker ps dump so next bounce surfaces the real act_runner naming convention — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:59:17 -04:00
bvandeusen 3531f373ee feat(settings): move BrowserExtensionCard from Maintenance to Overview tab — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen 36cc0622cb fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen e50f92d900 perf(ci): shard integration suite into 3 parallel jobs (int_api, int_imp, int_core) — newly feasible after act_runner capacity 2→6 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen ba8d9b112d fix(ext-ci): add diagnostic tracing to commit step to surface why run #309 reported success without producing the XPI side-commit — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen c451061ca5 fix(ext-ci): self-retrigger workflow on its own edits (path filter includes workflow file) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:24:31 -04:00
bvandeusen 47d760550d fix(ext-ci): glob AMO-renamed signed XPI + canonicalize to fabledcurator-<version>.xpi — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:22:01 -04:00
bvandeusen dc3bce7fc1 chore(ext): bump to 1.0.1 to trigger initial sign-and-publish — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen f657582f30 feat(import-ui): deep scan button, sticky settings tabs, tasks-above-filters, fix Scanning-undefined source_path — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen 111b952535 fix(ml): load SigLIP image-only processor to avoid SentencePiece dep — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:31:06 -04:00
bvandeusen a0470b5f60 feat(importer): _supersede() now applies the new (larger) file's sidecar — operator wanted to scan GS download dir to supersede smaller IR-migrated images AND wire up gallery-dl Post metadata, but supersede was file-only and silently dropped the sidecar. 
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.

Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 13:51:51 -04:00
bvandeusen b0bb7ae6cc ci: pip wheel cache (actions/cache on requirements.txt hash) + uv-when-available — ~2 min saved on warm runs, no risk 
uv falls back to pip install on runners without uv binary, so this
change is forward-compatible with the current ci-python image. When
the runner image gets uv pre-installed in a future bump, the warm
install path drops from ~2 min to ~10 seconds.

pytest-xdist parallelization is OUT OF SCOPE for this commit:
tests/conftest.py uses a TRUNCATE ALL TABLES RESTART IDENTITY CASCADE
fixture after every integration test against a single shared
database; xdist workers running in parallel would nuke each other's
mid-test state. A future refactor to per-worker databases or
per-worker schema isolation is the prerequisite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 13:47:27 -04:00
bvandeusen 1bbe478fd0 ci: report slowest 25 integration tests via pytest --durations=25 — instrumentation pass before deciding parallelization vs targeted slow-test fixes 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 13:09:25 -04:00
bvandeusen 5666fd5ca5 fix(ui): scan trigger immediate-feedback toast + delayed status re-poll — operator-flagged 'click does nothing' was actually scan_directory's skip-set finalizing the batch in <100ms when every file already had an ImportTask row, before refreshStatus could ever see the active state. Now the click always produces visible feedback (immediate 'Scan triggered' + 2s 'no new files' if it quick-finalizes). 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 13:06:11 -04:00
bvandeusen 3a359f6c5e fix(ui): Quick scan button always visible (disabled when active batch present) + inline Clear stuck action — operator was clicking a spinner area thinking it was the button because activeBatch hid the Quick scan button entirely 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 12:43:37 -04:00
bvandeusen b6a917ac81 feat(import): /api/import/clear-stuck endpoint + Clear stuck UI button — escape hatch for the autoretry-loop case the automatic sweep can't break 
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.

/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
  marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
  underlying cause is resolved

UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.

Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 12:37:07 -04:00
bvandeusen c361032554 feat(gallery): sort/group/jump by COALESCE(post.post_date, image_record.created_at) — surface migrated content at its original publish date, not FC scan date 
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.

Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
  ImageProvenance (only if currently NULL — preserves the canonical
  download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
  COALESCE(post.post_date, image_record.created_at), used in:
    * scroll() ORDER BY + cursor WHERE clauses
    * timeline() year/month group-by
    * jump_cursor() year/month filter
    * _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
  works for images without a post (NULL on the Post side, fall back
  to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
  /scroll exposes both alongside the existing created_at so the UI
  can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.

Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.

To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 12:30:18 -04:00