bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 33 Wiki Activity
374 Commits 3 Branches 33 Tags
9f54efdedfc81645525ee79ee61ff31dfb16e469
Commit Graph

172 Commits

Author SHA1 Message Date
bvandeusen 9f54efdedf fix(migrators): tag_apply phase 4 now covers deviantart + pixiv (was silently dropping IR PostMetadata from those platforms) 
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.

Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.

To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 11:25:45 -04:00
bvandeusen 6b1bb87647 fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 11:12:55 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever 
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:34:36 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output) 
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:25:30 -04:00
bvandeusen 9d5abb09f6 fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed 
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:00:29 -04:00
bvandeusen 832345a245 fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 01:28:54 -04:00
bvandeusen a0136fa30d fix(fc3k): add mime=image/jpeg to test ImageRecord ctors (NOT NULL) + reorder admin import after stdlib/3rd-party (ruff I001) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 01:12:52 -04:00
bvandeusen ad3d34a1fc fc3k: /api/admin endpoint integration tests — dry-run, confirm-mismatch, dispatch, Tier-A/B/C paths 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:45:31 -04:00
bvandeusen b5289ed372 fc3k: admin Celery task tests — registration, success, failure, missing-id idempotency 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:44:33 -04:00
bvandeusen f6aa805725 fc3k: cleanup_service unit tests — projections + mutations + file unlinks against real Postgres 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:43:51 -04:00
bvandeusen 718cc79905 fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:37:06 -04:00
bvandeusen 57a338f7e6 fc3h(tests): drop pinned migration-backup tests (retired surface; coverage moved to test_backup_service.py + test_api_system_backup.py) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:03:57 -04:00
bvandeusen d04983138a fc3h: /api/system/backup endpoint integration tests 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:02:32 -04:00
bvandeusen 9ec6fdb596 fc3h: Celery task integration tests (backup, restore, prune, nightly) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:01:39 -04:00
bvandeusen 86ad9b80e9 fc3h: backup_service unit tests (subprocess monkeypatched) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:00:40 -04:00
bvandeusen 37fcc74954 fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:22:42 -04:00
bvandeusen 36611cbe00 fc3i: tests for recover_stalled_task_runs + prune_task_runs 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:07:01 -04:00
bvandeusen 48ef22445a fc3i: integration tests for /api/system/activity/* endpoints 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:06:20 -04:00
bvandeusen e523d0ac94 fc3i: integration tests for Celery signal → task_run lifecycle 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:05:32 -04:00
bvandeusen b1d68929c5 fix(tests): drop test_post_apply_without_backup_rejected (gate retired in 5535677) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 20:15:10 -04:00
bvandeusen 538c1591e8 fc-3g-ext: IR Post/Provenance restore (tag_apply phase 4) + modal artist fallback 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:08:18 -04:00
bvandeusen 4437488899 fix(tests): unpin test_recover_interrupted_only_old from STUCK_THRESHOLD_MINUTES (was -5min fresh / now -30sec so threshold can move freely) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 12:40:27 -04:00
bvandeusen cd838ec904 fc3g: integration tests for /api/extension/* and /extension/<filename> 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 23:40:00 -04:00
bvandeusen 6f68bf5fa7 feat(ui): bump thumbnail sizes 1.5x, make Settings fluid, add subscription_count stat 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 23:18:22 -04:00
bvandeusen 9fb3d11412 fix(fc5): ruff I001 — consolidate unaliased imports + tests/test_gs_ingest credential_type assertion 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:29:09 -04:00
bvandeusen e5885f85c5 fix(fc5): ruff (E712, ASYNC230/240, I001) + Credential.credential_type + FileStorage in api_migrate test 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:20:42 -04:00
bvandeusen 89224a7895 fc5: /api/migrate blueprint — multipart upload for ingest kinds + JSON for the rest + apply-without-backup guard 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:03:24 -04:00
bvandeusen b5896427d4 fc5: ml_queue (fire tag_and_embed for unprocessed) + verify (row counts + sha256 sample) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:01:46 -04:00
bvandeusen 177d6728e1 fc5: tag_apply — joins manifest entries to ImageRecord by sha256 after filesystem scan 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:01:17 -04:00
bvandeusen 77b17f7bb6 fc5: ir_ingest — Tag rows (skip artist/post) + manifest JSON for sha256-keyed tag_apply 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:00:29 -04:00
bvandeusen a13a54206e fc5: gs_ingest — parsed-JSON ingest, Artist/Source/Credential, re-encrypts with FC key 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:59:43 -04:00
bvandeusen e1b057f494 fc5: backup + rollback services (pg_dump + tar.zst, manifest JSON, find-by-tag) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:12:17 -04:00
bvandeusen 51bdaf167b fc3f: GET /api/artists/directory — cursor + q + platform filters with validation 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 22:14:42 -04:00
bvandeusen 23e11d33ef fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 22:14:15 -04:00
bvandeusen 266137e3d0 fc3e: /api/posts blueprint — list (with cursor + filters) + detail 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 18:55:18 -04:00
bvandeusen 5925ebe1f1 fc3e: PostFeedService — cursor-paginated unified posts feed mirroring GalleryService.scroll 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 18:54:49 -04:00
bvandeusen 07edb73373 fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 18:53:35 -04:00
bvandeusen 28a4ed3d38 fc3d: settings API exposes scheduling knobs with bounds validation 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:04:19 -04:00
bvandeusen 69b3fa2ba9 fc3d: api_sources tests cover consecutive_failures + next_check_at fields 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:48 -04:00
bvandeusen ec004933a5 fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:02:36 -04:00
bvandeusen 9d5793f0ce fc3d: tick_due_sources Celery task on the scan queue 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:01:59 -04:00
bvandeusen 87fb534722 fc3d: DownloadService finalize hook — Source.consecutive_failures/last_error/last_checked_at 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:00:40 -04:00
bvandeusen 0493fdc466 fc3d: scheduler_service — pure logic for due-source selection + backoff 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 15:59:35 -04:00
bvandeusen 1fe259fa14 fix(fc3c): structured JPEGs + phash_threshold=0 so both fixtures attach 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:45:55 -04:00
bvandeusen c212691771 fix(fc3c): distinct test JPEG colors so both attach (sha256 dedup was kicking in) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:39:27 -04:00
bvandeusen e5da27f755 fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download' 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:01:26 -04:00
bvandeusen 717ceb4b5a fix(fc3c): explicit import of download task module in registration test 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 21:20:30 -04:00
bvandeusen e9814ca7e4 feat(fc3c): expose download_rate_limit_seconds + download_validate_files in /api/settings/import 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:44:58 -04:00
bvandeusen 71e6114f49 feat(fc3c): /api/downloads blueprint (list + detail w/ keyset pagination) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:44:28 -04:00
bvandeusen 95cfdff97d feat(fc3c): POST /api/sources/<id>/check — enqueue download 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:43:48 -04:00