bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 33 Wiki Activity
372 Commits 4 Branches 33 Tags
68cffce322f5bd9422fbf15538f50daa50610855
Commit Graph

82 Commits

Author SHA1 Message Date
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever 
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:34:36 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output) 
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:25:30 -04:00
bvandeusen 6df102b83d fc3k: cleanup_service mutations — unlink primitive, artist cascade, bulk image delete, tag delete, prune unused 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:39:45 -04:00
bvandeusen 2ae01d27e3 fc3k: cleanup_service projections — artist cascade, bulk delete, tag usage, unused tags 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:38:57 -04:00
bvandeusen 2b05f147f4 fc3h: migrators docstring — note backup/rollback retired to backup_service.py 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:00:01 -04:00
bvandeusen 70e1e010d1 fc3h: remove backend/app/services/migrators/backup.py + rollback.py (relocated to backup_service.py) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:59:18 -04:00
bvandeusen 319e7de547 fc3h: backup_service.py — DB + images backup/restore + unlink helpers (relocated, split per-kind) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:52:35 -04:00
bvandeusen c9a3f12847 fix(lint): tag_apply.py — one blank line between imports and module-level comment (ruff I001, third bounce on this rule) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:22:05 -04:00
bvandeusen 538c1591e8 fc-3g-ext: IR Post/Provenance restore (tag_apply phase 4) + modal artist fallback 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:08:18 -04:00
bvandeusen dd4874ae8d fix: 30min wall-clock timeout on backup subprocess + global padding-top on .fc-content so views don't start under the sticky TopNav 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 11:10:04 -04:00
bvandeusen 46d199450d fc3g: ExtensionService.quick_add_source — derive platform+slug, find-or-create Artist+Source 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 23:37:39 -04:00
bvandeusen f5efbea053 feat(fc5): /api/migrate/cleanup — delete every image attributed to one artist, files + thumbs + DB 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 12:09:56 -04:00
bvandeusen ec44c653fe fc5: strip SQLAlchemy +psycopg/+asyncpg driver suffix when passing URL to pg_dump/psql 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 22:54:19 -04:00
bvandeusen e5885f85c5 fix(fc5): ruff (E712, ASYNC230/240, I001) + Credential.credential_type + FileStorage in api_migrate test 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:20:42 -04:00
bvandeusen b5896427d4 fc5: ml_queue (fire tag_and_embed for unprocessed) + verify (row counts + sha256 sample) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:01:46 -04:00
bvandeusen 177d6728e1 fc5: tag_apply — joins manifest entries to ImageRecord by sha256 after filesystem scan 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:01:17 -04:00
bvandeusen 77b17f7bb6 fc5: ir_ingest — Tag rows (skip artist/post) + manifest JSON for sha256-keyed tag_apply 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:00:29 -04:00
bvandeusen a13a54206e fc5: gs_ingest — parsed-JSON ingest, Artist/Source/Credential, re-encrypts with FC key 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:59:43 -04:00
bvandeusen e1b057f494 fc5: backup + rollback services (pg_dump + tar.zst, manifest JSON, find-by-tag) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:12:17 -04:00
bvandeusen 23e11d33ef fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 22:14:15 -04:00
bvandeusen 5925ebe1f1 fc3e: PostFeedService — cursor-paginated unified posts feed mirroring GalleryService.scroll 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 18:54:49 -04:00
bvandeusen cd6983f728 fc3d: SourceRecord surfaces consecutive_failures + server-derived next_check_at 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:33 -04:00
bvandeusen 87fb534722 fc3d: DownloadService finalize hook — Source.consecutive_failures/last_error/last_checked_at 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:00:40 -04:00
bvandeusen 0493fdc466 fc3d: scheduler_service — pure logic for due-source selection + backoff 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 15:59:35 -04:00
bvandeusen e5da27f755 fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download' 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:01:26 -04:00
bvandeusen 29598c3f9a fix(fc3c): ruff — split SyncSession import, StrEnum, ASYNC240 noqa, blank-line 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:50:38 -04:00
bvandeusen 82fc6afe53 feat(fc3c): DownloadService orchestrator — 3-phase pipeline + Patreon retry 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:42:51 -04:00
bvandeusen 7a96b1d06c feat(fc3c): GalleryDLService — subprocess wrap + error classification + validate/quarantine 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:41:42 -04:00
bvandeusen 0c80c12984 feat(fc3c): PatreonResolver — vanity → campaign_id via requests 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:39:50 -04:00
bvandeusen 25e6e1b10f feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:39:17 -04:00
bvandeusen a67755c929 feat(fc3c): Importer.attach_in_place + _supersede(new_path=) + _copy_to_library helper 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:38:47 -04:00
bvandeusen 6fe8d199a0 fix(fc3b): replace FC-3a KNOWN_PLATFORMS w/ registry; drop /api/sources/platforms 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:37:17 -04:00
bvandeusen 4a9cada8c0 feat(fc3b): CredentialService — CRUD + on-demand cookies.txt materialisation 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:35:07 -04:00
bvandeusen 5596de6ed5 feat(fc3b): platforms.py — GS-6 registry (single source of truth) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:34:02 -04:00
bvandeusen 28842fd07d feat(fc3b): CredentialCrypto — Fernet with on-disk system key 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:33:34 -04:00
bvandeusen bf828c7b9d fix(fc3a): ruff I001 import-block fixes + use sqlalchemy.case (not func.case) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 14:22:53 -04:00
bvandeusen ecca91753f feat(fc3a): ArtistService.overview returns is_subscription for the badge 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 12:58:50 -04:00
bvandeusen 0270a23c1e feat(fc3a): ArtistService.find_or_create + autocomplete 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 12:54:47 -04:00
bvandeusen 8fa6c502d5 feat(fc3a): SourceService — CRUD + validation + is_subscription auto-flip 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 12:54:20 -04:00
bvandeusen d4d8976f29 feat(integrity): image-detail payload includes integrity_status 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 17:53:54 -04:00
bvandeusen 5dfdbf5a60 feat(deep-scan): Importer(deep=) re-derive on sha-match; import_file wires batch.scan_mode 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 15:32:26 -04:00
bvandeusen 74e34d359b feat(attachments): provenance payload attachments + download route 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 11:16:01 -04:00
bvandeusen f97551e2f6 feat(attachments): importer dispatch — archive extract + non-media capture 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 11:13:36 -04:00
bvandeusen df76fd75d8 feat(attachments): fail-soft archive_extractor (zip/cbz/rar/7z) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 11:11:37 -04:00
bvandeusen 1475bad48b feat(attachments): sha-addressed AttachmentStore 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 11:11:07 -04:00
bvandeusen 592c665701 feat(provenance): ML stops surfacing the artist category 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 21:36:43 -04:00
bvandeusen e798302cfa feat(provenance): _artists_for resolves via artist_id (closes vii-b gap) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 21:36:16 -04:00
bvandeusen aaf5d1ea70 feat(provenance): ArtistService resolves images via artist_id 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 21:34:30 -04:00
bvandeusen 914cf14d9e feat(provenance): importer sets image_record.artist_id, stops minting artist tags 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 21:33:55 -04:00
bvandeusen f6d5353b3b feat(provenance): per-image artist on gallery scroll payload 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 20:15:07 -04:00