bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 33 Wiki Activity
374 Commits 3 Branches 33 Tags
9f54efdedfc81645525ee79ee61ff31dfb16e469
Commit Graph

190 Commits

Author SHA1 Message Date
bvandeusen 9f54efdedf fix(migrators): tag_apply phase 4 now covers deviantart + pixiv (was silently dropping IR PostMetadata from those platforms) 
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.

Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.

To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 11:25:45 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever 
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:34:36 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output) 
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:25:30 -04:00
bvandeusen 9d5abb09f6 fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed 
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 02:00:29 -04:00
bvandeusen f096c9a5fb fc3k: register admin_bp in api/__init__ all_blueprints 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:42:40 -04:00
bvandeusen 676a86b514 fc3k: /api/admin tags prune-unused endpoint — Tier-A preview-then-commit flow 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:42:20 -04:00
bvandeusen 44cc625d4a fc3k: /api/admin tag endpoints — Tier-B delete + merge + usage-count helper 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:42:03 -04:00
bvandeusen f7ee122243 fc3k: /api/admin blueprint — Tier-C artist cascade + bulk image delete with sha8-keyed confirm tokens 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:40:53 -04:00
bvandeusen 7c6f11964a fc3k: celery_app — register admin tasks on maintenance queue 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:40:23 -04:00
bvandeusen 94c60c0af2 fc3k: admin Celery tasks — delete_artist_cascade_task + bulk_delete_images_task on maintenance queue 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:40:03 -04:00
bvandeusen 6df102b83d fc3k: cleanup_service mutations — unlink primitive, artist cascade, bulk image delete, tag delete, prune unused 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:39:45 -04:00
bvandeusen 2ae01d27e3 fc3k: cleanup_service projections — artist cascade, bulk delete, tag usage, unused tags 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 00:38:57 -04:00
bvandeusen 718cc79905 fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:37:06 -04:00
bvandeusen e78a35d333 fc3h: collapse multi-line sqlalchemy import in backup_run.py — fits under line-length=100, ruff I001 would bounce 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:12:06 -04:00
bvandeusen 2b05f147f4 fc3h: migrators docstring — note backup/rollback retired to backup_service.py 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 23:00:01 -04:00
bvandeusen 70e1e010d1 fc3h: remove backend/app/services/migrators/backup.py + rollback.py (relocated to backup_service.py) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:59:18 -04:00
bvandeusen d3d4320ed5 fc3h: retire backup + rollback from migrate API/task — moved to /api/system/backup/* per FC-3h 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:59:11 -04:00
bvandeusen 7d42cddb11 fc3h: /api/system/backup blueprint — trigger, list, get, patch, restore, delete, settings 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:57:25 -04:00
bvandeusen 1f01c4819a fc3h: celery_app — register backup tasks (include + maintenance route + Beat hourly tick + daily prune) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:56:06 -04:00
bvandeusen 06d527cb92 fc3h: prune_backups (daily retention) + backup_db_nightly (hourly tick, settings-gated) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:55:19 -04:00
bvandeusen e9ea376aed fc3h: restore_db_task + restore_images_task — restore creates 'restoring' marker row linked via restored_from_id 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:53:50 -04:00
bvandeusen 882cb491ba fc3h: backup_db_task + backup_images_task Celery tasks 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:53:26 -04:00
bvandeusen 319e7de547 fc3h: backup_service.py — DB + images backup/restore + unlink helpers (relocated, split per-kind) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:52:35 -04:00
bvandeusen e43312a129 fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:52:00 -04:00
bvandeusen c3e855bd9b fc3h: BackupRun model — artifact record for backup/restore runs 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:50:59 -04:00
bvandeusen 37fcc74954 fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:22:42 -04:00
bvandeusen 541e2bfe6a fc3i: /api/system/activity blueprint — queues, workers, runs, failures 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:04:34 -04:00
bvandeusen 7782672a51 fc3i: recover_stalled_task_runs + prune_task_runs maintenance tasks + Beat entries 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:03:33 -04:00
bvandeusen d12b51f6b7 fc3i: Celery signal handlers populate task_run on every task lifecycle event 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:01:47 -04:00
bvandeusen 79fee98db4 fc3i: TaskRun model — per-Celery-task lifecycle audit row 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 21:00:14 -04:00
bvandeusen 553567738e fix: drop migration backup-gate (FC-3h supersedes) + modal Escape via document-level listener so video focus can't swallow it 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:38:33 -04:00
bvandeusen c9a3f12847 fix(lint): tag_apply.py — one blank line between imports and module-level comment (ruff I001, third bounce on this rule) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:22:05 -04:00
bvandeusen 538c1591e8 fc-3g-ext: IR Post/Provenance restore (tag_apply phase 4) + modal artist fallback 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 14:08:18 -04:00
bvandeusen 96039fc983 fix(ext): wrap /api/extension/manifest filesystem work in asyncio.to_thread (ruff ASYNC240) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 12:20:36 -04:00
bvandeusen be0f472894 fix(workers): worker-level recovery — autoretry on transient errors + tightened sweep (5min) + import_media_file body-wrap so no path leaves rows stuck in 'processing' 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 11:46:36 -04:00
bvandeusen dd4874ae8d fix: 30min wall-clock timeout on backup subprocess + global padding-top on .fc-content so views don't start under the sticky TopNav 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 11:10:04 -04:00
bvandeusen 2e1aaffd93 fc3g: serve /extension/<filename> with x-xpinstall MIME so Firefox installs in one click 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 23:39:10 -04:00
bvandeusen 2065672a31 fc3g: /api/extension blueprint — quick-add-source + manifest endpoints, registered 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 23:38:35 -04:00
bvandeusen 46d199450d fc3g: ExtensionService.quick_add_source — derive platform+slug, find-or-create Artist+Source 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 23:37:39 -04:00
bvandeusen 7a5a71471e fix(scan): auto-finalize empty scans + system_stats picks running batch with actual work 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 21:25:57 -04:00
bvandeusen f5efbea053 feat(fc5): /api/migrate/cleanup — delete every image attributed to one artist, files + thumbs + DB 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 12:09:56 -04:00
bvandeusen f653c26680 fix(web): serve /images/<path> from disk so thumbnails+originals render instead of grey placeholders 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 11:51:08 -04:00
bvandeusen 061dc9e605 fix(tasks): collapse double blank line between imports and IMAGES_ROOT in import_file.py (ruff I001) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 11:05:35 -04:00
bvandeusen 7d8b9c3d90 fix(tasks): share one sync engine per worker process to stop connection-pool leak 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 10:44:16 -04:00
bvandeusen 8f25b27315 fix(import): scan_directory skips paths with existing non-failed ImportTask 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 10:25:27 -04:00
bvandeusen 6f68bf5fa7 feat(ui): bump thumbnail sizes 1.5x, make Settings fluid, add subscription_count stat 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 23:18:22 -04:00
bvandeusen ec44c653fe fc5: strip SQLAlchemy +psycopg/+asyncpg driver suffix when passing URL to pg_dump/psql 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 22:54:19 -04:00
bvandeusen 8188985029 fc5: lift Quart request body cap to 1 GiB so large IR exports don't 413 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 22:23:29 -04:00
bvandeusen 599697c912 fix(fc5): ruff I001 — alphabetical order of import statements (backup < gs_ingest < rollback) 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 14:04:10 -04:00
bvandeusen 9fb3d11412 fix(fc5): ruff I001 — consolidate unaliased imports + tests/test_gs_ingest credential_type assertion 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:29:09 -04:00