Commit Graph

16 Commits

Author SHA1 Message Date
bvandeusen 4da8d1d774 fix(importer): race-safe savepoint-based find-or-create for Source + Post (uq_source_artist_platform_url UniqueViolation operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:51:12 -04:00
bvandeusen c0c9e56fb9 fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:45:08 -04:00
bvandeusen 52d7905c43 perf(importer): cache phash candidates on Importer to fix archive-import soft-timeout (was O(M×N) per-member SELECTs) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:25:32 -04:00
bvandeusen e6ededbe8e feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:16:26 -04:00
bvandeusen 36cc0622cb fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen a0470b5f60 feat(importer): _supersede() now applies the new (larger) file's sidecar — operator wanted to scan GS download dir to supersede smaller IR-migrated images AND wire up gallery-dl Post metadata, but supersede was file-only and silently dropped the sidecar.
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.

Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:51:51 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:34:36 -04:00
bvandeusen e5da27f755 fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 22:01:26 -04:00
bvandeusen a67755c929 feat(fc3c): Importer.attach_in_place + _supersede(new_path=) + _copy_to_library helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:38:47 -04:00
bvandeusen 5dfdbf5a60 feat(deep-scan): Importer(deep=) re-derive on sha-match; import_file wires batch.scan_mode
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 15:32:26 -04:00
bvandeusen f97551e2f6 feat(attachments): importer dispatch — archive extract + non-media capture
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 11:13:36 -04:00
bvandeusen 914cf14d9e feat(provenance): importer sets image_record.artist_id, stops minting artist tags
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 21:33:55 -04:00
bvandeusen 9b79b66b76 feat(sidecar): importer creates Source/Post/ImageProvenance from sidecars (idempotent, fail-soft)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 14:46:17 -04:00
bvandeusen a510378603 feat(phash): importer near-dup dispatch + in-place supersession (clear ML)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 22:05:43 -04:00
bvandeusen b4e0d680f1 fix(fc2a): satisfy ruff 0.15.13 lint — UP017, UP042, I001
Ruff lint surfaced 23 violations across three rules; all addressed:

UP017 (Use datetime.UTC alias):
  Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC),
  also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for
  timezone.utc that ruff's pyupgrade rules prefer.

UP042 (StrEnum):
  Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)`
  with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and
  is the modern idiom. Behavior is equivalent for our usage (the .value
  attribute, str(member) semantics).

I001 (Import sorting):
  Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so
  ruff groups `backend.*` imports correctly. Without it, ruff treated
  them as third-party and demanded a different grouping. The existing
  import order is stdlib → third-party → first-party → local relative,
  which ruff now accepts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:11:35 -04:00
bvandeusen bb9c183ff7 feat(fc2a): add Importer service
Single-file pipeline: validate as image, apply filter rules (min dimensions,
transparency), SHA256 hash dedup, atomic copy to /images/{subdir}/, create
ImageRecord with origin='imported_filesystem' and integrity_status='unknown',
auto-derive top-level folder name as Artist + artist tag.

Importer is intentionally sync (consumed by a Celery worker process); the
async Quart side uses the same ORM through its own async session. The
db_sync fixture in conftest.py was added in Task 3 to support these tests.

Thumbnail generation is NOT inlined; the calling Celery task enqueues a
separate thumbnail task so the import queue keeps moving on big batches.
pHash dedup is FC-2d, not here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 12:07:01 -04:00