A download writes its files into the artist's folder and imports them in a
later step. A run that stopped in between left complete files no record
points at, and a walk that had already marked them seen never returned for
them; on the live instance that is about 95 GB of art.
"Import files the library is missing" (Cleanup, Duplicates & leftovers)
walks every folder named for an artist, matched by slug so a differently
cased folder still matches, and lists the images, videos and archives no
image, attachment or queued import points at. Stores, sidecars, staging
files and files changed in the last day are left alone. Preview shows the
counts per artist; Import queues the same list as a library import batch,
whose tasks file the folder's post record, attach each file where it lies
with its folder's artist and source, and remove the ones the library
already holds or the import filters reject. A file whose import crashed is
kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).
Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
quick lane — it lived on the 'ml' queue only by module colocation, which made
the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
installs keep working): OFF stops the four import hooks queueing embed work
nothing will consume and no-ops the manual backfill; switch lives on the
renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
that removes the container gives those up too.
Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.
**Split archive into its own task**
- New `import_archive_file` task: same body as import_media_file
(dispatch is by file-kind inside Importer.import_one) but
soft=30min / hard=35min. Shared `_run_import_task` helper holds the
flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
truth for media-vs-archive dispatch. Used by all three enqueue sites:
scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
is_archive(entry) (the model field already existed, anticipating
this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
task_routes `import_file.*` wildcard — no worker config change.
**Archive-aware recovery sweeps**
Both sweeps would otherwise preempt a legitimately-running archive:
- recover_interrupted_tasks (ImportTask 'processing' sweep): now
task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
35-min hard limit). Single UPDATE with an OR predicate over the two
(task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
the per-queue overrides added for ml. import_archive_file gets 40 min
while the 'import' queue stays at the 5-min default for single-file
imports. Precedence: task_name → queue → default, each pass excluding
rows claimed by a higher-precedence pass so every row is touched once.
**Tests**
- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
pins that a 10-min archive task-run survives, a 50-min one is flagged,
and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.
After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
import_media_file now enqueues tag_and_embed alongside generate_thumbnail
after a successful import. scripts/download_models.py snapshots Camie +
SigLIP into /models, idempotent (skips when present). The ml-worker
entrypoint runs it before starting the Celery worker so a fresh /models
volume self-heals on first boot. Downloader tests are pure-logic (no
network in CI).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff lint surfaced 23 violations across three rules; all addressed:
UP017 (Use datetime.UTC alias):
Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC),
also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for
timezone.utc that ruff's pyupgrade rules prefer.
UP042 (StrEnum):
Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)`
with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and
is the modern idiom. Behavior is equivalent for our usage (the .value
attribute, str(member) semantics).
I001 (Import sorting):
Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so
ruff groups `backend.*` imports correctly. Without it, ruff treated
them as third-party and demanded a different grouping. The existing
import order is stdlib → third-party → first-party → local relative,
which ruff now accepts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scan_directory walks ImportSettings.import_scan_path, creates an
ImportBatch, enumerates supported files into ImportTasks, and enqueues
import_media_file per task. import_media_file moves the task through
its state machine (pending → queued → processing → complete/skipped/failed),
updates ImportBatch counters atomically (UPDATE ... SET col = col + 1),
enqueues a thumbnail task on success, and marks the batch complete when
the last task drains.
generate_thumbnail runs on its own queue (thumbnail) so big imports
don't starve thumbnail throughput; failure here is logged and does not
fail the import.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5