Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.
Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
Both captured at correction time in TagService.add_to_image/remove_from_image
(source is lost on delete) into durable per-tag counters (head_metric), keyed
by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
(head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
+ head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
head quality, plus the snapshot time-series — the report to tune the precision
target + support floor.
Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.
What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.
NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Graduated heads can now apply their tag without a human — gated so it's safe:
- FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled,
default OFF) is on AND it has >= head_auto_apply_min_positives (default 30)
clean labels. A precise-looking but under-supported low-N head can't spray tags.
- auto_apply_sweep (heads.py): streams every embedded image in chunks, scores
against the eligible heads (numpy, no sklearn), applies each head's tag where
score >= its auto_apply_threshold and the tag isn't already applied/rejected,
with source='head_auto' (distinguishable + reversible). dry_run counts only.
- HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags
task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled)
+ recovery sweep + retention(20).
- API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled),
GET /api/heads/auto-apply (recent runs + per-concept report). Settings
head_auto_apply_enabled + min_positives via /api/ml/settings.
Tests: sweep applies above threshold, dry-run writes nothing, skips under-
supported + ungraduated heads; API disabled/dry-run/conflict guards.
NEXT (slice 2): the observability the operator asked for — per-concept misfire
(auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a
reporting API to tune. Slice 3: the UI (enable, preview, trends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before
I), and drop the inline comment that split heads.py's import block. Pure import
ordering — no behavior change. (run 1601 lint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).
Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
(connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
across chunks in _stream_to_file (HTTP has no total-download timeout), and
passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.
Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.
Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.
Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
External file-host fetches run to a 60-min hard limit (time_limit=3600,
per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min
default. recover_stalled_task_runs was phantom-flagging healthy in-flight
fetches as "RecoverySweep: no completion signal received within 5 min"
before the task's own timeout/error handling could surface the real error
(operator-flagged: target 414 swept at 6.6min).
The sweep already has per-queue/per-task overrides for long tasks, but
fetch_external_link was never added and its TaskRun records queue='default'
(no queue override) despite external.* routing to download. Add a task-name
override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it
robust regardless of the recorded queue. No new internal timeout needed —
the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception
already capture the real failure once the sweep stops preempting.
Pinned tests: external-fetch override survives a 10-min row / flags a 70-min
row on queue='default'; invariant guard asserts override >= hard time_limit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recover_stalled_task_runs used the 5-min default for the download queue,
but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT
(1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as
phantom 'RecoverySweep' failures — visible in System Activity but absent
from the Subscriptions view (the download finished ok, reset the source's
consecutive_failures; only the orphaned task_run kept the stamp, since
_finalize only updates rows still 'running').
Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard
limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real
hard kill is swept by the task-run and event sweeps together. Restores the
documented invariant (every override >= task time_limit). Regression test
pins the threshold above the hard limit so a future limit bump can't
silently re-break it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM
ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat
readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats.
- dbMaintenance store (loadStats / runVacuum) + test.
- Fix ruff I001: combine the two _sync_engine imports onto one line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.
- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
(VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
autocommit connection.
- GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.
Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).
Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
subprocess budgets (870 tick / 1170 backfill) sit below the soft
limit with phase-3 persist headroom. Promote to module constants and
guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
in-flight event with a real reason, mirror phase-3 source-health, and
decrement backfill so a chronically-slow source self-heals to tick
mode. The existing celery_signals handler only covered TaskRun, not
DownloadEvent — that was the gap.
Updates stale 900/1200 references in gallery_dl.py + maintenance.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
TimeoutExpired (matched the tick 870s/900s rationale). Backfill
runs that hit the cap let the next tick continue via the archive.
- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
without it cleanup_old_tasks' WHERE finished_at<cutoff never
reaped orphan-swept rows. recover_stalled_task_runs also now sets
duration_ms (matches celery_signals.finalize's millisecond math).
- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
on Source creation, mirroring SourceService.create. Without it,
Firefox quick-add on a creator with >20 unsynced posts walked the
full feed until subprocess timeout. Renamed the constant from
_NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.
- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
NO_NEW_CONTENT — the download path (line 712) already does, and
TIER_LIMITED proves auth reached the post and was told it was
tier-gated. Verify endpoint previously showed red on this and
prompted operators to rotate working cookies.
- prune_unused_tags now runs a single DELETE with the NOT-IN
predicate find_unused_tags uses, instead of SELECT-ids →
DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
would have surfaced on a tag explosion (>65k unused tags).
- credentials.upload() reflects the returned record into the store
cache (`.set(platform, rec)`) instead of evicting it; previously
the card briefly rendered "no credential" between upload and
loadAll().
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.
cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).
New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
download_source 1200s hard kill, so legitimately-running tasks are
never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
backoff is 2^N on that counter so per-event bumps would needlessly
inflate the next interval — sets last_error, stamps last_checked_at.
UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.
Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.
New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
derive the artist from the import path, find an ENABLED Source with a
real feed URL for (artist, platform). Returns None for filesystem-only
imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
set ImportTask.refetched=True, and trigger ONE download_source
re-check. Bounded by `refetched` so source-side corruption can't loop.
Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
tasks). Returns refetch_queued / no_source / already_refetched /
not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
manual button is the primary path; auto is opt-in since re-fetch
deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
→ stores.import.refetchTask → toast keyed on the result status.
Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.
Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.
**Split archive into its own task**
- New `import_archive_file` task: same body as import_media_file
(dispatch is by file-kind inside Importer.import_one) but
soft=30min / hard=35min. Shared `_run_import_task` helper holds the
flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
truth for media-vs-archive dispatch. Used by all three enqueue sites:
scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
is_archive(entry) (the model field already existed, anticipating
this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
task_routes `import_file.*` wildcard — no worker config change.
**Archive-aware recovery sweeps**
Both sweeps would otherwise preempt a legitimately-running archive:
- recover_interrupted_tasks (ImportTask 'processing' sweep): now
task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
35-min hard limit). Single UPDATE with an OR predicate over the two
(task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
the per-queue overrides added for ml. import_archive_file gets 40 min
while the 'import' queue stays at the 5-min default for single-file
imports. Precedence: task_name → queue → default, each pass excluding
rows claimed by a higher-precedence pass so every row is touched once.
**Tests**
- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
pins that a 10-min archive task-run survives, a 50-min one is flagged,
and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.
After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.
Root cause is two interacting limits, both undersized for video work:
tag_and_embed: soft_time_limit=300, time_limit=420
(sized for the image branch, ≈2 GPU ops)
recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues
The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.
**Two-part fix**
1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
(20 min). Sized for the video path's worst case; image runs return
in seconds and don't care.
2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
Queues with legitimately-long-running tasks (currently just `ml` at
25 min — 5-min buffer past the new hard kill) get their own
threshold; queues not in the dict use the default 5 min. The sweep
now issues one UPDATE per distinct threshold value, with
`queue.notin_(override_queues)` on the default pass so each row is
touched at most once.
Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
pins both directions: a 10-min-old ml row survives (fresh by 25-min
override), a 30-min-old ml row gets flagged.
After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff:
The remaining I001 errors came from ruff treating `alembic` as a first-
party module (because the alembic/ directory exists in the repo root)
rather than third-party. Ran `ruff check --fix` locally — auto-sorted
import groupings to put alembic/sqlalchemy alongside backend.* as first-
party, and trimmed redundant blank lines after a few import blocks.
Frontend:
`npm run check` (vue-tsc --noEmit) was failing because vue-tsc has no
tsconfig.json to read against, and the frontend is pure JS without
JSDoc annotations — vue-tsc had nothing to do. Skipping the step until
we add a tsconfig + convert to TS or add JSDoc annotations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff lint surfaced 23 violations across three rules; all addressed:
UP017 (Use datetime.UTC alias):
Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC),
also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for
timezone.utc that ruff's pyupgrade rules prefer.
UP042 (StrEnum):
Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)`
with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and
is the modern idiom. Behavior is equivalent for our usage (the .value
attribute, str(member) semantics).
I001 (Import sorting):
Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so
ruff groups `backend.*` imports correctly. Without it, ruff treated
them as third-party and demanded a different grouping. The existing
import order is stdlib → third-party → first-party → local relative,
which ruff now accepts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
recover_interrupted_tasks runs every 5 minutes, finds ImportTask rows
stuck in 'processing' for >30 minutes (well above any legitimate import
duration), and re-queues them. cleanup_old_tasks runs daily and deletes
finished tasks older than 7 days so the task table stays an operational
view rather than an archive.
Both thresholds match ImageRepo's precedent. The 30-min stuck threshold
is documented inline so a future reader can adjust it intentionally
rather than mistaking it for a 'magic number'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen