9 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	5259caa846	fix: dead external links are retried sparingly, not three times a day ... The daily recovery reset every dead link to a fresh three-attempt budget, so a link that can never succeed (an archive too big to import inside the fetch's hard limit, a file made private) was downloaded up to three times a day, every day. Recovery now gives a dead link one more attempt once it has been dead for 2**recoveries days (1, 2, 4, 8, 16), and stops after five. A recovered link that fails again goes straight back to dead. Retention prunes only links recovery has given up on, 30 days after their last attempt. A manual refetch of a post restores the full budgets. Migration 0118 adds external_link.recoveries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 19:13:46 -05:00
bvandeusenandClaude Opus 5.5	157f2803c0	fix: downloads stop leaving files on disk that the library doesn't use ... - An archive whose images were extracted was kept at its download path as well as in the attachments store. Both download paths now remove the downloaded copy. - A downloaded file the import filters reject (too small, transparent, single colour, unreadable) was left on disk with no record. A later walk found it, imported it again and rejected it again, every time. Rejected files are now removed; the seen ledger keeps them from being downloaded again. - Both paths share one policy, Importer.discard_after_attach, which never removes a file an ImageRecord points at. - An external-link fetch that fails, or whose worker was killed, left partial files (megatools' .megatmp.*) in the link's dir, and a killed fetch left its link 'downloading' for good. A fetch now clears unrecorded leftovers before starting and after failing. The sweep returns a link to retry once it has been 'downloading' past the task's hard limit, using a new external_link.started_at column (migration 0117). - The orphaned temp-file sweep now recognises .megatmp.* as well as .part and .partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:00:31 -05:00
bvandeusenandClaude Fable 5	19b962f1a7	feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch ... The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:53:08 -04:00
bvandeusenandClaude Opus 4.8	ee1b45f8cc	test(external): fix third fake_fetch stub still requiring timeout= ... test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the old `*, timeout` signature; the task now calls fetch_external() without it, so the stub raised TypeError in the integration lane (run 1191). Switch it to **kwargs like the other two. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 21:20:27 -04:00
bvandeusenandClaude Opus 4.8	4272a19d40	fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883) ... The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 21:15:49 -04:00
bvandeusenandClaude Opus 4.8	949c9abcc6	fix(external): path-safe unlink + per-link staging + orphan repair (#859) ... External downloads import IN PLACE, so the post-attach dedup-skip unlink could delete a file that IS an ImageRecord's backing file — orphaning the record and 404-ing on playback. Two sources of that: - Two links on the same post (same film from mega + gdrive) emitted the same filename into one external/<post_id>/ dir; the second overwrote the first. Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path. - The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it path-safe: only unlink when `f` is NOT the existing record's canonical file. Plus an operator-triggered orphan-repair maintenance task (prune_missing_file_records_task) to clean up records already orphaned by the bug: scans ImageRecords, deletes those whose file is gone (cascade), with an NFS-stall guard that aborts without deleting if a large sample is mostly missing. Wired through POST /api/admin/maintenance/prune-missing-files and a MissingFileRepairCard in the Maintenance panel. Tests: refetch-same-link keeps the canonical file; orphan repair deletes only real orphans and aborts on the mostly-missing guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 01:48:38 -04:00
bvandeusenandClaude Opus 4.8	05f226a8f6	feat(external): zip-parity provenance/tagging + thorough worker logging ... Operator-requested: a worker download must be tagged + provenance-associated exactly like an extracted zip, and the path must log well (we won't get it right first try). - _route_files now mirrors download_service._phase3_persist branch-for-branch: imported/superseded → collect member_image_ids+image_id (provenance-linked via the synthesized sidecar, same as extracted-zip members) → caller enqueues tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed → unlink + warn. - Logging at every stage: start (link/host/post/artist/attempt/url), requeue, fetch result (files/bytes) or fetch failure, per-file import decision, dead- letter transitions, and done (files/images/duration). - Parity test: an archive downloaded by the worker is extracted, provenance- linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for exactly the member images. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:45:56 -04:00
bvandeusen	82b26b8aaa	test(external): unique seeded artist per host (fix uq_artist_name in sweep test)	2026-06-14 15:33:49 -04:00
bvandeusenandClaude Opus 4.8	96e984cded	feat(external): download worker for file-host links (Phase 4b) ... tasks/external.py drives the external_link ledger: - fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern; requeue-with-countdown if busy), fetch via external_fetch into the artist library tree, then route each file through importer.attach_in_place via a synthesized sidecar so it links to the SAME post (archive→ImageRecords, else→PostAttachment; on-disk original removed for captured files, art stays); thumbnail+ML enqueue for new images; status downloaded | failed | dead with attempts/last_error/completed_at/duration. - sweep_external_links(): enqueue a bounded batch of actionable links. - recover_external_links() + prune_external_links(): recovery + retention (#89). - per-host enable read via getattr (forward-compatible; Settings UI adds the columns in 4d — defaults on, rule #26). Wiring: celery include + route (download lane) + beat (sweep 10m, recover + prune daily); download_service phase 3 enqueues a sweep after recording links. Integration tests: download+attach, failure, dead-letter, non-claimable, sweep. mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:44:07 -04:00