3 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	2bea4b0d31	fix: a Mega folder link's files in subfolders are imported, and a retry no longer deletes complete files ... _fetch_mega listed only the top level of the link dir, so a folder link's files in subfolders never reached the importer: the link either failed with "wrote no new file" or read downloaded with its subfolders left on disk. It now walks the tree. The worker cleared every unrecorded file in a link dir before each fetch and after a failed one. Combined with the daily recovery of dead links, that deletes complete downloads whose upstream is gone, which may be the only copy. Every fetcher stages to .part or .megatmp.* and renames on completion, so the clear now removes staging files only, and a successful fetch imports every complete unrecorded file in the link dir, including ones an earlier attempt finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:24:14 -05:00
bvandeusenandClaude Opus 4.8	4272a19d40	fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883) ... The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 21:15:49 -04:00
bvandeusenandClaude Opus 4.8	13253b18d1	feat(external): file-host fetcher subsystem (Phase 4a) ... Shared, reusable fetchers for the 5 off-platform hosts behind one signature (fetch_external(host, url, dest_dir, ...) -> FetchResult): - dropbox : force dl=1 + stream GET - pixeldrain : GET /api/file/{id} - mediafire : scrape the download page for the direct link + stream GET - gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs - mega : MEGAcmd `mega-get` subprocess (public link incl. #key) HTTP/gdown/subprocess go through module seams so unit tests run without network/gdown/MEGAcmd. fetch_external never raises — every backend failure (transport, non-200, scrape miss, subprocess error, stop) is captured on .error so the worker (next slice) records it and moves on. mega's binary lands in the runtime image in a later slice; the code is complete + tested now. Refs FC #830 (Phase 4a). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:29:20 -04:00