4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	2bea4b0d31	fix: a Mega folder link's files in subfolders are imported, and a retry no longer deletes complete files ... _fetch_mega listed only the top level of the link dir, so a folder link's files in subfolders never reached the importer: the link either failed with "wrote no new file" or read downloaded with its subfolders left on disk. It now walks the tree. The worker cleared every unrecorded file in a link dir before each fetch and after a failed one. Combined with the daily recovery of dead links, that deletes complete downloads whose upstream is gone, which may be the only copy. Every fetcher stages to .part or .megatmp.* and renames on completion, so the clear now removes staging files only, and a successful fetch imports every complete unrecorded file in the link dir, including ones an earlier attempt finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:24:14 -05:00
bvandeusenandClaude Opus 4.8	4272a19d40	fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883) ... The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 21:15:49 -04:00
bvandeusenandClaude Opus 4.8	bd2807cdd1	feat(external): mega.nz via megatools in the runtime image (Phase 4c) ... Use `megatools dl` (Debian-native apt package) for mega.nz public links rather than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds `megatools` to the runtime Dockerfile; the fetcher's mega backend now shells `megatools dl --path <dir> <url>` (key in the #fragment is preserved by the extractor). gdown (gdrive) is already a pip dep in the runtime image. NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile change is verified on the next dev→main merge, not by this dev push. The fetcher code path is unit-tested via the mocked _run_mega_get seam. With this, all 5 hosts download end-to-end once a celery download-worker runs. Refs FC #830 (Phase 4c). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:41:18 -04:00
bvandeusenandClaude Opus 4.8	13253b18d1	feat(external): file-host fetcher subsystem (Phase 4a) ... Shared, reusable fetchers for the 5 off-platform hosts behind one signature (fetch_external(host, url, dest_dir, ...) -> FetchResult): - dropbox : force dl=1 + stream GET - pixeldrain : GET /api/file/{id} - mediafire : scrape the download page for the direct link + stream GET - gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs - mega : MEGAcmd `mega-get` subprocess (public link incl. #key) HTTP/gdown/subprocess go through module seams so unit tests run without network/gdown/MEGAcmd. fetch_external never raises — every backend failure (transport, non-200, scrape miss, subprocess error, stop) is captured on .error so the worker (next slice) records it and moves on. mega's binary lands in the runtime image in a later slice; the code is complete + tested now. Refs FC #830 (Phase 4a). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:29:20 -04:00