fix: a Mega folder link's files in subfolders are imported, and a retry no longer deletes complete files
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Failing after 33s
CI and images / extension-test (push) Successful in 18s
CI and images / integration (push) Successful in 2m30s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped

_fetch_mega listed only the top level of the link dir, so a folder link's
files in subfolders never reached the importer: the link either failed with
"wrote no new file" or read downloaded with its subfolders left on disk.
It now walks the tree.

The worker cleared every unrecorded file in a link dir before each fetch
and after a failed one. Combined with the daily recovery of dead links,
that deletes complete downloads whose upstream is gone, which may be the
only copy. Every fetcher stages to .part or .megatmp.* and renames on
completion, so the clear now removes staging files only, and a successful
fetch imports every complete unrecorded file in the link dir, including
ones an earlier attempt finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 10:24:14 -05:00
co-authored by Claude Opus 5.5
parent 5259caa846
commit 2bea4b0d31
4 changed files with 136 additions and 25 deletions
+10 -2
View File
@@ -241,11 +241,19 @@ def _fetch_gdrive(url: str, dest_dir: Path, *, read_timeout: float,
def _fetch_mega(url: str, dest_dir: Path, *, read_timeout: float,
total_timeout: float,
should_stop: Callable[[], bool]) -> FetchResult:
before = set(dest_dir.iterdir()) if dest_dir.exists() else set()
# A folder link keeps its subfolders, so files are found at any depth.
# Dotfiles are megatools' `.megatmp.*` staging names.
def _complete_files() -> set[Path]:
return {
p for p in dest_dir.rglob("*")
if p.is_file() and not p.name.startswith(".")
} if dest_dir.exists() else set()
before = _complete_files()
# megatools is a subprocess: its timeout IS a total wall-clock cap (the read
# timeout has no analogue here), so the total budget applies directly.
_run_mega_get(url, str(dest_dir), timeout=total_timeout)
new = [p for p in dest_dir.iterdir() if p not in before and p.is_file()]
new = sorted(_complete_files() - before)
if not new:
return FetchResult(error="mega-get wrote no new file")
return FetchResult(files=new, bytes=sum(p.stat().st_size for p in new))