fix: downloads stop leaving files on disk that the library doesn't use
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
- An archive whose images were extracted was kept at its download path as well as in the attachments store. Both download paths now remove the downloaded copy. - A downloaded file the import filters reject (too small, transparent, single colour, unreadable) was left on disk with no record. A later walk found it, imported it again and rejected it again, every time. Rejected files are now removed; the seen ledger keeps them from being downloaded again. - Both paths share one policy, Importer.discard_after_attach, which never removes a file an ImageRecord points at. - An external-link fetch that fails, or whose worker was killed, left partial files (megatools' .megatmp.*) in the link's dir, and a killed fetch left its link 'downloading' for good. A fetch now clears unrecorded leftovers before starting and after failing. The sweep returns a link to retry once it has been 'downloading' past the task's hard limit, using a new external_link.started_at column (migration 0117). - The orphaned temp-file sweep now recognises .megatmp.* as well as .part and .partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1401,6 +1401,47 @@ class Importer:
|
||||
self.session.commit()
|
||||
return ImportResult(status="imported", image_id=record.id)
|
||||
|
||||
def discard_after_attach(self, path: Path, result: ImportResult) -> bool:
|
||||
"""Delete a downloaded file that `attach_in_place` has finished with,
|
||||
when keeping it would only leave a second copy or a rejected file on
|
||||
disk. Returns True when the file was removed. Both download paths (a
|
||||
subscription walk's import step and the external-link worker) apply
|
||||
this one policy:
|
||||
|
||||
- attached: the attachments store holds the canonical copy.
|
||||
- imported or superseded ARCHIVE: its new images are in the library and
|
||||
the archive itself is in the attachments store.
|
||||
- duplicate skip: the library already holds the content.
|
||||
- soft skip (too small, too transparent, single colour, not a readable
|
||||
image): the import filters rejected it. A file left on disk with no
|
||||
record is picked up by the next walk and imported again; the seen
|
||||
ledger is what stops it being downloaded again.
|
||||
- failed: the archive probe crashed, and a retry would crash again.
|
||||
|
||||
An imported or superseded media file is kept: it IS the record's file.
|
||||
So is any path an ImageRecord points at, whatever the status, because a
|
||||
re-fetch can write identical bytes over a record's own file."""
|
||||
if result.status in ("imported", "superseded"):
|
||||
redundant = is_archive(path)
|
||||
else:
|
||||
redundant = result.status in ("attached", "skipped", "failed")
|
||||
if not redundant:
|
||||
return False
|
||||
owned = self.session.execute(
|
||||
select(ImageRecord.id).where(
|
||||
ImageRecord.path.in_({str(path), str(path.resolve())})
|
||||
).limit(1)
|
||||
).first()
|
||||
if owned is not None:
|
||||
log.info("keeping %s: it is image %s's file", path.name, owned.id)
|
||||
return False
|
||||
try:
|
||||
path.unlink(missing_ok=True)
|
||||
except OSError as exc:
|
||||
log.warning("could not remove %s: %s", path, exc)
|
||||
return False
|
||||
return True
|
||||
|
||||
def _upsert_artist(self, name: str) -> Artist:
|
||||
slug = slugify(name)
|
||||
artist = self.session.execute(
|
||||
|
||||
Reference in New Issue
Block a user