feat: Cleanup can import files that are on disk but not in the library
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
A download writes its files into the artist's folder and imports them in a later step. A run that stopped in between left complete files no record points at, and a walk that had already marked them seen never returned for them; on the live instance that is about 95 GB of art. "Import files the library is missing" (Cleanup, Duplicates & leftovers) walks every folder named for an artist, matched by slug so a differently cased folder still matches, and lists the images, videos and archives no image, attachment or queued import points at. Stores, sidecars, staging files and files changed in the last day are left alone. Preview shows the counts per artist; Import queues the same list as a library import batch, whose tasks file the folder's post record, attach each file where it lies with its folder's artist and source, and remove the ones the library already holds or the import filters reject. A file whose import crashed is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -459,3 +459,29 @@ def reclaim_orphaned_attachments_task(self, dry_run: bool = True) -> dict:
|
||||
return cleanup_service.reclaim_orphaned_attachments(
|
||||
session, images_root=IMAGES_ROOT, dry_run=dry_run,
|
||||
)
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.admin.import_unimported_files_task",
|
||||
bind=True,
|
||||
autoretry_for=(OperationalError, DBAPIError),
|
||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
||||
# One walk of the artists' folders: a stat per file, so minutes on a large
|
||||
# library on network storage. The limits are the backstop for a wedged
|
||||
# filesystem, not the expected exit.
|
||||
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
|
||||
)
|
||||
def import_unimported_files_task(self, dry_run: bool = True) -> dict:
|
||||
"""Find files in the artists' folders that the library has no record of.
|
||||
dry_run (the default) returns what was found, per artist, without changing
|
||||
anything; apply queues them as a `library` import batch, which attaches each
|
||||
file where it lies and removes the ones the library already holds or the
|
||||
import filters reject."""
|
||||
from ..services import library_scan
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
if dry_run:
|
||||
_found, summary = library_scan.find_unimported(session, IMAGES_ROOT)
|
||||
return {**summary, "batch_id": None}
|
||||
return library_scan.queue_unimported(session, IMAGES_ROOT)
|
||||
|
||||
Reference in New Issue
Block a user