feat: Cleanup can import files that are on disk but not in the library
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

A download writes its files into the artist's folder and imports them in a
later step. A run that stopped in between left complete files no record
points at, and a walk that had already marked them seen never returned for
them; on the live instance that is about 95 GB of art.

"Import files the library is missing" (Cleanup, Duplicates & leftovers)
walks every folder named for an artist, matched by slug so a differently
cased folder still matches, and lists the images, videos and archives no
image, attachment or queued import points at. Stores, sidecars, staging
files and files changed in the last day are left alone. Preview shows the
counts per artist; Import queues the same list as a library import batch,
whose tasks file the folder's post record, attach each file where it lies
with its folder's artist and source, and remove the ones the library
already holds or the import filters reject. A file whose import crashed is
kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 10:31:37 -05:00
co-authored by Claude Opus 5.5
parent 1b25f598c5
commit 2fc71cfe4e
10 changed files with 559 additions and 3 deletions
@@ -0,0 +1,150 @@
<template>
<!-- Files a download wrote into an artist's folder but never imported: a run
that stopped between writing and importing leaves them there, and the
walk that marked them seen never returns. Preview lists them per artist;
Import queues them as an import batch that attaches each file where it
lies. -->
<MaintenanceTile
icon="mdi-folder-download"
title="Import files the library is missing"
blurb="Find art in the artists' folders that was downloaded but never imported, and import it where it is."
:open="applying || previewing"
>
<p class="text-body-2 mb-3">
A download writes its files into the artist's folder, then imports them.
A run that stops in between leaves complete files that no image points
at, and later downloads skip them as already seen. This walks every
artist's folder for images, videos and archives the library has
<strong>no record of</strong>. <strong>Preview</strong> first;
<strong>Import</strong> runs them through the normal import where they
lie. A file that turns out to be a copy of something already in the
library, or that the import filters reject, is removed from disk. Files
changed in the last day are left alone, so a download still running is
never caught mid-write.
</p>
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
<v-btn
color="primary" variant="tonal" rounded="pill"
:loading="previewing" :disabled="applying" @click="preview"
>
<v-icon start>mdi-magnify</v-icon> Preview
</v-btn>
<v-btn
color="primary" rounded="pill"
:loading="applying"
:disabled="previewing || !canApply"
@click="confirmOpen = true"
>
<v-icon start>mdi-folder-download</v-icon> Import
</v-btn>
</div>
<v-alert
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
density="comfortable"
>
<span v-if="applied && summary.batch_id">
Queued {{ summary.files }} file(s) ({{ humanBytes(summary.bytes) }}) for
import. Progress shows in the import queue below and on Activity.
</span>
<span v-else-if="hasWork">
{{ summary.files }} file(s), {{ humanBytes(summary.bytes) }}, are on
disk but not in the library<span v-if="summary.archives">, including
{{ summary.archives }} archive(s)</span>.
Click <strong>Import</strong> to bring them in.
</span>
<span v-else>Nothing to import: every file in the artists' folders is in the library.</span>
<div v-if="passedOver" class="mt-1 text-caption">
Left alone: {{ summary.recent }} changed in the last day,
{{ summary.not_media }} that aren't images, videos or archives, and
{{ summary.no_artist }} in folders that match no artist.
</div>
</v-alert>
<v-table v-if="hasWork && !applied" density="compact" class="mt-3">
<thead>
<tr>
<th>Artist</th>
<th class="text-right">Files</th>
<th class="text-right">Size</th>
<th>For example</th>
</tr>
</thead>
<tbody>
<tr v-for="a in summary.artists" :key="a.slug">
<td>{{ a.slug }}</td>
<td class="text-right">{{ a.files }}</td>
<td class="text-right">{{ humanBytes(a.bytes) }}</td>
<td class="text-caption text-medium-emphasis fc-sample">{{ a.samples[0] }}</td>
</tr>
</tbody>
</v-table>
<QueueStatusBar queue="import" queue-label="Import" />
<v-dialog v-model="confirmOpen" max-width="440">
<v-card>
<v-card-title>Import {{ summary?.files ?? 0 }} file(s)?</v-card-title>
<v-card-text class="text-body-2">
This imports <strong>{{ summary?.files ?? 0 }}</strong> file(s)
({{ humanBytes(summary?.bytes) }}) where they lie. Any that turn out
to be copies of art already in the library, or that the import
filters reject, are removed from disk.
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
<v-btn color="primary" @click="apply">Import</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</MaintenanceTile>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
import { humanBytes } from '../../utils/bytes.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue'
const confirmOpen = ref(false)
// The walk stats every file in the artists' folders, so on a large library on
// network storage it can take many minutes. 900 polls × 2s = 30 min, the
// task's own soft limit.
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
endpoint: '/api/admin/maintenance/import-unimported',
storageKey: 'fc.maint.importUnimported',
appliedToast: 'Files queued for import',
maxPolls: 900,
})
const hasWork = computed(() => !!summary.value && summary.value.files > 0)
const canApply = computed(() => hasWork.value && !applied.value)
const passedOver = computed(() => {
const s = summary.value
return !!s && (s.recent > 0 || s.not_media > 0 || s.no_artist > 0)
})
const summaryType = computed(() => {
if (applied.value) return 'success'
return hasWork.value ? 'info' : 'success'
})
function apply () {
confirmOpen.value = false
applyTask()
}
</script>
<style scoped>
.fc-sample {
max-width: 320px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
</style>
+4 -1
View File
@@ -22,13 +22,15 @@
<h3 class="fc-section__title">Duplicates &amp; leftovers</h3>
<p class="fc-section__hint">
Tidy post records, duplicates, locked-preview leftovers and attachments
that outlived what they belonged to.
that outlived what they belonged to, and import downloads that never
made it into the library.
</p>
<div class="fc-tile-grid">
<PostMaintenanceCard />
<VideoDedupCard />
<GatedPurgeCard />
<AttachmentReclaimCard />
<LibraryImportCard />
</div>
</section>
@@ -63,6 +65,7 @@ import PostMaintenanceCard from '../components/settings/PostMaintenanceCard.vue'
import VideoDedupCard from '../components/settings/VideoDedupCard.vue'
import GatedPurgeCard from '../components/settings/GatedPurgeCard.vue'
import AttachmentReclaimCard from '../components/settings/AttachmentReclaimCard.vue'
import LibraryImportCard from '../components/settings/LibraryImportCard.vue'
import TagMaintenanceCard from '../components/settings/TagMaintenanceCard.vue'
import DangerZoneCard from '../components/settings/DangerZoneCard.vue'
</script>