feat: Cleanup can import files that are on disk but not in the library
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
A download writes its files into the artist's folder and imports them in a later step. A run that stopped in between left complete files no record points at, and a walk that had already marked them seen never returned for them; on the live instance that is about 95 GB of art. "Import files the library is missing" (Cleanup, Duplicates & leftovers) walks every folder named for an artist, matched by slug so a differently cased folder still matches, and lists the images, videos and archives no image, attachment or queued import points at. Stores, sidecars, staging files and files changed in the last day are left alone. Preview shows the counts per artist; Import queues the same list as a library import batch, whose tasks file the folder's post record, attach each file where it lies with its folder's artist and source, and remove the ones the library already holds or the import filters reject. A file whose import crashed is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
<template>
|
||||
<!-- Files a download wrote into an artist's folder but never imported: a run
|
||||
that stopped between writing and importing leaves them there, and the
|
||||
walk that marked them seen never returns. Preview lists them per artist;
|
||||
Import queues them as an import batch that attaches each file where it
|
||||
lies. -->
|
||||
<MaintenanceTile
|
||||
icon="mdi-folder-download"
|
||||
title="Import files the library is missing"
|
||||
blurb="Find art in the artists' folders that was downloaded but never imported, and import it where it is."
|
||||
:open="applying || previewing"
|
||||
>
|
||||
<p class="text-body-2 mb-3">
|
||||
A download writes its files into the artist's folder, then imports them.
|
||||
A run that stops in between leaves complete files that no image points
|
||||
at, and later downloads skip them as already seen. This walks every
|
||||
artist's folder for images, videos and archives the library has
|
||||
<strong>no record of</strong>. <strong>Preview</strong> first;
|
||||
<strong>Import</strong> runs them through the normal import where they
|
||||
lie. A file that turns out to be a copy of something already in the
|
||||
library, or that the import filters reject, is removed from disk. Files
|
||||
changed in the last day are left alone, so a download still running is
|
||||
never caught mid-write.
|
||||
</p>
|
||||
|
||||
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
|
||||
<v-btn
|
||||
color="primary" variant="tonal" rounded="pill"
|
||||
:loading="previewing" :disabled="applying" @click="preview"
|
||||
>
|
||||
<v-icon start>mdi-magnify</v-icon> Preview
|
||||
</v-btn>
|
||||
<v-btn
|
||||
color="primary" rounded="pill"
|
||||
:loading="applying"
|
||||
:disabled="previewing || !canApply"
|
||||
@click="confirmOpen = true"
|
||||
>
|
||||
<v-icon start>mdi-folder-download</v-icon> Import
|
||||
</v-btn>
|
||||
</div>
|
||||
|
||||
<v-alert
|
||||
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
|
||||
density="comfortable"
|
||||
>
|
||||
<span v-if="applied && summary.batch_id">
|
||||
Queued {{ summary.files }} file(s) ({{ humanBytes(summary.bytes) }}) for
|
||||
import. Progress shows in the import queue below and on Activity.
|
||||
</span>
|
||||
<span v-else-if="hasWork">
|
||||
{{ summary.files }} file(s), {{ humanBytes(summary.bytes) }}, are on
|
||||
disk but not in the library<span v-if="summary.archives">, including
|
||||
{{ summary.archives }} archive(s)</span>.
|
||||
Click <strong>Import</strong> to bring them in.
|
||||
</span>
|
||||
<span v-else>Nothing to import: every file in the artists' folders is in the library.</span>
|
||||
|
||||
<div v-if="passedOver" class="mt-1 text-caption">
|
||||
Left alone: {{ summary.recent }} changed in the last day,
|
||||
{{ summary.not_media }} that aren't images, videos or archives, and
|
||||
{{ summary.no_artist }} in folders that match no artist.
|
||||
</div>
|
||||
</v-alert>
|
||||
|
||||
<v-table v-if="hasWork && !applied" density="compact" class="mt-3">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Artist</th>
|
||||
<th class="text-right">Files</th>
|
||||
<th class="text-right">Size</th>
|
||||
<th>For example</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr v-for="a in summary.artists" :key="a.slug">
|
||||
<td>{{ a.slug }}</td>
|
||||
<td class="text-right">{{ a.files }}</td>
|
||||
<td class="text-right">{{ humanBytes(a.bytes) }}</td>
|
||||
<td class="text-caption text-medium-emphasis fc-sample">{{ a.samples[0] }}</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</v-table>
|
||||
|
||||
<QueueStatusBar queue="import" queue-label="Import" />
|
||||
|
||||
<v-dialog v-model="confirmOpen" max-width="440">
|
||||
<v-card>
|
||||
<v-card-title>Import {{ summary?.files ?? 0 }} file(s)?</v-card-title>
|
||||
<v-card-text class="text-body-2">
|
||||
This imports <strong>{{ summary?.files ?? 0 }}</strong> file(s)
|
||||
({{ humanBytes(summary?.bytes) }}) where they lie. Any that turn out
|
||||
to be copies of art already in the library, or that the import
|
||||
filters reject, are removed from disk.
|
||||
</v-card-text>
|
||||
<v-card-actions>
|
||||
<v-spacer />
|
||||
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
|
||||
<v-btn color="primary" @click="apply">Import</v-btn>
|
||||
</v-card-actions>
|
||||
</v-card>
|
||||
</v-dialog>
|
||||
</MaintenanceTile>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed, ref } from 'vue'
|
||||
|
||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||
import { humanBytes } from '../../utils/bytes.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
const confirmOpen = ref(false)
|
||||
|
||||
// The walk stats every file in the artists' folders, so on a large library on
|
||||
// network storage it can take many minutes. 900 polls × 2s = 30 min, the
|
||||
// task's own soft limit.
|
||||
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
|
||||
endpoint: '/api/admin/maintenance/import-unimported',
|
||||
storageKey: 'fc.maint.importUnimported',
|
||||
appliedToast: 'Files queued for import',
|
||||
maxPolls: 900,
|
||||
})
|
||||
|
||||
const hasWork = computed(() => !!summary.value && summary.value.files > 0)
|
||||
const canApply = computed(() => hasWork.value && !applied.value)
|
||||
const passedOver = computed(() => {
|
||||
const s = summary.value
|
||||
return !!s && (s.recent > 0 || s.not_media > 0 || s.no_artist > 0)
|
||||
})
|
||||
const summaryType = computed(() => {
|
||||
if (applied.value) return 'success'
|
||||
return hasWork.value ? 'info' : 'success'
|
||||
})
|
||||
|
||||
function apply () {
|
||||
confirmOpen.value = false
|
||||
applyTask()
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.fc-sample {
|
||||
max-width: 320px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
</style>
|
||||
@@ -22,13 +22,15 @@
|
||||
<h3 class="fc-section__title">Duplicates & leftovers</h3>
|
||||
<p class="fc-section__hint">
|
||||
Tidy post records, duplicates, locked-preview leftovers and attachments
|
||||
that outlived what they belonged to.
|
||||
that outlived what they belonged to, and import downloads that never
|
||||
made it into the library.
|
||||
</p>
|
||||
<div class="fc-tile-grid">
|
||||
<PostMaintenanceCard />
|
||||
<VideoDedupCard />
|
||||
<GatedPurgeCard />
|
||||
<AttachmentReclaimCard />
|
||||
<LibraryImportCard />
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -63,6 +65,7 @@ import PostMaintenanceCard from '../components/settings/PostMaintenanceCard.vue'
|
||||
import VideoDedupCard from '../components/settings/VideoDedupCard.vue'
|
||||
import GatedPurgeCard from '../components/settings/GatedPurgeCard.vue'
|
||||
import AttachmentReclaimCard from '../components/settings/AttachmentReclaimCard.vue'
|
||||
import LibraryImportCard from '../components/settings/LibraryImportCard.vue'
|
||||
import TagMaintenanceCard from '../components/settings/TagMaintenanceCard.vue'
|
||||
import DangerZoneCard from '../components/settings/DangerZoneCard.vue'
|
||||
</script>
|
||||
|
||||
Reference in New Issue
Block a user