Files
FabledCurator/frontend/src/components/subscriptions/SubscriptionsTab.vue
T
bvandeusen 19aece1fc4
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m20s
feat(download): tick/backfill modes + partial-success classifier (plan #544)
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.

Two coupled changes, both operator-flagged 2026-06-01:

* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
  asks gallery-dl to exit after 20 contiguous archived items. Fresh
  subscriptions + new-content cases still walk normally; established
  subscription with zero new content exits in ~30s of HEAD requests
  instead of pegging the timeout. 20 (not 5) gives headroom against
  paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
  via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
  downloads use `skip: True` + 1800s timeout. Auto-decrements per run
  with early-reset to 0 when a clean run finds zero files (queue
  drained). N defaults to 3 — multiple runs give the system enough
  budget to finish a deep walk across timeout boundaries. New
  `POST /api/sources/{id}/backfill` arms the source; "Deep scan"
  button on each SourceRow (chip shows remaining count) wires it.

Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."

Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).

Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
2026-06-01 18:23:28 -04:00

592 lines
18 KiB
Vue
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<template>
<div>
<SchedulerStatusBar :status="store.scheduleStatus" class="fc-subs__sched" />
<div class="fc-subs__bar">
<v-btn color="accent" prepend-icon="mdi-plus" @click="openAddSource(null)">
Add subscription
</v-btn>
<v-btn variant="outlined" prepend-icon="mdi-account-plus" @click="showArtistDialog = true">
New artist
</v-btn>
<v-chip
:color="needsAttention ? 'error' : undefined"
:variant="needsAttention ? 'flat' : 'outlined'"
prepend-icon="mdi-alert-circle-outline"
@click="needsAttention = !needsAttention"
>
Needs attention
<v-chip v-if="attentionCount" size="x-small" class="ms-2" variant="tonal">
{{ attentionCount }}
</v-chip>
</v-chip>
<v-spacer />
<v-select
v-model="statusFilter"
:items="STATUS_OPTIONS"
:disabled="needsAttention"
density="compact" variant="outlined" hide-details
style="max-width: 180px"
/>
<v-text-field
v-model="search"
density="compact" variant="outlined" hide-details clearable
prepend-inner-icon="mdi-magnify"
placeholder="Search subscriptions"
style="max-width: 320px"
/>
</div>
<v-slide-y-transition>
<v-card
v-if="selected.length"
variant="tonal" color="info"
class="fc-subs__bulk mb-3"
>
<v-card-text class="d-flex align-center pa-3 ga-3">
<span class="text-body-2">
{{ selected.length }} selected
</span>
<v-spacer />
<v-btn size="small" variant="text" prepend-icon="mdi-toggle-switch" @click="bulkSetEnabled(true)">
Enable all
</v-btn>
<v-btn size="small" variant="text" prepend-icon="mdi-toggle-switch-off-outline" @click="bulkSetEnabled(false)">
Disable all
</v-btn>
<v-btn size="small" variant="text" color="error" prepend-icon="mdi-delete" @click="bulkDelete">
Delete
</v-btn>
<v-btn size="small" variant="text" @click="selected = []">Clear</v-btn>
</v-card-text>
</v-card>
</v-slide-y-transition>
<v-alert v-if="store.error" type="error" variant="tonal" closable class="mb-3">
{{ String(store.error) }}
</v-alert>
<div v-if="store.loading && groups.length === 0" class="fc-subs__loading">
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<div v-else-if="filteredGroups.length === 0" class="fc-subs__empty">
<p v-if="groups.length === 0">No subscriptions yet. Add your first artist.</p>
<p v-else>No subscriptions match the current filter.</p>
</div>
<v-card v-else class="fc-subs__card" variant="outlined">
<v-data-table
:headers="headers"
:items="filteredGroups"
item-value="key"
v-model="selected"
v-model:expanded="expanded"
:items-per-page="50"
:items-per-page-options="ITEMS_PER_PAGE_OPTIONS"
density="comfortable"
hover show-select show-expand
@click:row="onRowClick"
>
<template #item.name="{ item }">
<span class="fc-subs__name">{{ item.artist.name }}</span>
</template>
<template #item.platforms="{ item }">
<div class="fc-subs__chips">
<PlatformChip
v-for="p in item.platforms" :key="p"
:platform="p" size="x-small"
/>
</div>
</template>
<template #item.sources_count="{ item }">
<v-chip size="x-small" variant="tonal" label>
{{ item.sources.length }}
</v-chip>
</template>
<template #item.health="{ item }">
<SourceHealthDot
v-if="item.worstSource"
:source="item.worstSource"
:warning-threshold="failureThreshold"
/>
<span v-else class="fc-subs__zero"></span>
</template>
<template #item.last_activity="{ item }">
<span class="fc-subs__when">{{ formatRelative(item.lastActivity) }}</span>
</template>
<template #item.actions="{ item }">
<v-btn
icon size="small" variant="text"
:loading="anyChecking(item.sources)"
@click.stop="checkAll(item)"
>
<v-icon>mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Check all sources</v-tooltip>
</v-btn>
<v-btn icon size="small" variant="text" @click.stop="openAddSource(item.artist)">
<v-icon>mdi-plus</v-icon>
<v-tooltip activator="parent" location="top">Add source</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/posts?artist_id=${item.artist.id}`" @click.stop
>
<v-icon>mdi-rss</v-icon>
<v-tooltip activator="parent" location="top">View posts</v-tooltip>
</v-btn>
<v-btn
icon size="small" variant="text"
:to="`/artist/${item.artist.slug}`" @click.stop
>
<v-icon>mdi-account</v-icon>
<v-tooltip activator="parent" location="top">Open artist page</v-tooltip>
</v-btn>
</template>
<template #expanded-row="{ columns, item }">
<tr class="fc-subs__sources-row">
<td :colspan="columns.length" class="fc-subs__sources-cell">
<v-table density="compact" class="fc-subs__sources-table">
<thead>
<tr>
<th></th>
<th>Platform</th>
<th>URL</th>
<th>Enabled</th>
<th>Last check</th>
<th>Next check</th>
<th>Errors</th>
<th class="text-right">Actions</th>
</tr>
</thead>
<tbody>
<SourceRow
v-for="s in item.sources" :key="s.id" :source="s"
:checking="store.checkingIds.has(s.id)"
:warning-threshold="failureThreshold"
@edit="openEditSource"
@remove="removeSource"
@toggle="toggleSourceEnabled"
@check="onCheck"
@backfill="onBackfill"
/>
<tr v-if="item.sources.length === 0">
<td colspan="8" class="fc-subs__sources-empty">
No sources yet. Click + to add one.
</td>
</tr>
</tbody>
</v-table>
</td>
</tr>
</template>
</v-data-table>
</v-card>
<SourceFormDialog
v-model="showSourceDialog"
:source="editingSource"
:initial-artist="editingArtist"
@saved="onSourceSaved"
/>
<ArtistCreateDialog v-model="showArtistDialog" @created="onArtistCreated" />
</div>
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useSourcesStore } from '../../stores/sources.js'
import { usePlatformsStore } from '../../stores/platforms.js'
import { useImportStore } from '../../stores/import.js'
import SourceRow from './SourceRow.vue'
import SourceHealthDot from './SourceHealthDot.vue'
import SourceFormDialog from './SourceFormDialog.vue'
import ArtistCreateDialog from './ArtistCreateDialog.vue'
import PlatformChip from './PlatformChip.vue'
import SchedulerStatusBar from './SchedulerStatusBar.vue'
import { formatRelative } from '../../utils/date.js'
const ITEMS_PER_PAGE_OPTIONS = [
{ value: 25, title: '25' },
{ value: 50, title: '50' },
{ value: 100, title: '100' },
{ value: -1, title: 'All' },
]
const STATUS_OPTIONS = [
{ title: 'All status', value: 'all' },
{ title: 'Enabled', value: 'enabled' },
{ title: 'Disabled', value: 'disabled' },
{ title: 'Has errors', value: 'errors' },
{ title: 'Stale', value: 'stale' },
]
const route = useRoute()
const router = useRouter()
const store = useSourcesStore()
const platformsStore = usePlatformsStore()
const importStore = useImportStore()
const search = ref('')
const statusFilter = ref('all')
const needsAttention = ref(false)
const expanded = ref([])
const selected = ref([])
const showSourceDialog = ref(false)
const editingSource = ref(null)
const editingArtist = ref(null)
const showArtistDialog = ref(false)
const artistFilter = computed(() => {
const raw = route.query.artist_id
return raw == null ? null : Number(raw)
})
const failureThreshold = computed(() =>
importStore.settings?.download_failure_warning_threshold ?? 5,
)
async function refresh() {
await store.loadAll()
await platformsStore.loadAll()
store.loadScheduleStatus()
if (!importStore.settings) await importStore.loadSettings()
}
// Re-poll scheduler status every 30s so the "last ran" reading + health
// dot stay honest (the Beat tick fires every 60s).
let schedId = null
onMounted(() => {
refresh()
if (artistFilter.value != null) {
expanded.value = [`artist-${artistFilter.value}`]
}
schedId = setInterval(() => {
if (!document.hidden) store.loadScheduleStatus()
}, 30000)
})
onUnmounted(() => {
if (schedId) { clearInterval(schedId); schedId = null }
})
watch(() => route.query.artist_id, refresh)
const headers = [
{ title: 'Subscription', key: 'name', sortable: true, align: 'start' },
{ title: 'Platforms', key: 'platforms', sortable: false, align: 'start', width: 240 },
{ title: 'Sources', key: 'sources_count', sortable: true, align: 'start', width: 90 },
{ title: 'Health', key: 'health', sortable: false, align: 'start', width: 80 },
{ title: 'Last activity',key: 'last_activity', sortable: true, align: 'start', width: 140 },
{ title: 'Actions', key: 'actions', sortable: false, align: 'end', width: 200 },
]
const groups = computed(() => {
const all = store.sourcesByArtistGrouped()
return all.map((g) => {
const worstSource = pickWorstSource(g.sources, failureThreshold.value)
const lastActivity = pickLastActivity(g.sources)
const platforms = [...new Set(g.sources.map((s) => s.platform).filter(Boolean))]
return {
key: `artist-${g.artist.id}`,
artist: g.artist,
sources: g.sources,
sources_count: g.sources.length,
platforms,
worstSource,
lastActivity,
name: g.artist.name,
last_activity: lastActivity ?? '',
}
})
})
// A group "needs attention" if any of its sources has accumulated
// failures OR has never been checked — the ones worth acting on.
function groupNeedsAttention(g) {
return groupMatchesStatus(g, 'errors') || groupMatchesStatus(g, 'stale')
}
const attentionCount = computed(
() => groups.value.filter(groupNeedsAttention).length,
)
const filteredGroups = computed(() => {
let arr = groups.value
if (artistFilter.value != null) {
arr = arr.filter((g) => g.artist.id === artistFilter.value)
}
if (needsAttention.value) {
arr = arr.filter(groupNeedsAttention)
} else if (statusFilter.value !== 'all') {
arr = arr.filter((g) => groupMatchesStatus(g, statusFilter.value))
}
const q = search.value?.trim().toLowerCase()
if (q) {
arr = arr.filter(
(g) =>
g.artist.name.toLowerCase().includes(q) ||
g.sources.some(
(s) =>
(s.url || '').toLowerCase().includes(q) ||
(s.platform || '').toLowerCase().includes(q),
),
)
}
return arr
})
function groupMatchesStatus(g, status) {
if (status === 'enabled') return g.sources.some((s) => s.enabled)
if (status === 'disabled') return g.sources.every((s) => !s.enabled)
if (status === 'errors') return g.sources.some((s) => (s.consecutive_failures || 0) > 0)
if (status === 'stale') return g.sources.some((s) => !s.last_checked_at)
return true
}
function pickLastActivity(sources) {
let max = null
for (const s of sources) {
if (s.last_checked_at && (!max || s.last_checked_at > max)) max = s.last_checked_at
}
return max
}
function pickWorstSource(sources, threshold) {
if (!sources || sources.length === 0) return null
function level(s) {
if (!s.last_checked_at) return 0
const f = s.consecutive_failures || 0
if (f === 0) return 1
if (f < threshold) return 2
return 3
}
return sources.reduce((worst, s) => (level(s) > level(worst) ? s : worst), sources[0])
}
function onRowClick(_evt, { item }) {
const key = item.key
const idx = expanded.value.indexOf(key)
if (idx === -1) expanded.value = [...expanded.value, key]
else expanded.value = expanded.value.filter((k) => k !== key)
}
function openAddSource(artist) {
editingSource.value = null
editingArtist.value = artist
showSourceDialog.value = true
}
function openEditSource(source) {
editingSource.value = source
editingArtist.value = {
id: source.artist_id, name: source.artist_name, slug: source.artist_slug,
}
showSourceDialog.value = true
}
async function removeSource(source) {
await store.remove(source.id, source.artist_id)
await refresh()
}
async function toggleSourceEnabled({ source, enabled }) {
await store.update(source.id, { enabled }, source.artist_id)
await refresh()
}
async function onSourceSaved() {
showSourceDialog.value = false
await refresh()
}
function onArtistCreated(artist) {
showArtistDialog.value = false
openAddSource(artist)
}
async function onCheck(source) {
try {
const body = await store.checkNow(source.id)
toast({
text: `Check enqueued (event #${body.download_event_id})`,
type: 'success',
})
} catch (e) {
if (e?.body?.download_event_id) {
toast({
text: 'Already running — see Downloads',
type: 'info',
})
router.push({ path: '/subscriptions', query: { tab: 'downloads', source_id: source.id } })
} else {
toast({
text: `Check failed: ${e?.detail || e?.message || e}`,
type: 'error',
})
}
}
}
// Plan #544: arm a source for backfill mode (gallery-dl walks the full
// post history) for the next N download runs. Default 3 — enough budget
// to finish a deep creator without re-prompting the operator across
// timeout boundaries. The chip on the row reflects the remaining count.
async function onBackfill(source) {
const raw = globalThis.window?.prompt(
`Deep scan "${source.artist_name} (${source.platform})" — walk full history for the next how many download runs? (110, default 3)`,
'3',
)
if (raw == null) return
const runs = parseInt(raw, 10)
if (!Number.isFinite(runs) || runs < 1 || runs > 10) {
toast({ text: 'Deep scan: runs must be 110', type: 'error' })
return
}
try {
await store.setBackfill(source.id, runs, source.artist_id)
toast({
text: `Deep scan armed for ${runs} run${runs === 1 ? '' : 's'}`,
type: 'success',
})
await store.loadAll()
} catch (e) {
toast({
text: `Deep scan failed: ${e?.detail || e?.message || e}`,
type: 'error',
})
}
}
async function checkAll(group) {
let ok = 0
let conflict = 0
for (const s of group.sources) {
if (!s.enabled) continue
try {
await store.checkNow(s.id)
ok += 1
} catch (e) {
if (e?.body?.download_event_id) conflict += 1
}
}
const parts = []
if (ok) parts.push(`${ok} queued`)
if (conflict) parts.push(`${conflict} already running`)
toast({
text: parts.join(', ') || 'Nothing to check (no enabled sources)',
type: 'info',
})
}
function anyChecking(sources) {
return sources.some((s) => store.checkingIds.has(s.id))
}
function resolveSelectedGroups() {
return groups.value.filter((g) => selected.value.includes(g.key))
}
async function bulkSetEnabled(enabled) {
const groups = resolveSelectedGroups()
let changed = 0
for (const g of groups) {
for (const s of g.sources) {
if (s.enabled === enabled) continue
try {
await store.update(s.id, { enabled }, s.artist_id)
changed += 1
} catch { /* keep going */ }
}
}
await refresh()
toast({
text: `${changed} source${changed === 1 ? '' : 's'} ${enabled ? 'enabled' : 'disabled'}`,
type: 'success',
})
selected.value = []
}
async function bulkDelete() {
const groups = resolveSelectedGroups()
const total = groups.reduce((n, g) => n + g.sources.length, 0)
if (!globalThis.window?.confirm(
`Delete ${total} source${total === 1 ? '' : 's'} across ${groups.length} subscription${groups.length === 1 ? '' : 's'}? Artist rows remain.`,
)) return
let deleted = 0
for (const g of groups) {
for (const s of g.sources) {
try {
await store.remove(s.id, s.artist_id)
deleted += 1
} catch { /* keep going */ }
}
}
await refresh()
toast({
text: `${deleted} source${deleted === 1 ? '' : 's'} deleted`,
type: 'success',
})
selected.value = []
}
</script>
<style scoped>
.fc-subs__sched { margin-bottom: 10px; }
.fc-subs__bar {
display: flex; gap: 0.75rem; align-items: center;
flex-wrap: wrap;
/* Sticky to the top of the hub's scroll window so the filter/search
controls stay reachable while scrolling a long subscription list.
Opaque bg so table rows don't bleed through. Operator-flagged
2026-05-28. */
position: sticky;
top: 0;
z-index: 3;
background: rgb(var(--v-theme-background));
padding: 8px 0 1rem;
}
.fc-subs__bar .v-chip { cursor: pointer; }
.fc-subs__loading, .fc-subs__empty {
display: flex; justify-content: center; padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-subs__card {
background: rgb(var(--v-theme-surface));
}
.fc-subs__name { font-weight: 600; }
.fc-subs__chips {
display: flex; flex-wrap: wrap; gap: 4px;
}
.fc-subs__when {
color: rgb(var(--v-theme-on-surface-variant));
white-space: nowrap;
font-variant-numeric: tabular-nums;
}
.fc-subs__zero {
color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.6;
}
.fc-subs__sources-row td {
padding: 0 !important;
background: rgb(var(--v-theme-surface-light));
}
.fc-subs__sources-cell {
padding-left: 2rem !important;
border-top: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.15);
}
.fc-subs__sources-table {
background: transparent !important;
}
.fc-subs__sources-empty {
color: rgb(var(--v-theme-on-surface-variant));
text-align: center;
padding: 1rem;
}
.fc-subs__bulk { border-radius: 8px; }
</style>