b7d07324eee4235cbcea219701fd3c602090b2a5
15 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
b7d07324ee |
fix(subs): stop the lock/reload on source actions + regroup the row buttons
Lock/reload: every inline source action (check / backfill / recover / toggle / remove) ended by refetching the WHOLE subscription list (store.loadAll / refresh), which blocked the UI and re-rendered the table — collapsing the expanded row, which read as "locks then resets." The action APIs already return the updated source, so the store now patches that one row in place (_patchSource / _dropSource); the post-action loadAll/refresh calls are gone. Toggling enabled, starting/stopping a backfill, recovering, and removing are now instant and leave the expansion intact. Button regroup (operator-flagged: tiny, mis-clickable, not grouped by function): - New shared SourceActions.vue used by desktop SourceRow + mobile SourceCard. - Frequent actions stay as size="small" buttons: Check, Backfill/Stop. - Low-frequency / destructive actions move into a labelled overflow (⋮) menu — Preview backfill, Recover dropped near-duplicates, Remove source — so they can't be fat-fingered, and the labels spell out recover-vs-backfill. - Edit moves next to the source URL (its identity), out of the action cluster where it sat beside Remove. - Single-source Remove now confirms (it had no guard before). Tests: store patches/drops in place without dropping the cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e82c2ee57b |
feat(subscriptions): dry-run backfill preview — B4 preview (plan #708)
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
5b615b7ded |
feat(sources): pre-flight credential verify on backfill/recovery arm — #703 step 2
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
ec43e823e1 |
feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
618dafde85 |
feat(subscriptions): smarter-backfill UI — Start/Stop + state badge
Plan #693 (frontend). Backend landed in
|
||
|
bvandeusen
|
928e3037f0 |
fix(ui): purpose-built mobile layout for subscriptions hub
Vuetify's auto card-stack was too verbose (one subscription filled the whole phone screen) and the expanded sources still needed lateral scroll. Replace it below 600px (useDisplay) with a custom compact-card list: each subscription is a 2-line card (name + health + expand chevron, then platform chips + sources count + last activity) so several fit per screen. Expanding shows the action row + each source as a STACKED SourceCard (new) — platform/url/enabled/last/ next/errors/actions laid out vertically, no horizontal scroll. The mobile cards drive the same selected/expanded key arrays as the desktop data-table, so selection and bulk actions are unchanged. Desktop keeps the v-data-table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b08b12eb8f |
fix(ui): subscriptions table → card layout on mobile
The subscriptions v-data-table (select + expand + 6 cols + a nested 8-col sources sub-table) horizontally-scrolled on phones. Set mobile-breakpoint=600 so Vuetify stacks each subscription row into a label:value card below 600px; the custom item slots (platform chips, health dot, action buttons) render as card rows. The expanded sources detail reclaims its desktop indent on mobile and keeps its own horizontal scroll for the wide source columns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4fd6d4cc29 |
fix(ui): mobile pass 2 — Posts & Subscriptions filter bars
PostsFilterBar didn't wrap and its artist/platform fields had inline min-widths (240/180px) a media query can't override → horizontal overflow on phones. Moved widths to classes, added flex-wrap, and <600px each field takes a full-width row. SubscriptionsTab's status/search inline max-widths likewise moved to classes; <600px they go full-width and the v-spacer is dropped so the search isn't shoved around. Verified as already-fine (sweep false positives, no change): PostCard (default body is a stacked column; only goes row at container >=800px), SeriesReaderView (already has a <=768px block: nav drawer 150px, quick-nav stacks). The subscriptions v-data-table scrolls horizontally within its own wrapper, so it doesn't widen the page — a true mobile card layout is a larger follow-up if wanted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4bff1d8558 |
fix(audit-g4): status-enum miss batch
Five extension-miss findings from the 2026-06-02 audit, where a status
value was added on one side but a downstream consumer didn't pick it up.
- download_service._phase3_persist: explicit branches for
ImportResult.status in ('failed','refreshed'). For 'failed' (archive
probe crash from _import_archive), unlink the source file so the
filesystem scanner doesn't re-import and re-crash on the same
archive forever. 'refreshed' is currently unreachable from the
download path (no deep=True) but matches the importer's documented
contract; treat as 'attached'.
- gallery-dl backfill auto-complete now gates on dl_result.success +
no error_type, not just return_code==0 + files_downloaded==0.
VALIDATION_FAILED exits the subprocess with returncode=0 and
files_downloaded=0 when every file was quarantined, matching the
prior predicate exactly and zeroing the operator's armed backfill
budget on the FIRST quarantine run instead of decrementing.
- attach_in_place archive dispatch now threads artist + source_row
through _import_archive (and _import_media for archive members)
and _supersede. The path-walk fallback (_resolve_artist) is still
used by filesystem-import; the download path now binds
ImageProvenance to the explicit subscription Source instead of
rediscovering by (artist_id, platform).
- Three FE handlers now recognize status:'deferred' from
/api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting
"event #undefined"), SubscriptionsTab.checkAll (was counting
deferred as queued), DownloadEventRow.onRetry (was saying
"re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll
which already had it.
- celery_signals._queue_for now maps backup/admin/library_audit
prefixes to 'maintenance' (matching task_routes). TaskRun.queue
was returning 'default' for those rows, so per-queue dashboard
filters and per-queue threshold overrides (added in G3) silently
missed them.
|
||
|
bvandeusen
|
19aece1fc4 |
feat(download): tick/backfill modes + partial-success classifier (plan #544)
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
|
||
|
bvandeusen
|
42ddac9996 |
fix(subscriptions): fixed-height hub so only the tab content scrolls
The whole view scrolled instead of just the subscription list. Made the hub a viewport-height flex column (tabs stay fixed) with the v-window as the single internal scroll container; the per-tab sticky control bars now pin to the window top (top:48px -> 0). Operator-flagged 2026-05-28. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
eebc8e2413 |
refactor(dry-F2): centralize shared UI primitives (relative-time, toast, download-status)
- utils/date.js: add formatRelative(iso, {future,nullText}); migrate 6 sites
(SourceRow, SubscriptionsTab, SourceHealthDot, SchedulerStatusBar +
thin adapters in BackupRunsTable/SystemActivityTab for their '—' null text).
PostCard (30d->absolute) and CredentialCard (mo/y buckets) intentionally
keep bespoke formatters.
- utils/toast.js: toast(opts) wraps the globalThis.window?.__fcToast?.(...)
incantation; migrate 63 call sites across 24 files.
- utils/downloadStatus.js: single source for the download-event status enum
-> label/color/icon; collapse the 3 duplicate maps (DownloadStatChips,
DownloadsFilterPopover, DownloadEventRow).
Net -33 lines. Platform metadata was already centralized in platformColor.js.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
2358cedf3e |
feat(dashboards): scheduler health strip, failing-source rollup, 24h activity sparkline, credential staleness nudge
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick + GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources) + SchedulerStatusBar on the Subscriptions tab (re-polled every 30s). D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard on Downloads with per-source and bulk "retry" (re-runs the feed via /check). D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar chart by the stat chips (failures stacked in error color); refreshes on live poll. D4 credential staleness: surface last_verified age + "re-verify recommended" warning past 30d; also fixes the dead last_verified_at field-name mismatch so the verification row renders at all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a459d21a65 |
feat(dashboards): 1600px max-width, richer Downloads filters, needs-attention + sticky headers
Operator-flagged 2026-05-28: the Subscriptions/Downloads dashboards were full-bleed and thin on filtering. Chosen via AskUserQuestion. **Layout** - SubscriptionsView capped at a centered max-width 1600px (covers all three subtabs) so rows aren't a mile wide on ultrawide monitors. - Sticky control headers on both tabs (top: 48px, below the top nav, opaque bg) so filters/stat-chips stay reachable while scrolling a long list. **Downloads filters (all four requested)** - Stat chips are now clickable filters: click Queued/Running/Completed/ Failed/Skipped to filter the list to that status; the active chip is outlined + elevated; re-click clears. - Free-text search box over the loaded events (artist / platform / error substring). - Artist filter: the filter popover's numeric "Source ID" field is replaced with an artist autocomplete (sources.autocompleteArtist → artist_id, which /api/downloads already supports). Pill shows the artist name. - "Show no-change scans" toggle (default OFF): hides status=ok/skipped rows with 0 files (the scheduled scans that found nothing) so real downloads + failures stand out. **Subscriptions** - "Needs attention" quick-filter chip: one click to show only artists with sources that have errors OR have never been checked; chip shows the count and disables the status dropdown while active. Frontend-only — backend filter params (status/artist_id/date) and the /api/downloads endpoint already supported everything. |
||
|
bvandeusen
|
9322c984fd |
feat(subs-hub): collapse /credentials + /downloads into /subscriptions hub with three GS-style subtabs
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.
**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.
**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.
**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.
**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.
**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.
Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
|