3bb94674cfe0cc60b5aa3c0ca6dc85cd2475f824
106 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
a3c9499e93 |
feat(subs): kick off the first backfill walk immediately on source create
A new enabled source is armed for run-until-done backfill (#693) but would sit idle until the next scheduler tick (~60s). create_source now enqueues the first walk right away (pending DownloadEvent + download_source.delay), skipping only when the platform is in a rate-limit cooldown (the scheduler picks it up when that clears). Disabled sources still don't dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3c89223dcb |
feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714)
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps whatever casing it was created with. This adds a maintenance action that Title-Cases every existing tag (collapsing whitespace) and merges case/whitespace-variant duplicates into one. Backend: - tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor (prefer an already-canonical member → no rename/self-alias; else the best-connected tag → fewest FK repoints; else lowest id), merges the variants INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/ aliases/series_page repoints + protective ML aliases), then renames the survivor to canonical. Losers are deleted before the rename so there's no transient unique-index clash; commits per group and isolates failures per group. Idempotent — an already-canonical lone tag is a no-op. - normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i. - POST /api/admin/tags/normalize: dry_run=true returns a projection inline (group/collision/rename counts + sample); dry_run=false enqueues the task. Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup tab) — preview → apply (polls the activity dashboard to terminal status), behind a back-up-first warning. admin store gains normalizeTags(). Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept separate, ML-known loser keeps a protective alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
23f452021f |
fix(tags): Title-Case operator-entered tags at create endpoint only (plan #701)
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in the POST /api/tags handler so operator-entered tags get clean display casing. It is NOT applied in the shared find_or_create / rename paths — those are used by the ML tagger and allowlist matching, which must preserve the booru vocabulary's original casing (Title-Casing it broke apply_allowlist matching). find_or_create / rename keep case-insensitive lookup + clash detection so a differently-cased entry dedups onto the existing tag instead of forking. Tests updated to expect Title-Cased create output (sunset→Sunset, character:Saber→Character:saber, http://example.com→Http://example.com) and a dedicated normalize_tag_name unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
62cca64dce |
feat(downloads): live per-file progress on running events — #709
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
a497104661 |
feat(maintenance): re-extract archive attachments + link to post — #713 part 2
Existing PostAttachments that are actually archives (filed opaquely before the magic-byte gate) need extracting retroactively. cleanup_service. reextract_archive_attachments scans PostAttachments, magic-detects the archives, and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place in a temp dir — so the members extract and re-link to the SAME post via find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by sha256). Enqueues thumbnail+ML for new members. Wired as a maintenance-queue Celery task (tasks/admin) + POST /api/admin/maintenance/reextract-archives (202) + a "Re-extract archive attachments" card in Settings → Maintenance. Test: a zip stored under a mangled extension-less name extracts + links its member to the post via ImageProvenance, and a second run is a no-op (idempotent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e82c2ee57b |
feat(subscriptions): dry-run backfill preview — B4 preview (plan #708)
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
5b615b7ded |
feat(sources): pre-flight credential verify on backfill/recovery arm — #703 step 2
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
218bfebb92 |
feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)
The credential Verify button still ran gallery-dl --simulate for Patreon after the cutover — testing the wrong path (and prone to the vanity "Failed to extract campaign ID" the native resolver fixes). Wire it to the native ingester, behind a DRY dispatch so callers never branch on platform. - services/download_backends.py (new): the ONE place that knows which platforms are native vs gallery-dl. `uses_native_ingester(platform)` is the shared predicate; `verify_source_credential(...)` is the uniform probe (same (ok|None, message) contract for both backends). As a platform migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download routing and verify switch together. - PatreonClient.verify_auth(campaign_id): one authenticated /api/posts fetch → True (valid) / False (401/403/HTML-login) / None (drift or network — inconclusive, not a credential verdict). - patreon_ingester.verify_patreon_credential(): resolve campaign id, then verify_auth — the verify counterpart to the download path. - patreon_resolver.resolve_campaign_id_for_source(): extracted the override / id:-URL / vanity resolution into ONE helper now shared by the download ingester and verify (download_service no longer carries its own copy + regex; −`import re`). - download_service: routes on uses_native_ingester() instead of inline `== "patreon"` (3 sites); uses the shared resolver. - api/credentials: calls verify_source_credential — no platform branch. Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/ vanity/none), the dispatch predicate, verify_patreon_credential glue, credentials endpoint proves Patreon uses the native path (gallery-dl verify asserted not-called); repointed the gallery-dl verify test to subscribestar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ec43e823e1 |
feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
96fffaff64 |
feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5a6a95682d |
fix(cleanup): library scans survive navigation, reconnect on return
The transparency / single-color audit cards held the run + poll timer in local component state, so navigating away destroyed both and onMounted never reconnected — the Celery scan kept running and writing LibraryAuditRun, but the UI forgot it. Now each card, on mount, fetches its rule's latest run (GET /api/cleanup/audit?rule=<rule>&limit=1) and rehydrates: shows progress + resumes polling if still running, or shows the completed result (ready/applied/ error) so the operator can act on it after returning. Adds the ?rule= filter to the audit-history endpoint + cleanup store latestAuditForRule(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
91b0145bc8 |
feat(tags): 'Reset content tagging' admin action
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
79cd1234e2 |
feat(gallery): visual 'more like this' search (Phase 3 backend)
GalleryService.similar() ranks images by pgvector cosine distance to a source image's precomputed SigLIP embedding — no query-time ML inference. Composes with the Phase-1/2 scope filters (AND) but replaces the date sort (always nearest-first, bounded top-N, no cursor). Returns None for a missing source (→404), [] for a source with no embedding (video / pending ML); excludes self and NULL-embedding rows. New GET /api/gallery/similar?similar_to=<id>&limit=N. Image-detail payload gains has_embedding so the UI can hide the surface. Alembic 0036 adds an HNSW vector_cosine_ops index on siglip_embedding (1152<2000 dims) so the search is sub-50ms ANN instead of a full scan; one-time ~30-60s build over existing embeddings on deploy. Shared _gallery_images/_image_json helpers de-dup the scroll/similar builders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9fe534139a |
feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)
Extend the composable gallery filter with platform / untagged / no_artist / date_from / date_to, AND-composed with the existing tag/artist/media/sort params and threaded through scroll, timeline, and jump_cursor. Add GalleryService.facets() + GET /api/gallery/facets returning live counts scoped to the current filter with per-group minus-self semantics: platform counts (COUNT(DISTINCT image) incl. a null unsourced bucket), curation-flag counts (untagged / no_artist), and effective_date min/max bounds. The UNSOURCED_PLATFORM sentinel makes filesystem-imported content reachable via the platform facet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
914033db29 |
feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the periodic prune/backfill/recovery tasks churn dead tuples faster than autovacuum always keeps up — so explicit maintenance earns its keep here. - tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn). Scheduled weekly via Beat; also operator-triggerable. - _sync_engine.get_sync_engine(): expose the process engine for the autocommit connection. - GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct + last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a black box. - POST /api/admin/maintenance/vacuum: enqueue the task on demand. Tests: vacuum task runs + reports tables; db-stats shape; trigger queues. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3f30327fa5 |
feat(gallery): composable scroll filter (multi-tag AND, media, sort)
Phase 1 backend for the gallery filter bar. Extends scroll/timeline/jump
from a single mutually-exclusive filter to a composable one:
- tag_ids: image must carry ALL of them (one correlated EXISTS per tag —
AND, no row multiplication), replacing the single-tag JOIN.
- artist_id composes with tags; media_type ('image'|'video') narrows by
mime; post_id stays the exclusive post-detail path.
- sort ('newest'|'oldest') flips the effective_date/id cursor comparison
and ordering; the cursor value is unchanged (direction comes from the
request). jump_cursor honors sort too.
- Shared _apply_scope helper applied across scroll/timeline/jump so the
timeline sidebar reflects the filtered set. API _parse_filters parses
tag_id (comma list), artist_id, media, sort.
Tests: multi-tag AND, media filter, sort reversal (service + API);
post_id-excludes-others; single tag_id back-compat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
4f9464d215 |
feat(gallery,tags): clear active filters
Two gaps where a filter couldn't be removed:
- Gallery: a tag_id filter (from clicking a tag) had no indicator or clear
control — only post_id did (PostInfoHeader). Add an "Tag: <name> ✕" chip
that clears the filter by dropping tag_id from the URL. New lightweight
GET /api/tags/<id> resolves the name; the store fetches it on filter set.
- Tags view: the kind chip-group used mandatory="false" — a STRING ("false"
is truthy in JS), which made the group mandatory so the active kind chip
couldn't be deselected. Fixed to :mandatory="false" so the filter clears.
Tests: GET /tags/<id> shape + 404; gallery store resolves filterTagName.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
d9ab6e15c6 |
feat(tags): edit a character tag's fandom (backend)
No way existed to change which fandom a character tag belongs to after creation — PATCH /tags/<id> only renamed. - TagService.set_fandom(tag_id, fandom_id, merge=False): set / change / clear (fandom_id=None) a character's fandom, with the same validation as find_or_create. On a name collision in the target fandom it raises TagMergeConflict (→ 409, same shape as rename); merge=True resolves it by merging this tag INTO the existing character. - Extract _do_merge(source, target) from merge() so set_fandom can perform the deliberate CROSS-fandom merge the public merge() validation forbids. - PATCH /tags/<id> now accepts optional fandom_id (+ merge flag) alongside name, and returns fandom_id. Tests: set/change/clear, non-character + bad-ref rejection, collision raises, merge resolves; API set/clear + collision→merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
98673d4dca |
fix(audit-g5a): small architectural cleanups bundle
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.
- download_service: replace hardcoded ('discord','pixiv') tuple with
auth_type_for(platform) == 'token'. A 7th token-platform now picks
up the right credential path without touching this site.
- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
merge so the target's centroid reflects its new image set
immediately. Daily list_drifted catches it within 24h, but eager
recompute closes the suggestion-quality dip in the meantime.
- backfill_thumbnails added to beat_schedule (daily). The task
docstring claimed periodic Beat but the entry was never registered,
so the library got no self-healing thumbnail repair; only the
manual admin-UI button fired it.
- modal.createAndAdd pushes a kind='fandom' tag into
tagsStore.fandomCache so FandomPicker sees the new fandom on next
open. Was: cache-gated load (length===0) skipped refetch, new
fandom invisible until full page reload.
- cleanup cluster:
- Drop .webp from cleanup_service.unlink — thumbnailer only writes
.jpg/.png; the third tuple member was dead code.
- Drop effective_date from /api/gallery/scroll response — no FE
consumer reads it. Service still computes the attribute for
timeline ordering; this just trims the JSON.
- Rename store.recentMinute → store.recentRuns across the
systemActivity store + three consumers (SystemActivitySummary,
QueuesTable, SystemActivityTab). The data is the last 200 runs
(not actually "last minute"), so the name lied.
NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
|
||
|
bvandeusen
|
e30f50e6fe |
fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days). |
||
|
bvandeusen
|
9cbdb70e13 |
fix(thumbnails): surface backfill results + tighten validity check
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."
1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
returned just `{celery_task_id}` and the admin card said
"Enqueued." with no counts. There was no way to tell whether the
scan found 0 candidates, 5000 candidates, or whether the worker
even picked up the task. "Found nothing" was indistinguishable
from a broken queue.
Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
shared by the Celery task and the API endpoint. The API now runs
the scan in an executor and returns `{scanned, enqueued, ok,
regenerated}`. The actual thumbnail generation work still goes
to the thumbnail Celery queue per row via
`generate_thumbnail.delay()` — the scan itself is fast
(SELECT id+thumbnail_path + a file.stat() per row).
2. **`_thumb_is_valid` accepted header-only corrupt files.** The
magic-byte check passed for any 8-byte file starting with a JPEG
or PNG header, including empty/truncated/zero-pad files that
browsers render as broken. Backfill counted these as `ok` and
never regenerated.
Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
thumbnails are minimum ~2KB even on solid-color sources; header-
only corrupt files top out around 12 bytes. 256 is well above
the corrupt floor and well below any legitimate thumbnail.
Plus the admin card now shows the per-run counts instead of
"Enqueued.":
Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
|
||
|
bvandeusen
|
19aece1fc4 |
feat(download): tick/backfill modes + partial-success classifier (plan #544)
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
|
||
|
bvandeusen
|
af7b5c95e9 |
feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
Four coupled operator-asked changes to the view modal (Scribe plan #509): 1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus in onMounted/nextTick so the caret is in the input the moment the modal renders. No click needed to start typing. 2. **General suggestions expanded by default** — SuggestionsPanel's general-category group now mounts with `:default-open="true"`. Operator can collapse if too noisy, but the v1 frame shows them. 3. **Lower general threshold default 0.95 → 0.50** — MLSettings. suggestion_threshold_general default matches character. Alembic 0029 also bumps the existing singleton row's value if it's still at the old 0.95. Operator can re-tune from Settings → ML. 4. **Retire `copyright` + `artist` as ML suggestion categories** — neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never really existed as a copyright tag-kind). They were surfaced in the suggestions pipeline + threshold settings UI but had no follow- through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for, ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029 drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS, SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and MLThresholdSliders rows. Out of scope (intentional): `tag_kind` Postgres enum still includes `artist` for historic Tag row queryability (per the model comment); no operator pain reported, no enum-shrink needed. Tests: - test_surfaced_categories asserts {character, general}, excludes artist + copyright. - test_threshold_for_artist_is_unsurfaced extended to cover copyright. - test_get_and_patch_settings asserts new 0.50 default and the absent artist + copyright keys in the GET payload. |
||
|
bvandeusen
|
d65f0b2091 |
feat(extension): probe shows current state before click; v1.0.6
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 20s
extension / lint (push) Successful in 13s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m41s
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup): "the add source/subscription button idea to the firefox extension so it can tell me if a source/artist is added or not and offer an option to add it if it isn't." Plan tracked in Scribe task #507. ## Backend - `ExtensionService.probe(url)` — read-only resolution. Reuses `_derive` for platform+slug, then 2 SELECTs. Returns one of: - `source_match` (exact (artist, platform, url) Source exists) - `artist_match` (artist exists, this URL isn't a Source yet; collapses the sidecar-synthetic-only case from v26.06.01.0) - `new` (neither exists) - `unknown_platform` (URL didn't match any artist-page regex) - `GET /api/extension/probe?url=...` route with `X-Extension-Key` auth posture matching `/quick-add-source`. Read-only, side-effect free. - 6 backend tests in tests/test_api_extension.py covering each state + auth + invalid URL. ## Extension - `api.js`: `probeSource(url)` mirroring `quickAddSource` shape. - `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The latter strips the `/api` suffix from configured `apiUrl` (placeholder format per options.html) and opens `${base}/artist/{slug}` in a new tab via `browser.tabs.create`. - `content-script.js`: probe-first render — on page-load and SPA navigation, asks the backend for the URL's state and renders the chip in the matching color/copy on FIRST paint instead of flashing generic "Add" and updating after. Click handler branches: `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing ADD_AS_SOURCE flow (then re-probes so the chip flips green immediately, no wait for next nav). - `content-script.css`: three state-color modifiers (--new, --artist-match, --source-match) on the FC parchment-on-slate palette. Sage for already-added, amber for artist-exists, accent orange for new. ## Versioning - `extension/manifest.json` + `extension/package.json` → 1.0.6. build.yml's sign-extension job will fire on push to main since no `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the regenerated AMO keys end-to-end. ## Behavior on the sidecar-synthetic case Filesystem-imported "Dymkens"-style artist with only a sidecar synthetic Source: probe returns `artist_match` (not `new`), so the chip reads "+ Add Patreon source to Dymkens" rather than offering to recreate the artist. Clicking adds the real Source; existing `_source_for_sidecar` preference logic (v26.06.01.0) routes future gallery-dl Posts to the real one. |
||
|
bvandeusen
|
a5101494b6 |
feat(downloads): bulk retry respects cooldown; single-source RETRY overrides
Today's platform-cooldown commit (
|
||
|
bvandeusen
|
77e9859da3 |
fix(downloads): rewire MaintenanceMenu to the downloads pipeline
The maintenance dropdown in Subscriptions → Downloads was wired to the
filesystem-import pipeline (POST /api/import/retry-failed +
POST /api/import/clear-stuck) — the subtitles even said so ("Re-enqueue
every failed import task"), but it was contextually misplaced. From the
Downloads view "Retry failed" queued nothing the operator could see
because the action operated on import_task rows, not download_event
rows. Import-pipeline maintenance is already reachable from Settings →
Imports (ImportTaskList.vue), so removing the import wiring loses
nothing.
Rewired:
- "Retry failed" → bulk-retries the failing-sources list, same loop as
FailingSourcesCard's RETRY ALL (sourcesStore.checkNow per source).
Subtitle now matches: "Re-queue every currently failing source".
- "Force recovery sweep" → triggers recover_stalled_download_events on
demand via a new POST /api/downloads/recover-stalled endpoint. The
sweep also runs every 5 min on Beat; this is the manual fallback so
the operator doesn't have to wait for the next tick to clear newly
stranded events.
MaintenanceMenu is now stateless — emits retry-failed and recover-
stalled. DownloadsTab owns the handlers (reuses the existing
onRetryAll; new onRecoverStalled with a delayed refresh so swept rows
land in the failing rollup).
Operator-flagged 2026-05-29 — "the retry failed button in the
maintenance dropdown doesn't appear to queue anything but manual
requeues works."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
8649a13118 |
refactor(I5): remove one-and-done GS/IR migration tooling
The GS/IR migration cutover is complete, so the runbook tooling is dead weight. Removed: - services/migrators/ (gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup), tasks/migration.py, api/migrate.py (+ blueprint registration) - MigrationRun model; alembic 0027 drops the migration_run table - frontend LegacyMigrationCard + migration store (+ MaintenancePanel ref) - celery include + task route + celery_signals queue mapping for migration.* - the 1 GB MAX_CONTENT_LENGTH / MAX_FORM_MEMORY override (added solely for the ir_ingest upload) - migration-surface tests (test_api_migrate, test_migration_verify, test_ir_ingest, test_gs_ingest, test_tag_apply) Kept: the alembic schema-migration tests (test_migration_00XX — unrelated) and cleanup_service.py (the permanent artist-cascade/unlink home). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
00e2608ba1 |
feat(I3): always-on pipeline status indicator in the top nav
New /api/system/activity/summary aggregates scheduler health + per-queue pending depths + running count + 24h failure count in one cached call (safe to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a compact running/queued/failing chip with a scheduler-health dot that expands to a popover (scheduler, busy queues, counts, link to downloads). Polls the summary every 8s, paused when backgrounded. Reuses the queue-cache read via _queues_cached(). + API test for the summary shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c95b760294 |
feat(posts): in-context anchored feed with bidirectional infinite scroll
Provenance "View post" deep-links to /posts?post_id=X, which now opens the feed centered on that post with infinite load in BOTH directions. Backend: PostFeedService.scroll gains a direction (older|newer); new around(post_id) returns a window of newer + the post + older with a cursor for each end. /api/posts accepts ?around= and ?direction=. + API tests. Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends, newer prepends) with per-end cursors; PostsView's anchored mode scrolls to the post, observes top + bottom sentinels, and preserves scroll position on upward prepend so the page doesn't jump. Normal feed mode unchanged. Closes the remaining half of the post-navigation work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
171c486939 |
refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was repeated 15× across services, API, and 5 task modules. Added async load() + sync load_sync() classmethods on the model and migrated all 15 full-row sites (callers already imported ImportSettings, so no new imports; dropped download's now-orphaned select import). Left maintenance.py's deliberate column-select (import_scan_path only) as-is. Rest of the service layer was already adequately DRY — the Record/to_dict pattern is only 2 instances and the savepoint find-or-create recovery is correctly per-entity, so neither was forced into a shared abstraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
597c6d48d3 |
refactor(dry-B1): consolidate duplicated _bad error helper into api/_responses.py
8 blueprints each defined an identical _bad() (two variants: with/without detail). Extracted error_response() into api/_responses.py; each blueprint now imports it `as _bad` so call sites are unchanged. The detail-aware canonical subsumes both variants. Left settings.py's distinct _bad_int and the inline jsonify error sites (not duplicated helpers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2358cedf3e |
feat(dashboards): scheduler health strip, failing-source rollup, 24h activity sparkline, credential staleness nudge
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick + GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources) + SchedulerStatusBar on the Subscriptions tab (re-polled every 30s). D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard on Downloads with per-source and bulk "retry" (re-runs the feed via /check). D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar chart by the stat chips (failures stacked in error color); refreshes on live poll. D4 credential staleness: surface last_verified age + "re-verify recommended" warning past 30d; also fixes the dead last_verified_at field-name mismatch so the verification row renders at all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
56970fb66d |
feat(credentials+downloads): real credential Verify button + live download-activity polling
Operator-flagged 2026-05-28, two asks.
**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):
- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
gallery-dl in `--simulate --range 1-1` mode (no download) against the
URL with the materialized credentials, then reuses _categorize_error:
returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
the platform to probe, runs verify, and on success stamps
credential.last_verified (new CredentialService.mark_verified).
Returns {valid: bool|null, reason, last_verified?}. valid=null means
untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
result chip (Verified ✓ / Failed / Untestable) and a toast with the
reason. SettingsTab reloads on @verified so last_verified refreshes.
**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.
Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
|
||
|
bvandeusen
|
bf8eb4468f |
feat(tags): legacy-tag purge also catches source:* general tags
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
|
||
|
bvandeusen
|
e1fc65bd1b |
feat(provenance+tags): collapse provenance descriptions; purge retired-kind tags
Operator-flagged 2026-05-28, two asks. **1. Provenance posts ate the panel.** Each post card rendered its full description (180px scroll box) inline, so a few posts pushed everything else off-screen. ProvenancePanel now collapses the description by default behind a per-post "Show description ▾ / Hide ▴" toggle (state keyed by provenance_id, reset when the viewed image changes). Cards stay compact — platform/date/title/meta/actions — and the operator expands only the descriptions they want. **2. Purge tags of retired/system kinds.** The IR migration left `archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`) that FC no longer creates — the tag input only makes character/fandom/series/general, and provenance + artists are their own systems now. (meta/rating were already hard-deleted by alembic 0023.) - cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts (dry_run) or deletes tags whose kind ∈ (archive, post, artist). CASCADE clears the image_tag / alias / allowlist / etc. rows. - POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview returns per-kind counts + sample names — the preview IS the verification of exactly what'll be deleted before committing). - Tag Maintenance card gets a second section: "Preview retired-kind tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)". Tests: dry-run counts by kind (general survives), commit deletes only the retired kinds (general + character survive, retired count → 0). NOTE: a dry-run preview will show exactly which kinds/counts are present. If the operator's noisy tags turn out to be `general` (e.g. an IR `source:patreon` that fell back to general during migration), they won't be caught by the kind purge — the preview makes that visible so we can decide on a name-based pass separately. |
||
|
bvandeusen
|
dcfe55d731 |
feat(import-resilience L2): one-shot re-download for corrupt downloaded files
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its source, bounded to a single attempt. Operator-requested 2026-05-28. New backend/app/services/refetch_service.py: - resolve_refetch_source: parse the failed file's sidecar → platform, derive the artist from the import path, find an ENABLED Source with a real feed URL for (artist, platform). Returns None for filesystem-only imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic anchors (not pollable). - attempt_refetch: if not already refetched AND a Source resolves, delete the corrupt file (so gallery-dl's skip_existing re-fetches it), set ImportTask.refetched=True, and trigger ONE download_source re-check. Bounded by `refetched` so source-side corruption can't loop. Wiring: - Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed' tasks). Returns refetch_queued / no_source / already_refetched / not_found / not_failed. - Auto path in recover_interrupted_tasks: for each poison-pill row, if env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the manual button is the primary path; auto is opt-in since re-fetch deletes a file + re-runs the downloader). - Frontend: a cloud-refresh icon button on failed rows in ImportTaskList → stores.import.refetchTask → toast keyed on the result status. Filesystem imports with no upstream return no_source — the operator's only remediation there is replacing the file on disk, surfaced clearly in the toast. Tests: 404 unknown task, 400 non-failed task, no_source when unresolvable, and the full resolvable-source path (file deleted, refetched flag set, one download_source dispatched, second call is a no-op). The resolvable test repoints the migration-seeded import_settings(id=1) scan path rather than inserting a conflicting row. |
||
|
bvandeusen
|
a85880f965 |
fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct — the timeout covered the WHOLE archive, not per object. Importer._import_archive (importer.py:409) runs the full per-member pipeline (sha256 + pHash + dedup query + copy + provenance) for EVERY media member inline, all under import_media_file's single 300s soft limit. A single media file is sub-second; a multi-hundred-member archive blows the budget. They shared one task name and one timeout. **Split archive into its own task** - New `import_archive_file` task: same body as import_media_file (dispatch is by file-kind inside Importer.import_one) but soft=30min / hard=35min. Shared `_run_import_task` helper holds the flip-to-processing + resilience-contract wrapper; both tasks call it. - New `enqueue_import(task_id, task_type)` router — single source of truth for media-vs-archive dispatch. Used by all three enqueue sites: scan_directory, /api/import/retry-failed, recover_interrupted_tasks. - scan_directory now sets ImportTask.task_type = "archive" when is_archive(entry) (the model field already existed, anticipating this; scan was hardcoding "media"). - import_archive_file routes to the existing 'import' queue via the task_routes `import_file.*` wildcard — no worker config change. **Archive-aware recovery sweeps** Both sweeps would otherwise preempt a legitimately-running archive: - recover_interrupted_tasks (ImportTask 'processing' sweep): now task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the 35-min hard limit). Single UPDATE with an OR predicate over the two (task_type, cutoff) pairs; requeue routes via enqueue_import. - recover_stalled_task_runs (TaskRun 'running' sweep): now supports per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above the per-queue overrides added for ml. import_archive_file gets 40 min while the 'import' queue stays at the 5-min default for single-file imports. Precedence: task_name → queue → default, each pass excluding rows claimed by a higher-precedence pass so every row is touched once. **Tests** - test_import_archive_file_registered - test_recover_stalled_task_runs_archive_task_uses_longer_threshold — pins that a 10-min archive task-run survives, a 50-min one is flagged, and a same-queue 10-min media import is flagged at the default. - _make_task_run gains queue= + task_name= params. After deploy: archive imports get a 30-min budget and aren't preempted by either sweep; single-file imports keep their tight 5-min detection. |
||
|
bvandeusen
|
df6d89cb59 |
fix(secure-context): full audit — DestructiveConfirmModal.expectedTokenOverride + bulk-delete + min-dim use backend-computed tokens
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.
**Audit results across `frontend/src/`:**
crypto.subtle.digest — 2 sites:
- MinDimensionCard (fixed 2026-05-27)
- BulkEditorPanel (THIS FIX)
navigator.clipboard — 1 site, already guarded:
- utils/clipboard.js writeText with execCommand fallback
serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
cookieStore / queryLocalFonts / WebAuthn / geolocation
— NOT USED, nothing to fix
Extension scripts (background.js) use crypto.subtle but run from
moz-extension:// which IS a Secure Context — left as-is.
**BulkEditorPanel double bug**
The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:
1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
`delete-images-selection-<sha8>` while the backend expected
`delete-images-<sha8>`. The two would never match.
Fix:
- Backend `/api/admin/images/bulk-delete` dry-run response now returns
`confirm_token` (the canonical `delete-images-<sha8>` string).
Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
set, it bypasses the `${action}-${kind}-${runId}` formula and uses
the explicit string. This decouples the UI label (`kind`) from the
wire-format token (server-provided), so future endpoints can use a
kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
— no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
slicing the 8-char suffix off the backend's token and passing it
through `runId`; now passes the full backend token via
`expected-token-override` directly). Cleaner; one source of truth.
**Banked memory**
`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.
No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
|
||
|
bvandeusen
|
12be188ada |
feat(showcase): IR-parity R-key shuffle + stagger entry animation; fix(cleanup): min-dim Delete swallowed crypto.subtle TypeError on plain HTTP
**showcase R-key + entry animation**
Restores two behaviors lost during the FC-2 IR→Vue port. Operator-flagged
2026-05-27.
- ShowcaseView listens for keydown 'r'/'R' on window. Triggers
`store.shuffle()`. Skips when an input/textarea/contenteditable is
focused or a Vuetify overlay is open (the dialog/menu sets
`.v-overlay--active` on the body).
- MasonryGrid gains an opt-in `animateFromIndex` prop (default
`Number.POSITIVE_INFINITY` = off). When set, items with index ≥ the
threshold animate in with a stagger fade-in: 12px translateY,
0.25s ease, 60ms per item, capped by `prefers-reduced-motion`.
Stagger uses original-items-array index (resolved via an `idxById`
Map) so the reading order is preserved even after the masonry
distributes items across columns.
- ShowcaseView watches `store.images.length`: shrink-or-zero baseline
⇒ `animateFromIndex=0` (animate everything on initial load /
shuffle); grow ⇒ baseline=prevCount (animate only the appended
tail on infinite-scroll). Other MasonryGrid consumers (ArtistView's
Gallery tab) don't pass the prop, so they keep their current
no-animation behavior.
Direct port of IR's `app/static/js/showcase.js` keyboard handler +
`app/static/style.css` itemFadeIn keyframe.
**min-dim Delete: crypto.subtle TypeError fix**
The Delete button on the Cleanup → Minimum Dimensions card was
silently no-op'ing. Root cause: `crypto.subtle` is Secure-Context-gated
(undefined on plain-HTTP origins per the homelab posture). The card's
`onDeleteClick` computed the Tier-C confirm token via
`crypto.subtle.digest('SHA-256', ...)`, which threw TypeError before
`showModal.value = true`. The promise rejected, the click handler had
no `.catch`, the modal never opened — exactly the operator's reported
symptom.
Same shape as the v26.05.26.0 `navigator.clipboard` fix on the
ErrorDetailModal Copy button.
Fix: backend `/api/cleanup/min-dimension/preview` now returns
`confirm_token` (the canonical `delete-min-dim-<sha8>` string) in its
response. Frontend reads it from the preview response and feeds the
8-char suffix to DestructiveConfirmModal's `runId` prop — no
client-side crypto needed. Single source of truth.
Integration test `test_min_dimension_preview_returns_count` pinned to
also assert `body["confirm_token"]` matches the server-side compute.
|
||
|
bvandeusen
|
1fd54897d8 | fix(api): ruff UP017 — use datetime.UTC alias in /api/downloads/stats | ||
|
bvandeusen
|
9322c984fd |
feat(subs-hub): collapse /credentials + /downloads into /subscriptions hub with three GS-style subtabs
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.
**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.
**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.
**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.
**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.
**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.
Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
|
||
|
bvandeusen
|
8cdf0af0e1 |
feat(tags-api): IR-style kind:name parsing at POST /api/tags — when caller doesn't supply explicit kind, parse_kind_prefix runs on the name (artist:Eric → kind=artist, name='Eric'); explicit kind always wins for backward-compat; falls back to general when no recognized prefix is present. Updates the old "missing required" test that assumed kind was mandatory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b0bfbc585a |
fix(import-admin): retry-failed + clear-stuck — same UPDATE…WHERE pattern as the maintenance sweep, so neither endpoint can hit psycopg's 65535-parameter ceiling once accumulated row counts exceed ~65k
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
983da9e5b1 |
feat(thumb-backfill): /api/thumbnails/backfill endpoint — POST → 202 + celery_task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2505b197ae | feat(fc-cleanup): Pinia store + 3 cards + CleanupView + SettingsView tab + TagMaintenanceCard moved from Maintenance + ruff lint fixes — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> | ||
|
bvandeusen
|
0d0b236ac3 | feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> | ||
|
bvandeusen
|
a06ada4c9b | fix(ext-ui): direct :href install button (Firefox needs anchor click, not programmatic navigation) + manifest version detection ignores -latest.xpi alias — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> | ||
|
bvandeusen
|
e6ededbe8e | feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> | ||
|
bvandeusen
|
f657582f30 | feat(import-ui): deep scan button, sticky settings tabs, tasks-above-filters, fix Scanning-undefined source_path — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> | ||
|
bvandeusen
|
b6a917ac81 |
feat(import): /api/import/clear-stuck endpoint + Clear stuck UI button — escape hatch for the autoretry-loop case the automatic sweep can't break
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in
|