The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 blueprints each defined an identical _bad() (two variants: with/without
detail). Extracted error_response() into api/_responses.py; each blueprint
now imports it `as _bad` so call sites are unchanged. The detail-aware
canonical subsumes both variants. Left settings.py's distinct _bad_int and
the inline jsonify error sites (not duplicated helpers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick +
GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources)
+ SchedulerStatusBar on the Subscriptions tab (re-polled every 30s).
D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard
on Downloads with per-source and bulk "retry" (re-runs the feed via /check).
D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar
chart by the stat chips (failures stacked in error color); refreshes on live poll.
D4 credential staleness: surface last_verified age + "re-verify recommended"
warning past 30d; also fixes the dead last_verified_at field-name mismatch so
the verification row renders at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-28, two asks.
**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):
- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
gallery-dl in `--simulate --range 1-1` mode (no download) against the
URL with the materialized credentials, then reuses _categorize_error:
returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
the platform to probe, runs verify, and on success stamps
credential.last_verified (new CredentialService.mark_verified).
Returns {valid: bool|null, reason, last_verified?}. valid=null means
untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
result chip (Verified ✓ / Failed / Untestable) and a toast with the
reason. SettingsTab reloads on @verified so last_verified refreshes.
**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.
Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
Operator-flagged 2026-05-28, two asks.
**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.
**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)
- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
(dry_run) or deletes tags whose kind ∈ (archive, post, artist).
CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
returns per-kind counts + sample names — the preview IS the
verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".
Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).
NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.
New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
derive the artist from the import path, find an ENABLED Source with a
real feed URL for (artist, platform). Returns None for filesystem-only
imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
set ImportTask.refetched=True, and trigger ONE download_source
re-check. Bounded by `refetched` so source-side corruption can't loop.
Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
tasks). Returns refetch_queued / no_source / already_refetched /
not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
manual button is the primary path; auto is opt-in since re-fetch
deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
→ stores.import.refetchTask → toast keyed on the result status.
Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.
Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.
**Split archive into its own task**
- New `import_archive_file` task: same body as import_media_file
(dispatch is by file-kind inside Importer.import_one) but
soft=30min / hard=35min. Shared `_run_import_task` helper holds the
flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
truth for media-vs-archive dispatch. Used by all three enqueue sites:
scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
is_archive(entry) (the model field already existed, anticipating
this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
task_routes `import_file.*` wildcard — no worker config change.
**Archive-aware recovery sweeps**
Both sweeps would otherwise preempt a legitimately-running archive:
- recover_interrupted_tasks (ImportTask 'processing' sweep): now
task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
35-min hard limit). Single UPDATE with an OR predicate over the two
(task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
the per-queue overrides added for ml. import_archive_file gets 40 min
while the 'import' queue stays at the 5-min default for single-file
imports. Precedence: task_name → queue → default, each pass excluding
rows claimed by a higher-precedence pass so every row is touched once.
**Tests**
- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
pins that a 10-min archive task-run survives, a 50-min one is flagged,
and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.
After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.
**Audit results across `frontend/src/`:**
crypto.subtle.digest — 2 sites:
- MinDimensionCard (fixed 2026-05-27)
- BulkEditorPanel (THIS FIX)
navigator.clipboard — 1 site, already guarded:
- utils/clipboard.js writeText with execCommand fallback
serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
cookieStore / queryLocalFonts / WebAuthn / geolocation
— NOT USED, nothing to fix
Extension scripts (background.js) use crypto.subtle but run from
moz-extension:// which IS a Secure Context — left as-is.
**BulkEditorPanel double bug**
The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:
1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
`delete-images-selection-<sha8>` while the backend expected
`delete-images-<sha8>`. The two would never match.
Fix:
- Backend `/api/admin/images/bulk-delete` dry-run response now returns
`confirm_token` (the canonical `delete-images-<sha8>` string).
Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
set, it bypasses the `${action}-${kind}-${runId}` formula and uses
the explicit string. This decouples the UI label (`kind`) from the
wire-format token (server-provided), so future endpoints can use a
kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
— no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
slicing the 8-char suffix off the backend's token and passing it
through `runId`; now passes the full backend token via
`expected-token-override` directly). Cleaner; one source of truth.
**Banked memory**
`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.
No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
**showcase R-key + entry animation**
Restores two behaviors lost during the FC-2 IR→Vue port. Operator-flagged
2026-05-27.
- ShowcaseView listens for keydown 'r'/'R' on window. Triggers
`store.shuffle()`. Skips when an input/textarea/contenteditable is
focused or a Vuetify overlay is open (the dialog/menu sets
`.v-overlay--active` on the body).
- MasonryGrid gains an opt-in `animateFromIndex` prop (default
`Number.POSITIVE_INFINITY` = off). When set, items with index ≥ the
threshold animate in with a stagger fade-in: 12px translateY,
0.25s ease, 60ms per item, capped by `prefers-reduced-motion`.
Stagger uses original-items-array index (resolved via an `idxById`
Map) so the reading order is preserved even after the masonry
distributes items across columns.
- ShowcaseView watches `store.images.length`: shrink-or-zero baseline
⇒ `animateFromIndex=0` (animate everything on initial load /
shuffle); grow ⇒ baseline=prevCount (animate only the appended
tail on infinite-scroll). Other MasonryGrid consumers (ArtistView's
Gallery tab) don't pass the prop, so they keep their current
no-animation behavior.
Direct port of IR's `app/static/js/showcase.js` keyboard handler +
`app/static/style.css` itemFadeIn keyframe.
**min-dim Delete: crypto.subtle TypeError fix**
The Delete button on the Cleanup → Minimum Dimensions card was
silently no-op'ing. Root cause: `crypto.subtle` is Secure-Context-gated
(undefined on plain-HTTP origins per the homelab posture). The card's
`onDeleteClick` computed the Tier-C confirm token via
`crypto.subtle.digest('SHA-256', ...)`, which threw TypeError before
`showModal.value = true`. The promise rejected, the click handler had
no `.catch`, the modal never opened — exactly the operator's reported
symptom.
Same shape as the v26.05.26.0 `navigator.clipboard` fix on the
ErrorDetailModal Copy button.
Fix: backend `/api/cleanup/min-dimension/preview` now returns
`confirm_token` (the canonical `delete-min-dim-<sha8>` string) in its
response. Frontend reads it from the preview response and feeds the
8-char suffix to DestructiveConfirmModal's `runId` prop — no
client-side crypto needed. Single source of truth.
Integration test `test_min_dimension_preview_returns_count` pinned to
also assert `body["confirm_token"]` matches the server-side compute.
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.
**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.
**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.
**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.
**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.
**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.
Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.
/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
underlying cause is resolved
UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.
Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.
Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
ImageProvenance (only if currently NULL — preserves the canonical
download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
COALESCE(post.post_date, image_record.created_at), used in:
* scroll() ORDER BY + cursor WHERE clauses
* timeline() year/month group-by
* jump_cursor() year/month filter
* _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
works for images without a post (NULL on the Post side, fall back
to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
/scroll exposes both alongside the existing created_at so the UI
can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.
Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.
To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen