e30f50e6fec230990566d5d4aa19bcea16e721f9
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
e30f50e6fe |
fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days). |
||
|
bvandeusen
|
e66987f092 |
fix(audit-g2): async race / state-leak across eight stores
Extracts gallery.js's hand-rolled inflightId pattern into a new useInflightToken composable; adopts in every store that previously had no guard against late-response overwrites or wrong-image URL interpolation. Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62) flagged: - modal.removeTag rolled back the chip rail unconditionally even when only the secondary dismiss POST had failed — UI lied until refresh. And all tag-mutation URLs interpolated currentImageId AFTER an await, so a fast prev/next could route DELETE/POST to the wrong image. Both fixed: split try/catch (dismiss failure surfaces a warning, doesn't roll back the delete); imageId captured at call-time and used in URLs throughout. - suggestions.accept dereferenced currentImageId after the awaited POST /api/tags, so the subsequent /suggestions/accept could apply A's chosen tag to image B AND push it to B's allowlist. Fixed by capturing imageId at click-time + inflight guard on load(). Same shape across artist / downloads / artistDirectory / tagDirectory / posts stores: rapid filter/nav changes used to interleave responses (last-writer-wins). Now the late response is discarded and the most-recent request wins. Filter-change-during- search no longer drops the second fetch because the loading flag was still true from the first. gallery.js's inflightId removed in favor of the shared composable so the pattern stays consistent. |
||
|
bvandeusen
|
80ef9bce48 |
fix(test): credentials.upload reflects record into cache (not invalidate)
Stale assertion pinned the buggy pre-G1.6 behavior (the test name "upload invalidates the cache" literally describes the bug). The audit's correction makes upload mirror the returned record into the store so the card updates immediately — update the assertion to match the corrected behavior. |
||
|
bvandeusen
|
3898ce7be4 |
fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout 30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat TimeoutExpired (matched the tick 870s/900s rationale). Backfill runs that hit the cap let the next tick continue via the archive. - recover_interrupted_tasks orphan UPDATE now stamps finished_at; without it cleanup_old_tasks' WHERE finished_at<cutoff never reaped orphan-swept rows. recover_stalled_task_runs also now sets duration_ms (matches celery_signals.finalize's millisecond math). - ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3 on Source creation, mirroring SourceService.create. Without it, Firefox quick-add on a creator with >20 unsynced posts walked the full feed until subprocess timeout. Renamed the constant from _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module. - gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside NO_NEW_CONTENT — the download path (line 712) already does, and TIER_LIMITED proves auth reached the post and was told it was tier-gated. Verify endpoint previously showed red on this and prompted operators to rotate working cookies. - prune_unused_tags now runs a single DELETE with the NOT-IN predicate find_unused_tags uses, instead of SELECT-ids → DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that would have surfaced on a tag explosion (>65k unused tags). - credentials.upload() reflects the returned record into the store cache (`.set(platform, rec)`) instead of evicting it; previously the card briefly rendered "no credential" between upload and loadAll(). |
||
|
bvandeusen
|
91be9df671 |
fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount
attach_in_place mirrored only the media flow, so gallery-dl-downloaded zips/PDFs/audio bounced back as `skipped+invalid_image`, which download_service counted as an ingest error and flipped runs to status="error" despite N successful image attaches. Lustria patreon event #38998 (21 images + 1 OST zip) went red for exactly this reason. Now attach_in_place dispatches the same way as import_one: archives → _import_archive (extracts media members, captures archive as PostAttachment), non-media → _capture_attachment. Download_service accepts the new `attached` result and treats non-duplicate skips as soft skips, not ingest errors. Also: ShowcaseView always loadInitial() on mount, not just when the store is empty — Pinia persists across navigations and operator wants a fresh shuffle every time the showcase loads. |
||
|
bvandeusen
|
412edec028 |
fix(showcase): don't exhaust on all-dupe batches; retry up to 8x
/api/showcase returns a random sample; after enough scrolling, an unlucky 3-item batch can be entirely in the `seen` Set, which was flipping `exhausted=true` and surfacing "End." mid-scroll. The showcase is endless by intent — only a genuinely empty API response (library has 0 images) should mark it exhausted. Tiny-library fallback: cap retries at 8 to avoid spinning forever. |
||
|
bvandeusen
|
937421485d |
fix(modal): refresh tag chips after accepting a suggestion
Operator-flagged 2026-06-01: clicking Accept on a suggestion added the tag to the database but the modal's chip rail kept showing the old set. Suggestion would disappear from the suggestions list (suggestionsStore drops it from byCategory) and the toast would confirm "Tagged: …", but visually the modal looked unchanged until you closed and reopened it. Root cause: useSuggestionsStore.accept() POSTs to the backend and updates its own state, but never tells useModalStore that the backing image's tag set has changed. The addExistingTag and createAndAdd flows in TagPanel already call modal.reloadTags() after applying — the suggestions path needed the same refresh. Two-line fix in SuggestionsPanel: after store.accept() and store.aliasAccept(), call modal.reloadTags() so TagPanel's `modal.current?.tags` rebinds. |
||
|
bvandeusen
|
43b778aa04 |
fix(test): include 'scanned' in all backfill result assertions
Same 'change shared shape, miss pinned tests' trap as
|
||
|
bvandeusen
|
9cbdb70e13 |
fix(thumbnails): surface backfill results + tighten validity check
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."
1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
returned just `{celery_task_id}` and the admin card said
"Enqueued." with no counts. There was no way to tell whether the
scan found 0 candidates, 5000 candidates, or whether the worker
even picked up the task. "Found nothing" was indistinguishable
from a broken queue.
Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
shared by the Celery task and the API endpoint. The API now runs
the scan in an executor and returns `{scanned, enqueued, ok,
regenerated}`. The actual thumbnail generation work still goes
to the thumbnail Celery queue per row via
`generate_thumbnail.delay()` — the scan itself is fast
(SELECT id+thumbnail_path + a file.stat() per row).
2. **`_thumb_is_valid` accepted header-only corrupt files.** The
magic-byte check passed for any 8-byte file starting with a JPEG
or PNG header, including empty/truncated/zero-pad files that
browsers render as broken. Backfill counted these as `ok` and
never regenerated.
Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
thumbnails are minimum ~2KB even on solid-color sources; header-
only corrupt files top out around 12 bytes. 256 is well above
the corrupt floor and well below any legitimate thumbnail.
Plus the admin card now shows the per-run counts instead of
"Enqueued.":
Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
|
||
|
bvandeusen
|
bd06794647 |
fix(downloads): enqueue thumbnail + ML tasks per attached image
Operator-flagged 2026-06-01: downloaded images stayed at thumbnail_path=NULL until a periodic backfill sweep picked them up, surfacing as broken-thumbnail tiles in the gallery for hours after the download landed. Importer.attach_in_place deliberately skips inline thumbnail generation (importer.py:591-592) so the import queue stays moving — the CALLING task is responsible for the enqueue. tasks/import_file.py already did this (line 228-239). tasks/download.py / download_service did not — every gallery-dl-attached image landed un-thumbnailed. Fix in download_service._phase3_persist: after each `attach_in_place` returning status in (imported, superseded), fan out `generate_thumbnail.delay()` + `tag_and_embed.delay()` for each image_id. Lazy import avoids circular-import risk between download_service and the celery task modules that depend on it. Mirrors the existing pattern verbatim — single source of truth for "what fires after a successful attach" remains a comment in two places (filesystem-import task, download orchestrator) rather than a shared helper, because the contexts differ enough (sync session vs async orchestrator) that abstracting would obscure more than it'd share. Test covers the happy-path with two attached files: both get the thumbnail enqueue AND the ML enqueue, with image_ids drawn from ImportResult (so future supersede-on-attach paths stay covered). |
||
|
bvandeusen
|
f575cfb93b |
ux(failing-sources): visible row separators + clearer hover
The zebra striping in
|
||
|
bvandeusen
|
717b601c81 |
ux(failing-sources): zebra striping + hover highlight on rows
Operator-flagged 2026-06-01: with the failing-sources panel expanded, all rows have the same flat low-opacity background, no visual track from the artist name on the left to the Logs/Retry buttons on the right. Hard to tell which row a button belongs to when scanning down a list of 17. Three pure-CSS changes (no DOM, no logic): - 3px gap between rows (was 2px) — slightly more breathing room - Even rows get a darker surface tint (zebra striping) - Hover paints the whole row in a low-opacity error tint, so moving the cursor toward Logs/Retry lights up the whole horizontal band and the eye traces back to the artist name automatically |
||
|
bvandeusen
|
cfa4fb4084 |
fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create
`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial value (0) when checking that the override works. Now that create() pre-arms enabled sources to 3, the assertion is stale — the test was already verifying the override path, the pre-assertion just decorated it. Drop the pre-assertion; keep the override check. Other tests use raw Source(...) constructors (default to 0 via the column server_default), not SourceService.create(), so they're unaffected. |
||
|
bvandeusen
|
2aa2002f22 |
feat(subscriptions): newly added enabled sources start in backfill mode
A freshly created subscription has no gallery-dl archive yet, so the first poll in tick mode would walk forever — exit:20 doesn't trip until 20 contiguous archive-hits, and there are none. The wall-clock cap kicks in mid-walk, and the partial-success classifier (plan #544) gracefully labels it status=ok, but the operator still wonders why their new subscription isn't grabbing the back-catalog. Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`. Same default as the manual "Deep scan" button, same auto-decrement and auto-reset rules — once the queue drains (clean exit + zero new files) or the budget runs out, tick mode resumes naturally. Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'` that arrive disabled = False, or operator-added sources that start disabled deliberately) skip the pre-arm — they're never polled, no budget to waste. |
||
|
bvandeusen
|
66ff671f09 |
fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos
Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts hosted on Mux carry a JWT that encodes a playback restriction policy (`playback_restriction_id` in the token). The token signature alone is not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch. gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns 200 (token is valid, HEAD sends no Referer that Mux's policy rejects), but when yt-dlp follows up with the actual manifest GET it sends its own default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the single video fails — every other image in the same post still downloads. Static `downloader.ytdl.raw-options.http_headers` block now pins Referer and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch clears the policy check. Headers-only restrictions are now handled. Mux IP-range restrictions (if a creator opts into them) remain unfixable from our worker — those would need a Patreon-region IP. Per-video PARTIAL classifier (shipped in plan #544 last commit) already handles the residual case: a run that grabs all images and fails 1 video classifies as status=ok rather than flipping the whole event red. |
||
|
bvandeusen
|
19aece1fc4 |
feat(download): tick/backfill modes + partial-success classifier (plan #544)
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
|
||
|
bvandeusen
|
c9089b1d03 |
fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source
gallery_service._provenance_clause artist branch was correlating its bare Post reference to the outer query's primary_post_id outer-join, so the artist filter silently matched zero rows for images with no primary post. Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner FROM rather than treating it as a correlated outer table. Five sidecar/import tests still asserted that a synthetic Source row appears after a filesystem import. Alembic 0030 retired that behavior; the Post sits null-source and the artist linkage lives on Post.artist_id. Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent, test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar, and test_apply_sidecar_recovers_from_integrity_error to assert post.source_id IS NULL + post.artist_id linkage instead. |
||
|
bvandeusen
|
644d538bab | fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention | ||
|
bvandeusen
|
ff35da4743 | fix(lint): drop unused Source import after Post.artist_id refactor | ||
|
bvandeusen
|
2f66de2928 |
feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it. |
||
|
bvandeusen
|
94e7d20792 |
feat(downloads): Logs button on failing-sources rollup rows
Operator-asked 2026-06-01: each row in the "X sources are failing" rollup needs a button to surface the last run's stdout/stderr/error without leaving the Downloads tab to find the matching event row. Wired: - `downloadsStore.loadLastForSource(sourceId)` two-steps via `GET /api/downloads?source_id=N&limit=1` (most recent event) then the existing `loadOne(id)` for the full detail payload. Returns null if no events exist (toast warns). - `FailingSourcesCard` row: new text button `Logs` with `mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`, emits `view-logs` with the source record. - `DownloadsTab.onViewFailingLogs` is the handler — same `DownloadDetailModal` instance the event-row clicks use. |
||
|
bvandeusen
|
fb605af959 |
fix(posts): anchored view scroll inherits artist_id/platform filters
Operator-flagged 2026-06-01: clicking a Provenance post title from the
view modal opens the post in the posts feed scoped to that artist
(`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll
loaded UNFILTERED global posts instead of staying in the artist's
stream. Root cause: the posts store's loadAround/loadOlder/loadNewer
sent only `{around, cursor, direction}` — never the `artist_id` /
`platform` filters. Backend `/api/posts` accepts them on every path
(post_feed_service.scroll filters via `Source.artist_id`); the
frontend just wasn't passing them.
Fix:
- `posts.js` `loadAround(postId, filters)` now takes the filter
snapshot and stores it. `_aroundParams(extra)` mixes the active
filters into around/cursor calls.
- `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` +
`platform` from the route query.
|
||
|
bvandeusen
|
4c56cf121f |
fix(modal): Esc closes from tag input; Provenance cards scroll at ~2.5
Two operator-flagged UX gaps from 2026-06-01: 1. **Esc trapped inside the tag-entry field** The autofocused TagAutocomplete input made Esc-to-close unreachable because ImageViewer's keydown handler bailed early on `isTextEntry(ev.target)` for every key including Escape. Now Escape closes the modal from text inputs too — except when a Vuetify overlay is open (FandomPicker, autocomplete dropdown, suggestion 3-dot menu), in which case that overlay's own Esc-handling fires instead of closing the whole modal mid-interaction. Detected via `.v-overlay--active`. Arrow keys still gate on isTextEntry so the tag input handles typing without navigating images. 2. **Provenance cards expanded the side panel unbounded** When an image had many ImageProvenance entries the cards stack pushed the Tags section below the fold. Wrapped the cards in a `.fc-prov__cards` container with max-height 270px (≈2.5 cards) and thin overflow scrollbar. Title stays anchored at top; Tags panel sits at a consistent position below regardless of provenance count. |
||
|
bvandeusen
|
9564d073b9 |
fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash)
`${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner
executes the step via dash/BusyBox sh and errored with
`Bad substitution` (act_runner workflow shell, observed on the
|
||
|
bvandeusen
|
f87a06a6bd |
feat(modal): post title is the primary click (artist-scoped) + suggestion rows look like buttons
Two operator-asked modal UX changes (Scribe plan #514): 1. **Post title click → artist-scoped posts feed** - The bold post title in ProvenancePanel is now the primary click target. Click navigates to /posts?post_id=N&artist_id=A; the PostsView already filters on `artist_id`, so the user lands in that creator's stream, not the global one. - The redundant "View post" link is removed. "Show description" stays as the only action link below the meta line. - Title is styled as a button-link: accent color, hover underline, focus ring for keyboard nav (family rule 24 — UI quality bar). 2. **Suggestion rows look like buttons** - SuggestionItem becomes a chip-card: visible border, hover/focus background, unified container for name + score + actions (operator's "nothing visual to unify the buttons to their object" complaint). - Accept is an explicit tonal pill button labeled "Accept" (operator's pick over whole-row-clickable). 3-dot menu retains alias/dismiss, now `variant="outlined"` so it reads as a button. - Score is a fixed-width monospace pill on the right. - "+ new" badge upgraded to a pill chip with accent border so it's visibly an annotation, not part of the tag name. |
||
|
bvandeusen
|
5d284aae9f | fix(test): unpin general-threshold test from old 0.95 default (alembic 0029) | ||
|
bvandeusen
|
af7b5c95e9 |
feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
Four coupled operator-asked changes to the view modal (Scribe plan #509): 1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus in onMounted/nextTick so the caret is in the input the moment the modal renders. No click needed to start typing. 2. **General suggestions expanded by default** — SuggestionsPanel's general-category group now mounts with `:default-open="true"`. Operator can collapse if too noisy, but the v1 frame shows them. 3. **Lower general threshold default 0.95 → 0.50** — MLSettings. suggestion_threshold_general default matches character. Alembic 0029 also bumps the existing singleton row's value if it's still at the old 0.95. Operator can re-tune from Settings → ML. 4. **Retire `copyright` + `artist` as ML suggestion categories** — neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never really existed as a copyright tag-kind). They were surfaced in the suggestions pipeline + threshold settings UI but had no follow- through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for, ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029 drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS, SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and MLThresholdSliders rows. Out of scope (intentional): `tag_kind` Postgres enum still includes `artist` for historic Tag row queryability (per the model comment); no operator pain reported, no enum-shrink needed. Tests: - test_surfaced_categories asserts {character, general}, excludes artist + copyright. - test_threshold_for_artist_is_unsurfaced extended to cover copyright. - test_get_and_patch_settings asserts new 0.50 default and the absent artist + copyright keys in the GET payload. |
||
|
bvandeusen
|
8de7ccd07d |
build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m15s
extension / lint (pull_request) Successful in 16s
CI / lint (push) Successful in 2s
Adds an immutable per-commit docker tag to every main-push build:
`git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`,
alongside the existing floating `:main` + `:latest`. Implements the
new family release-posture rule "Tags are milestones, not gates —
commit-SHA images are the rollback unit" so rollback to any commit
on main is `docker pull …:c-<sha>` with no release ceremony required.
Behavior change summary:
- main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>}
- tag-push (opt-in vYY.MM.DD only, no .N): unchanged
- safety-net dev: unchanged
No code changes; the rule is about how the tag list is constructed.
Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still
fire them when the operator wants a labeled checkpoint.
|
||
|
bvandeusen
|
d65f0b2091 |
feat(extension): probe shows current state before click; v1.0.6
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 20s
extension / lint (push) Successful in 13s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m41s
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup): "the add source/subscription button idea to the firefox extension so it can tell me if a source/artist is added or not and offer an option to add it if it isn't." Plan tracked in Scribe task #507. ## Backend - `ExtensionService.probe(url)` — read-only resolution. Reuses `_derive` for platform+slug, then 2 SELECTs. Returns one of: - `source_match` (exact (artist, platform, url) Source exists) - `artist_match` (artist exists, this URL isn't a Source yet; collapses the sidecar-synthetic-only case from v26.06.01.0) - `new` (neither exists) - `unknown_platform` (URL didn't match any artist-page regex) - `GET /api/extension/probe?url=...` route with `X-Extension-Key` auth posture matching `/quick-add-source`. Read-only, side-effect free. - 6 backend tests in tests/test_api_extension.py covering each state + auth + invalid URL. ## Extension - `api.js`: `probeSource(url)` mirroring `quickAddSource` shape. - `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The latter strips the `/api` suffix from configured `apiUrl` (placeholder format per options.html) and opens `${base}/artist/{slug}` in a new tab via `browser.tabs.create`. - `content-script.js`: probe-first render — on page-load and SPA navigation, asks the backend for the URL's state and renders the chip in the matching color/copy on FIRST paint instead of flashing generic "Add" and updating after. Click handler branches: `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing ADD_AS_SOURCE flow (then re-probes so the chip flips green immediately, no wait for next nav). - `content-script.css`: three state-color modifiers (--new, --artist-match, --source-match) on the FC parchment-on-slate palette. Sage for already-added, amber for artist-exists, accent orange for new. ## Versioning - `extension/manifest.json` + `extension/package.json` → 1.0.6. build.yml's sign-extension job will fire on push to main since no `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the regenerated AMO keys end-to-end. ## Behavior on the sidecar-synthetic case Filesystem-imported "Dymkens"-style artist with only a sidecar synthetic Source: probe returns `artist_match` (not `new`), so the chip reads "+ Add Patreon source to Dymkens" rather than offering to recreate the artist. Clicking adds the real Source; existing `_source_for_sidecar` preference logic (v26.06.01.0) routes future gallery-dl Posts to the real one. |
||
|
bvandeusen
|
66f19d67f5 |
fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED |
||
|
bvandeusen
|
6fc8ae3106 |
fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing "phantom" subscriptions like `sidecar:patreon:dpmaker`: 1. `SourceService.list()` returned every Source, no filter on URL. alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into one canonical row per (artist, platform); when no real campaign URL was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>` enabled=false as a disabled anchor. The UI then listed those anchors as if they were polls — disabled, but visible. Fix: `list()` excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True` opts back in for admin tooling. 2. `importer._source_for_sidecar` picked the lowest-id Source for (artist, platform). When alembic 0022 had rewritten a per-post row into a synthetic anchor (lower id) AND the operator later added the real subscription (higher id), every gallery-dl download silently attached its Post to the SYNTHETIC instead of the real Source. Fix: prefer a non-`sidecar:%` URL when one exists; fall back to the synthetic; only create a new synthetic when nothing exists for (artist, platform). alembic 0028 is the data half: for every (artist, platform) with both a synthetic AND a real Source, pre-merge Post+ImageProvenance collisions on the canonical, bulk-repoint Posts/ImageProvenance/ DownloadEvent.source_id onto the real Source, and delete the synthetic. Lone synthetics (no real twin) are left intact — they anchor real imported content the operator may still want; the list-filter hides them so they no longer surface as phantoms. |
||
 bvandeusenandClaude Opus 4.7
|
a5101494b6 |
feat(downloads): bulk retry respects cooldown; single-source RETRY overrides
Today's platform-cooldown commit (
|
||
|
bvandeusenandClaude Opus 4.7
|
e3a7aff7a3 |
ux(showcase): pipeline fetches in-order, chunk size 3, keep the trickle
Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast but could let later chunks arrive ahead of earlier ones — even when each chunk is a random sample, that "later chunk loads first" risk made the load order non-deterministic. And the original goal behind asking for batching was a faster first-image-on-screen, which neither sequen- tial nor parallel really addressed cleanly. Switched the loadInitial flow to a PIPELINE: - Only one fetch in flight at any moment (in-order arrival, no race). - The NEXT fetch kicks off as soon as the current one resolves (NOT after its trickle finishes), so the next RTT overlaps the visible trickle window — round-trips are hidden behind the animation cadence. - PAGE 5 → 3 + INITIAL_BATCHES 12 → 20 (total still 60). Smaller chunk → first chunk's items appear sooner (a chunk of 3 trickles in 240ms, well within one RTT, so by the time chunk 2 is in-hand the first trickle is just finishing). Trickle, sequence-token guard, and infinite-scroll behaviour unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9cd6d09e60 |
ux(showcase): smooth one-at-a-time cadence + earlier infinite-scroll trigger
Operator-flagged 2026-05-30 (round 2): the batched-loads change improved consistency but still felt "chunky" — 5 items appeared together, then a ~200 ms API round-trip pause, then another 5. And infinite-scroll fired too late when a single tall image (long manga page) made one masonry column much taller than its siblings. **Cadence (showcase store):** - Fire all INITIAL_BATCHES fetches in PARALLEL — collapses the per- batch round-trip gap so all the data arrives in ~1 RTT instead of 12 sequential RTTs. - Trickle each response's items into images.value one at a time with APPEND_DELAY_MS = 80ms between each (≈ the MasonryGrid stagger animation, 70ms). User sees a smooth steady stream. - fetchPage (the infinite-scroll path) uses the same trickle so its 5-item appends also cascade one-by-one instead of popping together. - Sequence token guards against a fast shuffle / mount-then-shuffle interleaving two trickles into the same images.value. - Dropped useAsyncAction here — the parallel-fetch-then-trickle flow doesn't fit its single-wrap-call shape cleanly; inline loading/error state is clearer. **Infinite-scroll trigger (MasonryGrid):** - Pass `rootMargin: '2400px'` to useInfiniteScroll (was the 600px default). The masonry sentinel sits at the bottom of the container, whose height = MAX(column heights). A tall image in one column pushes the sentinel ~2× viewport below where the user is actually reading (the bottom of the SHORTER columns). 2400px ≈ 2-3 screen-heights of pre-emptive trigger, comfortable for typical tall manga heights. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
810baf63ac |
fix(import): archive probe → subprocess (was multiprocessing.Process)
Every archive import was failing immediately with "AssertionError:
daemonic processes are not allowed to have children" (operator-flagged
2026-05-30 — import_archive_file crashes in 49ms with the assertion).
Celery's prefork pool runs tasks in daemon processes; Python's
multiprocessing module refuses to let daemons spawn children, which is
exactly what probe_archive was doing via mp.get_context("spawn")
.Process. The Layer-3 crash-isolation feature added 2026-05-28 was
effectively a hard-blocker on the very import path it was meant to
protect.
Switched probe_archive to subprocess.run — no daemon restriction, still
isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill
the worker). The probe body lifted to a tiny runner module
(_archive_probe_runner) that imports the unchanged _run_probe helper
and prints a single JSON line; parent parses stdout, returns the
ProbeResult exactly as before (timeout, signal, OOM, clean-rejection,
ok — all preserved).
cwd for the subprocess is the repo root derived from __file__ parents
so `python -m backend.app.utils._archive_probe_runner` resolves both in
the Celery container and pytest, regardless of where the worker was
launched.
Test refactor: test_archive_probe_target_bomb_guard →
test_run_probe_bomb_guard. Same in-process call to the (renamed) probe
body so the monkeypatched cap still takes effect; the real subprocess
path is exercised by the existing test_probe_archive_valid_zip /
test_probe_archive_corrupt_zip_clean_rejection tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
44bb12a93d |
fix(thumbnails): derive URL from stored thumbnail_path, not (sha256, mime)
The showcase/gallery/artist/series/post-feed APIs were constructing
thumbnail URLs from (sha256, mime). The MIME-based extension predicate
("png if image/png or image/gif else jpg") DISAGREED with the
thumbnailer's actual on-disk extension predicate ("png if alpha else
jpg"). Result: every PNG source without transparency 404'd (URL asked
.png, disk had .jpg); every WebP/AVIF source with transparency 404'd
(URL asked .jpg, disk had .png) — despite the thumbnail file existing
on disk.
The backfill task couldn't catch these because backfill checks the
ACTUAL thumbnail_path stored on the record (correct), not the URL the
browser fetches (broken derivation). So records with valid on-disk
thumbnails kept showing as broken in the UI no matter how many times
backfill ran.
Operator-flagged 2026-05-30: "the generate thumbnails function appears
to not catch all of the failed thumbnail cases" — turned out to not be
a backfill bug at all.
Fix: thumbnail_url now takes (thumbnail_path, sha256, mime) and returns
the stored path verbatim — Quart serves /images/* 1:1 from the volume
(frontend.py:20-36), so the URL IS the disk path. Falls back to the old
sha256+mime derivation only when thumbnail_path is NULL (thumbnailer
hasn't run yet); that URL will 404 in the browser until backfill catches
it, same as before the path was tracked.
All 8 callers updated: showcase_service, gallery_service (2 sites),
artist_service, series_service, post_feed_service, tag_directory_service,
artist_directory_service. The four sites whose query was raw-tuple now
also SELECT ImageRecord.thumbnail_path.
Net effect: every record that has a valid on-disk thumbnail will now
render correctly, regardless of which extension the thumbnailer chose,
without any DB migration or backfill rerun needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
1eefed9ab3 |
fix(ui): finish showcase store batching (PAGE 60 → 5, INITIAL_BATCHES 12)
The showcase-store change in
|
||
|
bvandeusenandClaude Opus 4.7
|
adeee64a2d |
feat(ui): batch initial showcase/gallery loads into smaller chunks
Showcase and gallery were fetching the full initial batch (60 / 50 items) in a single API call. Operator-flagged 2026-05-30: items should stream in as small batches so they render progressively rather than blocking on one big response. - showcase store: PAGE 60 → 5, INITIAL_BATCHES = 12 (5 × 12 = 60, same total). loadInitial() fetches PAGE chunks in sequence; shuffle() delegates. fetchPage() still returns one PAGE-sized chunk so infinite-scroll also pulls 5 per trigger. - gallery store: loadMore() limit 50 → PAGE (5), INITIAL_BATCHES = 10. loadInitial() loops loadMore() up to INITIAL_BATCHES times, stopping early when nextCursor becomes null (end of data). - ShowcaseView: onMounted calls loadInitial() instead of fetchPage(). Gallery/setTagFilter/setPostFilter already routed through loadInitial. Net visual effect: each ~5-item batch lands and renders independently; combined with MasonryGrid's animateFromIndex logic, the showcase now cascades batches in as they arrive instead of one big pop-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
99b66aa85f |
fix(download): preserve partial output + classify timeouts richer
Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show anything other than that the task timedout and was cleaned up. I can't tell why it ran over or if it was stuck failed or there was just that much to get." The TimeoutExpired branch was returning a DownloadResult with no stdout, no stderr, no files_downloaded, and a generic "Download timed out after N seconds" message — even though subprocess.TimeoutExpired carries the partial output gallery-dl emitted before being killed. Now: - Capture e.stdout / e.stderr (coerced str if bytes; "" if None). - Count files_downloaded from partial stdout via _count_downloaded_files. - Surface a tail-of-stderr hint in error_message so the UI summary tells the operator at a glance whether it was "lots of content" (high count, clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung silent" (zero count, "no stderr output"). - Promote error_type to RATE_LIMITED when the partial stderr matches RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole 900s window is the timeout-shaped tail of a real rate limit, and the platform cooldown should kick in for the same reason. Existing test_download_timeout strengthened to also assert empty-partial case stays correctly TIMEOUT-classified with no preserved output. New test_download_timeout_preserves_partial_output_and_classifies covers the rich-partial-output → RATE_LIMITED promotion path. DownloadEvent.metadata already flows stdout/stderr/run_stats from DownloadResult via _phase3_persist — no UI change needed; the existing DownloadDetailModal will surface the captured output automatically once the build redeploys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
77f7a23410 |
feat(scheduler): order due sources by last_checked_at — most overdue first
select_due_sources returned rows in undefined order (Postgres-determined, typically PK). At tick rates that outpace download-queue throughput, a freshly-rerun source could keep getting re-queued ahead of one that's still waiting for its first attempt this cycle. Operator-flagged 2026-05-30: > if there are 8 hours before a source is due again and 40 full time > downloads can happen in that period that means that there's a chance > the first one to fire gets back into the download queue before item 41 > has a chance to get downloaded. Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source SELECT. Never-checked sources go first, then longest-since-checked, then ties broken by id. Combined with Celery's FIFO `download` queue, the oldest-overdue source in each tick now reaches a worker before any fresher one. Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue source, and a 2-min-overdue source come back in that exact order from select_due_sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
ff9e96e0e2 |
fix(tests): update scheduler-status shape pins for platform_cooldowns
|
||
|
bvandeusenandClaude Opus 4.7
|
61ce1ce13c |
feat(scheduler): platform-wide cooldown on RATE_LIMITED — burst prevention
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.
When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.
INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.
Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.
Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
d28db32012 |
fix(download): align gallery-dl subprocess timeout to Celery soft limit
SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's Celery task has soft_time_limit=900s and hard time_limit=1200s. So gallery-dl never hit its own subprocess.run timeout — Celery always killed it first. The hard SIGKILL leaves no terminal flip on the DownloadEvent, which then sat pending/running until the recovery sweep flipped it to error at 30 min from start. From the operator's seat that was a ~30–40 min "hang" on every retry of a broken source. Pinning the default to 900s (matching Celery's soft_time_limit) lets subprocess.run raise TimeoutExpired cleanly inside Celery's window, and the existing `except subprocess.TimeoutExpired` branch (gallery_dl.py:635) captures it as a clean error_type='timeout' with a real message. The DownloadEvent flips to error in ~15 min instead of waiting on the recovery sweep at 30. Per-source bumps still live in source.config_overrides for legitimately long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries the rationale and the Celery-soft-limit dependency in its comment. Operator-flagged 2026-05-30 on 59-source strand pile retries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
77e9859da3 |
fix(downloads): rewire MaintenanceMenu to the downloads pipeline
The maintenance dropdown in Subscriptions → Downloads was wired to the
filesystem-import pipeline (POST /api/import/retry-failed +
POST /api/import/clear-stuck) — the subtitles even said so ("Re-enqueue
every failed import task"), but it was contextually misplaced. From the
Downloads view "Retry failed" queued nothing the operator could see
because the action operated on import_task rows, not download_event
rows. Import-pipeline maintenance is already reachable from Settings →
Imports (ImportTaskList.vue), so removing the import wiring loses
nothing.
Rewired:
- "Retry failed" → bulk-retries the failing-sources list, same loop as
FailingSourcesCard's RETRY ALL (sourcesStore.checkNow per source).
Subtitle now matches: "Re-queue every currently failing source".
- "Force recovery sweep" → triggers recover_stalled_download_events on
demand via a new POST /api/downloads/recover-stalled endpoint. The
sweep also runs every 5 min on Beat; this is the manual fallback so
the operator doesn't have to wait for the next tick to clear newly
stranded events.
MaintenanceMenu is now stateless — emits retry-failed and recover-
stalled. DownloadsTab owns the handlers (reuses the existing
onRetryAll; new onRecoverStalled with a delayed refresh so swept rows
land in the failing rollup).
Operator-flagged 2026-05-29 — "the retry failed button in the
maintenance dropdown doesn't appear to queue anything but manual
requeues works."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
36f8ec80fd |
fix(ui): tooltip override needs !important — Vite reorders CSS chunks
104cac5's override sits at the same specificity as Vuetify's default
(`.v-tooltip > .v-overlay__content` on both sides). The fix relied on
source order — app.css imported after vuetify/styles in main.js — but
Vite's production bundler reorders node_modules CSS into the final
stylesheet unpredictably, so source order isn't a reliable winner.
!important on the two contrast properties (background + color) forces
the slate-on-parchment pair regardless of stylesheet load order. The
cosmetic border + shadow don't need it (Vuetify doesn't set them, so
nothing's competing).
Operator re-flagged 2026-05-29: tooltips still rendered light-on-light
on the deployed :latest after PR #33 —
|
||
|
bvandeusenandClaude Opus 4.7
|
e35fb1edf7 |
fix(scan): recovery sweep for stranded download events
The scan tick (scan.py:_tick_due_sources_async) inserts DownloadEvent(status='pending') and fires download_source.delay(). If the task dies before finalizing the event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind on the 1200s hard time_limit — the event stays in-flight forever. Every later tick then skips the source via the in-flight guard (scan.py:168), so Source.last_checked_at is never written and the operator sees "last check never" in the Subscriptions health column, permanently. cleanup_old_download_events only prunes terminal events (by design); no existing sweep covered the pending/running case. Operator confirmed 2026-05-29 with a diagnostic query: all 43 "never checked" sources were stranded behind stale in-flight events (eligible_stuck_inflight = 43, every other bucket zero). New recover_stalled_download_events task (Beat every 5 min): - Flips DownloadEvent rows pending/running > 30 min (10 min past the download_source 1200s hard kill, so legitimately-running tasks are never touched) to status='error' with a sentinel message. - Bumps each affected Source's consecutive_failures ONCE per source — backoff is 2^N on that counter so per-event bumps would needlessly inflate the next interval — sets last_error, stamps last_checked_at. UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that would hit the psycopg 65535-param ceiling on a large strand pile. Net: the 43 currently-stranded sources unstick on the first sweep after deploy, their health dots flip amber instead of unchecked, and the next scan tick re-queues them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
08420cd619 |
feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon
The plain-JS frontend re-declares two backend enum value-sets by hand: download-event statuses (downloadStatus.js) and platform keys (platformColor.js). With no TS codegen, drift is silent — the same class as the last_verified_at field mismatch. tests/test_fe_be_contract.py parses both JS mirrors and asserts they equal the backend canon (downloads._ALLOWED_STATUSES, platforms.known_platform_keys()) so a change on either side fails CI on whichever moved. Backend is the source of truth. Documented the invariant in both JS file headers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
8649a13118 |
refactor(I5): remove one-and-done GS/IR migration tooling
The GS/IR migration cutover is complete, so the runbook tooling is dead weight. Removed: - services/migrators/ (gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup), tasks/migration.py, api/migrate.py (+ blueprint registration) - MigrationRun model; alembic 0027 drops the migration_run table - frontend LegacyMigrationCard + migration store (+ MaintenancePanel ref) - celery include + task route + celery_signals queue mapping for migration.* - the 1 GB MAX_CONTENT_LENGTH / MAX_FORM_MEMORY override (added solely for the ir_ingest upload) - migration-surface tests (test_api_migrate, test_migration_verify, test_ir_ingest, test_gs_ingest, test_tag_apply) Kept: the alembic schema-migration tests (test_migration_00XX — unrelated) and cleanup_service.py (the permanent artist-cascade/unlink home). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
8979e0e377 |
refactor(I4): extract useInfiniteScroll composable; migrate 6 consumers
The IntersectionObserver-on-a-sentinel infinite-scroll pattern was copy- pasted in 7 places. New composables/useInfiniteScroll(sentinelRef, cb) owns the observer lifecycle (attach on mount, re-attach when the ref changes, disconnect on unmount). Migrated GalleryGrid, MasonryGrid (gallery+showcase), ArtistPostsTab, ArtistsView, TagsView, SeriesManageView. PostsView left manual on purpose — its anchored mode does bidirectional scroll-position preservation that doesn't fit the simple composable. I4(a) (timestamps) is effectively already done: the UTC displays were converted earlier; the remaining toLocaleString uses already render local time, so they're left as-is rather than churned for format-only consistency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
00e2608ba1 |
feat(I3): always-on pipeline status indicator in the top nav
New /api/system/activity/summary aggregates scheduler health + per-queue pending depths + running count + 24h failure count in one cached call (safe to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a compact running/queued/failing chip with a scheduler-health dot that expands to a popover (scheduler, busy queues, counts, link to downloads). Polls the summary every 8s, paused when backgrounded. Reuses the queue-cache read via _queues_cached(). + API test for the summary shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9ab5d709c8 |
fix(test): DownloadEventRow row shows platform, not artist name
The row template renders status + platform (PlatformChip) + time + counts; the artist isn't displayed there. Assert on 'Completed'/'Patreon' instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
44410db492 |
test(I2): vitest component smoke tests for high-risk UI
Adds @vue/test-utils + happy-dom and mounts CredentialCard, DownloadEventRow, PostCard, ActiveDownloadsPanel, QueueStatusBar, asserting they render without throwing and surface key content — catching the dead-binding / render-error class (e.g. the last_verified_at regression, guarded explicitly). Vuetify components are left unresolved (Vue renders unknown elements + slots), so no Vuetify-plugin setup is needed; only RouterLink is stubbed. Per-file happy-dom env via docblock keeps the existing node-env specs untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
e76aa36a29 |
ci(I1): dedicated fast-fail ruff lint lane
ruff is pre-installed in the ci-python image, so a new `lint` job runs it with no dependency install and fails in seconds — surfacing the common lint bounce class without waiting on the backend job's ~30-60s wheel install. Dropped the now-redundant ruff step from backend-lint-and-test (same job name, required-checks unchanged). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
c95b760294 |
feat(posts): in-context anchored feed with bidirectional infinite scroll
Provenance "View post" deep-links to /posts?post_id=X, which now opens the feed centered on that post with infinite load in BOTH directions. Backend: PostFeedService.scroll gains a direction (older|newer); new around(post_id) returns a window of newer + the post + older with a cursor for each end. /api/posts accepts ?around= and ?direction=. + API tests. Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends, newer prepends) with per-end cursors; PostsView's anchored mode scrolls to the post, observes top + bottom sentinels, and preserves scroll position on upward prepend so the page doesn't jump. Normal feed mode unchanged. Closes the remaining half of the post-navigation work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
35fe420701 |
feat(maintenance): live queue-depth bar under the backfill buttons
Each maintenance button (ML backfill, centroid recompute → ml queue; thumbnail backfill → thumbnail queue) now shows a status bar with the live pending count for its Celery queue, so the operator can see work is already queued/running before re-triggering and piling on. MaintenancePanel polls /api/system/activity/queues every 4s; QueueStatusBar reads the depth and turns warning-colored when the queue is busy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9e74c80e2f |
feat(downloads): front-and-center "active now" panel with live elapsed timers
New ActiveDownloadsPanel pinned at the top of the Downloads tab shows what's running (pulsing dot + live mm:ss timer counting from started_at) and what's queued, so the operator can see activity at a glance without the running filter. Backed by store.loadActive() which fetches running + pending independent of the feed filter; refreshed every 4s by the existing live poll. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
c87e8e0932 |
fix(ui): render absolute timestamps in the viewer's local timezone
Download event times showed raw UTC wall-clock (iso.slice). Added formatDateTime()/formatLocalDate() (local tz, robust to naive vs tz-aware ISO) and applied them to the download row + detail modal datetimes and the credential/artist date displays. formatPostDate stays UTC (date-only, locale-stable, unit-tested). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
8c3900b998 |
feat(showcase): dramatic 3D cascade entry — tiles tip back then settle into place
Replaced the subtle 12px fade with a perspective rotateX(-28deg) tilt that flips up and settles flat with a slight overshoot, staggered 70ms so tiles cascade in one at a time. Makes the showcase read as an experience rather than a quiet fade. Tunable (tilt/stagger/duration); reduced-motion safe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
972d9014ce |
feat(showcase): IR-parity hover animation on masonry thumbnails
FC already matched IR's staggered entry fade-in; this adds the missing piece — hover zoom (scale 1.03) + brighten (1.1) on each thumbnail, with overflow:hidden so the scaled image clips to the rounded card. Disabled under prefers-reduced-motion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
75b6b8056e |
feat(posts): plain bold titles; reserve View-original to the post card; provenance links to the posts feed
- Post titles arrived as stored HTML (e.g. <strong>…</strong>) rendering as literal markup. New toPlainText() strips tags; titles render plain + bold (ProvenancePanel and PostCard). - Removed "View original post" from the provenance panel (modal) — the open-original button lives on the PostCard (the post view). - Provenance "View post" now navigates to the /posts feed (post_id query), not the gallery image grid. (Feed in-context landing lands next.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
42ddac9996 |
fix(subscriptions): fixed-height hub so only the tab content scrolls
The whole view scrolled instead of just the subscription list. Made the hub a viewport-height flex column (tabs stay fixed) with the v-window as the single internal scroll container; the per-tab sticky control bars now pin to the window top (top:48px -> 0). Operator-flagged 2026-05-28. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
1322056b22 |
refactor(dry-F5): useTabQuery composable for ?tab= query-param routing
ArtistView and SubscriptionsView both two-way-bound a tab ref with the ?tab= query param via identical tab->URL and URL->tab watchers. Extracted useTabQuery(validTabs, defaultTab) — defaultTab accepts a string or a function (ArtistView's default is postCount-dependent), and resolve() is exposed so ArtistView can re-apply it after the artist loads. Both views drop their now-orphaned ref/useRouter imports. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
32bdde049f |
refactor(dry-B3): extract _get_or_create race-safe find-or-create in Importer
_find_or_create_source, _source_for_sidecar, and _find_or_create_post each repeated the SELECT → savepoint-INSERT → on-IntegrityError rollback+re-SELECT pattern. Extracted _get_or_create(stmt, factory): the statement is reused for the scalar_one_or_none lookup and the scalar_one post-conflict re-fetch, so all three are reproduced exactly. Centralizing the race-safe pattern in one place also reduces the risk of the copies drifting (the bug class banked 2026-05-26). Left _upsert_artist (no savepoint by design) and the ImageProvenance void ensure-exists block (no return / no re-select) alone — they don't fit. The rest of the ingest pipeline was already DRY: sidecar parsing lives in utils/sidecar.py, per-platform quirks in the platforms package, and _safe_ext/_categorize_error/_build_config are each single-instance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9d18dacbe8 |
fix(lint): UP037 — drop quotes from ImportSettings.load return annotations (py3.14 deferred annotations)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
171c486939 |
refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was repeated 15× across services, API, and 5 task modules. Added async load() + sync load_sync() classmethods on the model and migrated all 15 full-row sites (callers already imported ImportSettings, so no new imports; dropped download's now-orphaned select import). Left maintenance.py's deliberate column-select (import_scan_path only) as-is. Rest of the service layer was already adequately DRY — the Record/to_dict pattern is only 2 instances and the savepoint find-or-create recovery is correctly per-entity, so neither was forced into a shared abstraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
21c1b0a81c |
refactor(dry-B4): extract shared async_session_factory for Celery tasks
download/migration/scan each defined an identical _async_session_factory() (fresh per-invocation async engine — async connections are event-loop-bound so each asyncio.run() task needs its own engine, unlike the process-wide _sync_engine). Moved it to tasks/_async_session.py; the 3 files import it and drop their now-orphaned sqlalchemy.ext.asyncio / get_config imports (migration keeps AsyncSession for a type hint). Call-site try/finally dispose left as-is to avoid re-indenting the critical task bodies. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
597c6d48d3 |
refactor(dry-B1): consolidate duplicated _bad error helper into api/_responses.py
8 blueprints each defined an identical _bad() (two variants: with/without detail). Extracted error_response() into api/_responses.py; each blueprint now imports it `as _bad` so call sites are unchanged. The detail-aware canonical subsumes both variants. Left settings.py's distinct _bad_int and the inline jsonify error sites (not duplicated helpers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
a37dad33c7 |
refactor(dry-F1): shared useAsyncAction lifecycle helper for stores
New composables/useAsyncAction.js owns the loading/error/try-finally lifecycle. Migrated 11 stores: credentials, downloads, sources, posts (error=raw) + ml, artistDirectory, tagDirectory, showcase, suggestions, seriesReader, modal (error=message). The errorAs option preserves each store's existing error shape so store.error keeps the same type for components (~50 consumption sites unchanged). Stores whose catch also reset data (suggestions/seriesReader/modal) clear it upfront instead. Deliberately NOT migrated (special control flow, would change behavior): artist (conditional 404 catch + dual loading states), migration (rethrows), gallery (inflight-id stale-response guard), and the Shape-B no-catch / loaded-guard / keyed-cache stores. Net -77 lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
cabd73287a |
fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk
Removing the create_app import left 2 blank lines before pytestmark in 9 files where ruff isort wants 1. Also stripped two pre-existing whitespace-only blank lines surfaced by the file change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
def967a1a8 |
refactor(dry-S1): hoist app/client test fixtures into conftest
Removed the app/client fixtures duplicated across 36 test files (two variants: separate app + client(app), and a self-contained client() that called create_app inline) and the now-unused create_app imports. Both fixtures now live once in conftest.py. test_suggestions_bulk keeps its import (builds the app inline in two tests); test_health drops its local client + unused pytest_asyncio. Net -415 lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
eebc8e2413 |
refactor(dry-F2): centralize shared UI primitives (relative-time, toast, download-status)
- utils/date.js: add formatRelative(iso, {future,nullText}); migrate 6 sites
(SourceRow, SubscriptionsTab, SourceHealthDot, SchedulerStatusBar +
thin adapters in BackupRunsTable/SystemActivityTab for their '—' null text).
PostCard (30d->absolute) and CredentialCard (mo/y buckets) intentionally
keep bespoke formatters.
- utils/toast.js: toast(opts) wraps the globalThis.window?.__fcToast?.(...)
incantation; migrate 63 call sites across 24 files.
- utils/downloadStatus.js: single source for the download-event status enum
-> label/color/icon; collapse the 3 duplicate maps (DownloadStatChips,
DownloadsFilterPopover, DownloadEventRow).
Net -33 lines. Platform metadata was already centralized in platformColor.js.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
2358cedf3e |
feat(dashboards): scheduler health strip, failing-source rollup, 24h activity sparkline, credential staleness nudge
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick + GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources) + SchedulerStatusBar on the Subscriptions tab (re-polled every 30s). D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard on Downloads with per-source and bulk "retry" (re-runs the feed via /check). D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar chart by the stat chips (failures stacked in error color); refreshes on live poll. D4 credential staleness: surface last_verified age + "re-verify recommended" warning past 30d; also fixes the dead last_verified_at field-name mismatch so the verification row renders at all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
215a8993a1 |
fix(lint): noqa ASYNC109 on gallery_dl.verify timeout (subprocess.run deadline, not coroutine)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a459d21a65 |
feat(dashboards): 1600px max-width, richer Downloads filters, needs-attention + sticky headers
Operator-flagged 2026-05-28: the Subscriptions/Downloads dashboards were full-bleed and thin on filtering. Chosen via AskUserQuestion. **Layout** - SubscriptionsView capped at a centered max-width 1600px (covers all three subtabs) so rows aren't a mile wide on ultrawide monitors. - Sticky control headers on both tabs (top: 48px, below the top nav, opaque bg) so filters/stat-chips stay reachable while scrolling a long list. **Downloads filters (all four requested)** - Stat chips are now clickable filters: click Queued/Running/Completed/ Failed/Skipped to filter the list to that status; the active chip is outlined + elevated; re-click clears. - Free-text search box over the loaded events (artist / platform / error substring). - Artist filter: the filter popover's numeric "Source ID" field is replaced with an artist autocomplete (sources.autocompleteArtist → artist_id, which /api/downloads already supports). Pill shows the artist name. - "Show no-change scans" toggle (default OFF): hides status=ok/skipped rows with 0 files (the scheduled scans that found nothing) so real downloads + failures stand out. **Subscriptions** - "Needs attention" quick-filter chip: one click to show only artists with sources that have errors OR have never been checked; chip shows the count and disables the status dropdown while active. Frontend-only — backend filter params (status/artist_id/date) and the /api/downloads endpoint already supported everything. |
||
|
bvandeusen
|
73520b7cc3 |
feat(downloads): copy buttons in the download detail modal
Operator-flagged 2026-05-28: the download event detail modal had no way to copy the error / stdout / stderr text for researching a failure (parity with the ErrorDetailModal Copy button shipped in v26.05.26.0). Added per-block Copy buttons (Error, Errors & warnings, Raw stdout, Raw stderr — the stdout/stderr ones sit in the expansion-panel title with @click.stop so they don't toggle the panel) plus a "Copy all diagnostics" button in the footer that assembles a single block: header line (event id / platform / artist / status / timestamps) + error + errors&warnings + full stdout + stderr, ready to paste into an issue. All routed through utils/clipboard.js copyText() — the navigator.clipboard → execCommand fallback that works on the plain-HTTP homelab origin (per feedback_no_secure_context_apis). Each copy shows a confirmation toast. |
||
|
bvandeusen
|
56970fb66d |
feat(credentials+downloads): real credential Verify button + live download-activity polling
Operator-flagged 2026-05-28, two asks.
**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):
- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
gallery-dl in `--simulate --range 1-1` mode (no download) against the
URL with the materialized credentials, then reuses _categorize_error:
returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
the platform to probe, runs verify, and on success stamps
credential.last_verified (new CredentialService.mark_verified).
Returns {valid: bool|null, reason, last_verified?}. valid=null means
untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
result chip (Verified ✓ / Failed / Untestable) and a toast with the
reason. SettingsTab reloads on @verified so last_verified refreshes.
**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.
Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
|
||
|
bvandeusen
|
bf8eb4468f |
feat(tags): legacy-tag purge also catches source:* general tags
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
|
||
|
bvandeusen
|
e1fc65bd1b |
feat(provenance+tags): collapse provenance descriptions; purge retired-kind tags
Operator-flagged 2026-05-28, two asks. **1. Provenance posts ate the panel.** Each post card rendered its full description (180px scroll box) inline, so a few posts pushed everything else off-screen. ProvenancePanel now collapses the description by default behind a per-post "Show description ▾ / Hide ▴" toggle (state keyed by provenance_id, reset when the viewed image changes). Cards stay compact — platform/date/title/meta/actions — and the operator expands only the descriptions they want. **2. Purge tags of retired/system kinds.** The IR migration left `archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`) that FC no longer creates — the tag input only makes character/fandom/series/general, and provenance + artists are their own systems now. (meta/rating were already hard-deleted by alembic 0023.) - cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts (dry_run) or deletes tags whose kind ∈ (archive, post, artist). CASCADE clears the image_tag / alias / allowlist / etc. rows. - POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview returns per-kind counts + sample names — the preview IS the verification of exactly what'll be deleted before committing). - Tag Maintenance card gets a second section: "Preview retired-kind tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)". Tests: dry-run counts by kind (general survives), commit deletes only the retired kinds (general + character survive, retired count → 0). NOTE: a dry-run preview will show exactly which kinds/counts are present. If the operator's noisy tags turn out to be `general` (e.g. an IR `source:patreon` that fell back to general during migration), they won't be caught by the kind purge — the preview makes that visible so we can decide on a name-based pass separately. |
||
|
bvandeusen
|
104cac5dca |
fix(ui): tooltip readability — dark bg + parchment text (was light-on-light)
Operator-flagged 2026-05-28: tooltips (e.g. the action buttons in Subscriptions → Subscriptions) render near-white-on-near-white, unreadable. Cause: Vuetify's default v-tooltip pairs `on-surface-variant` text with an `surface-variant` background. FC's theme deliberately maps `on-surface-variant` to vellum (#C2BFB4 — a light cream, the correct muted-text token for captions/hints on the dark page) but never defines `surface-variant`, so Vuetify auto-generates a light-ish tooltip background. Light text on light bg. Fix is tooltip-specific so it doesn't disturb the (correctly light) muted-text token elsewhere. New app-global stylesheet frontend/src/styles/app.css, imported in main.js AFTER vuetify/styles (equal-specificity rule wins by source order), overrides `.v-tooltip > .v-overlay__content` to a dark elevated panel (surface-bright = slate #2C313A) with high-contrast parchment text (#E8E4D8) + a subtle border + shadow. Applies to every tooltip in the app, so the fix is consistent rather than per-component. |
||
|
bvandeusen
|
dcfe55d731 |
feat(import-resilience L2): one-shot re-download for corrupt downloaded files
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its source, bounded to a single attempt. Operator-requested 2026-05-28. New backend/app/services/refetch_service.py: - resolve_refetch_source: parse the failed file's sidecar → platform, derive the artist from the import path, find an ENABLED Source with a real feed URL for (artist, platform). Returns None for filesystem-only imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic anchors (not pollable). - attempt_refetch: if not already refetched AND a Source resolves, delete the corrupt file (so gallery-dl's skip_existing re-fetches it), set ImportTask.refetched=True, and trigger ONE download_source re-check. Bounded by `refetched` so source-side corruption can't loop. Wiring: - Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed' tasks). Returns refetch_queued / no_source / already_refetched / not_found / not_failed. - Auto path in recover_interrupted_tasks: for each poison-pill row, if env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the manual button is the primary path; auto is opt-in since re-fetch deletes a file + re-runs the downloader). - Frontend: a cloud-refresh icon button on failed rows in ImportTaskList → stores.import.refetchTask → toast keyed on the result status. Filesystem imports with no upstream return no_source — the operator's only remediation there is replacing the file on disk, surfaced clearly in the toast. Tests: 404 unknown task, 400 non-failed task, no_source when unresolvable, and the full resolvable-source path (file deleted, refetched flag set, one download_source dispatched, second call is a no-op). The resolvable test repoints the migration-seeded import_settings(id=1) scan path rather than inserting a conflicting row. |
||
|
bvandeusen
|
e3cdd0f92b |
feat(import-resilience L3): subprocess-isolated probes for video + archive
Layer 3 — prevent the hard worker crash rather than just recovering from it. The realistic process-crash vectors (operator's observed slow/heavy tasks) are video decode and archive extraction; images decode in-process and Pillow raises-and-skips cleanly, and a subprocess per image would wreck deep-scan throughput, so images are intentionally not probed. New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so the spawned child stays light): - probe_video(path): validates the container + first video stream via ffprobe (a separate binary — a decoder crash kills only ffprobe, not the worker). Returns width/height, which the importer didn't capture for videos before. crashed=True only on ffprobe timeout. - probe_archive(path): an uncompressed-size bomb guard (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a spawned child process. A decompression-bomb OOM or native-lib segfault on a malformed archive shows up as a non-zero child exit code → crashed=True, never a dead worker. ProbeResult.crashed distinguishes a HARD failure (subprocess killed / timed out — the poison-pill signature → caller returns terminal 'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap, integrity mismatch → caller's choice of skipped/attached). Wired: - importer._import_media video branch: probe_video before the pipeline; crash → failed, clean reject → invalid_image skip, ok → capture dims. - importer._import_archive: probe_archive before extract_archive; crash → failed, clean reject → still preserve the archive as a PostAttachment (matches extract_archive's fail-soft contract). - ml.tag_and_embed video branch: probe_video before sampling 10 frames, so a corrupt video is rejected (status='bad_video') instead of crashing the ml-worker on frame decode. Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct _inspect_archive size+integrity, in-process _archive_probe_target bomb guard (monkeypatch can't reach a spawned child, so the target is called directly), and a non-video → ok=False that's robust to ffprobe presence in CI. |
||
|
bvandeusen
|
e77afe8295 |
feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
|
||
|
bvandeusen
|
57a22d6098 | fix(tests): repair test_maintenance — skips_fresh_running tail was orphaned at module scope by the inserted ml/archive sweep tests (F841/F821) | ||
|
bvandeusen
|
a85880f965 |
fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct — the timeout covered the WHOLE archive, not per object. Importer._import_archive (importer.py:409) runs the full per-member pipeline (sha256 + pHash + dedup query + copy + provenance) for EVERY media member inline, all under import_media_file's single 300s soft limit. A single media file is sub-second; a multi-hundred-member archive blows the budget. They shared one task name and one timeout. **Split archive into its own task** - New `import_archive_file` task: same body as import_media_file (dispatch is by file-kind inside Importer.import_one) but soft=30min / hard=35min. Shared `_run_import_task` helper holds the flip-to-processing + resilience-contract wrapper; both tasks call it. - New `enqueue_import(task_id, task_type)` router — single source of truth for media-vs-archive dispatch. Used by all three enqueue sites: scan_directory, /api/import/retry-failed, recover_interrupted_tasks. - scan_directory now sets ImportTask.task_type = "archive" when is_archive(entry) (the model field already existed, anticipating this; scan was hardcoding "media"). - import_archive_file routes to the existing 'import' queue via the task_routes `import_file.*` wildcard — no worker config change. **Archive-aware recovery sweeps** Both sweeps would otherwise preempt a legitimately-running archive: - recover_interrupted_tasks (ImportTask 'processing' sweep): now task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the 35-min hard limit). Single UPDATE with an OR predicate over the two (task_type, cutoff) pairs; requeue routes via enqueue_import. - recover_stalled_task_runs (TaskRun 'running' sweep): now supports per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above the per-queue overrides added for ml. import_archive_file gets 40 min while the 'import' queue stays at the 5-min default for single-file imports. Precedence: task_name → queue → default, each pass excluding rows claimed by a higher-precedence pass so every row is touched once. **Tests** - test_import_archive_file_registered - test_recover_stalled_task_runs_archive_task_uses_longer_threshold — pins that a 10-min archive task-run survives, a 50-min one is flagged, and a same-queue 10-min media import is flagged at the default. - _make_task_run gains queue= + task_name= params. After deploy: archive imports get a 30-min budget and aren't preempted by either sweep; single-file imports keep their tight 5-min detection. |
||
|
bvandeusen
|
407de18ff6 |
fix(ml): video branch needs longer time limits; recovery sweep is now per-queue
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.
Root cause is two interacting limits, both undersized for video work:
tag_and_embed: soft_time_limit=300, time_limit=420
(sized for the image branch, ≈2 GPU ops)
recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues
The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.
**Two-part fix**
1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
(20 min). Sized for the video path's worst case; image runs return
in seconds and don't care.
2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
Queues with legitimately-long-running tasks (currently just `ml` at
25 min — 5-min buffer past the new hard kill) get their own
threshold; queues not in the dict use the default 5 min. The sweep
now issues one UPDATE per distinct threshold value, with
`queue.notin_(override_queues)` on the default pass so each row is
touched at most once.
Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
pins both directions: a 10-min-old ml row survives (fresh by 25-min
override), a 30-min-old ml row gets flagged.
After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
|
||
|
bvandeusen
|
b1b129ce9f |
feat(downloads-tab): A+B dashboard improvements — row restyle + date-grouped sections with failed-pinned
Operator-flagged 2026-05-27: the Downloads subtab "doesn't feel like a dashboard" — status was a tiny mdi icon at the far left, platform chip was neutral-tonal, errors were plain orange text floating on the right, and all 28 rows from the same hour visually had the same priority. **Row restyle (A):** - 4px colored left-edge bar by status (success/error/info/warning/grey) — visually scannable at the edge without parsing the chip text - Status chip with text label (Completed/Failed/Running/Queued/Skipped) + leading icon, tonal-colored. Replaces the bare mdi-icon. - Platform chip swapped to the color-coded subscriptions/PlatformChip (Patreon=red mdi-patreon, SubscribeStar=amber, HentaiFoundry=purple, Discord=indigo, Pixiv=blue, DeviantArt=green). - File count: tonal info chip when > 0, dim middle-dot when 0 (so scheduled "no-change" scans don't dominate the column visually). - Error: red tonal pill chip with leading icon, truncated to 60 chars with full text in the title tooltip. Replaces plain text. - Per-row actions (hidden at 50% opacity, fade to full on row hover): Retry (only when status=error AND source_id known — hits POST /api/sources/<id>/check via the existing sources.checkNow), Details (opens the detail modal), Open artist (navigates to the artist page). Clicks stop-propagation so they don't bubble to the row click. **Date-grouped sections (B):** - Events are bucketed into four sections: Today / Yesterday / Last 7 days / Earlier. Empty buckets are skipped. Buckets boundaries are computed against the operator's local-time start-of-day so "Today" matches their intuition. - Each section has a collapsible header with a row-count chip + a red "failed in this section" chip when any failures are in scope. - Within each section, status='error' rows are pinned to the top (operator's eye lands on failures first; successful scans flow below). - Collapsed state persists across refresh within the SubscriptionsView lifetime (reactive object, default all-expanded). DownloadEventRow grid widened to accommodate the status chip + actions column. PolyMasonry-style ellipsis on the artist link prevents long names from breaking the layout. No new endpoints; the Retry path reuses the existing /api/sources/<id>/check flow (the source-check endpoint was already in place, just not wired into a per-row button). |
||
|
bvandeusen
|
df6d89cb59 |
fix(secure-context): full audit — DestructiveConfirmModal.expectedTokenOverride + bulk-delete + min-dim use backend-computed tokens
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.
**Audit results across `frontend/src/`:**
crypto.subtle.digest — 2 sites:
- MinDimensionCard (fixed 2026-05-27)
- BulkEditorPanel (THIS FIX)
navigator.clipboard — 1 site, already guarded:
- utils/clipboard.js writeText with execCommand fallback
serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
cookieStore / queryLocalFonts / WebAuthn / geolocation
— NOT USED, nothing to fix
Extension scripts (background.js) use crypto.subtle but run from
moz-extension:// which IS a Secure Context — left as-is.
**BulkEditorPanel double bug**
The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:
1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
`delete-images-selection-<sha8>` while the backend expected
`delete-images-<sha8>`. The two would never match.
Fix:
- Backend `/api/admin/images/bulk-delete` dry-run response now returns
`confirm_token` (the canonical `delete-images-<sha8>` string).
Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
set, it bypasses the `${action}-${kind}-${runId}` formula and uses
the explicit string. This decouples the UI label (`kind`) from the
wire-format token (server-provided), so future endpoints can use a
kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
— no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
slicing the 8-char suffix off the backend's token and passing it
through `runId`; now passes the full backend token via
`expected-token-override` directly). Cleaner; one source of truth.
**Banked memory**
`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.
No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
|
||
|
bvandeusen
|
12be188ada |
feat(showcase): IR-parity R-key shuffle + stagger entry animation; fix(cleanup): min-dim Delete swallowed crypto.subtle TypeError on plain HTTP
**showcase R-key + entry animation**
Restores two behaviors lost during the FC-2 IR→Vue port. Operator-flagged
2026-05-27.
- ShowcaseView listens for keydown 'r'/'R' on window. Triggers
`store.shuffle()`. Skips when an input/textarea/contenteditable is
focused or a Vuetify overlay is open (the dialog/menu sets
`.v-overlay--active` on the body).
- MasonryGrid gains an opt-in `animateFromIndex` prop (default
`Number.POSITIVE_INFINITY` = off). When set, items with index ≥ the
threshold animate in with a stagger fade-in: 12px translateY,
0.25s ease, 60ms per item, capped by `prefers-reduced-motion`.
Stagger uses original-items-array index (resolved via an `idxById`
Map) so the reading order is preserved even after the masonry
distributes items across columns.
- ShowcaseView watches `store.images.length`: shrink-or-zero baseline
⇒ `animateFromIndex=0` (animate everything on initial load /
shuffle); grow ⇒ baseline=prevCount (animate only the appended
tail on infinite-scroll). Other MasonryGrid consumers (ArtistView's
Gallery tab) don't pass the prop, so they keep their current
no-animation behavior.
Direct port of IR's `app/static/js/showcase.js` keyboard handler +
`app/static/style.css` itemFadeIn keyframe.
**min-dim Delete: crypto.subtle TypeError fix**
The Delete button on the Cleanup → Minimum Dimensions card was
silently no-op'ing. Root cause: `crypto.subtle` is Secure-Context-gated
(undefined on plain-HTTP origins per the homelab posture). The card's
`onDeleteClick` computed the Tier-C confirm token via
`crypto.subtle.digest('SHA-256', ...)`, which threw TypeError before
`showModal.value = true`. The promise rejected, the click handler had
no `.catch`, the modal never opened — exactly the operator's reported
symptom.
Same shape as the v26.05.26.0 `navigator.clipboard` fix on the
ErrorDetailModal Copy button.
Fix: backend `/api/cleanup/min-dimension/preview` now returns
`confirm_token` (the canonical `delete-min-dim-<sha8>` string) in its
response. Frontend reads it from the preview response and feeds the
8-char suffix to DestructiveConfirmModal's `runId` prop — no
client-side crypto needed. Single source of truth.
Integration test `test_min_dimension_preview_returns_count` pinned to
also assert `body["confirm_token"]` matches the server-side compute.
|
||
|
bvandeusen
|
6d7116c090 | fix(platforms): ruff I001 in base.py — one blank line between imports and module-level constant (was two) | ||
|
bvandeusen
|
b447c42853 | fix(platforms): ruff I001 — drop unused __future__ import; switch __init__ to per-module imports for clean isort ordering | ||
|
bvandeusen
|
abafc3265e |
refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."
**Layout**
services/platforms/
base.py PlatformInfo dataclass + module-default key
chains + shared helpers (str_id_value, str_field)
__init__.py PLATFORMS dict + public API (auth_type_for,
known_platform_keys, to_dict,
external_post_id_keys_for, description_keys_for)
patreon.py metadata only — the reference platform, no quirks
subscribestar.py metadata + augment_cookies (18+ agreement) +
derive_post_url (synthetic /posts/<post_id>)
hentaifoundry.py metadata + augment_cookies (host-only PHPSESSID
duplicate) + derive_post_url (/pictures/user/...)
pixiv.py metadata + derive_post_url (/artworks/<id>)
discord.py metadata + derive_post_url
(channels/<server>/<channel>/<message>)
deviantart.py metadata only — un-audited; quirks to be added
when an operator first exercises DA
**PlatformInfo extensions**
Existing fields preserved. Four new optional fields:
external_post_id_keys: tuple[str, ...] | None
Override the sidecar external_post_id lookup chain. None falls
back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
("post_id", "id", "index", "message_id") — covers every current
platform.
description_keys: tuple[str, ...] | None
Override the description body lookup chain. None falls back to
DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
"message") — Discord's "message" body field is covered by the
default's trailing entry.
derive_post_url: Callable[[dict], str | None] | None
Synthesize the post permalink from sidecar metadata. None = trust
the bare `url` / `post_url` field (patreon, deviantart).
subscribestar/pixiv/hf/discord override this because their `url`
is the file CDN URL.
augment_cookies: Callable[[str], str] | None
Post-process the materialized cookies.txt before gallery-dl
consumes it. None = no-op. Used by subscribestar (age cookie) and
hentaifoundry (host-only PHPSESSID duplicate).
**Consumer changes**
- credential_service._augment_cookies(platform, netscape) shrunk from a
per-platform-conditional dispatcher (~80 lines of inlined helpers) to
a 5-line lookup: `info.augment_cookies(netscape) if info and
info.augment_cookies else netscape`. The platform-specific helper
bodies moved verbatim into the per-platform modules.
- sidecar.parse_sidecar similarly delegates: external_post_id chain via
external_post_id_keys_for(category), description chain via
description_keys_for(category), post_url via
PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
and inline _derive_post_url body both gone. Added a shared `_first_id`
helper for bool-safe id coercion.
**Public API preserved**
PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.
**Adding a new platform**
1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
pick it up automatically.
|
||
|
bvandeusen
|
2394e47370 |
fix(hentaifoundry): inject host-only PHPSESSID/CSRF duplicates + extension preserves browser hostOnly
Operator-flagged 2026-05-27: HF source check 401'd on
`HEAD /?enterAgree=1` even with valid login cookies. Root cause is the
combination of (1) gallery-dl's HF extractor checking
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching, and (2) the extension's cookies.js
forcibly rewriting every captured cookie to a leading-dot subdomain-wide
form. HF's PHPSESSID is browser-stored as host-only on
`www.hentai-foundry.com`; the rewrite re-anchored it to
`.hentai-foundry.com`, which `cookies.get(...)` no longer matches even
though the cookie is still sent on actual HTTP requests (RFC 6265
subdomain rules). The extractor falls into its unauthenticated
`?enterAgree=1` fallback, which 401s (Cloudflare or HF's anti-bot HEAD
gating).
Two-part fix, no operator action required for existing stored cookies:
1. **Backend** (`credential_service._augment_cookies`) — refactored from
the subscribestar-only single function into a per-platform dispatcher.
New `_augment_hentaifoundry` parses the materialized netscape file
and, for each `.hentai-foundry.com` entry whose name is PHPSESSID or
YII_CSRF_TOKEN, appends a host-only duplicate
(`www.hentai-foundry.com\tFALSE\t...`). Originals preserved. Three
new tests pin: injection fires + originals preserved; idempotent
when host-only already exists; doesn't touch unrelated cookies
(e.g. `_ga`).
2. **Extension** (`cookies.js`) — `toNetscapeFormat` now respects
`c.hostOnly` from the browser instead of blindly forcing a
leading-dot subdomain-wide form. Host-only cookies are written with
the bare host + FALSE flag; non-host-only cookies retain the
leading-dot + TRUE form. Forward-compat — fresh captures from
v1.0.5+ no longer need the backend's host-only duplication.
Extension bumped 1.0.4 → 1.0.5; manifest + package.json in lockstep.
After deploy: the next HF source check on the operator's already-stored
cookies will succeed because the materialized cookies.txt now contains
host-only PHPSESSID. No browser re-export needed.
|
||
|
bvandeusen
|
8243740a04 |
fix(subscribestar): inject 18_plus_agreement_generic age cookie to bypass server gate
Operator-flagged 2026-05-27: subscribestar source check aborted with `AbortExtraction: HTTP redirect to .../age_confirmation_warning`. The captured `_personalization_id` cookie in the browser-stored file had expired (annual rotation), and the user could not realistically refresh it: SubscribeStar's frontend JS uses localStorage to suppress the age-confirmation popup once dismissed, so a logged-in revisit doesn't re-show the popup and the server-side cookie is never re-issued. gallery-dl's own login flow (which FC doesn't exercise — cookies come from the extension instead) sidesteps this by manually setting `18_plus_agreement_generic=true` on `.subscribestar.adult`. The server accepts that as the age-confirmation marker. `credential_service._augment_cookies(platform, netscape)` mirrors that behavior: when the materialized cookies file is for subscribestar and the age cookie isn't already present, append a synthetic line for `.subscribestar.adult` with name=`18_plus_agreement_generic` value=`true` and a far-future expiry. No-op for other platforms; no-op if the cookie is already present (idempotent for manual pastes / extension captures that happen to include it). Three new tests pin: (a) injection fires for subscribestar, preserves existing cookies; (b) idempotent when already present (no double injection); (c) does NOT fire for non-subscribestar platforms (Patreon etc. don't get a foreign-domain cookie). Not a curator handling bug per se — the extension faithfully captured what the browser had. This is mirroring a documented gallery-dl workaround so the cookies-via-extension auth path doesn't degrade as the server-side cookie expires. |
||
|
bvandeusen
|
aa28bddeab | fix(alembic 0025): qualify ambiguous post.id / post.source_id in fragment-group SELECT (post JOIN source — both have id) | ||
|
bvandeusen
|
b7b313cc05 |
fix(alembic 0025): include HF + Discord post_url backfill (no longer 'deferred to deep-scan')
Operator-flagged: the claim that 'a future deep-scan via the new parser
will fix HF and Discord post_url' was conditional on the operator
actually running a deep-scan, which they might not do for ages. Until
then HF posts stay at post_url=NULL (HF sidecars have no `url` field)
and Discord posts stay pointing at cdn.discordapp.com/attachments/...
(the file URL, not the message permalink).
The migration was already opening sidecar files for SubscribeStar.
Generalizing the loop to also handle HF and Discord is a tiny addition
that closes the gap without operator intervention.
Per-platform Part 1 logic now:
subscribestar — read sidecar.post_id, overwrite external_post_id +
post_url with the derived /posts/<post_id> permalink.
hentaifoundry — read sidecar.user + .index, overwrite post_url with
/pictures/user/<u>/<i>. external_post_id (= index) unchanged.
discord — read sidecar.server_id + .channel_id + .message_id,
overwrite post_url with the discord.com/channels/.../<m> triple.
external_post_id (= message_id) unchanged.
Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv
post_url backfill) unchanged.
Posts whose related ImageRecord paths don't resolve on disk (orphan
filesystem state) are reported per-platform in the migration output —
those still need a future deep-scan, but the in-DB-with-on-disk-files
common case is now fully covered by the migration alone.
|
||
|
bvandeusen
|
bd3f996582 |
fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
|
||
|
bvandeusen
|
ae8c78ae09 |
fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar)
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading sentence inside `content` HTML. Confirmed against the operator's /mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every post's JSON has `title: ""` and a content like `<div>Lets say hello to you guys with my Belle <br><br><br></div>`. FC's sidecar parser, treating empty strings as missing, had been leaving post_title NULL on every subscribestar post since FC-3 shipped. Fix at two layers: 1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)` helper strips HTML tags, collapses whitespace, returns the first non-empty line truncated to 120 chars with ellipsis. `parse_sidecar` now falls back to this when `title` resolves to None and a `content`/`description`/`caption` value is present. Patreon's non-empty titles short-circuit the fallback so existing behavior is unchanged. Four new tests in test_sidecar_util.py pin: derivation from content, truncation at 120 chars, explicit-title precedence, no-content no-fallback. 2. `alembic 0024_backfill_post_title_from_description` — backfills the same logic across existing Post rows where `post_title IS NULL OR post_title = ''` AND description is present. Idempotent (re-running is a no-op once titles are populated). Downgrade is a no-op since there's no safe way to tell derived rows from genuine ones. After deploy + migration: subscribestar posts will surface a meaningful title in PostCard, post feed search, etc. |
||
|
bvandeusen
|
4d2c464045 |
feat(post-card): absorb PostModal into PostCard with click-to-expand
PostCard and PostModal competed for the same data and rendered redundant chrome (header twice, image grid twice, attachment list twice). The wider PostCard layout we shipped 2026-05-27 has enough real estate to be the canonical post surface, so collapse the two into one. Compact (default) state is unchanged: hero + 3-cell rail + truncated title + 3/5-line description + attachment count badge. Whole-card click expands in place. Expanded state shows: full title, mosaic of ALL post images via PostImageGrid (uncapped, lazy-loaded via getPostFull), full sanitized-HTML description with paragraph wrapping, attachments as downloadable pill links. Click the chevron in the header to collapse; mosaic image clicks open ImageViewer scoped to the post (modalStore's postImageIds path is preserved — only the comment changed). Per-card local state — no global modal store. Each PostCard owns its own expanded ref and lazy-loaded detail; collapsing a card discards neither (so re-expand is instant after the first fetch). Deleted: PostModal.vue, postModal.js store. Removed the App.vue mount. |
||
|
bvandeusen
|
b8ad17c68d |
fix(build): poll for ext-<version> release in tag-push build-web (race fix)
Cutting a release fires BOTH the push-to-main workflow AND the push-to-tag workflow in parallel. main-push runs sign-extension (AMO round-trip 1-5min) then publishes the ext-<version> Forgejo release; tag-push skips sign-extension (gated to main) and races straight to build-web's Download XPI step. Tag-push lost every time — got 404 from releases/tags/ext-<version> before main-push had finished signing. v26.05.27.0 hit this: tag-push build-web died on exit 22 because the ext-1.0.4 release wasn't published yet (it arrived ~4min later). Fix: wrap the release lookup in a 20-iteration sleep+retry loop, 30s between attempts (10min total upper bound, generous for AMO). main-push's signing eventually publishes the release; tag-push picks it up on a later poll. No more manual rerun of the failed job after every release cut. Banked the trap as reference_tag_push_main_push_race.md — same shape will recur any time a tag-push workflow consumes a main-push-produced artifact. |
||
|
bvandeusen
|
1fd54897d8 | fix(api): ruff UP017 — use datetime.UTC alias in /api/downloads/stats |