606 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusen	64ca858574	Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main	2026-06-02 08:26:58 -04:00
bvandeusen	91be9df671	fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount ... attach_in_place mirrored only the media flow, so gallery-dl-downloaded zips/PDFs/audio bounced back as `skipped+invalid_image`, which download_service counted as an ingest error and flipped runs to status="error" despite N successful image attaches. Lustria patreon event #38998 (21 images + 1 OST zip) went red for exactly this reason. Now attach_in_place dispatches the same way as import_one: archives → _import_archive (extracts media members, captures archive as PostAttachment), non-media → _capture_attachment. Download_service accepts the new `attached` result and treats non-duplicate skips as soft skips, not ingest errors. Also: ShowcaseView always loadInitial() on mount, not just when the store is empty — Pinia persists across navigations and operator wants a fresh shuffle every time the showcase loads.	2026-06-02 08:16:13 -04:00
bvandeusen	412edec028	fix(showcase): don't exhaust on all-dupe batches; retry up to 8x ... /api/showcase returns a random sample; after enough scrolling, an unlucky 3-item batch can be entirely in the `seen` Set, which was flipping `exhausted=true` and surfacing "End." mid-scroll. The showcase is endless by intent — only a genuinely empty API response (library has 0 images) should mark it exhausted. Tiny-library fallback: cap retries at 8 to avoid spinning forever.	2026-06-01 23:37:21 -04:00
bvandeusen	937421485d	fix(modal): refresh tag chips after accepting a suggestion ... Operator-flagged 2026-06-01: clicking Accept on a suggestion added the tag to the database but the modal's chip rail kept showing the old set. Suggestion would disappear from the suggestions list (suggestionsStore drops it from byCategory) and the toast would confirm "Tagged: …", but visually the modal looked unchanged until you closed and reopened it. Root cause: useSuggestionsStore.accept() POSTs to the backend and updates its own state, but never tells useModalStore that the backing image's tag set has changed. The addExistingTag and createAndAdd flows in TagPanel already call modal.reloadTags() after applying — the suggestions path needed the same refresh. Two-line fix in SuggestionsPanel: after store.accept() and store.aliasAccept(), call modal.reloadTags() so TagPanel's `modal.current?.tags` rebinds.	2026-06-01 23:02:57 -04:00
bvandeusen	9d0c0b7da8	Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main	2026-06-01 22:34:38 -04:00
bvandeusen	43b778aa04	fix(test): include 'scanned' in all backfill result assertions ... Same 'change shared shape, miss pinned tests' trap as cfa4fb4 — the prior commit updated only test_backfill_mixed_aggregate; three sibling tests in the same file also pinned the old {enqueued,ok,regenerated} shape without 'scanned' and CI bounced.	2026-06-01 22:09:02 -04:00
bvandeusen	9cbdb70e13	fix(thumbnails): surface backfill results + tighten validity check ... Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails but triggering backfill found nothing." 1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill` returned just `{celery_task_id}` and the admin card said "Enqueued." with no counts. There was no way to tell whether the scan found 0 candidates, 5000 candidates, or whether the worker even picked up the task. "Found nothing" was indistinguishable from a broken queue. Fix: refactor the scan into a sync helper (`_run_backfill_scan`) shared by the Celery task and the API endpoint. The API now runs the scan in an executor and returns `{scanned, enqueued, ok, regenerated}`. The actual thumbnail generation work still goes to the thumbnail Celery queue per row via `generate_thumbnail.delay()` — the scan itself is fast (SELECT id+thumbnail_path + a file.stat() per row). 2. **`_thumb_is_valid` accepted header-only corrupt files.** The magic-byte check passed for any 8-byte file starting with a JPEG or PNG header, including empty/truncated/zero-pad files that browsers render as broken. Backfill counted these as `ok` and never regenerated. Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real thumbnails are minimum ~2KB even on solid-color sources; header- only corrupt files top out around 12 bytes. 256 is well above the corrupt floor and well below any legitimate thumbnail. Plus the admin card now shows the per-run counts instead of "Enqueued.": Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok	2026-06-01 21:57:29 -04:00
bvandeusen	8e4d252ae4	Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main	2026-06-01 21:52:42 -04:00
bvandeusen	bd06794647	fix(downloads): enqueue thumbnail + ML tasks per attached image ... Operator-flagged 2026-06-01: downloaded images stayed at thumbnail_path=NULL until a periodic backfill sweep picked them up, surfacing as broken-thumbnail tiles in the gallery for hours after the download landed. Importer.attach_in_place deliberately skips inline thumbnail generation (importer.py:591-592) so the import queue stays moving — the CALLING task is responsible for the enqueue. tasks/import_file.py already did this (line 228-239). tasks/download.py / download_service did not — every gallery-dl-attached image landed un-thumbnailed. Fix in download_service._phase3_persist: after each `attach_in_place` returning status in (imported, superseded), fan out `generate_thumbnail.delay()` + `tag_and_embed.delay()` for each image_id. Lazy import avoids circular-import risk between download_service and the celery task modules that depend on it. Mirrors the existing pattern verbatim — single source of truth for "what fires after a successful attach" remains a comment in two places (filesystem-import task, download orchestrator) rather than a shared helper, because the contexts differ enough (sync session vs async orchestrator) that abstracting would obscure more than it'd share. Test covers the happy-path with two attached files: both get the thumbnail enqueue AND the ML enqueue, with image_ids drawn from ImportResult (so future supersede-on-attach paths stay covered).	2026-06-01 21:39:17 -04:00
bvandeusen	fdd3e01f56	Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main	2026-06-01 21:09:28 -04:00
bvandeusen	f575cfb93b	ux(failing-sources): visible row separators + clearer hover ... The zebra striping in 717b601 was too subtle — inside the tonal error card, surface-tint contrast washed out and rows still looked uniform. Operator-flagged the same issue twice. Replace the surface-tint approach with hard visual structure: - Bottom border on each row (1px white/0.08) — clearly delineated rows regardless of card background - Hover darkens to black/0.25 — much more obvious than the previous error-tint hover, gives the eye a clear horizontal track - Slightly more vertical padding (8px vs 6px) for tap-target comfort - Drop the gap; borders are the separator now	2026-06-01 20:57:41 -04:00
bvandeusen	c82fb308b6	Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main	2026-06-01 20:44:23 -04:00
bvandeusen	717b601c81	ux(failing-sources): zebra striping + hover highlight on rows ... Operator-flagged 2026-06-01: with the failing-sources panel expanded, all rows have the same flat low-opacity background, no visual track from the artist name on the left to the Logs/Retry buttons on the right. Hard to tell which row a button belongs to when scanning down a list of 17. Three pure-CSS changes (no DOM, no logic): - 3px gap between rows (was 2px) — slightly more breathing room - Even rows get a darker surface tint (zebra striping) - Hover paints the whole row in a low-opacity error tint, so moving the cursor toward Logs/Retry lights up the whole horizontal band and the eye traces back to the artist name automatically	2026-06-01 20:14:48 -04:00
bvandeusen	cfa4fb4084	fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create ... `test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial value (0) when checking that the override works. Now that create() pre-arms enabled sources to 3, the assertion is stale — the test was already verifying the override path, the pre-assertion just decorated it. Drop the pre-assertion; keep the override check. Other tests use raw Source(...) constructors (default to 0 via the column server_default), not SourceService.create(), so they're unaffected.	2026-06-01 20:12:55 -04:00
bvandeusen	2aa2002f22	feat(subscriptions): newly added enabled sources start in backfill mode ... A freshly created subscription has no gallery-dl archive yet, so the first poll in tick mode would walk forever — exit:20 doesn't trip until 20 contiguous archive-hits, and there are none. The wall-clock cap kicks in mid-walk, and the partial-success classifier (plan #544) gracefully labels it status=ok, but the operator still wonders why their new subscription isn't grabbing the back-catalog. Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`. Same default as the manual "Deep scan" button, same auto-decrement and auto-reset rules — once the queue drains (clean exit + zero new files) or the budget runs out, tick mode resumes naturally. Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'` that arrive disabled = False, or operator-added sources that start disabled deliberately) skip the pre-arm — they're never polled, no budget to waste.	2026-06-01 20:02:19 -04:00
bvandeusen	66ff671f09	fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos ... Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts hosted on Mux carry a JWT that encodes a playback restriction policy (`playback_restriction_id` in the token). The token signature alone is not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch. gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns 200 (token is valid, HEAD sends no Referer that Mux's policy rejects), but when yt-dlp follows up with the actual manifest GET it sends its own default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the single video fails — every other image in the same post still downloads. Static `downloader.ytdl.raw-options.http_headers` block now pins Referer and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch clears the policy check. Headers-only restrictions are now handled. Mux IP-range restrictions (if a creator opts into them) remain unfixable from our worker — those would need a Patreon-region IP. Per-video PARTIAL classifier (shipped in plan #544 last commit) already handles the residual case: a run that grabs all images and fails 1 video classifies as status=ok rather than flipping the whole event red.	2026-06-01 19:00:50 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00
bvandeusen	c9089b1d03	fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source ... gallery_service._provenance_clause artist branch was correlating its bare Post reference to the outer query's primary_post_id outer-join, so the artist filter silently matched zero rows for images with no primary post. Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner FROM rather than treating it as a correlated outer table. Five sidecar/import tests still asserted that a synthetic Source row appears after a filesystem import. Alembic 0030 retired that behavior; the Post sits null-source and the artist linkage lives on Post.artist_id. Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent, test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar, and test_apply_sidecar_recovers_from_integrity_error to assert post.source_id IS NULL + post.artist_id linkage instead.	2026-06-01 14:40:28 -04:00
bvandeusen	644d538bab	fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention	2026-06-01 14:23:18 -04:00
bvandeusen	ff35da4743	fix(lint): drop unused Source import after Post.artist_id refactor	2026-06-01 14:19:37 -04:00
bvandeusen	2f66de2928	feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics ... Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.	2026-06-01 14:17:52 -04:00
bvandeusen	8cf8d2ca4d	Merge pull request 'Modal Esc/overflow polish, artist-scoped post scroll, failing-sources Logs button' (#43) from dev into main	2026-06-01 12:10:11 -04:00
bvandeusen	94e7d20792	feat(downloads): Logs button on failing-sources rollup rows ... Operator-asked 2026-06-01: each row in the "X sources are failing" rollup needs a button to surface the last run's stdout/stderr/error without leaving the Downloads tab to find the matching event row. Wired: - `downloadsStore.loadLastForSource(sourceId)` two-steps via `GET /api/downloads?source_id=N&limit=1` (most recent event) then the existing `loadOne(id)` for the full detail payload. Returns null if no events exist (toast warns). - `FailingSourcesCard` row: new text button `Logs` with `mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`, emits `view-logs` with the source record. - `DownloadsTab.onViewFailingLogs` is the handler — same `DownloadDetailModal` instance the event-row clicks use.	2026-06-01 10:04:22 -04:00
bvandeusen	fb605af959	fix(posts): anchored view scroll inherits artist_id/platform filters ... Operator-flagged 2026-06-01: clicking a Provenance post title from the view modal opens the post in the posts feed scoped to that artist (`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll loaded UNFILTERED global posts instead of staying in the artist's stream. Root cause: the posts store's loadAround/loadOlder/loadNewer sent only `{around, cursor, direction}` — never the `artist_id` / `platform` filters. Backend `/api/posts` accepts them on every path (post_feed_service.scroll filters via `Source.artist_id`); the frontend just wasn't passing them. Fix: - `posts.js` `loadAround(postId, filters)` now takes the filter snapshot and stores it. `_aroundParams(extra)` mixes the active filters into around/cursor calls. - `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` + `platform` from the route query.	2026-06-01 08:24:43 -04:00
bvandeusen	4c56cf121f	fix(modal): Esc closes from tag input; Provenance cards scroll at ~2.5 ... Two operator-flagged UX gaps from 2026-06-01: 1. **Esc trapped inside the tag-entry field** The autofocused TagAutocomplete input made Esc-to-close unreachable because ImageViewer's keydown handler bailed early on `isTextEntry(ev.target)` for every key including Escape. Now Escape closes the modal from text inputs too — except when a Vuetify overlay is open (FandomPicker, autocomplete dropdown, suggestion 3-dot menu), in which case that overlay's own Esc-handling fires instead of closing the whole modal mid-interaction. Detected via `.v-overlay--active`. Arrow keys still gate on isTextEntry so the tag input handles typing without navigating images. 2. **Provenance cards expanded the side panel unbounded** When an image had many ImageProvenance entries the cards stack pushed the Tags section below the fold. Wrapped the cards in a `.fc-prov__cards` container with max-height 270px (≈2.5 cards) and thin overflow scrollbar. Title stays anchored at top; Tags panel sits at a consistent position below regardless of provenance count.	2026-06-01 08:17:54 -04:00
bvandeusen	b1d58bc3b8	Merge pull request 'fix(ci): POSIX-safe SHORT_SHA in build.yml' (#42) from dev into main	2026-06-01 08:03:17 -04:00
bvandeusen	9564d073b9	fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash) ... `${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner executes the step via dash/BusyBox sh and errored with `Bad substitution` (act_runner workflow shell, observed on the 65386f0 main-push run). Switched to `SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in both shells. Both build-web and build-ml tag-determination steps updated.	2026-06-01 07:53:33 -04:00
bvandeusen	65386f02a0	Merge pull request 'View modal batch: autofocus, suggestions UX, post-title click, retire copyright/artist' (#41) from dev into main	2026-06-01 07:01:42 -04:00
bvandeusen	f87a06a6bd	feat(modal): post title is the primary click (artist-scoped) + suggestion rows look like buttons ... Two operator-asked modal UX changes (Scribe plan #514): 1. **Post title click → artist-scoped posts feed** - The bold post title in ProvenancePanel is now the primary click target. Click navigates to /posts?post_id=N&artist_id=A; the PostsView already filters on `artist_id`, so the user lands in that creator's stream, not the global one. - The redundant "View post" link is removed. "Show description" stays as the only action link below the meta line. - Title is styled as a button-link: accent color, hover underline, focus ring for keyboard nav (family rule 24 — UI quality bar). 2. **Suggestion rows look like buttons** - SuggestionItem becomes a chip-card: visible border, hover/focus background, unified container for name + score + actions (operator's "nothing visual to unify the buttons to their object" complaint). - Accept is an explicit tonal pill button labeled "Accept" (operator's pick over whole-row-clickable). 3-dot menu retains alias/dismiss, now `variant="outlined"` so it reads as a button. - Score is a fixed-width monospace pill on the right. - "+ new" badge upgraded to a pill chip with accent border so it's visibly an annotation, not part of the tag name.	2026-06-01 02:21:59 -04:00
bvandeusen	5d284aae9f	fix(test): unpin general-threshold test from old 0.95 default (alembic 0029)	2026-06-01 02:18:54 -04:00
bvandeusen	af7b5c95e9	feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories ... Four coupled operator-asked changes to the view modal (Scribe plan #509): 1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus in onMounted/nextTick so the caret is in the input the moment the modal renders. No click needed to start typing. 2. **General suggestions expanded by default** — SuggestionsPanel's general-category group now mounts with `:default-open="true"`. Operator can collapse if too noisy, but the v1 frame shows them. 3. **Lower general threshold default 0.95 → 0.50** — MLSettings. suggestion_threshold_general default matches character. Alembic 0029 also bumps the existing singleton row's value if it's still at the old 0.95. Operator can re-tune from Settings → ML. 4. **Retire `copyright` + `artist` as ML suggestion categories** — neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never really existed as a copyright tag-kind). They were surfaced in the suggestions pipeline + threshold settings UI but had no follow- through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for, ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029 drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS, SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and MLThresholdSliders rows. Out of scope (intentional): `tag_kind` Postgres enum still includes `artist` for historic Tag row queryability (per the model comment); no operator pain reported, no enum-shrink needed. Tests: - test_surfaced_categories asserts {character, general}, excludes artist + copyright. - test_threshold_for_artist_is_unsurfaced extended to cover copyright. - test_get_and_patch_settings asserts new 0.50 default and the absent artist + copyright keys in the GET payload.	2026-06-01 02:08:10 -04:00
bvandeusen	667b05f14e	Merge pull request 'Extension probe-and-add (v1.0.6) + per-commit image tags' (#40) from dev into main ext-1.0.6	2026-06-01 01:44:03 -04:00
bvandeusen	8de7ccd07d	build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46 ... Adds an immutable per-commit docker tag to every main-push build: `git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`, alongside the existing floating `:main` + `:latest`. Implements the new family release-posture rule "Tags are milestones, not gates — commit-SHA images are the rollback unit" so rollback to any commit on main is `docker pull …:c-<sha>` with no release ceremony required. Behavior change summary: - main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>} - tag-push (opt-in vYY.MM.DD only, no .N): unchanged - safety-net dev: unchanged No code changes; the rule is about how the tag list is constructed. Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still fire them when the operator wants a labeled checkpoint.	2026-06-01 01:28:55 -04:00
bvandeusen	d65f0b2091	feat(extension): probe shows current state before click; v1.0.6 ... Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup): "the add source/subscription button idea to the firefox extension so it can tell me if a source/artist is added or not and offer an option to add it if it isn't." Plan tracked in Scribe task #507. ## Backend - `ExtensionService.probe(url)` — read-only resolution. Reuses `_derive` for platform+slug, then 2 SELECTs. Returns one of: - `source_match` (exact (artist, platform, url) Source exists) - `artist_match` (artist exists, this URL isn't a Source yet; collapses the sidecar-synthetic-only case from v26.06.01.0) - `new` (neither exists) - `unknown_platform` (URL didn't match any artist-page regex) - `GET /api/extension/probe?url=...` route with `X-Extension-Key` auth posture matching `/quick-add-source`. Read-only, side-effect free. - 6 backend tests in tests/test_api_extension.py covering each state + auth + invalid URL. ## Extension - `api.js`: `probeSource(url)` mirroring `quickAddSource` shape. - `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The latter strips the `/api` suffix from configured `apiUrl` (placeholder format per options.html) and opens `${base}/artist/{slug}` in a new tab via `browser.tabs.create`. - `content-script.js`: probe-first render — on page-load and SPA navigation, asks the backend for the URL's state and renders the chip in the matching color/copy on FIRST paint instead of flashing generic "Add" and updating after. Click handler branches: `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing ADD_AS_SOURCE flow (then re-probes so the chip flips green immediately, no wait for next nav). - `content-script.css`: three state-color modifiers (--new, --artist-match, --source-match) on the FC parchment-on-slate palette. Sage for already-added, amber for artist-exists, accent orange for new. ## Versioning - `extension/manifest.json` + `extension/package.json` → 1.0.6. build.yml's sign-extension job will fire on push to main since no `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the regenerated AMO keys end-to-end. ## Behavior on the sidecar-synthetic case Filesystem-imported "Dymkens"-style artist with only a sidecar synthetic Source: probe returns `artist_match` (not `new`), so the chip reads "+ Add Patreon source to Dymkens" rather than offering to recreate the artist. Clicking adds the real Source; existing `_source_for_sidecar` preference logic (v26.06.01.0) routes future gallery-dl Posts to the real one.	2026-06-01 00:41:17 -04:00
bvandeusen	856e9104b4	Merge pull request 'Sidecar synthetic anchor cleanup + tier-gated classifier fix' (#39) from dev into main v26.06.01.0	2026-06-01 00:16:58 -04:00
bvandeusen	66f19d67f5	fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp ... Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED	2026-05-31 23:30:39 -04:00
bvandeusen	6fc8ae3106	fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup ... Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing "phantom" subscriptions like `sidecar:patreon:dpmaker`: 1. `SourceService.list()` returned every Source, no filter on URL. alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into one canonical row per (artist, platform); when no real campaign URL was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>` enabled=false as a disabled anchor. The UI then listed those anchors as if they were polls — disabled, but visible. Fix: `list()` excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True` opts back in for admin tooling. 2. `importer._source_for_sidecar` picked the lowest-id Source for (artist, platform). When alembic 0022 had rewritten a per-post row into a synthetic anchor (lower id) AND the operator later added the real subscription (higher id), every gallery-dl download silently attached its Post to the SYNTHETIC instead of the real Source. Fix: prefer a non-`sidecar:%` URL when one exists; fall back to the synthetic; only create a new synthetic when nothing exists for (artist, platform). alembic 0028 is the data half: for every (artist, platform) with both a synthetic AND a real Source, pre-merge Post+ImageProvenance collisions on the canonical, bulk-repoint Posts/ImageProvenance/ DownloadEvent.source_id onto the real Source, and delete the synthetic. Lone synthetics (no real twin) are left intact — they anchor real imported content the operator may still want; the list-filter hides them so they no longer surface as phantoms.	2026-05-31 23:08:38 -04:00
bvandeusen	0397642b21	Merge pull request 'Showcase cadence tuning + cooldown-aware bulk retry' (#38) from dev into main	2026-05-30 23:50:36 -04:00
bvandeusen	a5101494b6	feat(downloads): bulk retry respects cooldown; single-source RETRY overrides ... Today's platform-cooldown commit (61ce1ce) only filtered the scan tick — manual /api/sources/<id>/check still bypassed it. Operator-flagged 2026-05-30: clicked "Retry failed" on a Patreon failure pile and saw every one go 'queued' without realising the cooldown wasn't in the loop. Bulk retry with N sources on a cooled-down platform bowls right back into the rate limit the cooldown is trying to prevent. **Backend (`/api/sources/<id>/check`):** - Reads optional `?force=true` query flag. - Without force: queries `active_platform_cooldowns` (renamed from the private `_platforms_in_cooldown` since it's now a cross-module API). If the source's platform is in cooldown, returns **202** with `{status: 'deferred', platform, cooldown_until}` — no event created, no dispatch. - With force: cooldown skipped entirely. - In-flight guard always applies (no point creating duplicate pendings). **Frontend (`sourcesStore.checkNow(id, {force=false})`):** new optional `force` flag → adds `?force=true` to the URL. **Frontend (`DownloadsTab`):** - `onRetrySource` (single-source RETRY click): passes `force: true` → explicit operator override, useful for rapid auth-fix testing. - `onRetryAll` (RETRY ALL + MaintenanceMenu "Retry failed"): no force → cooldown respected. Tallies `deferred` alongside `queued` / `already_running`; toast reads e.g. *"5 queued, 12 deferred (cooldown), 3 already running"*. That count is the operator's diagnostic answer for "is rate-limit the cause of most failures?" (12-of-20 deferred → yes; 0 deferred → no). **Auto-resume:** no new sweep needed. Deferred sources still have stale `last_checked_at`, so the next scan tick after the cooldown AppSetting expires picks them up via `select_due_sources` (which already filters on `active_platform_cooldowns`). Tests: two new — deferred-on-cooldown returns 202 with the right body and no dispatch; force=true overrides the cooldown and creates the event normally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 23:15:07 -04:00
bvandeusen	e3a7aff7a3	ux(showcase): pipeline fetches in-order, chunk size 3, keep the trickle ... Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast but could let later chunks arrive ahead of earlier ones — even when each chunk is a random sample, that "later chunk loads first" risk made the load order non-deterministic. And the original goal behind asking for batching was a faster first-image-on-screen, which neither sequen- tial nor parallel really addressed cleanly. Switched the loadInitial flow to a PIPELINE: - Only one fetch in flight at any moment (in-order arrival, no race). - The NEXT fetch kicks off as soon as the current one resolves (NOT after its trickle finishes), so the next RTT overlaps the visible trickle window — round-trips are hidden behind the animation cadence. - PAGE 5 → 3 + INITIAL_BATCHES 12 → 20 (total still 60). Smaller chunk → first chunk's items appear sooner (a chunk of 3 trickles in 240ms, well within one RTT, so by the time chunk 2 is in-hand the first trickle is just finishing). Trickle, sequence-token guard, and infinite-scroll behaviour unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 22:52:55 -04:00
bvandeusen	9cd6d09e60	ux(showcase): smooth one-at-a-time cadence + earlier infinite-scroll trigger ... Operator-flagged 2026-05-30 (round 2): the batched-loads change improved consistency but still felt "chunky" — 5 items appeared together, then a ~200 ms API round-trip pause, then another 5. And infinite-scroll fired too late when a single tall image (long manga page) made one masonry column much taller than its siblings. **Cadence (showcase store):** - Fire all INITIAL_BATCHES fetches in PARALLEL — collapses the per- batch round-trip gap so all the data arrives in ~1 RTT instead of 12 sequential RTTs. - Trickle each response's items into images.value one at a time with APPEND_DELAY_MS = 80ms between each (≈ the MasonryGrid stagger animation, 70ms). User sees a smooth steady stream. - fetchPage (the infinite-scroll path) uses the same trickle so its 5-item appends also cascade one-by-one instead of popping together. - Sequence token guards against a fast shuffle / mount-then-shuffle interleaving two trickles into the same images.value. - Dropped useAsyncAction here — the parallel-fetch-then-trickle flow doesn't fit its single-wrap-call shape cleanly; inline loading/error state is clearer. **Infinite-scroll trigger (MasonryGrid):** - Pass `rootMargin: '2400px'` to useInfiniteScroll (was the 600px default). The masonry sentinel sits at the bottom of the container, whose height = MAX(column heights). A tall image in one column pushes the sentinel ~2× viewport below where the user is actually reading (the bottom of the SHORTER columns). 2400px ≈ 2-3 screen-heights of pre-emptive trigger, comfortable for typical tall manga heights. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 22:48:58 -04:00
bvandeusen	237575447d	Merge pull request 'Thumbnail URL fix + archive daemon fix + batched initial loads' (#37) from dev into main	2026-05-30 22:01:43 -04:00
bvandeusen	810baf63ac	fix(import): archive probe → subprocess (was multiprocessing.Process) ... Every archive import was failing immediately with "AssertionError: daemonic processes are not allowed to have children" (operator-flagged 2026-05-30 — import_archive_file crashes in 49ms with the assertion). Celery's prefork pool runs tasks in daemon processes; Python's multiprocessing module refuses to let daemons spawn children, which is exactly what probe_archive was doing via mp.get_context("spawn") .Process. The Layer-3 crash-isolation feature added 2026-05-28 was effectively a hard-blocker on the very import path it was meant to protect. Switched probe_archive to subprocess.run — no daemon restriction, still isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill the worker). The probe body lifted to a tiny runner module (_archive_probe_runner) that imports the unchanged _run_probe helper and prints a single JSON line; parent parses stdout, returns the ProbeResult exactly as before (timeout, signal, OOM, clean-rejection, ok — all preserved). cwd for the subprocess is the repo root derived from __file__ parents so `python -m backend.app.utils._archive_probe_runner` resolves both in the Celery container and pytest, regardless of where the worker was launched. Test refactor: test_archive_probe_target_bomb_guard → test_run_probe_bomb_guard. Same in-process call to the (renamed) probe body so the monkeypatched cap still takes effect; the real subprocess path is exercised by the existing test_probe_archive_valid_zip / test_probe_archive_corrupt_zip_clean_rejection tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 21:48:51 -04:00
bvandeusen	44bb12a93d	fix(thumbnails): derive URL from stored thumbnail_path, not (sha256, mime) ... The showcase/gallery/artist/series/post-feed APIs were constructing thumbnail URLs from (sha256, mime). The MIME-based extension predicate ("png if image/png or image/gif else jpg") DISAGREED with the thumbnailer's actual on-disk extension predicate ("png if alpha else jpg"). Result: every PNG source without transparency 404'd (URL asked .png, disk had .jpg); every WebP/AVIF source with transparency 404'd (URL asked .jpg, disk had .png) — despite the thumbnail file existing on disk. The backfill task couldn't catch these because backfill checks the ACTUAL thumbnail_path stored on the record (correct), not the URL the browser fetches (broken derivation). So records with valid on-disk thumbnails kept showing as broken in the UI no matter how many times backfill ran. Operator-flagged 2026-05-30: "the generate thumbnails function appears to not catch all of the failed thumbnail cases" — turned out to not be a backfill bug at all. Fix: thumbnail_url now takes (thumbnail_path, sha256, mime) and returns the stored path verbatim — Quart serves /images/* 1:1 from the volume (frontend.py:20-36), so the URL IS the disk path. Falls back to the old sha256+mime derivation only when thumbnail_path is NULL (thumbnailer hasn't run yet); that URL will 404 in the browser until backfill catches it, same as before the path was tracked. All 8 callers updated: showcase_service, gallery_service (2 sites), artist_service, series_service, post_feed_service, tag_directory_service, artist_directory_service. The four sites whose query was raw-tuple now also SELECT ImageRecord.thumbnail_path. Net effect: every record that has a valid on-disk thumbnail will now render correctly, regardless of which extension the thumbnailer chose, without any DB migration or backfill rerun needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 16:55:38 -04:00
bvandeusen	1eefed9ab3	fix(ui): finish showcase store batching (PAGE 60 → 5, INITIAL_BATCHES 12) ... The showcase-store change in adeee64 didn't actually land — only gallery.js + ShowcaseView.vue made it into the commit. Without this, ShowcaseView mounts loadInitial() which doesn't exist yet, so the showcase blows up. Landing the matching store edit now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 14:53:24 -04:00
bvandeusen	adeee64a2d	feat(ui): batch initial showcase/gallery loads into smaller chunks ... Showcase and gallery were fetching the full initial batch (60 / 50 items) in a single API call. Operator-flagged 2026-05-30: items should stream in as small batches so they render progressively rather than blocking on one big response. - showcase store: PAGE 60 → 5, INITIAL_BATCHES = 12 (5 × 12 = 60, same total). loadInitial() fetches PAGE chunks in sequence; shuffle() delegates. fetchPage() still returns one PAGE-sized chunk so infinite-scroll also pulls 5 per trigger. - gallery store: loadMore() limit 50 → PAGE (5), INITIAL_BATCHES = 10. loadInitial() loops loadMore() up to INITIAL_BATCHES times, stopping early when nextCursor becomes null (end of data). - ShowcaseView: onMounted calls loadInitial() instead of fetchPage(). Gallery/setTagFilter/setPostFilter already routed through loadInitial. Net visual effect: each ~5-item batch lands and renders independently; combined with MasonryGrid's animateFromIndex logic, the showcase now cascades batches in as they arrive instead of one big pop-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 14:51:58 -04:00
bvandeusen	ed358757dc	Merge pull request 'Most-overdue-first scheduling + rich timeout diagnostics' (#36) from dev into main	2026-05-30 14:30:41 -04:00
bvandeusen	99b66aa85f	fix(download): preserve partial output + classify timeouts richer ... Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show anything other than that the task timedout and was cleaned up. I can't tell why it ran over or if it was stuck failed or there was just that much to get." The TimeoutExpired branch was returning a DownloadResult with no stdout, no stderr, no files_downloaded, and a generic "Download timed out after N seconds" message — even though subprocess.TimeoutExpired carries the partial output gallery-dl emitted before being killed. Now: - Capture e.stdout / e.stderr (coerced str if bytes; "" if None). - Count files_downloaded from partial stdout via _count_downloaded_files. - Surface a tail-of-stderr hint in error_message so the UI summary tells the operator at a glance whether it was "lots of content" (high count, clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung silent" (zero count, "no stderr output"). - Promote error_type to RATE_LIMITED when the partial stderr matches RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole 900s window is the timeout-shaped tail of a real rate limit, and the platform cooldown should kick in for the same reason. Existing test_download_timeout strengthened to also assert empty-partial case stays correctly TIMEOUT-classified with no preserved output. New test_download_timeout_preserves_partial_output_and_classifies covers the rich-partial-output → RATE_LIMITED promotion path. DownloadEvent.metadata already flows stdout/stderr/run_stats from DownloadResult via _phase3_persist — no UI change needed; the existing DownloadDetailModal will surface the captured output automatically once the build redeploys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 14:16:54 -04:00
bvandeusen	77f7a23410	feat(scheduler): order due sources by last_checked_at — most overdue first ... select_due_sources returned rows in undefined order (Postgres-determined, typically PK). At tick rates that outpace download-queue throughput, a freshly-rerun source could keep getting re-queued ahead of one that's still waiting for its first attempt this cycle. Operator-flagged 2026-05-30: > if there are 8 hours before a source is due again and 40 full time > downloads can happen in that period that means that there's a chance > the first one to fire gets back into the download queue before item 41 > has a chance to get downloaded. Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source SELECT. Never-checked sources go first, then longest-since-checked, then ties broken by id. Combined with Celery's FIFO `download` queue, the oldest-overdue source in each tick now reaches a worker before any fresher one. Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue source, and a 2-min-overdue source come back in that exact order from select_due_sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 12:08:43 -04:00
bvandeusen	d181f4afb8	Merge pull request 'Downloads burst-prevention + maintenance-menu fix + gdl timeout' (#35) from dev into main	2026-05-30 11:43:18 -04:00