Commit Graph

553 Commits

Author SHA1 Message Date
bvandeusen bd06794647 fix(downloads): enqueue thumbnail + ML tasks per attached image
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 21s
CI / intimp (push) Successful in 3m43s
CI / intapi (push) Successful in 7m53s
CI / intcore (push) Successful in 8m23s
Operator-flagged 2026-06-01: downloaded images stayed at
thumbnail_path=NULL until a periodic backfill sweep picked them up,
surfacing as broken-thumbnail tiles in the gallery for hours after
the download landed.

Importer.attach_in_place deliberately skips inline thumbnail
generation (importer.py:591-592) so the import queue stays moving —
the CALLING task is responsible for the enqueue. tasks/import_file.py
already did this (line 228-239). tasks/download.py / download_service
did not — every gallery-dl-attached image landed un-thumbnailed.

Fix in download_service._phase3_persist: after each
`attach_in_place` returning status in (imported, superseded), fan out
`generate_thumbnail.delay()` + `tag_and_embed.delay()` for each
image_id. Lazy import avoids circular-import risk between
download_service and the celery task modules that depend on it.

Mirrors the existing pattern verbatim — single source of truth for
"what fires after a successful attach" remains a comment in two
places (filesystem-import task, download orchestrator) rather than
a shared helper, because the contexts differ enough (sync session vs
async orchestrator) that abstracting would obscure more than it'd
share.

Test covers the happy-path with two attached files: both get the
thumbnail enqueue AND the ML enqueue, with image_ids drawn from
ImportResult (so future supersede-on-attach paths stay covered).
2026-06-01 21:39:17 -04:00
bvandeusen f575cfb93b ux(failing-sources): visible row separators + clearer hover
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m15s
The zebra striping in 717b601 was too subtle — inside the tonal error
card, surface-tint contrast washed out and rows still looked uniform.
Operator-flagged the same issue twice.

Replace the surface-tint approach with hard visual structure:
- Bottom border on each row (1px white/0.08) — clearly delineated rows
  regardless of card background
- Hover darkens to black/0.25 — much more obvious than the previous
  error-tint hover, gives the eye a clear horizontal track
- Slightly more vertical padding (8px vs 6px) for tap-target comfort
- Drop the gap; borders are the separator now
2026-06-01 20:57:41 -04:00
bvandeusen 717b601c81 ux(failing-sources): zebra striping + hover highlight on rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 8m29s
CI / intcore (push) Successful in 8m58s
Operator-flagged 2026-06-01: with the failing-sources panel expanded,
all rows have the same flat low-opacity background, no visual track
from the artist name on the left to the Logs/Retry buttons on the
right. Hard to tell which row a button belongs to when scanning down
a list of 17.

Three pure-CSS changes (no DOM, no logic):
- 3px gap between rows (was 2px) — slightly more breathing room
- Even rows get a darker surface tint (zebra striping)
- Hover paints the whole row in a low-opacity error tint, so moving
  the cursor toward Logs/Retry lights up the whole horizontal band
  and the eye traces back to the artist name automatically
2026-06-01 20:14:48 -04:00
bvandeusen cfa4fb4084 fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 28s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m32s
`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial
value (0) when checking that the override works. Now that create() pre-arms
enabled sources to 3, the assertion is stale — the test was already
verifying the override path, the pre-assertion just decorated it.

Drop the pre-assertion; keep the override check. Other tests use raw
Source(...) constructors (default to 0 via the column server_default),
not SourceService.create(), so they're unaffected.
2026-06-01 20:12:55 -04:00
bvandeusen 2aa2002f22 feat(subscriptions): newly added enabled sources start in backfill mode
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Failing after 8m23s
A freshly created subscription has no gallery-dl archive yet, so the
first poll in tick mode would walk forever — exit:20 doesn't trip
until 20 contiguous archive-hits, and there are none. The wall-clock
cap kicks in mid-walk, and the partial-success classifier (plan #544)
gracefully labels it status=ok, but the operator still wonders why
their new subscription isn't grabbing the back-catalog.

Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`.
Same default as the manual "Deep scan" button, same auto-decrement
and auto-reset rules — once the queue drains (clean exit + zero new
files) or the budget runs out, tick mode resumes naturally.

Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'`
that arrive disabled = False, or operator-added sources that start
disabled deliberately) skip the pre-arm — they're never polled, no
budget to waste.
2026-06-01 20:02:19 -04:00
bvandeusen 66ff671f09 fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m58s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m19s
Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.

gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.

Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.

Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.
2026-06-01 19:00:50 -04:00
bvandeusen 19aece1fc4 feat(download): tick/backfill modes + partial-success classifier (plan #544)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m20s
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.

Two coupled changes, both operator-flagged 2026-06-01:

* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
  asks gallery-dl to exit after 20 contiguous archived items. Fresh
  subscriptions + new-content cases still walk normally; established
  subscription with zero new content exits in ~30s of HEAD requests
  instead of pegging the timeout. 20 (not 5) gives headroom against
  paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
  via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
  downloads use `skip: True` + 1800s timeout. Auto-decrements per run
  with early-reset to 0 when a clean run finds zero files (queue
  drained). N defaults to 3 — multiple runs give the system enough
  budget to finish a deep walk across timeout boundaries. New
  `POST /api/sources/{id}/backfill` arms the source; "Deep scan"
  button on each SourceRow (chip shows remaining count) wires it.

Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."

Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).

Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
2026-06-01 18:23:28 -04:00
bvandeusen c9089b1d03 fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 8m18s
CI / intcore (push) Successful in 8m48s
gallery_service._provenance_clause artist branch was correlating its bare
Post reference to the outer query's primary_post_id outer-join, so the
artist filter silently matched zero rows for images with no primary post.
Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner
FROM rather than treating it as a correlated outer table.

Five sidecar/import tests still asserted that a synthetic Source row
appears after a filesystem import. Alembic 0030 retired that behavior;
the Post sits null-source and the artist linkage lives on Post.artist_id.
Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent,
test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar,
and test_apply_sidecar_recovers_from_integrity_error to assert
post.source_id IS NULL + post.artist_id linkage instead.
2026-06-01 14:40:28 -04:00
bvandeusen 644d538bab fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 3m46s
CI / intapi (push) Successful in 8m0s
CI / intcore (push) Failing after 8m45s
2026-06-01 14:23:18 -04:00
bvandeusen ff35da4743 fix(lint): drop unused Source import after Post.artist_id refactor
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 21s
CI / intapi (push) Failing after 2m16s
CI / intimp (push) Failing after 2m16s
CI / intcore (push) Failing after 2m20s
2026-06-01 14:19:37 -04:00
bvandeusen 2f66de2928 feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 2m15s
CI / intapi (push) Failing after 2m18s
CI / intcore (push) Failing after 2m17s
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.
2026-06-01 14:17:52 -04:00
bvandeusen 94e7d20792 feat(downloads): Logs button on failing-sources rollup rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m43s
CI / intcore (push) Successful in 8m10s
Operator-asked 2026-06-01: each row in the "X sources are failing"
rollup needs a button to surface the last run's stdout/stderr/error
without leaving the Downloads tab to find the matching event row.

Wired:
- `downloadsStore.loadLastForSource(sourceId)` two-steps via
  `GET /api/downloads?source_id=N&limit=1` (most recent event) then
  the existing `loadOne(id)` for the full detail payload. Returns
  null if no events exist (toast warns).
- `FailingSourcesCard` row: new text button `Logs` with
  `mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`,
  emits `view-logs` with the source record.
- `DownloadsTab.onViewFailingLogs` is the handler — same
  `DownloadDetailModal` instance the event-row clicks use.
2026-06-01 10:04:22 -04:00
bvandeusen fb605af959 fix(posts): anchored view scroll inherits artist_id/platform filters
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
Operator-flagged 2026-06-01: clicking a Provenance post title from the
view modal opens the post in the posts feed scoped to that artist
(`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll
loaded UNFILTERED global posts instead of staying in the artist's
stream. Root cause: the posts store's loadAround/loadOlder/loadNewer
sent only `{around, cursor, direction}` — never the `artist_id` /
`platform` filters. Backend `/api/posts` accepts them on every path
(post_feed_service.scroll filters via `Source.artist_id`); the
frontend just wasn't passing them.

Fix:
- `posts.js` `loadAround(postId, filters)` now takes the filter
  snapshot and stores it. `_aroundParams(extra)` mixes the active
  filters into around/cursor calls.
- `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` +
  `platform` from the route query.
2026-06-01 08:24:43 -04:00
bvandeusen 4c56cf121f fix(modal): Esc closes from tag input; Provenance cards scroll at ~2.5
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 29s
CI / intimp (push) Successful in 3m54s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m16s
Two operator-flagged UX gaps from 2026-06-01:

1. **Esc trapped inside the tag-entry field**
   The autofocused TagAutocomplete input made Esc-to-close unreachable
   because ImageViewer's keydown handler bailed early on
   `isTextEntry(ev.target)` for every key including Escape. Now Escape
   closes the modal from text inputs too — except when a Vuetify
   overlay is open (FandomPicker, autocomplete dropdown, suggestion
   3-dot menu), in which case that overlay's own Esc-handling fires
   instead of closing the whole modal mid-interaction. Detected via
   `.v-overlay--active`. Arrow keys still gate on isTextEntry so the
   tag input handles typing without navigating images.

2. **Provenance cards expanded the side panel unbounded**
   When an image had many ImageProvenance entries the cards stack
   pushed the Tags section below the fold. Wrapped the cards in a
   `.fc-prov__cards` container with max-height 270px (≈2.5 cards) and
   thin overflow scrollbar. Title stays anchored at top; Tags panel
   sits at a consistent position below regardless of provenance count.
2026-06-01 08:17:54 -04:00
bvandeusen 9564d073b9 fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m26s
CI / intcore (push) Successful in 8m3s
`${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner
executes the step via dash/BusyBox sh and errored with
`Bad substitution` (act_runner workflow shell, observed on the
65386f0 main-push run). Switched to
`SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in
both shells. Both build-web and build-ml tag-determination steps
updated.
2026-06-01 07:53:33 -04:00
bvandeusen f87a06a6bd feat(modal): post title is the primary click (artist-scoped) + suggestion rows look like buttons
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 16s
CI / intimp (push) Successful in 4m16s
CI / intapi (push) Successful in 8m16s
CI / intcore (push) Successful in 8m41s
Two operator-asked modal UX changes (Scribe plan #514):

1. **Post title click → artist-scoped posts feed**
   - The bold post title in ProvenancePanel is now the primary click
     target. Click navigates to /posts?post_id=N&artist_id=A; the
     PostsView already filters on `artist_id`, so the user lands in
     that creator's stream, not the global one.
   - The redundant "View post" link is removed. "Show description"
     stays as the only action link below the meta line.
   - Title is styled as a button-link: accent color, hover underline,
     focus ring for keyboard nav (family rule 24 — UI quality bar).

2. **Suggestion rows look like buttons**
   - SuggestionItem becomes a chip-card: visible border, hover/focus
     background, unified container for name + score + actions
     (operator's "nothing visual to unify the buttons to their object"
     complaint).
   - Accept is an explicit tonal pill button labeled "Accept"
     (operator's pick over whole-row-clickable). 3-dot menu retains
     alias/dismiss, now `variant="outlined"` so it reads as a button.
   - Score is a fixed-width monospace pill on the right.
   - "+ new" badge upgraded to a pill chip with accent border so
     it's visibly an annotation, not part of the tag name.
2026-06-01 02:21:59 -04:00
bvandeusen 5d284aae9f fix(test): unpin general-threshold test from old 0.95 default (alembic 0029)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m55s
CI / intcore (push) Successful in 8m46s
2026-06-01 02:18:54 -04:00
bvandeusen af7b5c95e9 feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Failing after 8m18s
Four coupled operator-asked changes to the view modal (Scribe plan #509):

1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
   in onMounted/nextTick so the caret is in the input the moment the
   modal renders. No click needed to start typing.

2. **General suggestions expanded by default** — SuggestionsPanel's
   general-category group now mounts with `:default-open="true"`.
   Operator can collapse if too noisy, but the v1 frame shows them.

3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
   suggestion_threshold_general default matches character. Alembic
   0029 also bumps the existing singleton row's value if it's still
   at the old 0.95. Operator can re-tune from Settings → ML.

4. **Retire `copyright` + `artist` as ML suggestion categories** —
   neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
   really existed as a copyright tag-kind). They were surfaced in the
   suggestions pipeline + threshold settings UI but had no follow-
   through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
   ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
   drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
   SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
   MLThresholdSliders rows.

Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.

Tests:
- test_surfaced_categories asserts {character, general}, excludes
  artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
  artist + copyright keys in the GET payload.
2026-06-01 02:08:10 -04:00
bvandeusen 8de7ccd07d build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m15s
extension / lint (pull_request) Successful in 16s
Adds an immutable per-commit docker tag to every main-push build:
`git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`,
alongside the existing floating `:main` + `:latest`. Implements the
new family release-posture rule "Tags are milestones, not gates —
commit-SHA images are the rollback unit" so rollback to any commit
on main is `docker pull …:c-<sha>` with no release ceremony required.

Behavior change summary:
- main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>}
- tag-push (opt-in vYY.MM.DD only, no .N): unchanged
- safety-net dev: unchanged

No code changes; the rule is about how the tag list is constructed.
Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still
fire them when the operator wants a labeled checkpoint.
2026-06-01 01:28:55 -04:00
bvandeusen d65f0b2091 feat(extension): probe shows current state before click; v1.0.6
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 20s
extension / lint (push) Successful in 13s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m41s
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup):
"the add source/subscription button idea to the firefox extension so
it can tell me if a source/artist is added or not and offer an option
to add it if it isn't." Plan tracked in Scribe task #507.

## Backend

- `ExtensionService.probe(url)` — read-only resolution. Reuses
  `_derive` for platform+slug, then 2 SELECTs. Returns one of:
  - `source_match` (exact (artist, platform, url) Source exists)
  - `artist_match` (artist exists, this URL isn't a Source yet;
     collapses the sidecar-synthetic-only case from v26.06.01.0)
  - `new` (neither exists)
  - `unknown_platform` (URL didn't match any artist-page regex)
- `GET /api/extension/probe?url=...` route with `X-Extension-Key`
  auth posture matching `/quick-add-source`. Read-only, side-effect
  free.
- 6 backend tests in tests/test_api_extension.py covering each state
  + auth + invalid URL.

## Extension

- `api.js`: `probeSource(url)` mirroring `quickAddSource` shape.
- `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The
  latter strips the `/api` suffix from configured `apiUrl` (placeholder
  format per options.html) and opens `${base}/artist/{slug}` in a new
  tab via `browser.tabs.create`.
- `content-script.js`: probe-first render — on page-load and SPA
  navigation, asks the backend for the URL's state and renders the
  chip in the matching color/copy on FIRST paint instead of flashing
  generic "Add" and updating after. Click handler branches:
  `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing
  ADD_AS_SOURCE flow (then re-probes so the chip flips green
  immediately, no wait for next nav).
- `content-script.css`: three state-color modifiers
  (--new, --artist-match, --source-match) on the FC parchment-on-slate
  palette. Sage for already-added, amber for artist-exists, accent
  orange for new.

## Versioning

- `extension/manifest.json` + `extension/package.json` → 1.0.6.
  build.yml's sign-extension job will fire on push to main since no
  `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the
  regenerated AMO keys end-to-end.

## Behavior on the sidecar-synthetic case

Filesystem-imported "Dymkens"-style artist with only a sidecar
synthetic Source: probe returns `artist_match` (not `new`), so the
chip reads "+ Add Patreon source to Dymkens" rather than offering to
recreate the artist. Clicking adds the real Source; existing
`_source_for_sidecar` preference logic (v26.06.01.0) routes future
gallery-dl Posts to the real one.
2026-06-01 00:41:17 -04:00
bvandeusen 66f19d67f5 fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 6m56s
CI / intcore (push) Successful in 7m35s
Three coupled operator-reported pains from the 2026-05-31 download
event audit:

1. `[patreon][warning] Not allowed to view post N` was bubbling up as
   an error event, bumping consecutive_failures and parking the source
   in "needs attention." The classifier's tier-gated branch was gated
   on `return_code in (1, 4)`. Gallery-dl returns a different exit
   code for mixed-failure runs (e.g. paywall warnings + a missing
   yt-dlp dep flipping the exit bits), so the branch never fired and
   the path fell through to UNKNOWN_ERROR. Widen the gate: when no
   source-level error fired AND tier-gated warnings are present,
   classify as TIER_LIMITED regardless of return code.

2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
   "stranded by recovery sweep (no terminal status after time_limit)"
   + empty stdout/stderr. Root cause: subprocess.run timeout (900s)
   and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
   wiped the in-memory captured output and the DownloadEvent ended up
   empty-logged 18 minutes later when the sweep finalized it. Drop
   gallery-dl's default subprocess timeout to 870s — a 30s margin
   shy of Celery's soft limit — so subprocess.TimeoutExpired always
   wins the race and captures the partial stdout/stderr via the
   existing handler.

3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
   firing on every video attachment, causing per-item download
   failures that masked legitimate tier-gated classification.
   Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
   video posts download normally and the per-item failure noise
   disappears.

Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
2026-05-31 23:30:39 -04:00
bvandeusen 6fc8ae3106 fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 8m20s
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:

1. `SourceService.list()` returned every Source, no filter on URL.
   alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
   one canonical row per (artist, platform); when no real campaign URL
   was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
   enabled=false as a disabled anchor. The UI then listed those
   anchors as if they were polls — disabled, but visible. Fix: `list()`
   excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
   opts back in for admin tooling.

2. `importer._source_for_sidecar` picked the lowest-id Source for
   (artist, platform). When alembic 0022 had rewritten a per-post row
   into a synthetic anchor (lower id) AND the operator later added the
   real subscription (higher id), every gallery-dl download silently
   attached its Post to the SYNTHETIC instead of the real Source. Fix:
   prefer a non-`sidecar:%` URL when one exists; fall back to the
   synthetic; only create a new synthetic when nothing exists for
   (artist, platform).

alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.
2026-05-31 23:08:38 -04:00
bvandeusen a5101494b6 feat(downloads): bulk retry respects cooldown; single-source RETRY overrides
Today's platform-cooldown commit (61ce1ce) only filtered the scan tick
— manual /api/sources/<id>/check still bypassed it. Operator-flagged
2026-05-30: clicked "Retry failed" on a Patreon failure pile and saw
every one go 'queued' without realising the cooldown wasn't in the
loop. Bulk retry with N sources on a cooled-down platform bowls right
back into the rate limit the cooldown is trying to prevent.

**Backend (`/api/sources/<id>/check`):**
- Reads optional `?force=true` query flag.
- Without force: queries `active_platform_cooldowns` (renamed from the
  private `_platforms_in_cooldown` since it's now a cross-module API).
  If the source's platform is in cooldown, returns **202** with
  `{status: 'deferred', platform, cooldown_until}` — no event created,
  no dispatch.
- With force: cooldown skipped entirely.
- In-flight guard always applies (no point creating duplicate pendings).

**Frontend (`sourcesStore.checkNow(id, {force=false})`):** new optional
`force` flag → adds `?force=true` to the URL.

**Frontend (`DownloadsTab`):**
- `onRetrySource` (single-source RETRY click): passes `force: true` →
  explicit operator override, useful for rapid auth-fix testing.
- `onRetryAll` (RETRY ALL + MaintenanceMenu "Retry failed"): no force →
  cooldown respected. Tallies `deferred` alongside `queued` /
  `already_running`; toast reads e.g. *"5 queued, 12 deferred
  (cooldown), 3 already running"*. That count is the operator's
  diagnostic answer for "is rate-limit the cause of most failures?"
  (12-of-20 deferred → yes; 0 deferred → no).

**Auto-resume:** no new sweep needed. Deferred sources still have stale
`last_checked_at`, so the next scan tick after the cooldown AppSetting
expires picks them up via `select_due_sources` (which already filters
on `active_platform_cooldowns`).

Tests: two new — deferred-on-cooldown returns 202 with the right body
and no dispatch; force=true overrides the cooldown and creates the
event normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 23:15:07 -04:00
bvandeusen e3a7aff7a3 ux(showcase): pipeline fetches in-order, chunk size 3, keep the trickle
Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast
but could let later chunks arrive ahead of earlier ones — even when
each chunk is a random sample, that "later chunk loads first" risk made
the load order non-deterministic. And the original goal behind asking
for batching was a faster first-image-on-screen, which neither sequen-
tial nor parallel really addressed cleanly.

Switched the loadInitial flow to a PIPELINE:
- Only one fetch in flight at any moment (in-order arrival, no race).
- The NEXT fetch kicks off as soon as the current one resolves (NOT
  after its trickle finishes), so the next RTT overlaps the visible
  trickle window — round-trips are hidden behind the animation cadence.
- PAGE 5 → 3 + INITIAL_BATCHES 12 → 20 (total still 60). Smaller chunk
  → first chunk's items appear sooner (a chunk of 3 trickles in 240ms,
  well within one RTT, so by the time chunk 2 is in-hand the first
  trickle is just finishing).

Trickle, sequence-token guard, and infinite-scroll behaviour unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 22:52:55 -04:00
bvandeusen 9cd6d09e60 ux(showcase): smooth one-at-a-time cadence + earlier infinite-scroll trigger
Operator-flagged 2026-05-30 (round 2): the batched-loads change improved
consistency but still felt "chunky" — 5 items appeared together, then a
~200 ms API round-trip pause, then another 5. And infinite-scroll fired
too late when a single tall image (long manga page) made one masonry
column much taller than its siblings.

**Cadence (showcase store):**
- Fire all INITIAL_BATCHES fetches in PARALLEL — collapses the per-
  batch round-trip gap so all the data arrives in ~1 RTT instead of 12
  sequential RTTs.
- Trickle each response's items into images.value one at a time with
  APPEND_DELAY_MS = 80ms between each (≈ the MasonryGrid stagger
  animation, 70ms). User sees a smooth steady stream.
- fetchPage (the infinite-scroll path) uses the same trickle so its
  5-item appends also cascade one-by-one instead of popping together.
- Sequence token guards against a fast shuffle / mount-then-shuffle
  interleaving two trickles into the same images.value.
- Dropped useAsyncAction here — the parallel-fetch-then-trickle flow
  doesn't fit its single-wrap-call shape cleanly; inline loading/error
  state is clearer.

**Infinite-scroll trigger (MasonryGrid):**
- Pass `rootMargin: '2400px'` to useInfiniteScroll (was the 600px
  default). The masonry sentinel sits at the bottom of the container,
  whose height = MAX(column heights). A tall image in one column pushes
  the sentinel ~2× viewport below where the user is actually reading
  (the bottom of the SHORTER columns). 2400px ≈ 2-3 screen-heights of
  pre-emptive trigger, comfortable for typical tall manga heights.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 22:48:58 -04:00
bvandeusen 810baf63ac fix(import): archive probe → subprocess (was multiprocessing.Process)
Every archive import was failing immediately with "AssertionError:
daemonic processes are not allowed to have children" (operator-flagged
2026-05-30 — import_archive_file crashes in 49ms with the assertion).
Celery's prefork pool runs tasks in daemon processes; Python's
multiprocessing module refuses to let daemons spawn children, which is
exactly what probe_archive was doing via mp.get_context("spawn")
.Process. The Layer-3 crash-isolation feature added 2026-05-28 was
effectively a hard-blocker on the very import path it was meant to
protect.

Switched probe_archive to subprocess.run — no daemon restriction, still
isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill
the worker). The probe body lifted to a tiny runner module
(_archive_probe_runner) that imports the unchanged _run_probe helper
and prints a single JSON line; parent parses stdout, returns the
ProbeResult exactly as before (timeout, signal, OOM, clean-rejection,
ok — all preserved).

cwd for the subprocess is the repo root derived from __file__ parents
so `python -m backend.app.utils._archive_probe_runner` resolves both in
the Celery container and pytest, regardless of where the worker was
launched.

Test refactor: test_archive_probe_target_bomb_guard →
test_run_probe_bomb_guard. Same in-process call to the (renamed) probe
body so the monkeypatched cap still takes effect; the real subprocess
path is exercised by the existing test_probe_archive_valid_zip /
test_probe_archive_corrupt_zip_clean_rejection tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 21:48:51 -04:00
bvandeusen 44bb12a93d fix(thumbnails): derive URL from stored thumbnail_path, not (sha256, mime)
The showcase/gallery/artist/series/post-feed APIs were constructing
thumbnail URLs from (sha256, mime). The MIME-based extension predicate
("png if image/png or image/gif else jpg") DISAGREED with the
thumbnailer's actual on-disk extension predicate ("png if alpha else
jpg"). Result: every PNG source without transparency 404'd (URL asked
.png, disk had .jpg); every WebP/AVIF source with transparency 404'd
(URL asked .jpg, disk had .png) — despite the thumbnail file existing
on disk.

The backfill task couldn't catch these because backfill checks the
ACTUAL thumbnail_path stored on the record (correct), not the URL the
browser fetches (broken derivation). So records with valid on-disk
thumbnails kept showing as broken in the UI no matter how many times
backfill ran.

Operator-flagged 2026-05-30: "the generate thumbnails function appears
to not catch all of the failed thumbnail cases" — turned out to not be
a backfill bug at all.

Fix: thumbnail_url now takes (thumbnail_path, sha256, mime) and returns
the stored path verbatim — Quart serves /images/* 1:1 from the volume
(frontend.py:20-36), so the URL IS the disk path. Falls back to the old
sha256+mime derivation only when thumbnail_path is NULL (thumbnailer
hasn't run yet); that URL will 404 in the browser until backfill catches
it, same as before the path was tracked.

All 8 callers updated: showcase_service, gallery_service (2 sites),
artist_service, series_service, post_feed_service, tag_directory_service,
artist_directory_service. The four sites whose query was raw-tuple now
also SELECT ImageRecord.thumbnail_path.

Net effect: every record that has a valid on-disk thumbnail will now
render correctly, regardless of which extension the thumbnailer chose,
without any DB migration or backfill rerun needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 16:55:38 -04:00
bvandeusen 1eefed9ab3 fix(ui): finish showcase store batching (PAGE 60 → 5, INITIAL_BATCHES 12)
The showcase-store change in adeee64 didn't actually land — only
gallery.js + ShowcaseView.vue made it into the commit. Without this,
ShowcaseView mounts loadInitial() which doesn't exist yet, so the
showcase blows up. Landing the matching store edit now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:53:24 -04:00
bvandeusen adeee64a2d feat(ui): batch initial showcase/gallery loads into smaller chunks
Showcase and gallery were fetching the full initial batch (60 / 50
items) in a single API call. Operator-flagged 2026-05-30: items should
stream in as small batches so they render progressively rather than
blocking on one big response.

- showcase store: PAGE 60 → 5, INITIAL_BATCHES = 12 (5 × 12 = 60, same
  total). loadInitial() fetches PAGE chunks in sequence; shuffle()
  delegates. fetchPage() still returns one PAGE-sized chunk so
  infinite-scroll also pulls 5 per trigger.
- gallery store: loadMore() limit 50 → PAGE (5), INITIAL_BATCHES = 10.
  loadInitial() loops loadMore() up to INITIAL_BATCHES times, stopping
  early when nextCursor becomes null (end of data).
- ShowcaseView: onMounted calls loadInitial() instead of fetchPage().
  Gallery/setTagFilter/setPostFilter already routed through loadInitial.

Net visual effect: each ~5-item batch lands and renders independently;
combined with MasonryGrid's animateFromIndex logic, the showcase now
cascades batches in as they arrive instead of one big pop-in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:51:58 -04:00
bvandeusen 99b66aa85f fix(download): preserve partial output + classify timeouts richer
Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show
anything other than that the task timedout and was cleaned up. I can't
tell why it ran over or if it was stuck failed or there was just that
much to get."

The TimeoutExpired branch was returning a DownloadResult with no stdout,
no stderr, no files_downloaded, and a generic "Download timed out after
N seconds" message — even though subprocess.TimeoutExpired carries the
partial output gallery-dl emitted before being killed.

Now:
- Capture e.stdout / e.stderr (coerced str if bytes; "" if None).
- Count files_downloaded from partial stdout via _count_downloaded_files.
- Surface a tail-of-stderr hint in error_message so the UI summary tells
  the operator at a glance whether it was "lots of content" (high count,
  clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung
  silent" (zero count, "no stderr output").
- Promote error_type to RATE_LIMITED when the partial stderr matches
  RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole
  900s window is the timeout-shaped tail of a real rate limit, and the
  platform cooldown should kick in for the same reason.

Existing test_download_timeout strengthened to also assert empty-partial
case stays correctly TIMEOUT-classified with no preserved output.
New test_download_timeout_preserves_partial_output_and_classifies covers
the rich-partial-output → RATE_LIMITED promotion path.

DownloadEvent.metadata already flows stdout/stderr/run_stats from
DownloadResult via _phase3_persist — no UI change needed; the existing
DownloadDetailModal will surface the captured output automatically once
the build redeploys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:16:54 -04:00
bvandeusen 77f7a23410 feat(scheduler): order due sources by last_checked_at — most overdue first
select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:

> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.

Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.

Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 12:08:43 -04:00
bvandeusen ff9e96e0e2 fix(tests): update scheduler-status shape pins for platform_cooldowns
61ce1ce added platform_cooldowns to scheduler_status(), and two pinned
key-set assertions tied to the old shape failed in CI. Updated:

- tests/test_api_sources.py::test_schedule_status_shape — direct
  /api/sources/schedule-status response.
- tests/test_api_system_activity.py::test_summary_returns_rollup_shape
  — nested under body["scheduler"] in the summary endpoint.

[[feedback-plan-grep-pinned-tests]] miss — should have grepped tests/
for `last_tick_at` / `auto_sources` before adding the new key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 11:25:36 -04:00
bvandeusen 61ce1ce13c feat(scheduler): platform-wide cooldown on RATE_LIMITED — burst prevention
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.

When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.

INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.

Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.

Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 11:01:40 -04:00
bvandeusen d28db32012 fix(download): align gallery-dl subprocess timeout to Celery soft limit
SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's
Celery task has soft_time_limit=900s and hard time_limit=1200s. So
gallery-dl never hit its own subprocess.run timeout — Celery always
killed it first. The hard SIGKILL leaves no terminal flip on the
DownloadEvent, which then sat pending/running until the recovery sweep
flipped it to error at 30 min from start. From the operator's seat that
was a ~30–40 min "hang" on every retry of a broken source.

Pinning the default to 900s (matching Celery's soft_time_limit) lets
subprocess.run raise TimeoutExpired cleanly inside Celery's window, and
the existing `except subprocess.TimeoutExpired` branch
(gallery_dl.py:635) captures it as a clean error_type='timeout' with a
real message. The DownloadEvent flips to error in ~15 min instead of
waiting on the recovery sweep at 30.

Per-source bumps still live in source.config_overrides for legitimately
long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries
the rationale and the Celery-soft-limit dependency in its comment.

Operator-flagged 2026-05-30 on 59-source strand pile retries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 02:25:04 -04:00
bvandeusen 77e9859da3 fix(downloads): rewire MaintenanceMenu to the downloads pipeline
The maintenance dropdown in Subscriptions → Downloads was wired to the
filesystem-import pipeline (POST /api/import/retry-failed +
POST /api/import/clear-stuck) — the subtitles even said so ("Re-enqueue
every failed import task"), but it was contextually misplaced. From the
Downloads view "Retry failed" queued nothing the operator could see
because the action operated on import_task rows, not download_event
rows. Import-pipeline maintenance is already reachable from Settings →
Imports (ImportTaskList.vue), so removing the import wiring loses
nothing.

Rewired:
- "Retry failed" → bulk-retries the failing-sources list, same loop as
  FailingSourcesCard's RETRY ALL (sourcesStore.checkNow per source).
  Subtitle now matches: "Re-queue every currently failing source".
- "Force recovery sweep" → triggers recover_stalled_download_events on
  demand via a new POST /api/downloads/recover-stalled endpoint. The
  sweep also runs every 5 min on Beat; this is the manual fallback so
  the operator doesn't have to wait for the next tick to clear newly
  stranded events.

MaintenanceMenu is now stateless — emits retry-failed and recover-
stalled. DownloadsTab owns the handlers (reuses the existing
onRetryAll; new onRecoverStalled with a delayed refresh so swept rows
land in the failing rollup).

Operator-flagged 2026-05-29 — "the retry failed button in the
maintenance dropdown doesn't appear to queue anything but manual
requeues works."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 00:59:05 -04:00
bvandeusen 36f8ec80fd fix(ui): tooltip override needs !important — Vite reorders CSS chunks
104cac5's override sits at the same specificity as Vuetify's default
(`.v-tooltip > .v-overlay__content` on both sides). The fix relied on
source order — app.css imported after vuetify/styles in main.js — but
Vite's production bundler reorders node_modules CSS into the final
stylesheet unpredictably, so source order isn't a reliable winner.

!important on the two contrast properties (background + color) forces
the slate-on-parchment pair regardless of stylesheet load order. The
cosmetic border + shadow don't need it (Vuetify doesn't set them, so
nothing's competing).

Operator re-flagged 2026-05-29: tooltips still rendered light-on-light
on the deployed :latest after PR #33104cac5 was in the bundle but
not winning. Also updated the file header so the source-order claim
isn't carried forward as gospel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 23:51:39 -04:00
bvandeusen e35fb1edf7 fix(scan): recovery sweep for stranded download events
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.

cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).

New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
  download_source 1200s hard kill, so legitimately-running tasks are
  never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
  backoff is 2^N on that counter so per-event bumps would needlessly
  inflate the next interval — sets last_error, stamps last_checked_at.

UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.

Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 21:40:52 -04:00
bvandeusen 08420cd619 feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon
The plain-JS frontend re-declares two backend enum value-sets by hand:
download-event statuses (downloadStatus.js) and platform keys
(platformColor.js). With no TS codegen, drift is silent — the same class
as the last_verified_at field mismatch.

tests/test_fe_be_contract.py parses both JS mirrors and asserts they equal
the backend canon (downloads._ALLOWED_STATUSES, platforms.known_platform_keys())
so a change on either side fails CI on whichever moved. Backend is the
source of truth. Documented the invariant in both JS file headers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 15:34:20 -04:00
bvandeusen 8649a13118 refactor(I5): remove one-and-done GS/IR migration tooling
The GS/IR migration cutover is complete, so the runbook tooling is dead
weight. Removed:
- services/migrators/ (gs_ingest, ir_ingest, tag_apply, ml_queue, verify,
  cleanup), tasks/migration.py, api/migrate.py (+ blueprint registration)
- MigrationRun model; alembic 0027 drops the migration_run table
- frontend LegacyMigrationCard + migration store (+ MaintenancePanel ref)
- celery include + task route + celery_signals queue mapping for migration.*
- the 1 GB MAX_CONTENT_LENGTH / MAX_FORM_MEMORY override (added solely for
  the ir_ingest upload)
- migration-surface tests (test_api_migrate, test_migration_verify,
  test_ir_ingest, test_gs_ingest, test_tag_apply)

Kept: the alembic schema-migration tests (test_migration_00XX — unrelated)
and cleanup_service.py (the permanent artist-cascade/unlink home).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 14:38:59 -04:00
bvandeusen 8979e0e377 refactor(I4): extract useInfiniteScroll composable; migrate 6 consumers
The IntersectionObserver-on-a-sentinel infinite-scroll pattern was copy-
pasted in 7 places. New composables/useInfiniteScroll(sentinelRef, cb)
owns the observer lifecycle (attach on mount, re-attach when the ref
changes, disconnect on unmount). Migrated GalleryGrid, MasonryGrid
(gallery+showcase), ArtistPostsTab, ArtistsView, TagsView, SeriesManageView.
PostsView left manual on purpose — its anchored mode does bidirectional
scroll-position preservation that doesn't fit the simple composable.

I4(a) (timestamps) is effectively already done: the UTC displays were
converted earlier; the remaining toLocaleString uses already render local
time, so they're left as-is rather than churned for format-only consistency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 14:03:49 -04:00
bvandeusen 00e2608ba1 feat(I3): always-on pipeline status indicator in the top nav
New /api/system/activity/summary aggregates scheduler health + per-queue
pending depths + running count + 24h failure count in one cached call (safe
to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a
compact running/queued/failing chip with a scheduler-health dot that expands
to a popover (scheduler, busy queues, counts, link to downloads). Polls the
summary every 8s, paused when backgrounded. Reuses the queue-cache read via
_queues_cached(). + API test for the summary shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:24:11 -04:00
bvandeusen 9ab5d709c8 fix(test): DownloadEventRow row shows platform, not artist name
The row template renders status + platform (PlatformChip) + time + counts;
the artist isn't displayed there. Assert on 'Completed'/'Patreon' instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:16:33 -04:00
bvandeusen 44410db492 test(I2): vitest component smoke tests for high-risk UI
Adds @vue/test-utils + happy-dom and mounts CredentialCard, DownloadEventRow,
PostCard, ActiveDownloadsPanel, QueueStatusBar, asserting they render without
throwing and surface key content — catching the dead-binding / render-error
class (e.g. the last_verified_at regression, guarded explicitly). Vuetify
components are left unresolved (Vue renders unknown elements + slots), so no
Vuetify-plugin setup is needed; only RouterLink is stubbed. Per-file
happy-dom env via docblock keeps the existing node-env specs untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 12:34:58 -04:00
bvandeusen e76aa36a29 ci(I1): dedicated fast-fail ruff lint lane
ruff is pre-installed in the ci-python image, so a new `lint` job runs it
with no dependency install and fails in seconds — surfacing the common lint
bounce class without waiting on the backend job's ~30-60s wheel install.
Dropped the now-redundant ruff step from backend-lint-and-test (same job
name, required-checks unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 12:22:33 -04:00
bvandeusen c95b760294 feat(posts): in-context anchored feed with bidirectional infinite scroll
Provenance "View post" deep-links to /posts?post_id=X, which now opens the
feed centered on that post with infinite load in BOTH directions.

Backend: PostFeedService.scroll gains a direction (older|newer); new
around(post_id) returns a window of newer + the post + older with a cursor
for each end. /api/posts accepts ?around= and ?direction=. + API tests.

Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends,
newer prepends) with per-end cursors; PostsView's anchored mode scrolls to
the post, observes top + bottom sentinels, and preserves scroll position on
upward prepend so the page doesn't jump. Normal feed mode unchanged.

Closes the remaining half of the post-navigation work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:12:18 -04:00
bvandeusen 35fe420701 feat(maintenance): live queue-depth bar under the backfill buttons
Each maintenance button (ML backfill, centroid recompute → ml queue;
thumbnail backfill → thumbnail queue) now shows a status bar with the live
pending count for its Celery queue, so the operator can see work is already
queued/running before re-triggering and piling on. MaintenancePanel polls
/api/system/activity/queues every 4s; QueueStatusBar reads the depth and
turns warning-colored when the queue is busy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 20:03:32 -04:00
bvandeusen 9e74c80e2f feat(downloads): front-and-center "active now" panel with live elapsed timers
New ActiveDownloadsPanel pinned at the top of the Downloads tab shows what's
running (pulsing dot + live mm:ss timer counting from started_at) and what's
queued, so the operator can see activity at a glance without the running
filter. Backed by store.loadActive() which fetches running + pending
independent of the feed filter; refreshed every 4s by the existing live poll.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:36:49 -04:00
bvandeusen c87e8e0932 fix(ui): render absolute timestamps in the viewer's local timezone
Download event times showed raw UTC wall-clock (iso.slice). Added
formatDateTime()/formatLocalDate() (local tz, robust to naive vs tz-aware
ISO) and applied them to the download row + detail modal datetimes and the
credential/artist date displays. formatPostDate stays UTC (date-only,
locale-stable, unit-tested).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:36:42 -04:00
bvandeusen 8c3900b998 feat(showcase): dramatic 3D cascade entry — tiles tip back then settle into place
Replaced the subtle 12px fade with a perspective rotateX(-28deg) tilt that
flips up and settles flat with a slight overshoot, staggered 70ms so tiles
cascade in one at a time. Makes the showcase read as an experience rather
than a quiet fade. Tunable (tilt/stagger/duration); reduced-motion safe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:01:04 -04:00
bvandeusen 972d9014ce feat(showcase): IR-parity hover animation on masonry thumbnails
FC already matched IR's staggered entry fade-in; this adds the missing
piece — hover zoom (scale 1.03) + brighten (1.1) on each thumbnail, with
overflow:hidden so the scaled image clips to the rounded card. Disabled
under prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:58:05 -04:00