f2e9ae07dc9fd5662b9eba853f7add25af4fcc20
678 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
f2e9ae07dc |
fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/ normalize (operator-flagged — it was the dominant entry in the 24h failures). It now runs ~10-min chunks and re-enqueues itself until the library is exhausted, matching the operator's preferred pattern (reasonable timeout → retry queued → other things process between). New columns (alembic 0039): resume_after_id persists the keyset cursor so a chunk continues where the last left off; last_progress_at lets the recovery sweep tell a progressing multi- chunk audit from a dead one (it now measures staleness from last_progress_at, not started_at). Matches accumulate across chunks. soft/hard limits dropped 2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop re-enqueues to resume instead of erroring the whole run. Tests: time-box → re-enqueue (status stays running); resume carries prior matches and appends new ones. Existing full-scan tests unchanged (small sets finish in one chunk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d9d502a60d |
fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
1819caaf5b |
feat(modal): keyboard-friendly tagging — fandom dialogs, Tab-accept, jump hotkey, cheatsheet
Operator-requested modal/tagging keyboard improvements: - A2/A3: fandom dialogs autofocus their autocomplete on open; in the character- creation FandomPicker, picking a fandom (keyboard Enter or click) confirms in one step. FandomSetDialog stays autofocus-only (its Save can trigger a merge). - B5: Tab accepts the highlighted autocomplete row (standard convention). - C9: T or / jumps focus to the tag input from anywhere in the modal. - C8: ? toggles a keyboard cheatsheet (corner hint advertises it; Esc closes the cheatsheet first, then the viewer). Builds on the same-batch regression fixes (kebab #711, ESC-after-accept #700, autocomplete scroll-into-view). B6 (keep focus after add) is covered — the input retains focus after adding a tag, and Esc now works after accepting a suggestion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
22dc516dc7 |
fix(modal): tag-chip kebab + ESC-after-accept + autocomplete scroll-into-view
Two regressions the operator re-flagged (the earlier "fixes" didn't work): #711 tag-chip kebab: TagPanel's kebab used the #activator + v-bind="props" v-menu pattern — the exact pattern SuggestionItem's own comment documents as NEVER toggling inside the teleported ImageViewer modal. Extracted TagChip.vue using the proven explicit pattern (activator="parent" + :open-on-click="false" + a manual v-model), mirroring the working suggestion kebab. Now opens. #700 ESC-after-accept: the guard suppressed close whenever ANY non-tooltip overlay was active anywhere in the DOM, so a stray overlay after accepting a suggestion (focus drops to <body>) blocked Esc. Now key off the event origin — only defer to an overlay when Esc is pressed from INSIDE its content (ev.target.closest('.v-overlay__content')); a stray overlay no longer traps the modal, and dialogs/menus still handle their own Esc. A1: TagAutocomplete arrow-nav now scrollIntoView's the highlighted row — the list is capped at 240px and arrowing past the fold left the active item off-screen (operator-flagged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4c42a15fa1 |
fix(patreon): a missing media file_name is a URL-basename fallback, not API drift
The native client treated a gallery image without `file_name` as schema drift and raised "Patreon API changed — ingester needs update", failing the whole walk (operator-flagged 2026-06-07: BlenderKnight post 73665615, kind=images). But the resource had a valid URL, and the code already derives a filename from the URL basename right below the raise — the same fallback gallery-dl uses. Patreon legitimately serves some images without file_name, so this isn't drift. Drop the require_file_name gate from _media_item: file_name is now optional for every kind (images/attachments/postfile), falling back to the URL basename. Genuine drift still raises — no resolvable URL, or a media id referenced by a relationship but absent from `included`. Test updated to assert the fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
14c4dd1ea0 |
test(patreon): adjust deterministic clock for the new per-media should_stop read
The mid-post time-box check (
|
||
|
bvandeusen
|
619e7712c2 |
fix(patreon): enforce the backfill time-box mid-post (stop overrunning to the soft limit)
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked between POSTS, but download_post downloads ALL of one post's media synchronously — so a single media-heavy post could run the chunk far past 600s, all the way to the Celery soft time limit (1350s), where it was killed and finalized as error (Pocketacer, event #41330: ran the full 22.5 min). download_post now polls a should_stop() deadline BEFORE each media item and the engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the budget and the remaining media (never marked seen) re-fetch next chunk. Bounds chunk overrun to one media download instead of one whole post. Also genericized the soft-limit salvage message — it claimed the "gallery-dl subprocess" failed, which is wrong for a native Patreon walk; it now describes the time-budget overrun + per-page checkpoint resume in platform-neutral terms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
416d8d71cd |
feat(patreon): resolve the creator's campaign from a single-post URL
A source URL like https://www.patreon.com/posts/mimic-in-dungeon-158372536 is a single-post permalink, not a creator page — the resolver grabbed "posts" as the vanity and failed (operator-flagged 2026-06-07). Add a resolution path: extract the trailing post id and follow it to the owning campaign via the Patreon post API (/api/posts/<id>?include=campaign), so pasting any post URL subscribes to that creator's whole feed. `posts/` is excluded from the vanity regex so it can't masquerade as a creator slug. Resolution order is now: cached override → id: URL → /posts/<id> → vanity (campaigns API + creator-page scrape). Tests cover the post→campaign resolve and that /posts/ URLs aren't treated as vanities. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a3c9499e93 |
feat(subs): kick off the first backfill walk immediately on source create
A new enabled source is armed for run-until-done backfill (#693) but would sit idle until the next scheduler tick (~60s). create_source now enqueues the first walk right away (pending DownloadEvent + download_source.delay), skipping only when the platform is in a rate-limit cooldown (the scheduler picks it up when that clears). Disabled sources still don't dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
87c7318125 |
feat(downloads): serialize same-platform downloads (Patreon concurrency cap)
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
e4e35163ab |
feat(subs): subscriptions UX batch — error reasons, single-source rows, health sort, bulk backfill
Operator-requested follow-ups: - #1 Failure reason on hover: the red error-count chip now shows source.last_error in a tooltip (desktop row + mobile card), so the cause (e.g. the new "vanity=cw" message) is visible without opening Downloads. - #2 Collapse the single-source case: a subscription with exactly one source now shows that source's URL + its own actions (Check / Backfill / ⋮ / Edit) inline on the artist row — no expand needed for the common case. Multi-source keeps the artist-level actions + expandable per-source table. - #3 Sort by health: the Health column is sortable on a numeric rank (never/ok/warn/fail) and the table defaults to worst-first, name as tiebreak. - #4 Drop Preview: removed the low-value bounded-peek action from the menu and all its wiring (backend endpoint + store fn left in place, unused). - #5 Backfill selected: a "Backfill" button in the bulk bar arms a run-until-done backfill on every enabled, not-already-running source in the selection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b7d07324ee |
fix(subs): stop the lock/reload on source actions + regroup the row buttons
Lock/reload: every inline source action (check / backfill / recover / toggle / remove) ended by refetching the WHOLE subscription list (store.loadAll / refresh), which blocked the UI and re-rendered the table — collapsing the expanded row, which read as "locks then resets." The action APIs already return the updated source, so the store now patches that one row in place (_patchSource / _dropSource); the post-action loadAll/refresh calls are gone. Toggling enabled, starting/stopping a backfill, recovering, and removing are now instant and leave the expansion intact. Button regroup (operator-flagged: tiny, mis-clickable, not grouped by function): - New shared SourceActions.vue used by desktop SourceRow + mobile SourceCard. - Frequent actions stay as size="small" buttons: Check, Backfill/Stop. - Low-frequency / destructive actions move into a labelled overflow (⋮) menu — Preview backfill, Recover dropped near-duplicates, Remove source — so they can't be fat-fingered, and the labels spell out recover-vs-backfill. - Edit moves next to the source URL (its identity), out of the action cluster where it sat beside Remove. - Single-source Remove now confirms (it had no guard before). Tests: store patches/drops in place without dropping the cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c65da42593 |
fix(patreon): handle the /cw/ creator-URL prefix in vanity extraction
A source URL like https://www.patreon.com/cw/Atole resolved vanity='cw' — the vanity regex only skipped a /c/ prefix, so Patreon's current /cw/ ("creator workspace") form fell through to the bare-vanity branch and captured the prefix instead of the slug. Every /cw/ source then failed campaign-id resolution (API + page-scrape both looked up "cw"). Operator-confirmed 2026-06-07 via the new error text: source_url='.../cw/Atole'; vanity='cw'. Add cw/ to the optional prefix group (ordered before c/ so the longer prefix wins), and have the creator-page fallback try the /cw/ form too. Test covers bare / c/ / cw/ extraction and that id: URLs stay non-vanity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
19eb4e9388 |
feat(import): surface WHY an archive was captured without extracting images
The recurring "post shows a zip but no images" report had no diagnostic: when
_import_archive captured an archive as a bare PostAttachment — because the
bomb-guard probe rejected it, or extraction yielded zero members (corrupt /
unsupported / missing extractor backend), or it held only non-media files — it
returned status="attached" silently.
Now those paths set ImportResult.error with the specific reason and log a
warning, and download_service records each as {file, reason} under the event's
metadata.unextracted_archives (None when every archive extracted cleanly). So
the next run names exactly which archives failed and why, instead of leaving the
operator to guess. No behaviour change to the happy path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
a559fabdd5 |
feat(patreon): scrape creator-page HTML as a campaign-id resolution fallback
Patreon's /api/campaigns?filter[vanity]= lookup returns empty data for creators
that plainly exist (operator-flagged 2026-06-06 — Atole etc. erroring at the
resolve step). gallery-dl never used that endpoint; it pulls the campaign id out
of the creator page's bootstrap JSON. Add the same as a fallback: when the API
misses, GET the creator page (bare + /c/ vanity paths) and scrape the first
campaign id from any known embedding ("id":"…","type":"campaign" /
"campaign":{"data":{"id" / /api/campaigns/<id> / "campaign_id"). API is still
tried first (cheap, structured); the page scrape only runs on a miss.
Tests: API-empty → page-scrape fallback resolves; _scrape_campaign_id pattern
coverage. Existing API-path tests unchanged (happy paths short-circuit before
the fallback; failure paths hit the guarded scrape and still return None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
711fd2bb75 |
style: drop aliased cross-module import (I001) — use a local _CAMPAIGNS_API constant
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3556a54260 |
feat(patreon): surface source URL + vanity in campaign-id resolution errors
Naming/lookup failures now report the source_url, the extracted vanity, and the exact campaigns-API lookup URL attempted, so a "could not resolve campaign id" error is diagnosable (wrong vanity? cookie/auth? creator renamed?) instead of opaque. Applied to all three resolution surfaces: the native download event, the dry-run preview, and the credential-verify probe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
df2310bc70 |
fix(tags): preserve acronym casing in tag normalization (DC, NSFW)
normalize_tag_name now only capitalizes the first letter of each word and leaves the rest of the word untouched (was lowercasing the tail, which turned DC→Dc / NSFW→Nsfw). This matches ml/tag_name._title_word, so a Camie-suggested tag keeps the exact casing the suggestion UI showed when it round-trips through POST /api/tags on Accept — addressing "auto-suggested tags must obey capitalization" and "don't mangle acronyms" in one rule. Trade-off (operator-chosen): all-caps input no longer folds to Title Case, so case-variant merging in #714 still folds the dominant lowercase-vs-Title case but leaves all-caps stylizations distinct (protecting acronyms wins). Tests updated + a new test documenting acronym preservation / non-folding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9374f63953 |
fix(posts): post-card thumbnail strip spans the full hero width
The hero collage's thumbnail rail hard-capped at 3 fixed-80px cells, left- aligned, so it never reached the edge of the (50%-width) hero. Make the rail a CSS grid of equal columns (1fr) at a fixed height that stretches to the hero's full width: show up to 5 thumbnails, and when a post has more images than fit, the last cell becomes the "+N" overflow tile (count unchanged). Column count is driven by --fc-rail-cols so the strip always reaches the hero edge regardless of image count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6332ae13fd |
style(tags): C416 — use dict(members) instead of identity comprehension (#714)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3c89223dcb |
feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714)
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps whatever casing it was created with. This adds a maintenance action that Title-Cases every existing tag (collapsing whitespace) and merges case/whitespace-variant duplicates into one. Backend: - tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor (prefer an already-canonical member → no rename/self-alias; else the best-connected tag → fewest FK repoints; else lowest id), merges the variants INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/ aliases/series_page repoints + protective ML aliases), then renames the survivor to canonical. Losers are deleted before the rename so there's no transient unique-index clash; commits per group and isolates failures per group. Idempotent — an already-canonical lone tag is a no-op. - normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i. - POST /api/admin/tags/normalize: dry_run=true returns a projection inline (group/collision/rename counts + sample); dry_run=false enqueues the task. Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup tab) — preview → apply (polls the activity dashboard to terminal status), behind a back-up-first warning. admin store gains normalizeTags(). Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept separate, ML-known loser keeps a protective alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
23f452021f |
fix(tags): Title-Case operator-entered tags at create endpoint only (plan #701)
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in the POST /api/tags handler so operator-entered tags get clean display casing. It is NOT applied in the shared find_or_create / rename paths — those are used by the ML tagger and allowlist matching, which must preserve the booru vocabulary's original casing (Title-Casing it broke apply_allowlist matching). find_or_create / rename keep case-insensitive lookup + clash detection so a differently-cased entry dedups onto the existing tag instead of forking. Tests updated to expect Title-Cased create output (sunset→Sunset, character:Saber→Character:saber, http://example.com→Http://example.com) and a dedicated normalize_tag_name unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
62cca64dce |
feat(downloads): live per-file progress on running events — #709
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
89dfa42e18 |
fix(showcase): over-sample + random-order to break near-dup clustering — #699
TABLESAMPLE SYSTEM_ROWS reads CONTIGUOUS rows from each sampled page, so
sequentially-imported near-duplicates (multi-image posts, variant sets) came
back adjacent and clustered in the showcase ("three near-identical in a row").
Sample limit*5 rows (spanning more pages) then ORDER BY random() before taking
limit — breaks the physical adjacency for much better spread, still cheap
(random() over a few hundred rows, not the whole table).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
2b69540ecc |
feat(tags): Title-Case normalization on create/rename — #701 (core)
Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).
normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.
Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
4fe53cdf6b |
fix(modal/tags): fandom list, modal kebab, ESC-after-accept — #712 #711 #700
#712 — fandom picker showed no existing fandoms: loadFandoms enumerated via /tags/autocomplete with q=' ', which the backend strips to empty → returns []. Switch to the cursor-paged /tags/directory?kind=fandom (loop all pages); drop the load-once guard so the dialog reflects fandoms created elsewhere. #711 — modal tag-chip kebab never opened: the kebab + menu were nested INSIDE the v-chip, which swallowed the click / mis-anchored the teleported menu. Un-nest it as a sibling v-btn using the standard v-menu activator slot (Vuetify wires the click and stacks the overlay above the modal natively). Removes the openTagId workaround. #700 — ESC didn't close the modal after accepting a suggested tag: the guard suppressed close while ANY .v-overlay--active existed, which includes tooltips — a lingering tooltip blocked the close. Exclude .v-tooltip from the guard so only real interactive overlays (menus/dialogs) keep ESC from closing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
cb9b286c53 |
fix(maintenance): stage re-extract under the artist dir so members link — #713 part 2 fix
The integration test caught it: _import_media re-derives the artist by path-walk from the attribution path (ignoring the explicit artist) AND _copy_to_library lands members relative to that path. Staging the archive in /tmp meant the artist didn't resolve (provenance skipped) and members would land in the temp dir. Stage under images_root/<slug>/<platform>/<post>/ instead so the artist resolves and members land in the real library; remove only the staged archive + sidecar afterward (members stay). Require a real artist+slug (skip + count otherwise). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a497104661 |
feat(maintenance): re-extract archive attachments + link to post — #713 part 2
Existing PostAttachments that are actually archives (filed opaquely before the magic-byte gate) need extracting retroactively. cleanup_service. reextract_archive_attachments scans PostAttachments, magic-detects the archives, and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place in a temp dir — so the members extract and re-link to the SAME post via find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by sha256). Enqueues thumbnail+ML for new members. Wired as a maintenance-queue Celery task (tasks/admin) + POST /api/admin/maintenance/reextract-archives (202) + a "Re-extract archive attachments" card in Settings → Maintenance. Test: a zip stored under a mangled extension-less name extracts + links its member to the post via ImageProvenance, and a second run is a no-op (idempotent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5bb25245a5 |
fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1
Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").
Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.
Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
911d535f56 |
fix(artists): card preview dead-space + empty-state flicker on first load
Two operator-reported Artists-view bugs. Layout: ArtistCard previews used aspect-ratio: 3/1 + max-height: 220px — when a lone grid column got wide (>~660px), the max-height made the aspect-ratio box shrink its own WIDTH to keep the ratio, leaving dead space beside the 3 thumbnails. Dropped max-height (kept the min-height floor) so the strip fills the card width; lowered the grid min 440→360px so a lone column stays <732px (strip ≲244px) and desktop gets 2+ columns. Flicker: isEmpty checked !loading, but on the first render loading is still false (the initial loadMore fires in onMounted, after paint) so "No artists match" flashed for a frame before the spinner/data. Added a reactive `loaded` flag (true after the first load attempt, reset on reset()); isEmpty now also requires it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e82c2ee57b |
feat(subscriptions): dry-run backfill preview — B4 preview (plan #708)
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
cd43439401 |
feat(ingester): graceful mid-walk cancel on Stop — B4 cancel (plan #708)
Owning the walk lets Stop interrupt a live backfill chunk instead of letting it run to its ~14.5-min time-box. ingest_core.run now polls _backfill_state at each page boundary (a short SELECT, never held across the walk) and bails with PARTIAL when an operator Stop has popped it. Latched on the first observed "running" state so a run invoked without one (unit test / stale call) never self-cancels. Progress is already checkpointed per-page, so a restart resumes from the cursor; Stop clears it for a clean reset. No UI change — the existing Stop button now just takes effect immediately. Tests: _still_running reads the state; a latched run bails PARTIAL at the next boundary when the state disappears (only the pre-cancel post ran). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
bde19944db |
test(patreon): fix _BoomSession stub for the B5 headers kwarg
test_one_failure_isolated's _BoomSession overrode get() without the headers kwarg _fetch_to_file now passes (B5 Range resume), so the call TypeError'd and both items errored. Add headers=None to match the base fake. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
402086c34c |
feat(patreon): resume partial media downloads via HTTP Range — B5 (plan #708)
Owning the media fetch means a mid-download transport cut no longer refetches from zero. _fetch_to_file now resumes: on a transient retry, if bytes already landed in the .part, it requests Range: bytes=<have>- and appends on a 206; falls back to a clean truncate-and-restart if the server ignores Range (200) or the range is past EOF (416). The .part staging means a non-range server never corrupts the output — worst case is the old behavior (refetch from zero). Tests: mid-stream cut resumes from the offset (asserts the Range header); a Range-ignoring server refetches clean (no double-write). Test session fakes updated to accept the new headers kwarg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
fb7383eea7 |
feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708)
Owning the native client means we see the 429 Retry-After header — previously discarded. PatreonAPIError now carries `retry_after`; on a PERSISTENT page-fetch 429 the client attaches the server's raw Retry-After seconds. New DownloadResult.retry_after_seconds; patreon_ingester._failure_result sets it on RATE_LIMITED. download_service._update_source_health passes it to set_platform_cooldown as `seconds=`, clamped to [60, 3600] (a tiny hint can't leave the platform effectively un-cooled; a huge one can't strand it for hours); no hint → the flat 900s default. So a rate-limited platform cools for as long as the server actually asks, not a fixed guess. Tests: terminal 429 surfaces retry_after (test_patreon_client); cooldown honors + clamps the hint, falls back to default when absent (test_download_service). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e47fa0cf4b |
refactor(downloads): unify phase-2 dispatch in download_backends — A5 (plan #707)
Mirror verify_source_credential: download_backends.run_download is now the single download entry, so that module is the ONE registry of how each platform both downloads AND verifies — the seam that makes adding a platform a bounded job (write its adapter construction next to its verify). The native-ingester construction + campaign-id resolution moves out of download_service into download_backends._run_native_ingester. download_service.download_source drops its `if uses_native_ingester ... else gdl.download` branch and calls one `self._run_download(...)` (a thin delegate to run_download passing the service's gdl + sync sessionmaker). mode (tick/backfill/ recovery) is still chosen there from the backfill state machine and threaded through. Removed the now-unused PatreonIngester / resolve_campaign_id_for_source imports from download_service. Tests: the phase-2 stub seam moves from svc._run_patreon_ingester to svc._run_download (helper + the db-release test); the two native-construction tests repoint to download_backends.run_download (patching download_backends.resolve_campaign_id_for_source / PatreonIngester). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b211900390 |
refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
697a86d31c |
fix(ingester): close #5 within-chunk live posts + #8 video transient retry
Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes both. #5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_ lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the badge didn't move during a chunk (up to ~14.5 min) and over-counted the re-walked resume page. The plan called for within-chunk live updates. Move ownership of _backfill_posts into the ingester: ingest_core writes a monotonic absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and once at the end, EXCLUDING the resumed page so it no longer inflates across chunks. download_service seeds posts_base from prior chunks and stops touching the key (the lifecycle now carries the ingester's committed value forward). #8 (per-media transient/permanent retry) covered only the plain-GET path (_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient (back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError) is permanent (yt-dlp already did its own network retries) → fail fast to the per-item/dead-letter path. Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist (test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite (test_download_service); video transient-retry + permanent-fail-fast (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9a2cd569c3 |
refactor(ingest): extract platform-agnostic Ingester core — roadmap #9 (plan #706)
Factor the native-ingest orchestration out of PatreonIngester into a reusable ingest_core.Ingester base, parametrized by client/downloader/ledger-models/ constraints/key/platform/error_base. PatreonIngester becomes a thin adapter: it resolves the Patreon client/downloader, wires the seen/dead-letter models + UNIQUE-constraint names + _ledger_key into super().__init__, and overrides _failure_result with the Patreon exception taxonomy. Behavior-preserving — no table rename, no migration; the public surface (PatreonIngester, _ledger_key, DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged. This is the strategic seam: SubscribeStar/etc. now migrate by writing a ~40-line adapter, not by re-implementing the tick/backfill/recovery walk, tiered skip, checkpoint, and dead-letter logic. run() moved to ingest_core, so the budget test's monotonic patch repoints to ingest_core.time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d592e0ca02 |
feat(patreon): within-pass transient retry for media GETs — #8 (was overstated as done)
Honest completion of roadmap #8. Previously a NON-429 media failure (a connection reset, a timeout, a truncated stream, a 5xx) was an immediate terminal "error" for the pass — only retried on the NEXT walk. Now _fetch_to_file retries TRANSIENT failures in-place with backoff (transport blips incl. mid-download, 429 honoring Retry-After, and 5xx; up to 3 tries), while PERMANENT failures (404 gone / 403 forbidden) fail fast straight to the error → dead-letter path — re-fetching them is pointless. This makes the transient-vs-permanent split explicit instead of leaning on the next-tick cycle. (#1's 429 backoff + #7's dead-letter covered most of #8's value; this is the missing in-pass transient piece I'd loosely marked "folded".) Tests: a connection blip / a 5xx is retried then succeeds; a 404 errors with NO retry; an exhausted transient becomes a terminal error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
7a872a3619 |
feat(patreon): dead-letter ledger for permanently-failing media — #705 step 2 (#7)
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux, persistently-corrupt bytes) used to re-error forever and re-burn chunks. New `patreon_failed_media` table (alembic 0038, chains 0037) records per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the ingester skips it on routine tick/backfill walks (tier-1.5, folded into the seen/skip predicate). Recovery BYPASSES it (the operator's "try everything again" re-attempts dead media). A clean download clears the row (recovered); errors/quarantines upsert-increment it. Surfaced as run_stats.dead_lettered_count. - New PatreonFailedMedia model + migration; ingester _dead_keys / _record_failures (on_conflict increment) / _clear_failures. - skip = seen | dead (empty in recovery); failures recorded post-fetch on short sessions (same pattern as the seen-ledger). Tests: a media erroring 3× is dead-lettered + skipped (no download attempt); recovery re-attempts a dead media and clears it on success; a clean download clears a sub-threshold failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4bb11ce7dc |
feat(patreon): incremental cursor checkpoint mid-walk — #705 step 1 (#6)
A worker SIGKILL (hard-time-limit backstop) mid-chunk lost the whole
chunk's walk — the cursor was only persisted at chunk boundaries by phase
3, so the next tick re-walked from the chunk start. Now the ingester
checkpoints _backfill_cursor to the DB at each page boundary (backfill/
recovery only) via an ATOMIC single-key UPDATE (config_overrides::jsonb →
jsonb_set('{_backfill_cursor}') → ::json), so it never clobbers operator
config or other backfill keys. On a crash the last mid-walk cursor
survives → the next chunk resumes near the crash, not the chunk start.
phase 3 still writes the final cursor (same value); this is the safety net.
Tests: a backfill walk leaves the last page's cursor in the DB (written by
the ingester, before any phase 3); a tick never checkpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
e42a86d995 |
test(patreon): fix #704 — quarantine status + budget-cut cursor assertions
Two test breaks from the structured-results change: - An existing downloader test pinned a corrupt file to status "error"; it's now the distinct "quarantined" status (the new behavior). Updated it + removed the duplicate I'd added. - The budget-cut ingester test asserted the checkpoint cursor was the last FULLY-processed page (CUR1); it's actually the page we were cut on (CUR2, entered + cursor emitted before the budget check), matching the prior parse_last_cursor(last) semantics. Corrected the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b2e59e7e17 |
feat(subscriptions): live posts-processed progress on backfill/recovery — #704 step 2
The running badge only showed the chunk counter; now it shows posts walked — real walk progress. The ingester already reports posts_processed per chunk (step 1); the backfill lifecycle accumulates it into config_overrides._backfill_posts across chunks. SourceRecord exposes backfill_posts; start_backfill/start_recovery clear it (fresh walk); stop clears it too. SourceRow/SourceCard badge renders "Recovering · 45 posts" (falls back to "(N)" chunks before any posts are counted). Per-chunk accumulation (no mid-walk DB write) — simple and race-free; a small over-count from each chunk re-walking its resumed page is fine for a progress indicator. Tests: lifecycle accumulates posts_processed across chunks; start clears a prior _backfill_posts and the record exposes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e53f8959af |
feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5b615b7ded |
feat(sources): pre-flight credential verify on backfill/recovery arm — #703 step 2
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
d6c15f4ea0 |
test(patreon): fake gdl needs real _rate_limit for native pacing — #703 step 1 fix
_run_patreon_ingester reads self.gdl._rate_limit for the native pacing config (max(0.5, rate_limit/4)); the MagicMock fake gdl broke the arithmetic. Give it real _rate_limit/_validate_files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3b2f7a41c3 |
feat(patreon): ingester rate-limit resilience — #703 step 1
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:
- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
`rate_limit`, read off self.gdl) as a pre-download sleep on real media
downloads only (skips don't pace); pace /api/posts page fetches with the
per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
RATE_LIMITED. Light 2-retry on a media-GET 429 too.
Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).
Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
218bfebb92 |
feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)
The credential Verify button still ran gallery-dl --simulate for Patreon after the cutover — testing the wrong path (and prone to the vanity "Failed to extract campaign ID" the native resolver fixes). Wire it to the native ingester, behind a DRY dispatch so callers never branch on platform. - services/download_backends.py (new): the ONE place that knows which platforms are native vs gallery-dl. `uses_native_ingester(platform)` is the shared predicate; `verify_source_credential(...)` is the uniform probe (same (ok|None, message) contract for both backends). As a platform migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download routing and verify switch together. - PatreonClient.verify_auth(campaign_id): one authenticated /api/posts fetch → True (valid) / False (401/403/HTML-login) / None (drift or network — inconclusive, not a credential verdict). - patreon_ingester.verify_patreon_credential(): resolve campaign id, then verify_auth — the verify counterpart to the download path. - patreon_resolver.resolve_campaign_id_for_source(): extracted the override / id:-URL / vanity resolution into ONE helper now shared by the download ingester and verify (download_service no longer carries its own copy + regex; −`import re`). - download_service: routes on uses_native_ingester() instead of inline `== "patreon"` (3 sites); uses the shared resolver. - api/credentials: calls verify_source_credential — no platform branch. Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/ vanity/none), the dispatch predicate, verify_patreon_credential glue, credentials endpoint proves Patreon uses the native path (gallery-dl verify asserted not-called); repointed the gallery-dl verify test to subscribestar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ec43e823e1 |
feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |