635 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusen	0533807669	Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main ext-1.0.7	2026-06-03 14:17:42 -04:00
bvandeusen	d3245f0c22	feat(ext): verify cookies in-browser before uploading (1.0.7) ... Pre-upload verify request: after capturing the live browser cookies, hit a known authenticated endpoint with credentials:'include' from the extension's background context. If the platform reports we're not logged in, abort the upload so we don't overwrite FC-side credentials with stale data. - platforms.js: add `verify` config per cookie-auth platform - hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF _init_site_filters; same 401 path the operator hit 2026-06-03) - patreon: GET /api/current_user (clean 401 when logged out) - subscribestar, deviantart: no stable auth endpoint, skip verify - cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}. ok=true/false/null tri-state — null = verify not configured, caller treats as "proceed". - background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the upload; failures bubble up with the platform's name + reason. - popup.js: success message now appends "(verified ✓)" when applicable. - manifest + package.json: 1.0.6 → 1.0.7.	2026-06-03 14:04:45 -04:00
bvandeusen	279dff3fb6	Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main	2026-06-03 13:18:44 -04:00
bvandeusen	e450145304	fix(ml): preserve digit-only tag names in normalize (year tags) ... Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005' returned None even though they're legitimate (booru year-tag shape). Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^', '+_+') still drop since they contain neither letters nor digits.	2026-06-03 13:09:35 -04:00
bvandeusen	a6e8d4b52e	feat(ml): normalize Camie suggestion names to human-readable ... Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`, `#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were surfacing raw in SuggestionsPanel — and worse, the SAME raw string was written to tag.name on Accept, polluting the DB with `underscored_lowercase` names that don't match the operator's "Title Case" tag convention. Add backend/app/services/ml/tag_name.py with a single normalize() applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing _(disambiguator) blocks iteratively, strip wrapping quotes, underscores to spaces, space after colon, title-case each word's first char, preserve hyphens/apostrophes/digits, drop entries with no letters). Wire into SuggestionService.for_image: - raw Camie key kept for alias_map lookup (alias rows are hand-curated against raw keys; don't disturb) - display_name = normalize(raw); None means drop the candidate - existing-tag lookup widened to case-insensitive match against BOTH raw and normalized forms so legacy underscore-named Tag rows accepted before this change still surface as "existing" not "+ new"	2026-06-03 13:00:08 -04:00
bvandeusen	37e66cddc4	Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main	2026-06-02 19:35:04 -04:00
bvandeusen	f1860866de	chore(modal): drop the ?image=N soft-compat from G5.4 ... Operator confirmed they have no existing ?image=N bookmarks to preserve, so the soft-compat read on initial mount + router.replace strip is dead weight. The modal is now purely a Pinia overlay — no URL involvement on open OR initial mount. Drops 33 lines plus the now-unused vue-router imports in both ArtistGalleryTab (entire onMounted gone) and GalleryView (just the ?image=N block; the post_id/tag_id filter handling stays).	2026-06-02 19:24:09 -04:00
bvandeusen	9cf6b2d363	Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main	2026-06-02 19:09:49 -04:00
bvandeusen	b181d779fe	fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)	2026-06-02 18:49:24 -04:00
bvandeusen	0fbb19dc24	fix(modal): TagPanel kebab — apply the wrapping-span fix that the prior commit missed ... The previous commit (8326e54) updated the CSS but the structural edit to the v-menu wrapper didn't take. Re-applying so the chip kebab actually opens.	2026-06-02 18:48:53 -04:00
bvandeusen	8326e5447a	fix(modal): kebab menus weren't opening (chip + suggestion rows) ... Operator-flagged 2026-06-02. Both kebab activators were broken: - TagPanel chip kebab had `@click.stop` directly on the v-icon activator. In Vue 3, an explicit @click on the same element as `v-bind="props"` overrides the spread onClick — so Vuetify's activator handler never fired. Menu never opened. - SuggestionItem kebab didn't have @click.stop, but for consistency and to make both kebabs follow the same shape, wrap it too. The fix: each kebab is now wrapped in a `<span @click.stop>`. The v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu; the bubbling click then reaches the span where stopPropagation absorbs it before it can affect a parent (the v-chip's close button in TagPanel's case).	2026-06-02 18:48:32 -04:00
bvandeusen	1fd594baaf	chore(ml): suggestion_threshold default 0.50 → 0.70 ... Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) surfaces too many low-confidence picks in the modal's Suggestions rail. 0.70 keeps the rail signal-rich while still showing more than the original 0.95 (which hid almost everything). Alembic 0033 updates the singleton row conditionally — only rows still at the old 0.50 default flip to 0.70. Operators who tuned to some other value via Settings → ML keep their pick. Settings UI already exposes both sliders (MLThresholdSliders.vue), so further tuning continues to work without a deploy.	2026-06-02 18:38:12 -04:00
bvandeusen	ecac6c4bda	fix(audit-g5): centroid version DB-as-truth + modal as overlay ... Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4). G5.1 — Centroid version no longer drifts: CentroidService now reads MLSettings.embedder_model_version (the DB row tag_and_embed already writes from) for both the centroid model- version stamp and the drift-detection comparison. Previously the centroid sites imported MODEL_VERSION from env, so the version stamped on centroids could disagree with the version stamped on the embeddings they were built from. By construction those now match, so list_drifted won't silently miss the env-vs-DB drift case. embedder.py keeps MODEL_VERSION as an env-driven constant for the actual model loader — that's a different concern (which weights are loaded) from the version-stamp that gets persisted alongside data. G5.4 — Modal is a Pinia-only overlay: The previous URL↔modal sync in GalleryView and ArtistGalleryTab leaked the modal across route changes (RouterLink to /artist/<slug> left the modal mounted on top of the new route) and re-opened it on history back/forward with stale ?image=N entries. Now: openImage() just calls modal.open(id) — no URL push. GalleryView's dead closeImage helper is deleted. A route.name watcher in App.vue closes the modal whenever the route changes, which auto-fixes RouterLink-in-modal and back/forward. Backward-compat: ?image=N is still honored on initial mount as a one-shot deep-link opener, then router.replace strips the query so the URL doesn't re-trigger and no extra history entry is added. Existing bookmarks / shared URLs keep working; new opens stay Pinia-only.	2026-06-02 18:28:57 -04:00
bvandeusen	6ef0fed41f	Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main	2026-06-02 18:07:25 -04:00
bvandeusen	9f7261b9c0	fix(audit-g5c): set CURATOR_BOOTSTRAP_NEW_KEY=1 in conftest ... CredentialCrypto's safety check fires on create_app() instantiation because the test environment has no pre-seeded Fernet key file. Set the bootstrap env var before any test imports so the auto-create path is allowed during tests.	2026-06-02 17:55:25 -04:00
bvandeusen	f05aaa707b	fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard ... Alembic 0032 adds Source.error_type (varchar(32), indexed). _update_source_health stamps it alongside last_error on status='error' and clears it on 'ok'. SourceRecord/to_dict exposes it. FailingSourcesCard renders a colored chip next to the consecutive- failures count, with a tooltip explaining the suggested operator action. Color reflects intent: - warning (yellow) — operator action needed (auth_error) - info (blue) — backend-paced (rate_limited / timeout / network_error / partial / tier_limited) - error (red) — likely terminal without intervention (not_found / access_denied / validation_failed / unsupported_url / http_error / unknown_error) Audit 2026-06-02: the backend computed 13 ErrorType categories but only the free-text last_error reached the operator. Bulk-triage by class ("all auth_error → rotate cookies", "12 rate_limited → just wait") required opening Logs per row.	2026-06-02 17:54:44 -04:00
bvandeusen	4df98171ab	fix(audit-g5c): refuse silent Fernet key regeneration on partial restore ... Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet key whenever the key file was missing — no log, no warning. The failure mode the audit flagged: a partial disaster restore where the DB was restored but `/images/secrets/` was lost would produce a working-looking system in which every authenticated download fails AUTH_ERROR until the operator re-uploads every credential by hand. Two opt-ins now needed for auto-creation: 1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR 2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup) Otherwise the constructor raises `MissingCredentialKey` so the app fails fast at startup and the operator can restore the key file from backup before encrypted_blob rows go undecryptable. Also: docstring path was wrong (said "images/data root" but actual location is `/images/secrets/credential_key.b64`) — corrected. Tests updated to pass `bootstrap_ok=True` explicitly, and two new tests cover the safety behavior (missing-key-raises, env-var-bootstraps).	2026-06-02 17:04:27 -04:00
bvandeusen	8d75ade1d5	fix(audit-g5b): race-poisoning in three find-or-create sites ... Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the race under concurrent writers / recovery-sweep replays, poisoning the outer transaction with an unrecoverable IntegrityError and crashing the calling task. Same shape as the banked 2026-05-26 ImageProvenance incident (see reference_scalar_one_or_none_duplicates memory). Mirrors the importer._get_or_create pattern in all three: savepoint via begin_nested + IntegrityError rollback to that savepoint + re- SELECT to grab the row the other worker committed first. - importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store is sha-addressed so both workers race to write the same on-disk path (shutil.copy2 + rename is idempotent), so no extra cleanup needed. - TagService.find_or_create: partial uniqueness index on (name, kind, COALESCE(fandom_id, -1)). The previous docstring claimed "INSERT ... ON CONFLICT" but the implementation was SELECT-then-INSERT with no recovery. - ArtistService.find_or_create: already had IntegrityError handling but did session.rollback() (unwinds the WHOLE transaction); now uses begin_nested + sp.rollback() so the surrounding request's progress isn't lost.	2026-06-02 17:02:12 -04:00
bvandeusen	75c63e1511	fix(audit-g5a): ruff isort — platforms after patreon_resolver	2026-06-02 16:55:56 -04:00
bvandeusen	98673d4dca	fix(audit-g5a): small architectural cleanups bundle ... Five small G5 findings from the 2026-06-02 audit. Each is local and follows an established FC pattern. - download_service: replace hardcoded ('discord','pixiv') tuple with auth_type_for(platform) == 'token'. A 7th token-platform now picks up the right credential path without touching this site. - /api/tags/<source_id>/merge enqueues recompute_centroid.delay after merge so the target's centroid reflects its new image set immediately. Daily list_drifted catches it within 24h, but eager recompute closes the suggestion-quality dip in the meantime. - backfill_thumbnails added to beat_schedule (daily). The task docstring claimed periodic Beat but the entry was never registered, so the library got no self-healing thumbnail repair; only the manual admin-UI button fired it. - modal.createAndAdd pushes a kind='fandom' tag into tagsStore.fandomCache so FandomPicker sees the new fandom on next open. Was: cache-gated load (length===0) skipped refetch, new fandom invisible until full page reload. - cleanup cluster: - Drop .webp from cleanup_service.unlink — thumbnailer only writes .jpg/.png; the third tuple member was dead code. - Drop effective_date from /api/gallery/scroll response — no FE consumer reads it. Service still computes the attribute for timeline ordering; this just trims the JSON. - Rename store.recentMinute → store.recentRuns across the systemActivity store + three consumers (SystemActivitySummary, QueuesTable, SystemActivityTab). The data is the last 200 runs (not actually "last minute"), so the name lied. NOT in this bundle: the duplicate tag-merge endpoint (/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests on the admin variant; consolidation is its own change.	2026-06-02 16:46:46 -04:00
bvandeusen	89b48f8f35	Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main	2026-06-02 16:15:00 -04:00
bvandeusen	4bff1d8558	fix(audit-g4): status-enum miss batch ... Five extension-miss findings from the 2026-06-02 audit, where a status value was added on one side but a downstream consumer didn't pick it up. - download_service._phase3_persist: explicit branches for ImportResult.status in ('failed','refreshed'). For 'failed' (archive probe crash from _import_archive), unlink the source file so the filesystem scanner doesn't re-import and re-crash on the same archive forever. 'refreshed' is currently unreachable from the download path (no deep=True) but matches the importer's documented contract; treat as 'attached'. - gallery-dl backfill auto-complete now gates on dl_result.success + no error_type, not just return_code==0 + files_downloaded==0. VALIDATION_FAILED exits the subprocess with returncode=0 and files_downloaded=0 when every file was quarantined, matching the prior predicate exactly and zeroing the operator's armed backfill budget on the FIRST quarantine run instead of decrementing. - attach_in_place archive dispatch now threads artist + source_row through _import_archive (and _import_media for archive members) and _supersede. The path-walk fallback (_resolve_artist) is still used by filesystem-import; the download path now binds ImageProvenance to the explicit subscription Source instead of rediscovering by (artist_id, platform). - Three FE handlers now recognize status:'deferred' from /api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting "event #undefined"), SubscriptionsTab.checkAll (was counting deferred as queued), DownloadEventRow.onRetry (was saying "re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll which already had it. - celery_signals._queue_for now maps backup/admin/library_audit prefixes to 'maintenance' (matching task_routes). TaskRun.queue was returning 'default' for those rows, so per-queue dashboard filters and per-queue threshold overrides (added in G3) silently missed them.	2026-06-02 16:04:59 -04:00
bvandeusen	d60e0b9494	Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main	2026-06-02 14:49:28 -04:00
bvandeusen	e30f50e6fe	fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts ... Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days).	2026-06-02 14:30:46 -04:00
bvandeusen	9c27a2d3c7	Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main	2026-06-02 14:17:12 -04:00
bvandeusen	e66987f092	fix(audit-g2): async race / state-leak across eight stores ... Extracts gallery.js's hand-rolled inflightId pattern into a new useInflightToken composable; adopts in every store that previously had no guard against late-response overwrites or wrong-image URL interpolation. Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62) flagged: - modal.removeTag rolled back the chip rail unconditionally even when only the secondary dismiss POST had failed — UI lied until refresh. And all tag-mutation URLs interpolated currentImageId AFTER an await, so a fast prev/next could route DELETE/POST to the wrong image. Both fixed: split try/catch (dismiss failure surfaces a warning, doesn't roll back the delete); imageId captured at call-time and used in URLs throughout. - suggestions.accept dereferenced currentImageId after the awaited POST /api/tags, so the subsequent /suggestions/accept could apply A's chosen tag to image B AND push it to B's allowlist. Fixed by capturing imageId at click-time + inflight guard on load(). Same shape across artist / downloads / artistDirectory / tagDirectory / posts stores: rapid filter/nav changes used to interleave responses (last-writer-wins). Now the late response is discarded and the most-recent request wins. Filter-change-during- search no longer drops the second fetch because the loading flag was still true from the first. gallery.js's inflightId removed in favor of the shared composable so the pattern stays consistent.	2026-06-02 14:07:58 -04:00
bvandeusen	93e37681b7	Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main	2026-06-02 13:29:17 -04:00
bvandeusen	80ef9bce48	fix(test): credentials.upload reflects record into cache (not invalidate) ... Stale assertion pinned the buggy pre-G1.6 behavior (the test name "upload invalidates the cache" literally describes the bug). The audit's correction makes upload mirror the returned record into the store so the card updates immediately — update the assertion to match the corrected behavior.	2026-06-02 13:11:39 -04:00
bvandeusen	3898ce7be4	fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit ... - BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout 30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat TimeoutExpired (matched the tick 870s/900s rationale). Backfill runs that hit the cap let the next tick continue via the archive. - recover_interrupted_tasks orphan UPDATE now stamps finished_at; without it cleanup_old_tasks' WHERE finished_at<cutoff never reaped orphan-swept rows. recover_stalled_task_runs also now sets duration_ms (matches celery_signals.finalize's millisecond math). - ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3 on Source creation, mirroring SourceService.create. Without it, Firefox quick-add on a creator with >20 unsynced posts walked the full feed until subprocess timeout. Renamed the constant from _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module. - gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside NO_NEW_CONTENT — the download path (line 712) already does, and TIER_LIMITED proves auth reached the post and was told it was tier-gated. Verify endpoint previously showed red on this and prompted operators to rotate working cookies. - prune_unused_tags now runs a single DELETE with the NOT-IN predicate find_unused_tags uses, instead of SELECT-ids → DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that would have surfaced on a tag explosion (>65k unused tags). - credentials.upload() reflects the returned record into the store cache (`.set(platform, rec)`) instead of evicting it; previously the card briefly rendered "no credential" between upload and loadAll().	2026-06-02 11:24:05 -04:00
bvandeusen	64ca858574	Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main	2026-06-02 08:26:58 -04:00
bvandeusen	91be9df671	fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount ... attach_in_place mirrored only the media flow, so gallery-dl-downloaded zips/PDFs/audio bounced back as `skipped+invalid_image`, which download_service counted as an ingest error and flipped runs to status="error" despite N successful image attaches. Lustria patreon event #38998 (21 images + 1 OST zip) went red for exactly this reason. Now attach_in_place dispatches the same way as import_one: archives → _import_archive (extracts media members, captures archive as PostAttachment), non-media → _capture_attachment. Download_service accepts the new `attached` result and treats non-duplicate skips as soft skips, not ingest errors. Also: ShowcaseView always loadInitial() on mount, not just when the store is empty — Pinia persists across navigations and operator wants a fresh shuffle every time the showcase loads.	2026-06-02 08:16:13 -04:00
bvandeusen	412edec028	fix(showcase): don't exhaust on all-dupe batches; retry up to 8x ... /api/showcase returns a random sample; after enough scrolling, an unlucky 3-item batch can be entirely in the `seen` Set, which was flipping `exhausted=true` and surfacing "End." mid-scroll. The showcase is endless by intent — only a genuinely empty API response (library has 0 images) should mark it exhausted. Tiny-library fallback: cap retries at 8 to avoid spinning forever.	2026-06-01 23:37:21 -04:00
bvandeusen	937421485d	fix(modal): refresh tag chips after accepting a suggestion ... Operator-flagged 2026-06-01: clicking Accept on a suggestion added the tag to the database but the modal's chip rail kept showing the old set. Suggestion would disappear from the suggestions list (suggestionsStore drops it from byCategory) and the toast would confirm "Tagged: …", but visually the modal looked unchanged until you closed and reopened it. Root cause: useSuggestionsStore.accept() POSTs to the backend and updates its own state, but never tells useModalStore that the backing image's tag set has changed. The addExistingTag and createAndAdd flows in TagPanel already call modal.reloadTags() after applying — the suggestions path needed the same refresh. Two-line fix in SuggestionsPanel: after store.accept() and store.aliasAccept(), call modal.reloadTags() so TagPanel's `modal.current?.tags` rebinds.	2026-06-01 23:02:57 -04:00
bvandeusen	9d0c0b7da8	Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main	2026-06-01 22:34:38 -04:00
bvandeusen	43b778aa04	fix(test): include 'scanned' in all backfill result assertions ... Same 'change shared shape, miss pinned tests' trap as cfa4fb4 — the prior commit updated only test_backfill_mixed_aggregate; three sibling tests in the same file also pinned the old {enqueued,ok,regenerated} shape without 'scanned' and CI bounced.	2026-06-01 22:09:02 -04:00
bvandeusen	9cbdb70e13	fix(thumbnails): surface backfill results + tighten validity check ... Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails but triggering backfill found nothing." 1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill` returned just `{celery_task_id}` and the admin card said "Enqueued." with no counts. There was no way to tell whether the scan found 0 candidates, 5000 candidates, or whether the worker even picked up the task. "Found nothing" was indistinguishable from a broken queue. Fix: refactor the scan into a sync helper (`_run_backfill_scan`) shared by the Celery task and the API endpoint. The API now runs the scan in an executor and returns `{scanned, enqueued, ok, regenerated}`. The actual thumbnail generation work still goes to the thumbnail Celery queue per row via `generate_thumbnail.delay()` — the scan itself is fast (SELECT id+thumbnail_path + a file.stat() per row). 2. **`_thumb_is_valid` accepted header-only corrupt files.** The magic-byte check passed for any 8-byte file starting with a JPEG or PNG header, including empty/truncated/zero-pad files that browsers render as broken. Backfill counted these as `ok` and never regenerated. Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real thumbnails are minimum ~2KB even on solid-color sources; header- only corrupt files top out around 12 bytes. 256 is well above the corrupt floor and well below any legitimate thumbnail. Plus the admin card now shows the per-run counts instead of "Enqueued.": Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok	2026-06-01 21:57:29 -04:00
bvandeusen	8e4d252ae4	Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main	2026-06-01 21:52:42 -04:00
bvandeusen	bd06794647	fix(downloads): enqueue thumbnail + ML tasks per attached image ... Operator-flagged 2026-06-01: downloaded images stayed at thumbnail_path=NULL until a periodic backfill sweep picked them up, surfacing as broken-thumbnail tiles in the gallery for hours after the download landed. Importer.attach_in_place deliberately skips inline thumbnail generation (importer.py:591-592) so the import queue stays moving — the CALLING task is responsible for the enqueue. tasks/import_file.py already did this (line 228-239). tasks/download.py / download_service did not — every gallery-dl-attached image landed un-thumbnailed. Fix in download_service._phase3_persist: after each `attach_in_place` returning status in (imported, superseded), fan out `generate_thumbnail.delay()` + `tag_and_embed.delay()` for each image_id. Lazy import avoids circular-import risk between download_service and the celery task modules that depend on it. Mirrors the existing pattern verbatim — single source of truth for "what fires after a successful attach" remains a comment in two places (filesystem-import task, download orchestrator) rather than a shared helper, because the contexts differ enough (sync session vs async orchestrator) that abstracting would obscure more than it'd share. Test covers the happy-path with two attached files: both get the thumbnail enqueue AND the ML enqueue, with image_ids drawn from ImportResult (so future supersede-on-attach paths stay covered).	2026-06-01 21:39:17 -04:00
bvandeusen	fdd3e01f56	Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main	2026-06-01 21:09:28 -04:00
bvandeusen	f575cfb93b	ux(failing-sources): visible row separators + clearer hover ... The zebra striping in 717b601 was too subtle — inside the tonal error card, surface-tint contrast washed out and rows still looked uniform. Operator-flagged the same issue twice. Replace the surface-tint approach with hard visual structure: - Bottom border on each row (1px white/0.08) — clearly delineated rows regardless of card background - Hover darkens to black/0.25 — much more obvious than the previous error-tint hover, gives the eye a clear horizontal track - Slightly more vertical padding (8px vs 6px) for tap-target comfort - Drop the gap; borders are the separator now	2026-06-01 20:57:41 -04:00
bvandeusen	c82fb308b6	Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main	2026-06-01 20:44:23 -04:00
bvandeusen	717b601c81	ux(failing-sources): zebra striping + hover highlight on rows ... Operator-flagged 2026-06-01: with the failing-sources panel expanded, all rows have the same flat low-opacity background, no visual track from the artist name on the left to the Logs/Retry buttons on the right. Hard to tell which row a button belongs to when scanning down a list of 17. Three pure-CSS changes (no DOM, no logic): - 3px gap between rows (was 2px) — slightly more breathing room - Even rows get a darker surface tint (zebra striping) - Hover paints the whole row in a low-opacity error tint, so moving the cursor toward Logs/Retry lights up the whole horizontal band and the eye traces back to the artist name automatically	2026-06-01 20:14:48 -04:00
bvandeusen	cfa4fb4084	fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create ... `test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial value (0) when checking that the override works. Now that create() pre-arms enabled sources to 3, the assertion is stale — the test was already verifying the override path, the pre-assertion just decorated it. Drop the pre-assertion; keep the override check. Other tests use raw Source(...) constructors (default to 0 via the column server_default), not SourceService.create(), so they're unaffected.	2026-06-01 20:12:55 -04:00
bvandeusen	2aa2002f22	feat(subscriptions): newly added enabled sources start in backfill mode ... A freshly created subscription has no gallery-dl archive yet, so the first poll in tick mode would walk forever — exit:20 doesn't trip until 20 contiguous archive-hits, and there are none. The wall-clock cap kicks in mid-walk, and the partial-success classifier (plan #544) gracefully labels it status=ok, but the operator still wonders why their new subscription isn't grabbing the back-catalog. Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`. Same default as the manual "Deep scan" button, same auto-decrement and auto-reset rules — once the queue drains (clean exit + zero new files) or the budget runs out, tick mode resumes naturally. Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'` that arrive disabled = False, or operator-added sources that start disabled deliberately) skip the pre-arm — they're never polled, no budget to waste.	2026-06-01 20:02:19 -04:00
bvandeusen	66ff671f09	fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos ... Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts hosted on Mux carry a JWT that encodes a playback restriction policy (`playback_restriction_id` in the token). The token signature alone is not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch. gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns 200 (token is valid, HEAD sends no Referer that Mux's policy rejects), but when yt-dlp follows up with the actual manifest GET it sends its own default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the single video fails — every other image in the same post still downloads. Static `downloader.ytdl.raw-options.http_headers` block now pins Referer and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch clears the policy check. Headers-only restrictions are now handled. Mux IP-range restrictions (if a creator opts into them) remain unfixable from our worker — those would need a Patreon-region IP. Per-video PARTIAL classifier (shipped in plan #544 last commit) already handles the residual case: a run that grabs all images and fails 1 video classifies as status=ok rather than flipping the whole event red.	2026-06-01 19:00:50 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00
bvandeusen	c9089b1d03	fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source ... gallery_service._provenance_clause artist branch was correlating its bare Post reference to the outer query's primary_post_id outer-join, so the artist filter silently matched zero rows for images with no primary post. Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner FROM rather than treating it as a correlated outer table. Five sidecar/import tests still asserted that a synthetic Source row appears after a filesystem import. Alembic 0030 retired that behavior; the Post sits null-source and the artist linkage lives on Post.artist_id. Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent, test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar, and test_apply_sidecar_recovers_from_integrity_error to assert post.source_id IS NULL + post.artist_id linkage instead.	2026-06-01 14:40:28 -04:00
bvandeusen	644d538bab	fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention	2026-06-01 14:23:18 -04:00
bvandeusen	ff35da4743	fix(lint): drop unused Source import after Post.artist_id refactor	2026-06-01 14:19:37 -04:00
bvandeusen	2f66de2928	feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics ... Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.	2026-06-01 14:17:52 -04:00