06f98acf3e25e303fa9590d7ebe5fdbbc7050c29
1040 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
5be6b9cada |
feat(explore): auto-focus the tag input on every image change
The workspace is built for rapid walk-and-tag, but the tag field was only focused once (TagAutocomplete's on-mount autofocus) — walking to a neighbour left focus behind, so the operator had to click the field each time (operator-asked 2026-06-26). TagPanel now exposes focusTagInput; ExploreView watches the focused image id and re-focuses the field on seed + every walk via nextTick. Reuses the existing focus path, so TagAutocomplete's mobile guard (no soft-keyboard pop on touch) is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4a1f255164 |
fix(modal): place meta + save block under Provenance, above Tags
Operator-clarified 2026-06-26: the dimensions/size/type + save (floppy) block should sit DIRECTLY above the Tags section — i.e. just under Provenance — not at the very top of the rail. Reorder the rail's main scroll area to Provenance → ImageMetaBar → TagPanel (Related stays pinned at the bottom). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
1728b43167 |
fix(modal): pin Related to rail bottom, floppy-disk download, drop suggestions cap
Right-rail layout fixes (operator-flagged 2026-06-26 — the prior change wasn't
the intended improvement):
- Pin the Related strip to the BOTTOM of the rail: the side becomes a flex
column with a scrolling main area (meta + provenance + tags + suggestions)
and a pinned Related footer (capped at 45% of the rail, scrolls past that).
Related now stays reachable no matter how long Tags/Suggestions run, and
self-collapses (no footer space) when there's nothing to show.
- Remove the 320px suggestions scroll cap (
|
||
|
bvandeusen
|
2d1cddd9b7 |
feat(explore): 3-pane tagging workspace — gallery | viewer | tag rail
Reworks Explore from "anchor + neighbour grid + cluster tag-gap rail" into a persistent 3-pane workspace that unfolds the image modal so you can tag while rabbit-holing (operator concept 2026-06-26): - LEFT neighbour grid (larger thumbs), click = walk; breadcrumb retained. - CENTER light viewer — reuses ImageCanvas + ImageMetaBar(:image) for the focused image; "Open full viewer" still launches the overlay modal. - RIGHT the modal's TagPanel, hosted on the anchor for modal-parity tagging (chips, autocomplete, suggestions + Accept, fandom-on-chip, T/"/" focus). Reuse without destabilising the audited modal store: TagPanel and SuggestionsPanel gain an optional `host` prop (default = modal store, so the image modal is unchanged); the explore store implements the same small tag-CRUD surface (current/currentImageId + reloadTags/addExistingTag/ removeTag/createAndAdd) over the anchor. ImageMetaBar gains an optional `image` prop for the same reason. Drops the mass/cluster tagger (TagGapPanel deleted; clusterIds/thumbById removed) — per-image tagging feeds the per-tag reference-embedding centroid better than bulk ops. Nav: keep the Explore tab but bare /explore now SEEDS a random image (GET /api/showcase?limit=1 → /explore/:id) so the tab kick-starts a rabbit hole; explicit meta.navOrder pins nav order (Explore after Gallery) since router.getRoutes() doesn't preserve declaration order. Note: the backend cluster tag-gaps route/service (#94a) is now frontend-orphaned — left in place; flag for a separate cleanup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
1aadf3267b |
fix(tags): correct directory image_count — fandom leg must correlate the outer tag
The directory card count regressed to a globally-inflated number (~every card showed the same ~469): the fandom leg used a doubly-nested correlated subquery — image_tag.tag_id IN (SELECT member.id WHERE member.fandom_id == Tag.id) — whose inner predicate did not correlate the outer Tag, so it matched EVERY character that has any fandom and counted all their images for every tag. The gallery scope and cleanup count were unaffected (they pass a literal tag id, a single-level subquery), which is why only the card diverged from the gallery. Rewrite the count as a single-level correlated scalar subquery: join `member` (the tag applied to the image) and match image_tag.tag_id == Tag.id (direct) OR member.fandom_id == Tag.id (a character of this fandom). Strengthen the directory test with a second unrelated fandom/character so a non-correlating fandom leg fails (count would read 4 instead of 3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
10434509d3 |
fix(tags): fandom views aggregate images via their characters
A fandom owns characters via Tag.fandom_id, but every image<->tag query went purely through direct image_tag rows, so a fandom only surfaced images literally tagged with it — images carrying one of its characters were invisible to its browse count, previews, and gallery filter. Derive membership at query time instead of materializing fandom rows (which would drift on every reassign/merge/remove). Add one shared predicate in tag_query.py — image_in_tag_scope / image_in_any_tag_scope: an image belongs to a tag if tagged with it directly OR (when the tag is a fandom) carrying a character whose fandom_id is that tag. The character leg is empty for non-fandom tags, so it applies uniformly with no kind branching. Route all read sites through it: - gallery _apply_scope: include, OR-groups, and symmetric exclude - directory image_count: correlated COUNT(DISTINCT) scalar subquery - directory previews: UNION direct + via-character, then ROW_NUMBER<=3 - cleanup count_tag_associations: Tier-B delete prompt now reports a fandom's true blast radius (was 0 for fandoms with no direct rows) find_unused_tags already protected fandoms via used_via_fandom; left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b85327a79d |
fix(celery): harden broker connection so workers ride out a Redis blip
A swarm overlay-network blip after the :latest redeploy left Redis healthy but transiently unreachable; a worker starting in that window crash-looped on the initial broker connect (kombu OperationalError) and needed a manual Redis reset to recover. Retry the broker forever on startup + at runtime (broker_connection_max_retries =None), add redis-transport socket options to the broker (short connect timeout, TCP keepalive, retry_on_timeout, periodic health check), and mirror the same on the Redis result backend. Now a transient outage self-heals when overlay routing returns instead of the worker exiting. Test pins the key resilience settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
3fcc4aeb43 |
fix(modal): scroll-cap the suggestions list so Related stays reachable
The right rail scrolls as a whole and ProvenancePanel already caps its cards + attachments, but SuggestionsPanel had no cap — a long suggestion set (the General bucket runs to dozens) stretched the rail and pushed the Related strip below the fold. Wrap the suggestion groups in a 320px max-height scroll box (hairline scrollbar matching the provenance regions), so suggestions scroll internally and Related stays visible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
7b712920a4 |
feat(explore): Explore view + tag-gap closing + modal meta/download (#94b–d, #4a/b)
Cluster C frontend, milestone #94. #94b Explore walk: new /explore/:imageId route + ExploreView + explore store. Anchor (reuse /api/gallery/image), neighbour grid (reuse /api/gallery/similar, 24), click a neighbour to re-anchor; in-memory breadcrumb that trims on backtrack (route is the source of truth). Empty/loading/error + no-embedding states. #94c tag-gap closing: components/explore/TagGapPanel — fetches /api/images/cluster/tag-gaps for the anchor+neighbours, a consensus-threshold slider (default 60%), per gap shows present/total + the missing thumbnails + 'Apply to N missing' → /api/tags/images/bulk/tags (source manual) → re-fetch. #94d entry points: 'Explore' button in the modal RelatedStrip; the TopNav entry comes free from the route's meta.title. #4a metadata HUD + #4b split Download: new modal ImageMetaBar (always-on, above ProvenancePanel) shows dimensions/size/type and a split Download button (default Download, chevron → Copy link via utils/clipboard — no clipboard-image, rule 95). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
0ecd1ce4f1 |
feat(explore): cluster-consensus tag-gaps service + route (#94a)
Cluster C, milestone #94. BulkTagService.tag_gaps(image_ids, threshold) finds tags applied to >= threshold fraction of a visual neighbour set but not all of it (the '7 of 10 share Miku; these 3 don't' signal). Each gap carries the laggard image ids minus any TagSuggestionRejection rows, so apply-to-cluster never re-proposes a tag a neighbour dismissed. 100%-common tags and <2-image sets are excluded. New POST /api/images/cluster/tag-gaps. Tests: consensus found / common excluded / missing ids; rejected laggard excluded from missing; tag dropped when all laggards rejected; <2 images empty; route shape + bad input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
0cd2f391ee |
test(allowlist): unique image paths in coverage tests (CI fix)
The new coverage tests' sequential shas (c{i:063d}) share their first 8 chars,
so deriving the image path from sha[:8] collided on uq_image_record_path. Use
the full sha in the path. Same hardening for test_api_suggestions._img.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
|
||
|
bvandeusen
|
e49cea3eba |
feat(tagging): allowlist tuning dashboard + post-accept toast + merge preview UI (#7c/#7d/#8b)
Cluster B frontend, milestone #99. #7c: AllowlistTable gains Applied + Covers columns and a live 'covers ~N at T' projection as the operator drags a row's threshold (debounced coverage call, then commits the threshold). allowlist store gains coverage(tagId, threshold) and refreshes coverage_count after a save. #7d: suggestions store surfaces a non-blocking toast when accept/alias newly allowlists a tag — '<verb>: <tag> — allowlisted, auto-applying to ~N images' (N is the projection; apply runs async). Falls back to the plain toast when the tag was already allowlisted. #8b: TagsView merge picker now previews the merge via usePreviewCommit before committing — shows images moving / already-on-target / series pages / alias-or- delete / a thumbnail sample, blocks the Merge button on an incompatible kind/fandom. adminStore.mergeTags gains a dryRun option. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
7127714316 |
feat(tags): non-mutating merge preview + admin dry_run (#8a)
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the same counts the apply produces (rule 93 parity) without mutating: images_moving (source links the apply UPDATEs), images_already_on_target (links it drops), source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom compatible flag (surfaced, not raised, so the UI can warn), and up to 6 thumbnails of the moving images. The admin /tags/<dest>/merge route gains a dry_run flag returning the preview JSON. Tests: preview moving-count == apply merged_count (parity), incompatible flagged without raising, self/missing raise, admin dry_run returns preview + no mutation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
e206778a5c |
feat(allowlist): coverage projection + applied-count + post-accept projection (#7a/#7b)
Cluster B, milestone #99. Backend for the allowlist tuning dashboard. #7a: AllowlistService.coverage(tag_id, threshold) counts distinct images with a prediction resolving to the tag (raw_name==tag.name OR (raw_name,category) in the tag's aliases) scoring >= threshold — the gross candidate pool, mirroring tasks.ml._confidence_for_tag resolution. list_all now carries applied_count (grouped image_tag count) + coverage_count (at the row's threshold). New GET /api/tags/<id>/allowlist/coverage?threshold= for the live what-if number. #7b: /suggestions/accept + /alias return {allowlisted, tag_id, tag_name, projected_count} (projection at the tag's threshold) instead of 204, so the UI can show a non-blocking 'auto-applying to ~N images' toast. Apply still runs async via apply_allowlist_tags — projected_count is an estimate. Tests: coverage by threshold (direct + alias-with-category), list applied vs coverage, coverage route (explicit/default/bad threshold), accept/alias payload (newly-allowlisted vs already-on-list). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
73dd301dbb |
feat(gallery): OR/exclude tag filtering — light chips + advanced builder (#6b/c/d)
Cluster A, milestone #97. Completes the frontend of the structured tag filter
(backend landed in
|
||
|
bvandeusen
|
23fab983a0 |
feat(gallery): tag→gallery nav from modal chips (#5) + OR/exclude tag scope (#6a)
Cluster A, milestone #97. #5: clicking an image-modal tag chip's body now closes the modal and opens the gallery filtered for that one tag (fresh filter); ✕/kebab stay as the explicit remove/rename controls. #6a (backend of OR/exclude filtering): gallery_service._apply_scope gains a structured tag model — tag_or_groups (AND-of-OR: one EXISTS(tag_id IN group) per group) + tag_exclude (NOT EXISTS(tag_id IN exclude)) — layered additively on the existing tag_ids AND path so cursors/facets/deep-links are untouched. Threaded through scroll/timeline/jump_cursor/facets/similar + facets common dict; _require_single_filter rejects post_id combined with OR/exclude. API parses tag_or (repeatable → one OR-group each) + tag_not (csv exclude). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
7c94d99b9f |
refactor(settings): canonical usePreviewCommit for maintenance preview→commit tiles (#753)
Frontend pattern-consistency sweep (note #1026, the last DRY-thread item). TagMaintenanceCard (4 flows) + PostMaintenanceCard (2 flows) each hand-rolled the same sync preview→commit state machine: a previewData/previewing/committing triple + onPreview/onCommit that dry-run-previews, then applies and collapses the projection (the apply shares the backend predicate, so afterward it's empty). Extract usePreviewCommit({preview, commit, emptyPreview}) owning that lifecycle. The 6 flows become declarative: supply the two thunks + the collapse shape. The normalize flow (commit dispatches a self-resuming background task, not a sync apply) omits emptyPreview so the projection stays and a truthy result = queued. Composable returns are aliased to the cards' existing local names, so the templates only change where they read the apply result (the success badges). Long-Celery-task cards (GatedPurge/VideoDedup) keep useMaintenanceTask — a different pattern (navigable-away task lifecycle), deliberately not merged. Exhaustiveness: no card hand-rolls the refs anymore; the only dryRun:false callers are these two cards, both via the composable. Added a vitest spec for the primitive (collapse static + fn, dispatch-variant, re-preview clears result). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
26589c3d98 |
refactor(admin-store): _guard + _dryRunPost consolidate the store actions (#753 Finding C)
DRY pass follow-up (note #1026). All 13 admin-store actions repeated the same lastError-capture/rethrow wrapper; the 6 Tier-A maintenance actions additionally repeated the dry_run POST shape. - _guard(fn): one copy of the lastError=null / try / catch(set lastError; rethrow) wrapper, used by all 13 actions. - _dryRunPost(url, {dryRun, ...extra}): the dry_run POST shape on top of _guard, used by the 6 maintenance actions. reconcile maps sourceId -> source_id. Public exports + every action signature unchanged (object-opts for Tier-A, positional for cascade/bulk/tag ops), so no card/view changes. Behavior identical. Added frontend spec (the admin store had none): _dryRunPost endpoint+body+default, sourceId->source_id mapping, and _guard capturing lastError + clearing on success. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6599a07468 |
refactor(admin): consolidate maintenance-trigger 202 responses onto _queued() (#753 Finding B)
DRY pass follow-up (note #1026). Five handlers returned the identical jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result). Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives, trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews. trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it). Added route-level tests for all five consumers (these trigger endpoints had no route coverage before): 202 + task_id via _queued, and the dry_run flag threading through to dedup/purge-gated. Behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6281cb1e66 |
refactor(admin): consolidate Tier-A dry-run/apply handlers onto one helper (#753)
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened process #594). Five Tier-A endpoints repeated the same get_json -> dry_run -> run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn, **kwargs); the five route handlers now delegate. reconcile keeps its source_id validation and passes it through **kwargs. The cleanup_service predicates were already shared between preview and apply (find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no backend-logic change; this is purely the HTTP-handler boilerplate. Consumers (all routed through the helper, verified no copy left behind): prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id), purge_legacy_tags, reset_content_tagging. Added route-level tests for prune-bare (apply) and reconcile (apply + source_id passthrough + invalid-source_id 400) — the two helper consumers that previously had only service-level coverage, so every consumer is exercised at the route. Findings B (queued-response helper) and C (store dry-run POST helper) identified but not applied this pass (operator scoped to A). The card preview->commit state machine is deferred to a frontend pattern-consistency sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
77d02f57ae |
fix(reconcile): preserve from_attachment_id when merging duplicate posts (#73/#87)
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in eff6427; closing it out after today's #87 work, which added a seam it didn't account for. _repoint_post_links drops a loser post's ImageProvenance row on the (image, post) uniqueness collision — and that row may now carry from_attachment_id (which archive the file was extracted from). For the exact gallery-dl->native case this targets, the keeper is the native stub (no archive) and the loser is the gallery-dl row that extracted the member, so a blind delete silently lost the containing-archive linkage. Carry from_attachment_id onto the keeper's surviving row (when NULL) before dropping the collision. The rarer PostAttachment-collision case (both dup posts captured the same archive blob) doesn't arise in the targeted scenario — the archive lives only on the gallery-dl post, so it re-points straight to the keeper and the FK stays valid. Test: collision merge preserves the loser's from_attachment_id on the keeper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
af7f0078bc |
fix(importer): guard _stamp_member_archive against a sidecar-less archive
A filesystem-imported archive with no adjacent sidecar has no Post, so _post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced post.id. _stamp_member_archive already no-ops on a None post_id (no post → no provenance to stamp); pass None instead of crashing. Caught by the existing test_reimport_archive_is_idempotent (no-sidecar zip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5269cd0709 |
feat(provenance): capture which archive an extracted image came from (#87)
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from
|
||
|
bvandeusen
|
068def2f24 |
fix(provenance): scope attachments to originating post + scroll-cap the list
The Attachments section aggregated PostAttachment rows across EVERY post an image was pHash-linked to. When one of those was a 'High Resolution Files' mega-bundle (dozens of unrelated archives), the list ballooned past the viewport and overwhelmed the modal's right rail. - for_image() now scopes attachments to ImageRecord.primary_post_id (the post the file was actually captured from), falling back to all linked posts only when primary_post_id is unset (older rows / filesystem imports). - ProvenancePanel wraps the list in a max-height scroll container with a count in the heading, mirroring the cards' independent-scroll treatment. Note: FC stores archives as opaque blobs and never records which archive an extracted image came from, so attachments can't yet be scoped tighter than the post. Capturing image->archive containment is tracked as separate work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
311fe0ee9c |
feat(settings): tidy Maintenance tab into compact tiles + center the views (pass 2)
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled viewport; expanding a tile to read into it is the only reason to scroll. - Every Maintenance card converted to the collapsible MaintenanceTile (collapsed by default = icon + short title + one-line blurb). Task cards (ML backfill, centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance) sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion thresholds, allowlist, aliases) grouped in one Tagging section as collapsible tiles; Backup is its own collapsible tile. - Three labeled sections mirror the Cleanup tab: Backfills and reprocessing / Tagging / Storage. - Center the whole Settings surface: SettingsView is now a centered, width-capped (1140px) column so the tab strip and every panel sit in a tidy centered measure (was full-width). CleanupView drops its own left-aligned max-width to fill it. All card logic unchanged - only the chrome. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
3b435dc0ba |
feat(settings): tidy Cleanup tab into sectioned compact tiles (pass 1)
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles (2026-06-18: keep both tabs, group inside, compact tiles in a grid). New common/MaintenanceTile.vue: a compact expandable tile (icon + short title + one-line blurb collapsed; click the header to expand the full controls/preview/ result inline; keyboard-accessible button + focus ring; tints the icon, keeps a running task expanded). Cleanup tab (this pass) restructured into 3 sections — Import-filter audits (Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts, Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy, Reset content tagging, Standardize casing) — each a responsive grid of tiles. PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are destructive content cleanup). All card logic unchanged — only the chrome. Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
eff64275fc |
feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
7f6345dccf |
fix(subscribestar): port gallery-dl date extraction (wrapped dates) + parse canary
Image posts wrap the post date in an <a> permalink (<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY the unwrapped case, so every image post got a null published_at and sorted to the top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's _data_from_post method: text up to the first </, then after the last > — handles both. Verified against the live raw feed (all 6 dates now parse). Robust logging (operator request): _parse_posts now logs per-page parse stats (posts / dated / with-body) and a WARNING canary when posts parse but NONE get a date or body while the raw markers are present — i.e. our extraction diverged from the live markup. Makes this failure class diagnosable from the worker log alone, no authed re-fetch needed. Test: a permalink-wrapped date parses to ISO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
479b7b54da |
style(subscribestar): drop quoted forward-ref annotation (UP037)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
976f581aa2 |
feat(subscribestar): port gallery-dl doc + audio attachment extraction
Some SubscribeStar posts deliver content only through document/audio attachments, which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them: - docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the href URL + doc_preview-title + data-upload-id (kind=attachment). - audio: scope uploads-audios..post-edit_form, split on audio_preview-data blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio). The existing downloader handles them unchanged (plain streaming GET; the file validator only inspects image/video extensions via is_validatable, so PDFs/zips/ audio pass straight through, no quarantine). Test covers doc + audio extraction (the cheunart sample has none, so this pins gallery-dl's documented markup shape). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
8771364cee |
fix(subscribestar): port gallery-dl's content + preview-skip extraction faithfully
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body regex either returned empty or over-captured into sibling upload divs and the 'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact _data_from_post rule — content between the post_content-text wrapper and the youtube-uploads div (literal markers), then strip the trix editor's <html><body>…</body></html> document wrapper to its inner. Verified against the live cheunart sample: clean per-post bodies, empty for genuinely text-less posts. Also port gallery-dl's _media_from_post preview guard: skip gallery items whose URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of the Patreon gated-preview bug (#874); this is why a locked post yields no media. Tests: body must not bleed into the pagination counter; trix html-document wrapper stripped; /previews items skipped. Fixture now includes the youtube- uploads close marker present in real markup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
559d29fe1c |
fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate)
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_ subscriber even with valid .adult cookies (confirmed present: _personalization_id + _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's ENTIRE flow — including the base Extractor._init_session I'd not read — the divergence is the HTTP request profile, not the cookies or parser. gallery-dl's default (cookies-only) mode sends, on EVERY request including the first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same- site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling looked unlike a browser → SubscribeStar gated the adult-creator page. Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept- Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the per-request XHR headers (both feed and load-more now use the shared profile). Test updated to assert the gallery-dl-parity profile instead of the old navigation-vs-XHR split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
204d341a99 |
fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift)
The native client split the feed on `<div class="post is-shown`, but `is-shown` is added by SubscribeStar's infinite-scroll JS when a post scrolls into view — present in a browser-SAVED page (what the Step-0 characterization used) but ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError. Align with gallery-dl's proven `_pagination`: split on the generic `<div class="post ` (trailing space rules out the hyphenated post-content/ post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating detection (/verify_subscriber, /age_confirmation_warning => auth, not drift). Regression tests: raw server markup without is-shown now parses; an age-wall redirect raises SubscribeStarAuthError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
9201b7b539 |
diag(subscribestar): name the interstitial in the drift error (title + type)
The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart still drifts — we're being served a full HTML page that isn't the feed. Add _describe_page(): the drift error now reports the page <title> + which known interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha), so the next run names the actual cause instead of "markup changed". Strong suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is UA-locked and our hardcoded UA likely differs from the cookie-capturing browser). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
78a3977f8a |
fix(subscribestar): initial feed GET is a navigation, not XHR (first-run drift)
First live run (cheunart) tripped the drift guard: "no posts and no recognizable feed container". The browser-saved page was normal (6 posts + posts_container-list), so the parser was fine — our live HTTP fetch got a different response. Cause: the client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content- negotiates an XHR full-page request to a non-HTML body → no container → drift. Fix: the session now uses browser-like navigation headers (Accept: html, NO X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on the "load more" endpoint (which is a genuine XHR). Drift message now reports the response length + a JSON hint so a recurrence is self-explaining. Regression test pins the header split (navigation initial GET, XHR load-more). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
d526447496 |
fix(dispatch): resolve native ingester class at call time (test monkeypatch)
The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester", _FakeIngester) no longer affected dispatch → the fake's run() never ran → KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the dict with a _native_ingester_cls() call-time lookup that reads the module globals, so monkeypatching the class names works again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
d8d8ecd78f |
feat(subscribestar): flip dispatch to the native ingester (#893, Step 5)
SubscribeStar now downloads + verifies through the native core ingester instead of gallery-dl — the go-live switch for milestone #71. - download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester / preview_source / verify_source_credential platform-aware. SubscribeStar's campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity. preview now catches the shared NativeIngestError (covers both platforms). - platform_lock: subscribestar serialized (one paced walk at a time). - gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no fallback once native works). - frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the recover/recapture actions show for subscribestar; download_service's cursor/mode/post_first + the preview endpoint already key on uses_native_ingester, so backfill/recovery/recapture/preview light up for free. - tests: download_backends (subscribestar native), platform_lock (serialized), and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials verify, gallery_dl_service skip-value, api_sources arm-no-preflight). post_is_gated stays best-effort (can't cause junk downloads); not gating this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
82551a89d1 |
feat(native-ingest): durable run logging that survives a worker kill (#899 L1/L3, DRY 3/3)
DRY pass commit 3 — the observability half. ingest_core accumulated ALL human-readable progress in log_lines → DownloadResult.stdout, persisted to the DownloadEvent ONLY at phase 3; the real logger was used almost nowhere. So a worker SIGKILL/OOM/hard-time-limit mid-walk left NO trace (the "task died, no trace" mode from the recovery-sweep work). Route run milestones through the container log too, each carrying source_id (L3 context): - run START (platform/mode/source/campaign/resume_cursor) - per-PAGE breadcrumb (posts/downloaded/skipped/errors/quarantined/gated/cursor) at each page boundary — pages are minutes apart on big backfills, so this shows how far a since-died walk got - final SUMMARY (same string as the stdout summary) - operator STOP (L2 — quarantines log.warning'd — already landed in commit 1's base _validate_path; failures log.warning via the base _failure_result; the #862 body canary already log.error's.) log_lines/stdout content is unchanged (summary just captured in a var), so existing assertions hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
ebe6ab9741 |
refactor(native-ingest): shared exception trio + base _failure_result (#899 DRY 2/3)
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).
- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
(status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
("Patreon API" / "SubscribeStar markup"), preserving the existing message
(test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
-unused DownloadResult/ErrorType/*Auth/*Drift imports.
Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
7ac5c7e522 |
refactor(native-ingest): extract native_ingest_common + BaseNativeDownloader (#899 DRY 1/3)
DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was importing patreon privates — wrong owner). New backend/app/services/ native_ingest_common.py is the neutral home for: - make_session (was _load_session ×2), retry_after_seconds + 429 constants, sanitize_segment, basename_from_url, post_dir_name, MediaOutcome / PostRecordOutcome. - BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume) and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it; each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone. - Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only). post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO). Tests repointed to the single source at every consumer (rule 93 / §8b parity): patreon_client/downloader, subscribestar_native. Exception-trio consolidation + base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
817a002c2b |
feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892)
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
f678819093 |
feat(subscribestar): seen/failed ledger models + migration 0054 (#889)
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
ee1b45f8cc |
test(external): fix third fake_fetch stub still requiring timeout=
test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the old `*, timeout` signature; the task now calls fetch_external() without it, so the stub raised TypeError in the integration lane (run 1191). Switch it to **kwargs like the other two. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
4272a19d40 |
fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
25e1e098fb |
fix(activity): record external.* TaskRun.queue as download, not default (#883)
celery_signals._queue_for is a hand-maintained mirror of task_routes that stamps TaskRun.queue in the prerun signal. It was missing the backend.app.tasks.external. prefix, so external fetches recorded queue='default' even though celery routes external.* → download and runs them on the download worker. The dashboard's per-queue filters and the per-queue recovery-sweep threshold therefore missed them — the same 'queue column lies default' gap the 2026-06-02 audit fixed for backup/admin/library_audit. Map external.* → download in _queue_for. Composes with the fetch_external_link task-name sweep override (#883), which wins by precedence regardless of the recorded queue. Pinned test asserts the mirror agrees with the actual route. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
258c77dfcd |
fix(maint): raise recovery-sweep threshold for fetch_external_link (#883)
External file-host fetches run to a 60-min hard limit (time_limit=3600, per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min default. recover_stalled_task_runs was phantom-flagging healthy in-flight fetches as "RecoverySweep: no completion signal received within 5 min" before the task's own timeout/error handling could surface the real error (operator-flagged: target 414 swept at 6.6min). The sweep already has per-queue/per-task overrides for long tasks, but fetch_external_link was never added and its TaskRun records queue='default' (no queue override) despite external.* routing to download. Add a task-name override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it robust regardless of the recorded queue. No new internal timeout needed — the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception already capture the real failure once the sweep stops preempting. Pinned tests: external-fetch override survives a 10-min row / flags a 70-min row on queue='default'; invariant guard asserts override >= hard time_limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
002279e63b |
test(artist-dir): deterministic sha256 in _seed_image (fix flaky uq collision)
test_artist_directory_service._seed_image built sha256 from abs(hash(suffix)) % 10000 — PYTHONHASHSEED-randomized hash() over only 10k buckets, so two suffixes in one test could birthday-collide and violate uq_image_record_sha256. Flaky per process seed: passed on dev (run 1179), failed on main (run 1182) with identical code. Use hashlib.sha256(suffix).hexdigest() for a stable, collision-free digest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
51201b459e |
fix(ml): per-task async engine for recompute_centroid (#881)
recompute_centroid + recompute_centroids were the only tasks still using the process-wide singleton extensions.get_session() under asyncio.run(). The async engine's asyncpg pool is bound to the loop it was created on; each Celery task runs a fresh asyncio.run() loop, so after the first invocation the cached engine handed loop-A connections to loop B and raised "Future attached to a different loop" — every recompute after the first in a worker process failed (~35ms, fails on first DB await). Convert both to the established per-task async_session_factory() pattern (NullPool engine created + disposed inside the task's own loop), matching scan/download/admin tasks. No get_session usages remain in tasks/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
e287802ecb |
fix(maint): resurface dedup/gated-purge results after navigate-away (#877)
Long-running maintenance tasks must survive navigating away or reloading
the page. VideoDedupCard + GatedPurgeCard held the in-flight Celery task id
only in component refs and polled task-result inline, so leaving the page
mid-run lost the id and the result was never shown — even though the task
finished on the worker.
New shared composable useMaintenanceTask: persists {taskId, mode, startedAt}
to localStorage on dispatch, re-attaches on mount, and re-shows the result
when the task finishes (the celery result backend retains the summary well
under result_expires). Stale-guard skips resume past 3h. Both cards refactored
onto it; card-specific computeds + confirm dialog kept.
Also fixed the QueueStatusBar lane: both cards watched queue="maintenance"
but tasks.admin.* routes to maintenance_long, so the bar never reflected
their own task — now queue="maintenance_long".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
540151290b |
feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
A one-shot Maintenance action to remove the blurred locked-preview images the ingester downloaded from tier-gated Patreon posts before #874. current_user_can_view was never persisted, so the cleanup re-walks each enabled Patreon source (read-only) to re-derive which posts are gated now and the blurred filehashes Patreon serves for them, then matches by CONTENT HASH against stored source_filehash. Because the hash is content-addressed, a real file downloaded when access existed has a different hash and can never match — regained-then-lost-access content is provably spared (operator's hard requirement). NULL source_filehash => unverifiable, kept + reported. On apply: delete matched ImageRecords + files (provenance cascades), clear seen/dead-letter ledger rows for those hashes so the real media re-ingests if access returns, and delete gated posts left bare. Shares one match predicate between preview and apply (rule 93). - cleanup_service: collect_gated_previews + purge_gated_previews - tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed) - api.admin: POST /maintenance/purge-gated-previews - GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply) - tests: collect predicate, hash-match delete/spare/unverifiable, ledger clear, bare-post removal, no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |