Cluster C frontend, milestone #94.
#94b Explore walk: new /explore/:imageId route + ExploreView + explore store.
Anchor (reuse /api/gallery/image), neighbour grid (reuse /api/gallery/similar,
24), click a neighbour to re-anchor; in-memory breadcrumb that trims on
backtrack (route is the source of truth). Empty/loading/error + no-embedding
states.
#94c tag-gap closing: components/explore/TagGapPanel — fetches
/api/images/cluster/tag-gaps for the anchor+neighbours, a consensus-threshold
slider (default 60%), per gap shows present/total + the missing thumbnails +
'Apply to N missing' → /api/tags/images/bulk/tags (source manual) → re-fetch.
#94d entry points: 'Explore' button in the modal RelatedStrip; the TopNav entry
comes free from the route's meta.title.
#4a metadata HUD + #4b split Download: new modal ImageMetaBar (always-on, above
ProvenancePanel) shows dimensions/size/type and a split Download button
(default Download, chevron → Copy link via utils/clipboard — no clipboard-image,
rule 95).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster C, milestone #94. BulkTagService.tag_gaps(image_ids, threshold) finds
tags applied to >= threshold fraction of a visual neighbour set but not all of
it (the '7 of 10 share Miku; these 3 don't' signal). Each gap carries the
laggard image ids minus any TagSuggestionRejection rows, so apply-to-cluster
never re-proposes a tag a neighbour dismissed. 100%-common tags and <2-image
sets are excluded. New POST /api/images/cluster/tag-gaps.
Tests: consensus found / common excluded / missing ids; rejected laggard
excluded from missing; tag dropped when all laggards rejected; <2 images empty;
route shape + bad input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
The new coverage tests' sequential shas (c{i:063d}) share their first 8 chars,
so deriving the image path from sha[:8] collided on uq_image_record_path. Use
the full sha in the path. Same hardening for test_api_suggestions._img.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B frontend, milestone #99.
#7c: AllowlistTable gains Applied + Covers columns and a live 'covers ~N at T'
projection as the operator drags a row's threshold (debounced coverage call,
then commits the threshold). allowlist store gains coverage(tagId, threshold)
and refreshes coverage_count after a save.
#7d: suggestions store surfaces a non-blocking toast when accept/alias newly
allowlists a tag — '<verb>: <tag> — allowlisted, auto-applying to ~N images'
(N is the projection; apply runs async). Falls back to the plain toast when
the tag was already allowlisted.
#8b: TagsView merge picker now previews the merge via usePreviewCommit before
committing — shows images moving / already-on-target / series pages / alias-or-
delete / a thumbnail sample, blocks the Merge button on an incompatible
kind/fandom. adminStore.mergeTags gains a dryRun option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the
same counts the apply produces (rule 93 parity) without mutating: images_moving
(source links the apply UPDATEs), images_already_on_target (links it drops),
source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom
compatible flag (surfaced, not raised, so the UI can warn), and up to 6
thumbnails of the moving images. The admin /tags/<dest>/merge route gains a
dry_run flag returning the preview JSON.
Tests: preview moving-count == apply merged_count (parity), incompatible flagged
without raising, self/missing raise, admin dry_run returns preview + no mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. Backend for the allowlist tuning dashboard.
#7a: AllowlistService.coverage(tag_id, threshold) counts distinct images with
a prediction resolving to the tag (raw_name==tag.name OR (raw_name,category) in
the tag's aliases) scoring >= threshold — the gross candidate pool, mirroring
tasks.ml._confidence_for_tag resolution. list_all now carries applied_count
(grouped image_tag count) + coverage_count (at the row's threshold). New
GET /api/tags/<id>/allowlist/coverage?threshold= for the live what-if number.
#7b: /suggestions/accept + /alias return {allowlisted, tag_id, tag_name,
projected_count} (projection at the tag's threshold) instead of 204, so the UI
can show a non-blocking 'auto-applying to ~N images' toast. Apply still runs
async via apply_allowlist_tags — projected_count is an estimate.
Tests: coverage by threshold (direct + alias-with-category), list applied vs
coverage, coverage route (explicit/default/bad threshold), accept/alias payload
(newly-allowlisted vs already-on-list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster A, milestone #97. Completes the frontend of the structured tag filter
(backend landed in 23fab98).
#6b store: gallery.filter gains tag_or (OR-groups) + tag_exclude; one model,
serialized via tag_or (repeated key) + tag_not across activeFilterParam/
filterToQuery/applyFilterFromQuery/cloneFilter/loadSimilar; _resolveLabels
resolves names for every referenced id. useApi now appends array param values
as a repeated key (tag_or=a&tag_or=b).
#6c light editor (GalleryFilterBar): autocomplete pick → include chip; click a
chip body to flip include↔exclude (exclude = red minus); ✕ removes. An
"N OR-groups" chip + an Advanced button open the builder.
#6d advanced editor (TagQueryBuilder.vue + common/TagPicker.vue): AND-of-OR
group builder + NOT list. Unifies includes+OR-groups into one groups view,
splits back to tag_ids/tag_or on Apply so the URL stays compact. Writes the
same model the light chips edit.
Store serialization round-trip tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster A, milestone #97. #5: clicking an image-modal tag chip's body now
closes the modal and opens the gallery filtered for that one tag (fresh
filter); ✕/kebab stay as the explicit remove/rename controls.
#6a (backend of OR/exclude filtering): gallery_service._apply_scope gains a
structured tag model — tag_or_groups (AND-of-OR: one EXISTS(tag_id IN group)
per group) + tag_exclude (NOT EXISTS(tag_id IN exclude)) — layered additively
on the existing tag_ids AND path so cursors/facets/deep-links are untouched.
Threaded through scroll/timeline/jump_cursor/facets/similar + facets common
dict; _require_single_filter rejects post_id combined with OR/exclude. API
parses tag_or (repeatable → one OR-group each) + tag_not (csv exclude).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Frontend pattern-consistency sweep (note #1026, the last DRY-thread item).
TagMaintenanceCard (4 flows) + PostMaintenanceCard (2 flows) each hand-rolled the
same sync preview→commit state machine: a previewData/previewing/committing
triple + onPreview/onCommit that dry-run-previews, then applies and collapses the
projection (the apply shares the backend predicate, so afterward it's empty).
Extract usePreviewCommit({preview, commit, emptyPreview}) owning that lifecycle.
The 6 flows become declarative: supply the two thunks + the collapse shape. The
normalize flow (commit dispatches a self-resuming background task, not a sync
apply) omits emptyPreview so the projection stays and a truthy result = queued.
Composable returns are aliased to the cards' existing local names, so the
templates only change where they read the apply result (the success badges).
Long-Celery-task cards (GatedPurge/VideoDedup) keep useMaintenanceTask — a
different pattern (navigable-away task lifecycle), deliberately not merged.
Exhaustiveness: no card hand-rolls the refs anymore; the only dryRun:false
callers are these two cards, both via the composable. Added a vitest spec for
the primitive (collapse static + fn, dispatch-variant, re-preview clears result).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass follow-up (note #1026). All 13 admin-store actions repeated the same
lastError-capture/rethrow wrapper; the 6 Tier-A maintenance actions additionally
repeated the dry_run POST shape.
- _guard(fn): one copy of the lastError=null / try / catch(set lastError; rethrow)
wrapper, used by all 13 actions.
- _dryRunPost(url, {dryRun, ...extra}): the dry_run POST shape on top of _guard,
used by the 6 maintenance actions. reconcile maps sourceId -> source_id.
Public exports + every action signature unchanged (object-opts for Tier-A,
positional for cascade/bulk/tag ops), so no card/view changes. Behavior identical.
Added frontend spec (the admin store had none): _dryRunPost endpoint+body+default,
sourceId->source_id mapping, and _guard capturing lastError + clearing on success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass follow-up (note #1026). Five handlers returned the identical
jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result).
Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives,
trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews.
trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it).
Added route-level tests for all five consumers (these trigger endpoints had no
route coverage before): 202 + task_id via _queued, and the dry_run flag threading
through to dedup/purge-gated. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened
process #594). Five Tier-A endpoints repeated the same get_json -> dry_run ->
run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn,
**kwargs); the five route handlers now delegate. reconcile keeps its source_id
validation and passes it through **kwargs.
The cleanup_service predicates were already shared between preview and apply
(find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no
backend-logic change; this is purely the HTTP-handler boilerplate.
Consumers (all routed through the helper, verified no copy left behind):
prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id),
purge_legacy_tags, reset_content_tagging.
Added route-level tests for prune-bare (apply) and reconcile (apply + source_id
passthrough + invalid-source_id 400) — the two helper consumers that previously
had only service-level coverage, so every consumer is exercised at the route.
Findings B (queued-response helper) and C (store dry-run POST helper) identified
but not applied this pass (operator scoped to A). The card preview->commit state
machine is deferred to a frontend pattern-consistency sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in
eff6427; closing it out after today's #87 work, which added a seam it didn't
account for. _repoint_post_links drops a loser post's ImageProvenance row on the
(image, post) uniqueness collision — and that row may now carry from_attachment_id
(which archive the file was extracted from). For the exact gallery-dl->native
case this targets, the keeper is the native stub (no archive) and the loser is the
gallery-dl row that extracted the member, so a blind delete silently lost the
containing-archive linkage. Carry from_attachment_id onto the keeper's surviving
row (when NULL) before dropping the collision.
The rarer PostAttachment-collision case (both dup posts captured the same archive
blob) doesn't arise in the targeted scenario — the archive lives only on the
gallery-dl post, so it re-points straight to the keeper and the FK stays valid.
Test: collision merge preserves the loser's from_attachment_id on the keeper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A filesystem-imported archive with no adjacent sidecar has no Post, so
_post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced
post.id. _stamp_member_archive already no-ops on a None post_id (no post → no
provenance to stamp); pass None instead of crashing. Caught by the existing
test_reimport_archive_is_idempotent (no-sidecar zip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from 068def2.
- ProvenancePanel: heading pluralizes ('Attachment' for a single file).
- Backfill: re-running reextract_archive_attachments (ArchiveReextractCard)
routes through _import_archive and stamps existing rows — no new code.
Tests: capture stamps on fresh import, nested-archive attribution, per-post
archive on dedup; for_image filters to the containing archive; reextract
backfill stamps the link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Attachments section aggregated PostAttachment rows across EVERY post an
image was pHash-linked to. When one of those was a 'High Resolution Files'
mega-bundle (dozens of unrelated archives), the list ballooned past the
viewport and overwhelmed the modal's right rail.
- for_image() now scopes attachments to ImageRecord.primary_post_id (the post
the file was actually captured from), falling back to all linked posts only
when primary_post_id is unset (older rows / filesystem imports).
- ProvenancePanel wraps the list in a max-height scroll container with a count
in the heading, mirroring the cards' independent-scroll treatment.
Note: FC stores archives as opaque blobs and never records which archive an
extracted image came from, so attachments can't yet be scoped tighter than the
post. Capturing image->archive containment is tracked as separate work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled
viewport; expanding a tile to read into it is the only reason to scroll.
- Every Maintenance card converted to the collapsible MaintenanceTile (collapsed
by default = icon + short title + one-line blurb). Task cards (ML backfill,
centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance)
sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion
thresholds, allowlist, aliases) grouped in one Tagging section as collapsible
tiles; Backup is its own collapsible tile.
- Three labeled sections mirror the Cleanup tab: Backfills and reprocessing /
Tagging / Storage.
- Center the whole Settings surface: SettingsView is now a centered, width-capped
(1140px) column so the tab strip and every panel sit in a tidy centered measure
(was full-width). CleanupView drops its own left-aligned max-width to fill it.
All card logic unchanged - only the chrome.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long
descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles
(2026-06-18: keep both tabs, group inside, compact tiles in a grid).
New common/MaintenanceTile.vue: a compact expandable tile (icon + short title +
one-line blurb collapsed; click the header to expand the full controls/preview/
result inline; keyboard-accessible button + focus ring; tints the
icon, keeps a running task expanded).
Cleanup tab (this pass) restructured into 3 sections — Import-filter audits
(Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts,
Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy,
Reset content tagging, Standardize casing) — each a responsive grid of tiles.
PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved
VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are
destructive content cleanup). All card logic unchanged — only the chrome.
Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Image posts wrap the post date in an <a> permalink
(<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts
don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY
the unwrapped case, so every image post got a null published_at and sorted to the
top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's
_data_from_post method: text up to the first </, then after the last > — handles
both. Verified against the live raw feed (all 6 dates now parse).
Robust logging (operator request): _parse_posts now logs per-page parse stats
(posts / dated / with-body) and a WARNING canary when posts parse but NONE get a
date or body while the raw markers are present — i.e. our extraction diverged
from the live markup. Makes this failure class diagnosable from the worker log
alone, no authed re-fetch needed.
Test: a permalink-wrapped date parses to ISO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SubscribeStar posts deliver content only through document/audio attachments,
which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them:
- docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the
href URL + doc_preview-title + data-upload-id (kind=attachment).
- audio: scope uploads-audios..post-edit_form, split on audio_preview-data
blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio).
The existing downloader handles them unchanged (plain streaming GET; the file
validator only inspects image/video extensions via is_validatable, so PDFs/zips/
audio pass straight through, no quarantine).
Test covers doc + audio extraction (the cheunart sample has none, so this pins
gallery-dl's documented markup shape).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body
regex either returned empty or over-captured into sibling upload divs and the
'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact
_data_from_post rule — content between the post_content-text wrapper and the
youtube-uploads div (literal markers), then strip the trix editor's
<html><body>…</body></html> document wrapper to its inner. Verified against the
live cheunart sample: clean per-post bodies, empty for genuinely text-less posts.
Also port gallery-dl's _media_from_post preview guard: skip gallery items whose
URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of
the Patreon gated-preview bug (#874); this is why a locked post yields no media.
Tests: body must not bleed into the pagination counter; trix html-document
wrapper stripped; /previews items skipped. Fixture now includes the youtube-
uploads close marker present in real markup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_
subscriber even with valid .adult cookies (confirmed present: _personalization_id
+ _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's
ENTIRE flow — including the base Extractor._init_session I'd not read — the
divergence is the HTTP request profile, not the cookies or parser.
gallery-dl's default (cookies-only) mode sends, on EVERY request including the
first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same-
site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint
is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling
looked unlike a browser → SubscribeStar gated the adult-creator page.
Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept-
Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the
per-request XHR headers (both feed and load-more now use the shared profile).
Test updated to assert the gallery-dl-parity profile instead of the old
navigation-vs-XHR split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native client split the feed on `<div class="post is-shown`, but `is-shown`
is added by SubscribeStar's infinite-scroll JS when a post scrolls into view —
present in a browser-SAVED page (what the Step-0 characterization used) but
ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live
feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError.
Align with gallery-dl's proven `_pagination`: split on the generic
`<div class="post ` (trailing space rules out the hyphenated post-content/
post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating
detection (/verify_subscriber, /age_confirmation_warning => auth, not drift).
Regression tests: raw server markup without is-shown now parses; an age-wall
redirect raises SubscribeStarAuthError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart
still drifts — we're being served a full HTML page that isn't the feed. Add
_describe_page(): the drift error now reports the page <title> + which known
interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha),
so the next run names the actual cause instead of "markup changed". Strong
suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is
UA-locked and our hardcoded UA likely differs from the cookie-capturing browser).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First live run (cheunart) tripped the drift guard: "no posts and no recognizable
feed container". The browser-saved page was normal (6 posts + posts_container-list),
so the parser was fine — our live HTTP fetch got a different response. Cause: the
client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the
initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content-
negotiates an XHR full-page request to a non-HTML body → no container → drift.
Fix: the session now uses browser-like navigation headers (Accept: html, NO
X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on
the "load more" endpoint (which is a genuine XHR). Drift message now reports the
response length + a JSON hint so a recurrence is self-explaining. Regression test
pins the header split (navigation initial GET, XHR load-more).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at
import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester",
_FakeIngester) no longer affected dispatch → the fake's run() never ran →
KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the
dict with a _native_ingester_cls() call-time lookup that reads the module globals,
so monkeypatching the class names works again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.
- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
_NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
/ preview_source / verify_source_credential platform-aware. SubscribeStar's
campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
recover/recapture actions show for subscribestar; download_service's
cursor/mode/post_first + the preview endpoint already key on
uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
verify, gallery_dl_service skip-value, api_sources arm-no-preflight).
post_is_gated stays best-effort (can't cause junk downloads); not gating this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 3 — the observability half. ingest_core accumulated ALL
human-readable progress in log_lines → DownloadResult.stdout, persisted to the
DownloadEvent ONLY at phase 3; the real logger was used almost nowhere. So a
worker SIGKILL/OOM/hard-time-limit mid-walk left NO trace (the "task died,
no trace" mode from the recovery-sweep work).
Route run milestones through the container log too, each carrying source_id (L3
context):
- run START (platform/mode/source/campaign/resume_cursor)
- per-PAGE breadcrumb (posts/downloaded/skipped/errors/quarantined/gated/cursor)
at each page boundary — pages are minutes apart on big backfills, so this shows
how far a since-died walk got
- final SUMMARY (same string as the stdout summary)
- operator STOP
(L2 — quarantines log.warning'd — already landed in commit 1's base
_validate_path; failures log.warning via the base _failure_result; the #862 body
canary already log.error's.) log_lines/stdout content is unchanged (summary just
captured in a var), so existing assertions hold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).
- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
(status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
("Patreon API" / "SubscribeStar markup"), preserving the existing message
(test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
-unused DownloadResult/ErrorType/*Auth/*Drift imports.
Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing
the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was
importing patreon privates — wrong owner). New backend/app/services/
native_ingest_common.py is the neutral home for:
- make_session (was _load_session ×2), retry_after_seconds + 429 constants,
sanitize_segment, basename_from_url, post_dir_name, MediaOutcome /
PostRecordOutcome.
- BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume)
and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it;
each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch
enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the
divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone.
- Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only).
post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO).
Tests repointed to the single source at every consumer (rule 93 / §8b parity):
patreon_client/downloader, subscribestar_native. Exception-trio consolidation +
base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core
ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger:
SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery
bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing
media stops re-burning backfill chunks). Per operator decision, dedicated
per-platform tables (not a generalized shared ledger).
filehash is String(128): a CDN content hash when the URL carries one, else a
synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key.
Registered in models/__init__; migration 0054 creates both tables (down 0053).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the
old `*, timeout` signature; the task now calls fetch_external() without it, so
the stub raised TypeError in the integration lane (run 1191). Switch it to
**kwargs like the other two.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).
Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
(connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
across chunks in _stream_to_file (HTTP has no total-download timeout), and
passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.
Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.
Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.
Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
celery_signals._queue_for is a hand-maintained mirror of task_routes that
stamps TaskRun.queue in the prerun signal. It was missing the
backend.app.tasks.external. prefix, so external fetches recorded
queue='default' even though celery routes external.* → download and runs
them on the download worker. The dashboard's per-queue filters and the
per-queue recovery-sweep threshold therefore missed them — the same
'queue column lies default' gap the 2026-06-02 audit fixed for
backup/admin/library_audit.
Map external.* → download in _queue_for. Composes with the fetch_external_link
task-name sweep override (#883), which wins by precedence regardless of the
recorded queue. Pinned test asserts the mirror agrees with the actual route.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bvandeusen