dev
503 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
10434509d3 |
fix(tags): fandom views aggregate images via their characters
A fandom owns characters via Tag.fandom_id, but every image<->tag query went purely through direct image_tag rows, so a fandom only surfaced images literally tagged with it — images carrying one of its characters were invisible to its browse count, previews, and gallery filter. Derive membership at query time instead of materializing fandom rows (which would drift on every reassign/merge/remove). Add one shared predicate in tag_query.py — image_in_tag_scope / image_in_any_tag_scope: an image belongs to a tag if tagged with it directly OR (when the tag is a fandom) carrying a character whose fandom_id is that tag. The character leg is empty for non-fandom tags, so it applies uniformly with no kind branching. Route all read sites through it: - gallery _apply_scope: include, OR-groups, and symmetric exclude - directory image_count: correlated COUNT(DISTINCT) scalar subquery - directory previews: UNION direct + via-character, then ROW_NUMBER<=3 - cleanup count_tag_associations: Tier-B delete prompt now reports a fandom's true blast radius (was 0 for fandoms with no direct rows) find_unused_tags already protected fandoms via used_via_fandom; left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b85327a79d |
fix(celery): harden broker connection so workers ride out a Redis blip
A swarm overlay-network blip after the :latest redeploy left Redis healthy but transiently unreachable; a worker starting in that window crash-looped on the initial broker connect (kombu OperationalError) and needed a manual Redis reset to recover. Retry the broker forever on startup + at runtime (broker_connection_max_retries =None), add redis-transport socket options to the broker (short connect timeout, TCP keepalive, retry_on_timeout, periodic health check), and mirror the same on the Redis result backend. Now a transient outage self-heals when overlay routing returns instead of the worker exiting. Test pins the key resilience settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
0ecd1ce4f1 |
feat(explore): cluster-consensus tag-gaps service + route (#94a)
Cluster C, milestone #94. BulkTagService.tag_gaps(image_ids, threshold) finds tags applied to >= threshold fraction of a visual neighbour set but not all of it (the '7 of 10 share Miku; these 3 don't' signal). Each gap carries the laggard image ids minus any TagSuggestionRejection rows, so apply-to-cluster never re-proposes a tag a neighbour dismissed. 100%-common tags and <2-image sets are excluded. New POST /api/images/cluster/tag-gaps. Tests: consensus found / common excluded / missing ids; rejected laggard excluded from missing; tag dropped when all laggards rejected; <2 images empty; route shape + bad input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
0cd2f391ee |
test(allowlist): unique image paths in coverage tests (CI fix)
The new coverage tests' sequential shas (c{i:063d}) share their first 8 chars,
so deriving the image path from sha[:8] collided on uq_image_record_path. Use
the full sha in the path. Same hardening for test_api_suggestions._img.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
|
||
|
bvandeusen
|
7127714316 |
feat(tags): non-mutating merge preview + admin dry_run (#8a)
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the same counts the apply produces (rule 93 parity) without mutating: images_moving (source links the apply UPDATEs), images_already_on_target (links it drops), source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom compatible flag (surfaced, not raised, so the UI can warn), and up to 6 thumbnails of the moving images. The admin /tags/<dest>/merge route gains a dry_run flag returning the preview JSON. Tests: preview moving-count == apply merged_count (parity), incompatible flagged without raising, self/missing raise, admin dry_run returns preview + no mutation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
e206778a5c |
feat(allowlist): coverage projection + applied-count + post-accept projection (#7a/#7b)
Cluster B, milestone #99. Backend for the allowlist tuning dashboard. #7a: AllowlistService.coverage(tag_id, threshold) counts distinct images with a prediction resolving to the tag (raw_name==tag.name OR (raw_name,category) in the tag's aliases) scoring >= threshold — the gross candidate pool, mirroring tasks.ml._confidence_for_tag resolution. list_all now carries applied_count (grouped image_tag count) + coverage_count (at the row's threshold). New GET /api/tags/<id>/allowlist/coverage?threshold= for the live what-if number. #7b: /suggestions/accept + /alias return {allowlisted, tag_id, tag_name, projected_count} (projection at the tag's threshold) instead of 204, so the UI can show a non-blocking 'auto-applying to ~N images' toast. Apply still runs async via apply_allowlist_tags — projected_count is an estimate. Tests: coverage by threshold (direct + alias-with-category), list applied vs coverage, coverage route (explicit/default/bad threshold), accept/alias payload (newly-allowlisted vs already-on-list). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
23fab983a0 |
feat(gallery): tag→gallery nav from modal chips (#5) + OR/exclude tag scope (#6a)
Cluster A, milestone #97. #5: clicking an image-modal tag chip's body now closes the modal and opens the gallery filtered for that one tag (fresh filter); ✕/kebab stay as the explicit remove/rename controls. #6a (backend of OR/exclude filtering): gallery_service._apply_scope gains a structured tag model — tag_or_groups (AND-of-OR: one EXISTS(tag_id IN group) per group) + tag_exclude (NOT EXISTS(tag_id IN exclude)) — layered additively on the existing tag_ids AND path so cursors/facets/deep-links are untouched. Threaded through scroll/timeline/jump_cursor/facets/similar + facets common dict; _require_single_filter rejects post_id combined with OR/exclude. API parses tag_or (repeatable → one OR-group each) + tag_not (csv exclude). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
6599a07468 |
refactor(admin): consolidate maintenance-trigger 202 responses onto _queued() (#753 Finding B)
DRY pass follow-up (note #1026). Five handlers returned the identical jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result). Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives, trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews. trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it). Added route-level tests for all five consumers (these trigger endpoints had no route coverage before): 202 + task_id via _queued, and the dry_run flag threading through to dedup/purge-gated. Behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6281cb1e66 |
refactor(admin): consolidate Tier-A dry-run/apply handlers onto one helper (#753)
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened process #594). Five Tier-A endpoints repeated the same get_json -> dry_run -> run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn, **kwargs); the five route handlers now delegate. reconcile keeps its source_id validation and passes it through **kwargs. The cleanup_service predicates were already shared between preview and apply (find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no backend-logic change; this is purely the HTTP-handler boilerplate. Consumers (all routed through the helper, verified no copy left behind): prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id), purge_legacy_tags, reset_content_tagging. Added route-level tests for prune-bare (apply) and reconcile (apply + source_id passthrough + invalid-source_id 400) — the two helper consumers that previously had only service-level coverage, so every consumer is exercised at the route. Findings B (queued-response helper) and C (store dry-run POST helper) identified but not applied this pass (operator scoped to A). The card preview->commit state machine is deferred to a frontend pattern-consistency sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
77d02f57ae |
fix(reconcile): preserve from_attachment_id when merging duplicate posts (#73/#87)
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in eff6427; closing it out after today's #87 work, which added a seam it didn't account for. _repoint_post_links drops a loser post's ImageProvenance row on the (image, post) uniqueness collision — and that row may now carry from_attachment_id (which archive the file was extracted from). For the exact gallery-dl->native case this targets, the keeper is the native stub (no archive) and the loser is the gallery-dl row that extracted the member, so a blind delete silently lost the containing-archive linkage. Carry from_attachment_id onto the keeper's surviving row (when NULL) before dropping the collision. The rarer PostAttachment-collision case (both dup posts captured the same archive blob) doesn't arise in the targeted scenario — the archive lives only on the gallery-dl post, so it re-points straight to the keeper and the FK stays valid. Test: collision merge preserves the loser's from_attachment_id on the keeper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5269cd0709 |
feat(provenance): capture which archive an extracted image came from (#87)
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from
|
||
|
bvandeusen
|
068def2f24 |
fix(provenance): scope attachments to originating post + scroll-cap the list
The Attachments section aggregated PostAttachment rows across EVERY post an image was pHash-linked to. When one of those was a 'High Resolution Files' mega-bundle (dozens of unrelated archives), the list ballooned past the viewport and overwhelmed the modal's right rail. - for_image() now scopes attachments to ImageRecord.primary_post_id (the post the file was actually captured from), falling back to all linked posts only when primary_post_id is unset (older rows / filesystem imports). - ProvenancePanel wraps the list in a max-height scroll container with a count in the heading, mirroring the cards' independent-scroll treatment. Note: FC stores archives as opaque blobs and never records which archive an extracted image came from, so attachments can't yet be scoped tighter than the post. Capturing image->archive containment is tracked as separate work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
eff64275fc |
feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
7f6345dccf |
fix(subscribestar): port gallery-dl date extraction (wrapped dates) + parse canary
Image posts wrap the post date in an <a> permalink (<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY the unwrapped case, so every image post got a null published_at and sorted to the top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's _data_from_post method: text up to the first </, then after the last > — handles both. Verified against the live raw feed (all 6 dates now parse). Robust logging (operator request): _parse_posts now logs per-page parse stats (posts / dated / with-body) and a WARNING canary when posts parse but NONE get a date or body while the raw markers are present — i.e. our extraction diverged from the live markup. Makes this failure class diagnosable from the worker log alone, no authed re-fetch needed. Test: a permalink-wrapped date parses to ISO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
976f581aa2 |
feat(subscribestar): port gallery-dl doc + audio attachment extraction
Some SubscribeStar posts deliver content only through document/audio attachments, which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them: - docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the href URL + doc_preview-title + data-upload-id (kind=attachment). - audio: scope uploads-audios..post-edit_form, split on audio_preview-data blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio). The existing downloader handles them unchanged (plain streaming GET; the file validator only inspects image/video extensions via is_validatable, so PDFs/zips/ audio pass straight through, no quarantine). Test covers doc + audio extraction (the cheunart sample has none, so this pins gallery-dl's documented markup shape). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
8771364cee |
fix(subscribestar): port gallery-dl's content + preview-skip extraction faithfully
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body regex either returned empty or over-captured into sibling upload divs and the 'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact _data_from_post rule — content between the post_content-text wrapper and the youtube-uploads div (literal markers), then strip the trix editor's <html><body>…</body></html> document wrapper to its inner. Verified against the live cheunart sample: clean per-post bodies, empty for genuinely text-less posts. Also port gallery-dl's _media_from_post preview guard: skip gallery items whose URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of the Patreon gated-preview bug (#874); this is why a locked post yields no media. Tests: body must not bleed into the pagination counter; trix html-document wrapper stripped; /previews items skipped. Fixture now includes the youtube- uploads close marker present in real markup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
559d29fe1c |
fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate)
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_ subscriber even with valid .adult cookies (confirmed present: _personalization_id + _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's ENTIRE flow — including the base Extractor._init_session I'd not read — the divergence is the HTTP request profile, not the cookies or parser. gallery-dl's default (cookies-only) mode sends, on EVERY request including the first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same- site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling looked unlike a browser → SubscribeStar gated the adult-creator page. Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept- Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the per-request XHR headers (both feed and load-more now use the shared profile). Test updated to assert the gallery-dl-parity profile instead of the old navigation-vs-XHR split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
204d341a99 |
fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift)
The native client split the feed on `<div class="post is-shown`, but `is-shown` is added by SubscribeStar's infinite-scroll JS when a post scrolls into view — present in a browser-SAVED page (what the Step-0 characterization used) but ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError. Align with gallery-dl's proven `_pagination`: split on the generic `<div class="post ` (trailing space rules out the hyphenated post-content/ post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating detection (/verify_subscriber, /age_confirmation_warning => auth, not drift). Regression tests: raw server markup without is-shown now parses; an age-wall redirect raises SubscribeStarAuthError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
78a3977f8a |
fix(subscribestar): initial feed GET is a navigation, not XHR (first-run drift)
First live run (cheunart) tripped the drift guard: "no posts and no recognizable feed container". The browser-saved page was normal (6 posts + posts_container-list), so the parser was fine — our live HTTP fetch got a different response. Cause: the client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content- negotiates an XHR full-page request to a non-HTML body → no container → drift. Fix: the session now uses browser-like navigation headers (Accept: html, NO X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on the "load more" endpoint (which is a genuine XHR). Drift message now reports the response length + a JSON hint so a recurrence is self-explaining. Regression test pins the header split (navigation initial GET, XHR load-more). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
d8d8ecd78f |
feat(subscribestar): flip dispatch to the native ingester (#893, Step 5)
SubscribeStar now downloads + verifies through the native core ingester instead of gallery-dl — the go-live switch for milestone #71. - download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester / preview_source / verify_source_credential platform-aware. SubscribeStar's campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity. preview now catches the shared NativeIngestError (covers both platforms). - platform_lock: subscribestar serialized (one paced walk at a time). - gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no fallback once native works). - frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the recover/recapture actions show for subscribestar; download_service's cursor/mode/post_first + the preview endpoint already key on uses_native_ingester, so backfill/recovery/recapture/preview light up for free. - tests: download_backends (subscribestar native), platform_lock (serialized), and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials verify, gallery_dl_service skip-value, api_sources arm-no-preflight). post_is_gated stays best-effort (can't cause junk downloads); not gating this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
7ac5c7e522 |
refactor(native-ingest): extract native_ingest_common + BaseNativeDownloader (#899 DRY 1/3)
DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was importing patreon privates — wrong owner). New backend/app/services/ native_ingest_common.py is the neutral home for: - make_session (was _load_session ×2), retry_after_seconds + 429 constants, sanitize_segment, basename_from_url, post_dir_name, MediaOutcome / PostRecordOutcome. - BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume) and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it; each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone. - Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only). post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO). Tests repointed to the single source at every consumer (rule 93 / §8b parity): patreon_client/downloader, subscribestar_native. Exception-trio consolidation + base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
817a002c2b |
feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892)
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
ee1b45f8cc |
test(external): fix third fake_fetch stub still requiring timeout=
test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the old `*, timeout` signature; the task now calls fetch_external() without it, so the stub raised TypeError in the integration lane (run 1191). Switch it to **kwargs like the other two. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
4272a19d40 |
fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
25e1e098fb |
fix(activity): record external.* TaskRun.queue as download, not default (#883)
celery_signals._queue_for is a hand-maintained mirror of task_routes that stamps TaskRun.queue in the prerun signal. It was missing the backend.app.tasks.external. prefix, so external fetches recorded queue='default' even though celery routes external.* → download and runs them on the download worker. The dashboard's per-queue filters and the per-queue recovery-sweep threshold therefore missed them — the same 'queue column lies default' gap the 2026-06-02 audit fixed for backup/admin/library_audit. Map external.* → download in _queue_for. Composes with the fetch_external_link task-name sweep override (#883), which wins by precedence regardless of the recorded queue. Pinned test asserts the mirror agrees with the actual route. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
258c77dfcd |
fix(maint): raise recovery-sweep threshold for fetch_external_link (#883)
External file-host fetches run to a 60-min hard limit (time_limit=3600, per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min default. recover_stalled_task_runs was phantom-flagging healthy in-flight fetches as "RecoverySweep: no completion signal received within 5 min" before the task's own timeout/error handling could surface the real error (operator-flagged: target 414 swept at 6.6min). The sweep already has per-queue/per-task overrides for long tasks, but fetch_external_link was never added and its TaskRun records queue='default' (no queue override) despite external.* routing to download. Add a task-name override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it robust regardless of the recorded queue. No new internal timeout needed — the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception already capture the real failure once the sweep stops preempting. Pinned tests: external-fetch override survives a 10-min row / flags a 70-min row on queue='default'; invariant guard asserts override >= hard time_limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
002279e63b |
test(artist-dir): deterministic sha256 in _seed_image (fix flaky uq collision)
test_artist_directory_service._seed_image built sha256 from abs(hash(suffix)) % 10000 — PYTHONHASHSEED-randomized hash() over only 10k buckets, so two suffixes in one test could birthday-collide and violate uq_image_record_sha256. Flaky per process seed: passed on dev (run 1179), failed on main (run 1182) with identical code. Use hashlib.sha256(suffix).hexdigest() for a stable, collision-free digest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
540151290b |
feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
A one-shot Maintenance action to remove the blurred locked-preview images the ingester downloaded from tier-gated Patreon posts before #874. current_user_can_view was never persisted, so the cleanup re-walks each enabled Patreon source (read-only) to re-derive which posts are gated now and the blurred filehashes Patreon serves for them, then matches by CONTENT HASH against stored source_filehash. Because the hash is content-addressed, a real file downloaded when access existed has a different hash and can never match — regained-then-lost-access content is provably spared (operator's hard requirement). NULL source_filehash => unverifiable, kept + reported. On apply: delete matched ImageRecords + files (provenance cascades), clear seen/dead-letter ledger rows for those hashes so the real media re-ingests if access returns, and delete gated posts left bare. Shares one match predicate between preview and apply (rule 93). - cleanup_service: collect_gated_previews + purge_gated_previews - tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed) - api.admin: POST /maintenance/purge-gated-previews - GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply) - tests: collect predicate, hash-match delete/spare/unverifiable, ledger clear, bare-post removal, no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
9422eadabe |
test(ingest): repair gated-post tests (#874 CI bounce)
The prior commit's Edit orphaned the recapture test's relink/stdout assertions into the new preview test (F821 res_recap/downloader2/m1) and the gated-skip test's written_paths check matched 'gated' in the tmp dir name. Restore the recapture assertions to their function and assert on the media basename instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
b3afc2437e |
fix(ingest): skip tier-gated Patreon posts entirely (#874)
Patreon serves only blurred locked-preview thumbnails for posts the authenticated account can't fully view; the native ingester was downloading those as real media. current_user_can_view was already in _FIELDS_POST but never read. Add PatreonClient.post_is_gated (gate ONLY on explicit current_user_can_view=False; missing/None → viewable, never over-filter) and skip gated posts at the top of the ingest_core run() and preview() loops — no media download AND no post-record stub (operator: 'no stub for gated content'). Skipped before the post-record block so gated posts never inflate the #862 body canary; surfaced as 'N gated-skipped' in the run summary. Same gate in preview() for preview/apply parity (rule 93). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
60a9c9e6ef |
refactor(ml): drop GPU code, cap inference threads by default (#747/#872)
GPU enablement (#872) cancelled — not worth the Pascal-specific build for a modest CPU→GPU win on an old P4. Remove the dead GPU code (device.py, the CUDA provider branch in tagger, the .to('cuda') path in embedder) so nothing carries it forward. Instead, bound CPU inference threads by default so the ml-worker is a predictable core consumer on a SHARED node — the intended scaling model is multiple worker replicas (each --concurrency=1, each its own cgroup limit), not one big container. ONNX Runtime and torch otherwise size their thread pools to ALL host cores, so each replica would grab every core and oversubscribe / starve the co-located DB+web. Cap both to _INTRA_OP_THREADS=4 (matches the prior per-worker cpus:4 unit): run N replicas where N×4 stays within the cores allotted to ML. - tagger: ort.SessionOptions().intra_op_num_threads = 4 (CPUExecutionProvider). - embedder: torch.set_num_threads(4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
db7e1f2b59 |
feat(ml): GPU-capable tagger + embedder with CPU fallback (#872)
Step 1 of GPU enablement (code only — CPU-safe, CI-green; the CUDA image is a separate step pending the host driver version). - New services/ml/device.py: FC_ML_DEVICE (auto|cuda|cpu) intent + VRAM knobs (FC_ML_ONNX_GPU_MEM_GB, FC_ML_TORCH_MEM_FRACTION). Per-worker-host bootstrap → env, not a DB setting (the GPU host runs CUDA, others CPU). - tagger: use CUDAExecutionProvider (with gpu_mem_limit) when requested AND the provider is actually present (onnxruntime-gpu), else CPUExecutionProvider. Logs the active providers. - embedder: move model + inputs to cuda when requested AND torch.cuda is available; cap torch's VRAM share; .detach().cpu() before numpy. fp32 kept so GPU embeddings stay in the same space as existing CPU ones. Both AND the env intent with the framework's real availability, so on CPU (CI / CPU onnxruntime / no GPU) they fall back cleanly — behavior unchanged. The 8GB P4 is shared by both frameworks, hence the conservative default caps. Tests: device env parsing. (tagger/embedder GPU paths are operator-verified on the GPU host — models aren't in CI.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
369e3de684 |
feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)
Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag firing on one frame survived at peak confidence, and a fixed count under-samples long multi-scene videos so real scene-local tags looked like noise. Redesign (operator-steered): - Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds` (default 4) across the 5–95% window — so a tag's frame-presence reflects real screen time independent of video length. Capped at `video_max_frames` (default 64): a long video stretches the spacing instead of exploding into hundreds of inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg timeout also cut 60s→30s). - Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in >= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on screen — duration-independent noise rejection), with confidence = MEAN over the frames it appears in (not max). Clamps the threshold to the sample count so a 1–2-frame short video still tags. - All three knobs are DB-backed ml_settings (migration 0053), patchable via /api/ml/settings + sliders in the ML settings card — replaces the VIDEO_ML_FRAMES env var (product-not-project). Tests: aggregation drops one-frame noise + means corroborated tags + clamps on short videos; settings round-trip + min>max validation. Replaced the _maxpool_predictions unit test. NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs CPU-only — is GPU enablement, tracked separately in #872. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
41652db20f |
feat(maintenance): retroactive video-dedup action — preview + apply (#871)
Phase 2 of #871: clean up the duplicate videos already in the library (the #859 "same video from multiple sources" clutter). Import-time dedup (Phase 1) only prevents NEW dups; this is the operator-triggered cleanup of existing ones. cleanup_service.dedup_videos(dry_run): - backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the existing library participates; idempotent (only NULL rows), writes a negative sentinel for un-probeable files so they're neither re-probed forever nor matched. - find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect, anchored per cluster to bound the span (no chain drift); keeper = highest pixel area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances. - apply: re-point each loser's post links to the keeper (so no post loses the video) THEN delete the redundant records + files via delete_images (cascade). dry_run shares the same discovery predicate and returns the projection only (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated). - dedup_videos_task (maintenance queue; summary → task_run.metadata). - POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so the card shows the dry-run projection before the destructive apply. - VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind a confirm dialog. Mounted in the Maintenance panel. Tests: dedup collapses + re-links the loser's post to the keeper + removes the file; dry-run deletes nothing; distinct durations aren't grouped; task registered. (Migration 0052 for duration_seconds already shipped with Phase 1.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
f154603811 |
feat(import): Tier-1 video near-dup by duration+aspect (#871)
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
8dee2f9628 |
feat(import): recurse nested archives + precise "no images" reason (#718)
Root cause (operator-confirmed via event metadata + lsar): a "High Resolution files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase). _import_archive only extracted one level — a nested-archive member failed is_supported and was skipped, so the real pages were silently dropped and the post showed "archive but no images". The disk scan found this pattern recurring across the attachment store. - Recurse into nested archives via _collect_archive_members: a member that is itself an archive is bomb-probed and extracted too, depth-capped at _ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so they link to the right Post. Each level is wrapped so one bad nested archive can't abort the import. The shared path means external (mega/gdrive) archives recurse too. - Replace the catch-all "held no supported members" string with a per-outcome tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped case is now recognised as BENIGN — images already in the library, re-linked to this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no longer false-flags in event metadata.unextracted_archives. Genuine failures carry the precise breakdown. Tests: nested zip-in-cbz imports both inner images + links them to the outer post; all-deduped archive returns attached with error=None and links images to both posts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
8b99dc9b81 |
test(downloader): fix test_sidecar_written_and_findable for post-first minimal sidecar (#856)
The per-media sidecar no longer carries title/url/content (post-first, #856) — update the assertion to expect image identity only (category/id/source_url). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
dcbc3ae335 |
refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
Milestone #67 step 2. On the native core ingester the Post becomes the single authoritative record for body/links/metadata, captured once per post by the post-record; the per-media import only links image provenance + localization. Before: every per-media sidecar carried the full post body, so a post with N images wrote the body N+1 times (post-record + N media) — redundant on disk and a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only body source). - patreon_downloader: the per-media sidecar is now minimal — {category, id, source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body resolution + detail-fetch (the post-record, written first in the walk, already did it). Body no longer duplicated next to each image. - importer: new per-instance `post_first` flag (Importer is per-task). When set, `_apply_sidecar` still writes source_filehash + provenance + primary_post_id but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata, so applying a body-less sidecar would clobber raw_metadata + re-sync links off empty data). Default False keeps gallery-dl writing post fields. - download_service: `_phase3_persist` sets importer.post_first = uses_native_ingester(platform) — the future-proof seam, so a platform migrating onto the native core flips to post-first automatically (step 3). Media imports before post-records but both unify on external_post_id, so the post ends with its body either way. Tests: per-media sidecar is minimal + never hits the detail fetcher; attach post_first=True links provenance/localization but writes no post body/title; post_first=False (gallery-dl) still applies them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
00607a309b |
feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)
If Patreon renames/restructures the post body field again (as content → content_json_string already did), every body silently comes back empty and we'd archive empty posts without noticing. Surface that as a loud failure. Research-grounded design (Patreon `content` is officially null|string, body has no post_type gate, gallery-dl independently added the same content_json_string fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction threshold would false-positive constantly. The robust, creator-independent break signature is "a meaningful sample of posts, a body extracted from NONE of them." - ingest_core counts posts_recorded / posts_with_body on the native post-record path (gallery-dl never enters it, so the canary is native-only by construction). - When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are literally "fix the field-set/parser, not creds"). Placed after the timeout/stop returns so it never masks a more specific failure. - Run summary always appends "bodies X/Y" for sub-threshold observability (a partial regression that still extracts some bodies shows in the Raw stdout). Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success; below the sample floor -> success (tick safety). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
949c9abcc6 |
fix(external): path-safe unlink + per-link staging + orphan repair (#859)
External downloads import IN PLACE, so the post-attach dedup-skip unlink could delete a file that IS an ImageRecord's backing file — orphaning the record and 404-ing on playback. Two sources of that: - Two links on the same post (same film from mega + gdrive) emitted the same filename into one external/<post_id>/ dir; the second overwrote the first. Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path. - The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it path-safe: only unlink when `f` is NOT the existing record's canonical file. Plus an operator-triggered orphan-repair maintenance task (prune_missing_file_records_task) to clean up records already orphaned by the bug: scans ImageRecords, deletes those whose file is gone (cascade), with an NFS-stall guard that aborts without deleting if a large sample is mostly missing. Wired through POST /api/admin/maintenance/prune-missing-files and a MissingFileRepairCard in the Maintenance panel. Tests: refetch-same-link keeps the canonical file; orphan repair deletes only real orphans and aborts on the mostly-missing guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
976107bbe8 |
fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)
THE empty-body root cause. Patreon deprecated the flat `content` HTML field — it returns null on the feed AND the detail endpoint, for every post type (confirmed against the live API: all 135 StickySpoodge posts, text_only/ image_file/poll alike). The real body now lives in `content_json_string` (a ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse fields[post]=content request omits it. Not credential, not post_type: a request shape gone stale. - NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks bold/italic/underline/strike/code/link, hardBreak, inline images, lists, headings; unknown nodes degrade to children). post_body_html(attrs) = the one resolver: legacy content HTML else convert content_json_string. - patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post]) and resolve via post_body_html (replaces the wrong sparse req + full-fetch fallback). - patreon_downloader._write_sidecar_data: resolve body via post_body_html (feed content_json_string) before the detail-fetch; memoize resolved HTML. - tests: prosemirror converter unit tests; client legacy + content_json_string paths; contract pins content_json_string. Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
eb811e11f6 |
refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
Two corrections from operator review: 1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI section — the native ingester now writes a per-post line into the run stdout (parity with gallery-dl's per-file stdout), so the per-post handling shows in the panel the operator already uses. 2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring the download_post -> MediaOutcome contract — and the downloader owns the read; ingest_core only formats the outcome into the log line. Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture' section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+ ' — EMPTY' when 0), so an empty body is self-explanatory by post_type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
bcc7266021 |
feat(downloads): per-post body-capture diagnostics in the event UI (#842)
Operator can't (and shouldn't have to) hunt worker logs to see why a recapture
left a post body empty. Surface per-post handling ON THE EVENT, in the UI.
The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a
per-post diagnostic {post_id, title, post_type, body_chars} with zero extra
fetching — a 0-char body next to its post_type explains an empty post at a
glance (e.g. polls/embeds whose body the API never returns).
- ingest_core: accumulate post_diagnostics; thread via DownloadResult
- download_service: write to DownloadEvent.metadata_['post_diagnostics']
- DownloadDetailModal: 'Post capture' section — totals + empty-body table
(post_type + chars, flagged) + all-posts table; included in Copy-all
- tests: ingester diag (post_type + body_chars), download_service metadata
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
3df191e255 |
fix(patreon): full-fetch fallback when sparse fieldset returns null content (#842)
Operator-flagged: 9 StickySpoodge posts had empty bodies in FC despite the body plainly existing + being accessible (creds refresh didn't help). All 9 are body-only / poll / embed / announcement posts with no downloadable gallery media — Patreon's detail endpoint returns content:null for these under the sparse fields[post]=content request even though the body exists. fetch_post_detail_content now re-fetches the FULL post resource once when the sparse request comes back empty: recovers the body when the sparse fieldset was the cause, and logs post_type when even the full resource is empty (body lives elsewhere). Only the empty cases pay the extra GET; the 126 already-working posts keep the fast sparse path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
b999480db5 |
feat(ingest): per-post body-capture + recapture diagnostics logging
Operator-flagged: a recapture 'caught nothing' for a post and there were no logs explaining why. Three silent spots now log, so a recapture's per-post outcome is diagnosable (retention bounds the volume): - patreon_client.fetch_post_detail_content: the 200-OK-but-null-content branch was silent — now logs 'fetched N chars' on success AND 'empty/null content (tier-gated or no text)' on the empty case (the most common silent miss). - patreon_downloader.write_post_record: logs each post's FINAL body outcome (captured N chars / NO body) read off the memoized attrs after detail-fetch. - ingest_core summary: appends post-record + relinked counts to the run summary (surfaces on the event stdout the operator already reads). - download_service phase3: logs how many on-disk images got source_filehash relinked (N/total) per recapture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
65ec29ba9b |
feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
96c29c370b |
feat(ingest): localize inline post-body images to local copies (Phase 2)
Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
5e1655384f |
feat(ingest): recapture body + links for every walked post (Phase 5)
Operator reframed backfill as inherent to the existing walk: you can't fill links the system never had by re-downloading media that's already on disk, so the body/link recapture has to ride the walk itself. Hoist the post-record capture out of the media-less branch so it runs for EVERY post — gated once per post by the synthetic post key in the seen-ledger (detail-fetch for an empty feed body happens at most once; recovery re-captures unconditionally). A normal BACKFILL now walks history and recaptures each post's body + external links (which phase 3 imports via upsert_post_record → _sync_external_links → the download sweep, all already wired). A tick captures new posts going forward. No separate button — the backfill is the backfill. Tests: media posts now also carry a synthetic post-key ledger row (count assertions +1); new test proves an already-on-disk media post still recaptures its body/links on a re-walk. Completes the core of #830 (Phase 5). Phase 2 (inline-image localization) remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
8dbf29f803 |
feat(external): per-host enable toggles in Settings (Phase 4d)
Operator lever: disable a single file host (e.g. mega.nz when it's banning) without touching the others. Five booleans on import_settings (extdl_<host>_enabled, default true — works out of the box, rule #26); the worker already reads them via getattr so no worker change. Migration 0050 + model fields + settings GET/PATCH (uniform boolean validation) + a 'External file-host downloads' card in the subscriptions Settings tab. Completes Phase 4. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
05f226a8f6 |
feat(external): zip-parity provenance/tagging + thorough worker logging
Operator-requested: a worker download must be tagged + provenance-associated exactly like an extracted zip, and the path must log well (we won't get it right first try). - _route_files now mirrors download_service._phase3_persist branch-for-branch: imported/superseded → collect member_image_ids+image_id (provenance-linked via the synthesized sidecar, same as extracted-zip members) → caller enqueues tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed → unlink + warn. - Logging at every stage: start (link/host/post/artist/attempt/url), requeue, fetch result (files/bytes) or fetch failure, per-file import decision, dead- letter transitions, and done (files/images/duration). - Parity test: an archive downloaded by the worker is extracted, provenance- linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for exactly the member images. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |