DRY pass follow-up (note #1026). Five handlers returned the identical
jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result).
Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives,
trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews.
trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it).
Added route-level tests for all five consumers (these trigger endpoints had no
route coverage before): 202 + task_id via _queued, and the dry_run flag threading
through to dedup/purge-gated. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened
process #594). Five Tier-A endpoints repeated the same get_json -> dry_run ->
run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn,
**kwargs); the five route handlers now delegate. reconcile keeps its source_id
validation and passes it through **kwargs.
The cleanup_service predicates were already shared between preview and apply
(find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no
backend-logic change; this is purely the HTTP-handler boilerplate.
Consumers (all routed through the helper, verified no copy left behind):
prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id),
purge_legacy_tags, reset_content_tagging.
Added route-level tests for prune-bare (apply) and reconcile (apply + source_id
passthrough + invalid-source_id 400) — the two helper consumers that previously
had only service-level coverage, so every consumer is exercised at the route.
Findings B (queued-response helper) and C (store dry-run POST helper) identified
but not applied this pass (operator scoped to A). The card preview->commit state
machine is deferred to a frontend pattern-consistency sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in
eff6427; closing it out after today's #87 work, which added a seam it didn't
account for. _repoint_post_links drops a loser post's ImageProvenance row on the
(image, post) uniqueness collision — and that row may now carry from_attachment_id
(which archive the file was extracted from). For the exact gallery-dl->native
case this targets, the keeper is the native stub (no archive) and the loser is the
gallery-dl row that extracted the member, so a blind delete silently lost the
containing-archive linkage. Carry from_attachment_id onto the keeper's surviving
row (when NULL) before dropping the collision.
The rarer PostAttachment-collision case (both dup posts captured the same archive
blob) doesn't arise in the targeted scenario — the archive lives only on the
gallery-dl post, so it re-points straight to the keeper and the FK stays valid.
Test: collision merge preserves the loser's from_attachment_id on the keeper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A filesystem-imported archive with no adjacent sidecar has no Post, so
_post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced
post.id. _stamp_member_archive already no-ops on a None post_id (no post → no
provenance to stamp); pass None instead of crashing. Caught by the existing
test_reimport_archive_is_idempotent (no-sidecar zip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from 068def2.
- ProvenancePanel: heading pluralizes ('Attachment' for a single file).
- Backfill: re-running reextract_archive_attachments (ArchiveReextractCard)
routes through _import_archive and stamps existing rows — no new code.
Tests: capture stamps on fresh import, nested-archive attribution, per-post
archive on dedup; for_image filters to the containing archive; reextract
backfill stamps the link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Attachments section aggregated PostAttachment rows across EVERY post an
image was pHash-linked to. When one of those was a 'High Resolution Files'
mega-bundle (dozens of unrelated archives), the list ballooned past the
viewport and overwhelmed the modal's right rail.
- for_image() now scopes attachments to ImageRecord.primary_post_id (the post
the file was actually captured from), falling back to all linked posts only
when primary_post_id is unset (older rows / filesystem imports).
- ProvenancePanel wraps the list in a max-height scroll container with a count
in the heading, mirroring the cards' independent-scroll treatment.
Note: FC stores archives as opaque blobs and never records which archive an
extracted image came from, so attachments can't yet be scoped tighter than the
post. Capturing image->archive containment is tracked as separate work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled
viewport; expanding a tile to read into it is the only reason to scroll.
- Every Maintenance card converted to the collapsible MaintenanceTile (collapsed
by default = icon + short title + one-line blurb). Task cards (ML backfill,
centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance)
sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion
thresholds, allowlist, aliases) grouped in one Tagging section as collapsible
tiles; Backup is its own collapsible tile.
- Three labeled sections mirror the Cleanup tab: Backfills and reprocessing /
Tagging / Storage.
- Center the whole Settings surface: SettingsView is now a centered, width-capped
(1140px) column so the tab strip and every panel sit in a tidy centered measure
(was full-width). CleanupView drops its own left-aligned max-width to fill it.
All card logic unchanged - only the chrome.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long
descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles
(2026-06-18: keep both tabs, group inside, compact tiles in a grid).
New common/MaintenanceTile.vue: a compact expandable tile (icon + short title +
one-line blurb collapsed; click the header to expand the full controls/preview/
result inline; keyboard-accessible button + focus ring; tints the
icon, keeps a running task expanded).
Cleanup tab (this pass) restructured into 3 sections — Import-filter audits
(Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts,
Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy,
Reset content tagging, Standardize casing) — each a responsive grid of tiles.
PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved
VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are
destructive content cleanup). All card logic unchanged — only the chrome.
Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Image posts wrap the post date in an <a> permalink
(<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts
don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY
the unwrapped case, so every image post got a null published_at and sorted to the
top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's
_data_from_post method: text up to the first </, then after the last > — handles
both. Verified against the live raw feed (all 6 dates now parse).
Robust logging (operator request): _parse_posts now logs per-page parse stats
(posts / dated / with-body) and a WARNING canary when posts parse but NONE get a
date or body while the raw markers are present — i.e. our extraction diverged
from the live markup. Makes this failure class diagnosable from the worker log
alone, no authed re-fetch needed.
Test: a permalink-wrapped date parses to ISO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SubscribeStar posts deliver content only through document/audio attachments,
which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them:
- docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the
href URL + doc_preview-title + data-upload-id (kind=attachment).
- audio: scope uploads-audios..post-edit_form, split on audio_preview-data
blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio).
The existing downloader handles them unchanged (plain streaming GET; the file
validator only inspects image/video extensions via is_validatable, so PDFs/zips/
audio pass straight through, no quarantine).
Test covers doc + audio extraction (the cheunart sample has none, so this pins
gallery-dl's documented markup shape).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body
regex either returned empty or over-captured into sibling upload divs and the
'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact
_data_from_post rule — content between the post_content-text wrapper and the
youtube-uploads div (literal markers), then strip the trix editor's
<html><body>…</body></html> document wrapper to its inner. Verified against the
live cheunart sample: clean per-post bodies, empty for genuinely text-less posts.
Also port gallery-dl's _media_from_post preview guard: skip gallery items whose
URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of
the Patreon gated-preview bug (#874); this is why a locked post yields no media.
Tests: body must not bleed into the pagination counter; trix html-document
wrapper stripped; /previews items skipped. Fixture now includes the youtube-
uploads close marker present in real markup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_
subscriber even with valid .adult cookies (confirmed present: _personalization_id
+ _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's
ENTIRE flow — including the base Extractor._init_session I'd not read — the
divergence is the HTTP request profile, not the cookies or parser.
gallery-dl's default (cookies-only) mode sends, on EVERY request including the
first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same-
site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint
is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling
looked unlike a browser → SubscribeStar gated the adult-creator page.
Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept-
Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the
per-request XHR headers (both feed and load-more now use the shared profile).
Test updated to assert the gallery-dl-parity profile instead of the old
navigation-vs-XHR split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native client split the feed on `<div class="post is-shown`, but `is-shown`
is added by SubscribeStar's infinite-scroll JS when a post scrolls into view —
present in a browser-SAVED page (what the Step-0 characterization used) but
ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live
feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError.
Align with gallery-dl's proven `_pagination`: split on the generic
`<div class="post ` (trailing space rules out the hyphenated post-content/
post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating
detection (/verify_subscriber, /age_confirmation_warning => auth, not drift).
Regression tests: raw server markup without is-shown now parses; an age-wall
redirect raises SubscribeStarAuthError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart
still drifts — we're being served a full HTML page that isn't the feed. Add
_describe_page(): the drift error now reports the page <title> + which known
interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha),
so the next run names the actual cause instead of "markup changed". Strong
suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is
UA-locked and our hardcoded UA likely differs from the cookie-capturing browser).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First live run (cheunart) tripped the drift guard: "no posts and no recognizable
feed container". The browser-saved page was normal (6 posts + posts_container-list),
so the parser was fine — our live HTTP fetch got a different response. Cause: the
client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the
initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content-
negotiates an XHR full-page request to a non-HTML body → no container → drift.
Fix: the session now uses browser-like navigation headers (Accept: html, NO
X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on
the "load more" endpoint (which is a genuine XHR). Drift message now reports the
response length + a JSON hint so a recurrence is self-explaining. Regression test
pins the header split (navigation initial GET, XHR load-more).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at
import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester",
_FakeIngester) no longer affected dispatch → the fake's run() never ran →
KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the
dict with a _native_ingester_cls() call-time lookup that reads the module globals,
so monkeypatching the class names works again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.
- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
_NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
/ preview_source / verify_source_credential platform-aware. SubscribeStar's
campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
recover/recapture actions show for subscribestar; download_service's
cursor/mode/post_first + the preview endpoint already key on
uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
verify, gallery_dl_service skip-value, api_sources arm-no-preflight).
post_is_gated stays best-effort (can't cause junk downloads); not gating this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 3 — the observability half. ingest_core accumulated ALL
human-readable progress in log_lines → DownloadResult.stdout, persisted to the
DownloadEvent ONLY at phase 3; the real logger was used almost nowhere. So a
worker SIGKILL/OOM/hard-time-limit mid-walk left NO trace (the "task died,
no trace" mode from the recovery-sweep work).
Route run milestones through the container log too, each carrying source_id (L3
context):
- run START (platform/mode/source/campaign/resume_cursor)
- per-PAGE breadcrumb (posts/downloaded/skipped/errors/quarantined/gated/cursor)
at each page boundary — pages are minutes apart on big backfills, so this shows
how far a since-died walk got
- final SUMMARY (same string as the stdout summary)
- operator STOP
(L2 — quarantines log.warning'd — already landed in commit 1's base
_validate_path; failures log.warning via the base _failure_result; the #862 body
canary already log.error's.) log_lines/stdout content is unchanged (summary just
captured in a var), so existing assertions hold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).
- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
(status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
("Patreon API" / "SubscribeStar markup"), preserving the existing message
(test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
-unused DownloadResult/ErrorType/*Auth/*Drift imports.
Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing
the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was
importing patreon privates — wrong owner). New backend/app/services/
native_ingest_common.py is the neutral home for:
- make_session (was _load_session ×2), retry_after_seconds + 429 constants,
sanitize_segment, basename_from_url, post_dir_name, MediaOutcome /
PostRecordOutcome.
- BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume)
and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it;
each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch
enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the
divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone.
- Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only).
post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO).
Tests repointed to the single source at every consumer (rule 93 / §8b parity):
patreon_client/downloader, subscribestar_native. Exception-trio consolidation +
base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core
ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger:
SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery
bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing
media stops re-burning backfill chunks). Per operator decision, dedicated
per-platform tables (not a generalized shared ledger).
filehash is String(128): a CDN content hash when the URL carries one, else a
synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key.
Registered in models/__init__; migration 0054 creates both tables (down 0053).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the
old `*, timeout` signature; the task now calls fetch_external() without it, so
the stub raised TypeError in the integration lane (run 1191). Switch it to
**kwargs like the other two.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).
Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
(connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
across chunks in _stream_to_file (HTTP has no total-download timeout), and
passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.
Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.
Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.
Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
celery_signals._queue_for is a hand-maintained mirror of task_routes that
stamps TaskRun.queue in the prerun signal. It was missing the
backend.app.tasks.external. prefix, so external fetches recorded
queue='default' even though celery routes external.* → download and runs
them on the download worker. The dashboard's per-queue filters and the
per-queue recovery-sweep threshold therefore missed them — the same
'queue column lies default' gap the 2026-06-02 audit fixed for
backup/admin/library_audit.
Map external.* → download in _queue_for. Composes with the fetch_external_link
task-name sweep override (#883), which wins by precedence regardless of the
recorded queue. Pinned test asserts the mirror agrees with the actual route.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
External file-host fetches run to a 60-min hard limit (time_limit=3600,
per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min
default. recover_stalled_task_runs was phantom-flagging healthy in-flight
fetches as "RecoverySweep: no completion signal received within 5 min"
before the task's own timeout/error handling could surface the real error
(operator-flagged: target 414 swept at 6.6min).
The sweep already has per-queue/per-task overrides for long tasks, but
fetch_external_link was never added and its TaskRun records queue='default'
(no queue override) despite external.* routing to download. Add a task-name
override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it
robust regardless of the recorded queue. No new internal timeout needed —
the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception
already capture the real failure once the sweep stops preempting.
Pinned tests: external-fetch override survives a 10-min row / flags a 70-min
row on queue='default'; invariant guard asserts override >= hard time_limit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_artist_directory_service._seed_image built sha256 from
abs(hash(suffix)) % 10000 — PYTHONHASHSEED-randomized hash() over only 10k
buckets, so two suffixes in one test could birthday-collide and violate
uq_image_record_sha256. Flaky per process seed: passed on dev (run 1179),
failed on main (run 1182) with identical code. Use
hashlib.sha256(suffix).hexdigest() for a stable, collision-free digest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recompute_centroid + recompute_centroids were the only tasks still using
the process-wide singleton extensions.get_session() under asyncio.run().
The async engine's asyncpg pool is bound to the loop it was created on;
each Celery task runs a fresh asyncio.run() loop, so after the first
invocation the cached engine handed loop-A connections to loop B and raised
"Future attached to a different loop" — every recompute after the first in
a worker process failed (~35ms, fails on first DB await).
Convert both to the established per-task async_session_factory() pattern
(NullPool engine created + disposed inside the task's own loop), matching
scan/download/admin tasks. No get_session usages remain in tasks/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Long-running maintenance tasks must survive navigating away or reloading
the page. VideoDedupCard + GatedPurgeCard held the in-flight Celery task id
only in component refs and polled task-result inline, so leaving the page
mid-run lost the id and the result was never shown — even though the task
finished on the worker.
New shared composable useMaintenanceTask: persists {taskId, mode, startedAt}
to localStorage on dispatch, re-attaches on mount, and re-shows the result
when the task finishes (the celery result backend retains the summary well
under result_expires). Stale-guard skips resume past 3h. Both cards refactored
onto it; card-specific computeds + confirm dialog kept.
Also fixed the QueueStatusBar lane: both cards watched queue="maintenance"
but tasks.admin.* routes to maintenance_long, so the bar never reflected
their own task — now queue="maintenance_long".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A one-shot Maintenance action to remove the blurred locked-preview images
the ingester downloaded from tier-gated Patreon posts before #874.
current_user_can_view was never persisted, so the cleanup re-walks each
enabled Patreon source (read-only) to re-derive which posts are gated now
and the blurred filehashes Patreon serves for them, then matches by
CONTENT HASH against stored source_filehash. Because the hash is
content-addressed, a real file downloaded when access existed has a
different hash and can never match — regained-then-lost-access content is
provably spared (operator's hard requirement). NULL source_filehash =>
unverifiable, kept + reported.
On apply: delete matched ImageRecords + files (provenance cascades),
clear seen/dead-letter ledger rows for those hashes so the real media
re-ingests if access returns, and delete gated posts left bare. Shares
one match predicate between preview and apply (rule 93).
- cleanup_service: collect_gated_previews + purge_gated_previews
- tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed)
- api.admin: POST /maintenance/purge-gated-previews
- GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply)
- tests: collect predicate, hash-match delete/spare/unverifiable, ledger
clear, bare-post removal, no-op
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit's Edit orphaned the recapture test's relink/stdout
assertions into the new preview test (F821 res_recap/downloader2/m1) and
the gated-skip test's written_paths check matched 'gated' in the tmp dir
name. Restore the recapture assertions to their function and assert on
the media basename instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Patreon serves only blurred locked-preview thumbnails for posts the
authenticated account can't fully view; the native ingester was
downloading those as real media. current_user_can_view was already in
_FIELDS_POST but never read.
Add PatreonClient.post_is_gated (gate ONLY on explicit
current_user_can_view=False; missing/None → viewable, never over-filter)
and skip gated posts at the top of the ingest_core run() and preview()
loops — no media download AND no post-record stub (operator: 'no stub
for gated content'). Skipped before the post-record block so gated posts
never inflate the #862 body canary; surfaced as 'N gated-skipped' in the
run summary. Same gate in preview() for preview/apply parity (rule 93).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bvandeusen