feat(subscriptions): dry-run backfill preview — B4 preview (plan #708)
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -144,6 +144,54 @@ async def _run_native_ingester(
|
||||
return dl_result, resolved_campaign_id
|
||||
|
||||
|
||||
async def preview_source(
|
||||
*,
|
||||
platform: str,
|
||||
url: str,
|
||||
source_id: int,
|
||||
config_overrides: dict | None,
|
||||
cookies_path: str | None,
|
||||
images_root: Path,
|
||||
sync_session_factory,
|
||||
page_limit: int = 3,
|
||||
) -> dict:
|
||||
"""Dry-run preview for a native platform (plan #708 B4): resolve the campaign
|
||||
id, then walk a few pages counting media not already seen/dead — no download.
|
||||
|
||||
Returns the preview dict (total_new / posts_scanned / pages_scanned /
|
||||
has_more / sample), or `{"error": msg}` on a resolve / auth / drift failure.
|
||||
Native-only — the caller gates on `uses_native_ingester`.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from .patreon_client import PatreonAPIError
|
||||
|
||||
campaign_id, _ = await resolve_campaign_id_for_source(
|
||||
url, cookies_path, config_overrides or {}
|
||||
)
|
||||
if not campaign_id:
|
||||
return {
|
||||
"error": (
|
||||
"Couldn't resolve the campaign id from the source URL "
|
||||
"(cookies expired, or the creator moved/renamed?)."
|
||||
)
|
||||
}
|
||||
ingester = PatreonIngester(
|
||||
images_root=images_root,
|
||||
cookies_path=cookies_path,
|
||||
session_factory=sync_session_factory,
|
||||
)
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
result = await loop.run_in_executor(
|
||||
None,
|
||||
lambda: ingester.preview(source_id, campaign_id, page_limit=page_limit),
|
||||
)
|
||||
except PatreonAPIError as exc:
|
||||
return {"error": f"Couldn't preview: {exc}"}
|
||||
return result
|
||||
|
||||
|
||||
async def verify_source_credential(
|
||||
*,
|
||||
platform: str,
|
||||
|
||||
Reference in New Issue
Block a user