dev
503 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
c65da42593 |
fix(patreon): handle the /cw/ creator-URL prefix in vanity extraction
A source URL like https://www.patreon.com/cw/Atole resolved vanity='cw' — the vanity regex only skipped a /c/ prefix, so Patreon's current /cw/ ("creator workspace") form fell through to the bare-vanity branch and captured the prefix instead of the slug. Every /cw/ source then failed campaign-id resolution (API + page-scrape both looked up "cw"). Operator-confirmed 2026-06-07 via the new error text: source_url='.../cw/Atole'; vanity='cw'. Add cw/ to the optional prefix group (ordered before c/ so the longer prefix wins), and have the creator-page fallback try the /cw/ form too. Test covers bare / c/ / cw/ extraction and that id: URLs stay non-vanity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a559fabdd5 |
feat(patreon): scrape creator-page HTML as a campaign-id resolution fallback
Patreon's /api/campaigns?filter[vanity]= lookup returns empty data for creators
that plainly exist (operator-flagged 2026-06-06 — Atole etc. erroring at the
resolve step). gallery-dl never used that endpoint; it pulls the campaign id out
of the creator page's bootstrap JSON. Add the same as a fallback: when the API
misses, GET the creator page (bare + /c/ vanity paths) and scrape the first
campaign id from any known embedding ("id":"…","type":"campaign" /
"campaign":{"data":{"id" / /api/campaigns/<id> / "campaign_id"). API is still
tried first (cheap, structured); the page scrape only runs on a miss.
Tests: API-empty → page-scrape fallback resolves; _scrape_campaign_id pattern
coverage. Existing API-path tests unchanged (happy paths short-circuit before
the fallback; failure paths hit the guarded scrape and still return None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
df2310bc70 |
fix(tags): preserve acronym casing in tag normalization (DC, NSFW)
normalize_tag_name now only capitalizes the first letter of each word and leaves the rest of the word untouched (was lowercasing the tail, which turned DC→Dc / NSFW→Nsfw). This matches ml/tag_name._title_word, so a Camie-suggested tag keeps the exact casing the suggestion UI showed when it round-trips through POST /api/tags on Accept — addressing "auto-suggested tags must obey capitalization" and "don't mangle acronyms" in one rule. Trade-off (operator-chosen): all-caps input no longer folds to Title Case, so case-variant merging in #714 still folds the dominant lowercase-vs-Title case but leaves all-caps stylizations distinct (protecting acronyms wins). Tests updated + a new test documenting acronym preservation / non-folding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3c89223dcb |
feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714)
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps whatever casing it was created with. This adds a maintenance action that Title-Cases every existing tag (collapsing whitespace) and merges case/whitespace-variant duplicates into one. Backend: - tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor (prefer an already-canonical member → no rename/self-alias; else the best-connected tag → fewest FK repoints; else lowest id), merges the variants INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/ aliases/series_page repoints + protective ML aliases), then renames the survivor to canonical. Losers are deleted before the rename so there's no transient unique-index clash; commits per group and isolates failures per group. Idempotent — an already-canonical lone tag is a no-op. - normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i. - POST /api/admin/tags/normalize: dry_run=true returns a projection inline (group/collision/rename counts + sample); dry_run=false enqueues the task. Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup tab) — preview → apply (polls the activity dashboard to terminal status), behind a back-up-first warning. admin store gains normalizeTags(). Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept separate, ML-known loser keeps a protective alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
23f452021f |
fix(tags): Title-Case operator-entered tags at create endpoint only (plan #701)
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in the POST /api/tags handler so operator-entered tags get clean display casing. It is NOT applied in the shared find_or_create / rename paths — those are used by the ML tagger and allowlist matching, which must preserve the booru vocabulary's original casing (Title-Casing it broke apply_allowlist matching). find_or_create / rename keep case-insensitive lookup + clash detection so a differently-cased entry dedups onto the existing tag instead of forking. Tests updated to expect Title-Cased create output (sunset→Sunset, character:Saber→Character:saber, http://example.com→Http://example.com) and a dedicated normalize_tag_name unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
62cca64dce |
feat(downloads): live per-file progress on running events — #709
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
2b69540ecc |
feat(tags): Title-Case normalization on create/rename — #701 (core)
Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).
normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.
Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
a497104661 |
feat(maintenance): re-extract archive attachments + link to post — #713 part 2
Existing PostAttachments that are actually archives (filed opaquely before the magic-byte gate) need extracting retroactively. cleanup_service. reextract_archive_attachments scans PostAttachments, magic-detects the archives, and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place in a temp dir — so the members extract and re-link to the SAME post via find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by sha256). Enqueues thumbnail+ML for new members. Wired as a maintenance-queue Celery task (tasks/admin) + POST /api/admin/maintenance/reextract-archives (202) + a "Re-extract archive attachments" card in Settings → Maintenance. Test: a zip stored under a mangled extension-less name extracts + links its member to the post via ImageProvenance, and a second run is a no-op (idempotent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5bb25245a5 |
fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1
Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").
Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.
Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
e82c2ee57b |
feat(subscriptions): dry-run backfill preview — B4 preview (plan #708)
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
cd43439401 |
feat(ingester): graceful mid-walk cancel on Stop — B4 cancel (plan #708)
Owning the walk lets Stop interrupt a live backfill chunk instead of letting it run to its ~14.5-min time-box. ingest_core.run now polls _backfill_state at each page boundary (a short SELECT, never held across the walk) and bails with PARTIAL when an operator Stop has popped it. Latched on the first observed "running" state so a run invoked without one (unit test / stale call) never self-cancels. Progress is already checkpointed per-page, so a restart resumes from the cursor; Stop clears it for a clean reset. No UI change — the existing Stop button now just takes effect immediately. Tests: _still_running reads the state; a latched run bails PARTIAL at the next boundary when the state disappears (only the pre-cancel post ran). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
bde19944db |
test(patreon): fix _BoomSession stub for the B5 headers kwarg
test_one_failure_isolated's _BoomSession overrode get() without the headers kwarg _fetch_to_file now passes (B5 Range resume), so the call TypeError'd and both items errored. Add headers=None to match the base fake. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
402086c34c |
feat(patreon): resume partial media downloads via HTTP Range — B5 (plan #708)
Owning the media fetch means a mid-download transport cut no longer refetches from zero. _fetch_to_file now resumes: on a transient retry, if bytes already landed in the .part, it requests Range: bytes=<have>- and appends on a 206; falls back to a clean truncate-and-restart if the server ignores Range (200) or the range is past EOF (416). The .part staging means a non-range server never corrupts the output — worst case is the old behavior (refetch from zero). Tests: mid-stream cut resumes from the offset (asserts the Range header); a Range-ignoring server refetches clean (no double-write). Test session fakes updated to accept the new headers kwarg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
fb7383eea7 |
feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708)
Owning the native client means we see the 429 Retry-After header — previously discarded. PatreonAPIError now carries `retry_after`; on a PERSISTENT page-fetch 429 the client attaches the server's raw Retry-After seconds. New DownloadResult.retry_after_seconds; patreon_ingester._failure_result sets it on RATE_LIMITED. download_service._update_source_health passes it to set_platform_cooldown as `seconds=`, clamped to [60, 3600] (a tiny hint can't leave the platform effectively un-cooled; a huge one can't strand it for hours); no hint → the flat 900s default. So a rate-limited platform cools for as long as the server actually asks, not a fixed guess. Tests: terminal 429 surfaces retry_after (test_patreon_client); cooldown honors + clamps the hint, falls back to default when absent (test_download_service). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e47fa0cf4b |
refactor(downloads): unify phase-2 dispatch in download_backends — A5 (plan #707)
Mirror verify_source_credential: download_backends.run_download is now the single download entry, so that module is the ONE registry of how each platform both downloads AND verifies — the seam that makes adding a platform a bounded job (write its adapter construction next to its verify). The native-ingester construction + campaign-id resolution moves out of download_service into download_backends._run_native_ingester. download_service.download_source drops its `if uses_native_ingester ... else gdl.download` branch and calls one `self._run_download(...)` (a thin delegate to run_download passing the service's gdl + sync sessionmaker). mode (tick/backfill/ recovery) is still chosen there from the backfill state machine and threaded through. Removed the now-unused PatreonIngester / resolve_campaign_id_for_source imports from download_service. Tests: the phase-2 stub seam moves from svc._run_patreon_ingester to svc._run_download (helper + the db-release test); the two native-construction tests repoint to download_backends.run_download (patching download_backends.resolve_campaign_id_for_source / PatreonIngester). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b211900390 |
refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707)
Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
697a86d31c |
fix(ingester): close #5 within-chunk live posts + #8 video transient retry
Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes both. #5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_ lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the badge didn't move during a chunk (up to ~14.5 min) and over-counted the re-walked resume page. The plan called for within-chunk live updates. Move ownership of _backfill_posts into the ingester: ingest_core writes a monotonic absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and once at the end, EXCLUDING the resumed page so it no longer inflates across chunks. download_service seeds posts_base from prior chunks and stops touching the key (the lifecycle now carries the ingester's committed value forward). #8 (per-media transient/permanent retry) covered only the plain-GET path (_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient (back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError) is permanent (yt-dlp already did its own network retries) → fail fast to the per-item/dead-letter path. Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist (test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite (test_download_service); video transient-retry + permanent-fail-fast (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9a2cd569c3 |
refactor(ingest): extract platform-agnostic Ingester core — roadmap #9 (plan #706)
Factor the native-ingest orchestration out of PatreonIngester into a reusable ingest_core.Ingester base, parametrized by client/downloader/ledger-models/ constraints/key/platform/error_base. PatreonIngester becomes a thin adapter: it resolves the Patreon client/downloader, wires the seen/dead-letter models + UNIQUE-constraint names + _ledger_key into super().__init__, and overrides _failure_result with the Patreon exception taxonomy. Behavior-preserving — no table rename, no migration; the public surface (PatreonIngester, _ledger_key, DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged. This is the strategic seam: SubscribeStar/etc. now migrate by writing a ~40-line adapter, not by re-implementing the tick/backfill/recovery walk, tiered skip, checkpoint, and dead-letter logic. run() moved to ingest_core, so the budget test's monotonic patch repoints to ingest_core.time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d592e0ca02 |
feat(patreon): within-pass transient retry for media GETs — #8 (was overstated as done)
Honest completion of roadmap #8. Previously a NON-429 media failure (a connection reset, a timeout, a truncated stream, a 5xx) was an immediate terminal "error" for the pass — only retried on the NEXT walk. Now _fetch_to_file retries TRANSIENT failures in-place with backoff (transport blips incl. mid-download, 429 honoring Retry-After, and 5xx; up to 3 tries), while PERMANENT failures (404 gone / 403 forbidden) fail fast straight to the error → dead-letter path — re-fetching them is pointless. This makes the transient-vs-permanent split explicit instead of leaning on the next-tick cycle. (#1's 429 backoff + #7's dead-letter covered most of #8's value; this is the missing in-pass transient piece I'd loosely marked "folded".) Tests: a connection blip / a 5xx is retried then succeeds; a 404 errors with NO retry; an exhausted transient becomes a terminal error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
7a872a3619 |
feat(patreon): dead-letter ledger for permanently-failing media — #705 step 2 (#7)
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux, persistently-corrupt bytes) used to re-error forever and re-burn chunks. New `patreon_failed_media` table (alembic 0038, chains 0037) records per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the ingester skips it on routine tick/backfill walks (tier-1.5, folded into the seen/skip predicate). Recovery BYPASSES it (the operator's "try everything again" re-attempts dead media). A clean download clears the row (recovered); errors/quarantines upsert-increment it. Surfaced as run_stats.dead_lettered_count. - New PatreonFailedMedia model + migration; ingester _dead_keys / _record_failures (on_conflict increment) / _clear_failures. - skip = seen | dead (empty in recovery); failures recorded post-fetch on short sessions (same pattern as the seen-ledger). Tests: a media erroring 3× is dead-lettered + skipped (no download attempt); recovery re-attempts a dead media and clears it on success; a clean download clears a sub-threshold failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4bb11ce7dc |
feat(patreon): incremental cursor checkpoint mid-walk — #705 step 1 (#6)
A worker SIGKILL (hard-time-limit backstop) mid-chunk lost the whole
chunk's walk — the cursor was only persisted at chunk boundaries by phase
3, so the next tick re-walked from the chunk start. Now the ingester
checkpoints _backfill_cursor to the DB at each page boundary (backfill/
recovery only) via an ATOMIC single-key UPDATE (config_overrides::jsonb →
jsonb_set('{_backfill_cursor}') → ::json), so it never clobbers operator
config or other backfill keys. On a crash the last mid-walk cursor
survives → the next chunk resumes near the crash, not the chunk start.
phase 3 still writes the final cursor (same value); this is the safety net.
Tests: a backfill walk leaves the last page's cursor in the DB (written by
the ingester, before any phase 3); a tick never checkpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
e42a86d995 |
test(patreon): fix #704 — quarantine status + budget-cut cursor assertions
Two test breaks from the structured-results change: - An existing downloader test pinned a corrupt file to status "error"; it's now the distinct "quarantined" status (the new behavior). Updated it + removed the duplicate I'd added. - The budget-cut ingester test asserted the checkpoint cursor was the last FULLY-processed page (CUR1); it's actually the page we were cut on (CUR2, entered + cursor emitted before the budget check), matching the prior parse_last_cursor(last) semantics. Corrected the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
b2e59e7e17 |
feat(subscriptions): live posts-processed progress on backfill/recovery — #704 step 2
The running badge only showed the chunk counter; now it shows posts walked — real walk progress. The ingester already reports posts_processed per chunk (step 1); the backfill lifecycle accumulates it into config_overrides._backfill_posts across chunks. SourceRecord exposes backfill_posts; start_backfill/start_recovery clear it (fresh walk); stop clears it too. SourceRow/SourceCard badge renders "Recovering · 45 posts" (falls back to "(N)" chunks before any posts are counted). Per-chunk accumulation (no mid-walk DB write) — simple and race-free; a small over-count from each chunk re-walking its resumed page is fine for a progress indicator. Tests: lifecycle accumulates posts_processed across chunks; start clears a prior _backfill_posts and the record exposes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e53f8959af |
feat(patreon): structured ingester results + quarantine surfacing — #704 step 1
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5b615b7ded |
feat(sources): pre-flight credential verify on backfill/recovery arm — #703 step 2
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
d6c15f4ea0 |
test(patreon): fake gdl needs real _rate_limit for native pacing — #703 step 1 fix
_run_patreon_ingester reads self.gdl._rate_limit for the native pacing config (max(0.5, rate_limit/4)); the MagicMock fake gdl broke the arithmetic. Give it real _rate_limit/_validate_files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3b2f7a41c3 |
feat(patreon): ingester rate-limit resilience — #703 step 1
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:
- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
`rate_limit`, read off self.gdl) as a pre-download sleep on real media
downloads only (skips don't pace); pace /api/posts page fetches with the
per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
RATE_LIMITED. Light 2-retry on a media-GET 429 too.
Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).
Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
218bfebb92 |
feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)
The credential Verify button still ran gallery-dl --simulate for Patreon after the cutover — testing the wrong path (and prone to the vanity "Failed to extract campaign ID" the native resolver fixes). Wire it to the native ingester, behind a DRY dispatch so callers never branch on platform. - services/download_backends.py (new): the ONE place that knows which platforms are native vs gallery-dl. `uses_native_ingester(platform)` is the shared predicate; `verify_source_credential(...)` is the uniform probe (same (ok|None, message) contract for both backends). As a platform migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download routing and verify switch together. - PatreonClient.verify_auth(campaign_id): one authenticated /api/posts fetch → True (valid) / False (401/403/HTML-login) / None (drift or network — inconclusive, not a credential verdict). - patreon_ingester.verify_patreon_credential(): resolve campaign id, then verify_auth — the verify counterpart to the download path. - patreon_resolver.resolve_campaign_id_for_source(): extracted the override / id:-URL / vanity resolution into ONE helper now shared by the download ingester and verify (download_service no longer carries its own copy + regex; −`import re`). - download_service: routes on uses_native_ingester() instead of inline `== "patreon"` (3 sites); uses the shared resolver. - api/credentials: calls verify_source_credential — no platform branch. Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/ vanity/none), the dispatch predicate, verify_patreon_credential glue, credentials endpoint proves Patreon uses the native path (gallery-dl verify asserted not-called); repointed the gallery-dl verify test to subscribestar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ec43e823e1 |
feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
682beafbc5 |
feat(patreon): drift detection + error categorization — build step 4 (plan #697)
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
96c30eba13 |
feat(patreon): phase-2 ingester integration — build step 3 (plan #697)
Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.
Three modes wired from config_overrides state:
- tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
contiguous already-have-it items.
- backfill: full-history time-boxed chunk, cursor checkpoint via
gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
lifecycle + parse_last_cursor verbatim).
- recovery: backfill that BYPASSES the tier-1 seen-ledger so
dropped-and-deleted near-dups get re-fetched and re-evaluated under
the current pHash threshold. Rides the #693 state machine via a
_backfill_bypass_seen flag, cleared on completion / stop.
The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.
Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
2ec7d86a3b |
feat(patreon): native media downloader — ingester build step 2b (plan #697)
PatreonDownloader.download_post: writes resolved MediaItems to gallery-dl's exact on-disk layout (<slug>/patreon/<date>_<id>_<title40>/<NN>_<file>) + a sidecar the importer's find_sidecar/parse_sidecar consume unchanged. Two-tier skip (injected seen predicate, then disk). Streamed GET (.part→rename) + file_validator quarantine; Mux/m3u8 video shells out to yt-dlp with Patreon Referer/Origin. Pure (no DB) — ledger + orchestration land in step 3. Unit tests stub the session + yt-dlp seams (no network/subprocess). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6222928746 |
feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id, filehash) — our own queryable replacement for gallery-dl's archive.sqlite3. Routine walks skip seen media; recovery mode bypasses the ledger. filehash is a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic 0037 (← 0036). Integration test covers dedup + savepoint recovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
1bdaa04aa2 |
test(patreon): fix self-contradictory attachment/postfile dedup assertion
The fixture gives the attachment and post_file the same filehash, so they correctly collapse to one item; the test asserted both survival and collapse. Rewrite to verify the cross-kind dedup (postfile kind covered by video test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
1c2dc7659a |
feat(patreon): native JSON-API client — ingester build step 1 (plan #697)
PatreonClient: cookie-auth requests session, /api/posts cursor pagination, JSON:API included flattening, per-post media extraction (images/image_large/ attachments/postfile/content) with filehash dedup, loud drift detection. Zero per-file HEADs — every media URL+file_name comes from the API. Not yet wired into download_service (later step). Pure-parsing unit tests + fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
96fffaff64 |
feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
593f65c9cc |
feat(download): cursor-paged Patreon backfill for large catalogs
Large Patreon creators (Anduo: weekly 50-120-image Reports back months = thousands of files) couldn't backfill: each run re-walked newest→oldest from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone exceeded the 1170s subprocess budget, so the run died during enumeration with 0 files written and NO forward progress — re-stranding every time (event #40411). Checkpoint gallery-dl's pagination cursor so each backfill window advances the frontier: - gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True for a fresh run); parse_last_cursor() pulls the last emitted 'Cursor: <token>' from stdout+stderr — survives a timed-out run since the TimeoutExpired path returns partial output. - download_service.py: phase2 stays in BACKFILL mode while a cursor is pending (even after the run budget drains) and threads resume_cursor; _apply_backfill_lifecycle() checkpoints the advancing cursor each non-completing run, completes on a clean rc=0 finish (walk reached bottom), and a stuck-guard clears the cursor after 2 non-advancing runs so a wedged walk can't re-strand forever. patreon-only (sole platform with a resumable cursor); other platforms keep the simple counter semantics. Cursor state lives in config_overrides JSON (patreon_campaign_id precedent) — no migration. Time-budget ladder (1170/1350/1500) unchanged. Plan #689. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5a6a95682d |
fix(cleanup): library scans survive navigation, reconnect on return
The transparency / single-color audit cards held the run + poll timer in local component state, so navigating away destroyed both and onMounted never reconnected — the Celery scan kept running and writing LibraryAuditRun, but the UI forgot it. Now each card, on mount, fetches its rule's latest run (GET /api/cleanup/audit?rule=<rule>&limit=1) and rehydrates: shows progress + resumes polling if still running, or shows the completed result (ready/applied/ error) so the operator can act on it after returning. Adds the ?rule= filter to the audit-history endpoint + cleanup store latestAuditForRule(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
91b0145bc8 |
feat(tags): 'Reset content tagging' admin action
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
79cd1234e2 |
feat(gallery): visual 'more like this' search (Phase 3 backend)
GalleryService.similar() ranks images by pgvector cosine distance to a source image's precomputed SigLIP embedding — no query-time ML inference. Composes with the Phase-1/2 scope filters (AND) but replaces the date sort (always nearest-first, bounded top-N, no cursor). Returns None for a missing source (→404), [] for a source with no embedding (video / pending ML); excludes self and NULL-embedding rows. New GET /api/gallery/similar?similar_to=<id>&limit=N. Image-detail payload gains has_embedding so the UI can hide the surface. Alembic 0036 adds an HNSW vector_cosine_ops index on siglip_embedding (1152<2000 dims) so the search is sub-50ms ANN instead of a full scan; one-time ~30-60s build over existing embeddings on deploy. Shared _gallery_images/_image_json helpers de-dup the scroll/similar builders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c802b26406 |
perf(ci): reuse a session-scoped engine for the per-test DB reset
The autouse integration teardown created a fresh SQLAlchemy engine + Postgres connection for EVERY test, then disposed it — --durations showed the 15 slowest ops in both long shards were all ~1.5-2s teardowns (the connect+SCRAM handshake, not test logic). Hoist the truncate engine to a session-scoped, pool_pre_ping'd fixture so the pooled connection is reused across teardowns; the TRUNCATE+restore still runs per test, so isolation is unchanged. Lazy create_engine means the no-DB unit job instantiates but never connects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9fe534139a |
feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)
Extend the composable gallery filter with platform / untagged / no_artist / date_from / date_to, AND-composed with the existing tag/artist/media/sort params and threaded through scroll, timeline, and jump_cursor. Add GalleryService.facets() + GET /api/gallery/facets returning live counts scoped to the current filter with per-group minus-self semantics: platform counts (COUNT(DISTINCT image) incl. a null unsourced bucket), curation-flag counts (untagged / no_artist), and effective_date min/max bounds. The UNSOURCED_PLATFORM sentinel makes filesystem-imported content reachable via the platform facet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
914033db29 |
feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the periodic prune/backfill/recovery tasks churn dead tuples faster than autovacuum always keeps up — so explicit maintenance earns its keep here. - tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn). Scheduled weekly via Beat; also operator-triggerable. - _sync_engine.get_sync_engine(): expose the process engine for the autocommit connection. - GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct + last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a black box. - POST /api/admin/maintenance/vacuum: enqueue the task on demand. Tests: vacuum task runs + reports tables; db-stats shape; trigger queues. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3f30327fa5 |
feat(gallery): composable scroll filter (multi-tag AND, media, sort)
Phase 1 backend for the gallery filter bar. Extends scroll/timeline/jump
from a single mutually-exclusive filter to a composable one:
- tag_ids: image must carry ALL of them (one correlated EXISTS per tag —
AND, no row multiplication), replacing the single-tag JOIN.
- artist_id composes with tags; media_type ('image'|'video') narrows by
mime; post_id stays the exclusive post-detail path.
- sort ('newest'|'oldest') flips the effective_date/id cursor comparison
and ordering; the cursor value is unchanged (direction comes from the
request). jump_cursor honors sort too.
- Shared _apply_scope helper applied across scroll/timeline/jump so the
timeline sidebar reflects the filtered set. API _parse_filters parses
tag_id (comma list), artist_id, media, sort.
Tests: multi-tag AND, media filter, sort reversal (service + API);
post_id-excludes-others; single tag_id back-compat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
4f9464d215 |
feat(gallery,tags): clear active filters
Two gaps where a filter couldn't be removed:
- Gallery: a tag_id filter (from clicking a tag) had no indicator or clear
control — only post_id did (PostInfoHeader). Add an "Tag: <name> ✕" chip
that clears the filter by dropping tag_id from the URL. New lightweight
GET /api/tags/<id> resolves the name; the store fetches it on filter set.
- Tags view: the kind chip-group used mandatory="false" — a STRING ("false"
is truthy in JS), which made the group mandatory so the active kind chip
couldn't be deselected. Fixed to :mandatory="false" so the filter clears.
Tests: GET /tags/<id> shape + 404; gallery store resolves filterTagName.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
d9ab6e15c6 |
feat(tags): edit a character tag's fandom (backend)
No way existed to change which fandom a character tag belongs to after creation — PATCH /tags/<id> only renamed. - TagService.set_fandom(tag_id, fandom_id, merge=False): set / change / clear (fandom_id=None) a character's fandom, with the same validation as find_or_create. On a name collision in the target fandom it raises TagMergeConflict (→ 409, same shape as rename); merge=True resolves it by merging this tag INTO the existing character. - Extract _do_merge(source, target) from merge() so set_fandom can perform the deliberate CROSS-fandom merge the public merge() validation forbids. - PATCH /tags/<id> now accepts optional fandom_id (+ merge flag) alongside name, and returns fandom_id. Tests: set/change/clear, non-character + bad-ref rejection, collision raises, merge resolves; API set/clear + collision→merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e05e0b9f37 |
perf(gallery): materialize indexed effective_date sort key
The gallery cursored on COALESCE(post.post_date, image_record.created_at) across the Post outer join — an expression spanning two tables that no index can serve, so every /scroll sorted a large slice of the library (and the old frontend fired ten serially). Materialize it: - image_record.effective_date column + ix_image_record_effective_date (effective_date DESC, id DESC); alembic 0035 backfills COALESCE(primary post's post_date, created_at) for existing rows. - gallery_service._effective_date_col() now returns the column, so scroll / timeline / jump / neighbors all order off the index instead of re-deriving the COALESCE. _neighbors reads record.effective_date directly (drops an extra Post lookup). - importer._apply_sidecar maintains it: when a primary post with a date is linked, effective_date = post.post_date; plain inserts keep the created_at-equivalent server default. Tests: sidecar import asserts effective_date == post.post_date; gallery ordering/timeline/jump test seeds set effective_date alongside created_at. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
576e16d14d |
fix(download): release DB connections across the gallery-dl subprocess
Backfill events were STILL stranding empty after the timeout-ladder fix.
Worker logs showed the salvage path working ("Download timeout for
anduo/patreon after 1170.0s (18 files written)") but then:
Retry in 3s: DBAPIError(ConnectionDoesNotExistError: connection was
closed in the middle of operation)
...succeeded in 0.149s <- in-flight guard no-op
Root cause: DownloadService held the async + sync DB connections checked
out across the entire (≤19.5-min backfill) gallery-dl subprocess. The
server reaps the idle connection, so phase 3's first query hits a dead
socket. That DBAPIError trips download_source's autoretry_for, the retry
re-enters _phase1_setup, sees the event still 'running', returns
in_flight and no-ops — leaving the event to be stranded empty by the
recovery sweep. pool_pre_ping was already on both engines but can't help
a *held* connection (it only validates on pool checkout).
Fix:
- DownloadService.download_source closes the async + sync sessions after
phase 1, before the subprocess, so phase 3 re-acquires a live
connection (matches the class's "Phase 2 — no DB connection" docstring).
- The per-task async engine switches to NullPool so phase 3 always opens
a fresh connection rather than a pooled one the server may have reaped.
Tests: assert connections are released before gdl.download runs and the
event still finalizes; assert the task engine uses NullPool. Also fixes a
stale 1800s->1170s comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
9cb24c9e1b |
style(test): fix ruff I001 import order in download task test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6590dcdb39 |
fix(download): salvage soft-time-limit kills + fix timeout ladder
Backfill downloads stranded with empty logs + a generic "stranded by recovery sweep" error. Root cause: the backfill gallery-dl subprocess timeout (1170s) exceeded download_source's Celery soft_time_limit (900s), so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The TimeoutExpired path (which captures partial stdout/stderr and finalizes the event) never ran, the event was left 'running', and phase 3 never decremented backfill_runs_remaining — so the source re-ran and re-stranded every tick (Anduo #39912). Two layers: 1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both subprocess budgets (870 tick / 1170 backfill) sit below the soft limit with phase-3 persist headroom. Promote to module constants and guard the invariant with a test. 2. Catch SoftTimeLimitExceeded in download_source and finalize the in-flight event with a real reason, mirror phase-3 source-health, and decrement backfill so a chronically-slow source self-heals to tick mode. The existing celery_signals handler only covered TaskRun, not DownloadEvent — that was the gap. Updates stale 900/1200 references in gallery_dl.py + maintenance.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |