Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native client treated a gallery image without `file_name` as schema drift
and raised "Patreon API changed — ingester needs update", failing the whole walk
(operator-flagged 2026-06-07: BlenderKnight post 73665615, kind=images). But the
resource had a valid URL, and the code already derives a filename from the URL
basename right below the raise — the same fallback gallery-dl uses. Patreon
legitimately serves some images without file_name, so this isn't drift.
Drop the require_file_name gate from _media_item: file_name is now optional for
every kind (images/attachments/postfile), falling back to the URL basename.
Genuine drift still raises — no resolvable URL, or a media id referenced by a
relationship but absent from `included`. Test updated to assert the fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mid-post time-box check (619e771) reads time.monotonic() once more per post,
so test_backfill_budget_cut_returns_partial_with_progress's discrete tick
sequence shifted — the >budget tick (200) landed on post1's first-item check
instead of post2's gate, cutting post1 to 0 files. Add the extra tick (20, still
under budget) so post1's item downloads, matching production where the gate and
the first should_stop are microseconds apart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).
download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.
Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/posts/mimic-in-dungeon-158372536 is a
single-post permalink, not a creator page — the resolver grabbed "posts" as the
vanity and failed (operator-flagged 2026-06-07). Add a resolution path: extract
the trailing post id and follow it to the owning campaign via the Patreon post
API (/api/posts/<id>?include=campaign), so pasting any post URL subscribes to
that creator's whole feed. `posts/` is excluded from the vanity regex so it
can't masquerade as a creator slug.
Resolution order is now: cached override → id: URL → /posts/<id> → vanity
(campaigns API + creator-page scrape). Tests cover the post→campaign resolve
and that /posts/ URLs aren't treated as vanities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new enabled source is armed for run-until-done backfill (#693) but would sit
idle until the next scheduler tick (~60s). create_source now enqueues the first
walk right away (pending DownloadEvent + download_source.delay), skipping only
when the platform is in a rate-limit cooldown (the scheduler picks it up when
that clears). Disabled sources still don't dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/cw/Atole resolved vanity='cw' — the
vanity regex only skipped a /c/ prefix, so Patreon's current /cw/ ("creator
workspace") form fell through to the bare-vanity branch and captured the prefix
instead of the slug. Every /cw/ source then failed campaign-id resolution
(API + page-scrape both looked up "cw"). Operator-confirmed 2026-06-07 via the
new error text: source_url='.../cw/Atole'; vanity='cw'.
Add cw/ to the optional prefix group (ordered before c/ so the longer prefix
wins), and have the creator-page fallback try the /cw/ form too. Test covers
bare / c/ / cw/ extraction and that id: URLs stay non-vanity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon's /api/campaigns?filter[vanity]= lookup returns empty data for creators
that plainly exist (operator-flagged 2026-06-06 — Atole etc. erroring at the
resolve step). gallery-dl never used that endpoint; it pulls the campaign id out
of the creator page's bootstrap JSON. Add the same as a fallback: when the API
misses, GET the creator page (bare + /c/ vanity paths) and scrape the first
campaign id from any known embedding ("id":"…","type":"campaign" /
"campaign":{"data":{"id" / /api/campaigns/<id> / "campaign_id"). API is still
tried first (cheap, structured); the page scrape only runs on a miss.
Tests: API-empty → page-scrape fallback resolves; _scrape_campaign_id pattern
coverage. Existing API-path tests unchanged (happy paths short-circuit before
the fallback; failure paths hit the guarded scrape and still return None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name now only capitalizes the first letter of each word and
leaves the rest of the word untouched (was lowercasing the tail, which turned
DC→Dc / NSFW→Nsfw). This matches ml/tag_name._title_word, so a Camie-suggested
tag keeps the exact casing the suggestion UI showed when it round-trips through
POST /api/tags on Accept — addressing "auto-suggested tags must obey
capitalization" and "don't mangle acronyms" in one rule.
Trade-off (operator-chosen): all-caps input no longer folds to Title Case, so
case-variant merging in #714 still folds the dominant lowercase-vs-Title case
but leaves all-caps stylizations distinct (protecting acronyms wins). Tests
updated + a new test documenting acronym preservation / non-folding.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps
whatever casing it was created with. This adds a maintenance action that
Title-Cases every existing tag (collapsing whitespace) and merges
case/whitespace-variant duplicates into one.
Backend:
- tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by
(kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor
(prefer an already-canonical member → no rename/self-alias; else the
best-connected tag → fewest FK repoints; else lowest id), merges the variants
INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/
aliases/series_page repoints + protective ML aliases), then renames the
survivor to canonical. Losers are deleted before the rename so there's no
transient unique-index clash; commits per group and isolates failures per
group. Idempotent — an already-canonical lone tag is a no-op.
- normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async
engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i.
- POST /api/admin/tags/normalize: dry_run=true returns a projection inline
(group/collision/rename counts + sample); dry_run=false enqueues the task.
Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup
tab) — preview → apply (polls the activity dashboard to terminal status),
behind a back-up-first warning. admin store gains normalizeTags().
Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag
dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept
separate, ML-known loser keeps a protective alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in
the POST /api/tags handler so operator-entered tags get clean display casing.
It is NOT applied in the shared find_or_create / rename paths — those are used
by the ML tagger and allowlist matching, which must preserve the booru
vocabulary's original casing (Title-Casing it broke apply_allowlist matching).
find_or_create / rename keep case-insensitive lookup + clash detection so a
differently-cased entry dedups onto the existing tag instead of forking.
Tests updated to expect Title-Cased create output (sunset→Sunset,
character:Saber→Character:saber, http://example.com→Http://example.com) and a
dedicated normalize_tag_name unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).
normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.
Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.
Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.
Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").
Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.
Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the walk lets Stop interrupt a live backfill chunk instead of letting it
run to its ~14.5-min time-box. ingest_core.run now polls _backfill_state at each
page boundary (a short SELECT, never held across the walk) and bails with PARTIAL
when an operator Stop has popped it. Latched on the first observed "running"
state so a run invoked without one (unit test / stale call) never self-cancels.
Progress is already checkpointed per-page, so a restart resumes from the cursor;
Stop clears it for a clean reset. No UI change — the existing Stop button now
just takes effect immediately.
Tests: _still_running reads the state; a latched run bails PARTIAL at the next
boundary when the state disappears (only the pre-cancel post ran).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_one_failure_isolated's _BoomSession overrode get() without the headers
kwarg _fetch_to_file now passes (B5 Range resume), so the call TypeError'd and
both items errored. Add headers=None to match the base fake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the media fetch means a mid-download transport cut no longer refetches
from zero. _fetch_to_file now resumes: on a transient retry, if bytes already
landed in the .part, it requests Range: bytes=<have>- and appends on a 206;
falls back to a clean truncate-and-restart if the server ignores Range (200) or
the range is past EOF (416). The .part staging means a non-range server never
corrupts the output — worst case is the old behavior (refetch from zero).
Tests: mid-stream cut resumes from the offset (asserts the Range header);
a Range-ignoring server refetches clean (no double-write). Test session fakes
updated to accept the new headers kwarg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the native client means we see the 429 Retry-After header — previously
discarded. PatreonAPIError now carries `retry_after`; on a PERSISTENT page-fetch
429 the client attaches the server's raw Retry-After seconds. New
DownloadResult.retry_after_seconds; patreon_ingester._failure_result sets it on
RATE_LIMITED. download_service._update_source_health passes it to
set_platform_cooldown as `seconds=`, clamped to [60, 3600] (a tiny hint can't
leave the platform effectively un-cooled; a huge one can't strand it for hours);
no hint → the flat 900s default. So a rate-limited platform cools for as long as
the server actually asks, not a fixed guess.
Tests: terminal 429 surfaces retry_after (test_patreon_client); cooldown honors
+ clamps the hint, falls back to default when absent (test_download_service).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror verify_source_credential: download_backends.run_download is now the single
download entry, so that module is the ONE registry of how each platform both
downloads AND verifies — the seam that makes adding a platform a bounded job
(write its adapter construction next to its verify). The native-ingester
construction + campaign-id resolution moves out of download_service into
download_backends._run_native_ingester.
download_service.download_source drops its `if uses_native_ingester ... else
gdl.download` branch and calls one `self._run_download(...)` (a thin delegate to
run_download passing the service's gdl + sync sessionmaker). mode (tick/backfill/
recovery) is still chosen there from the backfill state machine and threaded
through. Removed the now-unused PatreonIngester / resolve_campaign_id_for_source
imports from download_service.
Tests: the phase-2 stub seam moves from svc._run_patreon_ingester to
svc._run_download (helper + the db-release test); the two native-construction
tests repoint to download_backends.run_download (patching
download_backends.resolve_campaign_id_for_source / PatreonIngester).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates duplication that owning the native ingester left against the still-
live gallery-dl path, and fixes a parity gap the duplication hid.
A1 — shared quarantine: extract file_validator.quarantine_file (move to
_quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar).
gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both
call it. PARITY FIX: the native path now writes the provenance sidecar it
previously skipped — threads the media url through for source_url.
A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats
key set; gallery_dl._compute_run_stats and ingest_core both build through it so
the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0).
A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for
the Path call sites + memory pointer) and patreon_client both use it. Closes the
double-impl of the URL-encoded-basename ext gotcha.
A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level
truncate_log/extract_errors_warnings; download_service calls them directly
instead of reaching through self.gdl for native-result log shaping. The
staticmethods stay as thin delegators for existing callers/tests.
Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine
writes a .quarantine.json (test_patreon_downloader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.
#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).
#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.
Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Factor the native-ingest orchestration out of PatreonIngester into a reusable
ingest_core.Ingester base, parametrized by client/downloader/ledger-models/
constraints/key/platform/error_base. PatreonIngester becomes a thin adapter:
it resolves the Patreon client/downloader, wires the seen/dead-letter models +
UNIQUE-constraint names + _ledger_key into super().__init__, and overrides
_failure_result with the Patreon exception taxonomy. Behavior-preserving — no
table rename, no migration; the public surface (PatreonIngester, _ledger_key,
DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged.
This is the strategic seam: SubscribeStar/etc. now migrate by writing a
~40-line adapter, not by re-implementing the tick/backfill/recovery walk,
tiered skip, checkpoint, and dead-letter logic.
run() moved to ingest_core, so the budget test's monotonic patch repoints to
ingest_core.time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Honest completion of roadmap #8. Previously a NON-429 media failure (a
connection reset, a timeout, a truncated stream, a 5xx) was an immediate
terminal "error" for the pass — only retried on the NEXT walk. Now
_fetch_to_file retries TRANSIENT failures in-place with backoff (transport
blips incl. mid-download, 429 honoring Retry-After, and 5xx; up to 3 tries),
while PERMANENT failures (404 gone / 403 forbidden) fail fast straight to the
error → dead-letter path — re-fetching them is pointless. This makes the
transient-vs-permanent split explicit instead of leaning on the next-tick
cycle. (#1's 429 backoff + #7's dead-letter covered most of #8's value; this
is the missing in-pass transient piece I'd loosely marked "folded".)
Tests: a connection blip / a 5xx is retried then succeeds; a 404 errors with
NO retry; an exhausted transient becomes a terminal error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux,
persistently-corrupt bytes) used to re-error forever and re-burn chunks.
New `patreon_failed_media` table (alembic 0038, chains 0037) records
per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the
ingester skips it on routine tick/backfill walks (tier-1.5, folded into the
seen/skip predicate). Recovery BYPASSES it (the operator's "try everything
again" re-attempts dead media). A clean download clears the row (recovered);
errors/quarantines upsert-increment it. Surfaced as
run_stats.dead_lettered_count.
- New PatreonFailedMedia model + migration; ingester _dead_keys /
_record_failures (on_conflict increment) / _clear_failures.
- skip = seen | dead (empty in recovery); failures recorded post-fetch on
short sessions (same pattern as the seen-ledger).
Tests: a media erroring 3× is dead-lettered + skipped (no download attempt);
recovery re-attempts a dead media and clears it on success; a clean download
clears a sub-threshold failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A worker SIGKILL (hard-time-limit backstop) mid-chunk lost the whole
chunk's walk — the cursor was only persisted at chunk boundaries by phase
3, so the next tick re-walked from the chunk start. Now the ingester
checkpoints _backfill_cursor to the DB at each page boundary (backfill/
recovery only) via an ATOMIC single-key UPDATE (config_overrides::jsonb →
jsonb_set('{_backfill_cursor}') → ::json), so it never clobbers operator
config or other backfill keys. On a crash the last mid-walk cursor
survives → the next chunk resumes near the crash, not the chunk start.
phase 3 still writes the final cursor (same value); this is the safety net.
Tests: a backfill walk leaves the last page's cursor in the DB (written by
the ingester, before any phase 3); a tick never checkpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two test breaks from the structured-results change:
- An existing downloader test pinned a corrupt file to status "error";
it's now the distinct "quarantined" status (the new behavior). Updated
it + removed the duplicate I'd added.
- The budget-cut ingester test asserted the checkpoint cursor was the last
FULLY-processed page (CUR1); it's actually the page we were cut on (CUR2,
entered + cursor emitted before the budget check), matching the prior
parse_last_cursor(last) semantics. Corrected the assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The running badge only showed the chunk counter; now it shows posts walked
— real walk progress. The ingester already reports posts_processed per
chunk (step 1); the backfill lifecycle accumulates it into
config_overrides._backfill_posts across chunks. SourceRecord exposes
backfill_posts; start_backfill/start_recovery clear it (fresh walk); stop
clears it too. SourceRow/SourceCard badge renders "Recovering · 45 posts"
(falls back to "(N)" chunks before any posts are counted).
Per-chunk accumulation (no mid-walk DB write) — simple and race-free; a
small over-count from each chunk re-walking its resumed page is fine for a
progress indicator.
Tests: lifecycle accumulates posts_processed across chunks; start clears a
prior _backfill_posts and the record exposes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and
phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and
quarantine stats blank. We own the ingester, so it now RETURNS structured
data and phase 3 reads it directly.
- DownloadResult gains run_stats/cursor/posts_processed (None/0 on the
gallery-dl path, which keeps the text route).
- Ingester builds real run_stats from per-media outcome counts, sets the
checkpoint cursor structurally (no fake `Cursor:` stdout), and counts
posts processed. download_service phase 3 uses dl_result.run_stats when
present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint
dl_result.cursor instead of parse_last_cursor(stdout).
- #4 quarantine: PatreonDownloader reports a distinct "quarantined"
MediaOutcome (with the _quarantine dest); the ingester surfaces a real
files_quarantined + quarantined_paths + run_stats.quarantined_count
(was hardcoded 0). Quarantined media isn't written or marked seen.
- Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`)
removed from gallery_dl — the structured cursor replaced the scrape.
Tests: ingester result carries real run_stats/cursor/posts_processed +
quarantine counts; downloader quarantines an invalid file as "quarantined";
backfill cursor tests pass cursor= structurally; dropped the
parse_last_cursor tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_run_patreon_ingester reads self.gdl._rate_limit for the native pacing
config (max(0.5, rate_limit/4)); the MagicMock fake gdl broke the
arithmetic. Give it real _rate_limit/_validate_files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:
- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
`rate_limit`, read off self.gdl) as a pre-download sleep on real media
downloads only (skips don't pace); pace /api/posts page fetches with the
per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
RATE_LIMITED. Light 2-retry on a media-GET 429 too.
Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).
Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The credential Verify button still ran gallery-dl --simulate for Patreon
after the cutover — testing the wrong path (and prone to the vanity
"Failed to extract campaign ID" the native resolver fixes). Wire it to the
native ingester, behind a DRY dispatch so callers never branch on platform.
- services/download_backends.py (new): the ONE place that knows which
platforms are native vs gallery-dl. `uses_native_ingester(platform)` is
the shared predicate; `verify_source_credential(...)` is the uniform
probe (same (ok|None, message) contract for both backends). As a platform
migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download
routing and verify switch together.
- PatreonClient.verify_auth(campaign_id): one authenticated /api/posts
fetch → True (valid) / False (401/403/HTML-login) / None (drift or
network — inconclusive, not a credential verdict).
- patreon_ingester.verify_patreon_credential(): resolve campaign id, then
verify_auth — the verify counterpart to the download path.
- patreon_resolver.resolve_campaign_id_for_source(): extracted the
override / id:-URL / vanity resolution into ONE helper now shared by the
download ingester and verify (download_service no longer carries its own
copy + regex; −`import re`).
- download_service: routes on uses_native_ingester() instead of inline
`== "patreon"` (3 sites); uses the shared resolver.
- api/credentials: calls verify_source_credential — no platform branch.
Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/
vanity/none), the dispatch predicate, verify_patreon_credential glue,
credentials endpoint proves Patreon uses the native path (gallery-dl verify
asserted not-called); repointed the gallery-dl verify test to subscribestar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final step of the native Patreon ingester: a first-class Recovery action,
and removal of the now-dead gallery-dl Patreon path.
Recovery (rules #23/#24/#27 — full product, with UI):
- source_service.start_recovery arms the #693 backfill state machine PLUS
`_backfill_bypass_seen`, flipping download mode to recovery (bypass the
seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate
under the current pHash threshold). Stop via the shared stop_backfill.
- SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill
gains action="recover".
- Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow
+ SourceCard; the running badge labels "Recovering (N)" vs "Backfilling
(N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource +
SubscriptionsTab onRecover.
Cutover (rule #22 — no legacy):
- gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon
files/cursor branch in _build_config_for_source, and the patreon/Mux
yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the
other platforms' yt-dlp fetches; native ingester owns it now).
- download_service: removed the dead campaign-id-retry helpers
(_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and
_effective_url. Vanity→campaign resolution + resume_cursor + the
cursor/PARTIAL lifecycle stay — they serve the native ingester.
Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp
Referer, resume-cursor); repointed the generic skip-value config tests to
subscribestar; added start_recovery + recover-endpoint coverage
(backfill_bypass_seen). gallery-dl stays for the other 5 platforms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.
Three modes wired from config_overrides state:
- tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
contiguous already-have-it items.
- backfill: full-history time-boxed chunk, cursor checkpoint via
gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
lifecycle + parse_last_cursor verbatim).
- recovery: backfill that BYPASSES the tier-1 seen-ledger so
dropped-and-deleted near-dups get re-fetched and re-evaluated under
the current pHash threshold. Rides the #693 state machine via a
_backfill_bypass_seen flag, cleared on completion / stop.
The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.
Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PatreonDownloader.download_post: writes resolved MediaItems to gallery-dl's
exact on-disk layout (<slug>/patreon/<date>_<id>_<title40>/<NN>_<file>) +
a sidecar the importer's find_sidecar/parse_sidecar consume unchanged. Two-tier
skip (injected seen predicate, then disk). Streamed GET (.part→rename) +
file_validator quarantine; Mux/m3u8 video shells out to yt-dlp with Patreon
Referer/Origin. Pure (no DB) — ledger + orchestration land in step 3. Unit
tests stub the session + yt-dlp seams (no network/subprocess).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id,
filehash) — our own queryable replacement for gallery-dl's archive.sqlite3.
Routine walks skip seen media; recovery mode bypasses the ledger. filehash is
a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic
0037 (← 0036). Integration test covers dedup + savepoint recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fixture gives the attachment and post_file the same filehash, so they
correctly collapse to one item; the test asserted both survival and collapse.
Rewrite to verify the cross-kind dedup (postfile kind covered by video test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PatreonClient: cookie-auth requests session, /api/posts cursor pagination,
JSON:API included flattening, per-post media extraction (images/image_large/
attachments/postfile/content) with filehash dedup, loud drift detection.
Zero per-file HEADs — every media URL+file_name comes from the API. Not yet
wired into download_service (later step). Pure-parsing unit tests + fixture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).
- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
far under the 1350 soft limit. Hitting it = normal chunk boundary (the
TimeoutExpired path already captures partial output + the cursor), not a
near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
(running/complete/stalled). A running backfill auto-continues in chunks
across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
pause a pathological walk as 'stalled'. Replaces the N-runs counter
(backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
source dict exposes backfill_state + backfill_chunks.
Frontend (Start/Stop control + state badge) lands in the next push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Large Patreon creators (Anduo: weekly 50-120-image Reports back months =
thousands of files) couldn't backfill: each run re-walked newest→oldest
from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone
exceeded the 1170s subprocess budget, so the run died during enumeration
with 0 files written and NO forward progress — re-stranding every time
(event #40411).
Checkpoint gallery-dl's pagination cursor so each backfill window advances
the frontier:
- gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets
extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True
for a fresh run); parse_last_cursor() pulls the last emitted
'Cursor: <token>' from stdout+stderr — survives a timed-out run since the
TimeoutExpired path returns partial output.
- download_service.py: phase2 stays in BACKFILL mode while a cursor is
pending (even after the run budget drains) and threads resume_cursor;
_apply_backfill_lifecycle() checkpoints the advancing cursor each
non-completing run, completes on a clean rc=0 finish (walk reached
bottom), and a stuck-guard clears the cursor after 2 non-advancing runs
so a wedged walk can't re-strand forever.
patreon-only (sole platform with a resumable cursor); other platforms keep
the simple counter semantics. Cursor state lives in config_overrides JSON
(patreon_campaign_id precedent) — no migration. Time-budget ladder
(1170/1350/1500) unchanged.
Plan #689.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transparency / single-color audit cards held the run + poll timer in
local component state, so navigating away destroyed both and onMounted never
reconnected — the Celery scan kept running and writing LibraryAuditRun, but
the UI forgot it. Now each card, on mount, fetches its rule's latest run
(GET /api/cleanup/audit?rule=<rule>&limit=1) and rehydrates: shows progress +
resumes polling if still running, or shows the completed result (ready/applied/
error) so the operator can act on it after returning. Adds the ?rule= filter
to the audit-history endpoint + cleanup store latestAuditForRule().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GalleryService.similar() ranks images by pgvector cosine distance to a source
image's precomputed SigLIP embedding — no query-time ML inference. Composes
with the Phase-1/2 scope filters (AND) but replaces the date sort (always
nearest-first, bounded top-N, no cursor). Returns None for a missing source
(→404), [] for a source with no embedding (video / pending ML); excludes self
and NULL-embedding rows. New GET /api/gallery/similar?similar_to=<id>&limit=N.
Image-detail payload gains has_embedding so the UI can hide the surface.
Alembic 0036 adds an HNSW vector_cosine_ops index on siglip_embedding (1152<2000
dims) so the search is sub-50ms ANN instead of a full scan; one-time ~30-60s
build over existing embeddings on deploy. Shared _gallery_images/_image_json
helpers de-dup the scroll/similar builders.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The autouse integration teardown created a fresh SQLAlchemy engine + Postgres
connection for EVERY test, then disposed it — --durations showed the 15
slowest ops in both long shards were all ~1.5-2s teardowns (the connect+SCRAM
handshake, not test logic). Hoist the truncate engine to a session-scoped,
pool_pre_ping'd fixture so the pooled connection is reused across teardowns;
the TRUNCATE+restore still runs per test, so isolation is unchanged. Lazy
create_engine means the no-DB unit job instantiates but never connects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen