31 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	e5bdcd2596	feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428) ... - Recover and Recapture show on every native source. The menu gated them on a copied platform list ('patreon', 'subscribestar') that went stale when Discord moved over. Sources now carry `native_ingester` from the backend's own predicate. - A running backfill is due on every scheduler tick. Nothing queued a backfill's next chunk: each one waited for the source's regular interval, so an armed backfill sat idle until the next check (8h at the default) and a five-chunk walk took most of two days. The in-flight guard and the platform lock keep one chunk at a time. A failing source falls back to its backoff, and a stalled or out-of-budget walk stops being due. It also runs when the artist has auto-check off, since the operator started it by hand. - gallery-dl no longer carries Discord: its naming constants, platform defaults, sidecar-mirroring postprocessor and token injection are gone. The naming test moves to the native downloader and still renders against the real gallery-dl sidecar fixture. That is the guard that the files gallery-dl wrote are found on disk rather than fetched again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 23:14:48 -04:00
bvandeusenandClaude Opus 5.5	da2091a875	fix: a download is marked seen only after it is imported ... A run killed between download and import left its files on disk and in the seen-ledger but never in the library, and every later walk trusted the ledger. TamadaHeijun's 12PCG post lost 7 of 13 images this way (stranded run 90402), which read as the duplicates filter. The ingester now hands phase 3 a mark_seen_after_import hook, called after the import loop. A file on disk with no ImageRecord at its path is fed to import, not reconciled into the ledger. Recapture reaches files already orphaned, because it looks past the ledger to the disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:13:12 -04:00
bvandeusenandClaude Opus 5	9b82a95b7e	feat: Settings can re-download the Discord images the None naming broke (3999) ... The operator chose a clean re-download over relinking in place. The ~1,600 broken files can't be relinked reliably: their message ids are gone, and their sidecars collided. Settings → Maintenance → "Repair Discord downloads" previews, then applies: - Deletes every image whose path is `…/discord/None/<8 digits>_None_…`. Both the folder and the name are required, because that pair is only what the bug produced. It reuses cleanup_service.delete_images for the record and file deletes. - Sweeps the leftover collided sidecars from those folders and removes the emptied folders. - Only then clears gallery-dl's archive rows `discord%`, excluding `discordasset_%`. Upstream keys message attachments as `discord{message_id}_{num}`. Since the broken files lost their message ids, per-source forgetting is impossible. Every pre-fix Discord download is broken, and files fetched after the fix still exist on disk, so gallery-dl's `skip` won't re-fetch them. - Arms a fresh backfill on every Discord source. The apply defaults to preview at both the route and the task, runs on maintenance_long, and is never on a beat. The card uses the confirm-dialog pattern of AttachmentReclaimCard. Supporting refactors, with no behaviour change: - gallery_dl.archive_path() is the single definition of the archive location. - source_service.arm_backfill() is the mutation start_backfill already did, now shared with the sync repair. Tests (tests/test_discord_repair.py): - The archive clear leaves other platforms and Discord assets alone, and counting mutates nothing. - Case-twin artist folders are both found. - The folder sweep works. - An integration run shows only the broken image goes. A correctly named Discord file and a `None` folder under Patreon survive, and only Discord sources are re-armed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 20:52:32 -04:00
bvandeusenandClaude Opus 5	dc840feec7	fix: Discord downloads land as None/<date>_None_<name> with no post — name them from the keys gallery-dl really emits ... Operator report: a Discord source showed the wrong post time. Listing the real downloads found every Discord folder on the instance (7 artist folders, ~1,600 media) in a directory named `None`, as `<date>_None_<original name>`, next to sidecars named `<original name>.json`. The real sidecar settles why: - gallery-dl's discord extractor emits `channel` as a plain string, the message as `message_id`, and the attachment position as `num`. It has no `id` key. - The patterns asked for `{channel[name]}` and `{id}`. gallery-dl renders a missing field as "None" and carries on. - The sidecar was named `{filename}.json`, the attachment's ORIGINAL name. find_sidecar never pairs that with `<date>_None_<name>.png`, so no Discord file ever got a Post or a post date. The card fell back to downloaded_at. - Every `image.png` in a channel also overwrote the same `image.json`. Fix (gallery_dl.py): - The directory is `{channel}`. - The filename is `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, unique per attachment. - A discord-scoped metadata postprocessor names the sidecar exactly like the media minus its extension, so find_sidecar's first candidate matches. A per-source filename override re-derives the sidecar name. save_metadata=False drops it. Guard (tests/test_gallery_dl_naming.py) renders the patterns through Python's formatter against a sanitized copy of the real sidecar (same keys and types, invented values). A missing key or a subscript into a string raises, which is the loud failure gallery-dl doesn't give. A positive control shows both shipped patterns fail it. Existing broken downloads are NOT repaired by this. gallery-dl's archive already records them, so a re-run skips them, and their collided sidecars no longer describe them. That repair is a separate, destructive step for the operator to decide on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 20:01:11 -04:00
bvandeusenandClaude Opus 5	7751715b83	feat: a paywalled creator is no longer indistinguishable from a silent one (milestone 387 step A2) ... A2 of milestone 387. A1 made the gated-post count true; this makes it mean something. A native walk that reached the bottom returned `error_type=None` whether the creator had posted nothing or every post sat behind a tier we don't hold. `source.error_type` stayed NULL and the source read healthy and quiet. gallery-dl has classified this as TIER_LIMITED since the paywall-as-"needs attention" complaint; the native path never did. Two things had to move that the plan didn't foresee, both found by reading the consumers rather than by testing afterwards: The backfill lifecycle's completion test required `error_type is None`. Returning TIER_LIMITED naively would have dropped a fully-paywalled backfill into the not-finished branch — zero downloads means no progress, two strikes marks it "stalled" — so the creator we can see least would become the one we re-walk most. `walk_completed` now admits informational classes. `_update_source_health` only stamps `error_type` on status "error" and CLEARS it on "ok". Since TIER_LIMITED is a success, the chip was wiped by the very run that produced it — which is why FailingSourcesCard's `tier_limited` palette entry has never been reachable. An "ok" run now keeps an informational class while failures stay 0 and last_error stays clear: the run did not fail and must not earn a backoff. Deviation from the plan, deliberate: the filed step said classify only when `downloaded == 0`. gallery-dl doesn't condition on that, and diverging the two backends over the same concept is what rule 169 forbids — so the native path mirrors it. "There is content here you aren't paying for" is equally true in a week we also got the cheap posts. Pinned by a test, since the stricter rule looks more correct. The predicate, the wording and the completion test are defined once in gallery_dl.py and spread into both backends (snippet 3087), rather than re-derived per half — which is exactly how they drifted apart before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-09 15:40:19 -04:00
bvandeusenandClaude Opus 5	ddf896078c	refactor(platforms): retire deviantart end-to-end (#3069) ... Executes the 2026-07-05 product decision (FC downloaders = art-dedicated services only), which removed Twitter/X and Bluesky but left deviantart fully wired for seven weeks — the half-retired state rule 22 exists to prevent. Removed: the PlatformInfo module and its registry entry, the gallery-dl extractor block, extension_service's artist-page pattern, the extension's PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission and content-script match, the frontend icon/colour/label, and the operator- facing "supported platforms" list that still advertised it. Two judgment calls, both recorded in migration 0088: * existing `source` rows are DISABLED, not deleted. The row is the only record of the artist's DeviantArt URL. Disabling is also required for correctness rather than tidiness: with the platform unregistered the download path falls through to gallery-dl, which carries its OWN deviantart extractor, so an enabled row would have kept downloading from a dropped platform. * the `credential` row IS deleted — a live session cookie for a site FC will never call again. Adds the invariant whose absence is why manifest.json drifted in the first place: nothing tied its domain lists back to the platform table. The extension suite now asserts both directions, plus that no host permission belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at an operator-chosen URL the extension cannot enumerate). Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged extension change without a bump. No release is cut — build.yml's sign-extension job only runs on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:24:08 -04:00
bvandeusenandClaude Opus 4.8	8838b325fb	fix(recapture): link on-disk images to their post (#1288) ... Recapture disk-skips already-downloaded media, and upsert_post_record only writes Post fields — so a pre-existing image (e.g. one pulled under the old gallery-dl path, imported bare with no post) stays orphaned even after its post record is (re)written. Confirmed on the operator's instance: 329 pixiv images with primary_post_id NULL, 694 pixiv posts with content but no linked images, 0 duplicate posts. Fix: the recapture relink channel now carries the media's post_id (2- → 3-tuple path/url/post_id), and phase 3 calls importer.link_existing_image_to_post — match the on-disk image by path, find its Post by (source, external_post_id), upsert image_provenance + primary_post_id. Factored the provenance-linking out of _apply_sidecar into a shared _attach_provenance so the fresh-import and recapture-backlink paths can't diverge. Idempotent; generic across native platforms (no-op for already-linked Patreon/SubscribeStar). Re-running recapture now repairs orphaned images; future walks never orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 20:13:53 -04:00
bvandeusenandClaude Fable 5	0bbcdee3bd	feat(pixiv): flip dispatch to the native ingester (#129 step 4) ... pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the recover/recapture UI actions now route through PixivIngester. Campaign id is parsed straight from the source URL (numeric user id — no network resolver), with a platform-aware resolution-failure message. auth_token now rides the uniform adapter construction (token platforms use it, cookie platforms accept-and-ignore), and the preview endpoint fetches/threads it. The legacy gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the refresh-token config branches in download/verify) per no-legacy policy; gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-03 09:54:18 -04:00
bvandeusenandClaude Opus 4.8	d8d8ecd78f	feat(subscribestar): flip dispatch to the native ingester (#893, Step 5) ... SubscribeStar now downloads + verifies through the native core ingester instead of gallery-dl — the go-live switch for milestone #71. - download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester / preview_source / verify_source_credential platform-aware. SubscribeStar's campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity. preview now catches the shared NativeIngestError (covers both platforms). - platform_lock: subscribestar serialized (one paced walk at a time). - gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no fallback once native works). - frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the recover/recapture actions show for subscribestar; download_service's cursor/mode/post_first + the preview endpoint already key on uses_native_ingester, so backfill/recovery/recapture/preview light up for free. - tests: download_backends (subscribestar native), platform_lock (serialized), and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials verify, gallery_dl_service skip-value, api_sources arm-no-preflight). post_is_gated stays best-effort (can't cause junk downloads); not gating this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 12:03:48 -04:00
bvandeusenandClaude Opus 4.8	eb811e11f6	refactor(ingest): per-post handling into run stdout via a downloader outcome (#842) ... Two corrections from operator review: 1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI section — the native ingester now writes a per-post line into the run stdout (parity with gallery-dl's per-file stdout), so the per-post handling shows in the panel the operator already uses. 2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring the download_post -> MediaOutcome contract — and the downloader owns the read; ingest_core only formats the outcome into the log line. Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture' section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+ ' — EMPTY' when 0), so an empty body is self-explanatory by post_type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 23:27:38 -04:00
bvandeusenandClaude Opus 4.8	bcc7266021	feat(downloads): per-post body-capture diagnostics in the event UI (#842) ... Operator can't (and shouldn't have to) hunt worker logs to see why a recapture left a post body empty. Surface per-post handling ON THE EVENT, in the UI. The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a per-post diagnostic {post_id, title, post_type, body_chars} with zero extra fetching — a 0-char body next to its post_type explains an empty post at a glance (e.g. polls/embeds whose body the API never returns). - ingest_core: accumulate post_diagnostics; thread via DownloadResult - download_service: write to DownloadEvent.metadata_['post_diagnostics'] - DownloadDetailModal: 'Post capture' section — totals + empty-body table (post_type + chars, flagged) + all-posts table; included in Copy-all - tests: ingester diag (post_type + body_chars), download_service metadata Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 23:09:37 -04:00
bvandeusenandClaude Opus 4.8	65ec29ba9b	feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830) ... A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:58:40 -04:00
bvandeusenandClaude Opus 4.8	796e92540a	feat(patreon): capture media-less/text-only posts (post-only records) ... Today the ingest core does `if not media: continue`, so a post with no downloadable media (a pure-text post — which often holds the ONLY copy of an external mega/gdrive/pixeldrain link) never upserts a Post. Now the native ingester emits a post-only sidecar (`_post.json`) for every media-less post, gated through the seen-ledger via a synthetic `post:<id>` key so the body is detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on external_post_id so it UPDATES the same Post a media import would create — never doubles, never clobbers a populated body with an empty one. - gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path unaffected — all constructions are keyword). - ingest_core.py: media-less branch (optional client/downloader seams via getattr; stub clients in tests skip it as before). - patreon_client.py: post_record_key(post). patreon_downloader.py: write_post_record + _write_sidecar_data refactor (shared serializer). - importer.py: upsert_post_record. download_service.py: phase-3 import loop. - tests: client/downloader/ingester (gate + recovery)/importer (no-double). Slice 0b of milestone #64. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 12:44:46 -04:00
bvandeusenandClaude Opus 4.8	fb7383eea7	feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708) ... Owning the native client means we see the 429 Retry-After header — previously discarded. PatreonAPIError now carries `retry_after`; on a PERSISTENT page-fetch 429 the client attaches the server's raw Retry-After seconds. New DownloadResult.retry_after_seconds; patreon_ingester._failure_result sets it on RATE_LIMITED. download_service._update_source_health passes it to set_platform_cooldown as `seconds=`, clamped to [60, 3600] (a tiny hint can't leave the platform effectively un-cooled; a huge one can't strand it for hours); no hint → the flat 900s default. So a rate-limited platform cools for as long as the server actually asks, not a fixed guess. Tests: terminal 429 surfaces retry_after (test_patreon_client); cooldown honors + clamps the hint, falls back to default when absent (test_download_service). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 11:34:37 -04:00
bvandeusenandClaude Opus 4.8	b211900390	refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707) ... Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 11:11:25 -04:00
bvandeusenandClaude Opus 4.8	e53f8959af	feat(patreon): structured ingester results + quarantine surfacing — #704 step 1 ... The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and quarantine stats blank. We own the ingester, so it now RETURNS structured data and phase 3 reads it directly. - DownloadResult gains run_stats/cursor/posts_processed (None/0 on the gallery-dl path, which keeps the text route). - Ingester builds real run_stats from per-media outcome counts, sets the checkpoint cursor structurally (no fake `Cursor:` stdout), and counts posts processed. download_service phase 3 uses dl_result.run_stats when present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint dl_result.cursor instead of parse_last_cursor(stdout). - #4 quarantine: PatreonDownloader reports a distinct "quarantined" MediaOutcome (with the _quarantine dest); the ingester surfaces a real files_quarantined + quarantined_paths + run_stats.quarantined_count (was hardcoded 0). Quarantined media isn't written or marked seen. - Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`) removed from gallery_dl — the structured cursor replaced the scrape. Tests: ingester result carries real run_stats/cursor/posts_processed + quarantine counts; downloader quarantines an invalid file as "quarantined"; backfill cursor tests pass cursor= structurally; dropped the parse_last_cursor tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 23:39:28 -04:00
bvandeusenandClaude Opus 4.8	ec43e823e1	feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697) ... Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 22:18:03 -04:00
bvandeusenandClaude Opus 4.8	682beafbc5	feat(patreon): drift detection + error categorization — build step 4 (plan #697) ... Typed, loud failure mapping for the native Patreon ingester so a changed API shape or expired auth never silently zero-downloads as "success". - New ErrorType.API_DRIFT (free varchar error_type col → no migration): distinct from auth so the operator knows the fix is updating the ingester, not rotating cookies. - patreon_client: PatreonAPIError carries status_code; new PatreonAuthError for 401/403 + HTML-login/non-JSON bodies (reclassified from drift — expired-session is auth, actionable as "rotate cookies"). - patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR, PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR, transport→NETWORK_ERROR. (429 thus drives the platform cooldown.) - FailingSourcesCard: api_drift chip (red) + hint. Contract test (new test_patreon_contract.py): the recorded /api/posts fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the request params must still carry every field the parser depends on (file_name, image_urls/download_url, images/attachments_media/media includes, content/post_file/image post fields) — a trim of either trips a red build. Plus client HTTP-status classification tests and ingester error-type mapping tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 21:56:58 -04:00
bvandeusenandClaude Opus 4.8	96fffaff64	feat(download): smarter backfill — time-boxed chunks, run-until-done (backend) ... Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 15:02:46 -04:00
bvandeusenandClaude Opus 4.8	593f65c9cc	feat(download): cursor-paged Patreon backfill for large catalogs ... Large Patreon creators (Anduo: weekly 50-120-image Reports back months = thousands of files) couldn't backfill: each run re-walked newest→oldest from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone exceeded the 1170s subprocess budget, so the run died during enumeration with 0 files written and NO forward progress — re-stranding every time (event #40411). Checkpoint gallery-dl's pagination cursor so each backfill window advances the frontier: - gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True for a fresh run); parse_last_cursor() pulls the last emitted 'Cursor: <token>' from stdout+stderr — survives a timed-out run since the TimeoutExpired path returns partial output. - download_service.py: phase2 stays in BACKFILL mode while a cursor is pending (even after the run budget drains) and threads resume_cursor; _apply_backfill_lifecycle() checkpoints the advancing cursor each non-completing run, completes on a clean rc=0 finish (walk reached bottom), and a stuck-guard clears the cursor after 2 non-advancing runs so a wedged walk can't re-strand forever. patreon-only (sole platform with a resumable cursor); other platforms keep the simple counter semantics. Cursor state lives in config_overrides JSON (patreon_campaign_id precedent) — no migration. Time-budget ladder (1170/1350/1500) unchanged. Plan #689. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 23:44:31 -04:00
bvandeusenandClaude Opus 4.8	6590dcdb39	fix(download): salvage soft-time-limit kills + fix timeout ladder ... Backfill downloads stranded with empty logs + a generic "stranded by recovery sweep" error. Root cause: the backfill gallery-dl subprocess timeout (1170s) exceeded download_source's Celery soft_time_limit (900s), so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The TimeoutExpired path (which captures partial stdout/stderr and finalizes the event) never ran, the event was left 'running', and phase 3 never decremented backfill_runs_remaining — so the source re-ran and re-stranded every tick (Anduo #39912). Two layers: 1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both subprocess budgets (870 tick / 1170 backfill) sit below the soft limit with phase-3 persist headroom. Promote to module constants and guard the invariant with a test. 2. Catch SoftTimeLimitExceeded in download_source and finalize the in-flight event with a real reason, mirror phase-3 source-health, and decrement backfill so a chronically-slow source self-heals to tick mode. The existing celery_signals handler only covered TaskRun, not DownloadEvent — that was the gap. Updates stale 900/1200 references in gallery_dl.py + maintenance.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 16:56:13 -04:00
bvandeusen	3898ce7be4	fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit ... - BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout 30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat TimeoutExpired (matched the tick 870s/900s rationale). Backfill runs that hit the cap let the next tick continue via the archive. - recover_interrupted_tasks orphan UPDATE now stamps finished_at; without it cleanup_old_tasks' WHERE finished_at<cutoff never reaped orphan-swept rows. recover_stalled_task_runs also now sets duration_ms (matches celery_signals.finalize's millisecond math). - ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3 on Source creation, mirroring SourceService.create. Without it, Firefox quick-add on a creator with >20 unsynced posts walked the full feed until subprocess timeout. Renamed the constant from _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module. - gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside NO_NEW_CONTENT — the download path (line 712) already does, and TIER_LIMITED proves auth reached the post and was told it was tier-gated. Verify endpoint previously showed red on this and prompted operators to rotate working cookies. - prune_unused_tags now runs a single DELETE with the NOT-IN predicate find_unused_tags uses, instead of SELECT-ids → DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that would have surfaced on a tag explosion (>65k unused tags). - credentials.upload() reflects the returned record into the store cache (`.set(platform, rec)`) instead of evicting it; previously the card briefly rendered "no credential" between upload and loadAll().	2026-06-02 11:24:05 -04:00
bvandeusen	66ff671f09	fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos ... Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts hosted on Mux carry a JWT that encodes a playback restriction policy (`playback_restriction_id` in the token). The token signature alone is not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch. gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns 200 (token is valid, HEAD sends no Referer that Mux's policy rejects), but when yt-dlp follows up with the actual manifest GET it sends its own default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the single video fails — every other image in the same post still downloads. Static `downloader.ytdl.raw-options.http_headers` block now pins Referer and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch clears the policy check. Headers-only restrictions are now handled. Mux IP-range restrictions (if a creator opts into them) remain unfixable from our worker — those would need a Patreon-region IP. Per-video PARTIAL classifier (shipped in plan #544 last commit) already handles the residual case: a run that grabs all images and fails 1 video classifies as status=ok rather than flipping the whole event red.	2026-06-01 19:00:50 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00
bvandeusen	66f19d67f5	fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp ... Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED	2026-05-31 23:30:39 -04:00
bvandeusenandClaude Opus 4.7	99b66aa85f	fix(download): preserve partial output + classify timeouts richer ... Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show anything other than that the task timedout and was cleaned up. I can't tell why it ran over or if it was stuck failed or there was just that much to get." The TimeoutExpired branch was returning a DownloadResult with no stdout, no stderr, no files_downloaded, and a generic "Download timed out after N seconds" message — even though subprocess.TimeoutExpired carries the partial output gallery-dl emitted before being killed. Now: - Capture e.stdout / e.stderr (coerced str if bytes; "" if None). - Count files_downloaded from partial stdout via _count_downloaded_files. - Surface a tail-of-stderr hint in error_message so the UI summary tells the operator at a glance whether it was "lots of content" (high count, clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung silent" (zero count, "no stderr output"). - Promote error_type to RATE_LIMITED when the partial stderr matches RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole 900s window is the timeout-shaped tail of a real rate limit, and the platform cooldown should kick in for the same reason. Existing test_download_timeout strengthened to also assert empty-partial case stays correctly TIMEOUT-classified with no preserved output. New test_download_timeout_preserves_partial_output_and_classifies covers the rich-partial-output → RATE_LIMITED promotion path. DownloadEvent.metadata already flows stdout/stderr/run_stats from DownloadResult via _phase3_persist — no UI change needed; the existing DownloadDetailModal will surface the captured output automatically once the build redeploys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 14:16:54 -04:00
bvandeusenandClaude Opus 4.7	d28db32012	fix(download): align gallery-dl subprocess timeout to Celery soft limit ... SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's Celery task has soft_time_limit=900s and hard time_limit=1200s. So gallery-dl never hit its own subprocess.run timeout — Celery always killed it first. The hard SIGKILL leaves no terminal flip on the DownloadEvent, which then sat pending/running until the recovery sweep flipped it to error at 30 min from start. From the operator's seat that was a ~30–40 min "hang" on every retry of a broken source. Pinning the default to 900s (matching Celery's soft_time_limit) lets subprocess.run raise TimeoutExpired cleanly inside Celery's window, and the existing `except subprocess.TimeoutExpired` branch (gallery_dl.py:635) captures it as a clean error_type='timeout' with a real message. The DownloadEvent flips to error in ~15 min instead of waiting on the recovery sweep at 30. Per-source bumps still live in source.config_overrides for legitimately long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries the rationale and the Celery-soft-limit dependency in its comment. Operator-flagged 2026-05-30 on 59-source strand pile retries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 02:25:04 -04:00
bvandeusenandClaude Opus 4.7	215a8993a1	fix(lint): noqa ASYNC109 on gallery_dl.verify timeout (subprocess.run deadline, not coroutine) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 08:18:33 -04:00
bvandeusen	56970fb66d	feat(credentials+downloads): real credential Verify button + live download-activity polling ... Operator-flagged 2026-05-28, two asks. **1. Credential Verify (was missing vs GS — and now actually verifies).** GS's Verify was a stub (`TODO: implement actual verification` — just stamped last_verified). FC does a real check, which matters given the recent auth pain (subscribestar age cookie, HF host-only PHPSESSID): - GalleryDLService.verify(url, platform, cookies_path, auth_token) runs gallery-dl in `--simulate --range 1-1` mode (no download) against the URL with the materialized credentials, then reuses _categorize_error: returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other → inconclusive (reason surfaced). 45s timeout. - POST /api/credentials/<platform>/verify picks an enabled Source for the platform to probe, runs verify, and on success stamps credential.last_verified (new CredentialService.mark_verified). Returns {valid: bool|null, reason, last_verified?}. valid=null means untestable (no credential, or no enabled source to point at). - CredentialCard gains a Verify button (on credentialed cards) + a result chip (Verified ✓ / Failed / Untestable) and a toast with the reason. SettingsTab reloads on @verified so last_verified refreshes. **2. Live download-activity feedback.** The Downloads tab was static — no way to tell if downloads were succeeding without manually hitting Refresh. It now auto-polls: stats every 4s, and the event list too while anything is queued/running. Polling pauses when the tab is backgrounded (document.hidden) and the list reload is skipped on idle ticks to stay light. A pulsing "● live" indicator next to the stat chips shows when auto-refresh is active (queued+running > 0); honors prefers-reduced-motion. Tests: verify endpoint — untestable with no credential, untestable with no enabled source, valid+stamped on success (gallery-dl mocked), and auth-failure reported without stamping.	2026-05-28 07:52:20 -04:00
bvandeusenandClaude Opus 4.7	29598c3f9a	fix(fc3c): ruff — split SyncSession import, StrEnum, ASYNC240 noqa, blank-line ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 20:50:38 -04:00
bvandeusenandClaude Opus 4.7	7a96b1d06c	feat(fc3c): GalleryDLService — subprocess wrap + error classification + validate/quarantine ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 20:41:42 -04:00