0928d2f914796e69ce1a32afac72e32444d155b5
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1efece0f21 |
feat: a Discord source can remove the posts it took from people outside its poster list (#4486)
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m39s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m43s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 2s
"Remove posts from other posters" on a Discord source with an "Only posts by" list previews what goes, per poster, then deletes it behind a typed token: - posts whose record names a poster not on the list (by id, username or display name); posts that record no poster are left alone and counted; - images found only on those posts; one also on a kept post stays, and a synthetic drop's link never keeps one; - their attachments, and every drop that absorbed one of them, so the grouping sweep regroups what remains. Preview and apply share one predicate, and a parity test holds them to it. The post record now also saves the poster's display name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e5bdcd2596 |
feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
a copied platform list ('patreon', 'subscribestar') that went stale when
Discord moved over. Sources now carry `native_ingester` from the backend's
own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
backfill's next chunk: each one waited for the source's regular interval,
so an armed backfill sat idle until the next check (8h at the default) and
a five-chunk walk took most of two days. The in-flight guard and the
platform lock keep one chunk at a time. A failing source falls back to its
backoff, and a stalled or out-of-budget walk stops being due. It also runs
when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
defaults, sidecar-mirroring postprocessor and token injection are gone.
The naming test moves to the native downloader and still renders against
the real gallery-dl sidecar fixture. That is the guard that the files
gallery-dl wrote are found on disk rather than fetched again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
|
||
|
|
84e5448941 |
feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / integration (push) Successful in 2m27s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR |