Compare commits

...
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 eeb9263125 feat: the Discord Add panel's artist field autocompletes against FabledCurator's artists (milestone 429)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m23s
CI and images / build-agent (push) Successful in 5s
CI and images / sign-extension (push) Successful in 2m56s
CI and images / build-web (push) Successful in 1m37s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Successful in 1s
Operator, after the first live run: "I need the extension to offer an
autofill search function so it's easier to match an entry with an existing
artist."

- The field searches as soon as the panel opens (the server name, usually)
  and on every keystroke; matches list under it, with ↑/↓, Enter/Tab to pick,
  Esc to close, and a last "+ New artist" row.
- Inline autofill: the rest of the top match is filled in and selected, so
  typing on replaces it and Tab/Enter accepts it. Backspacing never refills.
- A result whose name IS the text, spacing and case aside, is picked on its
  own; the hint says in green which existing artist the source will join.
- /api/artists/autocomplete also matches ignoring spacing and punctuation
  ("Tamada Heijun" finds "TamadaHeijun"), ranked just below an exact match.
  The web UI's artist search gets it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 07:45:51 -04:00
bvandeusenandClaude Opus 5.5 4c75dd0f88 feat: the extension adds Discord channels to an artist you pick, and its tests gate the XPI (milestone 429)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 24s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / build-agent (push) Successful in 5s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3m13s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Server (#4420)
- extension_service gains a Discord pattern (server or channel, jump links,
  ptb/canary; not DMs or threads), mirrored in platforms.js and pinned by
  the shared artist-url-samples.json.
- probe on a Discord URL matches the source by ids under any artist, reports
  a whole-server source as covering the channel, suggests the artist who owns
  another source on the same server, and names server/channel via the stored
  token (best-effort, bounded, no rate-limit waits).
- quick-add takes artist_id / artist_name; Discord URLs are stored canonical.

Extension (#4421, #4422)
- Content script on discord.com; SPA navigation by URL polling (the old
  pushState patch ran in the isolated world and never fired); stale probes
  are dropped.
- Discord chip opens an Add panel: this channel or the whole server, and the
  suggested artist / a search / a new name.
- Popup: sources show artist, platform and state; a Discord token export is
  verified by FC and the result shown. Token capture covers ptb/canary.
- Pure logic in lib/chip.js and lib/popup-format.js, with specs.

CI (#4423)
- extension.yml's lane (web-ext lint, vitest, XPI contents) moves into
  build.yml as extension-test and joins the needs of sign-extension,
  build-web and build-agent. As a separate workflow it gated nothing: a red
  extension suite still signed and shipped the XPI (rule 177).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 23:31:38 -04:00
bvandeusenandClaude Opus 5.5 97045279a3 fix: a never-checked source still reports no next check while its first backfill is armed
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m51s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 2s
Every new source is created with a backfill armed, so e5bdcd2's backfill
branch in compute_next_check_at answered "now" for sources that have never
been checked, where the contract (test_create_response_includes_fc3d_fields)
is None. The never-checked case comes first again. Scheduling is unaffected:
select_due_sources already treats never-checked sources as due.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 23:19:23 -04:00
bvandeusenandClaude Opus 5.5 e5bdcd2596 feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
- Recover and Recapture show on every native source. The menu gated them on
  a copied platform list ('patreon', 'subscribestar') that went stale when
  Discord moved over. Sources now carry `native_ingester` from the backend's
  own predicate.
- A running backfill is due on every scheduler tick. Nothing queued a
  backfill's next chunk: each one waited for the source's regular interval,
  so an armed backfill sat idle until the next check (8h at the default) and
  a five-chunk walk took most of two days. The in-flight guard and the
  platform lock keep one chunk at a time. A failing source falls back to its
  backoff, and a stalled or out-of-budget walk stops being due. It also runs
  when the artist has auto-check off, since the operator started it by hand.
- gallery-dl no longer carries Discord: its naming constants, platform
  defaults, sidecar-mirroring postprocessor and token injection are gone.
  The naming test moves to the native downloader and still renders against
  the real gallery-dl sidecar fixture. That is the guard that the files
  gallery-dl wrote are found on disk rather than fetched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 23:14:48 -04:00
bvandeusenandClaude Opus 5.5 84e5448941 feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / integration (push) Successful in 2m27s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Discord was the last focus platform still on gallery-dl. This adds the native
path, mirrored from gallery-dl 1.32.13's discord extractor:

- discord_client: API v10 with the user token and gallery-dl's request
  profile (dated Firefox UA, Referer). Walks a server, category, forum,
  channel or thread in gallery-dl's order and pages each channel newest-first.
  Files are attachments, then embeds, then forwards, numbered across the
  message. The resume cursor is <channel>:<before>. Text-only messages are
  not posts, since gallery-dl never made them.
- discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans
  names on Linux (only `/` and control characters change), so existing files
  are skipped_disk rather than fetched again. Sidecars carry identity only.
  The message record keeps gallery-dl's keys, so parse_sidecar,
  derive_post_url and the drop grouping read it unchanged.
- The ledger keys on the attachment id (or a hash of an embed's URL path),
  not the file's position, which an edit can renumber. Migration 0111.
- DiscordIngester: token auth, body canary off (files-only drops are
  normal). Registered as native, verified by token, and serialised
  per-platform, since every source shares one user token.
- ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out
  on a multi-channel source now ends the quiet channel, not the whole walk.
  Clients without the seam behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 19:42:00 -04:00
bvandeusenandClaude Opus 5.5 058fa85606 fix: backfill_phash runs on the long maintenance lane, not the scheduler's quick one (4411)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m47s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
A whole-library rehash with a 35-minute limit was matched by the
maintenance.* glob. It held a scheduler process for its whole run, and the
minute ticks queued behind it. An exact-name route now sends it to
maintenance_long.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 19:26:48 -04:00
bvandeusenandClaude Opus 5.5 cc53d8db7b feat: a GPU agent on the CPU shows as degraded, in the System view and on its own page (4410)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-web (push) Successful in 2m46s
CI and images / smoke-web (push) Successful in 50s
CI and images / build-agent (push) Successful in 6m35s
CI and images / promote (push) Successful in 2s
torch and onnxruntime both fall back to the CPU without raising, so the agent
that ran CPU-bound for weeks after a driver update leased and checked in like
a healthy one.

- The agent sends its startup accel report on every lease and heartbeat.
- The server keeps a bounded copy on the roster row. A running agent with a
  runtime off the GPU becomes `degraded`, with a sentence naming the runtime
  and the reason.
- The top nav shows it amber.
- The agent page carries a banner, and its pill reads "CPU only".

Also: the bandwidth field gets the page's − / + stepper, and both number
fields drop the browser's spin arrows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 19:09:07 -04:00
bvandeusenandClaude Opus 5.5 e39ec1c550 docs(agent): a driver update leaves the CDI spec naming a stale nvidia-uvm device — how to spot and regenerate it (1451)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m29s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 6s
CI and images / smoke-web (push) Successful in 41s
CI and images / promote (push) Successful in 1s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 19:04:03 -04:00
bvandeusenandClaude Opus 5.5 42a40d71a4 fix: the agent's ONNX check asks CUDA for a device instead of trusting that the libraries loaded (1451)
CI and images / frontend-build (push) Successful in 24s
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m19s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-web (push) Successful in 5s
CI and images / smoke-web (push) Successful in 41s
CI and images / build-agent (push) Successful in 5m42s
CI and images / promote (push) Successful in 2s
It reported "onnx on GPU" beside torch failing cuInit with "CUDA unknown
error". Every library resolved, but no device could be used. The check now
calls cudaGetDeviceCount and reports the CUDA error when there is one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 18:28:26 -04:00
bvandeusenandClaude Opus 5.5 f127c50207 fix: the lane sizer reads the live pool, so it stops growing a lane past its cap (4409)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 24s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m16s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
It read celery's `max-concurrency` as the pool size, but prefork reports that
as the limit the pool booted with. pool_grow and pool_shrink never update it.
A lane booted at 1 therefore read 1 forever. The sweep sent `target - 1` on
every tick while work waited, and could never shrink, since 1 - 1 is 0. The
System tab showed Scheduler "6 / 1" at a cap of 2.

pool_size() counts `processes` and falls back to the limit only when there is
no process list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 18:21:11 -04:00
bvandeusenandClaude Opus 5.5 7a09dc3cda build: the agent installs one CUDA-13 stack instead of two, the web image drops ML packages it never imported, and Redis moves to 8 (1451, 1452)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
extension / lint (push) Successful in 16s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-web (push) Successful in 3m7s
CI and images / smoke-web (push) Successful in 59s
CI and images / build-agent (push) Successful in 6m41s
CI and images / promote (push) Successful in 2s
Agent:
- The image ran PyPI's CUDA-13 torch 2.14 and onnxruntime-gpu 1.30 on a
  CUDA 12.9 cudnn-runtime base. requirements.txt had silently replaced the
  Dockerfile's torch 2.6+cu124, because ultralytics pulls torchvision, which
  pulls its own torch. That left ~3 GB of base libraries and a ~3 GB torch
  nothing loaded: 10 GB compressed.
- Now: an nvidia/cuda 13.0.3 `base` image, with torch and torchvision
  installed together from cu130. CUDA and cuDNN come from the nvidia-* pip
  packages; onnxruntime-gpu declares its [cuda,cudnn] extras.
- fc_agent/accel.py preloads those libraries for onnxruntime. It then logs,
  and reports in /status, whether torch and the ONNX CUDA provider actually
  got the GPU, since both fall back to the CPU silently.

Web image:
- Drop opencv-python-headless and onnxruntime, plus the opencv-only apt libs.
  Both have been listed since the scaffold and nothing in backend/ imports
  them.
- torch/torchvision move to 2.14/0.29, and the unexplained caps are lifted
  (rule 154).

Redis: 8-alpine in both compose files and both CI service containers. That
gives an AGPLv3 licence option, where 7.4 was RSAL/SSPL only. The client
moves to >=8.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 17:45:39 -04:00
bvandeusenandClaude Opus 5.5 2587421f5b ci: a push publishes through a candidate tag and promotes only after the smoke (4310)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
extension / lint (push) Successful in 21s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m19s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 5s
CI and images / smoke-web (push) Successful in 41s
CI and images / promote (push) Successful in 1s
On a push, build-web and build-agent wrote :dev / :latest straight from the
build, and :c-<sha> right after it. smoke-web then booted the image, so it
could detect a broken image but not stop one reaching the tag deployments
follow. Rule 164's verify_with puts the check between build and push.

Both image jobs now build to :<channel>-candidate (the refresh keeps
:refresh-candidate). Their repoint step acts only on a reuse hit. promote
needs build-web, build-agent and smoke-web on every trigger, and writes
each built image's full tag list (channel, plus :c-<sha> on main) by manifest
PUT from that job's digest, reading each tag back. A failed or skipped smoke
leaves every tag on the last build that worked. The refresh path now also
promotes by digest rather than by the candidate tag's name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:51:29 -04:00
bvandeusenandClaude Opus 5.5 ff70f837d0 refactor: test row factories and the fetch stub have one copy each (3109)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 30s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m44s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
tests/factories.py holds image_row/make_image/make_image_async/make_tag. The
17 byte-identical _img/_tag helpers (15 modules) now import them under their
old names, so no call site changed. frontend/test/support/stubFetch.js
replaces 15 copies that differed only in formatting. Copies whose bodies
differ (other defaults, other columns, a url-only stub) are left as they
are; folding those needs a look at each caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:39:38 -04:00
bvandeusenandClaude Opus 5.5 0842df46e9 refactor: components read the obsidian surface from its token, not a typed rgba (3108)
Thirteen rgba(20, 23, 26, ...) across six components (the task counted
eleven). They now use --v-theme-background, which Vuetify maps to obsidian;
ArtistHeader's banner fade uses --fc-chrome-rgb, since it is nav-style
chrome. A spec fails if the literal reappears outside a comment. Values are
unchanged, so nothing should render differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:37:49 -04:00
bvandeusenandClaude Opus 5.5 f2e4ee4d17 fix: beat takes each job's last run from task_run, so a redeploy no longer resets the schedule (4408)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m18s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 53s
CI and images / promote (push) Skipped
Beat's default scheduler kept its memory in a shelve file nothing persists,
and a scheduler that remembers nothing waits a full interval before any job.
Since the one-container image, every redeploy restarts beat, and no daily or
weekly job had run since 2026-09-21 (cleanup, backup and download-event
pruning, membership sync, thumbnail backfill, integrity check, vacuum).

TaskRunScheduler seeds each entry's last_run_at at startup from task_run's
newest start for that task: an overdue job runs at once, one not yet due
waits the remainder, and one never recorded is due now. prune_task_runs now
keeps each task's newest row however old, or a weekly job would look
never-run a day after it ran and fire on every restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:36:58 -04:00
bvandeusenandClaude Opus 5.5 31020d9395 fix: a native chunk stops walking when its import work would overrun the task
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m15s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
The walk's time budget covered the walk alone, but phase 3 runs in the same
Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked
for about two minutes and handed phase 3 431 orphan imports and ~3000
relinks. Phase 3 ran for 20 minutes and died at the soft limit (event
90808). This predates the worker consolidation; the limits are unchanged
since June.

The walk now also stops when elapsed time plus phase 3's estimated cost
(2.5s per import, 0.25s per relink, measured on the live instance) passes
CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so
such a stop is a PARTIAL chunk boundary and the next chunk resumes the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:13:47 -04:00
bvandeusenandClaude Opus 5.5 ee59133b42 fix: a new view opens at its top instead of the last view's scroll offset
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m17s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
The router had no scrollBehavior, so Settings entered from a scrolled feed
opened scrolled too, its heading and tab strip under the nav. Back/forward
restores the saved position, and a new query on the same view keeps its
place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 16:03:38 -04:00
bvandeusenandClaude Opus 5.5 7cc34b5724 fix: a Discord message that re-posts a file is grouped, and joins the drop it repeats
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m17s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m40s
CI and images / smoke-web (push) Successful in 53s
CI and images / promote (push) Skipped
Grouping read a message's images through primary_post_id alone, which only
the first message imported with a file holds. The backfill runs newest-first,
so the original message usually owned nothing: 101 of Yellowroom's messages
(mostly 2018-2020) could never be grouped. Every image they carried also sat
in another message (296 links, measured on the live instance).

_message_images unions ownership with provenance for the candidate query,
the drop seed, the member image links, and the merge pass. Nothing is
re-owned. A later drop carrying the very file an earlier one carries now
merges by a new same_image route, which nearest-neighbour could not see
because it skips the image itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 15:21:05 -04:00
bvandeusenandClaude Opus 5.5 20cb813809 fix: a one-message Discord drop is labelled a Discord message, not a grouping (4390)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m16s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
It stays synthetic, since teaser matching looks only at drops, but the card
no longer claims "Grouped from 1 Discord message" / "grouped by
FabledCurator" over a single message. The marker chip stays and reads
"from Discord".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 15:16:41 -04:00
bvandeusenandClaude Opus 5.5 3aa899ad29 test: revisit tests' first walk completes phase 3 before the second
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m16s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 14:16:07 -04:00
bvandeusenandClaude Opus 5.5 da2091a875 fix: a download is marked seen only after it is imported
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
A run killed between download and import left its files on disk and in the
seen-ledger but never in the library, and every later walk trusted the
ledger. TamadaHeijun's 12PCG post lost 7 of 13 images this way (stranded
run 90402), which read as the duplicates filter.

The ingester now hands phase 3 a mark_seen_after_import hook, called after
the import loop. A file on disk with no ImageRecord at its path is fed to
import, not reconciled into the ledger. Recapture reaches files already
orphaned, because it looks past the ledger to the disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 14:13:12 -04:00
bvandeusenandClaude Opus 5.5 49d18c1757 fix: adding a found subscription sends its membership id (adopt 400 invalid_body)
membershipReconcile.adopt passed { membership_id } as request options, so
no body was sent. useApi now refuses any option other than body, params or
signal, so a payload in the wrong place fails loudly in the browser instead
of as a 400 from the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 14:13:12 -04:00
bvandeusenandClaude Opus 5.5 2f9e35390e fix: the matcher sees a drop's names, finds merged trickles, and catches up on its own (4390, 4392)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m17s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m45s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Three defects found while answering "is the linking automatic":

1. The name and image checks never worked on a live drop. The corpus keyed
   every image by primary_post_id, but a drop's images belong to its member
   messages and the drop claims them only through provenance, so a drop
   looked nameless and hashless. The tests attached images to the drop
   itself, which discord_grouping never does. Images now count under the
   post a reader sees them on: a message's absorbing drop, else the post
   itself.

2. The trickle merge (e08401c) dates a drop by its first stage, days before
   the release a teaser announces, which put merged trickles outside the 24h
   window. Candidates are now found and timed by their closest member
   message. The sweep follows recently grown drops by their messages' times
   the same way.

3. A pair left pending was skipped forever: the matcher skipped every
   recorded pair, not only decided ones. Pending pairs are now re-scored in
   place and linked once conclusive. Linked and dismissed pairs are still
   never touched.

Also, "Scan now" shared the sweep's 48-hour horizon, so it could not reach
the history it is described as being for. It now scores every post by an
artist with Discord drops (rescan(full=True)).

New tests build drops the way the grouper does, with images owned by the
member messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 14:01:07 -04:00
bvandeusenandClaude Opus 5.5 e08401c44a feat: a creator's trickle becomes one drop instead of a card per message (4390)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m54s
CI and images / smoke-web (push) Successful in 1m0s
CI and images / promote (push) Skipped
The operator, on a feed of "Grouped from 1 Discord message" cards: "the groups
are still single image even when they can clearly be seen as group". 665 of
Yellowroom's 714 drops were one message.

Both join paths demand cosine <= 0.10 to a drop's FIRST image. The stages of
one piece fail that: each is nearest the one before, not the first.
`svtt_wip4` never joined `svtt_wip3` from the day before.

Measured on artist 8 before writing it:
- phash cannot see it. Stages sit 68-134 bits apart; unrelated same-artist
  pairs have a median of 126 and a p5 of 110 (lesson #4400).
- The embedding's nearest neighbour can. Every stage of three real trickles
  had a sibling as its single nearest image in the artist's library. In a
  control over all 137 recent Discord images, a nearest neighbour that was
  another message within 7 days carried the same working name 53 times out
  of 53. Mismatches start past 7 days.

A new merge pass runs last in the sweep. A later drop folds into an earlier
one within discord_group_close_after_hours (168h, the measured 7 days) when
they share a gated leading working name, or when one's image is the other's
nearest neighbour. A drop reaching several earlier drops pulls them all
together, unless two of them are named as different pieces.

A merge carries teaser links across (the payload FK would otherwise cascade
them away). Growth is stamped at the messages' own time, so merging history
never jumps an old drop to the top of the feed. Each drop records the route
it merged by, and is checked once.

Offline replay over Yellowroom's 127 drops since 2025: they become 70 posts.
The Marin trickle ("Very early Marin" -> 3 screenshots -> MarinaraSauce_base)
becomes one post of 5 by nearest neighbour. svtt, 0-k1, cnni14 and 0adm come
together by name.

FAMILY_MAX_POSTS moves to post_naming, so the grouper and the teaser card
share one definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 13:47:25 -04:00
bvandeusenandClaude Opus 5.5 7aa074d1a9 feat: the unified card shows referenced images, their text, and an undo (4402)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 18s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m38s
CI and images / smoke-web (push) Successful in 59s
CI and images / promote (push) Skipped
The linked drop's images and the piece's variants join the teaser's rail,
after its own images, each with a corner badge. The meta line counts them
apart ("+N from Discord"), so a Patreon card never reads as though the creator
posted Discord's files there. The text of each referenced post sits under
the teaser's body.

If FC made the link, the card says so, names the matching working name and
offers Undo. If a person accepted it, it offers Unlink and makes no claim
about FC. Both dismiss the pair. The kept row then stops the next sweep from
linking it straight back.

The announcement settings card gets the two windows: how close a drop must be
before its own card is left out of the feed, and how far to reach for
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 11:25:52 -04:00
bvandeusenandClaude Opus 5.5 30a263a47a feat: a teaser's card references the drop it announced and the piece's variants (4402, 4401)
The operator's problem: a Patreon teaser is a pointer, and its card showed the
censored crop plus a text link while the content it pointed at sat on another
card. The fix is a REFERENCE, not an absorption: "the nested items on the
unified post are a duplicate or reference of existing content". Nothing is
written. Discord posts keep their own rows, dates and places in the feed.

- post_unification: for each teaser with a linked association, the drop's
  images and text, plus its variant family: Discord images sharing the
  seed's gated LEADING working name, or a phash near-duplicate, within a
  window of the teaser. One hop only, oldest first.
- Measured on artist 8 before writing it: of 121 message pairs 2-60 days
  apart that share a gated token, 106 share the leading name and all read
  as real families. Of the 15 sharing only a trailing word, 14 are sibling
  pieces and one is a plain collision (`bottom`, 56 days). The family cap
  is 8, not the pairing cap of 6, because `tentacooler` and `0-k1` (6 posts
  each) are real families.
- The feed drops a linked drop's own card only within discord_link_fold_hours
  of its teaser (default 24): "only hidden from the post view they're posted
  the same day". Older referenced posts stay where they landed.
- post_association.linked_by records whether FC or a person made the link,
  so the card can say so. Undo is the existing dismiss.
- `image0` (gallery-dl's fallback name) becomes a stopword.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 11:25:52 -04:00
bvandeusenandClaude Opus 5 f7b3e15014 feat: the drop carrying the teaser's own image links it (4392)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 29s
CI and images / integration (push) Successful in 2m12s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m43s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
Crop-to-source matching was held until the cheap signals could be shown
insufficient. They can: of artist 8's 27 teasers with a drop inside a day, 11
go unlinked, and five are screenshot teasers with no working name at all.

So it was tried, on exactly those pairs. Every teaser image correlated against
every window of every nearby drop image at five scales, ground truth being the
pairs the working name independently confirms, control being unrelated
same-artist posts a month away. **It does not separate** — true pairs score as
low as 0.401 while the control reaches 0.605, and no threshold divides them.

The reason is the one the naive version was rejected for, which turns out to
apply just as hard to the careful one: a single artist's work is
stylistically homogeneous, so a whole-image comparison between two of their
pieces is high whether or not it is the same piece. That is now written down
in the module docstring with its numbers, so the next person to reach for it
inherits the measurement instead of repeating it.

What survived asks a narrower question the measurement shows IS answerable:
not "is this a crop of that" but "is this the same image". Same pairs, same
control, using the pHash FC already stores on every image — pairs the name
confirms score 0, 0 and 20 bits of 256; the nearest unrelated pair in a
29-sample control scores 108. The threshold sits at 32, which is the number
gallery_service already calls a near-duplicate, inside a 76-bit gap.

It earns its place by being the only signal needing no cooperation from the
creator: it works on a teaser called `Screenshot 2026-08-13`, and on a creator
whose two platforms share no naming convention. It is quiet most of the time,
because a teaser is usually a crop rather than a copy — but where it fires it
is close to certain, and it recovers `Cute Selfie, Cute Dress` from the
unreachable list.

utils/phash warns the hash alone must not decide a MERGE, since variants of
one piece collide at this distance. That does not invert here — it is the
point. A merge destroys a file, so a variant colliding with its original is a
loss; this asks whether two POSTS are about the same piece, and a variant of
the drop's image is exactly that. Nothing is deleted either way.

Gated on posts like the other two: an image on many of the creator's posts is
a banner, not a piece. `_rarity` is public as `rarity` now that all three
signals share it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:27:20 -04:00
bvandeusenandClaude Opus 5 941f1c6e07 fix: run 7464's two failures — an unused loop target and a missed call site (4392)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m37s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Both mine, both from the same change.

`identity` is unpacked in the write loop and never read there — the decision
it feeds happens above it, when `conclusive` is built. flake8-bugbear is on
repo-wide and B007 is exactly this. Renamed `_identity`, and `linked +=
status == "linked"` spelled out as the `if` it actually is.

The second is worse, because it was a real assertion silently pointed at the
wrong shape. `match_post` now returns `(proposed, linked)`, and when I
rewrote the call sites I matched on `) == 0` — so the one comparison in the
file that reads `) == 1` kept comparing a tuple to an integer. Found by
walking the module's AST for every comparison against `match_post` and every
use of a name assigned from it, rather than grepping again with the pattern
that had already missed it once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:20:12 -04:00
bvandeusenandClaude Opus 5 b3e4491b0a feat: the announcement card offers the auto-link switch and says why (4392)
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m18s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
The card's copy had gone stale against its own backend in three places, which
matters more than usual here because every line of it is a claim about what FC
will do with the operator's library:

  * "nothing is linked until you accept one" is no longer true for a pair the
    creator's own file name settles. The switch that governs that is now on the
    card rather than only in the database.
  * "always needs two reasons — close in time and the post mentioning Discord"
    left out the shared-marker signal, and named 0.55 as the number below which
    no single signal can carry a pair. That number is 0.45 now.
  * "pairs only appear when a post lands near a drop and says it is about
    Discord" described the empty state as if the identity route did not exist.

A queued row now shows the working name behind it when there is one, and says
in the same breath why that name was not enough on its own. A review queue
that cannot explain itself is one the operator learns to click through without
reading, and this is the only signal that claims the two posts are the SAME
piece rather than that they happened near each other.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:16:07 -04:00
bvandeusenandClaude Opus 5 31de54e326 feat: FC links a conclusive pair itself instead of asking (4392)
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / build-web (push) Skipped
CI and images / integration (push) Failing after 2m15s
CI and images / sign-extension (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator, 2026-09-24: *"I don't want this to be manual that defeats the
convenience that I'm going for."*

Confirm-only was right while every signal was circumstantial. Time proximity
and a body that mentions Discord can never be more than suggestive, so asking
was the honest response to what FC actually knew. A shared working name is
different in kind: when the creator's own name for a piece appears in exactly
these two posts and nowhere else in their library, there is nothing left for
the operator to adjudicate, and asking is a chore FC invented for them.

AUTO_LINK_FLOOR is 1.0 and sits deliberately above IDENTITY_FLOOR's 0.75. The
gap between them IS the review queue — real evidence, not certain enough for
FC to act on alone. Measured on artist 8, of 15 name-sharing pairs: 11 are
conclusive, 2 more propose, 2 fall short of both.

Three refusals, because an auto-link is FC asserting something the operator
never saw:

  * Exactly one candidate may be conclusive. Two is not a tie to be broken by
    score — one post can tease two pieces dropped separately, and then each
    drop carries a different name from the same teaser, each individually
    conclusive. Two conclusive answers to "which drop is this" means the
    question was wrong, so both queue and FC says nothing.
  * Neither end may already be claimed by an accepted link. That link is the
    operator's decision and reassigning its other end would overrule them
    silently.
  * The whole thing is one setting, defaulting on, reversible in the UI —
    an accepted link is a row they can dismiss.

`match_post` now returns (proposed, linked) so a sweep can report what it did
on its own rather than only what it queued.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:14:04 -04:00
bvandeusenandClaude Opus 5 81f23991e9 fix: a sweep follows the drops FC has just authored, not only recent posts (4392)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m39s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 5s
CI and images / smoke-web (push) Canceled after 0s
CI and images / promote (push) Canceled after 0s
#4392's third cause, and the only one of the three about whether a pair is
SCORED AT ALL rather than how well.

A drop's post_date is backdated to its first message, but FC cannot author the
drop until that message has an embedding and the hourly grouper has run. So a
drop created this minute lands wherever its messages were — days or weeks back
in the feed. The sweep only looked at announcements published within twice the
window of NOW, so by the time the drop existed its neighbours were already
outside the horizon, and nothing brought the sweep back to them.

Measured on the live instance: a pair scoring 0.800 with `associations: []`
and an empty queue. The two scoring fixes in the previous commit would not
have helped it, because nothing scored it.

So the sweep now also gathers announcements sitting beside any drop whose
`last_grew_at`/`downloaded_at` is inside the horizon — the drop's own clock
rather than its backdated position. Capped at MAX_RECENT_DROPS so a backfill
authoring thousands at once does not quietly become the full-library rescan
the manual button exists for, and the id set is sorted before the loop because
a sweep visiting posts in a different order each run is one whose failures
cannot be reproduced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:09:09 -04:00
bvandeusenandClaude Opus 5 fd214f3a08 feat: the matcher gets an identity route, so a name can link what time cannot (4392)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Successful in 2m52s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m6s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
#4392 measured three compounding reasons the announcement matcher could not
fire for this creator. This is the first two.

**Vocabulary.** `discord` is not how these creators write once the audience
already knows where the server is. Across 20,558 Patreon bodies `discord`
appears in 486 and `the server` in 37 — but the distribution is the point, not
the totals: for artist 8, 21 of 42 posts say `discord`, 7 say `the server`,
and it is the RECENT ones that say the latter. A vocabulary list written from
old posts silently stops matching the posts that still need it.

**Two routes.** Circumstance and identity answer different questions.
Proximity and a declaration say two things happened near each other; a working
name the creator uses on these two posts and nowhere else says they are the
same piece. Averaging those makes the threshold uninterpretable and costs
both ways — adding identity as a fourth weight dilutes the others enough that
measured teaser/drop pairs an hour apart stop proposing, while capping
identity at a weight means the strongest evidence available can never carry a
pair alone. So identity may OVERRIDE, never dilute; below IDENTITY_FLOOR it is
recorded for the operator and moves nothing, which is the conservative
direction, since a wrong link asserts two different pieces are one.

The circumstantial bundle is now proximity/declared/marker at .45/.35/.20.
Rebalanced rather than extended: the existing guard still holds arithmetically
(no single weight reaches the threshold) and the existing pairs still score
where they did — a teaser and its drop an hour apart with an invite link go
from 0.815 to 0.781, both well over.

Measured against the live instance, artist 8: 17 candidate pairs, 14 carried
by a shared name, no false positives, and no drop claimed by two posts. The
operator's own example — `Anya -- 🍈🍈` and the drop 75 minutes before it —
links on the token `0-k`, which is what its two files are called on either
platform. Two pairs 21h apart that proximity scores 0.10 are reached, and two
more (`680lc`, `cnni18x`, four posts each) are the stated cost of the floor.

Both rare-token signals are artist-scoped and loaded once per artist per
sweep, not per candidate pair.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5 fb7563e263 fix: ruff UP031 — percent formatting in the new naming test (4392)
Run 7456's only failure. `"comic2_%02d.jpg" % i` is an f-string here; UP is
enabled repo-wide and I wrote the one construct it forbids.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5 c67a8c313e fix: count the POSTS a working name spans, not the files (4392)
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Counting files punished a piece for the one thing a working name is
guaranteed to do — have several exports. Measured across the operator's four
dual-platform artists: knuxy carries `p217` on four files spanning exactly two
posts, the Patreon post and the Discord drop, and ~200 comic-page tokens have
that shape. Every one scored half strength. Counting posts scores them 1.00
and still catches the habits, which span many posts rather than many files:
tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4. Knuxy's
ambiguous band drops from 268 tokens to 55.

Three false positives found the same way — by running the module over the real
library rather than reading it:

  * The screenshot guard matches from the start of the stem, so it never fired
    on the legacy `<post id>_media_<media id>_` era. `Screenshot 2025-07-27
    182450ab` sailed through and contributed `2025-07-27` — the same-day date
    collision this module's docstring exists to refuse.
  * tamadaheijun names screenshots in Japanese. A guard that knows only the
    English word is a guard for one artist.
  * `timeline 3-0002` spans three unrelated posts. Keeping hyphens inside
    tokens for `0-k`'s sake let the number survive whole, so an identity token
    now has to contain a letter — which is the property behind the old
    bare-year and all-digits rules anyway, and leaves `0-k`, `680lc` and `p59`
    untouched.

Also stopwords for `the`, `gif`, `main`, `patreon`, `capture`, `timeline`,
each measured carrying a false match inside the admitted band.

IDENTITY_FLOOR names what a shared name must reach to propose a link with no
corroboration. At 0.75, 13 of artist 8's 15 name-sharing pairs clear it,
including the operator's own example. The two that do not are real pairs this
signal will not carry alone — the stated cost of refusing the four-post band,
where conto's `illustration9` and `maid` also sit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 08:00:36 -04:00
bvandeusenandClaude Opus 5 ecbfd83029 feat: a creator's own working name links the teaser to the drop (4392)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 30s
CI and images / integration (push) Successful in 2m16s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m40s
CI and images / smoke-web (push) Successful in 57s
CI and images / promote (push) Skipped
The Patreon teaser is a censored crop of the Discord release, which is the
pair whole-image comparison handles worst — a censor bar is exactly the local
edit that moves a perceptual hash and blurs an embedding. But the creator
exports both from one file and the internal working name survives into both
platforms untouched, so the filename answers what the pixels will not.

Measured on artist 8 (520 images, 409 tokens): 17 candidate pairs, 14 of them
carried by a shared name, no false positives and nothing ambiguous. Ten are
byte-identical names either side — ConnFront/ConnFront, LoisLaneTB2/
LoisLaneTB2 — and the signal is orthogonal to timing, reaching a pair 23.8h
apart that proximity scores 0.005.

Both signals gate on rarity within ONE ARTIST's library, through one shared
_rarity so they cannot drift apart again. They already had: the filename side
grew a frequency cap and the text side never did, so a habitual emoji scored
the same 1.00 as a marker used twice. Measured, on real proposals: the
deciding evidence for one pair was 💦 (13 of 300 posts) and for another the
word "like" (43 posts). Both now score zero. The cap discards exactly what it
should — anya 8, riju 9, undyne 8, bea 16 are character names, and ungated
every Anya post would match every Anya drop.

A screenshot contributes nothing rather than the date match it could be
squeezed for: that date collides across platforms by construction, since the
teaser and the release go out the same day.

Not yet wired into post_association_service — that is the next commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 07:55:18 -04:00
bvandeusenandClaude Opus 5 e1cd629196 feat: the agent derives its version instead of being asked to remember it (4386)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
extension / lint (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-web (push) Successful in 2m1s
CI and images / smoke-web (push) Successful in 1m2s
CI and images / build-agent (push) Successful in 10m7s
CI and images / promote (push) Skipped
Operator: the agent's build string could not identify the agent. VERSION was a
literal in app.py an author was meant to bump, and nobody did — the September
image printed the same "2026-07-17.1" as the July one, so the one surface
meant to answer "did my pull work?" answered the same either way.

Nothing new was needed. scripts/artifacts.sh has derived a version per
artifact since milestone 313, and build-agent has been computing the agent's
on every run and printing it to the log. The image just never carried it.

Three values, never folded together (rule 149):

  FC_VERSION   YYYY.MM.DD.HHMM from the COMMIT its shipped files last changed
               in — identical on dev and main for the same source, which is
               what makes "am I running production's code?" answerable.
  FC_CHANNEL   a sibling field, never a suffix inside the name.
  FC_REVISION  the 12-char sha; the same string as the fc.revision LABEL, so
               the image and the registry cannot disagree about which commit
               this is.

The page SHOWS the version and COMPARES the revision. Those were one value
before, which is how a version acquires a second job and then cannot be
changed without breaking the reload banner. An unstamped local build reads
`unknown` and compares `local` — absent rather than empty, one spelling of
"cannot say".

scripts/artifacts.sh joins the AGENT path set in the same commit, and it had
to: a version has no backstop. A revision that is computed differently stops
matching the published label and forces a rebuild, so it self-corrects; a
version is compared against nothing, so a change to cmd_version alone would
leave the agent publishing the old format with nothing to contradict it. That
is #3202's finding, and the agent was rightly exempt only while it had no
version of its own. tests/test_artifact_paths.py pins it.

Also corrects two build.yml comments claiming agent/ had not changed since
2026-07-17. Both were already false — it changed 2026-09-23 — and one of them
is the stated rationale for the force_build escape hatch. Rewritten without
dates: how long an artifact has been quiet is a `git log` question, and its
answer in a comment is wrong the next time anyone commits (lesson #4383).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 07:07:31 -04:00
bvandeusenandClaude Opus 5 970d17f98a fix: a second _seed_seen shadowed the one the recapture tests call (4386)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m13s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m45s
CI and images / smoke-web (push) Successful in 1m6s
CI and images / promote (push) Skipped
Run 7422 integration: two #830 tests died with
`_seed_seen() got an unexpected keyword argument 'post_id'`. Not those tests'
fault — I defined a second `_seed_seen` at the BOTTOM of the module, taking a
list of media instead of one key, and Python's last definition wins for every
call site in the file including the ones 500 lines above it.

Renamed to `_seed_all_seen` and made it loop over the existing one-key helper,
so there is one definition of what seeding the ledger means. The nearby
failure mode is worth naming: a helper defined below the tests that use it is
invisible at the point of use, and shadowing produces a TypeError in code
nobody touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 06:39:49 -04:00
bvandeusenandClaude Opus 5 2b093958a4 feat: a tick keeps looking back 30 days, so an EDITED post is reached (4386)
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 40s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator, 2026-09-23, on a Floppystack post: "this post has been updated as
he implements hot fixes — any chance we have a way to scan for or see updated
posts so we can update ours to match and pull the new attachments and
pictures etc."

The download half already worked: extract_media reads the media list off the
LIVE feed response every walk, so a newly attached hotfix build is a ledger
key we have never seen. Only REACHING the post was missing — a tick stopped
after 20 contiguous already-have-it items, and a post edited three days after
publication sits well below twenty. Not a bug in the early-out; a count
cannot express "recent".

The early-out now needs BOTH conditions: the run of seen items AND a post
published before the horizon. Strictly a widening — window 0 is exactly the
old behaviour, and no window can make a tick stop EARLIER than it used to, so
a source paused for months still walks its whole unseen backlog. The horizon
is a floor on how far to look, never a ceiling.

Inside the window the post-record gate is bypassed too (write_post_record
revisit=True): the body is re-read from the feed response already in hand, so
a revisit costs zero requests, and a body that comes back empty writes
NOTHING rather than blanking one a detail-fetch had filled. Revisits are kept
out of the #862 body-drift canary's sample for the same reason — an empty
revisit is healthy, and counting it would walk the alarm toward firing on
good ticks.

The run summary names what changed ("3 post(s) updated (5 new file(s))") with
a line per post; the ask was to SEE updated posts, not only to end up with
their bytes.

download_revisit_days is a settings row, not a constant (rule 25) — how long
a creator keeps editing is a property of the creator. Default 30, 0 turns it
off. Migration 0108.

Also corrects two stale docstrings: both clients described post_meta as
feeding an Ingester.preview that no longer calls it. It had no consumer at
all until this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 19:21:06 -04:00
bvandeusenandClaude Opus 5 ffbe21098c fix: l is an ambiguous name and ruff says so (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m14s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m56s
CI and images / smoke-web (push) Successful in 1m1s
CI and images / promote (push) Skipped
E741, run 7410. Two comprehensions in the new endpoint tests. Every other
lane was green — backend unit, frontend and integration — and nothing
published, which is the gate doing its job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 18:55:36 -04:00
bvandeusenandClaude Opus 5 45bb7044f7 feat: the System tab reads a stored sample instead of inspecting per load (4295)
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 31s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m44s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5 7f1693a40d fix: the ML dial offered slots the machine had no cores to feed (4295)
CI and images / lint (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-version (push) Successful in 3s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 1m7s
CI and images / promote (push) Skipped
Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in
flight by Little's law, and the daily CCIP sweep dying on its 1800s soft
limit in a numpy matmul. The billiard/pool.py frame in that traceback is
the soft-timeout signal handler, not a pool fault.

Two causes, both mine.

1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile
   `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading
   "keep N_replicas x this within the cores allotted to ML" — a constraint
   stated where nothing could act on it. A large-memory host offered ~49
   slots, the operator took what the dial offered, and the lane asked the
   box for ~200 torch threads.

   The number moves onto the lane as `threads_per_slot`, the embedder
   reads it rather than restating it, and the ceiling is now the smaller
   of the two bounds. They fail differently on purpose: too little memory
   is honestly zero, because the first task would OOM the container; too
   few cores is merely slow, so it floors at one rather than making the
   lane unreachable on a small box.

2. `scheduled_ccip_auto_apply` scored one image per matmul, over every
   image in the library, on every daily run — ~119k products each too
   small to pay for its own BLAS setup. `char_maxima` does the same
   arithmetic in blocks bounded by elements, so its memory stays flat as
   either axis grows.

   Batching changes no arithmetic: a character's score for an image is a
   max over that image's figures AND that character's prototypes, and max
   does not care how it is grouped. Pinned against the old loop written
   out longhand, and against itself with the blocking forced to split
   every row.

The UI copy said the ML ceiling came from memory; it says cores or
memory, whichever runs out first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 16:46:22 -04:00
bvandeusenandClaude Opus 5 1353d346b3 fix: the cap dial waited out a broker round trip it did not need (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m13s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m59s
CI and images / smoke-web (push) Successful in 58s
CI and images / promote (push) Skipped
Operator: "when the number is changed the change should be queued so that
it isn't blocking of the webui or the system itself. we shouldn't have to
wait for the validation live."

Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection
being HELD across them, which is what had been stalling the whole site,
and left the press itself as slow as it was.

1. The store refetched after every write. GET /api/system/workers runs a
   celery inspect on an eleven-second budget, so the stepper stayed
   disabled through a round trip the press did not need. It now patches
   the row from the reply — cap, ceiling, enabled, the three fields that
   reply actually decides — and lets the 15s poll bring the live columns,
   which are measurements it must not invent.

2. The endpoint pushed to the broker before answering. Turning a lane off
   is four cancel_consumer messages; lowering a cap reads the live pool
   first. Now it stores the cap, answers `queued`, and hands the push to a
   Quart background task. Raising a cap was already free and stays free.

Nothing is lost by not waiting: the stored cap is what the system obeys
and the sizing pass re-reads it every minute. That sweep was already the
backstop for a push that failed, which under `no_live_workers` is every
push in the suite.

Also closes a hole the move exposed: the model fetch was gated on the
consumer change having landed, so raising ML off zero while the lane was
restarting stored the cap, let the sizing pass start the consumers a
minute later, and left the lane running with no model — nothing else ever
asks for one. It now fires on the transition and waits in the ml queue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 15:41:32 -04:00
bvandeusenandClaude Opus 5 48108a3569 fix: a lane that is OFF was not attributable to itself (4295)
CI and images / frontend-build (push) Successful in 25s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"clean up the stale service_seen rows"*.

**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.

A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:

1. The lanes table reported the lane **not answering** — the signal for a
   crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
   rendered as a display name — shown "running" beside the real lane's row
   going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
   lane present. ML ships at cap 0, so a fresh install was permanently
   unhealthy and Swarm restarts an unhealthy task forever.

That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.

`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.

The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.

Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 15:21:21 -04:00
bvandeusenandClaude Opus 5 a4c66601db fix: the heartbeat guard grepped its own explanation (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 24s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m9s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-web (push) Successful in 1m48s
CI and images / smoke-web (push) Successful in 55s
CI and images / build-agent (push) Successful in 8m56s
CI and images / promote (push) Skipped
Run 7375: the test that asserts the agent's heartbeat is not gated on holding
leases failed — on the docstring of the fix, which quotes the construct the
fix removed, because that is what a docstring explaining a fix does.

    assert "if ids:" not in loop

A source-TEXT assertion cannot tell code from prose about code. Parsed now:
the function's AST body, unparsed with the docstring node dropped, so the
guard reads only what executes.

Worth stating as the general shape, since this repo writes long explanatory
comments on purpose: any check that greps source for the absence of a pattern
is in tension with documenting why that pattern is gone. Either it excludes
the prose, or the next person to explain the fix breaks the guard that
protects it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 14:59:06 -04:00
175 changed files with 11683 additions and 1350 deletions
+249 -88
View File
@@ -41,10 +41,17 @@ on:
# produce a changelog, not an image.
# The escape hatch for the one thing skip-if-exists makes untestable: a
# build that WOULD be skipped. `agent/` has not changed since 2026-07-17, so
# every push since has correctly declined to build it — which also means the
# agent build path has not run in six weeks and cannot be exercised on
# demand. #3190 lives on exactly that path.
# build that WOULD be skipped. An artifact whose source is quiet has its
# build path correctly declined on every push — which also means that path
# is not exercised, and cannot be exercised on demand, for as long as the
# quiet lasts. #3190 lives on exactly such a path.
#
# This used to name the agent and a date ("unchanged since 2026-07-17"). It
# was already false when read on 2026-09-24 — the agent had changed the day
# before — and a rationale resting on a stale fact reads as settled
# reasoning forever after (lesson #4383). Which artifact is quiet is a
# question for `git log`, not for a comment; the hatch exists because ANY of
# them can be.
#
# Editing build.yml does not force one either, and that is deliberate: the
# workflow is not shipped bytes, so it is in no artifact's path set. Putting
@@ -67,11 +74,18 @@ on:
# The base-image refresh (milestone 326 step 4, #3154).
#
# Skip-if-exists is keyed on OUR source, so an artifact whose source stops
# moving stops picking up base-image updates. `agent/` last changed
# 2026-07-17; every push since has correctly declined to rebuild it, which
# also means it will serve that day's `nvidia/cuda` layers forever. Nothing
# is wrong until it has been unchanged for months, which is precisely why
# this is a calendar trigger and not a condition on the push path.
# moving stops picking up base-image updates — it will serve the base layers
# of its last build forever. The agent is the standing example because its
# base is the heaviest (`nvidia/cuda`, ~6.3 GB) and its source the quietest:
# 48 of 1111 commits touch it, about 4% (#3114).
#
# Deliberately no date here. Nothing is wrong until an artifact has been
# unchanged for months, and "how long has it been" is a `git log` question
# whose answer in a comment is wrong the next time anyone commits — which is
# exactly how the previous version of this paragraph came to claim the agent
# had not moved since 2026-07-17 while it had moved the day before. That is
# precisely why this is a CALENDAR trigger and not a condition on the push
# path: the calendar cannot be wrong about how much time has passed.
#
# Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
# the two are never diagnosing each other, and on the quietest day so a
@@ -228,7 +242,7 @@ jobs:
# rather than left running beside the new mechanism (rule 22).
#
# Two things are still worth asserting, and this is the only lane that can:
# the extension.yml suite runs on node:24-slim, which is exactly why
# the extension-test lane runs on node:24-slim, which is exactly why
# version.spec.js sticks to packaging.sh's git-free subcommands.
# 1. the derivation actually resolves on this commit
# 2. the derived string is one AMO will accept, checked against Mozilla's
@@ -364,6 +378,76 @@ jobs:
- run: npm run test:unit
- run: npm run build
# The extension's lane: web-ext lint, the vitest suite, and the check that
# asks web-ext what it ACTUALLY packaged. Moved in from `extension.yml`
# (deleted) at milestone 429 — the same move `ci.yml` made on 2026-09-23 and
# for the same reason: a separate workflow cannot gate this one, so a red
# extension suite still let `sign-extension` sign and `build-web` ship the
# XPI. It is a lane in THE GATE now. Its old path filter is gone with it; a
# filter here would make the lane skip, and a skipped lane blocks the publish.
#
# The vitest suite is also the JS half of the #3093 artist-pattern mirror —
# the Python half runs in backend-lint-and-test — so until this move one half
# of that guard could fail without stopping anything.
extension-test:
runs-on: python-ci
container:
image: node:24-bookworm-slim
steps:
- uses: actions/checkout@v4
# Not --no-save: vitest and web-ext are both real devDependencies now,
# and the suite needs vitest resolvable from node_modules.
- name: Install dev dependencies
run: cd extension && npm install --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
# Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
# package version-consistency checks. No browser, no network.
- name: Unit tests
run: cd extension && npm run test:unit
# Everything else about packaging is asserted against our own declaration
# of what ships. This is the only check that asks web-ext what it ACTUALLY
# put in the archive. Until now that was an unverified assumption about
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
# only because callers `set -f` first, so losing that quoting would
# silently start shipping dev files with no other signal.
- name: Verify XPI contents
run: |
set -eu
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
cd extension
npm run build
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
echo "=== packaged entries in $ZIP ==="
unzip -Z1 "$ZIP" | sort
echo "=== end ==="
ENTRIES=$(unzip -Z1 "$ZIP")
fail=0
# Must NOT ship: repo infrastructure with no business in a user's browser.
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
if echo "$ENTRIES" | grep -q "^$pat"; then
echo "ERROR: '$pat' was packaged into the XPI but must not be"
fail=1
fi
done
# Must ship: if an exclusion pattern ever over-matches, the extension
# breaks at runtime rather than at build time, so assert presence too.
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js' 'lib/chip.js' 'lib/popup-format.js'; do
if ! echo "$ENTRIES" | grep -q "^$req$"; then
echo "ERROR: '$req' is missing from the XPI"
fail=1
fi
done
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
if ! echo "$ENTRIES" | grep -q "^$dir"; then
echo "ERROR: nothing from '$dir' was packaged"
fail=1
fi
done
[ "$fail" -eq 0 ] || exit 1
echo "XPI contents verified."
# Single integration job — collapsed from a 3-way shard split on 2026-06-04.
# The shards existed to parallelize ~8.5min of integration tests; once the
# throwaway Postgres runs with fsync OFF (the durability step below) the whole
@@ -405,7 +489,7 @@ jobs:
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
image: redis:8-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
@@ -420,7 +504,7 @@ jobs:
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:8-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -507,7 +591,8 @@ jobs:
# itself through a job-level `if:` that could not read `env`, and a design
# where only a FAILED gate blocks would have published unverified images
# while reporting success.
needs: [lint, extension-version, backend-lint-and-test, frontend-build, integration]
needs: [lint, extension-version, backend-lint-and-test, frontend-build, extension-test,
integration]
# A pull_request run is the lanes and nothing else. This is the ONLY thing
# separating "validate a Renovate bump" from "publish a Renovate bump", so
# it is stated on each publishing job rather than inferred from a `needs`
@@ -836,13 +921,16 @@ jobs:
# reuse hit — deliberately NOT folded into `digest`, which the :c-<sha>
# repoint reads and which must keep meaning "what this run built" (#4290).
published_digest: ${{ steps.reuse.outputs.published_digest }}
# Every tag this commit should end up under (channel, plus :c-<sha> on
# main). `promote` writes them from `digest` once the smoke has passed.
tags: ${{ steps.tag.outputs.tags }}
# A plain `needs` — no `always()`. That expression existed to let a
# SKIPPED sign-extension through on a tag push while still blocking a
# FAILED one. With no tag trigger, sign-extension always runs, so the
# default behaviour is exactly what we want: a failed sign skips build-web
# rather than shipping an image without its XPI.
needs: [sign-extension, lint, extension-version, backend-lint-and-test,
frontend-build, integration]
frontend-build, extension-test, integration]
# The lanes in that list are THE GATE (2026-09-23) — see sign-extension's
# copy of this comment for why, including the property that a SKIPPED lane
# blocks as firmly as a failing one. They are repeated here rather than
@@ -1094,11 +1182,13 @@ jobs:
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
# whether the channel then has to be written separately.
#
# On a push the build writes the channel tag directly: the bytes came
# from a commit, and a commit is the thing CI tests. Nothing to hold
# it behind.
# Every build writes a CANDIDATE tag, never the channel (#4310):
# :dev-candidate / :latest-candidate on a push, :refresh-candidate on
# the refresh. `promote` moves the channel only after smoke-web has
# booted the bytes. The lanes prove the SOURCE; only the smoke proves
# the BYTES, and the two are not the same claim.
#
# On the scheduled refresh it writes a CANDIDATE tag instead. A
# The refresh is the case that made this obvious. A
# refresh rebuilds against freshly resolved base images, and the web
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
@@ -1128,7 +1218,10 @@ jobs:
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
# A push builds to a per-channel candidate too (#4310). The channel
# tag moves in `promote`, after smoke-web has booted these bytes —
# so a broken image never reaches the tag deployments follow.
echo "build_ref=$IMAGE:$T-candidate" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
@@ -1352,9 +1445,11 @@ jobs:
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max
# Only the web image carries these: it is the one with a UI and an
# HTTP surface to report them on. The ml and agent images have
# nothing to tell.
# The agent carries its own copy of these (plus FC_REVISION) since
# 2026-09-24. This used to say the agent "has nothing to tell" — true
# of the ml image, which no longer exists (#4311), and never true of
# the agent, which has a control page and a /status endpoint and was
# reporting a hand-written literal on both.
build-args: |
FC_CHANNEL=${{ steps.tag.outputs.channel }}
FC_VERSION=${{ steps.reuse.outputs.version }}
@@ -1402,6 +1497,16 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# A BUILD publishes nothing from here (#4310). Its bytes sit on the
# candidate tag until smoke-web has booted them; `promote` then
# writes the channel tag AND :c-<sha> from this run's digest. Writing
# :c-<sha> here would publish an immutable rollback tag for bytes
# that might then fail the smoke.
if [ -n "${BUILT_DIGEST:-}" ]; then
echo "repoint: built $BUILT_DIGEST this run — promote publishes it"
echo "repoint: after the smoke; nothing to write here."
exit 0
fi
# WHAT WE COPY FROM, which is not what we EXCLUDE (#4290).
#
# This step used to copy from the channel tag by NAME. Nothing
@@ -1495,14 +1600,9 @@ jobs:
# source tree, and not a static inspection: `ffmpeg -version` exiting 0 would
# pass while a codec removal broke every thumbnail in the library.
#
# Refresh-only. On a push the bytes came from a commit, and a commit is what
# the lanes above already test — and since 2026-09-23 they gate the build, so
# those bytes could not exist without them having passed.
#
# Reports a verdict; it does not yet gate the promote (milestone 362 step 4).
# Landing the gate and the thing it gates in one change would mean the first
# time anyone saw this job run would also be the first time it could stop a
# publish.
# It gates `promote` on every trigger (#4310). The lanes above prove the
# source; they cannot see the bytes, and a Dockerfile or base change breaks
# the bytes without touching the source.
smoke-web:
needs: [build-web]
# Every run that actually BUILT something, not just the weekly refresh.
@@ -1514,13 +1614,10 @@ jobs:
# and were smoked when they were built. Re-smoking them would burn two
# minutes to re-learn a fact.
#
# HONEST LIMIT, and it is the reason #4299 exists: on a push this runs
# AFTER build-web has written the channel tag, so it detects rather than
# gates. Rule 164's verify_with asks for the check BETWEEN build and push.
# Closing that needs the push path to adopt the candidate-then-promote
# shape the refresh already has — per-channel candidate tags, promote
# learning its channel, and the :c-<sha> repoint moving after the gate.
# That is a redesign of the production publish path and is its own task.
# This GATES the publish on every trigger (#4310): builds land on a
# candidate tag, and `promote` needs this job, so :dev / :latest and a
# built :c-<sha> move only after it passes. Rule 164's verify_with — the
# check BETWEEN build and push.
#
# NO `if:` — this job always runs (#4323). It used to be gated on the
# build having published something, which skipped it on a reuse hit. That
@@ -1567,7 +1664,7 @@ jobs:
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
image: redis:8-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
@@ -1593,7 +1690,7 @@ jobs:
# in a container against a mounted docker socket, so the services are
# SIBLINGS reachable by IP, not by hostname.
PG=$(docker ps --filter "name=smoke" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:7-alpine" -q | head -n1)
RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:8-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -1938,88 +2035,113 @@ jobs:
# smoked it, and it should not be read as if it were.
promote:
needs: [build-web, build-agent, smoke-web]
# Only a refresh publishes through a candidate; a push writes its channel
# tag directly from the build. Reads the same reuse-step decision the build
# took, via a job output — a job's `if:` cannot see the `env` context.
if: needs.build-web.outputs.candidate == 'true'
# THE PUBLISH (#4310). Both image jobs build to a candidate tag — a push to
# :dev-candidate / :latest-candidate, the weekly refresh to
# :refresh-candidate — and nothing names those bytes under a tag anybody
# deploys until this job runs. It runs only when every job in `needs`
# SUCCEEDED (a job-level `if:` without a status function implies
# success()), so a failed smoke — or a skipped one, which is not the same
# as a passing one (run 5290) — leaves :dev / :latest on the last build
# that worked. Rule 164's verify_with: the check sits BETWEEN build and
# push.
#
# Per image, from the DIGEST that image's job built (#4290): a tag can
# move between the build and this job, a digest cannot. An image whose job
# hit reuse built nothing and has nothing to promote — its build job
# already wrote its :c-<sha> from the channel tag, which "hit" proved
# carries this commit.
if: github.event_name != 'pull_request'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- name: Point the channel tags at the smoked candidates
- name: Point this commit's tags at the smoked builds
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
WEB_DIGEST: ${{ needs.build-web.outputs.digest }}
WEB_TAGS: ${{ needs.build-web.outputs.tags }}
AGENT_DIGEST: ${{ needs.build-agent.outputs.digest }}
AGENT_TAGS: ${{ needs.build-agent.outputs.tags }}
run: |
set -eu
# `latest` is not a guess: a refresh always builds `main` (BUILD_REF),
# and the "must have checked out main" guard in every build job fails
# the run if that did not hold. So the channel is main's.
TAG=latest
FAILED=""
MOVED=0
for NAME in fabledcurator fabledcurator-agent; do
promote() {
NAME="$1"; DIGEST="$2"; TAGS="$3"
REPO="bvandeusen/$NAME"
echo "promote: $REPO"
if [ -z "$DIGEST" ]; then
echo "promote: $NAME — no build this run (reuse hit); nothing to publish"
return 0
fi
echo "promote: $NAME $DIGEST"
# Registry auth is its own token exchange — `docker login`
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
# not hang the run until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the IMAGE manifest media types only. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing this
# whole approach exists to avoid creating.
# types too would let the registry hand back an index, and we would
# faithfully copy the thing this approach exists to avoid creating.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
"https://git.fabledsword.com/v2/$REPO/manifests/$DIGEST" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate $SRC ($CT)"
# NOT `imagetools create`. That wraps its source in an INDEX, and
# `.Image.Config.Labels` does not resolve through one — the
# fc.revision the reuse check reads off the channel tag would come
# back empty, every later push would miss and rebuild, and nothing
# would go red (#3183, run 4751). A manifest PUT is what "make this
# tag name that image" means at the registry: same bytes, same media
# type, same digest, no layer transfer.
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# tag name that image" means at the registry: same bytes, same
# media type, same digest, no layer transfer. (It also makes a
# built :c-<sha> a plain image rather than the index the old
# repoint left.)
IFS=,
for REF in $TAGS; do
TAG="${REF##*:}"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC" ]; then
echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2
FAILED="$FAILED $NAME"
continue
fi
echo "promote: $NAME:$TAG now names $NOW"
done
# Read it back. A PUT that returned 2xx but landed something else
# is exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$DIGEST" ]; then
echo "promote: FAILED — $NAME:$TAG is $NOW, expected $DIGEST" >&2
FAILED="$FAILED $NAME:$TAG"
continue
fi
echo "promote: $NAME:$TAG now names $NOW"
MOVED=$((MOVED + 1))
done
unset IFS
}
promote fabledcurator "$WEB_DIGEST" "$WEB_TAGS"
promote fabledcurator-agent "$AGENT_DIGEST" "$AGENT_TAGS"
if [ -n "$FAILED" ]; then
echo "" >&2
echo "promote: FAILED for:$FAILED" >&2
echo "promote: the channel tags are now INCONSISTENT — some images" >&2
echo "promote: moved and some did not. Re-run this refresh; the" >&2
echo "promote: candidates are still published and the promote is" >&2
echo "promote: idempotent." >&2
echo "promote: the tags are now INCONSISTENT — some moved and some" >&2
echo "promote: did not. Re-run this workflow; the builds are still" >&2
echo "promote: published by digest and the promote is idempotent." >&2
exit 1
fi
echo "promote: both channel tags moved"
echo "promote: $MOVED tag(s) written"
build-agent:
# THE GATE (2026-09-23). Every lane above must have PASSED before this job
# exists at all — so a red suite does not produce an image, let alone push
@@ -2032,7 +2154,14 @@ jobs:
# itself through a job-level `if:` that could not read `env`, and a design
# where only a FAILED gate blocks would have published unverified images
# while reporting success.
needs: [lint, extension-version, backend-lint-and-test, frontend-build, integration]
needs: [lint, extension-version, backend-lint-and-test, frontend-build, extension-test,
integration]
# What `promote` needs to publish this image once the smoke has passed:
# the manifest this run built (empty on a reuse hit) and every tag it
# belongs under. Same meaning as build-web's outputs of the same names.
outputs:
digest: ${{ steps.build.outputs.digest }}
tags: ${{ steps.tag.outputs.tags }}
# A pull_request run is the lanes and nothing else. This is the ONLY thing
# separating "validate a Renovate bump" from "publish a Renovate bump", so
# it is stated on each publishing job rather than inferred from a `needs`
@@ -2200,6 +2329,13 @@ jobs:
set -eu
DERIVED=$(sh scripts/artifacts.sh revision agent)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
# Baked into the agent image as FC_VERSION and reported by its
# control page and /status. A pure function of the revision — same
# commit, same string — so it adds no variability the reuse check
# would have to account for. Added 2026-09-24: the agent's version
# used to be a hand-written literal in app.py that nobody bumped, so
# the September image reported the same string as the July one.
echo "version=$(sh scripts/artifacts.sh version agent)" >> "$GITHUB_OUTPUT"
# The build clock, pinned to the same commit (#3265). Without it
# buildkit stamps the image config with the wall clock of the build,
@@ -2217,11 +2353,13 @@ jobs:
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
# whether the channel then has to be written separately.
#
# On a push the build writes the channel tag directly: the bytes came
# from a commit, and a commit is the thing CI tests. Nothing to hold
# it behind.
# Every build writes a CANDIDATE tag, never the channel (#4310):
# :dev-candidate / :latest-candidate on a push, :refresh-candidate on
# the refresh. `promote` moves the channel only after smoke-web has
# booted the bytes. The lanes prove the SOURCE; only the smoke proves
# the BYTES, and the two are not the same claim.
#
# On the scheduled refresh it writes a CANDIDATE tag instead. A
# The refresh is the case that made this obvious. A
# refresh rebuilds against freshly resolved base images, and the web
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
@@ -2250,7 +2388,10 @@ jobs:
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
# A push builds to a per-channel candidate too (#4310). The channel
# tag moves in `promote`, after smoke-web has booted these bytes —
# so a broken image never reaches the tag deployments follow.
echo "build_ref=$IMAGE:$T-candidate" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
@@ -2353,6 +2494,16 @@ jobs:
# decoration — an unstamped image is one that will always rebuild.
labels: |
fc.revision=${{ steps.reuse.outputs.revision }}
# What the agent reports about ITSELF, as opposed to the label above,
# which is what the registry reports about it. Three values kept
# apart (rule 149). FC_REVISION is the same string as the label, so
# the image and the registry cannot disagree about which commit this
# is — a build whose self-report names a different commit from the
# tag it was published under is unrollbackable in practice.
build-args: |
FC_CHANNEL=${{ steps.tag.outputs.channel }}
FC_VERSION=${{ steps.reuse.outputs.version }}
FC_REVISION=${{ steps.reuse.outputs.revision }}
# LOAD-BEARING, not a preference. On the default docker driver these
# were no-ops; on the docker-container driver above,
# build-push-action@v5 defaults provenance to TRUE when pushing.
@@ -2431,6 +2582,16 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# A BUILD publishes nothing from here (#4310). Its bytes sit on the
# candidate tag until smoke-web has booted them; `promote` then
# writes the channel tag AND :c-<sha> from this run's digest. Writing
# :c-<sha> here would publish an immutable rollback tag for bytes
# that might then fail the smoke.
if [ -n "${BUILT_DIGEST:-}" ]; then
echo "repoint: built $BUILT_DIGEST this run — promote publishes it"
echo "repoint: after the smoke; nothing to write here."
exit 0
fi
# WHAT WE COPY FROM, which is not what we EXCLUDE (#4290).
#
# This step used to copy from the channel tag by NAME. Nothing
-86
View File
@@ -1,86 +0,0 @@
name: extension
# Lint + unit tests. Deliberately NOT a publishing lane, which is why it is
# not part of build.yml's gate. The sign-and-publish dance moved into build.yml's
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
# because sign-extension runs as a build-web dependency in the SAME workflow,
# eliminating the prior race between build.yml and a separate extension.yml.
# Signed XPIs are cached in Forgejo Release Assets named `ext-<version>`.
on:
push:
branches: [dev, main]
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
# test/version.spec.js asserts things ABOUT build.yml — that it does not
# inline the packaged-file set, and that build.yml derives
# the shipped version rather than reading it out of the repo. A
# workflow-only edit can therefore break this suite, so it has to trigger
# it. build.yml joined the list at milestone 271 step 5, when the spec
# started asserting against it.
- '.forgejo/workflows/build.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
- '.forgejo/workflows/build.yml'
workflow_dispatch:
jobs:
lint:
runs-on: python-ci
container:
image: node:24-bookworm-slim
steps:
- uses: actions/checkout@v4
# Not --no-save: vitest and web-ext are both real devDependencies now,
# and the suite needs vitest resolvable from node_modules.
- name: Install dev dependencies
run: cd extension && npm install --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
# Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
# package version-consistency checks. No browser, no network.
- name: Unit tests
run: cd extension && npm run test:unit
# Everything else about packaging is asserted against our own declaration
# of what ships. This is the only check that asks web-ext what it ACTUALLY
# put in the archive. Until now that was an unverified assumption about
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
# only because callers `set -f` first, so losing that quoting would
# silently start shipping dev files with no other signal.
- name: Verify XPI contents
run: |
set -eu
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
cd extension
npm run build
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
echo "=== packaged entries in $ZIP ==="
unzip -Z1 "$ZIP" | sort
echo "=== end ==="
ENTRIES=$(unzip -Z1 "$ZIP")
fail=0
# Must NOT ship: repo infrastructure with no business in a user's browser.
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
if echo "$ENTRIES" | grep -q "^$pat"; then
echo "ERROR: '$pat' was packaged into the XPI but must not be"
fail=1
fi
done
# Must ship: if an exclusion pattern ever over-matches, the extension
# breaks at runtime rather than at build time, so assert presence too.
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
if ! echo "$ENTRIES" | grep -q "^$req$"; then
echo "ERROR: '$req' is missing from the XPI"
fail=1
fi
done
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
if ! echo "$ENTRIES" | grep -q "^$dir"; then
echo "ERROR: nothing from '$dir' was packaged"
fail=1
fi
done
[ "$fail" -eq 0 ] || exit 1
echo "XPI contents verified."
+3 -7
View File
@@ -36,11 +36,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libwebp7 \
libpng16-16 \
ca-certificates \
# opencv-python-headless (via requirements-ml.txt) links these even in its
# headless build. Came from Dockerfile.ml when the images merged
# (milestone 422 step 6).
libgl1 \
libglib2.0-0 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
@@ -61,7 +56,8 @@ RUN pip install -r requirements.txt
#
# torch 2.12.1+cpu wheel 192.3 MB
# torchvision 0.27.1+cpu 1.8 MB
# transformers / onnxruntime / opencv / sklearn and friends
# transformers / onnxruntime / opencv / sklearn and friends (opencv and
# onnxruntime since dropped, #1451 — nothing here imported them)
# 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB
# largest newly-pushed layer 222.07 MB
#
@@ -85,7 +81,7 @@ RUN pip install -r requirements.txt
# CPU-only torch from the PyTorch CPU index. Nothing here uses a GPU — the
# GPU agent is a separate service with its own image.
RUN pip install --index-url https://download.pytorch.org/whl/cpu \
"torch>=2.12,<3.0" "torchvision>=0.27,<0.28"
"torch>=2.14" "torchvision>=0.29"
RUN pip install -r requirements-ml.txt
# Where the model lands. Deliberately NOT a VOLUME instruction: that mints an
+6 -5
View File
@@ -271,11 +271,12 @@ Four deployable pieces, built by `.forgejo/workflows/build.yml`:
## CI / Forgejo setup
Three workflows: `build.yml` (the five verification lanes — lint,
extension-version check, backend unit tests, frontend build, integration — and
then sign + publish), `extension.yml` (extension lint, vitest, XPI content
verification), and `release.yml`, which runs only on a `v*` tag and publishes a
changelog without building anything.
Two workflows that matter here: `build.yml` (the six verification lanes — lint,
extension-version check, backend unit tests, frontend build, extension lint +
vitest + XPI content check, integration — and then sign + publish), and
`release.yml`, which runs only on a `v*` tag and publishes a changelog without
building anything. The extension lane was its own `extension.yml` until
milestone 429, which let a red extension suite sign and ship the XPI anyway.
**The lanes and the publish are one workflow on purpose.** They were two
(`ci.yml` and `build.yml`) until 2026-09-23, on the same push trigger, which
+40 -10
View File
@@ -1,10 +1,21 @@
# FabledCurator GPU agent — runs on the desktop with the GPU.
# CUDA 12.9 + cuDNN 9 runtime so onnxruntime-gpu can use the card (it needs
# cuDNN 9 — the plain -runtime image lacks it: "libcudnn.so.9: cannot open
# shared object file"); ffmpeg for video frames. Ubuntu 24.04 → Python 3.12.
# Stays on the CUDA-12 / cuDNN-9 line the default onnxruntime-gpu + torch are
# built against (CUDA 13 has only nascent ONNX Runtime support).
FROM nvidia/cuda:12.9.2-cudnn-runtime-ubuntu24.04
#
# The `base` flavour, not `cudnn-runtime`: CUDA and cuDNN arrive as the
# `nvidia-*` pip packages torch and onnxruntime-gpu depend on, so the base only
# has to hand the container the driver (it sets NVIDIA_VISIBLE_DEVICES /
# NVIDIA_DRIVER_CAPABILITIES for the Container Toolkit). Until #1451 this was
# `12.9.2-cudnn-runtime` under a `torch==2.6.0+cu124` — and requirements.txt then
# REPLACED that torch with PyPI's CUDA-13 build (ultralytics pulls torchvision,
# which pulls its matching torch), beside a CUDA-13 onnxruntime-gpu. The image
# ran CUDA 13 on a CUDA-12 base, carrying ~3 GB of base libraries and a ~3 GB
# torch nothing loaded: 10 GB compressed.
#
# 13.0 because that is the line both wheels are built for (torch's cu130 index,
# onnxruntime-gpu's `nvidia-cuda-runtime~=13.0`). Needs an NVIDIA driver that
# supports CUDA 13 (580+); fc_agent/accel.py logs at startup whether torch and
# onnxruntime actually got the GPU, since both fall back to the CPU silently.
# ffmpeg for video frames. Ubuntu 24.04 → Python 3.12.
FROM nvidia/cuda:13.0.3-base-ubuntu24.04
# PIP_BREAK_SYSTEM_PACKAGES: Ubuntu 24.04 marks its system Python as externally
# managed (PEP 668), so a global `pip install` errors without this. It's a
@@ -16,10 +27,12 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# torch from the CUDA-12.4 wheel index; its wheels bundle their own CUDA + cuDNN
# so they run on the 12.9 base and coexist with onnxruntime-gpu. Installed first
# + separately so the GPU build of torch is deterministic and layer-cached.
RUN pip3 install --no-cache-dir torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
# torch AND torchvision from the cu130 index, together and first. Installing
# torch alone is what let the next step swap it out: ultralytics needs
# torchvision, PyPI's torchvision pins its own torch, and pip replaced ours to
# match. With both present, requirements.txt finds them satisfied.
RUN pip3 install --no-cache-dir --index-url https://download.pytorch.org/whl/cu130 \
torch torchvision
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
COPY fc_agent ./fc_agent
@@ -27,6 +40,23 @@ COPY fc_agent ./fc_agent
# imgutils ONNX models + the transformers SigLIP weights both cache here; mount
# a volume to persist them across restarts (the SigLIP download is ~3.5 GB once).
ENV HF_HOME=/models
# Declared LAST on purpose, exactly as the web Dockerfile does: an ARG/ENV
# invalidates every layer below it, and these are the only values that differ
# between builds of otherwise identical source. Any earlier and the ~6.3 GB
# CUDA + torch layers could never be shared between the dev and main builds of
# one commit — which is the cost #3114 measured at 9m26s cold.
#
# Three values, never folded together (rule 149) — the NAME a person reads, the
# CHANNEL it came from, and the REVISION that identifies the content. See
# fc_agent/build_info.py; CI derives all three from scripts/artifacts.sh.
ARG FC_CHANNEL=""
ENV FC_CHANNEL=${FC_CHANNEL}
ARG FC_VERSION=""
ENV FC_VERSION=${FC_VERSION}
ARG FC_REVISION=""
ENV FC_REVISION=${FC_REVISION}
EXPOSE 8770
# The control UI; the worker is started from it (or POST /start).
+16 -1
View File
@@ -15,7 +15,22 @@ sudo pacman -S nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# verify:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi
# the header's CUDA version must be 13.0 or later (driver 580+)
```
### After a driver update: regenerate the CDI spec
If the agent's first log lines say `accel: torch is NOT on the GPU` or report
`cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the
toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The
`nvidia-uvm` device number changes between driver versions, and a spec
generated before the update hands the container a device node that no longer
exists (2026-09-24: host `511,0`, container `235,0`). Compare
`ls -l /dev/nvidia-uvm` on the host with the same inside the container, then:
```sh
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
# if your toolkit ships it, this keeps it current on every driver update:
sudo systemctl enable --now nvidia-cdi-refresh.path
```
## 1. Get a token
+124
View File
@@ -0,0 +1,124 @@
"""Which accelerator each runtime actually got — reported once, at startup.
The agent has two GPU runtimes and both fall back to the CPU without raising:
torch when the driver is too old for its CUDA build, and onnxruntime (the imgutils
detector + CCIP models) when its CUDA provider cannot load its libraries. A
fallback shows up only as slower work, and nothing reported it. On 2026-09-24 the
image turned out to be running a CUDA-13 torch and onnxruntime on a CUDA-12 base
(#1451), and whether the ONNX half was on the GPU could not be answered from
anything the agent had ever logged.
Also the fix for the likeliest way the ONNX half misses: onnxruntime-gpu's CUDA
provider finds libcudart/cuBLAS/cuDNN only on the loader path, and in this image
they live in the `nvidia-*` pip packages torch installs. `preload_dlls()` (ORT
1.21+) loads them from there, so the provider resolves them by soname.
Stdlib-only at import, so the unit suite can import it — torch and onnxruntime
are imported inside the functions.
"""
from __future__ import annotations
import ctypes
import importlib
import logging
from pathlib import Path
log = logging.getLogger("fc_agent.accel")
# Filled by report(); /status carries it so the page can show it too.
LAST: dict = {}
def torch_status(imp=importlib.import_module) -> dict:
try:
torch = imp("torch")
except Exception as e:
return {"device": "unavailable", "error": str(e)}
out = {"version": torch.__version__, "cuda_build": torch.version.cuda}
if torch.cuda.is_available():
out["device"] = "cuda"
out["gpu"] = torch.cuda.get_device_name(0)
else:
out["device"] = "cpu"
return out
def onnx_status(imp=importlib.import_module, load=ctypes.CDLL) -> dict:
try:
ort = imp("onnxruntime")
except Exception as e:
return {"device": "unavailable", "error": str(e)}
out = {"version": ort.__version__, "providers": list(ort.get_available_providers())}
if "CUDAExecutionProvider" not in out["providers"]:
out["device"] = "cpu"
return out
preload = getattr(ort, "preload_dlls", None)
if preload is not None:
try:
preload()
except Exception as e:
out["preload_error"] = str(e)
# "Available" only means the build HAS the provider. Loading its library is
# what resolves libcudart/cuBLAS/cuDNN — the step that fails when they are
# missing, and the one a session would otherwise fail silently on.
capi = Path(ort.__file__).parent / "capi"
try:
load(str(capi / "libonnxruntime_providers_shared.so"), mode=ctypes.RTLD_GLOBAL)
load(str(capi / "libonnxruntime_providers_cuda.so"))
except OSError as e:
out["device"] = "cpu"
out["error"] = str(e)
return out
# Loading proves the libraries resolve, NOT that a GPU can be used: on
# 2026-09-24 this reported "onnx on GPU" beside torch failing cuInit with
# "CUDA unknown error" (a driver update awaiting a reboot). Asking the CUDA
# runtime for a device initialises the driver the provider would use.
error = _cuda_device_error(load)
out["device"] = "cpu" if error else "cuda"
if error:
out["error"] = error
return out
def _cuda_device_error(load=ctypes.CDLL) -> str | None:
"""None when the CUDA runtime can reach a device, else why it cannot."""
try:
cudart = load("libcudart.so.13")
except OSError as e:
return str(e)
count = ctypes.c_int(0)
rc = cudart.cudaGetDeviceCount(ctypes.byref(count))
if rc != 0:
cudart.cudaGetErrorString.restype = ctypes.c_char_p
return f"cudaGetDeviceCount: {cudart.cudaGetErrorString(rc).decode()} ({rc})"
return None if count.value > 0 else "no CUDA device visible"
def summary() -> dict | None:
"""The report as FabledCurator stores it: each runtime's device, and why
when it is not the GPU. Sent on every lease and heartbeat, so the System
view can call a running agent that fell back to the CPU "degraded" rather
than "running" — the 2026-09-24 fallback went unseen for weeks because
only this agent's own log said so. None before report() has run."""
if not LAST:
return None
out = {}
for name, s in LAST.items():
entry = {"device": s.get("device")}
if s.get("error"):
entry["error"] = str(s["error"])[:200]
out[name] = entry
return out
def report() -> dict:
"""Check both runtimes, log the result, and keep it for /status."""
LAST.clear()
LAST.update(torch=torch_status(), onnx=onnx_status())
for name, s in LAST.items():
if s.get("device") == "cuda":
log.info("accel: %s on GPU (%s)", name, s)
else:
log.warning("accel: %s is NOT on the GPU — work runs on the CPU (%s)", name, s)
return dict(LAST)
+55 -11
View File
@@ -11,17 +11,22 @@ import logging
from fastapi import FastAPI, Request
from fastapi.responses import HTMLResponse, JSONResponse
from . import logbuf
from . import accel, logbuf
from .build_info import FC_CHANNEL, FC_REVISION, FC_VERSION, build_id, display_version
from .config import Config
from .gpu import read_gpu
from .worker import Worker
log = logging.getLogger("fc_agent.app")
# Bump on every agent change. The page embeds this and /status reports it; the UI
# warns to reload when they differ — so a stale browser-cached page can't be
# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
# DERIVED at image build time, not hand-maintained — see build_info. This was a
# literal an author was asked to bump on every agent change, and the September
# image printed the same "2026-07-17.1" as the July one, so the surface meant to
# answer "did my pull work?" answered the same either way.
#
# Two values with two jobs, kept apart (rule 149): the page SHOWS the version
# and COMPARES the build id. /status reports both, plus the raw fields, so a
# reader never has to take a formatted string apart to get at one of them.
logbuf.install()
cfg = Config.from_env()
@@ -42,6 +47,9 @@ async def _no_store(request, call_next):
@app.on_event("startup")
def _maybe_autostart() -> None:
# Before the worker: the report also preloads the CUDA libraries the ONNX
# models need, and it says in the log which runtimes landed on the GPU.
accel.report()
# With AUTO_START set, a container restart (host reboot, or `restart:
# unless-stopped` after a crash) resumes the worker on its own — the slots
# then ride out a still-down curator via lease backoff. Lets the agent
@@ -52,7 +60,14 @@ def _maybe_autostart() -> None:
@app.get("/", response_class=HTMLResponse)
def index() -> str:
return _PAGE.replace("__BUILD__", VERSION)
# Two substitutions, not one: `__VERSION__` is what a person reads in the
# meta line, `__BUILD_ID__` is what the script compares against /status to
# notice the page is a cached copy from a previous build.
return (
_PAGE
.replace("__VERSION__", display_version())
.replace("__BUILD_ID__", build_id())
)
@app.post("/start")
@@ -117,7 +132,15 @@ def status():
s["fc_url"] = cfg.fc_url
s["configured"] = bool(cfg.token)
s["queue"] = worker.latest_queue()
s["build"] = VERSION
# `build` is the comparison token the page checks — see build_info.
# `version`/`channel`/`revision` ride BESIDE it rather than inside it, so a
# reader wanting the version never has to parse it back out of something
# else. Absent rather than empty when the image carries no stamp.
s["build"] = build_id()
s["version"] = FC_VERSION or None
s["channel"] = FC_CHANNEL or None
s["revision"] = FC_REVISION or None
s["accel"] = accel.LAST or None
return JSONResponse(s)
@@ -169,7 +192,11 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
width:30px;height:32px;font:700 16px system-ui;cursor:pointer}
.step:hover{border-color:var(--acc)}
#conc,#bw{width:3.4rem;height:32px;text-align:center;font:700 16px system-ui;background:#11151a;
color:var(--fg);border:1px solid var(--bd);border-radius:8px}
color:var(--fg);border:1px solid var(--bd);border-radius:8px;appearance:textfield;-moz-appearance:textfield}
/* The browser's own spin arrows, hidden: the − / + beside each field are the
control, styled like the rest of the page (operator, 2026-09-24). */
#conc::-webkit-inner-spin-button,#conc::-webkit-outer-spin-button,
#bw::-webkit-inner-spin-button,#bw::-webkit-outer-spin-button{-webkit-appearance:none;margin:0}
.unit{color:var(--mut);font-size:12px;font-weight:600}
.hint{color:var(--mut);font-size:12px;margin-top:12px}
.tiles{display:grid;grid-template-columns:repeat(6,1fr);gap:8px;margin-bottom:16px}
@@ -203,11 +230,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
<div class=brand><span class=logo>◆</span> FabledCurator <span class=sub>GPU agent</span></div>
<div class=conn><span class="dot" id=dot></span><span id=connlbl>—</span></div>
</header>
<p class=meta>Server <code id=fc>—</code> · token <code id=cfg>—</code> · build <code id=build>__BUILD__</code></p>
<p class=meta>Server <code id=fc>—</code> · token <code id=cfg>—</code> · build <code id=build>__VERSION__</code></p>
<div id=verbanner class=banner style="display:none;background:#3a1212;border-color:#5a1717;color:#ffb3b3">
a newer agent version is running — reload this page (Ctrl+Shift+R) to update the controls
</div>
<div id=accelbanner class=banner style="display:none;background:#3a1212;border-color:#5a1717;color:#ffb3b3"></div>
<div id=banner class=banner style=display:none>
curator unreachable — holding work + retrying, resumes on its own (no restart needed)
</div>
@@ -225,7 +253,9 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
<button class=step onclick=setc(1)>+</button>
</div>
<div class=stepper title="aggregate download cap, downloads + video streams combined — 0 = unlimited">
<button class=step onclick=stepbw(-1)>−</button>
<input id=bw type=number min=0 step=1 value=8 onchange="setbw(this.value)">
<button class=step onclick=stepbw(1)>+</button>
<span class=unit>MB/s</span>
</div>
</div>
@@ -262,7 +292,7 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
</section>
</div>
<script>
const PAGE_BUILD="__BUILD__"
const PAGE_BUILD="__BUILD_ID__"
let CAP=8
// Optimistic transitional state on click, then apply the POST's own status
// response (it returns worker.status()) for instant feedback — don't wait on the
@@ -293,6 +323,14 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
await fetch('/auto',{method:'POST',headers:{'Content-Type':'application/json'},
body:JSON.stringify({value:on})});refresh()
}
function stepbw(d){ setbw((parseFloat(bw.value)||0)+d) }
// Runtimes that did NOT get the GPU, from the startup report. Both fall back
// to the CPU without raising, so this banner and the pill are the only place
// on this page a slow, CPU-bound agent announces itself.
function cpuOnly(s){
const a=s.accel||{}
return Object.keys(a).filter(k=>a[k] && a[k].device!=='cuda')
}
async function setbw(v){
v=Math.max(0,parseFloat(v)||0); bw.value=v
await fetch('/bandwidth',{method:'POST',headers:{'Content-Type':'application/json'},
@@ -363,11 +401,17 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
// unreachable curator; grey when stopped; red with no token.
let dc='dot', lbl='stopped'
if(!ok){ dc='dot red'; lbl='no token' }
else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable' }
else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable'
if(s.queue && cpuOnly(s).length){ dc='dot amber'; lbl='running · CPU only (degraded)' } }
else if(st==='starting'){ dc='dot amber'; lbl='starting…' }
else if(st==='stopping'){ dc='dot amber'; lbl='stopping…' }
dot.className=dc; connlbl.textContent=lbl
banner.style.display=(st==='running' && !s.queue)?'block':'none'
const slow=cpuOnly(s)
accelbanner.style.display=slow.length?'block':'none'
accelbanner.textContent=slow.length?('degraded — '+slow.join(' + ')+' not on the GPU, so that work runs on the CPU: '
+slow.map(k=>k+': '+(s.accel[k].error||s.accel[k].device)).join(' · ')
+'. After a driver update, regenerate the CDI spec (agent README).'):''
queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable'
}
}
+79
View File
@@ -0,0 +1,79 @@
"""What this agent build IS — stamped at image build time, not configurable.
The mirror of `backend/app/build_info.py`, for the same reasons and with the
same posture. Kept as its own module rather than as constants in `app.py`
because it is stdlib-only and therefore importable by the test suite, which
cannot import `app` (torch, transformers and ultralytics are not in the CI
image — see build.yml's "Agent syntax check").
## Why this replaced a hand-written string
`app.VERSION` used to be a literal an author was asked to bump, carrying a
version AND a changelog in one string:
VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle ..."
Nobody bumped it. The September image printed the identical string to the July
one, so the one surface that was supposed to answer *"did my pull work?"*
answered *"2026-07-17.1"* either way. An artifact that cannot identify itself
is worse than one that says nothing, because the stale value reads as an
answer.
The values are now derived by `scripts/artifacts.sh` from the commit its
shipped files last changed in — the same derivation the web image has used
since milestone 313, and the same one the reuse check already ran for the
agent and discarded.
## Three values, never folded together (rule 149)
* `FC_VERSION` — the NAME, `YYYY.MM.DD.HHMM` UTC, derived from COMMIT time.
For people to read and quote. Identical on `dev` and `main` for the same
source, which is the property that makes "am I running the same code as
production?" answerable at a glance.
* `FC_CHANNEL` — a SIBLING field, never a suffix inside the name.
* `FC_REVISION` — the 12-char commit sha, the artifact's IDENTITY. This is
what the reuse check already keys on as the `fc.revision` image label.
**Absent rather than empty when unknown.** A locally built image has no
stamp, and neither does any image predating this module. One spelling of
"cannot say" instead of two.
"""
import os
FC_VERSION = os.environ.get("FC_VERSION", "").strip()
FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
FC_REVISION = os.environ.get("FC_REVISION", "").strip()
def display_version() -> str:
"""The build, as a line for a human: `2026.09.24.1052 (dev)`.
`unknown` rather than a blank when unstamped — an empty slot in the meta
line reads as "no version", which is a different and false claim from "this
build does not carry one".
"""
if not FC_VERSION:
return "unknown"
return f"{FC_VERSION} ({FC_CHANNEL})" if FC_CHANNEL else FC_VERSION
def build_id() -> str:
"""The token the control page compares against `/status` to notice it is
showing a CACHED page from a previous build.
Deliberately NOT `display_version()`. That is the value for reading; this
is the value for deciding, and folding the two is what rule 149 is about.
The revision is the better discriminator of the two — two builds of the
same commit ARE the same agent and should not prompt a reload, and two
different commits always differ here even when they land in the same
minute and derive one version name.
A locally built image falls through to a constant, so the reload banner
cannot fire for it. That is honest rather than a gap: nothing in an
unstamped image knows what source it was built from, and a per-process
nonce would make every ordinary container RESTART claim a new version had
arrived — a false positive on the exact surface the banner exists to keep
trustworthy.
"""
return FC_REVISION or FC_VERSION or "local"
+9 -2
View File
@@ -7,6 +7,8 @@ import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from . import accel
class FcClient:
def __init__(self, base_url: str, token: str, agent_id: str):
@@ -72,7 +74,10 @@ class FcClient:
def lease(self, batch_size: int) -> list[dict]:
r = self.s.post(
f"{self.base}/api/gpu/jobs/lease",
json={"agent_id": self.agent_id, "batch_size": batch_size},
json={
"agent_id": self.agent_id, "batch_size": batch_size,
"accel": accel.summary(),
},
timeout=30,
)
r.raise_for_status()
@@ -90,7 +95,9 @@ class FcClient:
})
def heartbeat(self, job_ids: list[int]) -> None:
self._post_quiet("/api/gpu/jobs/heartbeat", {"job_ids": job_ids})
self._post_quiet(
"/api/gpu/jobs/heartbeat", {"job_ids": job_ids, "accel": accel.summary()},
)
def fail(self, job_id: int, error: str) -> None:
self._post_quiet("/api/gpu/jobs/fail", {"job_id": job_id, "error": error})
+6 -4
View File
@@ -1,10 +1,12 @@
# CCIP + figure detection (ONNX models, auto-downloaded from HuggingFace).
dghs-imgutils>=0.4
# GPU inference for the ONNX models. Swap to onnxruntime (CPU) for a slow
# server-side fallback run.
onnxruntime-gpu
# The crop EMBEDDER (concept bag). torch is installed separately in the
# Dockerfile from the CUDA-12.4 wheel index so the GPU build is deterministic;
# server-side fallback run. The extras declare the CUDA/cuDNN pip packages its
# CUDA provider loads (fc_agent/accel.py preloads them) rather than relying on
# torch happening to install the same ones.
onnxruntime-gpu[cuda,cudnn]
# The crop EMBEDDER (concept bag). torch + torchvision are installed separately
# in the Dockerfile from the cu130 wheel index, so pip never swaps them out;
# transformers loads whatever SigLIP-family model the server announces.
transformers>=4.45
# Crop PROPOSERS — small YOLO detectors (booru_yolo anatomy, COCO person, comic
@@ -0,0 +1,70 @@
"""service_seen — delete the roster rows the fixed code can no longer write.
Operator, 2026-09-23: *"clean up the stale service_seen rows"*. They were not
stale. They were PHANTOMS, written on purpose by code that identified a celery
worker from the queues it was consuming.
A lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list. The roster grouped on that empty set, wrote it under the
key `celery:` and rendered `role_display_name(())` as the display name — a row
called **`Worker ()`**, reported as running, beside the real lane's row going
stale because nothing updated it any more.
`worker_lanes.lane_for_node` fixes the cause: a worker is attributed by its
NODE NAME, which survives having no consumers. Nothing will write `celery:`
again.
## Why a migration and not a retention sweep
Lesson #4202: a guard that refuses to produce a bad value does not undo the
bad value already stored. The row is the thing that has to change.
And it must be deleted rather than aged out, because the roster deliberately
NEVER forgets — *"anything that has run at least once stays listed, that is
what lets a stopped one be noticed rather than simply vanishing"*. A row that
merely goes quiet is exactly what the roster is for. Only a row that cannot
correspond to anything real is safe to remove, and `celery:` is precisely
that: the empty queue set, which no correctly-attributed worker can produce.
## What is deliberately NOT deleted
**Celery rows with a real but unmatched queue set.** A deployment slicing
`CELERY_QUEUES` differently is supported and its rows are true. It is not this
migration's business to decide that somebody else's worker is obsolete.
**Agent rows, including a possible `agent:agent` from a build that omitted
`agent_id`.** Nothing here can tell an abandoned agent id from a second agent
that is currently down, and deleting a real one would hide a genuinely dead
GPU agent — the one thing the roster exists to show. If such a row is present
it needs a person to look at it, not a migration guessing.
Revision ID: 0106
Revises: 0105
Create Date: 2026-09-23
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0106"
down_revision: Union[str, None] = "0105"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Exactly the one key the empty queue set produced. Matched literally
# rather than by a LIKE or a prefix: `celery:` with nothing after it is
# the phantom, and `celery:ml` is a real lane.
op.execute(
sa.text("DELETE FROM service_seen WHERE key = :key").bindparams(key="celery:")
)
def downgrade() -> None:
# Nothing. The row carried no information — an empty queue set and a
# timestamp — and the roster re-learns anything real on its next refresh.
# Re-creating it would put a phantom back.
pass
@@ -0,0 +1,63 @@
"""worker_lane_sample — where the sizing sweep leaves what it measured.
Operator, 2026-09-23, on the System tab: *"there is a repull every time this
page loads — is there a reason this info isn't being tracked in the
background and stored in some way?"*
`/api/system/workers` ran a full celery inspect on every call — four
broadcasts on an eleven-second budget — and the page polls it every fifteen
seconds. `size_worker_lanes` was already inspecting on a timer to decide pool
sizes, computing exactly these numbers and discarding them. This table is
where they land instead, and the endpoint becomes a plain read.
## Why a new table rather than columns on `worker_lane`
`worker_lane` holds the one number an operator sets. Putting a measurement
beside it is the mistake alembic 0105 undid: `slots` sat next to `slots_cap`,
and a measurement next to a preference reads as a second preference.
No backfill. A row appears when the sweep first runs (within its period), and
until then the lane reads as not-yet-measured, which is true.
Revision ID: 0107
Revises: 0106
Create Date: 2026-09-23
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0107"
down_revision: Union[str, None] = "0106"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"worker_lane_sample",
sa.Column("lane", sa.String(length=32), primary_key=True),
# Nullable=False with no server_default: the sweep writes every column
# on every upsert, so a row only ever exists complete.
sa.Column("present", sa.Boolean(), nullable=False),
sa.Column("replicas", sa.Integer(), nullable=False),
# Nullable on purpose — unknown, never zero. A worker that answered
# without reporting its pool, and a queue the broker did not answer
# for, must not be summed as empty.
sa.Column("pool", sa.Integer(), nullable=True),
sa.Column("active", sa.Integer(), nullable=False),
sa.Column("reserved", sa.Integer(), nullable=False),
sa.Column("queue_depth", sa.Integer(), nullable=True),
sa.Column(
"measured_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
)
def downgrade() -> None:
op.drop_table("worker_lane_sample")
@@ -0,0 +1,54 @@
"""download_revisit_days — how far back a tick keeps looking for EDITED posts.
Operator, 2026-09-23, pointing at a Floppystack post: *"this post has been
updated as he implements hot fixes — any chance we have a way to scan for or
see updated posts so we can update ours to match and pull the new attachments
and pictures etc."*
A tick stopped after 20 contiguous already-have-it items. That is the right
instinct and the wrong unit: a post edited three days after publication sits
well below twenty seen items, so the walk turned around before reaching it. The
walk now needs BOTH a run of seen items and a post older than this many days
before it stops.
A settings row rather than a constant (rule 25) because the right window is a
property of the CREATOR, not of FabledCurator — one artist appends hotfix
builds for a fortnight, another never touches a post again. 0 turns the revisit
off entirely and restores the pure count early-out.
30 days is the operator's own number, 2026-09-23.
Revision ID: 0108
Revises: 0107
Create Date: 2026-09-23
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0108"
down_revision: Union[str, None] = "0107"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# server_default so the existing single settings row gets the window without
# a data migration — and so an install that predates this column reads 30
# rather than 0. 0 is a real, meaningful value here (revisit off), so the
# column must never be allowed to arrive at it by omission.
op.add_column(
"import_settings",
sa.Column(
"download_revisit_days",
sa.Integer(),
nullable=False,
server_default="30",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "download_revisit_days")
@@ -0,0 +1,44 @@
"""discord_link_auto — whether FC links a conclusive pair without asking.
Operator, 2026-09-24: *"I don't want this to be manual that defeats the
convenience that I'm going for."*
Confirm-only was right while every signal was circumstantial. Time proximity
and a body that mentions Discord can never be more than suggestive, so asking
was the honest response. A shared working name is different in kind: when the
creator's own name for a piece appears in exactly these two posts and nowhere
else in their library, there is nothing left for the operator to adjudicate,
and asking is just a chore FC invented for them.
Defaults ON, which is a real change of posture and deliberate. It only governs
the conclusive band — weaker evidence still queues — and a link is a row the
operator can dismiss, so the reversal is a click rather than a migration.
Revision ID: 0109
Revises: 0108
Create Date: 2026-09-24
"""
import sqlalchemy as sa
from alembic import op
revision = "0109"
down_revision = "0108"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"import_settings",
sa.Column(
"discord_link_auto",
sa.Boolean(),
nullable=False,
server_default=sa.text("true"),
),
)
def downgrade():
op.drop_column("import_settings", "discord_link_auto")
@@ -0,0 +1,59 @@
"""The unified post card — fold window, family window, and who linked a pair.
Milestone 388, #4402 and #4401. A Patreon teaser's card shows the Discord drop
it announced, and the rest of that piece's variants, by REFERENCE: nothing is
absorbed, nothing changes owner, and every Discord post keeps its own place.
Three columns:
* `import_settings.discord_link_fold_hours` — a linked drop leaves the feed
only when it is this close to its teaser (the same release, shown twice).
* `import_settings.discord_family_window_days` — how far from the teaser the
card reaches for variants. 60 is measured: named families spread up to 44
days on artist 8, every collision found over 500.
* `post_association.linked_by` — "fc" or "operator", so a link FC made by
itself can say so on the card and offer the undo the operator asked for.
Revision ID: 0110
Revises: 0109
Create Date: 2026-09-24
"""
import sqlalchemy as sa
from alembic import op
revision = "0110"
down_revision = "0109"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"import_settings",
sa.Column(
"discord_link_fold_hours",
sa.Float(),
nullable=False,
server_default=sa.text("24"),
),
)
op.add_column(
"import_settings",
sa.Column(
"discord_family_window_days",
sa.Float(),
nullable=False,
server_default=sa.text("60"),
),
)
op.add_column(
"post_association",
sa.Column("linked_by", sa.String(length=16), nullable=True),
)
def downgrade():
op.drop_column("post_association", "linked_by")
op.drop_column("import_settings", "discord_family_window_days")
op.drop_column("import_settings", "discord_link_fold_hours")
@@ -0,0 +1,75 @@
"""Discord native ingester ledgers — seen and dead-letter, per source.
Milestone 428, #4415. Discord moves off gallery-dl onto the native core, which
keeps its memory of what a source has already fetched in these two tables
instead of gallery-dl's archive. Same shape as the SubscribeStar pair.
Revision ID: 0111
Revises: 0110
Create Date: 2026-09-24
"""
import sqlalchemy as sa
from alembic import op
revision = "0111"
down_revision = "0110"
branch_labels = None
depends_on = None
def upgrade():
op.create_table(
"discord_seen_media",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("filehash", sa.String(length=128), nullable=False),
sa.Column("post_id", sa.String(length=64), nullable=True),
sa.Column(
"seen_at", sa.DateTime(timezone=True),
server_default=sa.text("now()"), nullable=False,
),
sa.ForeignKeyConstraint(
["source_id"], ["source.id"],
name=op.f("fk_discord_seen_media_source_id_source"), ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_seen_media")),
sa.UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"),
)
op.create_index(
op.f("ix_discord_seen_media_source_id"), "discord_seen_media", ["source_id"],
)
op.create_table(
"discord_failed_media",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("filehash", sa.String(length=128), nullable=False),
sa.Column("attempts", sa.Integer(), server_default="1", nullable=False),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column(
"first_failed_at", sa.DateTime(timezone=True),
server_default=sa.text("now()"), nullable=False,
),
sa.Column(
"last_failed_at", sa.DateTime(timezone=True),
server_default=sa.text("now()"), nullable=False,
),
sa.ForeignKeyConstraint(
["source_id"], ["source.id"],
name=op.f("fk_discord_failed_media_source_id_source"), ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_failed_media")),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_discord_failed_media_source_id",
),
)
op.create_index(
op.f("ix_discord_failed_media_source_id"), "discord_failed_media", ["source_id"],
)
def downgrade():
op.drop_index(op.f("ix_discord_failed_media_source_id"), table_name="discord_failed_media")
op.drop_table("discord_failed_media")
op.drop_index(op.f("ix_discord_seen_media_source_id"), table_name="discord_seen_media")
op.drop_table("discord_seen_media")
+20 -2
View File
@@ -19,6 +19,7 @@ from ..models import AppSetting
from ..services.extension_service import (
ExtensionService,
InvalidUrlError,
UnknownArtistError,
UnknownPlatformError,
)
from ..services.source_service import KNOWN_PLATFORMS
@@ -87,10 +88,14 @@ async def probe_source():
url = (request.args.get("url") or "").strip()
if not url:
return _bad("invalid_body", detail="url query parameter is required")
from .credentials import _get_crypto
async with get_session() as session:
if not await _ext_key_required(session):
return _bad("unauthorized", status=401)
result = await ExtensionService(session).probe(url)
# crypto lets a Discord probe name the server and channel with the
# stored token; every other platform ignores it.
result = await ExtensionService(session, _get_crypto()).probe(url)
return jsonify(result)
@@ -102,6 +107,15 @@ async def quick_add_source():
url = body.get("url")
if not isinstance(url, str) or not url.strip():
return _bad("invalid_body", detail="url is required")
# Optional: connect the new source to an existing artist (artist_id) or to
# the artist of that name (artist_name). A Discord channel names no
# creator, so the extension's Add panel always sends one of them.
artist_id = body.get("artist_id")
if artist_id is not None and (isinstance(artist_id, bool) or not isinstance(artist_id, int)):
return _bad("invalid_body", detail="artist_id must be an integer")
artist_name = body.get("artist_name")
if artist_name is not None and not isinstance(artist_name, str):
return _bad("invalid_body", detail="artist_name must be a string")
from .credentials import _get_crypto
@@ -111,7 +125,11 @@ async def quick_add_source():
try:
# crypto lets an add resolve the artist's display name via the
# stored credential (else it falls back to the URL handle). #130.
result = await ExtensionService(session, _get_crypto()).quick_add_source(url)
result = await ExtensionService(session, _get_crypto()).quick_add_source(
url, artist_id=artist_id, artist_name=artist_name,
)
except UnknownArtistError as exc:
return _bad("not_found", detail=str(exc), status=404)
except UnknownPlatformError as exc:
return _bad(
"unknown_platform",
+31 -2
View File
@@ -245,6 +245,29 @@ async def errors_recover(image_id: int):
# --- Agent (bearer token): lease / submit / heartbeat / fail ------------
def _accel_detail(body: dict) -> dict:
"""The agent's own report of which runtime got the GPU, kept on its roster
row so the System view can call a CPU-bound agent degraded (#4410).
Only a dict of {runtime: {device, error?}} is kept, and each value is
reduced to those two short strings: this is written on every lease, by a
client the server does not control. An agent that sends nothing (an
older build) simply has no `accel`, which reads as not-yet-reported.
"""
raw = body.get("accel")
if not isinstance(raw, dict):
return {}
accel = {}
for name, entry in list(raw.items())[:4]:
if not isinstance(entry, dict):
continue
clean = {"device": str(entry.get("device") or "")[:16]}
if entry.get("error"):
clean["error"] = str(entry["error"])[:200]
accel[str(name)[:16]] = clean
return {"accel": accel} if accel else {}
@gpu_bp.route("/jobs/lease", methods=["POST"])
async def lease():
body = await request.get_json(silent=True) or {}
@@ -267,7 +290,10 @@ async def lease():
key=f"agent:{agent_id}",
kind="agent",
display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
details={"agent_id": agent_id, "last_call": "lease", "leased": len(jobs)},
details={
"agent_id": agent_id, "last_call": "lease", "leased": len(jobs),
**_accel_detail(body),
},
)
ml = await MLSettings.load(session)
# image rows for url/mime in one shot
@@ -347,7 +373,10 @@ async def heartbeat():
key=f"agent:{agent_id}",
kind="agent",
display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
details={"agent_id": agent_id, "last_call": "heartbeat", "extended": n},
details={
"agent_id": agent_id, "last_call": "heartbeat", "extended": n,
**_accel_detail(body),
},
)
await session.commit()
return jsonify({"extended": n})
+3 -1
View File
@@ -207,6 +207,8 @@ async def rescan_associations():
be within the window to exist at all); this is the button for a first run
over a library that predates the feature."""
async with get_session() as session:
result = await association_rescan(session)
# full=True: the button reaches the whole history, which the hourly
# sweep's 48-hour horizon never does.
result = await association_rescan(session, full=True)
await session.commit()
return jsonify(result)
+20
View File
@@ -36,6 +36,7 @@ _EDITABLE_FIELDS = (
"download_validate_files",
"download_schedule_default_seconds",
"download_event_retention_days",
"download_revisit_days",
"download_failure_warning_threshold",
"series_suggest_enabled",
"series_suggest_threshold",
@@ -43,6 +44,9 @@ _EDITABLE_FIELDS = (
"discord_link_enabled",
"discord_link_threshold",
"discord_link_window_hours",
"discord_link_auto",
"discord_link_fold_hours",
"discord_family_window_days",
"extdl_mega_enabled",
"extdl_gdrive_enabled",
"extdl_mediafire_enabled",
@@ -113,6 +117,12 @@ async def update_import_settings():
v = body["download_schedule_default_seconds"]
if not isinstance(v, int) or isinstance(v, bool) or v < 60 or v > 86400:
return _bad_int("download_schedule_default_seconds", 60, 86400)
# 0 is a real value (revisit off), so the floor is 0, not 1 — and the
# ceiling is a year, past which a "tick" is a backfill wearing a hat.
if "download_revisit_days" in body:
v = body["download_revisit_days"]
if not isinstance(v, int) or isinstance(v, bool) or v < 0 or v > 365:
return _bad_int("download_revisit_days", 0, 365)
if "download_event_retention_days" in body:
v = body["download_event_retention_days"]
if not isinstance(v, int) or isinstance(v, bool) or v < 1 or v > 3650:
@@ -158,6 +168,10 @@ async def update_import_settings():
body["discord_link_enabled"], bool
):
return jsonify({"error": "discord_link_enabled must be a boolean"}), 400
if "discord_link_auto" in body and not isinstance(
body["discord_link_auto"], bool
):
return jsonify({"error": "discord_link_auto must be a boolean"}), 400
if "discord_link_threshold" in body:
v = body["discord_link_threshold"]
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
@@ -170,6 +184,12 @@ async def update_import_settings():
return jsonify(
{"error": "discord_link_window_hours must be a positive number"}
), 400
# Zero is meaningful for both: fold nothing, or reference no variants.
for key in ("discord_link_fold_hours", "discord_family_window_days"):
if key in body:
v = body[key]
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0:
return jsonify({"error": f"{key} must be a number >= 0"}), 400
if "wip_title_tagging_enabled" in body and not isinstance(
body["wip_title_tagging_enabled"], bool
):
+66 -15
View File
@@ -26,7 +26,6 @@ a true statement.
from __future__ import annotations
import asyncio
import logging
import time
from datetime import UTC, datetime
@@ -36,9 +35,7 @@ from sqlalchemy import select, text
from ..config import get_config
from ..extensions import get_session
from ..models import ServiceSeen
from ..services.service_roster import refresh_if_stale
log = logging.getLogger(__name__)
from ..services.worker_lanes import SWEEP_PERIOD_SECONDS
system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system")
@@ -53,14 +50,36 @@ system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system"
STALE_AFTER_SECONDS = 90
DOWN_AFTER_SECONDS = 300
# The celery roster is written by `size_worker_lanes` and by nothing else, so
# these thresholds are only meaningful against ITS cadence. Asserted at import
# rather than left to a reader, because this is precisely the comparison that
# was never made for the GPU agent: its lease poll backed off to 900s while
# the roster called it stopped at 300s, and both numbers were individually
# correct, in different directions, in different files (lesson #4355).
#
# Two clear sweeps before a part is even called STALE. One missed tick is
# routine — the sweep rides the maintenance queue and does an inspect that can
# take eleven seconds — and must not turn the page yellow.
_SWEEPS_BEFORE_STALE = 2
assert STALE_AFTER_SECONDS >= SWEEP_PERIOD_SECONDS * _SWEEPS_BEFORE_STALE, (
f"a {SWEEP_PERIOD_SECONDS}s sweep cannot keep a roster fresh against a "
f"{STALE_AFTER_SECONDS}s stale threshold: raise the threshold or shorten "
f"the sweep"
)
# Probes cross a process boundary, so they carry deadlines. A hung Postgres
# must make this endpoint say "postgres: down", not hang alongside it.
PROBE_TIMEOUT_SECONDS = 2.0
_OK, _STALE, _DOWN, _UNKNOWN = "ok", "stale", "down", "unknown"
# Checking in, but working at a fraction of its speed: a GPU agent whose
# runtimes fell back to the CPU (#4410). Below stale — a part that may have
# stopped is the more urgent question — and above unknown, because this one
# IS known to be wrong.
_DEGRADED = "degraded"
# Worst-first, so an overall verdict is just the max.
_SEVERITY = {_OK: 0, _UNKNOWN: 1, _STALE: 2, _DOWN: 3}
_SEVERITY = {_OK: 0, _UNKNOWN: 1, _DEGRADED: 2, _STALE: 3, _DOWN: 4}
def _age_state(age_seconds: float) -> str:
@@ -86,6 +105,39 @@ def _describe_learned(name: str, state: str, age: float, details: dict) -> str:
return f"{name} has not checked in for {ago} — treat it as stopped"
def _cpu_runtimes(details: dict) -> list[str]:
"""The runtimes an agent reported as NOT on the GPU, with why.
Both torch and onnxruntime fall back to the CPU without raising, so an
agent in that state leases, works and checks in exactly like a healthy
one. On 2026-09-24 one had been doing so since a driver update left a
stale CDI spec; the only sign was a line in the agent's own log.
"""
accel = details.get("accel")
if not isinstance(accel, dict):
return []
out = []
for name, entry in sorted(accel.items()):
if not isinstance(entry, dict) or entry.get("device") == "cuda":
continue
why = entry.get("error") or entry.get("device") or "unknown"
out.append(f"{name} ({why})")
return out
def _learned_state(name: str, state: str, age: float, details: dict) -> tuple[str, str]:
"""A roster row's state and its sentence, degraded included."""
if state == _OK:
cpu = _cpu_runtimes(details)
if cpu:
return _DEGRADED, (
f"{name} is running on the CPU — not on the GPU: {'; '.join(cpu)}. "
"After a driver update, regenerate the agent host's CDI spec "
"(agent README)."
)
return state, _describe_learned(name, state, age, details)
async def _probe_postgres(session) -> dict:
started = time.monotonic()
try:
@@ -151,26 +203,25 @@ async def system_health():
parts.append(pg)
if pg["state"] == _OK:
# Rate-limited inside; see service_roster on why the web process
# is the right observer.
try:
await refresh_if_stale(session)
await session.commit()
except Exception: # noqa: BLE001
log.warning("system health: roster refresh failed", exc_info=True)
# A PURE READ since 2026-09-23. This used to refresh the celery
# roster here, rate-limited to once per 20s — so the roster only
# advanced while somebody had a browser open, and a broadcast rode
# on a request. `size_worker_lanes` writes it now, on a timer, and
# the assertion below is what keeps that cadence honest.
rows = (
await session.execute(select(ServiceSeen).order_by(ServiceSeen.display_name))
).scalars().all()
for row in rows:
age = (now - row.last_seen_at).total_seconds()
state = _age_state(age)
state, detail = _learned_state(
row.display_name, _age_state(age), age, row.details or {},
)
parts.append({
"key": row.key,
"kind": row.kind,
"name": row.display_name,
"state": state,
"detail": _describe_learned(row.display_name, state, age, row.details or {}),
"detail": detail,
"last_seen_at": row.last_seen_at.isoformat(),
"first_seen_at": row.first_seen_at.isoformat(),
**{k: v for k, v in (row.details or {}).items() if k != "agent_id"},
+85 -24
View File
@@ -19,8 +19,9 @@ second into the first would make a read-only module a write one.
from __future__ import annotations
from datetime import UTC, datetime
from functools import partial
from quart import Blueprint, jsonify, request
from quart import Blueprint, current_app, jsonify, request
from ..extensions import get_session
from ..services.worker_control import (
@@ -30,7 +31,12 @@ from ..services.worker_control import (
push_lane_cap,
store_lane_cap,
)
from ..services.worker_lanes import LANES_BY_NAME
from ..services.worker_lanes import (
LANES_BY_NAME,
SWEEP_PERIOD_SECONDS,
Lane,
derived_ceiling,
)
from ._responses import error_response as _bad
workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers")
@@ -42,43 +48,63 @@ async def list_lanes():
Response: {lanes: [...], fetched_at: iso8601}
Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is
the surface an operator watches while dragging a stepper, and a cached
reply would show them the value from before their own change and read as
the control having failed.
One database read, and NO broker call. Operator, 2026-09-23: *"there is a
repull every time this page loads — is there a reason this info isn't
being tracked in the background and stored in some way?"*
It used to inspect the broker here, four broadcasts on an eleven-second
budget, four times a minute per open tab — while `size_worker_lanes` was
already inspecting on a timer and discarding the same numbers. The sweep
stores them now (`worker_lane_sample`) and this reads them.
So the live figures are up to `SWEEP_PERIOD_SECONDS` old, and each lane
carries the `measured_at` that says so. `sweep_period_seconds` is returned
alongside, so the UI can explain the age without hard-coding the cadence
in a second place.
"""
# The session closes BEFORE the broker work. Holding a Postgres connection
# across a celery inspect is what made this page block the whole site —
# see `worker_control.LaneSettings`. This endpoint polls every 15s and the
# inspect budget is 11s, so each poll was pinning a connection for most of
# the interval.
async with get_session() as session:
settings = await lane_settings(session)
lanes = await lane_view(settings)
return jsonify({
"lanes": lanes,
"lanes": lane_view(settings),
"fetched_at": datetime.now(UTC).isoformat(),
"sweep_period_seconds": SWEEP_PERIOD_SECONDS,
})
@workers_bp.route("/<name>", methods=["POST"])
async def update_lane(name: str):
"""Set a lane's cap. Stores it, then makes the live lane obey it.
"""Set a lane's cap. Stores it, answers, and makes the lane follow after.
ONE field, since 2026-09-23. It used to take `slots`, `slots_cap`,
`enabled` and `autoscale`; how many workers are running is now a
measurement the sizing pass owns, and `enabled` is `cap > 0`.
Two failure kinds, deliberately different statuses:
## The reply does not wait for the lane
Operator, 2026-09-23: *"when the number is changed the change should be
queued so that it isn't blocking of the webui or the system itself. we
shouldn't have to wait for the validation live."*
So the request does exactly one thing that can be slow — a row update —
and hands the broker work to a background task. Turning a lane off is
four `cancel_consumer` messages and a resize; lowering a cap is an
`inspect` on an eleven-second budget. Both used to happen between the
click and the response, with the stepper disabled the whole time.
Nothing is lost by not waiting: the cap in the database is what the
system obeys, the sizing pass re-reads it every minute, and the table
polls, so the live columns catch up on their own. If the web process dies
before the background task runs, that sweep is the backstop — which is
the same guarantee the awaited version had, since a push could fail
there too.
Refusals still happen inline, because they are decided from the value and
the machine's ceiling alone and never touch the broker:
* **400** — the value is not allowed (negative, or above what this
container can hold). Nothing was stored. The body carries `detail`,
which is the sentence the UI shows; a refused control with no reason
reads as a bug.
* **200 with `applied: false`** — the value WAS stored but could not be
pushed, because the lane is not currently answering. Not an error: the
sizing pass carries it within a minute, and the UI should say "saved,
not yet live" rather than "that didn't work".
"""
lane = LANES_BY_NAME.get(name)
if lane is None:
@@ -97,13 +123,48 @@ async def update_lane(name: str):
if not isinstance(value, int) or isinstance(value, bool):
return _bad("invalid_body", detail="slots_cap must be an integer")
# Store, close the session, THEN push. Same reason as the GET above, and
# more sharply here: a cap change could do three broker round trips, all
# of them previously with a connection held.
# Store, close the session, THEN hand off. The session must not be held
# across broker work — that is what made this page block the whole site
# (see `worker_control.LaneSettings`) — and now the request does not wait
# for that work either.
async with get_session() as session:
try:
was_cap = await store_lane_cap(session, lane, value)
except LaneUpdateRefused as exc:
return _bad("refused", detail=str(exc))
result = await push_lane_cap(lane, value, was_cap=was_cap)
return jsonify(result)
_schedule_push(lane, value, was_cap)
return jsonify({
"name": lane.name,
"slots_cap": value,
"ceiling": derived_ceiling(lane),
"enabled": value > 0,
# The value is stored; the live lane is being told separately. The UI
# patches its row from this and lets the next poll bring the live
# columns, rather than refetching and paying for an inspect it just
# avoided.
"queued": True,
# Raising the cap off zero is what downloads the model (step 6), and
# the background task does it. Reported here so the UI can say a
# download has started rather than leaving the operator to wonder why
# a lane they just turned on is busy.
"fetching_models": value > 0 and was_cap == 0 and bool(lane.models),
})
def _schedule_push(lane: Lane, slots_cap: int, was_cap: int) -> None:
"""Run the live push after the response has gone out.
A seam, not an abstraction: it is one call, and it exists so the tests can
hold the push still — a background task that outlived a test's patches
would reach the real broker during teardown.
Quart tracks the task on the app and awaits it at shutdown, so an
in-flight push survives a graceful restart. `partial` rather than passing
`was_cap=` through `add_background_task`, so nothing depends on how that
forwards keyword arguments.
"""
current_app.add_background_task(
partial(push_lane_cap, lane, slots_cap, was_cap=was_cap)
)
+77
View File
@@ -0,0 +1,77 @@
"""Celery beat that remembers when each job last ran — from task_run, not a file.
Celery's default PersistentScheduler keeps its memory in a shelve file in the
working directory. Nothing mounts that directory, so every container recreate
forgets it, and a scheduler that remembers nothing seeds every entry with
`last_run_at = now`: each job waits a FULL interval after startup. A daily job
therefore needs 24 hours without a redeploy to fire. Since the one-container
image (172e33d) every redeploy restarts beat, and on 2026-09-24 no daily or
weekly job had run since the 21st (#4408).
task_run already records every task that starts (celery_signals), indexed on
(task_name, started_at DESC), and prune_task_runs keeps the newest row of each
task however old it is. So on startup each entry takes its last_run_at from
there:
- a job that is overdue runs at once;
- a job that is not due waits only the remainder of its interval;
- a job that has never run is due now.
Beat keeps last_run_at in memory from then on, as the default scheduler does;
only the startup seed changes. If the database cannot be read, the entries keep
Celery's own default rather than beat failing to start.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from celery.beat import Scheduler
from sqlalchemy import func, select
log = logging.getLogger(__name__)
# Seed for a job with no recorded run: far enough back that any interval or
# crontab reads as due.
NEVER = datetime(2000, 1, 1, tzinfo=UTC)
def last_runs(session, task_names: list[str]) -> dict[str, datetime]:
"""The latest recorded start of each task, by task name."""
from .models import TaskRun
if not task_names:
return {}
rows = session.execute(
select(TaskRun.task_name, func.max(TaskRun.started_at))
.where(TaskRun.task_name.in_(sorted(set(task_names))))
.group_by(TaskRun.task_name)
).all()
return dict(rows)
def seed(entries, last: dict[str, datetime]) -> None:
"""Set each entry's last_run_at from `last`; a task with none is due now."""
for entry in entries:
entry.last_run_at = last.get(entry.task, NEVER)
class TaskRunScheduler(Scheduler):
"""An in-memory beat seeded from task_run history at startup."""
def setup_schedule(self):
super().setup_schedule()
try:
from .tasks._sync_engine import sync_session_factory
with sync_session_factory()() as session:
last = last_runs(session, [e.task for e in self.schedule.values()])
except Exception:
log.exception("beat: could not read task_run; every job waits a full interval")
return
seed(self.schedule.values(), last)
due = sum(1 for e in self.schedule.values() if e.is_due()[0])
log.info(
"beat: seeded %d job(s) from task_run, %d due now",
len(self.schedule), due,
)
+27 -7
View File
@@ -14,6 +14,7 @@ Queues:
from celery import Celery
from .config import get_config
from .services.worker_lanes import SWEEP_PERIOD_SECONDS
def make_celery() -> Celery:
@@ -61,6 +62,13 @@ def make_celery() -> Celery:
# can never starve the quick self-healing sweeps (operator-flagged
# 2026-06-07: a 2h audit blocked vacuum/backup/normalize for hours).
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
# The one long job in maintenance.py: a whole-library phash
# recompute (35 min hard limit; the library was cleared for
# re-hashing by migration 0098). On the quick lane it held a
# scheduler process for its whole run, and the minute ticks queued
# up behind it (2026-09-24: 7 waiting, "all workers busy for 18
# minutes"). An exact name wins over the glob above.
"backend.app.tasks.maintenance.backfill_phash": {"queue": "maintenance_long"},
"backend.app.tasks.backup.*": {"queue": "maintenance_long"},
"backend.app.tasks.admin.*": {"queue": "maintenance_long"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
@@ -113,7 +121,14 @@ def make_celery() -> Celery:
},
"size-worker-lanes": {
"task": "backend.app.tasks.maintenance.size_worker_lanes",
"schedule": 60.0, # every minute.
"schedule": SWEEP_PERIOD_SECONDS,
#
# The number lives in `services/worker_lanes` because three
# places must agree on it: this schedule, the freshness of the
# sample the System tab reads, and the roster staleness
# thresholds in `api/system_health` — which now depend on this
# sweep rather than on a browser being open, and assert their
# headroom over it at import.
#
# ONE entry, replacing `autoscale-worker-lanes` (60s) and
# `reconcile-worker-lanes` (300s) on 2026-09-23. They were two
@@ -121,14 +136,16 @@ def make_celery() -> Celery:
# existed to stop the reconcile undoing its work; with the
# stored `slots` gone there is nothing to disagree about.
#
# A minute because it reacts to a BACKLOG, and a five-minute
# reaction to a queue filling up is no reaction. It also now
# carries what the reconcile was for — a worker restarted at
# its ENV concurrency is corrected on the next tick rather
# than after five.
# Fast enough to react to a BACKLOG — a five-minute reaction to
# a queue filling up is no reaction. It also carries what the
# reconcile was for: a worker restarted at its ENV concurrency
# is corrected on the next tick rather than after five.
#
# Cheap when settled: one inspect plus one LLEN sweep, and no
# control messages at all once every lane matches.
# control messages at all once every lane matches. It is also
# now the ONLY thing that inspects — nothing on a request path
# does — so this is the whole broker cost of the System tab,
# whether nobody or ten tabs are watching.
},
"cleanup-old-tasks": {
"task": "backend.app.tasks.maintenance.cleanup_old_tasks",
@@ -343,6 +360,9 @@ def make_celery() -> Celery:
},
},
timezone="UTC",
# Beat's memory of when each job last ran comes from task_run, not a
# shelve file nothing persists — see beat_scheduler (#4408).
beat_scheduler="backend.app.beat_scheduler:TaskRunScheduler",
)
# FC-3i: register task_run signal handlers (side-effect import).
from . import celery_signals # noqa: F401
+6
View File
@@ -8,6 +8,8 @@ from .backup_run import BackupRun
from .base import Base
from .character_prototype import CcipPrototypeState, CharacterPrototype
from .credential import Credential
from .discord_failed_media import DiscordFailedMedia
from .discord_seen_media import DiscordSeenMedia
from .download_event import DownloadEvent
from .external_link import ExternalLink
from .gpu_job import GpuJob
@@ -45,6 +47,7 @@ from .tag_positive_confirmation import TagPositiveConfirmation
from .tag_suggestion_rejection import TagSuggestionRejection
from .task_run import TaskRun
from .worker_lane import WorkerLane
from .worker_lane_sample import WorkerLaneSample
__all__ = [
"Base",
@@ -55,6 +58,8 @@ __all__ = [
"BackupRun",
"Source",
"Credential",
"DiscordFailedMedia",
"DiscordSeenMedia",
"PatreonFailedMedia",
"PatreonSeenMedia",
"SubscribeStarFailedMedia",
@@ -96,4 +101,5 @@ __all__ = [
"TagSuggestionRejection",
"TaskRun",
"WorkerLane",
"WorkerLaneSample",
]
@@ -0,0 +1,36 @@
"""DiscordFailedMedia — per-source dead-letter ledger of Discord files that
keep failing to download or validate.
Mirror of SubscribeStarFailedMedia. After `attempts` reaches the dead-letter
threshold a routine walk skips the file (recovery still retries it); a later
clean download clears the row. `filehash` is the seen-ledger's key.
"""
from datetime import datetime
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.types import DateTime
from .base import Base
class DiscordFailedMedia(Base):
__tablename__ = "discord_failed_media"
__table_args__ = (
UniqueConstraint("source_id", "filehash", name="uq_discord_failed_media_source_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
)
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
last_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+34
View File
@@ -0,0 +1,34 @@
"""DiscordSeenMedia — per-source ledger of Discord files already downloaded.
Mirror of SubscribeStarSeenMedia for the native Discord ingester (milestone
428). `filehash` holds the ingester's per-file key, `<message_id>:<media_id>`:
the attachment id, or for an embed a hash of its URL path. Not the file's
position in the message — an edit that removes a file renumbers the rest
(see `discord_client.MediaItem`). The message record's own gate is the
synthetic `message:<id>` key in the same column.
"""
from datetime import datetime
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.types import DateTime
from .base import Base
class DiscordSeenMedia(Base):
__tablename__ = "discord_seen_media"
__table_args__ = (
UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
)
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
seen_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+54
View File
@@ -68,6 +68,16 @@ class ImportSettings(Base):
Integer, nullable=False, default=90,
server_default="90",
)
# How far back a routine tick keeps looking after it has run out of new
# posts, so a creator who EDITS an older post to attach a hotfix build is
# still reached (ingest_core.DEFAULT_REVISIT_DAYS carries the reasoning).
# A knob rather than a constant because how long a creator keeps editing is
# a property of the creator, not of FabledCurator: 0 turns the revisit off
# and restores the pure count early-out.
download_revisit_days: Mapped[int] = mapped_column(
Integer, nullable=False, default=30,
server_default="30",
)
download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5,
server_default="5",
@@ -129,6 +139,50 @@ class ImportSettings(Base):
server_default="24",
)
# Whether FC links a CONCLUSIVE pair without asking.
#
# Operator, 2026-09-24: *"I don't want this to be manual that defeats the
# convenience that I'm going for."* Confirm-only was the right default
# while the only signals were circumstantial — proximity and a body that
# mentions Discord can never be more than suggestive, and asking was the
# honest response to that. A shared working name is different in kind: when
# the name appears in exactly these two posts and nowhere else in the
# artist's library, there is nothing for the operator to adjudicate.
#
# Only the conclusive band is affected (post_association_service.
# AUTO_LINK_FLOOR). Everything weaker still queues, and an accepted link is
# a row the operator can dismiss, so this is reversible in the UI rather
# than only in the database.
discord_link_auto: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True,
server_default="true",
)
# The unified card (#4402). A linked Discord drop is NOT absorbed into its
# teaser — it keeps its own post, date and provenance, and the teaser's card
# shows it by REFERENCE. Operator, 2026-09-24: *"discord 'posts' land as
# normal and only hidden from the post view they're posted the same day."*
#
# So the drop's own card leaves the feed only when it sits within this many
# hours of the teaser that references it — the adjacency that reads as the
# same thing twice. Hours rather than a calendar day: a teaser at 23:00 and
# its drop at 01:00 are one release, and "the same day" has no timezone
# the server can know.
discord_link_fold_hours: Mapped[float] = mapped_column(
Float, nullable=False, default=24.0,
server_default="24",
)
# How far from the teaser the card reaches for the rest of a piece's
# variants — the wips, alts and censor passes a creator trickles out under
# one working name (#4401). Measured on artist 8: named families spread a
# median 5 days and up to 44, while every name collision found spreads
# over 500. A reference, not a regrouping, so a generous value costs one
# extra thumbnail at worst — never a post moved or hidden.
discord_family_window_days: Mapped[float] = mapped_column(
Float, nullable=False, default=60.0,
server_default="60",
)
# #830 off-platform file-host downloads — per-host enable lever (default on,
# rule #26). Column names are extdl_<host>_enabled so the worker reads them
# via getattr(settings, f"extdl_{host}_enabled", True).
+7
View File
@@ -91,6 +91,13 @@ class PostAssociation(Base):
status: Mapped[str] = mapped_column(
String(16), nullable=False, server_default="pending", index=True
)
# WHO linked it: "fc" when the matcher linked a conclusive pair by itself
# (discord_link_auto), "operator" when a person accepted it. The card needs
# this to be honest — a link FC asserted on its own says so and offers an
# undo, which the operator chose over a silent merge (#4402). NULL on a row
# that is not linked, and on rows linked before the column existed, all of
# which an operator accepted: auto-linking shipped in the same release.
linked_by: Mapped[str | None] = mapped_column(String(16), nullable=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+83
View File
@@ -0,0 +1,83 @@
"""worker_lane_sample — the last thing the sizing sweep measured about a lane.
Milestone 422, 2026-09-23. A MEASUREMENT table, deliberately separate from
`worker_lane`, which holds the one number an operator sets.
## Why this exists
Operator, 2026-09-23, looking at the System tab: *"there is a repull every
time this page loads — is there a reason this info isn't being tracked in the
background and stored in some way?"*
There was not a good one. `/api/system/workers` ran a full celery inspect —
four broadcast round trips on an eleven-second budget — on every call, and
the page polls it every fifteen seconds. Meanwhile `size_worker_lanes` was
already inspecting on a timer to decide pool sizes, computing exactly these
numbers, using them, and throwing them away. The browser then asked the
broker for them again.
So the sweep writes what it saw here, and the endpoint reads this table. The
request path makes no broker call at all any more.
## Why NOT columns on `worker_lane`
Because that is the mistake this milestone already made once and undid. That
table used to carry `slots` — how many workers were running — beside
`slots_cap`, and a measurement sitting next to a preference reads as a second
preference: the operator had to keep two numbers in agreement, and the
autoscaler had to be granted permission to move one of them.
The distinction is the whole design, so it is a table boundary. Nothing an
operator sets lives here; nothing here is ever an input to a decision about
what they wanted.
## Freshness is a value, not an assumption
`measured_at` is returned to the UI, which says how old the reading is rather
than implying it is live. A sample is a fact about a moment, and a page that
presents a one-minute-old number as current is how an operator ends up
mistrusting the whole surface.
"""
from datetime import datetime
from sqlalchemy import Boolean, DateTime, Integer, String, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class WorkerLaneSample(Base):
__tablename__ = "worker_lane_sample"
# The lane name from services/worker_lanes.LANES. One row per lane,
# overwritten in place: this is the LATEST reading, not a history. A time
# series would be a different table with a different retention problem,
# and nothing has asked for one.
lane: Mapped[str] = mapped_column(String(32), primary_key=True)
# Whether anything answered for this lane. NOT the same as "zero workers"
# — an unswept absence is not a verdict (snippet #3969). False here means
# the inspect came back without this lane, so every count below is
# meaningless and the UI must say "not answering" rather than "0".
present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
replicas: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Pool size of ONE process, nullable because a worker that answered
# without reporting its pool is unknown rather than empty.
pool: Mapped[int | None] = mapped_column(Integer, nullable=True)
active: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
reserved: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Redis LLEN across the lane's queues. Nullable for the same reason as
# `pool`: a queue the broker did not answer for is unknown, and summing it
# as zero would report a buried lane as idle.
queue_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
measured_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
+12 -4
View File
@@ -121,11 +121,19 @@ def _this_node_ok() -> tuple[bool, str]:
def _lanes_ok() -> tuple[bool, str]:
"""Every lane in the table is answering.
Deliberately ignores whether a lane is ENABLED: a disabled lane still runs
its process with its consumers cancelled, so it answers `inspect` and is
Deliberately ignores whether a lane is ON: a lane at cap 0 still runs its
process with its consumers cancelled, so it answers `inspect` and is
healthy. Health is "is the process alive"; whether it should be consuming
is a settings question the reconcile owns, and conflating them would make
turning a lane off in the UI mark the container unhealthy.
is a settings question the sizing pass owns, and conflating them would
make turning a lane off mark the container unhealthy.
That was not merely a risk — it was happening. Until 2026-09-23 a worker
was attributed to its lane by the queues it was CONSUMING, and a lane with
its consumers cancelled reports none, so it read as absent and this check
failed. ML ships at cap 0, so a fresh install was permanently unhealthy
and Swarm restarts an unhealthy task forever. The docstring above said the
right thing while the code did the opposite; `worker_lanes.lane_for_node`
is what makes it true.
"""
from ..services.worker_control import inspect_lanes_sync
from ..services.worker_lanes import LANES
+22 -7
View File
@@ -316,17 +316,32 @@ class ArtistService:
cleaned = (prefix or "").strip()
if not cleaned:
return []
like = f"%{cleaned.lower()}%"
prefix_like = f"{cleaned.lower()}%"
# Rank: exact (0) < prefix (1) < substring (2).
low = cleaned.lower()
like = f"%{low}%"
prefix_like = f"{low}%"
# Spacing- and punctuation-insensitive too, so "Tamada Heijun" finds
# "TamadaHeijun" and "sabu_art" finds "Sabu Art" — the same creator is
# spelled differently on every platform, and the browser extension's
# Add panel matches a Discord server name against these (milestone 429).
# [[:alnum:]] keeps non-Latin letters; a query with none (all
# punctuation) skips this arm rather than matching every artist.
squashed = "".join(ch for ch in low if ch.isalnum())
name_squashed = func.regexp_replace(func.lower(Artist.name), "[^[:alnum:]]", "", "g")
matches = [func.lower(Artist.name).like(like)]
if squashed:
matches.append(name_squashed.like(f"%{squashed}%"))
# Rank: exact (0) < exact ignoring spacing (1) < prefix (2) <
# substring (3) < substring ignoring spacing (4).
rank = case(
(func.lower(Artist.name) == cleaned.lower(), 0),
(func.lower(Artist.name).like(prefix_like), 1),
else_=2,
(func.lower(Artist.name) == low, 0),
(name_squashed == squashed, 1),
(func.lower(Artist.name).like(prefix_like), 2),
(func.lower(Artist.name).like(like), 3),
else_=4,
).label("rank")
rows = (await self.session.execute(
select(Artist, rank)
.where(func.lower(Artist.name).like(like))
.where(or_(*matches))
.order_by(rank, Artist.name.asc())
.limit(limit)
)).all()
+562
View File
@@ -0,0 +1,562 @@
"""Native Discord read client — the Discord counterpart to subscribestar_client.
Mirrors gallery-dl 1.32.13's `extractor/discord.py` (rule 130: gallery-dl is the
known-working base), adapted to the native core's client contract
(`ingest_core` module docstring): `iter_posts` / `extract_media`, plus the
post-first `post_record_key` and the `post_meta` date the revisit window reads.
What is mirrored exactly, because drift in any of it changes what we fetch or
where it lands on disk:
- API v10, `Authorization: <user token>` (a USER token, not a bot token).
- gallery-dl's request profile: its date-derived Firefox User-Agent,
`Accept: */*`, `Accept-Language`, `Referer: https://discord.com/`.
- `GET /channels/{id}/messages?limit=100&before=<last id>`, newest first,
stopping on a short page. Message types {0, 19, 21} only.
- The walk: a text/news channel's own messages then its threads, a forum's
threads, a category's children, a server's text/news/forum channels.
- Files: attachments, then embeds of type image/gifv/video (FC configures
`embeds: all`, which for files is the same three plus rich/link embeds that
carry an image), then forwarded `message_snapshots`, numbered from 1 across
the lot — the `num` in `{date}_{message_id}_{num}_{filename}`.
- Text: `content`, rich-embed author/title/description/fields/footer, poll.
Two deliberate departures, both about the walk order, neither about content:
- Threads are walked newest-CREATED first (by id), not by last-message time.
A backfill resumes from a checkpointed channel; last-message order shifts
between chunks whenever someone posts, which can move an unwalked thread
above the resume point and skip it. Creation order only ever grows at the
front, where the next tick finds it.
- A 403 on a thread or a nested channel skips that feed instead of failing
the walk. gallery-dl skips only nested channels; one private thread the
token cannot read would otherwise stop every channel after it.
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import hashlib
import logging
import re
import time
from collections.abc import Iterator
from dataclasses import dataclass
from datetime import date
from urllib.parse import unquote
import requests
from .native_ingest_common import (
NativeAuthError,
NativeDriftError,
NativeIngestError,
retry_after_seconds,
)
log = logging.getLogger(__name__)
API_ROOT = "https://discord.com/api/v10"
_ROOT = "https://discord.com"
_TIMEOUT_SECONDS = 60.0
_MESSAGES_BATCH = 100
_THREADS_BATCH = 25
# gallery-dl retries a 429 up to its default 4 retries, waiting
# `request_interval_429` (60s) between them. Discord's Retry-After is exact, so
# it is honoured when present; 60s is the fallback and the cap.
_MAX_429_RETRIES = 4
_429_WAIT_SECONDS = 60.0
# https://discord.com/developers/docs/resources/message#message-object-message-types
# DEFAULT, REPLY, CHAT_INPUT_COMMAND — the ones that carry user content.
MESSAGE_TYPES = frozenset({0, 19, 21})
# https://discord.com/developers/docs/resources/channel#channel-object-channel-types
_TEXT = frozenset({0, 5}) # text, announcement: messages + threads
_DIRECT = frozenset({1, 3, 10, 11, 12}) # DMs and threads: messages only
_FORUM = frozenset({15, 16}) # forum, media: threads only
_CATEGORY = 4
_SERVER_WALK = _TEXT | _FORUM
_EMBED_TYPES = frozenset({"image", "gifv", "video"})
_URL_RE = re.compile(
r"^(?:https?://)?(?:www\.|ptb\.|canary\.)?discord(?:app)?\.com/channels/"
r"(?P<server>@me|\d+)(?:/(?:\d+/threads/)?(?P<channel>\d+))?(?P<rest>/.*)?/?$"
)
class DiscordAPIError(NativeIngestError):
"""Base for native Discord client failures."""
class DiscordAuthError(DiscordAPIError, NativeAuthError):
"""401 (the token is invalid or expired) or a 403 on the channel the
source names. The fix is a new token, not a new client."""
class DiscordDriftError(DiscordAPIError, NativeDriftError):
"""A response did not have the shape the walk depends on."""
def firefox_user_agent(today: date | None = None) -> str:
"""gallery-dl's default User-Agent: a Firefox whose version advances every
four weeks (`util._ff_ver`, "147 on 2026-01-13"). Computed the same way so
the profile keeps matching the gallery-dl this replaced."""
ver = ((today or date.today()).toordinal() - 735_513) // 28
return (
f"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:{ver}.0) "
f"Gecko/20100101 Firefox/{ver}.0"
)
def nameext_from_url(url: str) -> tuple[str, str]:
"""gallery-dl's `text.nameext_from_url`: the URL's last path segment,
unquoted, split at the last dot when the extension is at most 16 chars
(lowercased); otherwise the whole name and no extension."""
filename = unquote(url.partition("?")[0].rpartition("/")[2])
name, _, ext = filename.rpartition(".")
if name and len(ext) <= 16:
return name, ext.lower()
return filename, ""
def parse_source_url(url: str) -> tuple[str | None, str | None]:
"""`(server_id, channel_id)` from a Discord channel/server URL. `server_id`
is None for a DM (`@me`); `channel_id` is None for a whole server. Raises
DiscordAPIError for anything else, including a link to a single message —
a message is not something a source can subscribe to."""
m = _URL_RE.match((url or "").strip())
if not m or (m.group("rest") or "").strip("/"):
raise DiscordAPIError(
f"Not a Discord channel or server link: {url!r} "
"(expected https://discord.com/channels/<server>[/<channel>])"
)
server = m.group("server")
channel = m.group("channel")
if server == "@me":
if not channel:
raise DiscordAPIError(f"A DM link needs a channel id: {url!r}")
return None, channel
return server, channel
def message_text(message: dict) -> str:
"""gallery-dl's `extract_message_text`: the body plus the text of rich
embeds and polls, newline-joined, empties dropped."""
parts = [message.get("content") or ""]
for embed in message.get("embeds") or []:
if embed.get("type") != "rich":
continue
parts.append((embed.get("author") or {}).get("name") or "")
parts.append(embed.get("title") or "")
parts.append(embed.get("description") or "")
for fld in embed.get("fields") or []:
parts.append(fld.get("name") or "")
parts.append(fld.get("value") or "")
parts.append((embed.get("footer") or {}).get("text") or "")
poll = message.get("poll")
if poll:
parts.append(((poll.get("question") or {}).get("text")) or "")
for answer in poll.get("answers") or []:
parts.append(((answer.get("poll_media") or {}).get("text")) or "")
return "\n".join(p for p in parts if p)
@dataclass
class MediaItem:
"""One file of a Discord message. `filename`/`extension` are gallery-dl's
split of the URL; `num` is its 1-based position across the message's files,
which is what names it on disk.
`media_id` is what the seen-ledger keys on, and it is deliberately NOT
`num`: an edit that removes a file renumbers the ones after it, and a
positional key would then call a different file seen. It is the
attachment's id, or for an embed (which has none) a hash of its URL path —
the query string is a signature that changes on every fetch. `filehash` is
always None; nothing in a signed CDN URL is a content hash."""
url: str
filename: str
extension: str
kind: str
post_id: str
num: int
media_id: str
filehash: str | None = None
class DiscordClient:
"""Synchronous Discord API v10 read client for one user token."""
def __init__(
self,
token: str | None,
*,
request_sleep: float = 0.0,
max_retries: int = _MAX_429_RETRIES,
session: requests.Session | None = None,
):
self._session = session or requests.Session()
self._session.headers.update({
"User-Agent": firefox_user_agent(),
"Accept": "*/*",
"Accept-Language": "en-US,en;q=0.5",
"Referer": _ROOT + "/",
})
if token:
self._session.headers["Authorization"] = token
self._token = token
self._request_sleep = request_sleep or 0.0
self._max_retries = max_retries
self._server: dict = {}
self._channels: dict[str, dict] = {}
self._skip_feed = False
# -- request -----------------------------------------------------------
def _get(self, endpoint: str, params: dict | None = None):
if not self._token:
raise DiscordAuthError("No Discord token is configured for this source")
if self._request_sleep > 0:
time.sleep(self._request_sleep)
url = API_ROOT + endpoint
attempt = 0
while True:
try:
resp = self._session.get(url, params=params, timeout=_TIMEOUT_SECONDS)
except requests.RequestException as exc:
raise DiscordAPIError(f"Discord request failed ({endpoint}): {exc}") from exc
if resp.status_code == 429 and attempt < self._max_retries:
attempt += 1
delay = retry_after_seconds(
resp, attempt, base=_429_WAIT_SECONDS, cap=_429_WAIT_SECONDS,
)
log.warning(
"Discord 429 (%s) — waiting %.1fs (retry %d/%d)",
endpoint, delay, attempt, self._max_retries,
)
time.sleep(delay)
continue
break
if resp.status_code == 401:
raise DiscordAuthError(
"Discord rejected the token (HTTP 401) — it is invalid or has "
"expired; copy a fresh one from the browser",
status_code=401,
)
if resp.status_code != 200:
raise DiscordAPIError(
f"Discord returned HTTP {resp.status_code} ({endpoint})",
status_code=resp.status_code,
retry_after=_retry_after(resp),
)
try:
return resp.json()
except ValueError as exc:
raise DiscordDriftError(
f"Discord returned non-JSON for {endpoint} ({len(resp.content)} bytes)"
) from exc
# -- metadata (gallery-dl parse_server / parse_channel) -----------------
def _load_server(self, server_id: str) -> None:
server = self._get(f"/guilds/{server_id}")
if not isinstance(server, dict) or "id" not in server:
raise DiscordDriftError(f"Discord server {server_id} came back without an id")
self._server = {
"server": server.get("name") or "",
"server_id": str(server["id"]),
"owner_id": server.get("owner_id"),
}
channels = self._get(f"/guilds/{server_id}/channels")
if not isinstance(channels, list):
raise DiscordDriftError(f"Discord server {server_id} channel list is not a list")
# Categories first, so every child can name its parent.
for channel in sorted(channels, key=lambda ch: ch.get("type") != _CATEGORY):
self._parse_channel(channel)
def _parse_channel(self, channel: dict) -> dict:
parent_id = channel.get("parent_id")
meta = {
"channel": channel.get("name") or "",
"channel_id": str(channel.get("id")),
"channel_type": channel.get("type"),
"channel_topic": channel.get("topic") or "",
"parent_id": parent_id,
"is_thread": "thread_metadata" in channel,
}
parent = self._channels.get(parent_id) if parent_id else None
if parent:
meta["parent"] = parent["channel"]
meta["parent_type"] = parent["channel_type"]
if meta["channel_type"] in {1, 3}:
recipients = channel.get("recipients") or []
meta["channel"] = "DMs"
meta["recipients"] = [u.get("username") for u in recipients]
meta["recipients_id"] = [u.get("id") for u in recipients]
self._channels[meta["channel_id"]] = meta
return meta
def _channel_meta(self, channel_id: str) -> dict:
if channel_id not in self._channels:
self._parse_channel(self._get(f"/channels/{channel_id}"))
return self._channels[channel_id]
def _threads(self, channel_id: str) -> list[dict]:
"""Every thread of a channel or forum, newest-created first (see the
module docstring for why not last-message order)."""
threads: list[dict] = []
offset = 0
while True:
data = self._get(f"/channels/{channel_id}/threads/search", {
"sort_by": "last_message_time",
"sort_order": "desc",
"limit": _THREADS_BATCH,
"offset": offset,
})
batch = (data.get("threads") or []) if isinstance(data, dict) else []
threads.extend(batch)
if len(batch) < _THREADS_BATCH:
break
offset += len(batch)
threads.sort(key=lambda t: int(t.get("id") or 0), reverse=True)
return threads
# -- the walk ------------------------------------------------------------
def _feeds(self, channel_id: str, *, safe: bool) -> Iterator[tuple[str, bool]]:
"""`(channel_id, safe)` for every message feed under `channel_id`, in
gallery-dl's order. `safe` feeds are skipped on a 403."""
try:
ctype = self._channel_meta(channel_id)["channel_type"]
except DiscordAPIError as exc:
if exc.status_code != 403:
raise
if not safe:
raise DiscordAuthError(
f"The Discord token cannot see channel {channel_id} (HTTP 403)",
status_code=403,
) from exc
log.info("Discord: no access to channel %s — skipped", channel_id)
return
if ctype in _TEXT or ctype in _DIRECT:
yield channel_id, safe
if ctype in _TEXT or ctype in _FORUM:
try:
threads = self._threads(channel_id)
except DiscordAPIError as exc:
if exc.status_code != 403:
raise
log.info("Discord: cannot list threads of %s — skipped", channel_id)
threads = []
for thread in threads:
yield self._parse_channel(thread)["channel_id"], True
elif ctype == _CATEGORY:
for child in list(self._channels.values()):
if child.get("parent_id") == channel_id:
yield from self._feeds(child["channel_id"], safe=True)
elif ctype not in _DIRECT and not safe:
raise DiscordAPIError(
f"Discord channel {channel_id} is of type {ctype}, which has no messages"
)
def _source_feeds(self, url: str) -> Iterator[tuple[str, bool]]:
server_id, channel_id = parse_source_url(url)
self._server, self._channels = {}, {}
if server_id is not None:
self._load_server(server_id)
if channel_id is not None:
yield from self._feeds(channel_id, safe=False)
return
for meta in list(self._channels.values()):
if meta["channel_type"] in _SERVER_WALK:
yield from self._feeds(meta["channel_id"], safe=True)
def skip_feed(self) -> None:
"""Optional core seam (#4413): end the current channel and go on to the
next one. A tick's early-out means THIS channel has nothing new, not
that the server has nothing new."""
self._skip_feed = True
def iter_posts(
self, campaign_id: str, cursor: str | None = None
) -> Iterator[tuple[dict, dict, str | None]]:
"""Yield `(message, channel_meta, page_cursor)` for every content
message the source reaches, channel by channel, each newest first.
`campaign_id` is the source URL. The cursor is `<channel_id>:<before>`
— the channel and the `before` id that fetched the page (empty for a
channel's first page) — so a backfill resumes inside the right channel
and re-fetches the page it was cut in. A cursor naming a channel the
walk no longer reaches (a deleted thread) restarts from the top rather
than walking nothing.
"""
resume_channel, _, resume_before = (cursor or "").partition(":")
resuming = bool(resume_channel)
feeds = list(self._source_feeds(campaign_id)) if resuming else None
if feeds is not None and resume_channel not in {cid for cid, _ in feeds}:
log.warning(
"Discord: resume channel %s is no longer in %s — restarting",
resume_channel, campaign_id,
)
resuming = False
for channel_id, safe in feeds if feeds is not None else self._source_feeds(campaign_id):
before = None
if resuming:
if channel_id != resume_channel:
continue
resuming = False
before = resume_before or None
yield from self._iter_channel(channel_id, before, safe=safe)
def _iter_channel(
self, channel_id: str, before: str | None, *, safe: bool
) -> Iterator[tuple[dict, dict, str | None]]:
self._skip_feed = False
meta = {**self._server, **self._channels.get(channel_id, {})}
while True:
page_cursor = f"{channel_id}:{before or ''}"
try:
messages = self._get(
f"/channels/{channel_id}/messages",
{"limit": _MESSAGES_BATCH, "before": before},
)
except DiscordAPIError as exc:
if exc.status_code != 403:
raise
if not safe:
raise DiscordAuthError(
f"The Discord token cannot read channel {channel_id} (HTTP 403)",
status_code=403,
) from exc
log.info("Discord: no access to messages of %s — skipped", channel_id)
return
if not isinstance(messages, list):
raise DiscordDriftError(
f"Discord messages of {channel_id} came back as "
f"{type(messages).__name__}, not a list"
)
for message in messages:
if message.get("type") not in MESSAGE_TYPES:
continue
message["_meta"] = meta
yield message, meta, page_cursor
if self._skip_feed:
return
if len(messages) < _MESSAGES_BATCH:
return
before = str(messages[-1]["id"])
# -- per-message -------------------------------------------------------
@staticmethod
def extract_media(post: dict, included: dict | None = None) -> list[MediaItem]:
"""gallery-dl's file list for one message: attachments, then the first
of video/image/thumbnail `proxy_url` of each file-bearing embed, then
the same for every forwarded snapshot; numbered from 1 across them."""
mid = str(post.get("id") or "")
snapshots = [post] + [
(s or {}).get("message") or {}
for s in post.get("message_snapshots") or []
if ((s or {}).get("message") or {}).get("type", 0) in MESSAGE_TYPES
]
found: list[tuple[str, str, str | None]] = []
for snap in snapshots:
for att in snap.get("attachments") or []:
if att.get("url"):
aid = att.get("id")
found.append((att["url"], "attachment", str(aid) if aid else None))
for embed in snap.get("embeds") or []:
if embed.get("type") not in _EMBED_TYPES:
continue
for fld in ("video", "image", "thumbnail"):
url = (embed.get(fld) or {}).get("proxy_url")
if url:
found.append((url, "embed", None))
break
items = []
for num, (url, kind, fid) in enumerate(found, start=1):
name, ext = nameext_from_url(url)
if fid is None:
path = url.partition("?")[0].encode()
fid = "u" + hashlib.sha1(path, usedforsecurity=False).hexdigest()[:32]
items.append(MediaItem(
url=url, filename=name, extension=ext, kind=kind, post_id=mid,
num=num, media_id=fid,
))
return items
@staticmethod
def post_meta(post: dict) -> dict:
"""No title (Discord has none); `date` is the message timestamp, ISO
with an offset — what the core's revisit window reads."""
return {"title": None, "date": post.get("timestamp")}
@classmethod
def post_record_key(cls, post: dict) -> tuple[str, str] | None:
"""`(message:<id>, <id>)` — gates the message record through the seen
ledger, like `post:<id>` on the other platforms.
None for a message with no files. gallery-dl wrote a sidecar only
beside a file, so a text-only chat line never became a post, and the
drop grouping (discord_grouping) is built on that: a channel's chatter
recorded as posts would bury the drops it exists to surface."""
mid = post.get("id")
mid = str(mid) if mid is not None else ""
if not mid or not cls.extract_media(post):
return None
return (f"message:{mid}", mid)
# -- verify ------------------------------------------------------------
def describe(self, server_id: str | None, channel_id: str | None) -> dict:
"""The display names behind a server/channel pair, for the browser
extension's Add panel. Best-effort per name: one that can't be read
comes back None, and the other is still returned."""
out: dict = {"server": None, "channel": None, "parent": None}
if server_id:
try:
out["server"] = (self._get(f"/guilds/{server_id}") or {}).get("name") or None
except DiscordAPIError:
pass
if channel_id:
try:
meta = self._parse_channel(self._get(f"/channels/{channel_id}"))
out["channel"] = meta.get("channel") or None
except (DiscordAPIError, AttributeError):
pass
return out
def verify_auth(self, url: str) -> tuple[bool | None, str]:
"""Is the token valid, and can it see what the source names?"""
try:
server_id, channel_id = parse_source_url(url)
except DiscordAPIError as exc:
return None, str(exc)
try:
me = self._get("/users/@me")
if channel_id is not None:
self._get(f"/channels/{channel_id}")
elif server_id is not None:
self._get(f"/guilds/{server_id}")
except DiscordAuthError as exc:
return False, f"Discord rejected the token — {exc}"
except DiscordAPIError as exc:
if exc.status_code in (403, 404):
return False, (
"The token is valid, but its account cannot see "
f"{'this channel' if channel_id else 'this server'} "
f"(HTTP {exc.status_code})"
)
return None, f"Couldn't verify (network/HTTP issue): {exc}"
who = (me or {}).get("username") if isinstance(me, dict) else None
return True, f"Token valid{f' ({who})' if who else ''} — the source is readable."
def _retry_after(resp: requests.Response) -> float | None:
hdr = resp.headers.get("Retry-After")
try:
return float(hdr) if hdr else None
except (TypeError, ValueError):
return None
+212
View File
@@ -0,0 +1,212 @@
"""Native Discord media downloader — the Discord counterpart to
subscribestar_downloader.
Writes files exactly where gallery-dl wrote them, so a cutover finds every
existing file on disk (`skipped_disk`) instead of fetching it again:
<images>/<artist>/discord/<channel>/<YYYYMMDD>_<message_id>_<NN>_<name>.<ext>
That is what FC's gallery-dl config produced (directory `{channel}`, filename
`{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, under the
per-source base directory `<images>/<artist>/<platform>`), retired from that
config once Discord moved here; tests/test_discord_naming.py pins the match
against a real gallery-dl sidecar. The name is cleaned the way gallery-dl cleans it
on Linux — `/` becomes `_` and control characters are removed, nothing else
(`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`,
whose Windows set would turn a `:` in a channel or file name into `_` and miss
the file gallery-dl wrote.
Post-first (rule 120): each file gets a minimal sidecar named like it minus the
extension (what `find_sidecar` pairs first), and the message itself gets one
record, `<YYYYMMDD>_<message_id>_post.json`, carrying gallery-dl's metadata keys
— `message_id` for the post id, `server_id`/`channel_id` for the permalink,
`message` for the body, `date` — so `parse_sidecar` reads it exactly as it read
the gallery-dl sidecars. Neither file carries an `id` or `post_id` key: both
outrank `message_id` in the post-id chain (`platforms.base`).
PURE: no DB; the seen-skip is an injected predicate.
"""
from __future__ import annotations
import json
import logging
import re
import time
from collections.abc import Callable
from datetime import UTC, datetime
from pathlib import Path
import requests
from .discord_client import firefox_user_agent, message_text
from .native_ingest_common import (
BaseNativeDownloader,
MediaOutcome,
PostRecordOutcome,
make_session,
)
log = logging.getLogger(__name__)
PLATFORM = "discord"
_CONTROL = re.compile("[\x00-\x1f\x7f]")
# gallery-dl falls back to the response's type for a URL with no extension;
# we never see the response before naming, and such URLs do not occur for
# Discord attachments or embed proxies in practice.
_NO_EXTENSION = "bin"
def gdl_clean(segment: str) -> str:
"""One path segment as gallery-dl writes it on Linux."""
return _CONTROL.sub("", segment.replace("/", "_"))
def message_date(post: dict) -> datetime | None:
raw = post.get("timestamp")
if not isinstance(raw, str) or not raw:
return None
try:
dt = datetime.fromisoformat(raw.replace("Z", "+00:00"))
except ValueError:
return None
return (dt if dt.tzinfo else dt.replace(tzinfo=UTC)).astimezone(UTC)
def channel_dir(images_root: Path, artist_slug: str, post: dict) -> Path:
"""gallery-dl's `{channel}` directory; an empty name adds no segment."""
base = Path(images_root) / artist_slug / PLATFORM
channel = gdl_clean(((post.get("_meta") or {}).get("channel") or "").strip())
return base / channel if channel else base
def media_stem(post: dict, media) -> str:
"""`<YYYYMMDD>_<message_id>_<NN>_<name>` — the file's name minus `.<ext>`."""
when = message_date(post)
day = f"{when:%Y%m%d}" if when else "None"
return gdl_clean(f"{day}_{post.get('id')}_{media.num:>02}_{media.filename}")
class DiscordDownloader(BaseNativeDownloader):
"""Download a message's files to gallery-dl's layout. The CDN gets
gallery-dl's browser profile and no token — gallery-dl sends the token only
to the API, and the CDN URLs are pre-signed."""
def __init__(
self,
images_root: Path,
cookies_path: str | None = None,
*,
validate: bool = True,
rate_limit: float = 0.0,
session: requests.Session | None = None,
):
super().__init__(
images_root, None, platform=PLATFORM,
validate=validate, rate_limit=rate_limit,
session=session if session is not None else make_session(None, extra_headers={
"User-Agent": firefox_user_agent(),
"Accept-Language": "en-US,en;q=0.5",
"Referer": "https://discord.com/",
}),
)
def download_post(
self,
post: dict,
media_items: list,
artist_slug: str,
*,
is_seen: Callable[[object], bool] = lambda m: False,
should_stop: Callable[[], bool] = lambda: False,
recapture: bool = False,
) -> list[MediaOutcome]:
"""Every file of one message; per-file outcomes, one failure isolated."""
folder = channel_dir(self.images_root, artist_slug, post)
outcomes: list[MediaOutcome] = []
for media in media_items:
if should_stop():
break
try:
outcomes.append(self._download_one(
post, media, folder, artist_slug, is_seen, recapture=recapture,
))
except Exception as exc: # resilient: isolate one item's failure
log.warning(
"Discord media failed (message %s, file %d): %s",
post.get("id"), media.num, exc,
)
outcomes.append(
MediaOutcome(media=media, status="error", path=None, error=str(exc))
)
return outcomes
def _download_one(
self,
post: dict,
media,
folder: Path,
artist_slug: str,
is_seen: Callable[[object], bool],
*,
recapture: bool = False,
) -> MediaOutcome:
seen = is_seen(media)
if seen and not recapture:
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
stem = media_stem(post, media)
path = folder / f"{stem}.{media.extension or _NO_EXTENSION}"
if path.exists(): # tier-2: gallery-dl (or an earlier walk) wrote it
return MediaOutcome(media=media, status="skipped_disk", path=path, error=None)
if seen: # recapture never re-fetches a seen file that is gone
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
folder.mkdir(parents=True, exist_ok=True)
if self._rate_limit > 0:
time.sleep(self._rate_limit)
out = self._fetch_get(media.url, path)
reason, quarantined = self._validate_path(out, artist_slug, media.url)
if reason is not None:
return MediaOutcome(media=media, status="quarantined", path=quarantined, error=reason)
sidecar = {"category": PLATFORM, "message_id": str(post.get("id") or "")}
sidecar["source_url"] = media.url
(folder / f"{stem}.json").write_text(json.dumps(sidecar, indent=2))
return MediaOutcome(media=media, status="downloaded", path=out, error=None)
def write_post_record(
self, post: dict, artist_slug: str, *, revisit: bool = False,
) -> PostRecordOutcome:
"""The message record — the one writer of a Discord post's body, date
and permalink ids. `revisit` re-reads a message already captured (an
edit); an empty re-read writes nothing, so it never blanks a body."""
mid = str(post.get("id") or "")
body = message_text(post)
if not mid or (revisit and not body.strip()):
return PostRecordOutcome(path=None, post_type=None, title=None, body_chars=0)
meta = post.get("_meta") or {}
author = post.get("author") or {}
record = {
"category": PLATFORM,
"message_id": mid,
"server": meta.get("server"),
"server_id": meta.get("server_id"),
"channel": meta.get("channel"),
"channel_id": meta.get("channel_id") or post.get("channel_id"),
"parent": meta.get("parent"),
"is_thread": meta.get("is_thread"),
"author": author.get("username"),
"author_id": author.get("id"),
"message": body,
"date": post.get("timestamp"),
}
folder = channel_dir(self.images_root, artist_slug, post)
folder.mkdir(parents=True, exist_ok=True)
when = message_date(post)
day = f"{when:%Y%m%d}" if when else "None"
path = folder / f"{day}_{mid}_post.json"
path.write_text(json.dumps(
{k: v for k, v in record.items() if v is not None}, indent=2,
))
return PostRecordOutcome(
path=path, post_type=None, title=None, body_chars=len(body),
)
+405 -10
View File
@@ -59,14 +59,23 @@ from __future__ import annotations
import logging
import math
from collections import Counter
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from sqlalchemy import Select, func, select, update
from sqlalchemy import Select, delete, func, select, union, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import ImageProvenance, ImageRecord, MLSettings, Post, Source
from ..models import (
ImageProvenance,
ImageRecord,
MLSettings,
Post,
PostAssociation,
Source,
)
from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies
log = logging.getLogger(__name__)
@@ -117,6 +126,31 @@ def cosine_distance(a, b) -> float:
return 1.0 - (dot / (na * nb))
def _message_images(posts):
"""(post_id, image_id) for every image a message carries — owned AND re-posted.
A message owns an image through `primary_post_id`, but only the FIRST
message imported with a given file does. The same file posted again is a
provenance link, and the backfill runs newest-first, so it is usually the
ORIGINAL message that ends up owning nothing. Reading ownership alone left
101 of Yellowroom's messages (2018–2020 mostly) ungroupable: every image
they carried also sat in another message.
`posts` is a list of ids or a select of them; filtering both branches by it
keeps the union to the messages in hand rather than the whole library.
Callers pass MESSAGE posts only — a drop's own provenance rows would read
as images it carries.
"""
owned = select(
ImageRecord.primary_post_id.label("post_id"), ImageRecord.id.label("image_id"),
).where(ImageRecord.primary_post_id.in_(posts))
reposted = select(
ImageProvenance.post_id.label("post_id"),
ImageProvenance.image_record_id.label("image_id"),
).where(ImageProvenance.post_id.in_(posts))
return union(owned, reposted).subquery()
def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select:
"""Ungrouped Discord message-posts, one representative image each, OLDEST
FIRST — which is the order `build_groups` requires.
@@ -134,13 +168,15 @@ def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select:
take the OLDEST candidates instead of the lowest-numbered ones.
"""
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
carried = _message_images(select(Post.id).where(Post.source_id == source_id))
inner = (
select(
Post.id.label("post_id"),
sort_key.label("occurred_at"),
ImageRecord.siglip_embedding.label("embedding"),
)
.join(ImageRecord, ImageRecord.primary_post_id == Post.id)
.join(carried, carried.c.post_id == Post.id)
.join(ImageRecord, ImageRecord.id == carried.c.image_id)
.where(
Post.source_id == source_id,
# Never absorb a post FC wrote, and never re-absorb one already
@@ -286,9 +322,10 @@ async def _link_member_images(
"""
if not member_ids:
return 0
image_rows = (await session.execute(
select(ImageRecord.id).where(ImageRecord.primary_post_id.in_(member_ids))
)).scalars().all()
carried = _message_images(member_ids)
image_rows = sorted(set((await session.execute(
select(carried.c.image_id)
)).scalars().all()))
if not image_rows:
return 0
await session.execute(
@@ -412,9 +449,11 @@ async def _group_seed(session: AsyncSession, post_id: int) -> list[float] | None
free to disagree; this way there is one.
"""
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
carried = _message_images(select(Post.id).where(Post.absorbed_by_post_id == post_id))
return (await session.execute(
select(ImageRecord.siglip_embedding)
.join(Post, ImageRecord.primary_post_id == Post.id)
.join(carried, carried.c.image_id == ImageRecord.id)
.join(Post, Post.id == carried.c.post_id)
.where(
Post.absorbed_by_post_id == post_id,
ImageRecord.siglip_embedding.is_not(None),
@@ -600,6 +639,352 @@ async def join_open_groups(
return joined
# ---------------------------------------------------------------------------
# #4390: a trickle is one drop — later drops of the same piece merge in.
# ---------------------------------------------------------------------------
#
# Operator, 2026-09-24, on a feed of "Grouped from 1 Discord message" cards:
# *"the groups are still single image even when they can clearly be seen as
# group"*. 665 of Yellowroom's 714 drops were one message.
#
# Both paths above join on cosine distance to the group's SEED, and the stages
# of one piece fail it: `svtt_wip4` did not join `svtt_wip3` from the day
# before. A creator trickles a piece out as sketch -> wip -> wip -> release, and
# each stage is nearest to the one before it, not to the first.
#
# Measured on artist 8 before any of this was written (#4390 log):
#
# * phash cannot see it. Stages of one piece sit 68-134 bits apart; unrelated
# pieces by the same artist sit at a median of 126, p5 110. Lesson #4400.
# * The embedding's NEAREST neighbour can. Every stage of three real trickles
# had a sibling stage as its single nearest image in the artist's whole
# library, while siblings further along ranked 40-100 — which is exactly why
# seed distance fails. Negative control over all 137 recent Discord images:
# where the nearest neighbour was another message within 7 days, the two
# carried the same working name 53 times out of 53. Disagreements start past
# 7 days.
# * The working name sees it directly, when there is one.
#
# So a later drop merges into an earlier one when the two are within
# `discord_group_close_after_hours` of each other (168h — the measured 7 days)
# AND either they share a gated LEADING working name, or one's image has the
# other's image as its nearest neighbour. A drop reaching SEVERAL earlier drops
# pulls them all together — unless two of them are named as different pieces,
# in which case nothing moves (see `_compatible`): leaving a drop alone is
# recoverable, a wrong merge asserts that unrelated art belongs together.
#
# Chaining is permitted here and was forbidden above, deliberately. The seed
# rule exists because tiny steps can drift from one piece to another; the
# measured precision of nearest-neighbour inside 7 days is what bounds drift
# for this route, and each link is between neighbours in time, never across a
# quiet week.
# How many unchecked drops one sweep examines per source. A first run over an
# established library drains over successive sweeps, oldest first, rather than
# issuing one nearest-neighbour query per image of the whole history at once.
TRICKLE_BATCH = 300
@dataclass
class _Drop:
post: Post
members: set[int]
first_at: datetime
last_at: datetime
names: set[str]
images: list[int]
nearest: set[int] | None
async def _nearest_message(
session: AsyncSession, *, artist_id: int, image_id: int, exclude: set[int],
) -> int | None:
"""The post that owns the nearest image in the artist's whole library.
The whole LIBRARY, not this source, because that is what was measured: a
neighbour that turns out to be a Patreon re-post simply yields no Discord
drop to merge into, which errs toward leaving things alone. `exclude` is
the drop's own messages — an image is always nearest to its own siblings
in the same drop, which says nothing.
"""
embedding = (await session.execute(
select(ImageRecord.siglip_embedding).where(ImageRecord.id == image_id)
)).scalar_one_or_none()
if embedding is None:
return None
stmt = (
select(ImageRecord.primary_post_id)
.where(
ImageRecord.artist_id == artist_id,
ImageRecord.id != image_id,
ImageRecord.siglip_embedding.is_not(None),
ImageRecord.primary_post_id.is_not(None),
)
.order_by(ImageRecord.siglip_embedding.cosine_distance(embedding))
.limit(1)
)
if exclude:
stmt = stmt.where(ImageRecord.primary_post_id.not_in(exclude))
return (await session.execute(stmt)).scalar_one_or_none()
async def _load_drops(session: AsyncSession, source: Source) -> list[_Drop]:
"""Every live drop of this source, with what the merge rule reads, oldest first."""
posts = (await session.execute(
select(Post).where(
Post.source_id == source.id,
Post.synthesized_by == DROP_GROUPER,
Post.absorbed_by_post_id.is_(None),
)
)).scalars().all()
if not posts:
return []
by_id = {p.id: p for p in posts}
msg_at = func.coalesce(Post.post_date, Post.downloaded_at)
members: dict[int, set[int]] = {pid: set() for pid in by_id}
times: dict[int, list[datetime]] = {pid: [] for pid in by_id}
for mid, owner, at in (await session.execute(
select(Post.id, Post.absorbed_by_post_id, msg_at)
.where(Post.absorbed_by_post_id.in_(list(by_id)))
)).all():
members[owner].add(mid)
times[owner].append(at)
owner_of = {m: d for d, ms in members.items() for m in ms}
names: dict[int, set[str]] = {pid: set() for pid in by_id}
images: dict[int, list[int]] = {pid: [] for pid in by_id}
if owner_of:
carried = _message_images(list(owner_of))
for iid, message, path in (await session.execute(
select(ImageRecord.id, carried.c.post_id, ImageRecord.path)
.join(carried, carried.c.image_id == ImageRecord.id)
.order_by(ImageRecord.id)
)).all():
drop = owner_of[message]
if iid in images[drop]:
continue # one file carried by two of the drop's messages
images[drop].append(iid)
if (name := leading_name(path)) is not None:
names[drop].add(name)
out = []
for pid, post in by_id.items():
if not times[pid]:
continue
stored = (post.synthesis_details or {}).get("nearest_message_ids")
out.append(_Drop(
post=post, members=members[pid],
first_at=min(times[pid]), last_at=max(times[pid]),
names=names[pid], images=images[pid],
nearest=set(stored) if stored is not None else None,
))
return sorted(out, key=lambda d: (d.first_at, d.post.id))
async def _name_posts(session: AsyncSession, artist_id: int) -> Counter[str]:
"""Post-span counts of the artist's working names — the same corpus the
teaser card and the announcement matcher count against."""
by_post: dict[int, list[str]] = {}
for pid, path in (await session.execute(
select(ImageRecord.primary_post_id, ImageRecord.path).where(
ImageRecord.artist_id == artist_id,
ImageRecord.primary_post_id.is_not(None),
)
)).all():
by_post.setdefault(pid, []).append(path)
return token_frequencies(by_post.values())
async def _repoint_associations(
session: AsyncSession, *, from_id: int, to_id: int,
) -> None:
"""Move announcement links from a drop about to merge onto the one it joins.
Without this the merge would silently undo a teaser link: the association's
payload FK cascades on delete. Where the teaser already points at the
surviving drop, the stronger claim is kept — a link over a proposal over a
dismissal — and the duplicate goes.
"""
rank = {"linked": 2, "pending": 1, "dismissed": 0}
moving = (await session.execute(
select(PostAssociation).where(PostAssociation.payload_post_id == from_id)
)).scalars().all()
for a in moving:
existing = (await session.execute(
select(PostAssociation).where(
PostAssociation.announcement_post_id == a.announcement_post_id,
PostAssociation.payload_post_id == to_id,
)
)).scalar_one_or_none()
if existing is None:
a.payload_post_id = to_id
continue
if rank.get(a.status, 0) > rank.get(existing.status, 0):
existing.status = a.status
existing.linked_by = a.linked_by
await session.delete(a)
await session.flush()
async def merge_trickles(
session: AsyncSession,
source: Source,
*,
gap: timedelta,
min_images: int,
cooldown: timedelta,
batch: int = TRICKLE_BATCH,
) -> int:
"""Fold later drops of the same piece into the earlier one. Returns merges."""
drops = await _load_drops(session, source)
if len(drops) < 2:
return 0
name_posts = await _name_posts(session, source.artist_id)
def gated(names: set[str]) -> set[str]:
return {n for n in names if rarity(name_posts.get(n, 0), FAMILY_MAX_POSTS) > 0}
alive: list[_Drop] = []
merged = 0
checked = 0
for drop in drops:
details = drop.post.synthesis_details or {}
if details.get("trickle_checked"):
alive.append(drop)
continue
if checked >= batch:
# Unchecked and out of budget: still a candidate for LATER drops'
# reverse edges, just not examined itself this run.
alive.append(drop)
continue
checked += 1
if drop.nearest is None:
found: set[int] = set()
for iid in drop.images:
pid = await _nearest_message(
session, artist_id=source.artist_id, image_id=iid,
exclude=drop.members,
)
if pid is not None:
found.add(pid)
drop.nearest = found
mine = gated(drop.names)
targets: dict[int, tuple[_Drop, str]] = {}
for earlier in alive:
if drop.first_at - earlier.last_at > gap:
continue
shared = mine & gated(earlier.names)
if set(drop.images) & set(earlier.images):
# The creator posted the very same file again — the strongest
# evidence there is, and one nearest-neighbour cannot see: it
# skips the image itself, which is the one they share.
targets[earlier.post.id] = (earlier, "same_image")
elif shared:
targets[earlier.post.id] = (earlier, f"name:{min(shared)}")
elif drop.nearest & earlier.members or (earlier.nearest or set()) & drop.members:
targets[earlier.post.id] = (earlier, "nearest")
record = dict(details)
record["nearest_message_ids"] = sorted(drop.nearest)
record["trickle_checked"] = True
drop.post.synthesis_details = record
if not targets or not _compatible(
[gated(t.names) for t, _route in targets.values()] + [mine]
):
alive.append(drop)
continue
# Every target is the same piece as this drop, so they are the same
# piece as each other: fold them all into the earliest, then this drop.
ordered = sorted(targets.values(), key=lambda tr: (tr[0].first_at, tr[0].post.id))
into = ordered[0][0]
for other, route in ordered[1:]:
await _merge_drop(
session, into=into, drop=other, route=route,
min_images=min_images, cooldown=cooldown,
)
alive.remove(other)
merged += 1
await _merge_drop(
session, into=into, drop=drop, route=ordered[0][1],
min_images=min_images, cooldown=cooldown,
)
merged += 1
return merged
def _compatible(name_sets: list[set[str]]) -> bool:
"""May drops carrying these working names become one post?
Refused only when two of them are NAMED AS DIFFERENT PIECES — both carry a
gated name, and they share none. An unnamed drop (a canvas screenshot)
fits anywhere, which is the whole of the Marin case: two early stages both
nearest to the same later one are one trickle, not an ambiguity.
What it does NOT refuse is a drop the creator made two pieces in
themselves. Measured on artist 8: one November message carries both
`AdL01_wip4` and `Year_20k_wip_z4`, so its drop holds both names, and a
later drop of either piece joins it on its own name. That is the creator's
co-posting carried forward — Discord shows those two together too — not a
bridge FC built.
"""
named = [n for n in name_sets if n]
return all(a & b for i, a in enumerate(named) for b in named[i + 1:])
async def _merge_drop(
session: AsyncSession,
*,
into: _Drop,
drop: _Drop,
route: str,
min_images: int,
cooldown: timedelta,
) -> None:
"""Absorb `drop`'s messages into `into`, carry its links over, delete it.
Growth is stamped at the merged messages' OWN time, not the wall clock.
Merging history must not drag a two-year-old drop to the top of the feed,
and the time the group actually grew is when those messages arrived.
"""
await _repoint_associations(session, from_id=drop.post.id, to_id=into.post.id)
grew_before = into.post.last_grew_at
await _absorb_into(
session, group=into.post, member_ids=sorted(drop.members),
source_id=into.post.source_id, now=drop.last_at,
min_images=min_images, cooldown=cooldown,
)
# Never backwards: a group that already grew later than these messages
# keeps that later date.
if grew_before is not None and grew_before > drop.last_at:
into.post.last_grew_at = grew_before
details = dict(into.post.synthesis_details or {})
if grew_before is not None and grew_before > drop.last_at:
details["last_grew_at"] = grew_before.isoformat()
# The honesty rule, extended: a grouping FC invented says what it was
# built from, and a merge says WHY — "name:svtt" or "nearest".
details["merged"] = [
*details.get("merged", []),
{"post_id": drop.post.id, "route": route, "message_ids": sorted(drop.members)},
]
details["nearest_message_ids"] = sorted((into.nearest or set()) | (drop.nearest or set()))
into.post.synthesis_details = details
into.members |= drop.members
into.names |= drop.names
into.images += drop.images
into.nearest = (into.nearest or set()) | (drop.nearest or set())
into.last_at = max(into.last_at, drop.last_at)
await session.execute(delete(ImageProvenance).where(ImageProvenance.post_id == drop.post.id))
await session.delete(drop.post)
await session.flush()
async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
"""Group every enabled Discord source. No-op when the switch is off.
@@ -612,6 +997,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
if not settings.discord_grouping_enabled:
return {
"enabled": False, "sources": 0, "posts_created": 0, "images_joined": 0,
"drops_merged": 0,
}
sources = (await session.execute(
@@ -626,6 +1012,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
created = 0
joined = 0
merged = 0
for source in sources:
joined += await join_open_groups(
session, source,
@@ -644,12 +1031,20 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
window_minutes=window_minutes,
now=now,
)
# Last, so the drops the two passes above just wrote are merged in
# the same sweep rather than showing as singletons for an hour.
merged += await merge_trickles(
session, source,
gap=timedelta(hours=float(settings.discord_group_close_after_hours)),
min_images=int(settings.discord_group_resurface_min_images),
cooldown=timedelta(hours=float(settings.discord_group_resurface_cooldown_hours)),
)
log.info(
"discord drop grouping: %d source(s), %d synthetic post(s) created, "
"%d image(s) joined to open groups",
len(sources), created, joined,
"%d image(s) joined to open groups, %d trickle drop(s) merged",
len(sources), created, joined, merged,
)
return {
"enabled": True, "sources": len(sources),
"posts_created": created, "images_joined": joined,
"posts_created": created, "images_joined": joined, "drops_merged": merged,
}
+85
View File
@@ -0,0 +1,85 @@
"""Native Discord ingester — the Discord ADAPTER over `ingest_core.Ingester`.
Thin counterpart to subscribestar_ingester (milestone 428). The walk's modes,
both ledgers, cursor checkpointing and the post-first capture live in the core;
this wires in the Discord client, downloader, ledger models and key.
Two things differ from the cookie platforms:
- Discord authenticates with a user TOKEN, so `auth_token` is the credential
here rather than an argument accepted and ignored.
- The body canary is off. It fails a walk whose first 30+ captured posts all
came back without text, on the theory that a creator nearly always writes
something; a Discord drop is routinely files and nothing else, so on
Discord that is an ordinary backfill, not a broken parser.
`campaign_id` is the source URL (a server, channel, thread or category link).
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import asyncio
from collections.abc import Callable
from pathlib import Path
from ..models import DiscordFailedMedia, DiscordSeenMedia
from .discord_client import DiscordAPIError, DiscordClient, MediaItem
from .discord_downloader import DiscordDownloader
from .ingest_core import Ingester
_LEDGER_KEY_MAX = 128
def _ledger_key(media: MediaItem) -> str:
"""`<message_id>:<media_id>` — stable across edits (see MediaItem)."""
return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
class DiscordIngester(Ingester):
"""Walk a Discord source's channels, download unseen files, return a
`DownloadResult`. `client` / `downloader` are injectable for tests."""
def __init__(
self,
images_root: Path,
cookies_path: str | None,
session_factory: Callable[[], object],
*,
validate: bool = True,
rate_limit: float = 0.0,
request_sleep: float = 0.0,
auth_token: str | None = None,
client: DiscordClient | None = None,
downloader: DiscordDownloader | None = None,
):
del cookies_path # Discord authenticates by token (uniform signature)
self.images_root = Path(images_root)
super().__init__(
client=client if client is not None else DiscordClient(
auth_token, request_sleep=request_sleep,
),
downloader=downloader if downloader is not None else DiscordDownloader(
self.images_root, validate=validate, rate_limit=rate_limit,
),
session_factory=session_factory,
seen_model=DiscordSeenMedia,
failed_model=DiscordFailedMedia,
seen_constraint="uq_discord_seen_media_source_id",
failed_constraint="uq_discord_failed_media_source_id",
ledger_key=_ledger_key,
platform="discord",
error_base=DiscordAPIError,
drift_label="Discord API",
body_canary=False,
)
async def verify_discord_credential(url: str, auth_token: str | None) -> tuple[bool | None, str]:
"""The uniform `(ok, message)` probe: is the token valid, and can its
account see the channel or server the source names?"""
if not auth_token:
return False, "No Discord token is saved — add one under Credentials."
client = DiscordClient(auth_token)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, client.verify_auth, url)
+24 -8
View File
@@ -23,16 +23,22 @@ from __future__ import annotations
import asyncio
from pathlib import Path
from .discord_ingester import DiscordIngester
from .gallery_dl import DownloadResult, ErrorType
from .ingest_core import DEFAULT_REVISIT_DAYS
from .patreon_ingester import PatreonIngester
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
from .platforms import known_platform_keys
from .subscribestar_ingester import SubscribeStarIngester
# Platforms whose download + verify go through the native ingester rather than
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
# they migrate too.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
# gallery-dl. gallery-dl still serves the rest (hentaifoundry) until it
# migrates too. Discord joined in milestone 428.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"})
# Native platforms whose feed id IS the source URL, so there is nothing to
# resolve: SubscribeStar's creator page, Discord's server/channel link.
_URL_IS_FEED = frozenset({"subscribestar", "discord"})
def _unsupported_platform_message(platform: str) -> str | None:
@@ -66,6 +72,7 @@ def _native_ingester_cls(platform: str):
return {
"patreon": PatreonIngester,
"subscribestar": SubscribeStarIngester,
"discord": DiscordIngester,
}[platform]
@@ -83,6 +90,7 @@ async def run_download(
mode: str | None,
gdl,
sync_session_factory,
revisit_days: int = DEFAULT_REVISIT_DAYS,
) -> tuple[DownloadResult, str | None]:
"""Uniform download across backends — the download counterpart to
`verify_source_credential`, so this module is the ONE place that knows how
@@ -106,7 +114,7 @@ async def run_download(
), None
if uses_native_ingester(platform):
return await _run_native_ingester(
ctx, source_config, mode, gdl, sync_session_factory
ctx, source_config, mode, gdl, sync_session_factory, revisit_days
)
result = await gdl.download(
url=ctx["url"],
@@ -124,10 +132,10 @@ async def _resolve_native_campaign_id(
platform: str, url: str, cookies_path: str | None, overrides: dict,
) -> tuple[str | None, str | None]:
"""`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
feed id IS the creator URL (no lookup → resolved None). Patreon resolves the
campaign id from the vanity URL (resolved non-None when a lookup actually ran,
so phase 3 caches it)."""
if platform == "subscribestar":
and Discord's feed id IS the source URL (no lookup → resolved None). Patreon
resolves the campaign id from the vanity URL (resolved non-None when a lookup
actually ran, so phase 3 caches it)."""
if platform in _URL_IS_FEED:
return url, None
return await resolve_campaign_id_for_source(url, cookies_path, overrides)
@@ -146,6 +154,7 @@ def _campaign_resolution_error(platform: str, url: str) -> str:
async def _run_native_ingester(
ctx: dict, source_config, mode: str | None, gdl, sync_session_factory,
revisit_days: int = DEFAULT_REVISIT_DAYS,
) -> tuple[DownloadResult, str | None]:
"""Run the native ingester for a native platform in a worker thread (sync
requests/subprocess). Patreon resolves a campaign id from the vanity URL;
@@ -210,6 +219,9 @@ async def _run_native_ingester(
mode=mode,
resume_cursor=source_config.resume_cursor,
time_budget_seconds=source_config.timeout,
# How far back a tick keeps looking for EDITED posts. The ingester
# applies it to ticks only; a backfill ignores it.
revisit_days=revisit_days,
posts_base=int(overrides.get("_backfill_posts", 0)),
# plan #709: live progress writes to this running event mid-walk.
event_id=ctx.get("event_id"),
@@ -246,6 +258,10 @@ async def verify_source_credential(
from .subscribestar_ingester import verify_subscribestar_credential
return await verify_subscribestar_credential(url, cookies_path, config_overrides)
if platform == "discord":
from .discord_ingester import verify_discord_credential
return await verify_discord_credential(url, auth_token)
from .patreon_ingester import verify_patreon_credential
return await verify_patreon_credential(url, cookies_path, config_overrides)
+16
View File
@@ -39,6 +39,7 @@ from .gallery_dl import (
walk_completed,
)
from .importer import Importer
from .ingest_core import DEFAULT_REVISIT_DAYS
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
@@ -60,6 +61,7 @@ class DownloadService:
importer: Importer,
cred_service: CredentialService,
sync_session_factory=None,
revisit_days: int = DEFAULT_REVISIT_DAYS,
):
self.async_session = async_session
self.sync_session = sync_session
@@ -71,6 +73,12 @@ class DownloadService:
# the multi-minute walk — see PatreonIngester). Only the patreon branch
# of phase 2 uses it; gallery-dl sources leave it None.
self.sync_session_factory = sync_session_factory
# ImportSettings.download_revisit_days — how far back a tick keeps
# looking for EDITED posts (ingest_core.DEFAULT_REVISIT_DAYS). Passed in
# rather than read here because the task already loads the settings row
# for rate_limit/validate_files, and a second load on every download
# would be the same row twice for one number.
self.revisit_days = revisit_days
async def download_source(self, source_id: int) -> int:
"""Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
@@ -178,6 +186,7 @@ class DownloadService:
return await run_download(
ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
gdl=self.gdl, sync_session_factory=self.sync_session_factory,
revisit_days=self.revisit_days,
)
async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
@@ -414,6 +423,13 @@ class DownloadService:
await loop.run_in_executor(None, _upsert)
# Only now is it safe to call this walk's media seen: every file above
# has been through the importer. Had the run died before here they stay
# unmarked, and the next walk imports them from disk (ingest_core).
mark_seen = getattr(dl_result, "mark_seen_after_import", None)
if mark_seen is not None:
await loop.run_in_executor(None, mark_seen)
# #830 recapture: backfill source_filehash on EXISTING on-disk images so
# their post-body inline <img src=CDN> remaps to the local copy. A
# SEPARATE non-deleting channel (NOT the import list — that would unlink
+188 -10
View File
@@ -21,6 +21,9 @@ from .source_service import BACKFILL_MAX_CHUNKS
log = logging.getLogger(__name__)
# The probe runs while the chip is drawing; names that take longer are skipped.
_NAME_LOOKUP_SECONDS = 6.0
class UnknownPlatformError(Exception):
"""URL didn't match any platform pattern."""
@@ -30,6 +33,10 @@ class InvalidUrlError(Exception):
"""URL was empty or missing a scheme."""
class UnknownArtistError(Exception):
"""quick-add named an `artist_id` that does not exist."""
# Mirrored byte-for-byte from extension/lib/platforms.js
# PLATFORM_ARTIST_PATTERNS. Keep these two copies in sync by hand —
# reviewers catch drift.
@@ -55,8 +62,39 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
re.IGNORECASE,
)),
# A Discord URL names a server or a channel, never a creator, so the slug is
# `<server>` or `<server>/<channel>` and the artist is chosen, not derived.
# A trailing message id (a jump link) still names its channel. DMs (`@me`)
# are not sources; thread links (`/threads/`) are left to the manual form.
("discord", re.compile(
r"^https?://(?:www\.|ptb\.|canary\.)?discord\.com/channels/"
r"(?P<slug>\d+(?:/\d+)?)(?:/\d+)?/?(?:[?#].*)?$",
re.IGNORECASE,
)),
]
DISCORD = "discord"
def canonical_source_url(platform: str, url: str, slug: str) -> str:
"""The URL a new source is stored under. Discord's is rebuilt from the ids
— the form the manual Add form and the ingester use — so a jump link, a
ptb/canary host or a trailing slash never makes a second source for the
same channel. Every other platform keeps the URL as given."""
if platform == DISCORD:
return f"https://discord.com/channels/{slug}"
return url
def _discord_ids(url: str) -> tuple[str | None, str | None] | None:
"""`(server_id, channel_id)` of a stored Discord source URL, None if it
does not parse (a DM or thread link, or an old malformed row)."""
from .discord_client import DiscordAPIError, parse_source_url
try:
return parse_source_url(url)
except DiscordAPIError:
return None
class ExtensionService:
def __init__(self, session: AsyncSession, crypto=None) -> None:
@@ -65,30 +103,63 @@ class ExtensionService:
# add-time. None → skip resolution, fall back to the handle.
self._crypto = crypto
async def quick_add_source(self, url: str) -> dict:
async def quick_add_source(
self,
url: str,
*,
artist_id: int | None = None,
artist_name: str | None = None,
) -> dict:
"""Add `url` as a source. `artist_id` connects it to an existing
artist, `artist_name` to that artist (created if new); with neither,
the artist is resolved from the platform as before."""
platform, raw_slug = self._derive(url)
url = canonical_source_url(platform, url, raw_slug)
# Identity by SOURCE handle (#130): an existing (platform, url) source
# keeps its artist on re-add — even if that artist was since renamed (its
# frozen slug no longer matches the current name). Only a genuinely new
# source resolves/creates an artist.
existing = (await self.session.execute(
select(Source).where(Source.platform == platform, Source.url == url)
)).scalar_one_or_none()
# frozen slug no longer matches the current name), and even when the
# add named a different artist. Only a genuinely new source
# resolves/creates an artist.
existing = await self._existing_source(platform, url)
if existing is not None:
artist = (await self.session.execute(
select(Artist).where(Artist.id == existing.artist_id)
)).scalar_one()
return self._shape(existing, artist, created_source=False, created_artist=False)
# New source → name the artist properly by resolving the real display
# name from the platform (falls back to the URL handle).
name = await self._resolve_artist_name(platform, raw_slug, url)
artist, created_artist = await self._find_or_create_artist(name)
if artist_id is not None:
artist = (await self.session.execute(
select(Artist).where(Artist.id == artist_id)
)).scalar_one_or_none()
if artist is None:
raise UnknownArtistError(f"no artist with id {artist_id}")
created_artist = False
else:
name = (artist_name or "").strip()
if not name:
# Name the artist properly by resolving the real display name
# from the platform (falls back to the URL handle).
name = await self._resolve_artist_name(platform, raw_slug, url)
artist, created_artist = await self._find_or_create_artist(name)
source, created_source = await self._find_or_create_source(
artist_id=artist.id, platform=platform, url=url,
)
return self._shape(source, artist, created_source, created_artist)
async def _existing_source(self, platform: str, url: str) -> Source | None:
"""The source this URL already is, whichever artist owns it. Discord
compares ids, not strings, so a row stored before canonicalisation (a
ptb host, a trailing slash) is still found."""
if platform != DISCORD:
return (await self.session.execute(
select(Source).where(Source.platform == platform, Source.url == url)
)).scalars().first()
want = _discord_ids(url)
rows = (await self.session.execute(
select(Source).where(Source.platform == DISCORD).order_by(Source.id)
)).scalars().all()
return next((s for s in rows if _discord_ids(s.url) == want), None)
@staticmethod
def _shape(source, artist, created_source: bool, created_artist: bool) -> dict:
return {
@@ -117,6 +188,11 @@ class ExtensionService:
platforms (and any failure — no credential, network error) fall back to
the URL handle, which is already readable.
The resolvers are sync, so they run in an executor."""
if platform == DISCORD:
# The server's name: what the operator knows the community as.
server_id = raw_slug.split("/", 1)[0]
names = await self._discord_names(server_id, None)
return names.get("server") or f"Discord {server_id}"
if self._crypto is None or platform not in ("patreon", "subscribestar"):
return raw_slug
import asyncio
@@ -166,6 +242,8 @@ class ExtensionService:
platform, raw_slug = self._derive(url)
except (UnknownPlatformError, InvalidUrlError):
return {"state": "unknown_platform"}
if platform == DISCORD:
return await self._probe_discord(raw_slug)
slug = slugify(raw_slug)
artist = (await self.session.execute(
@@ -205,6 +283,106 @@ class ExtensionService:
},
}
async def _probe_discord(self, raw_slug: str) -> dict:
"""probe for a Discord server or channel. The states mean what they
mean elsewhere, but the artist is never read off the URL:
- source_match: this channel is a source — or the whole server is
(`covered_by_server`), which already walks every channel;
- artist_match: another source on this server belongs to an artist,
the one this channel most likely belongs to too (a suggestion the
Add panel preselects, not a decision);
- new: nothing on this server yet.
`discord` carries the ids, both canonical URLs and the display names,
read with the stored token; a name that can't be read is None."""
server_id, _, channel_id = raw_slug.partition("/")
channel_id = channel_id or None
rows = (await self.session.execute(
select(Source, Artist)
.join(Artist, Artist.id == Source.artist_id)
.where(Source.platform == DISCORD)
.order_by(Source.id)
)).all()
exact = server_whole = on_server = None
for source, artist in rows:
ids = _discord_ids(source.url)
if ids is None or ids[0] != server_id:
continue
if ids[1] == channel_id and exact is None:
exact = (source, artist)
elif ids[1] is None and server_whole is None:
server_whole = (source, artist)
if on_server is None:
on_server = (source, artist)
names = await self._discord_names(server_id, channel_id)
base = f"https://discord.com/channels/{server_id}"
result: dict = {
"platform": DISCORD,
"slug": raw_slug,
"discord": {
"server_id": server_id,
"channel_id": channel_id,
"server_name": names.get("server"),
"channel_name": names.get("channel"),
"server_url": base,
"channel_url": f"{base}/{channel_id}" if channel_id else None,
},
}
hit = exact or server_whole
if hit is not None:
source, artist = hit
result.update(
state="source_match",
artist=self._artist_payload(artist),
source=self._source_payload(source),
covered_by_server=exact is None,
)
elif on_server is not None:
result.update(state="artist_match", artist=self._artist_payload(on_server[1]))
else:
result["state"] = "new"
return result
async def _discord_names(self, server_id: str | None, channel_id: str | None) -> dict:
"""Server/channel display names via the stored Discord token. Never
raises and never waits out a rate limit: it runs while the operator
looks at a page, so a slow or missing answer just means no names."""
if self._crypto is None:
return {}
import asyncio
from .credential_service import CredentialService
from .discord_client import DiscordClient
try:
token = await CredentialService(self.session, self._crypto).get_token(DISCORD)
if not token:
return {}
client = DiscordClient(token, max_retries=0)
loop = asyncio.get_running_loop()
return await asyncio.wait_for(
loop.run_in_executor(None, client.describe, server_id, channel_id),
timeout=_NAME_LOOKUP_SECONDS,
)
except Exception as exc: # names are decoration — never fail the call
log.info("Discord name lookup failed: %s", exc)
return {}
@staticmethod
def _artist_payload(artist) -> dict:
return {"id": artist.id, "name": artist.name, "slug": artist.slug}
@staticmethod
def _source_payload(source) -> dict:
return {
"id": source.id,
"artist_id": source.artist_id,
"platform": source.platform,
"url": source.url,
"enabled": source.enabled,
}
def _derive(self, url: str) -> tuple[str, str]:
if not isinstance(url, str) or not url.strip():
raise InvalidUrlError("url is empty")
+12 -73
View File
@@ -17,6 +17,7 @@ import subprocess
import sys
import tempfile
import time
from collections.abc import Callable
from dataclasses import dataclass, field
from datetime import UTC, datetime
from enum import StrEnum
@@ -94,48 +95,6 @@ BACKFILL_CHUNK_SECONDS = 600
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
# --- Discord naming ---------------------------------------------------------
#
# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory
# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits
# for an attachment: `channel` is a plain STRING (the channel's name), the
# message is `message_id`, the attachment's position in it is `num`, and there
# is NO `id` key at all.
#
# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as
# "None", so every Discord download since the platform was added landed in a
# directory called `None` as `<date>_None_<original name>`. Worse, the sidecar was
# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_
# sidecar` can never pair with `<date>_None_<name>.png`, so no Discord file ever
# got a Post or a post date, and (b) collides: every `image.png` in a channel
# overwrote the same `image.json`, so the one sidecar that survived described
# whichever message happened to be written last.
#
# The fix names the sidecar EXACTLY like the media minus its extension, so
# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match
# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders
# these patterns against a sanitized copy of the real sidecar, so a key that
# does not exist fails CI instead of silently becoming "None".
DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}"
DISCORD_DIRECTORY = ["{channel}"]
def sidecar_name_for(media_pattern: str) -> str | None:
"""The metadata filename pattern that names a sidecar exactly like its media.
Returns None for a pattern that does not end in `.{extension}`, since then
there is no media stem to mirror and the caller must fall back.
"""
suffix = ".{extension}"
if not media_pattern.endswith(suffix):
return None
return media_pattern[: -len(suffix)] + ".json"
def metadata_postprocessor(filename: str) -> dict:
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
def archive_path(images_root: Path) -> Path:
"""gallery-dl's download archive: the record of what it has already fetched.
@@ -230,6 +189,13 @@ class DownloadResult:
# the platform cooldown matches the hint instead of a flat default. None when
# unknown (no header, or not a rate-limit failure).
retry_after_seconds: float | None = None
# Native ingester only: marks this walk's fetched media seen in its ledger.
# Phase 3 calls it AFTER the import loop, never before — a file marked seen
# but not yet imported is invisible to every later walk, so a run killed in
# between orphaned it for good (TamadaHeijun's 12PCG post lost 7 of 13
# images to a stranded run, 2026-09-24). Unmarked, the next walk finds the
# file on disk with no ImageRecord and imports it. None on gallery-dl.
mark_seen_after_import: Callable[[], None] | None = None
def extract_errors_warnings(stderr: str) -> str:
@@ -417,27 +383,16 @@ class GalleryDLService:
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = {
# subscribestar removed — native-ingester platform now (#71); pixiv
# removed likewise (#129); deviantart removed at #3069 as a dropped
# platform, not a migrated one. The remaining entries are the
# gallery-dl platforms not yet migrated.
# removed likewise (#129); discord likewise (milestone 428, whose
# downloader keeps this config's on-disk naming); deviantart removed at
# #3069 as a dropped platform, not a migrated one. HentaiFoundry is the
# one platform left here, by the operator's choice not to migrate it.
"hentaifoundry": {
"content_types": ["all"],
"directory": [],
"filename": "{category}_{index:>03}_{title[:50]}.{extension}",
"include": "all",
},
"discord": {
"content_types": ["all"],
"directory": DISCORD_DIRECTORY,
"filename": DISCORD_FILENAME,
# Overrides the global `{filename}.json` sidecar for this extractor
# only — see the Discord naming note above.
"postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))],
"embeds": "all",
"stickers": True,
"reactions": False,
"threads": True,
},
}
def __init__(
@@ -552,17 +507,6 @@ class GalleryDLService:
if source_config.filename_pattern:
platform_section["filename"] = source_config.filename_pattern
# A platform that names its sidecar after its media must keep doing so
# under a per-source filename override, or the pairing breaks exactly the
# way Discord's did. No metadata wanted means no platform postprocessor
# either — the global list was already dropped above.
if "postprocessors" in platform_section:
mirrored = sidecar_name_for(platform_section.get("filename") or "")
if not source_config.save_metadata or mirrored is None:
platform_section.pop("postprocessors")
else:
platform_section["postprocessors"] = [metadata_postprocessor(mirrored)]
platform_section["metadata"] = source_config.save_metadata
return config
@@ -810,9 +754,6 @@ class GalleryDLService:
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})
config["extractor"]["discord"]["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
@@ -996,8 +937,6 @@ class GalleryDLService:
config = self._build_config_for_source(platform, source_config, artist_slug)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})["token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
+242 -13
View File
@@ -31,10 +31,12 @@ import json
import logging
import time
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from sqlalchemy import delete, func, select, text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..models import ImageRecord
from .gallery_dl import (
DownloadResult,
ErrorType,
@@ -51,6 +53,33 @@ log = logging.getLogger(__name__)
# per-file HEADs. Headroom against paywalled/undownloadable items interleaving.
_TICK_SEEN_THRESHOLD = 20
# How far back a tick keeps looking even once everything is already-have-it —
# the REVISIT WINDOW. Operator, 2026-09-23, holding up a Floppystack post:
# *"this post has been updated as he implements hot fixes — any chance we have a
# way to scan for or see updated posts so we can update ours to match and pull
# the new attachments and pictures etc."*
#
# A creator who edits a three-day-old post to append a hotfix build was
# structurally unreachable: that post sits twenty-odd already-seen items down
# the feed, so the count early-out above fired before the walk ever got to it.
# Not a bug in the early-out — a COUNT cannot express "recent".
#
# So the early-out now needs BOTH conditions: the run of already-seen items AND
# a post published before the horizon. Strictly a widening. Two properties this
# shape has and a plain "walk the last N days" would not:
#
# * window 0 is exactly the old behaviour, so the feature has an off switch
# that costs nothing to reason about;
# * no window can make a tick stop EARLIER than it used to. A source paused
# for months has an unseen backlog stretching well past any horizon, and
# the walk still runs to the end of it — the horizon is a FLOOR on how far
# to look, never a ceiling.
#
# The live value is `ImportSettings.download_revisit_days` (rule 25 — an
# operator tuning how far back their creators edit should not need a redeploy).
# This is the fallback for a caller that passes none.
DEFAULT_REVISIT_DAYS = 30
# plan #705 #7: after this many failed download/validate attempts a media is
# "dead-lettered" and skipped on routine tick/backfill walks (recovery still
# re-attempts it). Stops a permanently-broken media re-erroring forever.
@@ -77,6 +106,44 @@ _LIVE_PROGRESS_INTERVAL = 5.0
# recapture (the operator's schema-test flow) reaches the sample.
_CANARY_MIN_SAMPLE = 30
# The walk's time budget covers only the walk, but phase 3 runs in the SAME
# Celery task, under the same soft limit (tasks/download.py: 1350s). A walk that
# finds a lot of work for phase 3 must stop early and leave it to the next chunk,
# or phase 3 is killed mid-import: TamadaHeijun's recapture, 2026-09-24, walked
# for ~2 min and then spent 20 min importing 431 orphans and relinking ~3000
# on-disk files, and died at the soft limit.
#
# So the walk also stops when its elapsed time PLUS phase 3's estimated cost
# would pass CHUNK_TOTAL_SECONDS. Costs measured on the live instance: 431
# imports took 976s (~2.3s each: hash, pHash, sidecar, provenance); a relink is
# a sha256 over NFS, 0.15s for the 8.8 MB average file, plus a lookup.
# test_download_source_task pins CHUNK_TOTAL_SECONDS under the soft limit.
CHUNK_TOTAL_SECONDS = 1200.0
PHASE3_IMPORT_SECONDS = 2.5
PHASE3_RELINK_SECONDS = 0.25
def _parse_published(raw: object) -> datetime | None:
"""An ISO-8601 post date from either native client, as aware UTC.
Patreon's `published_at` is tz-aware with a `Z` or `+00:00` offset;
SubscribeStar's is NAIVE (`_parse_ss_datetime` renders a parsed local
timestamp with no zone). A naive value is read as UTC — the alternative is
discarding it, and a post whose date we refuse to read is a post the revisit
window can never reach.
Anything unparseable returns None, which reads downstream as "not provably
recent" and leaves the walk on its pre-revisit behaviour. Never raises: a
date we cannot read must not fail a walk that is otherwise working.
"""
if not isinstance(raw, str) or not raw.strip():
return None
try:
parsed = datetime.fromisoformat(raw.strip().replace("Z", "+00:00"))
except ValueError:
return None
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
class Ingester:
"""Generic native-ingest orchestration. Subclass with a platform adapter
@@ -132,11 +199,17 @@ class Ingester:
resume_cursor: str | None = None,
time_budget_seconds: float = 870.0,
seen_threshold: int = _TICK_SEEN_THRESHOLD,
revisit_days: int = DEFAULT_REVISIT_DAYS,
posts_base: int = 0,
event_id: int | None = None,
) -> DownloadResult:
"""Walk + download for one source, returning a gallery-dl-shaped result.
`revisit_days` is the tick's revisit window (see DEFAULT_REVISIT_DAYS):
inside it a tick neither early-outs nor trusts the post-record gate, so
a post edited after we first captured it is re-read and its new
attachments downloaded. 0 turns the window off.
`mode` is "tick" | "backfill" | "recovery" | "recapture". Recovery
bypasses the tier-1 seen-ledger AND the dead-letter ledger (tier-2 disk
still skips kept files). Recapture (#830) is the cheap "re-grab post
@@ -179,6 +252,29 @@ class Ingester:
# no media download, no post-record stub. Absent on stub/not-yet-migrated
# clients → nothing is ever treated as gated.
post_is_gated = getattr(self.client, "post_is_gated", None)
# The revisit window (see DEFAULT_REVISIT_DAYS). `post_meta` is an
# existing client seam — both native clients already implement it, for
# a preview sample whose caller has since gone, so this needed no new
# contract, only a live consumer for one. Absent seam, an unreadable
# date or a window of 0 → `horizon` never matches and the walk behaves
# exactly as it did before 2026-09-23.
#
# The window applies to TICKS only. A backfill is gated on purpose
# (capture each post once) and `recapture` mode already exists for the
# operator-driven "re-read every body" pass; a horizon there would be a
# third overlapping answer to a question that has two.
post_meta = getattr(self.client, "post_meta", None)
# #4413: optional client seam for a source that is several feeds walked
# one after another (a Discord server: every channel and thread). The
# tick early-out means "this feed has nothing new", and without the
# seam it ends the WHOLE walk — so the first quiet channel would hide
# every channel after it. With it, the early-out asks the client to
# move on and the walk continues. Absent → the early-out ends the walk,
# exactly as before (Patreon and SubscribeStar are one feed each).
skip_feed = getattr(self.client, "skip_feed", None)
horizon: datetime | None = None
if mode == "tick" and revisit_days > 0 and post_meta is not None:
horizon = datetime.now(UTC) - timedelta(days=revisit_days)
start = time.monotonic()
last_live = start # plan #709: last live-progress write timestamp
log_lines: list[str] = []
@@ -190,6 +286,9 @@ class Ingester:
# source_filehash and (b) link the on-disk image to its Post (#1288) —
# WITHOUT re-downloading or unlinking the file. Empty outside recapture.
relink: list[tuple[str, str, str]] = []
# Media handed to phase 3 for import. Marked seen by phase 3 once the
# import has run (`mark_seen_after_import`), not here — see there.
fetched: list[tuple[str, str]] = []
downloaded = 0
errors = 0
quarantined = 0
@@ -210,11 +309,18 @@ class Ingester:
# absolute across chunks instead of an inflating sum. posts_processed
# stays the gross per-chunk count used for the run summary.
chunk_new_posts = 0
# Posts inside the revisit window that we had already captured, and the
# media those revisits turned up. Reported in the run summary — the
# operator's ask was to SEE the updated posts, not only to end up with
# their files ("so we can update ours to match").
revisited = 0
revisit_downloads = 0
consecutive_seen = 0
emitted_cursor: str | None = None
reached_bottom = False
budget_hit = False
early_out = False
feeds_caught_up = 0 # #4413: feeds a tick left early via skip_feed
stopped = False # plan #708 B4: operator hit Stop mid-walk
cancel_armed = False # latched once we observe a live "running" state
@@ -236,6 +342,7 @@ class Ingester:
written_paths=written,
post_record_paths=list(post_records),
relink_source_paths=list(relink),
mark_seen_after_import=lambda: self._mark_seen(source_id, fetched),
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
@@ -311,7 +418,17 @@ class Ingester:
# Time-box check at the post boundary (coarse, like a gallery-dl
# chunk). Backfill/recovery resume from emitted_cursor next chunk.
if time.monotonic() - start >= time_budget_seconds:
# The second half is phase 3's share of the task — see
# CHUNK_TOTAL_SECONDS. A mid-page stop resumes the same page.
elapsed = time.monotonic() - start
phase3 = (
len(written) * PHASE3_IMPORT_SECONDS
+ len(relink) * PHASE3_RELINK_SECONDS
)
if (
elapsed >= time_budget_seconds
or elapsed + phase3 >= CHUNK_TOTAL_SECONDS
):
budget_hit = True
break
@@ -322,6 +439,20 @@ class Ingester:
# resume_cursor None, so everything counts.
if not (resume_cursor and page_cursor == resume_cursor):
chunk_new_posts += 1
# Inside the revisit window? Computed per post rather than
# "stop once one post is old" because the feed is only MOSTLY
# date-ordered — a pinned or re-pinned post can sit above older
# ones, and one such post must not end the walk.
in_window = False
if horizon is not None:
published = _parse_published((post_meta(post) or {}).get("date"))
in_window = published is not None and published >= horizon
# Set by the post-record block below when this post was already
# captured on an earlier walk. Stays False when the platform has
# no post-record seam, so the revisit accounting simply reports
# nothing rather than guessing.
post_already_recorded = False
downloaded_before = downloaded
# Tier-gated post (#874): the account can't fully view it, so
# Patreon serves only blurred locked-preview media. Skip it
# ENTIRELY — no media download AND no post-record stub (operator
@@ -353,11 +484,31 @@ class Ingester:
set() if recapture_records
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
rec = write_post_record(post, artist_slug)
posts_recorded += 1
if rec.body_chars:
posts_with_body += 1
post_already_recorded = pkey in already
# A post inside the revisit window is re-read even
# though the gate has it: that gate's whole job is to
# stop us paying for a post twice, and an EDITED post is
# not the same post. `revisit=True` keeps the cost at
# zero requests — the downloader re-reads the body from
# the feed response already in hand and declines to
# write at all if that body came back empty, so a
# detail-fetched body is never overwritten by a blank.
if not post_already_recorded or in_window:
rec = write_post_record(
post, artist_slug, revisit=post_already_recorded,
)
if not post_already_recorded:
# FIRST captures only feed the #862 body canary.
# A revisit legitimately comes back empty — a
# post whose body only ever arrived from the
# detail endpoint has none in the feed, and the
# downloader declines to write it. Counting
# those into the sample would walk the canary
# toward firing on healthy ticks, which is the
# one thing a drift alarm must never do.
posts_recorded += 1
if rec.body_chars:
posts_with_body += 1
if rec.path is not None:
post_records.append(str(rec.path))
self._mark_seen(source_id, [(pkey, ppid)])
@@ -367,7 +518,8 @@ class Ingester:
# a 0-char body is the "why is this one empty" answer.
log_lines.append(
f" post {ppid} [{rec.post_type or '?'}] "
f"body: {rec.body_chars} chars"
+ ("re-read, " if post_already_recorded else "")
+ f"body: {rec.body_chars} chars"
+ ("" if rec.body_chars else " — EMPTY")
+ (f" — {rec.title}" if rec.title else "")
)
@@ -400,6 +552,13 @@ class Ingester:
recapture=recapture,
)
# An on-disk file is only "done" if something imported it. One
# with no ImageRecord at its path was written by a run that died
# before phase 3 — it goes to import, not to the ledger.
imported_paths = self._recorded_paths([
str(o.path) for o in outcomes
if o.status == "skipped_disk" and o.path is not None
])
to_mark: list[tuple[str, str]] = []
to_clear: list[str] = [] # recovered → drop any dead-letter row
to_fail: list[tuple[str, str, str]] = [] # (key, post_id, error)
@@ -411,11 +570,29 @@ class Ingester:
downloaded += 1
if outcome.path is not None:
written.append(str(outcome.path))
to_mark.append((key, media_item.post_id))
fetched.append((key, media_item.post_id))
to_clear.append(key)
consecutive_seen = 0
elif (
outcome.status == "skipped_disk"
and outcome.path is not None
and str(outcome.path) not in imported_paths
):
# On disk, never imported: a prior run wrote it and died
# before phase 3. Import it now. Safe to feed to
# attach_in_place because no record owns this path —
# the unlink below is about a file that IS the record.
written.append(str(outcome.path))
fetched.append((key, media_item.post_id))
to_clear.append(key)
skipped_count += 1
consecutive_seen += 1
log_lines.append(
f" post {media_item.post_id} — on disk but never "
f"imported: {outcome.path.name}"
)
elif outcome.status == "skipped_disk":
# Already on disk (a prior run). Reconcile the ledger so a
# Already on disk and imported. Reconcile the ledger so a
# later tick skips it at tier-1 without a disk stat, but
# do NOT re-feed it to phase 3 — attach_in_place would see
# the duplicate sha256 and unlink the on-disk copy.
@@ -451,7 +628,15 @@ class Ingester:
to_fail.append((key, media_item.post_id, outcome.error or "error"))
# An error neither advances nor resets the run-of-seen.
if mode == "tick" and consecutive_seen >= seen_threshold:
# `not in_window` is the revisit window's half of the
# early-out: a run of already-seen items is only permission
# to stop once the walk is BELOW the horizon. Both halves,
# never either alone — see DEFAULT_REVISIT_DAYS.
if (
mode == "tick"
and not in_window
and consecutive_seen >= seen_threshold
):
early_out = True
break
@@ -465,6 +650,21 @@ class Ingester:
if to_fail:
self._record_failures(source_id, to_fail)
# An already-captured post that yielded NEW media is an edited
# post — the operator's Floppystack case, and the one thing in
# this walk worth naming individually in the run log. The media
# half needed no new detection: `extract_media` reads the media
# list off the live feed response, so a hotfix build appended
# last night is simply a ledger key we have never seen.
new_here = downloaded - downloaded_before
if post_already_recorded and new_here:
revisited += 1
revisit_downloads += new_here
log_lines.append(
f" post {post.get('id')} — updated: "
f"{new_here} new file(s)"
)
# plan #709: time-throttled live progress to the running event so
# the Downloads view ticks ~every 5s, independent of page size.
now = time.monotonic()
@@ -483,9 +683,15 @@ class Ingester:
})
if early_out:
break
if skip_feed is None:
break
skip_feed()
feeds_caught_up += 1
early_out = False
consecutive_seen = 0
else:
reached_bottom = True
# A walk that left feeds early did not read to their ends.
reached_bottom = not feeds_caught_up
except self._error_base as exc:
# The platform's client-error base — _failure_result (adapter)
# maps it to a typed error.
@@ -527,6 +733,13 @@ class Ingester:
# visible in the Raw stdout (e.g. "bodies 3/180" reads as off).
+ (f", bodies {posts_with_body}/{posts_recorded}" if posts_recorded else "")
+ (f", {gated_skipped} gated-skipped" if gated_skipped else "")
# Only when it happened: on a quiet tick this is 0 and saying so
# every run would bury the times it is not.
+ (
f", {revisited} post(s) updated ({revisit_downloads} new file(s))"
if revisited else ""
)
+ (f", {feeds_caught_up} feed(s) caught up" if feeds_caught_up else "")
+ (", reached end" if reached_bottom else "")
+ (", time-boxed" if budget_hit else "")
)
@@ -540,7 +753,13 @@ class Ingester:
# next chunk resumes from the emitted cursor. No progress → TIMEOUT,
# which feeds download_service's backfill stall-guard. rc<0 mirrors
# subprocess TimeoutExpired so completion detection stays false.
made_progress = downloaded > 0 or emitted_cursor != resume_cursor
# Work handed to phase 3 is progress too: a recapture chunk that
# stopped for its imports downloaded nothing, and may not have left
# its first page.
made_progress = (
downloaded > 0 or bool(written) or bool(relink)
or emitted_cursor != resume_cursor
)
if made_progress:
return _result(
success=False, return_code=-1,
@@ -746,6 +965,16 @@ class Ingester:
)
session.commit()
def _recorded_paths(self, paths: list[str]) -> set[str]:
"""Which of `paths` an ImageRecord already points at."""
if not paths:
return set()
with self.session_factory() as session:
rows = session.execute(
select(ImageRecord.path).where(ImageRecord.path.in_(paths))
).scalars().all()
return set(rows)
def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None:
"""Idempotent upsert of (filehash, post_id) seen-ledger rows for a page.
+52
View File
@@ -35,6 +35,58 @@ DEFAULT_SIM_THRESHOLD = 0.85
_FIGURE_KINDS = ("face", "figure")
# How many cosine scores to hold in memory at once, per matmul block.
# 4M float32 is 16 MB — small enough to stay in cache-friendly territory on the
# shared ml lane, large enough that the per-call overhead stops mattering.
_MAX_SCORE_ELEMS = 4_000_000
def char_maxima(q_by_image, allref, seg, np, *, max_elems=_MAX_SCORE_ELEMS):
"""(n_images, n_chars) — each image's best cosine to each character.
`q_by_image` is one L2-normalised `(n_figures, dim)` array per image, in
the order the answer comes back in. `allref` is every character's
prototypes stacked, and `seg` their per-character start offsets into it.
## Why this is batched, and why that is safe
`scheduled_ccip_auto_apply` did this one image at a time — a `(nq, dim) @
(dim, total)` product per image, over every image in the library on every
run. At ~119k images that is 119k separate matmuls, each too small to pay
for its own BLAS setup, and on 2026-09-23 the daily sweep hit its 1800s
soft limit on the operator's instance.
Batching changes no arithmetic. The score a character gets for an image is
a max over that image's figures AND over that character's prototypes, and
max does not care in what order or grouping it is taken — so reducing the
prototype axis first (per row, inside a block) and the figure axis after
(per image, across blocks) gives exactly what the per-image loop gave.
That equivalence is what `test_char_maxima_matches_the_per_image_loop`
pins, against the naive form written out longhand.
Blocked by ROWS rather than done in one product, because the full score
matrix is (all figures in the chunk x every prototype) and that grows with
the library on both axes. The block bound is on elements, so the memory
this uses stays flat as either axis grows.
"""
counts = [len(q) for q in q_by_image]
rows = np.vstack(q_by_image)
total = max(int(allref.shape[0]), 1)
block = max(1, max_elems // total)
per_row = np.empty((rows.shape[0], len(seg)), dtype=np.float32)
for a in range(0, rows.shape[0], block):
scores = rows[a:a + block] @ allref.T
per_row[a:a + block] = np.maximum.reduceat(scores, seg, axis=1)
# Start offset of each image's rows. Every image has at least one figure —
# it is in `q_by_image` because a region produced it — so these strictly
# increase, which is what `reduceat` needs to reduce rather than pass a row
# through untouched.
starts = np.cumsum([0] + counts[:-1])
return np.maximum.reduceat(per_row, starts, axis=0)
async def _settings_threshold(session: AsyncSession) -> float:
val = (
await session.execute(
+12 -3
View File
@@ -11,12 +11,21 @@ from pathlib import Path
import numpy as np
from PIL import Image, ImageFile
from ..worker_lanes import LANES_BY_NAME
ImageFile.LOAD_TRUNCATED_IMAGES = True
# Cap torch's intra-op threads so each ml-worker replica is a bounded core
# consumer on a shared node (torch otherwise uses all cores). Keep
# N_replicas × this within the cores allotted to ML to avoid oversubscription.
_INTRA_OP_THREADS = 4
# consumer on a shared node (torch otherwise uses all cores).
#
# Read from the lane rather than restated here. This was a literal 4 beside a
# comment reading "keep N_replicas x this within the cores allotted to ML" —
# a constraint written where nothing could act on it, and nothing did: the ML
# ceiling came from memory alone, offered the operator ~49 slots on a
# large-memory host, and the lane spent 2026-09-23 with ~200 torch threads on
# it. `derived_ceiling` now divides the cores by this number, which only means
# anything while the two are the same number.
_INTRA_OP_THREADS = LANES_BY_NAME["ml"].threads_per_slot
DEFAULT_MODEL_NAME = os.environ.get(
"SIGLIP_MODEL_NAME", "google/siglip-so400m-patch14-384"
+8 -2
View File
@@ -483,8 +483,14 @@ class PatreonClient:
@staticmethod
def post_meta(post: dict) -> dict:
"""Title + published date for a post — for the preview sample (plan #708
B4). Part of the client contract `ingest_core.Ingester.preview` calls."""
"""Title + published date for a post. Part of the client contract.
Written for a preview sample (plan #708 B4) whose caller has since gone;
as of 2026-09-23 its consumer is the core's REVISIT WINDOW, which needs
a post's date to know whether a tick is still inside it. Both native
clients answer in the same shape — an ISO-8601 string under `date`, or
None — so the core reads a date without knowing the platform.
"""
attrs = post.get("attributes") or {}
title = attrs.get("title")
published = attrs.get("published_at")
+32 -4
View File
@@ -340,7 +340,7 @@ class PatreonDownloader(BaseNativeDownloader):
def _write_sidecar_data(
self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
minimal: bool = False,
minimal: bool = False, detail_fetch: bool = True,
) -> Path:
"""Serialize the post's metadata to `sidecar_path`. The post-only record
(`write_post_record`) writes the FULL post (body/title/date/url); the
@@ -364,7 +364,13 @@ class PatreonDownloader(BaseNativeDownloader):
# dict — so a multi-image post fetches detail at most once, the post-record
# body-length read reuses it, and a fully-seen post (no fresh download → no
# sidecar write) never pays the extra GET.
if (not content or not content.strip()) and self._content_fetcher:
# `detail_fetch=False` on a REVISIT (a post inside the tick's revisit
# window that we already captured): re-read the body from the feed
# response we are holding and pay nothing. Without this a 30-day window
# would buy one detail GET per body-less post per tick, forever — a
# per-creator cost that grows with how prolific they are, to re-fetch a
# body we already stored.
if (not content or not content.strip()) and self._content_fetcher and detail_fetch:
fetched = self._content_fetcher(str(post.get("id") or ""))
if fetched:
content = fetched
@@ -386,7 +392,9 @@ class PatreonDownloader(BaseNativeDownloader):
sidecar_path.write_text(json.dumps(data, indent=2))
return sidecar_path
def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
def write_post_record(
self, post: dict, artist_slug: str, *, revisit: bool = False,
) -> PostRecordOutcome:
"""Write a post-ONLY sidecar (no media file) for a media-less post, so
the importer can still upsert the Post + its body — text posts often hold
the only copy of an external <a href> link. Named `_post.json`: the
@@ -397,6 +405,18 @@ class PatreonDownloader(BaseNativeDownloader):
Returns a PostRecordOutcome (path None when the post has no id) carrying
the captured body's shape — post_type + final char count — so the engine
can log per-post handling without re-reading the post itself.
`revisit=True` is the tick re-reading a post it already captured
(ingest_core's revisit window, #...). Two differences, both about not
making an update cost more than it is worth:
* no detail-fetch — the body comes from the feed response already in
hand, so a revisit costs zero requests;
* a body that comes back empty writes NOTHING and returns `path=None`.
On a first capture an empty body is the truth about the post; on a
revisit it usually just means this post's body only ever came from
the detail endpoint we just declined to call, and writing it would
blank a stored body to say something we never learned.
"""
attrs = post.get("attributes") or {}
title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
@@ -406,9 +426,17 @@ class PatreonDownloader(BaseNativeDownloader):
return PostRecordOutcome(
path=None, post_type=post_type, title=title, body_chars=0,
)
if revisit:
feed_body = post_body_html(attrs)
if not (isinstance(feed_body, str) and feed_body.strip()):
return PostRecordOutcome(
path=None, post_type=post_type, title=title, body_chars=0,
)
post_dir = self.images_root / artist_slug / "patreon" / post_dir_name(post)
post_dir.mkdir(parents=True, exist_ok=True)
path = self._write_sidecar_data(post, post_dir / "_post.json")
path = self._write_sidecar_data(
post, post_dir / "_post.json", detail_fetch=not revisit,
)
# _write_sidecar_data has by now memoized any detail-fetched body onto
# post["attributes"]["content"], so re-read it for the FINAL char count.
body = (post.get("attributes") or {}).get("content")
+4 -2
View File
@@ -23,8 +23,10 @@ log = logging.getLogger(__name__)
# Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
# here: each runs as a self-pacing subprocess and they're lower-volume. The
# native-ingester platforms are serialized (one paced scrape/API walk at a time).
# Add a platform here to cap it to a single concurrent walk.
SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar"})
# Add a platform here to cap it to a single concurrent walk. Discord most of
# all: every source walks on the operator's ONE user token, and parallel walks
# on a user account are both how its rate limit trips and what gets it flagged.
SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"})
_LOCK_PREFIX = "fc:download_lock:"
+506 -52
View File
@@ -15,24 +15,56 @@ where they already were, a wrong one actively misinforms and then propagates
into whatever reads the association. So the matcher's job is to make a SHORT
list worth reading, not a long list worth trusting.
## Signals, and the one deliberately NOT built
## Two routes, because the evidence is of two different kinds
CIRCUMSTANTIAL evidence says two things happened near each other. It is
additive, weighted, and no single one of its signals may reach the threshold:
1. **Time proximity.** The Patreon post exists in order to announce the drop,
so the two are minutes-to-hours apart. Nearly free, and strong.
2. **The post says so.** These announcements routinely name Discord or carry
an invite link, which is close to a declaration.
3. **A shared marker.** The creator's own tie-back — `🍈🍈` in the Patreon
title and `@everyone 🍈 🍈` in the Discord message — gated on how rare that
marker is in THIS artist's posts, because a habitual emoji is punctuation.
3. **Crop-to-source matching is HELD, on the plan's own instruction** — it is
real work with real false-positive risk, and it is only worth building once
1 and 2 are shown to be insufficient against the operator's actual artists.
Nothing here should be read as evidence it is unnecessary; it is deferred,
and the thing that would justify it is an empty review queue on a pair the
operator can see with their own eyes.
IDENTITY evidence says two things are the same thing, and it gets its own
route (see `IDENTITY_FLOOR`). Two signals, and the stronger one stands rather
than them being summed — saying "the same piece" twice is not more true:
Note also that a naive whole-image SigLIP similarity is NOT that signal. A
cropped teaser and its full version are exactly the pair a whole-image
comparison handles worst, so adding one as a "bonus" would mostly add noise
while looking like progress.
4. **A shared working name.** The creator exports the teaser and the release
from one file, and the internal name survives into both platforms
untouched. Measured on the operator's artist: `ConnFront` ↔ `ConnFront`.
This is the only signal that reaches a pair 23.8 hours apart, which
proximity scores at 0.005.
5. **The drop contains the teaser's image.** Rare, and near-certain when it
happens. It is the one signal needing no cooperation from the creator: it
works on a teaser called `Screenshot 2026-08-13`, and on a creator whose
two platforms share no naming convention.
## The one deliberately NOT built
**Crop-to-source matching stays held, and now for a measured reason rather
than a cautious one.**
It was deferred until the cheap signals could be shown insufficient. They can:
of artist 8's 27 teasers with a drop inside a day, 11 still go unlinked, and
five of those are screenshot teasers carrying no working name at all.
So it was tried, on those exact pairs. Every teaser image was correlated
against every window of every nearby drop image at five scales, with the pairs
the working name independently confirms as ground truth and unrelated
same-artist posts a month away as a control. **It does not separate.** True
pairs score as low as 0.401 while the control reaches 0.605 — the two
distributions overlap, and no threshold divides them.
The reason is the reason the naive version was rejected in the first place,
and it turns out to apply just as hard to the sophisticated one: one artist's
work is stylistically homogeneous, so any whole-image comparison between two
of their pieces is high whether or not it is the same piece. Signal 5 above is
what survived that experiment — it asks a narrower question ("is this the same
image") that the measurement shows is answerable, instead of a broader one
("is this a crop of that") that it shows is not.
## Creator identity comes free, so E4 is not actually a prerequisite
@@ -56,14 +88,27 @@ from __future__ import annotations
import logging
import re
from collections import Counter
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from sqlalchemy import func, or_, select
from sqlalchemy import and_, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import ImportSettings, Post, PostAssociation
from ..models import ImageRecord, ImportSettings, Post, PostAssociation
from ..utils.phash import hamming, hash_bits
from ..utils.text import html_to_plain
from .discord_grouping import DROP_GROUPER
from .post_naming import (
IDENTITY_FLOOR,
MAX_TOKEN_POSTS,
marker_frequencies,
marker_overlap,
rarity,
shared_identity,
token_frequencies,
working_name_tokens,
)
log = logging.getLogger(__name__)
@@ -76,18 +121,83 @@ log = logging.getLogger(__name__)
# enough" arithmetic rather than aspirational: on a busy day an artist posts
# several times, and a matcher that could pair on proximity alone would turn
# every busy day into false pairs. A guard test pins this.
WEIGHTS = {"proximity": 0.55, "declared": 0.45}
WEIGHTS = {"proximity": 0.45, "declared": 0.35, "marker": 0.20}
# A Discord INVITE in the body is close to a declaration; the bare word is
# weaker but still meaningful, because these posts are short and on-topic.
#
# "the server" and its possessives are here because the word `discord` is NOT
# how these creators actually write. Measured across 20,558 Patreon bodies:
# `discord` appears in 486 and `the server` in 37 — but the distribution is the
# point, not the totals. For the artist this step was built for, 21 of 42 posts
# say `discord` and 7 say `the server`, and it is the RECENT ones that say the
# latter: the phrasing drifted once the audience already knew where the server
# was. A vocabulary list written from old posts silently stops matching.
_INVITE = re.compile(r"discord\.(?:gg|com/invite)/", re.I)
_MENTION = re.compile(r"\bdiscord\b", re.I)
_MENTION = re.compile(r"\b(?:discord|(?:the|our|my)\s+server)\b", re.I)
DECLARED_INVITE = 1.0
DECLARED_MENTION = 0.6
MAX_CANDIDATES = 25
# How many just-grouped drops one sweep will look around. A ceiling on the
# `or_` the sweep builds, not a policy — a backfill that authors thousands of
# drops at once should not turn one sweep into a full-library rescan, which is
# the manual button's job.
MAX_RECENT_DROPS = 200
# What a shared name must reach before FC links a pair WITHOUT asking.
#
# 1.0, which under post_naming's post-span counting means the name appears in
# exactly these two posts and nowhere else in the artist's library. That is not
# "strong evidence" — within the library it is conclusive, and the remaining
# ways to be wrong are a mis-parse or the creator reusing a name for a genuinely
# different piece on the same day.
#
# Deliberately above IDENTITY_FLOOR, which is what a name needs to PROPOSE.
# The gap between them is the review queue: real evidence, not certain enough
# for FC to act on by itself. Measured on artist 8, 15 name-sharing pairs: 11
# are conclusive, 2 more propose, 2 fall short of both.
AUTO_LINK_FLOOR = 1.0
# When the drop simply CONTAINS the teaser's image — a pHash within this many
# of 256 bits.
#
# 32, the same number and unit `gallery_service._diversify_similar` already
# calls a near-duplicate. Measured on artist 8, comparing every teaser against
# every drop within a day: pairs the working name independently confirms score
# 0, 0 and 20, and the nearest unrelated same-artist pair in a 29-sample
# control scores **108**. A 76-bit gap, so the threshold is not finely tuned
# and does not need to be.
#
# `utils/phash` warns that the hash alone must not decide a MERGE, because
# variants of one piece collide at this distance. That warning does not invert
# here, it is the point: merging destroys a file, so a variant colliding with
# its original is a loss, while this is asking whether two POSTS are about the
# same piece — and a variant of the drop's image is exactly that. Nothing is
# deleted either way, so no pixel confirm is needed to accept.
DUPLICATE_MAX_DISTANCE = 32
@dataclass(frozen=True)
class _Corpus:
"""One artist's rare-token evidence, gathered once rather than per pair.
Both rare-token signals are scoped to a single artist — a working name and
a marker belong to the person who chose them — so the counts are useless
across artists and expensive to rebuild per candidate. A sweep touches an
artist's posts many times over; this is loaded on the first touch and kept
for the life of the service.
"""
tokens_by_post: dict[int, set[str]]
token_posts: Counter[str]
text_by_post: dict[int, str]
marker_posts: Counter[str]
hashes_by_post: dict[int, list[int]]
hash_posts: Counter[int]
def proximity_signal(gap: timedelta, window: timedelta) -> float:
"""1.0 when the two posts are simultaneous, decaying linearly to 0 at the
@@ -123,6 +233,40 @@ def declared_signal(description: str | None) -> float:
return 0.0
def shared_image(
left: list[int],
right: list[int],
hash_posts: Counter[int],
*,
max_distance: int = DUPLICATE_MAX_DISTANCE,
max_frequency: int = MAX_TOKEN_POSTS,
) -> float:
"""Strength in [0, 1] that the drop contains the teaser's own image.
IDENTITY evidence, and the only one of the three that needs no cooperation
from the creator — it works on a teaser named `Screenshot 2026-08-13`, and
on a creator whose two platforms share no naming convention at all. Where
it fires it is close to certain; it is simply quiet most of the time,
because a teaser is usually a crop rather than a copy.
Rarity-gated on POSTS like the other two: an image the creator puts on many
posts is a banner, not a piece.
"""
if not left or not right:
return 0.0
best = None
for a in left:
for b in right:
d = hamming(a, b)
if d is None or d > max_distance:
continue
span = max(hash_posts.get(a, 1), hash_posts.get(b, 1), 1)
strength = rarity(span, max_frequency)
if best is None or strength > best:
best = strength
return round(best, 4) if best is not None else 0.0
def weighted_score(signals: dict) -> float:
return round(sum(WEIGHTS[k] * signals.get(k, 0.0) for k in WEIGHTS), 4)
@@ -134,81 +278,298 @@ def _post_time(post: Post) -> datetime:
class PostAssociationService:
def __init__(self, session: AsyncSession):
self.session = session
self._corpora: dict[int, _Corpus] = {}
async def _decided(self, announcement_id: int) -> set[int]:
"""Payload posts already proposed for this announcement, in ANY status.
async def _corpus(self, artist_id: int) -> _Corpus:
if artist_id in self._corpora:
return self._corpora[artist_id]
Dismissed pairs are included deliberately: re-proposing a pair the
operator has already rejected on every subsequent scan is the single
behaviour that makes a review queue get ignored.
paths_by_post: dict[int, list[str]] = {}
hashes_by_post: dict[int, list[int]] = {}
# An image is counted under the post a reader SEES it on: a Discord
# message absorbed into a drop contributes to the drop, not to itself.
#
# Keyed on `primary_post_id` alone until 2026-09-24, which on the live
# instance meant a drop never had a name or a hash at all — its images
# are owned by its member messages, and the drop only claims them
# through provenance. The identity route therefore never fired there;
# the tests missed it because they attached images to the drop itself,
# which discord_grouping never does.
#
# Counting by the drop also makes the span honest: five wips of one
# piece posted as five messages and grouped into one drop are ONE post
# as far as "how many posts carry this name" is concerned.
owner = Post.__table__.alias("owner")
rows = await self.session.execute(
select(
func.coalesce(owner.c.absorbed_by_post_id, ImageRecord.primary_post_id),
ImageRecord.path, ImageRecord.phash,
)
.select_from(ImageRecord)
.join(owner, owner.c.id == ImageRecord.primary_post_id)
.where(
ImageRecord.artist_id == artist_id,
ImageRecord.primary_post_id.is_not(None),
)
)
for post_id, path, phash in rows:
paths_by_post.setdefault(post_id, []).append(path)
bits = hash_bits(phash)
if bits is not None:
hashes_by_post.setdefault(post_id, []).append(bits)
text_by_post: dict[int, str] = {}
rows = await self.session.execute(
select(Post.id, Post.post_title, Post.description).where(
Post.artist_id == artist_id
)
)
for post_id, title, description in rows:
text_by_post[post_id] = "\n".join(
part for part in (title, html_to_plain(description) or "") if part
)
corpus = _Corpus(
tokens_by_post={
pid: {t for path in paths for t in working_name_tokens(path)}
for pid, paths in paths_by_post.items()
},
# Both counts take POSTS, which is why they are built from these
# groupings rather than from flat lists — see post_naming.
token_posts=token_frequencies(paths_by_post.values()),
text_by_post=text_by_post,
marker_posts=marker_frequencies(text_by_post.values()),
hashes_by_post=hashes_by_post,
# An image the creator puts on many posts — a banner, a watermark
# plate, a recurring title card — is a habit exactly as a character
# name is, and gets gated the same way. Counted on the EXACT hash,
# which is what a reused file produces.
hash_posts=Counter(
h for hs in hashes_by_post.values() for h in set(hs)
),
)
self._corpora[artist_id] = corpus
return corpus
async def _decided(self, announcement_id: int) -> dict[int, PostAssociation]:
"""Pairs already recorded for this announcement, keyed by payload.
Linked and dismissed pairs are never touched again: re-proposing a pair
the operator has already rejected on every subsequent scan is the
single behaviour that makes a review queue get ignored.
A PENDING pair is different — nobody has decided it — so the caller
re-scores it. Otherwise a pair queued by an older, weaker matcher sits
in the queue forever even once the evidence is conclusive, which is the
chore the operator asked FC not to hand them.
"""
rows = (await self.session.execute(
select(PostAssociation.payload_post_id)
select(PostAssociation)
.where(PostAssociation.announcement_post_id == announcement_id)
)).scalars().all()
return set(rows)
return {a.payload_post_id: a for a in rows}
async def _candidate_groups(
self, announcement: Post, *, window: timedelta,
) -> list[Post]:
"""Synthetic Discord groupings by the SAME artist, inside the window.
) -> list[tuple[Post, datetime]]:
"""Synthetic Discord groupings by the SAME artist with a message inside
the window — each with the time of its message CLOSEST to the teaser.
Same-artist is the identity signal and it is free (see the module
docstring on E4). It is also a hard filter rather than a scored one:
two different creators posting minutes apart is a coincidence, not
evidence, and letting it score at all would mean a busy hour across the
library could out-vote everything else.
Measured on the MESSAGES, not the drop's own date. A drop is dated by
its first message, and since #4390 merges a creator's trickle into one
drop, that can be days before the release the teaser announces —
"Very early Marin" on Sep 7, `MarinaraSauce_base` on Sep 11. Matching
on the drop's date would put every merged trickle outside the window.
"""
at = _post_time(announcement)
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
return (await self.session.execute(
select(Post)
member = Post.__table__.alias("member")
member_at = func.coalesce(member.c.post_date, member.c.downloaded_at)
rows = list((await self.session.execute(
select(member.c.absorbed_by_post_id, member_at)
.where(
member.c.artist_id == announcement.artist_id,
member.c.absorbed_by_post_id.is_not(None),
member_at >= at - window,
member_at <= at + window,
)
)).all())
# The drop's own date counts too — the first message's, so it adds
# nothing for a real drop, but it keeps a drop with no member rows
# (hand-built, or one whose messages were removed) matchable.
own_at = func.coalesce(Post.post_date, Post.downloaded_at)
rows += (await self.session.execute(
select(Post.id, own_at).where(
Post.artist_id == announcement.artist_id,
Post.synthesized_by == DROP_GROUPER,
own_at >= at - window,
own_at <= at + window,
)
)).all()
closest: dict[int, datetime] = {}
for group_id, when in rows:
if group_id not in closest or abs(when - at) < abs(closest[group_id] - at):
closest[group_id] = when
if not closest:
return []
groups = (await self.session.execute(
select(Post).where(
Post.id.in_(list(closest)),
Post.synthesized_by == DROP_GROUPER,
Post.id != announcement.id,
sort_key >= at - window,
sort_key <= at + window,
)
.order_by(sort_key)
.limit(MAX_CANDIDATES)
)).scalars().all()
ranked = sorted(groups, key=lambda g: (abs(closest[g.id] - at), g.id))
return [(g, closest[g.id]) for g in ranked[:MAX_CANDIDATES]]
async def _claimed(self, announcement_id: int, payload_id: int) -> bool:
"""Is either end of this pair already spoken for by an accepted link?
An auto-link is FC asserting something the operator never saw, so it
only happens where there is nothing to contradict. A drop already
linked to a different announcement is exactly such a contradiction, and
resolving it is a judgement about which one is right — which is the
operator's, not FC's.
"""
return (await self.session.execute(
select(PostAssociation.id).where(
PostAssociation.status == "linked",
or_(
PostAssociation.payload_post_id == payload_id,
PostAssociation.announcement_post_id == announcement_id,
),
).limit(1)
)).scalar() is not None
async def match_post(
self, announcement_id: int, *, threshold: float, window_hours: float,
) -> int:
"""Score one announcement against nearby groupings. Returns proposals made."""
auto_link: bool = False,
) -> tuple[int, int]:
"""Score one announcement against nearby groupings.
Returns `(proposed, linked)` — how many pairs were written, and how
many of those were linked outright rather than queued.
"""
announcement = await self.session.get(Post, announcement_id)
if announcement is None or announcement.synthesized_by is not None:
# A synthetic post cannot announce anything — FC wrote it.
return 0
return 0, 0
window = timedelta(hours=window_hours)
declared = declared_signal(announcement.description)
already = await self._decided(announcement_id)
corpus = await self._corpus(announcement.artist_id)
here = corpus.tokens_by_post.get(announcement.id, set())
here_text = corpus.text_by_post.get(announcement.id, "")
here_hashes = corpus.hashes_by_post.get(announcement.id, [])
made = 0
for group in await self._candidate_groups(announcement, window=window):
if group.id in already:
scored: list[tuple[Post, float, dict, float]] = []
for group, group_at in await self._candidate_groups(announcement, window=window):
prior = already.get(group.id)
if prior is not None and prior.status != "pending":
continue
signals = {
named, token = shared_identity(
here,
corpus.tokens_by_post.get(group.id, set()),
corpus.token_posts,
)
# The two identity signals answer the same question by different
# means, so the stronger one stands rather than them being summed:
# a name and a shared image both say "the same piece", and saying
# it twice is not more true.
copied = shared_image(
here_hashes,
corpus.hashes_by_post.get(group.id, []),
corpus.hash_posts,
)
identity = max(named, copied)
circumstantial = {
"proximity": proximity_signal(
_post_time(group) - _post_time(announcement), window,
group_at - _post_time(announcement), window,
),
"declared": declared,
"marker": marker_overlap(
here_text,
corpus.text_by_post.get(group.id, ""),
corpus.marker_posts,
),
}
score = weighted_score(signals)
score = weighted_score(circumstantial)
# THE TWO ROUTES, and why identity is not simply a fourth weight.
#
# Circumstance and identity answer different questions. Proximity
# and a declaration say two things happened near each other and
# that one of them mentioned Discord; a working name the creator
# uses on these two posts and nowhere else says they are the same
# piece. Averaging those makes the threshold uninterpretable, and
# it costs both: adding identity as a weight dilutes the others
# enough that measured teaser/drop pairs an hour apart stop
# proposing, while capping identity's contribution at its weight
# means the strongest evidence available can never carry a pair on
# its own.
#
# So identity may override, never dilute. Below the floor it is
# recorded for the operator to read and moves nothing — which is
# the conservative direction, since a wrong link asserts that two
# different pieces are one.
if identity >= IDENTITY_FLOOR:
score = max(score, identity)
if score < threshold:
continue
signals = {**circumstantial, "identity": identity}
if copied:
signals["identity_image"] = copied
if token and named >= copied:
# Carried so the queue can say WHY. A review queue that cannot
# explain itself is one the operator learns to click through.
signals["identity_token"] = token
scored.append((group, score, signals, identity))
# Who, if anyone, FC links without asking.
#
# EXACTLY ONE candidate may be conclusive. Two drops sharing a name
# with one teaser at full strength is not a tie to be broken by score —
# it means the name identifies something other than what FC thinks it
# does, and the right response is to queue both and say nothing.
auto_id = None
if auto_link:
conclusive = [c for c in scored if c[3] >= AUTO_LINK_FLOOR]
if len(conclusive) == 1 and not await self._claimed(
announcement.id, conclusive[0][0].id
):
auto_id = conclusive[0][0].id
linked = 0
for group, score, signals, _identity in scored:
status = "linked" if group.id == auto_id else "pending"
if status == "linked":
linked += 1
prior = already.get(group.id)
if prior is not None:
# Re-scored in place: a pending pair keeps its row (and id),
# and only an upgrade to linked counts as news.
prior.score = score
prior.signals = signals
if status == "linked":
prior.status = "linked"
prior.linked_by = "fc"
continue
self.session.add(PostAssociation(
announcement_post_id=announcement.id,
payload_post_id=group.id,
score=score,
signals=signals,
status="pending",
status=status,
linked_by="fc" if status == "linked" else None,
))
made += 1
return made
return made, linked
async def list_pending(self) -> list[dict]:
rows = (await self.session.execute(
@@ -232,14 +593,19 @@ class PostAssociationService:
if a is None:
return None
a.status = "linked"
a.linked_by = "operator"
return {"id": a.id, "status": a.status}
async def dismiss(self, association_id: int) -> dict | None:
a = await self.session.get(PostAssociation, association_id)
if a is None:
return None
# Kept, not deleted — the row is what remembers the rejection.
# Kept, not deleted — the row is what remembers the rejection. It is
# also the undo for a link FC made itself (#4402): the unified card
# dismisses the pair, and the dismissed row stops the next sweep from
# linking it straight back.
a.status = "dismissed"
a.linked_by = None
return {"id": a.id, "status": a.status}
async def linked_for(self, post_ids: list[int]) -> dict[int, list[dict]]:
@@ -273,36 +639,124 @@ class PostAssociationService:
return out
async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict:
"""Score every recent non-synthetic post against nearby groupings."""
async def rescan(
session: AsyncSession, *, now: datetime | None = None, full: bool = False,
) -> dict:
"""Score recent non-synthetic posts against nearby groupings.
`full=True` scores EVERY post by an artist who has Discord drops at all —
the manual button's job, for history that predates the feature or that a
trickle merge (#4390) has just rearranged. It used to share the sweep's
48-hour horizon, so the button described as "a first run over a library
that predates the feature" could not reach that library.
"""
settings = await ImportSettings.load(session)
if not settings.discord_link_enabled:
return {"enabled": False, "scanned": 0, "proposed": 0}
return {"enabled": False, "scanned": 0, "proposed": 0, "linked": 0}
now = now or datetime.now(UTC)
window_hours = float(settings.discord_link_window_hours)
window = timedelta(hours=window_hours)
# Only look at announcements that could still have a partner in range —
# a full-library rescan is the manual button's job, not the sweep's.
horizon = now - timedelta(hours=window_hours * 2)
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
ids = (await session.execute(
if full:
with_drops = select(Post.artist_id).where(
Post.synthesized_by == DROP_GROUPER
).distinct()
ids = set((await session.execute(
select(Post.id).where(
Post.synthesized_by.is_(None),
Post.absorbed_by_post_id.is_(None),
Post.artist_id.in_(with_drops),
)
)).scalars().all())
return await _score(session, settings, ids, window_hours)
ids = set((await session.execute(
select(Post.id).where(
Post.synthesized_by.is_(None),
Post.absorbed_by_post_id.is_(None),
sort_key >= horizon,
)
)).scalars().all()
)).scalars().all())
# ...and announcements sitting next to a drop FC has only JUST authored.
#
# A drop's `post_date` is backdated to its first message, but FC cannot
# write the drop until the message has an embedding and the hourly grouper
# has run — so a drop created this minute can land weeks back in the feed.
# Its neighbours were last swept before it existed, and a sweep keyed only
# on how recent the ANNOUNCEMENT is will never look at them again.
#
# That is #4392's third cause, and it is the one that left a measured 0.800
# pair with an empty review queue on the live instance. The other two were
# about scoring; this one meant nothing was scored at all.
#
# The times are the drops' MESSAGES, not the drops' own dates: a drop that
# grew today by a trickle merge (#4390) is dated by its first stage, days
# earlier, while the teaser sits beside the message that just joined.
recent = (
select(Post.id).where(
Post.synthesized_by == DROP_GROUPER,
func.coalesce(Post.last_grew_at, Post.downloaded_at) >= horizon,
)
.order_by(func.coalesce(Post.last_grew_at, Post.downloaded_at).desc())
.limit(MAX_RECENT_DROPS)
)
member = Post.__table__.alias("member")
drop_times = set((await session.execute(
select(func.coalesce(member.c.post_date, member.c.downloaded_at))
.where(member.c.absorbed_by_post_id.in_(recent))
)).scalars().all())
# Plus each drop's own date — its first message's, so a duplicate for a
# real drop, but what a drop with no member rows is placed by.
drop_times |= set((await session.execute(
select(sort_key).where(Post.id.in_(recent))
)).scalars().all())
# The interval arithmetic is done in Python rather than SQL: a handful of
# literal ranges is portable, and `now - INTERVAL` is not.
ranges = [
and_(sort_key >= at - window, sort_key <= at + window)
for at in drop_times
if at is not None
]
if ranges:
ids |= set((await session.execute(
select(Post.id).where(
Post.synthesized_by.is_(None),
Post.absorbed_by_post_id.is_(None),
or_(*ranges),
)
)).scalars().all())
return await _score(session, settings, ids, window_hours)
async def _score(
session: AsyncSession, settings: ImportSettings, ids: set[int], window_hours: float,
) -> dict:
svc = PostAssociationService(session)
proposed = 0
for pid in ids:
proposed += await svc.match_post(
linked = 0
# Sorted because `ids` is now a union of two queries: set iteration order
# is arbitrary, and a sweep that visits posts in a different order each
# run is one whose failures cannot be reproduced.
for pid in sorted(ids):
made, auto = await svc.match_post(
pid,
threshold=float(settings.discord_link_threshold),
window_hours=window_hours,
auto_link=bool(settings.discord_link_auto),
)
proposed += made
linked += auto
log.info(
"discord announcement matcher: scanned %d post(s), proposed %d pair(s)",
len(ids), proposed,
"discord announcement matcher: scanned %d post(s), proposed %d pair(s), "
"linked %d outright",
len(ids), proposed, linked,
)
return {"enabled": True, "scanned": len(ids), "proposed": proposed}
return {
"enabled": True, "scanned": len(ids), "proposed": proposed,
"linked": linked,
}
+39 -1
View File
@@ -19,6 +19,7 @@ from ..models import (
ExternalLink,
ImageProvenance,
ImageRecord,
ImportSettings,
Post,
PostAttachment,
Source,
@@ -118,6 +119,15 @@ class PostFeedService:
# from. `around` and `get_post` deliberately do NOT apply this: reaching
# a member by id is how you inspect a grouping.
stmt = stmt.where(Post.absorbed_by_post_id.is_(None))
# A linked Discord drop is shown ON its teaser's card by reference
# (#4402), so its own card sitting beside that teaser is the same
# release twice. Only then is it left out — an older drop keeps its
# place, because a reference does not take anything out of history.
fold_hours = await self._fold_hours()
if fold_hours > 0:
from .post_unification import fold_clause
stmt = stmt.where(fold_clause(fold_hours))
if artist_id is not None:
stmt = stmt.where(Post.artist_id == artist_id)
if platform is not None:
@@ -169,9 +179,12 @@ class PostFeedService:
thumbs_map = await self._thumbnails_for(post_ids)
atts_map = await self._attachments_for(post_ids)
links_map = await self._links_for(post_ids)
unified_map = await self._unified_for([p for p, _, _ in rows])
items = [
self._to_dict(post, artist, source, thumbs_map, atts_map, links_map)
self._to_dict(
post, artist, source, thumbs_map, atts_map, links_map, unified_map,
)
for post, artist, source in rows
]
return {"items": items, "next_cursor": next_cursor}
@@ -213,6 +226,7 @@ class PostFeedService:
anchor_item = self._to_dict(
anchor_post, anchor_artist, anchor_source, thumbs_map, atts_map,
await self._links_for([anchor_post.id]),
await self._unified_for([anchor_post]),
)
return {
"items": newer["items"] + [anchor_item] + older["items"],
@@ -239,6 +253,7 @@ class PostFeedService:
item = self._to_dict(
post, artist, source, thumbs_map, atts_map,
await self._links_for([post.id]),
await self._unified_for([post]),
)
item["description_full"] = html_to_plain(post.description)
# Full (uncapped) translated description for the detail view (#143).
@@ -385,9 +400,27 @@ class PostFeedService:
return await PostAssociationService(self.session).linked_for(post_ids)
async def _unified_for(self, posts: list[Post]) -> dict[int, dict]:
"""The reference set each teaser's card shows (#4402). Local import for
the same reason as `_links_for`: it reaches the association service."""
from .post_unification import PostUnificationService
return await PostUnificationService(self.session).unified_for(posts)
async def _fold_hours(self) -> float:
"""`discord_link_fold_hours`, read without assuming the row exists.
The feed is the one surface that must not fail on a settings row the
caller never needed, so a missing row folds nothing rather than
raising — which is also exactly how the feed behaved before this.
"""
settings = await self.session.get(ImportSettings, 1)
return float(settings.discord_link_fold_hours) if settings is not None else 0.0
def _to_dict(
self, post: Post, artist: Artist, source: Source | None,
thumbs_map: dict, atts_map: dict, links_map: dict | None = None,
unified_map: dict | None = None,
) -> dict:
plain_full = html_to_plain(post.description) if post.description else None
if plain_full is None:
@@ -436,6 +469,11 @@ class PostFeedService:
# post is the drop). Always a list so the UI never branches on
# absence.
"associations": (links_map or {}).get(post.id, []),
# #4402. On a teaser with a linked drop: what the card shows BY
# REFERENCE — the drop's images, the piece's older variants, and
# the text of each — plus who made each link, so one FC made by
# itself can say so and offer the undo. None on every other post.
"unified": (unified_map or {}).get(post.id),
# Non-null on a chat message a synthetic post absorbed. The feed
# filters these out, but `around`/`get_post` still reach them, and
# the UI uses this to explain why a post it linked to is not in the
+426
View File
@@ -0,0 +1,426 @@
"""The creator's own working name for a piece, recovered from a filename.
Milestone 388. Pure functions, no DB and no ML — the whole point is that this
signal is free.
## What this is for
Two of the operator's artists post a censored or cropped teaser on Patreon and
the real release in their Discord. Matching those by IMAGE is the pair a
whole-image comparison handles worst: the teaser is a crop with a censor bar,
which is exactly the local edit that moves a perceptual hash and blurs a
semantic embedding.
But the creator names both exports after the same internal working title, and
that name survives into both platforms untouched. Measured on the live instance
2026-09-24, artist 8:
01_((0-k <-> 0-k_base (1.3h apart)
01_680LC <-> 680LC_Border (21.0h apart)
01_cnni18x <-> cnni18x (21.5h apart)
Three pairs, no false positives, and **two of them are 21 hours apart** — far
enough that time proximity scores them ~0.10 and could never propose them. The
naming signal is orthogonal to the timing one: each finds pairs the other
cannot, which is why both are kept rather than one being tuned to cover both.
## Why a filename and not a perceptual hash
A shared working-name token is IDENTITY evidence — `680lc` appearing on both
platforms is not a coincidence. Proximity is CIRCUMSTANTIAL: it says two things
happened near each other, never that they are the same thing. The distinction
drives the weighting in `post_association_service`, and it is why a rare enough
token is allowed to carry a proposal on its own while no amount of circumstance
is.
## The one false-positive class found, and why the fix is shaped this way
A first pass matched `01_Screenshot 2026-08-13 000004` to
`Screenshot_2026-08-13_032144` on the token `2026-08-13`, twice.
A screenshot filename is a camera artifact. It carries no working name, and the
date inside it collides across platforms on the same day BY CONSTRUCTION — the
teaser and the release are posted the same day, so their screenshot names
always share a date token. That is a signal that fires precisely when it is
least informative.
So a filename with no working name contributes NOTHING, rather than the
plausible-looking date match it could be squeezed for. Re-run with that rule:
the same three true pairs, zero false. Half of this creator's recent teasers
are screenshots, and those pairs are simply out of this signal's reach — which
is where crop-to-source matching earns its cost, and nowhere else.
"""
from __future__ import annotations
import re
from collections import Counter
from collections.abc import Iterable
from pathlib import PurePosixPath
# A screenshot name, on either platform. Patreon's importer writes
# `01_Screenshot 2026-08-13 000004`; gallery-dl's Discord naming writes
# `Screenshot_2026-09-22_003651`. Matched after the index/message prefixes are
# stripped, so both shapes reach this as a bare `Screenshot ...`.
# NOT `\b` after "shot": `\b` needs a word/non-word transition and `_` is a
# WORD character, so `Screenshot_2026-08-13_032144` — gallery-dl's Discord
# spelling — sailed straight past the guard while the space-separated Patreon
# spelling was caught. Found by running this against the live library rather
# than by reading it. Assert the next character is not a letter instead.
_SCREENSHOT = re.compile(r"^(?:screen[ _-]?shot(?![a-z])|\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8|\u30b9\u30af\u30b7\u30e7)", re.I)
# The importer's per-post media index: `01_`, `02_`. Not part of any name.
_MEDIA_INDEX = re.compile(r"^\d{1,3}_")
# gallery-dl's Discord filename pattern (#3999):
# `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}`.
_DISCORD_PREFIX = re.compile(r"^\d{8}_\d{6,}_\d{1,3}_")
# The legacy era (#4002): images sit FLAT at the artist root as
# `<post id>_media_<media id>_<name>`. Stripping it is not cosmetic — the
# SCREENSHOT guard below matches from the start of the stem, so while this
# prefix was left on, a legacy screenshot never looked like one. Measured on
# tamadaheijun: `109078417_media_334848471_Screenshot 2025-07-27 182450ab`
# sailed through and contributed `2025-07-27`, which is precisely the
# same-day date collision this module was built to refuse.
_LEGACY_PREFIX = re.compile(r"^\d+_media_\d+_")
# The importer's content-hash suffix, `__<10 hex>`, sometimes doubled on files
# that went through an older import era.
_HASH_SUFFIX = re.compile(r"(?:__[0-9a-f]{10})+$")
# Generic export decorations. Stripped as SUFFIXES so the stem survives:
# `cnni18x_wip3` and `cnni18x` must yield the same token, or a work-in-progress
# would never match the piece it became.
_DECORATION = re.compile(
r"(?:[_-]?(?:wip|base|final|alt|alts|edit|edits|border|clean|raw|hd|full|"
r"censored|uncensored|nsfw|sfw|ver|v)\d*)+$",
re.I,
)
# Tokens that carry no identity even when they survive the rules above.
_STOPWORDS = frozenset({
"img", "image", "untitled", "new", "test", "page", "final", "copy",
"post", "media", "file", "avatar", "cover", "banner", "icon", "splash",
# Each of these was MEASURED carrying a false match in the 3..6 frequency
# band, where the rarity gate still admits a token: `capture` across six
# unrelated knuxy posts, `the` and `patreon` out of legacy title-derived
# names, `main` out of `Anya Main CST`, `timeline` and `gif` out of
# tamadaheijun's exports.
"the", "gif", "main", "patreon", "capture", "timeline", "screenshot",
# Literally the string "None": issue #3999's Discord naming rendered
# `{user[name]}` as it for ~1,600 files, so it is the single most common
# "name" in the library and identifies nothing.
"none",
# gallery-dl's fallback when a Discord attachment has no filename of its
# own. Measured on artist 8: four unrelated images across 1,974 days, and
# the one false family the leading-name rule admitted inside 60 days.
"image0",
})
# A bare year: still needed for the TEXT signal, where words and numbers are
# tokenised separately.
_YEAR = re.compile(r"^(?:19|20)\d{2}$")
# An identity token must contain a LETTER. This replaces separate "all digits"
# and "bare year" rules with the property behind both, and it is the rule that
# holds once hyphens are kept inside tokens for `0-k`'s sake: without it
# `2025-07-27` and `3-0002` survive as single tokens, and both were measured
# linking unrelated posts — the second across three of them, out of
# tamadaheijun's `timeline 3-0002` exports.
#
# `0-k`, `680lc`, `cnni18x` and `p59` all keep a letter and are unaffected.
_HAS_LETTER = re.compile(r"[A-Za-z]")
MIN_TOKEN_LEN = 3
# A token appearing in more than this many of ONE ARTIST's POSTS is a habit,
# not an identity — a character name, a series tag, a recurring export preset.
#
# POSTS, not files, and the difference is not bookkeeping. Counting files
# punishes a piece for having many exports, which is the one thing a working
# name is GUARANTEED to do. Measured across the operator's four dual-platform
# artists: knuxy's comic pages carry `p217` on four files spread over exactly
# two posts — the Patreon post and the Discord drop — and file-counting scored
# every one of ~200 such tokens at half strength for it. Counting posts scores
# them 1.00 while still catching the real habits, which span many posts:
# tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4.
#
# Six rather than two, although two posts IS the shape of a teaser and its
# drop, because a creator legitimately revisits one working name: a wip post,
# then an alt, then the release. Measured on artist 8, `cnni18x` and `680lc`
# each span four posts and both are genuine. IDENTITY_FLOOR below is what
# decides how much span a link may carry on its own.
MAX_TOKEN_POSTS = 6
# A shared name at or above this strength is enough to propose a link with NO
# corroboration — it is identity evidence, and the whole reason this module
# exists is that identity survives where circumstance does not.
#
# 0.75 is a token spanning three posts or fewer. Measured on artist 8: of the
# 15 same-artist pairs that share a name, 13 clear this bar, including the
# operator's own example (`0-k`, three posts, 1.3h apart). The two that do not
# — `680lc` and `cnni18x`, four posts each, 21h apart — are real pairs this
# signal will not carry alone; they are the measured cost of not admitting the
# four-post band, where conto's `illustration9` and `maid` also sit.
IDENTITY_FLOOR = 0.75
# A LEADING name spanning this many of ONE ARTIST's posts is a habit — a
# character the creator returns to — not one piece's trickle. Used wherever a
# name gathers a FAMILY: the teaser card's variants (#4401) and the Discord
# grouper's trickle merge (#4390), so the two cannot disagree about what a
# family is.
#
# Its own value rather than MAX_TOKEN_POSTS (6), which is calibrated for
# PAIRING two posts and measured too tight for a family. On artist 8,
# `tentacooler` spans 6 posts over 7 days and `0-k1` 6 posts over 10: both real
# families, both gated out at 6. At 8, `anya` (7) passes the cap — and has no
# pair inside any family window, which is what the window is for.
FAMILY_MAX_POSTS = 8
def _strip_prefixes(stem: str) -> str:
"""Remove the framing each platform's importer adds around the real name."""
stem = _DISCORD_PREFIX.sub("", stem)
stem = _LEGACY_PREFIX.sub("", stem)
stem = _MEDIA_INDEX.sub("", stem)
return _HASH_SUFFIX.sub("", stem)
def _ordered_tokens(path: str) -> list[str]:
"""The identity-bearing tokens of one filename, in the order written.
The one tokenizer both public readings share, so the set of names and the
leading name cannot disagree about what counts as a name.
"""
stem = _strip_prefixes(PurePosixPath(path).stem)
if _SCREENSHOT.match(stem.strip()):
return []
out: list[str] = []
# Hyphens are kept INSIDE tokens — `0-k` is a real working name on the live
# instance, and splitting on hyphen would reduce it to a single character
# and then discard it for being too short.
for raw in re.split(r"[^0-9A-Za-z-]+", stem.lower()):
tok = _DECORATION.sub("", raw).strip("-")
if len(tok) < MIN_TOKEN_LEN:
continue
if tok in _STOPWORDS or not _HAS_LETTER.search(tok):
continue
if tok not in out:
out.append(tok)
return out
def working_name_tokens(path: str) -> set[str]:
"""The identity-bearing tokens in one image's filename.
Returns an EMPTY set for a name that carries no working title — a
screenshot, a bare number, a stopword. Empty means "no evidence", which the
caller must treat as silence rather than as a weak match; see the module
docstring for the false positive that rule exists for.
"""
return set(_ordered_tokens(path))
def leading_name(path: str) -> str | None:
"""The FIRST identity token of a filename — the piece, not its decoration.
Creators lead with what the piece is and trail with what this export of it
is: `Year_20k_wip1`, `not_sombra_21-cumpeen`, `Tentacooler_c_ins`. Content
words sit at the tail, and they span too FEW posts for any frequency cap
to catch — `nude`, `cum` and `top` are on three of artist 8's posts each.
Position is what separates them from a name.
Measured on artist 8, Discord messages 2-60 days apart sharing a gated
token: 121 pairs. The 106 sharing the leading name all read as one piece's
trickle; of the 15 sharing only a trailing word, 14 are sibling pieces
(`Bea_Machamp_Shiny_*` / `Bea_Machoke_Shiny_*`) and one is a plain
collision (`Undyne_insert_bottom_only-C` / `Lichgalclc_Lingerie_Bottom_21`).
None when the name carries no identity at all — see `working_name_tokens`.
"""
tokens = _ordered_tokens(path)
return tokens[0] if tokens else None
def token_frequencies(posts: Iterable[Iterable[str]]) -> Counter[str]:
"""How many of ONE ARTIST's POSTS each working-name token appears in.
Takes posts — each an iterable of that post's image paths — rather than a
flat list of paths, because the unit of the count is the post. See
MAX_TOKEN_POSTS for what that buys; the short version is that a piece with
six exports in one post has used its name once.
Scoped to the ARTIST, not the library: a working name belongs to the person
who chose it, and the same string can be one creator's piece and another's
boilerplate. Built once per artist per sweep, not per candidate pair.
"""
counts: Counter[str] = Counter()
for paths in posts:
counts.update({t for path in paths for t in working_name_tokens(path)})
return counts
def rarity(freq: int, max_frequency: int) -> float:
"""Rarity of one token within an artist's own corpus, in [0, 1].
Shared by EVERY rarity-gated signal deliberately. They carried one each
until 2026-09-24, and the copies drifted: the filename signal grew a
frequency gate and the marker signal never did, so a creator's habitual
emoji scored the same 1.00 as a marker they had used twice. One
definition cannot drift from itself.
Full strength at 2 rather than 1: a genuine match means the token is on
at least two things, so demanding uniqueness would reject every real
pair. Decays to zero AT the cap rather than falling off it, so nothing
sits on a cliff edge.
"""
if freq <= 2:
return 1.0
if freq >= max_frequency:
return 0.0
return (max_frequency - freq) / (max_frequency - 2)
def shared_identity(
left: Iterable[str],
right: Iterable[str],
frequencies: Counter[str],
*,
max_frequency: int = MAX_TOKEN_POSTS,
) -> tuple[float, str | None]:
"""Strength in [0, 1] that two sets of filenames name the SAME piece.
Returns `(strength, token)` — the token is carried back so the proposal can
say WHY it was made. A review queue that cannot explain itself is one the
operator learns to click through without reading.
Strength is a function of the winning token's rarity within the artist's
own library, not of how many tokens matched. One decisive token beats three
vague ones, and a token appearing across forty of this artist's posts is a
habit rather than an identity however exactly it matches.
`frequencies` must be the POST counts from `token_frequencies`.
"""
shared = {t for t in set(left) & set(right) if frequencies.get(t, 0) <= max_frequency}
if not shared:
return 0.0, None
# The rarest shared token decides — one decisive token beats three vague
# ones.
token = min(shared, key=lambda t: (frequencies.get(t, 0), -len(t), t))
strength = round(rarity(max(frequencies.get(token, 1), 1), max_frequency), 4)
# A token sitting exactly ON the cap decays to zero, and naming it anyway
# would hand the review queue a reason that carries no weight — "matched on
# loislanetb2", with nothing behind it. Measured: that token is on 6 of this
# artist's images. Report a token only when it is doing work.
return (strength, token) if strength > 0 else (0.0, None)
# --- the body/title signal ---------------------------------------------------
#
# The same idea applied to TEXT. The operator's example pair carries `🍈🍈` in
# the Patreon title and `@everyone 🍈 🍈` in the Discord message — a marker the
# creator uses to tie the two together, which no vocabulary list would predict.
#
# Rarity-gated, exactly as the filename signal is, and the gate is here because
# the first pass did NOT have one. Measured on artist 8, 300 posts:
#
# 💦 11 posts (4%) 🫴 6 🌰 5 🍗 5 🫣 4
#
# 💦 is punctuation for this creator — about one post in twenty-five. Ungated it
# scored a full 1.00 and was the DECIDING term in a proposal that proximity
# alone (0.441) could not carry. A habitual marker riding along with proximity
# is just proximity wearing a hat, which is the exact failure the matcher's
# threshold sits above 0.55 to prevent. The operator's 🍈🍈 is the opposite
# case: two posts, and they are the pair itself.
_WORD = re.compile(r"[0-9A-Za-z]{3,}")
# Anything outside the Basic Multilingual Plane's text ranges: emoji, symbols,
# kaomoji parts. These are the tokens creators actually use as markers, and
# they are rare enough in prose to be evidence on their own.
# U+1F000-1FAFF is the emoji planes; U+2190-2BFF covers arrows, dingbats and
# the miscellaneous-symbol blocks, which already contains U+2600-27BF.
_SYMBOL = re.compile(r"[\U0001F000-\U0001FAFF\u2190-\u2BFF]")
_COMMON_TEXT = frozenset({
"the", "and", "for", "you", "new", "out", "now", "this", "that", "with",
"everyone", "here", "post", "all", "art", "one", "get", "has", "are",
})
# A marker in more than this many of ONE ARTIST's posts is a signature, not a
# tie-back. A marker tying an announcement to its drop lands on two posts —
# the two.
#
# Tighter than MAX_TOKEN_POSTS. Both count posts, so the numbers are directly
# comparable and the gap between them is the claim being made: a working name
# is the creator's private label for one piece and may honestly recur as they
# revisit it, while a marker is public decoration and stops being evidence the
# moment it is reused. Measured on artist 8: 💦 spans 13 posts, the word
# "like" 43, and 🌗 — a real tie-back — exactly 2.
MAX_MARKER_POSTS = 4
def text_markers(text: str | None) -> set[str]:
"""Distinctive tokens in a post body or title: symbols, and rare-ish words.
Symbols count individually rather than as a run, so `🍈🍈` and `🍈 🍈` —
which is how the same marker appears on the two platforms — reduce to the
same token. Spacing is a platform's rendering, not the creator's intent.
"""
if not text:
return set()
out = {m.group(0) for m in _SYMBOL.finditer(text)}
out |= {
w.lower() for w in _WORD.findall(text)
if w.lower() not in _COMMON_TEXT and not _YEAR.match(w)
}
return out
def marker_frequencies(texts: Iterable[str | None]) -> Counter[str]:
"""How many of ONE ARTIST's posts each marker appears in.
Per POST, not per occurrence: a creator who repeats an emoji six times in
one body has used it once as far as identity goes. Scoped to the artist for
the same reason `token_frequencies` is — a marker is a personal habit, and
one creator's signature is another's whole vocabulary.
"""
counts: Counter[str] = Counter()
for t in texts:
counts.update(text_markers(t))
return counts
def marker_overlap(
left: str | None,
right: str | None,
frequencies: Counter[str],
*,
max_frequency: int = MAX_MARKER_POSTS,
) -> float:
"""Strength in [0, 1] that two texts share a DELIBERATE marker.
`frequencies` is required rather than defaulted to "no gate". An ungated
call is the bug this signature exists to make impossible to write by
accident, and a default would have kept it one keyword away.
Symbols weigh full and words a quarter, because prose shares words by
accident: a creator who writes "commission" in both posts on a Tuesday has
told us nothing that the timestamps did not already say.
There is no divisor. An earlier pass halved the total so that a long body
could not out-vote a short one, which the rarity gate now does properly —
and halving meant the operator's own 🍈🍈 pair, a marker on exactly two
posts, could reach only 0.5. One marker the creator uses nowhere else is
the whole signal, not half of it.
"""
shared = text_markers(left) & text_markers(right)
if not shared:
return 0.0
score = sum(
(1.0 if _SYMBOL.match(t) else 0.25) * rarity(frequencies.get(t, 1), max_frequency)
for t in shared
)
return round(min(1.0, score), 4)
+417
View File
@@ -0,0 +1,417 @@
"""The unified post card — a teaser shows what it points at (#4402, #4401).
Milestone 388. A Patreon teaser is a POINTER: a cropped, censored fragment
whose job is to say "the full set is in Discord". Until this module the card
rendered the fragment and a text link, and the reader had to make the join FC
had already made.
Operator, 2026-09-24: *"the teaser from the patreon post doesn't show the items
that it's supposed to reference so I'm trying to unify the teaser post with the
content it's meant to draw attention to."*
## A reference, never an absorption
`discord_grouping` folds chat messages into a synthetic post by transferring
ownership (`absorbed_by_post_id`). That is the wrong primitive here, and the
operator said so directly: *"the nested items on the unified post are a
duplicate or reference of existing content. that's why they can show similar
items and not erase or invalidate the way the discord items landed."*
So nothing here writes. The Discord posts keep their own rows, dates and
places in the feed; the teaser's card DISPLAYS them. That is also what makes
reaching back for older variants safe at all: a wrong reference shows one
extra thumbnail in one place, where a wrong regrouping would move content.
## What a teaser references
1. The Discord drops a `linked` PostAssociation joins it to (#4392) — accepted
by the operator, or linked by FC on a conclusive name match.
2. The rest of that piece's VARIANT FAMILY (#4401): the wips, alts and censor
passes a creator trickles out under one working name, days or weeks apart.
Families are found by the creator's LEADING working name, not by any shared
token and not by image similarity — see `post_naming.leading_name` for the
measurement, and lesson #4400 for why a whole-image comparison between two
works by one artist cannot separate "same piece" from "same artist".
"""
from __future__ import annotations
from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass
from datetime import datetime, timedelta
from sqlalchemy import and_, exists, extract, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import aliased
from ..models import (
ImageProvenance,
ImageRecord,
ImportSettings,
Post,
PostAssociation,
Source,
)
from ..utils.phash import hamming, hash_bits
from ..utils.text import html_to_plain, truncate_at_word
from .discord_grouping import PLATFORM as DISCORD
from .gallery_service import thumbnail_url
from .post_association_service import DUPLICATE_MAX_DISTANCE
from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies
# The text each referenced post contributes to the card, per post. The card
# clamps it again; this keeps a long Discord thread from making the feed
# payload the size of the thread.
TEXT_LIMIT = 280
@dataclass(frozen=True)
class Candidate:
"""One image as the family search sees it."""
image_id: int
post_id: int
path: str
phash: int | None
at: datetime
def family(
seed: Iterable[Candidate],
pool: Iterable[Candidate],
name_posts: Counter[str],
hash_posts: Counter[int],
*,
anchor: datetime,
window: timedelta,
max_posts: int = FAMILY_MAX_POSTS,
) -> list[Candidate]:
"""The images in `pool` that belong to the same piece as `seed`.
A member shares a seed image's LEADING working name, or is a perceptual
near-duplicate of one (the same file re-posted), and lies within `window`
of `anchor` — the teaser's date. Oldest first, so the card reads as the
trickle it was.
ONE hop from the seed, never transitive. Every measured family is one hop
from any of its members, because the members share the name; chaining is
what lets a family drift from `Year_20k` to whatever `Year_20k_Base`'s
other tokens happen to touch.
Both identity routes are rarity-gated against `max_posts`, exactly as the
matcher gates them. A leading name the creator uses across many posts is a
character, and a hash on many posts is a banner.
"""
seed = list(seed)
names = {
name for c in seed
if (name := leading_name(c.path)) is not None
and rarity(name_posts.get(name, 0), max_posts) > 0
}
hashes = [
c.phash for c in seed
if c.phash is not None and rarity(hash_posts.get(c.phash, 0), max_posts) > 0
]
taken = {c.image_id for c in seed}
out: list[Candidate] = []
for c in pool:
if c.image_id in taken or abs(c.at - anchor) > window:
continue
named = leading_name(c.path) in names
copied = c.phash is not None and any(
(d := hamming(c.phash, h)) is not None and d <= DUPLICATE_MAX_DISTANCE
for h in hashes
)
if named or copied:
out.append(c)
taken.add(c.image_id)
return sorted(out, key=lambda c: (c.at, c.image_id))
def _when(post: Post) -> datetime:
return post.post_date or post.downloaded_at
def _text(post: Post) -> str | None:
plain = html_to_plain(post.description) if post.description else None
if not plain or not plain.strip():
return None
return truncate_at_word(plain.strip(), TEXT_LIMIT)[0]
@dataclass
class _Artist:
"""Everything the family search needs about one artist, loaded once."""
rows: dict[int, tuple] # image_id -> (post_id, path, phash, sha, mime, thumb)
posts: dict[int, Post]
platform: dict[int, str | None] # post_id -> platform
name_posts: Counter[str]
hash_posts: Counter[int]
class PostUnificationService:
def __init__(self, session: AsyncSession):
self.session = session
self._artists: dict[int, _Artist] = {}
async def _artist(self, artist_id: int) -> _Artist:
if artist_id in self._artists:
return self._artists[artist_id]
posts: dict[int, Post] = {}
platform: dict[int, str | None] = {}
for post, plat in (await self.session.execute(
select(Post, Source.platform)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.artist_id == artist_id)
)).all():
posts[post.id] = post
platform[post.id] = plat
rows: dict[int, tuple] = {}
paths_by_post: dict[int, list[str]] = {}
hashes_by_post: dict[int, set[int]] = {}
for img_id, post_id, path, phash, sha, mime, thumb in (await self.session.execute(
select(
ImageRecord.id, ImageRecord.primary_post_id, ImageRecord.path,
ImageRecord.phash, ImageRecord.sha256, ImageRecord.mime,
ImageRecord.thumbnail_path,
).where(
ImageRecord.artist_id == artist_id,
ImageRecord.primary_post_id.is_not(None),
)
)).all():
bits = hash_bits(phash)
rows[img_id] = (post_id, path, bits, sha, mime, thumb)
paths_by_post.setdefault(post_id, []).append(path)
if bits is not None:
hashes_by_post.setdefault(post_id, set()).add(bits)
# Counted over EVERY post the artist has, exactly as the matcher's
# corpus counts them — a family is judged against the whole library,
# not against the slice inside the window, or a character name would
# look rare in any quiet month.
found = _Artist(
rows=rows,
posts=posts,
platform=platform,
name_posts=token_frequencies(paths_by_post.values()),
hash_posts=Counter(h for hs in hashes_by_post.values() for h in hs),
)
self._artists[artist_id] = found
return found
async def _drop_images(self, drop_ids: list[int]) -> dict[int, list[int]]:
"""drop post id -> its image ids, through provenance as the feed reads them.
A synthetic drop owns no image outright: its images belong to the
member messages, and `discord_grouping` gives the drop a provenance row
for each. The primary_post_id arm keeps any image that has one and no
row, the same union `PostFeedService._thumbnails_for` takes.
"""
out: dict[int, list[int]] = {pid: [] for pid in drop_ids}
if not drop_ids:
return out
links = (
select(
ImageProvenance.image_record_id.label("image_id"),
ImageProvenance.post_id.label("post_id"),
)
.where(ImageProvenance.post_id.in_(drop_ids))
.union(
select(
ImageRecord.id.label("image_id"),
ImageRecord.primary_post_id.label("post_id"),
).where(ImageRecord.primary_post_id.in_(drop_ids))
)
.subquery()
)
for img_id, pid in (await self.session.execute(
select(links.c.image_id, links.c.post_id).order_by(links.c.image_id)
)).all():
out[pid].append(img_id)
return out
async def unified_for(self, posts: Iterable[Post]) -> dict[int, dict]:
"""post id -> the card's reference set, for each post that HAS one.
Only teasers get one: a post with at least one `linked` association on
the announcing side. Every other post is absent from the result, and
the card renders exactly as it did before this module existed.
"""
teasers = {p.id: p for p in posts if p.synthesized_by is None}
if not teasers:
return {}
links = (await self.session.execute(
select(PostAssociation)
.where(
PostAssociation.status == "linked",
PostAssociation.announcement_post_id.in_(list(teasers)),
)
.order_by(PostAssociation.id)
)).scalars().all()
if not links:
return {}
settings = await self.session.get(ImportSettings, 1)
window = timedelta(days=float(
settings.discord_family_window_days if settings is not None else 60.0
))
drop_images = await self._drop_images(
sorted({a.payload_post_id for a in links})
)
by_teaser: dict[int, list[PostAssociation]] = {}
for a in links:
by_teaser.setdefault(a.announcement_post_id, []).append(a)
out: dict[int, dict] = {}
for teaser_id, assocs in by_teaser.items():
teaser = teasers[teaser_id]
artist = await self._artist(teaser.artist_id)
out[teaser_id] = self._compose(teaser, assocs, drop_images, artist, window)
return out
def _compose(
self,
teaser: Post,
assocs: list[PostAssociation],
drop_images: dict[int, list[int]],
artist: _Artist,
window: timedelta,
) -> dict:
def candidate(img_id: int) -> Candidate | None:
row = artist.rows.get(img_id)
if row is None:
return None
post_id, path, bits, *_ = row
post = artist.posts.get(post_id)
if post is None:
return None
return Candidate(img_id, post_id, path, bits, _when(post))
drop_ids = [a.payload_post_id for a in assocs]
shown = [i for d in drop_ids for i in drop_images.get(d, [])]
own = [i for i, row in artist.rows.items() if row[0] == teaser.id]
seed = [c for i in own + shown if (c := candidate(i)) is not None]
# Variants come from Discord only. That is where a creator trickles
# them out, it is the corpus the family rule was measured on, and it
# keeps one teaser from pulling a DIFFERENT teaser's crop onto its card.
pool = [
c for i, row in artist.rows.items()
if artist.platform.get(row[0]) == DISCORD
and (c := candidate(i)) is not None
]
variants = family(
seed, pool, artist.name_posts, artist.hash_posts,
anchor=_when(teaser), window=window,
)
def thumb(img_id: int, post_id: int, role: str) -> dict | None:
row = artist.rows.get(img_id)
if row is None:
return None
_pid, _path, _bits, sha, mime, tp = row
return {
"image_id": img_id,
"thumbnail_url": thumbnail_url(tp, sha, mime),
"mime": mime,
"post_id": post_id,
"role": role,
}
own_ids = set(own)
thumbnails: list[dict] = []
seen: set[int] = set(own_ids)
for drop_id in drop_ids:
for img_id in drop_images.get(drop_id, []):
if img_id in seen:
continue
if (t := thumb(img_id, drop_id, "drop")) is not None:
thumbnails.append(t)
seen.add(img_id)
for c in variants:
if c.image_id in seen:
continue
if (t := thumb(c.image_id, c.post_id, "variant")) is not None:
thumbnails.append(t)
seen.add(c.image_id)
# The text of every item the card unifies — the operator's *"the
# unified card should also contain the text for any of the items
# unified on it"*. A drop's own description already joins its member
# messages, so a variant's text is its MESSAGE, read off the member
# post that owns the image. A line said twice (`@everyone 🍈🍈` on
# every message of a drop) is shown once.
texts: list[dict] = []
said: set[str] = set()
def add_text(post: Post | None, role: str) -> None:
if post is None:
return
text = _text(post)
if text is None or text in said:
return
said.add(text)
texts.append({
"post_id": post.id,
"role": role,
"date": _when(post).isoformat(),
"text": text,
})
for drop_id in drop_ids:
add_text(artist.posts.get(drop_id), "drop")
for post_id in dict.fromkeys(c.post_id for c in variants):
add_text(artist.posts.get(post_id), "variant")
return {
"links": [
{
"association_id": a.id,
"post_id": a.payload_post_id,
# "fc" | "operator" | None (linked before the column
# existed — an operator accept, every one of them).
"linked_by": a.linked_by,
"token": (a.signals or {}).get("identity_token"),
}
for a in assocs
],
"thumbnails": thumbnails,
"variant_count": sum(1 for t in thumbnails if t["role"] == "variant"),
"texts": texts,
}
def fold_clause(fold_hours: float):
"""WHERE clause: this post is NOT a linked drop sitting beside its teaser.
Operator: *"discord 'posts' land as normal and only hidden from the post
view they're posted the same day."* Everything else stays — an older
variant the teaser also references is history, and a reference does not
remove it from history.
Built on `post_date`/`downloaded_at`, not the feed's `resurfaced_at`-led
sort key: whether two posts are the same release is a question about when
they were published, not about where the feed has since moved one.
"""
teaser = aliased(Post)
# The SQL-standard EXTRACT(epoch FROM …), which every Postgres accepts.
gap = func.abs(extract(
"epoch",
func.coalesce(Post.post_date, Post.downloaded_at)
- func.coalesce(teaser.post_date, teaser.downloaded_at),
))
return ~exists(
select(PostAssociation.id)
.join(teaser, teaser.id == PostAssociation.announcement_post_id)
.where(and_(
PostAssociation.payload_post_id == Post.id,
PostAssociation.status == "linked",
gap <= fold_hours * 3600,
))
)
+36 -1
View File
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
return active
def backfill_ready(source: Source) -> bool:
"""A deep walk the operator started, with budget left and no failure
backing it off — due NOW rather than at its next scheduled check.
A backfill runs one time-boxed chunk per download (plan #693), and nothing
queued the next chunk: each waited for the source's regular interval. At
the 8-hour default a freshly armed backfill sat untouched until the next
check (the operator armed one on 2026-09-25 and saw nothing happen) and a
five-chunk walk took most of two days. The tick's in-flight guard keeps
one chunk at a time per source and the platform lock one walk per
platform, so "due every tick" means "next chunk as soon as the last one
ends".
The failure gate is what keeps a broken source from retrying every
minute: any failed chunk raises `consecutive_failures`, which drops the
source back onto its backed-off interval. A chunk that fails to progress
twice marks the walk stalled (download_service), which ends it here too.
"""
co = source.config_overrides or {}
return (
co.get("_backfill_state") == "running"
and (source.backfill_runs_remaining or 0) > 0
and not (source.consecutive_failures or 0)
)
async def select_due_sources(session: AsyncSession) -> list[Source]:
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
cooldown is the preventive half of the burst-prevention pair (per-source
consecutive_failures backoff handles the offending source itself).
A running backfill (`backfill_ready`) is due on every tick, and whether
or not its artist is on auto-check — the operator started it by hand.
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
sources go first, then the longest-since-checked, so the most overdue
sources hit Celery's FIFO download queue first. Anti-starvation: if
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
.options(selectinload(Source.artist))
.join(Artist, Source.artist_id == Artist.id)
.where(Source.enabled.is_(True))
.where(Artist.auto_check.is_(True))
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
)).scalars().all()
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
for s in rows:
if s.platform in cooldowns:
continue
if backfill_ready(s):
due.append(s)
continue
if not s.artist.auto_check:
continue
interval = compute_effective_interval(s, s.artist, settings)
if s.last_checked_at is None:
due.append(s)
@@ -163,6 +196,8 @@ def compute_next_check_at(
"""Return the projected datetime of the next check, or None if never checked."""
if source.last_checked_at is None:
return None
if backfill_ready(source):
return datetime.now(UTC)
interval = compute_effective_interval(source, artist, settings)
return source.last_checked_at + timedelta(seconds=interval)
+100 -49
View File
@@ -31,19 +31,20 @@ from __future__ import annotations
import asyncio
import logging
from sqlalchemy import func, select
from sqlalchemy import func
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import ServiceSeen
from .worker_lanes import LANES
from .worker_lanes import LANES, lane_for_node
log = logging.getLogger(__name__)
# How stale the roster may be before a health request refreshes it. Comfortably
# under the staleness thresholds that decide a service is missing, so the
# verdict is never limited by how often anyone looked.
REFRESH_TTL_SECONDS = 20.0
# There is no refresh TTL any more. It existed because the HEALTH REQUEST
# refreshed the roster, rate-limited to 20s so that a page open in two tabs
# did not inspect twice as often. `size_worker_lanes` owns the refresh now, on
# `SWEEP_PERIOD_SECONDS`, so the cadence is a schedule rather than a side
# effect of someone looking.
# celery inspect is a broker round trip and this sits on a request path, so it
# gets a deadline (rule 156). A broker that has stopped answering must make the
@@ -118,7 +119,20 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
grouped: dict[tuple[str, ...], dict] = {}
for hostname, queues in active_queues.items():
key = tuple(sorted({q["name"] for q in queues}))
# Keyed on the LANE's queue set when the node name identifies one, so
# a lane keeps the same roster row whether or not it is consuming.
#
# Grouping on the ACTIVE queues alone meant a lane at cap 0 — which
# cancels its consumers — reported an empty set, landed under the key
# `celery:`, and rendered as a phantom row named `Worker ()` while its
# real row went stale beside it. Both symptoms on the operator's
# screen, 2026-09-23, from this one line.
#
# Deriving the key from `lane.queue_key` rather than inventing a new
# one keeps every existing row: it is the same string the lane already
# had while it was running.
lane = lane_for_node(hostname)
key = lane.queue_key if lane else tuple(sorted({q["name"] for q in queues}))
entry = grouped.setdefault(key, {"hostnames": [], "active": 0})
entry["hostnames"].append(hostname)
entry["active"] += len(active_tasks.get(hostname, []))
@@ -127,10 +141,13 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
return grouped
async def touch_service(
session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
) -> None:
"""Record that a part checked in just now.
def touch_service_stmt(*, key: str, kind: str, display_name: str, details: dict):
"""The upsert that records a check-in, as a statement.
Built here rather than inline so the async caller (an agent lease, over
the API) and the sync one (the sizing sweep, in a celery task) run the
SAME write. Two spellings of one upsert is the kind of duplication that
stays correct right up until one of them gains a column.
Upsert rather than read-modify-write: several web processes and several
agents can be doing this at once, and the last writer is simply the most
@@ -141,7 +158,7 @@ async def touch_service(
stmt = pg_insert(ServiceSeen).values(
key=key, kind=kind, display_name=display_name, details=details,
)
stmt = stmt.on_conflict_do_update(
return stmt.on_conflict_do_update(
index_elements=[ServiceSeen.key],
set_={
"kind": stmt.excluded.kind,
@@ -150,7 +167,75 @@ async def touch_service(
"last_seen_at": func.now(),
},
)
await session.execute(stmt)
async def touch_service(
session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
) -> None:
"""Record that a part checked in just now."""
await session.execute(touch_service_stmt(
key=key, kind=kind, display_name=display_name, details=details,
))
def _roster_rows(grouped: dict[tuple[str, ...], dict]) -> list[dict]:
"""The `touch_service` arguments for everything that answered.
Split from the write so the async and sync refreshes below share the
mapping as well as the statement — what a roster row IS should not depend
on which kind of session is writing it.
"""
return [
{
"key": "celery:" + ",".join(queues),
"kind": "celery",
"display_name": role_display_name(queues),
"details": {
"queues": list(queues),
"hostnames": entry["hostnames"],
"replicas": len(entry["hostnames"]),
"active": entry["active"],
},
}
for queues, entry in grouped.items()
]
def refresh_celery_roster_sync(session) -> None:
"""The roster refresh, from the sizing sweep's sync session.
## Why the sweep owns this now
It used to run on the request path, rate-limited to once every 20s by the
newest celery row. So the roster only advanced while somebody had a
browser open — the liveness of the workers was a function of whether
anyone was looking at them, which is the observer-effect version of the
bug this roster exists to prevent.
Operator, 2026-09-23: *"there is a repull every time this page loads — is
there a reason this info isn't being tracked in the background and stored
in some way?"*
Now a timer writes it and the page only reads. The cadence is
`SWEEP_PERIOD_SECONDS`, and `api/system_health` asserts it leaves headroom
under the staleness thresholds — because a sweep period and a stale
threshold chosen in different files and never compared is exactly how the
idle GPU agent came to read as stopped (lesson #4355).
Never raises. A failure means the roster does not advance, and the rows
going stale is then a TRUE report about a broker nobody can reach.
"""
try:
grouped = _inspect_celery_sync()
except Exception:
log.warning(
"service roster: celery inspect failed; roster not refreshed",
exc_info=True,
)
return
for row in _roster_rows(grouped):
session.execute(touch_service_stmt(**row))
session.commit()
async def refresh_celery_roster(session: AsyncSession) -> None:
@@ -173,39 +258,5 @@ async def refresh_celery_roster(session: AsyncSession) -> None:
log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True)
return
for queues, entry in grouped.items():
await touch_service(
session,
key="celery:" + ",".join(queues),
kind="celery",
display_name=role_display_name(queues),
details={
"queues": list(queues),
"hostnames": entry["hostnames"],
"replicas": len(entry["hostnames"]),
"active": entry["active"],
},
)
async def refresh_if_stale(session: AsyncSession) -> None:
"""Refresh the celery roster if nobody has for REFRESH_TTL_SECONDS.
Rate-limited by the data rather than by a lock: the gate is the newest
last_seen_at across the celery rows, which every web process can see. Two
processes racing through the gate costs one redundant inspect and writes
the same values twice, so the benign outcome needs no coordination to
prevent.
"""
newest = (
await session.execute(
select(func.max(ServiceSeen.last_seen_at)).where(ServiceSeen.kind == "celery")
)
).scalar_one_or_none()
if newest is not None:
age = (await session.execute(select(func.now()))).scalar_one() - newest
if age.total_seconds() < REFRESH_TTL_SECONDS:
return
await refresh_celery_roster(session)
for row in _roster_rows(grouped):
await touch_service(session, **row)
+8 -2
View File
@@ -18,6 +18,7 @@ from ..models import (
Source,
)
from .db_helpers import failing_sources_clause
from .download_backends import uses_native_ingester
from .gallery_dl import ErrorType
from .membership_reconcile import KEPT_KEY, STOPPED_KEY
from .membership_roster import gated_reasons_for_sources
@@ -125,6 +126,11 @@ class SourceRecord:
"backfill_posts": self.backfill_posts,
"tier_gated_count": self.tier_gated_count,
"gated_reason": self.gated_reason,
# Recover / recapture exist only on the native ingester. Sent so the
# UI asks the backend's own predicate instead of keeping a copy of
# the platform list — the copy said "patreon, subscribestar" for a
# day after Discord went native (milestone 428).
"native_ingester": uses_native_ingester(self.platform),
}
@@ -551,8 +557,8 @@ class SourceService:
whole source); the two flags are mutually exclusive, so arming recapture
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
fresh from the top. The flag is cleared on completion (download_service)
and on stop. Recapture is Patreon-only (the native ingester's post-record
capture); inert elsewhere. The UI gates the action to Patreon sources."""
and on stop. Recapture needs the native ingester's post-record capture,
so the UI offers it on native sources only (`native_ingester`)."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
+9 -2
View File
@@ -742,8 +742,15 @@ class SubscribeStarClient:
@staticmethod
def post_meta(post: dict) -> dict:
"""Title + date for the preview sample. Title is synthesized from the body
(SubscribeStar has no title field)."""
"""Title + date. Title is None — SubscribeStar has no title field, and
the importer synthesizes one from the body.
`date` is the contract the core's REVISIT WINDOW reads (2026-09-23):
ISO-8601 or None. NAIVE here, because `_parse_ss_datetime` renders a
parsed local timestamp with no zone; the core reads a naive date as UTC
rather than discarding it, since a date it refuses to read is a post
the window can never reach.
"""
attrs = post.get("attributes") or {}
return {"title": None, "date": attrs.get("published_at")}
@@ -184,10 +184,20 @@ class SubscribeStarDownloader(BaseNativeDownloader):
sidecar_path.write_text(json.dumps(data, indent=2))
return sidecar_path
def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
def write_post_record(
self, post: dict, artist_slug: str, *, revisit: bool = False,
) -> PostRecordOutcome:
"""Write the post-first `_post.json` (body/links/metadata) — the sole
writer of the post record on the native path. SubscribeStar's body is
already in the feed HTML, so no detail-fetch is needed."""
already in the feed HTML, so no detail-fetch is needed.
`revisit=True` is the tick re-reading a post it already captured
(ingest_core's revisit window). The no-detail-fetch half of that
contract is free here — there is no detail endpoint — but the
don't-blank-a-stored-body half still applies: a chunk that parsed with
no content must not overwrite a body we already have. Same guarantee as
the Patreon downloader, for the same reason, so a walk behaves the same
on both platforms."""
attrs = post.get("attributes") or {}
title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
@@ -196,6 +206,12 @@ class SubscribeStarDownloader(BaseNativeDownloader):
return PostRecordOutcome(
path=None, post_type=post_type, title=title, body_chars=0,
)
if revisit:
body = attrs.get("content")
if not (isinstance(body, str) and body.strip()):
return PostRecordOutcome(
path=None, post_type=post_type, title=title, body_chars=0,
)
post_dir = self.images_root / artist_slug / "subscribestar" / post_dir_name(post)
post_dir.mkdir(parents=True, exist_ok=True)
path = self._write_sidecar_data(post, post_dir / "_post.json")
+244 -60
View File
@@ -53,15 +53,17 @@ from dataclasses import dataclass, field
from datetime import UTC, datetime
from sqlalchemy import func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import TaskRun, WorkerLane
from ..models import TaskRun, WorkerLane, WorkerLaneSample
from .worker_lanes import (
LANES,
LANES_BY_QUEUE_KEY,
MIN_POOL_SLOTS,
Lane,
derived_ceiling,
lane_for_node,
)
log = logging.getLogger(__name__)
@@ -189,7 +191,13 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]:
return out
for hostname, queues in active_queues.items():
lane = _lane_for_queues(tuple(q["name"] for q in queues))
# The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its
# consumers cancelled and answers with an empty queue list, which
# matches no lane, so attributing by queues alone dropped every lane
# the operator had turned off and reported it as "not answering".
lane = lane_for_node(hostname) or _lane_for_queues(
tuple(q["name"] for q in queues)
)
if lane is None:
# A deployment slicing CELERY_QUEUES differently. Reported by the
# roster under its raw queue list; it simply has no lane row to
@@ -203,11 +211,10 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]:
state.reserved += len(reserved.get(hostname, []))
state.consuming.update(q["name"] for q in queues)
# `pool.max-concurrency` is the number pool_grow/pool_shrink move and
# the number the UI shows. Absent on a worker whose stats did not
# answer, which leaves pool=None — unknown, not zero.
pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency")
if isinstance(pool, int):
# Absent on a worker whose stats did not answer, which leaves
# pool=None — unknown, not zero.
pool = pool_size((stats.get(hostname) or {}).get("pool") or {})
if pool is not None:
state.pools[hostname] = pool
for state in out.values():
@@ -215,6 +222,27 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]:
return out
def pool_size(pool_stats: dict) -> int | None:
"""How many processes a prefork pool is running NOW, from `inspect stats`.
The length of `processes`, not `max-concurrency`. Celery reports
`max-concurrency` as the pool's `limit`, set once at boot; `pool_grow` and
`pool_shrink` hand straight to billiard and never touch it (celery 5.6
`concurrency/prefork.py`: `self.grow = P.grow`). So it read 1 forever on a
lane booted at 1, and the sizing sweep, computing `target - 1` on every
tick, grew a scheduler capped at 2 to six processes while the System tab
showed "6 / 1" (2026-09-24) — and could never shrink one, since 1 - 1 is 0.
`max-concurrency` is the fallback only for a pool that lists no processes
(a non-prefork pool), where it is the best number there is.
"""
procs = pool_stats.get("processes")
if isinstance(procs, list):
return len(procs)
limit = pool_stats.get("max-concurrency")
return limit if isinstance(limit, int) else None
def effective_slots(target: int) -> int:
"""What a pool can actually be set to. Never below one process.
@@ -334,6 +362,76 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]:
return rows
@dataclass(frozen=True)
class LaneSample:
"""What the sizing sweep last measured about one lane.
The same fields `LaneLiveState` carries, plus the queue depth and WHEN —
because this one is read from a table rather than from the broker, and a
reading with no timestamp invites being presented as current.
`measured_at=None` means no sweep has written this lane yet: a fresh
install inside its first period, or a stack whose beat is not running.
Distinct from `present=False` (something asked, nothing answered), and the
UI says different things about the two.
"""
present: bool = False
replicas: int = 0
pool: int | None = None
active: int = 0
reserved: int = 0
queue_depth: int | None = None
measured_at: datetime | None = None
def _lane_depth(lane: Lane, depths: dict[str, int | None]) -> int | None:
"""A lane's backlog across its queues — None when NOTHING answered.
A queue the broker did not answer for must not be summed as zero: an
unknown depth is not an empty one, and reporting a buried lane as idle is
the direction that matters.
"""
known = [depths.get(q) for q in lane.queues]
if not any(d is not None for d in known):
return None
return sum(d for d in known if d is not None)
def store_lane_samples_sync(session, live: dict[str, LaneLiveState], depths) -> None:
"""Write what the sweep just measured. SYNC — the celery task owns a sync
session, and this is the only place these rows are written.
Upsert per lane, last writer wins, same shape as `service_roster`'s
`touch_service`: two processes sweeping at once is a benign race that
needs no coordination, because both are recording what they actually saw.
A lane that did not answer is STILL written, with `present=False`. Skipping
it would leave the previous reading in place and let the page go on showing
a pool that is no longer there — the stale row would read as a current one
(lesson #4202: the row is the thing that has to change).
"""
now = datetime.now(UTC)
for lane in LANES:
state = live.get(lane.name) or LaneLiveState()
values = {
"lane": lane.name,
"present": state.present,
"replicas": state.replicas,
"pool": state.pool,
"active": state.active,
"reserved": state.reserved,
"queue_depth": _lane_depth(lane, depths),
"measured_at": now,
}
stmt = pg_insert(WorkerLaneSample).values(**values)
session.execute(stmt.on_conflict_do_update(
index_elements=[WorkerLaneSample.lane],
set_={k: v for k, v in values.items() if k != "lane"},
))
session.commit()
@dataclass
class LaneSettings:
"""What the DATABASE knows about the lanes — read and finished with before
@@ -359,58 +457,80 @@ class LaneSettings:
caps: dict[str, int]
oldest_by_queue: dict[str, datetime]
# The sizing sweep's last reading per lane. Since 2026-09-23 this is where
# the live numbers come from: the endpoint no longer inspects at all.
samples: dict[str, LaneSample] = field(default_factory=dict)
async def lane_settings(session: AsyncSession) -> LaneSettings:
"""Every DB read the lane view needs, in one short-lived session."""
"""Every DB read the lane view needs, in one short-lived session.
Which is now ALL of them. `lane_view` below takes what this returns and
talks to nothing.
"""
rows = await _rows_by_name(session)
samples = {
row.lane: LaneSample(
present=row.present,
replicas=row.replicas,
pool=row.pool,
active=row.active,
reserved=row.reserved,
queue_depth=row.queue_depth,
measured_at=row.measured_at,
)
for row in (
await session.execute(select(WorkerLaneSample))
).scalars()
}
return LaneSettings(
caps={name: row.slots_cap for name, row in rows.items()},
oldest_by_queue=await _oldest_running_by_queue(session),
samples=samples,
)
async def lane_view(settings: LaneSettings) -> list[dict]:
"""Every lane: what is configured, what is live, what it may grow to.
def lane_view(settings: LaneSettings) -> list[dict]:
"""Every lane: what is configured, what was last measured, what it may
grow to. NO broker call, and no database — `settings` is the whole input.
Takes the settings rather than a session ON PURPOSE — see `LaneSettings`.
Everything below this line is broker work, and no database connection is
held while it happens.
## It used to inspect, on every request
`pending` is the honest backlog — Redis depth PLUS reserved — because
Four broadcast round trips on an eleven-second budget, on a page that
polls every fifteen seconds. Operator, 2026-09-23: *"there is a repull
every time this page loads — is there a reason this info isn't being
tracked in the background and stored in some way?"*
There was one, and it had expired. The docstring here used to say the
endpoint was deliberately uncached because *"this is the surface an
operator watches while dragging a stepper, and a cached reply would show
them the value from before their own change"*. True while a cap change
refetched the table — and that refetch is exactly what was removed in
`1353d34`, so the UI now patches its own row from the write's reply and
nothing depends on this being live.
Meanwhile `size_worker_lanes` was already inspecting on a timer to decide
pool sizes: the same numbers, computed, used, and discarded, while the
browser asked the broker for them again four times a minute.
So the sweep writes `worker_lane_sample` and this reads it. The reading is
up to `SWEEP_PERIOD_SECONDS` old, and `measured_at` travels with it so the
UI can say so rather than implying it is current.
`pending` is still the honest backlog — depth PLUS reserved — because
celery prefetches and LLEN alone reads 0 while a worker holds tasks in
memory.
"""
# A deadline, because this is a request path and `to_thread` on its own is
# an await with no bound (rule 156). `inspect_lanes_sync` never raises and
# every inner call has its own timeout, so the only way past the budget is
# the thread not being scheduled — and a page that renders "not answering"
# is a better answer than one that does not render.
try:
live = await asyncio.wait_for(
asyncio.to_thread(inspect_lanes_sync),
timeout=INSPECT_BUDGET_SECONDS,
)
except TimeoutError:
log.warning(
"worker_control: inspect exceeded %ss; reporting every lane as "
"not answering", INSPECT_BUDGET_SECONDS,
)
live = {lane.name: LaneLiveState() for lane in LANES}
depths = await asyncio.to_thread(_queue_depths_sync)
oldest = settings.oldest_by_queue
now = datetime.now(UTC)
out = []
for lane in LANES:
cap = settings.caps[lane.name]
state = live[lane.name]
# None for a queue the broker did not answer for, which must not be
# silently summed as zero — an unknown depth is not an empty one.
known = [depths.get(q) for q in lane.queues]
depth = sum(d for d in known if d is not None) if any(
d is not None for d in known
) else None
# A lane with no row yet is not-measured, which is distinct from
# measured-as-absent. The default carries `measured_at=None`, and the
# UI says "not measured yet" rather than "not answering".
sample = settings.samples.get(lane.name) or LaneSample()
depth = sample.queue_depth
out.append({
"name": lane.name,
"display_name": lane.display_name,
@@ -437,17 +557,24 @@ async def lane_view(settings: LaneSettings) -> list[dict]:
for m in lane.models
],
"live": {
"present": state.present,
"replicas": state.replicas,
"pool": state.pool,
"active": state.active,
"reserved": state.reserved,
"present": sample.present,
"replicas": sample.replicas,
"pool": sample.pool,
"active": sample.active,
"reserved": sample.reserved,
},
"queue_depth": depth,
"pending": None if depth is None else depth + state.reserved,
"pending": None if depth is None else depth + sample.reserved,
# When the numbers above were read. Per lane rather than one for
# the response, because a lane whose row has never been written
# has no reading at all and must not borrow another lane's.
"measured_at": (
sample.measured_at.isoformat() if sample.measured_at else None
),
# How long the oldest still-running task on this lane has been
# going, in minutes. The operator asked for a trigger here — grow
# a lane whose tasks run past some duration — and it stayed a
# going, in minutes. Read from `task_run`, not from the sweep, so
# this one IS current. The operator asked for a trigger here —
# grow a lane whose tasks run past some duration — and it stayed a
# REPORT: a long task does not finish sooner because the lane
# gained a slot, so scaling on it would spend memory to change
# nothing. Shown so they can see a lane wedged on one slow job,
@@ -556,6 +683,13 @@ async def store_lane_cap(
async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
"""Make the running lane obey a cap that is already stored. NO database.
Runs OFF the request path since 2026-09-23 — the endpoint stores the cap,
answers, and hands this to a background task (operator: *"the change
should be queued so that it isn't blocking of the webui"*). Nothing here
changed as a result except who waits for it: the return value is now read
by the log rather than by a browser, and every branch below already
treated failure as "the sizing pass will carry it".
## What is pushed, and what is not
Consumers follow the cap immediately in BOTH directions: zero means off,
@@ -592,7 +726,22 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
# LOWERED on a running lane. Only this direction needs a message, and
# only when the pool is actually above the new cap — so it reads the
# live pool rather than resizing blind. A raise never reaches here.
live = await asyncio.to_thread(inspect_lanes_sync)
#
# Bounded (rule 156): `to_thread` on its own is an await with no
# deadline, and this runs in a background task where a hang would be
# silent rather than visible as a slow page. On a timeout the lane is
# simply not resized here and the sizing sweep carries it.
try:
live = await asyncio.wait_for(
asyncio.to_thread(inspect_lanes_sync),
timeout=INSPECT_BUDGET_SECONDS,
)
except TimeoutError:
log.warning(
"worker_control: inspect exceeded %ss lowering %s; leaving the "
"pool to the sizing pass", INSPECT_BUDGET_SECONDS, lane.name,
)
return _cap_result(lane, slots_cap, now_on, applied, error, False)
current = live[lane.name].pool
if current is not None and current > slots_cap:
applied, error = await asyncio.to_thread(
@@ -605,13 +754,38 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
# feature that is optional and clearly OFF.
#
# On the TRANSITION, so re-saving a cap on a lane already running does not
# re-enqueue. And only when the consumer change landed: enqueueing onto a
# queue nothing is consuming would leave the task pending with no
# explanation until the lane returns.
# re-enqueue.
#
# NOT gated on the consumer change having landed, which it was until
# 2026-09-23. The reasoning then was that enqueueing onto a queue nothing
# consumes leaves the task pending — true, and it is the right place for
# it to wait. Gated, a cap raised while the lane was restarting stored the
# cap, let the sizing pass start the consumers a minute later, and left
# the lane running with no model, because nothing else ever asks for one.
# A task parked on the `ml` queue is picked up the moment that happens.
fetching = False
if now_on and not was_on and lane.models and applied:
if now_on and not was_on and lane.models:
fetching = _enqueue_model_fetch()
# Nobody is waiting on this any more, so the log is where a push that did
# not land has to be visible. Not an error: the value is stored and the
# sizing pass carries it within a minute.
if not applied:
log.info(
"worker_control: %s cap %s stored, not pushed (%s); "
"the sizing pass will carry it",
lane.name, slots_cap, error,
)
return _cap_result(lane, slots_cap, now_on, applied, error, fetching)
def _cap_result(
lane: Lane, slots_cap: int, now_on: bool, applied: bool,
error: str | None, fetching: bool,
) -> dict:
"""The push's outcome. One builder, because `push_lane_cap` has two exits
and a second literal would be free to disagree with the first."""
return {
"name": lane.name,
"slots_cap": slots_cap,
@@ -716,7 +890,12 @@ def wanted_slots(cap: int, active: int, pending: int | None) -> int:
return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0)))
def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
def size_lanes_sync(
caps: dict[str, int],
*,
live: dict[str, LaneLiveState] | None = None,
depths: dict[str, int | None] | None = None,
) -> list[LaneSizing]:
"""Size every lane to its backlog, within the cap. The whole control loop.
`caps` is lane name -> slots_cap, read from the database by the caller.
@@ -724,6 +903,13 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
the session, and keeping the DB out of here is what lets it be called from
anywhere that already knows the caps.
`live` and `depths` are the measurements. Passing them in is not an
optimisation — it is how the caller gets to KEEP them. The sweep now
stores what it measured (`worker_lane_sample`) so the System tab reads a
table instead of inspecting on every page load, and that is only possible
if the same reading serves both purposes. Measured here when not given, so
every existing caller and test is unaffected.
## It must converge and then go quiet
One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a
@@ -740,8 +926,10 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
be a verdict drawn from an unswept read, and here it is worse than
useless: there is nothing to send the message to.
"""
live = inspect_lanes_sync()
depths = _queue_depths_sync()
if live is None:
live = inspect_lanes_sync()
if depths is None:
depths = _queue_depths_sync()
out: list[LaneSizing] = []
for lane in LANES:
@@ -775,11 +963,7 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
))
continue
known = [depths.get(q) for q in lane.queues]
depth = (
sum(d for d in known if d is not None)
if any(d is not None for d in known) else None
)
depth = _lane_depth(lane, depths)
pending = None if depth is None else depth + state.reserved
want = wanted_slots(cap, state.active, pending)
+117 -19
View File
@@ -139,8 +139,22 @@ class Lane:
# tells them to raise it rather than quietly consuming the machine.
default_slots_cap: int
# True when a slot costs a copy of the ML model rather than just a process.
# The only lane whose ceiling is decided by memory instead of by cores.
# Such a lane is bounded by memory AS WELL AS by cores, never instead of.
memory_bound: bool = False
# CPU threads ONE slot uses. More than one for a lane whose work is an
# inference library with its own thread pool: `services/ml/embedder.py`
# calls `torch.set_num_threads` with this number, so a slot is four cores'
# worth of demand rather than one process's.
#
# It lives here because the CEILING has to know it. It was a private
# constant in the embedder with a comment saying "keep N_replicas x this
# within the cores allotted to ML" — a rule stated where nothing could
# enforce it. Nothing did: the ML ceiling was computed from memory alone,
# so a large-memory host offered ~49 slots, the operator took them, and
# 2026-09-23's log shows ~200 torch threads fighting over the box —
# embeds at 107-246s each, and the daily CCIP sweep sharing that pool
# timing out at 1800s.
threads_per_slot: int = 1
# Models this lane downloads the first time it is enabled. Empty for every
# lane that needs none, which is how the UI knows whether to warn at all.
models: tuple[ModelRequirement, ...] = ()
@@ -209,6 +223,7 @@ LANES: tuple[Lane, ...] = (
entrypoint_role="ml-worker",
default_slots_cap=0,
memory_bound=True,
threads_per_slot=4,
models=(SIGLIP_MODEL,),
optional=True,
),
@@ -267,6 +282,31 @@ MIN_POOL_SLOTS = 1
# not to make a small machine unusable.
MIN_CEILING = 1
# How often `size_worker_lanes` runs — the beat schedule, and the freshness of
# everything the System tab shows.
#
# It is here, in the import-light module, because three places have to agree
# about it and they are in different packages: the beat entry in `celery_app`,
# the sample the sweep writes (`worker_lane_sample`), and the roster's
# staleness thresholds in `api/system_health`, which now depend on this sweep
# rather than on a browser being open.
#
# 30s, down from 60s, because the sweep became the ONLY writer of the celery
# roster on 2026-09-23. A part is called stale after 90s of silence, so a
# 60-second sweep left one missed tick between "normal" and "everything is
# yellow". That is the shape of lesson #4355 — a reader's threshold and an
# emitter's cadence chosen in different files and never compared — and the
# fix is headroom plus a test that asserts it, not a number that happens to
# work today.
#
# The cost is one inspect every 30s instead of every 60s; the saving is every
# inspect that used to run on a request path, which with a single tab open
# was roughly four a minute against this two. Consequence worth knowing: the
# pass also SHRINKS an idle lane by one slot per tick, so an idle lane now
# gives its workers back twice as fast. That is the direction the operator
# asked for — *"idle instances quiet down when not running"*.
SWEEP_PERIOD_SECONDS = 30.0
# What an unreadable limit yields. Low rather than unlimited, on purpose: not
# knowing how much memory there is must never read as "plenty". An unswept
# absence is not a verdict.
@@ -332,6 +372,59 @@ def container_cpu_count() -> int | None:
return os.cpu_count()
def lane_for_node(hostname: str) -> Lane | None:
"""`ml@7f3c9a1b` -> the ml lane. None for a node this build did not name.
## Why the node name, and not the queues it is consuming
Because a lane that is OFF is consuming nothing, and "nothing" identifies
no lane at all.
Both the roster and `inspect_lanes_sync` used to map a worker to its lane
through `active_queues()`. That is exact while the lane is running and
useless the moment it is not: a lane at cap 0 has its consumers cancelled,
so it answers the broadcast with an EMPTY queue list, matches no lane, and
is dropped. Three things followed, and the operator saw all three at once
on 2026-09-23:
1. The lanes table showed the lane as **not answering** — which is the
signal for a crashed worker, not for one the operator turned off.
2. The roster grew a phantom row called **`Worker ()`**, the empty queue
set rendered as a display name, "running" beside the real lane's row
going stale.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires
every lane in the table to be present, and its docstring asserted the
opposite of what the code did — *"a disabled lane still runs its
process with its consumers cancelled, so it answers inspect and is
healthy"*. It answers; it is not attributed. ML ships at cap 0, so a
fresh install would have been permanently unhealthy, and Swarm
restarts an unhealthy task forever.
The node name survives all of that: `gen_supervisord` sets
`CELERY_NODENAME={lane.name}` per program and the entrypoint passes it to
`celery -n`, so the identity travels with the PROCESS rather than with
what it happens to be doing. Falls back to the queue set for a deployment
that sets no node name — the multi-service compose stack, where every node
is `celery@<host>`.
"""
return LANES_BY_NAME.get(hostname.split("@", 1)[0])
def _cpu_bound_slots(lane: Lane) -> int:
"""How many slots this container's cores can feed, at `threads_per_slot`.
Never zero: a machine with fewer cores than one slot wants still runs the
lane, just slowly. That is a real trade an operator may want, and refusing
to offer the lane at all on a small box would make ML unreachable there —
unlike the memory bound, where the honest answer IS zero, because the
first task would OOM the container rather than merely be slow.
"""
cores = container_cpu_count()
if cores is None:
return UNKNOWN_CEILING
return max(MIN_CEILING, cores // lane.threads_per_slot)
def derived_ceiling(lane: Lane) -> int:
"""The most slots `lane` may be given on this container.
@@ -339,26 +432,31 @@ def derived_ceiling(lane: Lane) -> int:
is bounded by what it has NOW rather than by what it had when its row was
written.
"""
if lane.memory_bound:
total = container_memory_bytes()
if total is None:
log.warning(
"worker_lanes: cannot read a memory limit; capping %s at %d",
lane.name, UNKNOWN_CEILING,
)
return UNKNOWN_CEILING
usable = total - RESERVED_BYTES
if usable < ML_BYTES_PER_SLOT:
# Honestly zero. A box that cannot hold one model alongside the web
# process must be told it cannot run tagging, not sold a slot that
# will OOM the container the first time it is used.
return 0
return int(usable // ML_BYTES_PER_SLOT)
by_cpu = _cpu_bound_slots(lane)
if not lane.memory_bound:
return by_cpu
cores = container_cpu_count()
if cores is None:
total = container_memory_bytes()
if total is None:
log.warning(
"worker_lanes: cannot read a memory limit; capping %s at %d",
lane.name, UNKNOWN_CEILING,
)
return UNKNOWN_CEILING
return max(MIN_CEILING, cores)
usable = total - RESERVED_BYTES
if usable < ML_BYTES_PER_SLOT:
# Honestly zero. A box that cannot hold one model alongside the web
# process must be told it cannot run tagging, not sold a slot that
# will OOM the container the first time it is used.
return 0
# BOTH bounds, whichever binds first. Memory alone was the whole answer
# until 2026-09-23, and on a large-memory host that is the wrong one: RAM
# said ~49 slots, and each of those slots wants `threads_per_slot` cores.
# The operator raised the cap to what the dial offered and the lane
# starved itself — a control is not allowed to offer a number the machine
# cannot feed.
return min(int(usable // ML_BYTES_PER_SLOT), by_cpu)
def ceilings() -> dict[str, int]:
+5
View File
@@ -177,6 +177,10 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
settings = ImportSettings.load_sync(sync_session)
rate_limit = settings.download_rate_limit_seconds
validate_files = settings.download_validate_files
# How far back a tick keeps looking for EDITED posts. Read here
# with the other downloader knobs, off the row this block is
# already holding open.
revisit_days = settings.download_revisit_days
gdl = GalleryDLService(
images_root=IMAGES_ROOT,
@@ -207,6 +211,7 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
# the walk). Same factory the importer's sync session
# comes from — a different DB connection per checkout.
sync_session_factory=SyncFactory,
revisit_days=revisit_days,
)
return await svc.download_source(source_id)
finally:
+52 -8
View File
@@ -473,6 +473,10 @@ def prune_task_runs() -> dict:
(recover_stalled_task_runs) is the mechanism that flips them to
terminal state; prune doesn't touch in-flight state.
- 'retry' rows: treated as failures (>7d).
- The NEWEST row of each task is never deleted, whatever its age: it is
what the beat scheduler reads to know when a job last ran (#4408).
Without it a weekly job's last run would be pruned after a day, and beat
would think it had never run and fire it on every restart.
Returns dict of how many rows were deleted in each bucket.
"""
@@ -480,16 +484,19 @@ def prune_task_runs() -> dict:
now = datetime.now(UTC)
ok_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_OK_SECONDS)
fail_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_FAILURE_SECONDS)
newest = select(func.max(TaskRun.id)).group_by(TaskRun.task_name)
with SessionLocal() as session:
ok_deleted = session.execute(
delete(TaskRun)
.where(TaskRun.status == "ok")
.where(TaskRun.finished_at < ok_cutoff)
.where(TaskRun.id.not_in(newest))
).rowcount or 0
fail_deleted = session.execute(
delete(TaskRun)
.where(TaskRun.status.in_(["error", "timeout", "retry"]))
.where(TaskRun.finished_at < fail_cutoff)
.where(TaskRun.id.not_in(newest))
).rowcount or 0
session.commit()
return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@@ -1175,7 +1182,7 @@ def group_discord_drops() -> str:
return "disabled"
return (
f"sources={res['sources']} created={res['posts_created']} "
f"joined={res['images_joined']}"
f"joined={res['images_joined']} merged={res['drops_merged']}"
)
@@ -1184,12 +1191,16 @@ def group_discord_drops() -> str:
soft_time_limit=900, time_limit=1200,
)
def match_post_associations() -> str:
"""Milestone 388 E5: propose which Patreon post announced which Discord drop.
"""Milestone 388 E5: which Patreon post announced which Discord drop.
Proposes only — every pair lands in a review queue and nothing is linked
until the operator accepts. Maintenance lane for the same reason as the
grouper: no inference, no ML library, and it must not depend on the
optional ml-worker being present.
A CONCLUSIVE pair — one the creator's own working name identifies, where
that name appears in these two posts and nowhere else in their library — is
linked outright when `discord_link_auto` is on, because there is nothing
there for the operator to adjudicate. Everything weaker lands in the review
queue and stays unlinked until they accept it.
Maintenance lane for the same reason as the grouper: no inference, no ML
library, and it must not depend on the optional ml-worker being present.
"""
import asyncio
@@ -1368,9 +1379,31 @@ def size_worker_lanes() -> dict:
Returns every lane's outcome INCLUDING the ones it held, each with a
reason. A pass that only speaks when it acts cannot be debugged on the day
it does not.
## It is also the only thing that MEASURES, since 2026-09-23
It always inspected the broker to decide pool sizes, and then threw the
reading away — while `/api/system/workers` ran the same inspect on every
page load and the System tab polls it four times a minute. Operator:
*"there is a repull every time this page loads — is there a reason this
info isn't being tracked in the background and stored in some way?"*
So one inspect now feeds three things: the sizing decision, the stored
sample the System tab reads, and the celery roster. No request path
touches the broker any more.
Order matters. The sample is stored BEFORE the roster refresh, because
that refresh does its own broadcast and a broker that has just started
failing must not cost us the reading we already have.
"""
from ..models import WorkerLane
from ..services.worker_control import size_lanes_sync
from ..services.service_roster import refresh_celery_roster_sync
from ..services.worker_control import (
_queue_depths_sync,
inspect_lanes_sync,
size_lanes_sync,
store_lane_samples_sync,
)
# Read INSIDE the session. Reading a column off a detached instance
# happens to work while the attribute is still loaded and stops working
@@ -1387,7 +1420,18 @@ def size_worker_lanes() -> dict:
# let this task disagree with the seed it is meant to be enforcing.
return {"sized": []}
sized = size_lanes_sync(caps)
# Measured ONCE, here, and then used three times. Passing them down is
# what makes the reading keepable rather than an implementation detail of
# a function that returns decisions.
live = inspect_lanes_sync()
depths = _queue_depths_sync()
sized = size_lanes_sync(caps, live=live, depths=depths)
with _sync_session_factory()() as session:
store_lane_samples_sync(session, live, depths)
refresh_celery_roster_sync(session)
for d in sized:
if d.action not in ("held", "skipped"):
log.info(
+17 -6
View File
@@ -488,7 +488,7 @@ def scheduled_ccip_auto_apply() -> str:
from ..models import ImageRegion, MLSettings, Tag, TagKind
from ..models.tag import image_tag
from ..services.ml.ccip import _FIGURE_KINDS
from ..services.ml.ccip import _FIGURE_KINDS, char_maxima
from ..services.ml.training_data import _applied_or_rejected, _l2norm
SessionLocal = _sync_session_factory()
@@ -553,11 +553,22 @@ def scheduled_ccip_auto_apply() -> str:
by_img: dict[int, list] = {}
for iid, vec in rows:
by_img.setdefault(iid, []).append(vec)
for iid, vecs in by_img.items():
q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768)
colmax = (q @ allref.T).max(axis=0) # (total,)
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
for ci in np.where(charmax >= thr)[0]:
if not by_img:
continue
# One matmul per BLOCK of figures, not one per image. This loop ran
# over every image in the library on every daily run and did a
# matmul too small to pay for itself each time; it hit the 1800s
# soft limit on the operator's instance on 2026-09-23. Same
# arithmetic — see `char_maxima`.
iids = list(by_img)
charmax = char_maxima(
[_l2norm(np.asarray(by_img[i], dtype=np.float32), np) for i in iids],
allref, seg, np,
) # (n_img, n_chars)
for row, iid in enumerate(iids):
for ci in np.where(charmax[row] >= thr)[0]:
t = ref_tags[int(ci)]
if iid in skip[t]:
continue
+27
View File
@@ -80,6 +80,33 @@ def _seek_first_frame(pil_image) -> None:
pass
def hash_bits(hex_str: str | None) -> int | None:
"""A stored pHash hex string as an integer, or None if it is missing or
unparseable. Fails CLOSED, like every other gate in this module.
Parsed to an int rather than an imagehash object because the caller that
needs this compares one image against many: `int.bit_count()` on an XOR is
a machine instruction, where rebuilding a 16x16 boolean array per
comparison is not.
"""
if not hex_str:
return None
try:
return int(hex_str, 16)
except (TypeError, ValueError):
return None
def hamming(a: int | None, b: int | None) -> int | None:
"""Bits differing between two parsed hashes, or None if either is absent.
Out of 256 at HASH_SIZE 16.
"""
if a is None or b is None:
return None
return (a ^ b).bit_count()
def compute_phash(pil_image) -> str | None:
"""Perceptual hash of an opened PIL image, as a hex string. None on any
failure (videos/unreadable/non-image).
+3 -3
View File
@@ -15,7 +15,7 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
## Secondary runtime image
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
node:24-bookworm-slim — `build.yml`'s `extension-test` job only.
`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else
and installs nothing: it needs git and stdlib python, and builds no image.
@@ -29,8 +29,8 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
- `npm install --no-audit --no-fund` — in `frontend-build` job
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
- `unzip` — in `extension.yml`'s "Verify XPI contents" step, installed via apt
- `npm install --no-audit --no-fund` — in `build.yml`'s `extension-test` job (web-ext + vitest)
- `unzip` — in `extension-test`'s "Verify XPI contents" step, installed via apt
only when absent (`node:24-bookworm-slim` may or may not carry it). Debian
package, ~2s. Not worth baking into a shared image for a single consumer, per
`docs/process.md`'s ">1 project" rule.
+1 -1
View File
@@ -29,7 +29,7 @@
services:
redis:
image: redis:7-alpine
image: redis:8-alpine
volumes:
- redis_data:/data
healthcheck:
+16 -1
View File
@@ -48,7 +48,7 @@ x-celery-healthcheck: &celery_healthcheck
services:
redis:
image: redis:7-alpine
image: redis:8-alpine
volumes:
- redis_data:/data
healthcheck:
@@ -180,6 +180,10 @@ services:
environment:
<<: *app_env
CELERY_QUEUES: default,import,thumbnail,download
# Names the celery node for the roster. A lane whose consumers are
# cancelled reports no queues, so the NODE is the only thing left that
# identifies it — see services/worker_lanes.lane_for_node.
CELERY_NODENAME: worker
CELERY_CONCURRENCY: "2"
# /downloads dropped — nothing in the app references it (operator-flagged
# 2026-06-07: it wasn't mapped in prod and everything worked).
@@ -200,6 +204,10 @@ services:
environment:
<<: *app_env
CELERY_QUEUES: maintenance,scan
# Names the celery node for the roster. A lane whose consumers are
# cancelled reports no queues, so the NODE is the only thing left that
# identifies it — see services/worker_lanes.lane_for_node.
CELERY_NODENAME: scheduler
volumes:
- ./images:/images
- ./import:/import
@@ -223,6 +231,10 @@ services:
environment:
<<: *app_env
CELERY_QUEUES: maintenance_long
# Names the celery node for the roster. A lane whose consumers are
# cancelled reports no queues, so the NODE is the only thing left that
# identifies it — see services/worker_lanes.lane_for_node.
CELERY_NODENAME: maintenance_long
CELERY_CONCURRENCY: "1"
# Only /images: backups write to /images/_backups, audits read /images, and
# the admin tasks (re-extract/cascade-delete/normalize) operate on /images.
@@ -241,6 +253,9 @@ services:
deploy: *deploy_policy
environment:
<<: *app_env
# See the worker service — the node name is what identifies a lane
# whose consumers are cancelled.
CELERY_NODENAME: ml
volumes:
- ./images:/images:ro
- ./models:/models
+24 -3
View File
@@ -128,7 +128,8 @@ browser.webRequest.onBeforeSendHeaders.addListener(
saveDiscordToken(auth.value);
}
},
{ urls: ['https://discord.com/api/*'] },
// ptb/canary are Discord's beta clients; their API calls carry the same token.
{ urls: ['https://discord.com/api/*', 'https://*.discord.com/api/*'] },
['requestHeaders'],
);
@@ -213,7 +214,17 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
await api.uploadCredentials('discord', 'token', discordToken);
return { success: true };
// Then have FC try it against a Discord source, so a token Discord
// has already revoked shows up here rather than at the next check.
// A failed verify never undoes the upload: valid=null means FC could
// not test (no Discord source yet), not that the token is bad.
let verify = null;
try {
verify = await api.verifyCredential('discord');
} catch (e) {
verify = { valid: null, reason: e.message };
}
return { success: true, verify };
}
return { error: 'Unsupported platform.' };
} catch (e) {
@@ -256,7 +267,17 @@ browser.runtime.onMessage.addListener(async (msg) => {
case 'ADD_AS_SOURCE':
try {
return await api.quickAddSource(msg.url);
return await api.quickAddSource(msg.url, {
artistId: msg.artistId ?? null,
artistName: msg.artistName ?? null,
});
} catch (e) {
return { error: e.message };
}
case 'SEARCH_ARTISTS':
try {
return { artists: await api.searchArtists(msg.q || '') };
} catch (e) {
return { error: e.message };
}
+57
View File
@@ -40,3 +40,60 @@
from { transform: translateY(20px); opacity: 0; }
to { transform: translateY(0); opacity: 1; }
}
/* Discord Add panel — sits above the chip. Same slate/parchment palette. */
.fc-panel {
all: revert;
position: fixed; bottom: 76px; right: 24px; z-index: 2147483647;
box-sizing: border-box; width: 320px; max-width: calc(100vw - 48px);
padding: 14px 16px; border-radius: 10px;
background: rgb(20, 23, 26); color: rgb(232, 228, 216);
border: 1px solid rgb(60, 64, 70);
font: 14px/1.4 system-ui, sans-serif;
box-shadow: 0 8px 28px rgba(0, 0, 0, 0.5);
}
.fc-panel__title { font-weight: 600; color: rgb(244, 186, 122); }
.fc-panel__sub { color: rgb(170, 166, 156); font-size: 12px; margin-bottom: 8px; }
.fc-panel__label {
margin: 10px 0 4px; font-size: 11px; letter-spacing: 0.06em;
text-transform: uppercase; color: rgb(170, 166, 156);
}
.fc-panel__radio { display: flex; gap: 8px; align-items: center; padding: 2px 0; cursor: pointer; }
.fc-panel__radio input { margin: 0; accent-color: rgb(244, 186, 122); }
.fc-panel__input {
all: revert; box-sizing: border-box; width: 100%;
padding: 7px 9px; border-radius: 6px;
border: 1px solid rgb(70, 74, 80); background: rgb(12, 14, 16); color: inherit;
font: inherit;
}
.fc-panel__input:focus { outline: 2px solid rgb(244, 186, 122); outline-offset: -1px; }
.fc-panel__results { display: flex; flex-direction: column; max-height: 160px; overflow-y: auto; }
.fc-panel__result {
all: revert; text-align: left; padding: 6px 9px; border: none; border-radius: 4px;
background: transparent; color: inherit; font: inherit; cursor: pointer;
}
.fc-panel__result:hover, .fc-panel__result:focus { background: rgb(36, 40, 46); }
.fc-panel__hint { margin-top: 8px; font-size: 12px; color: rgb(170, 166, 156); }
.fc-panel__actions { display: flex; justify-content: flex-end; gap: 8px; margin-top: 12px; }
.fc-panel__btn {
all: revert; padding: 6px 14px; border-radius: 999px; cursor: pointer;
border: 1px solid rgb(70, 74, 80); background: transparent; color: inherit;
font: 500 13px/1.2 system-ui, sans-serif;
}
.fc-panel__btn--primary { border-color: rgb(244, 186, 122); background: rgb(244, 186, 122); color: rgb(20, 23, 26); }
.fc-panel__btn:disabled { opacity: 0.5; cursor: default; }
/* Artist autocomplete — the list sits under the field, inside the panel. */
.fc-panel__combo { position: relative; }
.fc-panel__results {
margin-top: 4px; border-radius: 6px;
background: rgb(12, 14, 16);
}
.fc-panel__results:empty { display: none; }
.fc-panel__results:not(:empty) { border: 1px solid rgb(70, 74, 80); padding: 3px; }
.fc-panel__result { display: flex; justify-content: space-between; align-items: center; width: 100%; box-sizing: border-box; }
.fc-panel__result--active { background: rgb(52, 44, 32); outline: 1px solid rgb(244, 186, 122); }
.fc-panel__result--new { color: rgb(244, 186, 122); }
.fc-panel__result-tag { font-size: 11px; color: rgb(140, 220, 160); }
.fc-panel__empty { padding: 6px 9px; font-size: 12px; color: rgb(170, 166, 156); }
.fc-panel__hint--match { color: rgb(140, 220, 160); }
+303 -71
View File
@@ -5,42 +5,61 @@
// Cached probe result for the current URL so click-handlers know which
// action to dispatch without round-tripping again.
let currentProbe = null;
// Bumped on every evaluate(): a probe that answers after the operator has
// navigated on is for a page they've left, and must not repaint the chip.
let generation = 0;
let lastUrl = window.location.href;
evaluate();
const reEval = () => evaluate();
window.addEventListener('popstate', reEval);
const origPush = history.pushState;
history.pushState = function () { origPush.apply(this, arguments); reEval(); };
// SPA navigation. Patreon, SubscribeStar and above all Discord change
// channel/page without a reload. Patching history.pushState from here never
// worked: a content script runs in an isolated world, so the page's own
// pushState is not the function we'd replace. Polling the URL is the one
// signal that sees every navigation, and costs a string compare.
window.addEventListener('popstate', () => onUrlMaybeChanged());
setInterval(onUrlMaybeChanged, 500);
function onUrlMaybeChanged() {
if (window.location.href === lastUrl) return;
lastUrl = window.location.href;
closePanel();
evaluate();
}
async function evaluate() {
const mine = ++generation;
const url = window.location.href;
const platform = getPlatformFromUrl(url);
const onArtist = platform && isArtistPage(url, platform);
const btn = document.getElementById('fc-add-source-btn');
if (!onArtist) {
if (btn) btn.remove();
removeButton();
currentProbe = null;
return;
}
// On artist pages, ask the backend what state the URL is in BEFORE
// injecting the button — so the chip can render the right state on
// first paint instead of flashing the generic "Add" copy and
// updating afterwards.
// Ask the backend what state the URL is in BEFORE drawing the button, so
// the chip renders the right state on first paint instead of flashing the
// generic "Add" copy and updating afterwards.
let probe;
try {
probe = await browser.runtime.sendMessage({ type: 'PROBE_SOURCE', url });
} catch (e) {
probe = { error: e?.message || 'probe failed' };
}
if (mine !== generation) return;
currentProbe = probe;
if (probe?.state === 'unknown_platform') {
if (btn) btn.remove();
removeButton();
return;
}
renderButton(probe);
}
function removeButton() {
document.getElementById('fc-add-source-btn')?.remove();
closePanel();
}
function renderButton(probe) {
let btn = document.getElementById('fc-add-source-btn');
if (!btn) {
@@ -51,79 +70,36 @@
}
// Reset state classes so re-renders (SPA navigation) don't stack.
btn.className = 'fc-add-source-btn';
btn.classList.add(`fc-add-source-btn--${stateModifier(probe)}`);
btn.textContent = labelFor(probe);
btn.classList.add(`fc-add-source-btn--${chipState(probe)}`);
btn.textContent = chipLabel(probe, PLATFORMS[probe?.platform]?.name || probe?.platform || '');
btn.disabled = false;
}
function stateModifier(probe) {
if (!probe || probe.error) return 'new';
return ({
source_match: 'source-match',
artist_match: 'artist-match',
new: 'new',
})[probe.state] || 'new';
}
function labelFor(probe) {
if (!probe || probe.error) return '+ Add to FabledCurator';
const platformName = platformDisplayName(probe.platform);
const artistName = probe.artist?.name;
switch (probe.state) {
case 'source_match':
return `✓ In FabledCurator · ${platformName}`;
case 'artist_match':
return `+ Add ${platformName} source to ${artistName || 'artist'}`;
case 'new':
default:
return '+ Add to FabledCurator';
}
}
function platformDisplayName(key) {
return PLATFORMS[key]?.name || key || '';
}
async function onClick() {
const btn = document.getElementById('fc-add-source-btn');
if (!btn) return;
btn.disabled = true;
const original = btn.textContent;
const probe = currentProbe;
if (probe?.state === 'source_match') {
btn.textContent = 'Opening…';
try {
const r = await browser.runtime.sendMessage({
type: 'OPEN_ARTIST_PAGE',
slug: probe.artist?.slug,
});
if (r?.error) showToast(`Error: ${r.error}`, 'error');
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
} finally {
btn.disabled = false;
btn.textContent = original;
}
await openArtist(btn, probe.artist?.slug);
return;
}
// A Discord URL names a channel, not a creator — ask which artist.
if (probe?.platform === 'discord') {
if (document.getElementById('fc-discord-panel')) closePanel();
else openDiscordPanel(probe);
return;
}
await add(btn, { url: window.location.href });
}
btn.textContent = 'Adding…';
async function openArtist(btn, slug) {
btn.disabled = true;
const original = btn.textContent;
btn.textContent = 'Opening…';
try {
const r = await browser.runtime.sendMessage({
type: 'ADD_AS_SOURCE',
url: window.location.href,
});
if (r?.error) {
showToast(`Error: ${r.error}`, 'error');
} else {
const verb = r.created_source ? 'Added' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success');
// Re-probe so the chip flips green without waiting for the next
// navigation.
evaluate();
return;
}
const r = await browser.runtime.sendMessage({ type: 'OPEN_ARTIST_PAGE', slug });
if (r?.error) showToast(`Error: ${r.error}`, 'error');
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
} finally {
@@ -132,6 +108,262 @@
}
}
// One add, shared by the one-click chip and the Discord panel. Resolves
// true on success.
async function add(btn, request) {
btn.disabled = true;
const original = btn.textContent;
btn.textContent = 'Adding…';
try {
const r = await browser.runtime.sendMessage({ type: 'ADD_AS_SOURCE', ...request });
if (r?.error) {
showToast(`Error: ${r.error}`, 'error');
return false;
}
const verb = r.created_source ? 'Added to' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success');
// Re-probe so the chip flips green without waiting for a navigation.
evaluate();
return true;
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
return false;
} finally {
btn.disabled = false;
btn.textContent = original;
}
}
// ---- Discord Add panel ----
// Where: this channel or the whole server. Who: the suggested artist, one
// found by search, or a new one by name. Built with createElement only —
// server, channel and artist names are other people's text.
function el(tag, props = {}, children = []) {
const node = document.createElement(tag);
const { class: className, text, ...rest } = props;
if (className) node.className = className;
if (text != null) node.textContent = text;
Object.assign(node, rest);
for (const c of children) node.appendChild(c);
return node;
}
function closePanel() {
document.getElementById('fc-discord-panel')?.remove();
}
function openDiscordPanel(probe) {
closePanel();
const d = probe.discord || {};
const choice = discordPanelDefaults(probe);
const scopeRow = (value, label, disabled) => {
const input = el('input', {
type: 'radio', name: 'fc-discord-scope', value,
checked: choice.scope === value, disabled,
});
input.addEventListener('change', () => { choice.scope = value; refresh(); });
return el('label', { class: 'fc-panel__radio' }, [input, el('span', { text: label })]);
};
// The artist field is an autocomplete over FC's artists: it searches as
// soon as the panel opens (with the prefilled name) and on every keystroke,
// lists the matches under the field, fills in the rest of the top match as
// you type (Tab or Enter accepts it, typing on replaces it), and picks an
// artist whose name IS the text, spacing and case aside, without being
// asked. ↑/↓ walk the list; the last row creates a new artist instead.
const nameInput = el('input', {
type: 'text', class: 'fc-panel__input', value: choice.artistName,
placeholder: 'Search artists or type a new name',
autocomplete: 'off', spellcheck: false,
role: 'combobox',
});
nameInput.setAttribute('aria-autocomplete', 'both');
nameInput.setAttribute('aria-expanded', 'false');
const results = el('div', { class: 'fc-panel__results', role: 'listbox' });
const hint = el('div', { class: 'fc-panel__hint' });
const addBtn = el('button', { class: 'fc-panel__btn fc-panel__btn--primary', text: 'Add' });
const cancelBtn = el('button', { class: 'fc-panel__btn', text: 'Cancel' });
const panel = el('div', { id: 'fc-discord-panel', class: 'fc-panel' }, [
el('div', { class: 'fc-panel__title', text: 'Add Discord source' }),
el('div', { class: 'fc-panel__sub', text: serverLabel(d) }),
el('div', { class: 'fc-panel__label', text: 'Follow' }),
scopeRow('channel', d.channel_id ? channelLabel(d) : 'this channel', !d.channel_id),
scopeRow('server', `Every channel in ${serverLabel(d)}`, false),
el('div', { class: 'fc-panel__label', text: 'Artist' }),
el('div', { class: 'fc-panel__combo' }, [nameInput, results]),
hint,
el('div', { class: 'fc-panel__actions' }, [cancelBtn, addBtn]),
]);
// Search state: the rows on screen, which one ↑/↓ has highlighted (-1 =
// none), and whether the list is open.
let rows = [];
let active = -1;
let listOpen = false;
function refresh() {
const req = discordAddRequest(choice);
addBtn.disabled = !req;
if (!req) hint.textContent = 'Pick an artist or type a name.';
else if (req.artistId != null) hint.textContent = `✓ Connects to ${choice.artist.name}, already in FabledCurator.`;
else hint.textContent = `Creates a new artist “${req.artistName}”.`;
hint.classList.toggle('fc-panel__hint--match', !!req && req.artistId != null);
}
function pick(artist) {
choice.artist = artist ? { id: artist.id, name: artist.name } : null;
if (artist) {
choice.artistName = artist.name;
nameInput.value = artist.name;
}
closeList();
refresh();
}
function closeList() {
listOpen = false;
active = -1;
results.replaceChildren();
nameInput.setAttribute('aria-expanded', 'false');
}
function renderList() {
const typed = nameInput.value.trim();
const exact = exactArtistMatch(typed, rows);
const items = rows.map((a, i) => {
const row = el('button', { type: 'button', class: 'fc-panel__result', role: 'option' }, [
el('span', { text: a.name }),
]);
if (choice.artist && choice.artist.id === a.id) {
row.appendChild(el('span', { class: 'fc-panel__result-tag', text: 'selected' }));
}
row.classList.toggle('fc-panel__result--active', i === active);
// mousedown, not click: it fires before the input's blur closes the list.
row.addEventListener('mousedown', (e) => { e.preventDefault(); pick(a); });
return row;
});
if (typed && !exact) {
const i = rows.length;
const create = el('button', { type: 'button', class: 'fc-panel__result fc-panel__result--new', role: 'option',
text: `+ New artist “${typed}”` });
create.classList.toggle('fc-panel__result--active', i === active);
create.addEventListener('mousedown', (e) => { e.preventDefault(); pick(null); choice.artistName = typed; refresh(); });
items.push(create);
}
if (typed && !rows.length) {
items.unshift(el('div', { class: 'fc-panel__empty', text: 'No FabledCurator artist matches.' }));
}
results.replaceChildren(...items);
listOpen = items.length > 0;
nameInput.setAttribute('aria-expanded', String(listOpen));
results.querySelector('.fc-panel__result--active')?.scrollIntoView({ block: 'nearest' });
}
let debounce = null;
let searchSeq = 0;
// `autofill` is false for deletions: filling the name back in as you
// backspace would make it impossible to delete.
function search(autofill) {
clearTimeout(debounce);
const q = nameInput.value.trim();
if (!q) { rows = []; closeList(); return; }
debounce = setTimeout(async () => {
const mine = ++searchSeq;
let r;
try {
r = await browser.runtime.sendMessage({ type: 'SEARCH_ARTISTS', q });
} catch {
return;
}
// Stale: a later keystroke has its own search coming.
if (mine !== searchSeq || r?.error || nameInput.value.trim() !== q) return;
rows = r.artists || [];
active = -1;
const exact = exactArtistMatch(q, rows);
if (exact && !choice.artist) {
choice.artist = { id: exact.id, name: exact.name };
} else if (autofill && document.activeElement === nameInput) {
const hit = inlineCompletion(nameInput.value, rows);
const caret = nameInput.value.length;
if (hit && nameInput.selectionStart === caret) {
nameInput.value = nameInput.value + hit.name.slice(caret);
nameInput.setSelectionRange(caret, hit.name.length);
choice.artist = { id: hit.id, name: hit.name };
choice.artistName = hit.name;
}
}
renderList();
refresh();
}, 150);
}
nameInput.addEventListener('input', (e) => {
choice.artistName = nameInput.value;
// Any edit un-picks: the artist is whatever the field now says.
choice.artist = null;
refresh();
search(!String(e.inputType || '').startsWith('delete'));
});
nameInput.addEventListener('focus', () => { if (rows.length) renderList(); });
nameInput.addEventListener('blur', () => closeList());
// Keep Discord's global shortcuts from eating keystrokes meant for us.
panel.addEventListener('keydown', (e) => {
e.stopPropagation();
const count = results.querySelectorAll('.fc-panel__result').length;
if (e.target === nameInput && listOpen && count) {
if (e.key === 'ArrowDown' || e.key === 'ArrowUp') {
e.preventDefault();
active = e.key === 'ArrowDown' ? (active + 1) % count : (active <= 0 ? count - 1 : active - 1);
renderList();
return;
}
if ((e.key === 'Enter' || e.key === 'Tab') && active >= 0) {
e.preventDefault();
results.querySelectorAll('.fc-panel__result')[active]
.dispatchEvent(new MouseEvent('mousedown', { cancelable: true }));
return;
}
if ((e.key === 'Tab' || e.key === 'Enter') && choice.artist
&& nameInput.selectionEnd > nameInput.selectionStart) {
// Accept the inline autofill — Enter too, but only to accept: the
// add itself takes a second Enter, once the hint names the artist.
e.preventDefault();
pick(choice.artist);
return;
}
}
if (e.key === 'Escape') {
if (listOpen) closeList();
else closePanel();
return;
}
if (e.key === 'Enter' && e.target === nameInput && !addBtn.disabled) addBtn.click();
});
cancelBtn.addEventListener('click', closePanel);
addBtn.addEventListener('click', async () => {
const req = discordAddRequest(choice);
if (!req) return;
const btn = document.getElementById('fc-add-source-btn');
addBtn.disabled = true;
const ok = await add(btn || addBtn, req);
if (ok) closePanel();
else refresh();
});
document.body.appendChild(panel);
refresh();
nameInput.focus();
nameInput.select();
// Search what the field opens with — the server's name, usually — so an
// artist it already matches is picked before the operator types anything.
if (!choice.artist && nameInput.value.trim()) search(false);
}
function showToast(text, kind) {
const t = document.createElement('div');
t.className = `fc-toast fc-toast--${kind}`;
+19 -2
View File
@@ -80,6 +80,17 @@ class FabledCuratorAPI {
getCredentials() {
return this.request('GET', '/credentials');
}
// Test the STORED credential against one of the platform's sources — the
// same check the web UI's Verify button runs. {valid: true|false|null, reason}.
verifyCredential(platform) {
return this.request('POST', `/credentials/${encodeURIComponent(platform)}/verify`);
}
// Artist search for the Discord Add panel — the web UI's autocomplete.
searchArtists(q, limit = 8) {
const qs = new URLSearchParams({ q, limit: String(limit) }).toString();
return this.request('GET', `/artists/autocomplete?${qs}`);
}
// FC-3a — sources.
listSources() {
@@ -90,8 +101,14 @@ class FabledCuratorAPI {
}
// FC-3g — extension-specific.
quickAddSource(url) {
return this.request('POST', '/extension/quick-add-source', { url });
// artistId connects the source to an existing artist, artistName to the
// artist of that name (created if new); with neither the server derives the
// artist from the URL. A Discord channel always sends one.
quickAddSource(url, { artistId = null, artistName = null } = {}) {
const body = { url };
if (artistId != null) body.artist_id = artistId;
else if (artistName) body.artist_name = artistName;
return this.request('POST', '/extension/quick-add-source', body);
}
probeSource(url) {
// Read-only existence check. Drives the content-script chip's
+107
View File
@@ -0,0 +1,107 @@
/**
* The content script's decisions, kept apart from its DOM so the specs can
* load them (test/chip.spec.js): which state the chip shows, what it says,
* and what the Discord Add panel starts out proposing.
*
* `probe` is /api/extension/probe's answer; `platformName` is the display
* name (PLATFORMS[key].name), passed in so this file needs no other lib.
*/
function chipState(probe) {
if (!probe || probe.error) return 'new';
return ({ source_match: 'source-match', artist_match: 'artist-match', new: 'new' })[probe.state] || 'new';
}
function chipLabel(probe, platformName) {
if (!probe || probe.error) return '+ Add to FabledCurator';
const artist = probe.artist?.name || 'artist';
if (probe.platform === 'discord') {
const d = probe.discord || {};
if (probe.state === 'source_match') {
return probe.covered_by_server
? `✓ Whole server in FabledCurator · ${artist}`
: `✓ In FabledCurator · ${artist}`;
}
// Every other Discord state opens the panel: the artist is always chosen.
return d.channel_id ? `+ Add ${channelLabel(d)} to FabledCurator` : '+ Add server to FabledCurator';
}
switch (probe.state) {
case 'source_match':
return `✓ In FabledCurator · ${platformName}`;
case 'artist_match':
return `+ Add ${platformName} source to ${artist}`;
default:
return '+ Add to FabledCurator';
}
}
/** `#name` when the probe could read it, else a neutral "this channel". */
function channelLabel(d) {
return d.channel_name ? `#${d.channel_name}` : 'this channel';
}
/** `name` when the probe could read it, else "this server". */
function serverLabel(d) {
return d.server_name || 'this server';
}
/**
* What the Discord Add panel opens with. The channel is the default scope
* when there is one: a server source walks every channel the token can read,
* which is rarely what a single art channel wants. The artist is the probe's
* suggestion (the owner of another source on this server), else a new artist
* named after the server.
*/
function discordPanelDefaults(probe) {
const d = probe?.discord || {};
const suggested = probe?.state === 'artist_match' && probe.artist ? probe.artist : null;
return {
scope: d.channel_id ? 'channel' : 'server',
channelUrl: d.channel_url || null,
serverUrl: d.server_url || null,
artist: suggested ? { id: suggested.id, name: suggested.name } : null,
artistName: suggested ? suggested.name : (d.server_name || ''),
};
}
/**
* The quick-add body for the panel's current choice. A picked artist goes by
* id — names can collide once slugified — and a typed name creates (or
* finds) that artist. null when there is nothing valid to send.
*/
function discordAddRequest(choice) {
const url = choice.scope === 'server' ? choice.serverUrl : choice.channelUrl;
if (!url) return null;
if (choice.artist && choice.artist.id != null && choice.artist.name === choice.artistName) {
return { url, artistId: choice.artist.id };
}
const name = (choice.artistName || '').trim();
return name ? { url, artistName: name } : null;
}
/**
* An artist name reduced to what identifies it: lowercase letters and digits
* of any script, nothing else — so "Tamada Heijun", "tamada_heijun" and
* "TamadaHeijun" are one name. Mirrors the server's autocomplete (#429).
*/
function squashName(name) {
return String(name || '').toLowerCase().replace(/[^\p{L}\p{N}]/gu, '');
}
/** The search result that IS the query, spacing aside, else null. */
function exactArtistMatch(query, results) {
const q = squashName(query);
if (!q) return null;
return (results || []).find((a) => squashName(a.name) === q) || null;
}
/**
* Inline autofill: the first result whose name extends what was typed
* (case-insensitive), so the panel can fill in the rest and select it —
* typing on overwrites it, Tab or Enter accepts it. null when none does.
*/
function inlineCompletion(typed, results) {
const t = String(typed || '').toLowerCase();
if (!t) return null;
return (results || []).find((a) => a.name.length > t.length && a.name.toLowerCase().startsWith(t)) || null;
}
+17 -1
View File
@@ -57,7 +57,8 @@ const PLATFORMS = {
domains: ['.discord.com', 'discord.com'],
authType: 'token',
color: '#5865F2',
urlPattern: /^https?:\/\/(www\.)?discord\.com/,
// ptb/canary are Discord's beta clients — same channels, same token.
urlPattern: /^https?:\/\/((www|ptb|canary)\.)?discord\.com/,
note: 'Open Discord in browser to capture token',
},
};
@@ -80,8 +81,23 @@ const PLATFORM_ARTIST_PATTERNS = {
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?:cw\/|c\/)?(?!(?:home|search|messages|notifications|library|settings|posts)(?:[\/?#]|$))[^/?#]+/i,
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
// A Discord URL names a server or channel, not a creator: the backend's slug
// is `<server>[/<channel>]` and the Add panel asks which artist it belongs
// to. A message jump link still names its channel; DMs (@me) and thread
// links don't match. Mirrors extension_service._PLATFORM_PATTERNS.
discord: /^https?:\/\/(?:www\.|ptb\.|canary\.)?discord\.com\/channels\/\d+(?:\/\d+)?(?:\/\d+)?\/?(?:[?#].*)?$/i,
};
/**
* `{serverId, channelId}` from a Discord channel/server URL the artist
* pattern accepts, else null. channelId is null for a whole-server URL.
*/
function parseDiscordUrl(url) {
if (!PLATFORM_ARTIST_PATTERNS.discord.test(url || '')) return null;
const m = /\/channels\/(\d+)(?:\/(\d+))?/.exec(url);
return m ? { serverId: m[1], channelId: m[2] || null } : null;
}
function getPlatformFromUrl(url) {
for (const [key, platform] of Object.entries(PLATFORMS)) {
if (platform.urlPattern.test(url)) return key;
+55
View File
@@ -0,0 +1,55 @@
/**
* The popup's wording, kept apart from its DOM so the specs can load it
* (test/popup-format.spec.js). Classic script, like the rest of lib/.
*/
/**
* One line of state for a source row, from /api/sources' fields, with the
* status class the popup colours it by ('ready' | 'error' | 'no-cookies' for
* a warning | '' for plain). The most actionable fact wins: an error before
* a running backfill, a backfill before the last-checked time.
*/
function sourceStatus(src, now = Date.now()) {
if (!src.enabled) return { text: 'Disabled', kind: '' };
if (src.last_error) {
const first = String(src.last_error).split('\n')[0].trim();
const short = first.length > 90 ? `${first.slice(0, 89)}…` : first;
return { text: `Error — ${short}`, kind: 'error' };
}
if (src.backfill_state === 'running') {
const n = src.backfill_chunks || 0;
return { text: n ? `Backfilling — ${n} chunk${n === 1 ? '' : 's'} done` : 'Backfill queued', kind: 'ready' };
}
if (src.backfill_state === 'stalled') return { text: 'Backfill stalled', kind: 'no-cookies' };
if (!src.last_checked_at) return { text: 'Not checked yet', kind: '' };
return { text: `Checked ${relativeTime(src.last_checked_at, now)}`, kind: '' };
}
// Same buckets and wording as the web UI's canonical formatRelative
// (frontend/src/utils/date.js, snippet #3959) — the extension can't import it (classic
// scripts, separate package), so it mirrors it: "42s ago", "5m ago", "3h ago",
// "2d ago", and "Never" for a missing or unreadable time.
function relativeTime(iso, now = Date.now()) {
const t = iso ? Date.parse(iso) : NaN;
if (Number.isNaN(t)) return 'Never';
const abs = Math.abs(now - t) / 1000;
let body;
if (abs < 60) body = `${Math.floor(abs)}s`;
else if (abs < 3600) body = `${Math.floor(abs / 60)}m`;
else if (abs < 86400) body = `${Math.floor(abs / 3600)}h`;
else body = `${Math.floor(abs / 86400)}d`;
return `${body} ago`;
}
/**
* The popup message after a Discord token export, from FC's verify of the
* stored token: {text, kind} with kind 'success' | 'warning' | 'error'.
* valid=null is "FC couldn't test it" (no Discord source yet, or a network
* hiccup) — a warning with FC's reason, never a failure.
*/
function tokenExportMessage(verify) {
if (!verify) return { text: 'Discord: token exported', kind: 'success' };
if (verify.valid === true) return { text: `Discord: token exported and verified ✓ — ${verify.reason}`, kind: 'success' };
if (verify.valid === false) return { text: `Discord: token exported, but FC's check failed — ${verify.reason}`, kind: 'error' };
return { text: `Discord: token exported (not verified — ${verify.reason})`, kind: 'warning' };
}
+3 -2
View File
@@ -56,9 +56,10 @@
"*://*.patreon.com/*",
"*://*.subscribestar.com/*",
"*://*.subscribestar.adult/*",
"*://*.hentai-foundry.com/*"
"*://*.hentai-foundry.com/*",
"*://*.discord.com/*"
],
"js": ["lib/platforms.js", "content/content-script.js"],
"js": ["lib/platforms.js", "lib/chip.js", "content/content-script.js"],
"css": ["content/content-script.css"],
"run_at": "document_idle"
}
+1
View File
@@ -47,6 +47,7 @@
</section>
<script src="../lib/platforms.js"></script>
<script src="../lib/popup-format.js"></script>
<script src="popup.js"></script>
</body>
</html>
+17 -5
View File
@@ -159,7 +159,11 @@ async function exportPlatformCookies(key, card) {
try {
const r = await browser.runtime.sendMessage({ type: 'EXPORT_COOKIES', platform: key });
if (r.error) showError(r.error);
else {
else if (key === 'discord') {
const m = tokenExportMessage(r.verify);
showStatusMessage(m.text, m.kind);
await loadPlatformStatus();
} else {
const n = r.cookieCount ?? null;
const verifiedSuffix = r.verified ? ' (verified ✓)' : '';
const msg = n !== null
@@ -205,7 +209,10 @@ async function loadSources() {
c.appendChild(mutedNote('No sources yet.'));
return;
}
for (const src of r.sources) c.appendChild(createSourceRow(src));
// Grouped by artist so a creator's Patreon and Discord sit together.
const sorted = [...r.sources].sort((a, b) =>
(a.artist_name || '').localeCompare(b.artist_name || '') || a.id - b.id);
for (const src of sorted) c.appendChild(createSourceRow(src));
}
function createSourceRow(src) {
@@ -215,11 +222,16 @@ function createSourceRow(src) {
info.className = 'info';
const name = document.createElement('div');
name.className = 'name';
name.textContent = `${src.platform} · #${src.id}`;
const platformName = PLATFORMS[src.platform]?.name || src.platform;
name.textContent = `${src.artist_name || `Source #${src.id}`} · ${platformName}`;
const state = sourceStatus(src);
const st = document.createElement('div');
st.className = `status ${state.kind}`;
st.textContent = state.text;
const url = document.createElement('div');
url.className = 'url';
url.textContent = src.url;
info.appendChild(name); info.appendChild(url);
info.appendChild(name); info.appendChild(st); info.appendChild(url);
const play = document.createElement('button');
play.className = 'play';
play.textContent = '▶';
@@ -229,7 +241,7 @@ function createSourceRow(src) {
const r = await browser.runtime.sendMessage({ type: 'CHECK_SOURCE', sourceId: src.id });
play.disabled = false;
if (r.error) showError(r.error);
else showSuccess(`Triggered check for source #${src.id}`);
else showSuccess(`Check queued for ${src.artist_name || `source #${src.id}`} (${platformName})`);
});
row.appendChild(info); row.appendChild(play);
return row;
+37
View File
@@ -134,5 +134,42 @@
{ "url": "https://www.hentai-foundry.com/pictures/popular", "why": "gallery listing, not a user" },
{ "url": "https://www.hentai-foundry.com/", "why": "site root" }
]
},
"discord": {
"match": [
{
"url": "https://discord.com/channels/111111111111111111/222222222222222222",
"slug": "111111111111111111/222222222222222222",
"why": "a channel: the slug is server/channel -- a Discord URL names a place, not a creator, so the artist is chosen in the Add panel"
},
{
"url": "https://discord.com/channels/111111111111111111",
"slug": "111111111111111111",
"why": "a whole server"
},
{
"url": "https://discord.com/channels/111111111111111111/222222222222222222/333333333333333333",
"slug": "111111111111111111/222222222222222222",
"why": "a message jump link still names its channel"
},
{
"url": "https://ptb.discord.com/channels/111111111111111111/222222222222222222",
"slug": "111111111111111111/222222222222222222",
"why": "the ptb and canary clients serve the same channels"
},
{
"url": "https://discord.com/channels/111111111111111111/222222222222222222/",
"slug": "111111111111111111/222222222222222222",
"why": "trailing slash is tolerated"
}
],
"no_match": [
{ "url": "https://discord.com/channels/@me", "why": "the DM list is not a source" },
{ "url": "https://discord.com/channels/@me/222222222222222222", "why": "a DM is not a source" },
{ "url": "https://discord.com/app", "why": "the app shell, no server open" },
{ "url": "https://discord.com/channels/111111111111111111/222222222222222222/threads/444444444444444444", "why": "thread links are left to the manual Add form" },
{ "url": "https://discord.com/servers/111111111111111111", "why": "a server-discovery page, not a channel" }
]
}
}
+143
View File
@@ -0,0 +1,143 @@
import { describe, it, expect } from 'vitest'
import { loadLib } from './helpers/loadLib.js'
const { chipState, chipLabel, discordPanelDefaults, discordAddRequest } = loadLib('chip.js', [
'chipState',
'chipLabel',
'discordPanelDefaults',
'discordAddRequest'
])
const discord = (extra = {}) => ({
platform: 'discord',
slug: '111/222',
discord: {
server_id: '111',
channel_id: '222',
server_name: 'Studio',
channel_name: 'drops',
server_url: 'https://discord.com/channels/111',
channel_url: 'https://discord.com/channels/111/222'
},
...extra
})
describe('chip state and label', () => {
it('keeps the one-click wording on creator platforms', () => {
expect(chipLabel({ state: 'new', platform: 'patreon' }, 'Patreon')).toBe('+ Add to FabledCurator')
expect(
chipLabel({ state: 'artist_match', platform: 'patreon', artist: { name: 'Atole' } }, 'Patreon')
).toBe('+ Add Patreon source to Atole')
expect(chipLabel({ state: 'source_match', platform: 'patreon' }, 'Patreon')).toBe(
'✓ In FabledCurator · Patreon'
)
})
it('offers the channel by name on Discord, whatever the suggestion', () => {
expect(chipLabel(discord({ state: 'new' }), 'Discord')).toBe('+ Add #drops to FabledCurator')
expect(chipLabel(discord({ state: 'artist_match', artist: { name: 'A' } }), 'Discord')).toBe(
'+ Add #drops to FabledCurator'
)
})
it('says "this channel" when the token could not read the name', () => {
const p = discord({ state: 'new' })
p.discord.channel_name = null
expect(chipLabel(p, 'Discord')).toBe('+ Add this channel to FabledCurator')
})
it('says whose source a Discord channel already is, and when the server covers it', () => {
const artist = { name: 'Tamada', slug: 'tamada' }
expect(chipLabel(discord({ state: 'source_match', artist, covered_by_server: false }), 'Discord')).toBe(
'✓ In FabledCurator · Tamada'
)
expect(chipLabel(discord({ state: 'source_match', artist, covered_by_server: true }), 'Discord')).toBe(
'✓ Whole server in FabledCurator · Tamada'
)
})
it('falls back to the generic add when the probe failed', () => {
expect(chipState({ error: 'x' })).toBe('new')
expect(chipLabel({ error: 'x' }, '')).toBe('+ Add to FabledCurator')
expect(chipState({ state: 'source_match' })).toBe('source-match')
})
})
describe('Discord Add panel', () => {
it('opens on the channel with the suggested artist preselected', () => {
const d = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
expect(d.scope).toBe('channel')
expect(d.artist).toEqual({ id: 7, name: 'Tamada' })
expect(d.artistName).toBe('Tamada')
})
it('proposes a new artist named after the server when nothing is suggested', () => {
const d = discordPanelDefaults(discord({ state: 'new' }))
expect(d.artist).toBe(null)
expect(d.artistName).toBe('Studio')
})
it('sends a picked artist by id', () => {
const choice = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
expect(discordAddRequest(choice)).toEqual({
url: 'https://discord.com/channels/111/222',
artistId: 7
})
})
it('sends a typed name once the picked artist has been edited away', () => {
const choice = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
choice.artistName = 'Tamada Alt'
expect(discordAddRequest(choice)).toEqual({
url: 'https://discord.com/channels/111/222',
artistName: 'Tamada Alt'
})
})
it('adds the whole server when the operator picks it', () => {
const choice = discordPanelDefaults(discord({ state: 'new' }))
choice.scope = 'server'
expect(discordAddRequest(choice).url).toBe('https://discord.com/channels/111')
})
it('has nothing to send without an artist', () => {
const choice = discordPanelDefaults(discord({ state: 'new' }))
choice.artistName = ' '
expect(discordAddRequest(choice)).toBe(null)
})
})
const { squashName, exactArtistMatch, inlineCompletion } = loadLib('chip.js', [
'squashName',
'exactArtistMatch',
'inlineCompletion'
])
describe('artist matching for the Add panel', () => {
const results = [
{ id: 1, name: 'TamadaHeijun' },
{ id: 2, name: 'Tamago' },
{ id: 3, name: 'Sabu Art' }
]
it('treats spacing, case and punctuation as the same name', () => {
expect(squashName('Tamada Heijun')).toBe('tamadaheijun')
expect(squashName('sabu_art!')).toBe('sabuart')
expect(squashName('玉田 平順')).toBe('玉田平順')
})
it('finds the result that is the query, spacing aside', () => {
expect(exactArtistMatch('tamada heijun', results)).toEqual({ id: 1, name: 'TamadaHeijun' })
expect(exactArtistMatch('Tama', results)).toBe(null)
expect(exactArtistMatch(' ', results)).toBe(null)
})
it('autofills the first name that extends what was typed', () => {
expect(inlineCompletion('tamad', results)).toEqual({ id: 1, name: 'TamadaHeijun' })
expect(inlineCompletion('Sab', results).name).toBe('Sabu Art')
// Nothing to add once the name is complete, or when nothing extends it.
expect(inlineCompletion('Sabu Art', results)).toBe(null)
expect(inlineCompletion('heijun', results)).toBe(null)
expect(inlineCompletion('', results)).toBe(null)
})
})
+42 -12
View File
@@ -7,10 +7,14 @@ import { loadLib } from './helpers/loadLib.js'
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
const manifest = JSON.parse(readFileSync(path.join(EXT_DIR, 'manifest.json'), 'utf8'))
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
'platforms.js',
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
)
const { getPlatformFromUrl, isArtistPage, parseDiscordUrl, PLATFORMS, PLATFORM_ARTIST_PATTERNS } =
loadLib('platforms.js', [
'getPlatformFromUrl',
'isArtistPage',
'parseDiscordUrl',
'PLATFORMS',
'PLATFORM_ARTIST_PATTERNS'
])
describe('getPlatformFromUrl', () => {
it('identifies each platform from a domain URL', () => {
@@ -88,8 +92,11 @@ describe('isArtistPage', () => {
)
})
it('returns false for a platform with no artist pattern (discord)', () => {
it('matches Discord server and channel pages, not DMs (milestone 429)', () => {
expect(isArtistPage('https://discord.com/channels/111/222', 'discord')).toBe(true)
expect(isArtistPage('https://discord.com/channels/111', 'discord')).toBe(true)
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
expect(isArtistPage('https://discord.com/channels/@me/222', 'discord')).toBe(false)
})
it('returns false for an unknown platform key', () => {
@@ -100,8 +107,8 @@ describe('isArtistPage', () => {
describe('platform table integrity', () => {
it('gives every artist pattern a corresponding platform entry', () => {
// A pattern keyed to a platform that no longer exists is dead code that
// silently never fires; the reverse (a platform with no pattern) is the
// legitimate discord case, so only this direction is an error.
// silently never fires; the reverse (a platform with no pattern) would be
// a platform the button never offers, a product choice rather than an error.
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
expect(Object.keys(PLATFORMS)).toContain(key)
}
@@ -125,7 +132,8 @@ describe('platform table integrity', () => {
const samples = {
patreon: 'https://www.patreon.com/cw/Atole',
subscribestar: 'https://subscribestar.adult/someone',
hentaifoundry: 'https://www.hentai-foundry.com/user/someone'
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
discord: 'https://ptb.discord.com/channels/111/222'
}
for (const [key, url] of Object.entries(samples)) {
expect(isArtistPage(url, key), `${key} artist pattern`).toBe(true)
@@ -152,7 +160,7 @@ describe('manifest.json agrees with the platform table', () => {
)
expect(owner, `no platform claims content-script match "${m}"`).toBeTruthy()
// The content script exists to draw the Add-as-source button, so a
// platform with no artist pattern (discord) has no business here.
// platform with no artist pattern has no business here.
expect(
PLATFORM_ARTIST_PATTERNS[owner[0]],
`"${m}" injects for ${owner[0]}, which has no artist pattern`
@@ -232,9 +240,8 @@ describe('the JS<->Py artist-pattern mirror (#3093)', () => {
it('has samples for every platform that has an artist pattern', () => {
// The guard's own coverage check: without it, deleting a platform's
// samples would make this block pass by testing less. Discord is
// deliberately in neither — it is channel-based, with no creator page to
// put a button on, so it has no artist pattern on either side.
// samples would make this block pass by testing less. Discord joined at
// milestone 429 — its slug is server/channel and the artist is chosen.
expect(Object.keys(samples).sort()).toEqual(Object.keys(PLATFORM_ARTIST_PATTERNS).sort())
})
@@ -247,3 +254,26 @@ describe('the JS<->Py artist-pattern mirror (#3093)', () => {
}
})
})
describe('parseDiscordUrl', () => {
it('reads the server and channel ids the Add panel offers', () => {
expect(parseDiscordUrl('https://discord.com/channels/111/222')).toEqual({
serverId: '111',
channelId: '222'
})
expect(parseDiscordUrl('https://discord.com/channels/111/222/333')).toEqual({
serverId: '111',
channelId: '222'
})
expect(parseDiscordUrl('https://discord.com/channels/111')).toEqual({
serverId: '111',
channelId: null
})
})
it('returns null for anything the artist pattern rejects', () => {
expect(parseDiscordUrl('https://discord.com/channels/@me/222')).toBe(null)
expect(parseDiscordUrl('https://discord.com/app')).toBe(null)
expect(parseDiscordUrl('')).toBe(null)
})
})
+53
View File
@@ -0,0 +1,53 @@
import { describe, it, expect } from 'vitest'
import { loadLib } from './helpers/loadLib.js'
const { sourceStatus, relativeTime, tokenExportMessage } = loadLib('popup-format.js', [
'sourceStatus',
'relativeTime',
'tokenExportMessage'
])
const NOW = Date.parse('2026-09-25T12:00:00Z')
const src = (extra = {}) => ({ enabled: true, last_error: null, backfill_state: null, ...extra })
describe('sourceStatus', () => {
it('puts an error first, trimmed to its first line', () => {
const s = sourceStatus(src({ last_error: 'Discord rejected the token\nstack…', backfill_state: 'running' }), NOW)
expect(s).toEqual({ text: 'Error — Discord rejected the token', kind: 'error' })
})
it('shows a running backfill and its progress', () => {
expect(sourceStatus(src({ backfill_state: 'running', backfill_chunks: 0 }), NOW).text).toBe('Backfill queued')
expect(sourceStatus(src({ backfill_state: 'running', backfill_chunks: 3 }), NOW).text).toBe(
'Backfilling — 3 chunks done'
)
})
it('says when a source was last checked, or that it never was', () => {
expect(sourceStatus(src({ last_checked_at: '2026-09-25T11:55:00Z' }), NOW).text).toBe('Checked 5m ago')
expect(sourceStatus(src({ last_checked_at: null }), NOW).text).toBe('Not checked yet')
})
it('shows a disabled source as disabled, whatever else it carries', () => {
expect(sourceStatus(src({ enabled: false, last_error: 'x' }), NOW).text).toBe('Disabled')
})
})
describe('relativeTime', () => {
it('uses the web UI formatRelative buckets', () => {
expect(relativeTime('2026-09-25T11:59:18Z', NOW)).toBe('42s ago')
expect(relativeTime('2026-09-25T09:00:00Z', NOW)).toBe('3h ago')
expect(relativeTime('2026-09-23T12:00:00Z', NOW)).toBe('2d ago')
expect(relativeTime('garbage', NOW)).toBe('Never')
})
})
describe('tokenExportMessage', () => {
it('distinguishes verified, rejected and untested tokens', () => {
expect(tokenExportMessage({ valid: true, reason: 'Token valid (me)' }).kind).toBe('success')
expect(tokenExportMessage({ valid: false, reason: 'Discord rejected the token' }).kind).toBe('error')
const untested = tokenExportMessage({ valid: null, reason: 'No enabled source' })
expect(untested.kind).toBe('warning')
expect(untested.text).toContain('No enabled source')
})
})
+8
View File
@@ -153,6 +153,14 @@ const health = computed(() => {
label: (worst?.detail || 'A part has stopped') + suffix,
}
}
// Running but slow — a GPU agent that fell back to the CPU (#4410). Worth
// the amber dot: nothing else anywhere says so.
if (overall === 'degraded') {
return {
icon: 'mdi-speedometer-slow', color: 'warning',
label: (worst?.detail || 'A part is running degraded') + suffix,
}
}
if (overall === 'stale') {
return {
icon: 'mdi-alert', color: 'warning',
@@ -103,9 +103,9 @@ const stats = computed(() => {
padding: 0.5rem 1rem;
background: linear-gradient(
to bottom,
rgba(20, 23, 26, 0.92) 0%,
rgba(20, 23, 26, 0.65) 60%,
rgba(20, 23, 26, 0) 100%
rgba(var(--fc-chrome-rgb), 0.92) 0%,
rgba(var(--fc-chrome-rgb), 0.65) 60%,
rgba(var(--fc-chrome-rgb), 0) 100%
);
backdrop-filter: blur(2px);
-webkit-backdrop-filter: blur(2px);
@@ -329,7 +329,7 @@ function pushFilter(mutate) {
opaque than the bar/nav so the controls stay legible. */
.fc-filterbar-wrap :deep(.v-field),
.fc-filterbar-wrap :deep(.v-btn-group) {
background-color: rgba(20, 23, 26, 0.72);
background-color: rgba(var(--v-theme-background), 0.72);
}
/* Media toggle (All / Images / Videos) as ONE cohesive segmented control.
FC's global VBtn { rounded: 'pill' } default made Vuetify 4 pill-round each
@@ -213,7 +213,7 @@ function nextFrame() {
--fc-side-w: 320px;
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
mid-opacity + blur so the page behind shows through faintly. */
background: rgba(20, 23, 26, 0.65);
background: rgba(var(--v-theme-background), 0.65);
backdrop-filter: blur(8px);
-webkit-backdrop-filter: blur(8px);
outline: none;
@@ -221,7 +221,7 @@ function nextFrame() {
}
.fc-viewer__close, .fc-viewer__nav {
position: absolute; top: 50%;
background: rgba(20, 23, 26, 0.7);
background: rgba(var(--v-theme-background), 0.7);
color: rgb(var(--v-theme-parchment, 232 228 216));
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 50%;
@@ -359,7 +359,7 @@ function nextFrame() {
z-index: 1;
/* Opaque obsidian so the scrolling panel never bleeds through the
haze behind the pinned image. */
background: rgb(20, 23, 26);
background: rgb(var(--v-theme-background));
}
.fc-viewer__side {
width: 100%;
+155 -8
View File
@@ -20,7 +20,7 @@
class="fc-post-card__synthetic" :title="synthesisTitle"
>
<v-icon icon="mdi-auto-fix" size="x-small" start />
grouped by FabledCurator
{{ synthesisChip }}
</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
@@ -30,6 +30,12 @@
<span v-if="totalImages" class="fc-post-card__meta">
· {{ totalImages }} image{{ totalImages === 1 ? '' : 's' }}
</span>
<!-- #4402. Referenced images are counted apart from the post's own: a
Patreon card must not read as though the creator posted Discord's
files on Patreon. -->
<span v-if="refImages.length" class="fc-post-card__meta fc-post-card__ref-meta">
· +{{ refImages.length }} from Discord
</span>
<!-- Only on a grouping that has actually grown. An ordinary post can
never show this, and a grouping that has not grown says nothing —
an "updated" label that is always present teaches you to ignore it. -->
@@ -66,9 +72,16 @@
<button
v-for="t in rail" :key="t.image_id" type="button"
class="fc-post-card__rail-cell"
aria-label="Open image" @click="openModal(t.image_id)"
:class="{ 'fc-post-card__rail-cell--ref': t.role }"
:title="t.role ? refTitle(t) : undefined"
:aria-label="t.role ? `Open ${refTitle(t)}` : 'Open image'"
@click="openModal(t.image_id)"
>
<img :src="t.thumbnail_url" alt="thumbnail" loading="lazy" />
<v-icon
v-if="t.role" icon="mdi-link-variant" size="x-small"
class="fc-post-card__ref-badge"
/>
</button>
<button
v-if="moreCount > 0" type="button"
@@ -138,9 +151,40 @@
query params" — so this works from Latest or Browse alike without
the card reaching for useRoute(), which it has no other reason to
know about. -->
<div v-if="associations.length" class="fc-post-card__assoc">
<!-- #4402: the unified card. A teaser is a pointer, so the card shows
what it points at — the drop's images join the rail above and
its text sits here — by REFERENCE: the Discord posts keep their
own place in the feed, and nothing here moves them. A link FC made
by itself says so and offers the undo the operator asked for. -->
<div v-if="unified" class="fc-post-card__unified">
<div
v-for="l in unified.links" :key="l.association_id"
class="fc-post-card__unified-head"
>
<v-icon icon="mdi-link-variant" size="x-small" />
<RouterLink
:to="{ query: { post_id: l.post_id } }"
class="fc-post-card__assoc-link"
>{{ linkLabel(l) }}</RouterLink>
<button
type="button" class="fc-post-card__undo"
:disabled="unlinking === l.association_id"
@click="undoLink(l)"
>{{ l.linked_by === 'fc' ? 'Undo' : 'Unlink' }}</button>
</div>
<div
v-for="t in unified.texts" :key="t.post_id"
class="fc-post-card__unified-text"
>
<div class="fc-post-card__unified-meta">
Discord · {{ shortDate(t.date) }}{{ t.role === 'variant' ? ' · variant' : '' }}
</div>
<p class="fc-post-card__unified-body">{{ t.text }}</p>
</div>
</div>
<div v-if="plainLinks.length" class="fc-post-card__assoc">
<RouterLink
v-for="a in associations" :key="a.id"
v-for="a in plainLinks" :key="a.id"
:to="{ query: { post_id: a.post_id } }"
class="fc-post-card__assoc-link"
>
@@ -191,6 +235,7 @@ import { RouterLink } from 'vue-router'
import { useModalStore } from '../../stores/modal.js'
import { usePostsStore } from '../../stores/posts.js'
import { toPlainText } from '../../utils/htmlSanitize.js'
import { toast } from '../../utils/toast.js'
import PostSeriesMenu from './PostSeriesMenu.vue'
import PostTranslationControl from './PostTranslationControl.vue'
@@ -207,8 +252,16 @@ const modal = useModalStore()
const detail = ref(null)
const attachments = computed(() => props.post.attachments || [])
const images = computed(() => props.post.thumbnails || [])
const totalImages = computed(() => images.value.length + (props.post.thumbnails_more || 0))
// #4402. A teaser's card also shows what it points at: the linked drop's
// images and the piece's other variants, each tagged with the post it really
// belongs to. They follow the post's own images so the teaser keeps its hero,
// and the post's own capped list stays a PREFIX of everything shown — which is
// what lets the "+N" tile and the modal playlist keep indexing correctly.
const unified = computed(() => props.post.unified || null)
const ownImages = computed(() => props.post.thumbnails || [])
const refImages = computed(() => unified.value?.thumbnails || [])
const images = computed(() => [...ownImages.value, ...refImages.value])
const totalImages = computed(() => ownImages.value.length + (props.post.thumbnails_more || 0))
const plainTitle = computed(() => toPlainText(props.post.post_title))
// #388 E2. Non-null `synthesized_by` means FC authored this row by grouping a
@@ -217,11 +270,18 @@ const plainTitle = computed(() => toPlainText(props.post.post_title))
// post dict by hand, must degrade to "not synthetic" rather than throw.
const synthesized = computed(() => Boolean(props.post.synthesized_by))
const messageCount = computed(() => props.post.synthesis?.message_count ?? 0)
// A drop of one message stays a synthetic post (teaser matching only looks at
// drops), but there is nothing grouped in it — #4390: "Grouped from 1 Discord
// message" described a wrapper, not a grouping. Say what it is instead.
const synthesisTitle = computed(() => {
const n = messageCount.value
if (!n) return 'Grouped from Discord'
return `Grouped from ${n} Discord message${n === 1 ? '' : 's'}`
if (n === 1) return 'Discord message'
return `Grouped from ${n} Discord messages`
})
const synthesisChip = computed(() =>
messageCount.value === 1 ? 'from Discord' : 'grouped by FabledCurator'
)
const hero = computed(() => images.value[0])
@@ -273,6 +333,44 @@ const grewAt = computed(() => (synthesized.value ? props.post.last_grew_at : nul
// never beside the artwork. Defaults to [] so a post dict from before the
// feature (or composed by hand) renders without a link rather than throwing.
const associations = computed(() => props.post.associations || [])
// The teaser side of a link is drawn by the unified block when there is one;
// only a link it does not cover (the drop's "announced by", or a feed payload
// from before #4402) keeps the plain text link.
const plainLinks = computed(() =>
unified.value
? associations.value.filter((a) => a.role !== 'announces')
: associations.value,
)
function linkLabel (l) {
if (l.linked_by === 'fc') {
return l.token
? `Linked by FabledCurator — matched on “${l.token}”`
: 'Linked by FabledCurator'
}
return 'Linked to its Discord drop'
}
function refTitle (t) {
return t.role === 'variant' ? 'a variant from Discord' : 'from the Discord drop'
}
function shortDate (iso) {
return new Date(iso).toLocaleDateString()
}
const unlinking = ref(null)
async function undoLink (l) {
unlinking.value = l.association_id
try {
await postsStore.unlink(props.post.id, l.association_id)
toast({ text: 'Unlinked — the Discord drop returns to the feed on the next load', type: 'success' })
} catch (e) {
toast({ text: `Unlink failed: ${e.message}`, type: 'error' })
} finally {
unlinking.value = null
}
}
const grewRelative = computed(() => (grewAt.value ? relativeFrom(grewAt.value) : ''))
const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString())
function relativeFrom (iso) {
@@ -298,7 +396,8 @@ async function fullImageIds () {
detail.value = await postsStore.getPostFull(props.post.id)
} catch { /* fall back to the capped feed list */ }
}
return (detail.value?.thumbnails || images.value).map((t) => t.image_id)
const own = detail.value?.thumbnails || ownImages.value
return [...own, ...refImages.value].map((t) => t.image_id)
}
async function openModal (imageId) {
@@ -447,6 +546,54 @@ function formatBytes (n) {
.fc-post-card__grew { color: rgb(var(--v-theme-accent)); }
.fc-post-card__assoc { margin-top: 8px; }
/* #4402 — the unified block. Quiet by design: it explains the references in
the rail, it does not compete with them. */
.fc-post-card__unified {
margin-top: 10px;
padding-left: 10px;
border-left: 2px solid rgba(var(--v-theme-accent), 0.5);
}
.fc-post-card__unified-head {
display: flex; align-items: center; flex-wrap: wrap; gap: 6px;
font-size: 0.8125rem;
color: rgb(var(--v-theme-accent));
}
.fc-post-card__undo {
padding: 0; border: 0; background: none; cursor: pointer;
font-size: 0.75rem; font-weight: 600;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__undo:hover { color: rgb(var(--v-theme-accent)); text-decoration: underline; }
.fc-post-card__undo:disabled { cursor: default; text-decoration: none; }
.fc-post-card__unified-text { margin-top: 6px; }
.fc-post-card__unified-meta {
font-size: 0.7rem; font-weight: 600; text-transform: uppercase;
letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__unified-body {
margin: 2px 0 0;
font-size: 0.85rem; line-height: 1.45;
white-space: pre-wrap;
color: rgb(var(--v-theme-on-surface));
display: -webkit-box;
-webkit-line-clamp: 3;
-webkit-box-orient: vertical;
overflow: hidden;
}
.fc-post-card__ref-meta { color: rgb(var(--v-theme-accent)); }
/* A referenced tile carries a corner badge: the images are Discord's, shown
here, and the card must not pass them off as the teaser's own. */
.fc-post-card__rail-cell { position: relative; }
.fc-post-card__rail-cell--ref { outline: 1px solid rgba(var(--v-theme-accent), 0.55); outline-offset: -1px; }
.fc-post-card__ref-badge {
position: absolute; top: 4px; right: 4px;
padding: 2px; border-radius: 4px;
background: rgba(var(--v-theme-surface), 0.85);
color: rgb(var(--v-theme-accent));
}
.fc-post-card__assoc-link {
display: inline-flex;
align-items: center;
@@ -6,9 +6,10 @@
>
<div class="text-caption fc-muted mb-3">
Some creators post a cropped fragment on Patreon to say the real thing has
landed in their Discord. FC proposes those pairs; nothing is linked until
you accept one. A wrong link would tell you two different pieces are the
same, so this stays a suggestion.
landed in their Discord. When the creator's own file name settles it —
the same working name on both posts and nowhere else in their library —
FC links the pair for you. Anything less certain waits here, because a
wrong link would tell you two different pieces are the same.
</div>
<v-switch
@@ -17,6 +18,17 @@
@update:model-value="store.setEnabled"
/>
<v-switch
v-model="auto" color="accent" hide-details density="compact"
:disabled="!enabled"
label="Link a pair outright when the file name settles it"
@update:model-value="store.setAuto"
/>
<div class="text-caption fc-muted mb-2">
Off, every pair waits for you here — including the ones there is nothing
left to decide about.
</div>
<v-row class="mt-2">
<v-col cols="12" sm="6">
<SettingNumberField
@@ -26,10 +38,13 @@
:disabled="!enabled" @change="store.setThreshold(Number(threshold))"
/>
<div class="text-caption fc-muted mt-1">
A pair always needs <strong>two</strong> reasons — being close in time
and the post itself mentioning Discord. Neither is enough alone at any
setting at or above 0.55, which is what stops a busy posting day from
producing false pairs.
A pair needs <strong>two</strong> reasons — close in time, the post
mentioning Discord or the server, or a marker the creator uses on
both and almost nowhere else. None is enough alone at any setting at
or above 0.45, which is what stops a busy posting day from producing
false pairs. A shared working name is weighed separately: it says the
two are the same piece rather than that they happened near each
other.
</div>
</v-col>
<v-col cols="12" sm="6">
@@ -46,6 +61,47 @@
</v-col>
</v-row>
<!-- #4402. What a link DOES once made: the teaser's card shows the drop
and the piece's other variants by reference. Nothing is moved — these
two windows only decide what the card shows and which duplicate the
feed leaves out. -->
<div class="text-body-2 mt-4 mb-1"><strong>On the teaser's card</strong></div>
<div class="text-caption fc-muted mb-2">
A linked teaser shows the Discord drop it announced — its images and its
message — and the other versions of the same piece: the wips, alts and
censor passes posted under the same working name. They stay where they
landed in the feed; the card only shows them together.
</div>
<v-row>
<v-col cols="12" sm="6">
<SettingNumberField
v-model="foldHours" label="Hide the drop's own card within (hours)"
:min="0" :step="1"
density="comfortable" max-width="none"
:disabled="!enabled" @change="store.setFoldHours(Number(foldHours))"
/>
<div class="text-caption fc-muted mt-1">
A drop this close to its teaser is the same release shown twice, so
only the teaser's card stays in the feed. A drop further away keeps
its own card. 0 hides nothing.
</div>
</v-col>
<v-col cols="12" sm="6">
<SettingNumberField
v-model="familyDays" label="Reach back for variants (days)"
:min="0" :step="1"
density="comfortable" max-width="none"
:disabled="!enabled" @change="store.setFamilyDays(Number(familyDays))"
/>
<div class="text-caption fc-muted mt-1">
How far either side of the teaser to look for the rest of the piece.
Measured on real drops: a piece's versions span up to about six
weeks, while unrelated pieces that happen to share a name are years
apart. 0 shows the drop alone.
</div>
</v-col>
</v-row>
<div class="d-flex align-center mt-4 mb-2" style="gap:12px">
<strong class="text-body-2">
{{ store.proposals.length }} waiting for review
@@ -57,8 +113,9 @@
</div>
<div v-if="!store.proposals.length" class="text-caption fc-muted">
Nothing proposed. That is the expected state most of the time — pairs only
appear when a post both lands near a drop and says it is about Discord.
Nothing waiting. That is the expected state most of the time — a pair the
file names settle is linked without asking, and everything else needs a
post to both land near a drop and point at Discord.
</div>
<div
@@ -80,6 +137,19 @@
{{ Math.round(p.score * 100) }}% ·
timing {{ Math.round((p.signals?.proximity ?? 0) * 100) }}% ·
says so {{ Math.round((p.signals?.declared ?? 0) * 100) }}%
<template v-if="p.signals?.marker">
· shared marker {{ Math.round(p.signals.marker * 100) }}%
</template>
</div>
<!-- The name, when there is one. A queue that cannot say WHY is one
you learn to click through without reading — and this is the only
signal that claims the two are the SAME piece rather than that
they happened near each other. -->
<div v-if="p.signals?.identity_token" class="text-caption fc-muted">
both call it
<code>{{ p.signals.identity_token }}</code>
({{ Math.round((p.signals.identity ?? 0) * 100) }}% — shared with
another post of theirs, so not conclusive on its own)
</div>
</div>
<v-btn size="small" variant="tonal" @click="store.accept(p.id)">Link</v-btn>
@@ -98,12 +168,18 @@ import SettingNumberField from '../common/SettingNumberField.vue'
const store = usePostAssociationsStore()
const enabled = ref(true)
const auto = ref(true)
const threshold = ref(0.6)
const windowHours = ref(24)
const foldHours = ref(24)
const familyDays = ref(60)
watch(() => store.enabled, (v) => { enabled.value = v }, { immediate: true })
watch(() => store.auto, (v) => { auto.value = v }, { immediate: true })
watch(() => store.threshold, (v) => { threshold.value = v }, { immediate: true })
watch(() => store.windowHours, (v) => { windowHours.value = v }, { immediate: true })
watch(() => store.foldHours, (v) => { foldHours.value = v }, { immediate: true })
watch(() => store.familyDays, (v) => { familyDays.value = v }, { immediate: true })
onMounted(async () => {
// Both swallow their own failures: a settings read that fails should not
@@ -150,6 +150,24 @@
</v-table>
</v-card>
<!-- Where the numbers came from, and when.
The worker counts are read by a background sweep and stored, not
fetched when this page loads (operator, 2026-09-23: *"is there a
reason this info isn't being tracked in the background and stored in
some way?"*). They are therefore up to one sweep old, and saying so
is the difference between a lagging number and a wrong one. -->
<p v-if="!store.lastError" class="fc-parts__measured mt-2 mb-0">
<template v-if="measured">
Waiting and worker counts were measured {{ formatRelative(measured) }}<span
v-if="sweepSeconds"
>, and are re-read every {{ Math.round(sweepSeconds) }}s</span>.
</template>
<template v-else>
Waiting and worker counts have not been measured yet — the first
reading lands within a minute of startup.
</template>
</p>
<!-- What the one control actually means. Operator, 2026-09-23: *"there's
nothing to describe what 'auto' means or why their needs to be or
should be on/off toggles."* There were three controls and no sentence
@@ -170,9 +188,9 @@
FabledCurator to work at all; ML tagging is the one that is genuinely
optional, and it ships at zero because switching it on downloads a
model. <em>Up to N</em> beneath each dial is what this machine can
hold — memory for ML tagging, processor cores for the rest — and it is
recalculated from the container's real limits every time this page
loads.
hold — processor cores for every lane, and for ML tagging whichever
runs out first, its cores or its memory. It is recalculated from the
container's real limits every time this page loads.
</p>
<p class="mb-2">
The shipped caps are one of each, which is right for a first boot and
@@ -218,8 +236,9 @@
</li>
</ul>
<p class="mb-0 text-caption">
Each worker loads its own copy, which is why this machine allows it
at most {{ lane.ceiling }}.
Each worker loads its own copy and asks for several processor cores
while it runs, which is why this machine allows it at most
{{ lane.ceiling }}.
<template v-if="lane.ceiling === 0">
It has too little memory to run this at all.
</template>
@@ -234,7 +253,7 @@ import { computed, onMounted, onUnmounted, ref } from 'vue'
import { laneStuckFor, useSystemActivityStore } from '../../stores/systemActivity.js'
import { useSystemHealthStore } from '../../stores/systemHealth.js'
import { formatRelative } from '../../utils/date.js'
import { laneAdvice, mergeParts } from '../../utils/systemParts.js'
import { laneAdvice, measuredAt, mergeParts } from '../../utils/systemParts.js'
const store = useSystemHealthStore()
const lanesStore = useSystemActivityStore()
@@ -272,6 +291,13 @@ const rows = computed(() => mergeParts(
store.parts, lanesStore.lanes?.lanes ?? [], laneStuckFor,
).map((row) => ({ ...row, advice: laneAdvice(row.lane) })))
// When the stored worker numbers were last read, and how often that happens.
// Both come from the endpoint rather than being restated here — the cadence
// is `SWEEP_PERIOD_SECONDS`, and a second copy of it in the UI would be free
// to drift from the schedule it is describing.
const measured = computed(() => measuredAt(lanesStore.lanes?.lanes ?? []))
const sweepSeconds = computed(() => lanesStore.lanes?.sweep_period_seconds ?? null)
const offOptionalLanes = computed(() =>
(lanesStore.lanes?.lanes ?? []).filter(
(l) => l.optional && l.slots_cap === 0 && l.models?.length,
@@ -299,16 +325,13 @@ async function apply(lane, fields) {
text: `${lane.display_name} is on. Downloading its model now — watch `
+ 'progress under Activity. It only happens once.',
}
} else if (reply && reply.applied === false) {
// Saved but not pushed — the lane is restarting, or the broker blipped.
// NOT an error: the reconcile carries it when the lane answers again,
// and saying "failed" would invite setting it a second time.
notice.value = {
type: 'info',
text: `Saved. ${lane.display_name} is not answering right now — `
+ 'it will pick this up within a few minutes.',
}
}
// There is no "saved but not applied" message any more, because there is
// nothing to report yet: the endpoint answers once the cap is STORED and
// tells the running lane afterwards, so the press cannot wait on a broker
// round trip (operator: "we shouldn't have to wait for the validation
// live"). Whether the lane has caught up is what its row says, and the
// row updates on the next poll a few seconds later.
} catch (e) {
// The endpoint's `detail` is written to be read by a person ("cap 10000 is
// above what this container can hold"). Surface it rather than a status
@@ -345,6 +368,7 @@ function step(lane, delta) {
.fc-sys__dot { width: 9px; height: 9px; border-radius: 50%; flex: 0 0 auto; }
.fc-sys__dot--ok { background: rgb(var(--v-theme-success)); }
.fc-sys__dot--stale { background: rgb(var(--v-theme-warning)); }
.fc-sys__dot--degraded { background: rgb(var(--v-theme-warning)); }
.fc-sys__dot--down { background: rgb(var(--v-theme-error)); }
.fc-sys__dot--unknown { background: rgb(var(--v-theme-on-surface) / 0.35); }
@@ -391,6 +415,9 @@ function step(lane, delta) {
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface) / 0.6);
}
.fc-parts__num { font-variant-numeric: tabular-nums; }
.fc-parts__measured {
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface) / 0.55);
}
.fc-parts__sub {
font-size: 0.68rem; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface) / 0.5); white-space: nowrap;
@@ -153,6 +153,24 @@
</div>
</v-col>
</v-row>
<v-row>
<v-col cols="12" sm="4">
<v-text-field
v-model.number="dl.download_revisit_days"
label="Re-check edited posts (days)"
type="number" :min="0" :max="365"
density="compact" hide-details
@blur="saveDownloader"
/>
<div class="fc-help">
A routine check keeps looking back this far after it runs out of
new posts, so a creator who edits an older post to add a file is
still picked up. New attachments download; the post text is
re-read from the same page, costing nothing extra. 0 turns this
off. Default 30.
</div>
</v-col>
</v-row>
<v-alert v-if="importStore.settingsError" type="error" variant="tonal" class="mt-2" closable>
{{ importStore.settingsError }}
</v-alert>
@@ -209,6 +227,7 @@ const dl = reactive({
download_schedule_default_seconds: 28800,
download_event_retention_days: 90,
download_failure_warning_threshold: 5,
download_revisit_days: 30,
extdl_mega_enabled: true,
extdl_gdrive_enabled: true,
extdl_mediafire_enabled: true,
@@ -45,8 +45,9 @@
>
<v-list-item-title>Recapture post text &amp; links</v-list-item-title>
<v-list-item-subtitle>
Re-grab every post's body + external links and localize inline images
already on disk — without re-downloading media
Re-grab every post's body + external links, localize inline images
already on disk, and import any downloaded file that never reached the
library — without re-downloading media
</v-list-item-subtitle>
</v-list-item>
<v-divider v-if="isNative && !running" />
@@ -75,10 +76,10 @@ const running = computed(() => props.source.backfill_state === 'running')
const recovering = computed(() => !!props.source.backfill_bypass_seen)
const recapturing = computed(() => !!props.source.backfill_recapture)
// Recover / recapture are native-ingester features (ledger-bypass re-walk and
// post-text re-grab), available to every native platform — not just Patreon.
// Mirrors backend download_backends.NATIVE_INGESTER_PLATFORMS.
const NATIVE_PLATFORMS = ['patreon', 'subscribestar']
const isNative = computed(() => NATIVE_PLATFORMS.includes(props.source.platform))
// post-text re-grab), available on every native platform. The backend says
// which those are (`native_ingester`); a copied list here went stale when
// Discord moved over.
const isNative = computed(() => !!props.source.native_ingester)
</script>
<style scoped>
@@ -679,7 +679,7 @@ async function onRecover(source) {
}
}
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
// external links and localize on-disk inline images, without re-downloading
// media. Reuses the backfill lifecycle/badge; stop via the same Stop control.
async function onRecapture(source) {
+16 -1
View File
@@ -10,7 +10,22 @@ export class ApiError extends Error {
}
}
async function request(method, url, { body, params, signal } = {}) {
const OPTIONS = new Set(['body', 'params', 'signal'])
async function request(method, url, opts = {}) {
// Refuse an option this wrapper does not know. `api.post(url, { id: 1 })`
// reads naturally and sends NO body — the payload lands in the options bag
// and is dropped — so the server answers `invalid_body` and the caller
// looks broken server-side. It shipped exactly that way in the Subscriptions
// "Add subscription" button (2026-09-24). Failing here names the mistake.
const unknown = Object.keys(opts).filter((k) => !OPTIONS.has(k))
if (unknown.length) {
throw new TypeError(
`useApi ${method} ${url}: unknown option(s) ${unknown.join(', ')} — ` +
'a request payload goes under `body`, a query under `params`'
)
}
const { body, params, signal } = opts
let fullUrl = url
if (params) {
const search = new URLSearchParams()
+17 -1
View File
@@ -94,9 +94,25 @@ const routes = [
const history =
typeof window !== 'undefined' ? createWebHistory() : createMemoryHistory()
// Where the window lands after a navigation. Without this the router keeps the
// window's scroll offset, so a view entered from halfway down another one opened
// halfway down itself, its heading and tab strip under the nav (operator-flagged
// 2026-09-24, Settings entered from a scrolled feed).
// - back/forward returns to where that entry was left;
// - the SAME view with a new query (a gallery filter, a Browse or
// Subscriptions tab) keeps its place — that is not a new page;
// - a new view opens at its top.
export function scrollBehavior(to, from, savedPosition) {
if (savedPosition) return savedPosition
if (from.matched.length && to.path === from.path) return false
if (to.hash) return { el: to.hash }
return { top: 0 }
}
const router = createRouter({
history,
routes
routes,
scrollBehavior
})
const DEFAULT_TITLE = 'FabledCurator'
+1 -1
View File
@@ -28,7 +28,7 @@ export const useMembershipReconcileStore = defineStore('membershipReconcile', ()
async function adopt (membershipId) {
try {
const res = await api.post('/api/sources/reconciliation/adopt', {
membership_id: membershipId
body: { membership_id: membershipId }
})
toast({
text: res.already_tracked
+33 -7
View File
@@ -6,16 +6,23 @@ import { useAsyncAction } from '../composables/useAsyncAction.js'
import { toast } from '../utils/toast.js'
// Backs the announcement review queue (#388 E5): "this Patreon post announced
// that Discord drop". Confirm-only, deliberately — a wrongly-asserted link
// tells the operator two different pieces are one, which is worse than no link
// at all, so accept is the ONLY thing that makes a link real. Mirrors
// seriesSuggestions (FC-6.3), which is the same shape for the same reason.
// that Discord drop". A wrongly-asserted link tells the operator two different
// pieces are one, which is worse than no link at all — so what reaches this
// queue is everything FC is NOT sure enough about to act on by itself.
//
// A pair the creator's own working name identifies, where that name is on
// these two posts and nowhere else in their library, is linked without asking
// (`auto`). Mirrors seriesSuggestions (FC-6.3) in shape.
export const usePostAssociationsStore = defineStore('postAssociations', () => {
const api = useApi()
const proposals = ref([])
const enabled = ref(true)
const auto = ref(true)
const threshold = ref(0.6)
const windowHours = ref(24)
// #4402 — what a link shows once made. See PostAssociationsCard.
const foldHours = ref(24)
const familyDays = ref(60)
const { loading, error, run } = useAsyncAction({ errorAs: 'message' })
async function load () {
@@ -28,8 +35,11 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => {
async function loadSettings () {
const s = await api.get('/api/settings/import')
enabled.value = s.discord_link_enabled
auto.value = s.discord_link_auto
threshold.value = s.discord_link_threshold
windowHours.value = s.discord_link_window_hours
foldHours.value = s.discord_link_fold_hours
familyDays.value = s.discord_family_window_days
}
async function saveSettings (patch) {
@@ -41,6 +51,11 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => {
await saveSettings({ discord_link_enabled: v })
}
async function setAuto (v) {
auto.value = v
await saveSettings({ discord_link_auto: v })
}
async function setThreshold (v) {
threshold.value = v
await saveSettings({ discord_link_threshold: v })
@@ -51,6 +66,16 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => {
await saveSettings({ discord_link_window_hours: v })
}
async function setFoldHours (v) {
foldHours.value = v
await saveSettings({ discord_link_fold_hours: v })
}
async function setFamilyDays (v) {
familyDays.value = v
await saveSettings({ discord_family_window_days: v })
}
async function accept (id) {
try {
await api.post(`/api/posts/associations/${id}/accept`, {})
@@ -76,8 +101,9 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => {
}
return {
proposals, enabled, threshold, windowHours, loading, error,
load, loadSettings, setEnabled, setThreshold, setWindowHours,
accept, dismiss, rescan
proposals, enabled, auto, threshold, windowHours, foldHours, familyDays,
loading, error,
load, loadSettings, setEnabled, setAuto, setThreshold, setWindowHours,
setFoldHours, setFamilyDays, accept, dismiss, rescan
}
})
+30 -1
View File
@@ -89,6 +89,35 @@ export const usePostsStore = defineStore('posts', () => {
return res
}
// Undo a link on a unified card (#4402). The operator chose "nest
// automatically, with visible undo" — so the undo is the review queue's own
// dismiss, whose kept row is what stops the next sweep linking the pair
// straight back. Patches the loaded teaser in place so the card drops its
// references at once; the drop's own card is not in the loaded page (it was
// folded out), so it returns on the next load rather than being guessed into
// position here.
async function unlink(postId, associationId) {
await api.post(`/api/posts/associations/${associationId}/dismiss`, {})
const item = items.value.find((p) => p.id === postId)
if (!item) return
item.associations = (item.associations || []).filter((a) => a.id !== associationId)
const unified = item.unified
if (!unified) return
const gone = unified.links.find((l) => l.association_id === associationId)
const links = unified.links.filter((l) => l.association_id !== associationId)
// Variants are keyed on the whole seed, so with one link left they may no
// longer all belong; with none left there is nothing to show at all. Drop
// what is certainly gone and let the next load recompute the rest.
item.unified = links.length
? {
...unified,
links,
thumbnails: unified.thumbnails.filter((t) => t.post_id !== gone?.post_id),
texts: unified.texts.filter((t) => t.post_id !== gone?.post_id),
}
: null
}
// Filter overlay for the around/older/newer (in-context anchored)
// path. Keep this distinct from `filters.value` (the down-only feed)
// so a normal-feed filter change doesn't leak into an active anchored
@@ -186,7 +215,7 @@ export const usePostsStore = defineStore('posts', () => {
return {
items, cursor, loading, done, error, filters,
cursorOlder, cursorNewer, doneOlder, doneNewer, anchorId,
loadInitial, loadMore, getPostFull, applyTranslationOverride,
loadInitial, loadMore, getPostFull, applyTranslationOverride, unlink,
loadAround, loadOlder, loadNewer,
}
})
+1 -1
View File
@@ -140,7 +140,7 @@ export const useSourcesStore = defineStore('sources', () => {
_patchSource(body)
return body
}
// #830: arm a recapture walk (Patreon-only) — re-grab every post's body +
// #830: arm a recapture walk (native platforms) — re-grab every post's body +
// external links and localize on-disk inline images, WITHOUT re-downloading
// media. Shares the backfill lifecycle/badge; stop via stopBackfill.
async function recaptureSource(id, artistIdHint = null) {
+25 -5
View File
@@ -88,17 +88,37 @@ export const useSystemActivityStore = defineStore('systemActivity', () => {
}
}
// Change one lane. Returns the endpoint's reply so the caller can tell a
// stored-but-not-yet-live change (`applied: false`) from a live one — the
// difference between "saved, the lane is restarting" and "that failed",
// which the UI must not collapse into one message.
// Change one lane, and PATCH the row from the reply.
//
// It refetched until 2026-09-23, on the reasoning that the server's whole
// table beats a local guess. The cost of that was the operator's:
//
// "something about changing the cap number is blocking to the website...
// it shouldn't be" / "the change should be queued so that it isn't
// blocking of the webui or the system itself. we shouldn't have to wait
// for the validation live."
//
// GET /api/system/workers costs a celery inspect — an eleven-second budget
// — so every press of `+` disabled the stepper until a broker round trip
// the press did not need had finished. The endpoint now answers as soon as
// the cap is stored and tells the lane separately.
//
// Only the three fields the reply actually decides are copied. The live
// columns (pool, active, pending) are measurements this reply does not
// carry and must not invent — the 15s poller brings them, a beat behind,
// which is what they are anyway.
//
// Deliberately NOT swallowing the error: a refused value (400) carries the
// sentence explaining why, and the card shows it. Returning null on failure
// would leave the operator with a control that silently did nothing.
async function setLane(name, fields) {
const reply = await api.post(`/api/system/workers/${name}`, { body: fields })
await loadLanes()
const row = lanes.value?.lanes?.find((l) => l.name === name)
if (row && reply) {
if (reply.slots_cap !== undefined) row.slots_cap = reply.slots_cap
if (reply.ceiling !== undefined) row.ceiling = reply.ceiling
if (reply.enabled !== undefined) row.enabled = reply.enabled
}
return reply
}
+24 -1
View File
@@ -24,7 +24,8 @@ export function queueKey(queues) {
}
// Worst first. A stopped datastore is why someone opened this tab.
export const SEVERITY = { down: 3, stale: 2, unknown: 1, ok: 0 }
// `degraded`: checking in, but slow — a GPU agent on the CPU (#4410).
export const SEVERITY = { down: 4, stale: 3, degraded: 2, unknown: 1, ok: 0 }
export function kindLabel(kind) {
if (kind === 'celery') return 'worker lane'
@@ -142,3 +143,25 @@ export function laneAdvice(lane) {
+ `Raise the cap to run more at once — this machine allows up to `
+ `${lane.ceiling}.`
}
// When the worker numbers in the table were actually read.
//
// They used to be fetched live on every page load — a celery broadcast per
// request, on a page that polls every 15s. Operator, 2026-09-23: *"there is a
// repull every time this page loads — is there a reason this info isn't being
// tracked in the background and stored in some way?"* It is now: a sweep
// measures on a timer and stores it, and the page reads the table.
//
// So the age has to be ON SCREEN. A number that is up to half a minute old,
// presented as current, is how someone ends up watching a queue "not move"
// that is in fact moving. The NEWEST across lanes, because they are written
// by one sweep in one pass — a lane lagging the others means its row has
// never been written, and that lane says so in its own detail line.
export function measuredAt(lanes) {
const stamps = (lanes || [])
.map((l) => l.measured_at)
.filter(Boolean)
.sort()
return stamps.length ? stamps[stamps.length - 1] : null
}
+1 -1
View File
@@ -369,7 +369,7 @@ onUnmounted(() => {
/* Center viewer. */
.fc-ex__viewer {
background: rgb(20, 23, 26);
background: rgb(var(--v-theme-background));
display: flex; flex-direction: column; min-width: 0; min-height: 0;
}
.fc-ex__canvas { flex: 1 1 auto; display: flex; min-height: 0; min-width: 0; }
+2 -2
View File
@@ -276,7 +276,7 @@ onUnmounted(() => {
.fc-reader__thumb img { width: 100%; height: auto; display: block; }
.fc-reader__thumbnum {
position: absolute; bottom: 4px; right: 4px;
background: rgba(20, 23, 26, 0.8); color: rgb(var(--v-theme-on-surface));
background: rgba(var(--v-theme-background), 0.8); color: rgb(var(--v-theme-on-surface));
padding: 1px 5px; border-radius: 4px; font-size: 0.7rem;
}
.fc-reader__content {
@@ -302,7 +302,7 @@ onUnmounted(() => {
.fc-reader__indicator {
position: fixed; bottom: 1.5rem; right: 1.5rem;
padding: 4px 10px; border-radius: 999px;
background: rgba(20, 23, 26, 0.6);
background: rgba(var(--v-theme-background), 0.6);
border: 1px solid rgb(var(--v-theme-surface-light));
font-size: 0.8rem; color: rgb(var(--v-theme-on-surface-variant));
z-index: 100; pointer-events: none;
+3 -3
View File
@@ -338,18 +338,18 @@ onMounted(() => {
position: absolute; top: 6px; left: 6px;
display: inline-flex; align-items: center; gap: 2px;
padding: 1px 6px; border-radius: 999px; font-size: 11px;
background: rgba(20, 23, 26, 0.75);
background: rgba(var(--v-theme-background), 0.75);
color: rgb(var(--v-theme-warning, var(--v-theme-accent)));
}
/* Kebab sits top-right of the cover, opposite the gap badge. Tinted backing so
it stays legible over any cover image. */
.fc-sbcard__kebab {
position: absolute; top: 4px; right: 4px;
background: rgba(20, 23, 26, 0.6) !important;
background: rgba(var(--v-theme-background), 0.6) !important;
border-radius: 50%;
color: rgb(var(--v-theme-on-surface));
}
.fc-sbcard__kebab:hover { background: rgba(20, 23, 26, 0.85) !important; }
.fc-sbcard__kebab:hover { background: rgba(var(--v-theme-background), 0.85) !important; }
.fc-sbcard__body { padding: 8px 10px; }
.fc-sbcard__name {
font-family: 'Fraunces', Georgia, serif; font-size: 15px; font-weight: 600;

Some files were not shown because too many files have changed in this diff Show More