652 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	23e062dd4a	test: the stall-sweep tests read the thresholds they check, and stay clear of the new import value (#4432) ... Run 7505 failed test_recover_stalled_task_runs_ml_queue_uses_longer_threshold. It restated the old 25-minute ml threshold as a 30-minute "stale" row, which is now inside the 40-minute window. The test now reads the value from QUEUE_STUCK_THRESHOLD_MINUTES. The archive test's fast-import row was exactly 10 minutes old, which is the new import threshold, so it passed only by the milliseconds between seeding and sweeping. It is now 15 minutes old. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:51:42 -04:00
bvandeusenandClaude Opus 5.5	a360d69ee8	fix: task runs record the lane Celery really routes them to, and no healthy long job is swept as stalled (#4432) ... celery_signals._queue_for was a hand-kept copy of task_routes and had drifted: - backup, admin and library_audit jobs, and backfill_phash, run on maintenance_long but were recorded as `maintenance`; - translation and gpu_queue jobs were recorded as `default`, where the 5-minute stall sweep failed healthy 35-minute translation runs. It now asks the router, cached per task name. A new guard test checks every registered task's hard time limit against the stall threshold the sweep would use for it. It also caught these sweeps, which failed healthy runs mid-flight and are fixed here: - ml's scheduled sweeps (35 min, previously swept at 25); - train_heads and apply_head_tags (65 min); - import_media_file (6 min, previously swept at 5); - the long lane, which now has its own 45-minute threshold. UI changes: - the admin job poller follows a job by celery_task_id, via a new filter on /runs, instead of by lane; - the archive re-extract and missing-file repair cards show the long lane's backlog, where their jobs actually wait; - the queue table lists maintenance_long. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:46:39 -04:00
bvandeusenandClaude Opus 5.5	efcb548ebf	fix: natively downloaded images take their post's date, not their download time (#4431) ... The native ingesters (Discord, Patreon, SubscribeStar) import a post's media before its record. Only the record (`_post.json`) carries the date: the Discord message timestamp, or the Patreon/SubscribeStar published_at. So each image was linked to a post with no date yet, and it kept its download time in both gallery date columns. The post itself was dated correctly once the record landed, but nothing went back to update its images. - upsert_post_record now re-dates the images already linked to the post. `effective_date` becomes the primary post's date, and `earliest_post_date` the earliest dated post the image is in, which are the same rules _attach_provenance applies. - Migration 0113 repairs the library from the posts. It writes only rows that differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:30:17 -04:00
bvandeusenandClaude Opus 5.5	7b1570f2a5	feat: retire the sketch/doodle WIP title tier, and the review strip asks "Is this a WIP?" (milestone 430) ... The soft tier (#1474) tagged `wip` on any post titled sketch, doodle or scribble: 6,096 of the library's 8,876 wip tags. Its conflict audit flagged most of them, because finished art scores >= 0.5 on some content head, which filled the Gallery strip with 2,086 cards. A "sketch" is usually finished work, so the operator retired the tier. The artist's own "WIP" / "work in progress" title rule and human wip tags stay. - Removed: the soft matcher, source and prefilter; the importer and backfill soft branches; soft_wip_conflict_audit with its task and beat; the wip_soft_title_tagging_enabled setting, API field and toggle; and wip_title_soft from _AUTO_SOURCES. - Migration 0112: a soft tag the operator confirmed, or kept from the strip, becomes `manual`. Every other soft tag is deleted, along with the open review cards whose tag is gone. The column is dropped. - Review strip (#4424): each card asks "Is this a WIP?" (or banner, and so on), and the buttons read "Is a WIP" / "Is not a WIP". The content tag it also scored on is shown as the reason it was flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 08:20:28 -04:00
bvandeusenandClaude Opus 5.5	83e1382812	fix: extension updates install the new build — no 12h-cached "latest" XPI, and the popup's Update opens FC's install page ... Operator, 2026-09-25: "the extension update trigger from inside the extension doesn't work and the manual update seems to not move it to the most recent version or at least mark it the most recent." - fabledcurator-latest.xpi was served with Quart's default `public, max-age=43200`: one URL whose bytes change every release, so a browser that had fetched it reinstalled the previous build for 12 hours (measured on the instance). It is now `no-cache` (the ETag keeps an unchanged file a 304); versioned XPIs are `immutable`. - The web Settings card installs/downloads the VERSIONED xpi_url, which can only ever be that build's bytes. - The popup's Update button did tabs.create() on the .xpi, which Firefox refuses (NS_ERROR_FAILURE on a 200: it only installs from a user click on a web page). It now opens FC's install card (/subscriptions?tab=settings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 08:01:02 -04:00
bvandeusenandClaude Opus 5.5	423275a1e5	feat: the artist picker on Patreon and SubscribeStar too — and the Patreon name is canon (milestone 429) ... Operator: "yes add the artist picker to patreon and subscribestar too. but generally we treat the patreon name as the canon" - Every add now goes through the panel. On Patreon/SubscribeStar it opens on the creator's display name (read only when the panel opens: probe?names=1), searches FC's artists with it, and auto-picks a match. An untouched URL handle sends no name, so the server still resolves it. - Patreon is canon: joining a Patreon source to an artist known by another name offers "Rename “x” to the Patreon name “X”", ticked by default. quick-add's use_platform_name renames server-side from the name it reads itself; name only, the slug never moves (#130); never to a URL handle when the name can't be read; ignored on SubscribeStar and Discord. - _platform_display_name returns None rather than the handle, bounded by the same 6s lookup budget as Discord's names. - panelDefaults / addRequest / renameOffer replace the Discord-only helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 07:50:48 -04:00
bvandeusenandClaude Opus 5.5	eeb9263125	feat: the Discord Add panel's artist field autocompletes against FabledCurator's artists (milestone 429) ... Operator, after the first live run: "I need the extension to offer an autofill search function so it's easier to match an entry with an existing artist." - The field searches as soon as the panel opens (the server name, usually) and on every keystroke; matches list under it, with ↑/↓, Enter/Tab to pick, Esc to close, and a last "+ New artist" row. - Inline autofill: the rest of the top match is filled in and selected, so typing on replaces it and Tab/Enter accepts it. Backspacing never refills. - A result whose name IS the text, spacing and case aside, is picked on its own; the hint says in green which existing artist the source will join. - /api/artists/autocomplete also matches ignoring spacing and punctuation ("Tamada Heijun" finds "TamadaHeijun"), ranked just below an exact match. The web UI's artist search gets it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 07:45:51 -04:00
bvandeusenandClaude Opus 5.5	4c75dd0f88	feat: the extension adds Discord channels to an artist you pick, and its tests gate the XPI (milestone 429) ... Server (#4420) - extension_service gains a Discord pattern (server or channel, jump links, ptb/canary; not DMs or threads), mirrored in platforms.js and pinned by the shared artist-url-samples.json. - probe on a Discord URL matches the source by ids under any artist, reports a whole-server source as covering the channel, suggests the artist who owns another source on the same server, and names server/channel via the stored token (best-effort, bounded, no rate-limit waits). - quick-add takes artist_id / artist_name; Discord URLs are stored canonical. Extension (#4421, #4422) - Content script on discord.com; SPA navigation by URL polling (the old pushState patch ran in the isolated world and never fired); stale probes are dropped. - Discord chip opens an Add panel: this channel or the whole server, and the suggested artist / a search / a new name. - Popup: sources show artist, platform and state; a Discord token export is verified by FC and the result shown. Token capture covers ptb/canary. - Pure logic in lib/chip.js and lib/popup-format.js, with specs. CI (#4423) - extension.yml's lane (web-ext lint, vitest, XPI contents) moves into build.yml as extension-test and joins the needs of sign-extension, build-web and build-agent. As a separate workflow it gated nothing: a red extension suite still signed and shipped the XPI (rule 177). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 23:31:38 -04:00
bvandeusenandClaude Opus 5.5	e5bdcd2596	feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428) ... - Recover and Recapture show on every native source. The menu gated them on a copied platform list ('patreon', 'subscribestar') that went stale when Discord moved over. Sources now carry `native_ingester` from the backend's own predicate. - A running backfill is due on every scheduler tick. Nothing queued a backfill's next chunk: each one waited for the source's regular interval, so an armed backfill sat idle until the next check (8h at the default) and a five-chunk walk took most of two days. The in-flight guard and the platform lock keep one chunk at a time. A failing source falls back to its backoff, and a stalled or out-of-budget walk stops being due. It also runs when the artist has auto-check off, since the operator started it by hand. - gallery-dl no longer carries Discord: its naming constants, platform defaults, sidecar-mirroring postprocessor and token injection are gone. The naming test moves to the native downloader and still renders against the real gallery-dl sidecar fixture. That is the guard that the files gallery-dl wrote are found on disk rather than fetched again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 23:14:48 -04:00
bvandeusenandClaude Opus 5.5	84e5448941	feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428) ... Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:42:00 -04:00
bvandeusenandClaude Opus 5.5	058fa85606	fix: backfill_phash runs on the long maintenance lane, not the scheduler's quick one (4411) ... A whole-library rehash with a 35-minute limit was matched by the maintenance.* glob. It held a scheduler process for its whole run, and the minute ticks queued behind it. An exact-name route now sends it to maintenance_long. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:26:48 -04:00
bvandeusenandClaude Opus 5.5	cc53d8db7b	feat: a GPU agent on the CPU shows as degraded, in the System view and on its own page (4410) ... torch and onnxruntime both fall back to the CPU without raising, so the agent that ran CPU-bound for weeks after a driver update leased and checked in like a healthy one. - The agent sends its startup accel report on every lease and heartbeat. - The server keeps a bounded copy on the roster row. A running agent with a runtime off the GPU becomes `degraded`, with a sentence naming the runtime and the reason. - The top nav shows it amber. - The agent page carries a banner, and its pill reads "CPU only". Also: the bandwidth field gets the page's − / + stepper, and both number fields drop the browser's spin arrows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:09:07 -04:00
bvandeusenandClaude Opus 5.5	42a40d71a4	fix: the agent's ONNX check asks CUDA for a device instead of trusting that the libraries loaded (1451) ... It reported "onnx on GPU" beside torch failing cuInit with "CUDA unknown error". Every library resolved, but no device could be used. The check now calls cudaGetDeviceCount and reports the CUDA error when there is one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 18:28:26 -04:00
bvandeusenandClaude Opus 5.5	f127c50207	fix: the lane sizer reads the live pool, so it stops growing a lane past its cap (4409) ... It read celery's `max-concurrency` as the pool size, but prefork reports that as the limit the pool booted with. pool_grow and pool_shrink never update it. A lane booted at 1 therefore read 1 forever. The sweep sent `target - 1` on every tick while work waited, and could never shrink, since 1 - 1 is 0. The System tab showed Scheduler "6 / 1" at a cap of 2. pool_size() counts `processes` and falls back to the limit only when there is no process list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 18:21:11 -04:00
bvandeusenandClaude Opus 5.5	7a09dc3cda	build: the agent installs one CUDA-13 stack instead of two, the web image drops ML packages it never imported, and Redis moves to 8 (1451, 1452) ... Agent: - The image ran PyPI's CUDA-13 torch 2.14 and onnxruntime-gpu 1.30 on a CUDA 12.9 cudnn-runtime base. requirements.txt had silently replaced the Dockerfile's torch 2.6+cu124, because ultralytics pulls torchvision, which pulls its own torch. That left ~3 GB of base libraries and a ~3 GB torch nothing loaded: 10 GB compressed. - Now: an nvidia/cuda 13.0.3 `base` image, with torch and torchvision installed together from cu130. CUDA and cuDNN come from the nvidia-* pip packages; onnxruntime-gpu declares its [cuda,cudnn] extras. - fc_agent/accel.py preloads those libraries for onnxruntime. It then logs, and reports in /status, whether torch and the ONNX CUDA provider actually got the GPU, since both fall back to the CPU silently. Web image: - Drop opencv-python-headless and onnxruntime, plus the opencv-only apt libs. Both have been listed since the scaffold and nothing in backend/ imports them. - torch/torchvision move to 2.14/0.29, and the unexplained caps are lifted (rule 154). Redis: 8-alpine in both compose files and both CI service containers. That gives an AGPLv3 licence option, where 7.4 was RSAL/SSPL only. The client moves to >=8.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 17:45:39 -04:00
bvandeusenandClaude Opus 5.5	ff70f837d0	refactor: test row factories and the fetch stub have one copy each (3109) ... tests/factories.py holds image_row/make_image/make_image_async/make_tag. The 17 byte-identical _img/_tag helpers (15 modules) now import them under their old names, so no call site changed. frontend/test/support/stubFetch.js replaces 15 copies that differed only in formatting. Copies whose bodies differ (other defaults, other columns, a url-only stub) are left as they are; folding those needs a look at each caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:39:38 -04:00
bvandeusenandClaude Opus 5.5	f2e4ee4d17	fix: beat takes each job's last run from task_run, so a redeploy no longer resets the schedule (4408) ... Beat's default scheduler kept its memory in a shelve file nothing persists, and a scheduler that remembers nothing waits a full interval before any job. Since the one-container image, every redeploy restarts beat, and no daily or weekly job had run since 2026-09-21 (cleanup, backup and download-event pruning, membership sync, thumbnail backfill, integrity check, vacuum). TaskRunScheduler seeds each entry's last_run_at at startup from task_run's newest start for that task: an overdue job runs at once, one not yet due waits the remainder, and one never recorded is due now. prune_task_runs now keeps each task's newest row however old, or a weekly job would look never-run a day after it ran and fire on every restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:36:58 -04:00
bvandeusenandClaude Opus 5.5	31020d9395	fix: a native chunk stops walking when its import work would overrun the task ... The walk's time budget covered the walk alone, but phase 3 runs in the same Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked for about two minutes and handed phase 3 431 orphan imports and ~3000 relinks. Phase 3 ran for 20 minutes and died at the soft limit (event 90808). This predates the worker consolidation; the limits are unchanged since June. The walk now also stops when elapsed time plus phase 3's estimated cost (2.5s per import, 0.25s per relink, measured on the live instance) passes CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so such a stop is a PARTIAL chunk boundary and the next chunk resumes the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:13:47 -04:00
bvandeusenandClaude Opus 5.5	7cc34b5724	fix: a Discord message that re-posts a file is grouped, and joins the drop it repeats ... Grouping read a message's images through primary_post_id alone, which only the first message imported with a file holds. The backfill runs newest-first, so the original message usually owned nothing: 101 of Yellowroom's messages (mostly 2018-2020) could never be grouped. Every image they carried also sat in another message (296 links, measured on the live instance). _message_images unions ownership with provenance for the candidate query, the drop seed, the member image links, and the merge pass. Nothing is re-owned. A later drop carrying the very file an earlier one carries now merges by a new same_image route, which nearest-neighbour could not see because it skips the image itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 15:21:05 -04:00
bvandeusenandClaude Opus 5.5	3aa899ad29	test: revisit tests' first walk completes phase 3 before the second ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:16:07 -04:00
bvandeusenandClaude Opus 5.5	da2091a875	fix: a download is marked seen only after it is imported ... A run killed between download and import left its files on disk and in the seen-ledger but never in the library, and every later walk trusted the ledger. TamadaHeijun's 12PCG post lost 7 of 13 images this way (stranded run 90402), which read as the duplicates filter. The ingester now hands phase 3 a mark_seen_after_import hook, called after the import loop. A file on disk with no ImageRecord at its path is fed to import, not reconciled into the ledger. Recapture reaches files already orphaned, because it looks past the ledger to the disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:13:12 -04:00
bvandeusenandClaude Opus 5.5	2f9e35390e	fix: the matcher sees a drop's names, finds merged trickles, and catches up on its own (4390, 4392) ... Three defects found while answering "is the linking automatic": 1. The name and image checks never worked on a live drop. The corpus keyed every image by primary_post_id, but a drop's images belong to its member messages and the drop claims them only through provenance, so a drop looked nameless and hashless. The tests attached images to the drop itself, which discord_grouping never does. Images now count under the post a reader sees them on: a message's absorbing drop, else the post itself. 2. The trickle merge (e08401c) dates a drop by its first stage, days before the release a teaser announces, which put merged trickles outside the 24h window. Candidates are now found and timed by their closest member message. The sweep follows recently grown drops by their messages' times the same way. 3. A pair left pending was skipped forever: the matcher skipped every recorded pair, not only decided ones. Pending pairs are now re-scored in place and linked once conclusive. Linked and dismissed pairs are still never touched. Also, "Scan now" shared the sweep's 48-hour horizon, so it could not reach the history it is described as being for. It now scores every post by an artist with Discord drops (rescan(full=True)). New tests build drops the way the grouper does, with images owned by the member messages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:01:07 -04:00
bvandeusenandClaude Opus 5.5	e08401c44a	feat: a creator's trickle becomes one drop instead of a card per message (4390) ... The operator, on a feed of "Grouped from 1 Discord message" cards: "the groups are still single image even when they can clearly be seen as group". 665 of Yellowroom's 714 drops were one message. Both join paths demand cosine <= 0.10 to a drop's FIRST image. The stages of one piece fail that: each is nearest the one before, not the first. `svtt_wip4` never joined `svtt_wip3` from the day before. Measured on artist 8 before writing it: - phash cannot see it. Stages sit 68-134 bits apart; unrelated same-artist pairs have a median of 126 and a p5 of 110 (lesson #4400). - The embedding's nearest neighbour can. Every stage of three real trickles had a sibling as its single nearest image in the artist's library. In a control over all 137 recent Discord images, a nearest neighbour that was another message within 7 days carried the same working name 53 times out of 53. Mismatches start past 7 days. A new merge pass runs last in the sweep. A later drop folds into an earlier one within discord_group_close_after_hours (168h, the measured 7 days) when they share a gated leading working name, or when one's image is the other's nearest neighbour. A drop reaching several earlier drops pulls them all together, unless two of them are named as different pieces. A merge carries teaser links across (the payload FK would otherwise cascade them away). Growth is stamped at the messages' own time, so merging history never jumps an old drop to the top of the feed. Each drop records the route it merged by, and is checked once. Offline replay over Yellowroom's 127 drops since 2025: they become 70 posts. The Marin trickle ("Very early Marin" -> 3 screenshots -> MarinaraSauce_base) becomes one post of 5 by nearest neighbour. svtt, 0-k1, cnni14 and 0adm come together by name. FAMILY_MAX_POSTS moves to post_naming, so the grouper and the teaser card share one definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 13:47:25 -04:00
bvandeusenandClaude Opus 5.5	30a263a47a	feat: a teaser's card references the drop it announced and the piece's variants (4402, 4401) ... The operator's problem: a Patreon teaser is a pointer, and its card showed the censored crop plus a text link while the content it pointed at sat on another card. The fix is a REFERENCE, not an absorption: "the nested items on the unified post are a duplicate or reference of existing content". Nothing is written. Discord posts keep their own rows, dates and places in the feed. - post_unification: for each teaser with a linked association, the drop's images and text, plus its variant family: Discord images sharing the seed's gated LEADING working name, or a phash near-duplicate, within a window of the teaser. One hop only, oldest first. - Measured on artist 8 before writing it: of 121 message pairs 2-60 days apart that share a gated token, 106 share the leading name and all read as real families. Of the 15 sharing only a trailing word, 14 are sibling pieces and one is a plain collision (`bottom`, 56 days). The family cap is 8, not the pairing cap of 6, because `tentacooler` and `0-k1` (6 posts each) are real families. - The feed drops a linked drop's own card only within discord_link_fold_hours of its teaser (default 24): "only hidden from the post view they're posted the same day". Older referenced posts stay where they landed. - post_association.linked_by records whether FC or a person made the link, so the card can say so. Undo is the existing dismiss. - `image0` (gallery-dl's fallback name) becomes a stopword. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 11:25:52 -04:00
bvandeusenandClaude Opus 5	f7b3e15014	feat: the drop carrying the teaser's own image links it (4392) ... Crop-to-source matching was held until the cheap signals could be shown insufficient. They can: of artist 8's 27 teasers with a drop inside a day, 11 go unlinked, and five are screenshot teasers with no working name at all. So it was tried, on exactly those pairs. Every teaser image correlated against every window of every nearby drop image at five scales, ground truth being the pairs the working name independently confirms, control being unrelated same-artist posts a month away. **It does not separate** — true pairs score as low as 0.401 while the control reaches 0.605, and no threshold divides them. The reason is the one the naive version was rejected for, which turns out to apply just as hard to the careful one: a single artist's work is stylistically homogeneous, so a whole-image comparison between two of their pieces is high whether or not it is the same piece. That is now written down in the module docstring with its numbers, so the next person to reach for it inherits the measurement instead of repeating it. What survived asks a narrower question the measurement shows IS answerable: not "is this a crop of that" but "is this the same image". Same pairs, same control, using the pHash FC already stores on every image — pairs the name confirms score 0, 0 and 20 bits of 256; the nearest unrelated pair in a 29-sample control scores 108. The threshold sits at 32, which is the number gallery_service already calls a near-duplicate, inside a 76-bit gap. It earns its place by being the only signal needing no cooperation from the creator: it works on a teaser called `Screenshot 2026-08-13`, and on a creator whose two platforms share no naming convention. It is quiet most of the time, because a teaser is usually a crop rather than a copy — but where it fires it is close to certain, and it recovers `Cute Selfie, Cute Dress` from the unreachable list. utils/phash warns the hash alone must not decide a MERGE, since variants of one piece collide at this distance. That does not invert here — it is the point. A merge destroys a file, so a variant colliding with its original is a loss; this asks whether two POSTS are about the same piece, and a variant of the drop's image is exactly that. Nothing is deleted either way. Gated on posts like the other two: an image on many of the creator's posts is a banner, not a piece. `_rarity` is public as `rarity` now that all three signals share it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:27:20 -04:00
bvandeusenandClaude Opus 5	941f1c6e07	fix: run 7464's two failures — an unused loop target and a missed call site (4392) ... Both mine, both from the same change. `identity` is unpacked in the write loop and never read there — the decision it feeds happens above it, when `conclusive` is built. flake8-bugbear is on repo-wide and B007 is exactly this. Renamed `_identity`, and `linked += status == "linked"` spelled out as the `if` it actually is. The second is worse, because it was a real assertion silently pointed at the wrong shape. `match_post` now returns `(proposed, linked)`, and when I rewrote the call sites I matched on `) == 0` — so the one comparison in the file that reads `) == 1` kept comparing a tuple to an integer. Found by walking the module's AST for every comparison against `match_post` and every use of a name assigned from it, rather than grepping again with the pattern that had already missed it once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:20:12 -04:00
bvandeusenandClaude Opus 5	31de54e326	feat: FC links a conclusive pair itself instead of asking (4392) ... Operator, 2026-09-24: *"I don't want this to be manual that defeats the convenience that I'm going for."* Confirm-only was right while every signal was circumstantial. Time proximity and a body that mentions Discord can never be more than suggestive, so asking was the honest response to what FC actually knew. A shared working name is different in kind: when the creator's own name for a piece appears in exactly these two posts and nowhere else in their library, there is nothing left for the operator to adjudicate, and asking is a chore FC invented for them. AUTO_LINK_FLOOR is 1.0 and sits deliberately above IDENTITY_FLOOR's 0.75. The gap between them IS the review queue — real evidence, not certain enough for FC to act on alone. Measured on artist 8, of 15 name-sharing pairs: 11 are conclusive, 2 more propose, 2 fall short of both. Three refusals, because an auto-link is FC asserting something the operator never saw: * Exactly one candidate may be conclusive. Two is not a tie to be broken by score — one post can tease two pieces dropped separately, and then each drop carries a different name from the same teaser, each individually conclusive. Two conclusive answers to "which drop is this" means the question was wrong, so both queue and FC says nothing. * Neither end may already be claimed by an accepted link. That link is the operator's decision and reassigning its other end would overrule them silently. * The whole thing is one setting, defaulting on, reversible in the UI — an accepted link is a row they can dismiss. `match_post` now returns (proposed, linked) so a sweep can report what it did on its own rather than only what it queued. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:14:04 -04:00
bvandeusenandClaude Opus 5	81f23991e9	fix: a sweep follows the drops FC has just authored, not only recent posts (4392) ... #4392's third cause, and the only one of the three about whether a pair is SCORED AT ALL rather than how well. A drop's post_date is backdated to its first message, but FC cannot author the drop until that message has an embedding and the hourly grouper has run. So a drop created this minute lands wherever its messages were — days or weeks back in the feed. The sweep only looked at announcements published within twice the window of NOW, so by the time the drop existed its neighbours were already outside the horizon, and nothing brought the sweep back to them. Measured on the live instance: a pair scoring 0.800 with `associations: []` and an empty queue. The two scoring fixes in the previous commit would not have helped it, because nothing scored it. So the sweep now also gathers announcements sitting beside any drop whose `last_grew_at`/`downloaded_at` is inside the horizon — the drop's own clock rather than its backdated position. Capped at MAX_RECENT_DROPS so a backfill authoring thousands at once does not quietly become the full-library rescan the manual button exists for, and the id set is sorted before the loop because a sweep visiting posts in a different order each run is one whose failures cannot be reproduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:09:09 -04:00
bvandeusenandClaude Opus 5	fd214f3a08	feat: the matcher gets an identity route, so a name can link what time cannot (4392) ... #4392 measured three compounding reasons the announcement matcher could not fire for this creator. This is the first two. **Vocabulary.** `discord` is not how these creators write once the audience already knows where the server is. Across 20,558 Patreon bodies `discord` appears in 486 and `the server` in 37 — but the distribution is the point, not the totals: for artist 8, 21 of 42 posts say `discord`, 7 say `the server`, and it is the RECENT ones that say the latter. A vocabulary list written from old posts silently stops matching the posts that still need it. **Two routes.** Circumstance and identity answer different questions. Proximity and a declaration say two things happened near each other; a working name the creator uses on these two posts and nowhere else says they are the same piece. Averaging those makes the threshold uninterpretable and costs both ways — adding identity as a fourth weight dilutes the others enough that measured teaser/drop pairs an hour apart stop proposing, while capping identity at a weight means the strongest evidence available can never carry a pair alone. So identity may OVERRIDE, never dilute; below IDENTITY_FLOOR it is recorded for the operator and moves nothing, which is the conservative direction, since a wrong link asserts two different pieces are one. The circumstantial bundle is now proximity/declared/marker at .45/.35/.20. Rebalanced rather than extended: the existing guard still holds arithmetically (no single weight reaches the threshold) and the existing pairs still score where they did — a teaser and its drop an hour apart with an invite link go from 0.815 to 0.781, both well over. Measured against the live instance, artist 8: 17 candidate pairs, 14 carried by a shared name, no false positives, and no drop claimed by two posts. The operator's own example — `Anya -- 🍈🍈` and the drop 75 minutes before it — links on the token `0-k`, which is what its two files are called on either platform. Two pairs 21h apart that proximity scores 0.10 are reached, and two more (`680lc`, `cnni18x`, four posts each) are the stated cost of the floor. Both rare-token signals are artist-scoped and loaded once per artist per sweep, not per candidate pair. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5	fb7563e263	fix: ruff UP031 — percent formatting in the new naming test (4392) ... Run 7456's only failure. `"comic2_%02d.jpg" % i` is an f-string here; UP is enabled repo-wide and I wrote the one construct it forbids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5	c67a8c313e	fix: count the POSTS a working name spans, not the files (4392) ... Counting files punished a piece for the one thing a working name is guaranteed to do — have several exports. Measured across the operator's four dual-platform artists: knuxy carries `p217` on four files spanning exactly two posts, the Patreon post and the Discord drop, and ~200 comic-page tokens have that shape. Every one scored half strength. Counting posts scores them 1.00 and still catches the habits, which span many posts rather than many files: tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4. Knuxy's ambiguous band drops from 268 tokens to 55. Three false positives found the same way — by running the module over the real library rather than reading it: * The screenshot guard matches from the start of the stem, so it never fired on the legacy `<post id>_media_<media id>_` era. `Screenshot 2025-07-27 182450ab` sailed through and contributed `2025-07-27` — the same-day date collision this module's docstring exists to refuse. * tamadaheijun names screenshots in Japanese. A guard that knows only the English word is a guard for one artist. * `timeline 3-0002` spans three unrelated posts. Keeping hyphens inside tokens for `0-k`'s sake let the number survive whole, so an identity token now has to contain a letter — which is the property behind the old bare-year and all-digits rules anyway, and leaves `0-k`, `680lc` and `p59` untouched. Also stopwords for `the`, `gif`, `main`, `patreon`, `capture`, `timeline`, each measured carrying a false match inside the admitted band. IDENTITY_FLOOR names what a shared name must reach to propose a link with no corroboration. At 0.75, 13 of artist 8's 15 name-sharing pairs clear it, including the operator's own example. The two that do not are real pairs this signal will not carry alone — the stated cost of refusing the four-post band, where conto's `illustration9` and `maid` also sit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:00:36 -04:00
bvandeusenandClaude Opus 5	ecbfd83029	feat: a creator's own working name links the teaser to the drop (4392) ... The Patreon teaser is a censored crop of the Discord release, which is the pair whole-image comparison handles worst — a censor bar is exactly the local edit that moves a perceptual hash and blurs an embedding. But the creator exports both from one file and the internal working name survives into both platforms untouched, so the filename answers what the pixels will not. Measured on artist 8 (520 images, 409 tokens): 17 candidate pairs, 14 of them carried by a shared name, no false positives and nothing ambiguous. Ten are byte-identical names either side — ConnFront/ConnFront, LoisLaneTB2/ LoisLaneTB2 — and the signal is orthogonal to timing, reaching a pair 23.8h apart that proximity scores 0.005. Both signals gate on rarity within ONE ARTIST's library, through one shared _rarity so they cannot drift apart again. They already had: the filename side grew a frequency cap and the text side never did, so a habitual emoji scored the same 1.00 as a marker used twice. Measured, on real proposals: the deciding evidence for one pair was 💦 (13 of 300 posts) and for another the word "like" (43 posts). Both now score zero. The cap discards exactly what it should — anya 8, riju 9, undyne 8, bea 16 are character names, and ungated every Anya post would match every Anya drop. A screenshot contributes nothing rather than the date match it could be squeezed for: that date collides across platforms by construction, since the teaser and the release go out the same day. Not yet wired into post_association_service — that is the next commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 07:55:18 -04:00
bvandeusenandClaude Opus 5	e1cd629196	feat: the agent derives its version instead of being asked to remember it (4386) ... Operator: the agent's build string could not identify the agent. VERSION was a literal in app.py an author was meant to bump, and nobody did — the September image printed the same "2026-07-17.1" as the July one, so the one surface meant to answer "did my pull work?" answered the same either way. Nothing new was needed. scripts/artifacts.sh has derived a version per artifact since milestone 313, and build-agent has been computing the agent's on every run and printing it to the log. The image just never carried it. Three values, never folded together (rule 149): FC_VERSION YYYY.MM.DD.HHMM from the COMMIT its shipped files last changed in — identical on dev and main for the same source, which is what makes "am I running production's code?" answerable. FC_CHANNEL a sibling field, never a suffix inside the name. FC_REVISION the 12-char sha; the same string as the fc.revision LABEL, so the image and the registry cannot disagree about which commit this is. The page SHOWS the version and COMPARES the revision. Those were one value before, which is how a version acquires a second job and then cannot be changed without breaking the reload banner. An unstamped local build reads `unknown` and compares `local` — absent rather than empty, one spelling of "cannot say". scripts/artifacts.sh joins the AGENT path set in the same commit, and it had to: a version has no backstop. A revision that is computed differently stops matching the published label and forces a rebuild, so it self-corrects; a version is compared against nothing, so a change to cmd_version alone would leave the agent publishing the old format with nothing to contradict it. That is #3202's finding, and the agent was rightly exempt only while it had no version of its own. tests/test_artifact_paths.py pins it. Also corrects two build.yml comments claiming agent/ had not changed since 2026-07-17. Both were already false — it changed 2026-09-23 — and one of them is the stated rationale for the force_build escape hatch. Rewritten without dates: how long an artifact has been quiet is a `git log` question, and its answer in a comment is wrong the next time anyone commits (lesson #4383). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 07:07:31 -04:00
bvandeusenandClaude Opus 5	970d17f98a	fix: a second _seed_seen shadowed the one the recapture tests call (4386) ... Run 7422 integration: two #830 tests died with `_seed_seen() got an unexpected keyword argument 'post_id'`. Not those tests' fault — I defined a second `_seed_seen` at the BOTTOM of the module, taking a list of media instead of one key, and Python's last definition wins for every call site in the file including the ones 500 lines above it. Renamed to `_seed_all_seen` and made it loop over the existing one-key helper, so there is one definition of what seeding the ledger means. The nearby failure mode is worth naming: a helper defined below the tests that use it is invisible at the point of use, and shadowing produces a TypeError in code nobody touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 06:39:49 -04:00
bvandeusenandClaude Opus 5	2b093958a4	feat: a tick keeps looking back 30 days, so an EDITED post is reached (4386) ... Operator, 2026-09-23, on a Floppystack post: "this post has been updated as he implements hot fixes — any chance we have a way to scan for or see updated posts so we can update ours to match and pull the new attachments and pictures etc." The download half already worked: extract_media reads the media list off the LIVE feed response every walk, so a newly attached hotfix build is a ledger key we have never seen. Only REACHING the post was missing — a tick stopped after 20 contiguous already-have-it items, and a post edited three days after publication sits well below twenty. Not a bug in the early-out; a count cannot express "recent". The early-out now needs BOTH conditions: the run of seen items AND a post published before the horizon. Strictly a widening — window 0 is exactly the old behaviour, and no window can make a tick stop EARLIER than it used to, so a source paused for months still walks its whole unseen backlog. The horizon is a floor on how far to look, never a ceiling. Inside the window the post-record gate is bypassed too (write_post_record revisit=True): the body is re-read from the feed response already in hand, so a revisit costs zero requests, and a body that comes back empty writes NOTHING rather than blanking one a detail-fetch had filled. Revisits are kept out of the #862 body-drift canary's sample for the same reason — an empty revisit is healthy, and counting it would walk the alarm toward firing on good ticks. The run summary names what changed ("3 post(s) updated (5 new file(s))") with a line per post; the ask was to SEE updated posts, not only to end up with their bytes. download_revisit_days is a settings row, not a constant (rule 25) — how long a creator keeps editing is a property of the creator. Default 30, 0 turns it off. Migration 0108. Also corrects two stale docstrings: both clients described post_meta as feeding an Ingester.preview that no longer calls it. It had no consumer at all until this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 19:21:06 -04:00
bvandeusenandClaude Opus 5	ffbe21098c	fix: l is an ambiguous name and ruff says so (4295) ... E741, run 7410. Two comprehensions in the new endpoint tests. Every other lane was green — backend unit, frontend and integration — and nothing published, which is the gate doing its job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:55:36 -04:00
bvandeusenandClaude Opus 5	45bb7044f7	feat: the System tab reads a stored sample instead of inspecting per load (4295) ... Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5	7f1693a40d	fix: the ML dial offered slots the machine had no cores to feed (4295) ... Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in flight by Little's law, and the daily CCIP sweep dying on its 1800s soft limit in a numpy matmul. The billiard/pool.py frame in that traceback is the soft-timeout signal handler, not a pool fault. Two causes, both mine. 1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading "keep N_replicas x this within the cores allotted to ML" — a constraint stated where nothing could act on it. A large-memory host offered ~49 slots, the operator took what the dial offered, and the lane asked the box for ~200 torch threads. The number moves onto the lane as `threads_per_slot`, the embedder reads it rather than restating it, and the ceiling is now the smaller of the two bounds. They fail differently on purpose: too little memory is honestly zero, because the first task would OOM the container; too few cores is merely slow, so it floors at one rather than making the lane unreachable on a small box. 2. `scheduled_ccip_auto_apply` scored one image per matmul, over every image in the library, on every daily run — ~119k products each too small to pay for its own BLAS setup. `char_maxima` does the same arithmetic in blocks bounded by elements, so its memory stays flat as either axis grows. Batching changes no arithmetic: a character's score for an image is a max over that image's figures AND that character's prototypes, and max does not care how it is grouped. Pinned against the old loop written out longhand, and against itself with the blocking forced to split every row. The UI copy said the ML ceiling came from memory; it says cores or memory, whichever runs out first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 16:46:22 -04:00
bvandeusenandClaude Opus 5	1353d346b3	fix: the cap dial waited out a broker round trip it did not need (4295) ... Operator: "when the number is changed the change should be queued so that it isn't blocking of the webui or the system itself. we shouldn't have to wait for the validation live." Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection being HELD across them, which is what had been stalling the whole site, and left the press itself as slow as it was. 1. The store refetched after every write. GET /api/system/workers runs a celery inspect on an eleven-second budget, so the stepper stayed disabled through a round trip the press did not need. It now patches the row from the reply — cap, ceiling, enabled, the three fields that reply actually decides — and lets the 15s poll bring the live columns, which are measurements it must not invent. 2. The endpoint pushed to the broker before answering. Turning a lane off is four cancel_consumer messages; lowering a cap reads the live pool first. Now it stores the cap, answers `queued`, and hands the push to a Quart background task. Raising a cap was already free and stays free. Nothing is lost by not waiting: the stored cap is what the system obeys and the sizing pass re-reads it every minute. That sweep was already the backstop for a push that failed, which under `no_live_workers` is every push in the suite. Also closes a hole the move exposed: the model fetch was gated on the consumer change having landed, so raising ML off zero while the lane was restarting stored the cap, let the sizing pass start the consumers a minute later, and left the lane running with no model — nothing else ever asks for one. It now fires on the transition and waits in the ml queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:41:32 -04:00
bvandeusenandClaude Opus 5	48108a3569	fix: a lane that is OFF was not attributable to itself (4295) ... Operator, 2026-09-23: *"clean up the stale service_seen rows"*. **They were not stale.** They were phantoms, written on purpose, and they will come back on every install that turns a lane off — so the rows are the smaller half of this. A celery worker was attributed to its lane by the queues it was CONSUMING. A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` with an empty list, matches no lane, and is dropped. Three consequences, all on the operator's screen at once: 1. The lanes table reported the lane **not answering** — the signal for a crashed worker, not for one the operator turned off. 2. The roster grew a phantom row named **`Worker ()`** — the empty queue set rendered as a display name — shown "running" beside the real lane's row going stale, because nothing updated it any more. 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every lane present. ML ships at cap 0, so a fresh install was permanently unhealthy and Swarm restarts an unhealthy task forever. That third one is the severe one, and its docstring asserted the opposite of what the code did — *"a disabled lane still runs its process with its consumers cancelled, so it answers inspect and is healthy"*. It answers. It was not attributed. A comment can be right about the intent and wrong about the program, and this one had been wrong since the consolidated container shipped. `worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels with the process rather than with what it happens to be doing. `gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the information was there and nothing read it. Falls back to the queue set for a deployment that names no node, and `docker-compose.yml` now sets one per service so the multi-service stack gets it too. The roster keys on the LANE's queue set when the node resolves, which is the same string the row already had while it was consuming — so an existing row keeps updating rather than a second one appearing. Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately NOT a retention sweep: the roster never forgets on purpose, so a quiet row is what it is FOR, and only a row that cannot correspond to anything real is safe to remove. An `agent:agent` row, if one exists, is left alone — nothing here can tell an abandoned agent id from a second agent that is genuinely down, and hiding a dead GPU agent is the one thing the roster must not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:21:21 -04:00
bvandeusenandClaude Opus 5	a4c66601db	fix: the heartbeat guard grepped its own explanation (4295) ... Run 7375: the test that asserts the agent's heartbeat is not gated on holding leases failed — on the docstring of the fix, which quotes the construct the fix removed, because that is what a docstring explaining a fix does. assert "if ids:" not in loop A source-TEXT assertion cannot tell code from prose about code. Parsed now: the function's AST body, unparsed with the docstring node dropped, so the guard reads only what executes. Worth stating as the general shape, since this repo writes long explanatory comments on purpose: any check that greps source for the absence of a pattern is in tension with documenting why that pattern is gone. Either it excludes the prose, or the next person to explain the fix breaks the guard that protects it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:59:06 -04:00
bvandeusenandClaude Opus 5	693759f2bb	fix: an idle GPU agent could not check in, so the roster called it stopped ... Operator, 2026-09-23: *"I'm running the gpu agent on my device and it currently reads as 'offline' but it's running and has checked in recently."* It had checked in — twelve minutes ago. Two cadences that never agreed: idle lease poll ceiling 900s agent/fc_agent/worker.py (sleep mode) heartbeat while idle never gated on holding leases roster "stopped" after 300s api/system_health.py The roster records an agent check-in on `lease` and `heartbeat`. The heartbeat loop was gated on `if ids:`, so an agent holding no leases sent nothing at all — leaving the lease poll as the only check-in, and sleep mode backs that off exponentially to a 900s ceiling. 900 against 300: an IDLE agent was structurally guaranteed to read as stopped. Nothing was broken; nothing was misconfigured; the two halves simply disagreed. Not a recent regression. Sleep mode landed 2026-07-02; the roster adopted the lease as its check-in on 2026-09-02 — *"A lease IS the check-in … Recorded on the call that was already happening"* — without noticing that the call it was piggybacking on had been deliberately slowed ten weeks earlier. The heartbeat now sends whether or not it holds leases. An empty one extends nothing (`id.in_([])` matches no rows) and costs one small POST every 45s — against the 6/min lease poll sleep mode exists to avoid, that is not a cadence worth protecting, and it is what makes "is the agent alive" answerable at all. Still gated on `self._running`: a worker that has been stopped is not checking in for work, and reporting it as present would be a different lie. Two things I could NOT determine from the code, both needing the live table: whether a stale `agent:agent` row exists from an older build that omitted `agent_id` (the server defaults it), and whether changing `AGENT_ID` has ever stranded an abandoned row — nothing prunes `service_seen`, so either would sit there reading "stopped" forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	5b6f2ba526	fix: a Postgres connection was held across every celery round trip (4295) ... Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	895589a578	fix: first boot raced itself for the credential key (4295) ... Run 7368's smoke, on an image whose five verification lanes were all green: WARNING Generating NEW Fernet credential key at /images/secrets/credential_key.b64 ... ValueError: Fernet key must be 32 url-safe base64-encoded bytes. Nothing to do with this batch's changes — it is a first-boot race that has been there since the key file existed, and it is a RACE rather than a certainty: the same code booted cleanly on the three runs before it. hypercorn starts several worker processes and each one builds the app, so on a first boot they all reach the bootstrap together. `write_bytes` creates the file at size zero and fills it a moment later, which gives the second process an `exists()` of True and a `read_bytes()` of `b""`. A first boot that fails one time in five is worse than one that fails every time, because it looks like the deployment rather than the code — and this is the very first thing a new install does. The key is now written to a temp file and `os.link`ed into place. `os.link` is the atomic part: it either creates the name or raises FileExistsError, and it cannot expose a half-written file. Deliberately NOT `os.replace`, which would succeed — two processes that both generated a key would each think they had won, and the loser's key would overwrite the one the winner had already handed to Fernet. The losing branch reads the winner's key back rather than returning its own, which is what keeps every worker on ONE key. Tested for AGREEMENT, not for the absence of a crash: eight threads through a barrier, and all eight must end up holding the same key. A race that left each worker with its own would pass a "did it raise" check and produce a system where a credential written by one worker cannot be read by the next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 13:04:26 -04:00
bvandeusenandClaude Opus 5	61641fbba7	fix: a test still described the control the cap replaced (4295) ... Run 7367, integration lane: FAILED test_a_cap_is_stored_even_when_it_cannot_be_pushed assert True is False The code was right. Raising a cap is PERMISSION, not a request — it deliberately does not grow the pool, because that would put workers on a lane with nothing to do, and the sizing pass spends the permission on its next tick if there is work. So nothing is pushed and `applied` is vacuously true. The test was carried over from when the number meant "run this many", where every write pushed. It asserted the old control's behaviour against the new one — lesson #4338's shape again: an assertion encoding the thing that changed, failing on the change rather than on a defect. Split into the two cases that actually exist now: - raising a cap stores it and pushes nothing, reporting applied; - turning a lane OFF does push, because consumers follow the cap immediately in both directions — off must take effect when it is asked for — so with nothing answering it reports `applied: false` with a reason, and the value is still stored for the sizing pass to carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:57:33 -04:00
bvandeusenandClaude Opus 5	364e050632	fix: a spliced-out test left its tail behind (4295) ... Run 7366, in both the lint and unit lanes — one bug wearing two hats: NameError: name 'slots' is not defined tests/test_worker_lanes.py:278 Rewriting `test_worker_lane_check_constraints` — parametrised over (slots, cap, ok) — I cut the parametrize decorator and the assertion but left the two lines after it, which still referenced all three names. `ast.parse` is happy with that; only running it is not. Both failures were the same fragment, which is the tell: the lint lane's F821 and the unit lane's NameError are the same fact reported twice. Checked the rest of today's splices for the same seam rather than fixing only the one CI named — a module-level undefined-name sweep over all four test files I rewrote today comes back clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:53:27 -04:00
bvandeusenandClaude Opus 5	830394ed5e	fix: the migration's DROP CONSTRAINT names doubled their own prefix (4295) ... Run 7365, integration lane: (psycopg.errors.UndefinedObject) constraint "ck_worker_lane_ck_worker_lane_slots_within_cap" of relation "worker_lane" does not exist That is #3275 exactly, from the other direction. alembic 0088 had to RENAME four constraints CREATED with a doubled prefix; this one tried to DROP two with the same doubling. `op.drop_constraint` runs its name through Base.metadata's naming convention, which prepends `ck_worker_lane_` to a string that already carries it — `op.f()` is what marks a name as final, and 0103 used it on the way in. The model test also went red, correctly: `test_worker_lane_check_constraints` was parametrised over (slots, cap) pairs and asserted all three constraints, and two of them went with the `slots` column. It is one unparametrised test now, asserting the whole remaining set rather than a membership — a constraint left behind naming a dropped column is not a harmless leftover, it is a table the migration cannot have produced. Worth recording: **the gate worked.** Run 7365 skipped `sign-extension`, `build-web`, `smoke-web`, `promote` and `build-agent`, and `:dev` still names the previous digest. That is the red-direction verification #4339 owed, and it arrived by accident rather than by a forced failure — which is the better evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:51:27 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5	274f7ffe21	ci: the tests gate the publish — ci.yml folds into build.yml ... Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*, then *"I don't want failing builds to publish anywhere going forward."* Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc` and `build-web` pushed `:dev` in the same minute, because the lanes and the build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and nothing about that was visible from either run. Two workflows cannot express the gate. A `needs:` edge only exists inside one graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted; `sign-extension`, `build-web` and `build-agent` now need all five. Nothing here is a new mechanism — it is the same edge that has gated `promote` since milestone 362 step 4, and it keeps that step's hardest-won property: **not running is not the same as passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane that silently skips itself blocks the publish exactly as a failing one does. Run 5290 is why that is worth stating. Scope, said plainly rather than implied: - Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension` publishes too, so it is gated with the rest. - Not gated, deliberately: `extension.yml` publishes nothing, and `release.yml` runs on a `v*` tag, generates notes rather than an artifact, and its commit already went through main's gated build. - `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its runs are the lanes and nothing else, via an `if:` on each publishing job rather than an inference from the `needs` chain. The cost, accepted knowingly: this workflow queues per branch and never cancels, so on two pushes in quick succession the second's lint feedback waits out the first's build. A slower red beats a fast red that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:15:42 -04:00