680 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	9715143492	fix: a restore with pg_restore 17 into Postgres 16 isn't stopped by a setting the server lacks ... pg_restore 17 opens with `SET transaction_timeout = 0`, which Postgres 16 refuses. Under --exit-on-error that one harmless statement ended every restore: the image ships pg_restore 17.11 and the bundled server is pg16, so a restore from the bucket (and the local restore) could never have loaded a dump. The backup round trip, now that CI has pg_dump, found it. pg_restore now runs to the end and its errors are judged afterwards: the known harmless one is ignored, and any other fails the restore with that error's words rather than the "Command was:" line that followed it. Also sorts an import the lint lane flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:00:00 -05:00
bvandeusenandClaude Opus 5.5	1e63f70bf7	fix: a media backup or restore cut off by a restart carries on where it stopped ... A media backup and a restore's file phase run as chains of 45-minute chunks, each queuing the next when it ends. A restart killed the chunk in flight, so nothing queued the next one; the run's row stayed "running", "Back up now" skipped it as already running, and only the 7-hour stall sweep ended it. - When the long-maintenance lane starts, it queues the next chunk of every media run still running and every restore fetching its files. - Each chunk writes its token to the run and hands it to the chunk it queues. A chunk queued by any other chunk is dropped (a killed chunk's message redelivered by acks_late, say), so a resume never doubles the chain; a chunk taken over stops at its next file. - The integration job installs postgresql-client when the CI image lacks it: the backup round trip in 375e7e2 runs the real pg_dump and pg_restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:53:57 -05:00
bvandeusenandClaude Opus 5.5	375e7e2eda	feat: restore a new install from the off-site bucket, database first, then every file ... Settings → Maintenance → Backups gains "Restore from the bucket": pick one of the bucket's dated dumps, type its confirmation, and FabledCurator saves the current database to the bucket, replaces it with the dump, migrates it to this version, then fetches every image and attachment back to the path its row records. Each file is checked against its sha256 on arrival and moved into place only then; a file already on disk is left alone, so "Fetch missing files" repairs a library that lost files without touching the database. - services/backup_restore.py: list/find/fetch dumps, migrate, keep this install's destination (the dump's sealed secret can't be opened here), library_files work list, restore_file (one fetch per sha, copies for duplicate paths, refuses paths outside /images). - restore_offsite_task: the database phase, then 45-minute file chunks with a heartbeat; a failed load leaves an empty migrated database and says why. The dump's own run row is marked ok, other runs then going "Interrupted". - backup_s3.get_file: sha/size check on arrival, ObjectMissing for a 404, download pacing. - backup_service: every dump leaves the credential table's rows out; restore_db reports pg_restore's last line, stops at the first error, ends other connections and waits at most 60 s for locks. - Backups and orphan repair wait while a restore runs; the stall sweep judges restores by heartbeat; restore runs are pruned to 20. - CLI: python -m backend.app.scripts.restore list|database|files. - README: backup and restore section; logins are in no backup. - Tests: a round trip through an in-process S3 bucket (tests/fake_s3.py) with real pg_dump/pg_restore, plus file, run, API and get_file cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:34:47 -05:00
bvandeusenandClaude Opus 5.5	d465198c4a	fix: the media backup task has its own stall threshold, and its test seeds a post for a shared archive ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:15:26 -05:00
bvandeusenandClaude Opus 5.5	0cf8edaa10	feat: images and attachments go to the off-site bucket once each, by fingerprint ... With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:07:15 -05:00
bvandeusenandClaude Opus 5.5	772482d794	feat: with an off-site destination set up, database backups stream to it ... The nightly database backup (and "Run DB backup now") now sends pg_dump's output straight to the off-site bucket as a multipart upload, with no copy staged on this server. The object is named database/fabledcurator-<UTC time>.dump and appears only once pg_dump has exited cleanly and every part has landed; a failed dump aborts the upload and records pg_dump's last line. Without a destination, dumps are written to /images/_backups as before. - backup_s3: Upload (start / add part / complete / abort), with a busy or silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and a Pacer that holds uploads to the new bandwidth limit. - Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default, 0 for none (migration 0120). - pg_dump runs under nice (and ionice where installed), locally too. - Backups card says where dumps go; the runs table gains "Saved to" and shows a failed run's error. Off-site rows have no restore/tag/delete here: restoring from the bucket is a later step (#6880). Task #6878, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:11:12 -05:00
bvandeusenandClaude Opus 5.5	bb10d30b2d	fix: two copies of one picture imported at once are filed as duplicates, and an archive's row always ends ... The sha256 lookup before inserting an image row can lose a race to another worker importing a second copy of the same picture. The unique index then refused the insert. A media import retried behind the whole import queue, and the recovery sweep ran it a second time. An archive swallowed the error, failed on the dead transaction, and its row stayed 'processing' because the failure mark could not commit either. - Importer._store_new_record inserts under a savepoint; on a conflict the file is filed as a duplicate of the row that won (_file_as_duplicate, now shared by every duplicate branch). _import_media removes its stray library copy. - _import_archive commits its post before extracting; the member loop rolls back a failed member and lets SoftTimeLimitExceeded through. - _mark_failed rolls back before writing the failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:35:51 -05:00
bvandeusenandClaude Opus 5.5	32cac46b8b	fix: the import recovery sweep no longer fails queued files that are waiting their turn ... The orphan sweep in recover_interrupted_tasks failed every 'queued' row older than 30 minutes. A queued row has its broker message, so in a batch bigger than half an hour of work the sweep failed the whole tail: a library import of 8,135 files lost 3,114 that way. Never-enqueued 'pending' rows are still swept after 30 minutes; a 'queued' row is swept only when the broker's import queue is empty, and is kept when the queue can't be read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 16:28:04 -05:00
bvandeusenandClaude Opus 5.5	4477d7d14f	feat: Settings can point backups at an off-site S3 bucket, with a Test destination check ... Settings → Maintenance → Storage gains "Off-site backup destination": the address (with its port), bucket, region, access key and secret key of an S3 bucket such as Fabled Archives. The secret key is sealed with the credential key, never returned by the API, and kept when the field is left blank. Test destination lists the bucket with the fields as typed and writes nothing, and names the cause when it fails ("nothing is listening there (check the port)", "the secret key is wrong", "answered, but not as S3"). services/backup_s3 is a standard-library SigV4 client (put, get, paged list) ported from Grimoire's; it is held to AWS's published signing examples and exercised against a local HTTP server. Migration 0118 adds the columns; no destination is set up until someone saves one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 16:04:57 -05:00
bvandeusenandClaude Opus 5.5	315399c4e5	fix: an unfinished download is never stored as an attachment, and the temp sweep keeps any file a library row points at ... The importer refuses a staging file (.part, .partial, .megatmp.*) at both entry points, the same way it refuses a sidecar .json, and discard_after_attach never deletes one: the age-gated temp sweep owns them. The sweep checks its stale candidates against image and attachment rows and keeps any a row points at. A Google Drive fetch's half-written file had been stored as <sha>.part; the sweep then deleted it by name and left the row with no file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 15:48:14 -05:00
bvandeusenandClaude Opus 5.5	2fc71cfe4e	feat: Cleanup can import files that are on disk but not in the library ... A download writes its files into the artist's folder and imports them in a later step. A run that stopped in between left complete files no record points at, and a walk that had already marked them seen never returned for them; on the live instance that is about 95 GB of art. "Import files the library is missing" (Cleanup, Duplicates & leftovers) walks every folder named for an artist, matched by slug so a differently cased folder still matches, and lists the images, videos and archives no image, attachment or queued import points at. Stores, sidecars, staging files and files changed in the last day are left alone. Preview shows the counts per artist; Import queues the same list as a library import batch, whose tasks file the folder's post record, attach each file where it lies with its folder's artist and source, and remove the ones the library already holds or the import filters reject. A file whose import crashed is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:31:37 -05:00
bvandeusenandClaude Opus 5.5	2bea4b0d31	fix: a Mega folder link's files in subfolders are imported, and a retry no longer deletes complete files ... _fetch_mega listed only the top level of the link dir, so a folder link's files in subfolders never reached the importer: the link either failed with "wrote no new file" or read downloaded with its subfolders left on disk. It now walks the tree. The worker cleared every unrecorded file in a link dir before each fetch and after a failed one. Combined with the daily recovery of dead links, that deletes complete downloads whose upstream is gone, which may be the only copy. Every fetcher stages to .part or .megatmp.* and renames on completion, so the clear now removes staging files only, and a successful fetch imports every complete unrecorded file in the link dir, including ones an earlier attempt finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 10:24:14 -05:00
bvandeusenandClaude Opus 5.5	5259caa846	fix: dead external links are retried sparingly, not three times a day ... The daily recovery reset every dead link to a fresh three-attempt budget, so a link that can never succeed (an archive too big to import inside the fetch's hard limit, a file made private) was downloaded up to three times a day, every day. Recovery now gives a dead link one more attempt once it has been dead for 2**recoveries days (1, 2, 4, 8, 16), and stops after five. A recovered link that fails again goes straight back to dead. Retention prunes only links recovery has given up on, 30 days after their last attempt. A manual refetch of a post restores the full budgets. Migration 0118 adds external_link.recoveries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 19:13:46 -05:00
bvandeusenandClaude Opus 5.5	157f2803c0	fix: downloads stop leaving files on disk that the library doesn't use ... - An archive whose images were extracted was kept at its download path as well as in the attachments store. Both download paths now remove the downloaded copy. - A downloaded file the import filters reject (too small, transparent, single colour, unreadable) was left on disk with no record. A later walk found it, imported it again and rejected it again, every time. Rejected files are now removed; the seen ledger keeps them from being downloaded again. - Both paths share one policy, Importer.discard_after_attach, which never removes a file an ImageRecord points at. - An external-link fetch that fails, or whose worker was killed, left partial files (megatools' .megatmp.*) in the link's dir, and a killed fetch left its link 'downloading' for good. A fetch now clears unrecorded leftovers before starting and after failing. The sweep returns a link to retry once it has been 'downloading' past the task's hard limit, using a new external_link.started_at column (migration 0117). - The orphaned temp-file sweep now recognises .megatmp.* as well as .part and .partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:00:31 -05:00
bvandeusenandClaude Opus 5.5	31c94d6beb	ci: the weekly refresh reuses main's signed extension; worker tests pin the machine ... Two more reasons CI could not go green, found while fixing #6864. 1. sign-extension's "never go backwards" guard refused every weekly base refresh since at least 2026-10-04 (runs 8023, 9015, 9400, 9818). The refresh builds main, whose extension version (2026.9.25.1201) trails the highest one dev has signed since (2026.9.28.1403), so the guard exited before the cache lookup that would have found main's version already signed and reused it. The web image (:latest) was never refreshed. A version that is already signed now passes the guard: it is reused from its release, not signed again, so it cannot be a downgrade. 2. tests/test_api_workers.py read each lane's ceiling from the CI container's CPU quota and memory limit. Since run 10242 (2026-11-05) the runner's container allows fewer slots than two tests set (6 worker slots, 2 ML), so both got 400 "above what this container can hold". An autouse fixture now pins a 16-core, 64 GiB container, as test_worker_lanes.py already does. Task #6864. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 10:39:32 -05:00
bvandeusenandClaude Opus 5.5	e92497cff3	test: set Celery config through a celery_conf fixture, not monkeypatch ... pytest 9.2.0 (released 2026-10-27, changelog #10644) changed how monkeypatch.setattr restores an attribute the object does not hold in its own __dict__: undo now deletes it instead of assigning the old value back. Celery's conf serves every key through __getattr__, so undo called delattr, the delete raised AttributeError, and pytest skipped the rest of that test's undos. That made 53 teardown errors across six files (cleanup API, task-run signals, tasks admin, backup, library audit, the soft-limit salvage test), and could leak patches into later tests. CI installs pytest unpinned, so the first run after the release picked it up: the weekly base refresh of main, 2026-11-01, and every run since. The gate held, so nothing was published. tests/conftest.py gains `celery_conf(**values)`: it records each key's value through conf[key], writes with conf.update(), and restores the saved values after the test. The six files use it. Task #6864. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 10:34:14 -05:00
bvandeusenandClaude Opus 5.5	3f5224800e	fix: phase 3 stops importing at a deadline and leaves the rest for the next chunk ... A Discord recapture chunk on #banana-land was killed at the 1350s soft limit twice (events 92354 on 2026-09-28 and 92623 on 2026-09-29). The walk's budget (CHUNK_TOTAL_SECONDS, #4407) assumed 2.5s per import; measured on those chunks, a real import of the source's doodles took about 5s, so ~290 imports the walk had planned to fit took about twice the time reserved. - PHASE3_DEADLINE_SECONDS (1200s from task start): on native platforms phase 3 starts no import past it. The files left stay on disk with no record and are not marked seen; the next chunk resumes on the page that holds the first of them and imports them through the existing "on disk but never imported" path. A walk that reached its end becomes PARTIAL so the backfill is not marked complete while files wait. - DownloadResult.written_cursors records each written file's page cursor; mark_seen_after_import takes skip_paths. - PHASE3_IMPORT_SECONDS 2.5 -> 5.0, the slow end of what was measured, so the walk rarely leans on the deadline. Task #4493. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 09:53:57 -05:00
bvandeusenandClaude Opus 5.5	0928d2f914	chore: the poster-picker comments name their task #4491 ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 10:03:06 -04:00
bvandeusenandClaude Opus 5.5	ef591edf10	feat: the extension picks a Discord source's posters from who posts in the channel, stored by id (#4488) ... On a Discord channel page the Add panel lists who posted in the newest 200 messages, the creator the server is named for (or owned by) ticked. On a channel FabledCurator already follows, the chip opens the same list with Save and Open artist. The list is kept as user ids, so a rename never quietly stops it matching; the name each was picked under rides beside it for display, and the Subscriptions dialog shows it. - DiscordClient.recent_posters + rank_posters (owner / name-matches-server; bots never suggested; it suggests, the operator ticks). - GET/POST /api/extension/discord/posters; quick-add takes discord_authors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 09:47:27 -04:00
bvandeusenandClaude Opus 5.5	1efece0f21	feat: a Discord source can remove the posts it took from people outside its poster list (#4486) ... "Remove posts from other posters" on a Discord source with an "Only posts by" list previews what goes, per poster, then deletes it behind a typed token: - posts whose record names a poster not on the list (by id, username or display name); posts that record no poster are left alone and counted; - images found only on those posts; one also on a kept post stays, and a synthetic drop's link never keeps one; - their attachments, and every drop that absorbed one of them, so the grouping sweep regroups what remains. Preview and apply share one predicate, and a parity test holds them to it. The post record now also saves the poster's display name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 08:34:06 -04:00
bvandeusenandClaude Opus 5.5	d6d3184361	fix: the single-color filter no longer takes line art for a blank image (#4483) ... Five of Todding's Discord doodles (pencil lines on white, 3000px) were skipped on import as "single color", so their posts showed text and no image. The predicate sampled a 64px BILINEAR thumbnail, which blends thin strokes into the paper, and 0.95 "one color" is below how white a doodle is. - Sample 256x256 by NEAREST, so each sample is a real pixel. - Default threshold 0.995: blank means essentially blank. Migration 0116 moves a stored 0.95 (the old default) with it; the settings slider now spans 0.9-1 in 0.005 steps so the value is reachable. - The Cleanup audit shares the predicate, so it stops flagging sketches too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 08:12:43 -04:00
bvandeusenandClaude Opus 5.5	29c22afcb7	fix: a Discord source takes only messages with an image attached, from the posters it names, and knows its channel's name (#4481) ... - A message with no image or video attachment is chat: extract_media returns nothing for it, so nothing downloads and no post record is written. A message that passes keeps every file, numbered as gallery-dl numbers them. - `discord_authors` in a source's config limits the walk to those posters (id, username or display name); the edit dialog has a field for it and no longer drops config keys it has no field for. - source.display_name (migration 0115), refreshed by every Discord walk, shows on Subscriptions in place of the two-id URL; a Discord post card names its channel from the record's `channel`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-28 07:03:47 -04:00
bvandeusenandClaude Opus 5.5	42a60e3e74	test: the patreon ingester tests run phase 3's seen-marking before counting post keys (#4436) ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 11:41:16 -04:00
bvandeusenandClaude Opus 5.5	621c2b8315	fix: a post record is marked seen only after it is upserted, and an hourly sweep dates the posts killed walks left undated (#4436) ... ingest_core marked a post's record key seen when it wrote _post.json, but the record reaches the database (and dates the post and its images) only in phase 3. A walk killed before phase 3 left posts the ledger called recorded and the database never dated; ticks early-out long before reaching them again. Keys now join the media in mark_seen_after_import. date_posts_from_records finds each undated native post's record under its artist's folder and upserts it with the post's own source. Hourly on maintenance_long; an empty query once everything is dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 11:36:17 -04:00
bvandeusenandClaude Opus 5.5	ca802bd885	fix: a non-image file lands on the source it was downloaded for, and the undated shells it left are folded into their real posts (#4435) ... The attachment path looked the post's source up by (artist, platform), taking the artist's lowest-id source. A Discord artist has one source per channel, so every archive or pdf from a later channel became an undated second post on the first channel's source. _post_for_sidecar now takes the downloading source, as upsert_post_record and _apply_sidecar already did. Migration 0114 re-points each shell's attachments to its dated twin and deletes the shell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:56:08 -04:00
bvandeusenandClaude Opus 5.5	e704c70f32	fix: the download boot hook imports a model-only module, keeping the membership roster off the fetch path (#4433) ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:32:41 -04:00
bvandeusenandClaude Opus 5.5	0fad744bfb	fix: a restart no longer strands downloads — the download lane closes its predecessor's runs as interrupted and frees the platform locks at boot (#4433) ... A walk that outlives the 90s stop grace is SIGKILLed: its event never finalizes and its platform lock is held for the 27-min TTL. The 30-min sweep then errored every stranded event and bumped consecutive_failures, backing sources off (and blocking backfills) as if the platform failed. On worker_ready, the process consuming 'download' ends pre-boot pending/running events as skipped with error_type 'interrupted', leaves the source untouched so the next tick resumes it, and releases the serialized platforms' locks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:29:05 -04:00
bvandeusenandClaude Opus 5.5	32874ca678	fix: the release page drops internal rule numbers and the miscounted "three" images ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:09:47 -04:00
bvandeusenandClaude Opus 5.5	23e062dd4a	test: the stall-sweep tests read the thresholds they check, and stay clear of the new import value (#4432) ... Run 7505 failed test_recover_stalled_task_runs_ml_queue_uses_longer_threshold. It restated the old 25-minute ml threshold as a 30-minute "stale" row, which is now inside the 40-minute window. The test now reads the value from QUEUE_STUCK_THRESHOLD_MINUTES. The archive test's fast-import row was exactly 10 minutes old, which is the new import threshold, so it passed only by the milliseconds between seeding and sweeping. It is now 15 minutes old. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:51:42 -04:00
bvandeusenandClaude Opus 5.5	a360d69ee8	fix: task runs record the lane Celery really routes them to, and no healthy long job is swept as stalled (#4432) ... celery_signals._queue_for was a hand-kept copy of task_routes and had drifted: - backup, admin and library_audit jobs, and backfill_phash, run on maintenance_long but were recorded as `maintenance`; - translation and gpu_queue jobs were recorded as `default`, where the 5-minute stall sweep failed healthy 35-minute translation runs. It now asks the router, cached per task name. A new guard test checks every registered task's hard time limit against the stall threshold the sweep would use for it. It also caught these sweeps, which failed healthy runs mid-flight and are fixed here: - ml's scheduled sweeps (35 min, previously swept at 25); - train_heads and apply_head_tags (65 min); - import_media_file (6 min, previously swept at 5); - the long lane, which now has its own 45-minute threshold. UI changes: - the admin job poller follows a job by celery_task_id, via a new filter on /runs, instead of by lane; - the archive re-extract and missing-file repair cards show the long lane's backlog, where their jobs actually wait; - the queue table lists maintenance_long. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:46:39 -04:00
bvandeusenandClaude Opus 5.5	efcb548ebf	fix: natively downloaded images take their post's date, not their download time (#4431) ... The native ingesters (Discord, Patreon, SubscribeStar) import a post's media before its record. Only the record (`_post.json`) carries the date: the Discord message timestamp, or the Patreon/SubscribeStar published_at. So each image was linked to a post with no date yet, and it kept its download time in both gallery date columns. The post itself was dated correctly once the record landed, but nothing went back to update its images. - upsert_post_record now re-dates the images already linked to the post. `effective_date` becomes the primary post's date, and `earliest_post_date` the earliest dated post the image is in, which are the same rules _attach_provenance applies. - Migration 0113 repairs the library from the posts. It writes only rows that differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:30:17 -04:00
bvandeusenandClaude Opus 5.5	7b1570f2a5	feat: retire the sketch/doodle WIP title tier, and the review strip asks "Is this a WIP?" (milestone 430) ... The soft tier (#1474) tagged `wip` on any post titled sketch, doodle or scribble: 6,096 of the library's 8,876 wip tags. Its conflict audit flagged most of them, because finished art scores >= 0.5 on some content head, which filled the Gallery strip with 2,086 cards. A "sketch" is usually finished work, so the operator retired the tier. The artist's own "WIP" / "work in progress" title rule and human wip tags stay. - Removed: the soft matcher, source and prefilter; the importer and backfill soft branches; soft_wip_conflict_audit with its task and beat; the wip_soft_title_tagging_enabled setting, API field and toggle; and wip_title_soft from _AUTO_SOURCES. - Migration 0112: a soft tag the operator confirmed, or kept from the strip, becomes `manual`. Every other soft tag is deleted, along with the open review cards whose tag is gone. The column is dropped. - Review strip (#4424): each card asks "Is this a WIP?" (or banner, and so on), and the buttons read "Is a WIP" / "Is not a WIP". The content tag it also scored on is shown as the reason it was flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 08:20:28 -04:00
bvandeusenandClaude Opus 5.5	83e1382812	fix: extension updates install the new build — no 12h-cached "latest" XPI, and the popup's Update opens FC's install page ... Operator, 2026-09-25: "the extension update trigger from inside the extension doesn't work and the manual update seems to not move it to the most recent version or at least mark it the most recent." - fabledcurator-latest.xpi was served with Quart's default `public, max-age=43200`: one URL whose bytes change every release, so a browser that had fetched it reinstalled the previous build for 12 hours (measured on the instance). It is now `no-cache` (the ETag keeps an unchanged file a 304); versioned XPIs are `immutable`. - The web Settings card installs/downloads the VERSIONED xpi_url, which can only ever be that build's bytes. - The popup's Update button did tabs.create() on the .xpi, which Firefox refuses (NS_ERROR_FAILURE on a 200: it only installs from a user click on a web page). It now opens FC's install card (/subscriptions?tab=settings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 08:01:02 -04:00
bvandeusenandClaude Opus 5.5	423275a1e5	feat: the artist picker on Patreon and SubscribeStar too — and the Patreon name is canon (milestone 429) ... Operator: "yes add the artist picker to patreon and subscribestar too. but generally we treat the patreon name as the canon" - Every add now goes through the panel. On Patreon/SubscribeStar it opens on the creator's display name (read only when the panel opens: probe?names=1), searches FC's artists with it, and auto-picks a match. An untouched URL handle sends no name, so the server still resolves it. - Patreon is canon: joining a Patreon source to an artist known by another name offers "Rename “x” to the Patreon name “X”", ticked by default. quick-add's use_platform_name renames server-side from the name it reads itself; name only, the slug never moves (#130); never to a URL handle when the name can't be read; ignored on SubscribeStar and Discord. - _platform_display_name returns None rather than the handle, bounded by the same 6s lookup budget as Discord's names. - panelDefaults / addRequest / renameOffer replace the Discord-only helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 07:50:48 -04:00
bvandeusenandClaude Opus 5.5	eeb9263125	feat: the Discord Add panel's artist field autocompletes against FabledCurator's artists (milestone 429) ... Operator, after the first live run: "I need the extension to offer an autofill search function so it's easier to match an entry with an existing artist." - The field searches as soon as the panel opens (the server name, usually) and on every keystroke; matches list under it, with ↑/↓, Enter/Tab to pick, Esc to close, and a last "+ New artist" row. - Inline autofill: the rest of the top match is filled in and selected, so typing on replaces it and Tab/Enter accepts it. Backspacing never refills. - A result whose name IS the text, spacing and case aside, is picked on its own; the hint says in green which existing artist the source will join. - /api/artists/autocomplete also matches ignoring spacing and punctuation ("Tamada Heijun" finds "TamadaHeijun"), ranked just below an exact match. The web UI's artist search gets it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 07:45:51 -04:00
bvandeusenandClaude Opus 5.5	4c75dd0f88	feat: the extension adds Discord channels to an artist you pick, and its tests gate the XPI (milestone 429) ... Server (#4420) - extension_service gains a Discord pattern (server or channel, jump links, ptb/canary; not DMs or threads), mirrored in platforms.js and pinned by the shared artist-url-samples.json. - probe on a Discord URL matches the source by ids under any artist, reports a whole-server source as covering the channel, suggests the artist who owns another source on the same server, and names server/channel via the stored token (best-effort, bounded, no rate-limit waits). - quick-add takes artist_id / artist_name; Discord URLs are stored canonical. Extension (#4421, #4422) - Content script on discord.com; SPA navigation by URL polling (the old pushState patch ran in the isolated world and never fired); stale probes are dropped. - Discord chip opens an Add panel: this channel or the whole server, and the suggested artist / a search / a new name. - Popup: sources show artist, platform and state; a Discord token export is verified by FC and the result shown. Token capture covers ptb/canary. - Pure logic in lib/chip.js and lib/popup-format.js, with specs. CI (#4423) - extension.yml's lane (web-ext lint, vitest, XPI contents) moves into build.yml as extension-test and joins the needs of sign-extension, build-web and build-agent. As a separate workflow it gated nothing: a red extension suite still signed and shipped the XPI (rule 177). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 23:31:38 -04:00
bvandeusenandClaude Opus 5.5	e5bdcd2596	feat: finish the Discord switchover — recapture on every native source, backfills that run, gallery-dl's Discord config retired (milestone 428) ... - Recover and Recapture show on every native source. The menu gated them on a copied platform list ('patreon', 'subscribestar') that went stale when Discord moved over. Sources now carry `native_ingester` from the backend's own predicate. - A running backfill is due on every scheduler tick. Nothing queued a backfill's next chunk: each one waited for the source's regular interval, so an armed backfill sat idle until the next check (8h at the default) and a five-chunk walk took most of two days. The in-flight guard and the platform lock keep one chunk at a time. A failing source falls back to its backoff, and a stalled or out-of-budget walk stops being due. It also runs when the artist has auto-check off, since the operator started it by hand. - gallery-dl no longer carries Discord: its naming constants, platform defaults, sidecar-mirroring postprocessor and token injection are gone. The naming test moves to the native downloader and still renders against the real gallery-dl sidecar fixture. That is the guard that the files gallery-dl wrote are found on disk rather than fetched again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 23:14:48 -04:00
bvandeusenandClaude Opus 5.5	84e5448941	feat: Discord on the native core ingester — client, downloader, ledgers, wiring (milestone 428) ... Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is <channel>:<before>. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:42:00 -04:00
bvandeusenandClaude Opus 5.5	058fa85606	fix: backfill_phash runs on the long maintenance lane, not the scheduler's quick one (4411) ... A whole-library rehash with a 35-minute limit was matched by the maintenance.* glob. It held a scheduler process for its whole run, and the minute ticks queued behind it. An exact-name route now sends it to maintenance_long. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:26:48 -04:00
bvandeusenandClaude Opus 5.5	cc53d8db7b	feat: a GPU agent on the CPU shows as degraded, in the System view and on its own page (4410) ... torch and onnxruntime both fall back to the CPU without raising, so the agent that ran CPU-bound for weeks after a driver update leased and checked in like a healthy one. - The agent sends its startup accel report on every lease and heartbeat. - The server keeps a bounded copy on the roster row. A running agent with a runtime off the GPU becomes `degraded`, with a sentence naming the runtime and the reason. - The top nav shows it amber. - The agent page carries a banner, and its pill reads "CPU only". Also: the bandwidth field gets the page's − / + stepper, and both number fields drop the browser's spin arrows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:09:07 -04:00
bvandeusenandClaude Opus 5.5	42a40d71a4	fix: the agent's ONNX check asks CUDA for a device instead of trusting that the libraries loaded (1451) ... It reported "onnx on GPU" beside torch failing cuInit with "CUDA unknown error". Every library resolved, but no device could be used. The check now calls cudaGetDeviceCount and reports the CUDA error when there is one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 18:28:26 -04:00
bvandeusenandClaude Opus 5.5	f127c50207	fix: the lane sizer reads the live pool, so it stops growing a lane past its cap (4409) ... It read celery's `max-concurrency` as the pool size, but prefork reports that as the limit the pool booted with. pool_grow and pool_shrink never update it. A lane booted at 1 therefore read 1 forever. The sweep sent `target - 1` on every tick while work waited, and could never shrink, since 1 - 1 is 0. The System tab showed Scheduler "6 / 1" at a cap of 2. pool_size() counts `processes` and falls back to the limit only when there is no process list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 18:21:11 -04:00
bvandeusenandClaude Opus 5.5	7a09dc3cda	build: the agent installs one CUDA-13 stack instead of two, the web image drops ML packages it never imported, and Redis moves to 8 (1451, 1452) ... Agent: - The image ran PyPI's CUDA-13 torch 2.14 and onnxruntime-gpu 1.30 on a CUDA 12.9 cudnn-runtime base. requirements.txt had silently replaced the Dockerfile's torch 2.6+cu124, because ultralytics pulls torchvision, which pulls its own torch. That left ~3 GB of base libraries and a ~3 GB torch nothing loaded: 10 GB compressed. - Now: an nvidia/cuda 13.0.3 `base` image, with torch and torchvision installed together from cu130. CUDA and cuDNN come from the nvidia-* pip packages; onnxruntime-gpu declares its [cuda,cudnn] extras. - fc_agent/accel.py preloads those libraries for onnxruntime. It then logs, and reports in /status, whether torch and the ONNX CUDA provider actually got the GPU, since both fall back to the CPU silently. Web image: - Drop opencv-python-headless and onnxruntime, plus the opencv-only apt libs. Both have been listed since the scaffold and nothing in backend/ imports them. - torch/torchvision move to 2.14/0.29, and the unexplained caps are lifted (rule 154). Redis: 8-alpine in both compose files and both CI service containers. That gives an AGPLv3 licence option, where 7.4 was RSAL/SSPL only. The client moves to >=8.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 17:45:39 -04:00
bvandeusenandClaude Opus 5.5	ff70f837d0	refactor: test row factories and the fetch stub have one copy each (3109) ... tests/factories.py holds image_row/make_image/make_image_async/make_tag. The 17 byte-identical _img/_tag helpers (15 modules) now import them under their old names, so no call site changed. frontend/test/support/stubFetch.js replaces 15 copies that differed only in formatting. Copies whose bodies differ (other defaults, other columns, a url-only stub) are left as they are; folding those needs a look at each caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:39:38 -04:00
bvandeusenandClaude Opus 5.5	f2e4ee4d17	fix: beat takes each job's last run from task_run, so a redeploy no longer resets the schedule (4408) ... Beat's default scheduler kept its memory in a shelve file nothing persists, and a scheduler that remembers nothing waits a full interval before any job. Since the one-container image, every redeploy restarts beat, and no daily or weekly job had run since 2026-09-21 (cleanup, backup and download-event pruning, membership sync, thumbnail backfill, integrity check, vacuum). TaskRunScheduler seeds each entry's last_run_at at startup from task_run's newest start for that task: an overdue job runs at once, one not yet due waits the remainder, and one never recorded is due now. prune_task_runs now keeps each task's newest row however old, or a weekly job would look never-run a day after it ran and fire on every restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:36:58 -04:00
bvandeusenandClaude Opus 5.5	31020d9395	fix: a native chunk stops walking when its import work would overrun the task ... The walk's time budget covered the walk alone, but phase 3 runs in the same Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked for about two minutes and handed phase 3 431 orphan imports and ~3000 relinks. Phase 3 ran for 20 minutes and died at the soft limit (event 90808). This predates the worker consolidation; the limits are unchanged since June. The walk now also stops when elapsed time plus phase 3's estimated cost (2.5s per import, 0.25s per relink, measured on the live instance) passes CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so such a stop is a PARTIAL chunk boundary and the next chunk resumes the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 16:13:47 -04:00
bvandeusenandClaude Opus 5.5	7cc34b5724	fix: a Discord message that re-posts a file is grouped, and joins the drop it repeats ... Grouping read a message's images through primary_post_id alone, which only the first message imported with a file holds. The backfill runs newest-first, so the original message usually owned nothing: 101 of Yellowroom's messages (mostly 2018-2020) could never be grouped. Every image they carried also sat in another message (296 links, measured on the live instance). _message_images unions ownership with provenance for the candidate query, the drop seed, the member image links, and the merge pass. Nothing is re-owned. A later drop carrying the very file an earlier one carries now merges by a new same_image route, which nearest-neighbour could not see because it skips the image itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 15:21:05 -04:00
bvandeusenandClaude Opus 5.5	3aa899ad29	test: revisit tests' first walk completes phase 3 before the second ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:16:07 -04:00
bvandeusenandClaude Opus 5.5	da2091a875	fix: a download is marked seen only after it is imported ... A run killed between download and import left its files on disk and in the seen-ledger but never in the library, and every later walk trusted the ledger. TamadaHeijun's 12PCG post lost 7 of 13 images this way (stranded run 90402), which read as the duplicates filter. The ingester now hands phase 3 a mark_seen_after_import hook, called after the import loop. A file on disk with no ImageRecord at its path is fed to import, not reconciled into the ledger. Recapture reaches files already orphaned, because it looks past the ledger to the disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:13:12 -04:00
bvandeusenandClaude Opus 5.5	2f9e35390e	fix: the matcher sees a drop's names, finds merged trickles, and catches up on its own (4390, 4392) ... Three defects found while answering "is the linking automatic": 1. The name and image checks never worked on a live drop. The corpus keyed every image by primary_post_id, but a drop's images belong to its member messages and the drop claims them only through provenance, so a drop looked nameless and hashless. The tests attached images to the drop itself, which discord_grouping never does. Images now count under the post a reader sees them on: a message's absorbing drop, else the post itself. 2. The trickle merge (e08401c) dates a drop by its first stage, days before the release a teaser announces, which put merged trickles outside the 24h window. Candidates are now found and timed by their closest member message. The sweep follows recently grown drops by their messages' times the same way. 3. A pair left pending was skipped forever: the matcher skipped every recorded pair, not only decided ones. Pending pairs are now re-scored in place and linked once conclusive. Linked and dismissed pairs are still never touched. Also, "Scan now" shared the sweep's 48-hour horizon, so it could not reach the history it is described as being for. It now scores every post by an artist with Discord drops (rescan(full=True)). New tests build drops the way the grouper does, with images owned by the member messages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 14:01:07 -04:00