116 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	dc8af8b1a7	feat: a learned roster, so a stopped part is observable (milestone 365 steps 1-2) ... Nothing in FabledCurator knew what was SUPPOSED to be running. `celery inspect` reports the workers that ANSWER, so a dead worker was a shorter list rather than a red light, and grep for any notion of expected services returned nothing. That is why Portainer was the only place an operator could see it: Portainer knows the intended set. `service_seen` is the memory that makes an absence observable — every part that has checked in, and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy. Keyed on those, this table would record a death and a birth every time the stack updates — and a status page that goes red on every deploy is a status page nobody reads, which is worse than not having one. CELERY_QUEUES is assigned per role in compose and survives container replacement, so it is the stable identity. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served, not how many containers exist. The GPU agent is keyed on agent_id, the identity its lease protocol already uses. gpu.py received it on both lease and heartbeat and threw it away — an idle agent with nothing to lease left no trace and was indistinguishable from one switched off a week ago. Now recorded on the calls that were already happening. **Who observes, corrected from the plan.** The plan said "record from the existing inspect path", which would only run when someone opened the Activity tab. Two other candidates and why they lost: - A beat sweep. If the scheduler dies the sweep stops, every row goes stale, and the page says everything is down when one thing is. An alarm that cannot distinguish "a part died" from "the observer died" is worse than none. - A background task in web. hypercorn runs --workers 4, so that is four concurrent inspect loops per container, forever. Taken instead: refresh on demand, rate-limited by the newest last_seen_at that every process can already see. The observer is then the thing serving the page — if web is down you get a browser error, not a confidently green page — and it self-limits with no coordination, since a race costs one redundant inspect that writes identical values. Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps FORWARD from 0089 rather than merely reproducing the schema. No secondary indexes: one row per moving part means every read is a handful of rows, and #3301 is the record of what speculative indexes cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 17:15:46 -04:00
bvandeusenandClaude Opus 5	aa71cbbdbf	db: the baseline was missing the three system-tag seeds (#3266) ... Integration caught it: 36 tests failing with NoResultFound, all on _system_tag(db, "banner") and its siblings. 0075 seeds three hygiene system tags — wip, banner, editor screenshot — and the first version of the baseline carried only the two settings singletons. This is the same defect class the baseline's own docstring warns about, which I then walked into anyway. The reason is worth recording: my scan for data statements used a regex requiring INSERT to sit immediately after the opening quote, so it saw op.execute("INSERT INTO ml_settings (id) VALUES (1)") and missed 0075, which builds the statement through sa.text() across several lines with bound parameters. The narrow pattern found two of three seeds and reported itself complete. The wider scan — grep for insert/bulk_insert across every revision in 725bf15 — turns up six data-writing migrations, and they separate mechanically: INSERT ... VALUES (literals) = SEED. Product data. Carry it. 0002 import_settings, 0003 ml_settings, 0075 system tags INSERT ... SELECT ... FROM tbl = BACKFILL. Derives from existing rows, inserts nothing on an empty database, correctly omitted. 0034 artist_visit, 0040 and 0047 series_chapter That rule is now in the docstring, because the next person collapsing a chain needs the rule more than they need the answer. 0075's adopt-before-insert guard is kept as WHERE NOT EXISTS. It cannot fire on the empty database this file runs against — it existed because an operator might already have hand-tagged `wip` — but it makes the statement re-runnable for free. Worth stating plainly: baseline.yml passed on the version without these rows, and would pass again. It compares schema, and a baseline missing every seed still produces a byte-identical schema. The integration suite is what caught this, which is the argument for the first-run check that #3271 should carry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-01 01:20:42 -04:00
bvandeusenandClaude Opus 5	973db73221	db: collapse alembic 0001..0089 into one baseline (#3266) ... 89 files and 6,300 lines become one file of 807. Nothing about the resulting schema changes; what goes away is the requirement that a new installation replay our development history to arrive at it. revision = "0089", down_revision = None. That pairing IS the migration strategy for existing installs, not a detail of it: a deployed database already has alembic_version = '0089' from running the real 0089, so alembic reads the version table, sees head reached, and does nothing. No stamp is required — which matters, because `alembic stamp` writes a version string without validating anything about the schema it is writing it against, and a wrong stamp is indistinguishable from a right one until the next migration fails. An empty database runs the file and records 0089. Both paths converge. The next migration is 0090, as it would have been; the numbering is continuous across the collapse on purpose. Autogenerate produced nearly all of this unaided, which was NOT true of the first attempt — that one was reverted because the generator silently dropped eleven indexes and three uniqueness guarantees. #3275 put those on the models first, so the HNSW index with its opclass, the COALESCE expression index, the partial uniques, 107 server_defaults and the enum CHECKs are all emitted now. Doing the reconciliation before the squash, rather than after, is what made this work. Hand-added, because none of it can live in a model: * CREATE EXTENSION vector / tsm_system_rows (0001, 0004) — database objects, not table metadata. * The pgvector import. Autogenerate writes qualified pgvector.sqlalchemy.vector.VECTOR references without importing the package, so its own output cannot run (run 4988). * THE TWO SEED ROWS. 0002 and 0003 did not only build schema — each inserted a settings singleton, and nothing in the app ever creates them: ImportSettings.load() and MLSettings.load() are select(...).scalar_one(), which RAISES NoResultFound rather than returning None. A models-only baseline would leave both tables empty and crash a fresh install on first settings access, while baseline.yml reported a perfect schema match. Only running the app against a new database finds that. Not carried over: 0023's DELETE FROM tag and 0047's series deletes, which are historical cleanups operating on rows an empty database lacks. downgrade() raises. A baseline's downgrade is "drop every table", which is a data-loss event wearing a migration as a disguise; offering it as one invites someone to run it. Restore from a backup. Also removed, per the plan: the 10 test_migration_*.py files (they assert intermediate states and backfills that no longer exist — a test that a column exists is already the model tests' job) and backend/app/utils/artist_backfill.py, whose only importer was 0008. Verified no other consumer anywhere in backend/ or tests/. baseline.yml changes with it. chain_ref now DEFAULTS to 725bf15, since the tree no longer carries a chain to compare against — that pinned commit is the last one that does. And the CheckConstraint repair is removed, because it never fired. I added it claiming autogenerate re-doubles a constraint name on the round trip and asserted in b979062 that it was "still correct and still needed". It is not: autogenerate wraps names in op.f(), which marks them already-formatted and blocks the convention from re-applying. Tested against the real candidate line — the regex matches nothing. What actually fixed the mismatch was 0088's renames alone. The doubling is a real hazard, but of hand-writing a pre-prefixed name, not of the generator; the comment asserting otherwise was worse than the dead code under it. The header comment is rewritten for the same reason — it described 87 revisions, and claimed the HNSW index could not be expressed in a model, which #3275 disproved. It now also states plainly what this check CANNOT see: it compares schema, so a green run means the schema is right, not that the baseline is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-01 01:14:37 -04:00
bvandeusenandClaude Opus 5	08418d54a3	db: index the seven unindexed FKs, drop the seven redundant ones (#3300, #3301) ... A structural sweep of the deployed schema, run AFTER 0088 got the models and the chain to exact agreement. That agreement is what 0088 achieved, and it is worth naming what it does not prove: a models-vs-chain diff shows the two describe the same schema, not that the schema is right. Everything here was wrong in BOTH. The one that matters: image_tag has PRIMARY KEY (image_record_id, tag_id) and no other index, so tag_id is unindexed. That is the gallery's tag filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the ON DELETE CASCADE from tag, both scanning the largest table in the schema. Six more FKs were unindexed on smaller tables; presentation_review.tag_id also CASCADEs. Dropped, on the other side: ix_image_record_sha256 was an exact duplicate of the index uq_image_record_sha256 already builds — two btrees on the same column of the highest-insert-rate table. The other six are single-column indexes a later composite superseded without the narrow one being retired; a btree on (a,b) already serves lookups on a. 0088 deliberately taught the models to declare BOTH sha256 indexes so they would describe reality. This changes the reality instead, and the models change with it — otherwise the next baseline.yml run reintroduces exactly the drift 0088 removed. CONCURRENTLY throughout, so building the image_tag index does not hold an ACCESS EXCLUSIVE lock over every write for the duration. The cost is that the migration cannot run in a transaction and so is not atomic: every statement is IF NOT EXISTS / IF EXISTS, making a re-run after a partial failure safe. The docstring carries the query for finding an INVALID index left by an interrupted CONCURRENTLY build. What the sweep found clean, for the record: all 43 tables have a primary key; all 51 FKs declare an explicit ON DELETE, so none silently blocks a delete; the three enum CHECKs match the code that writes them (rule 36). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 08:25:41 -04:00
bvandeusenandClaude Opus 5	389afe2f7b	db: the doubled CHECK list was six, not four (#3275) ... Run 5029 confirmed the four renames landed and surfaced two I had missed: external_link's host and status CHECKs are doubled the same way. They did not show in run 5026's diff because BOTH sides produced the doubled form back then — external_link.py pre-prefixed its names, so the models matched the chain's mistake. Switching all six models to bare names is what exposed the two the migration did not cover. The list in the file now comes from matching ck_(\w+?)_ck_\1_ against the chain's own pg_dump, rather than from reading migrations by eye. Reading by eye is what missed these, in the same way it earlier missed a UNIQUE constraint sitting two lines above the index being looked at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:33:50 -04:00
bvandeusenandClaude Opus 5	b979062dd7	db: rename the four double-prefixed CHECK constraints (#3275) ... Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one of them reverses an assumption I made in the previous commit. The doubled CHECK names are what the DATABASE has, not what the generator invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s, which — unlike uq/fk/ix — applies even to a constraint that already has a name, so four migrations that passed an already-prefixed name got it prefixed twice: ck_import_settings_ck_import_settings_singleton ck_ml_settings_ck_ml_settings_singleton ck_post_ck_post_translation_override ck_tag_ck_tag_fandom_requires_character The workflow repair added last commit is still correct and still needed — autogenerate really does re-double a name on the round trip — but it was making the MODELS side clean against a chain that is dirty. The comment in ml_settings.py claiming its bare name "matches migration 0003" was simply false; 0003 produces the doubled form. Nothing reads a CHECK constraint by name, so this has never done harm. But it is precisely the development-era residue the collapsed baseline exists to leave behind, and a public schema should not ship it — so 0088 renames the deployed constraints and all six models now declare bare names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no revalidation, which is why this is safe on post and tag. Guarded on pg_constraint scoped by conrelid, so it is a no-op on a database built from the models. ix_tag_fandom_id showed as a difference only because chain_ref was pinned to 0a5bbe8, which predates 0088 — the comparison was measuring the models against a chain missing the migration that closes the gap. chain_ref now defaults to blank, meaning "the chain in this ref". Pin it to a commit only after the collapse, when the tree no longer carries the revisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:29:57 -04:00
bvandeusenandClaude Opus 5	573228b9da	db: finish reconciling the models with the deployed schema (#3275) ... Closes the residue the first reconciliation pass left, and corrects a factual error I put into the record. sha256 was NOT missing a uniqueness guarantee. I read `op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and concluded duplicates were possible, without reading line 149 two lines above it: sa.UniqueConstraint("sha256", name="uq_image_record_sha256"), Uniqueness has held since the initial schema. The database expresses it as a CONSTRAINT plus a separate non-unique lookup index; the model said `unique=True, index=True`, which is one UNIQUE index under a different name. Same guarantee, different objects — which is exactly why the two schemas did not line up. The model now declares both objects. No DDL. 0088's docstring, which repeated the claim, is corrected in place. Two real divergences, both the MODEL over-claiming: * source: uq_source_artist_platform_url (alembic 0010) was declared nowhere in the models — source.py had no __table_args__ at all — so autogenerate would have proposed DROPPING it. * head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it nullable. Left nullable; the FK already cascades. Seven constraints renamed to what the chain actually created, rather than what base.py's naming convention renders: uq_series_page_image, uq_series_chapter_anchor_page, fk_series_chapter_anchor_page, fk_image_record_artist_id, fk_image_provenance_from_attachment, and the two hand-shortened fk_tsr_* names from 0003. Float server_defaults now mirror their own migration, per column. The chain is MIXED: a plain string renders DEFAULT '0.90'::double precision, sa.text() renders DEFAULT 0.90, and the migrations used both. Seven columns take text(); the rest stay strings. Two literals also disagreed outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against the migration's 0.90/0.50. baseline.yml gains two things. A repair for a SECOND generator defect in the same class as the missing pgvector import: base.py's ck convention contains %(constraint_name)s, so it applies even to a NAMED CheckConstraint — autogenerate writes the already-rendered name into the migration and running it applies the convention again, yielding ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage, not a claim the models make, so it is undone rather than counted. And the diff now runs twice. Column ORDER differs permanently between a schema built by 87 ADD COLUMNs and one built in a single shot — the operator's database keeps chain order forever, a fresh install gets model order — so a check that failed on it could never pass. The second pass SORTS column lines within each CREATE TABLE instead of deleting them, which cannot hide a column present on one side only, or one whose type, nullability or default differs. Ordered diff is reported as information; the order-insensitive one is the verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:24:00 -04:00
bvandeusen	5e1996e77f	db: reconcile the models with the deployed schema (#3275) ... Milestone 328's acceptance test compared a database built by the real 0001..0087 chain against one built from the models, and found ~130 places where they disagree. This closes them. Almost all were the MODEL being wrong, so almost all of this is model edits with no DDL — the database already had these things, nothing in it changes, and no deploy is needed for this part: * 92 columns gained server_default. The models carried Python-side `default=` only, so the ORM filled the value and the column had no database default. Anything inserting outside the ORM behaved differently from production. * Eleven indexes that existed only in migrations are now declared: the three backup_run reporting indexes, the two date-ordered image_record browse indexes, import_task and presentation_review, and the three task_run history indexes. All use text() for their DESC ordering and postgresql_where for the partial one. * Two UNIQUE indexes that autogenerate silently proposed DROPPING, because neither is expressible as a UniqueConstraint: uq_tag_name_kind_fandom — an EXPRESSION index over (name, kind, COALESCE(fandom_id, 0)) uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id IS NULL post.py already had a comment describing the second one. The comment was right; nothing declared it. * The two external_link enum CHECKs (host, status) — rule 36 territory, and absent from the model entirely. * Two indexes were named explicitly. A bare index=True generated ix_tag_alias_canonical_tag_id where the database has ix_tag_alias_canonical, so autogenerate proposed a drop+create of an index that was already there under another name. Same for tag_suggestion_rejection. Only ONE thing needed DDL, as 0088: tag.fandom_id is declared index=True but no migration ever created that index. Deliberately NOT here: image_record.sha256. The model says unique=True; 0001 created a plain index. Duplicates are possible today and the ORM believes otherwise. The fix depends on whether duplicates already exist — if they do, that is a dedupe decision, not a constraint — so it waits on an answer about live data. The real severity of #3275 is not the squash. It is that --autogenerate has been unsafe on this project: run against the old models it would have proposed dropping eleven indexes and two uniqueness guarantees.	2026-08-30 14:42:30 -04:00
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusenandClaude Opus 5	516521e7b0	refactor(platforms): drop migration 0088 — no deviantart rows exist (#3069) ... Operator confirms the instance has never used DeviantArt, so there is nothing for 0088 to quiesce. The migration only ever had two jobs — disable leftover `source` rows and delete a stale `credential` row — and both were guards against data that does not exist here. Removing it rather than keeping a no-op: a migration that runs on every deploy to touch zero rows is a permanent cost paid for a hypothetical, and it would read to a future reader as evidence that DeviantArt sources once existed. `platform` has no CHECK constraint, so retiring the key needs no schema change of its own. alembic head returns to 0087. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:26:02 -04:00
bvandeusenandClaude Opus 5	ddf896078c	refactor(platforms): retire deviantart end-to-end (#3069) ... Executes the 2026-07-05 product decision (FC downloaders = art-dedicated services only), which removed Twitter/X and Bluesky but left deviantart fully wired for seven weeks — the half-retired state rule 22 exists to prevent. Removed: the PlatformInfo module and its registry entry, the gallery-dl extractor block, extension_service's artist-page pattern, the extension's PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission and content-script match, the frontend icon/colour/label, and the operator- facing "supported platforms" list that still advertised it. Two judgment calls, both recorded in migration 0088: * existing `source` rows are DISABLED, not deleted. The row is the only record of the artist's DeviantArt URL. Disabling is also required for correctness rather than tidiness: with the platform unregistered the download path falls through to gallery-dl, which carries its OWN deviantart extractor, so an enabled row would have kept downloading from a dropped platform. * the `credential` row IS deleted — a live session cookie for a site FC will never call again. Adds the invariant whose absence is why manifest.json drifted in the first place: nothing tied its domain lists back to the platform table. The extension suite now asserts both directions, plus that no host permission belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at an operator-chosen URL the extension cannot enumerate). Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged extension change without a bump. No release is cut — build.yml's sign-extension job only runs on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:24:08 -04:00
bvandeusenandClaude Opus 4.8	af0d39ed52	feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474) ... Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 10:14:38 -04:00
bvandeusenandClaude Opus 4.8	ad2a5fc5fe	feat(system-tags): process vs chrome groups + WIP provisional auto-apply (#1464) ... Backend for the system-tag behavior refactor (milestone #157). editor screenshot moves from chrome (hidden) to the PROCESS group (shown, like wip); wip+editor gain provisional auto-apply so they stop needing endless manual identification — without a runaway loop. - tag.py: split PRESENTATION_SYSTEM_TAGS → CHROME_SYSTEM_TAGS (banner) + PROCESS_SYSTEM_TAGS (wip, editor screenshot). - heads.py: generalize presentation_auto_apply_sweep → system_tag_auto_apply_sweep (mode chrome|process). Same Guard 1 (skip human/confirmed) + Guard 2 (ring-loud conflict → PresentationReview). process mode uses source 'process_auto' and does NOT hide (hide is a gallery-query effect of group membership). - training_data._AUTO_SOURCES += 'process_auto' → the head never trains on its own auto-applied output; only wip_title/manual train it (the runaway break). - ml_settings: process_auto_apply_enabled (OFF, opt-in) + threshold + conflict threshold. presentation_review.mode ('chrome'|'process'). Migration 0086. - gallery_service: default-hide reads CHROME only (editor now shows); Explore neighbors exclude the whole PROCESS group. - tasks/ml + celery beat: scheduled_process_auto_apply (daily, opt-in); prune covers both modes. - api: ml_admin process_* CRUD+validation; hidden-review returns mode. - tests: rename chrome sweep calls; new test_process_auto_apply (apply, guards, mode flag, no-self-train); gallery test asserts editor now visible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 23:15:59 -04:00
bvandeusenandClaude Opus 4.8	571938781a	feat(tagging): title-based WIP auto-tagging (#1458) ... Auto-apply the `wip` system tag to posts whose TITLE explicitly declares work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision complement to the image-based ML `wip` head. WIP images are excluded from the Explore/gallery browse, so honouring the artist's own label keeps unfinished pieces out of the main browse. - services/wip_title.py: precision-first token-anchored matcher (swipe/wiped never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO NOTHING, chunked under the psycopg param ceiling). - importer: live hook on FRESH import only (never on deep-scan/supersede), so a manually-removed WIP tag is never re-applied by a routine re-scan. - maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep (coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a beat — a periodic re-run would silently undo manual removals. - ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating the live hook; GET/PATCH + POST /settings/wip-title/scan. - Settings UI: toggle + "Scan existing posts" button. - Tests: pure matcher unit tests + integration (apply idempotency, backfill precision). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:12:19 -04:00
bvandeusenandClaude Opus 4.8	aea2701c28	feat(translation): tunable acceptance floor (0.90) + per-post sticky override (#155) ... The gate at a fixed 0.80 couldn't catch the real pain: Interpreter (fresh == cached, verified by probe) confidently mis-detects short ASCII English like "... WIP Part 1" as German at 0.86 — above the floor — so it was accepted and a re-translate reproduced it. Confidence alone can't separate the 0.86 collision (genuine German lands there too), and single-word mis-flags sit at a confident 1.0 no floor catches. Two operator-approved levers: - Acceptance floor is now a live Settings value (ImportSettings. translation_min_confidence, default 0.90; surfaced in the Translation card), so it's tunable without a redeploy. _accept takes the threshold as a parameter. - Per-post sticky override (Post.translation_override: auto/force/original). 'force' stores a translation even below the floor (rescue a skipped legit-foreign title); 'original' keeps the original and clears any stored translation (kill a confident mis-flag no floor catches). The sweep honors it on every run and _reset_translations skips 'original', so the choice survives a Re-translate-all. POST /api/posts/<id>/translation-override applies it immediately (translate now when the service is up, else queue for the sweep). UI: PostTranslationControl on the posts-feed card. Migration 0084 (both columns + a CHECK on the override). The feed + provenance serializers expose translation_override. With a stricter floor the rollback finally works: raise it -> Re-translate all -> the 0.86 mis-flags are rejected and restored to the original; force / keep-original handle the residual either way. Tests: gate thresholds against the param (0.86 rejected at 0.90, explicit-floor cases); sweep force/original + re-translate-skips-original; override endpoint (validation, original clears, force queues when disabled, feed exposes it); settings min_confidence default/save/validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-10 22:38:25 -04:00
bvandeusenandClaude Opus 4.8	a3bc98a53c	feat(translation): Post translation columns + settings + migration (#143 step 1) ... Post gains post_title_translated / description_translated / translated_source_lang / translation_engine_version / translated_at — filled by the translate sweep so viewing is instant. ImportSettings gains translation_enabled (OFF by default), interpreter_base_url (EMPTY — no default host; the operator points it at their own Interpreter proxy behind a reverse proxy) and translation_target_lang (en), exposed + validated via /settings/import. Migration 0083. Settings defaults + patch + validation test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 12:20:31 -04:00
bvandeusenandClaude Opus 4.8	ab63d94249	feat(ml): presentation auto-hide settings + review table (#141 step 3) ... MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) + presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a FLAT threshold (decoupled from content-head graduation), plus the "also looks like content" conflict cut. New presentation_review table (image, presentation tag, conflict tag + score, created/resolved_at) records auto-hides flagged for review. Migration 0082 (columns + table), ml_admin API (editable + get_settings + _validate bounds), settings roundtrip/bounds test. The sweep that reads these knobs + the Settings UI land in step 4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 22:59:00 -04:00
bvandeusenandClaude Opus 4.8	7d3a3b4a83	revert(ml): keep head auto-apply precision at 0.97 (operator: general tuning was fine) ... Milestone 139 raised head_auto_apply_precision 0.97→0.98; operator confirmed the general-tag confidence was already well tuned, so revert that. The support floor (min_positives 30→50) and CCIP match confidence (0.92→0.95) stay. Migration 0081 (not yet deployed) edited to drop the precision bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 18:42:08 -04:00
bvandeusenandClaude Opus 4.8	cbc3e11a53	feat(ml): stricter auto-apply defaults to cut misfires (milestone 139) ... head_auto_apply_precision 0.97→0.98, head_auto_apply_min_positives 30→50, ccip_auto_apply_threshold 0.92→0.95 (operator-asked). Model defaults change for fresh installs; migration 0081 bumps the existing singleton row IFF still at the old default (won't clobber a deliberate operator change). ml_admin bounds already permit these. Fixed a stale comment in the auto-apply test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 18:08:54 -04:00
bvandeusenandClaude Opus 4.8	2cfbb284d5	feat(heads): incremental retraining — refit only changed tags (#1317 phase 2, m138) ... train_all_heads is now incremental by default: a per-tag training-data fingerprint (positive + rejection count/latest-timestamp, stored on tag_head.train_fingerprint) means a manual Retrain refits ONLY the tags whose data changed — O(what you touched), not O(all heads). The nightly scheduled_train_heads passes full=True to reconcile sampled-negative + hygiene drift across every head. First incremental run after deploy still refits everyone (NULL fingerprints), stamping them, then it's incremental. The refit decision + fingerprint are split into sklearn-free helpers (_head_fingerprints, _heads_needing_retrain) so the incremental logic is unit-tested directly (train_head itself needs scikit-learn). Migration 0080. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 16:36:30 -04:00
bvandeusenandClaude Opus 4.8	f24dc81764	feat(ccip): schema for precomputed incremental character prototypes (#1317, m138 step 1) ... Foundation for making CCIP character references a precomputed, INCREMENTAL artifact instead of a request-path rebuild (kills the per-accept ~4s suggestions stall; cost will scale with change, not library size): - character_prototype: a character's reference CCIP vectors, capped to MLSettings.ccip_prototype_cap so match cost doesn't grow with popularity. - ccip_prototype_state: per-character fingerprint (ref count + max region id) + updated_at → drives per-character incremental rebuilds and the matcher cache's reload-only-what-advanced. - MLSettings.ccip_ref_signature (cheap global change gate) + ccip_prototype_cap. Migration 0079. Schema + models only — the builder service, refresh task/beat, and matcher rewrite land in the following steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 15:58:11 -04:00
bvandeusenandClaude Opus 4.8	62ec70b9e4	feat(ml): detector config in MLSettings with working defaults (#134 step 1) ... Move the crop-proposer config (per-proposer enable + weights + conf, caps, dedupe IoU) into the DB so it's UI-tunable and can be announced to the GPU agent in the lease (like the embedder model) — no restart, agent env becomes bootstrap-only. Migration 0078 adds the columns with working server_defaults so existing rows + fresh installs crop out-of-the-box with all three proposers ON (operator: default-on): person=yolo11n.pt, anatomy=booru_yolo yolov11m_aa22 (URL, license unstated/private-homelab-OK), panel=mosesb best.pt. Plain columns, no CHECK enum. Steps 2 (lease announce + agent apply) and 3 (Settings UI) follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-05 19:35:59 -04:00
bvandeusenandClaude Opus 4.8	87d53db0cb	feat(artist): editable display name + rename surface; drop name-uniqueness (#130 step 1) ... First step of decoupling artist identity/storage/display. migration 0077 drops uq_artist_name so the display name is free text (two genuinely different creators can share a name); the slug stays the immutable, unique storage/identity key (the on-disk path component — untouched, so nothing moves). ArtistService.rename + PATCH /api/artists/<id> change the name ONLY. Frontend: inline pencil-edit on the artist header (mirrors TagCard), slug/route unaffected so no navigation. Fixes the operator's 'no surface to rename an artist' + the name-collision fragility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 22:04:20 -04:00
bvandeusenandClaude Fable 5	0563b2d750	feat(pixiv): ledger models + migration 0076 + PixivIngester adapter (#129 step 3) ... pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira — pximg URLs carry no content hash). PixivIngester wires client/downloader/ ledgers into ingest_core with drift label 'Pixiv app API' and the new body_canary=False opt-out: caption-less pixiv artists are common, so the zero-bodies #862 alarm would false-positive here — the client's response-shape drift checks cover that failure class instead. auth_token joins the uniform adapter constructor (pixiv is the first token-auth native platform). verify_pixiv_credential = one OAuth refresh, no feed walk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-03 09:45:14 -04:00
bvandeusenandClaude Fable 5	e9891ee9f3	feat(tags): system tags — is_system column, seeded hygiene tags, protection guards ... Training hygiene step 1 (milestone #128). Migration 0075 adds tag.is_system and seeds wip / banner / editor screenshot (kind=general), ADOPTING an existing same-(name,kind) tag case-insensitively instead of duplicating. These rows drive the upcoming training exclusions, so they are protected: rename and merge-away refuse system tags (merge-INTO stays allowed — folding an operator's old hygiene tag into the system row is the intended move; merge is the only tag-delete path, so that guard covers deletion). is_system rides every tag serialization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 23:14:49 -04:00
bvandeusenandClaude Fable 5	19b962f1a7	feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch ... The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:53:08 -04:00
bvandeusenandClaude Fable 5	eaea4308fc	chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved) ... The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:41:24 -04:00
bvandeusenandClaude Fable 5	a7abcc41ca	feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3) ... An errored GPU job's stored reason is a suspicion; the file probe is the verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own probe (sha256 + decode) on each errored image ONCE and writes both verdicts: ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' | 'file_ok', migration 0072). Every classification logs at WARNING so it surfaces in Logs/System Activity. - 'defect' rows are excluded from /retry_errors (re-running a known-bad file burns agent time re-minting the tombstone); response now reports defects_kept and the GpuAgentCard toast says so. - GET /api/gpu/errors: triage view — reason buckets (classify_reason), probe verdicts, per-job detail. POST /errors/triage runs the sweep now. - POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern — delete the defective copy + record (full cascade takes the tombstones too) and re-poll its subscription Source so a fresh copy re-imports and re-enters the pipeline; 'no_source' when nothing pollable resolves. - New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts, reason summary, probe-now, defect list with thumbnails + per-image Recover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:36:02 -04:00
bvandeusenandClaude Opus 4.8	c22f37d64d	feat(gallery): sort by earliest post date across all posts (new default) ... The gallery's newest/oldest sort keys off image_record.effective_date = COALESCE(primary post's post_date, created_at). The primary post is often the repost/download the file came from, so the grid led with download dates rather than when content was first posted (operator-flagged). Add a second materialized sort key, earliest_post_date = MIN(post_date) across ALL of an image's provenance posts (every post it appears in), else created_at — the original publish date. Mirrors the effective_date pattern so the sort stays a forward index scan. - alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill created_at baseline then MIN over image_provenance ⋈ post. - importer: recompute earliest_post_date whenever a dated post is linked (MIN over the image's provenance, which now includes the just-added row). - gallery_service: new sorts posted_new / posted_old key off earliest_post_date; cursor + year/month grouping follow the active column transparently. - api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads with original publish date. newest/oldest (effective_date) still available. - frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post date); existing effective-date sorts relabelled "Newest/Oldest added". - tests: service test asserts posted_new/posted_old key off earliest_post_date; frontend default-sort omission test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 10:46:09 -04:00
bvandeusenandClaude Opus 4.8	181f1c6a27	perf(gpu-queue): partial indexes + two-phase lease so leasing stays O(batch) ... The throughput bottleneck was curator-side, not the network. lease() claimed the lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a plain `status` index Postgres walked the primary key from id=1, skipping the entire prefix of already done/error rows before reaching pending ones. As `done` grew (69k+), every lease became an O(done) scan — leasing crawled, the DB saturated, and even /status (the queue GROUP BY count) stalled the agent. - Migration 0070 adds two partial indexes over just the live slice: pending rows indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery + orphan sweep). They stay tiny no matter how large the done/error history. - lease() split into two phases so each uses a partial index: claim pending first (id-ordered, O(batch)); reclaim expired leases only when pending can't fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim). - Model __table_args__ declares the indexes so ORM and schema agree. - Test: a done-prefix at low ids must not stop the lease reaching pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 21:12:12 -04:00
bvandeusenandClaude Opus 4.8	359bc5a283	feat(ml): default to SigLIP 2 (new installs) + model dropdown, no free-text (#1203) ... - Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in) — UPDATE applies ONLY where no image is embedded yet (fresh install), so an existing library is NOT silently invalidated; it switches deliberately via the dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2. - GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512 recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types. - GpuAgentCard: the two name/version text fields → a single model dropdown; Save sets name+version from the picked option (the current model is always selectable even if off-list). - embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1) to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the ml-worker's persistent HF_HOME. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 16:29:27 -04:00
bvandeusenandClaude Opus 4.8	bc6d43d3f2	refactor(ml): drop dead tagger/suggestion settings + columns (#1199) ... Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:41:25 -04:00
bvandeusenandClaude Opus 4.8	485387ff0b	refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189) ... Heads + CCIP are the tag source and head auto-apply is the earned propagation. The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply path firing in parallel with heads on every accept — exactly the un-earned spray the v2 pivot replaced. Retire both. Behavior change: accepting a suggestion now applies the tag to THAT image only (source='ml_accepted', a head-training positive) — it no longer allowlists + fans the tag across the library via Camie. Propagation is heads' earned auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was un-earned and bypassed the precision gate.) - tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction writes); backfill enqueues it for images with no embedding. - Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat + every enqueue caller (accept/alias/merge/per-image), api/allowlist.py + blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067), AllowlistTable.vue + allowlist store, the accept coverage-projection payload. - AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store the rail still needs); accept returns nothing, API returns {accepted, tag_id}. - tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on ML-applied image_tag sources (incl. head_auto) instead of the allowlist. - UI: MLBackfillCard relabelled to embedding-only; accept toast simplified; MaintenancePanel drops the allowlist tile. Left for a follow-up hygiene pass (now-inert, harmless): the dead settings columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*, video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders trim, and the Camie model download in download_models.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:04:31 -04:00
bvandeusenandClaude Opus 4.8	3d77a38a25	refactor(ml): remove the dead per-tag centroid subsystem (#1189) ... The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 11:48:09 -04:00
bvandeusenandClaude Opus 4.8	4daa3f2790	feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190) ... Make the SigLIP embedder an operator choice (drop-in to SigLIP 2: google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no schema change, better small-cue fidelity). A swap = set model + re-embed + retrain, all operator-driven; the GPU agent does the re-embed so it's fast. - settings: embedder_model_name is now a setting (migration 0065) alongside the existing embedder_model_version; both editable + validated (non-empty) in the ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model, model-agnostic), preferring the pre-downloaded local dir for the default so existing deploys don't re-download; rebuilds on a name change. - agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames) under the lease-announced model → POST /jobs/submit_embedding writes image_record.siglip_embedding + siglip_model_version. The lease now announces the model FROM THE SETTING (not a constant). - re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale- version images; 'siglip' now re-embeds concept crops whose version != current (so a swap re-triggers crops, not just the never-embedded back-catalogue). The CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't churn the library at 512px) — the GPU agent owns version re-embeds. Daily 'embed' + 'siglip' beats self-heal. - scoring: score_image only bags embeddings in the CURRENT model's space (whole- image gated by siglip_model_version, concept regions by embedding_version) so a mid-swap stale vector isn't scored by new-space heads; legacy NULL = current. - UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and "Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2. Tests: lease announces model + submit_embedding round-trip; enqueue 'embed' selects stale/unembedded; stale-version excluded from scoring; embedder model settable + empty rejected; siglip gate updated to current-version concept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 10:24:30 -04:00
bvandeusenandClaude Opus 4.8	b91a230f12	feat(ccip): automation + reference quality — keep identity flowing hands-free (#114) ... Works through the optional CCIP ideas + the "keep moving even if I forget" ask: AUTOMATION (no button needed): - Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored ones retried) on their own; the queue never goes idle waiting for a click. - CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so identity tags keep flowing. ON by default (opt-out, like head auto-apply); ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut), migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH /api/ml/settings; auto_applied count in /api/ccip/overview. REFERENCE QUALITY (the over-fire root cause): - character_references now draws ONLY from single-character images — on a multi-character image the tag is image-level, so every figure would otherwise pollute each character's prototypes (a 2-char image tagged 'Velma' made Daphne's figure a Velma reference). This is the contamination behind residual over-firing. - Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the reference load isn't redone on every modal open. Tests: multi-character image not used as a reference; auto-apply tags a confident match as ccip_auto. NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot interesting content"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 22:25:40 -04:00
bvandeusenandClaude Opus 4.8	625336b6b4	feat(ccip): tunable match threshold, default 0.85 (#114) ... Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got 3-10 character guesses dominated by the most-referenced characters (a 27-ref character clears a low bar on many images). A sweep showed 0.85 collapses the noise (noisy multi-matches 47→3) while keeping the confident single-character matches. - ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85. - Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999). - Slider in the GPU agent card ("Character-match strictness") — tune live, no redeploy, same observe-and-tune loop as auto-apply. Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 20:41:09 -04:00
bvandeusenandClaude Opus 4.8	b735432d02	feat(gpu): video-ready regions + the HTTP GPU-job queue engine (#114 slice 3) ... Answers "how are videos/all media handled by the GPU worker": a job is per ITEM, but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the existing cadence), each stored with a timestamp — so a video becomes a BAG of frame embeddings (fixes the mean-embedding muddle) instead of one washed-out vector. Stills → frame_time NULL; animated GIF/WebP treated like short video. - image_region.frame_time (migration 0061, not yet deployed so folded in): the source frame's seconds for video/animated media; NULL for stills. RegionService passes it through. A whole frame is just kind='frame'. - gpu_job + GpuJobService (migration 0062): the durable work list that keeps the desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat / complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues; the agent leases+submits over the web API; Redis/Postgres stay private. Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped heartbeat, complete, fail-requeue-then-error. region test now covers frame_time. NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the agent container + control UI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 11:18:28 -04:00
bvandeusenandClaude Opus 4.8	0ea7ecdea5	feat(regions): image_region storage + service for the crop pipeline (#114 slice 2) ... The storage backbone both crop jobs write to and read from. image_region = normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id; 'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP / Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The bbox doubles as grounded-tag provenance. Migration 0061. RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines don't clobber each other) + get_regions — the GPU agent's results endpoint will call the writer; the character matcher + bag scorer read. Server-side, no GPU. Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 10:36:52 -04:00
bvandeusenandClaude Opus 4.8	48c8811d69	feat(heads): auto-apply observability + on by default (#114 auto-apply B) ... Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:36:58 -04:00
bvandeusenandClaude Opus 4.8	74fef908d2	feat(heads): earned auto-apply — sweep mechanism, off by default (#114 auto-apply A) ... Graduated heads can now apply their tag without a human — gated so it's safe: - FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled, default OFF) is on AND it has >= head_auto_apply_min_positives (default 30) clean labels. A precise-looking but under-supported low-N head can't spray tags. - auto_apply_sweep (heads.py): streams every embedded image in chunks, scores against the eligible heads (numpy, no sklearn), applies each head's tag where score >= its auto_apply_threshold and the tag isn't already applied/rejected, with source='head_auto' (distinguishable + reversible). dry_run counts only. - HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled) + recovery sweep + retention(20). - API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled), GET /api/heads/auto-apply (recent runs + per-concept report). Settings head_auto_apply_enabled + min_positives via /api/ml/settings. Tests: sweep applies above threshold, dry-run writes nothing, skips under- supported + ungraduated heads; API disabled/dry-run/conflict guards. NEXT (slice 2): the observability the operator asked for — per-concept misfire (auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a reporting API to tune. Slice 3: the UI (enable, preview, trends). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:22:54 -04:00
bvandeusenandClaude Opus 4.8	22c3b54746	feat(heads): production per-concept heads — train + score backend (#114 A) ... The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its production form (the first of three slices that make heads the suggestion source, replacing Camie + centroid). - tag_head: one logistic-regression head per general/character concept with enough labelled positives. Weights (pgvector), honest CV-derived suggest threshold + earned-auto-apply point, and per-concept quality metrics. - head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the admin card shows live + historical status across navigation. - services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data loaders + metric math so production heads match measured eval numbers) and SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one image's embedding against all heads → the rail's suggestions, cached on (count, max trained_at) so a retrain invalidates without per-request loads. - tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress) + recover_stalled_head_training_runs sweep + retention(20) + 5-min beat (rule 89). - api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET /api/heads (count, graduated, last-trained, running, per-concept table, recent runs). - ml_settings: head_min_positives + head_auto_apply_precision, tunable via /api/ml/settings. Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B — this slice makes training + scoring exist and CI-verifiable. 'precision' column stored as precision_cv (SQL reserved word). Migration 0058. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-28 10:36:25 -04:00
bvandeusenandClaude Opus 4.8	b69c70ab2b	feat(tag-eval): "keep" records a confirmation so doubts stop resurfacing ... "Keep" on a doubted positive was a no-op, so the same confirmed-correct images came back in "head doubts" every run (operator-flagged: reinforcement keeps surfacing the same images). Add tag_positive_confirmation (mirror of tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and the eval excludes confirmed positives from the doubts list — exactly as rejected items already drop out of the suggest list. The tag stays a positive either way (confirmation is a "reviewed" marker, not a training change). - model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent). - tag_eval: _confirmed_ids + exclude from head_doubts_positive examples. - store.confirmTag + card "keep" calls it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-28 01:32:20 -04:00
bvandeusenandClaude Opus 4.8	6e3c5f697f	feat(ml): tag-eval backend — head-vs-centroid learning-curve eval (persisted) ... Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained- head spine on the operator's own data, reusing the SigLIP embeddings already stored on image_record — no re-embedding, no GPU. Per concept: train a logistic-regression HEAD (positives + negatives = explicit rejections + sampled unlabeled) vs the old single-CENTROID baseline; report cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged positives grow 10→30→100→300), and example image ids (head-would-suggest / head-doubts-positive) to eyeball. Persisted so the report SURVIVES navigation (operator-flagged): the run + full report live in a new tag_eval_run row (mirrors library_audit_run); the admin card will rehydrate from GET on mount, not transient state. - models.TagEvalRun + migration 0056; runs on the ml queue (only worker with numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue. - services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml .tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report). - recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat (rule 89). scikit-learn added to requirements-ml. - tests: param normalization + the rehydrate read-path + create/conflict. Frontend admin card (trigger + render persisted report) follows next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-27 22:49:10 -04:00
bvandeusenandClaude Opus 4.8	5269cd0709	feat(provenance): capture which archive an extracted image came from (#87) ... Images pulled out of a .zip/.rar previously kept no record of WHICH archive they came from — the member->archive link was computed during extraction and discarded, leaving only image->post. So the provenance modal could only scope attachments to the whole post, showing every archive a 'High Resolution Files' bundle carried instead of the one a given file lives in. - ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id (SET NULL), migration 0055. - importer: _import_archive stamps from_attachment_id on every member's provenance row for the post (new + superseded + deduped members), resolving the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only and idempotent, so it doesn't touch the dedup/supersede branches and the backfill is safe to re-run. Nested members link to the outer stored archive. - provenance_service.for_image: when the originating post's provenance row records from_attachment_id, return ONLY that archive; else fall back to the primary-post scoping from 068def2. - ProvenancePanel: heading pluralizes ('Attachment' for a single file). - Backfill: re-running reextract_archive_attachments (ArchiveReextractCard) routes through _import_archive and stamps existing rows — no new code. Tests: capture stamps on fresh import, nested-archive attribution, per-post archive on dedup; for_image filters to the containing archive; reextract backfill stamps the link. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-21 22:22:03 -04:00
bvandeusenandClaude Opus 4.8	f678819093	feat(subscribestar): seen/failed ledger models + migration 0054 (#889) ... Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 10:04:52 -04:00
bvandeusenandClaude Opus 4.8	369e3de684	feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747) ... Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag firing on one frame survived at peak confidence, and a fixed count under-samples long multi-scene videos so real scene-local tags looked like noise. Redesign (operator-steered): - Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds` (default 4) across the 5–95% window — so a tag's frame-presence reflects real screen time independent of video length. Capped at `video_max_frames` (default 64): a long video stretches the spacing instead of exploding into hundreds of inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg timeout also cut 60s→30s). - Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in >= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on screen — duration-independent noise rejection), with confidence = MEAN over the frames it appears in (not max). Clamps the threshold to the sample count so a 1–2-frame short video still tags. - All three knobs are DB-backed ml_settings (migration 0053), patchable via /api/ml/settings + sliders in the ML settings card — replaces the VIDEO_ML_FRAMES env var (product-not-project). Tests: aggregation drops one-frame noise + means corroborated tags + clamps on short videos; settings round-trip + min>max validation. Replaced the _maxpool_predictions unit test. NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs CPU-only — is GPU enablement, tracked separately in #872. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 11:07:00 -04:00
bvandeusenandClaude Opus 4.8	f154603811	feat(import): Tier-1 video near-dup by duration+aspect (#871) ... Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:17:36 -04:00
bvandeusenandClaude Opus 4.8	96c29c370b	feat(ingest): localize inline post-body images to local copies (Phase 2) ... Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 16:39:58 -04:00