602 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	fe4e0f2b71	feat: /api/system/health — one verdict for the whole stack (milestone 365 step 3) ... The single endpoint the nav indicator and the System page will both read. Composing a verdict is this module's job, not the UI's. Two kinds of part, answered differently. LEARNED — celery roles and the GPU agent, out of service_seen, where the question is "how long since it checked in" and the answer can be "it has not". PROBED — Postgres and Redis, always expected, never learned, because a last-seen for them would be actively misleading: that Redis answered thirty seconds ago says nothing about now. **The endpoint must never fail because something it checks has failed.** That inversion is easy to write by accident and it destroys the feature exactly when it is needed — a 500 when Redis is down instead of `redis: down`. Every probe is wrapped, every wait carries a deadline (rule 156), and the roster refresh swallows its own errors. The worst case is a part reported `unknown`, which is a true statement about the system. Postgres is probed first and gates the rest, because if it is unreachable nothing else can be read — and "the database is down" is the most useful single thing this can ever say. The staleness thresholds are the design risk, not the code, and they are deliberately generous: 90s to doubt, 300s to disbelieve. The constraint is a deploy rather than a crash — `docker compose up -d` rolls start-first, so a role is briefly served by two containers and then by neither while the old one drains. Thresholds tight enough to catch a crash in seconds would paint the page red on every update, and an alarm that cries wolf on every deploy is one nobody reads. Tune down only after watching a real deploy pass through. The numbers ship in the response so the UI can explain a `stale` without keeping a second copy of them. States are described in sentences rather than left as chips: "Scheduler has not checked in for 6 min — treat it as stopped" is what someone needs at the moment they are deciding whether to go and open Portainer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 17:17:22 -04:00
bvandeusenandClaude Opus 5	dc8af8b1a7	feat: a learned roster, so a stopped part is observable (milestone 365 steps 1-2) ... Nothing in FabledCurator knew what was SUPPOSED to be running. `celery inspect` reports the workers that ANSWER, so a dead worker was a shorter list rather than a red light, and grep for any notion of expected services returned nothing. That is why Portainer was the only place an operator could see it: Portainer knows the intended set. `service_seen` is the memory that makes an absence observable — every part that has checked in, and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy. Keyed on those, this table would record a death and a birth every time the stack updates — and a status page that goes red on every deploy is a status page nobody reads, which is worse than not having one. CELERY_QUEUES is assigned per role in compose and survives container replacement, so it is the stable identity. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served, not how many containers exist. The GPU agent is keyed on agent_id, the identity its lease protocol already uses. gpu.py received it on both lease and heartbeat and threw it away — an idle agent with nothing to lease left no trace and was indistinguishable from one switched off a week ago. Now recorded on the calls that were already happening. **Who observes, corrected from the plan.** The plan said "record from the existing inspect path", which would only run when someone opened the Activity tab. Two other candidates and why they lost: - A beat sweep. If the scheduler dies the sweep stops, every row goes stale, and the page says everything is down when one thing is. An alarm that cannot distinguish "a part died" from "the observer died" is worse than none. - A background task in web. hypercorn runs --workers 4, so that is four concurrent inspect loops per container, forever. Taken instead: refresh on demand, rate-limited by the newest last_seen_at that every process can already see. The observer is then the thing serving the page — if web is down you get a browser error, not a confidently green page — and it self-limits with no coordination, since a race costs one redundant inspect that writes identical values. Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps FORWARD from 0089 rather than merely reproducing the schema. No secondary indexes: one row per moving part means every read is a handful of rows, and #3301 is the record of what speculative indexes cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 17:15:46 -04:00
bvandeusenandClaude Opus 5	8a4af589f1	docs: write the install path for someone who is not the operator (#3271) ... FC has no login — no User model, no session auth, nothing. That was a deliberate call for a single-operator tool and it stays (operator, this session), but it was nowhere in the docs, and the app stores live Patreon / SubscribeStar / Pixiv session cookies on accounts that carry a payment method. Anyone standing this up from the README could reasonably have put it behind a TLS-terminating proxy and considered it handled. So the no-auth posture is now stated three times, in the three places someone decides where to bind the port: README has a "Before you expose it" section above the install instructions, .env.example explains why there is no auth variable in it, and the compose header says it before the first service. SECURITY.md claimed the opposite. It listed "a multi-user sharing ACL — instances can be shared" among the things worth protecting; there are no accounts to share between. That was rule 47 applied to a codebase that does not implement it, and it would have told a researcher FC holds a boundary it does not. Replaced with the real posture, including that TLS without an authenticating layer in front changes nothing. Also corrected, all of it stale rather than wrong-at-the-time: - EXTENSION_API_KEY was dead config. config.py read it into a field nothing consumed; the real key is generated into app_setting on first use and managed in the UI. Removed from config.py, compose and .env.example. - .env.example pointed at docs/superpowers/specs/… — there is no docs/ dir — and described the extension key as "lands in FC-3", closed 2026-05-21. - The /import mount comment described an FC-5 ImageRepo migration run from "Settings → Maintenance → Legacy migration", a surface with no frontend. - README said the extension installs from Settings → Maintenance. It is on Subscriptions → Settings. README is now split: running FC above the line, developing FC below it, with requirements, first run, the extension, upgrading and troubleshooting on the running side. First run documents the one real gap it found — a new installer with a library on disk has no button to import it, only POST /api/import/trigger, because the manual-scan UI was retired 2026-07-02 when that stopped mattering for an established install. Filed as #3367. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-02 00:16:23 -04:00
bvandeusenandClaude Opus 5	973db73221	db: collapse alembic 0001..0089 into one baseline (#3266) ... 89 files and 6,300 lines become one file of 807. Nothing about the resulting schema changes; what goes away is the requirement that a new installation replay our development history to arrive at it. revision = "0089", down_revision = None. That pairing IS the migration strategy for existing installs, not a detail of it: a deployed database already has alembic_version = '0089' from running the real 0089, so alembic reads the version table, sees head reached, and does nothing. No stamp is required — which matters, because `alembic stamp` writes a version string without validating anything about the schema it is writing it against, and a wrong stamp is indistinguishable from a right one until the next migration fails. An empty database runs the file and records 0089. Both paths converge. The next migration is 0090, as it would have been; the numbering is continuous across the collapse on purpose. Autogenerate produced nearly all of this unaided, which was NOT true of the first attempt — that one was reverted because the generator silently dropped eleven indexes and three uniqueness guarantees. #3275 put those on the models first, so the HNSW index with its opclass, the COALESCE expression index, the partial uniques, 107 server_defaults and the enum CHECKs are all emitted now. Doing the reconciliation before the squash, rather than after, is what made this work. Hand-added, because none of it can live in a model: * CREATE EXTENSION vector / tsm_system_rows (0001, 0004) — database objects, not table metadata. * The pgvector import. Autogenerate writes qualified pgvector.sqlalchemy.vector.VECTOR references without importing the package, so its own output cannot run (run 4988). * THE TWO SEED ROWS. 0002 and 0003 did not only build schema — each inserted a settings singleton, and nothing in the app ever creates them: ImportSettings.load() and MLSettings.load() are select(...).scalar_one(), which RAISES NoResultFound rather than returning None. A models-only baseline would leave both tables empty and crash a fresh install on first settings access, while baseline.yml reported a perfect schema match. Only running the app against a new database finds that. Not carried over: 0023's DELETE FROM tag and 0047's series deletes, which are historical cleanups operating on rows an empty database lacks. downgrade() raises. A baseline's downgrade is "drop every table", which is a data-loss event wearing a migration as a disguise; offering it as one invites someone to run it. Restore from a backup. Also removed, per the plan: the 10 test_migration_*.py files (they assert intermediate states and backfills that no longer exist — a test that a column exists is already the model tests' job) and backend/app/utils/artist_backfill.py, whose only importer was 0008. Verified no other consumer anywhere in backend/ or tests/. baseline.yml changes with it. chain_ref now DEFAULTS to 725bf15, since the tree no longer carries a chain to compare against — that pinned commit is the last one that does. And the CheckConstraint repair is removed, because it never fired. I added it claiming autogenerate re-doubles a constraint name on the round trip and asserted in b979062 that it was "still correct and still needed". It is not: autogenerate wraps names in op.f(), which marks them already-formatted and blocks the convention from re-applying. Tested against the real candidate line — the regex matches nothing. What actually fixed the mismatch was 0088's renames alone. The doubling is a real hazard, but of hand-writing a pre-prefixed name, not of the generator; the comment asserting otherwise was worse than the dead code under it. The header comment is rewritten for the same reason — it described 87 revisions, and claimed the HNSW index could not be expressed in a model, which #3275 disproved. It now also states plainly what this check CANNOT see: it compares schema, so a green run means the schema is right, not that the baseline is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-01 01:14:37 -04:00
bvandeusenandClaude Opus 5	08418d54a3	db: index the seven unindexed FKs, drop the seven redundant ones (#3300, #3301) ... A structural sweep of the deployed schema, run AFTER 0088 got the models and the chain to exact agreement. That agreement is what 0088 achieved, and it is worth naming what it does not prove: a models-vs-chain diff shows the two describe the same schema, not that the schema is right. Everything here was wrong in BOTH. The one that matters: image_tag has PRIMARY KEY (image_record_id, tag_id) and no other index, so tag_id is unindexed. That is the gallery's tag filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the ON DELETE CASCADE from tag, both scanning the largest table in the schema. Six more FKs were unindexed on smaller tables; presentation_review.tag_id also CASCADEs. Dropped, on the other side: ix_image_record_sha256 was an exact duplicate of the index uq_image_record_sha256 already builds — two btrees on the same column of the highest-insert-rate table. The other six are single-column indexes a later composite superseded without the narrow one being retired; a btree on (a,b) already serves lookups on a. 0088 deliberately taught the models to declare BOTH sha256 indexes so they would describe reality. This changes the reality instead, and the models change with it — otherwise the next baseline.yml run reintroduces exactly the drift 0088 removed. CONCURRENTLY throughout, so building the image_tag index does not hold an ACCESS EXCLUSIVE lock over every write for the duration. The cost is that the migration cannot run in a transaction and so is not atomic: every statement is IF NOT EXISTS / IF EXISTS, making a re-run after a partial failure safe. The docstring carries the query for finding an INVALID index left by an interrupted CONCURRENTLY build. What the sweep found clean, for the record: all 43 tables have a primary key; all 51 FKs declare an explicit ON DELETE, so none silently blocks a delete; the three enum CHECKs match the code that writes them (rule 36). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 08:25:41 -04:00
bvandeusenandClaude Opus 5	b1bd2531ad	style: sort JSON first in three sqlalchemy import blocks ... ruff's isort runs with order-by-type, which sorts ALL_CAPS names ahead of CamelCase ones, so `JSON` belongs at the head of the list rather than between `Integer` and `String`. Two of these (backup_run.py, post.py) have been failing lint since 5e1996e — I did not check the push CI after that commit, only the baseline workflow I had dispatched, so ci.yml has been red on dev across 5e1996e, ed2b1ad and d044e93. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:38:46 -04:00
bvandeusenandClaude Opus 5	b979062dd7	db: rename the four double-prefixed CHECK constraints (#3275) ... Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one of them reverses an assumption I made in the previous commit. The doubled CHECK names are what the DATABASE has, not what the generator invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s, which — unlike uq/fk/ix — applies even to a constraint that already has a name, so four migrations that passed an already-prefixed name got it prefixed twice: ck_import_settings_ck_import_settings_singleton ck_ml_settings_ck_ml_settings_singleton ck_post_ck_post_translation_override ck_tag_ck_tag_fandom_requires_character The workflow repair added last commit is still correct and still needed — autogenerate really does re-double a name on the round trip — but it was making the MODELS side clean against a chain that is dirty. The comment in ml_settings.py claiming its bare name "matches migration 0003" was simply false; 0003 produces the doubled form. Nothing reads a CHECK constraint by name, so this has never done harm. But it is precisely the development-era residue the collapsed baseline exists to leave behind, and a public schema should not ship it — so 0088 renames the deployed constraints and all six models now declare bare names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no revalidation, which is why this is safe on post and tag. Guarded on pg_constraint scoped by conrelid, so it is a no-op on a database built from the models. ix_tag_fandom_id showed as a difference only because chain_ref was pinned to 0a5bbe8, which predates 0088 — the comparison was measuring the models against a chain missing the migration that closes the gap. chain_ref now defaults to blank, meaning "the chain in this ref". Pin it to a commit only after the collapse, when the tree no longer carries the revisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:29:57 -04:00
bvandeusenandClaude Opus 5	573228b9da	db: finish reconciling the models with the deployed schema (#3275) ... Closes the residue the first reconciliation pass left, and corrects a factual error I put into the record. sha256 was NOT missing a uniqueness guarantee. I read `op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and concluded duplicates were possible, without reading line 149 two lines above it: sa.UniqueConstraint("sha256", name="uq_image_record_sha256"), Uniqueness has held since the initial schema. The database expresses it as a CONSTRAINT plus a separate non-unique lookup index; the model said `unique=True, index=True`, which is one UNIQUE index under a different name. Same guarantee, different objects — which is exactly why the two schemas did not line up. The model now declares both objects. No DDL. 0088's docstring, which repeated the claim, is corrected in place. Two real divergences, both the MODEL over-claiming: * source: uq_source_artist_platform_url (alembic 0010) was declared nowhere in the models — source.py had no __table_args__ at all — so autogenerate would have proposed DROPPING it. * head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it nullable. Left nullable; the FK already cascades. Seven constraints renamed to what the chain actually created, rather than what base.py's naming convention renders: uq_series_page_image, uq_series_chapter_anchor_page, fk_series_chapter_anchor_page, fk_image_record_artist_id, fk_image_provenance_from_attachment, and the two hand-shortened fk_tsr_* names from 0003. Float server_defaults now mirror their own migration, per column. The chain is MIXED: a plain string renders DEFAULT '0.90'::double precision, sa.text() renders DEFAULT 0.90, and the migrations used both. Seven columns take text(); the rest stay strings. Two literals also disagreed outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against the migration's 0.90/0.50. baseline.yml gains two things. A repair for a SECOND generator defect in the same class as the missing pgvector import: base.py's ck convention contains %(constraint_name)s, so it applies even to a NAMED CheckConstraint — autogenerate writes the already-rendered name into the migration and running it applies the convention again, yielding ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage, not a claim the models make, so it is undone rather than counted. And the diff now runs twice. Column ORDER differs permanently between a schema built by 87 ADD COLUMNs and one built in a single shot — the operator's database keeps chain order forever, a fresh install gets model order — so a check that failed on it could never pass. The second pass SORTS column lines within each CREATE TABLE instead of deleting them, which cannot hide a column present on one side only, or one whose type, nullability or default differs. Ordered diff is reported as information; the order-insensitive one is the verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:24:00 -04:00
bvandeusen	ed2b1adc2e	ci: compare the schema the MODELS produce against the migrations (#3275) ... baseline.yml only ever compared migrations against migrations. The question #3275 exists because nobody had ever asked the other one: does a database built from the MODELS match the one the chain produces? `mode: models` answers it. It applies the candidate autogenerated from the models instead of this tree's revisions, and diffs that against the chain. A clean run means --autogenerate is trustworthy again, which it demonstrably has not been: against the pre-reconciliation models it would have proposed dropping eleven indexes and two uniqueness guarantees. The two extensions are created by hand in that mode. They are database objects rather than table metadata, so no model can carry them — their absence is outside what this comparison asks about, and silently tolerating it is correct rather than a filter that hides a defect. Also declares the HNSW index on the ImageRecord model. SQLAlchemy can express an hnsw access method with an operator class (postgresql_using + postgresql_ops), so there was never a reason for it to live only in 0036. That removes the last item from the list of things a generated baseline cannot reproduce, leaving only the two extensions.	2026-08-30 14:43:26 -04:00
bvandeusen	5e1996e77f	db: reconcile the models with the deployed schema (#3275) ... Milestone 328's acceptance test compared a database built by the real 0001..0087 chain against one built from the models, and found ~130 places where they disagree. This closes them. Almost all were the MODEL being wrong, so almost all of this is model edits with no DDL — the database already had these things, nothing in it changes, and no deploy is needed for this part: * 92 columns gained server_default. The models carried Python-side `default=` only, so the ORM filled the value and the column had no database default. Anything inserting outside the ORM behaved differently from production. * Eleven indexes that existed only in migrations are now declared: the three backup_run reporting indexes, the two date-ordered image_record browse indexes, import_task and presentation_review, and the three task_run history indexes. All use text() for their DESC ordering and postgresql_where for the partial one. * Two UNIQUE indexes that autogenerate silently proposed DROPPING, because neither is expressible as a UniqueConstraint: uq_tag_name_kind_fandom — an EXPRESSION index over (name, kind, COALESCE(fandom_id, 0)) uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id IS NULL post.py already had a comment describing the second one. The comment was right; nothing declared it. * The two external_link enum CHECKs (host, status) — rule 36 territory, and absent from the model entirely. * Two indexes were named explicitly. A bare index=True generated ix_tag_alias_canonical_tag_id where the database has ix_tag_alias_canonical, so autogenerate proposed a drop+create of an index that was already there under another name. Same for tag_suggestion_rejection. Only ONE thing needed DDL, as 0088: tag.fandom_id is declared index=True but no migration ever created that index. Deliberately NOT here: image_record.sha256. The model says unique=True; 0001 created a plain index. Duplicates are possible today and the ORM believes otherwise. The fix depends on whether duplicates already exist — if they do, that is a dedupe decision, not a constraint — so it waits on an answer about live data. The real severity of #3275 is not the squash. It is that --autogenerate has been unsafe on this project: run against the old models it would have proposed dropping eleven indexes and two uniqueness guarantees.	2026-08-30 14:42:30 -04:00
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusen	bce894ba24	feat(settings): the instance reports which build it is (318 step 6) ... A dim line at the foot of Settings: `FabledCurator 2026.08.28.1249 · dev`. This is no longer a convenience. Milestone 318 stopped publishing version image tags, so an instance's own report is the ONLY answer to "which build is this?" — there is no registry name left to check it against. Note #3127 §5 says it directly: a wrong answer here has no second source to contradict it. Three states, kept distinct because collapsing any two of them lies: not asked yet render nothing asked, no version render "unknown" asked, has a version render it A blank footer reads as "no version", which is a different claim from "I cannot say". And a failed health call deliberately does NOT mark the build loaded — a network blip says nothing about the image, and presenting it as "unknown" would look like a defective build. Carried on /api/health rather than a new route: it answers at the same cost (two module constants, no I/O) and TopNav already fetches it app-wide, so a separate endpoint would mean a second request for two strings. Both fields are OMITTED when unset rather than sent empty. Absence already means "cannot say" — an image predating the field says exactly that by not having the key — so a second spelling would make every reader special-case it. The pre-existing test asserting the body is EXACTLY {"status": "ok"} is what keeps a well-meaning `or ""` default from creeping in. FC_CHANNEL now has one definition. It was read from the environment in extension.py and would have been read again here; the new build_info module holds both, and extension.py binds it as a module-level name so existing tests monkeypatch it exactly as before. Separate from config.py on purpose: those are operator settings meant to be changed, these describe the artifact. Channel sits beside the version, never inside it (rule 149), asserted from both ends. A `-dev` suffix would read as a 0 segment to the extension's parseInt comparator and make every dev build compare equal — #2993 exactly. Not hidden, per the operator and §7: the JS bundle and asset hashes fingerprint the build anyway, and "I'm on 2026.08.28.1249" is the single most useful line in a bug report.	2026-08-28 18:08:31 -04:00
bvandeusen	a7e626a67a	feat(extension): report the channel beside the version (step 7) ... Closes the half of the ask the signing work didn't: a way to tell a dev build from a main one. FC_CHANNEL is baked into the web image at build time and /api/extension/manifest reports it as its own key, next to version — the popup banner, the toolbar tooltip and the Settings card all name it. Beside the version, never inside it. A `1.0.3499884-dev` suffix is the obvious shortcut and it is the exact failure this design comes from: versionIsNewer parses each dotted segment with parseInt, so a suffixed segment reads as 0, every dev build compares equal to every other, and "no update available" stops being distinguishable from "I cannot read this version". The comparator already degrades rather than discarding (rule 150), which is a reason not to NEED the suffix, not a licence to add one. Two tests hold the line — one backend, asserting version and channel are separate keys; one frontend, asserting the rendered version text stays the bare derived number. Optional on the read side, and absent rather than defaulted. An image built before this field says nothing by not having the key; an image built without a channel now says nothing the same way, so there is one absence to handle instead of a second spelling of "unknown". Every reader drops the label entirely when it is missing and reads exactly as it did before. Reported verbatim rather than validated against {dev, main}: if an image declares something else, showing what it claims helps whoever is debugging more than dropping it would. FC_CHANNEL is declared LAST in the Dockerfile. An ARG invalidates every layer below it, and this is the one value that differs between the dev and main builds of identical source — earlier, and the two channels could never share a cached pip install. A tag push counts as main: a vYY.MM.DD tag is cut from main, so that image is a main-channel artifact wearing an immutable name. No channel switcher, deliberately. background.js:34 already records that Firefox's static update_url cannot apply, because every FC instance is a different host — so the extension asks its configured backend, and the channel IS the instance it points at. Switching is repointing apiUrl and reinstalling from that host. A separate setting would contradict each server build shipping its own extension. This commit touches packaged extension files, so it moves the derived version and will sign a new one via AMO — the first push to exercise the extension-changed path from dev end to end.	2026-08-27 11:47:30 -04:00
bvandeusen	5a0e1bbd03	perf(ml): batch the auto-apply sweeps' image_tag inserts (#3072) ... Item 1 of #3072. Both sweeps issued a single-row pg_insert(image_tag) from inside their per-image loop. Steady state that is nothing; a first sweep over a back-catalogue is one round-trip per applied tag, tens of thousands of them. Each chunk now collects its rows and writes them in one statement. The ticket suggested one insert per chunk PER TAG. A single multi-row VALUES carries every tag at once, so it is one statement per chunk full stop — and the sweeps already accumulate across all heads before they commit, so nothing had to be restructured to allow it. Not a new helper: wip_title.apply_wip_image_tags was already doing the chunked ON CONFLICT DO NOTHING insert, so that shape is extracted to services/image_tag_apply.insert_image_tags and all three writers share it. The extraction deliberately leaves wip_title's pre-SELECT behind rather than pulling it into the shared function — the sweeps don't need it (their `skip` sets already exclude applied and rejected images) and it exists only to produce an accurate count, which the sweeps also compute themselves. So the shared primitive returns nothing: psycopg reports rowcount -1 for a multi-row ON CONFLICT DO NOTHING insert, and a count taken from the statement would be a lie rather than an approximation. Ordering note for the system-tag sweep: tag rows are now written after that chunk's PresentationReview rows rather than interleaved before them. Safe — PresentationReview FKs to image_record and tag, not to image_tag. Chunk size stays 5000: 5000 rows x 3 bound params = 15000, inside Postgres' 65535-parameter ceiling with room to spare. tests/test_image_tag_apply.py covers the primitive directly, since it is now the single place three writers can be wrong at once — most importantly that a re-run never restamps a hand-applied tag's source, which would silently poison head training (it excludes the auto sources). Left alone: _insert_presentation_review is still per-row, and the retract path still deletes per-row. Both operate on sets that are small by construction, unlike the apply path. Refs #3072	2026-08-27 07:51:23 -04:00
bvandeusen	1ac448d881	refactor: four small cleanups from the review pass (#3072) ... Items 2-5 of #3072. Item 1 (the per-row sweep inserts) is separate. 2. .fc-bad was not merely duplicated — it is .fc-weak under a second name. Both local definitions were `color: rgb(var(--v-theme-error))`, identical to the global .fc-weak, and GpuAgentCard was already using .fc-weak to colour exactly what GpuActivityPanel coloured .fc-bad (an errored count, red when non-zero). So rather than promoting a synonym to app.css, both call sites now use .fc-weak and the local defs are gone. app.css's status-colour comment records why there is no .fc-bad, next to the existing note on why .fc-ok is deliberately NOT global. 3. GalleryItem.vue's obsidian literals now use --v-theme-background, which IS obsidian (vuetify-theme.js maps background -> surfaces. obsidian). Preferred over --fc-chrome-rgb: same value, but that variable is named for the nav fade, not for the palette entry. The ticket said these were the only three real uses in the tree. They are not — GalleryItem itself had two more in the artist-label gradient (fixed here, so the file is now consistent), and ~13 more live in SeriesView, SeriesReaderView, ImageViewer, ArtistHeader, ExploreView and GalleryFilterBar. Those are a separate sweep, filed rather than folded in here. 4. The attachment download path had two hand-formatted copies. One definition now, `attachment_download_url`, next to the model both serializers already import. The test pins it by MATCHING the built path against the app's real URL map rather than comparing to a literal — a string-equality test would still pass after someone renamed the route, which is the drift the helper exists to prevent. 5. Extension API key now compares with hmac.compare_digest. Compared as BYTES, not str: compare_digest's str form raises TypeError on non-ASCII, and this value comes straight from an attacker-controlled header, so the str form would turn a junk key into a 500 instead of a 403. Low stakes either way — the API is unauthenticated-by-design on a LAN — but it costs nothing. Refs #3072	2026-08-27 07:48:18 -04:00
bvandeusenandClaude Opus 5	ddf896078c	refactor(platforms): retire deviantart end-to-end (#3069) ... Executes the 2026-07-05 product decision (FC downloaders = art-dedicated services only), which removed Twitter/X and Bluesky but left deviantart fully wired for seven weeks — the half-retired state rule 22 exists to prevent. Removed: the PlatformInfo module and its registry entry, the gallery-dl extractor block, extension_service's artist-page pattern, the extension's PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission and content-script match, the frontend icon/colour/label, and the operator- facing "supported platforms" list that still advertised it. Two judgment calls, both recorded in migration 0088: * existing `source` rows are DISABLED, not deleted. The row is the only record of the artist's DeviantArt URL. Disabling is also required for correctness rather than tidiness: with the platform unregistered the download path falls through to gallery-dl, which carries its OWN deviantart extractor, so an enabled row would have kept downloading from a dropped platform. * the `credential` row IS deleted — a live session cookie for a site FC will never call again. Adds the invariant whose absence is why manifest.json drifted in the first place: nothing tied its domain lists back to the platform table. The extension suite now asserts both directions, plus that no host permission belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at an operator-chosen URL the extension cannot enumerate). Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged extension change without a bump. No release is cut — build.yml's sign-extension job only runs on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:24:08 -04:00
bvandeusenandClaude Opus 5	2e0f8f8c61	feat(cleanup): reclaim orphaned attachments — rows and store blobs (#3068) ... PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or artist left the row behind rather than taking it. Nothing ever pruned those rows, and nothing in the repo had ever unlinked a file under the attachment store — so both rows and bytes accumulated permanently, invisible to every existing diagnostic. Why a disk->DB reconciliation rather than a row sweep: the store is sha-addressed and idempotent, so ONE blob backs MANY rows. Deleting a row does not free its blob, and since the artist cascade (#3066) now deletes its attachment rows outright, a freed blob has no DB pointer left to find it by. Walking the store and asking "does any row still reference this sha?" catches orphans from every cause, including ones no future delete path will think to report. Preview and apply share `_orphan_attachment_conditions` (rule 93). The dry-run derives its surviving-sha set by NEGATING that same predicate, so it is honest about blobs the delete would free rather than counting them as still-referenced — the one place this was easy to get backwards, so it has its own parity test. Guards, each with a reason: - A blob is written before its row commits, so a just-stored file legitimately has no referencing row. Files under 6h are never judged — same guard and reasoning as ORPHAN_TEMP_MIN_AGE_HOURS. - `.partial` staging files belong to cleanup_orphaned_temp_files; skipped rather than raced. - The sha is parsed as the first 64 chars, not via Path.stem: store() takes the extension from the source filename, and a URL-encoded basename yields a multi-dot suffix that would make stem eat part of the sha. - A 900s walk budget reports partial=True instead of running to the task's hard limit (rule 89). - TASK_STUCK_THRESHOLD_MINUTES override at 30 (= time_limit 25 + 5). Without it a healthy 20-minute walk is phantom-flagged 'RecoverySweep' at the bare 5-min default — the #883 failure class; its invariant test is mirrored here. Defaults to the safe preview at both the task and the route, unlike the other maintenance triggers: this apply unlinks files. Operator-triggered only, never on a beat. Ships with its UI (rule 27): AttachmentReclaimCard in Cleanup → Duplicates & leftovers, built on the existing useMaintenanceTask/MaintenanceTile shapes, so a run survives navigating away. Surfaces files_failed and partial explicitly, since both change what the numbers mean. Also promotes humanBytes to utils/bytes.js — it was byte-identical in VideoDedupCard and GatedPurgeCard and this card would have been the third copy. The three divergent `formatBytes` helpers are deliberately left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:37:24 -04:00
bvandeusenandClaude Opus 5	2ce467e347	fix(cleanup): artist cascade preview counts posts and attachments (#3067) ... `project_artist_cascade` is documented as "a read-only projection of what delete_artist_cascade would touch" and drives the Tier-C confirm dialog, but it counted only images, sources, thumbs, import_tasks and bytes. It never counted posts or attachments, both of which the apply destroys. That is silent in the worst case. `Post.artist_id` is ondelete=CASCADE, so every post goes whether or not it carried an image — and FC has a large body-only post population (#1288 measured 694 pixiv posts with text and no images). Such an artist previewed as `images: 0`, reading as "empty, safe to remove", while the apply destroyed every captured body, description, external-link set and raw_metadata snapshot. The danger-zone card already promised "every image, source, post, and attachment" — the copy was honest and the numbers were not. Root of the drift: the preview re-derived its own predicates instead of sharing the apply's, the same shape as the 2026-06-08 fandom-tag deletion that rule 93 exists for. So rather than bolt on two counts, both halves now build from shared `_artist_{images,posts,attachments}_conditions` helpers, following the `_unused_tag_conditions` / `_bare_post_conditions` style already in the file. Sources keep no helper — the apply doesn't query them either, it gets them from the Artist.sources ORM cascade. The apply also now reports `posts_deleted` (counted before the delete, since the CASCADE leaves nothing to count after). Rule 93's second half asks for the apply to be tested, and parity is only assertable if both halves state the number. Adds a preview/apply parity test that runs both against one artist and asserts the three pairs agree AND that the rows actually went, plus a body-only-artist test covering the case that motivated this. The confirm dialog's counts grid renders every key, so posts and attachments surface there automatically; the prose summary line names posts explicitly, since that is the number that changes how an artist reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:23:29 -04:00
bvandeusenandClaude Opus 5	39cf81aea6	fix(cleanup): clear an artist's attachments before the cascade delete (#3066) ... `delete_artist_cascade` could abort partway through, and it aborted after the irreversible half. Deleting an artist CASCADEs to Post (post.artist_id is ondelete=CASCADE), which SET NULLs post_attachment.post_id — and `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE WHERE post_id IS NULL. So any two of that artist's attachments sharing a sha collapse onto one another and raise. That shape is ordinary, not corrupt: `_capture_attachment` deliberately writes one row per post over a single sha-addressed blob, so a creator who attaches the same pdf to two posts already has two such rows. A pre-existing filesystem-import row (post_id NULL) with the same sha collides on its own. The images and their on-disk files are deleted and committed in 500-row batches BEFORE the artist row is touched, so the failure landed after them: images gone, artist and posts alive, files unrecoverable. Fix: delete the artist's post_attachment rows explicitly first, matched by artist_id OR by the owning post's artist (artist_id is nullable, so neither arm alone covers every row). `_repoint_post_links` already guards the identical collision class in the reconcile path; this is its artist-cascade counterpart. Migration 0043 reasoned only about upgrade-time safety and never about this later SET NULL. The sha-addressed blobs are deliberately left on disk: one blob backs many rows, so unlinking needs a refcount pass, and this Tier-C op must not delete bytes its own preview never disclosed. Adds `attachments_deleted` to the summary, and two regression tests — the same sha on two posts, and an unrelated NULL-post row that must survive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:02:51 -04:00
bvandeusenandClaude Opus 4.8	05df51b749	fix(ml): drop unnecessary quotes on MLSettings.load annotations (UP037) ... Python 3.14 evaluates annotations lazily, so the self-referential return type needs no forward-ref quotes — matches ImportSettings.load. Fixes the ruff lint lane on the DRY-pass push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi	2026-07-13 21:49:30 -04:00
bvandeusenandClaude Opus 4.8	099e1e664c	refactor(patreon): DRY the campaigns-API request (#161) ... _lookup_via_api and resolve_display_name shared ~90% of their body (same endpoint, params, headers, error handling — differing only in which field they pluck from data[0]). Extract _campaigns_api_first(vanity, cookies_path) -> dict|None; callers pluck the campaign id vs the display name. Return-value behavior preserved (the display-name path additionally gains the helper's more granular warning logs). Covered by the existing test_patreon_resolver.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi	2026-07-13 21:41:34 -04:00
bvandeusenandClaude Opus 4.8	c87f8a1bb3	refactor(maintenance): DRY the recover_stalled head-run twins (#161) ... recover_stalled_head_training_runs and recover_stalled_head_auto_apply_runs were near-exact copies (coalesce-flip + keep-last-N prune, differing only in model + two constants). Extract _recover_stalled_runs(model, stall_minutes, keep_runs, label); the two tasks become thin wrappers. The other two recover tasks are deliberately NOT folded in (library-audit has no prune tail; backup uses a single started_at cutoff). test_recover_stalled_head_runs.py covers both wrappers (stalled→error, fresh survives) — previously untested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi	2026-07-13 21:41:34 -04:00
bvandeusenandClaude Opus 4.8	666b3a2ec8	refactor(ml): DRY pass — shared sweep helpers + table-driven settings (#161) ... Consolidate duplication accrued across the ML tagging + settings backend, behavior-preserving (over-DRY guard applied — the three auto-apply sweep BODIES stay separate; only their shared inner helpers are extracted). - _sigmoid / _conflict_scores / _insert_presentation_review (heads.py): the score→prob transform (6 inlined sites), the presentation conflict signal (2 sites), and the ring-loud PresentationReview insert (2 sites, single- sourced so the mode column can't drift on the shared composite PK). - _applied_or_rejected (training_data.py): the per-tag "applied ∪ rejected" skip-set, byte-identical at 3 sweep sites (heads.py x2, tasks/ml.py ccip). - ccip sweep divergence fixes: import ccip._FIGURE_KINDS + training_data._l2norm instead of local copies that silently drift when the canonical changes. - MLSettings.load / .load_sync classmethods (mirror ImportSettings); route all 8 scalar_one singleton reads through them (the session.get None-path stays). - GET serializers for MLSettings + ImportSettings are now table-driven off the same _EDITABLE tuples PATCH writes, so a new field can't be silently absent from GET (the split that historically dropped fields). - AUTO_APPLY_THRESHOLD_MIN/MAX constant single-sources the [0.5,0.999] operating range across the service clamp + the 5 API validators. - test_ml_dry_helpers.py pins _applied_or_rejected + _sigmoid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi	2026-07-13 21:41:24 -04:00
bvandeusenandClaude Opus 4.8	69b5637bd6	fix(extension): show Add-to-FC button on subscribed Patreon creators (#1485) ... Symptom (operator-flagged): the extension injected the Add-as-source button on a Patreon creator you had NOT subscribed to, but it disappeared once you were subscribed — the opposite of when it's useful. Root cause (extension logic, not Patreon security): Patreon serves a creator under three URL shapes — bare patreon.com/Atole, patreon.com/c/Atole, and patreon.com/cw/Atole (the 'creator workspace' URL you land on once subscribed; documented in patreon_resolver._VANITY_RE). The button's artist-page gate (PLATFORM_ARTIST_PATTERNS.patreon in platforms.js) and its byte-mirror probe pattern (_PLATFORM_PATTERNS in extension_service._derive) only matched the bare single-segment form and explicitly excluded c/. So the subscribed-view URL failed the gate → no button. The ingestion resolver already handled all three; only these two gates were too narrow. Fix: both regexes now accept optional cw/ and c/ prefixes and drop the strict single-segment end-anchor, so a creator's inner page (/cw/Atole/posts, /Atole/membership) also matches — robust to whatever exact shape the subscribed view uses. Nav-page exclusions (home/search/messages/notifications/library/ settings/posts + post permalinks) preserved. New unit test covers all three prefixes, sub-paths, and nav-page rejection (both regexes validated identically). Bump extension 1.0.7→1.0.8 so a fresh signed XPI ships the fix (also exercises batch-5 web-ext-10's AMO sign path end-to-end on the main build). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 17:50:07 -04:00
bvandeusenandClaude Opus 4.8	fac5ae6ce5	feat(explore): reach dial to escape dense clusters + anti-revisit (#1476) ... The Explore walk got stuck in dense signatures — neighbours all too similar, so forward-arrow couldn't escape and Random was the only exit. Root cause: MMR only diversifies WITHIN the nearest ~400 pool; in a dense cluster that whole pool is near-identical, so there's no escape route in it. - gallery_service.similar(reach=0.0, exclude_ids=None): reach>0 widens the pool (cap 400→1000) and _reach_sample strides across an outward-growing distance span so the set handed to MMR spans near→mid-far (guaranteed escape routes), not just the tight cluster. exclude_ids drops already-walked images. Gallery 'more like this' (reach=0) is unchanged. - api/gallery similar: parse reach + exclude_ids. - explore store: default reach 0.4 (auto-diversifies without touching the dial), pass the breadcrumb as exclude_ids, setReach action. - ExploreView: a Near↔Far reach slider in the trail. - tests: _reach_sample math (deeper ranks with higher reach, near kept); similar exclude_ids drops walked + reach path runs clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 12:06:22 -04:00
bvandeusenandClaude Opus 4.8	af0d39ed52	feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474) ... Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 10:14:38 -04:00
bvandeusenandClaude Opus 4.8	ad2a5fc5fe	feat(system-tags): process vs chrome groups + WIP provisional auto-apply (#1464) ... Backend for the system-tag behavior refactor (milestone #157). editor screenshot moves from chrome (hidden) to the PROCESS group (shown, like wip); wip+editor gain provisional auto-apply so they stop needing endless manual identification — without a runaway loop. - tag.py: split PRESENTATION_SYSTEM_TAGS → CHROME_SYSTEM_TAGS (banner) + PROCESS_SYSTEM_TAGS (wip, editor screenshot). - heads.py: generalize presentation_auto_apply_sweep → system_tag_auto_apply_sweep (mode chrome|process). Same Guard 1 (skip human/confirmed) + Guard 2 (ring-loud conflict → PresentationReview). process mode uses source 'process_auto' and does NOT hide (hide is a gallery-query effect of group membership). - training_data._AUTO_SOURCES += 'process_auto' → the head never trains on its own auto-applied output; only wip_title/manual train it (the runaway break). - ml_settings: process_auto_apply_enabled (OFF, opt-in) + threshold + conflict threshold. presentation_review.mode ('chrome'|'process'). Migration 0086. - gallery_service: default-hide reads CHROME only (editor now shows); Explore neighbors exclude the whole PROCESS group. - tasks/ml + celery beat: scheduled_process_auto_apply (daily, opt-in); prune covers both modes. - api: ml_admin process_* CRUD+validation; hidden-review returns mode. - tests: rename chrome sweep calls; new test_process_auto_apply (apply, guards, mode flag, no-self-train); gallery test asserts editor now visible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 23:15:59 -04:00
bvandeusenandClaude Opus 4.8	0da0e47784	fix(wip-title): count inserts via pre-SELECT, not driver rowcount ... apply_wip_image_tags relied on result.rowcount, but psycopg reports -1 for a multi-row INSERT ... ON CONFLICT DO NOTHING (executemany path), so the return count (and the backfill's reported total) was wrong. Compute the count from a pre-SELECT of already-tagged ids within the same transaction; keep ON CONFLICT DO NOTHING as a race-safety belt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:22:25 -04:00
bvandeusenandClaude Opus 4.8	503c8854bc	style(importer): sort wip_title import into the local-import block ... Ruff I001 fixup for 5719387 — the new .wip_title import belongs after .thumbnailer (alphabetical), not after .archive_extractor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:16:11 -04:00
bvandeusenandClaude Opus 4.8	571938781a	feat(tagging): title-based WIP auto-tagging (#1458) ... Auto-apply the `wip` system tag to posts whose TITLE explicitly declares work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision complement to the image-based ML `wip` head. WIP images are excluded from the Explore/gallery browse, so honouring the artist's own label keeps unfinished pieces out of the main browse. - services/wip_title.py: precision-first token-anchored matcher (swipe/wiped never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO NOTHING, chunked under the psycopg param ceiling). - importer: live hook on FRESH import only (never on deep-scan/supersede), so a manually-removed WIP tag is never re-applied by a routine re-scan. - maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep (coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a beat — a periodic re-run would silently undo manual removals. - ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating the live hook; GET/PATCH + POST /settings/wip-title/scan. - Settings UI: toggle + "Scan existing posts" button. - Tests: pure matcher unit tests + integration (apply idempotency, backfill precision). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:12:19 -04:00
bvandeusenandClaude Opus 4.8	aea2701c28	feat(translation): tunable acceptance floor (0.90) + per-post sticky override (#155) ... The gate at a fixed 0.80 couldn't catch the real pain: Interpreter (fresh == cached, verified by probe) confidently mis-detects short ASCII English like "... WIP Part 1" as German at 0.86 — above the floor — so it was accepted and a re-translate reproduced it. Confidence alone can't separate the 0.86 collision (genuine German lands there too), and single-word mis-flags sit at a confident 1.0 no floor catches. Two operator-approved levers: - Acceptance floor is now a live Settings value (ImportSettings. translation_min_confidence, default 0.90; surfaced in the Translation card), so it's tunable without a redeploy. _accept takes the threshold as a parameter. - Per-post sticky override (Post.translation_override: auto/force/original). 'force' stores a translation even below the floor (rescue a skipped legit-foreign title); 'original' keeps the original and clears any stored translation (kill a confident mis-flag no floor catches). The sweep honors it on every run and _reset_translations skips 'original', so the choice survives a Re-translate-all. POST /api/posts/<id>/translation-override applies it immediately (translate now when the service is up, else queue for the sweep). UI: PostTranslationControl on the posts-feed card. Migration 0084 (both columns + a CHECK on the override). The feed + provenance serializers expose translation_override. With a stricter floor the rollback finally works: raise it -> Re-translate all -> the 0.86 mis-flags are rejected and restored to the original; force / keep-original handle the residual either way. Tests: gate thresholds against the param (0.86 rejected at 0.90, explicit-floor cases); sweep force/original + re-translate-skips-original; override endpoint (validation, original clears, force queues when disabled, feed exposes it); settings min_confidence default/save/validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-10 22:38:25 -04:00
bvandeusenandClaude Opus 4.8	a444cf82d1	refactor(tags): unify suggestion source — one canonical DB-tag dropdown, drop dead raw/alias machinery (#154) ... Every tag suggestion is a canonical DB tag now (tagging-v2 #114: heads + CCIP score EXISTING concept tags). The pre-heads apparatus for model-predicted tags that didn't exist in the DB — creates_new_tag / raw_name / via_alias, the /suggestions/alias endpoint + add_alias_and_accept, AliasPickerDialog, and the store's aliasAccept/removeAlias — was dead and is removed. The type-to-add dropdown was TWO row sources (server autocomplete + the image's ML suggestions) merged with a dedup that dropped the %-bearing suggestion row when the debounced server hit landed — the operator's "confidence % flickers then vanishes". Now it's ONE list of DB-tag matches, each annotated with the model's confidence (join by canonical_tag_id) when the tag was scored for this image. No dedup, no flicker; picking a suggested tag still records acceptance via TagPanel.findPending. Single per-image fetch: score_image now reports above_threshold per row (computed vs the head's own suggest cut, separate from the inclusion floor), so the rail makes ONE min=0 request and derives the panel (above_threshold) and the dropdown (all, text-filtered) client-side — the two /suggestions calls collapse to one. Manual "Create 'X' as <kind>" (novel typed names) is unchanged; the alias table + tag-side alias admin + auto-apply alias matching are untouched. Tests: gate/serializer assertions updated (above_threshold; dropped dead-field + alias-endpoint checks); frontend spec seeds via the single load and covers the byCategory/aboveByCategory split. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-10 15:22:51 -04:00
bvandeusenandClaude Opus 4.8	6ab7fd5c7f	tune(translation): set latin acceptance floor to 0.80 (#1376) ... Calibrated against fresh probes once Interpreter returned real langdetect confidence: genuine German detected at 1.0, a correctly-detected but ambiguous latin string at 0.86. Set _MIN_LATIN_CONFIDENCE to 0.80 (below that band) so legitimate ambiguous non-English still translates while genuinely-unsure guesses are rejected. Real langdetect also fixed the original mis-flag at the source, so this floor is a safety net, not the primary fix. Pin 0.86-accepted in the gate test to guard against bumping the floor back up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-09 16:40:02 -04:00
bvandeusenandClaude Opus 4.8	7ddf94220d	feat(translation): accept/reject gate on Interpreter's detection confidence (#1376) ... Interpreter now returns a real per-detection confidence (source stays "auto"), so curator can reject the mis-detections it was blindly storing — e.g. a short English title mis-labelled as German and rewritten into the archive. The gate consumes ONLY Interpreter's own reported detection — curator does no language detection of its own (Scribe rule 133): a field is stored when the engine actually translated it AND either the detected language is CJK (script-detected, reliably high — ja/ko/zh trusted outright, incl. pure-kanji Japanese that lands as zh ~0.75) or the reported confidence clears a latin-script floor (_MIN_LATIN_CONFIDENCE = 0.90). A latin detection below the floor keeps the original and marks the post handled; a missing confidence fails open. The client already sent source="auto" and parsed confidence, so this is purely the gate + tests. Tests: pinned interpreter-client test now asserts source stays "auto"; new pure-unit gate tests (CJK trusted / latin floor / case-insensitive / fail-open) in the fast lane; end-to-end reject-low-latin, accept-high-latin, accept-low-cjk sweeps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-09 16:17:56 -04:00
bvandeusenandClaude Opus 4.8	ffdbdbaf07	feat(translation): 8h sweep + drain button + detection probe (#1376) ... Throughput: translate_posts now runs every 8h (was daily) as the steady-state cadence for newly-imported posts, and the Settings "Translate now" button runs it in drain mode (run-until-done, no reset) so one press clears the whole untranslated backlog instead of a single 300-post chunk. The interrupt/backoff re-enqueue now preserves the drain flag so a bulk drain resumes cleanly after an Interpreter restart. Misdetection groundwork: surface the detector's confidence from the Interpreter client (it was in the detectedLanguage payload but discarded) and add a read-only "Test translation" box — POST /settings/translation/ probe + TranslationCard UI — that shows detected language + confidence + engine + result for pasted text, without saving. Lets the operator see why a short/abbreviation-heavy English title gets mis-detected so the detection guard (min-length + confidence floor) can be tuned from real numbers. The guard itself follows once the mis-detected cases are probed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy	2026-07-08 20:13:49 -04:00
bvandeusenandClaude Opus 4.8	3c7ab44e74	feat(translation): per-field language detection for mixed-language posts ... _translate_one translated [title, description] in ONE Interpreter call and keyed the whole-post passthrough on the aggregate detected_lang (the FIRST item). So an English title + non-English description detected "en" and marked the post handled, leaving the description untranslated. Now each field is translated independently (its own detected_lang / passthrough) and the non-target field is stored on its own; translated_source_lang reflects the translated field's language. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 23:31:04 -04:00
bvandeusenandClaude Opus 4.8	4371ddb7e7	feat(translation): live re-translate progress in the Settings card ... translation/status now reports `active` (a translate/retranslate sweep is running, from the TaskRun table) and `last_run` (the most recent finished run's task + status). The Settings card polls live while a sweep runs, showing a spinner + "Translating… N remaining" that ticks down, and flags a last run that ended in error/timeout. No migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 22:20:32 -04:00
bvandeusenandClaude Opus 4.8	0b78264d62	feat(maintenance): daily janitor for orphaned .part/.partial staging files ... Downloads/imports stage into <name>.part / <name>.partial then os.replace() into place, so a kill mid-write leaves a discardable temp — never a corrupt final. cleanup_orphaned_temp_files sweeps ones left behind under the images root, only older than 6h so an in-flight download's staging file is never removed. Daily beat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 22:15:33 -04:00
bvandeusenandClaude Opus 4.8	9eae636047	feat(translation): pooled Interpreter session + manual sweep resumes after drain ... - interpreter_client: shared requests.Session with a connect-only retry (connect=2, no status retries — we map 429/5xx ourselves) so a proxy reload is smoothed and the keep-alive connection is pooled across the sweep. - translate_posts: on an interrupt (drain), re-enqueue after the Retry-After hint / default backoff instead of waiting for the daily beat; self-terminates via the health gate. Steady-state one-chunk-per-run on success is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 22:10:05 -04:00
bvandeusenandClaude Opus 4.8	f8105046dc	feat(explore): exclude WIP-tagged work from the Explore rabbit-hole ... Explore's neighbour grid (/api/gallery/similar → gallery_service.similar) now takes an Explore-only exclude_wip flag that drops `wip` system-tagged images from the candidates, alongside the banner/editor presentation tags. The gallery's own "similar" button is unchanged (keeps wip, #1274) — only the Explore store passes exclude_wip=1. The anchor itself may still be a WIP; only neighbours are filtered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 22:10:05 -04:00
bvandeusenandClaude Opus 4.8	d631ed023c	style(translation): use datetime.UTC alias (ruff UP017) ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 21:04:17 -04:00
bvandeusenandClaude Opus 4.8	c64261593d	feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience ... Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs mid-flight. Give in-flight work room to drain and make interrupted work resume cleanly instead of stalling. - docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s / scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown can actually drain before SIGKILL. - celery_app.py: task_reject_on_worker_lost=True — a task killed past the grace window is re-queued (safe: idempotent + chunked, recovery sweeps re-drive stragglers). - interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter behind a reverse proxy no longer raises an opaque HTTPError. - translation.py: thread retry_after out of _translate_batch; retranslate_posts resumes after the Retry-After hint (or 60s default, capped 900s) on an interrupt with _reset_done=True, self-terminating via the health gate. - tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff. No migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 21:01:00 -04:00
bvandeusenandClaude Opus 4.8	1f6d94f51d	feat(translation): re-translate on model change — artist-scoped + global re-run (#146) ... retranslate_posts resets the 5 translation columns to NULL for a scoped set of posts (all, or WHERE artist_id IN ids) then reuses the untranslated sweep to re-run them, chasing the tail until drained (run-until-done). Interpreter cache keys on engine_version so a changed model re-translates, an unchanged one is cache-fast. Reset only happens when the service is configured+healthy so translations are never wiped when they can't be rebuilt. New POST /settings/translation/retranslate (artist_id | all=true). UI: per-artist 'Re-translate posts' on the Artist Management tab + 'Re-translate all' in the Settings Translation card, both with confirm dialogs. No migration (reuses m143 columns). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 20:03:19 -04:00
bvandeusenandClaude Opus 4.8	6a255482ea	feat(translation): "Test connection" button — on-demand Interpreter health check (#143) ... New POST /api/settings/translation/test pings /v1/health for a GIVEN base URL (not the saved one), so the operator can verify a URL before enabling it. TranslationCard gains a Test-connection button that reports reachable/unreachable inline and updates the status dot. Endpoint test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 13:00:37 -04:00
bvandeusenandClaude Opus 4.8	ead60978e3	feat(translation): expose translated fields in post feed + provenance payloads (#143 step 4) ... post_feed_service (card + detail) and provenance_service._post_dict now include post_title_translated, description_translated (card-truncated / detail-uncapped) and translated_source_lang, keeping the originals for the toggle. Feed serialization test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 12:31:46 -04:00
bvandeusenandClaude Opus 4.8	7a4de7278d	feat(translation): backfill sweep + beat + manual trigger (#143 step 3) ... tasks/translation.py — translate_posts: picks untranslated posts (title OR description non-empty), per-post [title, description] batch via the Interpreter client, stores translations + detected lang + engine_version; passthrough / already-target posts are marked handled with no stored translation. 503 or a connection error interrupts (retry next cycle), 400 stops (fix config), per-post commit keeps progress; wall-clock bounded. Wired into celery (maintenance_long lane) + a daily beat. No-op unless enabled + base URL set + healthy. GET /settings/translation/status + POST .../run for the Settings card. Task tests (stubbed client, monkeypatched session). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 12:29:28 -04:00
bvandeusenandClaude Opus 4.8	7f8073c4c8	feat(translation): Interpreter client — batch translate + health (#143 step 2) ... services/interpreter_client.py: sync (requests) client for the LibreTranslate- compatible /v1/translate — no new dep, mirrors the platform clients. translate() maps translatedText[]↔texts (order + length), returns detected_lang + engine_version (aggregate = first item, fine for a per-post [title, description] batch); passthrough items come back unchanged in their slot. InterpreterUnavailable on 503 / connection error (retry later), InterpreterBadRequest on 400. health() checks /v1/health engines.llm. 10 unit tests with mocked HTTP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 12:23:49 -04:00
bvandeusenandClaude Opus 4.8	a3bc98a53c	feat(translation): Post translation columns + settings + migration (#143 step 1) ... Post gains post_title_translated / description_translated / translated_source_lang / translation_engine_version / translated_at — filled by the translate sweep so viewing is instant. ImportSettings gains translation_enabled (OFF by default), interpreter_base_url (EMPTY — no default host; the operator points it at their own Interpreter proxy behind a reverse proxy) and translation_target_lang (en), exposed + validated via /settings/import. Migration 0083. Settings defaults + patch + validation test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-07 12:20:31 -04:00
bvandeusenandClaude Opus 4.8	2bcaa20b22	feat(ml): schedule presentation auto-hide sweep + retention (#141 step 6) ... scheduled_presentation_auto_apply (daily beat) runs presentation_auto_apply_sweep — idempotent, so an interrupted run just re-runs next cycle (that's the recovery), wall-clock bounded by soft/hard task time limits. prune_presentation_reviews (daily beat) drops RESOLVED review flags older than 30 days (rule 89 retention). Tests run both tasks via a monkeypatched session factory. Milestone 141 complete: the presentation-chrome auto-hide + conflict-flagged review is now live end-to-end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 23:46:09 -04:00