d0b0458d279574f373262a9c1e97dcb32508877a
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5
|
235393c08b |
feat: link the Patreon teaser to the Discord drop it announced (388 step E5)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
The point of the milestone rather than its tail. Two of the operator's artists post a deliberately cropped fragment on Patreon to signal that the real thing has landed in their Discord; this proposes those pairs. Confirm-only, following the FC-6.3 series matcher. A wrongly-asserted association tells the operator two different pieces are one, which is strictly worse than no link: no link leaves them where they already were, a wrong one actively misinforms and then propagates into whatever reads it. So the matcher's job is a SHORT list worth reading, not a long list worth trusting. **The threshold sits above every single signal weight, and that is the design.** Proximity is 0.55, declaration 0.45, the cut 0.60 — so neither signal can carry a pair alone. That makes "time proximity alone is never sufficient" an arithmetic property rather than an aspiration: on a busy day an artist posts several times, and a matcher that could pair on proximity alone would turn every one of those days into false pairs until the review queue got abandoned. A guard test asserts the relationship against WEIGHTS directly, so it survives any refactor of the scorer, and says in its own failure message not to fix it by lowering the assertion. **Crop-to-source matching is HELD, on the plan's instruction** — real work with real false-positive risk, worth building only once signals 1 and 2 are shown insufficient against the operator's actual artists. Worth stating: a naive whole-image SigLIP similarity is NOT that signal. A cropped teaser and its full version are precisely the pair a whole-image comparison handles worst, so adding one as a "bonus" would mostly add noise while looking like progress. Two premises in the plan corrected in the building: * **E4 is not actually a prerequisite.** A Patreon Source and a Discord Source the operator has added under one Artist already share `Post.artist_id`, and the synthetic grouping inherits it. E4 EXTENDS this to creators FC has to learn the association for; it is not needed to represent one FC was told. Same-artist is then a hard filter, not a scored signal — two different creators posting minutes apart is a coincidence, not evidence. * **`link_extract` cannot supply the declaration signal.** It exists, but `SUPPORTED_HOSTS` is file hosts only and `host_for()` returns None for a Discord URL, so no ExternalLink row is ever written for one. The signal reads the post body directly instead. And a bug my own test would have caught: `declared_signal` stripped the HTML before looking for an invite, but `html_to_plain` discards attributes and these creators put the invite in an anchor's `href` — so the strongest form of the signal was being thrown away, leaving only whatever the link text said. The invite now matches the raw body; the bare mention still matches stripped text, so `\bdiscord\b` is tested against prose rather than against markup. Dismissed rows are kept, not deleted: the row is what remembers the rejection, and re-proposing a rejected pair on every scan is the one behaviour that makes a review queue get ignored. Both FKs CASCADE, so E3's one-DELETE reversal cannot leave a proposal pointing at a post that no longer exists. Only ACCEPTED links reach the post payload. A pending proposal is a question for the review queue, not a claim to render beside the artwork. UI (rule 27) follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9 |
||
|
bvandeusenandClaude Opus 5
|
b979062dd7 |
db: rename the four double-prefixed CHECK constraints (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 5s
CI / extension-version (push) Successful in 5s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-ml (push) Successful in 44s
Build images / build-web (push) Successful in 41s
CI / integration (push) Successful in 3m52s
Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one
of them reverses an assumption I made in the previous commit.
The doubled CHECK names are what the DATABASE has, not what the generator
invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s,
which — unlike uq/fk/ix — applies even to a constraint that already has a
name, so four migrations that passed an already-prefixed name got it
prefixed twice:
ck_import_settings_ck_import_settings_singleton
ck_ml_settings_ck_ml_settings_singleton
ck_post_ck_post_translation_override
ck_tag_ck_tag_fandom_requires_character
The workflow repair added last commit is still correct and still needed —
autogenerate really does re-double a name on the round trip — but it was
making the MODELS side clean against a chain that is dirty. The
comment in ml_settings.py claiming its bare name "matches migration 0003"
was simply false; 0003 produces the doubled form.
Nothing reads a CHECK constraint by name, so this has never done harm.
But it is precisely the development-era residue the collapsed baseline
exists to leave behind, and a public schema should not ship it — so 0088
renames the deployed constraints and all six models now declare bare
names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no
revalidation, which is why this is safe on post and tag. Guarded on
pg_constraint scoped by conrelid, so it is a no-op on a database built
from the models.
ix_tag_fandom_id showed as a difference only because chain_ref was pinned
to
|
||
|
bvandeusenandClaude Opus 5
|
573228b9da |
db: finish reconciling the models with the deployed schema (#3275)
CI / lint (push) Failing after 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 34s
Build images / build-ml (push) Successful in 53s
Build images / build-web (push) Successful in 44s
CI / integration (push) Successful in 4m5s
CI / backend-lint-and-test (push) Successful in 1m6s
Closes the residue the first reconciliation pass left, and corrects a
factual error I put into the record.
sha256 was NOT missing a uniqueness guarantee. I read
`op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and
concluded duplicates were possible, without reading line 149 two lines
above it:
sa.UniqueConstraint("sha256", name="uq_image_record_sha256"),
Uniqueness has held since the initial schema. The database expresses it
as a CONSTRAINT plus a separate non-unique lookup index; the model said
`unique=True, index=True`, which is one UNIQUE index under a different
name. Same guarantee, different objects — which is exactly why the two
schemas did not line up. The model now declares both objects. No DDL.
0088's docstring, which repeated the claim, is corrected in place.
Two real divergences, both the MODEL over-claiming:
* source: uq_source_artist_platform_url (alembic 0010) was declared
nowhere in the models — source.py had no __table_args__ at all — so
autogenerate would have proposed DROPPING it.
* head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it
nullable. Left nullable; the FK already cascades.
Seven constraints renamed to what the chain actually created, rather than
what base.py's naming convention renders: uq_series_page_image,
uq_series_chapter_anchor_page, fk_series_chapter_anchor_page,
fk_image_record_artist_id, fk_image_provenance_from_attachment, and the
two hand-shortened fk_tsr_* names from 0003.
Float server_defaults now mirror their own migration, per column. The
chain is MIXED: a plain string renders DEFAULT '0.90'::double precision,
sa.text() renders DEFAULT 0.90, and the migrations used both. Seven
columns take text(); the rest stay strings. Two literals also disagreed
outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against
the migration's 0.90/0.50.
baseline.yml gains two things. A repair for a SECOND generator defect in
the same class as the missing pgvector import: base.py's ck convention
contains %(constraint_name)s, so it applies even to a NAMED
CheckConstraint — autogenerate writes the already-rendered name into the
migration and running it applies the convention again, yielding
ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage,
not a claim the models make, so it is undone rather than counted.
And the diff now runs twice. Column ORDER differs permanently between a
schema built by 87 ADD COLUMNs and one built in a single shot — the
operator's database keeps chain order forever, a fresh install gets model
order — so a check that failed on it could never pass. The second pass
SORTS column lines within each CREATE TABLE instead of deleting them,
which cannot hide a column present on one side only, or one whose type,
nullability or default differs. Ordered diff is reported as information;
the order-insensitive one is the verdict.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
|
||
|
bvandeusen
|
5e1996e77f |
db: reconcile the models with the deployed schema (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s
Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.
Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:
* 92 columns gained server_default. The models carried Python-side
`default=` only, so the ORM filled the value and the column had no
database default. Anything inserting outside the ORM behaved
differently from production.
* Eleven indexes that existed only in migrations are now declared:
the three backup_run reporting indexes, the two date-ordered
image_record browse indexes, import_task and presentation_review,
and the three task_run history indexes. All use text() for their DESC
ordering and postgresql_where for the partial one.
* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
because neither is expressible as a UniqueConstraint:
uq_tag_name_kind_fandom — an EXPRESSION index over
(name, kind, COALESCE(fandom_id, 0))
uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
IS NULL
post.py already had a comment describing the second one. The comment
was right; nothing declared it.
* The two external_link enum CHECKs (host, status) — rule 36 territory,
and absent from the model entirely.
* Two indexes were named explicitly. A bare index=True generated
ix_tag_alias_canonical_tag_id where the database has
ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
index that was already there under another name. Same for
tag_suggestion_rejection.
Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.
Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.
The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
|
||
|
bvandeusenandClaude Opus 4.8
|
af0d39ed52 |
feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474)
Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
571938781a |
feat(tagging): title-based WIP auto-tagging (#1458)
Auto-apply the `wip` system tag to posts whose TITLE explicitly declares
work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision
complement to the image-based ML `wip` head. WIP images are excluded from the
Explore/gallery browse, so honouring the artist's own label keeps unfinished
pieces out of the main browse.
- services/wip_title.py: precision-first token-anchored matcher (swipe/wiped
never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO
NOTHING, chunked under the psycopg param ceiling).
- importer: live hook on FRESH import only (never on deep-scan/supersede), so a
manually-removed WIP tag is never re-applied by a routine re-scan.
- maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep
(coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a
beat — a periodic re-run would silently undo manual removals.
- ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating
the live hook; GET/PATCH + POST /settings/wip-title/scan.
- Settings UI: toggle + "Scan existing posts" button.
- Tests: pure matcher unit tests + integration (apply idempotency, backfill
precision).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
aea2701c28 |
feat(translation): tunable acceptance floor (0.90) + per-post sticky override (#155)
The gate at a fixed 0.80 couldn't catch the real pain: Interpreter (fresh == cached, verified by probe) confidently mis-detects short ASCII English like "... WIP Part 1" as German at 0.86 — above the floor — so it was accepted and a re-translate reproduced it. Confidence alone can't separate the 0.86 collision (genuine German lands there too), and single-word mis-flags sit at a confident 1.0 no floor catches. Two operator-approved levers: - Acceptance floor is now a live Settings value (ImportSettings. translation_min_confidence, default 0.90; surfaced in the Translation card), so it's tunable without a redeploy. _accept takes the threshold as a parameter. - Per-post sticky override (Post.translation_override: auto/force/original). 'force' stores a translation even below the floor (rescue a skipped legit-foreign title); 'original' keeps the original and clears any stored translation (kill a confident mis-flag no floor catches). The sweep honors it on every run and _reset_translations skips 'original', so the choice survives a Re-translate-all. POST /api/posts/<id>/translation-override applies it immediately (translate now when the service is up, else queue for the sweep). UI: PostTranslationControl on the posts-feed card. Migration 0084 (both columns + a CHECK on the override). The feed + provenance serializers expose translation_override. With a stricter floor the rollback finally works: raise it -> Re-translate all -> the 0.86 mis-flags are rejected and restored to the original; force / keep-original handle the residual either way. Tests: gate thresholds against the param (0.86 rejected at 0.90, explicit-floor cases); sweep force/original + re-translate-skips-original; override endpoint (validation, original clears, force queues when disabled, feed exposes it); settings min_confidence default/save/validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy |
||
|
bvandeusenandClaude Opus 4.8
|
a3bc98a53c |
feat(translation): Post translation columns + settings + migration (#143 step 1)
Post gains post_title_translated / description_translated / translated_source_lang / translation_engine_version / translated_at — filled by the translate sweep so viewing is instant. ImportSettings gains translation_enabled (OFF by default), interpreter_base_url (EMPTY — no default host; the operator points it at their own Interpreter proxy behind a reverse proxy) and translation_target_lang (en), exposed + validated via /settings/import. Migration 0083. Settings defaults + patch + validation test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Opus 4.8
|
8dbf29f803 |
feat(external): per-host enable toggles in Settings (Phase 4d)
Operator lever: disable a single file host (e.g. mega.nz when it's banning) without touching the others. Five booleans on import_settings (extdl_<host>_enabled, default true — works out of the box, rule #26); the worker already reads them via getattr so no worker change. Migration 0050 + model fields + settings GET/PATCH (uniform boolean validation) + a 'External file-host downloads' card in the subscriptions Settings tab. Completes Phase 4. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
c0fd80e694 |
feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3)
Confirm-only "this post may continue this series" matcher. - series_suggestion table (post_id, series_tag_id, score, signals jsonb, status pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings knobs (series_suggest_enabled, series_suggest_threshold). - series_match_service: weighted additive score (title-stem / same-artist / page-continuity / shared-distinctive-tags), no single signal gating. The title "pattern" is derived on the fly from the post titles already in a series, so it sharpens as more are confirmed (no persisted state to drift). Candidates are bounded to the post's artist. match_post upserts pending suggestions (UNIQUE + on-conflict, respecting prior added/dismissed decisions). - accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'. - rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from a post-id cursor (maintenance_long lane), like normalize_tags_task. - API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan. - Settings: enabled + threshold exposed via /settings/import. - Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE dedup. Frontend (Suggestions tab + settings card) lands next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9d18dacbe8 |
fix(lint): UP037 — drop quotes from ImportSettings.load return annotations (py3.14 deferred annotations)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
171c486939 |
refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was repeated 15× across services, API, and 5 task modules. Added async load() + sync load_sync() classmethods on the model and migrated all 15 full-row sites (callers already imported ImportSettings, so no new imports; dropped download's now-orphaned select import). Left maintenance.py's deliberate column-select (import_scan_path only) as-is. Rest of the service layer was already adequately DRY — the Record/to_dict pattern is only 2 instances and the savepoint find-or-create recovery is correctly per-entity, so neither was forced into a shared abstraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
e43312a129 |
fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
6d67e6e987 |
fc3d: models — Source.consecutive_failures + ImportSettings scheduling knobs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
159d4cb046 |
feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9ef5e5047d |
feat(phash): import_settings.phash_threshold (migration 0006) exposed + validated in API
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
23d2fb24ac |
fix(fc2a): three unit-test failures — async fixture decorator, name convention, lazy celery includes
test_health.py: @pytest.fixture on an `async def` function is rejected by pytest-asyncio 1.x strict mode. Switched to @pytest_asyncio.fixture. backend/app/models/import_settings.py: My constraint name was 'ck_import_settings_singleton' and Base.metadata's naming convention applies 'ck_<table>_<name>' on top, so the final ORM name was 'ck_import_settings_ck_import_settings_singleton' (double prefix). The migration creates the DB constraint as 'ck_import_settings_singleton' via raw alembic, so they didn't match. Fix: bare name 'singleton' in the model → convention produces 'ck_import_settings_singleton', matching the migration's literal name. tests/test_tasks_register.py: Celery's include=[...] parameter on the constructor is lazy — task modules aren't imported until a worker boots. The test only imported the Celery instance, so the @celery.task decorators in scan.py / import_file.py / thumbnail.py never ran. Fix: explicit `import` of those modules for side-effect at the top of the test file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
9eb1614fbf |
feat(fc2a): schema migration 0002 — tag kinds + fandom hierarchy + import lifecycle
Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating), Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a kind-aware uniqueness index so the same name can exist across kinds and the same character name can exist in different fandoms. Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs. Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e populates this via the integrity verifier. Drops the unused tag.namespace column from FC-1 — superseded by kind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |