Commit Graph

25 Commits

Author SHA1 Message Date
bvandeusen 0f7cd3cb76 fix(migration-0022): pre-merge colliding Posts before the bulk reparent — Postgres fires uq_post_source_external_id row-by-row during UPDATE, so the post-reparent merge-collisions step never ran (operator's v26.05.26.1 deploy hit it: 'duplicate key (source_id, external_post_id)=(42, 6166997)'). Detect (keep, drop) Post pairs whose external_post_id already exists under canonical, merge the drop into keep, then bulk-reparent the rest cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:48:49 -04:00
bvandeusen 0bc5767a2b fix(importer): Source = one per (artist, platform), not one per post — filesystem importer's sidecar paths now reuse the artist's existing subscription Source (or create one synthetic anchor with enabled=False) instead of fabricating a new Source per post URL. Alembic 0022 consolidates existing per-post Sources to canonical (prefers campaign URL; falls back to sidecar:<platform>:<slug>) and re-parents Posts + ImageProvenance, merging Post collisions.
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:19:29 -04:00
bvandeusen 397021dcbd fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 13:47:55 -04:00
bvandeusen 929d3fc092 feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:17:42 -04:00
bvandeusen e6ededbe8e feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:16:26 -04:00
bvandeusen e43312a129 fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:00 -04:00
bvandeusen 8f2732a56f fc3h: alembic 0017 — backup_run table with indexes + partial-tag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:51:22 -04:00
bvandeusen 0cda46fcdb fc3i: alembic 0016 — task_run table with single + composite indexes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:00:46 -04:00
bvandeusen da9c19b2dc fc5: migration 0015 + MigrationRun model (kind/status as String(32), API-validated)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 08:11:42 -04:00
bvandeusen 5f2ebd66b7 fc3d: migration 0014 — source.consecutive_failures + import_settings scheduling knobs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 15:58:30 -04:00
bvandeusen 159d4cb046 feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:36:19 -04:00
bvandeusen 2a18016b69 feat(fc3b): migration 0012 + AppSetting model
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:33:14 -04:00
bvandeusen e623e97be2 feat(fc3b): migration 0011 — credential schema aligned with GS wire fields
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:32:52 -04:00
bvandeusen a13a5878eb feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 12:53:34 -04:00
bvandeusen 8f69478227 feat(attachments): PostAttachment model + 0009 (table + batch counter)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 11:10:53 -04:00
bvandeusen 63e7185811 feat(provenance): image_record.artist_id + 0008 backfill/drop-artist-tags
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 21:33:11 -04:00
bvandeusen 15dac50367 feat(post): add nullable description + attachment_count (migration 0007)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 09:38:43 -04:00
bvandeusen 9ef5e5047d feat(phash): import_settings.phash_threshold (migration 0006) exposed + validated in API
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 22:04:13 -04:00
bvandeusen b83baa7d7f feat(series): SeriesPage model + 0005 migration (unique image, ordered pages)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:33:12 -04:00
bvandeusen a638c469e0 feat(fc2c-i): migration 0004 enables tsm_system_rows 2026-05-15 15:48:29 -04:00
bvandeusen 906804140c feat(fc2b): schema migration 0003 — ML pipeline tables
Renames image_record.wd14_* -> tagger_* (we're on Camie now, not WD14).
Adds tag_allowlist (auto-apply opt-in, per-tag confidence),
tag_suggestion_rejection (per-image dismissals), tag_alias (composite
(string, category) -> canonical tag, resolved at read time),
tag_reference_embedding (per-tag SigLIP centroids), and the ml_settings
singleton (per-category + centroid thresholds, model version pins).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 07:35:22 -04:00
bvandeusen 80a5690740 fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig)
Ruff:
The remaining I001 errors came from ruff treating `alembic` as a first-
party module (because the alembic/ directory exists in the repo root)
rather than third-party. Ran `ruff check --fix` locally — auto-sorted
import groupings to put alembic/sqlalchemy alongside backend.* as first-
party, and trimmed redundant blank lines after a few import blocks.

Frontend:
`npm run check` (vue-tsc --noEmit) was failing because vue-tsc has no
tsconfig.json to read against, and the frontend is pure JS without
JSDoc annotations — vue-tsc had nothing to do. Skipping the step until
we add a tsconfig + convert to TS or add JSDoc annotations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:24:25 -04:00
bvandeusen 9eb1614fbf feat(fc2a): schema migration 0002 — tag kinds + fandom hierarchy + import lifecycle
Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating),
Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a
kind-aware uniqueness index so the same name can exist across kinds and the same
character name can exist in different fandoms.

Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a
single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs.

Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e
populates this via the integrity verifier.

Drops the unused tag.namespace column from FC-1 — superseded by kind.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 12:03:41 -04:00
bvandeusen d6a156dcd2 feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration
Implements the data model from spec §3 in one go so FC-2/FC-3 don't need
schema-adding migrations of their own. Artist is the unified entity for
both gallery 'artist:' tags and GallerySubscriber Subscriptions
(is_subscription flag). ImageProvenance is many-to-one, enabling the
enrich-on-duplicate rule for downloaded content that pHash-matches an
existing record.

The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m;
swapping models in FC-2 will require a column-width migration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 07:34:29 -04:00
bvandeusen a03655039c feat: add SQLAlchemy declarative base, Alembic environment, and gitignore fix
Configures stable constraint naming so autogeneration produces clean diffs.
Alembic uses the sync psycopg driver while the runtime app uses asyncpg.

Also fixes a .gitignore bug caught during this task: the bare 'models/'
rule for the ML weights volume was matching backend/app/models/ (Python
package). Anchored all volume rules to repo root (/images/, /import/,
/downloads/, /models/, /postgres_data/, /redis_data/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 07:32:58 -04:00