5 Commits

Include renames

Author	SHA1	Message	Date
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusen	aa28bddeab	fix(alembic 0025): qualify ambiguous post.id / post.source_id in fragment-group SELECT (post JOIN source — both have id)	2026-05-27 15:45:42 -04:00
bvandeusen	b7b313cc05	fix(alembic 0025): include HF + Discord post_url backfill (no longer 'deferred to deep-scan') ... Operator-flagged: the claim that 'a future deep-scan via the new parser will fix HF and Discord post_url' was conditional on the operator actually running a deep-scan, which they might not do for ages. Until then HF posts stay at post_url=NULL (HF sidecars have no `url` field) and Discord posts stay pointing at cdn.discordapp.com/attachments/... (the file URL, not the message permalink). The migration was already opening sidecar files for SubscribeStar. Generalizing the loop to also handle HF and Discord is a tiny addition that closes the gap without operator intervention. Per-platform Part 1 logic now: subscribestar — read sidecar.post_id, overwrite external_post_id + post_url with the derived /posts/<post_id> permalink. hentaifoundry — read sidecar.user + .index, overwrite post_url with /pictures/user/<u>/<i>. external_post_id (= index) unchanged. discord — read sidecar.server_id + .channel_id + .message_id, overwrite post_url with the discord.com/channels/.../<m> triple. external_post_id (= message_id) unchanged. Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv post_url backfill) unchanged. Posts whose related ImageRecord paths don't resolve on disk (orphan filesystem state) are reported per-platform in the migration output — those still need a future deep-scan, but the in-DB-with-on-disk-files common case is now fully covered by the migration alone.	2026-05-27 15:38:18 -04:00
bvandeusen	bd3f996582	fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms ... Audit of one sample sidecar per platform on the operator's /mnt/Data/Patreon/ archive surfaced three parser bugs that have been silently corrupting non-Patreon Posts since FC-3 shipped: 1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the per-attachment id in `id` (e.g. 711509) and the actual post id in `post_id` (e.g. 360360). FC's external_post_id chain had `id` winning, so every multi-image SubscribeStar post was fragmented into N Post rows in the database. Reorder the chain to `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no `post_id`), HF (uses `index`), Discord (uses `message_id`) all unaffected. 2. Discord `message` field not captured. Discord posts put the body in `message`, not `content`. Append it to the description fallback chain `("content", "description", "caption", "message")`. 3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New `_derive_post_url(platform, data)` helper synthesizes proper permalinks from per-platform fields: subscribestar → https://www.subscribestar.com/posts/<post_id> pixiv → https://www.pixiv.net/artworks/<id> hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index> discord → https://discord.com/channels/<server>/<channel>/<message> Patreon's bare `url` IS a real permalink and is used as-is. For the four file-URL platforms, the bare `url` is NEVER trusted: derive or return None rather than persist a CDN URL. Tests: - `test_parse_core_fields_and_id_priority` flipped to assert post_id wins over id. - New `test_parse_id_used_when_no_post_id` covers the Patreon real shape. - New `test_parse_message_used_as_description_fallback` covers Discord bodies. - Five new tests cover per-platform post_url derivation (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields → None). Cleanup migration alembic 0025_fix_subscribestar_post_ids: - For each SubscribeStar Post: find a related ImageRecord.path, walk to its sidecar JSON, read `post_id`, overwrite Post.external_post_id + post_url with the corrected values. - After all updates, every group of Posts under one source sharing the same NEW external_post_id is a fragment-set — merge to a canonical row using the same ImageProvenance pre-delete + repoint dance as alembic 0022 (banked pattern). - Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape url with the derived `/artworks/<id>` permalink. - HF and Discord post_url backfills skipped — HF would need the `user` field (not stored on Post), Discord needs server/channel triple. Both will be corrected by a deep-scan re-applying sidecars through the new parser. Idempotent: re-running on already-corrected data is a no-op.	2026-05-27 15:35:25 -04:00