3 Commits

Include renames

Author	SHA1	Message	Date
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00