4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusen	f05aaa707b	fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard ... Alembic 0032 adds Source.error_type (varchar(32), indexed). _update_source_health stamps it alongside last_error on status='error' and clears it on 'ok'. SourceRecord/to_dict exposes it. FailingSourcesCard renders a colored chip next to the consecutive- failures count, with a tooltip explaining the suggested operator action. Color reflects intent: - warning (yellow) — operator action needed (auth_error) - info (blue) — backend-paced (rate_limited / timeout / network_error / partial / tier_limited) - error (red) — likely terminal without intervention (not_found / access_denied / validation_failed / unsupported_url / http_error / unknown_error) Audit 2026-06-02: the backend computed 13 ErrorType categories but only the free-text last_error reached the operator. Bulk-triage by class ("all auth_error → rotate cookies", "12 rate_limited → just wait") required opening Logs per row.	2026-06-02 17:54:44 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00
bvandeusenandClaude Opus 4.7	6d67e6e987	fc3d: models — Source.consecutive_failures + ImportSettings scheduling knobs ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 15:58:42 -04:00
bvandeusenandClaude Opus 4.7	d6a156dcd2	feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration ... Implements the data model from spec §3 in one go so FC-2/FC-3 don't need schema-adding migrations of their own. Artist is the unified entity for both gallery 'artist:' tags and GallerySubscriber Subscriptions (is_subscription flag). ImageProvenance is many-to-one, enabling the enrich-on-duplicate rule for downloaded content that pHash-matches an existing record. The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m; swapping models in FC-2 will require a column-width migration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:34:29 -04:00