Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX #44

Merged

bvandeusen merged 9 commits from dev into main 2026-06-01 20:44:24 -04:00

Conversation 0 Commits 9 Files Changed 40

+1099 -217

bvandeusen commented 2026-06-01 16:51:27 -04:00

Owner

Copy Link

Copy Source

Summary

Five thematically distinct batches landed on dev since the last main cut.
All five are CI-green; head is 717b601.

1. Nullable Post.source_id + denormalized Post.artist_id (alembic 0030)

• FK becomes nullable, ondelete=SET NULL. Post.artist_id denormalized as NOT NULL FK CASCADE.
• Partial unique index on (artist_id, external_post_id) WHERE source_id IS NULL keeps the post-uniqueness contract for filesystem-imported content.
• Drops the sidecar:<platform>:<slug> synthetic Source rows that used to satisfy the old NOT NULL.
• _lookup_source_for_sidecar returns the real Source when a subscription exists, otherwise None. The Post just sits null-source.
• Services rewritten (post_feed, artist, provenance, gallery): artist filters go through Post.artist_id; Source is now LEFT JOIN'd; dict surfaces emit source: null when absent.
• Gallery EXISTS subquery aliases Post to avoid SQLAlchemy correlating to the outer primary_post_id join.
• Frontend renders e.source?.platform ?? 'filesystem import'.

2. Tick/backfill download modes + PARTIAL classifier (Scribe plan #544, alembic 0031)

• New Source.backfill_runs_remaining counter. When > 0, downloads use gallery-dl's skip: True + 1800s timeout (walk full history); when 0, downloads use skip: "exit:20" + the default 870s (catch-up mode, exits gallery-dl after 20 contiguous archived items).
• Auto-decrements per run, auto-resets to 0 on clean exit + zero files (queue drained).
• New POST /api/sources/{id}/backfill arms a source (1..10 runs, default 3). "Deep scan" button + remaining-count chip on each SourceRow.
• New ErrorType.PARTIAL classifies runs that downloaded ≥1 file but didn't finish in budget. Downstream maps to status="ok" so timeout-mid-walk no longer flips events red.

3. Newly added enabled sources start in backfill mode

• SourceService.create() pre-arms backfill_runs_remaining=3 when enabled=True. Disabled-on-create sources (sidecar synthetics, deliberately-disabled adds) stay at 0.
• Saves the operator from having to click Deep scan manually on every new subscription.

4. Mux video Referer/Origin forwarding for yt-dlp

• DaferQ patreon failed every video because yt-dlp's m3u8 fetch got 403'd by Mux's JWT playback restriction (Referer/Origin check, separate from token validity).
• Static downloader.ytdl.raw-options.http_headers block in _get_default_config now pins Referer: https://www.patreon.com/ + Origin: https://www.patreon.com so yt-dlp's manifest fetch clears Mux's policy.
• IP-range restrictions remain unfixable from our worker, but PARTIAL classifier (#2) already labels the residual case as status="ok".

5. Failing-sources UX

• FailingSourcesCard rows get zebra striping + hover highlight. Operator-flagged: with 17 rows, eye couldn't trace from artist name on the left to Logs/Retry buttons on the right.

Test plan

• CI green on dev (717b601) — six jobs, all passing
• Alembic 0030 + 0031 apply cleanly
• Operator confirms post-merge:
  • No phantom sidecar:* sources in Subscriptions UI
  • Artist-filtered gallery still returns filesystem-imported images
  • New subscription gets Deep scan chip + walks full history on first poll
  • DaferQ video posts no longer fail with 403
  • Failing-sources panel hover/zebra makes Logs button targeting easier

🤖 Generated with Claude Code

## Summary Five thematically distinct batches landed on `dev` since the last main cut. All five are CI-green; head is `717b601`. ### 1. Nullable `Post.source_id` + denormalized `Post.artist_id` (alembic 0030) - FK becomes nullable, `ondelete=SET NULL`. `Post.artist_id` denormalized as NOT NULL FK CASCADE. - Partial unique index on `(artist_id, external_post_id) WHERE source_id IS NULL` keeps the post-uniqueness contract for filesystem-imported content. - Drops the `sidecar:<platform>:<slug>` synthetic Source rows that used to satisfy the old NOT NULL. - `_lookup_source_for_sidecar` returns the real Source when a subscription exists, otherwise None. The Post just sits null-source. - Services rewritten (post_feed, artist, provenance, gallery): artist filters go through `Post.artist_id`; `Source` is now LEFT JOIN'd; dict surfaces emit `source: null` when absent. - Gallery EXISTS subquery aliases `Post` to avoid SQLAlchemy correlating to the outer `primary_post_id` join. - Frontend renders `e.source?.platform ?? 'filesystem import'`. ### 2. Tick/backfill download modes + PARTIAL classifier (Scribe plan #544, alembic 0031) - New `Source.backfill_runs_remaining` counter. When `> 0`, downloads use gallery-dl's `skip: True` + 1800s timeout (walk full history); when `0`, downloads use `skip: "exit:20"` + the default 870s (catch-up mode, exits gallery-dl after 20 contiguous archived items). - Auto-decrements per run, auto-resets to 0 on clean exit + zero files (queue drained). - New `POST /api/sources/{id}/backfill` arms a source (1..10 runs, default 3). "Deep scan" button + remaining-count chip on each `SourceRow`. - New `ErrorType.PARTIAL` classifies runs that downloaded ≥1 file but didn't finish in budget. Downstream maps to `status="ok"` so timeout-mid-walk no longer flips events red. ### 3. Newly added enabled sources start in backfill mode - `SourceService.create()` pre-arms `backfill_runs_remaining=3` when `enabled=True`. Disabled-on-create sources (sidecar synthetics, deliberately-disabled adds) stay at 0. - Saves the operator from having to click Deep scan manually on every new subscription. ### 4. Mux video Referer/Origin forwarding for yt-dlp - DaferQ patreon failed every video because yt-dlp's m3u8 fetch got 403'd by Mux's JWT playback restriction (Referer/Origin check, separate from token validity). - Static `downloader.ytdl.raw-options.http_headers` block in `_get_default_config` now pins `Referer: https://www.patreon.com/` + `Origin: https://www.patreon.com` so yt-dlp's manifest fetch clears Mux's policy. - IP-range restrictions remain unfixable from our worker, but PARTIAL classifier (#2) already labels the residual case as `status="ok"`. ### 5. Failing-sources UX - `FailingSourcesCard` rows get zebra striping + hover highlight. Operator-flagged: with 17 rows, eye couldn't trace from artist name on the left to Logs/Retry buttons on the right. ## Test plan - [x] CI green on dev (`717b601`) — six jobs, all passing - [x] Alembic 0030 + 0031 apply cleanly - [ ] Operator confirms post-merge: - No phantom `sidecar:*` sources in Subscriptions UI - Artist-filtered gallery still returns filesystem-imported images - New subscription gets Deep scan chip + walks full history on first poll - DaferQ video posts no longer fail with 403 - Failing-sources panel hover/zebra makes Logs button targeting easier 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 4 commits 2026-06-01 16:51:27 -04:00

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics ... 2f66de2928

Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.

fix(lint): drop unused Source import after Post.artist_id refactor ff35da4743

fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention 644d538bab

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source ... c9089b1d03

gallery_service._provenance_clause artist branch was correlating its bare
Post reference to the outer query's primary_post_id outer-join, so the
artist filter silently matched zero rows for images with no primary post.
Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner
FROM rather than treating it as a correlated outer table.

Five sidecar/import tests still asserted that a synthetic Source row
appears after a filesystem import. Alembic 0030 retired that behavior;
the Post sits null-source and the artist linkage lives on Post.artist_id.
Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent,
test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar,
and test_apply_sidecar_recovers_from_integrity_error to assert
post.source_id IS NULL + post.artist_id linkage instead.

bvandeusen added 1 commit 2026-06-01 18:23:31 -04:00

feat(download): tick/backfill modes + partial-success classifier (plan #544) ... 19aece1fc4

Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.

Two coupled changes, both operator-flagged 2026-06-01:

* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
  asks gallery-dl to exit after 20 contiguous archived items. Fresh
  subscriptions + new-content cases still walk normally; established
  subscription with zero new content exits in ~30s of HEAD requests
  instead of pegging the timeout. 20 (not 5) gives headroom against
  paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
  via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
  downloads use `skip: True` + 1800s timeout. Auto-decrements per run
  with early-reset to 0 when a clean run finds zero files (queue
  drained). N defaults to 3 — multiple runs give the system enough
  budget to finish a deep walk across timeout boundaries. New
  `POST /api/sources/{id}/backfill` arms the source; "Deep scan"
  button on each SourceRow (chip shows remaining count) wires it.

Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."

Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).

Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)

bvandeusen added 1 commit 2026-06-01 19:00:52 -04:00

fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos ... 66ff671f09

Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.

gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.

Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.

Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.

bvandeusen added 1 commit 2026-06-01 20:02:20 -04:00

feat(subscriptions): newly added enabled sources start in backfill mode ... 2aa2002f22

A freshly created subscription has no gallery-dl archive yet, so the
first poll in tick mode would walk forever — exit:20 doesn't trip
until 20 contiguous archive-hits, and there are none. The wall-clock
cap kicks in mid-walk, and the partial-success classifier (plan #544)
gracefully labels it status=ok, but the operator still wonders why
their new subscription isn't grabbing the back-catalog.

Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`.
Same default as the manual "Deep scan" button, same auto-decrement
and auto-reset rules — once the queue drains (clean exit + zero new
files) or the budget runs out, tick mode resumes naturally.

Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'`
that arrive disabled = False, or operator-added sources that start
disabled deliberately) skip the pre-arm — they're never polled, no
budget to waste.

bvandeusen added 1 commit 2026-06-01 20:12:57 -04:00

fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create ... cfa4fb4084

`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial
value (0) when checking that the override works. Now that create() pre-arms
enabled sources to 3, the assertion is stale — the test was already
verifying the override path, the pre-assertion just decorated it.

Drop the pre-assertion; keep the override check. Other tests use raw
Source(...) constructors (default to 0 via the column server_default),
not SourceService.create(), so they're unaffected.

bvandeusen added 1 commit 2026-06-01 20:14:50 -04:00

ux(failing-sources): zebra striping + hover highlight on rows ... 717b601c81

Operator-flagged 2026-06-01: with the failing-sources panel expanded,
all rows have the same flat low-opacity background, no visual track
from the artist name on the left to the Logs/Retry buttons on the
right. Hard to tell which row a button belongs to when scanning down
a list of 17.

Three pure-CSS changes (no DOM, no logic):
- 3px gap between rows (was 2px) — slightly more breathing room
- Even rows get a darker surface tint (zebra striping)
- Hover paints the whole row in a low-opacity error tint, so moving
  the cursor toward Logs/Retry lights up the whole horizontal band
  and the eye traces back to the artist name automatically

bvandeusen changed title from Nullable Post.source_id + denormalized Post.artist_id (retire sidecar synthetics) to Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX 2026-06-01 20:33:35 -04:00

bvandeusen merged commit c82fb308b6 into main 2026-06-01 20:44:24 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#44