Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.
## Schema (alembic 0030)
- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
from source.artist_id in the migration. Indexed for the artist-filter
queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
NULL. Deleting a Source detaches its Posts instead of destroying
archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
(artist_id, external_post_id) WHERE source_id IS NULL — guards
filesystem-import dedup since the existing source-bound unique
ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
image_provenance first, then DELETE FROM source WHERE url LIKE
'sidecar:%'. The Dymkens cleanup.
## Model + service changes
- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
`_lookup_source_for_sidecar` (returns None when no subscription).
`_find_or_create_post` takes required `artist_id`; matches on
(source_id, external_post_id) for source-bound posts or
(artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
Source so NULL-source posts surface); artist_service date_row/
activity/post_count; provenance_service.for_image/for_post (LEFT
JOIN); gallery_service._provenance_exists_where_artist via
Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
NULL-source rows.
## Frontend
- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
?? 'filesystem import'` so NULL-source posts get a clear
"filesystem import" affordance instead of a NaN crash.
## Tests
- `test_importer_upsert_helpers`: removed the four synthetic-anchor
tests; added `_find_or_create_post_idempotent_with_null_source`
(dedup via the partial unique index) and
`_lookup_source_for_sidecar_returns_*` (existing-subscription +
none cases). The existing `_find_or_create_post_idempotent` now
also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
gains `artist_id=<artist>.id`. The `_seed_post` helper in
`test_post_feed_service` looks up artist_id from the source row so
callsites stay one-arg.
## Verification on deploy
After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
posts; PostCard same.
## Out of scope
- UI to manage/delete orphan NULL-source Posts. Data model is right;
UI follows if operator wants it.
Four coupled operator-asked changes to the view modal (Scribe plan #509):
1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
in onMounted/nextTick so the caret is in the input the moment the
modal renders. No click needed to start typing.
2. **General suggestions expanded by default** — SuggestionsPanel's
general-category group now mounts with `:default-open="true"`.
Operator can collapse if too noisy, but the v1 frame shows them.
3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
suggestion_threshold_general default matches character. Alembic
0029 also bumps the existing singleton row's value if it's still
at the old 0.95. Operator can re-tune from Settings → ML.
4. **Retire `copyright` + `artist` as ML suggestion categories** —
neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
really existed as a copyright tag-kind). They were surfaced in the
suggestions pipeline + threshold settings UI but had no follow-
through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
MLThresholdSliders rows.
Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.
Tests:
- test_surfaced_categories asserts {character, general}, excludes
artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
artist + copyright keys in the GET payload.
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:
1. `SourceService.list()` returned every Source, no filter on URL.
alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
one canonical row per (artist, platform); when no real campaign URL
was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
enabled=false as a disabled anchor. The UI then listed those
anchors as if they were polls — disabled, but visible. Fix: `list()`
excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
opts back in for admin tooling.
2. `importer._source_for_sidecar` picked the lowest-id Source for
(artist, platform). When alembic 0022 had rewritten a per-post row
into a synthetic anchor (lower id) AND the operator later added the
real subscription (higher id), every gallery-dl download silently
attached its Post to the SYNTHETIC instead of the real Source. Fix:
prefer a non-`sidecar:%` URL when one exists; fall back to the
synthetic; only create a new synthetic when nothing exists for
(artist, platform).
alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
Operator-flagged: the claim that 'a future deep-scan via the new parser
will fix HF and Discord post_url' was conditional on the operator
actually running a deep-scan, which they might not do for ages. Until
then HF posts stay at post_url=NULL (HF sidecars have no `url` field)
and Discord posts stay pointing at cdn.discordapp.com/attachments/...
(the file URL, not the message permalink).
The migration was already opening sidecar files for SubscribeStar.
Generalizing the loop to also handle HF and Discord is a tiny addition
that closes the gap without operator intervention.
Per-platform Part 1 logic now:
subscribestar — read sidecar.post_id, overwrite external_post_id +
post_url with the derived /posts/<post_id> permalink.
hentaifoundry — read sidecar.user + .index, overwrite post_url with
/pictures/user/<u>/<i>. external_post_id (= index) unchanged.
discord — read sidecar.server_id + .channel_id + .message_id,
overwrite post_url with the discord.com/channels/.../<m> triple.
external_post_id (= message_id) unchanged.
Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv
post_url backfill) unchanged.
Posts whose related ImageRecord paths don't resolve on disk (orphan
filesystem state) are reported per-platform in the migration output —
those still need a future deep-scan, but the in-DB-with-on-disk-files
common case is now fully covered by the migration alone.
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. Confirmed against the operator's
/mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every
post's JSON has `title: ""` and a content like
`<div>Lets say hello to you guys with my Belle <br><br><br></div>`.
FC's sidecar parser, treating empty strings as missing, had been leaving
post_title NULL on every subscribestar post since FC-3 shipped.
Fix at two layers:
1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)`
helper strips HTML tags, collapses whitespace, returns the first
non-empty line truncated to 120 chars with ellipsis. `parse_sidecar`
now falls back to this when `title` resolves to None and a
`content`/`description`/`caption` value is present. Patreon's
non-empty titles short-circuit the fallback so existing behavior is
unchanged. Four new tests in test_sidecar_util.py pin: derivation
from content, truncation at 120 chars, explicit-title precedence,
no-content no-fallback.
2. `alembic 0024_backfill_post_title_from_description` — backfills the
same logic across existing Post rows where `post_title IS NULL OR
post_title = ''` AND description is present. Idempotent (re-running
is a no-op once titles are populated). Downgrade is a no-op since
there's no safe way to tell derived rows from genuine ones.
After deploy + migration: subscribestar posts will surface a meaningful
title in PostCard, post feed search, etc.
extension/manifest.json: add content_security_policy.extension_pages = "script-src 'self'; object-src 'self';" — explicitly omits the upgrade-insecure-requests directive that MV3 inherits by default. Without this, every fetch(http://curator.../...) silently upgrades to https:// at the browser layer (Sec-Fetch-Site=same-origin, NS_ERROR_GENERATE_FAILURE), regardless of about:config. Bump XPI version 1.0.3 → 1.0.4 so a fresh signed build replaces the cached one. Operator-troubleshot 2026-05-26 via Inspect-the-extension dev tools showing the silent scheme upgrade.
alembic 0023: drop ck_tag_fandom_requires_character before the tag_kind type swap and recreate after. Postgres can't resolve `kind = 'character'` across the rename (column on tag_kind_old, literal binds to new tag_kind → "operator does not exist"). Same dance on downgrade. Banked under reference_tag_kind_enum_swap_check_drop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating),
Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a
kind-aware uniqueness index so the same name can exist across kinds and the same
character name can exist in different fandoms.
Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a
single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs.
Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e
populates this via the integrity verifier.
Drops the unused tag.namespace column from FC-1 — superseded by kind.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the data model from spec §3 in one go so FC-2/FC-3 don't need
schema-adding migrations of their own. Artist is the unified entity for
both gallery 'artist:' tags and GallerySubscriber Subscriptions
(is_subscription flag). ImageProvenance is many-to-one, enabling the
enrich-on-duplicate rule for downloaded content that pHash-matches an
existing record.
The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m;
swapping models in FC-2 will require a column-width migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Configures stable constraint naming so autogeneration produces clean diffs.
Alembic uses the sync psycopg driver while the runtime app uses asyncpg.
Also fixes a .gitignore bug caught during this task: the bare 'models/'
rule for the ML weights volume was matching backend/app/models/ (Python
package). Anchored all volume rules to repo root (/images/, /import/,
/downloads/, /models/, /postgres_data/, /redis_data/).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen