Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag
firing on one frame survived at peak confidence, and a fixed count under-samples
long multi-scene videos so real scene-local tags looked like noise.
Redesign (operator-steered):
- Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds`
(default 4) across the 5–95% window — so a tag's frame-presence reflects real
screen time independent of video length. Capped at `video_max_frames` (default
64): a long video stretches the spacing instead of exploding into hundreds of
inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg
timeout also cut 60s→30s).
- Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in
>= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on
screen — duration-independent noise rejection), with confidence = MEAN over the
frames it appears in (not max). Clamps the threshold to the sample count so a
1–2-frame short video still tags.
- All three knobs are DB-backed ml_settings (migration 0053), patchable via
/api/ml/settings + sliders in the ML settings card — replaces the
VIDEO_ML_FRAMES env var (product-not-project).
Tests: aggregation drops one-frame noise + means corroborated tags + clamps on
short videos; settings round-trip + min>max validation. Replaced the
_maxpool_predictions unit test.
NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs
CPU-only — is GPU enablement, tracked separately in #872.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.
Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.
- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.
Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).
Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):
- utils.paths.filehash_from_url — one source of truth for the extractor;
patreon_client._filehash now delegates so capture- and render-time
hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
every inline image whose filehash maps to a stored image of THIS
artist; unmatched / pre-Phase-2 images keep hotlinking.
Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.
Completes Phase 4. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.
- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
/attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
(_apply_sidecar + upsert_post_record) so a re-import never resets a link's
status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
no-dup on reimport), serializer.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add-from-post no longer appends straight into the run — it STAGES the post's
pages as pending (per-page status; page_number NULL), grouped by source post,
so the operator drops junk (text-free alts, bumpers) and places the keepers
into the sequence with clean series-global numbering.
- migration 0048: series_page.status ('placed' default | 'pending') + nullable
page_number.
- series_service: placed/pending split everywhere (list_pages returns the
placed run + a `pending` section grouped by source post; reorder/cover/
list_series operate on placed only); add_post stages pending; new
place_pending(image_ids, before_image_id=None) flips pending→placed spliced
before a page (or appended) and renumbers; junk removal reuses remove_images.
- api/tags: /add-post now returns staged count; new POST /series/<id>/pending/
place.
- frontend: PostSeriesMenu navigates to the series after staging; seriesManage
store surfaces `pending` + placePending; SeriesManageView gains a pending
tray (per-post groups, place-all / place-one / drop-junk).
- tests: pending staging, place (append + insert-before), ignore-already-
placed, drop-junk, route guard; updated add_post + match-accept expectations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframe a series from "ordered chapters that own pages" to ONE flat,
series-global ordered run of pages with optional cosmetic chapter DIVIDERS
over it. A chapter no longer wraps content — it's a labeled divider anchored
to the page that begins it; a page's chapter is derived as the nearest
preceding divider. This is what lets installments assembled from multiple
sources sit in one continuous, correctly-numbered sequence (operator's
Goblin Juice case).
- migration 0047: flatten each series to a series-global page_number
(preserving today's reading order); convert each existing chapter to a
divider anchored at its first page (keeping title/stated_part); drop
series_page.chapter_id; reshape series_chapter (anchor_page_id UNIQUE FK,
drop chapter_number/is_placeholder/stated_page_start/end). Loss-safe for
content; drops empty placeholder chapters + a redundant page-1 divider.
- series_page: page_number is now the series-global order; no chapter_id.
- series_chapter: anchored divider (anchor_page_id, title, stated_part).
- series_service: flat list_pages (one run + derived dividers + per-page
source_post + part_gaps), series-wide reorder/renumber, divider CRUD
(create/update/move/delete); retired per-chapter reorder/merge/placement.
- api/tags: drop chapter_id from add; /chapters endpoints are divider
create/update/delete (removed chapter reorder/merge/page-reorder).
- series_match_service: series "end" reads max(series_page.stated_page);
accept appends via add_post. tag_service series-merge appends src's pages
after tgt's max so the merged series stays one clean run.
- frontend: seriesManage store + SeriesManageView → one continuous
drag-reorder grid with inline divider bars + series-global page numbers;
reader walks the flat run, headings from dividers; PostSeriesMenu copy.
- tests reworked across the series suite for the divider model.
Phase 2 (pending staging for add-from-post) is separate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize tagger predictions out of the image_record.tagger_predictions JSON
blob into a queryable per-prediction table. Step 1 of the cutover (expand):
additive + low-risk — reads still use the JSON, this just adds the table and
keeps it populated.
- ImagePrediction(image_record_id, raw_name, category, score) — stores the
RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution
is unchanged. Indexed for per-image reads + by (raw_name, score).
- Migration 0045: create table + set-based backfill from the JSON via
json_each (fast post-#764-prune). The old column stays (vestigial) and is
dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the
hot image_record table, so it waits for a quiesced-worker window.
- tag_and_embed dual-writes the rows (delete-then-insert, idempotent);
tagger_store_floor already applied in infer().
Next: switch suggestion + allowlist reads to the table, then drop the JSON
write. Plan-task #768.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default
0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70).
Storing every tag down to 0.05 from a ~10k-tag tagger is what grew
image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70
and the centroid/learned path covers lower-confidence preferred tags, so the
sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim
land next; this only changes the write gate for NEW imports).
- ml_settings.tagger_store_floor (migration 0044, default 0.70)
- tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor
- ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold
below the floor (nothing below the floor is stored, so the gap surfaces
nothing) — server backstop for the UI slider clamp
- Settings → ML: store-floor slider + caption; category sliders min-bound to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator evidence 2026-06-07: 0.95 was too strict, skipping confident-enough
auto-applications of accepted tags. Newly-accepted tags now allowlist at 0.90;
existing entries keep their stored value and per-tag thresholds stay tunable in
the allowlist table. No migration — min_confidence has no DB server_default, so
the Python insert default governs new rows only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator feedback: thumbnails too small to judge order, no obvious way to mark
'this installment is Part 2', and the permanent two-pane picker was busy and
competed with the ordering work.
- Full-width parts, each a card with a big page grid (150px, contain so whole
pages are visible) and drag-to-reorder; positional page number as a badge.
- Editable Part # (hero field) backed by new series_chapter.stated_part —
separate from the auto-managed chapter_number, mirroring the page_number vs
stated_page split so reorder/delete renumbering can't wipe a hand-set part.
Missing-Part hints when consecutive parts' stated_part jump >1.
- Each part labels its source post (derived from pages' primary_post_id) and
shows the printed-page range with clear labels.
- Picker demoted to an on-demand right slide-over ('Add pages') with a target-
part selector; part actions (move/merge/delete) collapsed into an overflow ⋮.
alembic 0042 adds series_chapter.stated_part (nullable int).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirm-only "this post may continue this series" matcher.
- series_suggestion table (post_id, series_tag_id, score, signals jsonb, status
pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings
knobs (series_suggest_enabled, series_suggest_threshold).
- series_match_service: weighted additive score (title-stem / same-artist /
page-continuity / shared-distinctive-tags), no single signal gating. The title
"pattern" is derived on the fly from the post titles already in a series, so it
sharpens as more are confirmed (no persisted state to drift). Candidates are
bounded to the post's artist. match_post upserts pending suggestions (UNIQUE +
on-conflict, respecting prior added/dismissed decisions).
- accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'.
- rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from
a post-id cursor (maintenance_long lane), like normalize_tags_task.
- API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan.
- Settings: enabled + threshold exposed via /settings/import.
- Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE
dedup.
Frontend (Suggestions tab + settings card) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an ordered chapter layer to series. Reading order becomes
(series_chapter.chapter_number, series_page.page_number); a chapter may be a
placeholder reserving a slot, and carries an optional parsed stated-page range
used to flag missing-page gaps. An image still lives in at most one series ⇒ one
chapter (image_id stays UNIQUE).
- models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) +
stated_page. Migration 0040 backfills every existing series into one
auto-chapter holding its current flat pages — no data loss.
- SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter
assignment, reorder_pages, chapter-aware set_cover; list_pages now returns
chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide
reorder operates on the single default chapter and rejects multi-chapter series.
- API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an
optional chapter_id.
- TagService.merge now repoints series_chapter too, so a merged series' chapters
(and their pages) survive the source tag's deletion instead of cascading away.
- Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to
supply chapter_id.
Frontend (chapter-aware manage view + reader) lands next; until then the
existing UI keeps working via the flat pages[] + single default chapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux,
persistently-corrupt bytes) used to re-error forever and re-burn chunks.
New `patreon_failed_media` table (alembic 0038, chains 0037) records
per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the
ingester skips it on routine tick/backfill walks (tier-1.5, folded into the
seen/skip predicate). Recovery BYPASSES it (the operator's "try everything
again" re-attempts dead media). A clean download clears the row (recovered);
errors/quarantines upsert-increment it. Surfaced as
run_stats.dead_lettered_count.
- New PatreonFailedMedia model + migration; ingester _dead_keys /
_record_failures (on_conflict increment) / _clear_failures.
- skip = seen | dead (empty in recovery); failures recorded post-fetch on
short sessions (same pattern as the seen-ledger).
Tests: a media erroring 3× is dead-lettered + skipped (no download attempt);
recovery re-attempts a dead media and clears it on success; a clean download
clears a sub-threshold failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id,
filehash) — our own queryable replacement for gallery-dl's archive.sqlite3.
Routine walks skip seen media; recovery mode bypasses the ledger. filehash is
a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic
0037 (← 0036). Integration test covers dedup + savepoint recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gallery cursored on COALESCE(post.post_date, image_record.created_at)
across the Post outer join — an expression spanning two tables that no
index can serve, so every /scroll sorted a large slice of the library
(and the old frontend fired ten serially). Materialize it:
- image_record.effective_date column + ix_image_record_effective_date
(effective_date DESC, id DESC); alembic 0035 backfills
COALESCE(primary post's post_date, created_at) for existing rows.
- gallery_service._effective_date_col() now returns the column, so scroll
/ timeline / jump / neighbors all order off the index instead of
re-deriving the COALESCE. _neighbors reads record.effective_date
directly (drops an extra Post lookup).
- importer._apply_sidecar maintains it: when a primary post with a date is
linked, effective_date = post.post_date; plain inserts keep the
created_at-equivalent server default.
Tests: sidecar import asserts effective_date == post.post_date; gallery
ordering/timeline/jump test seeds set effective_date alongside created_at.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-artist "+N" accent pill on the artists directory and a "N new since
last visit" banner inside ArtistView. Counts new IMAGES (not posts) so
multi-image posts increment correctly.
- alembic 0034: artist_visit (artist_id PK, last_viewed_at NOT NULL).
Seeds every existing artist with last_viewed_at=NOW() so the badge
starts at 0 across the board — no noisy "5000 unseen images" on
first deploy.
- ArtistService.find_or_create autoseeds a visit row alongside new
artists, so freshly imported content doesn't read as unseen.
- ArtistService.overview reads pre-visit last_viewed_at, counts images
created since, then atomically UPSERTs last_viewed_at=NOW() via
postgres ON CONFLICT DO UPDATE (no SELECT-then-INSERT race per
reference_scalar_one_or_none_duplicates). Returns the pre-update
count as `unseen_count_at_visit` so the banner has data.
- ArtistDirectoryService.list_artists adds an `unseen_count` aggregate
to each card via LEFT JOIN artist_visit + conditional COUNT. NULL
last_viewed_at (artist created before this code shipped) defensively
counts as "never visited" → all images unseen.
- Frontend: ArtistCard renders an accent pill in the preview-strip
corner when unseen_count > 0 (capped at 99+); ArtistView shows a
closable v-alert banner on initial load when
unseen_count_at_visit > 0, re-arms on slug change.
Single-row-per-artist (no user_id) — rule #47 multi-user ACL is
aspirational; widens to (user_id, artist_id) PK when User lands, per
rule #22.
Scribe plan #597.
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).
Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.
Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.
FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
- warning (yellow) — operator action needed (auth_error)
- info (blue) — backend-paced (rate_limited / timeout /
network_error / partial / tier_limited)
- error (red) — likely terminal without intervention
(not_found / access_denied / validation_failed /
unsupported_url / http_error / unknown_error)
Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.
## Schema (alembic 0030)
- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
from source.artist_id in the migration. Indexed for the artist-filter
queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
NULL. Deleting a Source detaches its Posts instead of destroying
archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
(artist_id, external_post_id) WHERE source_id IS NULL — guards
filesystem-import dedup since the existing source-bound unique
ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
image_provenance first, then DELETE FROM source WHERE url LIKE
'sidecar:%'. The Dymkens cleanup.
## Model + service changes
- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
`_lookup_source_for_sidecar` (returns None when no subscription).
`_find_or_create_post` takes required `artist_id`; matches on
(source_id, external_post_id) for source-bound posts or
(artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
Source so NULL-source posts surface); artist_service date_row/
activity/post_count; provenance_service.for_image/for_post (LEFT
JOIN); gallery_service._provenance_exists_where_artist via
Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
NULL-source rows.
## Frontend
- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
?? 'filesystem import'` so NULL-source posts get a clear
"filesystem import" affordance instead of a NaN crash.
## Tests
- `test_importer_upsert_helpers`: removed the four synthetic-anchor
tests; added `_find_or_create_post_idempotent_with_null_source`
(dedup via the partial unique index) and
`_lookup_source_for_sidecar_returns_*` (existing-subscription +
none cases). The existing `_find_or_create_post_idempotent` now
also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
gains `artist_id=<artist>.id`. The `_seed_post` helper in
`test_post_feed_service` looks up artist_id from the source row so
callsites stay one-arg.
## Verification on deploy
After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
posts; PostCard same.
## Out of scope
- UI to manage/delete orphan NULL-source Posts. Data model is right;
UI follows if operator wants it.
Four coupled operator-asked changes to the view modal (Scribe plan #509):
1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
in onMounted/nextTick so the caret is in the input the moment the
modal renders. No click needed to start typing.
2. **General suggestions expanded by default** — SuggestionsPanel's
general-category group now mounts with `:default-open="true"`.
Operator can collapse if too noisy, but the v1 frame shows them.
3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
suggestion_threshold_general default matches character. Alembic
0029 also bumps the existing singleton row's value if it's still
at the old 0.95. Operator can re-tune from Settings → ML.
4. **Retire `copyright` + `artist` as ML suggestion categories** —
neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
really existed as a copyright tag-kind). They were surfaced in the
suggestions pipeline + threshold settings UI but had no follow-
through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
MLThresholdSliders rows.
Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.
Tests:
- test_surfaced_categories asserts {character, general}, excludes
artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
artist + copyright keys in the GET payload.
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
Base.metadata's convention applies ck_%(table_name)s_%(constraint_name)s.
ml_settings and tag_allowlist passed already-prefixed names
(ck_ml_settings_singleton / ck_tag_allowlist_confidence_range), so the
ORM-side names came out doubled (ck_ml_settings_ck_ml_settings_singleton
etc.) and the migration-0003 smoke tests failed.
Same class of bug fixed in FC-2a for ImportSettings — should have applied
that lesson here. Bare names ('singleton', 'confidence_range') let the
convention produce the final names that match migration 0003's literal
DDL. Migration unchanged; only the model __table_args__.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test_health.py:
@pytest.fixture on an `async def` function is rejected by pytest-asyncio
1.x strict mode. Switched to @pytest_asyncio.fixture.
backend/app/models/import_settings.py:
My constraint name was 'ck_import_settings_singleton' and Base.metadata's
naming convention applies 'ck_<table>_<name>' on top, so the final ORM
name was 'ck_import_settings_ck_import_settings_singleton' (double prefix).
The migration creates the DB constraint as 'ck_import_settings_singleton'
via raw alembic, so they didn't match. Fix: bare name 'singleton' in the
model → convention produces 'ck_import_settings_singleton', matching the
migration's literal name.
tests/test_tasks_register.py:
Celery's include=[...] parameter on the constructor is lazy — task
modules aren't imported until a worker boots. The test only imported
the Celery instance, so the @celery.task decorators in scan.py /
import_file.py / thumbnail.py never ran. Fix: explicit `import` of
those modules for side-effect at the top of the test file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen