Nothing in FabledCurator knew what was SUPPOSED to be running. `celery
inspect` reports the workers that ANSWER, so a dead worker was a shorter list
rather than a red light, and grep for any notion of expected services returned
nothing. That is why Portainer was the only place an operator could see it:
Portainer knows the intended set.
`service_seen` is the memory that makes an absence observable — every part
that has checked in, and when it last did.
**Keyed on the queue set, not the worker hostname.** Celery's worker names
here are `celery@<container id>`, minted fresh on every deploy. Keyed on those,
this table would record a death and a birth every time the stack updates — and
a status page that goes red on every deploy is a status page nobody reads,
which is worse than not having one. CELERY_QUEUES is assigned per role in
compose and survives container replacement, so it is the stable identity. Two
replicas of a role are therefore ONE row, which is right: the question is
whether the role is served, not how many containers exist.
The GPU agent is keyed on agent_id, the identity its lease protocol already
uses. gpu.py received it on both lease and heartbeat and threw it away — an
idle agent with nothing to lease left no trace and was indistinguishable from
one switched off a week ago. Now recorded on the calls that were already
happening.
**Who observes, corrected from the plan.** The plan said "record from the
existing inspect path", which would only run when someone opened the Activity
tab. Two other candidates and why they lost:
- A beat sweep. If the scheduler dies the sweep stops, every row goes stale,
and the page says everything is down when one thing is. An alarm that cannot
distinguish "a part died" from "the observer died" is worse than none.
- A background task in web. hypercorn runs --workers 4, so that is four
concurrent inspect loops per container, forever.
Taken instead: refresh on demand, rate-limited by the newest last_seen_at that
every process can already see. The observer is then the thing serving the page
— if web is down you get a browser error, not a confidently green page — and
it self-limits with no coordination, since a race costs one redundant inspect
that writes identical values.
Migration 0090 is the first written on the collapsed baseline (milestone 328),
so it is also the first evidence the chain steps FORWARD from 0089 rather than
merely reproducing the schema. No secondary indexes: one row per moving part
means every read is a handful of rows, and #3301 is the record of what
speculative indexes cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
Items 2-5 of #3072. Item 1 (the per-row sweep inserts) is separate.
2. .fc-bad was not merely duplicated — it is .fc-weak under a second
name. Both local definitions were `color: rgb(var(--v-theme-error))`,
identical to the global .fc-weak, and GpuAgentCard was already using
.fc-weak to colour exactly what GpuActivityPanel coloured .fc-bad (an
errored count, red when non-zero). So rather than promoting a synonym
to app.css, both call sites now use .fc-weak and the local defs are
gone. app.css's status-colour comment records why there is no .fc-bad,
next to the existing note on why .fc-ok is deliberately NOT global.
3. GalleryItem.vue's obsidian literals now use --v-theme-background,
which IS obsidian (vuetify-theme.js maps background -> surfaces.
obsidian). Preferred over --fc-chrome-rgb: same value, but that
variable is named for the nav fade, not for the palette entry.
The ticket said these were the only three real uses in the tree. They
are not — GalleryItem itself had two more in the artist-label
gradient (fixed here, so the file is now consistent), and ~13 more
live in SeriesView, SeriesReaderView, ImageViewer, ArtistHeader,
ExploreView and GalleryFilterBar. Those are a separate sweep, filed
rather than folded in here.
4. The attachment download path had two hand-formatted copies. One
definition now, `attachment_download_url`, next to the model both
serializers already import. The test pins it by MATCHING the built
path against the app's real URL map rather than comparing to a
literal — a string-equality test would still pass after someone
renamed the route, which is the drift the helper exists to prevent.
5. Extension API key now compares with hmac.compare_digest. Compared as
BYTES, not str: compare_digest's str form raises TypeError on
non-ASCII, and this value comes straight from an attacker-controlled
header, so the str form would turn a junk key into a 500 instead of a
403. Low stakes either way — the API is unauthenticated-by-design on
a LAN — but it costs nothing.
Refs #3072
MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) +
presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a
FLAT threshold (decoupled from content-head graduation), plus the "also looks
like content" conflict cut. New presentation_review table (image, presentation
tag, conflict tag + score, created/resolved_at) records auto-hides flagged for
review. Migration 0082 (columns + table), ml_admin API (editable + get_settings
+ _validate bounds), settings roundtrip/bounds test. The sweep that reads these
knobs + the Settings UI land in step 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Foundation for making CCIP character references a precomputed, INCREMENTAL
artifact instead of a request-path rebuild (kills the per-accept ~4s suggestions
stall; cost will scale with change, not library size):
- character_prototype: a character's reference CCIP vectors, capped to
MLSettings.ccip_prototype_cap so match cost doesn't grow with popularity.
- ccip_prototype_state: per-character fingerprint (ref count + max region id) +
updated_at → drives per-character incremental rebuilds and the matcher cache's
reload-only-what-advanced.
- MLSettings.ccip_ref_signature (cheap global change gate) + ccip_prototype_cap.
Migration 0079. Schema + models only — the builder service, refresh task/beat,
and matcher rewrite land in the following steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar
ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira
— pximg URLs carry no content hash). PixivIngester wires client/downloader/
ledgers into ingest_core with drift label 'Pixiv app API' and the new
body_canary=False opt-out: caption-less pixiv artists are common, so the
zero-bodies #862 alarm would false-positive here — the client's
response-shape drift checks cover that failure class instead. auth_token
joins the uniform adapter constructor (pixiv is the first token-auth native
platform). verify_pixiv_credential = one OAuth refresh, no feed walk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding +
trained head' spine; the operator accepted the tagging system and dropped the
harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint,
tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry,
TagEvalRun model + table (migration 0073), and its tests.
The eval's data loaders + metric helpers were NOT eval-specific — the nightly
heads trainer runs on them — so they moved verbatim to
services/ml/training_data.py (heads.py import updated; behavior unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP.
Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING
read them — suggestions come from heads+CCIP and apply_allowlist_tags applies
via Camie predictions, not centroids. Pure dead wiring; remove it.
Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the
daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger,
the tag_reference_embedding table + model, the centroid_similarity_threshold +
min_reference_images settings (migration 0066), the CentroidRecomputeCard +
its store action + MaintenancePanel tile, and the centroid slider in
MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the
allowlist branch already covers "could re-emit"); tag merge no longer clears a
table that no longer exists.
NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction,
and the allowlist bulk-apply — accepting a suggestion still allowlists + applies
it across the library. The tag-eval "centroid" baseline metric is unrelated
(in-memory) and stays. (image_record.centroid_scores JSON column also remains —
separate legacy field, its own micro-cleanup.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Answers "how are videos/all media handled by the GPU worker": a job is per ITEM,
but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the
existing cadence), each stored with a timestamp — so a video becomes a BAG of
frame embeddings (fixes the mean-embedding muddle) instead of one washed-out
vector. Stills → frame_time NULL; animated GIF/WebP treated like short video.
- image_region.frame_time (migration 0061, not yet deployed so folded in): the
source frame's seconds for video/animated media; NULL for stills. RegionService
passes it through. A whole frame is just kind='frame'.
- gpu_job + GpuJobService (migration 0062): the durable work list that keeps the
desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE
SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat /
complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues;
the agent leases+submits over the web API; Redis/Postgres stay private.
Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped
heartbeat, complete, fail-requeue-then-error. region test now covers frame_time.
NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the
agent container + control UI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The storage backbone both crop jobs write to and read from. image_region =
normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id;
'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP /
Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The
bbox doubles as grounded-tag provenance. Migration 0061.
RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines
don't clobber each other) + get_regions — the GPU agent's results endpoint will
call the writer; the character matcher + bag scorer read. Server-side, no GPU.
Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.
Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
Both captured at correction time in TagService.add_to_image/remove_from_image
(source is lost on delete) into durable per-tag counters (head_metric), keyed
by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
(head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
+ head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
head quality, plus the snapshot time-series — the report to tune the precision
target + support floor.
Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.
What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.
NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Graduated heads can now apply their tag without a human — gated so it's safe:
- FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled,
default OFF) is on AND it has >= head_auto_apply_min_positives (default 30)
clean labels. A precise-looking but under-supported low-N head can't spray tags.
- auto_apply_sweep (heads.py): streams every embedded image in chunks, scores
against the eligible heads (numpy, no sklearn), applies each head's tag where
score >= its auto_apply_threshold and the tag isn't already applied/rejected,
with source='head_auto' (distinguishable + reversible). dry_run counts only.
- HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags
task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled)
+ recovery sweep + retention(20).
- API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled),
GET /api/heads/auto-apply (recent runs + per-concept report). Settings
head_auto_apply_enabled + min_positives via /api/ml/settings.
Tests: sweep applies above threshold, dry-run writes nothing, skips under-
supported + ungraduated heads; API disabled/dry-run/conflict guards.
NEXT (slice 2): the observability the operator asked for — per-concept misfire
(auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a
reporting API to tune. Slice 3: the UI (enable, preview, trends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before
I), and drop the inline comment that split heads.py's import block. Pure import
ordering — no behavior change. (run 1601 lint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
"Keep" on a doubted positive was a no-op, so the same confirmed-correct images
came back in "head doubts" every run (operator-flagged: reinforcement keeps
surfacing the same images). Add tag_positive_confirmation (mirror of
tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and
the eval excludes confirmed positives from the doubts list — exactly as rejected
items already drop out of the suggest list. The tag stays a positive either way
(confirmation is a "reviewed" marker, not a training change).
- model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent).
- tag_eval: _confirmed_ids + exclude from head_doubts_positive examples.
- store.confirmTag + card "keep" calls it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core
ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger:
SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery
bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing
media stops re-burning backfill chunks). Per operator decision, dedicated
per-platform tables (not a generalized shared ledger).
filehash is String(128): a CDN content hash when the URL carries one, else a
synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key.
Registered in models/__init__; migration 0054 creates both tables (down 0053).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.
- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
/attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
(_apply_sidecar + upsert_post_record) so a re-import never resets a link's
status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
no-dup on reimport), serializer.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize tagger predictions out of the image_record.tagger_predictions JSON
blob into a queryable per-prediction table. Step 1 of the cutover (expand):
additive + low-risk — reads still use the JSON, this just adds the table and
keeps it populated.
- ImagePrediction(image_record_id, raw_name, category, score) — stores the
RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution
is unchanged. Indexed for per-image reads + by (raw_name, score).
- Migration 0045: create table + set-based backfill from the JSON via
json_each (fast post-#764-prune). The old column stays (vestigial) and is
dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the
hot image_record table, so it waits for a quiesced-worker window.
- tag_and_embed dual-writes the rows (delete-then-insert, idempotent);
tagger_store_floor already applied in infer().
Next: switch suggestion + allowlist reads to the table, then drop the JSON
write. Plan-task #768.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirm-only "this post may continue this series" matcher.
- series_suggestion table (post_id, series_tag_id, score, signals jsonb, status
pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings
knobs (series_suggest_enabled, series_suggest_threshold).
- series_match_service: weighted additive score (title-stem / same-artist /
page-continuity / shared-distinctive-tags), no single signal gating. The title
"pattern" is derived on the fly from the post titles already in a series, so it
sharpens as more are confirmed (no persisted state to drift). Candidates are
bounded to the post's artist. match_post upserts pending suggestions (UNIQUE +
on-conflict, respecting prior added/dismissed decisions).
- accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'.
- rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from
a post-id cursor (maintenance_long lane), like normalize_tags_task.
- API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan.
- Settings: enabled + threshold exposed via /settings/import.
- Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE
dedup.
Frontend (Suggestions tab + settings card) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an ordered chapter layer to series. Reading order becomes
(series_chapter.chapter_number, series_page.page_number); a chapter may be a
placeholder reserving a slot, and carries an optional parsed stated-page range
used to flag missing-page gaps. An image still lives in at most one series ⇒ one
chapter (image_id stays UNIQUE).
- models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) +
stated_page. Migration 0040 backfills every existing series into one
auto-chapter holding its current flat pages — no data loss.
- SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter
assignment, reorder_pages, chapter-aware set_cover; list_pages now returns
chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide
reorder operates on the single default chapter and rejects multi-chapter series.
- API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an
optional chapter_id.
- TagService.merge now repoints series_chapter too, so a merged series' chapters
(and their pages) survive the source tag's deletion instead of cascading away.
- Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to
supply chapter_id.
Frontend (chapter-aware manage view + reader) lands next; until then the
existing UI keeps working via the flat pages[] + single default chapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux,
persistently-corrupt bytes) used to re-error forever and re-burn chunks.
New `patreon_failed_media` table (alembic 0038, chains 0037) records
per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the
ingester skips it on routine tick/backfill walks (tier-1.5, folded into the
seen/skip predicate). Recovery BYPASSES it (the operator's "try everything
again" re-attempts dead media). A clean download clears the row (recovered);
errors/quarantines upsert-increment it. Surfaced as
run_stats.dead_lettered_count.
- New PatreonFailedMedia model + migration; ingester _dead_keys /
_record_failures (on_conflict increment) / _clear_failures.
- skip = seen | dead (empty in recovery); failures recorded post-fetch on
short sessions (same pattern as the seen-ledger).
Tests: a media erroring 3× is dead-lettered + skipped (no download attempt);
recovery re-attempts a dead media and clears it on success; a clean download
clears a sub-threshold failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id,
filehash) — our own queryable replacement for gallery-dl's archive.sqlite3.
Routine walks skip seen media; recovery mode bypasses the ledger. filehash is
a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic
0037 (← 0036). Integration test covers dedup + savepoint recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-artist "+N" accent pill on the artists directory and a "N new since
last visit" banner inside ArtistView. Counts new IMAGES (not posts) so
multi-image posts increment correctly.
- alembic 0034: artist_visit (artist_id PK, last_viewed_at NOT NULL).
Seeds every existing artist with last_viewed_at=NOW() so the badge
starts at 0 across the board — no noisy "5000 unseen images" on
first deploy.
- ArtistService.find_or_create autoseeds a visit row alongside new
artists, so freshly imported content doesn't read as unseen.
- ArtistService.overview reads pre-visit last_viewed_at, counts images
created since, then atomically UPSERTs last_viewed_at=NOW() via
postgres ON CONFLICT DO UPDATE (no SELECT-then-INSERT race per
reference_scalar_one_or_none_duplicates). Returns the pre-update
count as `unseen_count_at_visit` so the banner has data.
- ArtistDirectoryService.list_artists adds an `unseen_count` aggregate
to each card via LEFT JOIN artist_visit + conditional COUNT. NULL
last_viewed_at (artist created before this code shipped) defensively
counts as "never visited" → all images unseen.
- Frontend: ArtistCard renders an accent pill in the preview-strip
corner when unseen_count > 0 (capped at 99+); ArtistView shows a
closable v-alert banner on initial load when
unseen_count_at_visit > 0, re-arms on slug change.
Single-row-per-artist (no user_id) — rule #47 multi-user ACL is
aspirational; widens to (user_id, artist_id) PK when User lands, per
rule #22.
Scribe plan #597.
Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating),
Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a
kind-aware uniqueness index so the same name can exist across kinds and the same
character name can exist in different fandoms.
Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a
single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs.
Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e
populates this via the integrity verifier.
Drops the unused tag.namespace column from FC-1 — superseded by kind.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the data model from spec §3 in one go so FC-2/FC-3 don't need
schema-adding migrations of their own. Artist is the unified entity for
both gallery 'artist:' tags and GallerySubscriber Subscriptions
(is_subscription flag). ImageProvenance is many-to-one, enabling the
enrich-on-duplicate rule for downloaded content that pHash-matches an
existing record.
The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m;
swapping models in FC-2 will require a column-width migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Configures stable constraint naming so autogeneration produces clean diffs.
Alembic uses the sync psycopg driver while the runtime app uses asyncpg.
Also fixes a .gitignore bug caught during this task: the bare 'models/'
rule for the ML weights volume was matching backend/app/models/ (Python
package). Anchored all volume rules to repo root (/images/, /import/,
/downloads/, /models/, /postgres_data/, /redis_data/).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
