39 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	751e7ddb9f	feat: the membership sweep, and the state that makes its failures readable (387 C3) ... A daily sweep that walks each platform's roster into `platform_membership`. Daily because memberships change on a BILLING cycle, not a download cadence. **`membership_sync` is the part that earns its keep.** Without it three very different situations are one indistinguishable state — the account subscribes to nothing, the sweep never ran, the sweep failed — and all three leave zero rows in `platform_membership`. "You are tracking 12 sources you no longer subscribe to" is correct in the first case and an invitation to cancel things the operator is actively paying for in the other two. So C4 gates its CONCLUSIONS on `last_success_at`, not merely its display, and `roster_is_fresh` is computed server-side so no caller can forget to. Two timestamps rather than one: `last_attempt_at` moves every run, `last_success_at` only on a clean walk. The gap between them is the signal — a sweep hammering a broken credential every day must not look healthy because it ran recently, and there is a test for exactly that. Rejected shortcuts, both tempting: `MAX(platform_membership.last_seen_at)` cannot tell "synced fine, found nothing" from "never synced"; `task_run` is worse, since its retention prunes ok rows after 24h and a sweep that last succeeded three days ago would leave no trace at all. **The fetch completes before anything is written.** That ordering is the safety property: a walk that dies mid-pagination writes nothing, so a failure can never leave a roster half this week's and half last week's. `touch_membership` never deletes, so a failure cannot empty the roster either — but "intact" should mean intact, not merely non-empty. Rule 89's four, each where it actually lives: recovery is "run it again" (upsert, no deletes); retention is C1's age-out-never-delete, because disappearing IS the signal; the wall-clock deadline is per-platform and distinct from the per-REQUEST timeout the client already has (rule 156 — a paginated roster answering every page slowly-but-within-timeout would never trip that one and would sit on a worker indefinitely); duration comes from the existing TaskRun signal plumbing. **A bug caught in review, not production:** the broad `except Exception` would have swallowed Celery's SoftTimeLimitExceeded — which is an ORDINARY Exception subclass, not a BaseException — letting the sweep run past the soft limit into the hard one, where it is SIGKILLed mid-transaction. A sweep that cannot be stopped is worse than one that fails. Now re-raised explicitly, with a test that also asserts SoftTimeLimitExceeded is still an Exception, so the re-raise cannot quietly become dead code. Rule 164 is why this ships with UI rather than backend-only: a roster that never synced must be VISIBLE as such. The card says "never synced" in words and states no count at all — rendering it as 0 is the precise conflation the whole step exists to prevent — while a real zero behind a real sync is reported as zero, because that one IS an answer. Pinned in both directions. Three independent gates decide whether a platform is swept — registered here, client exposes `iter_memberships`, credential exists — each silent, so adding SubscribeStar (D1) is one line and nothing else. A missing credential is not an error: recording a failure would light up the UI for a feature never enabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 23:40:48 -04:00
bvandeusenandClaude Opus 5	235393c08b	feat: link the Patreon teaser to the Discord drop it announced (388 step E5) ... The point of the milestone rather than its tail. Two of the operator's artists post a deliberately cropped fragment on Patreon to signal that the real thing has landed in their Discord; this proposes those pairs. Confirm-only, following the FC-6.3 series matcher. A wrongly-asserted association tells the operator two different pieces are one, which is strictly worse than no link: no link leaves them where they already were, a wrong one actively misinforms and then propagates into whatever reads it. So the matcher's job is a SHORT list worth reading, not a long list worth trusting. **The threshold sits above every single signal weight, and that is the design.** Proximity is 0.55, declaration 0.45, the cut 0.60 — so neither signal can carry a pair alone. That makes "time proximity alone is never sufficient" an arithmetic property rather than an aspiration: on a busy day an artist posts several times, and a matcher that could pair on proximity alone would turn every one of those days into false pairs until the review queue got abandoned. A guard test asserts the relationship against WEIGHTS directly, so it survives any refactor of the scorer, and says in its own failure message not to fix it by lowering the assertion. **Crop-to-source matching is HELD, on the plan's instruction** — real work with real false-positive risk, worth building only once signals 1 and 2 are shown insufficient against the operator's actual artists. Worth stating: a naive whole-image SigLIP similarity is NOT that signal. A cropped teaser and its full version are precisely the pair a whole-image comparison handles worst, so adding one as a "bonus" would mostly add noise while looking like progress. Two premises in the plan corrected in the building: * **E4 is not actually a prerequisite.** A Patreon Source and a Discord Source the operator has added under one Artist already share `Post.artist_id`, and the synthetic grouping inherits it. E4 EXTENDS this to creators FC has to learn the association for; it is not needed to represent one FC was told. Same-artist is then a hard filter, not a scored signal — two different creators posting minutes apart is a coincidence, not evidence. * **`link_extract` cannot supply the declaration signal.** It exists, but `SUPPORTED_HOSTS` is file hosts only and `host_for()` returns None for a Discord URL, so no ExternalLink row is ever written for one. The signal reads the post body directly instead. And a bug my own test would have caught: `declared_signal` stripped the HTML before looking for an invite, but `html_to_plain` discards attributes and these creators put the invite in an anchor's `href` — so the strongest form of the signal was being thrown away, leaving only whatever the link text said. The invite now matches the raw body; the bare mention still matches stripped text, so `\bdiscord\b` is tested against prose rather than against markup. Dismissed rows are kept, not deleted: the row is what remembers the rejection, and re-proposing a rejected pair on every scan is the one behaviour that makes a review queue get ignored. Both FKs CASCADE, so E3's one-DELETE reversal cannot leave a proposal pointing at a post that no longer exists. Only ACCEPTED links reach the post payload. A pending proposal is a question for the review queue, not a claim to render beside the artwork. UI (rule 27) follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 11:47:28 -04:00
bvandeusenandClaude Opus 5	4fe792b61c	feat: platform_membership — the learned roster of what the account pays for (milestone 387 step C1) ... FC knows which creators it was TOLD to follow and nothing about which ones the operator is subscribed to. Those two sets drift both ways and neither drift is currently visible: a subscription FC doesn't track is content the operator believes they're archiving and aren't, and a source walked after the subscription lapsed is requests spent on a wall reported as a creator gone quiet. Sibling of service_seen (milestone 365) and the same insight — an absence is only observable against a record of presence. touch_membership reuses the recorded touch_service shape (snippet 3447): upsert rather than read-modify-write, first_seen_at deliberately outside the update set because it's the one field that makes a later DISAPPEARANCE readable as a lapse rather than as a creator we never knew. Nothing populates it yet, and that's the intended intermediate state. The sweep (C3) needs a client seam (C2) that needs Patreon's real response characterised from a captured sample (C0), which needs the operator's browser session. The table's SHAPE doesn't wait on that, because it's deliberately free-form exactly where C0's findings would otherwise dictate a column. status is an unconstrained String holding the PLATFORM's own word, not a normalised FC value. Rule 36 considered and declined, same reasoning service_seen.kind records: the vocabulary isn't ours to invent, and picking a lowest-common-denominator enum before any platform has been characterised would bake a guess into the schema. The service owns the whitelist and the mapping; the column owns the evidence. MEMBERSHIP_STATUS ships EMPTY, guarded by a test that fails if anyone adds an entry — every one must come from a characterised response, not from API docs. That's rule 130 at the one place it's easiest to break, and the failure message says so. has_paid_access returns None, never False, for a word it hasn't been taught. The difference is load-bearing: False means the operator lost access, which C4 turns into an offer to disable the source, so asserting it from an unrecognised word would tell them to cancel a subscription they're still paying for. Retention decided here rather than deferred (rule 89): a membership that stops appearing is aged out on time, never deleted on absence — deleting would destroy the signal at the moment it became interesting. Rule 90 check, done on the right thing this time: the per-test TRUNCATE teardown derives its table list from Base.metadata.sorted_tables, so the new table is picked up automatically; test_models asserts a subset, so it doesn't break. 0091 follows 0090 on the collapsed baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 10:58:25 -04:00
bvandeusenandClaude Opus 5	dc8af8b1a7	feat: a learned roster, so a stopped part is observable (milestone 365 steps 1-2) ... Nothing in FabledCurator knew what was SUPPOSED to be running. `celery inspect` reports the workers that ANSWER, so a dead worker was a shorter list rather than a red light, and grep for any notion of expected services returned nothing. That is why Portainer was the only place an operator could see it: Portainer knows the intended set. `service_seen` is the memory that makes an absence observable — every part that has checked in, and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy. Keyed on those, this table would record a death and a birth every time the stack updates — and a status page that goes red on every deploy is a status page nobody reads, which is worse than not having one. CELERY_QUEUES is assigned per role in compose and survives container replacement, so it is the stable identity. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served, not how many containers exist. The GPU agent is keyed on agent_id, the identity its lease protocol already uses. gpu.py received it on both lease and heartbeat and threw it away — an idle agent with nothing to lease left no trace and was indistinguishable from one switched off a week ago. Now recorded on the calls that were already happening. **Who observes, corrected from the plan.** The plan said "record from the existing inspect path", which would only run when someone opened the Activity tab. Two other candidates and why they lost: - A beat sweep. If the scheduler dies the sweep stops, every row goes stale, and the page says everything is down when one thing is. An alarm that cannot distinguish "a part died" from "the observer died" is worse than none. - A background task in web. hypercorn runs --workers 4, so that is four concurrent inspect loops per container, forever. Taken instead: refresh on demand, rate-limited by the newest last_seen_at that every process can already see. The observer is then the thing serving the page — if web is down you get a browser error, not a confidently green page — and it self-limits with no coordination, since a race costs one redundant inspect that writes identical values. Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps FORWARD from 0089 rather than merely reproducing the schema. No secondary indexes: one row per moving part means every read is a handful of rows, and #3301 is the record of what speculative indexes cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 17:15:46 -04:00
bvandeusen	1ac448d881	refactor: four small cleanups from the review pass (#3072) ... Items 2-5 of #3072. Item 1 (the per-row sweep inserts) is separate. 2. .fc-bad was not merely duplicated — it is .fc-weak under a second name. Both local definitions were `color: rgb(var(--v-theme-error))`, identical to the global .fc-weak, and GpuAgentCard was already using .fc-weak to colour exactly what GpuActivityPanel coloured .fc-bad (an errored count, red when non-zero). So rather than promoting a synonym to app.css, both call sites now use .fc-weak and the local defs are gone. app.css's status-colour comment records why there is no .fc-bad, next to the existing note on why .fc-ok is deliberately NOT global. 3. GalleryItem.vue's obsidian literals now use --v-theme-background, which IS obsidian (vuetify-theme.js maps background -> surfaces. obsidian). Preferred over --fc-chrome-rgb: same value, but that variable is named for the nav fade, not for the palette entry. The ticket said these were the only three real uses in the tree. They are not — GalleryItem itself had two more in the artist-label gradient (fixed here, so the file is now consistent), and ~13 more live in SeriesView, SeriesReaderView, ImageViewer, ArtistHeader, ExploreView and GalleryFilterBar. Those are a separate sweep, filed rather than folded in here. 4. The attachment download path had two hand-formatted copies. One definition now, `attachment_download_url`, next to the model both serializers already import. The test pins it by MATCHING the built path against the app's real URL map rather than comparing to a literal — a string-equality test would still pass after someone renamed the route, which is the drift the helper exists to prevent. 5. Extension API key now compares with hmac.compare_digest. Compared as BYTES, not str: compare_digest's str form raises TypeError on non-ASCII, and this value comes straight from an attacker-controlled header, so the str form would turn a junk key into a 500 instead of a 403. Low stakes either way — the API is unauthenticated-by-design on a LAN — but it costs nothing. Refs #3072	2026-08-27 07:48:18 -04:00
bvandeusenandClaude Opus 4.8	ab63d94249	feat(ml): presentation auto-hide settings + review table (#141 step 3) ... MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) + presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a FLAT threshold (decoupled from content-head graduation), plus the "also looks like content" conflict cut. New presentation_review table (image, presentation tag, conflict tag + score, created/resolved_at) records auto-hides flagged for review. Migration 0082 (columns + table), ml_admin API (editable + get_settings + _validate bounds), settings roundtrip/bounds test. The sweep that reads these knobs + the Settings UI land in step 4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 22:59:00 -04:00
bvandeusenandClaude Opus 4.8	f24dc81764	feat(ccip): schema for precomputed incremental character prototypes (#1317, m138 step 1) ... Foundation for making CCIP character references a precomputed, INCREMENTAL artifact instead of a request-path rebuild (kills the per-accept ~4s suggestions stall; cost will scale with change, not library size): - character_prototype: a character's reference CCIP vectors, capped to MLSettings.ccip_prototype_cap so match cost doesn't grow with popularity. - ccip_prototype_state: per-character fingerprint (ref count + max region id) + updated_at → drives per-character incremental rebuilds and the matcher cache's reload-only-what-advanced. - MLSettings.ccip_ref_signature (cheap global change gate) + ccip_prototype_cap. Migration 0079. Schema + models only — the builder service, refresh task/beat, and matcher rewrite land in the following steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-06 15:58:11 -04:00
bvandeusenandClaude Fable 5	0563b2d750	feat(pixiv): ledger models + migration 0076 + PixivIngester adapter (#129 step 3) ... pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira — pximg URLs carry no content hash). PixivIngester wires client/downloader/ ledgers into ingest_core with drift label 'Pixiv app API' and the new body_canary=False opt-out: caption-less pixiv artists are common, so the zero-bodies #862 alarm would false-positive here — the client's response-shape drift checks cover that failure class instead. auth_token joins the uniform adapter constructor (pixiv is the first token-auth native platform). verify_pixiv_credential = one OAuth refresh, no feed walk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-03 09:45:14 -04:00
bvandeusenandClaude Fable 5	eaea4308fc	chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved) ... The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:41:24 -04:00
bvandeusenandClaude Opus 4.8	485387ff0b	refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189) ... Heads + CCIP are the tag source and head auto-apply is the earned propagation. The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply path firing in parallel with heads on every accept — exactly the un-earned spray the v2 pivot replaced. Retire both. Behavior change: accepting a suggestion now applies the tag to THAT image only (source='ml_accepted', a head-training positive) — it no longer allowlists + fans the tag across the library via Camie. Propagation is heads' earned auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was un-earned and bypassed the precision gate.) - tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction writes); backfill enqueues it for images with no embedding. - Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat + every enqueue caller (accept/alias/merge/per-image), api/allowlist.py + blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067), AllowlistTable.vue + allowlist store, the accept coverage-projection payload. - AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store the rail still needs); accept returns nothing, API returns {accepted, tag_id}. - tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on ML-applied image_tag sources (incl. head_auto) instead of the allowlist. - UI: MLBackfillCard relabelled to embedding-only; accept toast simplified; MaintenancePanel drops the allowlist tile. Left for a follow-up hygiene pass (now-inert, harmless): the dead settings columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*, video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders trim, and the Camie model download in download_models.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:04:31 -04:00
bvandeusenandClaude Opus 4.8	3d77a38a25	refactor(ml): remove the dead per-tag centroid subsystem (#1189) ... The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 11:48:09 -04:00
bvandeusenandClaude Opus 4.8	b735432d02	feat(gpu): video-ready regions + the HTTP GPU-job queue engine (#114 slice 3) ... Answers "how are videos/all media handled by the GPU worker": a job is per ITEM, but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the existing cadence), each stored with a timestamp — so a video becomes a BAG of frame embeddings (fixes the mean-embedding muddle) instead of one washed-out vector. Stills → frame_time NULL; animated GIF/WebP treated like short video. - image_region.frame_time (migration 0061, not yet deployed so folded in): the source frame's seconds for video/animated media; NULL for stills. RegionService passes it through. A whole frame is just kind='frame'. - gpu_job + GpuJobService (migration 0062): the durable work list that keeps the desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat / complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues; the agent leases+submits over the web API; Redis/Postgres stay private. Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped heartbeat, complete, fail-requeue-then-error. region test now covers frame_time. NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the agent container + control UI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 11:18:28 -04:00
bvandeusenandClaude Opus 4.8	0ea7ecdea5	feat(regions): image_region storage + service for the crop pipeline (#114 slice 2) ... The storage backbone both crop jobs write to and read from. image_region = normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id; 'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP / Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The bbox doubles as grounded-tag provenance. Migration 0061. RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines don't clobber each other) + get_regions — the GPU agent's results endpoint will call the writer; the character matcher + bag scorer read. Server-side, no GPU. Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 10:36:52 -04:00
bvandeusenandClaude Opus 4.8	48c8811d69	feat(heads): auto-apply observability + on by default (#114 auto-apply B) ... Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:36:58 -04:00
bvandeusenandClaude Opus 4.8	74fef908d2	feat(heads): earned auto-apply — sweep mechanism, off by default (#114 auto-apply A) ... Graduated heads can now apply their tag without a human — gated so it's safe: - FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled, default OFF) is on AND it has >= head_auto_apply_min_positives (default 30) clean labels. A precise-looking but under-supported low-N head can't spray tags. - auto_apply_sweep (heads.py): streams every embedded image in chunks, scores against the eligible heads (numpy, no sklearn), applies each head's tag where score >= its auto_apply_threshold and the tag isn't already applied/rejected, with source='head_auto' (distinguishable + reversible). dry_run counts only. - HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled) + recovery sweep + retention(20). - API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled), GET /api/heads/auto-apply (recent runs + per-concept report). Settings head_auto_apply_enabled + min_positives via /api/ml/settings. Tests: sweep applies above threshold, dry-run writes nothing, skips under- supported + ungraduated heads; API disabled/dry-run/conflict guards. NEXT (slice 2): the observability the operator asked for — per-concept misfire (auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a reporting API to tune. Slice 3: the UI (enable, preview, trends). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:22:54 -04:00
bvandeusenandClaude Opus 4.8	1ed0895e8d	style(heads): fix import ordering (ruff I001) ... Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before I), and drop the inline comment that split heads.py's import block. Pure import ordering — no behavior change. (run 1601 lint) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-28 10:41:12 -04:00
bvandeusenandClaude Opus 4.8	22c3b54746	feat(heads): production per-concept heads — train + score backend (#114 A) ... The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its production form (the first of three slices that make heads the suggestion source, replacing Camie + centroid). - tag_head: one logistic-regression head per general/character concept with enough labelled positives. Weights (pgvector), honest CV-derived suggest threshold + earned-auto-apply point, and per-concept quality metrics. - head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the admin card shows live + historical status across navigation. - services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data loaders + metric math so production heads match measured eval numbers) and SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one image's embedding against all heads → the rail's suggestions, cached on (count, max trained_at) so a retrain invalidates without per-request loads. - tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress) + recover_stalled_head_training_runs sweep + retention(20) + 5-min beat (rule 89). - api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET /api/heads (count, graduated, last-trained, running, per-concept table, recent runs). - ml_settings: head_min_positives + head_auto_apply_precision, tunable via /api/ml/settings. Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B — this slice makes training + scoring exist and CI-verifiable. 'precision' column stored as precision_cv (SQL reserved word). Migration 0058. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-28 10:36:25 -04:00
bvandeusenandClaude Opus 4.8	b69c70ab2b	feat(tag-eval): "keep" records a confirmation so doubts stop resurfacing ... "Keep" on a doubted positive was a no-op, so the same confirmed-correct images came back in "head doubts" every run (operator-flagged: reinforcement keeps surfacing the same images). Add tag_positive_confirmation (mirror of tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and the eval excludes confirmed positives from the doubts list — exactly as rejected items already drop out of the suggest list. The tag stays a positive either way (confirmation is a "reviewed" marker, not a training change). - model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent). - tag_eval: _confirmed_ids + exclude from head_doubts_positive examples. - store.confirmTag + card "keep" calls it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-28 01:32:20 -04:00
bvandeusenandClaude Opus 4.8	6e3c5f697f	feat(ml): tag-eval backend — head-vs-centroid learning-curve eval (persisted) ... Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained- head spine on the operator's own data, reusing the SigLIP embeddings already stored on image_record — no re-embedding, no GPU. Per concept: train a logistic-regression HEAD (positives + negatives = explicit rejections + sampled unlabeled) vs the old single-CENTROID baseline; report cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged positives grow 10→30→100→300), and example image ids (head-would-suggest / head-doubts-positive) to eyeball. Persisted so the report SURVIVES navigation (operator-flagged): the run + full report live in a new tag_eval_run row (mirrors library_audit_run); the admin card will rehydrate from GET on mount, not transient state. - models.TagEvalRun + migration 0056; runs on the ml queue (only worker with numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue. - services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml .tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report). - recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat (rule 89). scikit-learn added to requirements-ml. - tests: param normalization + the rehydrate read-path + create/conflict. Frontend admin card (trigger + render persisted report) follows next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-27 22:49:10 -04:00
bvandeusenandClaude Opus 4.8	f678819093	feat(subscribestar): seen/failed ledger models + migration 0054 (#889) ... Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 10:04:52 -04:00
bvandeusenandClaude Opus 4.8	d96918d777	feat(posts): extract + record external file-host links (Phase 3) ... Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded in post bodies so they're never silently dropped, and surface them in the post view. The download worker (Phase 4) walks these rows. - link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups. Reusable by every platform (runs off Post.description). - external_link model + migration 0049: post_id/artist_id/host/url/label/status /attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl. worker statuses up front) + (post_id,url) unique. - importer._sync_external_links: insert-missing on both import paths (_apply_sidecar + upsert_post_record) so a re-import never resets a link's status; runs for all platforms. - post_feed_service.get_post: returns external_links (detail-only). - PostCard: renders the links (host chip + label + status) once expanded. - tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record + no-dup on reimport), serializer. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:15:36 -04:00
bvandeusenandClaude Opus 4.8	79089b50b0	feat(ml): image_prediction table + backfill + dual-write (#768 step 1) ... Normalize tagger predictions out of the image_record.tagger_predictions JSON blob into a queryable per-prediction table. Step 1 of the cutover (expand): additive + low-risk — reads still use the JSON, this just adds the table and keeps it populated. - ImagePrediction(image_record_id, raw_name, category, score) — stores the RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution is unchanged. Indexed for per-image reads + by (raw_name, score). - Migration 0045: create table + set-based backfill from the JSON via json_each (fast post-#764-prune). The old column stays (vestigial) and is dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the hot image_record table, so it waits for a quiesced-worker window. - tag_and_embed dual-writes the rows (delete-then-insert, idempotent); tagger_store_floor already applied in infer(). Next: switch suggestion + allowlist reads to the table, then drop the JSON write. Plan-task #768. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 15:55:32 -04:00
bvandeusenandClaude Opus 4.8	c0fd80e694	feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3) ... Confirm-only "this post may continue this series" matcher. - series_suggestion table (post_id, series_tag_id, score, signals jsonb, status pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings knobs (series_suggest_enabled, series_suggest_threshold). - series_match_service: weighted additive score (title-stem / same-artist / page-continuity / shared-distinctive-tags), no single signal gating. The title "pattern" is derived on the fly from the post titles already in a series, so it sharpens as more are confirmed (no persisted state to drift). Candidates are bounded to the post's artist. match_post upserts pending suggestions (UNIQUE + on-conflict, respecting prior added/dismissed decisions). - accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'. - rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from a post-id cursor (maintenance_long lane), like normalize_tags_task. - API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan. - Settings: enabled + threshold exposed via /settings/import. - Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE dedup. Frontend (Suggestions tab + settings card) lands next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 18:58:18 -04:00
bvandeusenandClaude Opus 4.8	1804a2c622	feat(series): chapter layer over series_page — backend (FC-6.1) ... Adds an ordered chapter layer to series. Reading order becomes (series_chapter.chapter_number, series_page.page_number); a chapter may be a placeholder reserving a slot, and carries an optional parsed stated-page range used to flag missing-page gaps. An image still lives in at most one series ⇒ one chapter (image_id stays UNIQUE). - models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) + stated_page. Migration 0040 backfills every existing series into one auto-chapter holding its current flat pages — no data loss. - SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter assignment, reorder_pages, chapter-aware set_cover; list_pages now returns chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide reorder operates on the single default chapter and rejects multi-chapter series. - API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an optional chapter_id. - TagService.merge now repoints series_chapter too, so a merged series' chapters (and their pages) survive the source tag's deletion instead of cascading away. - Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to supply chapter_id. Frontend (chapter-aware manage view + reader) lands next; until then the existing UI keeps working via the flat pages[] + single default chapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 16:31:55 -04:00
bvandeusenandClaude Opus 4.8	7a872a3619	feat(patreon): dead-letter ledger for permanently-failing media — #705 step 2 (#7) ... A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux, persistently-corrupt bytes) used to re-error forever and re-burn chunks. New `patreon_failed_media` table (alembic 0038, chains 0037) records per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the ingester skips it on routine tick/backfill walks (tier-1.5, folded into the seen/skip predicate). Recovery BYPASSES it (the operator's "try everything again" re-attempts dead media). A clean download clears the row (recovered); errors/quarantines upsert-increment it. Surfaced as run_stats.dead_lettered_count. - New PatreonFailedMedia model + migration; ingester _dead_keys / _record_failures (on_conflict increment) / _clear_failures. - skip = seen | dead (empty in recovery); failures recorded post-fetch on short sessions (same pattern as the seen-ledger). Tests: a media erroring 3× is dead-lettered + skipped (no download attempt); recovery re-attempts a dead media and clears it on success; a clean download clears a sub-threshold failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 00:04:24 -04:00
bvandeusenandClaude Opus 4.8	6222928746	feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697) ... patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id, filehash) — our own queryable replacement for gallery-dl's archive.sqlite3. Routine walks skip seen media; recovery mode bypasses the ledger. filehash is a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic 0037 (← 0036). Integration test covers dedup + savepoint recovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 19:26:57 -04:00
bvandeusen	b65e956ad2	feat(artist): "new since last visit" badge + banner ... Per-artist "+N" accent pill on the artists directory and a "N new since last visit" banner inside ArtistView. Counts new IMAGES (not posts) so multi-image posts increment correctly. - alembic 0034: artist_visit (artist_id PK, last_viewed_at NOT NULL). Seeds every existing artist with last_viewed_at=NOW() so the badge starts at 0 across the board — no noisy "5000 unseen images" on first deploy. - ArtistService.find_or_create autoseeds a visit row alongside new artists, so freshly imported content doesn't read as unseen. - ArtistService.overview reads pre-visit last_viewed_at, counts images created since, then atomically UPSERTs last_viewed_at=NOW() via postgres ON CONFLICT DO UPDATE (no SELECT-then-INSERT race per reference_scalar_one_or_none_duplicates). Returns the pre-update count as `unseen_count_at_visit` so the banner has data. - ArtistDirectoryService.list_artists adds an `unseen_count` aggregate to each card via LEFT JOIN artist_visit + conditional COUNT. NULL last_viewed_at (artist created before this code shipped) defensively counts as "never visited" → all images unseen. - Frontend: ArtistCard renders an accent pill in the preview-strip corner when unseen_count > 0 (capped at 99+); ArtistView shows a closable v-alert banner on initial load when unseen_count_at_visit > 0, re-arms on slug change. Single-row-per-artist (no user_id) — rule #47 multi-user ACL is aspirational; widens to (user_id, artist_id) PK when User lands, per rule #22. Scribe plan #597.	2026-06-03 15:27:11 -04:00
bvandeusenandClaude Opus 4.7	8649a13118	refactor(I5): remove one-and-done GS/IR migration tooling ... The GS/IR migration cutover is complete, so the runbook tooling is dead weight. Removed: - services/migrators/ (gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup), tasks/migration.py, api/migrate.py (+ blueprint registration) - MigrationRun model; alembic 0027 drops the migration_run table - frontend LegacyMigrationCard + migration store (+ MaintenancePanel ref) - celery include + task route + celery_signals queue mapping for migration.* - the 1 GB MAX_CONTENT_LENGTH / MAX_FORM_MEMORY override (added solely for the ir_ingest upload) - migration-surface tests (test_api_migrate, test_migration_verify, test_ir_ingest, test_gs_ingest, test_tag_apply) Kept: the alembic schema-migration tests (test_migration_00XX — unrelated) and cleanup_service.py (the permanent artist-cascade/unlink home). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-29 14:38:59 -04:00
bvandeusen	929d3fc092	feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 01:17:42 -04:00
bvandeusenandClaude Opus 4.7	c3e855bd9b	fc3h: BackupRun model — artifact record for backup/restore runs ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 22:50:59 -04:00
bvandeusenandClaude Opus 4.7	79fee98db4	fc3i: TaskRun model — per-Celery-task lifecycle audit row ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 21:00:14 -04:00
bvandeusenandClaude Opus 4.7	da9c19b2dc	fc5: migration 0015 + MigrationRun model (kind/status as String(32), API-validated) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-22 08:11:42 -04:00
bvandeusenandClaude Opus 4.7	2a18016b69	feat(fc3b): migration 0012 + AppSetting model ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 18:33:14 -04:00
bvandeusenandClaude Opus 4.7	8f69478227	feat(attachments): PostAttachment model + 0009 (table + batch counter) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 11:10:53 -04:00
bvandeusenandClaude Opus 4.7	b83baa7d7f	feat(series): SeriesPage model + 0005 migration (unique image, ordered pages) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 23:33:12 -04:00
bvandeusenandClaude Opus 4.7	906804140c	feat(fc2b): schema migration 0003 — ML pipeline tables ... Renames image_record.wd14_* -> tagger_* (we're on Camie now, not WD14). Adds tag_allowlist (auto-apply opt-in, per-tag confidence), tag_suggestion_rejection (per-image dismissals), tag_alias (composite (string, category) -> canonical tag, resolved at read time), tag_reference_embedding (per-tag SigLIP centroids), and the ml_settings singleton (per-category + centroid thresholds, model version pins). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 07:35:22 -04:00
bvandeusenandClaude Opus 4.7	9eb1614fbf	feat(fc2a): schema migration 0002 — tag kinds + fandom hierarchy + import lifecycle ... Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating), Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a kind-aware uniqueness index so the same name can exist across kinds and the same character name can exist in different fandoms. Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs. Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e populates this via the integrity verifier. Drops the unused tag.namespace column from FC-1 — superseded by kind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 12:03:41 -04:00
bvandeusenandClaude Opus 4.7	d6a156dcd2	feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration ... Implements the data model from spec §3 in one go so FC-2/FC-3 don't need schema-adding migrations of their own. Artist is the unified entity for both gallery 'artist:' tags and GallerySubscriber Subscriptions (is_subscription flag). ImageProvenance is many-to-one, enabling the enrich-on-duplicate rule for downloaded content that pHash-matches an existing record. The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m; swapping models in FC-2 will require a column-width migration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:34:29 -04:00
bvandeusenandClaude Opus 4.7	a03655039c	feat: add SQLAlchemy declarative base, Alembic environment, and gitignore fix ... Configures stable constraint naming so autogeneration produces clean diffs. Alembic uses the sync psycopg driver while the runtime app uses asyncpg. Also fixes a .gitignore bug caught during this task: the bare 'models/' rule for the ML weights volume was matching backend/app/models/ (Python package). Anchored all volume rules to repo root (/images/, /import/, /downloads/, /models/, /postgres_data/, /redis_data/). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:32:58 -04:00