Recapture disk-skips already-downloaded media, and upsert_post_record only
writes Post fields — so a pre-existing image (e.g. one pulled under the old
gallery-dl path, imported bare with no post) stays orphaned even after its post
record is (re)written. Confirmed on the operator's instance: 329 pixiv images
with primary_post_id NULL, 694 pixiv posts with content but no linked images, 0
duplicate posts.
Fix: the recapture relink channel now carries the media's post_id (2- → 3-tuple
path/url/post_id), and phase 3 calls importer.link_existing_image_to_post — match
the on-disk image by path, find its Post by (source, external_post_id), upsert
image_provenance + primary_post_id. Factored the provenance-linking out of
_apply_sidecar into a shared _attach_provenance so the fresh-import and
recapture-backlink paths can't diverge. Idempotent; generic across native
platforms (no-op for already-linked Patreon/SubscribeStar). Re-running recapture
now repairs orphaned images; future walks never orphan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags are kind=general, so their suggestions previously landed in the
General group. Give them their own 'system' suggestion category so the operator
reviews them apart from content tags: _current_heads maps is_system heads to
category 'system' (still trained as general heads, still gated by the 0.65
floor). Frontend: CATEGORY_ORDER/LABELS gain 'system'; SuggestionsPanel renders
a 'System' group first (small, collapsible, open — false positives easy to spot
and reject); the typed-dropdown shows the shield icon for system entries. Safe:
system-tag suggestions always carry a canonical_tag_id, so the create-by-kind
path (which would send 'system' as a TagKind) is never hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The core writes the post record BEFORE extract_media, but the ugoira frame
delays were only memoized DURING extract_media — so write_post_record never saw
them and ugoira_frames was always empty in the record. Extract a memoized
_ugoira_meta (frames + zip url share ONE /v1/ugoira/metadata call regardless of
order) and inject client.fetch_ugoira_frames into the downloader (mirrors
Patreon's content_fetcher) so write_post_record populates the frames itself.
Zero extra API calls — the fetch is shared/memoized with extract_media. A
recapture now backfills the timings onto existing ugoira posts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags (wip/banner/editor) already get heads (kind=general) and aren't
filtered from suggestions, but they surfaced only at each head's precision-tuned
suggest_threshold — high enough to hide the borderline/false-positive guesses the
operator wants to SEE and REJECT (hard-negative mining: 'negatively reinforce
what isn't a system tag'). score_image now uses a flat _SYSTEM_TAG_SUGGEST_FLOOR
(0.65, operator-set) for system-tag heads instead of their auto threshold;
content-tag heads keep their own, and the typed-dropdown threshold_override still
overrides everything. _current_heads carries Tag.is_system into the head meta to
drive it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Two pre-merge fixes:
1. SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on
the .art domain (keeps 302'ing to /age_confirmation_warning even with the
cookie — Elasid #54116), but the same creator is reachable on .adult where
the cookie works. _normalize_ss_host rewrites subscribestar.art →
subscribestar.adult at request time (stored Source.url untouched), logged so
it's visible in walk logs. .com/.adult pass through.
2. Disabling a source now clears its failure state (last_error, error_type,
consecutive_failures) so subs you pause (not paying for) stop lingering as
'failing'. Only the explicit disable clears — an unrelated edit to an
already-disabled source leaves state alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The native downloader used the Windows-safe sanitize_segment, but gallery-dl on
Linux (path-restrict auto→'/', path-remove default control chars, path-strip
auto→'') replaces ONLY '/' and deletes control chars — the Windows-forbidden set
(<>:"|?*) and trailing dots/spaces stay RAW in on-disk titles. Any pixiv title
with those chars would therefore miss the tier-2 disk-skip and re-download the
whole work at cutover (seen-ledger starts empty). Replace sanitize_segment with
gdl_clean_filename, a byte-exact mirror of gallery-dl 1.32.5 build_filename
(verified against path.py). Directory + template already matched; this closes the
last parity gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The real phase-1 ctx has always carried auth_token; the native branch now
threads it into the adapter constructors, so the stub ctx must match the
contract (kept the strict ctx[...] read — it catches exactly this drift).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the
recover/recapture UI actions now route through PixivIngester. Campaign id is
parsed straight from the source URL (numeric user id — no network resolver),
with a platform-aware resolution-failure message. auth_token now rides the
uniform adapter construction (token platforms use it, cookie platforms
accept-and-ignore), and the preview endpoint fetches/threads it. The legacy
gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the
refresh-token config branches in download/verify) per no-legacy policy;
gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The stub payload is PNG bytes regardless of target extension, so the real
validator quarantined the .jpg cases; and extracting the ugoira work hit the
API seam of a fake session with no .post. Validation/quarantine plumbing
stays covered by the Patreon downloader tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar
ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira
— pximg URLs carry no content hash). PixivIngester wires client/downloader/
ledgers into ingest_core with drift label 'Pixiv app API' and the new
body_canary=False opt-out: caption-less pixiv artists are common, so the
zero-bodies #862 alarm would false-positive here — the client's
response-shape drift checks cover that failure class instead. auth_token
joins the uniform adapter constructor (pixiv is the first token-auth native
platform). verify_pixiv_credential = one OAuth refresh, no feed walk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivDownloader writes originals to the exact pre-cutover gallery-dl layout
(<artist_slug>/pixiv/pixiv/{id}_{title[:50]}_{NN}.{ext} — flat, double
platform segment) so tier-2 disk-skip recognizes existing files. Post-first:
per-media sidecar is identity-only; the post record (_post_<id>.json — id
suffix because the flat layout would collide a bare _post.json) carries the
enrichment: tags + EN translations, rating from x_restrict, series,
view/bookmark/comment counts, AI flag, dimensions, author, and ugoira frame
delays (the zip has no timings). i.pximg.net media GETs ride the app-header
profile (403 without the app-api Referer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivClient mirrors gallery-dl 1.32.5's PixivAppAPI request profile exactly
(iOS app headers, OAuth refresh with X-Client-Time/X-Client-Hash,
/v1/user/illusts pagination via next_url — whose query string doubles as the
resumable page cursor). Post-first seams (post_record_key / post_is_gated /
post_meta) + extract_media covering multi-page, single-page, ugoira zip
(600x600→1920x1080 swap, frame delays memoized for the post record), and
limit_* placeholder gating. No PHPSESSID web fallback: FC holds only the
refresh token, same effective coverage as the gallery-dl path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The four remaining run-1895 failures were stale expectations, not
predicate bugs — prune/reset returned the right counts, but these tests
verified no-deletion by counting the ENTIRE tag table (or asserting the
full kind set), which now includes the three seeded hygiene tags that
survive prunes and resets by design. Filter is_system=false with a
pointer to #128 so future system tags cannot re-break them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
UI: shield marker + tooltip on TagChip and TagCard; system tags hide
rename/merge/delete affordances (chip kebab entirely — set-fandom never
applies to their general kind; remove stays, un-tagging is normal use).
Aliases stay available: mapping model outputs ONTO a system tag is
useful. Directory cards carry is_system.
Every destructive path that could take out a system row is now guarded,
found by sweeping run 1891s off-by-three failures — each one was a
surface that would have eaten the seeded tags:
- prune-unused: predicate exempts is_system (they ship with zero
applications and matched every unused condition)
- reset-content: predicate exempts is_system AND keeps their
applications — hygiene flags describe the file, not content tagging
- admin tag DELETE: refused with system_tag error
- normalize_existing_tags: scan excludes is_system — canonicalization
would recase wip -> Wip behind TagService.rename's guard, breaking
the name-keyed presentation lookup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Step 3 of milestone #128. Presentation-tagged images cluster on UI
chrome rather than content, so near any one of them they fill the whole
more-like-this grid. Excluded from candidates in the ONE whole-image
similarity surface (gallery similar mode, explore walk, and RelatedStrip
all ride GalleryService.similar) — the anchor itself may be a banner,
and wip stays surfaced: only the training pipelines exclude it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
TRUNCATE ... RESTART IDENTITY resets every sequence to 1, and the
baseline restore re-inserts seeded rows WITH their explicit ids —
leaving each sequence pointing below MAX(id). Harmless while the only
baseline rows lived in tables tests never sequence-insert into
(ml_settings id=1); migration 0075 seeded tag rows and every Tag insert
after the first truncate collided on pk_tag id=1 (205 failures, run
1888 — find_or_create then surfaced it as NoResultFound via its
conflict-recovery re-select). setval every restored table with a serial
id column past its restored rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Step 2 of milestone #128. _hygiene_excluded_ids (training_data.py) is the
one shared predicate: images carrying any system tag are dropped from
every OTHER concepts head training — not positives (a rough wip tagged
as a character drags the head toward generic-sketch) and not rejection
or sampled negatives (a wip OF character X is not evidence against X).
A system tags own head trains on them unfiltered; that is what makes
auto-flagging banners work. Selection is split out of train_head as the
sklearn-free head_training_ids so CI (no sklearn) can pin the behavior.
CCIP: reference prototypes skip hygiene-tagged images — a faceless wip
figure region must never become an identity reference — and the ref
cache signature now counts hygiene applications, since tagging an image
wip changes the reference set without touching character/region counts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Training hygiene step 1 (milestone #128). Migration 0075 adds
tag.is_system and seeds wip / banner / editor screenshot (kind=general),
ADOPTING an existing same-(name,kind) tag case-insensitively instead of
duplicating. These rows drive the upcoming training exclusions, so they
are protected: rename and merge-away refuse system tags (merge-INTO
stays allowed — folding an operator's old hygiene tag into the system
row is the intended move; merge is the only tag-delete path, so that
guard covers deletion). is_system rides every tag serialization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Spreading canonical_tag_id onto a raw suggestion changed its _keyOf
identity, so _dropEverywhere missed the actual list row and the panel
kept showing an already-accepted suggestion. Pass the resolved id as an
option instead; pinned with a raw-suggestion spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Three tag-flow gaps in the view modal (and the Explore workspace, which
shares TagPanel):
- the type-to-add dropdown now filters both its sections against the
imageledger applied tags reactively, so a just-added tag disappears
from search the moment the chip rail updates instead of after a
modal refresh
- manually picking or creating a tag the model also suggested routes
through the suggestion-accept flow: the acceptance is recorded for
head training and the row leaves the panel, instead of the add
silently bypassing the feedback loop
- removing a tag reloads the suggestion lists, so a model-suggested tag
returns to the suggestions area (flagged rejected, one-click
reversible) rather than vanishing until the next modal open
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The cookie was pinned to .subscribestar.adult only; cookies are
domain-scoped, so sources on subscribestar.art (Elasid, event #54116)
never sent it and every poll 302d to /age_confirmation_warning. Emit
one line per domain (.com/.adult/.art) with a per-domain presence
check, and admit .art in the platform url_pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator-flagged: the recovered defective files live DEEP in their artists'
back-catalogues — the normal download cadence (by design, via the seen-gates)
will never re-walk them, so recovery's source re-check alone can't bring them
back. The durable per-post handle is the ExternalLink row, which survives the
image delete:
- services/external_links.refetch_links_for_post: reset settled links to
pending (fresh attempt budget, in-flight left alone) + dispatch their
fetches; sha-dedupe at import discards payload files that still exist, so
only the missing file lands.
- recover_defective_image now captures the image's post ids BEFORE the delete
cascades provenance away and resets those posts' links — future recoveries
are surgical automatically (response gains links_reset; source re-check
stays for gallery-dl-native files within walk reach).
- POST /api/admin/posts/refetch-external {external_post_id, source_id?} — the
manual tool for the three files recovered before this fix existed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator-flagged on the deployed .5 build: the autoscaler grew the pool 1→8
against an EMPTY queue (an empty buffer read as 'GPU starving' regardless of
WHY), and every downloader kept polling lease every 10s all night.
- New idle signal straight from the lease results: an empty lease sets _idle,
any jobs clear it. The occupancy-low branch now distinguishes three cases:
queue empty → shed to ONE polling downloader; pinned at the bandwidth cap →
shed toward 3; cap headroom + work flowing → grow.
- Idle lease polls back off exponentially per downloader to
IDLE_POLL_MAX_SECONDS (15 min) and reset the moment work appears — so an
idle night costs one HTTP call per 15 min, and new work is noticed within
at most ~15 min (operator-accepted trade-off).
- UI hint: 'idle — queue empty, lease poll backed off'; /status gains idle.
Agent build 2026-07-02.6.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The Subscriptions Settings tab's bare text-h6 headers adopt the same
uppercase accent section-title + hint convention Maintenance/Cleanup use, with
a one-line hint per section (extension / credentials / downloader / external
file-hosts / schedule defaults). Every settings-ish surface now reads
identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Daily-use reorder of the Subscriptions tab: needs-attention strip first
(FailingSourcesCard moves up from below the Downloads fold — a broken
subscription was invisible unless you went looking), then a new Recent
arrivals card (real downloads only, no-change scans filtered out, artist
links), then the source list. Both cards render nothing when there's nothing
to say.
Retry logic moves into the downloads store (retrySource / retryAllFailing) so
the needs-attention card and the Downloads maintenance menu share one
implementation — single-retry forces past cooldown, bulk keeps cooldown
enforcement, same tally shape. The card's Logs button deep-links into the
Downloads tab pre-filtered (?source_id now watched, not just read on mount).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The Activity tab only knew Celery — the GPU agent (the majority of processing)
and the download pipeline were invisible there. Two new self-polling panels:
- GpuActivityPanel: queue depths + triage verdicts (defects / file-ok /
unprobed, top reason buckets) with a jump to Maintenance -> Failed
processing. The triage detail refetches only when the error count moves.
- DownloadsActivityPanel: 24h stat chips + failing-source names with a jump
into Subscriptions.
Both panels join the Activity tab under Queues+workers AND double as the
Overview health strip (side-by-side grid under the Celery summary) — one
component set, so Overview answers 'is everything healthy?' across all
systems. SystemStatsCards reviewed: content still accurate, left as-is.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Settings IA per the approved A3 design (the old layout was the two-app merge
fossilized):
- Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual
/import scans stay API-level; imports arrive via downloads/extension, heal
via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm
moves to Maintenance → 'Ingestion & filters' and loads its own settings; the
import store shrinks to settings-only (no remaining consumers of the
scan/task-list machinery). Overview's pending banner now points at Activity.
- Maintenance regrouped: Ingestion & filters / GPU agent & embeddings
(GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders,
Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive
re-extract demoted last) / Storage.
- One extension home: BrowserExtensionCard moves from Settings → Overview to
Subscriptions → Settings, above the API key bar it authenticates.
- Single-color import filter WIRED: skip_single_color/threshold existed since
FC-2 but nothing read them (the audit module's docstring said as much) —
now enforced on both import paths via the audit's canonical predicate
(tolerance 30, matching the Cleanup card default; animated images exempt
like the transparency check). Default stays off; test added.
- Dead weight: PlaceholderView (zero refs) and the permanently-disabled
'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed
(celery queue docstring, threshold comment citing retired tasks, ml
package docstring, HeadsCard 'replaces Camie' blurb).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).
Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
quick lane — it lived on the 'ml' queue only by module colocation, which made
the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
installs keep working): OFF stops the four import hooks queueing embed work
nothing will consume and no-ops the manual backfill; switch lives on the
renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
that removes the container gives those up too.
Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg
downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real
constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now
gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the
cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint
says 'holding at the bandwidth cap' and /status reports bw_capped, so the
behavior is legible without tests that need the ML stack.
Reset content tagging: stays a FULL-instance reset (operator's call), but now
lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a
preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale
counts are rejected server-side). Copy no longer claims suggestions repopulate:
it says plainly the heads' training examples are deleted and re-tagging starts
fresh. Moved out of TagMaintenanceCard into DangerZoneCard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding +
trained head' spine; the operator accepted the tagging system and dropped the
harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint,
tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry,
TagEvalRun model + table (migration 0073), and its tests.
The eval's data loaders + metric helpers were NOT eval-specific — the nightly
heads trainer runs on them — so they moved verbatim to
services/ml/training_data.py (heads.py import updated; behavior unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's
daily embed backfill for the same NULL-embedding images at ~100x the cost
(B1 audit verdict, milestone #124). The backfill TASK stays — the manual
/api/ml/backfill button remains the deliberate CPU fallback pending B3.
- purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on
the live library before removal (A2 audit, milestone #123). Fully retired
per rule 22: tile, store action, route, service fn, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited;
live MB/s dial + net readout in the control UI) is charged by every still
download (streamed chunk reads) and every ffmpeg video stream (metered from
outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget).
Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8
unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's
browser included). Capping the aggregate keeps the desktop usable and still
beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).
One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.
Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
reason instead of another release; strikes never count while stopping, and
clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
GpuJobService so they can't drift) converts expired leases with >=5 lease
grants and pending jobs with >=10 to 'error', preserving the last stored
failure reason. Backstops old agent builds.
Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head
existed, TagEvalCard whenever a persisted run existed — so a fresh Settings
load greeted the operator with several tiles already expanded. All three now
force-open only while their task is actually running (the #877 resurface
behavior on the busy-driven tiles is untouched).
MaintenanceTile additionally persists MANUAL expand/collapse per tile in
localStorage, so the section reloads the way the operator left it; a forced
open while a task runs stays transient and is never saved as a preference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
sample_frames_from_url now returns (frames, reason) — reason carries the
SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found",
or the timeout) instead of only logging it agent-side. The worker folds it
into the failure it reports, so curator's GpuJob.error reads e.g.
no frames sampled from video — ffmpeg exit 183: moov atom not found ...
instead of the bare "(unprocessable)". The errored-jobs list becomes
self-describing: after a retry sweep, surviving errors name their real
defect without needing the agent log. Return-value plumbing (not shared
state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the "no frames sampled from video (unprocessable)" flood
(operator-flagged 2026-07-02, whole 62k-70k image block + others): the
sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4)
frames — which is ZERO for any clip shorter than ~2s. Short animation loops
(0.5s, 1.75s — verified against two originals from different artists) are
complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames,
exited 0, and the agent failed the job as unprocessable. Long videos worked,
so only the short-clip class flooded.
Fix: sample with select ("first frame always, then one per interval of
timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range
yuv420p sources don't trip the mjpeg encoder's full-range strictness
(secondary failure observed on a 4440x2760 clip). Verified locally against
both failing originals (frames extracted, PIL-clean) and a synthetic 15s
video (4 frames at t=0/4/8/12 — long-video behavior unchanged).
Observability (why this hid for weeks): ffmpeg's stderr was discarded, so
every failure logged only "no frames sampled". stderr now goes to a temp
file and its tail is logged on any produced-no-frames/timeout failure — the
log names the actual ffmpeg reason from now on. Also: frames written before
a mid-stream ffmpeg error are now kept (partial > nothing).
VERSION → 2026-07-02.1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
THE root cause of "the Status section doesn't update" (chased across several
rounds; the backend was always healthy). `#capn` (the max-concurrency number)
was nested inside `#conchint`:
<div id=conchint>… · max <b id=capn>8</b></div>
and applyStatus() ran, every call: `capn.textContent=CAP` AND
`conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces
ALL of conchint's children — destroying the <b id=capn> node. So:
call 1: capn exists → tiles update → conchint.textContent DELETES capn
call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) →
applyStatus throws on its FIRST line → aborts before any tile →
frozen.
This is exactly the observed "ticks a couple times then freezes", and why
/gpu + /logs (which never touch capn) kept updating fine.
The capn write was redundant anyway — conchint.textContent already renders
the max. Remove the nested <b id=capn> element and the capn.textContent line;
the hint still shows "· max N". VERSION → .10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen