Fold the auto-tag accept into the chip: a provisional auto-tag now shows a compact
green ✓ / red ✗ pair IN the pill (replacing the ✕), and the "auto" text label is
dropped — the yes/no is signal enough (operator-asked). ✓ confirms (trains +
shields from retraction), ✗ removes (records a negative). The name still
ellipsis-truncates so the pair stays reachable.
onConfirm now returns focus to the tag input like onRemove already does, so the
input is the cursor's resting position after any chip action in both the modal
and Explore views.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The conflict-flag review strip only appeared when "Show hidden" was toggled on, so
misfires could go unnoticed — defeating the point of flagging them. Fetch pending
flags on mount and show the strip whenever there are any, independent of the
toggle (operator-flagged). Gated to the main gallery (not the post-detail view),
matching the filter bar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
scheduled_presentation_auto_apply (daily beat) runs presentation_auto_apply_sweep
— idempotent, so an interrupted run just re-runs next cycle (that's the recovery),
wall-clock bounded by soft/hard task time limits. prune_presentation_reviews
(daily beat) drops RESOLVED review flags older than 30 days (rule 89 retention).
Tests run both tasks via a monkeypatched session factory. Milestone 141 complete:
the presentation-chrome auto-hide + conflict-flagged review is now live end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
When "Show hidden" is on, a review strip appears atop the gallery listing the
auto-hidden chrome flagged "also looks like content" (most-concerning first):
thumbnail + "also looks like <X>" + Keep hidden / Un-hide. Un-hide removes the
presentation tag (image returns to the gallery) and trains the head; Keep
resolves the flag. Self-hides when there's nothing to review; theme-token styled.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GET /api/gallery/hidden-review lists unresolved presentation auto-hide flags
(image + presentation tag + conflict tag/score), most-concerning first. POST
.../keep resolves the flag (the tag stays). POST .../unhide removes the
presentation tag (image returns to the gallery), records a TagSuggestionRejection
so the head learns it misfired, and resolves the flag. Tests for list/keep/unhide.
Frontend review strip (shown when Show-hidden is on) next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
HeadsCard gains a "Hide presentation chrome" section: on/off switch + "Hide
confidence" (presentation_auto_apply_threshold) + "Flag if content ≥"
(presentation_conflict_threshold), wired to MLSettings via patchSettings and
loaded on mount. Makes the step-4 sweep's thresholds operator-tunable
(config-in-UI). wip is called out as never auto-hidden.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
presentation_auto_apply_sweep fires banner/editor-screenshot heads at the FLAT
presentation threshold (source=presentation_auto). Two guards: (1) hard-skip any
image already carrying a human/confirmed content tag — you valued it, so the model
can't bury it; (2) if an auto-hide ALSO scores >= presentation_conflict_threshold
on a content head, hide it but record a PresentationReview row (conflict tag +
score) for the Hidden view.
_auto_apply_heads now excludes system tags, so a graduated wip/banner can't fire
via the content path (and wip never auto-applies at all). presentation_auto added
to _AUTO_SOURCES so auto-hidden chrome never self-trains. Tests: applies,
hard-skip valued, conflict-flag, disabled no-op, ignores wip, content-path
excludes system. Settings UI + scheduling land next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
MLSettings gains presentation_auto_apply_enabled / _threshold (default 0.90) +
presentation_conflict_threshold (default 0.50): banner/editor auto-hide with a
FLAT threshold (decoupled from content-head graduation), plus the "also looks
like content" conflict cut. New presentation_review table (image, presentation
tag, conflict tag + score, created/resolved_at) records auto-hides flagged for
review. Migration 0082 (columns + table), ml_admin API (editable + get_settings
+ _validate bounds), settings roundtrip/bounds test. The sweep that reads these
knobs + the Settings UI land in step 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
A Curation-group chip in the facet panel flips include_hidden, threaded through
the gallery filter store (default model, activeFilterParam, applyFilterFromQuery,
cloneFilter, filterToQuery) and counted in the refine badge. Off by default → the
gallery hides banner/editor-screenshot chrome; on → it's revealed. Backend
already honors include_hidden (step 1). The dedicated conflict-flagged review
surface (only the set-aside items) lands in step 5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The /similar route splats **filters into similar(), which doesn't take the new
include_hidden kwarg → TypeError → 500 (test_gallery_similar). Drop it like
post_id; similar() has its own presentation exclusion (#1274), so the
gallery-browse flag doesn't apply there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The default gallery + facets now implicitly exclude images carrying a
presentation system tag (banner / editor screenshot), reusing the tag-scope
EXISTS machinery. Suppressed when the operator explicitly filters FOR a
presentation tag OR passes include_hidden (the Hidden view — step 2). `wip` is
NOT hidden (real, in-progress art). include_hidden threaded through
scroll/timeline/jump_cursor/facets + the gallery API _parse_filters. Test covers
default-hide, include_hidden, explicit-filter-shows, and wip-stays-visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
A character+fandom(+AUTO) chip could exceed the tag-panel width, pushing the
close ✕ off the right edge so the tag couldn't be removed (operator-flagged with
a "Mirko - Rumi Usagiyama → My Hero Academia AUTO" screenshot). Cap the chip at
100% of the rail and make the NAME the elastic part (ellipsis-truncates), so the
✕, fandom, and AUTO badge stay reachable; the full name stays on the hover title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The ✓ Keep on a provisional auto-tag chip was a smaller tinted-outline variant;
make it the SAME filled green circle (white ✓, 26px, opacity 0.9→1 + scale on
hover, accent focus ring) as the suggestion accept button (.fc-act--yes in
SuggestionItem) so "accept this tag" reads identically across surfaces
(operator-asked).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- heads.py: conf_map = dict(conf) instead of a dict comprehension (ruff C416).
- postCard.spec.js: the modal-playlist rename (postImageIds→playlistIds) missed
this frontend test (grep was src-only); update the expected call args.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The confirm-UI change added source + confirmed to serialize_tag; two exact-dict
unit tests in test_tag_query.py failed on the new keys. Add them (default
None/False for rows without image scope).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The image modal cycled GLOBAL neighbours; now the gallery hands it a snapshot of
the currently-filtered, ordered id list so prev/next moves through exactly what
you're viewing — the filtered-playlist behaviour lost in the ImageRepo→FC move.
Generalized the modal store's post-scoped cycle into a `playlistIds` playlist
reused by both GalleryView and PostCard (falls back to global neighbours when no
playlist is passed, e.g. Explore's "open full viewer").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Completes "no self-training": unconfirmed auto-applied character tags no longer
seed CCIP references — character_references + the prototype builder
(_current_fingerprints/_rebuild_one) gain a shared _positive_char_tag filter
(human-applied OR operator-confirmed), mirroring the head-positive exclusion.
Confirming a tag also has to move the change-detectors, or an incremental
refresh/Retrain right after a confirm wouldn't fold the tag in (only the nightly
full pass would): the CCIP global gate now counts character confirmations, and
the head training fingerprint counts confirmations. Test for the CCIP path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Auto-applied tags are provisional (they don't train the model + can be retracted
until confirmed), so surface and confirm them:
- Backend: list_for_image + get_image_with_tags now include `source` + a
`confirmed` flag on each applied tag (via serialize_tag, image-scoped; defaulted
for autocomplete/directory callers).
- Frontend: TagChip badges an unconfirmed auto-tag with an "auto" pill + a
one-click Keep/confirm (✓) → POST /images/<id>/tags/<id>/confirm, which promotes
it to a training positive and shields it from the retraction sweep; TagPanel
reloads so the badge + button drop once confirmed.
Contract test for the source/confirmed payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The keystroke debounce cleared the timer but not an already-fired fetch, so a
slower earlier-prefix response ("s") could land after "sex" and overwrite the
dropdown with wrong-prefix matches (operator-flagged with a "sex"→Stockings/
Super Mario screenshot). Gate each autocomplete response on a useInflightToken
(cancel on every keystroke, isCurrent() after the await) so only the latest
query's results are applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Milestone 139 raised head_auto_apply_precision 0.97→0.98; operator confirmed the
general-tag confidence was already well tuned, so revert that. The support floor
(min_positives 30→50) and CCIP match confidence (0.92→0.95) stay. Migration 0081
(not yet deployed) edited to drop the precision bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Each category section header gets a subtle "Reject rest" action that dismisses
every still-unhandled suggestion in it at once (store.dismissRemaining, parallel
dispatch). Canonical tags persist a rejection and stay flagged (reversible,
one-click un-reject); raw creates-new-tag rows drop client-side. Shows only when
the section has unhandled items. No confirm dialog — it's fully reversible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Makes auto-apply truly "soft" for heads: _ids_with_tag (head positives) and
_eligible_tag_ids (graduation count) now count human-applied + operator-confirmed
tags only, via a shared _AUTO_SOURCES (head_auto/ccip_auto/ml_auto) exclusion.
Unconfirmed auto-applied tags no longer train the head that judges them, so a
misfire can't reinforce itself and the retraction sweep can actually drop it.
Confirming a tag (TagPositiveConfirmation) promotes it to a positive AND protects
it from retraction. sklearn-free tests. CCIP reference exclusion is the companion
piece, next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The stricter head_auto_apply_min_positives (30→50, migration 0081) dropped the
_head helper's default n_pos=30 below the support floor, so the "supported head"
sweep tests saw the head as ineligible (n_applied 0). Move the default to 60; the
explicit n_pos=5 under-supported test stays correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Daily scheduled_retract_auto_tags re-scores standing auto-applied tags and drops
the ones the model no longer supports:
- retract_auto_applied_heads: per graduated head, re-score its source='head_auto'
images (bounded — only the images already carrying the auto-tag, not the whole
library) and remove ones now < auto_apply_threshold.
- retract_auto_applied_ccip: per source='ccip_auto' character tag, max-cosine the
image's figure vectors vs that character's prototypes; remove ones now below the
ccip auto-apply threshold.
Both SKIP operator-confirmed tags (TagPositiveConfirmation) and are SILENT — a low
score isn't proof the tag was wrong, so no hard negative is recorded (that's
reserved for an operator removal). No-op unless the relevant auto-apply switch is
on. New daily beat. sklearn-free tests for both paths + the disabled no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
head_auto_apply_precision 0.97→0.98, head_auto_apply_min_positives 30→50,
ccip_auto_apply_threshold 0.92→0.95 (operator-asked). Model defaults change for
fresh installs; migration 0081 bumps the existing singleton row IFF still at the
old default (won't clobber a deliberate operator change). ml_admin bounds already
permit these. Fixed a stale comment in the auto-apply test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
train_all_heads is now incremental by default: a per-tag training-data
fingerprint (positive + rejection count/latest-timestamp, stored on
tag_head.train_fingerprint) means a manual Retrain refits ONLY the tags whose
data changed — O(what you touched), not O(all heads). The nightly
scheduled_train_heads passes full=True to reconcile sampled-negative + hygiene
drift across every head. First incremental run after deploy still refits
everyone (NULL fingerprints), stamping them, then it's incremental.
The refit decision + fingerprint are split into sklearn-free helpers
(_head_fingerprints, _heads_needing_retrain) so the incremental logic is
unit-tested directly (train_head itself needs scikit-learn). Migration 0080.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now sources character references from character_prototype via a
per-character in-process cache (_load_prototypes) that reloads ONLY the
characters whose ccip_prototype_state.updated_at advanced — no request-path
rebuild, so the per-accept ~4s stall is gone once the store is populated. Cold
start (store empty pre-first-refresh) falls back to the legacy on-the-fly
reference build, so character suggestions work immediately post-deploy and the
background refresh populates the store within ~15 min. Match math + grounding
are unchanged; existing tests exercise the legacy fallback, and a new test
covers matching from the populated prototype store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- refresh_character_prototypes celery task wraps the incremental builder (sync
ml worker); returns skipped / rebuilt=N removed=N.
- Beat: every ~15 min (cheap global-gate no-op when idle) + a nightly full=True
reconcile as belt-and-suspenders.
- train_heads enqueues it on success, so the Retrain button AND the nightly head
retrain refresh CCIP on the SAME trigger — unified lifecycle, as asked.
The initial (cold) full build loads the whole reference set once in the
background, never on a /suggestions request.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
refresh_character_prototypes (sync, celery ml worker):
- Cheap GLOBAL gate (a few COUNTs) → no-op when nothing that affects references
changed since the last refresh (the operator's "only recompute if something
was tagged" trigger).
- Else a per-character fingerprint diff (one GROUP BY: ref count + max region id)
rebuilds ONLY the characters whose references moved — each capped to
MLSettings.ccip_prototype_cap — and drops characters that lost all refs.
Cost scales with WHAT changed, not library size. Reuses ccip's reference
predicate (single-character, non-hygiene, figure CCIP) so prototypes match the
legacy matcher exactly. The async matcher (next step) will READ the table.
Tests: gate no-op when idle, only-changed-character rebuild, capping,
single-character exclusion, lost-reference cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Foundation for making CCIP character references a precomputed, INCREMENTAL
artifact instead of a request-path rebuild (kills the per-accept ~4s suggestions
stall; cost will scale with change, not library size):
- character_prototype: a character's reference CCIP vectors, capped to
MLSettings.ccip_prototype_cap so match cost doesn't grow with popularity.
- ccip_prototype_state: per-character fingerprint (ref count + max region id) +
updated_at → drives per-character incremental rebuilds and the matcher cache's
reload-only-what-advanced.
- MLSettings.ccip_ref_signature (cheap global change gate) + ccip_prototype_cap.
Migration 0079. Schema + models only — the builder service, refresh task/beat,
and matcher rewrite land in the following steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- TagAutocomplete: focus the inner input with preventScroll so handing focus
back after an accept/reject stops yanking the rail up to the field (which sits
above the suggestions list — you had to scroll back down every time). Applies
to both the image modal and the Explore rail.
- ExploreView: pad the right rail's scroll bottom (88px) so the bottom-right
snackbar floats over empty space instead of covering the last suggestions and
their accept/reject controls.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The overlay label showed only the crop origin (booru:head / panel / person-m).
Put the hovered TAG on top as the headline and demote the crop origin to a small,
dimmed, monospace subline — the tag is what you're evaluating; the origin is just
provenance. Threads the tag name through the fcSuggestionHover payload ({g, tag})
from both setters (SuggestionItem for suggestions, TagChip for applied chips).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PreviewDialog.vue was orphaned — nothing mounted it — so the dry-run preview
was unreachable. Rather than wire it up, remove the whole chain: its unique
value (a capped 3-page count of what a backfill would grab) is low, and its
adjacent needs are already covered — auth validation by verify_source_credential,
and actually fetching recent posts by "Check now". Operator decision 2026-07-06.
Removed:
- frontend: PreviewDialog.vue + sources store previewSource()
- backend: POST /api/sources/<id>/preview route, download_backends.preview_source,
IngestCore.preview() + its now-unused NativeIngestError import
- tests: the 3 ingester preview tests
Nothing else referenced the chain (verified). Shared campaign-resolution and
ledger helpers stay — they're used by run()/verify_source_credential.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Applied tags aren't scored live, so compute the grounding on demand: run the
tag's head over the image's max-over-bag (whole-image + concept crops), argmax
→ the region that best explains the tag on this image, mirroring what
score_image records for live suggestions.
- heads.py: extract _image_bag (now shared by score_image) + ground_applied_tag.
Returns (grounding, has_head): has_head False = no head to localize with →
no overlay; grounding None = the whole-image vector won → whole-image frame.
- tags.py: GET /api/images/<id>/tags/<id>/grounding → {grounding, has_head}.
- TagChip/TagPanel: applied chips inject fcSuggestionHover and fetch grounding
on hover (cached per image+tag, race-guarded), reusing Step 3's overlay in
both the modal and Explore. No new frontend overlay code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The payoff: hover a suggestion in the rail and the exact crop that produced it
lights up on the image (a booru:head, a panel, a figure); a null-grounding tag
shows a subtle dashed whole-image frame ('global vector won, not a crop').
ImageCanvas gains a grounding overlay that tracks the <img>'s live bounding rect
(correct under object-fit letterboxing + pan/zoom) and draws the normalized bbox
+ a detector/kind label. SuggestionItem sets the hovered grounding via
provide/inject (no 4-level event relay through TagPanel/SuggestionsPanel/group);
ImageViewer AND ExploreView provide it + pass it to their canvas. Overlay is
pointer-events:none so it never blocks pan/zoom/click. Videos out of v1 scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
match_image now tracks WHICH query figure produced the winning cosine per
character (argmax over the per-figure best-reference sim) and attaches its bbox as
grounding {bbox,kind:'figure',detector}. SuggestionService carries it: a CCIP-only
character hit grounds to its figure; a 'both' hit keeps the head's localized crop
if it had one, else falls back to the CCIP figure — so corroborated characters
stay grounded. Test: a character match carries the matched figure's bbox+kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
score_image now keeps the ARGMAX beside the max-over-bag: which bag row won each
head. The region query also selects bbox/kind/detector_version, a parallel
bag_meta maps each row → its region (None for the whole-image vector), and every
hit gains grounding {bbox,kind,detector} (null when the global vector won). Threaded
through SuggestionService (new Suggestion.grounding field) → /api/.../suggestions
payload. This is the data the #1206 hover-overlay draws. CCIP-only hits ground null
for now (figure grounding = step 2). Tests: winning crop grounds the tag with its
bbox+kind; whole-image win → grounding None.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Exposes the detector config (per-proposer enable + weights + confidence, caps,
dedupe IoU) in Settings → Tagging, backed by MLSettings via /api/ml/settings.
ml_admin adds the detector fields to _EDITABLE + GET payload + validation (conf
0..1, caps >=1, IoU 0..1). New CropProposersCard.vue (mirrors HeadsCard) with
working defaults pre-filled, per-field live-save (no restart — the agent picks
changes up on its next lease), weights-format help, switch-revert on error.
Closes milestone #134: all three proposers are on out-of-the-box and tunable in
the UI. Test: detector defaults GET + patch round-trip + range validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The GPU lease now carries the crop-proposer config from MLSettings in a per-job
'detectors' block (same pattern as embed_model_name). The agent's worker builds
its Proposers from the announced config via _effective_cfg (lease block overlaid
on env) + _proposers_for (rebuilds only when a config signature changes) — so an
operator's UI edit takes effect on the next lease with NO restart, and env is now
just the bootstrap fallback until the server announces. enabled-off maps to empty
weights (proposer skipped); dedupe_iou + max_regions also come from the effective
cfg. Test: lease announces the detectors block with the seeded default weights.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Move the crop-proposer config (per-proposer enable + weights + conf, caps,
dedupe IoU) into the DB so it's UI-tunable and can be announced to the GPU agent
in the lease (like the embedder model) — no restart, agent env becomes
bootstrap-only. Migration 0078 adds the columns with working server_defaults so
existing rows + fresh installs crop out-of-the-box with all three proposers ON
(operator: default-on): person=yolo11n.pt, anatomy=booru_yolo yolov11m_aa22 (URL,
license unstated/private-homelab-OK), panel=mosesb best.pt. Plain columns, no
CHECK enum. Steps 2 (lease announce + agent apply) and 3 (Settings UI) follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Parity with pixiv (operator ask): the extension add now resolves the real
display name for our other native platforms too, not just the URL handle.
patreon_resolver.resolve_display_name reads the campaigns API's
attributes.name; SubscribeStarClient.resolve_display_name pulls the creator
name off the profile page (og:title, else the <title> stripped of the
SubscribeStar suffix). extension_service._resolve_artist_name dispatches per
platform (pixiv=token, patreon/subscribestar=cookies via get_cookies_path),
best-effort in an executor, falling back to the readable URL handle on any
failure. Still all curator core — the extension is unchanged (sends only the
URL). gallery-dl platforms keep the handle (readable, no native client).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Final piece of the artist decoupling. (1) Identity-by-source: quick_add_source
resolves the artist by an existing (platform, url) Source first, so a re-add
reuses the artist even after it was renamed (its frozen slug no longer matches
the name) — a slug-based lookup would have duplicated it. (2) Pixiv naming: a new
pixiv source resolves the real display name via the app API (PixivClient
.resolve_display_name → /v1/user/detail) using the stored token, so the artist is
'Kurotsuchi Machi' not '12345678' — and its name-derived slug matches what a
native download produces, unifying them. Falls back to the numeric id when no
token/crypto. ExtensionService gains the crypto seam; the endpoint passes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator ask: a surface to merge new sources into existing artists (consolidate
the singleton artist a fresh add spins up). Enabled by the #130 slug decoupling —
the storage path is immutable, so re-attribution moves NO files. SourceService
.reassign moves the source, re-points its posts (Post.source_id==S) and the
images it contributed (ImageProvenance via S, scoped to the old artist so shared
images aren't stolen), and deletes the old artist if it's left fully empty (else
clears its subscription flag). POST /api/sources/<id>/reassign. Frontend: a
'Move…' action per source on the artist Management tab → artist-autocomplete
picker → confirm → routes to the target (whose slug is stable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
First step of decoupling artist identity/storage/display. migration 0077 drops
uq_artist_name so the display name is free text (two genuinely different creators
can share a name); the slug stays the immutable, unique storage/identity key (the
on-disk path component — untouched, so nothing moves). ArtistService.rename +
PATCH /api/artists/<id> change the name ONLY. Frontend: inline pencil-edit on the
artist header (mirrors TagCard), slug/route unaffected so no navigation. Fixes the
operator's 'no surface to rename an artist' + the name-collision fragility.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Recapture disk-skips already-downloaded media, and upsert_post_record only
writes Post fields — so a pre-existing image (e.g. one pulled under the old
gallery-dl path, imported bare with no post) stays orphaned even after its post
record is (re)written. Confirmed on the operator's instance: 329 pixiv images
with primary_post_id NULL, 694 pixiv posts with content but no linked images, 0
duplicate posts.
Fix: the recapture relink channel now carries the media's post_id (2- → 3-tuple
path/url/post_id), and phase 3 calls importer.link_existing_image_to_post — match
the on-disk image by path, find its Post by (source, external_post_id), upsert
image_provenance + primary_post_id. Factored the provenance-linking out of
_apply_sidecar into a shared _attach_provenance so the fresh-import and
recapture-backlink paths can't diverge. Idempotent; generic across native
platforms (no-op for already-linked Patreon/SubscribeStar). Re-running recapture
now repairs orphaned images; future walks never orphan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags are kind=general, so their suggestions previously landed in the
General group. Give them their own 'system' suggestion category so the operator
reviews them apart from content tags: _current_heads maps is_system heads to
category 'system' (still trained as general heads, still gated by the 0.65
floor). Frontend: CATEGORY_ORDER/LABELS gain 'system'; SuggestionsPanel renders
a 'System' group first (small, collapsible, open — false positives easy to spot
and reject); the typed-dropdown shows the shield icon for system entries. Safe:
system-tag suggestions always carry a canonical_tag_id, so the create-by-kind
path (which would send 'system' as a TagKind) is never hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The core writes the post record BEFORE extract_media, but the ugoira frame
delays were only memoized DURING extract_media — so write_post_record never saw
them and ugoira_frames was always empty in the record. Extract a memoized
_ugoira_meta (frames + zip url share ONE /v1/ugoira/metadata call regardless of
order) and inject client.fetch_ugoira_frames into the downloader (mirrors
Patreon's content_fetcher) so write_post_record populates the frames itself.
Zero extra API calls — the fetch is shared/memoized with extract_media. A
recapture now backfills the timings onto existing ugoira posts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
System tags (wip/banner/editor) already get heads (kind=general) and aren't
filtered from suggestions, but they surfaced only at each head's precision-tuned
suggest_threshold — high enough to hide the borderline/false-positive guesses the
operator wants to SEE and REJECT (hard-negative mining: 'negatively reinforce
what isn't a system tag'). score_image now uses a flat _SYSTEM_TAG_SUGGEST_FLOOR
(0.65, operator-set) for system-tag heads instead of their auto threshold;
content-tag heads keep their own, and the typed-dropdown threshold_override still
overrides everything. _current_heads carries Tag.is_system into the head meta to
drive it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Two pre-merge fixes:
1. SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on
the .art domain (keeps 302'ing to /age_confirmation_warning even with the
cookie — Elasid #54116), but the same creator is reachable on .adult where
the cookie works. _normalize_ss_host rewrites subscribestar.art →
subscribestar.adult at request time (stored Source.url untouched), logged so
it's visible in walk logs. .com/.adult pass through.
2. Disabling a source now clears its failure state (last_error, error_type,
consecutive_failures) so subs you pause (not paying for) stop lingering as
'failing'. Only the explicit disable clears — an unrelated edit to an
already-disabled source leaves state alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The native downloader used the Windows-safe sanitize_segment, but gallery-dl on
Linux (path-restrict auto→'/', path-remove default control chars, path-strip
auto→'') replaces ONLY '/' and deletes control chars — the Windows-forbidden set
(<>:"|?*) and trailing dots/spaces stay RAW in on-disk titles. Any pixiv title
with those chars would therefore miss the tier-2 disk-skip and re-download the
whole work at cutover (seen-ledger starts empty). Replace sanitize_segment with
gdl_clean_filename, a byte-exact mirror of gallery-dl 1.32.5 build_filename
(verified against path.py). Directory + template already matched; this closes the
last parity gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The real phase-1 ctx has always carried auth_token; the native branch now
threads it into the adapter constructors, so the stub ctx must match the
contract (kept the strict ctx[...] read — it catches exactly this drift).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the
recover/recapture UI actions now route through PixivIngester. Campaign id is
parsed straight from the source URL (numeric user id — no network resolver),
with a platform-aware resolution-failure message. auth_token now rides the
uniform adapter construction (token platforms use it, cookie platforms
accept-and-ignore), and the preview endpoint fetches/threads it. The legacy
gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the
refresh-token config branches in download/verify) per no-legacy policy;
gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The stub payload is PNG bytes regardless of target extension, so the real
validator quarantined the .jpg cases; and extracting the ugoira work hit the
API seam of a fake session with no .post. Validation/quarantine plumbing
stays covered by the Patreon downloader tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar
ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira
— pximg URLs carry no content hash). PixivIngester wires client/downloader/
ledgers into ingest_core with drift label 'Pixiv app API' and the new
body_canary=False opt-out: caption-less pixiv artists are common, so the
zero-bodies #862 alarm would false-positive here — the client's
response-shape drift checks cover that failure class instead. auth_token
joins the uniform adapter constructor (pixiv is the first token-auth native
platform). verify_pixiv_credential = one OAuth refresh, no feed walk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivDownloader writes originals to the exact pre-cutover gallery-dl layout
(<artist_slug>/pixiv/pixiv/{id}_{title[:50]}_{NN}.{ext} — flat, double
platform segment) so tier-2 disk-skip recognizes existing files. Post-first:
per-media sidecar is identity-only; the post record (_post_<id>.json — id
suffix because the flat layout would collide a bare _post.json) carries the
enrichment: tags + EN translations, rating from x_restrict, series,
view/bookmark/comment counts, AI flag, dimensions, author, and ugoira frame
delays (the zip has no timings). i.pximg.net media GETs ride the app-header
profile (403 without the app-api Referer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
PixivClient mirrors gallery-dl 1.32.5's PixivAppAPI request profile exactly
(iOS app headers, OAuth refresh with X-Client-Time/X-Client-Hash,
/v1/user/illusts pagination via next_url — whose query string doubles as the
resumable page cursor). Post-first seams (post_record_key / post_is_gated /
post_meta) + extract_media covering multi-page, single-page, ugoira zip
(600x600→1920x1080 swap, frame delays memoized for the post record), and
limit_* placeholder gating. No PHPSESSID web fallback: FC holds only the
refresh token, same effective coverage as the gallery-dl path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The four remaining run-1895 failures were stale expectations, not
predicate bugs — prune/reset returned the right counts, but these tests
verified no-deletion by counting the ENTIRE tag table (or asserting the
full kind set), which now includes the three seeded hygiene tags that
survive prunes and resets by design. Filter is_system=false with a
pointer to #128 so future system tags cannot re-break them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
UI: shield marker + tooltip on TagChip and TagCard; system tags hide
rename/merge/delete affordances (chip kebab entirely — set-fandom never
applies to their general kind; remove stays, un-tagging is normal use).
Aliases stay available: mapping model outputs ONTO a system tag is
useful. Directory cards carry is_system.
Every destructive path that could take out a system row is now guarded,
found by sweeping run 1891s off-by-three failures — each one was a
surface that would have eaten the seeded tags:
- prune-unused: predicate exempts is_system (they ship with zero
applications and matched every unused condition)
- reset-content: predicate exempts is_system AND keeps their
applications — hygiene flags describe the file, not content tagging
- admin tag DELETE: refused with system_tag error
- normalize_existing_tags: scan excludes is_system — canonicalization
would recase wip -> Wip behind TagService.rename's guard, breaking
the name-keyed presentation lookup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Step 3 of milestone #128. Presentation-tagged images cluster on UI
chrome rather than content, so near any one of them they fill the whole
more-like-this grid. Excluded from candidates in the ONE whole-image
similarity surface (gallery similar mode, explore walk, and RelatedStrip
all ride GalleryService.similar) — the anchor itself may be a banner,
and wip stays surfaced: only the training pipelines exclude it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
TRUNCATE ... RESTART IDENTITY resets every sequence to 1, and the
baseline restore re-inserts seeded rows WITH their explicit ids —
leaving each sequence pointing below MAX(id). Harmless while the only
baseline rows lived in tables tests never sequence-insert into
(ml_settings id=1); migration 0075 seeded tag rows and every Tag insert
after the first truncate collided on pk_tag id=1 (205 failures, run
1888 — find_or_create then surfaced it as NoResultFound via its
conflict-recovery re-select). setval every restored table with a serial
id column past its restored rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Step 2 of milestone #128. _hygiene_excluded_ids (training_data.py) is the
one shared predicate: images carrying any system tag are dropped from
every OTHER concepts head training — not positives (a rough wip tagged
as a character drags the head toward generic-sketch) and not rejection
or sampled negatives (a wip OF character X is not evidence against X).
A system tags own head trains on them unfiltered; that is what makes
auto-flagging banners work. Selection is split out of train_head as the
sklearn-free head_training_ids so CI (no sklearn) can pin the behavior.
CCIP: reference prototypes skip hygiene-tagged images — a faceless wip
figure region must never become an identity reference — and the ref
cache signature now counts hygiene applications, since tagging an image
wip changes the reference set without touching character/region counts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Training hygiene step 1 (milestone #128). Migration 0075 adds
tag.is_system and seeds wip / banner / editor screenshot (kind=general),
ADOPTING an existing same-(name,kind) tag case-insensitively instead of
duplicating. These rows drive the upcoming training exclusions, so they
are protected: rename and merge-away refuse system tags (merge-INTO
stays allowed — folding an operator's old hygiene tag into the system
row is the intended move; merge is the only tag-delete path, so that
guard covers deletion). is_system rides every tag serialization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Spreading canonical_tag_id onto a raw suggestion changed its _keyOf
identity, so _dropEverywhere missed the actual list row and the panel
kept showing an already-accepted suggestion. Pass the resolved id as an
option instead; pinned with a raw-suggestion spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Three tag-flow gaps in the view modal (and the Explore workspace, which
shares TagPanel):
- the type-to-add dropdown now filters both its sections against the
imageledger applied tags reactively, so a just-added tag disappears
from search the moment the chip rail updates instead of after a
modal refresh
- manually picking or creating a tag the model also suggested routes
through the suggestion-accept flow: the acceptance is recorded for
head training and the row leaves the panel, instead of the add
silently bypassing the feedback loop
- removing a tag reloads the suggestion lists, so a model-suggested tag
returns to the suggestions area (flagged rejected, one-click
reversible) rather than vanishing until the next modal open
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The cookie was pinned to .subscribestar.adult only; cookies are
domain-scoped, so sources on subscribestar.art (Elasid, event #54116)
never sent it and every poll 302d to /age_confirmation_warning. Emit
one line per domain (.com/.adult/.art) with a per-domain presence
check, and admit .art in the platform url_pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator-flagged: the recovered defective files live DEEP in their artists'
back-catalogues — the normal download cadence (by design, via the seen-gates)
will never re-walk them, so recovery's source re-check alone can't bring them
back. The durable per-post handle is the ExternalLink row, which survives the
image delete:
- services/external_links.refetch_links_for_post: reset settled links to
pending (fresh attempt budget, in-flight left alone) + dispatch their
fetches; sha-dedupe at import discards payload files that still exist, so
only the missing file lands.
- recover_defective_image now captures the image's post ids BEFORE the delete
cascades provenance away and resets those posts' links — future recoveries
are surgical automatically (response gains links_reset; source re-check
stays for gallery-dl-native files within walk reach).
- POST /api/admin/posts/refetch-external {external_post_id, source_id?} — the
manual tool for the three files recovered before this fix existed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Operator-flagged on the deployed .5 build: the autoscaler grew the pool 1→8
against an EMPTY queue (an empty buffer read as 'GPU starving' regardless of
WHY), and every downloader kept polling lease every 10s all night.
- New idle signal straight from the lease results: an empty lease sets _idle,
any jobs clear it. The occupancy-low branch now distinguishes three cases:
queue empty → shed to ONE polling downloader; pinned at the bandwidth cap →
shed toward 3; cap headroom + work flowing → grow.
- Idle lease polls back off exponentially per downloader to
IDLE_POLL_MAX_SECONDS (15 min) and reset the moment work appears — so an
idle night costs one HTTP call per 15 min, and new work is noticed within
at most ~15 min (operator-accepted trade-off).
- UI hint: 'idle — queue empty, lease poll backed off'; /status gains idle.
Agent build 2026-07-02.6.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The Subscriptions Settings tab's bare text-h6 headers adopt the same
uppercase accent section-title + hint convention Maintenance/Cleanup use, with
a one-line hint per section (extension / credentials / downloader / external
file-hosts / schedule defaults). Every settings-ish surface now reads
identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Daily-use reorder of the Subscriptions tab: needs-attention strip first
(FailingSourcesCard moves up from below the Downloads fold — a broken
subscription was invisible unless you went looking), then a new Recent
arrivals card (real downloads only, no-change scans filtered out, artist
links), then the source list. Both cards render nothing when there's nothing
to say.
Retry logic moves into the downloads store (retrySource / retryAllFailing) so
the needs-attention card and the Downloads maintenance menu share one
implementation — single-retry forces past cooldown, bulk keeps cooldown
enforcement, same tally shape. The card's Logs button deep-links into the
Downloads tab pre-filtered (?source_id now watched, not just read on mount).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The Activity tab only knew Celery — the GPU agent (the majority of processing)
and the download pipeline were invisible there. Two new self-polling panels:
- GpuActivityPanel: queue depths + triage verdicts (defects / file-ok /
unprobed, top reason buckets) with a jump to Maintenance -> Failed
processing. The triage detail refetches only when the error count moves.
- DownloadsActivityPanel: 24h stat chips + failing-source names with a jump
into Subscriptions.
Both panels join the Activity tab under Queues+workers AND double as the
Overview health strip (side-by-side grid under the Celery summary) — one
component set, so Overview answers 'is everything healthy?' across all
systems. SystemStatsCards reviewed: content still accurate, left as-is.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Settings IA per the approved A3 design (the old layout was the two-app merge
fossilized):
- Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual
/import scans stay API-level; imports arrive via downloads/extension, heal
via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm
moves to Maintenance → 'Ingestion & filters' and loads its own settings; the
import store shrinks to settings-only (no remaining consumers of the
scan/task-list machinery). Overview's pending banner now points at Activity.
- Maintenance regrouped: Ingestion & filters / GPU agent & embeddings
(GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders,
Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive
re-extract demoted last) / Storage.
- One extension home: BrowserExtensionCard moves from Settings → Overview to
Subscriptions → Settings, above the API key bar it authenticates.
- Single-color import filter WIRED: skip_single_color/threshold existed since
FC-2 but nothing read them (the audit module's docstring said as much) —
now enforced on both import paths via the audit's canonical predicate
(tolerance 30, matching the Cleanup card default; animated images exempt
like the transparency check). Default stays off; test added.
- Dead weight: PlaceholderView (zero refs) and the permanently-disabled
'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed
(celery queue docstring, threshold comment citing retired tasks, ml
package docstring, HeadsCard 'replaces Camie' blurb).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).
Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
quick lane — it lived on the 'ml' queue only by module colocation, which made
the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
installs keep working): OFF stops the four import hooks queueing embed work
nothing will consume and no-ops the manual backfill; switch lives on the
renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
that removes the container gives those up too.
Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg
downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real
constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now
gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the
cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint
says 'holding at the bandwidth cap' and /status reports bw_capped, so the
behavior is legible without tests that need the ML stack.
Reset content tagging: stays a FULL-instance reset (operator's call), but now
lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a
preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale
counts are rejected server-side). Copy no longer claims suggestions repopulate:
it says plainly the heads' training examples are deleted and re-tagging starts
fresh. Moved out of TagMaintenanceCard into DangerZoneCard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding +
trained head' spine; the operator accepted the tagging system and dropped the
harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint,
tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry,
TagEvalRun model + table (migration 0073), and its tests.
The eval's data loaders + metric helpers were NOT eval-specific — the nightly
heads trainer runs on them — so they moved verbatim to
services/ml/training_data.py (heads.py import updated; behavior unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's
daily embed backfill for the same NULL-embedding images at ~100x the cost
(B1 audit verdict, milestone #124). The backfill TASK stays — the manual
/api/ml/backfill button remains the deliberate CPU fallback pending B3.
- purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on
the live library before removal (A2 audit, milestone #123). Fully retired
per rule 22: tile, store action, route, service fn, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
One shared TokenBucket (default 8 MB/s; BANDWIDTH_LIMIT_MB_S, 0 = unlimited;
live MB/s dial + net readout in the control UI) is charged by every still
download (streamed chunk reads) and every ffmpeg video stream (metered from
outside via /proc/<pid>/io and SIGSTOP/SIGCONTed into budget).
Why: D1 re-measurement 2026-07-02 — the idle link moves ~38 MB/s, but 8
unthrottled downloaders bufferbloated it to ~1-1.5 MB/s PER STREAM (operator's
browser included). Capping the aggregate keeps the desktop usable and still
beats the collapsed sweep throughput it replaces. Agent build 2026-07-02.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).
One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.
Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
reason instead of another release; strikes never count while stopping, and
clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
GpuJobService so they can't drift) converts expired leases with >=5 lease
grants and pending jobs with >=10 to 'error', preserving the last stored
failure reason. Backstops old agent builds.
Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head
existed, TagEvalCard whenever a persisted run existed — so a fresh Settings
load greeted the operator with several tiles already expanded. All three now
force-open only while their task is actually running (the #877 resurface
behavior on the busy-driven tiles is untouched).
MaintenanceTile additionally persists MANUAL expand/collapse per tile in
localStorage, so the section reloads the way the operator left it; a forced
open while a task runs stays transient and is never saved as a preference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
sample_frames_from_url now returns (frames, reason) — reason carries the
SPECIFIC cause on failure (ffmpeg's stderr tail, e.g. "moov atom not found",
or the timeout) instead of only logging it agent-side. The worker folds it
into the failure it reports, so curator's GpuJob.error reads e.g.
no frames sampled from video — ffmpeg exit 183: moov atom not found ...
instead of the bare "(unprocessable)". The errored-jobs list becomes
self-describing: after a retry sweep, surviving errors name their real
defect without needing the agent log. Return-value plumbing (not shared
state) so concurrent downloaders stay isolated. Agent VERSION → 2026-07-02.2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the "no frames sampled from video (unprocessable)" flood
(operator-flagged 2026-07-02, whole 62k-70k image block + others): the
sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4)
frames — which is ZERO for any clip shorter than ~2s. Short animation loops
(0.5s, 1.75s — verified against two originals from different artists) are
complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames,
exited 0, and the agent failed the job as unprocessable. Long videos worked,
so only the short-clip class flooded.
Fix: sample with select ("first frame always, then one per interval of
timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range
yuv420p sources don't trip the mjpeg encoder's full-range strictness
(secondary failure observed on a 4440x2760 clip). Verified locally against
both failing originals (frames extracted, PIL-clean) and a synthetic 15s
video (4 frames at t=0/4/8/12 — long-video behavior unchanged).
Observability (why this hid for weeks): ffmpeg's stderr was discarded, so
every failure logged only "no frames sampled". stderr now goes to a temp
file and its tail is logged on any produced-no-frames/timeout failure — the
log names the actual ffmpeg reason from now on. Also: frames written before
a mid-stream ffmpeg error are now kept (partial > nothing).
VERSION → 2026-07-02.1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
THE root cause of "the Status section doesn't update" (chased across several
rounds; the backend was always healthy). `#capn` (the max-concurrency number)
was nested inside `#conchint`:
<div id=conchint>… · max <b id=capn>8</b></div>
and applyStatus() ran, every call: `capn.textContent=CAP` AND
`conchint.textContent = '…max '+CAP`. Setting conchint.textContent replaces
ALL of conchint's children — destroying the <b id=capn> node. So:
call 1: capn exists → tiles update → conchint.textContent DELETES capn
call 2+: `capn.textContent` → "capn is not defined" (ReferenceError) →
applyStatus throws on its FIRST line → aborts before any tile →
frozen.
This is exactly the observed "ticks a couple times then freezes", and why
/gpu + /logs (which never touch capn) kept updating fine.
The capn write was redundant anyway — conchint.textContent already renders
the max. Remove the nested <b id=capn> element and the capn.textContent line;
the hint still shows "· max N". VERSION → .10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two follow-ups from live debugging of "work/min never populates" and
"stopped never reached".
1) jobs/min + downloads/min are now computed in the BACKEND on a fixed
cadence (_rate_loop, EWMA) and reported ready-to-show. The rates were
derived client-side from poll deltas with a dt<30s guard — but a
backgrounded/unfocused browser tab throttles its timers to ~1/min, so
every delta exceeded 30s and the guard blanked the rates forever. A
server-side rate is independent of how often the tab polls. Frontend just
displays s.jobs_per_min / s.downloads_per_min. VERSION → .9.
2) ffmpeg video sampling is now killable on Stop. A downloader stuck in a
slow/reconnecting decode (observed: 47s, 230s for one video) couldn't see
the stop signal until ffmpeg returned, so Stop detached still-running
threads and work kept flowing long after — "stopped" that wasn't really
stopped. sample_frames_from_url now runs ffmpeg via Popen and polls a
`should_stop` callback every 0.5s, terminating (then killing) the process
at once on Stop or the per-video timeout. A stop-killed job is handed back
(transient), not failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Status pill hung on "stopping" forever (operator-flagged 2026-07-01).
Root cause: the backend had no lifecycle state — status() only returned
running/stopped — so the UI FABRICATED "stopping" in JS as `!running &&
active>0`. That pill only cleared when the backend's `active` counter hit 0,
but stop() (a) blocked the HTTP handler on lease-release calls to curator and
(b) left `active>0` whenever a consumer wedged mid-submit/release to an
overloaded curator → "stopping" that never resolved.
Give the backend a real, truthful state it drives itself:
stopped → starting → running → stopping → stopped
- start(): → starting; a downloader flips it to running on its FIRST
successful lease (so "running" means curator is actually answering, not
just "Start was clicked"). If curator's down it honestly stays "starting".
- stop(): → stopping; returns immediately (no handler block). A background
monitor waits for the worker threads to actually exit, releases leases,
then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can
NEVER hold the UI in "stopping" again. In-flight work is handed back safely.
- Buttons follow the real state (Start only from stopped; both disabled
through the transition), so you can't fight a transition.
- Log every Start/Stop button press (routes) and every transition (worker),
so the Logs panel shows exactly what each button did.
Frontend now trusts s.state (drops the active>0 hack); VERSION → .8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The buffer / on-GPU / downloader counts flip many times a second, so a 3s
status poll only ever samples noise — the tiles looked frozen (same value
twice) or random (wildly different), reading as "the Status section doesn't
update" when the backend was in fact live (operator-flagged 2026-07-01).
Replace the three instantaneous gauge tiles with two derived RATE tiles:
- jobs / min — GPU throughput, from the monotonic `processed` counter
- downloads / min — fetch throughput, from a new monotonic `downloaded`
counter (bumped when a job is decoded into the buffer)
Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the
reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped
at 0 (agent restart resets the counters), and skip a backgrounded-tab gap.
The still-useful instantaneous state is demoted, not lost: buffer stays as
the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited
out` (transient) gets promoted to a tile.
backend: worker.status() gains `downloaded`; `_bump(downloaded=)`.
frontend: retiled Status + rate math in applyStatus; VERSION → .7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE
file into memory (r.content) just to sample a few frames, which buffered ~1GB in
RAM and — on any slow/contended media store — got cut off mid-download
(ChunkedEncodingError), failed, and re-leased forever. Measured the media read at
~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak
helps; the file simply shouldn't be fully downloaded.
Environment-agnostic fix (works for any deployment, completes even when slow):
- media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL.
It Range-reads only the video index + up to max_frames of content — never the
whole file — and reconnect flags resume a dropped transfer instead of failing.
Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over
speed. Removes the bytes-based sample_frames (dead once videos stream).
- worker._download_decode: videos now stream (no fetch_image, no RAM blowup);
stills still download+decode. On an ffmpeg miss, probe curator liveness
(client.is_reachable) → fail the job if curator is up (unprocessable file, stops
the infinite re-lease) vs release if curator is down (transient, survives a
redeploy). Auth header passed so it works whether or not /images is gated.
Build marker 2026-07-01.6. Refs issue #1225.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
"curator unreachable" was printed for every transient error, hiding whether a
single file's transfer stalled (ReadTimeout — curator is up, that stream is slow)
or curator itself is down (ConnectTimeout/ConnectionError) or errored (HTTP 5xx).
Those need completely different fixes, and we've been diagnosing the download
slowness blind.
Add _transient_reason(exc) → a specific label (HTTP <code>, else the exception
class: ReadTimeout / ConnectTimeout / ConnectionError / …) and use it in both
transient paths:
- downloader: "fetch failed job <id> (image <id>, ReadTimeout) — released, backing off"
- consumer: "submit failed job <id> (<reason>) — released, re-lease later"
Now the logs say which failure it actually is (and which image), so we can tell a
slow/stalled transfer apart from an unreachable curator. Build marker 2026-07-01.5.
Refs issue #1225.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The image library is on a CIFS/SMB share (mounted rsize=4 MiB, actimeo=1), and
Quart's FileBody streams in 8 KiB chunks — so serving one large original was
~19k network round-trips to the storage server, i.e. 30–58s per download
(operator-flagged). That's what starved the GPU agent (constant "curator
unreachable" backoff) AND slowed the browser: every byte is read off CIFS and
streamed through the Python app (no reverse-proxy sendfile), and only 2 hypercorn
workers meant the agent + the browser's thumbnail grid queued behind each other.
In-container fix, no new service:
- Raise FileBody.buffer_size 8 KiB → 4 MiB in create_app, matching the mount's
read size: one round-trip per read, ~500× fewer. buffer_size is the MAX read so
small thumbnails still read in one gulp, and Range/mime/ETag/conditional
handling lives on Response — all preserved. Guarded so a Quart-internal change
can't break boot.
- HYPERCORN_WORKERS default 2 → 4 so concurrent /images requests stop queuing.
Expected: large-file transfers drop from ~40s toward link speed (a few seconds)
for the agent and the browser. See issue #1223.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The gallery's newest/oldest sort keys off image_record.effective_date =
COALESCE(primary post's post_date, created_at). The primary post is often the
repost/download the file came from, so the grid led with download dates rather
than when content was first posted (operator-flagged).
Add a second materialized sort key, earliest_post_date = MIN(post_date) across
ALL of an image's provenance posts (every post it appears in), else created_at —
the original publish date. Mirrors the effective_date pattern so the sort stays a
forward index scan.
- alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill
created_at baseline then MIN over image_provenance ⋈ post.
- importer: recompute earliest_post_date whenever a dated post is linked (MIN over
the image's provenance, which now includes the just-added row).
- gallery_service: new sorts posted_new / posted_old key off earliest_post_date;
cursor + year/month grouping follow the active column transparently.
- api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads
with original publish date. newest/oldest (effective_date) still available.
- frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post
date); existing effective-date sorts relabelled "Newest/Oldest added".
- tests: service test asserts posted_new/posted_old key off earliest_post_date;
frontend default-sort omission test updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator hit an outage after the machine slept overnight: the agent showed
"curator unreachable" in a loop while curator's API (lease) was actually fine and
the browser could still load images — just slowly. Root cause is a feedback loop
in the new pipeline: every download streams a full original through curator's
single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the
buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for
that reason — so the agent piled on more concurrent large-file GETs, saturating
curator's web workers + NFS, which slowed curator (and its browser) further and
produced more failures → more downloaders. Classic congestion collapse.
- Failure-aware autoscaling: if transient download failures rose since the last
decision, SHRINK the downloader pool toward the floor instead of growing — the
empty buffer is caused by failures, not the GPU starving. It ramps back up only
once downloads succeed again.
- DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving
path is too many; 8 keeps a fast GPU fed without stampeding curator.
- fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a
large-but-flowing download still completes, but a stuck/dead connection fails in
60s instead of hanging a downloader for 3 min and piling up stuck requests.
Build marker 2026-07-01.4.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator wants the Explore "related" rail to span more — the #1188 diversifier
was tuned conservatively. Push all three knobs so it reaches further across
clusters instead of clumping near the anchor:
- MMR lam 0.55 → 0.40 — weight the diversity penalty harder (the main dial).
- candidate pool min(200, max(limit*5, 60)) → min(400, max(limit*8, 100)) — a
wider nearest-cosine pool so MMR has genuinely distinct neighbourhoods to pick
from, not just the near-dupes.
- pHash dup_threshold 6 → 8 — collapse more near-duplicate reposts/clones,
freeing rail slots for distinct picks.
Still deterministic (same set per image, just more spread) and relevance-anchored
via the lam*sim-to-anchor term. Backend-only; no migration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Near-static videos are the dominant GPU load: sampled into up to 64 frames, each
re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU
perceptual-hash frame dedup upstream of the GPU so the redundant frames are never
processed at all (not just their embeds).
- media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy
keep — a frame survives only if its hash differs from every kept frame by
>= min_distance bits (Hamming). A static run collapses to one frame; genuinely
distinct scenes all survive. Order + frame_time preserved.
- Called in worker._download_decode right after sample_frames, so it runs in the
decode stage on the downloader thread (CPU) — the GPU consumers only ever see
deduped frames, and buffered video items shrink (less RAM too).
- Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief
localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M`
when it drops any, so video load reduction is visible.
Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal
axis. Build marker 2026-07-01.3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Figure boxes are already NMS-merged (iou 0.6) and each YOLO detector self-NMSes,
but the combined per-frame crop pile (figure→concept ∪ anatomy component→concept
∪ panel) was embedded with no cross-proposer dedup — so genuine near-duplicates
slipped through (a figure box ≈ an anatomy component on a solo bust; overlapping
booru head classes on one head), embedding the same region twice and burning a
slot against max_regions.
Add detectors.dedupe_crops(): a greedy, high-IoU (default 0.85), kind-aware pass
over the pending (crop, template) list right before embed_batch — drop boxes that
overlap ≥ iou within the same kind, keep the highest score. The high threshold is
deliberate: it collapses only true near-identical boxes while preserving
intentional nested crops across scopes (a whole figure vs a small head component
sit well below it) and distinct kinds (concept vs panel). Env-tunable DEDUPE_IOU
(≥1.0 disables). Runs on CPU before the GPU work, so it cuts both embed cost and
region count. Temporal (cross-frame) dedup deferred. Build marker 2026-07-01.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms),
so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit)
left the fast GPU idle during every download. Rearchitect worker.py into a
producer/consumer pipeline:
downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU
consumers (detect+embed→submit)
- Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER
count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a
2nd consumer if it has util/VRAM headroom and lifts throughput, else trim).
- Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping
RAM + lease look-ahead. VRAM pressure sheds a consumer immediately.
- Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU;
curator's 180s TTL would otherwise reclaim them mid-buffer).
- Preserves all resilience: lease exp-backoff, submit-path retry (#169),
release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools
and releases every held lease at once (single held-set as source of truth).
- Consumers SHARE one embedder + proposers instance (a 2nd consumer adds
concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots).
- UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU ·
processed · errors, a buffer-occupancy meter, and a consumers/waited-out line;
the dial now tunes downloaders. Build marker 2026-07-01.1.
Also fix the operator-flagged detector warning: yolo11n + the comic-panel model
threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load-
time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3
crop proposers and spamming the log per image. Disable that fusion (unfused
inference is correct, marginally slower) and permanently self-disable a proposer
on the first inference failure instead of re-throwing forever.
Refs milestone 122.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Image 81602 turned out to be a 156 MB mp4, not a huge still: the agent samples
up to 64 frames × ~32 regions/frame → ~2000 regions (the 413) and 64 frames of
detect+CCIP+embed (the 38s). The MAX_REGIONS backstop (#171) only truncated the
SUBMIT — the GPU work was already spent. Break out of the frame loop once
accumulated regions reach max_regions, so a long video costs ~a few frames of
GPU (~2-3s), not all 64 (~38s). The whole-image 'embed' task is unaffected (it
mean-pools all frames and returns before this loop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Three safety/robustness fixes from the operator's run logs:
- Cap figures per frame (MAX_FIGURES, default 8) like components/panels already
are. Uncapped, a huge/busy image yielded hundreds of figure boxes → hundreds
of per-figure CCIP calls + crops → a 38s job AND a submit too big to accept
(image 81602 looped on 413). This is the acute fix.
- Global per-JOB backstop (MAX_REGIONS, default 128): if total regions still
exceed the cap (long video), keep the highest-scoring and log the drop, so a
submit body can never blow past curator's limit.
- Stale "active" meter: stop() now resets _active to 0 (no slots remain, so the
meter must read 0 at once), and _bump clamps at 0 so a slot finishing after the
reset can't drive it negative.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Trusted local library, not an upload surface, so a legitimately large image
(90–95M px, operator-flagged) must load. PIL only WARNS at the 89M-px default but
RAISES DecompressionBombError at ~179M px, which would fail those jobs. Set
Image.MAX_IMAGE_PIXELS = None. (The agent works off individual extracted files —
curator's archive_extractor unpacks zip/cbz/rar/7z at import — so this is about
big single images, not archives.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Bump the GPU-agent base image from 12.4.1-cudnn-runtime-ubuntu22.04 (Python 3.10,
CUDA 12.4, early-2024) to 12.9.2-cudnn-runtime-ubuntu24.04:
- Ubuntu 24.04 LTS → Python 3.12 — one modern runtime, no more 3.10.
- CUDA 12.9 + cuDNN 9 — current within the CUDA-12 / cuDNN-9 line that the
default onnxruntime-gpu wheel AND torch cu124 are built against. NOT CUDA 13:
ONNX Runtime's CUDA-13 support is still nascent (separate wheels + open
"Unsupported CUDA version: 13" reports), and torch bundles cu124 anyway. The
GPU (Ampere/Ada, 12 GB) is fine on either — this is a library-alignment call,
not a hardware limit.
- PIP_BREAK_SYSTEM_PACKAGES=1: 24.04 marks system Python externally-managed
(PEP 668); a single-purpose container owns its environment, so global installs
are fine and simplest.
- agent/ruff.toml pinned to py312 (was py310) so CI lints against the real
runtime; from __future__ import annotations stays (PEP 649 lazy annotations
are 3.14, so self-refs still evaluate on 3.12).
CI builds the image but has no GPU — validate on the desktop after pull that it
starts and loads CUDAExecutionProvider (not CPU fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The agent container (CUDA base, Python 3.10) crashed on startup with
`NameError: name 'Config' is not defined` — an earlier `ruff --fix` unquoted the
`from_env(cls) -> Config` self-reference, which is safe on CI's Python 3.14
(PEP 649 lazy annotations) but is evaluated at class-definition time on 3.10.
CI lint/compile run on 3.14, so it slipped through.
- config.py: `from __future__ import annotations` so the self-referential
annotation is a string, never evaluated — works on 3.10 and every version.
- agent/ruff.toml: pin the agent to `target-version = "py310"` (its real runtime)
and inherit the root rules. Ruff now flags exactly this class as F821, so CI's
lint lane catches it instead of shipping a broken image. (CI otherwise lints on
3.14, masking 3.10 issues.)
- client.py: submit path now retries in-place. A dedicated session with a
urllib3 Retry (connect/read/status, 0.5s backoff, 500/502/503/504, POST) so a
momentary blip after the GPU work is done doesn't discard it and force a full
re-download + recompute elsewhere. A duplicate submit after a lost response is
a harmless 409 no-op. Lease/fetch keep the plain session + loop-level backoff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Addresses operator reports: Stop never finishes, the agent polls curator
constantly, and stale-cached pages get mistaken for a failed deploy.
- Stop is prompt: flip _running BEFORE any lock so /status + worker loops see
"stopped" immediately, and add a stop/shrink checkpoint in _process (after
decode, before the expensive detect+embed) that releases the job and bails —
so a Stop doesn't wait out heavy GPU work.
- Lazy curator polling: the queue snapshot is fetched only while a browser is
actually watching (a /status hit within UI_IDLE_GRACE) and on a 5s cadence,
not a constant background loop. The work loop's own lease/submit is curator's
only visitor otherwise — nothing polls just to poll.
- Build marker: VERSION is embedded in the page and reported on /status; the UI
shows a "reload" banner when they differ, so a browser-cached page can't be
mistaken for "the new image didn't deploy" (complements the no-store header).
CI: the lint lane now also `ruff check`s agent/ and compileall-parses it, so the
GPU agent is linted + syntax-checked before its image builds (build.yml only
`docker build`s it). Fixed the agent's pre-existing UP037/B905 so it passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Instrument the job pipeline so we can see where wall-clock actually goes and
decide — on data, not theory — whether a download/compute split is worth
building. Each stage is timed per job and a rolling breakdown is logged every
30s to the agent console, e.g.:
timing/30s — lease 8ms · download 310ms · decode 40ms · gpu 165ms · submit 70ms | wall/job 585ms (214 jobs)
- lease timed around client.lease() in the slot loop (per batch).
- download = fetch_image; decode = image/frame decode; gpu = detect + CCIP +
batched embed; submit = the results POST. One-time model load is excluded
from the gpu figure.
- Thread-safe accumulator (stage -> [sum, count]) summarised + reset by a small
daemon reporter thread; logs only when there was work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The throughput bottleneck was curator-side, not the network. lease() claimed the
lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a
plain `status` index Postgres walked the primary key from id=1, skipping the
entire prefix of already done/error rows before reaching pending ones. As `done`
grew (69k+), every lease became an O(done) scan — leasing crawled, the DB
saturated, and even /status (the queue GROUP BY count) stalled the agent.
- Migration 0070 adds two partial indexes over just the live slice: pending rows
indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery
+ orphan sweep). They stay tiny no matter how large the done/error history.
- lease() split into two phases so each uses a partial index: claim pending
first (id-ordered, O(batch)); reclaim expired leases only when pending can't
fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim).
- Model __table_args__ declares the indexes so ORM and schema agree.
- Test: a done-prefix at low ids must not stop the lease reaching pending.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator: the status tiles (state/active/processed) and the Start/Stop buttons
freeze while the GPU meters stay live. Root cause: /status made an INLINE
blocking curator call (queue_status) on every poll, and with curator buried
under a 112k-job backlog that call stalled — freezing the whole status refresh
(the GPU bars survived because /gpu is a lock-free local read). Made worse by the
old util-band autoscaler, which grew workers toward the 32 cap forever because
util plateaus ~50% on this IO-bound load and never hit the 70 grow threshold —
piling load onto curator and the agent process.
- /status is now a pure in-memory read: worker.status() is lock-free, and the
curator queue snapshot is refreshed by a background poller (never inline).
- Autoscaler replaced with a smoothed, throughput-aware climb that SETTLES:
samples util every 2s and EWMA-smooths it (raw util swings 0↔99), then every
~24s grows by one only while each grow keeps lifting smoothed jobs/s; when a
grow stops helping it backs off one and holds, re-probing occasionally. No
runaway, no flopping.
- GPU util bar now shows a smoothed value: the agent's own EWMA (util_smooth,
exposed on /gpu) when running, else smoothed client-side — so it glides
instead of bouncing 0↔99.
- act() aborts a slow Start/Stop POST after 8s so the buttons can't stick; the
now-always-fast /status refresh recovers state regardless.
- Log pane: bound the page to the viewport (height:100vh) so the Logs card
scrolls INTERNALLY instead of overflowing off-screen; cap the ring buffer at
400 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The control page is a static string served with no cache headers, so after
pulling a fresh agent image the browser kept showing the OLD UI until a hard
refresh (operator-flagged). Add a no-store middleware covering the page and the
status/gpu/logs polls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator: the buttons fire but the status view doesn't reflect the change. Cause:
act() ignored the POST's own status response and waited on the separate /status
poll (which lags behind the curator queue call). Now:
- act() applies the POST's returned status immediately for instant feedback, and
shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled)
the moment it's clicked.
- A stop that still has in-flight jobs draining shows "stopping" until active
hits 0, then resolves to "stopped" on its own.
- applyStatus() guards the /status-only fields (connection pill + queue) so the
lean action response can't blank them — the Start/Stop path deliberately skips
the slow curator call to stay snappy.
Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a
time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll +
autoscaler + /status share a single subprocess instead of piling up in the
server thread pool (which was what made clicks feel dead under load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two operator-reported issues with the GPU agent:
1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb
probed +1, judged it over a too-short noisy throughput window, saw no clear
gain and reverted -1 — every tick. Replace it with a GPU-utilization-band
controller: HOLD while smoothed util sits in a healthy band, grow only on
clear spare capacity (util below the low mark + VRAM headroom), shrink under
saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced
(DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool.
Load stays consistent instead of probe/reverting.
2. GPU util/VRAM bars only updated on manual refresh. They rode the /status
poll, which blocks on the curator queue call (slow when curator is busy), so
the meters froze between refreshes. Give them a dedicated /gpu endpoint
(local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop
the curator queue-status timeout 15s -> 5s so /status itself stays snappy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Operator meant the LOG section should fill down the viewport (vertical), not the
whole page going full-width horizontally. Restore the centered column (820px),
make .wrap a full-height flex column, and let the Logs card flex to fill the
remaining height to the bottom (drop the fixed 230px log-pane cap).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s,
truncated-image failures):
- BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo
components + panels) and embed them in ONE forward pass instead of one
forward+lock per crop. The per-crop path serialised every crop through the
inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating);
batching gives a real GPU-bound workload + far higher throughput. CCIP still
runs per figure inline.
- LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly-
truncated scraped images now load instead of failing the job 3× then erroring
("image file is truncated (N bytes not processed)").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- Page fills the viewport horizontally (drop the 780px cap).
- Copy button on the Logs card → copies the console (clipboard API on localhost,
textarea-execCommand fallback), with a brief "Copied" confirmation.
- Silence httpx/httpcore/huggingface_hub/urllib3/filelock/uvicorn.access/
ultralytics to WARNING so the console shows agent activity (detector loads,
job errors, autoscale moves) instead of per-request HF-download spam.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- logbuf.py: bounded in-memory log ring buffer + a logging.Handler on the root
logger; GET /logs serves it; the control page polls it into a console pane —
so runs are monitorable without `docker logs`. worker now logs autoscale moves
(one line per change, with jobs/s + util + VRAM) and job failures (job + image
+ reason); detectors already log load/disable.
- Restyled the whole control page: a proper dark layout with a header + live
connection pill, cards (Control / Status / Logs), a styled Auto switch +
worker stepper, status tiles, separate GPU-util and VRAM meters, and the log
console. No longer feels like an afterthought; all the existing control hooks
are preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The new per-job workload (3 detectors + several SigLIP embeds) is far more
GPU-bound than the old I/O-bound CCIP pass, so the right worker count shifted and
is hard to guess. Add an Auto mode (default ON) that finds it:
- _control_loop samples jobs/sec + GPU util/VRAM every ~6s and hill-climbs the
target: grow while throughput keeps improving and VRAM stays under budget,
revert a step that doesn't help, back off under memory pressure (VRAM >= 90%),
then settle and periodically re-probe (the GPU/IO balance shifts over a run).
- A manual concurrency set is an override → leaves Auto; an "Auto" toggle in the
control UI re-enables it. status() reports `auto`; the dial reflects the
auto-chosen count (read-only) while Auto is on.
- AUTO_SCALE env (default on) + compose doc. Agent py-compiled (outside CI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
- Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in)
— UPDATE applies ONLY where no image is embedded yet (fresh install), so an
existing library is NOT silently invalidated; it switches deliberately via the
dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2.
- GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512
recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types.
- GpuAgentCard: the two name/version text fields → a single model dropdown;
Save sets name+version from the picked option (the current model is always
selectable even if off-list).
- embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1)
to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the
ml-worker's persistent HF_HOME.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The siglip/ccip backfills skip images that already have current-version regions,
so adding crop detectors only affected NEW images — the back-catalogue would
never be re-cropped. Add a reprocess trigger that resets every done/error job of
a task back to pending, so the agent re-runs the FULL pipeline (figure detection
+ CCIP + concept/panel crops) over the whole library under the current detectors.
- reprocess_gpu_jobs(task='ccip') task + POST /api/gpu/reprocess.
- gpu store reprocess() + GpuAgentCard "Re-process library (re-detect + re-crop)"
button with a confirm (it's heavy).
- Test: a done job resets to pending (attempts cleared).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Better region PROPOSERS feeding the existing crop→SigLIP→max-over-bag heads (no
change to the learned-tagging approach; no per-tag cost — propose once, embed
each region, all heads in one matmul).
- detectors.py: lazy ultralytics YOLO wrapper, each proposer independently
optional + guarded (a bad weight spec / inference error self-disables that one,
logged, never breaks the worker). Weights resolve from an ultralytics name |
http(s) URL | "hf_repo::file", cached under HF_HOME. NMS merge so a figure two
detectors both find collapses to one crop.
- worker: figure boxes = imgutils detect_person ∪ general COCO person (merged)
→ CCIP + concept (anime + Western/realistic coverage); booru_yolo anatomy
components (head/cat-head/anatomy/…) → concept crops; comic panels → kind=
'panel' concept crops. Capped per frame (MAX_COMPONENTS/MAX_PANELS).
- config + compose: PERSON_WEIGHTS (default yolo11n.pt, works OOB),
ANATOMY_WEIGHTS + PANEL_WEIGHTS (operator sets booru_yolo URL + mosesb panel
hf::file; empty = off). ultralytics added to requirements.
- backend: image_region 'kind' doc notes 'panel'; no migration (free String,
and the bag scorer keys on a non-null siglip_embedding, not the kind, so any
SigLIP region joins the bag automatically).
Agent is outside CI — py-compiled here; operator tests on the GPU and checks
Western-vs-anime crop quality via /api/ccip observability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Fast, read-only, indexed aggregates shaped for ANALYSIS (not the paged UI
directory, which is alphabetical + builds previews and timed out at 10 min on a
full count sweep).
- GET /api/tags/top — top tags by image count, desc. ?kind, ?limit (cap 500),
?min_count, ?source=all|human|manual|accepted|auto (human=manual+ml_accepted,
auto=head_auto+ccip_auto+ml_auto). One GROUP BY over image_tag (indexed on
tag_id).
- GET /api/tags/<id>/stats — per-tag dataset health: total + per-source counts
(manual/accepted/head_auto/ccip_auto), human vs auto rollups, rejection count,
and whether a trained head exists. Backs concept-readiness + source-split
analysis.
Plain-HTTP homelab posture, no auth change. Tests cover ranking, source filter,
min_count, the source breakdown, and 404.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Hygiene follow-up to the Camie retirement (#1189) — these were left inert to
bound that change; nothing reads them now. Migration 0068 drops:
- ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_
character/general (already dead pre-retirement — scoring uses per-head
thresholds), video_min_tag_frames (only the deleted video-prediction
aggregator used it).
- image_record: tagger_model_version (no writer), centroid_scores (dead JSON
cache, no reader).
Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant
+ video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding
card (interval + max frames only); importer no longer resets the dropped cols;
download_models drops the Camie fetch; stale CASCADE comments in cleanup_service
no longer name the removed tables. Tests updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP.
Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING
read them — suggestions come from heads+CCIP and apply_allowlist_tags applies
via Camie predictions, not centroids. Pure dead wiring; remove it.
Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the
daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger,
the tag_reference_embedding table + model, the centroid_similarity_threshold +
min_reference_images settings (migration 0066), the CentroidRecomputeCard +
its store action + MaintenancePanel tile, and the centroid slider in
MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the
allowlist branch already covers "could re-emit"); tag merge no longer clears a
table that no longer exists.
NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction,
and the allowlist bulk-apply — accepting a suggestion still allowlists + applies
it across the library. The tag-eval "centroid" baseline metric is unrelated
(in-memory) and stays. (image_record.centroid_scores JSON column also remains —
separate legacy field, its own micro-cleanup.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Pure nearest-cosine piled near-identical images into the neighbour grid — a
reposted banner filled all 24 slots, and once you wandered into a B&W /
comic-panel cluster every neighbour was more of the same with no way back to
colour without the Random button (operator-reported, with screenshot).
similar() now over-fetches a wide candidate pool (5x the requested limit, cap
200), then diversifies down to `limit`:
- pHash near-duplicate collapse: drop candidates within 6 Hamming bits of the
anchor or an already-kept candidate, so a repost (and the anchor's own clones)
appears at most once.
- MMR re-rank: greedily pick for closeness-to-anchor minus similarity-to-already
-picked (lambda 0.55), so the result SPANS clusters instead of returning 40
variations of one image. Falls back to nearest-order on any failure / small
pool, so existing nearest-first behaviour is unchanged when there's nothing to
diversify.
Frontend forwardTarget drops the now-redundant skip-nearest-third hack (the list
is already diversified server-side) — plain random-over-unvisited gives the
variance now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
For redeploying curator while away with nobody to restart the agent:
- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
fault. On a transport error it hands the job back (best effort) and signals
the loop to back off — instead of calling fail(), which would burn the job's
server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
resets on the first successful lease, so a long outage is gentle and recovery
is automatic within ≤60s of curator returning. Sleeps are interruptible so
Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
so a host reboot / crash-restart (restart: unless-stopped) self-heals with
nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
work" banner so the recovering state reads as recovery, not failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray,
lactation) that the whole-image SigLIP vector washes out: the GPU agent now
embeds figure crops with SigLIP too, stored as kind='concept' regions, and the
suggestion rail scores each image as a BAG (whole-image + every concept crop),
taking each head's MAX over the bag. The whole-image vector is always in the
bag, so this can never score lower than before.
Model-agnostic by construction: the server ANNOUNCES the embedding model
(HF name + version) in the lease, so the agent loads whatever the heads were
trained in and stays in lock-step — a model swap is a server setting + a
re-embed migration, never an agent change.
- agent: model-agnostic CropEmbedder (torch/transformers get_image_features,
fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits
figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the
back-catalogue backfill never churns figure/CCIP regions; torch cu124 +
transformers in the image.
- server: lease announces embed_model_name/embed_version; score_image is
max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill
'siglip' gates on a missing concept region (drains the back-catalogue,
retries failures, no double-enqueue); daily siglip-backfill beat; UI button;
/api/ccip/overview reports images_with_concept_siglip.
- v1 scope: suggestion rail only — auto-apply stays whole-image (conservative;
heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Works through the optional CCIP ideas + the "keep moving even if I forget" ask:
AUTOMATION (no button needed):
- Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored
ones retried) on their own; the queue never goes idle waiting for a click.
- CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so
identity tags keep flowing. ON by default (opt-out, like head auto-apply);
ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut),
migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips
already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH
/api/ml/settings; auto_applied count in /api/ccip/overview.
REFERENCE QUALITY (the over-fire root cause):
- character_references now draws ONLY from single-character images — on a
multi-character image the tag is image-level, so every figure would otherwise
pollute each character's prototypes (a 2-char image tagged 'Velma' made
Daphne's figure a Velma reference). This is the contamination behind residual
over-firing.
- Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the
reference load isn't redone on every modal open.
Tests: multi-character image not used as a reference; auto-apply tags a confident
match as ccip_auto.
NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot
interesting content").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Accept already re-focused the tag input (so you keep typing without re-clicking);
reject (✗) and un-reject (↶) went straight to the store and skipped it. Route
them through onDismiss/onUndismiss which emit 'dismissed', and wire that to
focusTagInput in TagPanel — same return-to-input behaviour as accept. TagPanel is
shared, so this covers both the image modal and the Explore workspace. The
field's mobile-focus guard is preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two reports: → sometimes "loops back", and the walk gets stuck on near-identical
images. Cause: forwardTarget picked a uniformly-random neighbour from the 24
NEAREST, so it (a) often landed on an image already in the trail — which snaps
the cursor back into history and makes → bounce between visited nodes — and (b)
only ever offered near-duplicates.
forwardTarget now: excludes already-visited neighbours (→ opens something new,
no snap-back), and skips the closest third of the (similarity-sorted) pool so the
jump favours the more-varied remainder instead of lookalikes. Neighbour pool
widened 24→40 for more variety to browse + jump into. The post-← browser-forward
walk through visited crumbs is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got
3-10 character guesses dominated by the most-referenced characters (a 27-ref
character clears a low bar on many images). A sweep showed 0.85 collapses the
noise (noisy multi-matches 47→3) while keeping the confident single-character
matches.
- ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image
reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85.
- Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999).
- Slider in the GPU agent card ("Character-match strictness") — tune live, no
redeploy, same observe-and-tune loop as auto-apply.
Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
At 8 workers the GPU sat at ~5% util / <5GB VRAM — the pipeline is I/O-bound
(downloading + decoding images over HTTP), so the GPU starves until many workers
overlap that I/O. Raise MAX_CONCURRENCY 8→32 and make the UI worker control a
number input (reaching 32 by ±1 was tedious); the cap is reported via /status so
the UI clamps to it. Also size the shared requests pool (pool_maxsize=64) — the
default 10 would have throttled 32 workers + spammed "connection pool is full".
Verified by running; watch GPU util/VRAM climb as you dial up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
A naive convert('RGB') on a palette-with-transparency image (common: character
PNGs on a clear background) lets PIL guess the transparent pixels — black-ish
artifacts that bleed into the crop + the CCIP embedding (and the "should be
converted to RGBA" warning). to_rgb() composites over white first for a clean,
consistent background; used by both stills and video frames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Control UI gains what the operator asked for:
- GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how
hard the card is working while you're at the desktop.
- Worker count is now a live − / + control (POST /concurrency), not just an env:
the worker is a pool of independent slots (shared model, so slots add concurrent
inference, not N× VRAM). Dial up for speed, down to free the card. Replaces
pause/resume with Start/Stop + the worker dial.
- Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back
via client.release() so they're re-picked immediately (pairs with the server
recovery sweep).
Not CI-tested (agent/ outside CI) — verified by running.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
So work an agent orphaned gets picked back up quickly, three layers:
- GpuJobService.release(): a graceful agent stop hands its still-leased jobs back
to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease.
- GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a
60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the
queue counts honest even when nothing is leasing.
- Lease TTL 300→180s: still well above any single job (a capped-frame video embed
is tens of seconds, and a live worker heartbeats), but a hard crash recovers
faster once the sweep fires.
Tests: release returns-to-pending (token-scoped), recover_orphaned resets only
expired leases, release API round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
onnxruntime-gpu needs cuDNN 9; the plain cuda:12.4.1-runtime image lacks it
(libcudnn.so.9 missing → CUDAExecutionProvider falls back to CPU). Switch to
the -cudnn-runtime variant which bundles cuDNN 9.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
compose file (pull the published image, GPU reservation, model-cache volume,
.env for the token) so the agent runs with `docker compose up -d` instead of a
long docker run. A copy + .env template also placed in ~/Documents/fc-gpu-agent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Build + push fabledcurator-agent alongside web/ml (own CUDA + onnxruntime-gpu
image, context=agent/, same tag cadence: main → :main/:latest/:c-<sha>, tag →
:<version>). So the operator PULLS + runs it on the GPU machine instead of
building locally. README switched to docker pull.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The last piece: a Dockerised desktop-GPU worker that talks to FC ONLY over HTTP
(lease → fetch pixels → detect figures + CCIP-embed → submit), so Redis/Postgres
stay private. New top-level agent/ (outside CI scope — verified by running it):
- fc_agent/worker.py: the lease/compute/submit loop, concurrency 1, start/pause/
stop (stop frees the card; unprocessed leases expire + re-queue).
- fc_agent/models.py: imgutils wrappers — detect_person (figures) + CCIP embed.
The two API seams to verify against the installed dghs-imgutils (flagged).
- fc_agent/media.py: stills + video frame sampling (ffmpeg) at FC's cadence →
per-frame instances (the bag).
- fc_agent/crops.py: vendored crop primitive. client.py: the FC HTTP client.
- fc_agent/app.py: FastAPI localhost control UI (start/pause/stop + progress +
queue depth). Dockerfile (CUDA + onnxruntime-gpu + ffmpeg) + requirements +
README (token → build → run --gpus all → Start; CPU-fallback path).
This completes the CCIP pipeline end to end: agent produces region CCIP vectors →
RegionService stores → matcher suggests characters → rail. Verified by running on
the desktop (not CI). README calls out the imgutils API + model-string checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
So the work can be checked through an API as the agent fills in vectors (same
pattern as /api/heads/metrics):
- GET /api/ccip/overview: regions by kind, images with figure CCIP vectors, the
per-character reference counts (which characters have enough examples to match
on), and the embedding versions present.
- GET /api/ccip/images/<id>: that image's stored regions (bbox, frame_time,
has_ccip/has_siglip, versions) + the CCIP character matches it would get — for
spot-checking detector + matcher output.
Read-only, no GPU. (Queue depth is already at /api/gpu/status.)
Tests: overview coverage counts + per-character refs; per-image regions + matches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
SuggestionService.for_image now merges CCIP character matches with the SigLIP
head suggestions — they're complementary, not exclusive: CCIP is the identity-
specialized signal but needs a detected figure; the heads work whole-image but
conflate identity with style. Merged by tag: 'both' when they corroborate
(higher score wins), 'ccip' / 'head' otherwise. Cheap when no CCIP vectors exist
yet (match_image returns early without a figure vector), so it's a no-op until
the agent runs. Suggestion.source is now 'head' | 'ccip' | 'both'.
Test: a character with a CCIP reference figure surfaces (source='ccip') on a new
image whose figure matches.
NEXT: the agent container (real CCIP/detector models, hands-on) that produces the
vectors this consumes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The server-side brain that turns stored CCIP vectors into character suggestions
— no GPU. character_references() gathers each character tag's prototype vectors
(figure/face-region CCIP embeddings on images carrying that tag); match_image()
cosine-matches an image's figure vectors against every character (multi-
prototype: best over a character's examples), surfacing those above a tunable
threshold as {tag_id, name, category:'character', score, source:'ccip'},
excluding already-applied characters. v1 = cosine on raw CCIP vectors; the exact
CCIP metric/threshold gets validated against the model in the hands-on eval.
Tests (synthetic vectors): same-character match across images, no-match for an
orthogonal figure, already-applied exclusion, no-figure-vectors empty.
NEXT: merge CCIP character suggestions into the rail; the agent container that
actually produces the vectors (hands-on, GPU — not CI-verifiable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The FC-side control surface the operator asked for: Settings → Tagging → "GPU
agent". Generate/reveal/copy/rotate the agent bearer token (with the FC URL to
point the agent at), see the live job-queue depth (pending/in-flight/done/
errored, polled), and a "Queue character embedding (CCIP)" button that triggers
the library backfill. Plain-HTTP-safe copy (copyText resolves on success,
throws on fail). Closes the "how do I get the token in the UI" gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The thin HTTP surface over the queue so the desktop agent stays HTTP-only:
- Agent endpoints (Authorization: Bearer <token>): POST /api/gpu/jobs/lease
(returns jobs + image_url + mime + video frame cadence), /submit (stores
regions via RegionService + closes the job; 409 on a stale lease), /heartbeat,
/fail. Token validated against AppSetting (mirrors the extension-key pattern,
constant-time compare).
- Admin (browser): GET/POST /api/gpu/token[/rotate] (generate + show the agent
token), GET /api/gpu/status (queue counts), POST /api/gpu/backfill → dispatches
enqueue_gpu_backfill.
- enqueue_gpu_backfill(task): one INSERT…SELECT enqueues a job per image lacking
one for the task (scales to the full library; idempotent).
Agent flow: lease over HTTP → fetch pixels via the normal FC image URL → compute
on the GPU → submit. Redis/Postgres never exposed.
Tests: bearer required (+ wrong-token 401), lease→submit round-trip (region+CCIP
vector stored, job done via /status), stale-lease 409, backfill enqueue +
idempotency.
NEXT: the agent container + control UI, then the CCIP detector/embedder + matcher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Answers "how are videos/all media handled by the GPU worker": a job is per ITEM,
but the agent fans a VIDEO into per-frame instances (ffmpeg in the agent, the
existing cadence), each stored with a timestamp — so a video becomes a BAG of
frame embeddings (fixes the mean-embedding muddle) instead of one washed-out
vector. Stills → frame_time NULL; animated GIF/WebP treated like short video.
- image_region.frame_time (migration 0061, not yet deployed so folded in): the
source frame's seconds for video/animated media; NULL for stills. RegionService
passes it through. A whole frame is just kind='frame'.
- gpu_job + GpuJobService (migration 0062): the durable work list that keeps the
desktop agent HTTP-only — enqueue (dedupes (image,task)) / lease (FOR UPDATE
SKIP LOCKED, re-claims expired leases so the queue self-heals) / heartbeat /
complete / fail (re-queues until MAX_ATTEMPTS then 'error'). The server enqueues;
the agent leases+submits over the web API; Redis/Postgres stay private.
Tests: enqueue dedupe, lease-then-skip-when-held, expired-lease reclaim, scoped
heartbeat, complete, fail-requeue-then-error. region test now covers frame_time.
NEXT: the thin HTTP API (lease/submit/heartbeat) + bearer-token auth, then the
agent container + control UI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The storage backbone both crop jobs write to and read from. image_region =
normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id;
'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP /
Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The
bbox doubles as grounded-tag provenance. Migration 0061.
RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines
don't clobber each other) + get_regions — the GPU agent's results endpoint will
call the writer; the character matcher + bag scorer read. Server-side, no GPU.
Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The trunk of both crop jobs — CCIP figure-crops and SigLIP concept-crops call
the SAME crop_region(): normalized-bbox crop with optional context padding,
edge-clamping, and the lower-bound size floor (max of a fraction-of-short-side
and an absolute pixel floor) below which a region is too small to embed and
returns None. Only the proposer (where) and embedder (what) differ; the crop is
shared. Pure Pillow — importable + testable anywhere (the GPU agent imports it
for the crop step). Unit-lane tests (no DB): region pixels, floor rejection,
edge clamp, pad expansion, out-size resize.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The Explore center pane hardcoded ImageCanvas, so a video anchor (e.g. a 169 MB
MP4) tried to load the MP4 into an <img> and showed only the alt text — the
thumbnail worked but the "main image" never rendered. Branch on
mime.startsWith('video/') to VideoCanvas (with mime), exactly like the image
modal. The anchor payload already carries mime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The autocomplete suppressed the Create row whenever any existing tag matched
name+kind — but characters are unique by (name, kind, fandom), so a same-named
character in a different fandom (e.g. another "Raven") is a valid distinct tag.
allowCreate now always offers Create for the character kind; the fandom picker
disambiguates and find_or_create is idempotent if the same fandom is re-picked.
The Create row reads "Create another \"Raven\" character (different fandom)" when
a same-name character already exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Surfaces earned auto-apply + its observability in Settings → Tagging → Concept
heads:
- Auto-apply section: an on/off switch (writes head_auto_apply_enabled), the
precision-target + min-examples-to-fire tuning inputs, a Preview (dry-run →
"would apply N", per-concept chips) and Apply-now button, with live run state.
- "How auto-apply is landing": per-concept table from /api/heads/metrics —
applied volume, misfires, realized misfire rate (green/amber/red), and missed
(under-fires) — the signal to tune the precision target from.
store: autoApply(dryRun) / autoApplyStatus() / metrics(). Card polls the sweep
to completion, then refreshes counts + metrics. Completes the auto-apply task.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
test_auto_apply_disabled_blocks_real_run assumed head_auto_apply_enabled
defaulted False; it now defaults True (opt-out), so a real sweep is accepted
(202). Set the switch off in the test to exercise the disabled→400 path.
(run 1629)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
dict(session.execute(...)) on a bare Result invokes the mapping protocol (a
Result has .keys() = column names) and subscripts it → "CursorResult is not
subscriptable". Materialize with .all() so dict() consumes rows as key-value
pairs. The API path already did this; the snapshot task missed it. Caught by
test_snapshot_records_timeseries_point (run 1628).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration
0059 + model default flipped. The support (>=30) + measured-precision gates keep
it safe and every auto-tag is reversible.
Observability so the operator can tune from real data:
- MISFIRE = an auto-applied (source='head_auto') tag the operator later removes.
UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it).
Both captured at correction time in TagService.add_to_image/remove_from_image
(source is lost on delete) into durable per-tag counters (head_metric), keyed
by tag so they survive head retrain/prune.
- Daily snapshot_head_metrics writes a per-concept time-series point
(head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires
+ head quality; 180-day retention; daily beat.
- GET /api/heads/metrics: per-concept current counts + realized misfire rate +
head quality, plus the snapshot time-series — the report to tune the precision
target + support floor.
Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual
removal isn't a misfire, headless manual add isn't an under-fire), snapshot
time-series, metrics API.
What's the autofire threshold? There's no single number — each graduated head
derives its OWN probability cutoff from its PR curve: the operating point that
holds precision >= head_auto_apply_precision (0.97) at max recall. The global
knobs are that target + the >=30 support floor.
NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Graduated heads can now apply their tag without a human — gated so it's safe:
- FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled,
default OFF) is on AND it has >= head_auto_apply_min_positives (default 30)
clean labels. A precise-looking but under-supported low-N head can't spray tags.
- auto_apply_sweep (heads.py): streams every embedded image in chunks, scores
against the eligible heads (numpy, no sklearn), applies each head's tag where
score >= its auto_apply_threshold and the tag isn't already applied/rejected,
with source='head_auto' (distinguishable + reversible). dry_run counts only.
- HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags
task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled)
+ recovery sweep + retention(20).
- API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled),
GET /api/heads/auto-apply (recent runs + per-concept report). Settings
head_auto_apply_enabled + min_positives via /api/ml/settings.
Tests: sweep applies above threshold, dry-run writes nothing, skips under-
supported + ungraduated heads; API disabled/dry-run/conflict guards.
NEXT (slice 2): the observability the operator asked for — per-concept misfire
(auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a
reporting API to tune. Slice 3: the UI (enable, preview, trends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Two cadences for keeping heads in sync with your tagging:
- PASSIVE: a nightly `scheduled_train_heads` beat (skips if a run is already
in flight; creates+commits the run row before dispatching train_heads so the
ml worker always finds it). Folds the day's accepts/rejects + newly-eligible
concepts into the heads without anyone clicking.
- ACTIVE: a "Retrain heads" button in the Explore trail bar — bank the +/-
feedback you just gave while walking content, without a trip to Settings.
Shared logic in a new useHeadTraining composable (trigger + poll + start/finish
toasts), used by the Explore button; reflects an already-running run (incl. the
nightly one) on mount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Arrow keys walk the Explore breadcrumb trail: ← steps back, → goes forward to
an already-visited item or — with no forward history — jumps to a RANDOM
neighbour to keep the rabbit-hole going (operator-asked).
The trail gains a cursor (browser back/forward semantics): stepping back no
longer trims the forward branch, so → can return to it; a genuinely new walk
off a back-step truncates the stale branch then appends. The crumb-bar "current"
highlight follows the cursor, not the tip.
Arrows are ignored while typing a tag, but still navigate when the tag input is
focused-but-empty (it auto-focuses after every walk, so otherwise arrow-nav
would dead-end after one step). Modifier-key combos pass through untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The rail's Suggestions now come from the trained per-concept heads. SuggestionService.for_image scores the image's frozen SigLIP embedding against
every head (heads.score_image) and surfaces concepts above each head's own
suggest threshold; the typed-dropdown's min=0 "show everything" mode maps to a
flat floor so any head-scored concept can still be picked. Already-applied tags
drop; rejected tags stay flagged + reversible (unchanged).
REMOVED from the suggestion path (rule 22, no fallback): the Camie
ImagePrediction candidate/alias/merge pipeline and the per-tag centroid
augmentation, plus the now-dead SuggestionService internals (_load_predictions,
_threshold_for, _settings, self.aliases, self.centroids). Head suggestions are
always canonical tags, so raw_name/via_alias are null/false and the rail's
alias kebab is inert by data (its removal + the Camie ingest-tagger rip are the
flagged follow-up). for_selection (bulk consensus) now aggregates head
suggestions unchanged.
Tests rewritten to the head path: test_ml_suggestions (surfaces/applied/
rejected-reversible/override/no-embedding/no-heads), test_suggestions_bulk
(consensus), test_api_suggestions (get + dropped the Camie-alias roundtrip),
and test_ml_artist_retired (artist not head-eligible via _HEAD_KINDS).
DEPLOY NOTE: after this lands, the rail is empty until you run Train heads
(Settings → Tagging → Concept heads) — deploy, train, then the rail populates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The UI for the heads subsystem: Settings → Tagging → "Concept heads". Shows
head count, auto-apply-ready count, and last-trained; a Train/Retrain button
(one run at a time, polls while running, surfaces a failed run's error); an
empty state guiding the operator to tag first; and a per-concept table (name,
category, +tags, AP, P, R, auto-apply ⚡) sorted strongest-first so weak/under-
tagged concepts are obvious. Rehydrates status from GET /api/heads on mount so
it survives navigation. Pulls head_min_positives from ML settings for copy.
Slice C (swap the rail's suggestions to heads, remove Camie + centroid) is next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before
I), and drop the inline comment that split heads.py's import block. Pure import
ordering — no behavior change. (run 1601 lint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
test_rejected_tag_surfaced_flagged_then_reversible asserted "Rejectme" but an
existing tag keeps its stored name ("rejectme"), so the suggestion's
display_name is lowercase. Match by canonical_tag_id instead (casing-robust).
The feature was correct — only the assertion was wrong (run 1595 integration).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands
its production form (the first of three slices that make heads the suggestion
source, replacing Camie + centroid).
- tag_head: one logistic-regression head per general/character concept with
enough labelled positives. Weights (pgvector), honest CV-derived suggest
threshold + earned-auto-apply point, and per-concept quality metrics.
- head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the
admin card shows live + historical status across navigation.
- services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data
loaders + metric math so production heads match measured eval numbers) and
SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one
image's embedding against all heads → the rail's suggestions, cached on
(count, max trained_at) so a retrain invalidates without per-request loads.
- tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress)
+ recover_stalled_head_training_runs sweep + retention(20) + 5-min beat
(rule 89).
- api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET
/api/heads (count, graduated, last-trained, running, per-concept table,
recent runs).
- ml_settings: head_min_positives + head_auto_apply_precision, tunable via
/api/ml/settings.
Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B —
this slice makes training + scoring exist and CI-verifiable. 'precision' column
stored as precision_cv (SQL reserved word). Migration 0058.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
A red-✗ dismissal no longer makes the suggestion vanish. The rejected tag
stays in the rail — dimmed, struck-through, with a "rejected" pill and a
one-click undo (↶) in place of the ✗ — so a misclick is recoverable and the
operator can see what they've said no to (operator-asked 2026-06-27).
Backend: SuggestionService.for_image now KEEPS rejected tags, flagged
rejected=True, sorted to the bottom of their category, instead of dropping
them. New AllowlistService.undismiss + POST /suggestions/undismiss clears the
TagSuggestionRejection. Rejected items are still excluded from bulk consensus
(for_selection) and the type-to-add dropdown, whose jobs are unchanged.
Frontend: store.dismiss flags in place (canonical tags) rather than dropping;
new store.undismiss reverts. SuggestionItem renders the rejected state and
swaps ✗→↶; ✓ still accepts (which clears the rejection server-side).
Tests: rejected-surfaced-flagged-then-reversible (service) + undismiss
endpoint idempotency (API).
Completes #1134's reversible-rejection half. Heads-as-suggestion-source is
the remaining piece.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Replace the single "Accept" pill in the modal Suggestions rail with the
eval card's green ✓ / red ✗ language: ✓ accepts the tag (positive), ✗
dismisses it for this image — which already persists a TagSuggestionRejection
(hard negative the heads train on). The pair occupies ~the footprint of the
old pill, so per-image rejection becomes a one-click peer of accepting
instead of being buried in the kebab.
Dismiss moves off the 3-dot menu, so the kebab now only carries alias
actions and is hidden when none apply (centroid hits with no alias option).
Toward #1134 (native per-image negatives in the rail). The bigger piece —
heads as a suggestion source feeding this panel — is still ahead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
"Keep" on a doubted positive was a no-op, so the same confirmed-correct images
came back in "head doubts" every run (operator-flagged: reinforcement keeps
surfacing the same images). Add tag_positive_confirmation (mirror of
tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and
the eval excludes confirmed positives from the doubts list — exactly as rejected
items already drop out of the suggest list. The tag stays a positive either way
(confirmation is a "reviewed" marker, not a training change).
- model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent).
- tag_eval: _confirmed_ids + exclude from head_doubts_positive examples.
- store.confirmTag + card "keep" calls it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"head would suggest" drew from the whole negative pool, which INCLUDES the
images the operator rejected. A rejected near-miss (e.g. an orc under "goblin")
is a hard negative that still scores high, so it kept resurfacing as a fresh
suggestion every run (operator-flagged: "same items keep appearing"). Exclude
already-rejected ids from the suggest list — once you've said no, it's gone.
(head doubts = lowest-scoring positives is unchanged; genuinely-hard true
positives legitimately recur there.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two additions driven by "what's the commit threshold?" + "find more tags":
1. High-precision operating point (Bar 4). Per concept, report the threshold that
maximizes recall while holding precision >= a target (default 0.97, configurable
via `precision_target`) — i.e. "could this fire without a human, and how much
would it catch?" `head.auto_apply` = {target, threshold, precision, recall} or
null if the target is unreachable. Surfaced on the card.
2. Server-side concept auto-discovery. `auto_top_n` param unions the explicit
concept list with the N most-tagged general tags (one fast DB query) so the
eval can broaden itself without hand-listing — replaces the slow HTTP directory
paging. Card gains "+ auto-add top-N" and precision-target inputs.
No migration; numpy/sklearn stay lazy. Existing _normalize_params test still
holds (new keys additive; None still falls back to DEFAULT_CONCEPTS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking an example in the maintenance card navigated to /explore/<id> —
heavier than wanted (operator: just want a bigger look). Open the existing
app-wide ImageViewer modal via modal.open(id) instead: bigger image + tags
in place, no navigation away from Settings. The ✓/✗ actions are unaffected
(separate overlay buttons).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the learn-from-tags loop directly on the eval lists (operator-flagged:
no surface to confirm/refine the head's suggestions). Each thumbnail gets a
green ✓ / red ✗ that writes the SAME tables the head trains on:
- suggest + ✓ → apply tag (new positive, POST /images/<id>/tags)
- suggest + ✗ → record rejection (hard negative, suggestions/dismiss)
- doubt + ✗ → remove tag + record rejection (kill bad positive, add negative)
- doubt + ✓ → keep (stays a positive, no write)
Acted thumbs grey out with a badge; re-run to see the head sharpen. Thumb still
links to /explore/<id>. All endpoints already existed — no backend change.
Inline is the starting point; longer-term the modal Suggestions rail gets the
red "No" (negative) so per-image rejection is native there too (next slice).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 56px example thumbs were too small to judge a label (operator-flagged).
Bump to 120px and wrap each in a link to /explore/<id> (new tab) so the
"head doubts / would suggest" galleries double as a review-and-fix queue —
click a doubted positive, land on it in Explore, correct the tag, re-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Frontend for #1130. A maintenance tile in Settings → Tagging:
- Editable concept list + "Run eval" → POST /api/tag-eval (one running at a time).
- Rehydrates on mount via the persisted run (getRun by latest id) and polls while
running — so the report SURVIVES navigation (operator-flagged); the task runs
backend-side regardless and the card reconnects to its row.
- Renders the saved report: per-concept head-vs-centroid metrics table (AP/F1/
precision/recall) with Δ AP, the learning curve (AP @ N positives), and
thumbnail galleries (head-would-suggest / head-doubts-positive) for eyeballing.
Backend: _examples now stores thumbnail_urls (not just ids) so the report is a
self-contained artifact that renders without per-id lookups on reload.
No new top-level surface — slots into the existing maintenance area.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained-
head spine on the operator's own data, reusing the SigLIP embeddings already
stored on image_record — no re-embedding, no GPU.
Per concept: train a logistic-regression HEAD (positives + negatives = explicit
rejections + sampled unlabeled) vs the old single-CENTROID baseline; report
cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged
positives grow 10→30→100→300), and example image ids (head-would-suggest /
head-doubts-positive) to eyeball.
Persisted so the report SURVIVES navigation (operator-flagged): the run + full
report live in a new tag_eval_run row (mirrors library_audit_run); the admin
card will rehydrate from GET on mount, not transient state.
- models.TagEvalRun + migration 0056; runs on the ml queue (only worker with
numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue.
- services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml
.tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report).
- recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat
(rule 89). scikit-learn added to requirements-ml.
- tests: param normalization + the rehydrate read-path + create/conflict.
Frontend admin card (trigger + render persisted report) follows next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Allowlist / Alias / ImportTask tables scroll their bodies (height=360/480) but
the column headers scrolled away with the rows, so you lost the column labels
(operator-flagged 2026-06-27). Add Vuetify `fixed-header` so the header row
stays pinned while the body scrolls.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panes grid had no explicit row, so the implicit `auto` row sized to its
tallest pane's content. With Provenance + Tags + a long Suggestions list, the
rail outgrew the fixed-height workspace, spilled over and made the WHOLE page
scrollable — showing as a weird empty gap at the top (operator-flagged
2026-06-26). grid-template-rows: minmax(0, 1fr) bounds the row to the container
so each pane's own overflow-y:auto scrolls internally instead. Reset to `none`
in the stacked (<=1100px) layout where the page is meant to scroll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post title/description frequently names the character, so surface it while
tagging in Explore (operator-asked 2026-06-26). ProvenancePanel gains optional
imageId/image props (default = modal store, so the modal is unchanged) since
provenance is its own system loaded by id; ExploreView renders it above TagPanel
in the right rail, hosted on the anchor. Self-collapses when the image has no
provenance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Explore is a rapid walk-and-tag surface, so focus must keep returning to the tag
input with no extra click (operator-asked 2026-06-26). Two gaps closed:
- Navigation hardening: refocus on every focused-image change (neighbour click,
breadcrumb, Random image, seed) now runs nextTick → requestAnimationFrame, so
it lands AFTER the post-navigation re-render/paint instead of being stolen
back by the neighbour-grid re-render.
- All tag actions refocus, in both Explore and the modal: tag add (existing/new)
and remove now hand focus back like accept-suggestion already did; and the
rename + fandom-assignment dialogs refocus on @after-leave (fires after
Vuetify's own focus-return to the activator, so ours wins).
TagAutocomplete's mobile guard is preserved throughout (no soft-keyboard pop on
touch). Modal behaviour gains the same stickier focus — consistent, low-risk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cluster tag-gap feature's only UI (Explore's TagGapPanel) was removed in the
3-pane rework, leaving the backend that fed it with no caller. Surgical removal:
- drop the POST /api/images/cluster/tag-gaps route (cluster_tag_gaps)
- drop BulkTagService.tag_gaps (+ the now-unused `import math`)
- drop the tag_gaps tests (test_bulk_tag_service, test_api_bulk_tags)
BulkTagService's common_tags / bulk_add / bulk_remove stay — they still back the
gallery bulk editor. Pure deletion, no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The workspace is built for rapid walk-and-tag, but the tag field was only
focused once (TagAutocomplete's on-mount autofocus) — walking to a neighbour
left focus behind, so the operator had to click the field each time
(operator-asked 2026-06-26).
TagPanel now exposes focusTagInput; ExploreView watches the focused image id and
re-focuses the field on seed + every walk via nextTick. Reuses the existing
focus path, so TagAutocomplete's mobile guard (no soft-keyboard pop on touch) is
preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-clarified 2026-06-26: the dimensions/size/type + save (floppy) block
should sit DIRECTLY above the Tags section — i.e. just under Provenance — not at
the very top of the rail. Reorder the rail's main scroll area to Provenance →
ImageMetaBar → TagPanel (Related stays pinned at the bottom).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Right-rail layout fixes (operator-flagged 2026-06-26 — the prior change wasn't
the intended improvement):
- Pin the Related strip to the BOTTOM of the rail: the side becomes a flex
column with a scrolling main area (meta + provenance + tags + suggestions)
and a pinned Related footer (capped at 45% of the rail, scrolls past that).
Related now stays reachable no matter how long Tags/Suggestions run, and
self-collapses (no footer space) when there's nothing to show.
- Remove the 320px suggestions scroll cap (3fcc4ae) — it was a workaround "so
Related stays reachable"; pinning Related is the proper fix, so suggestions
flow in the single main scroll instead of a nested scrollbar.
- Shrink the Download button to a floppy-disk save icon (mdi-content-save); the
meta (dimensions/size/type) + save action now sit as a compact top block
(meta left, icon right). Copy link moves into the adjacent kebab menu.
ImageMetaBar is shared with the Explore center pane, so the compact save
control applies there too (parity). Mobile (<=900px) keeps the single body
scroll — no nested scroll, Related flows at the end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks Explore from "anchor + neighbour grid + cluster tag-gap rail" into a
persistent 3-pane workspace that unfolds the image modal so you can tag while
rabbit-holing (operator concept 2026-06-26):
- LEFT neighbour grid (larger thumbs), click = walk; breadcrumb retained.
- CENTER light viewer — reuses ImageCanvas + ImageMetaBar(:image) for the
focused image; "Open full viewer" still launches the overlay modal.
- RIGHT the modal's TagPanel, hosted on the anchor for modal-parity tagging
(chips, autocomplete, suggestions + Accept, fandom-on-chip, T/"/" focus).
Reuse without destabilising the audited modal store: TagPanel and
SuggestionsPanel gain an optional `host` prop (default = modal store, so the
image modal is unchanged); the explore store implements the same small
tag-CRUD surface (current/currentImageId + reloadTags/addExistingTag/
removeTag/createAndAdd) over the anchor. ImageMetaBar gains an optional
`image` prop for the same reason.
Drops the mass/cluster tagger (TagGapPanel deleted; clusterIds/thumbById
removed) — per-image tagging feeds the per-tag reference-embedding centroid
better than bulk ops.
Nav: keep the Explore tab but bare /explore now SEEDS a random image
(GET /api/showcase?limit=1 → /explore/:id) so the tab kick-starts a rabbit
hole; explicit meta.navOrder pins nav order (Explore after Gallery) since
router.getRoutes() doesn't preserve declaration order.
Note: the backend cluster tag-gaps route/service (#94a) is now frontend-orphaned
— left in place; flag for a separate cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The directory card count regressed to a globally-inflated number (~every
card showed the same ~469): the fandom leg used a doubly-nested correlated
subquery — image_tag.tag_id IN (SELECT member.id WHERE member.fandom_id ==
Tag.id) — whose inner predicate did not correlate the outer Tag, so it
matched EVERY character that has any fandom and counted all their images for
every tag. The gallery scope and cleanup count were unaffected (they pass a
literal tag id, a single-level subquery), which is why only the card diverged
from the gallery.
Rewrite the count as a single-level correlated scalar subquery: join `member`
(the tag applied to the image) and match image_tag.tag_id == Tag.id (direct)
OR member.fandom_id == Tag.id (a character of this fandom). Strengthen the
directory test with a second unrelated fandom/character so a non-correlating
fandom leg fails (count would read 4 instead of 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fandom owns characters via Tag.fandom_id, but every image<->tag query
went purely through direct image_tag rows, so a fandom only surfaced
images literally tagged with it — images carrying one of its characters
were invisible to its browse count, previews, and gallery filter.
Derive membership at query time instead of materializing fandom rows
(which would drift on every reassign/merge/remove). Add one shared
predicate in tag_query.py — image_in_tag_scope / image_in_any_tag_scope:
an image belongs to a tag if tagged with it directly OR (when the tag is
a fandom) carrying a character whose fandom_id is that tag. The character
leg is empty for non-fandom tags, so it applies uniformly with no kind
branching. Route all read sites through it:
- gallery _apply_scope: include, OR-groups, and symmetric exclude
- directory image_count: correlated COUNT(DISTINCT) scalar subquery
- directory previews: UNION direct + via-character, then ROW_NUMBER<=3
- cleanup count_tag_associations: Tier-B delete prompt now reports a
fandom's true blast radius (was 0 for fandoms with no direct rows)
find_unused_tags already protected fandoms via used_via_fandom; left as is.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A swarm overlay-network blip after the :latest redeploy left Redis healthy but
transiently unreachable; a worker starting in that window crash-looped on the
initial broker connect (kombu OperationalError) and needed a manual Redis reset
to recover.
Retry the broker forever on startup + at runtime (broker_connection_max_retries
=None), add redis-transport socket options to the broker (short connect timeout,
TCP keepalive, retry_on_timeout, periodic health check), and mirror the same on
the Redis result backend. Now a transient outage self-heals when overlay routing
returns instead of the worker exiting.
Test pins the key resilience settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
The right rail scrolls as a whole and ProvenancePanel already caps its cards +
attachments, but SuggestionsPanel had no cap — a long suggestion set (the
General bucket runs to dozens) stretched the rail and pushed the Related strip
below the fold. Wrap the suggestion groups in a 320px max-height scroll box
(hairline scrollbar matching the provenance regions), so suggestions scroll
internally and Related stays visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster C frontend, milestone #94.
#94b Explore walk: new /explore/:imageId route + ExploreView + explore store.
Anchor (reuse /api/gallery/image), neighbour grid (reuse /api/gallery/similar,
24), click a neighbour to re-anchor; in-memory breadcrumb that trims on
backtrack (route is the source of truth). Empty/loading/error + no-embedding
states.
#94c tag-gap closing: components/explore/TagGapPanel — fetches
/api/images/cluster/tag-gaps for the anchor+neighbours, a consensus-threshold
slider (default 60%), per gap shows present/total + the missing thumbnails +
'Apply to N missing' → /api/tags/images/bulk/tags (source manual) → re-fetch.
#94d entry points: 'Explore' button in the modal RelatedStrip; the TopNav entry
comes free from the route's meta.title.
#4a metadata HUD + #4b split Download: new modal ImageMetaBar (always-on, above
ProvenancePanel) shows dimensions/size/type and a split Download button
(default Download, chevron → Copy link via utils/clipboard — no clipboard-image,
rule 95).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster C, milestone #94. BulkTagService.tag_gaps(image_ids, threshold) finds
tags applied to >= threshold fraction of a visual neighbour set but not all of
it (the '7 of 10 share Miku; these 3 don't' signal). Each gap carries the
laggard image ids minus any TagSuggestionRejection rows, so apply-to-cluster
never re-proposes a tag a neighbour dismissed. 100%-common tags and <2-image
sets are excluded. New POST /api/images/cluster/tag-gaps.
Tests: consensus found / common excluded / missing ids; rejected laggard
excluded from missing; tag dropped when all laggards rejected; <2 images empty;
route shape + bad input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
The new coverage tests' sequential shas (c{i:063d}) share their first 8 chars,
so deriving the image path from sha[:8] collided on uq_image_record_path. Use
the full sha in the path. Same hardening for test_api_suggestions._img.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B frontend, milestone #99.
#7c: AllowlistTable gains Applied + Covers columns and a live 'covers ~N at T'
projection as the operator drags a row's threshold (debounced coverage call,
then commits the threshold). allowlist store gains coverage(tagId, threshold)
and refreshes coverage_count after a save.
#7d: suggestions store surfaces a non-blocking toast when accept/alias newly
allowlists a tag — '<verb>: <tag> — allowlisted, auto-applying to ~N images'
(N is the projection; apply runs async). Falls back to the plain toast when
the tag was already allowlisted.
#8b: TagsView merge picker now previews the merge via usePreviewCommit before
committing — shows images moving / already-on-target / series pages / alias-or-
delete / a thumbnail sample, blocks the Merge button on an incompatible
kind/fandom. adminStore.mergeTags gains a dryRun option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the
same counts the apply produces (rule 93 parity) without mutating: images_moving
(source links the apply UPDATEs), images_already_on_target (links it drops),
source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom
compatible flag (surfaced, not raised, so the UI can warn), and up to 6
thumbnails of the moving images. The admin /tags/<dest>/merge route gains a
dry_run flag returning the preview JSON.
Tests: preview moving-count == apply merged_count (parity), incompatible flagged
without raising, self/missing raise, admin dry_run returns preview + no mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster B, milestone #99. Backend for the allowlist tuning dashboard.
#7a: AllowlistService.coverage(tag_id, threshold) counts distinct images with
a prediction resolving to the tag (raw_name==tag.name OR (raw_name,category) in
the tag's aliases) scoring >= threshold — the gross candidate pool, mirroring
tasks.ml._confidence_for_tag resolution. list_all now carries applied_count
(grouped image_tag count) + coverage_count (at the row's threshold). New
GET /api/tags/<id>/allowlist/coverage?threshold= for the live what-if number.
#7b: /suggestions/accept + /alias return {allowlisted, tag_id, tag_name,
projected_count} (projection at the tag's threshold) instead of 204, so the UI
can show a non-blocking 'auto-applying to ~N images' toast. Apply still runs
async via apply_allowlist_tags — projected_count is an estimate.
Tests: coverage by threshold (direct + alias-with-category), list applied vs
coverage, coverage route (explicit/default/bad threshold), accept/alias payload
(newly-allowlisted vs already-on-list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster A, milestone #97. Completes the frontend of the structured tag filter
(backend landed in 23fab98).
#6b store: gallery.filter gains tag_or (OR-groups) + tag_exclude; one model,
serialized via tag_or (repeated key) + tag_not across activeFilterParam/
filterToQuery/applyFilterFromQuery/cloneFilter/loadSimilar; _resolveLabels
resolves names for every referenced id. useApi now appends array param values
as a repeated key (tag_or=a&tag_or=b).
#6c light editor (GalleryFilterBar): autocomplete pick → include chip; click a
chip body to flip include↔exclude (exclude = red minus); ✕ removes. An
"N OR-groups" chip + an Advanced button open the builder.
#6d advanced editor (TagQueryBuilder.vue + common/TagPicker.vue): AND-of-OR
group builder + NOT list. Unifies includes+OR-groups into one groups view,
splits back to tag_ids/tag_or on Apply so the URL stays compact. Writes the
same model the light chips edit.
Store serialization round-trip tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Cluster A, milestone #97. #5: clicking an image-modal tag chip's body now
closes the modal and opens the gallery filtered for that one tag (fresh
filter); ✕/kebab stay as the explicit remove/rename controls.
#6a (backend of OR/exclude filtering): gallery_service._apply_scope gains a
structured tag model — tag_or_groups (AND-of-OR: one EXISTS(tag_id IN group)
per group) + tag_exclude (NOT EXISTS(tag_id IN exclude)) — layered additively
on the existing tag_ids AND path so cursors/facets/deep-links are untouched.
Threaded through scroll/timeline/jump_cursor/facets/similar + facets common
dict; _require_single_filter rejects post_id combined with OR/exclude. API
parses tag_or (repeatable → one OR-group each) + tag_not (csv exclude).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
Frontend pattern-consistency sweep (note #1026, the last DRY-thread item).
TagMaintenanceCard (4 flows) + PostMaintenanceCard (2 flows) each hand-rolled the
same sync preview→commit state machine: a previewData/previewing/committing
triple + onPreview/onCommit that dry-run-previews, then applies and collapses the
projection (the apply shares the backend predicate, so afterward it's empty).
Extract usePreviewCommit({preview, commit, emptyPreview}) owning that lifecycle.
The 6 flows become declarative: supply the two thunks + the collapse shape. The
normalize flow (commit dispatches a self-resuming background task, not a sync
apply) omits emptyPreview so the projection stays and a truthy result = queued.
Composable returns are aliased to the cards' existing local names, so the
templates only change where they read the apply result (the success badges).
Long-Celery-task cards (GatedPurge/VideoDedup) keep useMaintenanceTask — a
different pattern (navigable-away task lifecycle), deliberately not merged.
Exhaustiveness: no card hand-rolls the refs anymore; the only dryRun:false
callers are these two cards, both via the composable. Added a vitest spec for
the primitive (collapse static + fn, dispatch-variant, re-preview clears result).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass follow-up (note #1026). All 13 admin-store actions repeated the same
lastError-capture/rethrow wrapper; the 6 Tier-A maintenance actions additionally
repeated the dry_run POST shape.
- _guard(fn): one copy of the lastError=null / try / catch(set lastError; rethrow)
wrapper, used by all 13 actions.
- _dryRunPost(url, {dryRun, ...extra}): the dry_run POST shape on top of _guard,
used by the 6 maintenance actions. reconcile maps sourceId -> source_id.
Public exports + every action signature unchanged (object-opts for Tier-A,
positional for cascade/bulk/tag ops), so no card/view changes. Behavior identical.
Added frontend spec (the admin store had none): _dryRunPost endpoint+body+default,
sourceId->source_id mapping, and _guard capturing lastError + clearing on success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass follow-up (note #1026). Five handlers returned the identical
jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result).
Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives,
trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews.
trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it).
Added route-level tests for all five consumers (these trigger endpoints had no
route coverage before): 202 + task_id via _queued, and the dry_run flag threading
through to dedup/purge-gated. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened
process #594). Five Tier-A endpoints repeated the same get_json -> dry_run ->
run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn,
**kwargs); the five route handlers now delegate. reconcile keeps its source_id
validation and passes it through **kwargs.
The cleanup_service predicates were already shared between preview and apply
(find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no
backend-logic change; this is purely the HTTP-handler boilerplate.
Consumers (all routed through the helper, verified no copy left behind):
prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id),
purge_legacy_tags, reset_content_tagging.
Added route-level tests for prune-bare (apply) and reconcile (apply + source_id
passthrough + invalid-source_id 400) — the two helper consumers that previously
had only service-level coverage, so every consumer is exercised at the route.
Findings B (queued-response helper) and C (store dry-run POST helper) identified
but not applied this pass (operator scoped to A). The card preview->commit state
machine is deferred to a frontend pattern-consistency sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in
eff6427; closing it out after today's #87 work, which added a seam it didn't
account for. _repoint_post_links drops a loser post's ImageProvenance row on the
(image, post) uniqueness collision — and that row may now carry from_attachment_id
(which archive the file was extracted from). For the exact gallery-dl->native
case this targets, the keeper is the native stub (no archive) and the loser is the
gallery-dl row that extracted the member, so a blind delete silently lost the
containing-archive linkage. Carry from_attachment_id onto the keeper's surviving
row (when NULL) before dropping the collision.
The rarer PostAttachment-collision case (both dup posts captured the same archive
blob) doesn't arise in the targeted scenario — the archive lives only on the
gallery-dl post, so it re-points straight to the keeper and the FK stays valid.
Test: collision merge preserves the loser's from_attachment_id on the keeper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A filesystem-imported archive with no adjacent sidecar has no Post, so
_post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced
post.id. _stamp_member_archive already no-ops on a None post_id (no post → no
provenance to stamp); pass None instead of crashing. Caught by the existing
test_reimport_archive_is_idempotent (no-sidecar zip).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.
- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
(SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
provenance row for the post (new + superseded + deduped members), resolving
the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
and idempotent, so it doesn't touch the dedup/supersede branches and the
backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
records from_attachment_id, return ONLY that archive; else fall back to the
primary-post scoping from 068def2.
- ProvenancePanel: heading pluralizes ('Attachment' for a single file).
- Backfill: re-running reextract_archive_attachments (ArchiveReextractCard)
routes through _import_archive and stamps existing rows — no new code.
Tests: capture stamps on fresh import, nested-archive attribution, per-post
archive on dedup; for_image filters to the containing archive; reextract
backfill stamps the link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Attachments section aggregated PostAttachment rows across EVERY post an
image was pHash-linked to. When one of those was a 'High Resolution Files'
mega-bundle (dozens of unrelated archives), the list ballooned past the
viewport and overwhelmed the modal's right rail.
- for_image() now scopes attachments to ImageRecord.primary_post_id (the post
the file was actually captured from), falling back to all linked posts only
when primary_post_id is unset (older rows / filesystem imports).
- ProvenancePanel wraps the list in a max-height scroll container with a count
in the heading, mirroring the cards' independent-scroll treatment.
Note: FC stores archives as opaque blobs and never records which archive an
extracted image came from, so attachments can't yet be scoped tighter than the
post. Capturing image->archive containment is tracked as separate work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled
viewport; expanding a tile to read into it is the only reason to scroll.
- Every Maintenance card converted to the collapsible MaintenanceTile (collapsed
by default = icon + short title + one-line blurb). Task cards (ML backfill,
centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance)
sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion
thresholds, allowlist, aliases) grouped in one Tagging section as collapsible
tiles; Backup is its own collapsible tile.
- Three labeled sections mirror the Cleanup tab: Backfills and reprocessing /
Tagging / Storage.
- Center the whole Settings surface: SettingsView is now a centered, width-capped
(1140px) column so the tab strip and every panel sit in a tidy centered measure
(was full-width). CleanupView drops its own left-aligned max-width to fill it.
All card logic unchanged - only the chrome.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long
descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles
(2026-06-18: keep both tabs, group inside, compact tiles in a grid).
New common/MaintenanceTile.vue: a compact expandable tile (icon + short title +
one-line blurb collapsed; click the header to expand the full controls/preview/
result inline; keyboard-accessible button + focus ring; tints the
icon, keeps a running task expanded).
Cleanup tab (this pass) restructured into 3 sections — Import-filter audits
(Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts,
Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy,
Reset content tagging, Standardize casing) — each a responsive grid of tiles.
PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved
VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are
destructive content cleanup). All card logic unchanged — only the chrome.
Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Image posts wrap the post date in an <a> permalink
(<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts
don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY
the unwrapped case, so every image post got a null published_at and sorted to the
top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's
_data_from_post method: text up to the first </, then after the last > — handles
both. Verified against the live raw feed (all 6 dates now parse).
Robust logging (operator request): _parse_posts now logs per-page parse stats
(posts / dated / with-body) and a WARNING canary when posts parse but NONE get a
date or body while the raw markers are present — i.e. our extraction diverged
from the live markup. Makes this failure class diagnosable from the worker log
alone, no authed re-fetch needed.
Test: a permalink-wrapped date parses to ISO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SubscribeStar posts deliver content only through document/audio attachments,
which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them:
- docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the
href URL + doc_preview-title + data-upload-id (kind=attachment).
- audio: scope uploads-audios..post-edit_form, split on audio_preview-data
blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio).
The existing downloader handles them unchanged (plain streaming GET; the file
validator only inspects image/video extensions via is_validatable, so PDFs/zips/
audio pass straight through, no quarantine).
Test covers doc + audio extraction (the cheunart sample has none, so this pins
gallery-dl's documented markup shape).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body
regex either returned empty or over-captured into sibling upload divs and the
'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact
_data_from_post rule — content between the post_content-text wrapper and the
youtube-uploads div (literal markers), then strip the trix editor's
<html><body>…</body></html> document wrapper to its inner. Verified against the
live cheunart sample: clean per-post bodies, empty for genuinely text-less posts.
Also port gallery-dl's _media_from_post preview guard: skip gallery items whose
URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of
the Patreon gated-preview bug (#874); this is why a locked post yields no media.
Tests: body must not bleed into the pagination counter; trix html-document
wrapper stripped; /previews items skipped. Fixture now includes the youtube-
uploads close marker present in real markup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_
subscriber even with valid .adult cookies (confirmed present: _personalization_id
+ _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's
ENTIRE flow — including the base Extractor._init_session I'd not read — the
divergence is the HTTP request profile, not the cookies or parser.
gallery-dl's default (cookies-only) mode sends, on EVERY request including the
first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same-
site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint
is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling
looked unlike a browser → SubscribeStar gated the adult-creator page.
Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept-
Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the
per-request XHR headers (both feed and load-more now use the shared profile).
Test updated to assert the gallery-dl-parity profile instead of the old
navigation-vs-XHR split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native client split the feed on `<div class="post is-shown`, but `is-shown`
is added by SubscribeStar's infinite-scroll JS when a post scrolls into view —
present in a browser-SAVED page (what the Step-0 characterization used) but
ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live
feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError.
Align with gallery-dl's proven `_pagination`: split on the generic
`<div class="post ` (trailing space rules out the hyphenated post-content/
post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating
detection (/verify_subscriber, /age_confirmation_warning => auth, not drift).
Regression tests: raw server markup without is-shown now parses; an age-wall
redirect raises SubscribeStarAuthError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart
still drifts — we're being served a full HTML page that isn't the feed. Add
_describe_page(): the drift error now reports the page <title> + which known
interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha),
so the next run names the actual cause instead of "markup changed". Strong
suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is
UA-locked and our hardcoded UA likely differs from the cookie-capturing browser).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First live run (cheunart) tripped the drift guard: "no posts and no recognizable
feed container". The browser-saved page was normal (6 posts + posts_container-list),
so the parser was fine — our live HTTP fetch got a different response. Cause: the
client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the
initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content-
negotiates an XHR full-page request to a non-HTML body → no container → drift.
Fix: the session now uses browser-like navigation headers (Accept: html, NO
X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on
the "load more" endpoint (which is a genuine XHR). Drift message now reports the
response length + a JSON hint so a recurrence is self-explaining. Regression test
pins the header split (navigation initial GET, XHR load-more).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at
import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester",
_FakeIngester) no longer affected dispatch → the fake's run() never ran →
KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the
dict with a _native_ingester_cls() call-time lookup that reads the module globals,
so monkeypatching the class names works again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.
- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
_NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
/ preview_source / verify_source_credential platform-aware. SubscribeStar's
campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
recover/recapture actions show for subscribestar; download_service's
cursor/mode/post_first + the preview endpoint already key on
uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
verify, gallery_dl_service skip-value, api_sources arm-no-preflight).
post_is_gated stays best-effort (can't cause junk downloads); not gating this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 3 — the observability half. ingest_core accumulated ALL
human-readable progress in log_lines → DownloadResult.stdout, persisted to the
DownloadEvent ONLY at phase 3; the real logger was used almost nowhere. So a
worker SIGKILL/OOM/hard-time-limit mid-walk left NO trace (the "task died,
no trace" mode from the recovery-sweep work).
Route run milestones through the container log too, each carrying source_id (L3
context):
- run START (platform/mode/source/campaign/resume_cursor)
- per-PAGE breadcrumb (posts/downloaded/skipped/errors/quarantined/gated/cursor)
at each page boundary — pages are minutes apart on big backfills, so this shows
how far a since-died walk got
- final SUMMARY (same string as the stdout summary)
- operator STOP
(L2 — quarantines log.warning'd — already landed in commit 1's base
_validate_path; failures log.warning via the base _failure_result; the #862 body
canary already log.error's.) log_lines/stdout content is unchanged (summary just
captured in a var), so existing assertions hold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).
- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
(status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
("Patreon API" / "SubscribeStar markup"), preserving the existing message
(test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
-unused DownloadResult/ErrorType/*Auth/*Drift imports.
Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing
the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was
importing patreon privates — wrong owner). New backend/app/services/
native_ingest_common.py is the neutral home for:
- make_session (was _load_session ×2), retry_after_seconds + 429 constants,
sanitize_segment, basename_from_url, post_dir_name, MediaOutcome /
PostRecordOutcome.
- BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume)
and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it;
each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch
enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the
divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone.
- Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only).
post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO).
Tests repointed to the single source at every consumer (rule 93 / §8b parity):
patreon_client/downloader, subscribestar_native. Exception-trio consolidation +
base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core
ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger:
SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery
bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing
media stops re-burning backfill chunks). Per operator decision, dedicated
per-platform tables (not a generalized shared ledger).
filehash is String(128): a CDN content hash when the URL carries one, else a
synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key.
Registered in models/__init__; migration 0054 creates both tables (down 0053).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_downloaded_archive_gets_provenance_and_tagging's fake_fetch still had the
old `*, timeout` signature; the task now calls fetch_external() without it, so
the stub raised TypeError in the integration lane (run 1191). Switch it to
**kwargs like the other two.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).
Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
(connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
across chunks in _stream_to_file (HTTP has no total-download timeout), and
passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.
Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.
Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.
Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
celery_signals._queue_for is a hand-maintained mirror of task_routes that
stamps TaskRun.queue in the prerun signal. It was missing the
backend.app.tasks.external. prefix, so external fetches recorded
queue='default' even though celery routes external.* → download and runs
them on the download worker. The dashboard's per-queue filters and the
per-queue recovery-sweep threshold therefore missed them — the same
'queue column lies default' gap the 2026-06-02 audit fixed for
backup/admin/library_audit.
Map external.* → download in _queue_for. Composes with the fetch_external_link
task-name sweep override (#883), which wins by precedence regardless of the
recorded queue. Pinned test asserts the mirror agrees with the actual route.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
External file-host fetches run to a 60-min hard limit (time_limit=3600,
per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min
default. recover_stalled_task_runs was phantom-flagging healthy in-flight
fetches as "RecoverySweep: no completion signal received within 5 min"
before the task's own timeout/error handling could surface the real error
(operator-flagged: target 414 swept at 6.6min).
The sweep already has per-queue/per-task overrides for long tasks, but
fetch_external_link was never added and its TaskRun records queue='default'
(no queue override) despite external.* routing to download. Add a task-name
override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it
robust regardless of the recorded queue. No new internal timeout needed —
the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception
already capture the real failure once the sweep stops preempting.
Pinned tests: external-fetch override survives a 10-min row / flags a 70-min
row on queue='default'; invariant guard asserts override >= hard time_limit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_artist_directory_service._seed_image built sha256 from
abs(hash(suffix)) % 10000 — PYTHONHASHSEED-randomized hash() over only 10k
buckets, so two suffixes in one test could birthday-collide and violate
uq_image_record_sha256. Flaky per process seed: passed on dev (run 1179),
failed on main (run 1182) with identical code. Use
hashlib.sha256(suffix).hexdigest() for a stable, collision-free digest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recompute_centroid + recompute_centroids were the only tasks still using
the process-wide singleton extensions.get_session() under asyncio.run().
The async engine's asyncpg pool is bound to the loop it was created on;
each Celery task runs a fresh asyncio.run() loop, so after the first
invocation the cached engine handed loop-A connections to loop B and raised
"Future attached to a different loop" — every recompute after the first in
a worker process failed (~35ms, fails on first DB await).
Convert both to the established per-task async_session_factory() pattern
(NullPool engine created + disposed inside the task's own loop), matching
scan/download/admin tasks. No get_session usages remain in tasks/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Long-running maintenance tasks must survive navigating away or reloading
the page. VideoDedupCard + GatedPurgeCard held the in-flight Celery task id
only in component refs and polled task-result inline, so leaving the page
mid-run lost the id and the result was never shown — even though the task
finished on the worker.
New shared composable useMaintenanceTask: persists {taskId, mode, startedAt}
to localStorage on dispatch, re-attaches on mount, and re-shows the result
when the task finishes (the celery result backend retains the summary well
under result_expires). Stale-guard skips resume past 3h. Both cards refactored
onto it; card-specific computeds + confirm dialog kept.
Also fixed the QueueStatusBar lane: both cards watched queue="maintenance"
but tasks.admin.* routes to maintenance_long, so the bar never reflected
their own task — now queue="maintenance_long".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A one-shot Maintenance action to remove the blurred locked-preview images
the ingester downloaded from tier-gated Patreon posts before #874.
current_user_can_view was never persisted, so the cleanup re-walks each
enabled Patreon source (read-only) to re-derive which posts are gated now
and the blurred filehashes Patreon serves for them, then matches by
CONTENT HASH against stored source_filehash. Because the hash is
content-addressed, a real file downloaded when access existed has a
different hash and can never match — regained-then-lost-access content is
provably spared (operator's hard requirement). NULL source_filehash =>
unverifiable, kept + reported.
On apply: delete matched ImageRecords + files (provenance cascades),
clear seen/dead-letter ledger rows for those hashes so the real media
re-ingests if access returns, and delete gated posts left bare. Shares
one match predicate between preview and apply (rule 93).
- cleanup_service: collect_gated_previews + purge_gated_previews
- tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed)
- api.admin: POST /maintenance/purge-gated-previews
- GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply)
- tests: collect predicate, hash-match delete/spare/unverifiable, ledger
clear, bare-post removal, no-op
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit's Edit orphaned the recapture test's relink/stdout
assertions into the new preview test (F821 res_recap/downloader2/m1) and
the gated-skip test's written_paths check matched 'gated' in the tmp dir
name. Restore the recapture assertions to their function and assert on
the media basename instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Patreon serves only blurred locked-preview thumbnails for posts the
authenticated account can't fully view; the native ingester was
downloading those as real media. current_user_can_view was already in
_FIELDS_POST but never read.
Add PatreonClient.post_is_gated (gate ONLY on explicit
current_user_can_view=False; missing/None → viewable, never over-filter)
and skip gated posts at the top of the ingest_core run() and preview()
loops — no media download AND no post-record stub (operator: 'no stub
for gated content'). Skipped before the post-record block so gated posts
never inflate the #862 body canary; surfaced as 'N gated-skipped' in the
run summary. Same gate in preview() for preview/apply parity (rule 93).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GPU enablement (#872) cancelled — not worth the Pascal-specific build for a
modest CPU→GPU win on an old P4. Remove the dead GPU code (device.py, the CUDA
provider branch in tagger, the .to('cuda') path in embedder) so nothing carries
it forward.
Instead, bound CPU inference threads by default so the ml-worker is a predictable
core consumer on a SHARED node — the intended scaling model is multiple worker
replicas (each --concurrency=1, each its own cgroup limit), not one big
container. ONNX Runtime and torch otherwise size their thread pools to ALL host
cores, so each replica would grab every core and oversubscribe / starve the
co-located DB+web. Cap both to _INTRA_OP_THREADS=4 (matches the prior per-worker
cpus:4 unit): run N replicas where N×4 stays within the cores allotted to ML.
- tagger: ort.SessionOptions().intra_op_num_threads = 4 (CPUExecutionProvider).
- embedder: torch.set_num_threads(4).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 1 of GPU enablement (code only — CPU-safe, CI-green; the CUDA image is a
separate step pending the host driver version).
- New services/ml/device.py: FC_ML_DEVICE (auto|cuda|cpu) intent + VRAM knobs
(FC_ML_ONNX_GPU_MEM_GB, FC_ML_TORCH_MEM_FRACTION). Per-worker-host bootstrap →
env, not a DB setting (the GPU host runs CUDA, others CPU).
- tagger: use CUDAExecutionProvider (with gpu_mem_limit) when requested AND the
provider is actually present (onnxruntime-gpu), else CPUExecutionProvider. Logs
the active providers.
- embedder: move model + inputs to cuda when requested AND torch.cuda is
available; cap torch's VRAM share; .detach().cpu() before numpy. fp32 kept so
GPU embeddings stay in the same space as existing CPU ones.
Both AND the env intent with the framework's real availability, so on CPU
(CI / CPU onnxruntime / no GPU) they fall back cleanly — behavior unchanged.
The 8GB P4 is shared by both frameworks, hence the conservative default caps.
Tests: device env parsing. (tagger/embedder GPU paths are operator-verified on
the GPU host — models aren't in CI.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag
firing on one frame survived at peak confidence, and a fixed count under-samples
long multi-scene videos so real scene-local tags looked like noise.
Redesign (operator-steered):
- Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds`
(default 4) across the 5–95% window — so a tag's frame-presence reflects real
screen time independent of video length. Capped at `video_max_frames` (default
64): a long video stretches the spacing instead of exploding into hundreds of
inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg
timeout also cut 60s→30s).
- Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in
>= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on
screen — duration-independent noise rejection), with confidence = MEAN over the
frames it appears in (not max). Clamps the threshold to the sample count so a
1–2-frame short video still tags.
- All three knobs are DB-backed ml_settings (migration 0053), patchable via
/api/ml/settings + sliders in the ML settings card — replaces the
VIDEO_ML_FRAMES env var (product-not-project).
Tests: aggregation drops one-frame noise + means corroborated tags + clamps on
short videos; settings round-trip + min>max validation. Replaced the
_maxpool_predictions unit test.
NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs
CPU-only — is GPU enablement, tracked separately in #872.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2 of #871: clean up the duplicate videos already in the library (the #859
"same video from multiple sources" clutter). Import-time dedup (Phase 1) only
prevents NEW dups; this is the operator-triggered cleanup of existing ones.
cleanup_service.dedup_videos(dry_run):
- backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the
existing library participates; idempotent (only NULL rows), writes a negative
sentinel for un-probeable files so they're neither re-probed forever nor matched.
- find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect,
anchored per cluster to bound the span (no chain drift); keeper = highest pixel
area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances.
- apply: re-point each loser's post links to the keeper (so no post loses the
video) THEN delete the redundant records + files via delete_images (cascade).
dry_run shares the same discovery predicate and returns the projection only
(rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated).
- dedup_videos_task (maintenance queue; summary → task_run.metadata).
- POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so
the card shows the dry-run projection before the destructive apply.
- VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind
a confirm dialog. Mounted in the Maintenance panel.
Tests: dedup collapses + re-links the loser's post to the keeper + removes the
file; dry-run deletes nothing; distinct durations aren't grouped; task registered.
(Migration 0052 for duration_seconds already shipped with Phase 1.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.
Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.
- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.
Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).
Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dedup branches of _import_media linked the existing image to the new post via
_apply_sidecar(artist=None), relying on the SIDECAR to carry the artist. But an
archive member's artist comes from its path, and under post-first the per-media
sidecar is minimal (no artist) — so a re-packed / cross-posted archive image
deduped and was left UNLINKED from the new post, i.e. the post showed "no images".
Resolve the path-anchored artist (derive_top_level_artist) up-front in
_import_media and pass it to both enrich-on-duplicate branches (sha256 + phash
larger_exists) and the new-record path. Drop the now-dead _attach_artist helper
(its logic is inlined at the single new-record call site).
Surfaced by the new test_archive_all_deduped_is_benign_not_flagged (was asserting
2==4: the second post got no provenance links).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause (operator-confirmed via event metadata + lsar): a "High Resolution
files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase).
_import_archive only extracted one level — a nested-archive member failed
is_supported and was skipped, so the real pages were silently dropped and the post
showed "archive but no images". The disk scan found this pattern recurring across
the attachment store.
- Recurse into nested archives via _collect_archive_members: a member that is
itself an archive is bomb-probed and extracted too, depth-capped at
_ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so
they link to the right Post. Each level is wrapped so one bad nested archive
can't abort the import. The shared path means external (mega/gdrive) archives
recurse too.
- Replace the catch-all "held no supported members" string with a per-outcome
tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped
case is now recognised as BENIGN — images already in the library, re-linked to
this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no
longer false-flags in event metadata.unextracted_archives. Genuine failures
carry the precise breakdown.
Tests: nested zip-in-cbz imports both inner images + links them to the outer post;
all-deduped archive returns attached with error=None and links images to both posts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Milestone #67 step 3. Spell out, at the IngestCore.run seam resolution, that
post_record_key + write_post_record are the post-first contract a platform
implements when migrating onto the native core ingester — the post-record owns
the body/links, the per-media sidecar carries image identity only. The import
side is already self-enforcing via uses_native_ingester → importer.post_first.
Durable directive recorded as FC project rule #120.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-media sidecar no longer carries title/url/content (post-first, #856) —
update the assertion to expect image identity only (category/id/source_url).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.
Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).
- patreon_downloader: the per-media sidecar is now minimal — {category, id,
source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
resolution + detail-fetch (the post-record, written first in the walk, already
did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
`_apply_sidecar` still writes source_filehash + provenance + primary_post_id
but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
so applying a body-less sidecar would clobber raw_metadata + re-sync links off
empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
uses_native_ingester(platform) — the future-proof seam, so a platform migrating
onto the native core flips to post-first automatically (step 3). Media imports
before post-records but both unify on external_post_id, so the post ends with
its body either way.
Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If Patreon renames/restructures the post body field again (as content →
content_json_string already did), every body silently comes back empty and we'd
archive empty posts without noticing. Surface that as a loud failure.
Research-grounded design (Patreon `content` is officially null|string, body has
no post_type gate, gallery-dl independently added the same content_json_string
fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction
threshold would false-positive constantly. The robust, creator-independent break
signature is "a meaningful sample of posts, a body extracted from NONE of them."
- ingest_core counts posts_recorded / posts_with_body on the native post-record
path (gallery-dl never enters it, so the canary is native-only by construction).
- When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return
ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are
literally "fix the field-set/parser, not creds"). Placed after the timeout/stop
returns so it never masks a more specific failure.
- Run summary always appends "bodies X/Y" for sub-threshold observability (a
partial regression that still extracts some bodies shows in the Raw stdout).
Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success;
below the sample floor -> success (tick safety).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
External downloads import IN PLACE, so the post-attach dedup-skip unlink could
delete a file that IS an ImageRecord's backing file — orphaning the record and
404-ing on playback. Two sources of that:
- Two links on the same post (same film from mega + gdrive) emitted the same
filename into one external/<post_id>/ dir; the second overwrote the first.
Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path.
- The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it
path-safe: only unlink when `f` is NOT the existing record's canonical file.
Plus an operator-triggered orphan-repair maintenance task
(prune_missing_file_records_task) to clean up records already orphaned by the
bug: scans ImageRecords, deletes those whose file is gone (cascade), with an
NFS-stall guard that aborts without deleting if a large sample is mostly missing.
Wired through POST /api/admin/maintenance/prune-missing-files and a
MissingFileRepairCard in the Maintenance panel.
Tests: refetch-same-link keeps the canonical file; orphan repair deletes only
real orphans and aborts on the mostly-missing guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Text-only Patreon posts (WIP/announcement/poll — the bulk of a creator's feed)
rendered a big empty 'No images attached to this post' placeholder taking half
the card. Render the media column only when the post HAS images; image-less posts
let the title + body span the full width. Removes the now-dead PostEmptyThumbs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
THE empty-body root cause. Patreon deprecated the flat `content` HTML field —
it returns null on the feed AND the detail endpoint, for every post type
(confirmed against the live API: all 135 StickySpoodge posts, text_only/
image_file/poll alike). The real body now lives in `content_json_string` (a
ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse
fields[post]=content request omits it. Not credential, not post_type: a request
shape gone stale.
- NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks
bold/italic/underline/strike/code/link, hardBreak, inline images, lists,
headings; unknown nodes degrade to children). post_body_html(attrs) = the one
resolver: legacy content HTML else convert content_json_string.
- patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite
fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post])
and resolve via post_body_html (replaces the wrong sparse req + full-fetch
fallback).
- patreon_downloader._write_sidecar_data: resolve body via post_body_html
(feed content_json_string) before the detail-fetch; memoize resolved HTML.
- tests: prosemirror converter unit tests; client legacy + content_json_string
paths; contract pins content_json_string.
Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-media path (_apply_sidecar) and the post-record path (upsert_post_record)
each carried a VERBATIM copy of the post-field write (url/title/date/description/
attachment_count/raw_metadata + external-link sync). Two copies of one concept =
the divergence risk #753 targets. Consolidate into one _apply_post_fields(post,
sd) helper both call — a single predicate for how a post body/links get stored,
so the two sources can't drift. Behavior identical (fill-with-non-empty); both
paths already covered by existing importer tests.
Groundwork for the planned post-first ingest model (single authoritative post
record; media attaches to it) as more platforms move onto the native ingester.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two corrections from operator review:
1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI
section — the native ingester now writes a per-post line into the run stdout
(parity with gallery-dl's per-file stdout), so the per-post handling shows in
the panel the operator already uses.
2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record
now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring
the download_post -> MediaOutcome contract — and the downloader owns the read;
ingest_core only formats the outcome into the log line.
Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture'
section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+
' — EMPTY' when 0), so an empty body is self-explanatory by post_type.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator can't (and shouldn't have to) hunt worker logs to see why a recapture
left a post body empty. Surface per-post handling ON THE EVENT, in the UI.
The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a
per-post diagnostic {post_id, title, post_type, body_chars} with zero extra
fetching — a 0-char body next to its post_type explains an empty post at a
glance (e.g. polls/embeds whose body the API never returns).
- ingest_core: accumulate post_diagnostics; thread via DownloadResult
- download_service: write to DownloadEvent.metadata_['post_diagnostics']
- DownloadDetailModal: 'Post capture' section — totals + empty-body table
(post_type + chars, flagged) + all-posts table; included in Copy-all
- tests: ingester diag (post_type + body_chars), download_service metadata
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator-flagged: 9 StickySpoodge posts had empty bodies in FC despite the body
plainly existing + being accessible (creds refresh didn't help). All 9 are
body-only / poll / embed / announcement posts with no downloadable gallery
media — Patreon's detail endpoint returns content:null for these under the
sparse fields[post]=content request even though the body exists.
fetch_post_detail_content now re-fetches the FULL post resource once when the
sparse request comes back empty: recovers the body when the sparse fieldset was
the cause, and logs post_type when even the full resource is empty (body lives
elsewhere). Only the empty cases pay the extra GET; the 126 already-working
posts keep the fast sparse path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator-flagged: a recapture 'caught nothing' for a post and there were no
logs explaining why. Three silent spots now log, so a recapture's per-post
outcome is diagnosable (retention bounds the volume):
- patreon_client.fetch_post_detail_content: the 200-OK-but-null-content branch
was silent — now logs 'fetched N chars' on success AND 'empty/null content
(tier-gated or no text)' on the empty case (the most common silent miss).
- patreon_downloader.write_post_record: logs each post's FINAL body outcome
(captured N chars / NO body) read off the memoized attrs after detail-fetch.
- ingest_core summary: appends post-record + relinked counts to the run summary
(surfaces on the event stdout the operator already reads).
- download_service phase3: logs how many on-disk images got source_filehash
relinked (N/total) per recapture.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.
New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.
- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):
- utils.paths.filehash_from_url — one source of truth for the extractor;
patreon_client._filehash now delegates so capture- and render-time
hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
every inline image whose filehash maps to a stored image of THIS
artist; unmatched / pre-Phase-2 images keep hotlinking.
Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.
Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.
Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.
Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.
Completes Phase 4. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).
- _route_files now mirrors download_service._phase3_persist branch-for-branch:
imported/superseded → collect member_image_ids+image_id (provenance-linked via
the synthesized sidecar, same as extracted-zip members) → caller enqueues
tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
fetch result (files/bytes) or fetch failure, per-file import decision, dead-
letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
exactly the member images.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.
NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.
With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
requeue-with-countdown if busy), fetch via external_fetch into the artist
library tree, then route each file through importer.attach_in_place via a
synthesized sidecar so it links to the SAME post (archive→ImageRecords,
else→PostAttachment; on-disk original removed for captured files, art stays);
thumbnail+ML enqueue for new images; status downloaded | failed | dead with
attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
columns in 4d — defaults on, rule #26).
Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.
mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.
- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
/attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
(_apply_sidecar + upsert_post_record) so a re-import never resets a link's
status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
no-dup on reimport), serializer.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as
plain text. Now:
- html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings,
inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay
stripped; their text is preserved). Benefits the existing ProvenancePanel too.
- post_feed_service.get_post: add sanitized `description_html` to the DETAIL
response (the feed list stays lightweight plain text by design).
- PostCard.vue: render description_html via v-html once expanded (fetched with
detail); collapsed + no-detail fallback stay plain text. Styled close to the
source (headings, images max-width, accent links, lists, quotes, code).
Tests: sanitizer (headings/img/lists survive, img javascript: src dropped);
get_post returns sanitized description_html.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test_download_service stubs build dl_result as a SimpleNamespace that
doesn't set the new field; read it via getattr (matching the existing
retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any
caller that predates the field.
Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.
- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).
Slice 0b of milestone #64. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).
First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Series tab strip and the Browse search/sort (and Suggestions controls)
scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab
controls into one sticky header pinned under the 64px TopNav. The controls
had to leave v-window — it clips sticky children — so they're driven by the
tab from the header instead of living inside each window-item.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The headline bug: aliases created from the modal NEVER resolved. Create
sent the normalized display name ('Sword', 'Uchiha Sasuke') while
resolution keys on the raw booru model key ('sword', 'uchiha_sasuke',
case-sensitive) — so the mapping was stored under a key nothing looks up,
and the prediction kept reappearing unaliased. The raw key wasn't even in
the /suggestions response, so the modal couldn't send it.
- Suggestion now carries raw_name (the model key an alias must use) and
via_alias (surfaced via an operator alias); both serialized by the API.
- Modal alias-create sends raw_name, not display_name (the fix). Aliased
suggestions show an 'alias' badge and a 'Remove alias' action; 'Treat as
alias for…' is hidden for centroid hits (no model key) and already-aliased
rows.
- Tag-side management: TagCard ⋮ → 'Aliases…' opens a dialog listing the
model keys that fold into a tag, with remove (GET /api/tags/<id>/aliases +
AliasService.list_for_tag). Creation stays in the modal suggestion flow.
Tests: full API round-trip locking the raw-key contract (raw_name exposed →
alias authored with it → resolves + via_alias on a later image);
list_for_tag (service + API); via_alias/raw_name on the existing service
suggestion tests. No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An image whose on-disk path contains '#' (post folders like 'BLUE#59')
served its hash-named thumbnail fine but 404'd the original: the unencoded
'#' in image_url was parsed by the browser as a URL fragment, so
'#59/01_timelapse.jpg' never reached the /images route. Add a shared
image_url(path) helper that percent-encodes the path (safe='/') and route
the 3 raw builders (gallery detail + 2 in series) through it. Not a
cleanup-tool deletion — the file is on disk; only the URL was wrong.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-asked: the tab strip and search field were stacked; place them
side-by-side in a single flex bar (tabs left, search + scope chips right),
wrapping to two rows only on narrow viewports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Browse tab nav scrolled away (operator didn't know it existed) and
Posts had no search. Roll the tab strip + a shared search field into one
sticky block pinned under the 64px TopNav.
- Posts gains server-side text search: PostFeedService.scroll()/around()
+ /api/posts accept q (ILIKE over post_title OR description), applied
INSIDE the artist/platform WHERE so search stays scoped to the active
filter. Scope shown as clearable chips next to the search field.
- Artists/Tags search consolidates into the sticky bar: their inner
search boxes are removed; they react to route.query.q (q is deep-
linkable, e.g. /browse?tab=posts&q=foo). Platform/kind filters stay.
- Posts empty state now distinguishes 'no matches' from 'no posts yet'.
Tests: posts q-search matches title|description and stays artist-scoped
(service); q passthrough (api).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tinted backing was set on the square .fc-kebab wrapper span while the
button is round, so a translucent square showed behind the round ⋮.
border-radius:50% makes the backing a circle matching the button.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
add_post now stamps the post's parsed START (constant) on every staged
pending page so the group start survives junk removal; list_pages
surfaces it as start_page. Update the stale per-page [9,10,11] assertion
to check grp["start_page"] == 9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the auto-renumbered 1..N position key with operator-OWNED page
numbers: sparse, gaps allowed, editable, never auto-renumbered. Order follows
the numbers; unnumbered pages sort to the tail. This is the fix for the model
that clobbered hand-set numbers on the flatten — numbers are now data, not a
derived sequence.
- series_service: drop the renumber-on-reorder/remove; order by page_number
NULLS LAST; new set_page_number(image_id, n|None); list_pages returns `gaps`
(one entry per missing-number run) + each pending group's parsed `start_page`;
set_cover renumbers below the current min; place_pending(image_ids, start_page)
numbers placed pages sequentially from the start (drop junk first → numbers
line up); add_post stamps the parsed start on staged pages.
- api/tags: POST /series/<id>/pages/number (set one page's number); /pending/
place takes start_page; removed /reorder.
- frontend: per-card editable number input; one gap block per gap with
drop-on-edge to assign the adjacent number (middle → type); append drop zone;
pending tray gets a "from page N" field + "Place from page N".
- tests reworked: sparse numbers + gaps, place-from-start, set-page-number route.
No migration; nothing destructive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add-from-post no longer appends straight into the run — it STAGES the post's
pages as pending (per-page status; page_number NULL), grouped by source post,
so the operator drops junk (text-free alts, bumpers) and places the keepers
into the sequence with clean series-global numbering.
- migration 0048: series_page.status ('placed' default | 'pending') + nullable
page_number.
- series_service: placed/pending split everywhere (list_pages returns the
placed run + a `pending` section grouped by source post; reorder/cover/
list_series operate on placed only); add_post stages pending; new
place_pending(image_ids, before_image_id=None) flips pending→placed spliced
before a page (or appended) and renumbers; junk removal reuses remove_images.
- api/tags: /add-post now returns staged count; new POST /series/<id>/pending/
place.
- frontend: PostSeriesMenu navigates to the series after staging; seriesManage
store surfaces `pending` + placePending; SeriesManageView gains a pending
tray (per-post groups, place-all / place-one / drop-junk).
- tests: pending staging, place (append + insert-before), ignore-already-
placed, drop-junk, route guard; updated add_post + match-accept expectations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframe a series from "ordered chapters that own pages" to ONE flat,
series-global ordered run of pages with optional cosmetic chapter DIVIDERS
over it. A chapter no longer wraps content — it's a labeled divider anchored
to the page that begins it; a page's chapter is derived as the nearest
preceding divider. This is what lets installments assembled from multiple
sources sit in one continuous, correctly-numbered sequence (operator's
Goblin Juice case).
- migration 0047: flatten each series to a series-global page_number
(preserving today's reading order); convert each existing chapter to a
divider anchored at its first page (keeping title/stated_part); drop
series_page.chapter_id; reshape series_chapter (anchor_page_id UNIQUE FK,
drop chapter_number/is_placeholder/stated_page_start/end). Loss-safe for
content; drops empty placeholder chapters + a redundant page-1 divider.
- series_page: page_number is now the series-global order; no chapter_id.
- series_chapter: anchored divider (anchor_page_id, title, stated_part).
- series_service: flat list_pages (one run + derived dividers + per-page
source_post + part_gaps), series-wide reorder/renumber, divider CRUD
(create/update/move/delete); retired per-chapter reorder/merge/placement.
- api/tags: drop chapter_id from add; /chapters endpoints are divider
create/update/delete (removed chapter reorder/merge/page-reorder).
- series_match_service: series "end" reads max(series_page.stated_page);
accept appends via add_post. tag_service series-merge appends src's pages
after tgt's max so the merged series stays one clean run.
- frontend: seriesManage store + SeriesManageView → one continuous
drag-reorder grid with inline divider bars + series-global page numbers;
reader walks the flat run, headings from dividers; PostSeriesMenu copy.
- tests reworked across the series suite for the divider model.
Phase 2 (pending staging for add-from-post) is separate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline INSERT…SELECT backfill in migration 0045 wrapped the table
creation and a ~100 GB pass over image_record.tagger_predictions in one
transaction: nothing committed until the end, it was unmonitorable, and an
earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A
deploy got stuck on it for ~2h with image_prediction never appearing.
Split the concerns:
- 0045 now creates ONLY the table + indexes (instant DDL → web boots).
- New backend.app.tasks.admin.backfill_image_predictions_task copies the
>= store-floor predictions from the JSON into image_prediction, batched by
id window and committed per chunk: live progress, resumable (re-enqueues
from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each
stays in the DB executor streaming each window — no Python-side 100 GB load,
no materialization.
- POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card
to trigger the one-time run after upgrading.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MATERIALIZED-CTE scalar guard forced Postgres to materialize all object
rows with their full JSON (~100 GB) to temp before json_each — on NFS that's a
huge spill and pathologically slow (risks disk-full). Replace with an inline
CASE that feeds json_each an empty object for non-object rows: same scalar
guard, but a single streaming pass with no materialization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some image_record rows store tagger_predictions as a JSON scalar/null rather
than an object; json_each throws 'cannot deconstruct a scalar' on those,
rolling back the whole migration. Filter to json_typeof = 'object' in a
MATERIALIZED CTE so the guard runs before json_each ever evaluates a scalar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #764 in-place prune (rewrite tagger_predictions to >=0.70) is too slow on
100 GB of TOAST and fails at its soft limit (interrupts a query mid-flight ->
'another command is already in progress'). #768 supersedes it: extract only
the >=floor predictions into image_prediction via this set-based backfill,
then drop the column (step 3) — reading 100 GB once + writing ~840k small rows
beats rewriting 100 GB in place.
So this backfill no longer assumes the prune ran: it filters by
ml_settings.tagger_store_floor (default 0.70) itself, handling the full or
partially-pruned JSON identically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch every prediction READER off the JSON column onto the normalized
image_prediction table. Parity by construction: each reader loads the same
{raw_name: {category, confidence}} dict it consumed before (via small
_load_predictions helpers), so all downstream threshold/alias/merge/consensus
logic is byte-identical — only the data source changed.
- suggestions.SuggestionService.for_image (and for_selection via it)
- ml.apply_allowlist_tags (iterates images that have prediction rows)
- importer re-import reset deletes the image's prediction rows
The tagger_predictions JSON column is still dual-written (step 1) so it stays
valid during transition; the backfill task's NULL check still works. Removing
the JSON write + DROP column + retiring the #764 prune is the cleanup
follow-up (needs a quiesced-worker window for the DROP lock).
Tests: shared tests/_prediction_helpers.seed_predictions seeds the table;
read-path tests (suggestions, bulk consensus, allowlist apply, API) seed there
instead of ImageRecord.tagger_predictions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize tagger predictions out of the image_record.tagger_predictions JSON
blob into a queryable per-prediction table. Step 1 of the cutover (expand):
additive + low-risk — reads still use the JSON, this just adds the table and
keeps it populated.
- ImagePrediction(image_record_id, raw_name, category, score) — stores the
RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution
is unchanged. Indexed for per-image reads + by (raw_name, score).
- Migration 0045: create table + set-based backfill from the JSON via
json_each (fast post-#764-prune). The old column stays (vestigial) and is
dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the
hot image_record table, so it waits for a quiesced-worker window.
- tag_and_embed dual-writes the rows (delete-then-insert, idempotent);
tagger_store_floor already applied in infer().
Next: switch suggestion + allowlist reads to the table, then drop the JSON
write. Plan-task #768.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DB backup polish (plan-task #764 Q3):
- pg_dump now uses custom format (-Fc): compressed (much smaller on NFS) and
restored via pg_restore. Artifact extension .sql → .dump; restore_db swaps
psql -f for pg_restore -d. BackupRun.sql_path field name kept (it's just the
db artifact path).
- Reconcile the subprocess guardrails: the DB timeout was 720s with a stale
'Celery soft is 10 min' comment, but backup_db_task's soft limit is actually
1800s — so the bounded-kill fired 18 min early. Set DB=1700s / images=21000s,
each just under its task's Celery soft limit so _run_bounded stays the
primary guard (an NFS D-state hang defeats Celery's own SIGKILL).
Real shrink of the DB is the #764 prune; this makes each dump smaller/faster
on top of that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-time backfill that actually shrinks the DB: drops stored
tagger_predictions entries below ml_settings.tagger_store_floor from every
image_record row, and clamps any allowlist min_confidence below the floor up
to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate
so backfilled rows match new imports. Keyset by id ASC, idempotent,
self-resumes on the soft time limit; runs on the maintenance_long lane.
pg_dump copies live data only, so this alone fixes the #739 backup timeout —
the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional
disk-return step, brief because post-prune the live data is tiny.
- admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions
- PrunePredictionsCard in the Maintenance panel (shows the current floor)
- tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consumer #4 of the store-floor change (#764). An allowlist tag can't
auto-apply more permissively than the ingest floor — predictions below
tagger_store_floor aren't stored, so a lower min_confidence behaves
identically to the floor. update_threshold now clamps to max(value, floor);
the AllowlistTable confidence input min-binds to the live floor and clamps
on edit. Keeps the stored threshold honest about actual apply behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default
0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70).
Storing every tag down to 0.05 from a ~10k-tag tagger is what grew
image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70
and the centroid/learned path covers lower-confidence preferred tags, so the
sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim
land next; this only changes the write gate for NEW imports).
- ml_settings.tagger_store_floor (migration 0044, default 0.70)
- tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor
- ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold
below the floor (nothing below the floor is stored, so the gap surfaces
nothing) — server backstop for the UI slider clamp
- Settings → ML: store-floor slider + caption; category sliders min-bound to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recover_stalled_task_runs used the 5-min default for the download queue,
but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT
(1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as
phantom 'RecoverySweep' failures — visible in System Activity but absent
from the Subscriptions view (the download finished ok, reset the source's
consecutive_failures; only the orphaned task_run kept the stamp, since
_finalize only updates rows still 'running').
Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard
limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real
hard kill is swept by the task-run and event sweeps together. Restores the
documented invariant (every override >= task time_limit). Regression test
pins the threshold above the hard limit so a future limit bump can't
silently re-break it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recent failures gains a client-side search over the already-loaded 24h
rows (task/queue/target/error), shown as a filtered/total count alongside
the existing error-type chips. All recent activity gains a debounced
server-side task-name search (new `task` ILIKE param on /runs) so it
spans the full history, not just the loaded page. LIKE wildcards are
escaped so task names' literal underscores match literally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom self-join (resolve a character's fandom NAME via Tag.fandom_id->Tag)
and the {id,name,kind,fandom_id,fandom_name} dict were hand-written in
TagService.autocomplete/.list_for_image, GalleryService.get_image_with_tags and
the api/tags handlers — the last few grown by this session's fandom-on-chip
feature. Consolidate to services/tag_query: fandom_join_alias() + tag_columns()
build the select; serialize_tag(row) builds the dict. Now a new tag field is
added in one place.
Over-DRY guard: TagDirectoryService selects the full Tag ORM + an image-count
aggregate (a different select shape) — left as its own variant. §8b: the
fandom_lookup alias lives only in tag_query; gallery + both api/tags handlers
serialize via serialize_tag. Test: serialize_tag handles enum + string kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
encode_cursor/decode_cursor (base64 <iso8601>|<id>) were defined identically in
gallery_service AND post_feed_service, with artist_service importing gallery's
copy. Two implementations of one cursor format silently break pagination in
whichever feed drifts. Extract to services/pagination.py; gallery/post_feed/
artist all import it. Dropped now-unused base64/datetime imports.
§8b: encode_cursor/decode_cursor now defined only in pagination.py. Existing
cursor round-trip tests still cover it via the re-export. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI lint flagged UP047 — use the native generic syntax def get_or_create[T](...)
instead of typing.TypeVar on Python 3.14.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The find-or-create dance — SELECT, then a SAVEPOINT INSERT that recovers (not a
full rollback) on IntegrityError when a concurrent worker inserted first — was
hand-rolled identically in 4 async sites: ArtistService.find_or_create,
TagService.find_or_create, ExtensionService._find_or_create_artist and
._find_or_create_source. Divergent copies of exactly this pattern are how the
duplicate-row/race bugs in reference_scalar_one_or_none_duplicates crept in, so
it now lives once in services/db_helpers.get_or_create (returns (row, created);
factory adds+flushes+returns the row; caller owns the outer commit).
Over-DRY guard: SourceService's IntegrityError sites RAISE DuplicateSourceError
(reject-on-conflict, a different concept) — left alone. Importer._get_or_create
is the lone SYNC consumer (already shared by 2 callers) — stays separate, can't
cross the sync/async boundary. §8b: no hand-rolled async find-or-create remains.
Test: get_or_create creates then returns existing without re-invoking the factory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The icon+title v-card-title heading (d-flex align-center + gap + <v-icon size=small> +
<span>) was hand-rolled identically in 13 cards/dialogs (15 heading instances).
Consolidate to <CardHeading icon title> (components/common) with an iconColor
prop (error headings) and a default slot for trailing content (spacer+actions,
inline status chip). Adopted everywhere the pattern appears — all-or-nothing per
the hardened DRY process.
Over-DRY guard: plain text-only <v-card-title> one-liners are NOT this pattern
and stay; DownloadDetailModal leads with a status CHIP (not an icon), a different
concept, left alone. §8b: the only remaining d-flex align-center v-card-title is
that intentional variant. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The preview sample-name grid (scrollable monospace chip grid) was hand-rolled
5 times with verbatim-duplicated markup + CSS — TagMaintenanceCard (×4) and
PostMaintenanceCard. Consolidate to <SampleNameGrid> (components/common): pass
:names for the plain case, default slot for the normalize from→to chips
(styled via :slotted .fc-name). Removed the duplicated .fc-name-grid/.fc-name
CSS from both cards.
Over-DRY guard: only the verbatim-duplicated grid is merged — each card's
preview/commit logic and result-count lines genuinely differ and stay put;
MinDimensionCard's typed-token confirm is a separate variant, untouched.
§8b: fc-name-grid now lives only in SampleNameGrid. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The management view showed the series name but had no way to change it post-
creation (rename was only on the browse-card kebab). Add a pencil next to the
title that opens TagRenameDialog (reuses the canonical rename → PATCH
/api/tags/<id> with its collision→merge flow, since a series IS a
Tag(kind=series)); the new name reflects in place. Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The muted-text token was redefined identically in 12 component <style scoped>
blocks. Consolidate to one global utility in styles/app.css; remove the 12
copies. Keeps the explicit on-surface-variant (vellum) token, NOT Vuetify's
opacity-based text-medium-emphasis (per the muted-text-token rule). Behavior-
preserving: every class=fc-muted usage now resolves to the single source.
§8b exhaustiveness caught (and I fixed) my own sed clobbering the new app.css
rule — now exactly one .fc-muted definition exists, zero component-local.
Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First pattern-consistency DRY pass (process #594). The overflow kebab was
hand-rolled 7 ways in two divergent activator strategies — Pattern A
(#activator + v-bind) which silently breaks inside the teleported image modal
(#711), and Pattern B (manual v-model + activator=parent + open-on-click=false +
z-index 2400) the modal kebabs needed as a workaround.
New <KebabMenu> (components/common) bakes in the modal-safe strategy
UNIVERSALLY, so every kebab works in modal and non-modal contexts — folding the
latent #711-class bug fix into all five Pattern-A sites. Menu items go in the
default slot; variations (size/variant/location/label/min-width) are props.
Adopted across all 7: TagChip, SuggestionItem, TagCard, SeriesView card,
SeriesManageView, BackupRunsTable, SourceActions. Exhaustiveness (§8b):
mdi-dots-vertical now lives only in KebabMenu. Labeled dropdowns / nav menus /
filter popovers are a different concept and left alone. Seeded the pattern
catalog so new code reuses the primitive. Test: KebabMenu renders slot items +
trigger label/glyph + presentational props.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Posts, Artists, and Tags are the three 'browse the library by an axis'
surfaces; Subscriptions stays purely management (operator-asked 2026-06-09).
New BrowseView renders them as tabs (?tab=posts|artists|tags); only the active
tab mounts. The old standalone paths become redirects into the matching tab,
preserving deep-link query (/posts?post_id=N → /browse?tab=posts&post_id=N) and
keeping the route names so existing { name: 'posts'|'artists'|'tags' } links and
path pushes still resolve. Nav now reads Showcase · Gallery · Browse · Series ·
Subscriptions, with Settings pinned right.
Test: /browse resolves; /tags and /artists redirect into their tabs; a posts
deep link survives the redirect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Settings is configuration, not content, but sat mid-row (between Series and
Posts). Pull it out of the centered content links and pin it to the right as a
gear+label, matching the convention that config lives at the right edge. Mobile
is unchanged — Settings stays in the hamburger menu (navRoutes still includes
it). Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The typed dropdown sourced the threshold-filtered panel list (>= 0.70 general),
so low-confidence actions/features the model DID predict never appeared — forcing
hand-typed custom tags instead of accepting the model's canonical formatting.
Add a threshold override: SuggestionService.for_image(threshold_override=) and
GET /images/<id>/suggestions?min=<f> surface EVERY stored prediction (down to the
0.05 store floor), alias-resolved and normalized, still excluding applied/rejected
and unsurfaced categories. The suggestions store gains allByCategory + loadAll
(min=0); the dropdown searches that full set (cap 20), while the Suggestions panel
stays curated at the configured threshold. Accept/dismiss drop from both lists.
Operator-asked 2026-06-09. Test: a 0.30 general prediction is hidden by default
but surfaced with threshold_override=0.0; unsurfaced categories still excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A character chip with a fandom only rendered a bare arrow. Surface the fandom
NAME inline, truncated to 15 chars (full name in the tooltip). Resolve the name
via a Tag self-join in both tag paths the modal uses — list_for_image
(/api/images/<id>/tags) and gallery get_image_with_tags
(/api/gallery/image/<id>) — so chips show the fandom on first open and after any
reload. Falls back to the bare arrow when only fandom_id is known. Operator-asked
2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Series browse tab had no way to find a series in a long grid and no
per-series actions. Add a search field (instant client-side name/artist filter
over the already-loaded list) and a kebab on each card with Rename (reuses
TagRenameDialog → PATCH /api/tags/<id>, with its collision-merge flow) and
Delete (confirm dialog → DELETE /api/admin/tags/<id>; series_page/chapter/
suggestion cascade, images kept). Gap badge moved to the cover's top-left so the
kebab can sit top-right. Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accepting an auto-suggested tag (Suggestions panel or the autocomplete
dropdown) left focus on <body>, so the operator had to re-click the tag field
to add the next one. Expose TagAutocomplete.focus (the existing mobile-aware
focusInput) and call it after accept from both paths; SuggestionsPanel emits
'accepted' for the parent to refocus. Operator-asked 2026-06-08.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The character pointing at the fandom had no image associations, so it was
itself unused and inflated the dry-run count to 2. Tag it on a real image so
it is used (the real-world shape) — the fandom survives via a live character.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom/chapter exclusions added in fb05c5e only touched find_unused_tags
(the preview SAMPLE). prune_unused_tags re-implemented the predicate inline for
the dry-run COUNT and the live DELETE with only the image_tag + series_page
checks — so the preview showed a safe list of names while the delete removed
every fandom (and chaptered series). Operator-flagged 2026-06-08: real data loss
— assigned fandoms deleted, their characters SET-NULLed.
Extract _unused_tag_conditions() as the single source of truth and use it for
the preview, the count, AND the delete, so they can never diverge again. Added a
prune-commit test asserting the LIVE delete spares a character's fandom and a
chaptered series.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-flagged (again) on the tag-merge picker: Enter on the dropdown re-opens
it instead of accepting the selection. I'd already patched this twice (fandom
picker + fandom set dialog) with copy-pasted capture-phase handlers, so DRY it.
New composable useAcceptOnEnter(accept): tracks the menu state and, on a
capture-phase Enter, lets Vuetify pick when the menu is open but calls accept()
(and blocks the re-open) when it's closed. Applied to every confirm-style picker:
- TagsView merge-into picker (the reported one)
- AliasPickerDialog
- PostSeriesMenu add-to-existing
- FandomPicker + FandomSetDialog (refactored off their bespoke handlers)
One behavior, one place to change it.
Operator: 10-frame max-pooled tagging on video produces a lot of noisy tags, and
the sampling burns time/GPU. Drop the VIDEO_ML_FRAMES default to 6 (still env-
overridable). Fewer frames = less per-frame noise into the max-pool and a smaller
frame-sampling budget. Quality/perf of the whole video path is being reviewed
separately.
The task logged nothing and SoftTimeLimitExceeded stringifies to empty, so a
timeout surfaced as a bare 'SoftTimeLimitExceeded()' with no clue which file or
why (operator-flagged 2026-06-08).
- Log start (id/path/mime/bytes/video?), per-phase timing (load_models, video
probe/sample/infer, tag, embed, persist), and a success summary.
- Track a + file ; on SoftTimeLimitExceeded log it and re-raise
SoftTimeLimitExceeded WITH that context (keeps the 'timeout' task_run status
but gives the activity a real error_message: which file, which phase, elapsed).
- On other exceptions, log context then re-raise the ORIGINAL (preserves
autoretry for OSError/DBAPIError/OperationalError).
Now a stuck run names the culprit — most likely a slow video (frame sampling is
up to 10x60s ffmpeg) or a huge image; the phase log will say which.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_unused_tags only excluded tags with image_tag or series_page references, so
it flagged every fandom as 'unused' — fandoms are NEVER applied to images (a
character carries its fandom via tag.fandom_id), and the FK is ondelete=SET NULL,
so deleting one silently strips the fandom off all its characters
(operator-flagged 2026-06-08: artist-OC fandoms showing as unused).
Exclude tags referenced as a character's fandom_id, and (same class of gap) tags
referenced by a series_chapter (an all-placeholder series has chapters but no
pages yet). A genuinely orphaned fandom with no characters is still swept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_create_protective_aliases scanned every image_record's tagger_predictions JSON
(unindexed full scan, ~59k rows) to find the categories a merged-away tag's name
was predicted under. That scan ran inside the merge transaction AFTER it had
locked series_page — on a large library it held that lock for minutes and is what
blocked migration 0040 (and starved the standardization task into its 40-min
timeout).
The scan was redundant: the tagger's tag_to_category map is one-to-one (a name has
exactly one category) and a tag's kind is set from that category when created, so
kind already IS the tagger's category for the name. The scan only ever rediscovered
the kind. Build the single protective alias from src_kind directly — no scan, no
lock-holding slow step in the merge.
Rewrote test_alias_per_observed_prediction_category (which encoded the
can't-actually-happen one-name-two-categories case) → test_protective_alias_uses_tag_kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverses the advisory-lock approach (7309d1d) — it treated a replica race that
wasn't the cause and added a new indefinite-hang mode (a sibling/stale migrator
holding the xact lock).
Real cause of the 0040 hang (operator-diagnosed 2026-06-07): web has always been
a single replica. The migration's ALTER series_page queued behind a concurrent
tag-merge that held a series_page lock for minutes — _do_merge repoints
series_page then runs _create_protective_aliases, an unindexed full scan of
image_record (JSON column, ~59k rows). Migrations ran with no lock_timeout, so
the DDL hung indefinitely and silently.
Fix: SET lock_timeout (default 30s, env-overridable) on the migration connection
before alembic's transaction. A blocked DDL now fails fast with 'canceling
statement due to lock timeout'; the entrypoint exits non-zero so the deploy
retries / surfaces loudly instead of wedging. General protection for every
future migration. (The slow _create_protective_aliases scan — the actual lock
holder — is the separate perf fix still under discussion.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.
- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
async factory). A blocked merge now raises, the per-group handler rolls back +
counts an error, and the loop continues — one stuck group can't strand the
chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
diagnosable instead of silent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image-modal tag kebab's rename dialog still showed a leftover stub
('Merging two tags into one lands in FC-2c') on a name collision, dead-ending
the operator. The merge machinery has existed for a while — the Tags view
already resolves rename collisions this way. Wire TagRenameDialog to it: on the
409 collision hint, show the same merge confirmation FandomSetDialog uses
(target name, image associations moved, alias kept) and POST /api/tags/<id>/merge
into the existing tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator evidence 2026-06-07: 0.95 was too strict, skipping confident-enough
auto-applications of accepted tags. Newly-accepted tags now allowlist at 0.90;
existing entries keep their stored value and per-tag thresholds stay tunable in
the allowlist table. No migration — min_confidence has no DB server_default, so
the Python insert default governs new rows only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'Fandom for <character>' dialog (FandomSetDialog) used plain autofocus and
had no Enter handling, so Enter re-opened the dropdown instead of submitting —
the same bug FandomPicker already fixed. Mirror that flow: parent v-dialogs
focus the field via @after-enter→focusSearch (reliable past the focus-trap);
capture-phase Enter Saves the changed selection instead of re-opening the menu;
Tab jumps to the new-fandom field; creating a fandom returns focus to the
dropdown so a single Enter saves it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator feedback: thumbnails too small to judge order, no obvious way to mark
'this installment is Part 2', and the permanent two-pane picker was busy and
competed with the ordering work.
- Full-width parts, each a card with a big page grid (150px, contain so whole
pages are visible) and drag-to-reorder; positional page number as a badge.
- Editable Part # (hero field) backed by new series_chapter.stated_part —
separate from the auto-managed chapter_number, mirroring the page_number vs
stated_page split so reorder/delete renumbering can't wipe a hand-set part.
Missing-Part hints when consecutive parts' stated_part jump >1.
- Each part labels its source post (derived from pages' primary_post_id) and
shows the printed-page range with clear labels.
- Picker demoted to an on-demand right slide-over ('Add pages') with a target-
part selector; part actions (move/merge/delete) collapsed into an overflow ⋮.
alembic 0042 adds series_chapter.stated_part (nullable int).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every web replica runs 'alembic upgrade head' in its entrypoint, so under
docker stack deploy two replicas can boot at once and race the same DDL —
0040 raced in prod (operator-flagged 2026-06-07): one backend wedged on the
series_page lock while a second tried to re-CREATE series_chapter, and the
loser died with AdminShutdown, crash-looping the web service.
Wrap run_migrations() in a transaction-scoped pg_advisory_xact_lock acquired
BEFORE the version table is read. The first replica to reach it migrates and
holds the lock for the whole upgrade; siblings block, then find the version
already at head and apply nothing. Works regardless of replica count and
needs no Swarm depends_on ordering (which stack deploy ignores anyway).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.3 with the UI.
- SeriesView gains tabs: Browse (the existing grid) + Suggestions.
- Suggestions tab: pending matches as rows (post → series, per-signal strength
chips, score), Add (→ chapter) / Skip (→ dismiss); a "Matching on" toggle and
a threshold field (both DB-backed via /settings/import), and a Rescan button
that enqueues the background matcher.
- seriesSuggestions store wires load / accept / dismiss / rescan / settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirm-only "this post may continue this series" matcher.
- series_suggestion table (post_id, series_tag_id, score, signals jsonb, status
pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings
knobs (series_suggest_enabled, series_suggest_threshold).
- series_match_service: weighted additive score (title-stem / same-artist /
page-continuity / shared-distinctive-tags), no single signal gating. The title
"pattern" is derived on the fly from the post titles already in a series, so it
sharpens as more are confirmed (no persisted state to drift). Candidates are
bounded to the post's artist. match_post upserts pending suggestions (UNIQUE +
on-conflict, respecting prior added/dismissed decisions).
- accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'.
- rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from
a post-id cursor (maintenance_long lane), like normalize_tags_task.
- API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan.
- Settings: enabled + threshold exposed via /settings/import.
- Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE
dedup.
Frontend (Suggestions tab + settings card) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.2 with the UI.
- PostSeriesMenu: a "Series ▾" control on each post card — "New series from
this post" (promote → navigates to manage) and "Add to existing series…"
(dialog with a browsable picker loaded from GET /api/series, client-side
filtered — avoids the empty-autocomplete #712 trap).
- SeriesView (/series): a top-level Series browse grid — cover, name, artist,
chapter/page counts, gap badge; sort recent|name|size; cards → manage/read.
meta.title adds it to the nav automatically (peer of Posts).
- seriesBrowse store for the list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post-aware on-ramp + the data behind the missing Series browse view.
- page_number_parser: conservative stated-page parser (pages 9-12 / page 5 /
[3/8] / 3 of 8), keyword-gated to avoid false positives. Pure + unit-tested.
- SeriesService.promote_post_to_series: a self-contained post becomes its own
series — series tag named after the post, one chapter, the post's images as
pages (ordered by capture order; stated pages parsed from title/description).
- SeriesService.add_post_as_chapter: append a post as the next chapter of an
existing series, titled after the post and slotted by parsed page number
(a "pages 1-4" post lands ahead of the "pages 9-12" chapter).
- SeriesService.list_series: browse cards — cover thumb, artist, chapter/page
counts, gap flag, last-updated; sort recent|name|size + filter by artist.
- API: GET /api/series, POST /api/series/from-post, POST /api/series/<id>/add-post.
- Resolver uses ImageRecord.primary_post_id (same linkage the posts feed renders).
Frontend (Add-to-series control + Series view + nav) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.1: the series management UI now works in chapters.
- SeriesManageView: chapters as cards (inline-rename, stated-page range inputs,
move up/down, merge-into-previous, delete, pick-as-add-target), pages
drag-reorder WITHIN a chapter, a "gap: N-M missing" badge between chapters
with a stated-page hole, and Add chapter / Add placeholder. The picker adds
the selection into the targeted chapter.
- seriesManage store: chapter CRUD + reorderChapters/moveChapter/mergeChapter/
reorderPages actions; consumes chapters[]/gaps[]; addSelected targets a chapter.
- Reader: page_number is now within-chapter, so anchors switched to a global
`seq` (reading-order position) — fixes scroll/jump/active collisions across
chapters — plus chapter-title dividers at each chapter boundary.
- Updated seriesManage.spec to the chaptered store shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an ordered chapter layer to series. Reading order becomes
(series_chapter.chapter_number, series_page.page_number); a chapter may be a
placeholder reserving a slot, and carries an optional parsed stated-page range
used to flag missing-page gaps. An image still lives in at most one series ⇒ one
chapter (image_id stays UNIQUE).
- models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) +
stated_page. Migration 0040 backfills every existing series into one
auto-chapter holding its current flat pages — no data loss.
- SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter
assignment, reorder_pages, chapter-aware set_cover; list_pages now returns
chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide
reorder operates on the single default chapter and rejects multi-chapter series.
- API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an
optional chapter_id.
- TagService.merge now repoints series_chapter too, so a merged series' chapters
(and their pages) survive the source tag's deletion instead of cascading away.
- Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to
supply chapter_id.
Frontend (chapter-aware manage view + reader) lands next; until then the
existing UI keeps working via the flat pages[] + single default chapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prod stack runs under Docker Swarm (docker stack deploy), which SILENTLY
IGNORES `shm_size` — container inspect showed ShmSize still 64MB after the
a183be7 fix, and vacuum_analyze kept hitting DiskFull resizing a ~64MB POSIX
DSM segment in /dev/shm (operator-flagged 2026-06-07). Replace the ignored
`shm_size: 512m` with a tmpfs mount on /dev/shm (size 512MB), which Swarm AND
plain Compose both honor. Requires a stack redeploy to take effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Enter handler listened in the bubbling phase, so Vuetify's own input handler
(which opens the menu on Enter) fired first and my accept logic saw the menu
already opening and bailed — Enter popped the dropdown instead of submitting.
Bind it in the capture phase so it runs first, and stop the event when a fandom
is already selected so Vuetify never reopens the menu (operator-flagged
2026-06-07).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ArtistView set document.title to "<artist> — FabledCurator" on load but nothing
reset it when navigating away, so the artist name stuck on the Showcase/Gallery
tab title (operator-flagged 2026-06-07). Add a router.afterEach that sets the
title from meta.title on every navigation; detail views with no meta.title reset
to the default and then set their own dynamic title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Solid-color members phash-collapse to distance 0, so the second archive's member
deduped away ("held no supported members") and members_imported was 0. Use
structurally distinct patterned jpegs so both members import — the resume cursor
mechanics were already correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reextract_archive_attachments loaded ALL PostAttachments and ran in one pass up
to a 30-min soft limit, then died without re-enqueueing — a large archive
backlog would only ever partially process. And a naive re-run can't advance: an
already-extracted archive is still an archive on disk, so it'd re-extract the
same first batch forever.
Give it a real cursor + time-box + self-resume (mirrors normalize_tags_task,
operator-asked 2026-06-07: reasonable timeout, then re-queue so other work keeps
flowing):
- service scans attachments with id > after_id in ascending order, time-boxes
the chunk, and reports partial=True + resume_after_id (last scanned id).
- task passes a 600s budget and re-enqueues itself from the cursor until the
scan is exhausted. Routes on the maintenance_long lane.
- This is independent of the maintenance_long lane isolation (already shipped) —
that stops long tasks starving the quick maintenance queue; this stops the
re-extract itself dying on a big backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image's Camie suggestions now appear in the tag input's dropdown as you
type, filtered to the query and de-duped against the server autocomplete hits,
so the operator can pick a suggestion without hunting for it in the Suggestions
panel below (operator-asked 2026-06-07).
- Unified `rows` model (hits → matching suggestions → create row) so the
highlight index maps 1:1 to a row across all three sections; arrow/Enter/Tab
drive the whole list.
- Suggestion rows are marked (accent left-border + mdi-auto-fix score chip) and
show a "new" hint when the suggestion would create a tag.
- Picking a suggestion emits accept-suggestion → TagPanel runs the SAME accept
path as the Suggestions panel (creates raw tags, records acceptance, drops it
from the panel), then refreshes the chip rail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-specified flow for character-tag creation: focus starts in the
fandom search dropdown; Tab moves to the new-fandom field where Enter
creates; creating fills the dropdown and returns focus there; Enter in the
dropdown accepts the selection.
- Drive focus from the dialog's @after-enter (autofocus is unreliable inside
a v-dialog — the focus-trap steals it post-mount); FandomPicker exposes
focusSearch.
- Drop the @update:model-value auto-confirm that closed the dialog the instant
selectedId was set — that's what broke create-then-accept (creating set the
value and immediately confirmed). Enter now accepts (menu-closed + value),
while an open menu lets Vuetify pick the highlighted item first.
- Tab from search → new-fandom field; Enter there creates, then focus returns
to the dropdown for a single Enter-to-accept.
- Restore focus to the tag input after the dialog confirms/cancels so the
keyboard flow continues into the next tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
THE actual root cause of the "dead" tag-chip kebab (operator inspected it
2026-06-07: the menu was ghosted, blurred, behind the sidebar). The teleported
v-menu landed below .fc-viewer (z-index 2000) and the modal's
backdrop-filter: blur(8px) smeared it — so it opened the whole time, just
underneath. Every prior "fix" (un-nesting the button, the explicit activator
pattern) was chasing a click/activation problem that never existed.
Set :z-index="2400" on the tag-chip and suggestion kebab menus so they paint
above the modal. (Dialogs already render on top — only the anchored menus tied
with the modal's z-index and lost.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The old size-36 v-progress-circular sat tiny in the top-left because
.fc-viewer__media doesn't center its children (the canvas centers itself).
Replace it with a 108px dual counter-rotating accent-ring spinner as a centered,
non-interactive overlay over the modal (operator-flagged 2026-06-07).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-flagged: /downloads was never mapped in prod and everything worked —
confirmed nothing in the app references a filesystem /downloads (only the
unrelated /api/downloads route). Dropped the dead mount from web/worker/
scheduler, and scoped the new maintenance-long worker to just /images (backups
write to /images/_backups; audits + admin tasks all operate on /images).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The constant + comment landed BETWEEN @celery.task(...) and the function def,
which is a syntax error that broke the whole tasks.admin import (cascaded to
lint E999 + every backend/integration test). Move it above the decorator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two more maintenance-queue failures from the operator's 24h list:
- vacuum_analyze died with "could not resize shared memory segment to 67MB: No
space left on device" — Docker's default /dev/shm is 64MB, too small for
VACUUM (ANALYZE)'s parallel-worker shared memory. Set the postgres service
shm_size: 512m.
- backup_db_task timed out at its 12-min limit once the DB grew; a pg_dump can't
be chunked, so raise it to 30/35 min. (A long backup still briefly holds the
concurrency-1 lane — the structural fix is a dedicated lane for long one-shots.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested modal/tagging keyboard improvements:
- A2/A3: fandom dialogs autofocus their autocomplete on open; in the character-
creation FandomPicker, picking a fandom (keyboard Enter or click) confirms in
one step. FandomSetDialog stays autofocus-only (its Save can trigger a merge).
- B5: Tab accepts the highlighted autocomplete row (standard convention).
- C9: T or / jumps focus to the tag input from anywhere in the modal.
- C8: ? toggles a keyboard cheatsheet (corner hint advertises it; Esc closes the
cheatsheet first, then the viewer).
Builds on the same-batch regression fixes (kebab #711, ESC-after-accept #700,
autocomplete scroll-into-view). B6 (keep focus after add) is covered — the input
retains focus after adding a tag, and Esc now works after accepting a suggestion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two regressions the operator re-flagged (the earlier "fixes" didn't work):
#711 tag-chip kebab: TagPanel's kebab used the #activator + v-bind="props"
v-menu pattern — the exact pattern SuggestionItem's own comment documents as
NEVER toggling inside the teleported ImageViewer modal. Extracted TagChip.vue
using the proven explicit pattern (activator="parent" + :open-on-click="false"
+ a manual v-model), mirroring the working suggestion kebab. Now opens.
#700 ESC-after-accept: the guard suppressed close whenever ANY non-tooltip
overlay was active anywhere in the DOM, so a stray overlay after accepting a
suggestion (focus drops to <body>) blocked Esc. Now key off the event origin —
only defer to an overlay when Esc is pressed from INSIDE its content
(ev.target.closest('.v-overlay__content')); a stray overlay no longer traps the
modal, and dialogs/menus still handle their own Esc.
A1: TagAutocomplete arrow-nav now scrollIntoView's the highlighted row — the
list is capped at 240px and arrowing past the fold left the active item
off-screen (operator-flagged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native client treated a gallery image without `file_name` as schema drift
and raised "Patreon API changed — ingester needs update", failing the whole walk
(operator-flagged 2026-06-07: BlenderKnight post 73665615, kind=images). But the
resource had a valid URL, and the code already derives a filename from the URL
basename right below the raise — the same fallback gallery-dl uses. Patreon
legitimately serves some images without file_name, so this isn't drift.
Drop the require_file_name gate from _media_item: file_name is now optional for
every kind (images/attachments/postfile), falling back to the URL basename.
Genuine drift still raises — no resolvable URL, or a media id referenced by a
relationship but absent from `included`. Test updated to assert the fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mid-post time-box check (619e771) reads time.monotonic() once more per post,
so test_backfill_budget_cut_returns_partial_with_progress's discrete tick
sequence shifted — the >budget tick (200) landed on post1's first-item check
instead of post2's gate, cutting post1 to 0 files. Add the extra tick (20, still
under budget) so post1's item downloads, matching production where the gate and
the first should_stop are microseconds apart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).
download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.
Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/posts/mimic-in-dungeon-158372536 is a
single-post permalink, not a creator page — the resolver grabbed "posts" as the
vanity and failed (operator-flagged 2026-06-07). Add a resolution path: extract
the trailing post id and follow it to the owning campaign via the Patreon post
API (/api/posts/<id>?include=campaign), so pasting any post URL subscribes to
that creator's whole feed. `posts/` is excluded from the vanity regex so it
can't masquerade as a creator slug.
Resolution order is now: cached override → id: URL → /posts/<id> → vanity
(campaigns API + creator-page scrape). Tests cover the post→campaign resolve
and that /posts/ URLs aren't treated as vanities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new enabled source is armed for run-until-done backfill (#693) but would sit
idle until the next scheduler tick (~60s). create_source now enqueues the first
walk right away (pending DownloadEvent + download_source.delay), skipping only
when the platform is in a rate-limit cooldown (the scheduler picks it up when
that clears). Disabled sources still don't dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested follow-ups:
- #1 Failure reason on hover: the red error-count chip now shows source.last_error
in a tooltip (desktop row + mobile card), so the cause (e.g. the new
"vanity=cw" message) is visible without opening Downloads.
- #2 Collapse the single-source case: a subscription with exactly one source now
shows that source's URL + its own actions (Check / Backfill / ⋮ / Edit) inline
on the artist row — no expand needed for the common case. Multi-source keeps
the artist-level actions + expandable per-source table.
- #3 Sort by health: the Health column is sortable on a numeric rank
(never/ok/warn/fail) and the table defaults to worst-first, name as tiebreak.
- #4 Drop Preview: removed the low-value bounded-peek action from the menu and
all its wiring (backend endpoint + store fn left in place, unused).
- #5 Backfill selected: a "Backfill" button in the bulk bar arms a run-until-done
backfill on every enabled, not-already-running source in the selection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lock/reload: every inline source action (check / backfill / recover / toggle /
remove) ended by refetching the WHOLE subscription list (store.loadAll /
refresh), which blocked the UI and re-rendered the table — collapsing the
expanded row, which read as "locks then resets." The action APIs already return
the updated source, so the store now patches that one row in place
(_patchSource / _dropSource); the post-action loadAll/refresh calls are gone.
Toggling enabled, starting/stopping a backfill, recovering, and removing are now
instant and leave the expansion intact.
Button regroup (operator-flagged: tiny, mis-clickable, not grouped by function):
- New shared SourceActions.vue used by desktop SourceRow + mobile SourceCard.
- Frequent actions stay as size="small" buttons: Check, Backfill/Stop.
- Low-frequency / destructive actions move into a labelled overflow (⋮) menu —
Preview backfill, Recover dropped near-duplicates, Remove source — so they
can't be fat-fingered, and the labels spell out recover-vs-backfill.
- Edit moves next to the source URL (its identity), out of the action cluster
where it sat beside Remove.
- Single-source Remove now confirms (it had no guard before).
Tests: store patches/drops in place without dropping the cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/cw/Atole resolved vanity='cw' — the
vanity regex only skipped a /c/ prefix, so Patreon's current /cw/ ("creator
workspace") form fell through to the bare-vanity branch and captured the prefix
instead of the slug. Every /cw/ source then failed campaign-id resolution
(API + page-scrape both looked up "cw"). Operator-confirmed 2026-06-07 via the
new error text: source_url='.../cw/Atole'; vanity='cw'.
Add cw/ to the optional prefix group (ordered before c/ so the longer prefix
wins), and have the creator-page fallback try the /cw/ form too. Test covers
bare / c/ / cw/ extraction and that id: URLs stay non-vanity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recurring "post shows a zip but no images" report had no diagnostic: when
_import_archive captured an archive as a bare PostAttachment — because the
bomb-guard probe rejected it, or extraction yielded zero members (corrupt /
unsupported / missing extractor backend), or it held only non-media files — it
returned status="attached" silently.
Now those paths set ImportResult.error with the specific reason and log a
warning, and download_service records each as {file, reason} under the event's
metadata.unextracted_archives (None when every archive extracted cleanly). So
the next run names exactly which archives failed and why, instead of leaving the
operator to guess. No behaviour change to the happy path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon's /api/campaigns?filter[vanity]= lookup returns empty data for creators
that plainly exist (operator-flagged 2026-06-06 — Atole etc. erroring at the
resolve step). gallery-dl never used that endpoint; it pulls the campaign id out
of the creator page's bootstrap JSON. Add the same as a fallback: when the API
misses, GET the creator page (bare + /c/ vanity paths) and scrape the first
campaign id from any known embedding ("id":"…","type":"campaign" /
"campaign":{"data":{"id" / /api/campaigns/<id> / "campaign_id"). API is still
tried first (cheap, structured); the page scrape only runs on a miss.
Tests: API-empty → page-scrape fallback resolves; _scrape_campaign_id pattern
coverage. Existing API-path tests unchanged (happy paths short-circuit before
the fallback; failure paths hit the guarded scrape and still return None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Naming/lookup failures now report the source_url, the extracted vanity, and the
exact campaigns-API lookup URL attempted, so a "could not resolve campaign id"
error is diagnosable (wrong vanity? cookie/auth? creator renamed?) instead of
opaque. Applied to all three resolution surfaces: the native download event,
the dry-run preview, and the credential-verify probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name now only capitalizes the first letter of each word and
leaves the rest of the word untouched (was lowercasing the tail, which turned
DC→Dc / NSFW→Nsfw). This matches ml/tag_name._title_word, so a Camie-suggested
tag keeps the exact casing the suggestion UI showed when it round-trips through
POST /api/tags on Accept — addressing "auto-suggested tags must obey
capitalization" and "don't mangle acronyms" in one rule.
Trade-off (operator-chosen): all-caps input no longer folds to Title Case, so
case-variant merging in #714 still folds the dominant lowercase-vs-Title case
but leaves all-caps stylizations distinct (protecting acronyms wins). Tests
updated + a new test documenting acronym preservation / non-folding.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hero collage's thumbnail rail hard-capped at 3 fixed-80px cells, left-
aligned, so it never reached the edge of the (50%-width) hero. Make the rail a
CSS grid of equal columns (1fr) at a fixed height that stretches to the hero's
full width: show up to 5 thumbnails, and when a post has more images than fit,
the last cell becomes the "+N" overflow tile (count unchanged). Column count is
driven by --fc-rail-cols so the strip always reaches the hero edge regardless
of image count.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps
whatever casing it was created with. This adds a maintenance action that
Title-Cases every existing tag (collapsing whitespace) and merges
case/whitespace-variant duplicates into one.
Backend:
- tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by
(kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor
(prefer an already-canonical member → no rename/self-alias; else the
best-connected tag → fewest FK repoints; else lowest id), merges the variants
INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/
aliases/series_page repoints + protective ML aliases), then renames the
survivor to canonical. Losers are deleted before the rename so there's no
transient unique-index clash; commits per group and isolates failures per
group. Idempotent — an already-canonical lone tag is a no-op.
- normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async
engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i.
- POST /api/admin/tags/normalize: dry_run=true returns a projection inline
(group/collision/rename counts + sample); dry_run=false enqueues the task.
Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup
tab) — preview → apply (polls the activity dashboard to terminal status),
behind a back-up-first warning. admin store gains normalizeTags().
Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag
dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept
separate, ML-known loser keeps a protective alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in
the POST /api/tags handler so operator-entered tags get clean display casing.
It is NOT applied in the shared find_or_create / rename paths — those are used
by the ML tagger and allowlist matching, which must preserve the booru
vocabulary's original casing (Title-Casing it broke apply_allowlist matching).
find_or_create / rename keep case-insensitive lookup + clash detection so a
differently-cased entry dedups onto the existing tag instead of forking.
Tests updated to expect Title-Cased create output (sunset→Sunset,
character:Saber→Character:saber, http://example.com→Http://example.com) and a
dedicated normalize_tag_name unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TABLESAMPLE SYSTEM_ROWS reads CONTIGUOUS rows from each sampled page, so
sequentially-imported near-duplicates (multi-image posts, variant sets) came
back adjacent and clustered in the showcase ("three near-identical in a row").
Sample limit*5 rows (spanning more pages) then ORDER BY random() before taking
limit — breaks the physical adjacency for much better spread, still cheap
(random() over a few hundred rows, not the whole table).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).
normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.
Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#712 — fandom picker showed no existing fandoms: loadFandoms enumerated via
/tags/autocomplete with q=' ', which the backend strips to empty → returns [].
Switch to the cursor-paged /tags/directory?kind=fandom (loop all pages); drop the
load-once guard so the dialog reflects fandoms created elsewhere.
#711 — modal tag-chip kebab never opened: the kebab + menu were nested INSIDE the
v-chip, which swallowed the click / mis-anchored the teleported menu. Un-nest it
as a sibling v-btn using the standard v-menu activator slot (Vuetify wires the
click and stacks the overlay above the modal natively). Removes the openTagId
workaround.
#700 — ESC didn't close the modal after accepting a suggested tag: the guard
suppressed close while ANY .v-overlay--active existed, which includes tooltips —
a lingering tooltip blocked the close. Exclude .v-tooltip from the guard so only
real interactive overlays (menus/dialogs) keep ESC from closing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The integration test caught it: _import_media re-derives the artist by path-walk
from the attribution path (ignoring the explicit artist) AND _copy_to_library
lands members relative to that path. Staging the archive in /tmp meant the
artist didn't resolve (provenance skipped) and members would land in the temp
dir. Stage under images_root/<slug>/<platform>/<post>/ instead so the artist
resolves and members land in the real library; remove only the staged archive +
sidecar afterward (members stay). Require a real artist+slug (skip + count
otherwise).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.
Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.
Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").
Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.
Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two operator-reported Artists-view bugs.
Layout: ArtistCard previews used aspect-ratio: 3/1 + max-height: 220px — when a
lone grid column got wide (>~660px), the max-height made the aspect-ratio box
shrink its own WIDTH to keep the ratio, leaving dead space beside the 3
thumbnails. Dropped max-height (kept the min-height floor) so the strip fills the
card width; lowered the grid min 440→360px so a lone column stays <732px (strip
≲244px) and desktop gets 2+ columns.
Flicker: isEmpty checked !loading, but on the first render loading is still false
(the initial loadMore fires in onMounted, after paint) so "No artists match"
flashed for a frame before the spinner/data. Added a reactive `loaded` flag (true
after the first load attempt, reset on reset()); isEmpty now also requires it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the walk lets an operator gauge "is this source worth a backfill?" before
arming one. ingest_core.Ingester.preview walks the first few feed pages and
counts media NOT already in the seen/dead ledgers, downloading nothing
(read-only). download_backends.preview_source resolves the campaign id + runs it
(native-only, mirrors verify_source_credential / run_download); POST
/api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]}
(409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient
gains post_meta(post) for the sample's title/date.
UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard
opens PreviewDialog — self-fetches with loading / error / empty / result states
and a "Start backfill" shortcut. Store action previewSource.
Tests: preview counts new media without downloading + samples only posts with
new items; page_limit caps the walk + flags has_more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the walk lets Stop interrupt a live backfill chunk instead of letting it
run to its ~14.5-min time-box. ingest_core.run now polls _backfill_state at each
page boundary (a short SELECT, never held across the walk) and bails with PARTIAL
when an operator Stop has popped it. Latched on the first observed "running"
state so a run invoked without one (unit test / stale call) never self-cancels.
Progress is already checkpointed per-page, so a restart resumes from the cursor;
Stop clears it for a clean reset. No UI change — the existing Stop button now
just takes effect immediately.
Tests: _still_running reads the state; a latched run bails PARTIAL at the next
boundary when the state disappears (only the pre-cancel post ran).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_one_failure_isolated's _BoomSession overrode get() without the headers
kwarg _fetch_to_file now passes (B5 Range resume), so the call TypeError'd and
both items errored. Add headers=None to match the base fake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the media fetch means a mid-download transport cut no longer refetches
from zero. _fetch_to_file now resumes: on a transient retry, if bytes already
landed in the .part, it requests Range: bytes=<have>- and appends on a 206;
falls back to a clean truncate-and-restart if the server ignores Range (200) or
the range is past EOF (416). The .part staging means a non-range server never
corrupts the output — worst case is the old behavior (refetch from zero).
Tests: mid-stream cut resumes from the offset (asserts the Range header);
a Range-ignoring server refetches clean (no double-write). Test session fakes
updated to accept the new headers kwarg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owning the native client means we see the 429 Retry-After header — previously
discarded. PatreonAPIError now carries `retry_after`; on a PERSISTENT page-fetch
429 the client attaches the server's raw Retry-After seconds. New
DownloadResult.retry_after_seconds; patreon_ingester._failure_result sets it on
RATE_LIMITED. download_service._update_source_health passes it to
set_platform_cooldown as `seconds=`, clamped to [60, 3600] (a tiny hint can't
leave the platform effectively un-cooled; a huge one can't strand it for hours);
no hint → the flat 900s default. So a rate-limited platform cools for as long as
the server actually asks, not a fixed guess.
Tests: terminal 429 surfaces retry_after (test_patreon_client); cooldown honors
+ clamps the hint, falls back to default when absent (test_download_service).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror verify_source_credential: download_backends.run_download is now the single
download entry, so that module is the ONE registry of how each platform both
downloads AND verifies — the seam that makes adding a platform a bounded job
(write its adapter construction next to its verify). The native-ingester
construction + campaign-id resolution moves out of download_service into
download_backends._run_native_ingester.
download_service.download_source drops its `if uses_native_ingester ... else
gdl.download` branch and calls one `self._run_download(...)` (a thin delegate to
run_download passing the service's gdl + sync sessionmaker). mode (tick/backfill/
recovery) is still chosen there from the backfill state machine and threaded
through. Removed the now-unused PatreonIngester / resolve_campaign_id_for_source
imports from download_service.
Tests: the phase-2 stub seam moves from svc._run_patreon_ingester to
svc._run_download (helper + the db-release test); the two native-construction
tests repoint to download_backends.run_download (patching
download_backends.resolve_campaign_id_for_source / PatreonIngester).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates duplication that owning the native ingester left against the still-
live gallery-dl path, and fixes a parity gap the duplication hid.
A1 — shared quarantine: extract file_validator.quarantine_file (move to
_quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar).
gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both
call it. PARITY FIX: the native path now writes the provenance sidecar it
previously skipped — threads the media url through for source_url.
A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats
key set; gallery_dl._compute_run_stats and ingest_core both build through it so
the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0).
A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for
the Path call sites + memory pointer) and patreon_client both use it. Closes the
double-impl of the URL-encoded-basename ext gotcha.
A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level
truncate_log/extract_errors_warnings; download_service calls them directly
instead of reaching through self.gdl for native-result log shaping. The
staticmethods stay as thin delegators for existing callers/tests.
Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine
writes a .quarantine.json (test_patreon_downloader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.
#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).
#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.
Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Factor the native-ingest orchestration out of PatreonIngester into a reusable
ingest_core.Ingester base, parametrized by client/downloader/ledger-models/
constraints/key/platform/error_base. PatreonIngester becomes a thin adapter:
it resolves the Patreon client/downloader, wires the seen/dead-letter models +
UNIQUE-constraint names + _ledger_key into super().__init__, and overrides
_failure_result with the Patreon exception taxonomy. Behavior-preserving — no
table rename, no migration; the public surface (PatreonIngester, _ledger_key,
DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged.
This is the strategic seam: SubscribeStar/etc. now migrate by writing a
~40-line adapter, not by re-implementing the tick/backfill/recovery walk,
tiered skip, checkpoint, and dead-letter logic.
run() moved to ingest_core, so the budget test's monotonic patch repoints to
ingest_core.time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Honest completion of roadmap #8. Previously a NON-429 media failure (a
connection reset, a timeout, a truncated stream, a 5xx) was an immediate
terminal "error" for the pass — only retried on the NEXT walk. Now
_fetch_to_file retries TRANSIENT failures in-place with backoff (transport
blips incl. mid-download, 429 honoring Retry-After, and 5xx; up to 3 tries),
while PERMANENT failures (404 gone / 403 forbidden) fail fast straight to the
error → dead-letter path — re-fetching them is pointless. This makes the
transient-vs-permanent split explicit instead of leaning on the next-tick
cycle. (#1's 429 backoff + #7's dead-letter covered most of #8's value; this
is the missing in-pass transient piece I'd loosely marked "folded".)
Tests: a connection blip / a 5xx is retried then succeeds; a 404 errors with
NO retry; an exhausted transient becomes a terminal error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux,
persistently-corrupt bytes) used to re-error forever and re-burn chunks.
New `patreon_failed_media` table (alembic 0038, chains 0037) records
per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the
ingester skips it on routine tick/backfill walks (tier-1.5, folded into the
seen/skip predicate). Recovery BYPASSES it (the operator's "try everything
again" re-attempts dead media). A clean download clears the row (recovered);
errors/quarantines upsert-increment it. Surfaced as
run_stats.dead_lettered_count.
- New PatreonFailedMedia model + migration; ingester _dead_keys /
_record_failures (on_conflict increment) / _clear_failures.
- skip = seen | dead (empty in recovery); failures recorded post-fetch on
short sessions (same pattern as the seen-ledger).
Tests: a media erroring 3× is dead-lettered + skipped (no download attempt);
recovery re-attempts a dead media and clears it on success; a clean download
clears a sub-threshold failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A worker SIGKILL (hard-time-limit backstop) mid-chunk lost the whole
chunk's walk — the cursor was only persisted at chunk boundaries by phase
3, so the next tick re-walked from the chunk start. Now the ingester
checkpoints _backfill_cursor to the DB at each page boundary (backfill/
recovery only) via an ATOMIC single-key UPDATE (config_overrides::jsonb →
jsonb_set('{_backfill_cursor}') → ::json), so it never clobbers operator
config or other backfill keys. On a crash the last mid-walk cursor
survives → the next chunk resumes near the crash, not the chunk start.
phase 3 still writes the final cursor (same value); this is the safety net.
Tests: a backfill walk leaves the last page's cursor in the DB (written by
the ingester, before any phase 3); a tick never checkpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two test breaks from the structured-results change:
- An existing downloader test pinned a corrupt file to status "error";
it's now the distinct "quarantined" status (the new behavior). Updated
it + removed the duplicate I'd added.
- The budget-cut ingester test asserted the checkpoint cursor was the last
FULLY-processed page (CUR1); it's actually the page we were cut on (CUR2,
entered + cursor emitted before the budget check), matching the prior
parse_last_cursor(last) semantics. Corrected the assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The running badge only showed the chunk counter; now it shows posts walked
— real walk progress. The ingester already reports posts_processed per
chunk (step 1); the backfill lifecycle accumulates it into
config_overrides._backfill_posts across chunks. SourceRecord exposes
backfill_posts; start_backfill/start_recovery clear it (fresh walk); stop
clears it too. SourceRow/SourceCard badge renders "Recovering · 45 posts"
(falls back to "(N)" chunks before any posts are counted).
Per-chunk accumulation (no mid-walk DB write) — simple and race-free; a
small over-count from each chunk re-walking its resumed page is fine for a
progress indicator.
Tests: lifecycle accumulates posts_processed across chunks; start clears a
prior _backfill_posts and the record exposes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native ingester faked gallery-dl stdout (`Cursor:` lines, summary) and
phase 3 regex-scraped it back — so Patreon run-stats were mostly zero and
quarantine stats blank. We own the ingester, so it now RETURNS structured
data and phase 3 reads it directly.
- DownloadResult gains run_stats/cursor/posts_processed (None/0 on the
gallery-dl path, which keeps the text route).
- Ingester builds real run_stats from per-media outcome counts, sets the
checkpoint cursor structurally (no fake `Cursor:` stdout), and counts
posts processed. download_service phase 3 uses dl_result.run_stats when
present; the backfill lifecycle + TIMEOUT→PARTIAL block checkpoint
dl_result.cursor instead of parse_last_cursor(stdout).
- #4 quarantine: PatreonDownloader reports a distinct "quarantined"
MediaOutcome (with the _quarantine dest); the ingester surfaces a real
files_quarantined + quarantined_paths + run_stats.quarantined_count
(was hardcoded 0). Quarantined media isn't written or marked seen.
- Cleanup: parse_last_cursor + _CURSOR_RE (and the now-unused `import re`)
removed from gallery_dl — the structured cursor replaced the scrape.
Tests: ingester result carries real run_stats/cursor/posts_processed +
quarantine counts; downloader quarantines an invalid file as "quarantined";
backfill cursor tests pass cursor= structurally; dropped the
parse_last_cursor tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Before arming a deep walk on a native-ingester platform (Patreon — where
verify is one cheap API page), POST /sources/{id}/backfill {start|recover}
runs the shared verify_source_credential first and REFUSES (409 + reason)
only on a definitive rejection (verify→False, e.g. expired cookies). It
proceeds on valid (True) or inconclusive (None — a network blip must not
block). Gated to native platforms: gallery-dl verify is a slow --simulate
subprocess, too heavy for an arm action. The credential read happens in a
session that's CLOSED before the verify network call (no held conn).
Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the
reason in .body, not .detail) so the rejection text surfaces in the toast.
Tests: arm blocked on rejection (409, source not armed), proceeds on
inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight.
An autouse fixture stubs verify to 'valid' for the existing backfill
endpoint tests so they stay network-free.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_run_patreon_ingester reads self.gdl._rate_limit for the native pacing
config (max(0.5, rate_limit/4)); the MagicMock fake gdl broke the
arithmetic. Give it real _rate_limit/_validate_files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:
- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
`rate_limit`, read off self.gdl) as a pre-download sleep on real media
downloads only (skips don't pace); pace /api/posts page fetches with the
per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
RATE_LIMITED. Light 2-retry on a media-GET 429 too.
Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).
Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The credential Verify button still ran gallery-dl --simulate for Patreon
after the cutover — testing the wrong path (and prone to the vanity
"Failed to extract campaign ID" the native resolver fixes). Wire it to the
native ingester, behind a DRY dispatch so callers never branch on platform.
- services/download_backends.py (new): the ONE place that knows which
platforms are native vs gallery-dl. `uses_native_ingester(platform)` is
the shared predicate; `verify_source_credential(...)` is the uniform
probe (same (ok|None, message) contract for both backends). As a platform
migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download
routing and verify switch together.
- PatreonClient.verify_auth(campaign_id): one authenticated /api/posts
fetch → True (valid) / False (401/403/HTML-login) / None (drift or
network — inconclusive, not a credential verdict).
- patreon_ingester.verify_patreon_credential(): resolve campaign id, then
verify_auth — the verify counterpart to the download path.
- patreon_resolver.resolve_campaign_id_for_source(): extracted the
override / id:-URL / vanity resolution into ONE helper now shared by the
download ingester and verify (download_service no longer carries its own
copy + regex; −`import re`).
- download_service: routes on uses_native_ingester() instead of inline
`== "patreon"` (3 sites); uses the shared resolver.
- api/credentials: calls verify_source_credential — no platform branch.
Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/
vanity/none), the dispatch predicate, verify_patreon_credential glue,
credentials endpoint proves Patreon uses the native path (gallery-dl verify
asserted not-called); repointed the gallery-dl verify test to subscribestar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final step of the native Patreon ingester: a first-class Recovery action,
and removal of the now-dead gallery-dl Patreon path.
Recovery (rules #23/#24/#27 — full product, with UI):
- source_service.start_recovery arms the #693 backfill state machine PLUS
`_backfill_bypass_seen`, flipping download mode to recovery (bypass the
seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate
under the current pHash threshold). Stop via the shared stop_backfill.
- SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill
gains action="recover".
- Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow
+ SourceCard; the running badge labels "Recovering (N)" vs "Backfilling
(N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource +
SubscriptionsTab onRecover.
Cutover (rule #22 — no legacy):
- gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon
files/cursor branch in _build_config_for_source, and the patreon/Mux
yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the
other platforms' yt-dlp fetches; native ingester owns it now).
- download_service: removed the dead campaign-id-retry helpers
(_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and
_effective_url. Vanity→campaign resolution + resume_cursor + the
cursor/PARTIAL lifecycle stay — they serve the native ingester.
Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp
Referer, resume-cursor); repointed the generic skip-value config tests to
subscribestar; added start_recovery + recover-endpoint coverage
(backfill_bypass_seen). gallery-dl stays for the other 5 platforms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.
Three modes wired from config_overrides state:
- tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
contiguous already-have-it items.
- backfill: full-history time-boxed chunk, cursor checkpoint via
gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
lifecycle + parse_last_cursor verbatim).
- recovery: backfill that BYPASSES the tier-1 seen-ledger so
dropped-and-deleted near-dups get re-fetched and re-evaluated under
the current pHash threshold. Rides the #693 state machine via a
_backfill_bypass_seen flag, cleared on completion / stop.
The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.
Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PatreonDownloader.download_post: writes resolved MediaItems to gallery-dl's
exact on-disk layout (<slug>/patreon/<date>_<id>_<title40>/<NN>_<file>) +
a sidecar the importer's find_sidecar/parse_sidecar consume unchanged. Two-tier
skip (injected seen predicate, then disk). Streamed GET (.part→rename) +
file_validator quarantine; Mux/m3u8 video shells out to yt-dlp with Patreon
Referer/Origin. Pure (no DB) — ledger + orchestration land in step 3. Unit
tests stub the session + yt-dlp seams (no network/subprocess).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id,
filehash) — our own queryable replacement for gallery-dl's archive.sqlite3.
Routine walks skip seen media; recovery mode bypasses the ledger. filehash is
a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic
0037 (← 0036). Integration test covers dedup + savepoint recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fixture gives the attachment and post_file the same filehash, so they
correctly collapse to one item; the test asserted both survival and collapse.
Rewrite to verify the cross-kind dedup (postfile kind covered by video test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PatreonClient: cookie-auth requests session, /api/posts cursor pagination,
JSON:API included flattening, per-post media extraction (images/image_large/
attachments/postfile/content) with filehash dedup, loud drift detection.
Zero per-file HEADs — every media URL+file_name comes from the API. Not yet
wired into download_service (later step). Pure-parsing unit tests + fixture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan #693 (frontend). Backend landed in 96fffaf.
- sources store: setBackfill(runs) → startBackfill/stopBackfill ({action}).
- SubscriptionsTab: the deep-scan window.prompt for N runs becomes a
Start/Stop toggle keyed on source.backfill_state.
- SourceRow + SourceCard: the 'backfill (N×)' chip becomes a state badge —
Backfilling (chunk N) / Backfilled / Stalled — and the scan button
toggles between Backfill (mdi-magnify-scan) and Stop (mdi-stop).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).
- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
far under the 1350 soft limit. Hitting it = normal chunk boundary (the
TimeoutExpired path already captures partial output + the cursor), not a
near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
(running/complete/stalled). A running backfill auto-continues in chunks
across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
pause a pathological walk as 'stalled'. Replaces the N-runs counter
(backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
source dict exposes backfill_state + backfill_chunks.
Frontend (Start/Stop control + state badge) lands in the next push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-06-05 (mobile):
- TagAutocomplete no longer autofocuses on the ≤900px stacked layout —
focusing popped the soft keyboard, shrinking the viewport and shoving the
pinned image + nav/close controls out of view. matchMedia gate (safe on
plain HTTP); desktop autofocus unchanged.
- ImageViewer stacked layout: the body now scrolls and the media pane is
sticky (height:55vh, top:0), so the image + prev/next/close + integrity
badge stay pinned while the metadata panel scrolls beneath. Replaces the
old fixed image + 40vh internally-scrolled panel that could push the
controls off. Prev/next re-centered over the image band; opaque obsidian
bg so the scrolling panel doesn't bleed through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Large Patreon creators (Anduo: weekly 50-120-image Reports back months =
thousands of files) couldn't backfill: each run re-walked newest→oldest
from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone
exceeded the 1170s subprocess budget, so the run died during enumeration
with 0 files written and NO forward progress — re-stranding every time
(event #40411).
Checkpoint gallery-dl's pagination cursor so each backfill window advances
the frontier:
- gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets
extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True
for a fresh run); parse_last_cursor() pulls the last emitted
'Cursor: <token>' from stdout+stderr — survives a timed-out run since the
TimeoutExpired path returns partial output.
- download_service.py: phase2 stays in BACKFILL mode while a cursor is
pending (even after the run budget drains) and threads resume_cursor;
_apply_backfill_lifecycle() checkpoints the advancing cursor each
non-completing run, completes on a clean rc=0 finish (walk reached
bottom), and a stuck-guard clears the cursor after 2 non-advancing runs
so a wedged walk can't re-strand forever.
patreon-only (sole platform with a resumable cursor); other platforms keep
the simple counter semantics. Cursor state lives in config_overrides JSON
(patreon_campaign_id precedent) — no migration. Time-budget ladder
(1170/1350/1500) unchanged.
Plan #689.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-confirmed on a fresh build: both the tag-chip and suggestion
kebabs still never opened. The prior 8326e54 'fix' only wrapped them in a
<span @click.stop> — inert for SuggestionItem (no parent capture) — and
never addressed why the `#activator`/`v-bind="props"` click failed to
toggle the menu inside the teleported ImageViewer modal. The dialogs in
that same modal open via v-model and work, so drive the menus the same way:
- The activator (v-btn / v-icon) toggles a reactive flag with @click.stop
(which also shields the chip's close button / any parent).
- The v-menu binds that flag (v-model / :model-value) and uses
activator="parent" with :open-on-click="false" purely for positioning,
so opening no longer depends on Vuetify's activator-click path.
- TagPanel tracks a single openTagId (one chip menu open at a time).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Not all characters belong to a fandom (original characters, unsorted).
The create flow forced every new character through FandomPicker, whose
only outcomes were 'Use this fandom' (disabled until one is picked) or
Cancel (which aborts the whole creation) — there was no way to confirm a
character with no fandom.
- FandomPicker: add a 'No fandom' action that emits confirm(null).
- TagAutocomplete.onFandomChosen: pass fandom_id: null when null is
emitted.
Backend already supported this end to end (Tag.fandom_id nullable, the
CHECK only forbids fandom_id on non-character kinds, tag_service
find_or_create defaults fandom_id=None, API reads body.get). A fandom can
still be assigned later from the chip kebab's 'Set fandom…'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phash_threshold knob (controls whether edits/variants of an image are
dropped as near-duplicates on import) was buried at the bottom of the import
filters form and labelled opaquely, so it read as 'missing'. Hoist it to the
TOP of the form as a 'Near-duplicate sensitivity' section: a labelled slider
(Exact / Strict / Default / Loose stops, 0-16) for the gist + the precise
number field, both bound to phash_threshold, with copy that says plainly to
lower it if variants are being dropped.
Also swap the import-tab order to filters → trigger → recent-tasks (filters on
top per operator); the task list stays directly under the trigger for hit/miss
feedback adjacency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Post cards no longer expand the whole card on click (the old two-click path
to the images). The card is compact-only now:
- Hero / rail thumbs / the +N tile are buttons that open the post-scoped image
modal (modal.open(id, { postImageIds })) so you view big + arrow through ALL
the post's images. The feed caps thumbnails at 6, so for posts with more we
lazily getPostFull to get the complete id list; +N opens at the first hidden
image.
- The description is the ONLY in-place expansion: a Show more / Show less toggle
shown only when the text is actually truncated (server description_truncated
flag OR a measured CSS-clamp overflow, ResizeObserver-guarded). Expanding
loads description_full when server-truncated and renders it unclamped.
- Attachments: download chips now render inline in the compact card (the feed
already carries download_url), since the expanded view is gone.
Removes PostImageGrid.vue (the mosaic, now unused). Tests cover show-more
visibility + image-click opening the scoped modal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transparency / single-color audit cards held the run + poll timer in
local component state, so navigating away destroyed both and onMounted never
reconnected — the Celery scan kept running and writing LibraryAuditRun, but
the UI forgot it. Now each card, on mount, fetches its rule's latest run
(GET /api/cleanup/audit?rule=<rule>&limit=1) and rehydrates: shows progress +
resumes polling if still running, or shows the completed result (ready/applied/
error) so the operator can act on it after returning. Adds the ?rule= filter
to the audit-history endpoint + cleanup store latestAuditForRule().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'See all similar' takeover fetched /api/gallery/similar fine (200, ~100
results) but the grid showed nothing: GalleryGrid renders ONLY by iterating
store.dateGroups, and similar-mode returns date_groups=[] (results are ranked
by cosine distance, not chronological). Zero groups → zero tiles despite
store.images being full. Add a flat fallback: when there are no date groups
but images exist, render them as one ungrouped list in ranked order (no date
headers). The modal Related strip was unaffected (it renders its images
directly). Test locks both the flat and grouped paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vuetify's auto card-stack was too verbose (one subscription filled the whole
phone screen) and the expanded sources still needed lateral scroll. Replace it
below 600px (useDisplay) with a custom compact-card list: each subscription is
a 2-line card (name + health + expand chevron, then platform chips + sources
count + last activity) so several fit per screen. Expanding shows the action
row + each source as a STACKED SourceCard (new) — platform/url/enabled/last/
next/errors/actions laid out vertically, no horizontal scroll. The mobile
cards drive the same selected/expanded key arrays as the desktop data-table,
so selection and bulk actions are unchanged. Desktop keeps the v-data-table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The subscriptions v-data-table (select + expand + 6 cols + a nested 8-col
sources sub-table) horizontally-scrolled on phones. Set mobile-breakpoint=600
so Vuetify stacks each subscription row into a label:value card below 600px;
the custom item slots (platform chips, health dot, action buttons) render as
card rows. The expanded sources detail reclaims its desktop indent on mobile
and keeps its own horizontal scroll for the wide source columns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PostsFilterBar didn't wrap and its artist/platform fields had inline
min-widths (240/180px) a media query can't override → horizontal overflow on
phones. Moved widths to classes, added flex-wrap, and <600px each field takes
a full-width row. SubscriptionsTab's status/search inline max-widths likewise
moved to classes; <600px they go full-width and the v-spacer is dropped so the
search isn't shoved around.
Verified as already-fine (sweep false positives, no change): PostCard (default
body is a stacked column; only goes row at container >=800px), SeriesReaderView
(already has a <=768px block: nav drawer 150px, quick-nav stacks). The
subscriptions v-data-table scrolls horizontally within its own wrapper, so it
doesn't widen the page — a true mobile card layout is a larger follow-up if
wanted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The top nav packed brand + health + pipeline chip + ~7 inline links + an
action slot into one flex row, colliding/overflowing on phones (operator:
'almost unusable'). Below 768px the links now fold into a hamburger v-menu;
below 480px the brand text hides (glyph still brands). Plus the primary
browsing path:
- BulkEditorPanel: fixed 320px -> min(320px, 90vw) so it can't swallow the screen.
- GalleryFilterBar: <600px gives search its own full-width row (its 200px
min-width was jamming the wrapping bar); sort grows.
- GalleryFacetPanel: <480px wraps groups + lets the side-by-side date inputs
grow full-width.
- ArtistsView grid: minmax(min(440px,100%),1fr) so a card never overflows
(single column on phones).
- GalleryView: hide the year/month timeline strip <600px.
ImageViewer already stacks its side panel below the image <900px (left as-is).
Secondary surfaces (Posts/Subscriptions filter bars, SubscriptionsTab table,
SeriesReader, PostCard) still need a mobile pass — follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Showcase reveal cadence 80ms -> 160ms (slower, more deliberate one-at-a-time
cascade per operator). Bump showcase.spec timer advances to cover 60x160ms.
- Gallery filter bar now uses the EXACT gradiated-obsidian frost + blur as
TopNav (was a flat rgba(...,0.55) block), so the two read as one continuous
piece of chrome with images visibly scrolling under both; the nav's
transparent bottom edge against the bar's opaque top leaves a faint seam that
separates them at the very top of scroll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Modal 'Related' strip (RelatedStrip.vue) — top-12 similar thumbs, fetched on
its own DEFERRED, single-flighted path (200ms after the modal is up) so it
never blocks or slows the modal; collapses silently on empty/slow/error and is
hidden when the image has no embedding (has_embedding flag). 'See all similar'
closes the modal and navigates the gallery to ?similar_to=<id>.
Gallery store: similar_to filter field + loadSimilar() (ranked, hasMore=false,
no timeline); applyFilterFromQuery routes similar-mode to /similar with the
scope filters composed; cloneFilter/filterToQuery carry similar_to. Filter bar:
clearable 'Similar to #id' chip, sort hidden in similar-mode; timeline sidebar
hidden too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GalleryService.similar() ranks images by pgvector cosine distance to a source
image's precomputed SigLIP embedding — no query-time ML inference. Composes
with the Phase-1/2 scope filters (AND) but replaces the date sort (always
nearest-first, bounded top-N, no cursor). Returns None for a missing source
(→404), [] for a source with no embedding (video / pending ML); excludes self
and NULL-embedding rows. New GET /api/gallery/similar?similar_to=<id>&limit=N.
Image-detail payload gains has_embedding so the UI can hide the surface.
Alembic 0036 adds an HNSW vector_cosine_ops index on siglip_embedding (1152<2000
dims) so the search is sub-50ms ANN instead of a full scan; one-time ~30-60s
build over existing embeddings on deploy. Shared _gallery_images/_image_json
helpers de-dup the scroll/similar builders.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With fsync-off the whole integration suite runs in ~45s (was ~13min across
shards), so the 3-way split only triplicated the ~2min fixed overhead
(container + install + migrate) and consumed 3 of 6 runner slots for no
wall-clock gain. Merge intapi/intimp/intcore into one `integration` job:
spin up once, install once, migrate once, run `pytest -m integration` over
the whole suite. Frees 2 runner slots (6 jobs -> 4) and drops ~140 lines of
near-duplicate YAML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option 1 (pooling the teardown connection) left teardowns at ~1.5-2s/test, so
the cost is the per-test TRUNCATE's commit forcing an fsync, not the connect
handshake. Each shard now ALTER SYSTEM SETs fsync/synchronous_commit/
full_page_writes off + pg_reload_conf() right after deps install, before
alembic — sighup/user-context GUCs apply with no restart. The DB is ephemeral
(rebuilt per run) so fsync-off is safe; the step is non-fatal so a perms
surprise can't red a shard. Speeds up every test's commit (setup inserts +
the teardown TRUNCATE), stacking on the pooled engine from the prior commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The autouse integration teardown created a fresh SQLAlchemy engine + Postgres
connection for EVERY test, then disposed it — --durations showed the 15
slowest ops in both long shards were all ~1.5-2s teardowns (the connect+SCRAM
handshake, not test logic). Hoist the truncate engine to a session-scoped,
pool_pre_ping'd fixture so the pooled connection is reused across teardowns;
the TRUNCATE+restore still runs per test, so isolation is unchanged. Lazy
create_engine means the no-DB unit job instantiates but never connects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The buffered cascade revealed tiles on an 80ms timer regardless of image
load, so the flip-in animation played on a gray placeholder and the thumbnail
popped in afterward. Worse, MasonryGrid ALSO applied a per-index
animation-delay (index×70ms) that compounded on top of the insert cadence,
so the cascade visibly dragged and desynced as it grew.
Now the producer preloads each queued thumbnail (decode pipelined ahead) and
the consumer awaits that decode before pushing the item — every tile animates
in fully loaded, strictly one at a time. Drop the compounding CSS stagger;
the store's one-item-at-a-time push is the sole pacer, so each tile animates
the instant it mounts. New utils/preloadImage.js (load+decode+timeout gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite facets() common/plat_scope as dict literals (C408). Open the refine
panel via a watch on hasRefineFilters rather than reading filter state at
bar-setup time — the parent applies the URL query in its onMounted, after the
bar child has set up, so the initial read was always the default (empty) state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'Refine ▾' toggle to the gallery filter bar that expands a full-width
GalleryFacetPanel below it, inside the same sticky hazey chrome. The panel
offers platform chips (with live counts + a 'No platform' unsourced bucket),
two count-badged curation-flag toggles (Untagged / No artist), and a from/to
date range bounded by the facet min/max.
Store gains the platform/untagged/no_artist/date_from/date_to filter params
(URL-mirrored, AND-composed) and a panel-gated, single-flighted loadFacets()
that fetches /api/gallery/facets scoped to the active filter. Shared
cloneFilter/filterToQuery helpers keep the bar and panel writing one URL
format. The panel auto-opens on deep-link when refine filters are present and
refetches counts (debounced) on every filter change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the composable gallery filter with platform / untagged / no_artist /
date_from / date_to, AND-composed with the existing tag/artist/media/sort
params and threaded through scroll, timeline, and jump_cursor.
Add GalleryService.facets() + GET /api/gallery/facets returning live counts
scoped to the current filter with per-group minus-self semantics: platform
counts (COUNT(DISTINCT image) incl. a null unsourced bucket), curation-flag
counts (untagged / no_artist), and effective_date min/max bounds. The
UNSOURCED_PLATFORM sentinel makes filesystem-imported content reachable via
the platform facet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Image viewer (#609):
- The next (▶) arrow was offset from the viewport edge (right:16px) so it
floated over the 320px metadata side panel. Offset it off a shared
--fc-side-w var so it sits at the image's right edge instead; full-width
again below 900px when the panel stacks under the image.
- Arrow nav was fully disabled whenever a text field was focused. Now it
yields to the caret ONLY when the field has text; an empty tag-entry field
still navigates ←/→. Extracted to utils/textEntry.js (arrowNavAllowed).
ESC behaviour unchanged (already closes the modal, overlay-aware).
Test: arrowNavAllowed — empty/non-text → navigate, text present → don't.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM
ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat
readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats.
- dbMaintenance store (loadStats / runVacuum) + test.
- Fix ruff I001: combine the two _sync_engine imports onto one line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.
- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
(VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
autocommit connection.
- GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.
Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Filter bar gets the same obsidian translucent + backdrop-blur as the
TopNav so the two read as one piece of chrome.
- margin-top:-8px cancels the v-container's pt-2 so the bar sits flush at
64px even at scroll 0 — fixes the gap/separation when scrolled to top.
- Inputs/toggles get a more-opaque backing so they stay legible on the haze.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cascade "burped" — chunks appeared unevenly — because the old pipeline
coupled display to fetch timing: it trickled each batch right after its
fetch and assumed the next round-trip would land inside the ~240ms trickle
window. When a fetch ran long (TABLESAMPLE hits random, sometimes-cold
blocks; RTT jitter) the animation starved, then a clump burst in.
Decouple the two:
- Producer (_fill) races ahead fetching batches into a buffer up to a
target depth, refilling when it dips below BUFFER_MIN.
- Consumer (_drain) reveals one item every CADENCE_MS regardless of when
fetches land; it only waits if the buffer genuinely starves.
A small PRIME buffer precedes the drain so it doesn't starve at the front;
the buffer (BUFFER_MIN×CADENCE runway) absorbs per-fetch jitter so images
appear at an even pace. Public store API (loadInitial/shuffle/fetchPage/
images/loading/hasMore/isEmpty) unchanged — ShowcaseView/MasonryGrid need
no change.
Test (fake timers): fire-order + dedup, one-item-per-cadence rate limit,
empty-library flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gallery now has in-view filtering, styled like the app's sticky v-tabs
chrome (pinned at top:64px under TopNav).
- GalleryFilterBar: combined tag+artist autocomplete (searches
/api/tags + /api/artists), closable filter chips (multi-tag AND),
media toggle (All/Images/Videos), Newest/Oldest sort, Clear. Writes all
state to the URL via router.push.
- gallery store: filter is now { tag_ids, artist_id, media_type, sort,
post_id }; applyFilterFromQuery makes the URL the single source of truth
(deep-linkable, back-button works); chip labels resolved by id or
pre-noted on pick. Replaces the standalone tag chip + setTag/PostFilter.
- GalleryView: renders the bar (hidden in post-detail), syncs route.query
→ store on mount + every query change.
Also untracks the transient .claude/scheduled_tasks.lock committed in
3f30327 and gitignores it.
Tests: store parses query → composable scroll params, post_id exclusivity,
newest-sort omitted, label pre-seed, single initial fetch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 backend for the gallery filter bar. Extends scroll/timeline/jump
from a single mutually-exclusive filter to a composable one:
- tag_ids: image must carry ALL of them (one correlated EXISTS per tag —
AND, no row multiplication), replacing the single-tag JOIN.
- artist_id composes with tags; media_type ('image'|'video') narrows by
mime; post_id stays the exclusive post-detail path.
- sort ('newest'|'oldest') flips the effective_date/id cursor comparison
and ordering; the cursor value is unchanged (direction comes from the
request). jump_cursor honors sort too.
- Shared _apply_scope helper applied across scroll/timeline/jump so the
timeline sidebar reflects the filtered set. API _parse_filters parses
tag_id (comma list), artist_id, media, sort.
Tests: multi-tag AND, media filter, sort reversal (service + API);
post_id-excludes-others; single tag_id back-compat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two gaps where a filter couldn't be removed:
- Gallery: a tag_id filter (from clicking a tag) had no indicator or clear
control — only post_id did (PostInfoHeader). Add an "Tag: <name> ✕" chip
that clears the filter by dropping tag_id from the URL. New lightweight
GET /api/tags/<id> resolves the name; the store fetches it on filter set.
- Tags view: the kind chip-group used mandatory="false" — a STRING ("false"
is truthy in JS), which made the group mandatory so the active kind chip
couldn't be deselected. Fixed to :mandatory="false" so the filter clears.
Tests: GET /tags/<id> shape + 404; gallery store resolves filterTagName.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the missing UI to change a character tag's fandom, in both places:
- FandomSetDialog (shared): pick an existing fandom, create a new one, or
clear it; on a name collision in the target fandom it surfaces a merge
confirmation and resolves via setFandom(merge:true). Reuses the tags
store's fandom cache.
- TagCard kebab gains "Set fandom…" for character tags (→ TagsView opens
the dialog, reloads on success).
- TagPanel chip kebab gains "Set fandom…" for character tags (→ reloads the
modal's tag list on success).
- tags store: setFandom(tagId, fandomId, {merge}) action + test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
No way existed to change which fandom a character tag belongs to after
creation — PATCH /tags/<id> only renamed.
- TagService.set_fandom(tag_id, fandom_id, merge=False): set / change /
clear (fandom_id=None) a character's fandom, with the same validation as
find_or_create. On a name collision in the target fandom it raises
TagMergeConflict (→ 409, same shape as rename); merge=True resolves it by
merging this tag INTO the existing character.
- Extract _do_merge(source, target) from merge() so set_fandom can perform
the deliberate CROSS-fandom merge the public merge() validation forbids.
- PATCH /tags/<id> now accepts optional fandom_id (+ merge flag) alongside
name, and returns fandom_id.
Tests: set/change/clear, non-character + bad-ref rejection, collision
raises, merge resolves; API set/clear + collision→merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gallery cursored on COALESCE(post.post_date, image_record.created_at)
across the Post outer join — an expression spanning two tables that no
index can serve, so every /scroll sorted a large slice of the library
(and the old frontend fired ten serially). Materialize it:
- image_record.effective_date column + ix_image_record_effective_date
(effective_date DESC, id DESC); alembic 0035 backfills
COALESCE(primary post's post_date, created_at) for existing rows.
- gallery_service._effective_date_col() now returns the column, so scroll
/ timeline / jump / neighbors all order off the index instead of
re-deriving the COALESCE. _neighbors reads record.effective_date
directly (drops an extra Post lookup).
- importer._apply_sidecar maintains it: when a primary post with a date is
linked, effective_date = post.post_date; plain inserts keep the
created_at-equivalent server default.
Tests: sidecar import asserts effective_date == post.post_date; gallery
ordering/timeline/jump test seeds set effective_date alongside created_at.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 5×10 metadata batching only staggered the cheap layer (JSON);
thumbnails load as independent <img> requests and clustered, so tiles
"popped in together" after a wait. Two changes:
- GalleryItem reveals each tile when ITS OWN thumbnail fires @load (with
an onMounted complete-check for cached thumbs), playing a showcase-style
flip-up entrance. Tiles now cascade in natural load order instead of all
at once. Honors prefers-reduced-motion.
- gallery store does ONE initial fetch (limit=50) instead of 10 serial
/scroll round-trips. Fewer RTTs, faster first paint; the reveal-on-load
is what makes appearance progressive now. Infinite scroll pulls 25/trigger.
Tests: GalleryItem gains is-loaded only after @load; loadInitial issues
exactly one scroll request at the initial limit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backfill events were STILL stranding empty after the timeout-ladder fix.
Worker logs showed the salvage path working ("Download timeout for
anduo/patreon after 1170.0s (18 files written)") but then:
Retry in 3s: DBAPIError(ConnectionDoesNotExistError: connection was
closed in the middle of operation)
...succeeded in 0.149s <- in-flight guard no-op
Root cause: DownloadService held the async + sync DB connections checked
out across the entire (≤19.5-min backfill) gallery-dl subprocess. The
server reaps the idle connection, so phase 3's first query hits a dead
socket. That DBAPIError trips download_source's autoretry_for, the retry
re-enters _phase1_setup, sees the event still 'running', returns
in_flight and no-ops — leaving the event to be stranded empty by the
recovery sweep. pool_pre_ping was already on both engines but can't help
a *held* connection (it only validates on pool checkout).
Fix:
- DownloadService.download_source closes the async + sync sessions after
phase 1, before the subprocess, so phase 3 re-acquires a live
connection (matches the class's "Phase 2 — no DB connection" docstring).
- The per-task async engine switches to NullPool so phase 3 always opens
a fresh connection rather than a pooled one the server may have reaped.
Tests: assert connections are released before gdl.download runs and the
event still finalizes; assert the task engine uses NullPool. Also fixes a
stale 1800s->1170s comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).
Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
subprocess budgets (870 tick / 1170 backfill) sit below the soft
limit with phase-3 persist headroom. Promote to module constants and
guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
in-flight event with a real reason, mirror phase-3 source-health, and
decrement backfill so a chronically-slow source self-heals to tick
mode. The existing celery_signals handler only covered TaskRun, not
DownloadEvent — that was the gap.
Updates stale 900/1200 references in gallery_dl.py + maintenance.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CI bounces on b65e956:
1. ruff UP017 — Python 3.14's preferred form is `datetime.UTC`, not
`timezone.utc`. Switch the test's two TZ literals.
2. test_directory_card_shape pinned the card key set to the pre-feature
shape; `unseen_count` was added to the API payload but the pin
wasn't updated. Same shape as the recurring 'plan-grep-pinned-tests'
trap — should have grepped tests/ for card.keys() before pushing.
Per-artist "+N" accent pill on the artists directory and a "N new since
last visit" banner inside ArtistView. Counts new IMAGES (not posts) so
multi-image posts increment correctly.
- alembic 0034: artist_visit (artist_id PK, last_viewed_at NOT NULL).
Seeds every existing artist with last_viewed_at=NOW() so the badge
starts at 0 across the board — no noisy "5000 unseen images" on
first deploy.
- ArtistService.find_or_create autoseeds a visit row alongside new
artists, so freshly imported content doesn't read as unseen.
- ArtistService.overview reads pre-visit last_viewed_at, counts images
created since, then atomically UPSERTs last_viewed_at=NOW() via
postgres ON CONFLICT DO UPDATE (no SELECT-then-INSERT race per
reference_scalar_one_or_none_duplicates). Returns the pre-update
count as `unseen_count_at_visit` so the banner has data.
- ArtistDirectoryService.list_artists adds an `unseen_count` aggregate
to each card via LEFT JOIN artist_visit + conditional COUNT. NULL
last_viewed_at (artist created before this code shipped) defensively
counts as "never visited" → all images unseen.
- Frontend: ArtistCard renders an accent pill in the preview-strip
corner when unseen_count > 0 (capped at 99+); ArtistView shows a
closable v-alert banner on initial load when
unseen_count_at_visit > 0, re-arms on slug change.
Single-row-per-artist (no user_id) — rule #47 multi-user ACL is
aspirational; widens to (user_id, artist_id) PK when User lands, per
rule #22.
Scribe plan #597.
Pre-upload verify request: after capturing the live browser cookies,
hit a known authenticated endpoint with credentials:'include' from the
extension's background context. If the platform reports we're not
logged in, abort the upload so we don't overwrite FC-side credentials
with stale data.
- platforms.js: add `verify` config per cookie-auth platform
- hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF
_init_site_filters; same 401 path the operator hit 2026-06-03)
- patreon: GET /api/current_user (clean 401 when logged out)
- subscribestar, deviantart: no stable auth endpoint, skip verify
- cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}.
ok=true/false/null tri-state — null = verify not configured, caller
treats as "proceed".
- background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the
upload; failures bubble up with the platform's name + reason.
- popup.js: success message now appends "(verified ✓)" when applicable.
- manifest + package.json: 1.0.6 → 1.0.7.
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`,
`#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were
surfacing raw in SuggestionsPanel — and worse, the SAME raw string was
written to tag.name on Accept, polluting the DB with `underscored_lowercase`
names that don't match the operator's "Title Case" tag convention.
Add backend/app/services/ml/tag_name.py with a single normalize()
applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing
_(disambiguator) blocks iteratively, strip wrapping quotes, underscores
to spaces, space after colon, title-case each word's first char,
preserve hyphens/apostrophes/digits, drop entries with no letters).
Wire into SuggestionService.for_image:
- raw Camie key kept for alias_map lookup (alias rows are hand-curated
against raw keys; don't disturb)
- display_name = normalize(raw); None means drop the candidate
- existing-tag lookup widened to case-insensitive match against BOTH
raw and normalized forms so legacy underscore-named Tag rows accepted
before this change still surface as "existing" not "+ new"
Operator confirmed they have no existing ?image=N bookmarks to
preserve, so the soft-compat read on initial mount + router.replace
strip is dead weight. The modal is now purely a Pinia overlay — no
URL involvement on open OR initial mount.
Drops 33 lines plus the now-unused vue-router imports in both
ArtistGalleryTab (entire onMounted gone) and GalleryView (just the
?image=N block; the post_id/tag_id filter handling stays).
Operator-flagged 2026-06-02. Both kebab activators were broken:
- TagPanel chip kebab had `@click.stop` directly on the v-icon
activator. In Vue 3, an explicit @click on the same element as
`v-bind="props"` overrides the spread onClick — so Vuetify's
activator handler never fired. Menu never opened.
- SuggestionItem kebab didn't have @click.stop, but for consistency
and to make both kebabs follow the same shape, wrap it too.
The fix: each kebab is now wrapped in a `<span @click.stop>`. The
v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu;
the bubbling click then reaches the span where stopPropagation
absorbs it before it can affect a parent (the v-chip's close button
in TagPanel's case).
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).
Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.
Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4).
G5.1 — Centroid version no longer drifts:
CentroidService now reads MLSettings.embedder_model_version (the DB
row tag_and_embed already writes from) for both the centroid model-
version stamp and the drift-detection comparison. Previously the
centroid sites imported MODEL_VERSION from env, so the version stamped
on centroids could disagree with the version stamped on the embeddings
they were built from. By construction those now match, so list_drifted
won't silently miss the env-vs-DB drift case.
embedder.py keeps MODEL_VERSION as an env-driven constant for the
actual model loader — that's a different concern (which weights are
loaded) from the version-stamp that gets persisted alongside data.
G5.4 — Modal is a Pinia-only overlay:
The previous URL↔modal sync in GalleryView and ArtistGalleryTab
leaked the modal across route changes (RouterLink to /artist/<slug>
left the modal mounted on top of the new route) and re-opened it
on history back/forward with stale ?image=N entries.
Now: openImage() just calls modal.open(id) — no URL push.
GalleryView's dead closeImage helper is deleted. A route.name
watcher in App.vue closes the modal whenever the route changes,
which auto-fixes RouterLink-in-modal and back/forward.
Backward-compat: ?image=N is still honored on initial mount as a
one-shot deep-link opener, then router.replace strips the query so
the URL doesn't re-trigger and no extra history entry is added.
Existing bookmarks / shared URLs keep working; new opens stay
Pinia-only.
CredentialCrypto's safety check fires on create_app() instantiation
because the test environment has no pre-seeded Fernet key file. Set
the bootstrap env var before any test imports so the auto-create path
is allowed during tests.
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.
FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
- warning (yellow) — operator action needed (auth_error)
- info (blue) — backend-paced (rate_limited / timeout /
network_error / partial / tier_limited)
- error (red) — likely terminal without intervention
(not_found / access_denied / validation_failed /
unsupported_url / http_error / unknown_error)
Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.
Two opt-ins now needed for auto-creation:
1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)
Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.
Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.
Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the
race under concurrent writers / recovery-sweep replays, poisoning the
outer transaction with an unrecoverable IntegrityError and crashing
the calling task. Same shape as the banked 2026-05-26 ImageProvenance
incident (see reference_scalar_one_or_none_duplicates memory).
Mirrors the importer._get_or_create pattern in all three: savepoint
via begin_nested + IntegrityError rollback to that savepoint + re-
SELECT to grab the row the other worker committed first.
- importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store
is sha-addressed so both workers race to write the same on-disk path
(shutil.copy2 + rename is idempotent), so no extra cleanup needed.
- TagService.find_or_create: partial uniqueness index on
(name, kind, COALESCE(fandom_id, -1)). The previous docstring
claimed "INSERT ... ON CONFLICT" but the implementation was
SELECT-then-INSERT with no recovery.
- ArtistService.find_or_create: already had IntegrityError handling
but did session.rollback() (unwinds the WHOLE transaction); now
uses begin_nested + sp.rollback() so the surrounding request's
progress isn't lost.
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.
- download_service: replace hardcoded ('discord','pixiv') tuple with
auth_type_for(platform) == 'token'. A 7th token-platform now picks
up the right credential path without touching this site.
- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
merge so the target's centroid reflects its new image set
immediately. Daily list_drifted catches it within 24h, but eager
recompute closes the suggestion-quality dip in the meantime.
- backfill_thumbnails added to beat_schedule (daily). The task
docstring claimed periodic Beat but the entry was never registered,
so the library got no self-healing thumbnail repair; only the
manual admin-UI button fired it.
- modal.createAndAdd pushes a kind='fandom' tag into
tagsStore.fandomCache so FandomPicker sees the new fandom on next
open. Was: cache-gated load (length===0) skipped refetch, new
fandom invisible until full page reload.
- cleanup cluster:
- Drop .webp from cleanup_service.unlink — thumbnailer only writes
.jpg/.png; the third tuple member was dead code.
- Drop effective_date from /api/gallery/scroll response — no FE
consumer reads it. Service still computes the attribute for
timeline ordering; this just trims the JSON.
- Rename store.recentMinute → store.recentRuns across the
systemActivity store + three consumers (SystemActivitySummary,
QueuesTable, SystemActivityTab). The data is the last 200 runs
(not actually "last minute"), so the name lied.
NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
Five extension-miss findings from the 2026-06-02 audit, where a status
value was added on one side but a downstream consumer didn't pick it up.
- download_service._phase3_persist: explicit branches for
ImportResult.status in ('failed','refreshed'). For 'failed' (archive
probe crash from _import_archive), unlink the source file so the
filesystem scanner doesn't re-import and re-crash on the same
archive forever. 'refreshed' is currently unreachable from the
download path (no deep=True) but matches the importer's documented
contract; treat as 'attached'.
- gallery-dl backfill auto-complete now gates on dl_result.success +
no error_type, not just return_code==0 + files_downloaded==0.
VALIDATION_FAILED exits the subprocess with returncode=0 and
files_downloaded=0 when every file was quarantined, matching the
prior predicate exactly and zeroing the operator's armed backfill
budget on the FIRST quarantine run instead of decrementing.
- attach_in_place archive dispatch now threads artist + source_row
through _import_archive (and _import_media for archive members)
and _supersede. The path-walk fallback (_resolve_artist) is still
used by filesystem-import; the download path now binds
ImageProvenance to the explicit subscription Source instead of
rediscovering by (artist_id, platform).
- Three FE handlers now recognize status:'deferred' from
/api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting
"event #undefined"), SubscriptionsTab.checkAll (was counting
deferred as queued), DownloadEventRow.onRetry (was saying
"re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll
which already had it.
- celery_signals._queue_for now maps backup/admin/library_audit
prefixes to 'maintenance' (matching task_routes). TaskRun.queue
was returning 'default' for those rows, so per-queue dashboard
filters and per-queue threshold overrides (added in G3) silently
missed them.
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
Extracts gallery.js's hand-rolled inflightId pattern into a new
useInflightToken composable; adopts in every store that previously
had no guard against late-response overwrites or wrong-image URL
interpolation.
Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62)
flagged:
- modal.removeTag rolled back the chip rail unconditionally even
when only the secondary dismiss POST had failed — UI lied until
refresh. And all tag-mutation URLs interpolated currentImageId
AFTER an await, so a fast prev/next could route DELETE/POST to
the wrong image. Both fixed: split try/catch (dismiss failure
surfaces a warning, doesn't roll back the delete); imageId
captured at call-time and used in URLs throughout.
- suggestions.accept dereferenced currentImageId after the awaited
POST /api/tags, so the subsequent /suggestions/accept could
apply A's chosen tag to image B AND push it to B's allowlist.
Fixed by capturing imageId at click-time + inflight guard on
load().
Same shape across artist / downloads / artistDirectory /
tagDirectory / posts stores: rapid filter/nav changes used to
interleave responses (last-writer-wins). Now the late response is
discarded and the most-recent request wins. Filter-change-during-
search no longer drops the second fetch because the loading flag
was still true from the first.
gallery.js's inflightId removed in favor of the shared composable
so the pattern stays consistent.
Stale assertion pinned the buggy pre-G1.6 behavior (the test name
"upload invalidates the cache" literally describes the bug). The
audit's correction makes upload mirror the returned record into the
store so the card updates immediately — update the assertion to
match the corrected behavior.
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
TimeoutExpired (matched the tick 870s/900s rationale). Backfill
runs that hit the cap let the next tick continue via the archive.
- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
without it cleanup_old_tasks' WHERE finished_at<cutoff never
reaped orphan-swept rows. recover_stalled_task_runs also now sets
duration_ms (matches celery_signals.finalize's millisecond math).
- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
on Source creation, mirroring SourceService.create. Without it,
Firefox quick-add on a creator with >20 unsynced posts walked the
full feed until subprocess timeout. Renamed the constant from
_NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.
- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
NO_NEW_CONTENT — the download path (line 712) already does, and
TIER_LIMITED proves auth reached the post and was told it was
tier-gated. Verify endpoint previously showed red on this and
prompted operators to rotate working cookies.
- prune_unused_tags now runs a single DELETE with the NOT-IN
predicate find_unused_tags uses, instead of SELECT-ids →
DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
would have surfaced on a tag explosion (>65k unused tags).
- credentials.upload() reflects the returned record into the store
cache (`.set(platform, rec)`) instead of evicting it; previously
the card briefly rendered "no credential" between upload and
loadAll().
attach_in_place mirrored only the media flow, so gallery-dl-downloaded
zips/PDFs/audio bounced back as `skipped+invalid_image`, which
download_service counted as an ingest error and flipped runs to
status="error" despite N successful image attaches. Lustria patreon
event #38998 (21 images + 1 OST zip) went red for exactly this reason.
Now attach_in_place dispatches the same way as import_one: archives →
_import_archive (extracts media members, captures archive as
PostAttachment), non-media → _capture_attachment. Download_service
accepts the new `attached` result and treats non-duplicate skips as
soft skips, not ingest errors.
Also: ShowcaseView always loadInitial() on mount, not just when the
store is empty — Pinia persists across navigations and operator wants
a fresh shuffle every time the showcase loads.
/api/showcase returns a random sample; after enough scrolling, an
unlucky 3-item batch can be entirely in the `seen` Set, which was
flipping `exhausted=true` and surfacing "End." mid-scroll. The
showcase is endless by intent — only a genuinely empty API response
(library has 0 images) should mark it exhausted. Tiny-library
fallback: cap retries at 8 to avoid spinning forever.
Operator-flagged 2026-06-01: clicking Accept on a suggestion added
the tag to the database but the modal's chip rail kept showing the
old set. Suggestion would disappear from the suggestions list
(suggestionsStore drops it from byCategory) and the toast would
confirm "Tagged: …", but visually the modal looked unchanged
until you closed and reopened it.
Root cause: useSuggestionsStore.accept() POSTs to the backend and
updates its own state, but never tells useModalStore that the
backing image's tag set has changed. The addExistingTag and
createAndAdd flows in TagPanel already call modal.reloadTags()
after applying — the suggestions path needed the same refresh.
Two-line fix in SuggestionsPanel: after store.accept() and
store.aliasAccept(), call modal.reloadTags() so TagPanel's
`modal.current?.tags` rebinds.
Same 'change shared shape, miss pinned tests' trap as cfa4fb4 — the
prior commit updated only test_backfill_mixed_aggregate; three sibling
tests in the same file also pinned the old {enqueued,ok,regenerated}
shape without 'scanned' and CI bounced.
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."
1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
returned just `{celery_task_id}` and the admin card said
"Enqueued." with no counts. There was no way to tell whether the
scan found 0 candidates, 5000 candidates, or whether the worker
even picked up the task. "Found nothing" was indistinguishable
from a broken queue.
Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
shared by the Celery task and the API endpoint. The API now runs
the scan in an executor and returns `{scanned, enqueued, ok,
regenerated}`. The actual thumbnail generation work still goes
to the thumbnail Celery queue per row via
`generate_thumbnail.delay()` — the scan itself is fast
(SELECT id+thumbnail_path + a file.stat() per row).
2. **`_thumb_is_valid` accepted header-only corrupt files.** The
magic-byte check passed for any 8-byte file starting with a JPEG
or PNG header, including empty/truncated/zero-pad files that
browsers render as broken. Backfill counted these as `ok` and
never regenerated.
Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
thumbnails are minimum ~2KB even on solid-color sources; header-
only corrupt files top out around 12 bytes. 256 is well above
the corrupt floor and well below any legitimate thumbnail.
Plus the admin card now shows the per-run counts instead of
"Enqueued.":
Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
Operator-flagged 2026-06-01: downloaded images stayed at
thumbnail_path=NULL until a periodic backfill sweep picked them up,
surfacing as broken-thumbnail tiles in the gallery for hours after
the download landed.
Importer.attach_in_place deliberately skips inline thumbnail
generation (importer.py:591-592) so the import queue stays moving —
the CALLING task is responsible for the enqueue. tasks/import_file.py
already did this (line 228-239). tasks/download.py / download_service
did not — every gallery-dl-attached image landed un-thumbnailed.
Fix in download_service._phase3_persist: after each
`attach_in_place` returning status in (imported, superseded), fan out
`generate_thumbnail.delay()` + `tag_and_embed.delay()` for each
image_id. Lazy import avoids circular-import risk between
download_service and the celery task modules that depend on it.
Mirrors the existing pattern verbatim — single source of truth for
"what fires after a successful attach" remains a comment in two
places (filesystem-import task, download orchestrator) rather than
a shared helper, because the contexts differ enough (sync session vs
async orchestrator) that abstracting would obscure more than it'd
share.
Test covers the happy-path with two attached files: both get the
thumbnail enqueue AND the ML enqueue, with image_ids drawn from
ImportResult (so future supersede-on-attach paths stay covered).
The zebra striping in 717b601 was too subtle — inside the tonal error
card, surface-tint contrast washed out and rows still looked uniform.
Operator-flagged the same issue twice.
Replace the surface-tint approach with hard visual structure:
- Bottom border on each row (1px white/0.08) — clearly delineated rows
regardless of card background
- Hover darkens to black/0.25 — much more obvious than the previous
error-tint hover, gives the eye a clear horizontal track
- Slightly more vertical padding (8px vs 6px) for tap-target comfort
- Drop the gap; borders are the separator now
Operator-flagged 2026-06-01: with the failing-sources panel expanded,
all rows have the same flat low-opacity background, no visual track
from the artist name on the left to the Logs/Retry buttons on the
right. Hard to tell which row a button belongs to when scanning down
a list of 17.
Three pure-CSS changes (no DOM, no logic):
- 3px gap between rows (was 2px) — slightly more breathing room
- Even rows get a darker surface tint (zebra striping)
- Hover paints the whole row in a low-opacity error tint, so moving
the cursor toward Logs/Retry lights up the whole horizontal band
and the eye traces back to the artist name automatically
`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial
value (0) when checking that the override works. Now that create() pre-arms
enabled sources to 3, the assertion is stale — the test was already
verifying the override path, the pre-assertion just decorated it.
Drop the pre-assertion; keep the override check. Other tests use raw
Source(...) constructors (default to 0 via the column server_default),
not SourceService.create(), so they're unaffected.
A freshly created subscription has no gallery-dl archive yet, so the
first poll in tick mode would walk forever — exit:20 doesn't trip
until 20 contiguous archive-hits, and there are none. The wall-clock
cap kicks in mid-walk, and the partial-success classifier (plan #544)
gracefully labels it status=ok, but the operator still wonders why
their new subscription isn't grabbing the back-catalog.
Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`.
Same default as the manual "Deep scan" button, same auto-decrement
and auto-reset rules — once the queue drains (clean exit + zero new
files) or the budget runs out, tick mode resumes naturally.
Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'`
that arrive disabled = False, or operator-added sources that start
disabled deliberately) skip the pre-arm — they're never polled, no
budget to waste.
Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.
gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.
Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.
Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
gallery_service._provenance_clause artist branch was correlating its bare
Post reference to the outer query's primary_post_id outer-join, so the
artist filter silently matched zero rows for images with no primary post.
Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner
FROM rather than treating it as a correlated outer table.
Five sidecar/import tests still asserted that a synthetic Source row
appears after a filesystem import. Alembic 0030 retired that behavior;
the Post sits null-source and the artist linkage lives on Post.artist_id.
Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent,
test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar,
and test_apply_sidecar_recovers_from_integrity_error to assert
post.source_id IS NULL + post.artist_id linkage instead.
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.
## Schema (alembic 0030)
- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
from source.artist_id in the migration. Indexed for the artist-filter
queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
NULL. Deleting a Source detaches its Posts instead of destroying
archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
(artist_id, external_post_id) WHERE source_id IS NULL — guards
filesystem-import dedup since the existing source-bound unique
ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
image_provenance first, then DELETE FROM source WHERE url LIKE
'sidecar:%'. The Dymkens cleanup.
## Model + service changes
- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
`_lookup_source_for_sidecar` (returns None when no subscription).
`_find_or_create_post` takes required `artist_id`; matches on
(source_id, external_post_id) for source-bound posts or
(artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
Source so NULL-source posts surface); artist_service date_row/
activity/post_count; provenance_service.for_image/for_post (LEFT
JOIN); gallery_service._provenance_exists_where_artist via
Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
NULL-source rows.
## Frontend
- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
?? 'filesystem import'` so NULL-source posts get a clear
"filesystem import" affordance instead of a NaN crash.
## Tests
- `test_importer_upsert_helpers`: removed the four synthetic-anchor
tests; added `_find_or_create_post_idempotent_with_null_source`
(dedup via the partial unique index) and
`_lookup_source_for_sidecar_returns_*` (existing-subscription +
none cases). The existing `_find_or_create_post_idempotent` now
also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
gains `artist_id=<artist>.id`. The `_seed_post` helper in
`test_post_feed_service` looks up artist_id from the source row so
callsites stay one-arg.
## Verification on deploy
After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
posts; PostCard same.
## Out of scope
- UI to manage/delete orphan NULL-source Posts. Data model is right;
UI follows if operator wants it.
Operator-asked 2026-06-01: each row in the "X sources are failing"
rollup needs a button to surface the last run's stdout/stderr/error
without leaving the Downloads tab to find the matching event row.
Wired:
- `downloadsStore.loadLastForSource(sourceId)` two-steps via
`GET /api/downloads?source_id=N&limit=1` (most recent event) then
the existing `loadOne(id)` for the full detail payload. Returns
null if no events exist (toast warns).
- `FailingSourcesCard` row: new text button `Logs` with
`mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`,
emits `view-logs` with the source record.
- `DownloadsTab.onViewFailingLogs` is the handler — same
`DownloadDetailModal` instance the event-row clicks use.
Operator-flagged 2026-06-01: clicking a Provenance post title from the
view modal opens the post in the posts feed scoped to that artist
(`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll
loaded UNFILTERED global posts instead of staying in the artist's
stream. Root cause: the posts store's loadAround/loadOlder/loadNewer
sent only `{around, cursor, direction}` — never the `artist_id` /
`platform` filters. Backend `/api/posts` accepts them on every path
(post_feed_service.scroll filters via `Source.artist_id`); the
frontend just wasn't passing them.
Fix:
- `posts.js` `loadAround(postId, filters)` now takes the filter
snapshot and stores it. `_aroundParams(extra)` mixes the active
filters into around/cursor calls.
- `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` +
`platform` from the route query.
Two operator-flagged UX gaps from 2026-06-01:
1. **Esc trapped inside the tag-entry field**
The autofocused TagAutocomplete input made Esc-to-close unreachable
because ImageViewer's keydown handler bailed early on
`isTextEntry(ev.target)` for every key including Escape. Now Escape
closes the modal from text inputs too — except when a Vuetify
overlay is open (FandomPicker, autocomplete dropdown, suggestion
3-dot menu), in which case that overlay's own Esc-handling fires
instead of closing the whole modal mid-interaction. Detected via
`.v-overlay--active`. Arrow keys still gate on isTextEntry so the
tag input handles typing without navigating images.
2. **Provenance cards expanded the side panel unbounded**
When an image had many ImageProvenance entries the cards stack
pushed the Tags section below the fold. Wrapped the cards in a
`.fc-prov__cards` container with max-height 270px (≈2.5 cards) and
thin overflow scrollbar. Title stays anchored at top; Tags panel
sits at a consistent position below regardless of provenance count.
`${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner
executes the step via dash/BusyBox sh and errored with
`Bad substitution` (act_runner workflow shell, observed on the
65386f0 main-push run). Switched to
`SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in
both shells. Both build-web and build-ml tag-determination steps
updated.
Two operator-asked modal UX changes (Scribe plan #514):
1. **Post title click → artist-scoped posts feed**
- The bold post title in ProvenancePanel is now the primary click
target. Click navigates to /posts?post_id=N&artist_id=A; the
PostsView already filters on `artist_id`, so the user lands in
that creator's stream, not the global one.
- The redundant "View post" link is removed. "Show description"
stays as the only action link below the meta line.
- Title is styled as a button-link: accent color, hover underline,
focus ring for keyboard nav (family rule 24 — UI quality bar).
2. **Suggestion rows look like buttons**
- SuggestionItem becomes a chip-card: visible border, hover/focus
background, unified container for name + score + actions
(operator's "nothing visual to unify the buttons to their object"
complaint).
- Accept is an explicit tonal pill button labeled "Accept"
(operator's pick over whole-row-clickable). 3-dot menu retains
alias/dismiss, now `variant="outlined"` so it reads as a button.
- Score is a fixed-width monospace pill on the right.
- "+ new" badge upgraded to a pill chip with accent border so
it's visibly an annotation, not part of the tag name.
Four coupled operator-asked changes to the view modal (Scribe plan #509):
1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
in onMounted/nextTick so the caret is in the input the moment the
modal renders. No click needed to start typing.
2. **General suggestions expanded by default** — SuggestionsPanel's
general-category group now mounts with `:default-open="true"`.
Operator can collapse if too noisy, but the v1 frame shows them.
3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
suggestion_threshold_general default matches character. Alembic
0029 also bumps the existing singleton row's value if it's still
at the old 0.95. Operator can re-tune from Settings → ML.
4. **Retire `copyright` + `artist` as ML suggestion categories** —
neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
really existed as a copyright tag-kind). They were surfaced in the
suggestions pipeline + threshold settings UI but had no follow-
through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
MLThresholdSliders rows.
Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.
Tests:
- test_surfaced_categories asserts {character, general}, excludes
artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
artist + copyright keys in the GET payload.
Adds an immutable per-commit docker tag to every main-push build:
`git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`,
alongside the existing floating `:main` + `:latest`. Implements the
new family release-posture rule "Tags are milestones, not gates —
commit-SHA images are the rollback unit" so rollback to any commit
on main is `docker pull …:c-<sha>` with no release ceremony required.
Behavior change summary:
- main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>}
- tag-push (opt-in vYY.MM.DD only, no .N): unchanged
- safety-net dev: unchanged
No code changes; the rule is about how the tag list is constructed.
Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still
fire them when the operator wants a labeled checkpoint.
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup):
"the add source/subscription button idea to the firefox extension so
it can tell me if a source/artist is added or not and offer an option
to add it if it isn't." Plan tracked in Scribe task #507.
## Backend
- `ExtensionService.probe(url)` — read-only resolution. Reuses
`_derive` for platform+slug, then 2 SELECTs. Returns one of:
- `source_match` (exact (artist, platform, url) Source exists)
- `artist_match` (artist exists, this URL isn't a Source yet;
collapses the sidecar-synthetic-only case from v26.06.01.0)
- `new` (neither exists)
- `unknown_platform` (URL didn't match any artist-page regex)
- `GET /api/extension/probe?url=...` route with `X-Extension-Key`
auth posture matching `/quick-add-source`. Read-only, side-effect
free.
- 6 backend tests in tests/test_api_extension.py covering each state
+ auth + invalid URL.
## Extension
- `api.js`: `probeSource(url)` mirroring `quickAddSource` shape.
- `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The
latter strips the `/api` suffix from configured `apiUrl` (placeholder
format per options.html) and opens `${base}/artist/{slug}` in a new
tab via `browser.tabs.create`.
- `content-script.js`: probe-first render — on page-load and SPA
navigation, asks the backend for the URL's state and renders the
chip in the matching color/copy on FIRST paint instead of flashing
generic "Add" and updating after. Click handler branches:
`source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing
ADD_AS_SOURCE flow (then re-probes so the chip flips green
immediately, no wait for next nav).
- `content-script.css`: three state-color modifiers
(--new, --artist-match, --source-match) on the FC parchment-on-slate
palette. Sage for already-added, amber for artist-exists, accent
orange for new.
## Versioning
- `extension/manifest.json` + `extension/package.json` → 1.0.6.
build.yml's sign-extension job will fire on push to main since no
`ext-1.0.6` Forgejo/Gitea release exists yet — exercises the
regenerated AMO keys end-to-end.
## Behavior on the sidecar-synthetic case
Filesystem-imported "Dymkens"-style artist with only a sidecar
synthetic Source: probe returns `artist_match` (not `new`), so the
chip reads "+ Add Patreon source to Dymkens" rather than offering to
recreate the artist. Clicking adds the real Source; existing
`_source_for_sidecar` preference logic (v26.06.01.0) routes future
gallery-dl Posts to the real one.
Three coupled operator-reported pains from the 2026-05-31 download
event audit:
1. `[patreon][warning] Not allowed to view post N` was bubbling up as
an error event, bumping consecutive_failures and parking the source
in "needs attention." The classifier's tier-gated branch was gated
on `return_code in (1, 4)`. Gallery-dl returns a different exit
code for mixed-failure runs (e.g. paywall warnings + a missing
yt-dlp dep flipping the exit bits), so the branch never fired and
the path fell through to UNKNOWN_ERROR. Widen the gate: when no
source-level error fired AND tier-gated warnings are present,
classify as TIER_LIMITED regardless of return code.
2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
"stranded by recovery sweep (no terminal status after time_limit)"
+ empty stdout/stderr. Root cause: subprocess.run timeout (900s)
and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
wiped the in-memory captured output and the DownloadEvent ended up
empty-logged 18 minutes later when the sweep finalized it. Drop
gallery-dl's default subprocess timeout to 870s — a 30s margin
shy of Celery's soft limit — so subprocess.TimeoutExpired always
wins the race and captures the partial stdout/stderr via the
existing handler.
3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
firing on every video attachment, causing per-item download
failures that masked legitimate tier-gated classification.
Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
video posts download normally and the per-item failure noise
disappears.
Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:
1. `SourceService.list()` returned every Source, no filter on URL.
alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
one canonical row per (artist, platform); when no real campaign URL
was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
enabled=false as a disabled anchor. The UI then listed those
anchors as if they were polls — disabled, but visible. Fix: `list()`
excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
opts back in for admin tooling.
2. `importer._source_for_sidecar` picked the lowest-id Source for
(artist, platform). When alembic 0022 had rewritten a per-post row
into a synthetic anchor (lower id) AND the operator later added the
real subscription (higher id), every gallery-dl download silently
attached its Post to the SYNTHETIC instead of the real Source. Fix:
prefer a non-`sidecar:%` URL when one exists; fall back to the
synthetic; only create a new synthetic when nothing exists for
(artist, platform).
alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.
Today's platform-cooldown commit (61ce1ce) only filtered the scan tick
— manual /api/sources/<id>/check still bypassed it. Operator-flagged
2026-05-30: clicked "Retry failed" on a Patreon failure pile and saw
every one go 'queued' without realising the cooldown wasn't in the
loop. Bulk retry with N sources on a cooled-down platform bowls right
back into the rate limit the cooldown is trying to prevent.
**Backend (`/api/sources/<id>/check`):**
- Reads optional `?force=true` query flag.
- Without force: queries `active_platform_cooldowns` (renamed from the
private `_platforms_in_cooldown` since it's now a cross-module API).
If the source's platform is in cooldown, returns **202** with
`{status: 'deferred', platform, cooldown_until}` — no event created,
no dispatch.
- With force: cooldown skipped entirely.
- In-flight guard always applies (no point creating duplicate pendings).
**Frontend (`sourcesStore.checkNow(id, {force=false})`):** new optional
`force` flag → adds `?force=true` to the URL.
**Frontend (`DownloadsTab`):**
- `onRetrySource` (single-source RETRY click): passes `force: true` →
explicit operator override, useful for rapid auth-fix testing.
- `onRetryAll` (RETRY ALL + MaintenanceMenu "Retry failed"): no force →
cooldown respected. Tallies `deferred` alongside `queued` /
`already_running`; toast reads e.g. *"5 queued, 12 deferred
(cooldown), 3 already running"*. That count is the operator's
diagnostic answer for "is rate-limit the cause of most failures?"
(12-of-20 deferred → yes; 0 deferred → no).
**Auto-resume:** no new sweep needed. Deferred sources still have stale
`last_checked_at`, so the next scan tick after the cooldown AppSetting
expires picks them up via `select_due_sources` (which already filters
on `active_platform_cooldowns`).
Tests: two new — deferred-on-cooldown returns 202 with the right body
and no dispatch; force=true overrides the cooldown and creates the
event normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast
but could let later chunks arrive ahead of earlier ones — even when
each chunk is a random sample, that "later chunk loads first" risk made
the load order non-deterministic. And the original goal behind asking
for batching was a faster first-image-on-screen, which neither sequen-
tial nor parallel really addressed cleanly.
Switched the loadInitial flow to a PIPELINE:
- Only one fetch in flight at any moment (in-order arrival, no race).
- The NEXT fetch kicks off as soon as the current one resolves (NOT
after its trickle finishes), so the next RTT overlaps the visible
trickle window — round-trips are hidden behind the animation cadence.
- PAGE 5 → 3 + INITIAL_BATCHES 12 → 20 (total still 60). Smaller chunk
→ first chunk's items appear sooner (a chunk of 3 trickles in 240ms,
well within one RTT, so by the time chunk 2 is in-hand the first
trickle is just finishing).
Trickle, sequence-token guard, and infinite-scroll behaviour unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-30 (round 2): the batched-loads change improved
consistency but still felt "chunky" — 5 items appeared together, then a
~200 ms API round-trip pause, then another 5. And infinite-scroll fired
too late when a single tall image (long manga page) made one masonry
column much taller than its siblings.
**Cadence (showcase store):**
- Fire all INITIAL_BATCHES fetches in PARALLEL — collapses the per-
batch round-trip gap so all the data arrives in ~1 RTT instead of 12
sequential RTTs.
- Trickle each response's items into images.value one at a time with
APPEND_DELAY_MS = 80ms between each (≈ the MasonryGrid stagger
animation, 70ms). User sees a smooth steady stream.
- fetchPage (the infinite-scroll path) uses the same trickle so its
5-item appends also cascade one-by-one instead of popping together.
- Sequence token guards against a fast shuffle / mount-then-shuffle
interleaving two trickles into the same images.value.
- Dropped useAsyncAction here — the parallel-fetch-then-trickle flow
doesn't fit its single-wrap-call shape cleanly; inline loading/error
state is clearer.
**Infinite-scroll trigger (MasonryGrid):**
- Pass `rootMargin: '2400px'` to useInfiniteScroll (was the 600px
default). The masonry sentinel sits at the bottom of the container,
whose height = MAX(column heights). A tall image in one column pushes
the sentinel ~2× viewport below where the user is actually reading
(the bottom of the SHORTER columns). 2400px ≈ 2-3 screen-heights of
pre-emptive trigger, comfortable for typical tall manga heights.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every archive import was failing immediately with "AssertionError:
daemonic processes are not allowed to have children" (operator-flagged
2026-05-30 — import_archive_file crashes in 49ms with the assertion).
Celery's prefork pool runs tasks in daemon processes; Python's
multiprocessing module refuses to let daemons spawn children, which is
exactly what probe_archive was doing via mp.get_context("spawn")
.Process. The Layer-3 crash-isolation feature added 2026-05-28 was
effectively a hard-blocker on the very import path it was meant to
protect.
Switched probe_archive to subprocess.run — no daemon restriction, still
isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill
the worker). The probe body lifted to a tiny runner module
(_archive_probe_runner) that imports the unchanged _run_probe helper
and prints a single JSON line; parent parses stdout, returns the
ProbeResult exactly as before (timeout, signal, OOM, clean-rejection,
ok — all preserved).
cwd for the subprocess is the repo root derived from __file__ parents
so `python -m backend.app.utils._archive_probe_runner` resolves both in
the Celery container and pytest, regardless of where the worker was
launched.
Test refactor: test_archive_probe_target_bomb_guard →
test_run_probe_bomb_guard. Same in-process call to the (renamed) probe
body so the monkeypatched cap still takes effect; the real subprocess
path is exercised by the existing test_probe_archive_valid_zip /
test_probe_archive_corrupt_zip_clean_rejection tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The showcase/gallery/artist/series/post-feed APIs were constructing
thumbnail URLs from (sha256, mime). The MIME-based extension predicate
("png if image/png or image/gif else jpg") DISAGREED with the
thumbnailer's actual on-disk extension predicate ("png if alpha else
jpg"). Result: every PNG source without transparency 404'd (URL asked
.png, disk had .jpg); every WebP/AVIF source with transparency 404'd
(URL asked .jpg, disk had .png) — despite the thumbnail file existing
on disk.
The backfill task couldn't catch these because backfill checks the
ACTUAL thumbnail_path stored on the record (correct), not the URL the
browser fetches (broken derivation). So records with valid on-disk
thumbnails kept showing as broken in the UI no matter how many times
backfill ran.
Operator-flagged 2026-05-30: "the generate thumbnails function appears
to not catch all of the failed thumbnail cases" — turned out to not be
a backfill bug at all.
Fix: thumbnail_url now takes (thumbnail_path, sha256, mime) and returns
the stored path verbatim — Quart serves /images/* 1:1 from the volume
(frontend.py:20-36), so the URL IS the disk path. Falls back to the old
sha256+mime derivation only when thumbnail_path is NULL (thumbnailer
hasn't run yet); that URL will 404 in the browser until backfill catches
it, same as before the path was tracked.
All 8 callers updated: showcase_service, gallery_service (2 sites),
artist_service, series_service, post_feed_service, tag_directory_service,
artist_directory_service. The four sites whose query was raw-tuple now
also SELECT ImageRecord.thumbnail_path.
Net effect: every record that has a valid on-disk thumbnail will now
render correctly, regardless of which extension the thumbnailer chose,
without any DB migration or backfill rerun needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The showcase-store change in adeee64 didn't actually land — only
gallery.js + ShowcaseView.vue made it into the commit. Without this,
ShowcaseView mounts loadInitial() which doesn't exist yet, so the
showcase blows up. Landing the matching store edit now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Showcase and gallery were fetching the full initial batch (60 / 50
items) in a single API call. Operator-flagged 2026-05-30: items should
stream in as small batches so they render progressively rather than
blocking on one big response.
- showcase store: PAGE 60 → 5, INITIAL_BATCHES = 12 (5 × 12 = 60, same
total). loadInitial() fetches PAGE chunks in sequence; shuffle()
delegates. fetchPage() still returns one PAGE-sized chunk so
infinite-scroll also pulls 5 per trigger.
- gallery store: loadMore() limit 50 → PAGE (5), INITIAL_BATCHES = 10.
loadInitial() loops loadMore() up to INITIAL_BATCHES times, stopping
early when nextCursor becomes null (end of data).
- ShowcaseView: onMounted calls loadInitial() instead of fetchPage().
Gallery/setTagFilter/setPostFilter already routed through loadInitial.
Net visual effect: each ~5-item batch lands and renders independently;
combined with MasonryGrid's animateFromIndex logic, the showcase now
cascades batches in as they arrive instead of one big pop-in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show
anything other than that the task timedout and was cleaned up. I can't
tell why it ran over or if it was stuck failed or there was just that
much to get."
The TimeoutExpired branch was returning a DownloadResult with no stdout,
no stderr, no files_downloaded, and a generic "Download timed out after
N seconds" message — even though subprocess.TimeoutExpired carries the
partial output gallery-dl emitted before being killed.
Now:
- Capture e.stdout / e.stderr (coerced str if bytes; "" if None).
- Count files_downloaded from partial stdout via _count_downloaded_files.
- Surface a tail-of-stderr hint in error_message so the UI summary tells
the operator at a glance whether it was "lots of content" (high count,
clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung
silent" (zero count, "no stderr output").
- Promote error_type to RATE_LIMITED when the partial stderr matches
RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole
900s window is the timeout-shaped tail of a real rate limit, and the
platform cooldown should kick in for the same reason.
Existing test_download_timeout strengthened to also assert empty-partial
case stays correctly TIMEOUT-classified with no preserved output.
New test_download_timeout_preserves_partial_output_and_classifies covers
the rich-partial-output → RATE_LIMITED promotion path.
DownloadEvent.metadata already flows stdout/stderr/run_stats from
DownloadResult via _phase3_persist — no UI change needed; the existing
DownloadDetailModal will surface the captured output automatically once
the build redeploys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:
> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.
Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.
Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61ce1ce added platform_cooldowns to scheduler_status(), and two pinned
key-set assertions tied to the old shape failed in CI. Updated:
- tests/test_api_sources.py::test_schedule_status_shape — direct
/api/sources/schedule-status response.
- tests/test_api_system_activity.py::test_summary_returns_rollup_shape
— nested under body["scheduler"] in the summary endpoint.
[[feedback-plan-grep-pinned-tests]] miss — should have grepped tests/
for `last_tick_at` / `auto_sources` before adding the new key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.
When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.
INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.
Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.
Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's
Celery task has soft_time_limit=900s and hard time_limit=1200s. So
gallery-dl never hit its own subprocess.run timeout — Celery always
killed it first. The hard SIGKILL leaves no terminal flip on the
DownloadEvent, which then sat pending/running until the recovery sweep
flipped it to error at 30 min from start. From the operator's seat that
was a ~30–40 min "hang" on every retry of a broken source.
Pinning the default to 900s (matching Celery's soft_time_limit) lets
subprocess.run raise TimeoutExpired cleanly inside Celery's window, and
the existing `except subprocess.TimeoutExpired` branch
(gallery_dl.py:635) captures it as a clean error_type='timeout' with a
real message. The DownloadEvent flips to error in ~15 min instead of
waiting on the recovery sweep at 30.
Per-source bumps still live in source.config_overrides for legitimately
long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries
the rationale and the Celery-soft-limit dependency in its comment.
Operator-flagged 2026-05-30 on 59-source strand pile retries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The maintenance dropdown in Subscriptions → Downloads was wired to the
filesystem-import pipeline (POST /api/import/retry-failed +
POST /api/import/clear-stuck) — the subtitles even said so ("Re-enqueue
every failed import task"), but it was contextually misplaced. From the
Downloads view "Retry failed" queued nothing the operator could see
because the action operated on import_task rows, not download_event
rows. Import-pipeline maintenance is already reachable from Settings →
Imports (ImportTaskList.vue), so removing the import wiring loses
nothing.
Rewired:
- "Retry failed" → bulk-retries the failing-sources list, same loop as
FailingSourcesCard's RETRY ALL (sourcesStore.checkNow per source).
Subtitle now matches: "Re-queue every currently failing source".
- "Force recovery sweep" → triggers recover_stalled_download_events on
demand via a new POST /api/downloads/recover-stalled endpoint. The
sweep also runs every 5 min on Beat; this is the manual fallback so
the operator doesn't have to wait for the next tick to clear newly
stranded events.
MaintenanceMenu is now stateless — emits retry-failed and recover-
stalled. DownloadsTab owns the handlers (reuses the existing
onRetryAll; new onRecoverStalled with a delayed refresh so swept rows
land in the failing rollup).
Operator-flagged 2026-05-29 — "the retry failed button in the
maintenance dropdown doesn't appear to queue anything but manual
requeues works."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
104cac5's override sits at the same specificity as Vuetify's default
(`.v-tooltip > .v-overlay__content` on both sides). The fix relied on
source order — app.css imported after vuetify/styles in main.js — but
Vite's production bundler reorders node_modules CSS into the final
stylesheet unpredictably, so source order isn't a reliable winner.
!important on the two contrast properties (background + color) forces
the slate-on-parchment pair regardless of stylesheet load order. The
cosmetic border + shadow don't need it (Vuetify doesn't set them, so
nothing's competing).
Operator re-flagged 2026-05-29: tooltips still rendered light-on-light
on the deployed :latest after PR #33 — 104cac5 was in the bundle but
not winning. Also updated the file header so the source-order claim
isn't carried forward as gospel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.
cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).
New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
download_source 1200s hard kill, so legitimately-running tasks are
never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
backoff is 2^N on that counter so per-event bumps would needlessly
inflate the next interval — sets last_error, stamps last_checked_at.
UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.
Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plain-JS frontend re-declares two backend enum value-sets by hand:
download-event statuses (downloadStatus.js) and platform keys
(platformColor.js). With no TS codegen, drift is silent — the same class
as the last_verified_at field mismatch.
tests/test_fe_be_contract.py parses both JS mirrors and asserts they equal
the backend canon (downloads._ALLOWED_STATUSES, platforms.known_platform_keys())
so a change on either side fails CI on whichever moved. Backend is the
source of truth. Documented the invariant in both JS file headers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The IntersectionObserver-on-a-sentinel infinite-scroll pattern was copy-
pasted in 7 places. New composables/useInfiniteScroll(sentinelRef, cb)
owns the observer lifecycle (attach on mount, re-attach when the ref
changes, disconnect on unmount). Migrated GalleryGrid, MasonryGrid
(gallery+showcase), ArtistPostsTab, ArtistsView, TagsView, SeriesManageView.
PostsView left manual on purpose — its anchored mode does bidirectional
scroll-position preservation that doesn't fit the simple composable.
I4(a) (timestamps) is effectively already done: the UTC displays were
converted earlier; the remaining toLocaleString uses already render local
time, so they're left as-is rather than churned for format-only consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New /api/system/activity/summary aggregates scheduler health + per-queue
pending depths + running count + 24h failure count in one cached call (safe
to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a
compact running/queued/failing chip with a scheduler-health dot that expands
to a popover (scheduler, busy queues, counts, link to downloads). Polls the
summary every 8s, paused when backgrounded. Reuses the queue-cache read via
_queues_cached(). + API test for the summary shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The row template renders status + platform (PlatformChip) + time + counts;
the artist isn't displayed there. Assert on 'Completed'/'Patreon' instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds @vue/test-utils + happy-dom and mounts CredentialCard, DownloadEventRow,
PostCard, ActiveDownloadsPanel, QueueStatusBar, asserting they render without
throwing and surface key content — catching the dead-binding / render-error
class (e.g. the last_verified_at regression, guarded explicitly). Vuetify
components are left unresolved (Vue renders unknown elements + slots), so no
Vuetify-plugin setup is needed; only RouterLink is stubbed. Per-file
happy-dom env via docblock keeps the existing node-env specs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ruff is pre-installed in the ci-python image, so a new `lint` job runs it
with no dependency install and fails in seconds — surfacing the common lint
bounce class without waiting on the backend job's ~30-60s wheel install.
Dropped the now-redundant ruff step from backend-lint-and-test (same job
name, required-checks unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provenance "View post" deep-links to /posts?post_id=X, which now opens the
feed centered on that post with infinite load in BOTH directions.
Backend: PostFeedService.scroll gains a direction (older|newer); new
around(post_id) returns a window of newer + the post + older with a cursor
for each end. /api/posts accepts ?around= and ?direction=. + API tests.
Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends,
newer prepends) with per-end cursors; PostsView's anchored mode scrolls to
the post, observes top + bottom sentinels, and preserves scroll position on
upward prepend so the page doesn't jump. Normal feed mode unchanged.
Closes the remaining half of the post-navigation work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each maintenance button (ML backfill, centroid recompute → ml queue;
thumbnail backfill → thumbnail queue) now shows a status bar with the live
pending count for its Celery queue, so the operator can see work is already
queued/running before re-triggering and piling on. MaintenancePanel polls
/api/system/activity/queues every 4s; QueueStatusBar reads the depth and
turns warning-colored when the queue is busy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New ActiveDownloadsPanel pinned at the top of the Downloads tab shows what's
running (pulsing dot + live mm:ss timer counting from started_at) and what's
queued, so the operator can see activity at a glance without the running
filter. Backed by store.loadActive() which fetches running + pending
independent of the feed filter; refreshed every 4s by the existing live poll.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Download event times showed raw UTC wall-clock (iso.slice). Added
formatDateTime()/formatLocalDate() (local tz, robust to naive vs tz-aware
ISO) and applied them to the download row + detail modal datetimes and the
credential/artist date displays. formatPostDate stays UTC (date-only,
locale-stable, unit-tested).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaced the subtle 12px fade with a perspective rotateX(-28deg) tilt that
flips up and settles flat with a slight overshoot, staggered 70ms so tiles
cascade in one at a time. Makes the showcase read as an experience rather
than a quiet fade. Tunable (tilt/stagger/duration); reduced-motion safe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FC already matched IR's staggered entry fade-in; this adds the missing
piece — hover zoom (scale 1.03) + brighten (1.1) on each thumbnail, with
overflow:hidden so the scaled image clips to the rounded card. Disabled
under prefers-reduced-motion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Post titles arrived as stored HTML (e.g. <strong>…</strong>) rendering as
literal markup. New toPlainText() strips tags; titles render plain + bold
(ProvenancePanel and PostCard).
- Removed "View original post" from the provenance panel (modal) — the
open-original button lives on the PostCard (the post view).
- Provenance "View post" now navigates to the /posts feed (post_id query),
not the gallery image grid. (Feed in-context landing lands next.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The whole view scrolled instead of just the subscription list. Made the
hub a viewport-height flex column (tabs stay fixed) with the v-window as
the single internal scroll container; the per-tab sticky control bars now
pin to the window top (top:48px -> 0). Operator-flagged 2026-05-28.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ArtistView and SubscriptionsView both two-way-bound a tab ref with the
?tab= query param via identical tab->URL and URL->tab watchers. Extracted
useTabQuery(validTabs, defaultTab) — defaultTab accepts a string or a
function (ArtistView's default is postCount-dependent), and resolve() is
exposed so ArtistView can re-apply it after the artist loads. Both views
drop their now-orphaned ref/useRouter imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_find_or_create_source, _source_for_sidecar, and _find_or_create_post each
repeated the SELECT → savepoint-INSERT → on-IntegrityError rollback+re-SELECT
pattern. Extracted _get_or_create(stmt, factory): the statement is reused for
the scalar_one_or_none lookup and the scalar_one post-conflict re-fetch, so
all three are reproduced exactly. Centralizing the race-safe pattern in one
place also reduces the risk of the copies drifting (the bug class banked
2026-05-26).
Left _upsert_artist (no savepoint by design) and the ImageProvenance void
ensure-exists block (no return / no re-select) alone — they don't fit.
The rest of the ingest pipeline was already DRY: sidecar parsing lives in
utils/sidecar.py, per-platform quirks in the platforms package, and
_safe_ext/_categorize_error/_build_config are each single-instance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
download/migration/scan each defined an identical _async_session_factory()
(fresh per-invocation async engine — async connections are event-loop-bound
so each asyncio.run() task needs its own engine, unlike the process-wide
_sync_engine). Moved it to tasks/_async_session.py; the 3 files import it
and drop their now-orphaned sqlalchemy.ext.asyncio / get_config imports
(migration keeps AsyncSession for a type hint). Call-site try/finally
dispose left as-is to avoid re-indenting the critical task bodies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 blueprints each defined an identical _bad() (two variants: with/without
detail). Extracted error_response() into api/_responses.py; each blueprint
now imports it `as _bad` so call sites are unchanged. The detail-aware
canonical subsumes both variants. Left settings.py's distinct _bad_int and
the inline jsonify error sites (not duplicated helpers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New composables/useAsyncAction.js owns the loading/error/try-finally
lifecycle. Migrated 11 stores: credentials, downloads, sources, posts
(error=raw) + ml, artistDirectory, tagDirectory, showcase, suggestions,
seriesReader, modal (error=message). The errorAs option preserves each
store's existing error shape so store.error keeps the same type for
components (~50 consumption sites unchanged). Stores whose catch also
reset data (suggestions/seriesReader/modal) clear it upfront instead.
Deliberately NOT migrated (special control flow, would change behavior):
artist (conditional 404 catch + dual loading states), migration (rethrows),
gallery (inflight-id stale-response guard), and the Shape-B no-catch /
loaded-guard / keyed-cache stores.
Net -77 lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removing the create_app import left 2 blank lines before pytestmark in 9
files where ruff isort wants 1. Also stripped two pre-existing
whitespace-only blank lines surfaced by the file change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removed the app/client fixtures duplicated across 36 test files (two
variants: separate app + client(app), and a self-contained client() that
called create_app inline) and the now-unused create_app imports. Both
fixtures now live once in conftest.py. test_suggestions_bulk keeps its
import (builds the app inline in two tests); test_health drops its local
client + unused pytest_asyncio.
Net -415 lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick +
GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources)
+ SchedulerStatusBar on the Subscriptions tab (re-polled every 30s).
D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard
on Downloads with per-source and bulk "retry" (re-runs the feed via /check).
D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar
chart by the stat chips (failures stacked in error color); refreshes on live poll.
D4 credential staleness: surface last_verified age + "re-verify recommended"
warning past 30d; also fixes the dead last_verified_at field-name mismatch so
the verification row renders at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-28: the Subscriptions/Downloads dashboards were
full-bleed and thin on filtering. Chosen via AskUserQuestion.
**Layout**
- SubscriptionsView capped at a centered max-width 1600px (covers all
three subtabs) so rows aren't a mile wide on ultrawide monitors.
- Sticky control headers on both tabs (top: 48px, below the top nav,
opaque bg) so filters/stat-chips stay reachable while scrolling a
long list.
**Downloads filters (all four requested)**
- Stat chips are now clickable filters: click Queued/Running/Completed/
Failed/Skipped to filter the list to that status; the active chip is
outlined + elevated; re-click clears.
- Free-text search box over the loaded events (artist / platform /
error substring).
- Artist filter: the filter popover's numeric "Source ID" field is
replaced with an artist autocomplete (sources.autocompleteArtist →
artist_id, which /api/downloads already supports). Pill shows the
artist name.
- "Show no-change scans" toggle (default OFF): hides status=ok/skipped
rows with 0 files (the scheduled scans that found nothing) so real
downloads + failures stand out.
**Subscriptions**
- "Needs attention" quick-filter chip: one click to show only artists
with sources that have errors OR have never been checked; chip shows
the count and disables the status dropdown while active.
Frontend-only — backend filter params (status/artist_id/date) and the
/api/downloads endpoint already supported everything.
Operator-flagged 2026-05-28: the download event detail modal had no way
to copy the error / stdout / stderr text for researching a failure
(parity with the ErrorDetailModal Copy button shipped in v26.05.26.0).
Added per-block Copy buttons (Error, Errors & warnings, Raw stdout, Raw
stderr — the stdout/stderr ones sit in the expansion-panel title with
@click.stop so they don't toggle the panel) plus a "Copy all
diagnostics" button in the footer that assembles a single block: header
line (event id / platform / artist / status / timestamps) + error +
errors&warnings + full stdout + stderr, ready to paste into an issue.
All routed through utils/clipboard.js copyText() — the navigator.clipboard
→ execCommand fallback that works on the plain-HTTP homelab origin
(per feedback_no_secure_context_apis). Each copy shows a confirmation
toast.
Operator-flagged 2026-05-28, two asks.
**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):
- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
gallery-dl in `--simulate --range 1-1` mode (no download) against the
URL with the materialized credentials, then reuses _categorize_error:
returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
the platform to probe, runs verify, and on success stamps
credential.last_verified (new CredentialService.mark_verified).
Returns {valid: bool|null, reason, last_verified?}. valid=null means
untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
result chip (Verified ✓ / Failed / Untestable) and a toast with the
reason. SettingsTab reloads on @verified so last_verified refreshes.
**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.
Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
Operator-flagged 2026-05-28, two asks.
**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.
**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)
- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
(dry_run) or deletes tags whose kind ∈ (archive, post, artist).
CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
returns per-kind counts + sample names — the preview IS the
verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".
Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).
NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
Operator-flagged 2026-05-28: tooltips (e.g. the action buttons in
Subscriptions → Subscriptions) render near-white-on-near-white,
unreadable.
Cause: Vuetify's default v-tooltip pairs `on-surface-variant` text with
an `surface-variant` background. FC's theme deliberately maps
`on-surface-variant` to vellum (#C2BFB4 — a light cream, the correct
muted-text token for captions/hints on the dark page) but never defines
`surface-variant`, so Vuetify auto-generates a light-ish tooltip
background. Light text on light bg.
Fix is tooltip-specific so it doesn't disturb the (correctly light)
muted-text token elsewhere. New app-global stylesheet
frontend/src/styles/app.css, imported in main.js AFTER vuetify/styles
(equal-specificity rule wins by source order), overrides
`.v-tooltip > .v-overlay__content` to a dark elevated panel
(surface-bright = slate #2C313A) with high-contrast parchment text
(#E8E4D8) + a subtle border + shadow. Applies to every tooltip in the
app, so the fix is consistent rather than per-component.
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.
New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
derive the artist from the import path, find an ENABLED Source with a
real feed URL for (artist, platform). Returns None for filesystem-only
imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
set ImportTask.refetched=True, and trigger ONE download_source
re-check. Bounded by `refetched` so source-side corruption can't loop.
Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
tasks). Returns refetch_queued / no_source / already_refetched /
not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
manual button is the primary path; auto is opt-in since re-fetch
deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
→ stores.import.refetchTask → toast keyed on the result status.
Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.
Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.
New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):
- probe_video(path): validates the container + first video stream via
ffprobe (a separate binary — a decoder crash kills only ffprobe, not
the worker). Returns width/height, which the importer didn't capture
for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
(MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
spawned child process. A decompression-bomb OOM or native-lib
segfault on a malformed archive shows up as a non-zero child exit
code → crashed=True, never a dead worker.
ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).
Wired:
- importer._import_media video branch: probe_video before the pipeline;
crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
→ failed, clean reject → still preserve the archive as a
PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
so a corrupt video is rejected (status='bad_video') instead of
crashing the ml-worker on frame decode.
Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.
**Split archive into its own task**
- New `import_archive_file` task: same body as import_media_file
(dispatch is by file-kind inside Importer.import_one) but
soft=30min / hard=35min. Shared `_run_import_task` helper holds the
flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
truth for media-vs-archive dispatch. Used by all three enqueue sites:
scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
is_archive(entry) (the model field already existed, anticipating
this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
task_routes `import_file.*` wildcard — no worker config change.
**Archive-aware recovery sweeps**
Both sweeps would otherwise preempt a legitimately-running archive:
- recover_interrupted_tasks (ImportTask 'processing' sweep): now
task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
35-min hard limit). Single UPDATE with an OR predicate over the two
(task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
the per-queue overrides added for ml. import_archive_file gets 40 min
while the 'import' queue stays at the 5-min default for single-file
imports. Precedence: task_name → queue → default, each pass excluding
rows claimed by a higher-precedence pass so every row is touched once.
**Tests**
- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
pins that a 10-min archive task-run survives, a 50-min one is flagged,
and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.
After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.
Root cause is two interacting limits, both undersized for video work:
tag_and_embed: soft_time_limit=300, time_limit=420
(sized for the image branch, ≈2 GPU ops)
recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues
The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.
**Two-part fix**
1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
(20 min). Sized for the video path's worst case; image runs return
in seconds and don't care.
2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
Queues with legitimately-long-running tasks (currently just `ml` at
25 min — 5-min buffer past the new hard kill) get their own
threshold; queues not in the dict use the default 5 min. The sweep
now issues one UPDATE per distinct threshold value, with
`queue.notin_(override_queues)` on the default pass so each row is
touched at most once.
Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
pins both directions: a 10-min-old ml row survives (fresh by 25-min
override), a 30-min-old ml row gets flagged.
After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
Operator-flagged 2026-05-27: the Downloads subtab "doesn't feel like a
dashboard" — status was a tiny mdi icon at the far left, platform chip
was neutral-tonal, errors were plain orange text floating on the right,
and all 28 rows from the same hour visually had the same priority.
**Row restyle (A):**
- 4px colored left-edge bar by status (success/error/info/warning/grey)
— visually scannable at the edge without parsing the chip text
- Status chip with text label (Completed/Failed/Running/Queued/Skipped)
+ leading icon, tonal-colored. Replaces the bare mdi-icon.
- Platform chip swapped to the color-coded subscriptions/PlatformChip
(Patreon=red mdi-patreon, SubscribeStar=amber, HentaiFoundry=purple,
Discord=indigo, Pixiv=blue, DeviantArt=green).
- File count: tonal info chip when > 0, dim middle-dot when 0 (so
scheduled "no-change" scans don't dominate the column visually).
- Error: red tonal pill chip with leading icon, truncated to 60 chars
with full text in the title tooltip. Replaces plain text.
- Per-row actions (hidden at 50% opacity, fade to full on row hover):
Retry (only when status=error AND source_id known — hits
POST /api/sources/<id>/check via the existing sources.checkNow),
Details (opens the detail modal), Open artist (navigates to the
artist page). Clicks stop-propagation so they don't bubble to the
row click.
**Date-grouped sections (B):**
- Events are bucketed into four sections: Today / Yesterday /
Last 7 days / Earlier. Empty buckets are skipped. Buckets boundaries
are computed against the operator's local-time start-of-day so
"Today" matches their intuition.
- Each section has a collapsible header with a row-count chip + a
red "failed in this section" chip when any failures are in scope.
- Within each section, status='error' rows are pinned to the top
(operator's eye lands on failures first; successful scans flow
below).
- Collapsed state persists across refresh within the SubscriptionsView
lifetime (reactive object, default all-expanded).
DownloadEventRow grid widened to accommodate the status chip + actions
column. PolyMasonry-style ellipsis on the artist link prevents long
names from breaking the layout.
No new endpoints; the Retry path reuses the existing /api/sources/<id>/check
flow (the source-check endpoint was already in place, just not wired
into a per-row button).
2026-05-27 22:18:02 -04:00
587 changed files with 60417 additions and 9057 deletions
# Relax durability on the throwaway CI Postgres so the per-test
# TRUNCATE's commit-fsync — the integration teardown's dominant cost
# (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
# skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
# is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
# NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
# surprise can't red the job; fabledcurator is the postgres image's
# bootstrap superuser.
python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
return_bad("source_disabled",detail="enable the source first")
# Cooldown gate (unless explicitly overridden). Checked before the
# in-flight guard because a deferred retry doesn't need to create
# or check for an event at all.
ifnotforce:
cooldowns=awaitactive_platform_cooldowns(session)
expires_at=cooldowns.get(source.platform)
ifexpires_atisnotNone:
returnjsonify({
"status":"deferred",
"platform":source.platform,
"cooldown_until":expires_at.isoformat(),
}),202
in_flight=(awaitsession.execute(
select(DownloadEvent.id).where(
DownloadEvent.source_id==source_id,
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.