The one-time backfill that actually shrinks the DB: drops stored
tagger_predictions entries below ml_settings.tagger_store_floor from every
image_record row, and clamps any allowlist min_confidence below the floor up
to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate
so backfilled rows match new imports. Keyset by id ASC, idempotent,
self-resumes on the soft time limit; runs on the maintenance_long lane.
pg_dump copies live data only, so this alone fixes the #739 backup timeout —
the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional
disk-return step, brief because post-prune the live data is tiny.
- admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions
- PrunePredictionsCard in the Maintenance panel (shows the current floor)
- tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consumer #4 of the store-floor change (#764). An allowlist tag can't
auto-apply more permissively than the ingest floor — predictions below
tagger_store_floor aren't stored, so a lower min_confidence behaves
identically to the floor. update_threshold now clamps to max(value, floor);
the AllowlistTable confidence input min-binds to the live floor and clamps
on edit. Keeps the stored threshold honest about actual apply behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default
0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70).
Storing every tag down to 0.05 from a ~10k-tag tagger is what grew
image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70
and the centroid/learned path covers lower-confidence preferred tags, so the
sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim
land next; this only changes the write gate for NEW imports).
- ml_settings.tagger_store_floor (migration 0044, default 0.70)
- tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor
- ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold
below the floor (nothing below the floor is stored, so the gap surfaces
nothing) — server backstop for the UI slider clamp
- Settings → ML: store-floor slider + caption; category sliders min-bound to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recover_stalled_task_runs used the 5-min default for the download queue,
but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT
(1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as
phantom 'RecoverySweep' failures — visible in System Activity but absent
from the Subscriptions view (the download finished ok, reset the source's
consecutive_failures; only the orphaned task_run kept the stamp, since
_finalize only updates rows still 'running').
Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard
limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real
hard kill is swept by the task-run and event sweeps together. Restores the
documented invariant (every override >= task time_limit). Regression test
pins the threshold above the hard limit so a future limit bump can't
silently re-break it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recent failures gains a client-side search over the already-loaded 24h
rows (task/queue/target/error), shown as a filtered/total count alongside
the existing error-type chips. All recent activity gains a debounced
server-side task-name search (new `task` ILIKE param on /runs) so it
spans the full history, not just the loaded page. LIKE wildcards are
escaped so task names' literal underscores match literally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom self-join (resolve a character's fandom NAME via Tag.fandom_id->Tag)
and the {id,name,kind,fandom_id,fandom_name} dict were hand-written in
TagService.autocomplete/.list_for_image, GalleryService.get_image_with_tags and
the api/tags handlers — the last few grown by this session's fandom-on-chip
feature. Consolidate to services/tag_query: fandom_join_alias() + tag_columns()
build the select; serialize_tag(row) builds the dict. Now a new tag field is
added in one place.
Over-DRY guard: TagDirectoryService selects the full Tag ORM + an image-count
aggregate (a different select shape) — left as its own variant. §8b: the
fandom_lookup alias lives only in tag_query; gallery + both api/tags handlers
serialize via serialize_tag. Test: serialize_tag handles enum + string kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
encode_cursor/decode_cursor (base64 <iso8601>|<id>) were defined identically in
gallery_service AND post_feed_service, with artist_service importing gallery's
copy. Two implementations of one cursor format silently break pagination in
whichever feed drifts. Extract to services/pagination.py; gallery/post_feed/
artist all import it. Dropped now-unused base64/datetime imports.
§8b: encode_cursor/decode_cursor now defined only in pagination.py. Existing
cursor round-trip tests still cover it via the re-export. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI lint flagged UP047 — use the native generic syntax def get_or_create[T](...)
instead of typing.TypeVar on Python 3.14.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The find-or-create dance — SELECT, then a SAVEPOINT INSERT that recovers (not a
full rollback) on IntegrityError when a concurrent worker inserted first — was
hand-rolled identically in 4 async sites: ArtistService.find_or_create,
TagService.find_or_create, ExtensionService._find_or_create_artist and
._find_or_create_source. Divergent copies of exactly this pattern are how the
duplicate-row/race bugs in reference_scalar_one_or_none_duplicates crept in, so
it now lives once in services/db_helpers.get_or_create (returns (row, created);
factory adds+flushes+returns the row; caller owns the outer commit).
Over-DRY guard: SourceService's IntegrityError sites RAISE DuplicateSourceError
(reject-on-conflict, a different concept) — left alone. Importer._get_or_create
is the lone SYNC consumer (already shared by 2 callers) — stays separate, can't
cross the sync/async boundary. §8b: no hand-rolled async find-or-create remains.
Test: get_or_create creates then returns existing without re-invoking the factory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The typed dropdown sourced the threshold-filtered panel list (>= 0.70 general),
so low-confidence actions/features the model DID predict never appeared — forcing
hand-typed custom tags instead of accepting the model's canonical formatting.
Add a threshold override: SuggestionService.for_image(threshold_override=) and
GET /images/<id>/suggestions?min=<f> surface EVERY stored prediction (down to the
0.05 store floor), alias-resolved and normalized, still excluding applied/rejected
and unsurfaced categories. The suggestions store gains allByCategory + loadAll
(min=0); the dropdown searches that full set (cap 20), while the Suggestions panel
stays curated at the configured threshold. Accept/dismiss drop from both lists.
Operator-asked 2026-06-09. Test: a 0.30 general prediction is hidden by default
but surfaced with threshold_override=0.0; unsurfaced categories still excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A character chip with a fandom only rendered a bare arrow. Surface the fandom
NAME inline, truncated to 15 chars (full name in the tooltip). Resolve the name
via a Tag self-join in both tag paths the modal uses — list_for_image
(/api/images/<id>/tags) and gallery get_image_with_tags
(/api/gallery/image/<id>) — so chips show the fandom on first open and after any
reload. Falls back to the bare arrow when only fandom_id is known. Operator-asked
2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom/chapter exclusions added in fb05c5e only touched find_unused_tags
(the preview SAMPLE). prune_unused_tags re-implemented the predicate inline for
the dry-run COUNT and the live DELETE with only the image_tag + series_page
checks — so the preview showed a safe list of names while the delete removed
every fandom (and chaptered series). Operator-flagged 2026-06-08: real data loss
— assigned fandoms deleted, their characters SET-NULLed.
Extract _unused_tag_conditions() as the single source of truth and use it for
the preview, the count, AND the delete, so they can never diverge again. Added a
prune-commit test asserting the LIVE delete spares a character's fandom and a
chaptered series.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator: 10-frame max-pooled tagging on video produces a lot of noisy tags, and
the sampling burns time/GPU. Drop the VIDEO_ML_FRAMES default to 6 (still env-
overridable). Fewer frames = less per-frame noise into the max-pool and a smaller
frame-sampling budget. Quality/perf of the whole video path is being reviewed
separately.
The task logged nothing and SoftTimeLimitExceeded stringifies to empty, so a
timeout surfaced as a bare 'SoftTimeLimitExceeded()' with no clue which file or
why (operator-flagged 2026-06-08).
- Log start (id/path/mime/bytes/video?), per-phase timing (load_models, video
probe/sample/infer, tag, embed, persist), and a success summary.
- Track a + file ; on SoftTimeLimitExceeded log it and re-raise
SoftTimeLimitExceeded WITH that context (keeps the 'timeout' task_run status
but gives the activity a real error_message: which file, which phase, elapsed).
- On other exceptions, log context then re-raise the ORIGINAL (preserves
autoretry for OSError/DBAPIError/OperationalError).
Now a stuck run names the culprit — most likely a slow video (frame sampling is
up to 10x60s ffmpeg) or a huge image; the phase log will say which.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_unused_tags only excluded tags with image_tag or series_page references, so
it flagged every fandom as 'unused' — fandoms are NEVER applied to images (a
character carries its fandom via tag.fandom_id), and the FK is ondelete=SET NULL,
so deleting one silently strips the fandom off all its characters
(operator-flagged 2026-06-08: artist-OC fandoms showing as unused).
Exclude tags referenced as a character's fandom_id, and (same class of gap) tags
referenced by a series_chapter (an all-placeholder series has chapters but no
pages yet). A genuinely orphaned fandom with no characters is still swept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_create_protective_aliases scanned every image_record's tagger_predictions JSON
(unindexed full scan, ~59k rows) to find the categories a merged-away tag's name
was predicted under. That scan ran inside the merge transaction AFTER it had
locked series_page — on a large library it held that lock for minutes and is what
blocked migration 0040 (and starved the standardization task into its 40-min
timeout).
The scan was redundant: the tagger's tag_to_category map is one-to-one (a name has
exactly one category) and a tag's kind is set from that category when created, so
kind already IS the tagger's category for the name. The scan only ever rediscovered
the kind. Build the single protective alias from src_kind directly — no scan, no
lock-holding slow step in the merge.
Rewrote test_alias_per_observed_prediction_category (which encoded the
can't-actually-happen one-name-two-categories case) → test_protective_alias_uses_tag_kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.
- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
async factory). A blocked merge now raises, the per-group handler rolls back +
counts an error, and the loop continues — one stuck group can't strand the
chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
diagnosable instead of silent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator evidence 2026-06-07: 0.95 was too strict, skipping confident-enough
auto-applications of accepted tags. Newly-accepted tags now allowlist at 0.90;
existing entries keep their stored value and per-tag thresholds stay tunable in
the allowlist table. No migration — min_confidence has no DB server_default, so
the Python insert default governs new rows only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator feedback: thumbnails too small to judge order, no obvious way to mark
'this installment is Part 2', and the permanent two-pane picker was busy and
competed with the ordering work.
- Full-width parts, each a card with a big page grid (150px, contain so whole
pages are visible) and drag-to-reorder; positional page number as a badge.
- Editable Part # (hero field) backed by new series_chapter.stated_part —
separate from the auto-managed chapter_number, mirroring the page_number vs
stated_page split so reorder/delete renumbering can't wipe a hand-set part.
Missing-Part hints when consecutive parts' stated_part jump >1.
- Each part labels its source post (derived from pages' primary_post_id) and
shows the printed-page range with clear labels.
- Picker demoted to an on-demand right slide-over ('Add pages') with a target-
part selector; part actions (move/merge/delete) collapsed into an overflow ⋮.
alembic 0042 adds series_chapter.stated_part (nullable int).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirm-only "this post may continue this series" matcher.
- series_suggestion table (post_id, series_tag_id, score, signals jsonb, status
pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings
knobs (series_suggest_enabled, series_suggest_threshold).
- series_match_service: weighted additive score (title-stem / same-artist /
page-continuity / shared-distinctive-tags), no single signal gating. The title
"pattern" is derived on the fly from the post titles already in a series, so it
sharpens as more are confirmed (no persisted state to drift). Candidates are
bounded to the post's artist. match_post upserts pending suggestions (UNIQUE +
on-conflict, respecting prior added/dismissed decisions).
- accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'.
- rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from
a post-id cursor (maintenance_long lane), like normalize_tags_task.
- API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan.
- Settings: enabled + threshold exposed via /settings/import.
- Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE
dedup.
Frontend (Suggestions tab + settings card) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post-aware on-ramp + the data behind the missing Series browse view.
- page_number_parser: conservative stated-page parser (pages 9-12 / page 5 /
[3/8] / 3 of 8), keyword-gated to avoid false positives. Pure + unit-tested.
- SeriesService.promote_post_to_series: a self-contained post becomes its own
series — series tag named after the post, one chapter, the post's images as
pages (ordered by capture order; stated pages parsed from title/description).
- SeriesService.add_post_as_chapter: append a post as the next chapter of an
existing series, titled after the post and slotted by parsed page number
(a "pages 1-4" post lands ahead of the "pages 9-12" chapter).
- SeriesService.list_series: browse cards — cover thumb, artist, chapter/page
counts, gap flag, last-updated; sort recent|name|size + filter by artist.
- API: GET /api/series, POST /api/series/from-post, POST /api/series/<id>/add-post.
- Resolver uses ImageRecord.primary_post_id (same linkage the posts feed renders).
Frontend (Add-to-series control + Series view + nav) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an ordered chapter layer to series. Reading order becomes
(series_chapter.chapter_number, series_page.page_number); a chapter may be a
placeholder reserving a slot, and carries an optional parsed stated-page range
used to flag missing-page gaps. An image still lives in at most one series ⇒ one
chapter (image_id stays UNIQUE).
- models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) +
stated_page. Migration 0040 backfills every existing series into one
auto-chapter holding its current flat pages — no data loss.
- SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter
assignment, reorder_pages, chapter-aware set_cover; list_pages now returns
chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide
reorder operates on the single default chapter and rejects multi-chapter series.
- API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an
optional chapter_id.
- TagService.merge now repoints series_chapter too, so a merged series' chapters
(and their pages) survive the source tag's deletion instead of cascading away.
- Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to
supply chapter_id.
Frontend (chapter-aware manage view + reader) lands next; until then the
existing UI keeps working via the flat pages[] + single default chapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reextract_archive_attachments loaded ALL PostAttachments and ran in one pass up
to a 30-min soft limit, then died without re-enqueueing — a large archive
backlog would only ever partially process. And a naive re-run can't advance: an
already-extracted archive is still an archive on disk, so it'd re-extract the
same first batch forever.
Give it a real cursor + time-box + self-resume (mirrors normalize_tags_task,
operator-asked 2026-06-07: reasonable timeout, then re-queue so other work keeps
flowing):
- service scans attachments with id > after_id in ascending order, time-boxes
the chunk, and reports partial=True + resume_after_id (last scanned id).
- task passes a 600s budget and re-enqueues itself from the cursor until the
scan is exhausted. Routes on the maintenance_long lane.
- This is independent of the maintenance_long lane isolation (already shipped) —
that stops long tasks starving the quick maintenance queue; this stops the
re-extract itself dying on a big backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The constant + comment landed BETWEEN @celery.task(...) and the function def,
which is a syntax error that broke the whole tasks.admin import (cascaded to
lint E999 + every backend/integration test). Move it above the decorator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two more maintenance-queue failures from the operator's 24h list:
- vacuum_analyze died with "could not resize shared memory segment to 67MB: No
space left on device" — Docker's default /dev/shm is 64MB, too small for
VACUUM (ANALYZE)'s parallel-worker shared memory. Set the postgres service
shm_size: 512m.
- backup_db_task timed out at its 12-min limit once the DB grew; a pg_dump can't
be chunked, so raise it to 30/35 min. (A long backup still briefly holds the
concurrency-1 lane — the structural fix is a dedicated lane for long one-shots.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native client treated a gallery image without `file_name` as schema drift
and raised "Patreon API changed — ingester needs update", failing the whole walk
(operator-flagged 2026-06-07: BlenderKnight post 73665615, kind=images). But the
resource had a valid URL, and the code already derives a filename from the URL
basename right below the raise — the same fallback gallery-dl uses. Patreon
legitimately serves some images without file_name, so this isn't drift.
Drop the require_file_name gate from _media_item: file_name is now optional for
every kind (images/attachments/postfile), falling back to the URL basename.
Genuine drift still raises — no resolvable URL, or a media id referenced by a
relationship but absent from `included`. Test updated to assert the fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).
download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.
Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/posts/mimic-in-dungeon-158372536 is a
single-post permalink, not a creator page — the resolver grabbed "posts" as the
vanity and failed (operator-flagged 2026-06-07). Add a resolution path: extract
the trailing post id and follow it to the owning campaign via the Patreon post
API (/api/posts/<id>?include=campaign), so pasting any post URL subscribes to
that creator's whole feed. `posts/` is excluded from the vanity regex so it
can't masquerade as a creator slug.
Resolution order is now: cached override → id: URL → /posts/<id> → vanity
(campaigns API + creator-page scrape). Tests cover the post→campaign resolve
and that /posts/ URLs aren't treated as vanities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new enabled source is armed for run-until-done backfill (#693) but would sit
idle until the next scheduler tick (~60s). create_source now enqueues the first
walk right away (pending DownloadEvent + download_source.delay), skipping only
when the platform is in a rate-limit cooldown (the scheduler picks it up when
that clears). Disabled sources still don't dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/cw/Atole resolved vanity='cw' — the
vanity regex only skipped a /c/ prefix, so Patreon's current /cw/ ("creator
workspace") form fell through to the bare-vanity branch and captured the prefix
instead of the slug. Every /cw/ source then failed campaign-id resolution
(API + page-scrape both looked up "cw"). Operator-confirmed 2026-06-07 via the
new error text: source_url='.../cw/Atole'; vanity='cw'.
Add cw/ to the optional prefix group (ordered before c/ so the longer prefix
wins), and have the creator-page fallback try the /cw/ form too. Test covers
bare / c/ / cw/ extraction and that id: URLs stay non-vanity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recurring "post shows a zip but no images" report had no diagnostic: when
_import_archive captured an archive as a bare PostAttachment — because the
bomb-guard probe rejected it, or extraction yielded zero members (corrupt /
unsupported / missing extractor backend), or it held only non-media files — it
returned status="attached" silently.
Now those paths set ImportResult.error with the specific reason and log a
warning, and download_service records each as {file, reason} under the event's
metadata.unextracted_archives (None when every archive extracted cleanly). So
the next run names exactly which archives failed and why, instead of leaving the
operator to guess. No behaviour change to the happy path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon's /api/campaigns?filter[vanity]= lookup returns empty data for creators
that plainly exist (operator-flagged 2026-06-06 — Atole etc. erroring at the
resolve step). gallery-dl never used that endpoint; it pulls the campaign id out
of the creator page's bootstrap JSON. Add the same as a fallback: when the API
misses, GET the creator page (bare + /c/ vanity paths) and scrape the first
campaign id from any known embedding ("id":"…","type":"campaign" /
"campaign":{"data":{"id" / /api/campaigns/<id> / "campaign_id"). API is still
tried first (cheap, structured); the page scrape only runs on a miss.
Tests: API-empty → page-scrape fallback resolves; _scrape_campaign_id pattern
coverage. Existing API-path tests unchanged (happy paths short-circuit before
the fallback; failure paths hit the guarded scrape and still return None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Naming/lookup failures now report the source_url, the extracted vanity, and the
exact campaigns-API lookup URL attempted, so a "could not resolve campaign id"
error is diagnosable (wrong vanity? cookie/auth? creator renamed?) instead of
opaque. Applied to all three resolution surfaces: the native download event,
the dry-run preview, and the credential-verify probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name now only capitalizes the first letter of each word and
leaves the rest of the word untouched (was lowercasing the tail, which turned
DC→Dc / NSFW→Nsfw). This matches ml/tag_name._title_word, so a Camie-suggested
tag keeps the exact casing the suggestion UI showed when it round-trips through
POST /api/tags on Accept — addressing "auto-suggested tags must obey
capitalization" and "don't mangle acronyms" in one rule.
Trade-off (operator-chosen): all-caps input no longer folds to Title Case, so
case-variant merging in #714 still folds the dominant lowercase-vs-Title case
but leaves all-caps stylizations distinct (protecting acronyms wins). Tests
updated + a new test documenting acronym preservation / non-folding.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps
whatever casing it was created with. This adds a maintenance action that
Title-Cases every existing tag (collapsing whitespace) and merges
case/whitespace-variant duplicates into one.
Backend:
- tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by
(kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor
(prefer an already-canonical member → no rename/self-alias; else the
best-connected tag → fewest FK repoints; else lowest id), merges the variants
INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/
aliases/series_page repoints + protective ML aliases), then renames the
survivor to canonical. Losers are deleted before the rename so there's no
transient unique-index clash; commits per group and isolates failures per
group. Idempotent — an already-canonical lone tag is a no-op.
- normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async
engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i.
- POST /api/admin/tags/normalize: dry_run=true returns a projection inline
(group/collision/rename counts + sample); dry_run=false enqueues the task.
Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup
tab) — preview → apply (polls the activity dashboard to terminal status),
behind a back-up-first warning. admin store gains normalizeTags().
Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag
dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept
separate, ML-known loser keeps a protective alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tag_name (per-word capitalize + whitespace collapse) is applied in
the POST /api/tags handler so operator-entered tags get clean display casing.
It is NOT applied in the shared find_or_create / rename paths — those are used
by the ML tagger and allowlist matching, which must preserve the booru
vocabulary's original casing (Title-Casing it broke apply_allowlist matching).
find_or_create / rename keep case-insensitive lookup + clash detection so a
differently-cased entry dedups onto the existing tag instead of forking.
Tests updated to expect Title-Cased create output (sunset→Sunset,
character:Saber→Character:saber, http://example.com→Http://example.com) and a
dedicated normalize_tag_name unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.
Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TABLESAMPLE SYSTEM_ROWS reads CONTIGUOUS rows from each sampled page, so
sequentially-imported near-duplicates (multi-image posts, variant sets) came
back adjacent and clustered in the showcase ("three near-identical in a row").
Sample limit*5 rows (spanning more pages) then ORDER BY random() before taking
limit — breaks the physical adjacency for much better spread, still cheap
(random() over a few hundred rows, not the whole table).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).
normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.
Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The integration test caught it: _import_media re-derives the artist by path-walk
from the attribution path (ignoring the explicit artist) AND _copy_to_library
lands members relative to that path. Staging the archive in /tmp meant the
artist didn't resolve (provenance skipped) and members would land in the temp
dir. Stage under images_root/<slug>/<platform>/<post>/ instead so the artist
resolves and members land in the real library; remove only the staged archive +
sidecar afterward (members stay). Require a real artist+slug (skip + count
otherwise).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.
Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.
Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").
Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.
Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen