Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):
- utils.paths.filehash_from_url — one source of truth for the extractor;
patreon_client._filehash now delegates so capture- and render-time
hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
every inline image whose filehash maps to a stored image of THIS
artist; unmatched / pre-Phase-2 images keep hotlinking.
Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.
Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.
Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.
Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.
Completes Phase 4. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).
- _route_files now mirrors download_service._phase3_persist branch-for-branch:
imported/superseded → collect member_image_ids+image_id (provenance-linked via
the synthesized sidecar, same as extracted-zip members) → caller enqueues
tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
fetch result (files/bytes) or fetch failure, per-file import decision, dead-
letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
exactly the member images.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.
NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.
With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
requeue-with-countdown if busy), fetch via external_fetch into the artist
library tree, then route each file through importer.attach_in_place via a
synthesized sidecar so it links to the SAME post (archive→ImageRecords,
else→PostAttachment; on-disk original removed for captured files, art stays);
thumbnail+ML enqueue for new images; status downloaded | failed | dead with
attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
columns in 4d — defaults on, rule #26).
Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.
mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.
- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
/attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
(_apply_sidecar + upsert_post_record) so a re-import never resets a link's
status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
no-dup on reimport), serializer.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as
plain text. Now:
- html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings,
inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay
stripped; their text is preserved). Benefits the existing ProvenancePanel too.
- post_feed_service.get_post: add sanitized `description_html` to the DETAIL
response (the feed list stays lightweight plain text by design).
- PostCard.vue: render description_html via v-html once expanded (fetched with
detail); collapsed + no-detail fallback stay plain text. Styled close to the
source (headings, images max-width, accent links, lists, quotes, code).
Tests: sanitizer (headings/img/lists survive, img javascript: src dropped);
get_post returns sanitized description_html.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test_download_service stubs build dl_result as a SimpleNamespace that
doesn't set the new field; read it via getattr (matching the existing
retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any
caller that predates the field.
Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.
- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).
Slice 0b of milestone #64. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).
First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Series tab strip and the Browse search/sort (and Suggestions controls)
scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab
controls into one sticky header pinned under the 64px TopNav. The controls
had to leave v-window — it clips sticky children — so they're driven by the
tab from the header instead of living inside each window-item.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The headline bug: aliases created from the modal NEVER resolved. Create
sent the normalized display name ('Sword', 'Uchiha Sasuke') while
resolution keys on the raw booru model key ('sword', 'uchiha_sasuke',
case-sensitive) — so the mapping was stored under a key nothing looks up,
and the prediction kept reappearing unaliased. The raw key wasn't even in
the /suggestions response, so the modal couldn't send it.
- Suggestion now carries raw_name (the model key an alias must use) and
via_alias (surfaced via an operator alias); both serialized by the API.
- Modal alias-create sends raw_name, not display_name (the fix). Aliased
suggestions show an 'alias' badge and a 'Remove alias' action; 'Treat as
alias for…' is hidden for centroid hits (no model key) and already-aliased
rows.
- Tag-side management: TagCard ⋮ → 'Aliases…' opens a dialog listing the
model keys that fold into a tag, with remove (GET /api/tags/<id>/aliases +
AliasService.list_for_tag). Creation stays in the modal suggestion flow.
Tests: full API round-trip locking the raw-key contract (raw_name exposed →
alias authored with it → resolves + via_alias on a later image);
list_for_tag (service + API); via_alias/raw_name on the existing service
suggestion tests. No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An image whose on-disk path contains '#' (post folders like 'BLUE#59')
served its hash-named thumbnail fine but 404'd the original: the unencoded
'#' in image_url was parsed by the browser as a URL fragment, so
'#59/01_timelapse.jpg' never reached the /images route. Add a shared
image_url(path) helper that percent-encodes the path (safe='/') and route
the 3 raw builders (gallery detail + 2 in series) through it. Not a
cleanup-tool deletion — the file is on disk; only the URL was wrong.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-asked: the tab strip and search field were stacked; place them
side-by-side in a single flex bar (tabs left, search + scope chips right),
wrapping to two rows only on narrow viewports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Browse tab nav scrolled away (operator didn't know it existed) and
Posts had no search. Roll the tab strip + a shared search field into one
sticky block pinned under the 64px TopNav.
- Posts gains server-side text search: PostFeedService.scroll()/around()
+ /api/posts accept q (ILIKE over post_title OR description), applied
INSIDE the artist/platform WHERE so search stays scoped to the active
filter. Scope shown as clearable chips next to the search field.
- Artists/Tags search consolidates into the sticky bar: their inner
search boxes are removed; they react to route.query.q (q is deep-
linkable, e.g. /browse?tab=posts&q=foo). Platform/kind filters stay.
- Posts empty state now distinguishes 'no matches' from 'no posts yet'.
Tests: posts q-search matches title|description and stays artist-scoped
(service); q passthrough (api).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tinted backing was set on the square .fc-kebab wrapper span while the
button is round, so a translucent square showed behind the round ⋮.
border-radius:50% makes the backing a circle matching the button.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
add_post now stamps the post's parsed START (constant) on every staged
pending page so the group start survives junk removal; list_pages
surfaces it as start_page. Update the stale per-page [9,10,11] assertion
to check grp["start_page"] == 9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the auto-renumbered 1..N position key with operator-OWNED page
numbers: sparse, gaps allowed, editable, never auto-renumbered. Order follows
the numbers; unnumbered pages sort to the tail. This is the fix for the model
that clobbered hand-set numbers on the flatten — numbers are now data, not a
derived sequence.
- series_service: drop the renumber-on-reorder/remove; order by page_number
NULLS LAST; new set_page_number(image_id, n|None); list_pages returns `gaps`
(one entry per missing-number run) + each pending group's parsed `start_page`;
set_cover renumbers below the current min; place_pending(image_ids, start_page)
numbers placed pages sequentially from the start (drop junk first → numbers
line up); add_post stamps the parsed start on staged pages.
- api/tags: POST /series/<id>/pages/number (set one page's number); /pending/
place takes start_page; removed /reorder.
- frontend: per-card editable number input; one gap block per gap with
drop-on-edge to assign the adjacent number (middle → type); append drop zone;
pending tray gets a "from page N" field + "Place from page N".
- tests reworked: sparse numbers + gaps, place-from-start, set-page-number route.
No migration; nothing destructive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add-from-post no longer appends straight into the run — it STAGES the post's
pages as pending (per-page status; page_number NULL), grouped by source post,
so the operator drops junk (text-free alts, bumpers) and places the keepers
into the sequence with clean series-global numbering.
- migration 0048: series_page.status ('placed' default | 'pending') + nullable
page_number.
- series_service: placed/pending split everywhere (list_pages returns the
placed run + a `pending` section grouped by source post; reorder/cover/
list_series operate on placed only); add_post stages pending; new
place_pending(image_ids, before_image_id=None) flips pending→placed spliced
before a page (or appended) and renumbers; junk removal reuses remove_images.
- api/tags: /add-post now returns staged count; new POST /series/<id>/pending/
place.
- frontend: PostSeriesMenu navigates to the series after staging; seriesManage
store surfaces `pending` + placePending; SeriesManageView gains a pending
tray (per-post groups, place-all / place-one / drop-junk).
- tests: pending staging, place (append + insert-before), ignore-already-
placed, drop-junk, route guard; updated add_post + match-accept expectations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframe a series from "ordered chapters that own pages" to ONE flat,
series-global ordered run of pages with optional cosmetic chapter DIVIDERS
over it. A chapter no longer wraps content — it's a labeled divider anchored
to the page that begins it; a page's chapter is derived as the nearest
preceding divider. This is what lets installments assembled from multiple
sources sit in one continuous, correctly-numbered sequence (operator's
Goblin Juice case).
- migration 0047: flatten each series to a series-global page_number
(preserving today's reading order); convert each existing chapter to a
divider anchored at its first page (keeping title/stated_part); drop
series_page.chapter_id; reshape series_chapter (anchor_page_id UNIQUE FK,
drop chapter_number/is_placeholder/stated_page_start/end). Loss-safe for
content; drops empty placeholder chapters + a redundant page-1 divider.
- series_page: page_number is now the series-global order; no chapter_id.
- series_chapter: anchored divider (anchor_page_id, title, stated_part).
- series_service: flat list_pages (one run + derived dividers + per-page
source_post + part_gaps), series-wide reorder/renumber, divider CRUD
(create/update/move/delete); retired per-chapter reorder/merge/placement.
- api/tags: drop chapter_id from add; /chapters endpoints are divider
create/update/delete (removed chapter reorder/merge/page-reorder).
- series_match_service: series "end" reads max(series_page.stated_page);
accept appends via add_post. tag_service series-merge appends src's pages
after tgt's max so the merged series stays one clean run.
- frontend: seriesManage store + SeriesManageView → one continuous
drag-reorder grid with inline divider bars + series-global page numbers;
reader walks the flat run, headings from dividers; PostSeriesMenu copy.
- tests reworked across the series suite for the divider model.
Phase 2 (pending staging for add-from-post) is separate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline INSERT…SELECT backfill in migration 0045 wrapped the table
creation and a ~100 GB pass over image_record.tagger_predictions in one
transaction: nothing committed until the end, it was unmonitorable, and an
earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A
deploy got stuck on it for ~2h with image_prediction never appearing.
Split the concerns:
- 0045 now creates ONLY the table + indexes (instant DDL → web boots).
- New backend.app.tasks.admin.backfill_image_predictions_task copies the
>= store-floor predictions from the JSON into image_prediction, batched by
id window and committed per chunk: live progress, resumable (re-enqueues
from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each
stays in the DB executor streaming each window — no Python-side 100 GB load,
no materialization.
- POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card
to trigger the one-time run after upgrading.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MATERIALIZED-CTE scalar guard forced Postgres to materialize all object
rows with their full JSON (~100 GB) to temp before json_each — on NFS that's a
huge spill and pathologically slow (risks disk-full). Replace with an inline
CASE that feeds json_each an empty object for non-object rows: same scalar
guard, but a single streaming pass with no materialization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some image_record rows store tagger_predictions as a JSON scalar/null rather
than an object; json_each throws 'cannot deconstruct a scalar' on those,
rolling back the whole migration. Filter to json_typeof = 'object' in a
MATERIALIZED CTE so the guard runs before json_each ever evaluates a scalar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #764 in-place prune (rewrite tagger_predictions to >=0.70) is too slow on
100 GB of TOAST and fails at its soft limit (interrupts a query mid-flight ->
'another command is already in progress'). #768 supersedes it: extract only
the >=floor predictions into image_prediction via this set-based backfill,
then drop the column (step 3) — reading 100 GB once + writing ~840k small rows
beats rewriting 100 GB in place.
So this backfill no longer assumes the prune ran: it filters by
ml_settings.tagger_store_floor (default 0.70) itself, handling the full or
partially-pruned JSON identically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch every prediction READER off the JSON column onto the normalized
image_prediction table. Parity by construction: each reader loads the same
{raw_name: {category, confidence}} dict it consumed before (via small
_load_predictions helpers), so all downstream threshold/alias/merge/consensus
logic is byte-identical — only the data source changed.
- suggestions.SuggestionService.for_image (and for_selection via it)
- ml.apply_allowlist_tags (iterates images that have prediction rows)
- importer re-import reset deletes the image's prediction rows
The tagger_predictions JSON column is still dual-written (step 1) so it stays
valid during transition; the backfill task's NULL check still works. Removing
the JSON write + DROP column + retiring the #764 prune is the cleanup
follow-up (needs a quiesced-worker window for the DROP lock).
Tests: shared tests/_prediction_helpers.seed_predictions seeds the table;
read-path tests (suggestions, bulk consensus, allowlist apply, API) seed there
instead of ImageRecord.tagger_predictions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Normalize tagger predictions out of the image_record.tagger_predictions JSON
blob into a queryable per-prediction table. Step 1 of the cutover (expand):
additive + low-risk — reads still use the JSON, this just adds the table and
keeps it populated.
- ImagePrediction(image_record_id, raw_name, category, score) — stores the
RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution
is unchanged. Indexed for per-image reads + by (raw_name, score).
- Migration 0045: create table + set-based backfill from the JSON via
json_each (fast post-#764-prune). The old column stays (vestigial) and is
dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the
hot image_record table, so it waits for a quiesced-worker window.
- tag_and_embed dual-writes the rows (delete-then-insert, idempotent);
tagger_store_floor already applied in infer().
Next: switch suggestion + allowlist reads to the table, then drop the JSON
write. Plan-task #768.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DB backup polish (plan-task #764 Q3):
- pg_dump now uses custom format (-Fc): compressed (much smaller on NFS) and
restored via pg_restore. Artifact extension .sql → .dump; restore_db swaps
psql -f for pg_restore -d. BackupRun.sql_path field name kept (it's just the
db artifact path).
- Reconcile the subprocess guardrails: the DB timeout was 720s with a stale
'Celery soft is 10 min' comment, but backup_db_task's soft limit is actually
1800s — so the bounded-kill fired 18 min early. Set DB=1700s / images=21000s,
each just under its task's Celery soft limit so _run_bounded stays the
primary guard (an NFS D-state hang defeats Celery's own SIGKILL).
Real shrink of the DB is the #764 prune; this makes each dump smaller/faster
on top of that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-time backfill that actually shrinks the DB: drops stored
tagger_predictions entries below ml_settings.tagger_store_floor from every
image_record row, and clamps any allowlist min_confidence below the floor up
to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate
so backfilled rows match new imports. Keyset by id ASC, idempotent,
self-resumes on the soft time limit; runs on the maintenance_long lane.
pg_dump copies live data only, so this alone fixes the #739 backup timeout —
the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional
disk-return step, brief because post-prune the live data is tiny.
- admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions
- PrunePredictionsCard in the Maintenance panel (shows the current floor)
- tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consumer #4 of the store-floor change (#764). An allowlist tag can't
auto-apply more permissively than the ingest floor — predictions below
tagger_store_floor aren't stored, so a lower min_confidence behaves
identically to the floor. update_threshold now clamps to max(value, floor);
the AllowlistTable confidence input min-binds to the live floor and clamps
on edit. Keeps the stored threshold honest about actual apply behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default
0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70).
Storing every tag down to 0.05 from a ~10k-tag tagger is what grew
image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70
and the centroid/learned path covers lower-confidence preferred tags, so the
sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim
land next; this only changes the write gate for NEW imports).
- ml_settings.tagger_store_floor (migration 0044, default 0.70)
- tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor
- ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold
below the floor (nothing below the floor is stored, so the gap surfaces
nothing) — server backstop for the UI slider clamp
- Settings → ML: store-floor slider + caption; category sliders min-bound to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recover_stalled_task_runs used the 5-min default for the download queue,
but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT
(1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as
phantom 'RecoverySweep' failures — visible in System Activity but absent
from the Subscriptions view (the download finished ok, reset the source's
consecutive_failures; only the orphaned task_run kept the stamp, since
_finalize only updates rows still 'running').
Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard
limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real
hard kill is swept by the task-run and event sweeps together. Restores the
documented invariant (every override >= task time_limit). Regression test
pins the threshold above the hard limit so a future limit bump can't
silently re-break it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recent failures gains a client-side search over the already-loaded 24h
rows (task/queue/target/error), shown as a filtered/total count alongside
the existing error-type chips. All recent activity gains a debounced
server-side task-name search (new `task` ILIKE param on /runs) so it
spans the full history, not just the loaded page. LIKE wildcards are
escaped so task names' literal underscores match literally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom self-join (resolve a character's fandom NAME via Tag.fandom_id->Tag)
and the {id,name,kind,fandom_id,fandom_name} dict were hand-written in
TagService.autocomplete/.list_for_image, GalleryService.get_image_with_tags and
the api/tags handlers — the last few grown by this session's fandom-on-chip
feature. Consolidate to services/tag_query: fandom_join_alias() + tag_columns()
build the select; serialize_tag(row) builds the dict. Now a new tag field is
added in one place.
Over-DRY guard: TagDirectoryService selects the full Tag ORM + an image-count
aggregate (a different select shape) — left as its own variant. §8b: the
fandom_lookup alias lives only in tag_query; gallery + both api/tags handlers
serialize via serialize_tag. Test: serialize_tag handles enum + string kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
encode_cursor/decode_cursor (base64 <iso8601>|<id>) were defined identically in
gallery_service AND post_feed_service, with artist_service importing gallery's
copy. Two implementations of one cursor format silently break pagination in
whichever feed drifts. Extract to services/pagination.py; gallery/post_feed/
artist all import it. Dropped now-unused base64/datetime imports.
§8b: encode_cursor/decode_cursor now defined only in pagination.py. Existing
cursor round-trip tests still cover it via the re-export. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI lint flagged UP047 — use the native generic syntax def get_or_create[T](...)
instead of typing.TypeVar on Python 3.14.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The find-or-create dance — SELECT, then a SAVEPOINT INSERT that recovers (not a
full rollback) on IntegrityError when a concurrent worker inserted first — was
hand-rolled identically in 4 async sites: ArtistService.find_or_create,
TagService.find_or_create, ExtensionService._find_or_create_artist and
._find_or_create_source. Divergent copies of exactly this pattern are how the
duplicate-row/race bugs in reference_scalar_one_or_none_duplicates crept in, so
it now lives once in services/db_helpers.get_or_create (returns (row, created);
factory adds+flushes+returns the row; caller owns the outer commit).
Over-DRY guard: SourceService's IntegrityError sites RAISE DuplicateSourceError
(reject-on-conflict, a different concept) — left alone. Importer._get_or_create
is the lone SYNC consumer (already shared by 2 callers) — stays separate, can't
cross the sync/async boundary. §8b: no hand-rolled async find-or-create remains.
Test: get_or_create creates then returns existing without re-invoking the factory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The icon+title v-card-title heading (d-flex align-center + gap + <v-icon size=small> +
<span>) was hand-rolled identically in 13 cards/dialogs (15 heading instances).
Consolidate to <CardHeading icon title> (components/common) with an iconColor
prop (error headings) and a default slot for trailing content (spacer+actions,
inline status chip). Adopted everywhere the pattern appears — all-or-nothing per
the hardened DRY process.
Over-DRY guard: plain text-only <v-card-title> one-liners are NOT this pattern
and stay; DownloadDetailModal leads with a status CHIP (not an icon), a different
concept, left alone. §8b: the only remaining d-flex align-center v-card-title is
that intentional variant. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The preview sample-name grid (scrollable monospace chip grid) was hand-rolled
5 times with verbatim-duplicated markup + CSS — TagMaintenanceCard (×4) and
PostMaintenanceCard. Consolidate to <SampleNameGrid> (components/common): pass
:names for the plain case, default slot for the normalize from→to chips
(styled via :slotted .fc-name). Removed the duplicated .fc-name-grid/.fc-name
CSS from both cards.
Over-DRY guard: only the verbatim-duplicated grid is merged — each card's
preview/commit logic and result-count lines genuinely differ and stay put;
MinDimensionCard's typed-token confirm is a separate variant, untouched.
§8b: fc-name-grid now lives only in SampleNameGrid. Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The management view showed the series name but had no way to change it post-
creation (rename was only on the browse-card kebab). Add a pencil next to the
title that opens TagRenameDialog (reuses the canonical rename → PATCH
/api/tags/<id> with its collision→merge flow, since a series IS a
Tag(kind=series)); the new name reflects in place. Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The muted-text token was redefined identically in 12 component <style scoped>
blocks. Consolidate to one global utility in styles/app.css; remove the 12
copies. Keeps the explicit on-surface-variant (vellum) token, NOT Vuetify's
opacity-based text-medium-emphasis (per the muted-text-token rule). Behavior-
preserving: every class=fc-muted usage now resolves to the single source.
§8b exhaustiveness caught (and I fixed) my own sed clobbering the new app.css
rule — now exactly one .fc-muted definition exists, zero component-local.
Catalog updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First pattern-consistency DRY pass (process #594). The overflow kebab was
hand-rolled 7 ways in two divergent activator strategies — Pattern A
(#activator + v-bind) which silently breaks inside the teleported image modal
(#711), and Pattern B (manual v-model + activator=parent + open-on-click=false +
z-index 2400) the modal kebabs needed as a workaround.
New <KebabMenu> (components/common) bakes in the modal-safe strategy
UNIVERSALLY, so every kebab works in modal and non-modal contexts — folding the
latent #711-class bug fix into all five Pattern-A sites. Menu items go in the
default slot; variations (size/variant/location/label/min-width) are props.
Adopted across all 7: TagChip, SuggestionItem, TagCard, SeriesView card,
SeriesManageView, BackupRunsTable, SourceActions. Exhaustiveness (§8b):
mdi-dots-vertical now lives only in KebabMenu. Labeled dropdowns / nav menus /
filter popovers are a different concept and left alone. Seeded the pattern
catalog so new code reuses the primitive. Test: KebabMenu renders slot items +
trigger label/glyph + presentational props.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Posts, Artists, and Tags are the three 'browse the library by an axis'
surfaces; Subscriptions stays purely management (operator-asked 2026-06-09).
New BrowseView renders them as tabs (?tab=posts|artists|tags); only the active
tab mounts. The old standalone paths become redirects into the matching tab,
preserving deep-link query (/posts?post_id=N → /browse?tab=posts&post_id=N) and
keeping the route names so existing { name: 'posts'|'artists'|'tags' } links and
path pushes still resolve. Nav now reads Showcase · Gallery · Browse · Series ·
Subscriptions, with Settings pinned right.
Test: /browse resolves; /tags and /artists redirect into their tabs; a posts
deep link survives the redirect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Settings is configuration, not content, but sat mid-row (between Series and
Posts). Pull it out of the centered content links and pin it to the right as a
gear+label, matching the convention that config lives at the right edge. Mobile
is unchanged — Settings stays in the hamburger menu (navRoutes still includes
it). Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The typed dropdown sourced the threshold-filtered panel list (>= 0.70 general),
so low-confidence actions/features the model DID predict never appeared — forcing
hand-typed custom tags instead of accepting the model's canonical formatting.
Add a threshold override: SuggestionService.for_image(threshold_override=) and
GET /images/<id>/suggestions?min=<f> surface EVERY stored prediction (down to the
0.05 store floor), alias-resolved and normalized, still excluding applied/rejected
and unsurfaced categories. The suggestions store gains allByCategory + loadAll
(min=0); the dropdown searches that full set (cap 20), while the Suggestions panel
stays curated at the configured threshold. Accept/dismiss drop from both lists.
Operator-asked 2026-06-09. Test: a 0.30 general prediction is hidden by default
but surfaced with threshold_override=0.0; unsurfaced categories still excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A character chip with a fandom only rendered a bare arrow. Surface the fandom
NAME inline, truncated to 15 chars (full name in the tooltip). Resolve the name
via a Tag self-join in both tag paths the modal uses — list_for_image
(/api/images/<id>/tags) and gallery get_image_with_tags
(/api/gallery/image/<id>) — so chips show the fandom on first open and after any
reload. Falls back to the bare arrow when only fandom_id is known. Operator-asked
2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Series browse tab had no way to find a series in a long grid and no
per-series actions. Add a search field (instant client-side name/artist filter
over the already-loaded list) and a kebab on each card with Rename (reuses
TagRenameDialog → PATCH /api/tags/<id>, with its collision-merge flow) and
Delete (confirm dialog → DELETE /api/admin/tags/<id>; series_page/chapter/
suggestion cascade, images kept). Gap badge moved to the cover's top-left so the
kebab can sit top-right. Operator-asked 2026-06-09.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accepting an auto-suggested tag (Suggestions panel or the autocomplete
dropdown) left focus on <body>, so the operator had to re-click the tag field
to add the next one. Expose TagAutocomplete.focus (the existing mobile-aware
focusInput) and call it after accept from both paths; SuggestionsPanel emits
'accepted' for the parent to refocus. Operator-asked 2026-06-08.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The character pointing at the fandom had no image associations, so it was
itself unused and inflated the dry-run count to 2. Tag it on a real image so
it is used (the real-world shape) — the fandom survives via a live character.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom/chapter exclusions added in fb05c5e only touched find_unused_tags
(the preview SAMPLE). prune_unused_tags re-implemented the predicate inline for
the dry-run COUNT and the live DELETE with only the image_tag + series_page
checks — so the preview showed a safe list of names while the delete removed
every fandom (and chaptered series). Operator-flagged 2026-06-08: real data loss
— assigned fandoms deleted, their characters SET-NULLed.
Extract _unused_tag_conditions() as the single source of truth and use it for
the preview, the count, AND the delete, so they can never diverge again. Added a
prune-commit test asserting the LIVE delete spares a character's fandom and a
chaptered series.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-flagged (again) on the tag-merge picker: Enter on the dropdown re-opens
it instead of accepting the selection. I'd already patched this twice (fandom
picker + fandom set dialog) with copy-pasted capture-phase handlers, so DRY it.
New composable useAcceptOnEnter(accept): tracks the menu state and, on a
capture-phase Enter, lets Vuetify pick when the menu is open but calls accept()
(and blocks the re-open) when it's closed. Applied to every confirm-style picker:
- TagsView merge-into picker (the reported one)
- AliasPickerDialog
- PostSeriesMenu add-to-existing
- FandomPicker + FandomSetDialog (refactored off their bespoke handlers)
One behavior, one place to change it.
Operator: 10-frame max-pooled tagging on video produces a lot of noisy tags, and
the sampling burns time/GPU. Drop the VIDEO_ML_FRAMES default to 6 (still env-
overridable). Fewer frames = less per-frame noise into the max-pool and a smaller
frame-sampling budget. Quality/perf of the whole video path is being reviewed
separately.
The task logged nothing and SoftTimeLimitExceeded stringifies to empty, so a
timeout surfaced as a bare 'SoftTimeLimitExceeded()' with no clue which file or
why (operator-flagged 2026-06-08).
- Log start (id/path/mime/bytes/video?), per-phase timing (load_models, video
probe/sample/infer, tag, embed, persist), and a success summary.
- Track a + file ; on SoftTimeLimitExceeded log it and re-raise
SoftTimeLimitExceeded WITH that context (keeps the 'timeout' task_run status
but gives the activity a real error_message: which file, which phase, elapsed).
- On other exceptions, log context then re-raise the ORIGINAL (preserves
autoretry for OSError/DBAPIError/OperationalError).
Now a stuck run names the culprit — most likely a slow video (frame sampling is
up to 10x60s ffmpeg) or a huge image; the phase log will say which.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_unused_tags only excluded tags with image_tag or series_page references, so
it flagged every fandom as 'unused' — fandoms are NEVER applied to images (a
character carries its fandom via tag.fandom_id), and the FK is ondelete=SET NULL,
so deleting one silently strips the fandom off all its characters
(operator-flagged 2026-06-08: artist-OC fandoms showing as unused).
Exclude tags referenced as a character's fandom_id, and (same class of gap) tags
referenced by a series_chapter (an all-placeholder series has chapters but no
pages yet). A genuinely orphaned fandom with no characters is still swept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_create_protective_aliases scanned every image_record's tagger_predictions JSON
(unindexed full scan, ~59k rows) to find the categories a merged-away tag's name
was predicted under. That scan ran inside the merge transaction AFTER it had
locked series_page — on a large library it held that lock for minutes and is what
blocked migration 0040 (and starved the standardization task into its 40-min
timeout).
The scan was redundant: the tagger's tag_to_category map is one-to-one (a name has
exactly one category) and a tag's kind is set from that category when created, so
kind already IS the tagger's category for the name. The scan only ever rediscovered
the kind. Build the single protective alias from src_kind directly — no scan, no
lock-holding slow step in the merge.
Rewrote test_alias_per_observed_prediction_category (which encoded the
can't-actually-happen one-name-two-categories case) → test_protective_alias_uses_tag_kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverses the advisory-lock approach (7309d1d) — it treated a replica race that
wasn't the cause and added a new indefinite-hang mode (a sibling/stale migrator
holding the xact lock).
Real cause of the 0040 hang (operator-diagnosed 2026-06-07): web has always been
a single replica. The migration's ALTER series_page queued behind a concurrent
tag-merge that held a series_page lock for minutes — _do_merge repoints
series_page then runs _create_protective_aliases, an unindexed full scan of
image_record (JSON column, ~59k rows). Migrations ran with no lock_timeout, so
the DDL hung indefinitely and silently.
Fix: SET lock_timeout (default 30s, env-overridable) on the migration connection
before alembic's transaction. A blocked DDL now fails fast with 'canceling
statement due to lock timeout'; the entrypoint exits non-zero so the deploy
retries / surfaces loudly instead of wedging. General protection for every
future migration. (The slow _create_protective_aliases scan — the actual lock
holder — is the separate perf fix still under discussion.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.
- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
async factory). A blocked merge now raises, the per-group handler rolls back +
counts an error, and the loop continues — one stuck group can't strand the
chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
diagnosable instead of silent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image-modal tag kebab's rename dialog still showed a leftover stub
('Merging two tags into one lands in FC-2c') on a name collision, dead-ending
the operator. The merge machinery has existed for a while — the Tags view
already resolves rename collisions this way. Wire TagRenameDialog to it: on the
409 collision hint, show the same merge confirmation FandomSetDialog uses
(target name, image associations moved, alias kept) and POST /api/tags/<id>/merge
into the existing tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator evidence 2026-06-07: 0.95 was too strict, skipping confident-enough
auto-applications of accepted tags. Newly-accepted tags now allowlist at 0.90;
existing entries keep their stored value and per-tag thresholds stay tunable in
the allowlist table. No migration — min_confidence has no DB server_default, so
the Python insert default governs new rows only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'Fandom for <character>' dialog (FandomSetDialog) used plain autofocus and
had no Enter handling, so Enter re-opened the dropdown instead of submitting —
the same bug FandomPicker already fixed. Mirror that flow: parent v-dialogs
focus the field via @after-enter→focusSearch (reliable past the focus-trap);
capture-phase Enter Saves the changed selection instead of re-opening the menu;
Tab jumps to the new-fandom field; creating a fandom returns focus to the
dropdown so a single Enter saves it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator feedback: thumbnails too small to judge order, no obvious way to mark
'this installment is Part 2', and the permanent two-pane picker was busy and
competed with the ordering work.
- Full-width parts, each a card with a big page grid (150px, contain so whole
pages are visible) and drag-to-reorder; positional page number as a badge.
- Editable Part # (hero field) backed by new series_chapter.stated_part —
separate from the auto-managed chapter_number, mirroring the page_number vs
stated_page split so reorder/delete renumbering can't wipe a hand-set part.
Missing-Part hints when consecutive parts' stated_part jump >1.
- Each part labels its source post (derived from pages' primary_post_id) and
shows the printed-page range with clear labels.
- Picker demoted to an on-demand right slide-over ('Add pages') with a target-
part selector; part actions (move/merge/delete) collapsed into an overflow ⋮.
alembic 0042 adds series_chapter.stated_part (nullable int).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every web replica runs 'alembic upgrade head' in its entrypoint, so under
docker stack deploy two replicas can boot at once and race the same DDL —
0040 raced in prod (operator-flagged 2026-06-07): one backend wedged on the
series_page lock while a second tried to re-CREATE series_chapter, and the
loser died with AdminShutdown, crash-looping the web service.
Wrap run_migrations() in a transaction-scoped pg_advisory_xact_lock acquired
BEFORE the version table is read. The first replica to reach it migrates and
holds the lock for the whole upgrade; siblings block, then find the version
already at head and apply nothing. Works regardless of replica count and
needs no Swarm depends_on ordering (which stack deploy ignores anyway).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.3 with the UI.
- SeriesView gains tabs: Browse (the existing grid) + Suggestions.
- Suggestions tab: pending matches as rows (post → series, per-signal strength
chips, score), Add (→ chapter) / Skip (→ dismiss); a "Matching on" toggle and
a threshold field (both DB-backed via /settings/import), and a Rescan button
that enqueues the background matcher.
- seriesSuggestions store wires load / accept / dismiss / rescan / settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirm-only "this post may continue this series" matcher.
- series_suggestion table (post_id, series_tag_id, score, signals jsonb, status
pending|added|dismissed, UNIQUE(post,series)); migration 0041 + two settings
knobs (series_suggest_enabled, series_suggest_threshold).
- series_match_service: weighted additive score (title-stem / same-artist /
page-continuity / shared-distinctive-tags), no single signal gating. The title
"pattern" is derived on the fly from the post titles already in a series, so it
sharpens as more are confirmed (no persisted state to drift). Candidates are
bounded to the post's artist. match_post upserts pending suggestions (UNIQUE +
on-conflict, respecting prior added/dismissed decisions).
- accept reuses add_post_as_chapter then marks 'added'; dismiss marks 'dismissed'.
- rescan_series_suggestions_task: settings-gated, time-boxed + self-resuming from
a post-id cursor (maintenance_long lane), like normalize_tags_task.
- API: GET /series/suggestions, POST .../<id>/accept|dismiss, POST .../rescan.
- Settings: enabled + threshold exposed via /settings/import.
- Tests: pure scoring helpers + matcher/accept/dismiss/rescan lifecycle + UNIQUE
dedup.
Frontend (Suggestions tab + settings card) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.2 with the UI.
- PostSeriesMenu: a "Series ▾" control on each post card — "New series from
this post" (promote → navigates to manage) and "Add to existing series…"
(dialog with a browsable picker loaded from GET /api/series, client-side
filtered — avoids the empty-autocomplete #712 trap).
- SeriesView (/series): a top-level Series browse grid — cover, name, artist,
chapter/page counts, gap badge; sort recent|name|size; cards → manage/read.
meta.title adds it to the nav automatically (peer of Posts).
- seriesBrowse store for the list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post-aware on-ramp + the data behind the missing Series browse view.
- page_number_parser: conservative stated-page parser (pages 9-12 / page 5 /
[3/8] / 3 of 8), keyword-gated to avoid false positives. Pure + unit-tested.
- SeriesService.promote_post_to_series: a self-contained post becomes its own
series — series tag named after the post, one chapter, the post's images as
pages (ordered by capture order; stated pages parsed from title/description).
- SeriesService.add_post_as_chapter: append a post as the next chapter of an
existing series, titled after the post and slotted by parsed page number
(a "pages 1-4" post lands ahead of the "pages 9-12" chapter).
- SeriesService.list_series: browse cards — cover thumb, artist, chapter/page
counts, gap flag, last-updated; sort recent|name|size + filter by artist.
- API: GET /api/series, POST /api/series/from-post, POST /api/series/<id>/add-post.
- Resolver uses ImageRecord.primary_post_id (same linkage the posts feed renders).
Frontend (Add-to-series control + Series view + nav) lands next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes FC-6.1: the series management UI now works in chapters.
- SeriesManageView: chapters as cards (inline-rename, stated-page range inputs,
move up/down, merge-into-previous, delete, pick-as-add-target), pages
drag-reorder WITHIN a chapter, a "gap: N-M missing" badge between chapters
with a stated-page hole, and Add chapter / Add placeholder. The picker adds
the selection into the targeted chapter.
- seriesManage store: chapter CRUD + reorderChapters/moveChapter/mergeChapter/
reorderPages actions; consumes chapters[]/gaps[]; addSelected targets a chapter.
- Reader: page_number is now within-chapter, so anchors switched to a global
`seq` (reading-order position) — fixes scroll/jump/active collisions across
chapters — plus chapter-title dividers at each chapter boundary.
- Updated seriesManage.spec to the chaptered store shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an ordered chapter layer to series. Reading order becomes
(series_chapter.chapter_number, series_page.page_number); a chapter may be a
placeholder reserving a slot, and carries an optional parsed stated-page range
used to flag missing-page gaps. An image still lives in at most one series ⇒ one
chapter (image_id stays UNIQUE).
- models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) +
stated_page. Migration 0040 backfills every existing series into one
auto-chapter holding its current flat pages — no data loss.
- SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter
assignment, reorder_pages, chapter-aware set_cover; list_pages now returns
chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide
reorder operates on the single default chapter and rejects multi-chapter series.
- API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an
optional chapter_id.
- TagService.merge now repoints series_chapter too, so a merged series' chapters
(and their pages) survive the source tag's deletion instead of cascading away.
- Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to
supply chapter_id.
Frontend (chapter-aware manage view + reader) lands next; until then the
existing UI keeps working via the flat pages[] + single default chapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prod stack runs under Docker Swarm (docker stack deploy), which SILENTLY
IGNORES `shm_size` — container inspect showed ShmSize still 64MB after the
a183be7 fix, and vacuum_analyze kept hitting DiskFull resizing a ~64MB POSIX
DSM segment in /dev/shm (operator-flagged 2026-06-07). Replace the ignored
`shm_size: 512m` with a tmpfs mount on /dev/shm (size 512MB), which Swarm AND
plain Compose both honor. Requires a stack redeploy to take effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Enter handler listened in the bubbling phase, so Vuetify's own input handler
(which opens the menu on Enter) fired first and my accept logic saw the menu
already opening and bailed — Enter popped the dropdown instead of submitting.
Bind it in the capture phase so it runs first, and stop the event when a fandom
is already selected so Vuetify never reopens the menu (operator-flagged
2026-06-07).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ArtistView set document.title to "<artist> — FabledCurator" on load but nothing
reset it when navigating away, so the artist name stuck on the Showcase/Gallery
tab title (operator-flagged 2026-06-07). Add a router.afterEach that sets the
title from meta.title on every navigation; detail views with no meta.title reset
to the default and then set their own dynamic title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Solid-color members phash-collapse to distance 0, so the second archive's member
deduped away ("held no supported members") and members_imported was 0. Use
structurally distinct patterned jpegs so both members import — the resume cursor
mechanics were already correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reextract_archive_attachments loaded ALL PostAttachments and ran in one pass up
to a 30-min soft limit, then died without re-enqueueing — a large archive
backlog would only ever partially process. And a naive re-run can't advance: an
already-extracted archive is still an archive on disk, so it'd re-extract the
same first batch forever.
Give it a real cursor + time-box + self-resume (mirrors normalize_tags_task,
operator-asked 2026-06-07: reasonable timeout, then re-queue so other work keeps
flowing):
- service scans attachments with id > after_id in ascending order, time-boxes
the chunk, and reports partial=True + resume_after_id (last scanned id).
- task passes a 600s budget and re-enqueues itself from the cursor until the
scan is exhausted. Routes on the maintenance_long lane.
- This is independent of the maintenance_long lane isolation (already shipped) —
that stops long tasks starving the quick maintenance queue; this stops the
re-extract itself dying on a big backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The image's Camie suggestions now appear in the tag input's dropdown as you
type, filtered to the query and de-duped against the server autocomplete hits,
so the operator can pick a suggestion without hunting for it in the Suggestions
panel below (operator-asked 2026-06-07).
- Unified `rows` model (hits → matching suggestions → create row) so the
highlight index maps 1:1 to a row across all three sections; arrow/Enter/Tab
drive the whole list.
- Suggestion rows are marked (accent left-border + mdi-auto-fix score chip) and
show a "new" hint when the suggestion would create a tag.
- Picking a suggestion emits accept-suggestion → TagPanel runs the SAME accept
path as the Suggestions panel (creates raw tags, records acceptance, drops it
from the panel), then refreshes the chip rail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-specified flow for character-tag creation: focus starts in the
fandom search dropdown; Tab moves to the new-fandom field where Enter
creates; creating fills the dropdown and returns focus there; Enter in the
dropdown accepts the selection.
- Drive focus from the dialog's @after-enter (autofocus is unreliable inside
a v-dialog — the focus-trap steals it post-mount); FandomPicker exposes
focusSearch.
- Drop the @update:model-value auto-confirm that closed the dialog the instant
selectedId was set — that's what broke create-then-accept (creating set the
value and immediately confirmed). Enter now accepts (menu-closed + value),
while an open menu lets Vuetify pick the highlighted item first.
- Tab from search → new-fandom field; Enter there creates, then focus returns
to the dropdown for a single Enter-to-accept.
- Restore focus to the tag input after the dialog confirms/cancels so the
keyboard flow continues into the next tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
THE actual root cause of the "dead" tag-chip kebab (operator inspected it
2026-06-07: the menu was ghosted, blurred, behind the sidebar). The teleported
v-menu landed below .fc-viewer (z-index 2000) and the modal's
backdrop-filter: blur(8px) smeared it — so it opened the whole time, just
underneath. Every prior "fix" (un-nesting the button, the explicit activator
pattern) was chasing a click/activation problem that never existed.
Set :z-index="2400" on the tag-chip and suggestion kebab menus so they paint
above the modal. (Dialogs already render on top — only the anchored menus tied
with the modal's z-index and lost.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The old size-36 v-progress-circular sat tiny in the top-left because
.fc-viewer__media doesn't center its children (the canvas centers itself).
Replace it with a 108px dual counter-rotating accent-ring spinner as a centered,
non-interactive overlay over the modal (operator-flagged 2026-06-07).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-flagged: /downloads was never mapped in prod and everything worked —
confirmed nothing in the app references a filesystem /downloads (only the
unrelated /api/downloads route). Dropped the dead mount from web/worker/
scheduler, and scoped the new maintenance-long worker to just /images (backups
write to /images/_backups; audits + admin tasks all operate on /images).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The constant + comment landed BETWEEN @celery.task(...) and the function def,
which is a syntax error that broke the whole tasks.admin import (cascaded to
lint E999 + every backend/integration test). Move it above the decorator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two more maintenance-queue failures from the operator's 24h list:
- vacuum_analyze died with "could not resize shared memory segment to 67MB: No
space left on device" — Docker's default /dev/shm is 64MB, too small for
VACUUM (ANALYZE)'s parallel-worker shared memory. Set the postgres service
shm_size: 512m.
- backup_db_task timed out at its 12-min limit once the DB grew; a pg_dump can't
be chunked, so raise it to 30/35 min. (A long backup still briefly holds the
concurrency-1 lane — the structural fix is a dedicated lane for long one-shots.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested modal/tagging keyboard improvements:
- A2/A3: fandom dialogs autofocus their autocomplete on open; in the character-
creation FandomPicker, picking a fandom (keyboard Enter or click) confirms in
one step. FandomSetDialog stays autofocus-only (its Save can trigger a merge).
- B5: Tab accepts the highlighted autocomplete row (standard convention).
- C9: T or / jumps focus to the tag input from anywhere in the modal.
- C8: ? toggles a keyboard cheatsheet (corner hint advertises it; Esc closes the
cheatsheet first, then the viewer).
Builds on the same-batch regression fixes (kebab #711, ESC-after-accept #700,
autocomplete scroll-into-view). B6 (keep focus after add) is covered — the input
retains focus after adding a tag, and Esc now works after accepting a suggestion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two regressions the operator re-flagged (the earlier "fixes" didn't work):
#711 tag-chip kebab: TagPanel's kebab used the #activator + v-bind="props"
v-menu pattern — the exact pattern SuggestionItem's own comment documents as
NEVER toggling inside the teleported ImageViewer modal. Extracted TagChip.vue
using the proven explicit pattern (activator="parent" + :open-on-click="false"
+ a manual v-model), mirroring the working suggestion kebab. Now opens.
#700 ESC-after-accept: the guard suppressed close whenever ANY non-tooltip
overlay was active anywhere in the DOM, so a stray overlay after accepting a
suggestion (focus drops to <body>) blocked Esc. Now key off the event origin —
only defer to an overlay when Esc is pressed from INSIDE its content
(ev.target.closest('.v-overlay__content')); a stray overlay no longer traps the
modal, and dialogs/menus still handle their own Esc.
A1: TagAutocomplete arrow-nav now scrollIntoView's the highlighted row — the
list is capped at 240px and arrowing past the fold left the active item
off-screen (operator-flagged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native client treated a gallery image without `file_name` as schema drift
and raised "Patreon API changed — ingester needs update", failing the whole walk
(operator-flagged 2026-06-07: BlenderKnight post 73665615, kind=images). But the
resource had a valid URL, and the code already derives a filename from the URL
basename right below the raise — the same fallback gallery-dl uses. Patreon
legitimately serves some images without file_name, so this isn't drift.
Drop the require_file_name gate from _media_item: file_name is now optional for
every kind (images/attachments/postfile), falling back to the URL basename.
Genuine drift still raises — no resolvable URL, or a media id referenced by a
relationship but absent from `included`. Test updated to assert the fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mid-post time-box check (619e771) reads time.monotonic() once more per post,
so test_backfill_budget_cut_returns_partial_with_progress's discrete tick
sequence shifted — the >budget tick (200) landed on post1's first-item check
instead of post2's gate, cutting post1 to 0 files. Add the extra tick (20, still
under budget) so post1's item downloads, matching production where the gate and
the first should_stop are microseconds apart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A backfill chunk's time-box (BACKFILL_CHUNK_SECONDS=600) was only checked
between POSTS, but download_post downloads ALL of one post's media
synchronously — so a single media-heavy post could run the chunk far past 600s,
all the way to the Celery soft time limit (1350s), where it was killed and
finalized as error (Pocketacer, event #41330: ran the full 22.5 min).
download_post now polls a should_stop() deadline BEFORE each media item and the
engine passes `now - start >= time_budget_seconds`, so a heavy post stops at the
budget and the remaining media (never marked seen) re-fetch next chunk. Bounds
chunk overrun to one media download instead of one whole post.
Also genericized the soft-limit salvage message — it claimed the "gallery-dl
subprocess" failed, which is wrong for a native Patreon walk; it now describes
the time-budget overrun + per-page checkpoint resume in platform-neutral terms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source URL like https://www.patreon.com/posts/mimic-in-dungeon-158372536 is a
single-post permalink, not a creator page — the resolver grabbed "posts" as the
vanity and failed (operator-flagged 2026-06-07). Add a resolution path: extract
the trailing post id and follow it to the owning campaign via the Patreon post
API (/api/posts/<id>?include=campaign), so pasting any post URL subscribes to
that creator's whole feed. `posts/` is excluded from the vanity regex so it
can't masquerade as a creator slug.
Resolution order is now: cached override → id: URL → /posts/<id> → vanity
(campaigns API + creator-page scrape). Tests cover the post→campaign resolve
and that /posts/ URLs aren't treated as vanities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new enabled source is armed for run-until-done backfill (#693) but would sit
idle until the next scheduler tick (~60s). create_source now enqueues the first
walk right away (pending DownloadEvent + download_source.delay), skipping only
when the platform is in a rate-limit cooldown (the scheduler picks it up when
that clears). Disabled sources still don't dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.
download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
