00607a309b34cdaa702acb384214d19b06936a84
778 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
00607a309b |
feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)
If Patreon renames/restructures the post body field again (as content → content_json_string already did), every body silently comes back empty and we'd archive empty posts without noticing. Surface that as a loud failure. Research-grounded design (Patreon `content` is officially null|string, body has no post_type gate, gallery-dl independently added the same content_json_string fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction threshold would false-positive constantly. The robust, creator-independent break signature is "a meaningful sample of posts, a body extracted from NONE of them." - ingest_core counts posts_recorded / posts_with_body on the native post-record path (gallery-dl never enters it, so the canary is native-only by construction). - When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are literally "fix the field-set/parser, not creds"). Placed after the timeout/stop returns so it never masks a more specific failure. - Run summary always appends "bodies X/Y" for sub-threshold observability (a partial regression that still extracts some bodies shows in the Raw stdout). Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success; below the sample floor -> success (tick safety). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
949c9abcc6 |
fix(external): path-safe unlink + per-link staging + orphan repair (#859)
External downloads import IN PLACE, so the post-attach dedup-skip unlink could delete a file that IS an ImageRecord's backing file — orphaning the record and 404-ing on playback. Two sources of that: - Two links on the same post (same film from mega + gdrive) emitted the same filename into one external/<post_id>/ dir; the second overwrote the first. Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path. - The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it path-safe: only unlink when `f` is NOT the existing record's canonical file. Plus an operator-triggered orphan-repair maintenance task (prune_missing_file_records_task) to clean up records already orphaned by the bug: scans ImageRecords, deletes those whose file is gone (cascade), with an NFS-stall guard that aborts without deleting if a large sample is mostly missing. Wired through POST /api/admin/maintenance/prune-missing-files and a MissingFileRepairCard in the Maintenance panel. Tests: refetch-same-link keeps the canonical file; orphan repair deletes only real orphans and aborts on the mostly-missing guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
f897e2534b |
feat(posts): full-width body for image-less posts (drop dead 'no images' box)
Text-only Patreon posts (WIP/announcement/poll — the bulk of a creator's feed) rendered a big empty 'No images attached to this post' placeholder taking half the card. Render the media column only when the post HAS images; image-less posts let the title + body span the full width. Removes the now-dead PostEmptyThumbs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
976107bbe8 |
fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)
THE empty-body root cause. Patreon deprecated the flat `content` HTML field — it returns null on the feed AND the detail endpoint, for every post type (confirmed against the live API: all 135 StickySpoodge posts, text_only/ image_file/poll alike). The real body now lives in `content_json_string` (a ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse fields[post]=content request omits it. Not credential, not post_type: a request shape gone stale. - NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks bold/italic/underline/strike/code/link, hardBreak, inline images, lists, headings; unknown nodes degrade to children). post_body_html(attrs) = the one resolver: legacy content HTML else convert content_json_string. - patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post]) and resolve via post_body_html (replaces the wrong sparse req + full-fetch fallback). - patreon_downloader._write_sidecar_data: resolve body via post_body_html (feed content_json_string) before the detail-fetch; memoize resolved HTML. - tests: prosemirror converter unit tests; client legacy + content_json_string paths; contract pins content_json_string. Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
0d51b93aa7 |
refactor(importer): single _apply_post_fields predicate for both ingest paths (#842/#753)
The per-media path (_apply_sidecar) and the post-record path (upsert_post_record) each carried a VERBATIM copy of the post-field write (url/title/date/description/ attachment_count/raw_metadata + external-link sync). Two copies of one concept = the divergence risk #753 targets. Consolidate into one _apply_post_fields(post, sd) helper both call — a single predicate for how a post body/links get stored, so the two sources can't drift. Behavior identical (fill-with-non-empty); both paths already covered by existing importer tests. Groundwork for the planned post-first ingest model (single authoritative post record; media attaches to it) as more platforms move onto the native ingester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
eb811e11f6 |
refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
Two corrections from operator review: 1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI section — the native ingester now writes a per-post line into the run stdout (parity with gallery-dl's per-file stdout), so the per-post handling shows in the panel the operator already uses. 2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring the download_post -> MediaOutcome contract — and the downloader owns the read; ingest_core only formats the outcome into the log line. Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture' section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+ ' — EMPTY' when 0), so an empty body is self-explanatory by post_type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
bcc7266021 |
feat(downloads): per-post body-capture diagnostics in the event UI (#842)
Operator can't (and shouldn't have to) hunt worker logs to see why a recapture
left a post body empty. Surface per-post handling ON THE EVENT, in the UI.
The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a
per-post diagnostic {post_id, title, post_type, body_chars} with zero extra
fetching — a 0-char body next to its post_type explains an empty post at a
glance (e.g. polls/embeds whose body the API never returns).
- ingest_core: accumulate post_diagnostics; thread via DownloadResult
- download_service: write to DownloadEvent.metadata_['post_diagnostics']
- DownloadDetailModal: 'Post capture' section — totals + empty-body table
(post_type + chars, flagged) + all-posts table; included in Copy-all
- tests: ingester diag (post_type + body_chars), download_service metadata
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
3df191e255 |
fix(patreon): full-fetch fallback when sparse fieldset returns null content (#842)
Operator-flagged: 9 StickySpoodge posts had empty bodies in FC despite the body plainly existing + being accessible (creds refresh didn't help). All 9 are body-only / poll / embed / announcement posts with no downloadable gallery media — Patreon's detail endpoint returns content:null for these under the sparse fields[post]=content request even though the body exists. fetch_post_detail_content now re-fetches the FULL post resource once when the sparse request comes back empty: recovers the body when the sparse fieldset was the cause, and logs post_type when even the full resource is empty (body lives elsewhere). Only the empty cases pay the extra GET; the 126 already-working posts keep the fast sparse path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
b999480db5 |
feat(ingest): per-post body-capture + recapture diagnostics logging
Operator-flagged: a recapture 'caught nothing' for a post and there were no logs explaining why. Three silent spots now log, so a recapture's per-post outcome is diagnosable (retention bounds the volume): - patreon_client.fetch_post_detail_content: the 200-OK-but-null-content branch was silent — now logs 'fetched N chars' on success AND 'empty/null content (tier-gated or no text)' on the empty case (the most common silent miss). - patreon_downloader.write_post_record: logs each post's FINAL body outcome (captured N chars / NO body) read off the memoized attrs after detail-fetch. - ingest_core summary: appends post-record + relinked counts to the run summary (surfaces on the event stdout the operator already reads). - download_service phase3: logs how many on-disk images got source_filehash relinked (N/total) per recapture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
65ec29ba9b |
feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
96c29c370b |
feat(ingest): localize inline post-body images to local copies (Phase 2)
Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
5e1655384f |
feat(ingest): recapture body + links for every walked post (Phase 5)
Operator reframed backfill as inherent to the existing walk: you can't fill links the system never had by re-downloading media that's already on disk, so the body/link recapture has to ride the walk itself. Hoist the post-record capture out of the media-less branch so it runs for EVERY post — gated once per post by the synthetic post key in the seen-ledger (detail-fetch for an empty feed body happens at most once; recovery re-captures unconditionally). A normal BACKFILL now walks history and recaptures each post's body + external links (which phase 3 imports via upsert_post_record → _sync_external_links → the download sweep, all already wired). A tick captures new posts going forward. No separate button — the backfill is the backfill. Tests: media posts now also carry a synthetic post-key ledger row (count assertions +1); new test proves an already-on-disk media post still recaptures its body/links on a re-walk. Completes the core of #830 (Phase 5). Phase 2 (inline-image localization) remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
8dbf29f803 |
feat(external): per-host enable toggles in Settings (Phase 4d)
Operator lever: disable a single file host (e.g. mega.nz when it's banning) without touching the others. Five booleans on import_settings (extdl_<host>_enabled, default true — works out of the box, rule #26); the worker already reads them via getattr so no worker change. Migration 0050 + model fields + settings GET/PATCH (uniform boolean validation) + a 'External file-host downloads' card in the subscriptions Settings tab. Completes Phase 4. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
05f226a8f6 |
feat(external): zip-parity provenance/tagging + thorough worker logging
Operator-requested: a worker download must be tagged + provenance-associated exactly like an extracted zip, and the path must log well (we won't get it right first try). - _route_files now mirrors download_service._phase3_persist branch-for-branch: imported/superseded → collect member_image_ids+image_id (provenance-linked via the synthesized sidecar, same as extracted-zip members) → caller enqueues tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed → unlink + warn. - Logging at every stage: start (link/host/post/artist/attempt/url), requeue, fetch result (files/bytes) or fetch failure, per-file import decision, dead- letter transitions, and done (files/images/duration). - Parity test: an archive downloaded by the worker is extracted, provenance- linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for exactly the member images. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
bd2807cdd1 |
feat(external): mega.nz via megatools in the runtime image (Phase 4c)
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds `megatools` to the runtime Dockerfile; the fetcher's mega backend now shells `megatools dl --path <dir> <url>` (key in the #fragment is preserved by the extractor). gdown (gdrive) is already a pip dep in the runtime image. NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile change is verified on the next dev→main merge, not by this dev push. The fetcher code path is unit-tested via the mocked _run_mega_get seam. With this, all 5 hosts download end-to-end once a celery download-worker runs. Refs FC #830 (Phase 4c). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
82b26b8aaa | test(external): unique seeded artist per host (fix uq_artist_name in sweep test) | ||
|
bvandeusen
|
96e984cded |
feat(external): download worker for file-host links (Phase 4b)
tasks/external.py drives the external_link ledger: - fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern; requeue-with-countdown if busy), fetch via external_fetch into the artist library tree, then route each file through importer.attach_in_place via a synthesized sidecar so it links to the SAME post (archive→ImageRecords, else→PostAttachment; on-disk original removed for captured files, art stays); thumbnail+ML enqueue for new images; status downloaded | failed | dead with attempts/last_error/completed_at/duration. - sweep_external_links(): enqueue a bounded batch of actionable links. - recover_external_links() + prune_external_links(): recovery + retention (#89). - per-host enable read via getattr (forward-compatible; Settings UI adds the columns in 4d — defaults on, rule #26). Wiring: celery include + route (download lane) + beat (sweep 10m, recover + prune daily); download_service phase 3 enqueues a sweep after recording links. Integration tests: download+attach, failure, dead-letter, non-claimable, sweep. mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
13253b18d1 |
feat(external): file-host fetcher subsystem (Phase 4a)
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
896e4f248c | style(importer): fix ruff import ordering for link_extract | ||
|
bvandeusen
|
d96918d777 |
feat(posts): extract + record external file-host links (Phase 3)
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded in post bodies so they're never silently dropped, and surface them in the post view. The download worker (Phase 4) walks these rows. - link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups. Reusable by every platform (runs off Post.description). - external_link model + migration 0049: post_id/artist_id/host/url/label/status /attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl. worker statuses up front) + (post_id,url) unique. - importer._sync_external_links: insert-missing on both import paths (_apply_sidecar + upsert_post_record) so a re-import never resets a link's status; runs for all platforms. - post_feed_service.get_post: returns external_links (detail-only). - PostCard: renders the links (host chip + label + status) once expanded. - tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record + no-dup on reimport), serializer. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c342c73a25 |
feat(posts): faithful (semantic) HTML rendering of post bodies
Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as plain text. Now: - html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings, inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay stripped; their text is preserved). Benefits the existing ProvenancePanel too. - post_feed_service.get_post: add sanitized `description_html` to the DETAIL response (the feed list stays lightweight plain text by design). - PostCard.vue: render description_html via v-html once expanded (fetched with detail); collapsed + no-detail fallback stay plain text. Styled close to the source (headings, images max-width, accent links, lists, quotes, code). Tests: sanitizer (headings/img/lists survive, img javascript: src dropped); get_post returns sanitized description_html. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ca25f688c3 |
fix(download): tolerate dl_result without post_record_paths
The test_download_service stubs build dl_result as a SimpleNamespace that doesn't set the new field; read it via getattr (matching the existing retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any caller that predates the field. |
||
|
bvandeusen
|
796e92540a |
feat(patreon): capture media-less/text-only posts (post-only records)
Today the ingest core does `if not media: continue`, so a post with no downloadable media (a pure-text post — which often holds the ONLY copy of an external mega/gdrive/pixeldrain link) never upserts a Post. Now the native ingester emits a post-only sidecar (`_post.json`) for every media-less post, gated through the seen-ledger via a synthetic `post:<id>` key so the body is detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on external_post_id so it UPDATES the same Post a media import would create — never doubles, never clobbers a populated body with an empty one. - gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path unaffected — all constructions are keyword). - ingest_core.py: media-less branch (optional client/downloader seams via getattr; stub clients in tests skip it as before). - patreon_client.py: post_record_key(post). patreon_downloader.py: write_post_record + _write_sidecar_data refactor (shared serializer). - importer.py: upsert_post_record. download_service.py: phase-3 import loop. - tests: client/downloader/ingester (gate + recovery)/importer (no-double). Slice 0b of milestone #64. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2c67c27044 |
feat(patreon): capture full post body via adaptive detail-fetch
The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).
First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
7fcef53d5b |
fix(series): sticky tabs + controls on the Series view
The Series tab strip and the Browse search/sort (and Suggestions controls) scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab controls into one sticky header pinned under the 64px TopNav. The controls had to leave v-window — it clips sticky children — so they're driven by the tab from the header instead of living inside each window-item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5c3f8ebd70 |
fix(aliases): store modal alias under raw model key + make aliases visible/manageable
The headline bug: aliases created from the modal NEVER resolved. Create
sent the normalized display name ('Sword', 'Uchiha Sasuke') while
resolution keys on the raw booru model key ('sword', 'uchiha_sasuke',
case-sensitive) — so the mapping was stored under a key nothing looks up,
and the prediction kept reappearing unaliased. The raw key wasn't even in
the /suggestions response, so the modal couldn't send it.
- Suggestion now carries raw_name (the model key an alias must use) and
via_alias (surfaced via an operator alias); both serialized by the API.
- Modal alias-create sends raw_name, not display_name (the fix). Aliased
suggestions show an 'alias' badge and a 'Remove alias' action; 'Treat as
alias for…' is hidden for centroid hits (no model key) and already-aliased
rows.
- Tag-side management: TagCard ⋮ → 'Aliases…' opens a dialog listing the
model keys that fold into a tag, with remove (GET /api/tags/<id>/aliases +
AliasService.list_for_tag). Creation stays in the modal suggestion flow.
Tests: full API round-trip locking the raw-key contract (raw_name exposed →
alias authored with it → resolves + via_alias on a later image);
list_for_tag (service + API); via_alias/raw_name on the existing service
suggestion tests. No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
7c4b24c80d |
fix(images): percent-encode original-image URLs ('#' in paths 404'd)
An image whose on-disk path contains '#' (post folders like 'BLUE#59') served its hash-named thumbnail fine but 404'd the original: the unencoded '#' in image_url was parsed by the browser as a URL fragment, so '#59/01_timelapse.jpg' never reached the /images route. Add a shared image_url(path) helper that percent-encodes the path (safe='/') and route the 3 raw builders (gallery detail + 2 in series) through it. Not a cleanup-tool deletion — the file is on disk; only the URL was wrong. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3e1303ea3c |
fix(browse): put tabs and search on one row
Operator-asked: the tab strip and search field were stacked; place them side-by-side in a single flex bar (tabs left, search + scope chips right), wrapping to two rows only on narrow viewports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2c544ad5af |
feat(browse): sticky tabs + per-tab search bar (server-side, scope-aware)
The Browse tab nav scrolled away (operator didn't know it existed) and Posts had no search. Roll the tab strip + a shared search field into one sticky block pinned under the 64px TopNav. - Posts gains server-side text search: PostFeedService.scroll()/around() + /api/posts accept q (ILIKE over post_title OR description), applied INSIDE the artist/platform WHERE so search stays scoped to the active filter. Scope shown as clearable chips next to the search field. - Artists/Tags search consolidates into the sticky bar: their inner search boxes are removed; they react to route.query.q (q is deep- linkable, e.g. /browse?tab=posts&q=foo). Platform/kind filters stay. - Posts empty state now distinguishes 'no matches' from 'no posts yet'. Tests: posts q-search matches title|description and stays artist-scoped (service); q passthrough (api). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
90c68f8b2a |
fix(series): round the kebab backing on series cards
The tinted backing was set on the square .fc-kebab wrapper span while the button is round, so a translucent square showed behind the round ⋮. border-radius:50% makes the backing a circle matching the button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3e22e78aa4 |
test(series): assert group start_page, not per-page stated_page
add_post now stamps the post's parsed START (constant) on every staged pending page so the group start survives junk removal; list_pages surfaces it as start_page. Update the stale per-page [9,10,11] assertion to check grp["start_page"] == 9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
013b9d7f06 |
feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)
Replaces the auto-renumbered 1..N position key with operator-OWNED page numbers: sparse, gaps allowed, editable, never auto-renumbered. Order follows the numbers; unnumbered pages sort to the tail. This is the fix for the model that clobbered hand-set numbers on the flatten — numbers are now data, not a derived sequence. - series_service: drop the renumber-on-reorder/remove; order by page_number NULLS LAST; new set_page_number(image_id, n|None); list_pages returns `gaps` (one entry per missing-number run) + each pending group's parsed `start_page`; set_cover renumbers below the current min; place_pending(image_ids, start_page) numbers placed pages sequentially from the start (drop junk first → numbers line up); add_post stamps the parsed start on staged pages. - api/tags: POST /series/<id>/pages/number (set one page's number); /pending/ place takes start_page; removed /reorder. - frontend: per-card editable number input; one gap block per gap with drop-on-edge to assign the adjacent number (middle → type); append drop zone; pending tray gets a "from page N" field + "Place from page N". - tests reworked: sparse numbers + gaps, place-from-start, set-page-number route. No migration; nothing destructive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
7bb765b6ed |
feat(series): pending staging for add-from-post (#789 Phase 2)
Add-from-post no longer appends straight into the run — it STAGES the post's
pages as pending (per-page status; page_number NULL), grouped by source post,
so the operator drops junk (text-free alts, bumpers) and places the keepers
into the sequence with clean series-global numbering.
- migration 0048: series_page.status ('placed' default | 'pending') + nullable
page_number.
- series_service: placed/pending split everywhere (list_pages returns the
placed run + a `pending` section grouped by source post; reorder/cover/
list_series operate on placed only); add_post stages pending; new
place_pending(image_ids, before_image_id=None) flips pending→placed spliced
before a page (or appended) and renumbers; junk removal reuses remove_images.
- api/tags: /add-post now returns staged count; new POST /series/<id>/pending/
place.
- frontend: PostSeriesMenu navigates to the series after staging; seriesManage
store surfaces `pending` + placePending; SeriesManageView gains a pending
tray (per-post groups, place-all / place-one / drop-junk).
- tests: pending staging, place (append + insert-before), ignore-already-
placed, drop-junk, route guard; updated add_post + match-accept expectations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
59746d213d |
feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)
Reframe a series from "ordered chapters that own pages" to ONE flat, series-global ordered run of pages with optional cosmetic chapter DIVIDERS over it. A chapter no longer wraps content — it's a labeled divider anchored to the page that begins it; a page's chapter is derived as the nearest preceding divider. This is what lets installments assembled from multiple sources sit in one continuous, correctly-numbered sequence (operator's Goblin Juice case). - migration 0047: flatten each series to a series-global page_number (preserving today's reading order); convert each existing chapter to a divider anchored at its first page (keeping title/stated_part); drop series_page.chapter_id; reshape series_chapter (anchor_page_id UNIQUE FK, drop chapter_number/is_placeholder/stated_page_start/end). Loss-safe for content; drops empty placeholder chapters + a redundant page-1 divider. - series_page: page_number is now the series-global order; no chapter_id. - series_chapter: anchored divider (anchor_page_id, title, stated_part). - series_service: flat list_pages (one run + derived dividers + per-page source_post + part_gaps), series-wide reorder/renumber, divider CRUD (create/update/move/delete); retired per-chapter reorder/merge/placement. - api/tags: drop chapter_id from add; /chapters endpoints are divider create/update/delete (removed chapter reorder/merge/page-reorder). - series_match_service: series "end" reads max(series_page.stated_page); accept appends via add_post. tag_service series-merge appends src's pages after tgt's max so the merged series stays one clean run. - frontend: seriesManage store + SeriesManageView → one continuous drag-reorder grid with inline divider bars + series-global page numbers; reader walks the flat run, headings from dividers; PostSeriesMenu copy. - tests reworked across the series suite for the divider model. Phase 2 (pending staging for add-from-post) is separate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3610ba495f |
feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)
Read cutover verified in prod (suggestions + allowlist read image_prediction; backfill complete at 908k rows / 51k images). Removes the old JSON column and everything that fed it: - ImageRecord.tagger_predictions column removed; migration 0046 DROPs it. tagger_model_version kept as the "tagged / current?" signal the backfill sweep reads (needs-tagging check switched to tagger_model_version IS NULL). - tag_and_embed no longer dual-writes the JSON — image_prediction is the only write path. - importer re-import reset drops the JSON line (image_prediction rows are already deleted on re-import). - Retired the one-time #768 backfill task + the #764 prune task, their admin endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard). - Tests seed/assert via image_prediction; stale column refs removed. Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run `VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's the source of truth now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
65211a3f2f |
fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
The inline INSERT…SELECT backfill in migration 0045 wrapped the table creation and a ~100 GB pass over image_record.tagger_predictions in one transaction: nothing committed until the end, it was unmonitorable, and an earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A deploy got stuck on it for ~2h with image_prediction never appearing. Split the concerns: - 0045 now creates ONLY the table + indexes (instant DDL → web boots). - New backend.app.tasks.admin.backfill_image_predictions_task copies the >= store-floor predictions from the JSON into image_prediction, batched by id window and committed per chunk: live progress, resumable (re-enqueues from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each stays in the DB executor streaming each window — no Python-side 100 GB load, no materialization. - POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card to trigger the one-time run after upgrading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e6d5f67f11 |
perf(migration): 0045 streams json_each via inline CASE guard (no temp spill)
The MATERIALIZED-CTE scalar guard forced Postgres to materialize all object rows with their full JSON (~100 GB) to temp before json_each — on NFS that's a huge spill and pathologically slow (risks disk-full). Replace with an inline CASE that feeds json_each an empty object for non-object rows: same scalar guard, but a single streaming pass with no materialization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a712cef92d |
fix(migration): 0045 backfill guards json_each against non-object rows
Some image_record rows store tagger_predictions as a JSON scalar/null rather than an object; json_each throws 'cannot deconstruct a scalar' on those, rolling back the whole migration. Filter to json_typeof = 'object' in a MATERIALIZED CTE so the guard runs before json_each ever evaluates a scalar. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
75eab188c8 |
fix(migration): 0045 backfill filters to >= store floor (supersedes #764 prune)
The #764 in-place prune (rewrite tagger_predictions to >=0.70) is too slow on 100 GB of TOAST and fails at its soft limit (interrupts a query mid-flight -> 'another command is already in progress'). #768 supersedes it: extract only the >=floor predictions into image_prediction via this set-based backfill, then drop the column (step 3) — reading 100 GB once + writing ~840k small rows beats rewriting 100 GB in place. So this backfill no longer assumes the prune ran: it filters by ml_settings.tagger_store_floor (default 0.70) itself, handling the full or partially-pruned JSON identically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
0319812b45 |
style: group tests._prediction_helpers import with backend (ruff I001)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
22cdf0f334 |
feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)
Switch every prediction READER off the JSON column onto the normalized
image_prediction table. Parity by construction: each reader loads the same
{raw_name: {category, confidence}} dict it consumed before (via small
_load_predictions helpers), so all downstream threshold/alias/merge/consensus
logic is byte-identical — only the data source changed.
- suggestions.SuggestionService.for_image (and for_selection via it)
- ml.apply_allowlist_tags (iterates images that have prediction rows)
- importer re-import reset deletes the image's prediction rows
The tagger_predictions JSON column is still dual-written (step 1) so it stays
valid during transition; the backfill task's NULL check still works. Removing
the JSON write + DROP column + retiring the #764 prune is the cleanup
follow-up (needs a quiesced-worker window for the DROP lock).
Tests: shared tests/_prediction_helpers.seed_predictions seeds the table;
read-path tests (suggestions, bulk consensus, allowlist apply, API) seed there
instead of ImageRecord.tagger_predictions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
79089b50b0 |
feat(ml): image_prediction table + backfill + dual-write (#768 step 1)
Normalize tagger predictions out of the image_record.tagger_predictions JSON blob into a queryable per-prediction table. Step 1 of the cutover (expand): additive + low-risk — reads still use the JSON, this just adds the table and keeps it populated. - ImagePrediction(image_record_id, raw_name, category, score) — stores the RAW tagger vocab name (not tag_id) so read-time alias→canonical resolution is unchanged. Indexed for per-image reads + by (raw_name, score). - Migration 0045: create table + set-based backfill from the JSON via json_each (fast post-#764-prune). The old column stays (vestigial) and is dropped in a later follow-up — DROP needs an ACCESS EXCLUSIVE lock on the hot image_record table, so it waits for a quiesced-worker window. - tag_and_embed dual-writes the rows (delete-then-insert, idempotent); tagger_store_floor already applied in infer(). Next: switch suggestion + allowlist reads to the table, then drop the JSON write. Plan-task #768. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
7a40a50fe9 |
fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)
DB backup polish (plan-task #764 Q3): - pg_dump now uses custom format (-Fc): compressed (much smaller on NFS) and restored via pg_restore. Artifact extension .sql → .dump; restore_db swaps psql -f for pg_restore -d. BackupRun.sql_path field name kept (it's just the db artifact path). - Reconcile the subprocess guardrails: the DB timeout was 720s with a stale 'Celery soft is 10 min' comment, but backup_db_task's soft limit is actually 1800s — so the bounded-kill fired 18 min early. Set DB=1700s / images=21000s, each just under its task's Celery soft limit so _run_bounded stays the primary guard (an NFS D-state hang defeats Celery's own SIGKILL). Real shrink of the DB is the #764 prune; this makes each dump smaller/faster on top of that. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d55e52ae9b |
feat(admin): prune_low_confidence_predictions backfill task + UI (#764)
The one-time backfill that actually shrinks the DB: drops stored tagger_predictions entries below ml_settings.tagger_store_floor from every image_record row, and clamps any allowlist min_confidence below the floor up to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate so backfilled rows match new imports. Keyset by id ASC, idempotent, self-resumes on the soft time limit; runs on the maintenance_long lane. pg_dump copies live data only, so this alone fixes the #739 backup timeout — the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional disk-return step, brief because post-prune the live data is tiny. - admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions - PrunePredictionsCard in the Maintenance panel (shows the current floor) - tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c8b815afe6 |
feat(ml): clamp allowlist min_confidence to the tagger store floor
Consumer #4 of the store-floor change (#764). An allowlist tag can't auto-apply more permissively than the ingest floor — predictions below tagger_store_floor aren't stored, so a lower min_confidence behaves identically to the floor. update_threshold now clamps to max(value, floor); the AllowlistTable confidence input min-binds to the live floor and clamps on edit. Keeps the stored threshold honest about actual apply behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3f92669f12 |
feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default 0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70). Storing every tag down to 0.05 from a ~10k-tag tagger is what grew image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70 and the centroid/learned path covers lower-confidence preferred tags, so the sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim land next; this only changes the write gate for NEW imports). - ml_settings.tagger_store_floor (migration 0044, default 0.70) - tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor - ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold below the floor (nothing below the floor is stored, so the gap surfaces nothing) — server backstop for the UI slider clamp - Settings → ML: store-floor slider + caption; category sliders min-bound to it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9ba3db75fd |
fix(maintenance): download queue needs a sweep threshold above its 25-min time_limit
recover_stalled_task_runs used the 5-min default for the download queue, but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT (1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as phantom 'RecoverySweep' failures — visible in System Activity but absent from the Subscriptions view (the download finished ok, reset the source's consecutive_failures; only the orphaned task_run kept the stamp, since _finalize only updates rows still 'running'). Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real hard kill is swept by the task-run and event sweeps together. Restores the documented invariant (every override >= task time_limit). Regression test pins the threshold above the hard limit so a future limit bump can't silently re-break it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
70d4017cf6 |
feat(activity): search/filter on both Activity-tab panes
Recent failures gains a client-side search over the already-loaded 24h rows (task/queue/target/error), shown as a filtered/total count alongside the existing error-type chips. All recent activity gains a debounced server-side task-name search (new `task` ILIKE param on /runs) so it spans the full history, not just the loaded page. LIKE wildcards are escaped so task names' literal underscores match literally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
14c244bd3d |
refactor(tags): shared tag_query for fandom self-join + serialization (DRY sweep)
The fandom self-join (resolve a character's fandom NAME via Tag.fandom_id->Tag)
and the {id,name,kind,fandom_id,fandom_name} dict were hand-written in
TagService.autocomplete/.list_for_image, GalleryService.get_image_with_tags and
the api/tags handlers — the last few grown by this session's fandom-on-chip
feature. Consolidate to services/tag_query: fandom_join_alias() + tag_columns()
build the select; serialize_tag(row) builds the dict. Now a new tag field is
added in one place.
Over-DRY guard: TagDirectoryService selects the full Tag ORM + an image-count
aggregate (a different select shape) — left as its own variant. §8b: the
fandom_lookup alias lives only in tag_query; gallery + both api/tags handlers
serialize via serialize_tag. Test: serialize_tag handles enum + string kind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
074c5868fb |
refactor(services): shared pagination cursor (DRY sweep)
encode_cursor/decode_cursor (base64 <iso8601>|<id>) were defined identically in gallery_service AND post_feed_service, with artist_service importing gallery's copy. Two implementations of one cursor format silently break pagination in whichever feed drifts. Extract to services/pagination.py; gallery/post_feed/ artist all import it. Dropped now-unused base64/datetime imports. §8b: encode_cursor/decode_cursor now defined only in pagination.py. Existing cursor round-trip tests still cover it via the re-export. Catalog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |