Commit Graph

892 Commits

Author SHA1 Message Date
bvandeusen cffdd93418 Merge pull request 'Nested-archive extraction (#718) + post-first ingest (#67) + post-body canary (#862)' (#109) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m19s
2026-06-15 21:37:39 -04:00
bvandeusen b48ba60830 fix(import): resolve artist from path for enrich-on-duplicate (#718)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m19s
The dedup branches of _import_media linked the existing image to the new post via
_apply_sidecar(artist=None), relying on the SIDECAR to carry the artist. But an
archive member's artist comes from its path, and under post-first the per-media
sidecar is minimal (no artist) — so a re-packed / cross-posted archive image
deduped and was left UNLINKED from the new post, i.e. the post showed "no images".

Resolve the path-anchored artist (derive_top_level_artist) up-front in
_import_media and pass it to both enrich-on-duplicate branches (sha256 + phash
larger_exists) and the new-record path. Drop the now-dead _attach_artist helper
(its logic is inlined at the single new-record call site).

Surfaced by the new test_archive_all_deduped_is_benign_not_flagged (was asserting
2==4: the second post got no provenance links).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:25:55 -04:00
bvandeusen 8dee2f9628 feat(import): recurse nested archives + precise "no images" reason (#718)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Failing after 3m13s
Root cause (operator-confirmed via event metadata + lsar): a "High Resolution
files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase).
_import_archive only extracted one level — a nested-archive member failed
is_supported and was skipped, so the real pages were silently dropped and the post
showed "archive but no images". The disk scan found this pattern recurring across
the attachment store.

- Recurse into nested archives via _collect_archive_members: a member that is
  itself an archive is bomb-probed and extracted too, depth-capped at
  _ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so
  they link to the right Post. Each level is wrapped so one bad nested archive
  can't abort the import. The shared path means external (mega/gdrive) archives
  recurse too.
- Replace the catch-all "held no supported members" string with a per-outcome
  tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped
  case is now recognised as BENIGN — images already in the library, re-linked to
  this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no
  longer false-flags in event metadata.unextracted_archives. Genuine failures
  carry the precise breakdown.

Tests: nested zip-in-cbz imports both inner images + links them to the outer post;
all-deduped archive returns attached with error=None and links images to both posts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:18:02 -04:00
bvandeusen 41aa8fe39e docs(ingest): document the post-first migration contract at the native seam (#857)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m16s
Milestone #67 step 3. Spell out, at the IngestCore.run seam resolution, that
post_record_key + write_post_record are the post-first contract a platform
implements when migrating onto the native core ingester — the post-record owns
the body/links, the per-media sidecar carries image identity only. The import
side is already self-enforcing via uses_native_ingester → importer.post_first.
Durable directive recorded as FC project rule #120.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:20:03 -04:00
bvandeusen 8b99dc9b81 test(downloader): fix test_sidecar_written_and_findable for post-first minimal sidecar (#856)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m18s
The per-media sidecar no longer carries title/url/content (post-first, #856) —
update the assertion to expect image identity only (category/id/source_url).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:15:04 -04:00
bvandeusen dcbc3ae335 refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Failing after 25s
CI / integration (push) Successful in 3m20s
Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.

Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).

- patreon_downloader: the per-media sidecar is now minimal — {category, id,
  source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
  resolution + detail-fetch (the post-record, written first in the walk, already
  did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
  `_apply_sidecar` still writes source_filehash + provenance + primary_post_id
  but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
  so applying a body-less sidecar would clobber raw_metadata + re-sync links off
  empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
  uses_native_ingester(platform) — the future-proof seam, so a platform migrating
  onto the native core flips to post-first automatically (step 3). Media imports
  before post-records but both unify on external_post_id, so the post ends with
  its body either way.

Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:10:21 -04:00
bvandeusen 00607a309b feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m17s
If Patreon renames/restructures the post body field again (as content →
content_json_string already did), every body silently comes back empty and we'd
archive empty posts without noticing. Surface that as a loud failure.

Research-grounded design (Patreon `content` is officially null|string, body has
no post_type gate, gallery-dl independently added the same content_json_string
fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction
threshold would false-positive constantly. The robust, creator-independent break
signature is "a meaningful sample of posts, a body extracted from NONE of them."

- ingest_core counts posts_recorded / posts_with_body on the native post-record
  path (gallery-dl never enters it, so the canary is native-only by construction).
- When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return
  ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are
  literally "fix the field-set/parser, not creds"). Placed after the timeout/stop
  returns so it never masks a more specific failure.
- Run summary always appends "bodies X/Y" for sub-threshold observability (a
  partial regression that still extracts some bodies shows in the Raw stdout).

Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success;
below the sample floor -> success (tick safety).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 08:56:11 -04:00
bvandeusen fd84be40dd Merge pull request 'External-attach orphan fix (#859) + image-less post display' (#108) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 40s
CI / integration (push) Successful in 3m19s
2026-06-15 01:55:36 -04:00
bvandeusen 949c9abcc6 fix(external): path-safe unlink + per-link staging + orphan repair (#859)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m17s
External downloads import IN PLACE, so the post-attach dedup-skip unlink could
delete a file that IS an ImageRecord's backing file — orphaning the record and
404-ing on playback. Two sources of that:

- Two links on the same post (same film from mega + gdrive) emitted the same
  filename into one external/<post_id>/ dir; the second overwrote the first.
  Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path.
- The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it
  path-safe: only unlink when `f` is NOT the existing record's canonical file.

Plus an operator-triggered orphan-repair maintenance task
(prune_missing_file_records_task) to clean up records already orphaned by the
bug: scans ImageRecords, deletes those whose file is gone (cascade), with an
NFS-stall guard that aborts without deleting if a large sample is mostly missing.
Wired through POST /api/admin/maintenance/prune-missing-files and a
MissingFileRepairCard in the Maintenance panel.

Tests: refetch-same-link keeps the canonical file; orphan repair deletes only
real orphans and aborts on the mostly-missing guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:48:38 -04:00
bvandeusen f897e2534b feat(posts): full-width body for image-less posts (drop dead 'no images' box)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m16s
Text-only Patreon posts (WIP/announcement/poll — the bulk of a creator's feed)
rendered a big empty 'No images attached to this post' placeholder taking half
the card. Render the media column only when the post HAS images; image-less posts
let the title + body span the full width. Removes the now-dead PostEmptyThumbs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:42:14 -04:00
bvandeusen 79f510d7f8 Merge pull request 'Merge dev → main: read post body from content_json_string (the empty-body fix) (#842)' (#107) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m24s
2026-06-15 00:19:45 -04:00
bvandeusen 976107bbe8 fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m13s
THE empty-body root cause. Patreon deprecated the flat `content` HTML field —
it returns null on the feed AND the detail endpoint, for every post type
(confirmed against the live API: all 135 StickySpoodge posts, text_only/
image_file/poll alike). The real body now lives in `content_json_string` (a
ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse
fields[post]=content request omits it. Not credential, not post_type: a request
shape gone stale.

- NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks
  bold/italic/underline/strike/code/link, hardBreak, inline images, lists,
  headings; unknown nodes degrade to children). post_body_html(attrs) = the one
  resolver: legacy content HTML else convert content_json_string.
- patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite
  fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post])
  and resolve via post_body_html (replaces the wrong sparse req + full-fetch
  fallback).
- patreon_downloader._write_sidecar_data: resolve body via post_body_html
  (feed content_json_string) before the detail-fetch; memoize resolved HTML.
- tests: prosemirror converter unit tests; client legacy + content_json_string
  paths; contract pins content_json_string.

Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:14:52 -04:00
bvandeusen 59181069da Merge pull request 'Merge dev → main: per-post stdout diagnostics + post-field write DRY (#842/#753)' (#106) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 41s
Build images / build-web (push) Successful in 2m15s
Build images / build-ml (push) Successful in 2m55s
CI / integration (push) Successful in 3m11s
2026-06-14 23:45:39 -04:00
bvandeusen 0d51b93aa7 refactor(importer): single _apply_post_fields predicate for both ingest paths (#842/#753)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 42s
CI / integration (push) Successful in 3m22s
The per-media path (_apply_sidecar) and the post-record path (upsert_post_record)
each carried a VERBATIM copy of the post-field write (url/title/date/description/
attachment_count/raw_metadata + external-link sync). Two copies of one concept =
the divergence risk #753 targets. Consolidate into one _apply_post_fields(post,
sd) helper both call — a single predicate for how a post body/links get stored,
so the two sources can't drift. Behavior identical (fill-with-non-empty); both
paths already covered by existing importer tests.

Groundwork for the planned post-first ingest model (single authoritative post
record; media attaches to it) as more platforms move onto the native ingester.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:39:30 -04:00
bvandeusen eb811e11f6 refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m18s
Two corrections from operator review:
1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI
   section — the native ingester now writes a per-post line into the run stdout
   (parity with gallery-dl's per-file stdout), so the per-post handling shows in
   the panel the operator already uses.
2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record
   now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring
   the download_post -> MediaOutcome contract — and the downloader owns the read;
   ingest_core only formats the outcome into the log line.

Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture'
section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+
' — EMPTY' when 0), so an empty body is self-explanatory by post_type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:27:38 -04:00
bvandeusen 428ecd8642 Merge pull request 'Merge dev → main: per-post body-capture diagnostics in the event UI (#842)' (#105) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m11s
2026-06-14 23:14:14 -04:00
bvandeusen bcc7266021 feat(downloads): per-post body-capture diagnostics in the event UI (#842)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m32s
Operator can't (and shouldn't have to) hunt worker logs to see why a recapture
left a post body empty. Surface per-post handling ON THE EVENT, in the UI.

The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a
per-post diagnostic {post_id, title, post_type, body_chars} with zero extra
fetching — a 0-char body next to its post_type explains an empty post at a
glance (e.g. polls/embeds whose body the API never returns).

- ingest_core: accumulate post_diagnostics; thread via DownloadResult
- download_service: write to DownloadEvent.metadata_['post_diagnostics']
- DownloadDetailModal: 'Post capture' section — totals + empty-body table
  (post_type + chars, flagged) + all-posts table; included in Copy-all
- tests: ingester diag (post_type + body_chars), download_service metadata

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:09:37 -04:00
bvandeusen ed1e04b831 Merge pull request 'Merge dev → main: recapture body-fetch fix + diagnostics (#842)' (#104) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m15s
2026-06-14 22:31:41 -04:00
bvandeusen 3df191e255 fix(patreon): full-fetch fallback when sparse fieldset returns null content (#842)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 1m0s
CI / integration (push) Successful in 3m27s
Operator-flagged: 9 StickySpoodge posts had empty bodies in FC despite the body
plainly existing + being accessible (creds refresh didn't help). All 9 are
body-only / poll / embed / announcement posts with no downloadable gallery
media — Patreon's detail endpoint returns content:null for these under the
sparse fields[post]=content request even though the body exists.

fetch_post_detail_content now re-fetches the FULL post resource once when the
sparse request comes back empty: recovers the body when the sparse fieldset was
the cause, and logs post_type when even the full resource is empty (body lives
elsewhere). Only the empty cases pay the extra GET; the 126 already-working
posts keep the fast sparse path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:26:56 -04:00
bvandeusen b999480db5 feat(ingest): per-post body-capture + recapture diagnostics logging
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m30s
Operator-flagged: a recapture 'caught nothing' for a post and there were no
logs explaining why. Three silent spots now log, so a recapture's per-post
outcome is diagnosable (retention bounds the volume):

- patreon_client.fetch_post_detail_content: the 200-OK-but-null-content branch
  was silent — now logs 'fetched N chars' on success AND 'empty/null content
  (tier-gated or no text)' on the empty case (the most common silent miss).
- patreon_downloader.write_post_record: logs each post's FINAL body outcome
  (captured N chars / NO body) read off the memoized attrs after detail-fetch.
- ingest_core summary: appends post-record + relinked counts to the run summary
  (surfaces on the event stdout the operator already reads).
- download_service phase3: logs how many on-disk images got source_filehash
  relinked (N/total) per recapture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:10:59 -04:00
bvandeusen f5156bd847 Merge pull request 'Merge dev → main: Recapture mode (#842) — re-grab post bodies/links + localize on-disk inline images' (#103) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 2m30s
Build images / build-ml (push) Successful in 3m27s
CI / integration (push) Successful in 3m29s
2026-06-14 21:21:49 -04:00
bvandeusen 65ec29ba9b feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s
A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:58:40 -04:00
bvandeusen dfc3922d24 Merge pull request 'Merge dev → main: #830 rich post capture + external-host downloads (+ #768/#789/#739)' (#102) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 48s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m47s
CI / integration (push) Successful in 3m20s
2026-06-14 19:16:09 -04:00
bvandeusen 96c29c370b feat(ingest): localize inline post-body images to local copies (Phase 2)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m14s
Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):

- utils.paths.filehash_from_url — one source of truth for the extractor;
  patreon_client._filehash now delegates so capture- and render-time
  hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
  match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
  persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
  every inline image whose filehash maps to a stored image of THIS
  artist; unmatched / pre-Phase-2 images keep hotlinking.

Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:39:58 -04:00
bvandeusen 5e1655384f feat(ingest): recapture body + links for every walked post (Phase 5)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m14s
Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.

Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.

Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.

Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 16:14:40 -04:00
bvandeusen 8dbf29f803 feat(external): per-host enable toggles in Settings (Phase 4d)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m23s
Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.

Completes Phase 4. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 15:57:42 -04:00
bvandeusen 05f226a8f6 feat(external): zip-parity provenance/tagging + thorough worker logging
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m9s
Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).

- _route_files now mirrors download_service._phase3_persist branch-for-branch:
  imported/superseded → collect member_image_ids+image_id (provenance-linked via
  the synthesized sidecar, same as extracted-zip members) → caller enqueues
  tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
  on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
  unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
  fetch result (files/bytes) or fetch failure, per-file import decision, dead-
  letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
  linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
  exactly the member images.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 15:45:56 -04:00
bvandeusen bd2807cdd1 feat(external): mega.nz via megatools in the runtime image (Phase 4c)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m17s
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.

NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.

With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 15:41:18 -04:00
bvandeusen 82b26b8aaa test(external): unique seeded artist per host (fix uq_artist_name in sweep test)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m18s
2026-06-14 15:33:49 -04:00
bvandeusen 96e984cded feat(external): download worker for file-host links (Phase 4b)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Failing after 3m14s
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
  duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
  requeue-with-countdown if busy), fetch via external_fetch into the artist
  library tree, then route each file through importer.attach_in_place via a
  synthesized sidecar so it links to the SAME post (archive→ImageRecords,
  else→PostAttachment; on-disk original removed for captured files, art stays);
  thumbnail+ML enqueue for new images; status downloaded | failed | dead with
  attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
  columns in 4d — defaults on, rule #26).

Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.

mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:44:07 -04:00
bvandeusen 13253b18d1 feat(external): file-host fetcher subsystem (Phase 4a)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m21s
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox    : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire  : scrape the download page for the direct link + stream GET
- gdrive     : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega       : MEGAcmd `mega-get` subprocess (public link incl. #key)

HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.

Refs FC #830 (Phase 4a).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:29:20 -04:00
bvandeusen 896e4f248c style(importer): fix ruff import ordering for link_extract
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 42s
CI / integration (push) Successful in 3m20s
2026-06-14 13:19:51 -04:00
bvandeusen d96918d777 feat(posts): extract + record external file-host links (Phase 3)
CI / lint (push) Failing after 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m20s
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.

- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
  redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
  Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
  /attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
  worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
  (_apply_sidecar + upsert_post_record) so a re-import never resets a link's
  status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
  no-dup on reimport), serializer.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:15:36 -04:00
bvandeusen c342c73a25 feat(posts): faithful (semantic) HTML rendering of post bodies
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m20s
Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as
plain text. Now:
- html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings,
  inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay
  stripped; their text is preserved). Benefits the existing ProvenancePanel too.
- post_feed_service.get_post: add sanitized `description_html` to the DETAIL
  response (the feed list stays lightweight plain text by design).
- PostCard.vue: render description_html via v-html once expanded (fetched with
  detail); collapsed + no-detail fallback stay plain text. Styled close to the
  source (headings, images max-width, accent links, lists, quotes, code).

Tests: sanitizer (headings/img/lists survive, img javascript: src dropped);
get_post returns sanitized description_html.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 13:02:21 -04:00
bvandeusen ca25f688c3 fix(download): tolerate dl_result without post_record_paths
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 36s
CI / integration (push) Successful in 3m9s
The test_download_service stubs build dl_result as a SimpleNamespace that
doesn't set the new field; read it via getattr (matching the existing
retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any
caller that predates the field.
2026-06-14 12:50:34 -04:00
bvandeusen 796e92540a feat(patreon): capture media-less/text-only posts (post-only records)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Failing after 3m26s
Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.

- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
  unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
  getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
  write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).

Slice 0b of milestone #64. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:44:46 -04:00
bvandeusen 2c67c27044 feat(patreon): capture full post body via adaptive detail-fetch
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m24s
The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).

First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 12:29:06 -04:00
bvandeusen 7fcef53d5b fix(series): sticky tabs + controls on the Series view
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m14s
The Series tab strip and the Browse search/sort (and Suggestions controls)
scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab
controls into one sticky header pinned under the 64px TopNav. The controls
had to leave v-window — it clips sticky children — so they're driven by the
tab from the header instead of living inside each window-item.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 20:30:27 -04:00
bvandeusen 3eb08e926b Merge pull request 'fix(aliases): modal raw-key bug + alias visibility/management' (#101) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m13s
2026-06-12 14:02:00 -04:00
bvandeusen 5c3f8ebd70 fix(aliases): store modal alias under raw model key + make aliases visible/manageable
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m7s
The headline bug: aliases created from the modal NEVER resolved. Create
sent the normalized display name ('Sword', 'Uchiha Sasuke') while
resolution keys on the raw booru model key ('sword', 'uchiha_sasuke',
case-sensitive) — so the mapping was stored under a key nothing looks up,
and the prediction kept reappearing unaliased. The raw key wasn't even in
the /suggestions response, so the modal couldn't send it.

- Suggestion now carries raw_name (the model key an alias must use) and
  via_alias (surfaced via an operator alias); both serialized by the API.
- Modal alias-create sends raw_name, not display_name (the fix). Aliased
  suggestions show an 'alias' badge and a 'Remove alias' action; 'Treat as
  alias for…' is hidden for centroid hits (no model key) and already-aliased
  rows.
- Tag-side management: TagCard ⋮ → 'Aliases…' opens a dialog listing the
  model keys that fold into a tag, with remove (GET /api/tags/<id>/aliases +
  AliasService.list_for_tag). Creation stays in the modal suggestion flow.

Tests: full API round-trip locking the raw-key contract (raw_name exposed →
alias authored with it → resolves + via_alias on a later image);
list_for_tag (service + API); via_alias/raw_name on the existing service
suggestion tests. No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 13:05:58 -04:00
bvandeusen 9e81ced359 Merge pull request 'fix(images): percent-encode original-image URLs ('#' in paths 404'd)' (#100) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 36s
CI / integration (push) Successful in 3m15s
2026-06-12 00:44:56 -04:00
bvandeusen 7c4b24c80d fix(images): percent-encode original-image URLs ('#' in paths 404'd)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m17s
An image whose on-disk path contains '#' (post folders like 'BLUE#59')
served its hash-named thumbnail fine but 404'd the original: the unencoded
'#' in image_url was parsed by the browser as a URL fragment, so
'#59/01_timelapse.jpg' never reached the /images route. Add a shared
image_url(path) helper that percent-encodes the path (safe='/') and route
the 3 raw builders (gallery detail + 2 in series) through it. Not a
cleanup-tool deletion — the file is on disk; only the URL was wrong.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 00:36:44 -04:00
bvandeusen 11e9f5af60 Merge pull request 'fix(browse): tabs and search on one row' (#99) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 11s
CI / integration (push) Successful in 3m13s
2026-06-12 00:28:38 -04:00
bvandeusen 3e1303ea3c fix(browse): put tabs and search on one row
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m12s
Operator-asked: the tab strip and search field were stacked; place them
side-by-side in a single flex bar (tabs left, search + scope chips right),
wrapping to two rows only on narrow viewports.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 00:25:03 -04:00
bvandeusen 909fa37b15 Merge pull request 'Browse search + series numbering rework + kebab fix' (#98) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 2m50s
CI / integration (push) Successful in 3m16s
2026-06-12 00:14:31 -04:00
bvandeusen 2c544ad5af feat(browse): sticky tabs + per-tab search bar (server-side, scope-aware)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m10s
The Browse tab nav scrolled away (operator didn't know it existed) and
Posts had no search. Roll the tab strip + a shared search field into one
sticky block pinned under the 64px TopNav.

- Posts gains server-side text search: PostFeedService.scroll()/around()
  + /api/posts accept q (ILIKE over post_title OR description), applied
  INSIDE the artist/platform WHERE so search stays scoped to the active
  filter. Scope shown as clearable chips next to the search field.
- Artists/Tags search consolidates into the sticky bar: their inner
  search boxes are removed; they react to route.query.q (q is deep-
  linkable, e.g. /browse?tab=posts&q=foo). Platform/kind filters stay.
- Posts empty state now distinguishes 'no matches' from 'no posts yet'.

Tests: posts q-search matches title|description and stays artist-scoped
(service); q passthrough (api).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 00:04:06 -04:00
bvandeusen 90c68f8b2a fix(series): round the kebab backing on series cards
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 35s
CI / backend-lint-and-test (push) Successful in 56s
CI / integration (push) Successful in 3m14s
The tinted backing was set on the square .fc-kebab wrapper span while the
button is round, so a translucent square showed behind the round ⋮.
border-radius:50% makes the backing a circle matching the button.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 23:42:39 -04:00
bvandeusen 3e22e78aa4 test(series): assert group start_page, not per-page stated_page
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m11s
add_post now stamps the post's parsed START (constant) on every staged
pending page so the group start survives junk removal; list_pages
surfaces it as start_page. Update the stale per-page [9,10,11] assertion
to check grp["start_page"] == 9.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 23:17:14 -04:00
bvandeusen 013b9d7f06 feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Failing after 3m17s
Replaces the auto-renumbered 1..N position key with operator-OWNED page
numbers: sparse, gaps allowed, editable, never auto-renumbered. Order follows
the numbers; unnumbered pages sort to the tail. This is the fix for the model
that clobbered hand-set numbers on the flatten — numbers are now data, not a
derived sequence.

- series_service: drop the renumber-on-reorder/remove; order by page_number
  NULLS LAST; new set_page_number(image_id, n|None); list_pages returns `gaps`
  (one entry per missing-number run) + each pending group's parsed `start_page`;
  set_cover renumbers below the current min; place_pending(image_ids, start_page)
  numbers placed pages sequentially from the start (drop junk first → numbers
  line up); add_post stamps the parsed start on staged pages.
- api/tags: POST /series/<id>/pages/number (set one page's number); /pending/
  place takes start_page; removed /reorder.
- frontend: per-card editable number input; one gap block per gap with
  drop-on-edge to assign the adjacent number (middle → type); append drop zone;
  pending tray gets a "from page N" field + "Place from page N".
- tests reworked: sparse numbers + gaps, place-from-start, set-page-number route.

No migration; nothing destructive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 23:10:30 -04:00
bvandeusen dfab8f65ff Merge pull request 'feat(series): flat sequence + cosmetic dividers + pending staging (#789)' (#97) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 2m13s
Build images / build-ml (push) Successful in 2m49s
CI / integration (push) Successful in 3m9s
2026-06-11 22:07:57 -04:00