Merge dev → main: #830 rich post capture + external-host downloads (+ #768/#789/#739) #102

Merged

bvandeusen merged 15 commits from dev into main 2026-06-14 19:16:09 -04:00

Conversation 0 Commits 15 Files Changed 40

+2677 -75

bvandeusen commented 2026-06-14 19:16:02 -04:00

Owner

Copy Link

Copy Source

Promotes accumulated dev work to main so the runtime image rebuilds. Primary driver: this build bakes megatools + gdown into the image (Dockerfile + requirements.txt) — required to actually run the off-platform downloaders from #830 Phase 4.

What lands

#830 — rich post capture + faithful render + external-host downloads (all phases 0–5)

• Full HTML body capture for every walked post (media + text-only), faithful sanitized render in the Curator view.
• Inline images served locally (CDN-filehash join), migration 0051.
• External file-host link extraction (mega/gdrive/mediafire/dropbox/pixeldrain), external_link table (0049), per-host Settings toggles (0050).
• Downloader subsystem: external_fetch.py (gdown for Drive, megatools dl for mega) + tasks/external.py worker (claim, per-host lock, attach-in-place, dead-letter/recovery/retention) on the download queue.

#768 — normalized image_prediction table; dropped the 100 GB tagger_predictions TOAST (0045/0046).
#789 — flat series sequence + cosmetic chapter dividers + global page numbering + pending staging (0047/0048).
#739 — DB-backup hardening: split stall threshold (40 min), local tempfile dump + bounded-kill, -Fc dumps.
Plus tag-alias raw-key fix, browse/series UX polish, DRY sweeps.

Migration head: 0051.

Deploy / test after merge

1. This push to main triggers build.yml → publishes :main + :latest with megatools/gdown.
2. Force a real re-pull of :latest on the services (a moved tag does NOT auto re-pull) — the download worker especially.
3. Enable hosts in Settings → External, then run a backfill on a source with off-platform links.

CI is green on dev head (5e16553/96c29c3).

Promotes accumulated `dev` work to `main` so the runtime image rebuilds. **Primary driver: this build bakes `megatools` + `gdown` into the image** (`Dockerfile` + `requirements.txt`) — required to actually run the off-platform downloaders from #830 Phase 4. ## What lands **#830 — rich post capture + faithful render + external-host downloads (all phases 0–5)** - Full HTML body capture for every walked post (media + text-only), faithful sanitized render in the Curator view. - Inline images served locally (CDN-filehash join), migration 0051. - External file-host link extraction (mega/gdrive/mediafire/dropbox/pixeldrain), `external_link` table (0049), per-host Settings toggles (0050). - Downloader subsystem: `external_fetch.py` (gdown for Drive, `megatools dl` for mega) + `tasks/external.py` worker (claim, per-host lock, attach-in-place, dead-letter/recovery/retention) on the `download` queue. **#768** — normalized `image_prediction` table; dropped the 100 GB `tagger_predictions` TOAST (0045/0046). **#789** — flat series sequence + cosmetic chapter dividers + global page numbering + pending staging (0047/0048). **#739** — DB-backup hardening: split stall threshold (40 min), local tempfile dump + bounded-kill, `-Fc` dumps. Plus tag-alias raw-key fix, browse/series UX polish, DRY sweeps. Migration head: **0051**. ## Deploy / test after merge 1. This push to `main` triggers `build.yml` → publishes `:main` + `:latest` with megatools/gdown. 2. **Force a real re-pull** of `:latest` on the services (a moved tag does NOT auto re-pull) — the `download` worker especially. 3. Enable hosts in Settings → External, then run a backfill on a source with off-platform links. CI is green on `dev` head (5e16553/96c29c3).

bvandeusen added 15 commits 2026-06-14 19:16:02 -04:00

fix(series): sticky tabs + controls on the Series view ... 7fcef53d5b

The Series tab strip and the Browse search/sort (and Suggestions controls)
scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab
controls into one sticky header pinned under the 64px TopNav. The controls
had to leave v-window — it clips sticky children — so they're driven by the
tab from the header instead of living inside each window-item.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(patreon): capture full post body via adaptive detail-fetch ... 2c67c27044

The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).

First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(patreon): capture media-less/text-only posts (post-only records) ... 796e92540a

Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.

- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
  unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
  getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
  write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).

Slice 0b of milestone #64. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(download): tolerate dl_result without post_record_paths ... ca25f688c3

The test_download_service stubs build dl_result as a SimpleNamespace that
doesn't set the new field; read it via getattr (matching the existing
retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any
caller that predates the field.

feat(posts): faithful (semantic) HTML rendering of post bodies ... c342c73a25

Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as
plain text. Now:
- html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings,
  inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay
  stripped; their text is preserved). Benefits the existing ProvenancePanel too.
- post_feed_service.get_post: add sanitized `description_html` to the DETAIL
  response (the feed list stays lightweight plain text by design).
- PostCard.vue: render description_html via v-html once expanded (fetched with
  detail); collapsed + no-detail fallback stay plain text. Styled close to the
  source (headings, images max-width, accent links, lists, quotes, code).

Tests: sanitizer (headings/img/lists survive, img javascript: src dropped);
get_post returns sanitized description_html.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(posts): extract + record external file-host links (Phase 3) ... d96918d777

Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.

- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
  redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
  Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
  /attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
  worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
  (_apply_sidecar + upsert_post_record) so a re-import never resets a link's
  status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
  no-dup on reimport), serializer.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

style(importer): fix ruff import ordering for link_extract 896e4f248c

feat(external): file-host fetcher subsystem (Phase 4a) ... 13253b18d1

Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox    : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire  : scrape the download page for the direct link + stream GET
- gdrive     : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega       : MEGAcmd `mega-get` subprocess (public link incl. #key)

HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.

Refs FC #830 (Phase 4a).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(external): download worker for file-host links (Phase 4b) ... 96e984cded

tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
  duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
  requeue-with-countdown if busy), fetch via external_fetch into the artist
  library tree, then route each file through importer.attach_in_place via a
  synthesized sidecar so it links to the SAME post (archive→ImageRecords,
  else→PostAttachment; on-disk original removed for captured files, art stays);
  thumbnail+ML enqueue for new images; status downloaded | failed | dead with
  attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
  columns in 4d — defaults on, rule #26).

Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.

mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(external): unique seeded artist per host (fix uq_artist_name in sweep test) 82b26b8aaa

feat(external): mega.nz via megatools in the runtime image (Phase 4c) ... bd2807cdd1

Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.

NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.

With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(external): zip-parity provenance/tagging + thorough worker logging ... 05f226a8f6

Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).

- _route_files now mirrors download_service._phase3_persist branch-for-branch:
  imported/superseded → collect member_image_ids+image_id (provenance-linked via
  the synthesized sidecar, same as extracted-zip members) → caller enqueues
  tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
  on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
  unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
  fetch result (files/bytes) or fetch failure, per-file import decision, dead-
  letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
  linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
  exactly the member images.

Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(external): per-host enable toggles in Settings (Phase 4d) ... 8dbf29f803

Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.

Completes Phase 4. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ingest): recapture body + links for every walked post (Phase 5) ... 5e1655384f

Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.

Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.

Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.

Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ingest): localize inline post-body images to local copies (Phase 2) ... 96c29c370b

Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):

- utils.paths.filehash_from_url — one source of truth for the extractor;
  patreon_client._filehash now delegates so capture- and render-time
  hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
  match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
  persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
  every inline image whose filehash maps to a stored image of THIS
  artist; unmatched / pre-Phase-2 images keep hotlinking.

Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit dfc3922d24 into main 2026-06-14 19:16:09 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#102