Promotes accumulated dev work to main so the runtime image rebuilds. Primary driver: this build bakes megatools + gdown into the image (Dockerfile + requirements.txt) — required to actually run the off-platform downloaders from #830 Phase 4.
This push to main triggers build.yml → publishes :main + :latest with megatools/gdown.
Force a real re-pull of :latest on the services (a moved tag does NOT auto re-pull) — the download worker especially.
Enable hosts in Settings → External, then run a backfill on a source with off-platform links.
CI is green on dev head (5e16553/96c29c3).
Promotes accumulated `dev` work to `main` so the runtime image rebuilds. **Primary driver: this build bakes `megatools` + `gdown` into the image** (`Dockerfile` + `requirements.txt`) — required to actually run the off-platform downloaders from #830 Phase 4.
## What lands
**#830 — rich post capture + faithful render + external-host downloads (all phases 0–5)**
- Full HTML body capture for every walked post (media + text-only), faithful sanitized render in the Curator view.
- Inline images served locally (CDN-filehash join), migration 0051.
- External file-host link extraction (mega/gdrive/mediafire/dropbox/pixeldrain), `external_link` table (0049), per-host Settings toggles (0050).
- Downloader subsystem: `external_fetch.py` (gdown for Drive, `megatools dl` for mega) + `tasks/external.py` worker (claim, per-host lock, attach-in-place, dead-letter/recovery/retention) on the `download` queue.
**#768** — normalized `image_prediction` table; dropped the 100 GB `tagger_predictions` TOAST (0045/0046).
**#789** — flat series sequence + cosmetic chapter dividers + global page numbering + pending staging (0047/0048).
**#739** — DB-backup hardening: split stall threshold (40 min), local tempfile dump + bounded-kill, `-Fc` dumps.
Plus tag-alias raw-key fix, browse/series UX polish, DRY sweeps.
Migration head: **0051**.
## Deploy / test after merge
1. This push to `main` triggers `build.yml` → publishes `:main` + `:latest` with megatools/gdown.
2. **Force a real re-pull** of `:latest` on the services (a moved tag does NOT auto re-pull) — the `download` worker especially.
3. Enable hosts in Settings → External, then run a backfill on a source with off-platform links.
CI is green on `dev` head (5e16553/96c29c3).
The Series tab strip and the Browse search/sort (and Suggestions controls)
scrolled away on a long grid (operator-asked). Hoist the tabs + active-tab
controls into one sticky header pinned under the 64px TopNav. The controls
had to leave v-window — it clips sticky children — so they're driven by the
tab from the header instead of living inside each window-item.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).
First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.
- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).
Slice 0b of milestone #64. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test_download_service stubs build dl_result as a SimpleNamespace that
doesn't set the new field; read it via getattr (matching the existing
retry_after_seconds pattern) so phase 3 doesn't AttributeError on stubs or any
caller that predates the field.
Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as
plain text. Now:
- html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings,
inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay
stripped; their text is preserved). Benefits the existing ProvenancePanel too.
- post_feed_service.get_post: add sanitized `description_html` to the DETAIL
response (the feed list stays lightweight plain text by design).
- PostCard.vue: render description_html via v-html once expanded (fetched with
detail); collapsed + no-detail fallback stay plain text. Styled close to the
source (headings, images max-width, accent links, lists, quotes, code).
Tests: sanitizer (headings/img/lists survive, img javascript: src dropped);
get_post returns sanitized description_html.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded
in post bodies so they're never silently dropped, and surface them in the post
view. The download worker (Phase 4) walks these rows.
- link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon
redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups.
Reusable by every platform (runs off Post.description).
- external_link model + migration 0049: post_id/artist_id/host/url/label/status
/attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl.
worker statuses up front) + (post_id,url) unique.
- importer._sync_external_links: insert-missing on both import paths
(_apply_sidecar + upsert_post_record) so a re-import never resets a link's
status; runs for all platforms.
- post_feed_service.get_post: returns external_links (detail-only).
- PostCard: renders the links (host chip + label + status) once expanded.
- tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record +
no-dup on reimport), serializer.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
requeue-with-countdown if busy), fetch via external_fetch into the artist
library tree, then route each file through importer.attach_in_place via a
synthesized sidecar so it links to the SAME post (archive→ImageRecords,
else→PostAttachment; on-disk original removed for captured files, art stays);
thumbnail+ML enqueue for new images; status downloaded | failed | dead with
attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
columns in 4d — defaults on, rule #26).
Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.
mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.
NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.
With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).
- _route_files now mirrors download_service._phase3_persist branch-for-branch:
imported/superseded → collect member_image_ids+image_id (provenance-linked via
the synthesized sidecar, same as extracted-zip members) → caller enqueues
tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
fetch result (files/bytes) or fetch failure, per-file import decision, dead-
letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
exactly the member images.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator lever: disable a single file host (e.g. mega.nz when it's banning)
without touching the others. Five booleans on import_settings
(extdl_<host>_enabled, default true — works out of the box, rule #26); the
worker already reads them via getattr so no worker change. Migration 0050 +
model fields + settings GET/PATCH (uniform boolean validation) + a
'External file-host downloads' card in the subscriptions Settings tab.
Completes Phase 4. Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.
Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.
Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.
Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):
- utils.paths.filehash_from_url — one source of truth for the extractor;
patreon_client._filehash now delegates so capture- and render-time
hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
every inline image whose filehash maps to a stored image of THIS
artist; unmatched / pre-Phase-2 images keep hotlinking.
Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Promotes accumulated
devwork tomainso the runtime image rebuilds. Primary driver: this build bakesmegatools+gdowninto the image (Dockerfile+requirements.txt) — required to actually run the off-platform downloaders from #830 Phase 4.What lands
#830 — rich post capture + faithful render + external-host downloads (all phases 0–5)
external_linktable (0049), per-host Settings toggles (0050).external_fetch.py(gdown for Drive,megatools dlfor mega) +tasks/external.pyworker (claim, per-host lock, attach-in-place, dead-letter/recovery/retention) on thedownloadqueue.#768 — normalized
image_predictiontable; dropped the 100 GBtagger_predictionsTOAST (0045/0046).#789 — flat series sequence + cosmetic chapter dividers + global page numbering + pending staging (0047/0048).
#739 — DB-backup hardening: split stall threshold (40 min), local tempfile dump + bounded-kill,
-Fcdumps.Plus tag-alias raw-key fix, browse/series UX polish, DRY sweeps.
Migration head: 0051.
Deploy / test after merge
maintriggersbuild.yml→ publishes:main+:latestwith megatools/gdown.:lateston the services (a moved tag does NOT auto re-pull) — thedownloadworker especially.CI is green on
devhead (5e16553/96c29c3).The feed endpoint (/api/posts) returns `content` empty for many posts, so post bodies — their formatting, inline <img>, and external <a href> links — were never captured (the post showed "(no description)"). Enrich an empty feed body from the per-post detail endpoint (/api/posts/{id}) before writing the importer sidecar, memoized by mutating the shared post dict so a multi-image post fetches detail exactly once and fully-seen posts (no fresh download) pay nothing. Best-effort by design: a body we can't fetch returns None and never fails the walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is keyed on external_post_id; an empty body parses to None and isn't applied). First slice of milestone #64 (rich post capture + faithful rendering + external-host downloads). Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Shared, reusable fetchers for the 5 off-platform hosts behind one signature (fetch_external(host, url, dest_dir, ...) -> FetchResult): - dropbox : force dl=1 + stream GET - pixeldrain : GET /api/file/{id} - mediafire : scrape the download page for the direct link + stream GET - gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs - mega : MEGAcmd `mega-get` subprocess (public link incl. #key) HTTP/gdown/subprocess go through module seams so unit tests run without network/gdown/MEGAcmd. fetch_external never raises — every backend failure (transport, non-200, scrape miss, subprocess error, stop) is captured on .error so the worker (next slice) records it and moves on. mega's binary lands in the runtime image in a later slice; the code is complete + tested now. Refs FC #830 (Phase 4a). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>