bvandeusen
8a0237eeea
The failing "poison" jobs were 800MB+ 4K VR videos: the agent pulled the ENTIRE file into memory (r.content) just to sample a few frames, which buffered ~1GB in RAM and — on any slow/contended media store — got cut off mid-download (ChunkedEncodingError), failed, and re-leased forever. Measured the media read at ~4–6 MB/s (raw off the share, curator out of the path), so no serving-layer tweak helps; the file simply shouldn't be fully downloaded. Environment-agnostic fix (works for any deployment, completes even when slow): - media.sample_frames_from_url(): point ffmpeg straight at curator's /images URL. It Range-reads only the video index + up to max_frames of content — never the whole file — and reconnect flags resume a dropped transfer instead of failing. Generous, env-tunable timeout (FFMPEG_TIMEOUT, default 1200s) = completion over speed. Removes the bytes-based sample_frames (dead once videos stream). - worker._download_decode: videos now stream (no fetch_image, no RAM blowup); stills still download+decode. On an ffmpeg miss, probe curator liveness (client.is_reachable) → fail the job if curator is up (unprocessable file, stops the infinite re-lease) vs release if curator is down (transient, survives a redeploy). Auth header passed so it works whether or not /images is gated. Build marker 2026-07-01.6. Refs issue #1225. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
FabledCurator
Self-hosted media curation — gallery, ML tagging, and subscription-driven downloading in one app. Part of the FabledSword family.
Combines what was ImageRepo (gallery, ML, importer) and GallerySubscriber (gallery-dl wrapper, subscriptions, credential capture) into a single product.
Status
Pre-v1. Not yet functional.
Quick start
For local development and testing, just:
docker compose up -d
# UI: http://localhost:8080
That uses sane dev defaults baked into docker-compose.yml and the dev
override (docker-compose.override.yml, auto-merged) — local builds, DEBUG
logging, exposed Postgres + Redis ports on the host. No .env required.
For a production-like deployment, override the dev defaults via shell env
or a .env file (see .env.example for the variable names) and use:
docker compose -f docker-compose.yml up -d
# (skips the override so containers pull registry images)
Deployment posture
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
CI / Forgejo setup
The repo's workflows expect:
-
Runner label
python-ci— a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Bothci.ymlandbuild.ymluse this label. The runner image (runner-base:python-ci) is built fromCI-Runner/CI-python/in the operator's workspace;make pushfrom that directory builds and pushes a new image when toolchain pins change. -
Repo secret
RELEASE_TOKEN— a Forgejo PAT with the following scopes:write:package+read:package— fordocker pushtogit.fabledsword.comwrite:release— for future release-cutting workflowswrite:issue— for future issue-management automation
Generate at https://git.fabledsword.com/user/settings/applications. The injected
GITHUB_TOKENcannot be used because it lackswrite:package.
License
Personal project; use at your own discretion.