bvandeusen
ccbb5cbc9e
Operator hit an outage after the machine slept overnight: the agent showed "curator unreachable" in a loop while curator's API (lease) was actually fine and the browser could still load images — just slowly. Root cause is a feedback loop in the new pipeline: every download streams a full original through curator's single Python file-serving path, and the autoscaler grows DOWNLOADERS whenever the buffer is empty. When downloads are merely SLOW/failing, the buffer is empty for that reason — so the agent piled on more concurrent large-file GETs, saturating curator's web workers + NFS, which slowed curator (and its browser) further and produced more failures → more downloaders. Classic congestion collapse. - Failure-aware autoscaling: if transient download failures rose since the last decision, SHRINK the downloader pool toward the floor instead of growing — the empty buffer is caused by failures, not the GPU starving. It ramps back up only once downloads succeed again. - DL_MAX 24 → 8: 24 concurrent large-file downloads through one Python serving path is too many; 8 keeps a fast GPU fed without stampeding curator. - fetch_image timeout 180 → (10, 60): the read timeout is between-bytes, so a large-but-flowing download still completes, but a stuck/dead connection fails in 60s instead of hanging a downloader for 3 min and piling up stuck requests. Build marker 2026-07-01.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
FabledCurator
Self-hosted media curation — gallery, ML tagging, and subscription-driven downloading in one app. Part of the FabledSword family.
Combines what was ImageRepo (gallery, ML, importer) and GallerySubscriber (gallery-dl wrapper, subscriptions, credential capture) into a single product.
Status
Pre-v1. Not yet functional.
Quick start
For local development and testing, just:
docker compose up -d
# UI: http://localhost:8080
That uses sane dev defaults baked into docker-compose.yml and the dev
override (docker-compose.override.yml, auto-merged) — local builds, DEBUG
logging, exposed Postgres + Redis ports on the host. No .env required.
For a production-like deployment, override the dev defaults via shell env
or a .env file (see .env.example for the variable names) and use:
docker compose -f docker-compose.yml up -d
# (skips the override so containers pull registry images)
Deployment posture
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
CI / Forgejo setup
The repo's workflows expect:
-
Runner label
python-ci— a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Bothci.ymlandbuild.ymluse this label. The runner image (runner-base:python-ci) is built fromCI-Runner/CI-python/in the operator's workspace;make pushfrom that directory builds and pushes a new image when toolchain pins change. -
Repo secret
RELEASE_TOKEN— a Forgejo PAT with the following scopes:write:package+read:package— fordocker pushtogit.fabledsword.comwrite:release— for future release-cutting workflowswrite:issue— for future issue-management automation
Generate at https://git.fabledsword.com/user/settings/applications. The injected
GITHUB_TOKENcannot be used because it lackswrite:package.
License
Personal project; use at your own discretion.