bvandeusen
96fffaff64
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FabledCurator
Self-hosted media curation — gallery, ML tagging, and subscription-driven downloading in one app. Part of the FabledSword family.
Combines what was ImageRepo (gallery, ML, importer) and GallerySubscriber (gallery-dl wrapper, subscriptions, credential capture) into a single product.
Status
Pre-v1. Not yet functional.
Quick start
For local development and testing, just:
docker compose up -d
# UI: http://localhost:8080
That uses sane dev defaults baked into docker-compose.yml and the dev
override (docker-compose.override.yml, auto-merged) — local builds, DEBUG
logging, exposed Postgres + Redis ports on the host. No .env required.
For a production-like deployment, override the dev defaults via shell env
or a .env file (see .env.example for the variable names) and use:
docker compose -f docker-compose.yml up -d
# (skips the override so containers pull registry images)
Deployment posture
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
CI / Forgejo setup
The repo's workflows expect:
-
Runner label
python-ci— a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Bothci.ymlandbuild.ymluse this label. The runner image (runner-base:python-ci) is built fromCI-Runner/CI-python/in the operator's workspace;make pushfrom that directory builds and pushes a new image when toolchain pins change. -
Repo secret
RELEASE_TOKEN— a Forgejo PAT with the following scopes:write:package+read:package— fordocker pushtogit.fabledsword.comwrite:release— for future release-cutting workflowswrite:issue— for future issue-management automation
Generate at https://git.fabledsword.com/user/settings/applications. The injected
GITHUB_TOKENcannot be used because it lackswrite:package.
License
Personal project; use at your own discretion.