bvandeusen
bd3f996582
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
FabledCurator
Self-hosted media curation — gallery, ML tagging, and subscription-driven downloading in one app. Part of the FabledSword family.
Combines what was ImageRepo (gallery, ML, importer) and GallerySubscriber (gallery-dl wrapper, subscriptions, credential capture) into a single product.
Status
Pre-v1. Not yet functional.
Quick start
For local development and testing, just:
docker compose up -d
# UI: http://localhost:8080
That uses sane dev defaults baked into docker-compose.yml and the dev
override (docker-compose.override.yml, auto-merged) — local builds, DEBUG
logging, exposed Postgres + Redis ports on the host. No .env required.
For a production-like deployment, override the dev defaults via shell env
or a .env file (see .env.example for the variable names) and use:
docker compose -f docker-compose.yml up -d
# (skips the override so containers pull registry images)
Deployment posture
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
CI / Forgejo setup
The repo's workflows expect:
-
Runner label
python-ci— a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Bothci.ymlandbuild.ymluse this label. The runner image (runner-base:python-ci) is built fromCI-Runner/CI-python/in the operator's workspace;make pushfrom that directory builds and pushes a new image when toolchain pins change. -
Repo secret
RELEASE_TOKEN— a Forgejo PAT with the following scopes:write:package+read:package— fordocker pushtogit.fabledsword.comwrite:release— for future release-cutting workflowswrite:issue— for future issue-management automation
Generate at https://git.fabledsword.com/user/settings/applications. The injected
GITHUB_TOKENcannot be used because it lackswrite:package.
License
Personal project; use at your own discretion.