Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d`
gives three containers — FabledCurator, Postgres, Redis — where the stack
previously needed seven.
THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app
services separately and remains the right shape for a Swarm deployment spread
across hosts, where per-service rolling rollback and placement constraints
matter. This adds a compose file; it deletes none.
`entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES
and execs it as PID 1. Generated rather than checked in because a static
.conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of
the queue names — after celery_app.task_routes and the three collapsed in
steps 1, 2 and 4. Every one of those had already drifted when found.
Generating gives a stronger guarantee than "they match today": a lane added
to LANES gets a process, and a queue cannot end up with no consumer because
someone missed a file.
supervisord over s6-overlay: one pip dependency on an image already Python,
with per-program stop timeouts and stopasgroup. The process-group part is not
a detail — celery's prefork pool forks children, and a TERM reaching only the
parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and
zombie handling) comes from `init: true` instead. Nothing in FC talks to the
supervisor, so the choice is reversible without touching product code.
FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements
live only in Dockerfile.ml until step 6 merges the images, so an `ml` program
here would fail to import on every restart forever. `--with-ml` is the flag
step 6 turns on.
THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught:
1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord
parses that key as a COMMA-separated list, so it reads as
CELERY_QUEUES=default plus three malformed entries and the worker lane
would have consumed only `default`. Silent: the worker starts, reports
healthy, never picks up an import. Now quoted, and the test asserts the
quoted form rather than the bare substring, which passed either way.
2. The generator emitted `entrypoint.sh <lane.name>`, but `maintenance_long`
is not a role — compose runs it as the plain `worker` role with different
queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh
to assert every role a lane names actually exists.
3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY.
Harmless while only compose started it and set none; with a generated value
being passed, the lane would have sat at 1 until the reconcile noticed,
with nothing saying why.
The healthcheck asserts BOTH halves — hypercorn answers and every configured
lane is answering the broker. That is the failure mode consolidation creates:
docker can no longer see the lanes as separate services, so a web-only check
would report a healthy container with every lane inside it dead. It
deliberately ignores the `enabled` flag: a disabled lane still has a running
process with its consumers cancelled, and marking the container unhealthy for
turning tagging off would be wrong.
stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s)
rather than the average, with a test asserting no program's stopwaitsecs can
exceed what compose allows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.
NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.
With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three coupled operator-reported pains from the 2026-05-31 download
event audit:
1. `[patreon][warning] Not allowed to view post N` was bubbling up as
an error event, bumping consecutive_failures and parking the source
in "needs attention." The classifier's tier-gated branch was gated
on `return_code in (1, 4)`. Gallery-dl returns a different exit
code for mixed-failure runs (e.g. paywall warnings + a missing
yt-dlp dep flipping the exit bits), so the branch never fired and
the path fell through to UNKNOWN_ERROR. Widen the gate: when no
source-level error fired AND tier-gated warnings are present,
classify as TIER_LIMITED regardless of return code.
2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
"stranded by recovery sweep (no terminal status after time_limit)"
+ empty stdout/stderr. Root cause: subprocess.run timeout (900s)
and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
wiped the in-memory captured output and the DownloadEvent ended up
empty-logged 18 minutes later when the sweep finalized it. Drop
gallery-dl's default subprocess timeout to 870s — a 30s margin
shy of Celery's soft limit — so subprocess.TimeoutExpired always
wins the race and captures the partial stdout/stderr via the
existing handler.
3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
firing on every video attachment, causing per-item download
failures that masked legitimate tier-gated classification.
Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
video posts download normally and the per-item failure noise
disappears.
Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
CI failed at 'pip install -r requirements.txt' with Pillow 10.4.0 sdist
building from source under Python 3.14 and exiting with KeyError: '__version__'
(no 3.14 wheel exists for the 10.x line). The same risk applied to
asyncpg 0.29 (also no 3.14 wheel) and to psycopg 3.1.x (dep resolver was
backtracking through every minor trying to find a 3.14-compatible combo).
Bumps:
- pillow: 10.2-10.x -> 11.1-11.x (11.x has 3.13+ wheels)
- asyncpg: 0.29 -> 0.30 (0.30 added 3.13/3.14 wheels)
- psycopg: 3.1.x -> 3.2.x (current line, stable resolver)
- cryptography: 42 -> 44-45 (current abi3 wheels; the 42 line
worked via abi3 but bumping aligns
with the newer cffi-on-3.14 path)
This is the operator-flagged risk from FC-1 ("scientific stack may still be
catching up on 3.14 wheels — worth confirming before locking in"), realized.
The fix is to pin newer minimums, not to back off to 3.13.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pins runtime and ML deps separately so the regular web image stays lean.
Configures ruff for py312 with bugbear, async, and pyupgrade lints enabled.
psycopg sync driver included up-front for alembic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusenandClaude Opus 5