1253 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	0f98e46200	docs: the merged image's cost, measured — and it corrects my own estimate (4296) ... I wrote in ffcd130's Dockerfile comment that merging ML in means "everyone pulls it, including the many who will never turn tagging on", framed as a real cost the milestone accepted. My working estimate behind that was ~4GB. Measured from run 7273's build-web log: torch 2.12.1+cpu wheel 192.3 MB torchvision 0.27.1+cpu 1.8 MB transformers / onnxruntime / opencv / sklearn and friends 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB largest newly-pushed layer 222.07 MB The ML code adds a few HUNDRED MB, not gigabytes. The `--index-url` CPU resolution is what makes that true — the default PyPI torch wheel carries the CUDA runtime and is ~2GB by itself, and the log confirms 2.12.1+cpu resolved, so it is working as intended rather than as intended-but-unverified. Why this matters beyond a comment being wrong: it settles the trade this step was explicitly asked to weigh and could not, and it reverses how close the call looked. Baking the weights in adds ~3.5GB to every pull for a feature many adopters never enable; shipping the code and fetching on demand adds ~350MB. An order of magnitude, where the estimate had them within 15% of each other. Off-by-default is not a judgement call here, it is arithmetic. The gigabytes were always in the MODEL, and the model is not in the image. Two things NOT measured, still: the total image size (the push only transfers layers the registry lacks, so a push log cannot give it) and the per-slot resident RAM, which stays flagged `measured=False` in the lane table and renders as "about" in the UI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:02:47 -04:00
bvandeusenandClaude Opus 5	f174981b07	fix: ml and web are one image, so they are one path set (4296) ... Run 7271. `test_every_copied_path_is_in_the_artifacts_path_set` — a guard I did not know this repo had — caught the merge's real consequence within minutes, and it is precisely #3202's class: ml: Dockerfile copies 'frontend/package.json' into the ml image, but no include in scripts/artifacts.sh covers it. web: Dockerfile copies 'requirements-ml.txt' into the web image, but no include covers it. Both follow from step 6 and neither was visible by reading my own diff. The merged Dockerfile carries the frontend-builder stage, so the ml image now copies frontend files; and the web image now installs the ML requirements. Left as they were, each artifact's derived version would have stopped moving when those files changed, and a pinned build would serve stale bytes — silently, since nothing fails when a version simply does not advance. The fix is structural rather than two more entries. `fabledcurator-ml` IS web's image now: same Dockerfile, same context, same bytes, published under a second name only because the operator's Swarm stack still references it. So ML_PATHS is WEB_PATHS by assignment, and `cmd_paths` gives ml the same deriver and extension append — the XPI is in those bytes too. Two lists describing one image is the duplication this milestone has been collapsing all day. It existed for about an hour and the guard found it first, which is the argument for the guard. I also predicted this failure would be the image build. It was not; the build passed and the unit lane failed. Worth noting because the prediction was confident and wrong, and reading the log took one call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:57:09 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusenandClaude Opus 5	ecbd325437	feat: an optional lane says it is optional, and what enabling it costs (4296) ... Operator, 2026-09-22: "since the ml-worker is optional it should be shown as such in the UI and have a warning about what it does and that it pulls the models and what models and their projected size and ram requirements to run." The card previously said "a few GB, once" — a number sourced from nothing, which is exactly the hand-wave I had flagged in this step's own survey log as something that should be measured rather than asserted. ONE FACT CORRECTED WHILE WRITING THE COPY. I had named the lane "ML tagging". It downloads an EMBEDDER: google/siglip-so400m-patch14-384. WD14 tagging is the GPU agent's job — celery_app.py:5 still names both, but that has been stale since B3 (#1238), when the agent took over and this lane was left as the CPU embed fallback for stacks running no agent (see MLSettings.cpu_embed_enabled). Telling someone the lane "does tagging" would have been wrong in exactly the way this request exists to prevent. The facts are structured data on the lane, not prose in a component: ModelRequirement(repo, approx_download_bytes, approx_resident_bytes, measured). The API carries them; the card renders them. Numbers come from the system, wording from the UI. ML_BYTES_PER_SLOT IS NOW DERIVED from that requirement rather than stated separately. They have to be one number: the figure quoted to the operator before they enable the lane and the figure the cap enforces. Two copies could disagree, and the UI would promise a slot the cap then refuses. `measured=False` travels with the numbers and the card renders "about". They are estimates from the checkpoint's parameter count and dtype — ~877M params at fp32 is ~3.5GB of weights — not from a build. This decides whether someone's server survives, so it is labelled rather than rounded into something that reads like a fact. A test asserts the flag is false, to be flipped in the same commit that records a real measurement. The card now shows: an "optional" chip in the row itself (someone scanning the table should not have to enable a lane to learn it was never required), and before the switch, what the lane does, that you only need it if you are NOT running the GPU agent, the repo id, the download size, the per-slot RAM, and why the ceiling is what it is — including saying plainly when a box has too little memory to run it at all. Keyed on the lane's own `optional` flag, not on the name 'ml', so a second optional lane gets the same treatment without anyone remembering to add it. A test asserts no REQUIRED lane declares a model: if one ever needs a download, it stops being required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:45:22 -04:00
bvandeusenandClaude Opus 5	e579333455	docs: record that the ml :ro loss is a ruled non-issue (4295) ... Operator, 2026-09-22: "I don't care about the :ro loss thank you for calling it out repeated but I don't care." Raised three times across the milestone body, this file's header and two reports. Recorded as settled at the point someone would rediscover it, so it is not raised a fourth time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:38:49 -04:00
bvandeusenandClaude Opus 5	172e33de9a	feat: run web and every worker lane in one container (4295) ... Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d` gives three containers — FabledCurator, Postgres, Redis — where the stack previously needed seven. THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app services separately and remains the right shape for a Swarm deployment spread across hosts, where per-service rolling rollback and placement constraints matter. This adds a compose file; it deletes none. `entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES and execs it as PID 1. Generated rather than checked in because a static .conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of the queue names — after celery_app.task_routes and the three collapsed in steps 1, 2 and 4. Every one of those had already drifted when found. Generating gives a stronger guarantee than "they match today": a lane added to LANES gets a process, and a queue cannot end up with no consumer because someone missed a file. supervisord over s6-overlay: one pip dependency on an image already Python, with per-program stop timeouts and stopasgroup. The process-group part is not a detail — celery's prefork pool forks children, and a TERM reaching only the parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and zombie handling) comes from `init: true` instead. Nothing in FC talks to the supervisor, so the choice is reversible without touching product code. FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements live only in Dockerfile.ml until step 6 merges the images, so an `ml` program here would fail to import on every restart forever. `--with-ml` is the flag step 6 turns on. THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught: 1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord parses that key as a COMMA-separated list, so it reads as CELERY_QUEUES=default plus three malformed entries and the worker lane would have consumed only `default`. Silent: the worker starts, reports healthy, never picks up an import. Now quoted, and the test asserts the quoted form rather than the bare substring, which passed either way. 2. The generator emitted `entrypoint.sh <lane.name>`, but `maintenance_long` is not a role — compose runs it as the plain `worker` role with different queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh to assert every role a lane names actually exists. 3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY. Harmless while only compose started it and set none; with a generated value being passed, the lane would have sat at 1 until the reconcile noticed, with nothing saying why. The healthcheck asserts BOTH halves — hypercorn answers and every configured lane is answering the broker. That is the failure mode consolidation creates: docker can no longer see the lanes as separate services, so a web-only check would report a healthy container with every lane inside it dead. It deliberately ignores the `enabled` flag: a disabled lane still has a running process with its consumers cancelled, and marking the container unhealthy for turning tagging off would be wrong. stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s) rather than the average, with a test asserting no program's stopwaitsecs can exceed what compose allows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:32:27 -04:00
bvandeusenandClaude Opus 5	da48edf7da	feat: the worker-lanes card — see each lane, change its slots (4294) ... Milestone 422 step 4. Rule 27: no UI, no ship. This is where the previous three steps become usable. REACHABLE AT: Settings -> Activity -> Worker lanes, directly under the "Queues + workers" pane. Written from opening the view, not from memory of having built it — lesson #4282, and #3463 is the same trap landing inside milestone 365, where the System page shipped with no navigation to it. Under that pane deliberately, not in the System tab. System answers "is everything running", where every control would be about something broken. This is about something working that should work harder, and it belongs beside the backlog it reacts to: you watch a queue grow and give that lane another slot without leaving the pane. Per lane: queues, pending, busy, a stepper, an enable switch. - PENDING is depth + reserved, not LLEN. Celery prefetches, so LLEN alone reads 0 while a worker holds tasks in memory — the number that would make someone think a buried lane was idle. - NOT ANSWERING, never "stopped". present=false means nothing replied; saying stopped would send the operator looking for a crash that has not happened. - THE CEILING IS ON SCREEN, with "(memory)" on the ML lane. It is the one number here the operator cannot change, so it has to justify itself; a greyed stepper with no explanation reads as a bug. - `busy` is per lane, so adjusting one does not freeze the others. TWO OUTCOMES THAT MUST NOT COLLAPSE INTO ONE MESSAGE. A stored-but-unpushed change (applied:false — the lane is restarting) is information: the value is saved and the reconcile will carry it, so the card says so and invites waiting. A refused value (400) is an error and shows the endpoint's own sentence. Collapsing them would make one of the two invite the wrong action. A BUG CAUGHT BEFORE COMMIT: the card read `e.detail?.detail`, but ApiError puts the parsed body on `.body` and `.message` on the short `error` key. It would have shown the operator the bare word "refused" with no reason — the exact failure that line exists to prevent. The test would not have caught it either: `rejects.toThrow()` passes whether the sentence is reachable or not. It now asserts `err.body.detail` specifically. Also: queueOptions in SystemActivityTab was a fourth hand-kept copy of the queue list and had drifted — `maintenance_long` was missing, so activity on that lane could not be filtered for at all despite four task routes pointing there. Added, and DELIBERATELY left as a written-out list rather than derived like the other three were: this filters task_run HISTORY, so a derived list would hide the filter for any queue that has rows but no longer has a lane — precisely when someone is looking — and would be empty whenever the endpoint is down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:23:14 -04:00
bvandeusenandClaude Opus 5	5f8c63f61b	feat: reconcile every running lane back to its stored slots (4293) ... Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by its supervisor comes back at its ENV concurrency, silently below whatever the operator set, and nothing on step 2's write path would ever notice. Storing the value made it survivable; this makes it survive. A BEAT TASK, NOT A HOOK IN WEB — a deliberate deviation from the step as written, for a reason already recorded in this codebase. Step 3 said "web applies the stored values after it starts". It cannot: service_roster.py documents that hypercorn runs --workers 4, so anything in before_serving becomes four concurrent loops per container hammering the broker forever. service_roster's own answer — refresh on demand from whichever request arrives — was also rejected, because the two solve different problems. A stale ROSTER only misleads someone looking at it, so recomputing when they look is exactly right. A lane running at the wrong size is doing less work than it was told to whether or not anyone is watching, and the case that matters is a deploy at 3am followed by a backlog nobody is awake to see. So: unattended, every 5 minutes, on the quick `maintenance` lane beside the other recovery sweeps. Accepted cost — a dead scheduler stops reconciliation, but a dead scheduler already stops every other sweep and the roster reports it, so this adds no new blind spot. A BUG I WROTE AND CAUGHT BEFORE COMMITTING. The first version called set_lane_enabled_sync unconditionally, so a settled system re-sent add_consumer for every queue on every tick — forever. Harmless per call (add_consumer on an already-consumed queue does nothing), unbounded in aggregate, and completely invisible. That is lesson #4183's failure mode exactly, in the very function whose docstring cites it. Worse, my test would not have caught it: it asserted only on grew/shrank. LaneLiveState now carries `consuming` — which queues a lane is actually serving, distinct from the queues it was configured with — so the reconcile compares before acting. The test now asserts ALL FOUR control families are silent on a settled tick, plus a new case for an already-disabled lane, which is the other half of the same fixed point. An absent lane is SKIPPED, not corrected. present=False means nothing answered, not zero slots; correcting it would be a conclusion from an unswept read (snippet #3969), and there would be nothing to send the message to. One lane failing does not stop the others. One inspect serves every lane: the two setters now take an optional pre-fetched LaneLiveState, so a tick costs one broker round trip rather than one per lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:08:38 -04:00
bvandeusenandClaude Opus 5	a9c1b421a7	feat: change a lane's slots on a running system, over the broker (4292) ... Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to its live pool; `POST /api/system/workers/<name>` changes it. NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because restarting a dead worker needs a socket the web container deliberately does not have. That holds for restarting a CONTAINER; it does not hold for changing how much work a RUNNING worker does. celery's pool_grow / pool_shrink / add_consumer / cancel_consumer send a message over the Redis the app already uses, and the worker resizes itself. No new privilege, no new surface, and the security question that deferred this is never raised. PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a restart drops every lane to its env concurrency — so a UI that only pushed would lose the setting on the next deploy with nothing to show for it (lesson #4202). Storing alone would describe nothing until something restarted. A failed PUSH is not a failed setting: 200 with `applied: false` and a reason, so the UI says "saved, not yet live" rather than "that didn't work". Step 3's reconcile carries it when the lane answers again. PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH destination, so while `worker` runs `replicas: 2` a single delta from an aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY means — one process's pool — and each replica is driven to it from its OWN current size, so replicas that drifted apart converge rather than moving in lockstep. I wrote this wrong first: the docstring claimed per-replica while the code computed one delta from the max across replicas. LaneLiveState now carries `pools` per hostname and exposes `pool` as a property. A replica already at the target is sent nothing at all — the reachable fixed point step 3's periodic reconcile needs, or it re-issues a grow of zero every tick forever (lesson #4183). A replica that answered inspect but not stats is NAMED in the error rather than skipped silently, since otherwise it would run at a size the UI claims it does not. `present=False` is not "zero slots", it is "nothing answered" — kept distinct throughout, because step 3 skips an absent lane rather than correcting it. /workers now also reports pool size (from `insp.stats()`) and RESERVED count. Celery prefetches, so tasks that have left the Redis list but not started are invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker memory. `pending` is depth + reserved. The UI is misleading without this and step 7's autoscaler would be simply wrong. Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES, whose own comment admitted the coupling ("must match celery_app.task_routes") and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed. Now derived from LANES. The rendered order changes to lane grouping, which is the better shape for a lane-oriented UI. Separate blueprint rather than folding into system_activity, which states in its first line that it is read-only and answers a different question — its /workers is keyed on celery HOSTNAME and reports which nodes answered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:01:23 -04:00
bvandeusenandClaude Opus 5	5974a1bfbc	fix: two errors in the worker-lane tests (4291) ... Both mine, both in tests/test_worker_lanes.py, neither in the code under test. Run 7242. RUFF I001 — two blank lines between the import block and the first module-level comment. Rule 102 names this exact trap ("exactly ONE blank line between imports and a module-level constant/comment/pytestmark") and I was pointed at that rule repeatedly before opening it. SIX FAILURES in test_worker_lane_check_constraints — the test asserted bare constraint names, but Base.metadata's naming_convention has already applied the `ck_worker_lane_` prefix by the time __table__.constraints is read. The failure output is worth keeping: it shows the model emits exactly the three intended constraints, prefixed once — ck_worker_lane_slots_non_negative slots >= 0 ck_worker_lane_cap_non_negative slots_cap >= 0 ck_worker_lane_slots_within_cap slots <= slots_cap — which is the model behaving correctly, and confirms the migration's op.f() names match what the ORM produces. The assertion is now an equality against the prefixed names plus an explicit check for a doubled prefix. That is strictly more valuable than what I wrote: a bare-name assertion would have passed just as happily against `ck_worker_lane_ck_worker_lane_slots_within_cap`, which is the defect alembic 0088 had to rename four constraints for (#3275). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:53:30 -04:00
bvandeusenandClaude Opus 5	84f13135ce	feat: worker lanes become rows — slots, a settable cap, a derived ceiling (4291) ... Milestone 422 step 1. The data model the rest of the milestone reads. No behaviour change: nothing consumes these rows yet, and every lane still boots at its CELERY_CONCURRENCY env value. Three numbers, not two, per the operator's distinction — the derived value is a cap ON the cap: slots <= slots_cap <= derived_ceiling (live) (operator) (computed) They can always lower their own cap; they cannot raise it past what the container can hold. The ceiling is never stored, so a row written on a 32GB host and later run in a 4GB container is bounded by the 4GB. `services/worker_lanes.py` is the one place that knows the lane set. `models/worker_lane.py` holds only what an operator may change. Two deviations from the step as written, both deliberate: QUEUES ARE NOT A COLUMN. The step body said the row carries its `-Q` list, but a lane's queues are decided by celery_app's task_routes, not by preference — an operator cannot move a backup off maintenance_long. Storing them would create a row that can contradict the routing table, with nothing to notice until a queue had no consumer. So queues are code, slots are data. `test_every_routed_queue_has_a_lane_that_serves_it` reads the real routing table and fails if a route is ever added without a lane. ROLE_NAMES IS NOW DERIVED, not left alone. It was a hand-kept second copy of "queue set -> display name" and had already drifted: maintenance_long is a live lane with four task routes and a dedicated worker in the operator's stack, and the roster did not know its name — so the System tab labelled it `Worker (maintenance_long)`. Adding a lane table beside it would have made three copies. The ceiling honours cgroup limits rather than the host's. `os.cpu_count()` reports the HOST's cores from inside a container, so a 4-core quota on a 32-core host would otherwise offer 32 slots — and the operator's own stack sets `cpus: '4.0'` on ml-worker, so that is real configuration, not a hypothetical. Memory reads cgroup v2 then v1, and recognises v1's PAGE_SIZE-aligned LONG_MAX sentinel by magnitude rather than treating it as petabytes. Every uncertain case fails LOW. An unreadable limit yields UNKNOWN_CEILING, never unlimited — not knowing how much memory there is must not read as plenty. A box too small to hold one model beside the web process gets an ML ceiling of 0 rather than a floor of 1: offering a slot that OOMs the container the first time it is used is exactly what this exists to prevent. ML_BYTES_PER_SLOT is 4 GiB and is UNMEASURED — flagged as such in the code, with the method for replacing it with a real figure. It decides whether a stranger's server survives enabling tagging, so it errs toward refusing a slot that would have fitted. Seeded one-of-each with ml at 0 and disabled (alembic 0103). ML off is step 6's requirement arriving early: enabling the lane is what triggers the SigLIP download, and rule 164 permits a runtime fetch only for a feature that is optional and clearly off. The seed values are literals rather than an import of LANES — a migration is a statement about one moment, and importing the live defaults would silently change what this revision does on a fresh database in 2027. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:48:25 -04:00
bvandeusenandClaude Opus 5	aa2bb665b9	fix: serialise builds per branch, and pin :c-<sha> to the digest this run built (4290) ... Two independent defects from one missing mechanism. `grep -rn concurrency .forgejo/workflows/` returned nothing, so two pushes to a branch ran build.yml in full parallel with no ordering. Both jobs read `fc.revision` off the channel tag before either has pushed, so both miss the reuse check and both build. Whichever finishes LAST owns the tag. Half 1 — the rolling tag. A slower older build leaves `:dev` carrying content older than the commit that moved it. Family rule 146 says a rolling channel refreshes itself; this is the case where it quietly does not. Self-healing on the next push. Half 2, and the reason this is not filed low — `:c-<sha>`. The repoint step wrote every non-channel tag by copying the channel tag BY NAME, so the immutable rollback tag named whatever `:latest` pointed at when that step happened to run. Lose the race and `:c-<shaA>` names the OTHER run's bytes. Rule 145 makes that tag the rollback unit and immutable, so this does not break immutability — it makes the tag wrong from birth, and immutability then guarantees nobody ever corrects it. Nothing goes red; it surfaces the day someone rolls back and gets a commit they did not choose. Not self-healing. Two fixes, deliberately both: A. A workflow-level `concurrency` group keyed on `github.ref`, so dev and main never block each other. `cancel-in-progress: false` — queue, never cancel: cancelling could kill sign-extension mid-AMO-upload, leaving the version registered at AMO with no cached asset, which is the unrecoverable stuck state that job's rollback trap exists to prevent, reached by another door. Not keyed on BUILD_REF because the group is evaluated before any job starts and cannot read the `env` context. B. Each build-push-action step gains `id: build`, and the repoint step copies from `$IMAGE@${{ steps.build.outputs.digest }}` — the manifest THIS run pushed — rather than from the channel tag by name. On a reuse hit there is no digest and the channel tag remains correct by definition: "hit" means that tag already carries this commit's fc.revision. B is not redundant with A. A depends on this Gitea honouring a key whose failure mode is silent, and this file has been burned by exactly that before (the `format()` note records `true == 'true'` evaluating FALSE on run 5270, every lane green, the feature simply not happening). B holds the :c-<sha> correctness property whether or not A is honoured. The loop's exclusion is now keyed on CHANNEL_REF rather than on SOURCE, and that is load-bearing: SOURCE may now be a digest ref, which never equals a tag string, so testing against it would stop excluding the channel tag. imagetools would index-wrap it and `.Image.Config.Labels` would stop resolving, killing the reuse label permanently — #3183 arriving again. Neither half is verifiable on a dev push: dev's tag list is the channel tag alone, so the repoint correctly does nothing there. B is observable on the next merge to main (:c-<sha>'s digest must equal the build step's), and A by pushing twice in quick succession and reading the run list for a queued second run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 22:07:53 -04:00
bvandeusenandClaude Opus 5	bd92fb46b3	test: pin the JS<->Py artist-pattern mirror with a shared sample table (3093) ... `PLATFORM_ARTIST_PATTERNS` (extension/lib/platforms.js) and `_PLATFORM_PATTERNS` (extension_service.py) are two hand-kept copies of one table whose only guard was the comment "keep in sync by hand; reviewers catch drift" — the same guarantee manifest.json had before #3069, where deviantart sat in the manifest for seven weeks after the product dropped it. Drift here is worse than the manifest case, because the two copies gate opposite halves of ONE interaction: the JS copy decides whether the "Add to FC" button appears, the Python copy decides whether the resulting POST is accepted. JS looser than Py shows a button that 400s; Py looser than JS silently never offers a button for a URL the backend would take. #1485 (the Patreon /c/ and /cw/ shapes) was the second of those, and its fix had to be applied to both files by hand. The two-runtimes objection to a shared SOURCE file is fair, so this tests the invariant instead of the source. `extension/test/artist-url-samples.json` is one table of 25 URL samples — match (with the expected slug) and no_match, each with a `why` — read by BOTH suites and asserted against each one's own copy of the patterns. Neither runtime imports the other; a change to one copy alone turns the other runtime's suite red. Both halves also assert their own coverage: the sample platforms must equal the platforms that actually have an artist pattern, and every platform must have samples in both directions. Without that, deleting a platform's samples would make the guard pass by testing less. Discord is deliberately in neither table — it is channel-based and has no creator page to put a button on. The no_match half asserts `_derive` RAISES rather than merely missing the platform: it tries every pattern in turn, so a nav page some other platform's pattern happened to swallow would still be accepted by the backend — the same defect wearing a different platform name. Samples live under extension/test/ because that path is excluded from both the XPI file set and the extension version derivation (packaging.sh: NOT_PACKAGED_TRACKED and NOT_VERSION_RELEVANT both carry `test/**`), so adding samples ships no bytes and forces no re-sign. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 21:04:53 -04:00
bvandeusenandClaude Opus 5	24b10d0ffa	feat: retire pixiv entirely — delete its code, its ledgers, its credential (3977, 3978, 3979) ... Milestone #406 phase 2, with issue #3980 folded in. Phase 1 (2026-09-13) unregistered pixiv so nothing could reach it; the code has sat in the tree uncalled since. DeviantArt is why the second half is not left for later — #3069 retired it in code on 2026-08-27 and its stored session was still in the database seven weeks on. Step 5 — the code. Deletes pixiv_client, pixiv_downloader, pixiv_ingester, platforms/pixiv and their three test modules and fixture, then edits out every remaining reference: the dispatch entry, the campaign-id and verify branches in download_backends, the display-name branch in extension_service, and the comments that still described pixiv as live. The consolidation check the step asked for comes back negative: native_ingest_common has seven non-pixiv callers (patreon, subscribestar, membership_reconcile, membership_roster, ingest_core), so nothing there drops to a single user. Step 6 — the data, alembic 0102. Drops pixiv_seen_media and pixiv_failed_media, and deletes credential rows whose platform is not registered. Written as "not registered" rather than "pixiv" at the step's explicit ask, which is what makes one migration cover two retirements: the pixiv OAuth refresh token and DeviantArt's leftover session (#3980). It is also the only way either row can go — the credentials UI renders one card per platform from /api/platforms and looks the credential up by key, so an unregistered platform's row has no card and no Remove button. Pixiv's Source rows are KEPT, changing the milestone's original data table on the operator's call. `platform` is stored only on Source; neither Post nor ImageRecord carries it. Both FKs are ON DELETE SET NULL, so a delete would not lose the art — but it would drop every pixiv image into the gallery's __unsourced__ bucket and strip the platform chip off every pixiv post. The rows stay disabled (0097) and unregistered, so nothing schedules or downloads through them. Keeping them costs nothing and keeps the attribution that "the art already downloaded from pixiv stays" is about. Step 7 — the guard. test_pixiv_code_and_tables_are_gone asserts absence from the module table and from Base.metadata, not from prose (snippet #3352's trap). The extension and registry negative assertions were already in place from phase 1. The final sweep found one real residue step 4 missed: extension/README.md still advertised pixiv support and carried a "Pixiv OAuth" manual-test item. Also replaces the two deleted dispatch tests with one over the whole NATIVE_INGESTER_PLATFORMS set, so adding a platform and forgetting its ingester class now fails at unit level rather than as a mid-download KeyError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 20:46:15 -04:00
bvandeusenandClaude Opus 5	5aa8e3d81b	fix: a stopped source is not a failing one, and cannot be deep-scanned (4279) ... Ebi77 sat in the "1 source is failing" banner for six days with no action available, reading `stranded by recovery sweep (no terminal status after time_limit)`. Four things lined up: 1. The membership sweep did its job — saw `former_patron`, disabled the source, cleared its failure state. Clean at 02:50. 2. Twenty minutes later a deep scan was armed on it. `/backfill` had a credential pre-flight but NO `enabled` guard, while `/check` has carried one all along. The two trigger endpoints disagreed, and the ungated one is the one that arms the long walk. 3. Without a membership the walk cannot finish, never reaches a terminal status, and the recovery sweep strands it with consecutive_failures = 1. 4. Nothing could clear that. A disabled source is never scheduled, so no successful run resets the count; `SourceService.update` clears only on an explicit disable and it was already disabled; and the banner's Retry routes to `/check`, which refuses a disabled source. The card offered a button structurally incapable of acting on the only source it was showing. `failing_sources_clause()` now means "enabled AND erroring". That also settles a disagreement its two callers already had: the scheduler's count paired it with `enabled.is_(True)` and `SourceService.list(failing=True)` did not, so one counted Ebi77 and the other did not — exactly the drift the note above that function warns about, which is why the test belongs IN the predicate rather than beside it. The scheduler's now-duplicate clause is dropped so one place decides. `/backfill` gains the guard for start/recover/recapture. `stop` stays open on a disabled source, or arming becomes a one-way door. Migration 0101 clears failure state on sources that are already disabled — the predicate fixes what the surfaces report, not what the rows carry, and the rows are why the operator had no way out (lesson #4202). It matches what `update` already does on an explicit disable, so rows disabled by any other path come into line. Enabled sources are untouched: a real failure on a live source must keep showing, which the second new test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 19:25:42 -04:00
bvandeusenandClaude Opus 5	11a01a9686	revert: remove the placement reconciler — it manufactured the problem it solved ... Milestone #421 built a sweep that compared each image's `artist_id` to the name of the directory holding its file, and called every mismatch a misplaced image. It reported 33,789 of 63,605 as wrongly filed. That number described the comparison, not the library. What it actually was: 32,475 (97.1%) one artist's own folder, spelled differently — Telepurte/ vs telepurte/. Same artist, same art. 657 ( 2.0%) loose at the images root 328 ( 1.0%) in a folder named after a different artist And the 1% did not mean what the tool assumed either. `ImageProvenance` records the post and source every file was downloaded from — the authoritative answer, which the tool never consulted. Querying it for all 328: 144 provenance agrees with the record (move would be right) 87 provenance agrees with the FOLDER (the record is wrong; move wrong) 53 provenance names SEVERAL artists (no single correct folder) 41 no provenance at all 3 agrees with neither So the sweep would have misfiled or arbitrarily picked for ~41% of the only set it was really needed for. The system already knew where each file came from; the tool inferred it from a column and a directory name instead. Operator, 2026-09-21: *"the current system consistently records where items are and where they came from this is just complicating something works and doesn't need fixing."* Correct on both counts. Removed: the service, the tasks, the model and migration 0099's table, the /api/cleanup/layout and /placement/* endpoints, the Maintenance card and its store actions, and the tests. 0100 drops the table (rule #22 — no legacy). KEPT deliberately, per the operator: - `utils.paths.canonical_subdir` — new filesystem imports derive their directory from the artist's slug, matching what the downloader always did. Not part of this tool; removing it would be churn that fixes nothing. - The 327 files run 1 moved (InsoUwu/ -> insouwu/). Same artist either way, and the gallery renders them correctly. - Everything from #4223 (three-gate dedup, 256-bit pHash) and #4234 (backup credential exclusion). Those fixed problems that were actually reported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 18:17:13 -04:00
bvandeusenandClaude Opus 5	2dd9b956d5	feat: File placement card — survey, plan per artist, apply, put back (4246, slice 3c) ... The UI half of the reconciler, in Maintenance. Survey shows how many images sit in the wrong artist's folder and which folders they are in; each artist gets its own Plan button; each run can be reviewed, applied, and put back. Deliberately NOT using useMaintenanceTask. That composable stashes a task id in localStorage so a result survives navigate-away, which is the right answer when the only record is a Celery result. Here the runs are database rows — so a reload, another machine, or coming back tomorrow simply shows the same state, because the state IS the row. The card polls the runs endpoint instead. Copy avoids the vocabulary this work has been tripping over: "folder", "put back", "in the wrong folder" rather than artist_id, revert and canonical. The one thing the operator most needs to know — nothing here changes who an image belongs to — is what the blurb says first. Three things I had assumed and checked instead: MaintenanceTile lives in common/ not settings/; there is no generic ConfirmDialog (BackupCard uses a purpose-built modal), so this uses a plain v-dialog; and `loadArtistNames` did not exist — it does now, mapping id to name so a run row reads "Conto" rather than "#47". The name deliberately is not denormalised into the run: it belongs to the artist and would go stale on a rename. Also extracted `stubFetch` to frontend/test/stubFetch.js. The shape ledger flagged it as a byte-identical duplicate across five specs and this would have been the sixth; the others keep their copies until each is next touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:32:51 -04:00
bvandeusenandClaude Opus 5	a4bdbcaca4	test: import the placement task module so its names reach celery.tasks (4246) ... Three registration assertions failed: a task name only enters `celery.tasks` when its module is imported, and nothing in the test process imported `library_placement`. The API routes import it lazily inside the handlers, and the registration test runs before any handler test triggers that. `include=[...]` is what gets the module imported in a real WORKER, so production registration was never in question — the test was asserting something only observable after an import it never performed. test_tasks_admin already carries the convention verbatim (`import backend.app.tasks.admin # noqa: F401 — register tasks`); I wrote the assertion from what I meant instead of copying the idiom next to it. Same mistake shape as the commit-vs-flush bounce one commit ago: the pattern was already in the suite both times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:21:20 -04:00
bvandeusenandClaude Opus 5	db63377754	test: placement API tests must commit, not flush (4246) ... Five of the new tests failed in CI: every one that created a LibraryPlacementRun and then read it back through the client. The ones that touched no rows passed. A flush stays inside the test's own transaction, and the app under test runs on a separate session and connection — so the endpoint queried a database where the row did not exist yet and got its 404 / empty list honestly. `_seed_runs` in test_api_system_backup already commits for this reason; the idiom was there to copy and I did not look first. Recorded in the helper's docstring rather than just fixed, since the next person writing a create-then-fetch API test will reach for flush too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:15:52 -04:00
bvandeusenandClaude Opus 5	abe449b4f2	feat: placement reconciler tasks + API (4246, slice 3b) ... Three Celery tasks wrapping the 3a service, and the endpoints that drive them. Routed to `maintenance_long` alongside backups: 33k renames on NFS have no business in the quick lane where the self-healing sweeps live (the 2026-06-07 starvation). A durability bug in 3a, found by thinking about what a crash costs rather than by a failing test: `apply_run` wrote its ledger only at the end, so a worker dying at row 30,000 of 33,789 would have taken the undo information for the first 29,999 with it — and that ledger is the ONLY record of where those files came from. It now persists every 200 moves. Two things fell out of writing that: - `_persist` reassigns `run.moves`, so the loop had to snapshot the plan first rather than iterate the attribute it rewrites. - the reassignment is itself load-bearing: SQLAlchemy does not track in-place mutation of a JSONB list, so an `.append()` alone would never reach the database and the ledger would have stayed silently empty. Re-running a partially-applied plan is safe — the moved rows no longer match their `from` and refuse as "row moved since planning" — but `apply_placement` deliberately has NO autoretry: re-entering a half-applied plan should be the operator's call after reading what happened, not the queue's. Endpoints gate on run state as well as the service does, so a stray POST cannot re-apply an applied run. The list response omits `moves` (an applied whole-library run carries tens of thousands of entries); the detail endpoint includes them, because that detail IS the preview read before agreeing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:13:25 -04:00
bvandeusenandClaude Opus 5	9ccc460c69	feat: placement reconciler — plan, apply, revert (4246, slice 3a) ... Milestone #421 step 3, reframed on the operator's steer: not a one-off migration but the system that keeps the tree true. The 33,789 misplaced rows the survey found are just its first run. The placement half was already done, verified by reading each writer rather than assuming: downloads have always written `<root>/<slug>/<platform>/` (gallery_dl.py:523), attach_in_place leaves files where the downloader put them, and `_copy_to_library` / `_supersede` became canonical in #4244. So nothing is written off-canon today; what remains is the backlog and a standing check for future drift. `LibraryPlacementRun` (migration 0099) holds the plan as JSONB, and that one structure does three jobs: it is the PREVIEW the operator reads, the list the APPLY executes (rather than re-deriving the set, so the two cannot disagree), and — because `from` is retained — the UNDO. The undo is the point. It makes a 33,789-file operation something to do one artist at a time, look at in the gallery, and reverse if it reads wrong. That settles whether artist_id or the folder held the truth (spike #4257) by doing rather than by arguing it from a 50-row sample. An applied run is therefore HISTORY, not state — lesson #4226's trap, since it is the only record of where those files used to be. The model and the migration both say so: any future retention here may prune ready/cancelled/ error runs, never an applied one. Everything fails closed. The apply re-checks each row against what the plan recorded — source still there, destination still free, row still pointing where the plan said — because a download or a supersede can land in between. A refusal is recorded with its reason and the run continues; one stale row is not a reason to abandon the other 33,788. The row is updated only after its rename lands, so a failed move can never leave `path` naming a file that is not there. Writing the collision test caught the code disagreeing with its own comment: it claimed the first of two rows wanting one destination and skipped the second, silently picking a winner by iteration order. Now it counts first and filters after, so genuinely neither is planned. Thumbnails are sha-addressed, not path-keyed, so they do not move — pinned by a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 12:46:06 -04:00
bvandeusenandClaude Opus 5	9728407539	docs: record why the misplaced-rows LIKE needs no escaping (4245) ... `startswith` compiles to LIKE, where `_` and `%` are wildcards, and the call does not escape them. That is safe only because `slugify` reduces a slug to [a-z0-9-] — an invariant living in a different module, which is exactly the kind of thing that gets widened later without anyone connecting the two. Worth naming because `poch4n_art` is a real directory here: if slugs ever carried underscores, that prefix would start matching `poch4nXart` and the sweep would quietly mis-file one artist's rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 11:05:37 -04:00
bvandeusenandClaude Opus 5	fc982f74b9	feat: survey which image rows sit outside their artist's canonical directory (4245) ... Step 2 of milestone #421. The disk survey counted FOLDERS; this counts ROWS, which is the number that matters — every move in step 3 is a row update, and `ImageRecord.path` is the only pointer at the bytes. `library_layout.py` holds the decision in two shared pieces, and both halves of the consolidation spread them rather than restating them (rule 93, the _x_conditions shape from snippet #3087): - `_misplaced_conditions(root, artist_id, slug)` — rows of one artist whose file is not under that artist's directory. The prefix carries a trailing separator deliberately: without it `ara` matches everything under `arbuzbudesh/`, and one artist reads as fully placed while another's rows are silently skipped. Both are real artists here, hence the test. - `destination_for(path, root, slug)` — where a row's file belongs, or None when it must not be moved: outside the images root, or under one of the reserved stores (`thumbs`, `attachments`, `cookies`, `secrets`, `_backups`, `_quarantine`). Relocating those would move the thumbnail cache or the credential key into an artist folder. `destination_for` diverges from `canonical_subdir` in exactly one case, and the docstring says why: a file at the images ROOT with a known artist moves under that artist here, where the import-time helper leaves it alone. The two answer different questions — an empty subdir at import means no artist was resolved, while a row that already carries an artist_id is an anomaly with a known correct home. The 660 unattributed files have no artist_id at all, so no predicate reaches them; they are counted and left for task #4247. `GET /api/cleanup/layout` exposes it. `?check_disk=1` additionally stats every destination for collisions and missing sources — the conditions the apply refuses on — but it is off by default so the count-only pass answers "how big is this" in seconds instead of timing the request out on NFS. Nothing here writes; a test asserts that against both the row and the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 11:03:03 -04:00
bvandeusenandClaude Opus 5	01c906e6ad	fix: ruff I001 — canonical_subdir sorts before derive_subdir ... Rule 102's import-order trap, hit by inserting into the block rather than re-sorting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:49:38 -04:00
bvandeusenandClaude Opus 5	30337a6c11	fix: library paths follow the artist's slug, not the import folder's name (4244) ... Step 1 of milestone #421. The images tree has 57 directory families for what the database says are single artists — `Conto`/`conto`, `InCaseArt`/`incaseart`, `StickySpoodge`/`Stickyspoodge`/`stickyspoodge`, and so on down to a four-way split for Pocket Ace Games. There was never a duplicate Artist row. `/api/artists/names` returns exactly one per artist. The files simply get written to two places for one row: `_copy_to_library` built its destination from `derive_subdir`, which mirrors the IMPORT tree's folder name verbatim, while the download path leaves files where the ingester wrote them — under the slug. Two writers, two conventions, one artist. This is the half that stops it re-growing, and it has to land before anything moves existing files: consolidate first and the next filesystem import out of a capitalised folder re-creates the directory that was just emptied. `canonical_subdir` replaces the top-level segment with the artist's slug and leaves everything below it alone — the post hierarchy is the downloader's business. Two deliberate pass-throughs: no resolved artist (nothing authoritative to canonicalise against) and an empty subdir (a file at the images root, whose fate is task #4247, not a side effect of this helper). `_supersede` resolves the KEPT row's artist for the same reason — a supersede rewrites `existing.path`, so writing it anywhere else would move a row back out of the tree being consolidated. ImageRecord carries `artist_id` with no relationship attribute, so that is a session lookup rather than an attribute. test_import_one_happy_path pinned the old `Alice/` destination and now pins `alice/` (rule 90). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:48:46 -04:00
bvandeusenandClaude Opus 5	6915cbbbe1	fix: the images backup carried the key to the accounts it backs up (4234) ... Listing a 2026-05 tarball while investigating the 4.3T `_backups` pile showed its second and third entries: images/secrets/ images/secrets/credential_key.b64 That is the key that decrypts the stored Patreon/SubscribeStar session credentials, and `cookies/` sat beside it — both unexcluded, so this was true of every images backup taken today, not just the old ones. An images tarball is supposed to be a media archive; one that carries the operator's account keys is a credential leak wearing a backup's name, in a single file that is easy to copy to another disk or restore somewhere less protected. Encryption at rest buys nothing when the key travels in the same archive. `secrets` and `cookies` join `_backups` and `_quarantine` in one named tuple, each with its reason recorded — the recursion that produced 4.3T of nested tarballs is the cautionary tale for why the list is worth explaining rather than just listing. A restore no longer re-establishes credentials. You sign in again, which is the correct outcome for a media backup. Tests cover both new names and that every exclude stays root-relative — a bare `secrets` would also match an artist folder of that name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:35:35 -04:00
bvandeusenandClaude Opus 5	3313c3b10a	feat: a report that shows what the near-dup gates decide about real artwork (4223) ... The three pixel constants added with the #4223 fix were chosen without ever measuring real files — CI only has synthetic split/solid fixtures, and FC verifies nowhere else. This prints the measurements they should have been chosen from: per pair, the hash distance, the mean drift, the changed-pixel fraction, the verdict, and which gate produced it. It drives the real find_similar with the real confirm rather than restating the decision, so it cannot drift from what the importer does. Read-only: opens files, touches no database. Also splits fingerprint_diff out of fingerprints_match — same computation, now returning the numbers instead of only the boolean, so the report can show how far a pair sat from a limit rather than which side of it it fell on. Runs inside the published :dev image (PIL + imagehash already there, no local env needed) with the art folder mounted read-only — rule 147's channel, so nothing has to reach main to be tried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 06:30:54 -04:00
bvandeusenandClaude Opus 5	84f873eb1b	fix: ruff UP037 — unquote the fingerprint_path return annotation ... PIL's Image is imported at module scope, so the annotation never needed to be a string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:17:15 -04:00
bvandeusenandClaude Opus 5	a05cb66635	fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223) ... The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:16:14 -04:00
bvandeusenandClaude Opus 5	3fe9d0a612	fix: the Latest feed's filter dropdowns opened empty ... Operator: "the filters in the latest feed, drop down but don't have values". Two separate causes: - Platform: PostsFilterBar built its items from `platformsStore.platforms`. The platforms store has never had that property; it exposes `list` and `byKey`. The read returned undefined, `|| []` turned that into an empty list, and nothing failed. ArtistsView had copied the same read, so the Browse → Artists platform filter was empty too. Both now read `list` and show platform names rather than raw keys. - Artist: the autocomplete searched the server only after something was typed (autocomplete returns [] for an empty query by design, which its tests pin). Opening the dropdown therefore showed an empty menu. PostsFilterBar now loads every artist once from a new lightweight `GET /api/artists/names` (id, name, slug; alphabetical; no joins) and filters client-side, so the list is there on open. A deep-linked artist_id now also shows the artist's real name instead of "Artist #id". Guard: frontend/test/storeUsage.spec.js scans src for `platformsStore.<name>` and fails on any name the store doesn't define, since the frontend CI has no type-checker to catch this. A positive control shows the shipped `platformsStore.platforms` read is flagged, and a vacuity check confirms the scan really walks the tree. tests/test_api_artists_create.py covers /names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 22:38:27 -04:00
bvandeusenandClaude Opus 5	7ff8915147	feat: a source stops pulling once its membership ends, and resumes on resubscribe (3995) ... Operator, 2026-09-13: "if I kill a subscription on patreon I would like the pulling to stop on curator as well", with auto-resume chosen. This reverses the 2026-09-11 "report only" decision for lapsed sources. membership_reconcile.apply_membership_lapses runs in sync_memberships right after each platform's successful sync, so it only ever acts on the roster just written. It stops a source (enabled=false, with the same failure-state reset as a manual disable, #1285) only when all of these hold: - the roster is fresh - the source's matched membership says has_paid_access is False (lapsed, or a free follow) - the paid-through date has passed, where the platform gives one (Patreon's member.access_expires_at; SubscribeStar gives none, so it stops at once) - the source is enabled - the operator hasn't chosen to keep it It never acts on absence. A source with no matched membership keeps pulling, because a rename or a never-walked source produces the same absence. An unrecognised status is never a lapse either. It resumes only sources carrying its own `_membership_stopped` marker, once the membership is paid again. The operator outranks the sweep both ways (SourceService.update): - turning a stopped source back on marks it `_membership_kept`, so the next sweep leaves it alone until it's paid again - turning a source off by hand drops the marker, so the sweep never switches it back on Both are `_`-prefixed app-managed config keys, which operator edits already preserve. No migration. The roster/fetch line holds. This is a source-level action by the sweep. No download path reads the roster, and the scheduler still selects on `enabled` alone. test_no_fetch_path_can_read_the_roster is unchanged. UI: SourceRow shows a neutral "Membership ended" chip, with the status and the resume/keep explanation, ahead of the other chips. The sweep's task summary reports stopped/resumed counts. Tests (tests/test_membership_lapses.py): - a lapse stops the source with a clean slate and keeps the id cache - paid-through is honoured - absence, an unknown status and a stale roster never stop anything - a resume touches only what the sweep stopped - a manual on sticks, a manual off drops the marker, and a kept source is released once paid Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 22:08:06 -04:00
bvandeusenandClaude Opus 5	4d84ab2816	fix: the filter rail goes, and the subscriptions card stops listing what you don't pay for — and can be dismissed ... Operator live review of #253 on the wide window. The images landed well, but: - "the left rail feels like wasted space". The filters and check status move back into one row above the feed (PostsView). On a wide window the row lines up with the feed column, clear of the day gutter, and the container recentres without the 280px rail. The narrow layout is as before, with the status still first. This retires #407 option E; A and D stay. - The "you can't see" link landed on the Subscriptions reconcile card, which listed 9 sources as `former_patron` under "sources your roster doesn't account for". The operator doesn't want sources they no longer pay for listed there, and wants the card dismissable "unless something changes": - MembershipReconcileCard renders only "creators you subscribe to but don't follow here" and the stale-roster warning. - Stopping pulls on lapsed sources is the membership sweep's job instead (#3995, next). - A close button dismisses the card. The dismissal is keyed to a fingerprint of what the card says (offered membership ids plus stale-roster flags), so a new subscription or a roster going stale brings it back. It's stored per browser, which is enough for a single-operator instance. The backend still computes tracked_not_subscribed. #3995 acts on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 22:04:02 -04:00
bvandeusenandClaude Opus 5	9b82a95b7e	feat: Settings can re-download the Discord images the None naming broke (3999) ... The operator chose a clean re-download over relinking in place. The ~1,600 broken files can't be relinked reliably: their message ids are gone, and their sidecars collided. Settings → Maintenance → "Repair Discord downloads" previews, then applies: - Deletes every image whose path is `…/discord/None/<8 digits>_None_…`. Both the folder and the name are required, because that pair is only what the bug produced. It reuses cleanup_service.delete_images for the record and file deletes. - Sweeps the leftover collided sidecars from those folders and removes the emptied folders. - Only then clears gallery-dl's archive rows `discord%`, excluding `discordasset_%`. Upstream keys message attachments as `discord{message_id}_{num}`. Since the broken files lost their message ids, per-source forgetting is impossible. Every pre-fix Discord download is broken, and files fetched after the fix still exist on disk, so gallery-dl's `skip` won't re-fetch them. - Arms a fresh backfill on every Discord source. The apply defaults to preview at both the route and the task, runs on maintenance_long, and is never on a beat. The card uses the confirm-dialog pattern of AttachmentReclaimCard. Supporting refactors, with no behaviour change: - gallery_dl.archive_path() is the single definition of the archive location. - source_service.arm_backfill() is the mutation start_backfill already did, now shared with the sync repair. Tests (tests/test_discord_repair.py): - The archive clear leaves other platforms and Discord assets alone, and counting mutates nothing. - Case-twin artist folders are both found. - The folder sweep works. - An integration run shows only the broken image goes. A correctly named Discord file and a `None` folder under Patreon survive, and only Discord sources are re-armed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 20:52:32 -04:00
bvandeusenandClaude Opus 5	dc840feec7	fix: Discord downloads land as None/<date>_None_<name> with no post — name them from the keys gallery-dl really emits ... Operator report: a Discord source showed the wrong post time. Listing the real downloads found every Discord folder on the instance (7 artist folders, ~1,600 media) in a directory named `None`, as `<date>_None_<original name>`, next to sidecars named `<original name>.json`. The real sidecar settles why: - gallery-dl's discord extractor emits `channel` as a plain string, the message as `message_id`, and the attachment position as `num`. It has no `id` key. - The patterns asked for `{channel[name]}` and `{id}`. gallery-dl renders a missing field as "None" and carries on. - The sidecar was named `{filename}.json`, the attachment's ORIGINAL name. find_sidecar never pairs that with `<date>_None_<name>.png`, so no Discord file ever got a Post or a post date. The card fell back to downloaded_at. - Every `image.png` in a channel also overwrote the same `image.json`. Fix (gallery_dl.py): - The directory is `{channel}`. - The filename is `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, unique per attachment. - A discord-scoped metadata postprocessor names the sidecar exactly like the media minus its extension, so find_sidecar's first candidate matches. A per-source filename override re-derives the sidecar name. save_metadata=False drops it. Guard (tests/test_gallery_dl_naming.py) renders the patterns through Python's formatter against a sanitized copy of the real sidecar (same keys and types, invented values). A missing key or a subscript into a string raises, which is the loud failure gallery-dl doesn't give. A positive control shows both shipped patterns fail it. Existing broken downloads are NOT repaired by this. gallery-dl's archive already records them, so a re-run skips them, and their collided sidecars no longer describe them. That repair is a separate, destructive step for the operator to decide on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 20:01:11 -04:00
bvandeusenandClaude Opus 5	98c3b74260	feat: the Latest feed uses a wide window — filmstrip cards, a day gutter and a filter rail (407 A, D, E) ... On the operator's 3432px window the feed was a 900px column, 26% of the width. They picked three options from a to-scale layout study. A — filmstrip card (PostCard.vue): - A card measures itself with a ResizeObserver. At 1100px or wider its hero gets a fixed height, clamp(260px, 34vh, 460px), and the extra images move into a 2-column grid of squares beside it. The grid cells are sized from the hero height, so the grid ends flush with the hero. - The rail cap is 4 cells in this layout (2×2, the last becoming "+N") and 5 in the narrow layout, which is unchanged. - The description clamp drops to 4 lines, because long reads happen in the expanded view. - The hero has a height, not a width, so a wide card can't grow into a full-screen post. That was the operator's constraint. D — day gutter (PostsView.vue): - The normal feed groups consecutive posts by local day (Today, Yesterday, a weekday, or a date), with post and artist counts for what has loaded. - Runs rather than date buckets, because the sort key includes resurfaced_at, which the payload doesn't carry. A resurfaced grouping gets its own heading where it actually appears, instead of being pulled out of order. E — filter rail (PostsView.vue): - At 1600px and wider, the filters and status ribbon stack in a sticky 280px left rail, and each day's heading sits in a sticky 150px gutter beside its posts. - Below 1600px the layout is exactly the old one, including the 900px column. The in-context (post_id) view gets the wide column but no rail or day grouping, so anchor scrolling is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 19:48:46 -04:00
bvandeusenandClaude Opus 5	4b4e532c56	fix: the membership status vocabulary moves beside Membership, so the Patreon client never imports the roster ... 240f11c made PatreonClient._membership import has_paid_access from membership_roster. test_gated_reason::test_no_fetch_path_can_read_the_roster failed on it, correctly: native_ingest_common is a fetch root, patreon_client is reachable from it, and no fetch path may be able to reach the roster. The roster is allowed to explain a skip, never to cause one. MEMBERSHIP_STATUS and has_paid_access are pure platform knowledge with no database behind them. They move to native_ingest_common, next to the Membership type they interpret (the same move C7 made for Membership itself). membership_roster, membership_reconcile, patreon_client and the tests import them from there. There is no re-export from membership_roster. The guard is unchanged. The lapsed-orphan skip from 240f11c stays as it was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 16:07:26 -04:00
bvandeusenandClaude Opus 5	240f11c5aa	fix: one lapsed membership to a deleted creator no longer fails the Patreon roster ... The first live sweep failed with "Patreon member resource has no campaign relationship". The operator ran FC's exact /api/members request in a browser and shared the response. - It has 104 rows. The browser's settings page shows 43, because FC sends no membership-type filter and so also gets lapses back to 2016. - Exactly one row has no `campaign` key at all: a former_patron whose membership ended in 2017. Its included reward has no campaign link either, so the creator's page is gone. - Every other row, including the 4 is_active=false campaigns, has a campaign. _membership returns None for a campaign-less row only when has_paid_access says it is definitely not paying (a known lapsed status, or a free member), and iter_memberships skips it. That changes no conclusion. A lapsed membership and an absent one both mean "not paying", and no Source can match a campaign with no id. An active or unrecognised membership without a campaign still raises, because dropping a membership that might be paid would read downstream as a cancellation. Paging still counts the rows the server sent, not the rows kept. A test pins that, so a skip can't re-read an offset or stop a page short. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 16:01:50 -04:00
bvandeusenandClaude Opus 5	57c880a623	feat: the full brand mark as a faint backdrop behind every page ... The operator asked for one large version of the logo as the site background. It is pinned to the viewport behind .fc-content at about 6% strength, so it shows in the gutters and on bare page ground while cards and the nav cover it. The series reader is immersive, skips the shell, and never draws over it. It is a layered background (the page colour at 94% over logo.svg), not an overlay element with opacity. An overlay needs the content z-indexed above it, which turns every page into one stacking context under the nav's z-index 1000 and can trap an in-page overlay beneath the nav. A background changes nothing about stacking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 15:58:40 -04:00
bvandeusenandClaude Opus 5	dd766eb976	fix: the favicon and nav brand mark rendered blank — "--" inside an XML comment ... 7ca6ee0's favicon.svg had a comment naming the `--fc-chrome-rgb` custom property. `--` is illegal inside an XML comment, so the file is not well-formed XML, and a browser renders an SVG-as-image only if it parses. It fails without any error, just a blank image. Because TopNav's brand glyph is the same file, the tab icon and the nav mark both went missing on the #251 deploy. logo.svg parses and was unaffected. The comment now names the property without the hyphens, and says why it has to. tests/test_public_svgs.py parses every SVG under frontend/public. It includes a vacuity guard (the directory really contains the two known files) and a positive control (the exact defect raises ParseError), so the check can actually fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 15:57:19 -04:00
bvandeusenandClaude Opus 5	e3c516d6be	fix: a retired platform leaves the chip maps, and a fixture stops creating pixiv sources (406) ... e3fd8c6 failed two lanes, both on things pixiv's retirement correctly started refusing. Unit - test_fe_be_contract pins platformColor.js's ICONS keys to known_platform_keys(). I had kept pixiv's icon, colour and label "so existing pixiv posts don't look broken". That was wrong, and the file's own header already said why: unknown platforms fall back to a grey chip with the raw key, and that fallback is deliberately what a retired platform hits - it is how a pre-3069 deviantart row already renders. So pixiv leaves all three maps, and its posts show as a grey "pixiv" chip rather than a broken one. The header now says outright that a retired platform is removed, never kept, since the milestone plan itself got this backwards. Integration - four reassign tests built their fixture source through SourceService.create with platform="pixiv", which the validator now rejects. Reassign never reads the platform and never moves files, so any registered platform serves; the fixture uses hentaifoundry. 1294 other integration tests passed on e3fd8c6, so nothing else used pixiv through a validating path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 11:52:06 -04:00
bvandeusenandClaude Opus 5	c2f9e9cc08	docs: stop claiming pixiv, and stop claiming everything gallery-dl supports (406 step 4) ... Ships with the switch-off rather than with the code removal: a doc that promises a platform the product refuses is the Install and Public Surface area's characteristic defect. pixiv comes out of README (twice), SECURITY.md (twice), .env.example and the compose header. The stored-credential warnings now name Patreon and SubscribeStar - the accounts that usually carry a payment method. One correction beyond pixiv. README said FabledCurator follows creators on Patreon, SubscribeStar, Pixiv "and anything gallery-dl supports". That was already false: a platform not in the registry is rejected, however capable gallery-dl is. It now names the real set, which rule 171 records: Patreon, SubscribeStar, Discord and HentaiFoundry. The 3422 docs guards still hold - the key path and bootstrap variable are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9 ext-2026.9.13.1545	2026-09-13 11:45:49 -04:00
bvandeusenandClaude Opus 5	e3fd8c67d4	feat: switch pixiv off — unregistered, unreachable, and refused at dispatch (406 phase 1) ... Milestone 406 retires pixiv (rule 171) in two phases at the operator's explicit ask: switch it off, then later delete its code. This is the switch-off. Steps 2 and 3 ship together because each is a half-state of the other: unregistered but still in the extension, pixiv creator pages would offer a button the backend then refuses. Reachability removed, never gated (rule 22 - no flag, no `if platform == "pixiv"`): - platforms registry: pixiv unregistered, so /api/platforms, the source validator and quick-add all refuse it through their existing unknown-platform paths. - NATIVE_INGESTER_PLATFORMS: pixiv removed. - extension_service: pixiv's quick-add URL pattern removed (the Python half of the JS mirror). - extension: pixiv's host permissions, content-script match, platform entry and artist pattern removed; popup's pixiv branches removed; and the whole pixiv PKCE OAuth flow cut out of background.js. That last one could not wait for phase 2 - a webRequest listener on a host the manifest no longer grants is at best dead and at worst a startup failure for the entire background script. On startup the extension now also removes any pixiv refresh token a browser still holds in storage, for the same reason as the server-side credential cleanup (3980). - frontend: the extension card stops listing pixiv; SourceActions' copy of the native list drops it. platformColor keeps rendering a pixiv key so existing pixiv posts do not look broken. The guard, and why a registry change alone was not enough. A source outlives its platform: the live instance still had one ENABLED pixiv source (step 1). Tracing it: the scheduler only selects enabled rows and every platform lookup uses .get(), so a disabled row is inert - but re-enabling it and pressing Check would have routed pixiv, no longer native, straight into the gallery-dl branch, which still has a pixiv extractor. And a worker can pick up a still-enabled row before a deploy's migration runs. So run_download and verify_source_credential - the two functions every download and credential probe pass through - now refuse any platform not in the registry: an unsupported_url failure for downloads, and an inconclusive (None, not False) verify, since nothing was probed so nothing was rejected. Generic by registration, so it covers deviantart's leftovers too. Positive-controlled: a supported gallery-dl platform must still reach gallery-dl, or a guard that refused everything would pass (rule 167). Migration 0097 disables sources on retired platforms (pixiv, deviantart) and clears their failure state exactly as disabling through the app does (1285), so the stale row stops being scheduled and stops showing as failing. Nothing is deleted: removing a source can collide with uq_post_artist_external_id_null_source on real data, which is phase 2's step 6 to check. No post or image is touched. Tests: the known-platform lists drop pixiv and gain retirement assertions beside deviantart's; pixiv's positive extension cases become negative guards; the pixiv sidecar post-URL test is deleted with the behaviour it tested; quick-add rejects a pixiv URL. The pixiv client/downloader/ingester suites stay - that code stays until phase 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 11:45:49 -04:00
bvandeusenandClaude Opus 5	0835da8a91	fix: give the roster's column zip an explicit strict=False (387 D1, B905) ... ef91fcf failed ruff's B905 lane on one zip(labels, cells) without strict=. Tests, integration and the frontend were already green on that SHA. strict=False is the deliberate side, not the quiet one. strict=True raises a bare ValueError - not SubscribeStarDriftError - and would fail the whole roster sync over a column mismatch in `details`, which nothing reads yet. That would take down reconciliation and the gated-post reasons over a cosmetic markup change, while creator identity (id, slug) never depended on the columns at all. But a shifted column would mislabel details silently (a price filed under "discord"), so a count mismatch now logs a canary warning, mirroring the feed parser's existing parse canary: diagnosable from the worker log, never fatal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 10:56:18 -04:00
bvandeusenandClaude Opus 5	ef91fcfd26	feat: SubscribeStar joins the membership roster (387 D1) ... The second platform through the seam note 3970 contracted, characterized first from a live capture of the account's /subscriptions page (note 3989). The capture lives in the gitignored captures dir; the committed fixture is hand-built with invented values and was verified tag-for-tag against it - card wrappers, both table heads, and every distinct row shape - before any code depended on it. What the page is, and the three decisions it forced: The table IS the status. SubscribeStar has no per-row status word: a creator is either in the active_subscriptions card or the cancelled_subscriptions one. The card's data-identifier is stored verbatim as Membership.status and mapped in MEMBERSHIP_STATUS, keyed on the identifier rather than the table class because the cancelled table's class names the same list differently (for-unsubscribed_users). The creator's numeric data-user-id is the key, not the slug. A slug re-keys when a creator renames; the old row stops appearing; and a disappearance is exactly what reconciliation reads as a lapse. Keyed on the slug, a rename would have told a paying subscriber they had cancelled. The slug rides as vanity, where the identity join already looks for a handle. Price is kept as text, never parsed into amount_cents. A bare $ names no currency and a page price is not proven to be the charge - 3970 finding 4. Tier names live behind a per-row modal and are not fetched. Refusals, because SubscribeStar offers nothing like Patreon's meta.pagination.total and every conclusion downstream is drawn from absence. The parser raises when: the active card is missing (auth error on a login/age wall, drift otherwise); a row lacks a numeric creator id or a creator link; anything renders after a card's table; or the page carries a page= link. Both cards are paginatable (app#embed_pagination) and the captured account was too small to show what pagination looks like, so possible pagination is a roster FC cannot prove complete. A loud error on a larger account beats a quiet half-list. A missing cancelled card is not drift, and a creator in both tables is reported once, as active. Fetched from subscribestar.adult, not the .art the capture came from: FC's requests never clear the .art age wall with the 18+ cookie (1259, 1284). Whether /subscriptions on .adult authenticates exactly as .art did in the browser is untested - if not, the sweep records a visible error and C6 shows its unavailable rung. The seam leak D1 found. Note 3970 promised a second platform would be one builders line plus the client method. The sweep instead called current_user_id() on every client, which only Patreon's has, so SubscribeStar would have raised AttributeError on the first sweep. roster_user_id probes it with getattr, the same way the sweep already probes iter_memberships. Two existing tests were passing for the wrong reason and now can fail: - "a platform that has never been characterised says nothing" named SubscribeStar, and stayed green only because active_patron is not a SubscribeStar word. Now uses hentaifoundry, with a positive SubscribeStar test beside it. - the freshness test gave SubscribeStar a Patreon word, so the vocabulary excluded it and deleting the freshness gate outright would have left it green. It now uses cancelled_subscriptions, making the gate the only thing that excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 10:53:01 -04:00
bvandeusenandClaude Opus 5	529d4bff57	test: tie the install docs to the code they quote (3422 follow-up) ... 3422 was already fixed. Commit 86abaf0 applied option 1 in full and it is on main: .env.example carries the bootstrap section with the backup warning, README explains the refusal and why it is deliberate, docker-compose forwards the variable, and the milestone-362 smoke gate that FOUND the bug now sets it (build.yml:1159) and passes. The issue's premise - "CURATOR_BOOTSTRAP_NEW_KEY appears nowhere outside backend/" - is stale. What was left is the dependency that fix created. README.md and .env.example now both print the literal error text, the literal key path and the variable name, because a stranger greps for the string their terminal showed them. That is the right call and it means two user-facing files now depend on this module's wording with nothing connecting them - the install surface's characteristic defect, one rename away from a README that sends strangers to a path that does not exist. Three guards, all presence checks on both sides. An absence check against prose would pass for the wrong reason the moment a sentence were reworded (snippet 3352): - the raised message still contains the sentence README reproduces, the variable both docs say to set, and the restore-rather-than-mint alternative the whole refusal rests on; - both docs still name _CREDENTIAL_KEY_PATH and the variable, read from the code rather than retyped, so a rename fails here; - compose still forwards the variable - without that line the docs' "set it in .env" is silently inert and fails identically to not setting it. Option 2 (mint when the credential table is empty) is deliberately NOT done. The issue's own guidance is "(1) now, (2) if the friction proves annoying", and the friction has not been reported. Worth recording that its predicate checks out exactly: the Fernet key protects Credential.encrypted_blob and nothing else - no other Fernet user exists - so "no credential rows means nothing can be made undecryptable" is provable rather than probable. The cost is placement: create_app() is sync and constructs the key before any engine exists, so the check cannot live where the failure is. entrypoint.sh, which already runs alembic against the DB, is the natural seam. Only the web role is affected; the Celery roles build the key lazily inside tasks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-12 21:31:21 -04:00
bvandeusenandClaude Opus 5	eb6e0df858	feat: the empty front door offers to find what you already subscribe to (387 C6) ... B4's on-ramp read "add a credential, add a source" - the manual path, and the one that makes a new installer retype creators they have already told Patreon about. With C4 in place the app can just look them up. This is the step where the milestone's original framing actually lands on screen. Four rungs, chosen by ONE predicate rather than four independent v-ifs. The rungs are mutually exclusive by construction and the rung shown matches what is actually POSSIBLE: no credential -> Add a credential. Discovery is not offered, because a button that cannot work is worse than its absence. credential, never synced -> Find what you already subscribe to. Queues the C3 sweep. synced, unmatched > 0 -> the count, linking into C4's bucket 1. synced, nothing unmatched-> the manual add, because there is genuinely nothing to discover. The fifth state is rule 164's. A sweep that has been ATTEMPTED and never succeeded reads as "couldn't reach patreon", with the error type and the manual path still open - not a spinner, not a crash, not a retry button that will fail identically. An install with no outbound network lands here. That is deliberately narrower than "there is an error": a roster that synced once and failed since is NOT unavailable. It has a roster, just an ageing one, and C4's freshness gate already handles that. Collapsing the two would hide a usable roster behind an error banner. Still offer, never auto-add. An empty front door is exactly where "just add all thirty" is most tempting and most wrong - thirty backfills on first boot - so the rung carries the count and sends them to the picker. No new machinery: three existing stores (credentials, membershipSync, membershipReconcile). Reconcile is fetched only once something has synced, since bucket 1's count is meaningless before that. Every load swallows its failure, because this screen renders on an install that can reach nothing. Ten tests on top of B4's, including the mutual-exclusion property asserted directly. Its phrases are each pinned to a single template line and both sides whitespace-normalised - a phrase spanning a line break would never match, and a mutual-exclusion check whose phrases never match passes vacuously (rule 167). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-12 20:19:42 -04:00
bvandeusenandClaude Opus 5	4533e036ac	refactor: the membership seam's contract type is the seam's, not Patreon's (387 C7) ... Membership moves from patreon_client to native_ingest_common, beside PostRecordOutcome, for exactly the reason that one lives there: it is the seam's contract rather than the first platform's. Left where it was, D1 would have had to import the shape it implements from the module of the platform it is being mirrored FROM - which inverts the dependency and is how a seam advertised as portable quietly stays Patreon-shaped. Found by C7's own pass, which is the point of running C7 before D1 rather than writing it up afterwards: this is invisible while there is only one implementer and load-bearing the moment there are two. No behaviour change. Three files, no shim (rule 122): patreon_client imports it, the dataclass keeps its docstring, and the test imports from the seam's home. The docstring gains what the contract owes a second platform - that a missing field supplies the empty answer and never a guess: no tiers -> [], no pledge -> None (absent stays distinguishable from zero, since "free" and "we don't know" are different answers), no vanity -> None with identity falling back to the URL tail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-12 20:05:11 -04:00
bvandeusenandClaude Opus 5	6f5ea5d1d3	fix: the provenance panel hotlinked the CDN for images already on disk (3965) ... Two surfaces render a post's HTML body. get_post did _localize_inline_images(sanitize_post_html(...)); provenance_service._post_dict called sanitize_post_html alone. So opening the Provenance panel fetched images from Patreon's CDN for files FC had already downloaded - the archive reaching out to the platform to display what it had archived, which is the thing 830 Phase 2 set out to stop. The same bodies also break when a CDN URL expires or a post is removed, while the identical local copy sits unused. The fix is the shape, not the call. Those were two separately-callable halves and only the first looked mandatory, so a second caller was always going to do half of it. render_post_body in the new services/post_body.py is the whole pipeline in one call, and sanitizing-without-localizing is no longer a reachable operation. _localize_inline_images moves there verbatim; post_feed_service loses five imports that went with it. _post_dict becomes async and takes the session. Both call sites are already inside async methods, so for_image's list comprehension awaits per entry - fine, because localization issues ZERO queries for a body with no inline <img>, which is most of them. Recorded that early exit in the module so the loop isn't "optimized" into a batch without a measurement. Deliberately NOT fixed: provenance still names the same columns url/title/date where the feed says post_url/post_title/post_date, and its description_translated is full text where the feed truncates to DESCRIPTION_LIMIT. That is 3965's wider half - a breaking payload change for ProvenancePanel with no second reason to spend it today. Noted in _post_dict's docstring so the next reader knows it was seen and left. Four regression tests on the provenance path, covering both entry points plus the two refusals the feed already pins: an uncaptured image stays hotlinked (a broken local path is worse than an intact remote one), and a filehash owned by another artist never leaks in. Reverting render_post_body to a bare sanitize fails the first two. Recorded as snippet 3968. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-12 19:57:00 -04:00
bvandeusenandClaude Opus 5	2862fadcb1	fix: the roster guard's import walk resolved package __init__ imports wrongly (387 C5) ... Two of C5's three structural tests errored in CI with ValueError: PosixPath('.') has an empty name. The walk special-cased a package's __init__.py, dropping the __init__ component before computing what `from .` refers to — which made `api/__init__.py`'s `from . import health` resolve to the app root instead of to `api`, and `celery_app.py`'s `from . import celery_signals` resolve to the empty string, which is what actually crashed. The special case was never needed: `parts[:-1]` already gives the CONTAINING package for both forms, because `services/foo.py` drops `foo` to leave `services` and `api/__init__.py` drops `__init__` to leave `api` — exactly what `from .` means inside each. The level slice is now clamped at 0 as well; an import climbing past backend/app left the tree, and the unclamped negative index wrapped and resolved to the wrong module rather than to nothing. A module directly under backend/app doing `from . import x` still yields no package prefix, and there the alias alone IS the dotted name - handled explicitly rather than by falling through into a Path built from an empty string. The two positive controls earned their place immediately: they are what failed. Without them the walk would have resolved almost nothing and test_no_fetch_path_can_read_the_roster would have passed on a broken walker, reading as coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-11 23:02:18 -04:00
bvandeusenandClaude Opus 5	aa765f0a72	feat: say why the posts are invisible, without ever deciding they are (387 C5) ... A3 made a tier-gated source say "47 posts you can't see". The roster turns that into a reason: the membership ended, or the tier doesn't reach these posts, or it's a free follow. Rendered under A3's count in the health tooltip, quieter than the count it explains. FREE is a fourth case the step didn't enumerate, and it earns its own sentence. has_paid_access collapses "former patron" and "current free follower" to the same False, so deriving the reason from that boolean would tell a free follower "you're not a patron any more" - a false statement about a state they were never in. gated_reason reads the status axis first, calling has_paid_access with is_free_member forced off, then splits on the free flag. Silence is the default, and there are four ways into it: campaign absent from the roster, roster stale, platform never swept, status word not yet characterised. All four send null and the count stands alone. The frontend has no fallback sentence either - a default would turn "we don't know why" into a reason, which is the one thing this step must not do. The line that must not be crossed is pinned structurally rather than by inspection: test_no_fetch_path_can_read_the_roster walks the transitive first-party imports from the fetch roots and asserts the roster is unreachable. FC runs no local verification (rule 85), so a guard cannot be falsified by hand before it lands - it carries two positive controls instead, proving the walker finds roster imports that ARE there, one direct and one through a hop, so the real assertion can never pass merely because the walk resolved nothing. C4's identity loop moved to membership_roster.pair_sources_with_memberships when C5 became its second caller; two copies would let the Subscriptions row and the reconciliation card disagree about which creator a source IS. Three test files were each building PlatformMembership rows with their own drifting helper - consolidated into tests/roster_builders.py, same family as issue 3109. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-11 22:59:29 -04:00