Compare commits

...
Author SHA1 Message Date
bvandeusen fb2c4d5b80 fix(extension): packaging.sh decides the version, so it must move the version
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 18s
extension / lint (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 30s
Build images / sign-extension (push) Successful in 1m41s
Build images / build-web (push) Successful in 11s
CI / integration (push) Successful in 3m42s
extension / lint (pull_request) Successful in 21s
`packaging.sh pathspec` excluded `extension/scripts/**` — the same list
web-ext ignores. But the two lists answer different questions, and this is
the one place they disagree.

packaging.sh is not packaged into the XPI. It does decide the version
string, and build.yml stamps that string into the manifest.json that IS
packaged. Changing how the version is computed therefore changes the shipped
bytes, and the derivation has to see it.

Harmless while every push rebuilt the web image. Step 4 made the rebuild
conditional on the derived revision moving, which turns it into a silent
failure: a packaging.sh change yields a NEW version, so sign-extension misses
its ext-<version> cache and signs — while build-web sees an unmoved revision,
reuses the published image and ships the OLD XPI. One orphaned AMO signature,
and an instance serving code the registry calls current. Found while checking
the ground under step 5, which changes the version format and is exactly the
commit that would have hit it.

Split the list rather than widening the shared one: NOT_VERSION_RELEVANT
drives the pathspec, NOT_PACKAGED_TRACKED still drives web-ext's ignore
list, and scripts/ stays out of the XPI. The two directions are not
symmetric, which is why the version list is the narrower one — too wide
costs a re-sign and a rebuild for a change that ships nothing new, too
narrow serves stale bytes and says nothing.

No version churn: the last packaging.sh commit predates the current
extension revision, so the derived version is unchanged at 1.0.3500147 and
web's revision stays a7e626a67a.

Both suites now pin the disagreement from their own side, because the
tempting fix for either half is to make the lists one again — version.spec.js
asserts the pathspec does NOT exclude scripts while the ignore list still
does, and test_artifact_paths.py asserts packaging.sh is inside the extension
and web path sets.
2026-08-28 08:49:06 -04:00
bvandeusen 609bc82acc ci: reuse the published image instead of rebuilding it (milestone 313 step 4)
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 17s
extension / lint (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m43s
Before building, each job asks the registry whether this artifact's content
is already published. On a hit it skips the build entirely and repoints the
channel and date tags at the existing manifest with `imagetools create` —
registry-side, no layer transfer, seconds. This is the step that stops a
push touching only `agent/` from rebuilding web and ml, and stops a merge to
main rebuilding what dev already built.

The question is asked with a new `artifacts.sh identity`, not with the date
tag: the date tag is day-precise and last-one-wins, so two different builds
share it and it cannot answer "is this content published?". The commit sha
would move on every push and never hit, which is the redundant rebuild being
removed. The revision does both jobs — content-unique, and stable across
pushes that did not touch the artifact.

Identity is channel-qualified for web and only for web, because web is the
only image that takes a build-arg: FC_CHANNEL is baked in and reported by
/api/extension/manifest, so its dev and main builds of one revision are
genuinely different images. ml and agent take none, which is what lets a
merge reuse dev's build rather than rebuilding the agent's CUDA image to
produce bytes that already exist. tests/test_artifact_identity.py reads the
Dockerfiles and fails if that list drifts from the ARG declarations, in
either direction — collapsing the channels ships an instance that reports
the wrong one, and splitting them needlessly rebuilds every merge.

Failure direction is deliberate: an inspect that errors for any reason reads
as a miss and the build runs. Only a real 200 skips one.

A tag-push never claims the identity. It rebuilds a revision main already
published, and image configs are not bit-reproducible, so re-pushing
r-<rev> would point an immutable tag at fresh bytes — rule 145's exact
prohibition. It publishes only its own :v... label and otherwise reuses.

Base-image freshness, decided rather than left implicit: an artifact whose
source stops moving stops picking up base updates under its pinned tag. That
is what a pin means, and rule 145 already says the refresh belongs on the
moving tag instead. Filed as #3154 rather than folded in here, because the
naive version regresses :latest on the next unrelated push.

ci.yml's backend lane gains fetch-depth: 0 — the new tests derive real
revisions, and on a depth-1 clone that derivation returns the tip sha or
fails, so the lane would go green while asserting nothing.

The three build jobs' shadow steps are renamed and re-commented: those
values stopped being informational at step 3, and a step captioned "nothing
reads this" beside steps that do is worse than no caption.
2026-08-28 08:24:58 -04:00
bvandeusen 7a20c55441 ci: publish a per-artifact date tag on main builds (milestone 313 step 3)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 6s
Build images / build-web (push) Successful in 5s
extension / lint (push) Successful in 17s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m44s
Each main build now also publishes <image>:YYYY.M.D — the date of the
commit that artifact's shipped files last changed in. Purely additive;
:main, :latest, :c-<sha> and the dev path are untouched, so reverting this
commit reverts the behaviour.

    main push -> :main, :latest, :c-<sha>, :2026.8.27
    dev push  -> :dev

Per artifact, so an image whose files did not change keeps the tag it
already had. On this commit the agent reads 2026.7.17 while web and ml
read 2026.8.27 — six weeks apart, from one push. Step 4 turns that into
not rebuilding it.

Day precision, and a second main build the same day replaces the first.
Operator's call, and the reasoning is theirs: same-day work is not
something worth pinning. A rollback goes to a day, not to the fourth merge
of a Tuesday afternoon. It also makes retention mean "the last N days"
rather than "the last N pushes".

CALVER is computed inside the main branch rather than at the top of the
step, and hard-fails when empty. There is no `set -e` here, so an
unconditional assignment that failed would have left it empty and published
the tag `fabledcurator:` — an invalid name, from a step that still reported
success. It is also simply unused on the dev and tag paths.

Fixed a stale comment while in this block rather than leaving it for step
7: it claimed release tags carry "no `.N` per family release-posture rule".
Rule 148 was amended on 2026-08-24 to REQUIRE the suffix, after the ban
caused a same-day tag to be retargeted and a release deleted to make room
(note 2813). FC's own tags already carry suffixes; only the comment was
asserting the superseded rule.

Verified before pushing: the derivation holds across 200 commits of real
history — a derived revision always touches its own path set, the version
never decreases along any parent->child edge, and web tracks all three
extension-only commits in the log. That last one is the direction that
would serve stale bytes on a pin.
2026-08-27 21:56:52 -04:00
bvandeusen 0c43fa3eb2 ci: shadow the per-artifact derived versions (milestone 313 step 2)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
extension / lint (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m44s
Every build job now logs the tag, version and revision its artifact would
get. Nothing reads them; no `set -e`, and each derivation falls back to
UNAVAILABLE, so a broken script cannot fail a build. Same discipline as
milestone 271 step 2, which is what made that cutover safe to do in one
commit.

Also fixes a landmine the plan named but had not checked: build-ml and
build-agent were checking out at depth 1. Both now use fetch-depth: 0.

That mattered more than it looks. A depth-1 clone sees one commit, so
`git log HEAD -- <shipped paths>` either returns that commit's timestamp —
plausible, and wrong — or returns nothing. For build-ml on this push it
would have returned today's date, because HEAD touches backend/, and
nothing downstream would have questioned it. For build-agent it would have
returned nothing at all, since no single commit here touches agent/, and
artifacts.sh exits non-zero rather than guessing. One direction is silent
and one is loud; only the loud one was ever going to get noticed.

What to read from the shadow lines over the next few pushes, in order of
how badly each would bite:

  * a push touching the extension must move BOTH the extension and web,
    because build-web bakes the XPI in. If web does not move, its path set
    is too narrow and a pinned web image will serve an extension it does
    not name.
  * a push touching only agent/ must leave web and ml still. If they move,
    their sets are too wide and they will rebuild for changes they do not
    ship.
  * a docs-only push must move nothing.
  * dev and main must derive the same values for the same source.

Step 3 only lets these values name a tag once those hold.
2026-08-27 21:41:14 -04:00
bvandeusen cf06c81db9 build: one definition per artifact of what it ships (milestone 313 step 1)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 41s
CI / backend-lint-and-test (push) Successful in 2m4s
CI / integration (push) Successful in 4m15s
Build images / build-web (push) Successful in 4m50s
Build images / build-ml (push) Successful in 5m43s
Build images / build-agent (push) Successful in 10m45s
scripts/artifacts.sh generalises what packaging.sh established for the
extension: four published artifacts, four path sets, four independent
versions derived from the newest commit touching each set.

Measured on this commit, and this is the point of the whole thing:

  web        tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a
  ml         tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a
  agent      tag=2026.7.17  version=2026.7.17.1657  rev=57e5243
  extension  tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a

The agent is six weeks behind because agent/fc_agent has not changed since
57e5243. Today it rebuilds and re-tags on every push regardless; from step
4 it will not.

Three outputs, because they answer different questions and conflating them
is how this goes wrong:

  tag       YYYY.M.D        the published image tag. Day precision, per the
                            operator: same-day work is not worth pinning, so
                            a second build that day replaces the first.
  version   YYYY.M.D.HHMM   the ordering key. The extension needs this and
                            cannot use `tag`: Firefox compares it to decide
                            whether an update exists, so two same-day builds
                            must be distinguishable or the second hits the
                            ext-<version> cache and ships stale bytes. That
                            is issue #2397's failure mode exactly.
  revision  <sha>           content identity. Because `tag` is only
                            day-precise, "does this tag already exist" cannot
                            decide whether a build can be skipped — two
                            different builds legitimately share a tag. Step 4
                            keys on this instead.

Path sets read from the Dockerfiles rather than guessed. Notable calls:

  - web includes the extension's packaged set, because build.yml bakes the
    signed XPI into frontend/public/extension/ before the docker build. Miss
    that and :latest serves a NEW extension under an unchanged web version.
  - web excludes frontend/test: vite builds from src/, index.html and
    public/, so a spec change lands in the builder layer but never in dist.
  - agent is agent/Dockerfile + agent/requirements.txt + agent/fc_agent,
    NOT agent/. README.md, ruff.toml and docker-compose.yml sit in that
    directory and never reach the image.
  - every set includes its own Dockerfile and requirements: a base-image
    bump changes the artifact as surely as a source edit does.
  - the extension's set is read from packaging.sh, not restated. One
    definition, per #2397.

tests/test_artifact_paths.py guards both directions of being wrong, since
both are silent. Too narrow — a COPY'd file missing from the set — means the
version does not move when the content does, and a pin serves stale bytes.
Too wide means re-versioning for a change the artifact does not ship. The
test parses each Dockerfile's COPY lines and compares them against the
declaration, so adding a COPY without updating the set fails the lane.

No workflow reads any of this yet. Step 2 shadows it.
2026-08-27 21:36:17 -04:00
bvandeusen 0db38cc111 ci: log in to the registry with the docker CLI, not docker/login-action
Build images / sign-extension (push) Successful in 4s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 32s
CI / lint (push) Successful in 6s
CI / extension-version (push) Successful in 6s
extension / lint (push) Successful in 45s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m59s
CI / integration (push) Successful in 4m9s
Build images / build-agent (push) Failing after 10m58s
extension / lint (pull_request) Successful in 27s
build-ml failed at the login step twice on a7e626a, five seconds in, with
MODULE_NOT_FOUND on the action's own dist/index.js. Not the token — the
secret resolved to *** and the action never ran far enough to use it.

The cause is a race in act_runner's shared action cache, not corruption.
A remote action is cached at one /root/.cache/act/<hash> per runner, and
build-web, build-ml and build-agent all start in the same second and all
want docker/login-action@v3. One job re-clones that directory — emptying
and repopulating it — while another walks it to copy into its container,
and the walker lstat()s a file that just vanished. The two failures named
DIFFERENT missing files, eslint.config.mjs then jest.config.ts, which is
what rules out a bad cache entry and points at the race: a dangling entry
would name the same file every time.

Re-running does not help, because the re-run starts the three jobs
simultaneously again. It reproduced immediately.

Dropping the action removes FC from that race for this step. Logging in is
one command, the docker CLI is already in the CI image per
ci-requirements.md, and the same reasoning as family rule 5 applies: a
marketplace action buys nothing when the tool is baked into the image the
workflow already selected. Password on stdin, never as an argument — an
argument lands in the process table and draws docker's own deprecation
warning.

This narrows the exposure rather than closing it. All three jobs also
share docker/build-push-action@v5 and can race on it the same way; that
one has not lost yet, and replacing it means hand-rolling buildx
invocation including the build-args and provenance handling, which is a
bigger change than this failure justifies. Recorded on #3118.

Live consequence being cleared: fabledcurator-ml:dev was left a commit
behind fabledcurator:dev, which is the stale-pairing trap the trigger
comment on 239b1ed warns about.
2026-08-27 12:08:46 -04:00
bvandeusen a7e626a67a feat(extension): report the channel beside the version (step 7)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
extension / lint (push) Successful in 28s
CI / integration (push) Successful in 3m52s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Failing after 5s
Build images / build-agent (push) Successful in 13s
Build images / build-web (push) Successful in 2m4s
Closes the half of the ask the signing work didn't: a way to tell a dev
build from a main one. FC_CHANNEL is baked into the web image at build
time and /api/extension/manifest reports it as its own key, next to
version — the popup banner, the toolbar tooltip and the Settings card all
name it.

Beside the version, never inside it. A `1.0.3499884-dev` suffix is the
obvious shortcut and it is the exact failure this design comes from:
versionIsNewer parses each dotted segment with parseInt, so a suffixed
segment reads as 0, every dev build compares equal to every other, and
"no update available" stops being distinguishable from "I cannot read this
version". The comparator already degrades rather than discarding (rule
150), which is a reason not to NEED the suffix, not a licence to add one.
Two tests hold the line — one backend, asserting version and channel are
separate keys; one frontend, asserting the rendered version text stays the
bare derived number.

Optional on the read side, and absent rather than defaulted. An image
built before this field says nothing by not having the key; an image built
without a channel now says nothing the same way, so there is one absence
to handle instead of a second spelling of "unknown". Every reader drops
the label entirely when it is missing and reads exactly as it did before.
Reported verbatim rather than validated against {dev, main}: if an image
declares something else, showing what it claims helps whoever is debugging
more than dropping it would.

FC_CHANNEL is declared LAST in the Dockerfile. An ARG invalidates every
layer below it, and this is the one value that differs between the dev and
main builds of identical source — earlier, and the two channels could
never share a cached pip install. A tag push counts as main: a vYY.MM.DD
tag is cut from main, so that image is a main-channel artifact wearing an
immutable name.

No channel switcher, deliberately. background.js:34 already records that
Firefox's static update_url cannot apply, because every FC instance is a
different host — so the extension asks its configured backend, and the
channel IS the instance it points at. Switching is repointing apiUrl and
reinstalling from that host. A separate setting would contradict each
server build shipping its own extension.

This commit touches packaged extension files, so it moves the derived
version and will sign a new one via AMO — the first push to exercise the
extension-changed path from dev end to end.
2026-08-27 11:47:30 -04:00
bvandeusen fe48e77821 ci(extension): retire the manual-bump guard, true up the docs (step 5)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 22s
extension / lint (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 1m57s
Build images / build-ml (push) Successful in 2m38s
CI / integration (push) Successful in 3m50s
The guard asked whether a packaged extension file changed without the
version moving. Since step 4 nobody moves the version by hand, so it was
checking a fact that had stopped existing — and it was not merely dead
weight: it would have failed the lane on every real extension change,
demanding a bump that decides nothing. Removed rather than left running
beside the new mechanism (rule 22).

What replaces it is thinner and true. The extension-version lane now
asserts the derivation resolves on this commit, that the derived value is
the plain dotted-numeric shape AMO accepts, and that MAJOR.MINOR agrees
between manifest.json and package.json. MAJOR.MINOR is the one part still
hand-set, and packaging.sh reads it from manifest.json ALONE, so a
divergence ships a version package.json disagrees with. The lane keeps
fetch-depth: 0 — checking that the derivation survives a real checkout is
half its remaining value.

Deliberately not checked there: that the derived value beats what is
already signed. That guard belongs in build.yml, where it compares against
the real ext-* releases. Comparing against origin/main in a lane would be
wrong, because dev legitimately derives a LOWER value whenever main is
ahead on the extension, and a lane that fails for being behind is a lane
people learn to ignore.

packaging.sh is down to two consumers from three. version.spec.js's
"ci.yml derives its pathspec" test would have gone red on that, so it is
rewritten to assert the property rather than the consumer: no workflow
inlines an :(exclude)extension/ literal, across all three. That keeps the
#2397 anti-regression value while surviving consumers coming and going.
A second test pins build.yml to packaging.sh version and fails if it goes
back to grepping the committed value — which is not a style regression but
the #3092 bug itself. build.yml joins extension.yml's trigger paths, since
the suite now asserts against it.

The lockstep test narrows from the whole version string to MAJOR.MINOR.
The committed patch numbers are inert now; asserting on them would fail
for a difference that changes nothing.

Docs. extension/README.md's Release section described extension.yml
signing on main and committing the XPI into frontend/public/ — untrue
since 2026-05-25, and it told the reader to hand-bump both files, which is
now exactly the wrong instruction. Rewritten, with a Versioning section
that says plainly that editing the patch number does nothing and why the
key is commit time rather than a count. ci-requirements.md drops the third
packaging.sh consumer and names every job that needs full history. Root
README no longer claims the extension is signed on main only.
2026-08-27 11:29:44 -04:00
bvandeusen 9eb946b21b ci(extension): sign on dev too, and bundle the XPI into :dev (step 6)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-ml (push) Successful in 2m40s
CI / integration (push) Successful in 3m55s
Build images / sign-extension (push) Successful in 4m43s
Build images / build-web (push) Successful in 2m11s
Build images / build-agent (push) Successful in 10m13s
The step the milestone exists for. sign-extension ungates from main-only
to main-or-dev, and build-web downloads the XPI on dev as well, so a dev
push produces an image carrying the extension that is being developed
rather than requiring a merge to try one.

Not two signatures. The version is the commit TIME of the newest packaged
extension change, so dev and main derive the SAME number for the same
source. A dev push that changes the extension signs it; the merge to main
finds the ext-<version> release already there, hits the cache, and bundles
the byte-identical XPI into :latest with no second AMO call. One signature
per extension CHANGE, shared by both channels. That property is what makes
two channels affordable at all, and it is why step 4 had to land first:
ungating this while the version was still the hand-set 1.0.11 would have
found the existing ext-1.0.11 release, skipped AMO, and bundled main's
stale XPI into :dev — a dev channel confidently serving old code.

Tags stay excluded. The tag path deliberately skips signing and polls for
the release instead (the 2026-05-27 race).

The ext-<version> release's target_commitish moves from the literal "main"
to $GITHUB_SHA. Either branch can create that release now, and tagging a
dev-signed XPI against a main commit that need not even contain the source
it was built from is a lie that costs nothing to avoid.

Known, not addressed here: two concurrent builds that both derive the same
unsigned version will both call AMO and the loser gets a 409. The window
already existed between main and tag pushes; dev signing widens it. It
fails loudly rather than shipping anything wrong, and the rollback trap
cleans up the empty release. Filed separately.

Also unchanged here: ci.yml's manual-bump guard is still in place and
still false. It does not fire on this commit — nothing packaged changed —
but it will fail the lane on the next extension change, demanding a bump
that no longer decides anything. Step 5 next.
2026-08-27 10:56:58 -04:00
bvandeusen 5447a40e97 ci(extension): the derived version drives signing (milestone 271 step 4)
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
CI / backend-lint-and-test (push) Successful in 30s
extension / lint (push) Successful in 27s
Build images / sign-extension (push) Skipped
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
Build images / build-web (push) Successful in 2m8s
Build images / build-ml (push) Successful in 2m48s
CI / integration (push) Successful in 3m52s
Cutover. sign-extension no longer reads the version out of the repo — it
runs packaging.sh version and stamps the result into manifest.json and
package.json in the working tree before web-ext sees them. Never
committed back: the commit carrying the bump would itself be a change to
the extension and would move the version again.

Shadow mode ends here, in both build.yml and ci.yml. It had one job —
validate the formula at zero cost before a real AMO version was burned —
and CI confirmed it on 239b1ed: shadow: manual=1.0.11 derived=1.0.3499884.

build-web re-derives rather than being handed the value, so it gains
fetch-depth: 0. It was the outstanding landmine: a depth-1 clone derives a
WRONG, too-low version rather than failing, and would then 404 fetching a
release that exists under its real name. sign-extension and
extension-version already had full history.

New guard, and it stays permanently: refuse to sign when the derived
version is strictly OLDER than the highest ext-* release already signed.
Firefox rejects a downgrade and AMO never releases a burned version, so
backwards is unrecoverable — it strands every install that took the higher
one. Strictly older, not older-or-equal: equality is the ordinary case,
an unchanged extension deriving the same version it did last build, which
is exactly what makes the ext-<version> cache hit and holds AMO to one
call per extension CHANGE rather than per push. The release list is
paginated because ext-* shares it with the v* tags, and the bound fails
rather than calling the highest it happened to see the highest there is.

First derived value is 1.0.3499884 against a highest-signed ext-1.0.10, so
the backfill direction is right by six orders of magnitude. 1.0.11 sits in
the repo and was never signed; nothing is stranded by skipping past it.

Still main-only. Step 6 ungates sign-extension to dev, which is what
actually puts an XPI on :dev.

Note for step 5: ci.yml's manual-bump guard is now false. It still demands
a hand bump when a packaged file changes, and that bump no longer decides
anything — the derived value overwrites it at build time. Harmless but
pointless, and it should be retired before the next extension change.
2026-08-27 10:45:17 -04:00
bvandeusen 239b1ed8d9 ci: build :dev images again so the dev channel can carry a build
Build images / sign-extension (push) Skipped
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 29s
extension / lint (push) Successful in 30s
Build images / build-web (push) Successful in 2m23s
Build images / build-ml (push) Successful in 3m20s
CI / integration (push) Successful in 3m52s
Build images / build-agent (push) Successful in 9m26s
build.yml triggered on main and tags only. The 2026-05-26 comment gave
the reason: "operator tests from :latest after merge-to-main, not from
the dev branch image. Saves one full docker build per dev push."

That trade has since been named as a fault. Family rule 147 — main IS
production, test on :dev, never by shipping — and rule 146 — a rolling
channel refreshes itself, and a channel that can only be refreshed by
shipping is not a channel. 146's note on 147 describes this exact shape:
the pressure to test by shipping does not come from carelessness, it
comes from :dev being unable to carry the build.

Two live consequences, not hypotheticals:
  - docker-compose.yml pins fabledcurator:dev, an image nothing has
    published since May. The registry-image path of the documented
    quick-start could not have worked.
  - trying an extension change required merging to main, because
    sign-extension is gated to main and :dev did not exist to carry an
    XPI. Shipping was the only way to test.

All three images build on dev. Deliberate: a :dev web image paired with
a stale :dev ml or agent is a worse trap than no dev channel, because
the mismatch surfaces as a runtime failure rather than a missing tag.
The cost the 2026-05-26 note was avoiding is real and is now paid on
every dev push — layer reuse should keep ml's cost to the COPY layers,
but if it bites, narrowing is a `paths:` filter away.

:dev only. The dev path never writes :c-<sha>: that is the rollback unit
(rule 145), and a rolling tag may legitimately carry newer contents than
the :c-<sha> of the same commit.

This does NOT yet put an XPI on :dev — sign-extension is still gated to
main, and ungating it has to wait for the derived version to control
publishing, or dev would sign the hand-set 1.0.11, hit the existing
cache and ship main's stale XPI. That is the next step.
2026-08-27 09:26:58 -04:00
bvandeusen cd5444e3ae ci(extension): derive the version from commit TIME, not commit count (#3092)
Rule 149: an artifact's ordering key must be time-derived, never a commit
count. packaging.sh's cmd_patch was a count.

Why that matters here rather than in the abstract. A count is per-branch:
dev and main count different histories of the SAME code. Today only main
signs, so nothing has ordered the two against each other and the fault is
invisible. The moment dev also publishes an extension, the two versions
order by which branch accumulated more commits rather than by which is
newer — and a squash-merge makes it permanent, because main gains one
commit where dev gained five. dev then climbs away from main and a dev
install can never cross back.

That is Roundtable's 2026-08-24 incident (Scribe #2993) in a different
repo: their versionCode was the branch's commit count, and it produced a
channel you could enter and not leave. Measured on this repo today the
old formula gives main=23, dev=24 — one apart, which is exactly how the
inversion stays invisible until it strands somebody.

New formula: minutes since 2020-01-01 of the LATEST commit touching a
packaged extension file. Same anchor and unit Roundtable settled on.

Commit time, not build time, and the difference is load-bearing:

  - stable while the extension is unchanged, so the ext-<version>
    signature cache still hits and AMO is called once per extension
    CHANGE rather than once per push. Build-time minutes would re-sign
    on every push and never let two channels share a signature.
  - after a merge, main sees the same commit and derives the same
    number, so :latest reuses the signature :dev already produced for
    byte-identical code. Same code, same version, one signing.
  - monotonic: max() over a set that only gains members. Verified
    across all 24 extension-touching commits, zero non-monotonic steps.
  - reproducible from any checkout.

Derives 1.0.3499884 on dev, 1.0.3465860 on main — both far above the
last hand-set 1.0.11, so milestone 271's backfill guard is satisfied by
construction rather than by an offset.

Still shadow-only: nothing reads the derived value yet. Both shadow
steps log it, and ci.yml's runs on dev too, so both channels' numbers
are visible — that is the pair that has to stay ordered. Prior shadow
observations describe the OLD formula and prove nothing about this one,
so the window restarts; ci.yml says so at the step.

New requirement recorded in ci-requirements.md: a depth-1 clone derives
a wrong, too-low value rather than failing, so fetch-depth: 0 is
load-bearing wherever packaging.sh version is called.

Refs #3092, milestone 271
2026-08-27 09:26:58 -04:00
bvandeusen 5a0e1bbd03 perf(ml): batch the auto-apply sweeps' image_tag inserts (#3072)
CI / extension-version (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 4m2s
Item 1 of #3072. Both sweeps issued a single-row pg_insert(image_tag)
from inside their per-image loop. Steady state that is nothing; a first
sweep over a back-catalogue is one round-trip per applied tag, tens of
thousands of them. Each chunk now collects its rows and writes them in
one statement.

The ticket suggested one insert per chunk PER TAG. A single multi-row
VALUES carries every tag at once, so it is one statement per chunk full
stop — and the sweeps already accumulate across all heads before they
commit, so nothing had to be restructured to allow it.

Not a new helper: wip_title.apply_wip_image_tags was already doing the
chunked ON CONFLICT DO NOTHING insert, so that shape is extracted to
services/image_tag_apply.insert_image_tags and all three writers share
it. The extraction deliberately leaves wip_title's pre-SELECT behind
rather than pulling it into the shared function — the sweeps don't need
it (their `skip` sets already exclude applied and rejected images) and
it exists only to produce an accurate count, which the sweeps also
compute themselves. So the shared primitive returns nothing: psycopg
reports rowcount -1 for a multi-row ON CONFLICT DO NOTHING insert, and a
count taken from the statement would be a lie rather than an
approximation.

Ordering note for the system-tag sweep: tag rows are now written after
that chunk's PresentationReview rows rather than interleaved before
them. Safe — PresentationReview FKs to image_record and tag, not to
image_tag.

Chunk size stays 5000: 5000 rows x 3 bound params = 15000, inside
Postgres' 65535-parameter ceiling with room to spare.

tests/test_image_tag_apply.py covers the primitive directly, since it is
now the single place three writers can be wrong at once — most
importantly that a re-run never restamps a hand-applied tag's source,
which would silently poison head training (it excludes the auto
sources).

Left alone: _insert_presentation_review is still per-row, and the
retract path still deletes per-row. Both operate on sets that are small
by construction, unlike the apply path.

Refs #3072
2026-08-27 07:51:23 -04:00
bvandeusen 1ac448d881 refactor: four small cleanups from the review pass (#3072)
CI / extension-version (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 4m57s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
Items 2-5 of #3072. Item 1 (the per-row sweep inserts) is separate.

2. .fc-bad was not merely duplicated — it is .fc-weak under a second
   name. Both local definitions were `color: rgb(var(--v-theme-error))`,
   identical to the global .fc-weak, and GpuAgentCard was already using
   .fc-weak to colour exactly what GpuActivityPanel coloured .fc-bad (an
   errored count, red when non-zero). So rather than promoting a synonym
   to app.css, both call sites now use .fc-weak and the local defs are
   gone. app.css's status-colour comment records why there is no .fc-bad,
   next to the existing note on why .fc-ok is deliberately NOT global.

3. GalleryItem.vue's obsidian literals now use --v-theme-background,
   which IS obsidian (vuetify-theme.js maps background -> surfaces.
   obsidian). Preferred over --fc-chrome-rgb: same value, but that
   variable is named for the nav fade, not for the palette entry.

   The ticket said these were the only three real uses in the tree. They
   are not — GalleryItem itself had two more in the artist-label
   gradient (fixed here, so the file is now consistent), and ~13 more
   live in SeriesView, SeriesReaderView, ImageViewer, ArtistHeader,
   ExploreView and GalleryFilterBar. Those are a separate sweep, filed
   rather than folded in here.

4. The attachment download path had two hand-formatted copies. One
   definition now, `attachment_download_url`, next to the model both
   serializers already import. The test pins it by MATCHING the built
   path against the app's real URL map rather than comparing to a
   literal — a string-equality test would still pass after someone
   renamed the route, which is the drift the helper exists to prevent.

5. Extension API key now compares with hmac.compare_digest. Compared as
   BYTES, not str: compare_digest's str form raises TypeError on
   non-ASCII, and this value comes straight from an attacker-controlled
   header, so the str form would turn a junk key into a 500 instead of a
   403. Low stakes either way — the API is unauthenticated-by-design on
   a LAN — but it costs nothing.

Refs #3072
2026-08-27 07:48:18 -04:00
bvandeusen bfc5135f19 docs: true up README — status, the five pieces, CI (#3070)
CI / extension-version (push) Successful in 2s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 5m0s
README.md was last touched in 4aff9c5 (2026-05-14) and several of its
most visible lines had gone false:

- "Pre-v1. Not yet functional." — FC has been continuously deployed for
  months. Replaced with what main/dev actually mean for what's running.
- "Node 22 pre-installed" — ci-requirements.md and extension.yml both say
  node 24, and frontend/package.json requires >=24.
- "Runner label python-ci — a runner with Python 3.14, ruff and Node 22
  pre-installed ... The runner image (runner-base:python-ci) is built
  from CI-Runner/CI-python/" — describes runs-on as selecting the
  toolchain. It doesn't: runs-on is a scheduling label, and every job
  names its own container.image (ci-python:3.14, or node:24-bookworm-slim
  for the extension lane).
- "Both ci.yml and build.yml use this label" — there are three workflows.

The CI section now points at ci-requirements.md rather than restating
it, so the two can't drift apart again; that file is current and is the
one the CI-runner process expects.

Added a "What's in here" table for the five deployable pieces — the
extension, the GPU agent and the ML image were unmentioned, three of the
five. Also corrected the RELEASE_TOKEN write:release scope, which is no
longer "for future release-cutting workflows": it backs the ext-<version>
releases that cache the signed XPI.

Refs #3070
2026-08-27 07:39:49 -04:00
bvandeusenandClaude Opus 5 89155478a8 test(refetch): cover the Layer-2 auto-refetch remediation (#3071)
CI / extension-version (push) Successful in 2s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m59s
refetch_service was the only module under backend/app/services/ with no
test file — and not an inert one: it runs unattended off the recovery
sweep and deletes a file from disk before asking a downloader to replace
it. The frontend cites it by name as the reason the Import tab could be
retired ("imports heal themselves").

The ticket described it as having zero direct coverage. That is true of
the module, but not of the code: test_api_import_admin.py already drives
the happy path end-to-end through the refetch route — file deleted, task
flagged, one dispatch, second attempt a no-op. These 17 tests therefore
target what the route tests cannot reach rather than restating them:

  * every branch of resolve_refetch_source — disabled Source, a
    `sidecar:<platform>:<slug>` synthetic anchor, a platform mismatch,
    the gallery-dl `NN_` numbering-prefix sidecar, the lowest-id pick
    among several candidates, and each of the five ways it declines
    (no sidecar, unreadable JSON, non-object JSON, no platform, no
    artist folder / no matching Artist row).

  * that the file SURVIVES when nothing re-pollable resolves. This is
    the assertion the module exists for: `no_source` is the common case
    on a filesystem-only library, where the file on disk is the
    operator's only copy. The route-level no_source test cannot catch a
    regression here — its path never existed, so an unconditional unlink
    would pass it.

  * that the `refetched` bound is checked BEFORE the unlink, so a second
    sweep leaves the re-downloaded file alone rather than deleting it
    again.

  * that an unlink failure is logged and stepped over, not raised —
    a raise would abort the whole sweep for every other poison-pill row
    in the batch. Exercised with a real IsADirectoryError rather than a
    patched pathlib.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 07:33:14 -04:00
bvandeusenandClaude Opus 5 516521e7b0 refactor(platforms): drop migration 0088 — no deviantart rows exist (#3069)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
CI / integration (push) Successful in 3m43s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 28s
Operator confirms the instance has never used DeviantArt, so there is
nothing for 0088 to quiesce. The migration only ever had two jobs —
disable leftover `source` rows and delete a stale `credential` row — and
both were guards against data that does not exist here.

Removing it rather than keeping a no-op: a migration that runs on every
deploy to touch zero rows is a permanent cost paid for a hypothetical,
and it would read to a future reader as evidence that DeviantArt sources
once existed. `platform` has no CHECK constraint, so retiring the key
needs no schema change of its own.

alembic head returns to 0087.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 07:26:02 -04:00
bvandeusenandClaude Opus 5 ddf896078c refactor(platforms): retire deviantart end-to-end (#3069)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 23s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m43s
Executes the 2026-07-05 product decision (FC downloaders = art-dedicated
services only), which removed Twitter/X and Bluesky but left deviantart
fully wired for seven weeks — the half-retired state rule 22 exists to
prevent.

Removed: the PlatformInfo module and its registry entry, the gallery-dl
extractor block, extension_service's artist-page pattern, the extension's
PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission
and content-script match, the frontend icon/colour/label, and the operator-
facing "supported platforms" list that still advertised it.

Two judgment calls, both recorded in migration 0088:

  * existing `source` rows are DISABLED, not deleted. The row is the only
    record of the artist's DeviantArt URL. Disabling is also required for
    correctness rather than tidiness: with the platform unregistered the
    download path falls through to gallery-dl, which carries its OWN
    deviantart extractor, so an enabled row would have kept downloading
    from a dropped platform.
  * the `credential` row IS deleted — a live session cookie for a site FC
    will never call again.

Adds the invariant whose absence is why manifest.json drifted in the first
place: nothing tied its domain lists back to the platform table. The
extension suite now asserts both directions, plus that no host permission
belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at
an operator-chosen URL the extension cannot enumerate).

Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged
extension change without a bump. No release is cut — build.yml's
sign-extension job only runs on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 07:24:08 -04:00
bvandeusenandClaude Opus 5 2e0f8f8c61 feat(cleanup): reclaim orphaned attachments — rows and store blobs (#3068)
CI / extension-version (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 28s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 24s
CI / integration (push) Successful in 3m47s
PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or
artist left the row behind rather than taking it. Nothing ever pruned those
rows, and nothing in the repo had ever unlinked a file under the attachment
store — so both rows and bytes accumulated permanently, invisible to every
existing diagnostic.

Why a disk->DB reconciliation rather than a row sweep: the store is
sha-addressed and idempotent, so ONE blob backs MANY rows. Deleting a row
does not free its blob, and since the artist cascade (#3066) now deletes
its attachment rows outright, a freed blob has no DB pointer left to find
it by. Walking the store and asking "does any row still reference this
sha?" catches orphans from every cause, including ones no future delete
path will think to report.

Preview and apply share `_orphan_attachment_conditions` (rule 93). The
dry-run derives its surviving-sha set by NEGATING that same predicate, so
it is honest about blobs the delete would free rather than counting them as
still-referenced — the one place this was easy to get backwards, so it has
its own parity test.

Guards, each with a reason:
- A blob is written before its row commits, so a just-stored file legitimately
  has no referencing row. Files under 6h are never judged — same guard and
  reasoning as ORPHAN_TEMP_MIN_AGE_HOURS.
- `.partial` staging files belong to cleanup_orphaned_temp_files; skipped
  rather than raced.
- The sha is parsed as the first 64 chars, not via Path.stem: store() takes
  the extension from the source filename, and a URL-encoded basename yields a
  multi-dot suffix that would make stem eat part of the sha.
- A 900s walk budget reports partial=True instead of running to the task's
  hard limit (rule 89).
- TASK_STUCK_THRESHOLD_MINUTES override at 30 (= time_limit 25 + 5). Without
  it a healthy 20-minute walk is phantom-flagged 'RecoverySweep' at the bare
  5-min default — the #883 failure class; its invariant test is mirrored here.

Defaults to the safe preview at both the task and the route, unlike the other
maintenance triggers: this apply unlinks files. Operator-triggered only,
never on a beat.

Ships with its UI (rule 27): AttachmentReclaimCard in Cleanup → Duplicates &
leftovers, built on the existing useMaintenanceTask/MaintenanceTile shapes, so
a run survives navigating away. Surfaces files_failed and partial explicitly,
since both change what the numbers mean.

Also promotes humanBytes to utils/bytes.js — it was byte-identical in
VideoDedupCard and GatedPurgeCard and this card would have been the third
copy. The three divergent `formatBytes` helpers are deliberately left alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 22:37:24 -04:00
bvandeusenandClaude Opus 5 2ce467e347 fix(cleanup): artist cascade preview counts posts and attachments (#3067)
CI / lint (push) Successful in 5s
CI / backend-lint-and-test (push) Successful in 34s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 23s
CI / integration (push) Successful in 4m32s
`project_artist_cascade` is documented as "a read-only projection of what
delete_artist_cascade would touch" and drives the Tier-C confirm dialog,
but it counted only images, sources, thumbs, import_tasks and bytes. It
never counted posts or attachments, both of which the apply destroys.

That is silent in the worst case. `Post.artist_id` is ondelete=CASCADE, so
every post goes whether or not it carried an image — and FC has a large
body-only post population (#1288 measured 694 pixiv posts with text and no
images). Such an artist previewed as `images: 0`, reading as "empty, safe
to remove", while the apply destroyed every captured body, description,
external-link set and raw_metadata snapshot. The danger-zone card already
promised "every image, source, post, and attachment" — the copy was honest
and the numbers were not.

Root of the drift: the preview re-derived its own predicates instead of
sharing the apply's, the same shape as the 2026-06-08 fandom-tag deletion
that rule 93 exists for. So rather than bolt on two counts, both halves now
build from shared `_artist_{images,posts,attachments}_conditions` helpers,
following the `_unused_tag_conditions` / `_bare_post_conditions` style
already in the file. Sources keep no helper — the apply doesn't query them
either, it gets them from the Artist.sources ORM cascade.

The apply also now reports `posts_deleted` (counted before the delete,
since the CASCADE leaves nothing to count after). Rule 93's second half
asks for the apply to be tested, and parity is only assertable if both
halves state the number.

Adds a preview/apply parity test that runs both against one artist and
asserts the three pairs agree AND that the rows actually went, plus a
body-only-artist test covering the case that motivated this.

The confirm dialog's counts grid renders every key, so posts and
attachments surface there automatically; the prose summary line names
posts explicitly, since that is the number that changes how an artist
reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 22:23:29 -04:00
bvandeusenandClaude Opus 5 39cf81aea6 fix(cleanup): clear an artist's attachments before the cascade delete (#3066)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m49s
`delete_artist_cascade` could abort partway through, and it aborted after
the irreversible half. Deleting an artist CASCADEs to Post (post.artist_id
is ondelete=CASCADE), which SET NULLs post_attachment.post_id — and
`uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE
WHERE post_id IS NULL. So any two of that artist's attachments sharing a
sha collapse onto one another and raise.

That shape is ordinary, not corrupt: `_capture_attachment` deliberately
writes one row per post over a single sha-addressed blob, so a creator who
attaches the same pdf to two posts already has two such rows. A
pre-existing filesystem-import row (post_id NULL) with the same sha
collides on its own.

The images and their on-disk files are deleted and committed in 500-row
batches BEFORE the artist row is touched, so the failure landed after
them: images gone, artist and posts alive, files unrecoverable.

Fix: delete the artist's post_attachment rows explicitly first, matched by
artist_id OR by the owning post's artist (artist_id is nullable, so neither
arm alone covers every row). `_repoint_post_links` already guards the
identical collision class in the reconcile path; this is its artist-cascade
counterpart. Migration 0043 reasoned only about upgrade-time safety and
never about this later SET NULL.

The sha-addressed blobs are deliberately left on disk: one blob backs many
rows, so unlinking needs a refcount pass, and this Tier-C op must not
delete bytes its own preview never disclosed.

Adds `attachments_deleted` to the summary, and two regression tests — the
same sha on two posts, and an unrelated NULL-post row that must survive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 22:02:51 -04:00
Claude 11dd324f89 fix(extension): exclude the test/ and scripts/ directory entries from the XPI
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
extension / lint (push) Successful in 19s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m57s
extension / lint (pull_request) Successful in 34s
The XPI-content check added in 1c6452e did its job on its first run: the
archive carried empty `test/` and `scripts/` entries.

`test/**` matches the files inside a directory but not the directory entry
itself, and web-ext writes an entry for the directory separately -- so the
contents were correctly excluded while the empty directories shipped anyway.
Nothing harmful reached users (no dev code, just two empty entries), but our
single declaration claimed these do not ship and something was shipping.

Both forms are now listed per directory. The bare name alone would not do:
minimatch's `test` does not match `test/url.spec.js`, so dropping the glob
would ship the contents instead.

Verified locally: the derived version is unchanged at 1.0.19, confirming the
added entries are redundant for the git pathspec and only affect web-ext.

Refs #2400

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 16:20:54 -04:00
Claude 1c6452e10e ci(extension): shadow the derived version + verify real XPI contents
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
extension / lint (push) Failing after 28s
CI / backend-lint-and-test (push) Successful in 47s
CI / integration (push) Successful in 4m1s
Milestone #271 steps 2 and 3. Neither changes what gets published.

STEP 2 -- shadow mode.

build.yml's sign-extension and ci.yml's extension-version guard now log the
version that WOULD be derived from git history alongside the hand-maintained
one. Nothing reads the derived value, and neither site can fail because of it.

This exists because `web-ext sign` is one-shot per version: AMO 409s on a
repeat, so a wrong formula burns a real version number that cannot be
reclaimed. Comparing the two across real builds is the only way to validate it
at zero cost. sign-extension runs on main only, so main pushes are the sole
source of truth for whether the derived number moves exactly when the shipped
extension changes -- the dev-side log is a convenience, not the evidence.

sign-extension now checks out with fetch-depth: 0. The derived version is a
commit count and a depth-1 clone cannot produce one.

STEP 3 -- XPI content verification.

Every other packaging assertion checks our declaration against itself. This is
the first that asks web-ext what it ACTUALLY wrote into the archive.

That assumption was both unverified and fragile: `test/**` only survives to
web-ext because callers `set -f` before substituting it, so losing that
quoting would silently start shipping dev files with no other signal. The step
builds the XPI and asserts test/, scripts/, vitest.config.js, package.json,
package-lock.json, README.md and node_modules are absent -- and, because an
over-matching exclusion would break the extension at runtime rather than at
build time, that manifest.json, all four lib/*.js and every UI directory are
present.

unzip is installed only when missing; node:24-bookworm-slim may not carry it.

Refs #2399, #2400

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 16:15:12 -04:00
Claude 597b91d29b refactor(extension): one definition of what ships in the XPI
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
extension / lint (push) Successful in 20s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m59s
Milestone #271 step 1. Groundwork for deriving the extension version from git;
no behavior change yet -- nothing consumes `version` so far.

"Which files end up in the XPI" was stated in two places and about to become
three. Three hand-kept copies of one fact is what allowed #2397, where the
publish path could republish a stale XPI because its cache key had no link to
the content it stood for.

New extension/scripts/packaging.sh holds the single declaration and exposes:
  ignore       web-ext --ignore-files values
  pathspec     :(exclude)extension/... for git
  version      <MAJOR.MINOR from manifest>.<commit count over packaged files>
  major-minor  /  patch

Consumers now delegate instead of restating it:
- extension/package.json -- all four web-ext scripts
- .forgejo/workflows/ci.yml -- the extension-version guard's exclusions
- (step 4) the rev-list that derives the version

scripts/** joins the non-packaged set; the script must not ship to users.

Two shell hazards, both load-bearing:

The script runs `set -euf`. Its lists are iterated with deliberate word
splitting, and without -f the shell ALSO globs them -- invoking `pathspec`
from a directory where test/ exists (exactly how ci.yml calls it) would expand
`test/**` into the individual spec files and silently stop covering anything
added later. A caller's own `set -f` cannot prevent this: the script is a
separate sh process and does not inherit it.

Callers additionally need their own `set -f` for the substituted RESULT, which
is a different expansion. version.spec.js asserts every --ignore-files caller
sets it, that the pathspec comes through with `test/**` literal and no
.spec.js paths, and that neither consumer has reinstated a hardcoded list --
the easy future regression is "simplifying" by inlining one again.

Verified: all five subcommands plus the usage/exit-2 path. Derived version on
main (8300029) is 1.0.19, matching dev. Last published is 1.0.10, so the
eventual cutover moves strictly upward and needs no offset -- Firefox refuses
downgrades. (An earlier note recorded 18; that was measured against a stale
origin/main from before the PR #234 merge.)

Refs #2398

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 15:33:11 -04:00
64 changed files with 3543 additions and 408 deletions
+722 -78
View File
@@ -2,10 +2,18 @@ name: Build images
on:
push:
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
# merge-to-main, not from the dev branch image. Saves one full docker
# build per dev push.
branches: [main]
# `:dev` builds were dropped 2026-05-26 to save a docker build per dev
# push, on the reasoning that "operator tests from `:latest` after
# merge-to-main". Restored 2026-08-27: that is testing by shipping, and
# family rules 146/147 now name it directly — `main` IS production, and a
# channel that can only be refreshed by shipping is not a channel. The
# pressure to merge in order to try something does not come from
# carelessness; it comes from `:dev` being unable to carry the build.
#
# All three images build on dev, deliberately: a `:dev` web image paired
# with a stale `:dev` ml or agent is a worse trap than no dev channel at
# all, since the mismatch only shows up as a runtime failure.
branches: [main, dev]
# Tag-push triggers an immutable per-version image build (e.g.
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
@@ -25,28 +33,156 @@ jobs:
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
# Result is uploaded as an Actions artifact for build-web to consume.
#
# Why this lives in build.yml (not a separate workflow): the merge-commit's
# docker image tagged `:latest` MUST carry the XPI. A separate sign workflow
# racing build.yml leaves `:latest` without the XPI for ~5min (until the
# commit-back triggers another build). Inline ordering eliminates the race.
# Why this lives in build.yml (not a separate workflow): the image a push
# publishes MUST carry the XPI. A separate sign workflow racing build.yml
# leaves that image without one for ~5min (until the commit-back triggers
# another build). Inline ordering eliminates the race.
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
# branch to manage). AMO blocks re-signing the same version (returns 409),
# so signing is intentionally one-shot per version bump.
# so signing is intentionally one-shot per version.
#
# BOTH branches sign (milestone 271 step 6, 2026-08-27). Not two signatures:
# the version is the commit TIME of the newest packaged-extension change, so
# dev and main derive the SAME number for the same extension source. A dev
# push that changes the extension signs it; the merge to main then finds the
# ext-<version> release already there, hits the cache, and bundles the
# byte-identical XPI into `:latest` with no second AMO call. One signature
# per extension CHANGE, shared by both channels — that is what makes two
# channels affordable, and it is why step 4 (derived version) had to land
# first. Ungating this while the version was still the hand-set 1.0.11 would
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
# `:dev` — a dev channel confidently serving old code.
#
# Tags stay excluded: the tag path deliberately skips signing and polls for
# the release instead (see build-web's race note, 2026-05-27).
sign-extension:
if: github.ref == 'refs/heads/main'
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Full history is load-bearing, not a convenience: the version this
# job signs is derived from the commit TIME of the newest packaged
# extension change. A depth-1 clone sees one commit and derives a
# wrong, too-low value rather than failing (ci-requirements.md).
fetch-depth: 0
- name: Resolve extension version
# The version is DERIVED, not read from the repo (milestone 271 step 4,
# cut over 2026-08-27). `packaging.sh version` returns MAJOR.MINOR from
# manifest.json plus a patch component that is the commit TIME of the
# newest change to a PACKAGED extension file, in minutes since
# 2020-01-01 — family rule 149, never a commit count, which orders by
# branch rather than by recency.
#
# The committed "version" in manifest.json / package.json no longer
# decides anything: the stamp step below overwrites it in the working
# tree before web-ext ever reads it. It is deliberately NOT committed
# back — the commit carrying the bump would itself be a change to the
# extension and would move the version again. The repo holds the source;
# the build derives the label.
- name: Derive extension version
id: extver
run: |
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
set -eu
VERSION=$(sh extension/scripts/packaging.sh version)
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Resolved extension version: $VERSION"
echo "Derived extension version: $VERSION"
# Firefox refuses a downgrade and AMO never releases a burned version,
# so a version that moves BACKWARDS is unrecoverable: it strands every
# install that already took the higher one. Two ways it could happen —
# a checkout without full history (derives too low), or a rewritten
# history that drops the newest packaged commit.
#
# The test is `derived < highest already signed`, strictly. Equality is
# the ORDINARY case, not a fault: an unchanged extension derives the same
# version it did last build, which is exactly what lets the ext-<version>
# cache hit and holds AMO to one call per extension CHANGE. Only moving
# backwards is a failure, so this runs on every path — cache hit
# included — rather than only before a sign.
# --- shadow mode (milestone 313, step 2) -----------------------------
# Informational ONLY. Nothing reads this and it must never fail the
# build — no `set -e`, and every derivation falls back to UNAVAILABLE.
#
# What to watch across pushes, because this is what step 3 will trust:
# * a push touching only agent/ moves the agent and leaves web and ml
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# * dev and main derive the same values for the same source.
- name: Shadow — derived artifact version (informational)
run: |
set -u
A=extension
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
- name: Guard — the derived version must never go backwards
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
DERIVED: ${{ steps.extver.outputs.version }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request
API = ("https://git.fabledsword.com/api/v1/repos/"
"bvandeusen/FabledCurator/releases")
headers = {"Authorization": "token " + os.environ["TOKEN"]}
# Paginated rather than first-page-only: ext-* releases share this
# list with the v* release tags, so one page would start missing them
# as those accumulate. The bound FAILS rather than silently scanning
# part of the list and calling the highest it saw the highest there is.
tags = []
for page in range(1, 21):
req = urllib.request.Request(
f"{API}?limit=50&page={page}", headers=headers)
with urllib.request.urlopen(req, timeout=30) as resp:
batch = json.load(resp)
if not batch:
break
tags += [r.get("tag_name", "") for r in batch]
else:
sys.exit("guard: >1000 releases — pagination bound reached")
def parse(v):
try:
return tuple(int(part) for part in v.split("."))
except ValueError:
return None
derived_s = os.environ["DERIVED"]
derived = parse(derived_s)
if derived is None:
sys.exit(f"guard: derived version {derived_s!r} is not numeric")
signed = sorted(
(v, t) for t in tags if t.startswith("ext-")
for v in [parse(t[4:])] if v
)
if not signed:
print("guard: no ext-* release yet — nothing to go backwards from")
raise SystemExit(0)
hi, hi_tag = signed[-1]
print(f"guard: derived={derived_s} highest already signed={hi_tag}")
if derived < hi:
sys.exit(
f"REFUSING TO SIGN: derived {derived_s} is OLDER than the "
f"already-signed {hi_tag}. Firefox would reject it as a "
f"downgrade, and AMO will not release the burned version. "
f"First thing to check: did this job check out with "
f"fetch-depth: 0?"
)
print("guard: ok")
PY
- name: Check Forgejo release-asset cache
id: cache
@@ -84,6 +220,29 @@ jobs:
# removal — sign-extension's job is just to ensure the cache
# exists on Forgejo; the build-web side reads it independently).
# web-ext signs whatever manifest.json says, so the derived value has to
# reach the tree before signing. package.json is written too: the two are
# required to agree (ci.yml's guard), and a local `npm run build` reads
# it. Working tree only — never committed, per the note on the derive
# step.
- name: Stamp the derived version into manifest.json + package.json
env:
DERIVED: ${{ steps.extver.outputs.version }}
run: |
python3 - <<'PY'
import json, os
version = os.environ["DERIVED"]
for path in ("extension/manifest.json", "extension/package.json"):
with open(path) as fh:
doc = json.load(fh)
doc["version"] = version
with open(path, "w") as fh:
json.dump(doc, fh, indent=2)
fh.write("\n")
print(f"{path}: version -> {version}")
PY
- name: Sign via AMO (cache miss)
if: steps.cache.outputs.cached != 'true'
run: |
@@ -110,6 +269,11 @@ jobs:
# created it so an upload failure below can roll back (don't
# leave an empty release tombstone that the next run's
# cache-check mistakes for a partial-failure state).
#
# target_commitish is the signing commit, not a branch name: since
# step 6 either branch can create this release, and hard-coding
# `main` would tag a dev-signed XPI against a main commit that may
# not even contain the extension source it was built from.
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
@@ -117,7 +281,7 @@ jobs:
CREATED_BY_US=false
else
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"main\"}" \
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"$GITHUB_SHA\"}" \
-o release.json \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
CREATED_BY_US=true
@@ -160,19 +324,208 @@ jobs:
build-web:
needs: [sign-extension]
# sign-extension is main-only; on dev it's skipped, build-web still runs.
# sign-extension runs on main and dev, and is skipped on a tag push (which
# polls for the release instead). Either is fine to build on; a FAILED sign
# is not — this condition lets success and skipped through, so a failure
# skips build-web rather than shipping an image without the XPI.
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Full history: this job RE-DERIVES the extension version rather than
# being handed it, and a depth-1 clone derives a wrong, too-low value
# rather than failing — which would 404 the download of a release
# that exists perfectly well under its real name.
fetch-depth: 0
- name: Download signed XPI from Forgejo release asset (main + tags)
# Fires on main-push AND on tag-push. Tag-push builds re-package the
# same source code as the preceding main-push build but with an
# immutable version tag — they need the XPI too, otherwise the
# versioned image ships without the signed extension.
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
# never be the reason an image does not ship.
#
# What it should say:
# * a push touching only agent/ moves the agent and leaves web and ml
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
set -u
A=web
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
- name: Determine tag
id: tag
run: |
# Three trigger shapes:
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
# plus `.N` when the day already carries a tag —
# family rule 148, amended 2026-08-24 after a
# same-day tag was retargeted and a release
# deleted to make room, note 2813).
# Publish ONLY the immutable version tag;
# don't touch :latest (the main-push build
# for the merge commit already did that).
# refs/heads/main → push to main: publish :main + :latest
# (floating) AND :c-<short_sha> (immutable
# per-commit rollback substrate, per family
# release-posture rule "Tags are milestones,
# not gates — commit-SHA images are the
# rollback unit"). Rollback to any commit
# becomes `docker pull …:c-<sha>` without a
# release ceremony.
# refs/heads/dev → push to dev: publish :dev, the rolling test
# channel (family rule 146). Rolling means it may
# carry newer contents than the :c-<sha> of the
# same commit; it never writes :c-<sha> itself,
# because that is the rollback unit (rule 145).
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
# THIS artifact's shipped files last changed in. Day precision is
# deliberate — same-day work is not something worth pinning, so a
# second main build the same day replaces the first rather than
# accumulating a tag nobody would roll back to.
#
# Derived per artifact, so an image whose files did not change keeps
# the tag it already had: the agent reads 2026.7.17 today while web
# reads 2026.8.27 — and the reuse step below turns that into a
# skipped build rather than a rebuild of bytes that already exist.
# `channel` is baked into the image as FC_CHANNEL and reported by
# /api/extension/manifest (milestone 271 step 7). A tag-push counts as
# `main`: a vYY.MM.DD tag is cut from main, so that image is a
# main-channel artifact wearing an immutable name.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
CALVER=$(sh scripts/artifacts.sh tag web)
# Guarded, and computed only on this path. There is no `set -e` in
# this step, so a failed derivation would otherwise leave CALVER
# empty and publish the tag `fabledcurator:` — an invalid
# name, from a green step. An empty pin must never reach the
# registry.
if [ -z "$CALVER" ]; then
echo "ERROR: could not derive a web version tag" >&2
exit 1
fi
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator:${CALVER}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
echo "channel=dev" >> "$GITHUB_OUTPUT"
fi
# A shell step, not docker/login-action@v3, because the action's shared
# cache races itself (#3118). act_runner caches a remote action under one
# /root/.cache/act/<hash> per runner, and build-web, build-ml and
# build-agent all start in the same second and all want this same action.
# One job re-clones the directory — which empties and repopulates it —
# while another is walking it to copy into its container, and the walker
# lstat()s a file that has just vanished. It failed twice on 2026-08-27,
# naming a DIFFERENT missing file each time (`eslint.config.mjs`, then
# `jest.config.ts`), which is what rules out a corrupt cache and points at
# a race. The loser dies with MODULE_NOT_FOUND on dist/index.js before the
# action runs at all, so the secret is never even reached.
#
# Nothing is lost by dropping it: logging in is one command, the docker
# CLI is already in the CI image (ci-requirements.md), and the same
# reasoning as family rule 5 applies — a marketplace action buys nothing
# when the tool is baked into the image the workflow already selected.
#
# Password on stdin, never as an argument: an argument lands in the
# process table and draws docker's own deprecation warning.
- name: Login to Forgejo registry
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
#
# The failure direction is deliberate. An inspect that errors for ANY
# reason — network, auth, a registry hiccup — reads as a miss and the
# build runs. Only a genuine 200 skips one, so there is no path here
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
CHANNEL: ${{ steps.tag.outputs.channel }}
TAGS: ${{ steps.tag.outputs.tags }}
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity web "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# A tag-push builds a revision that main already published, so it
# must NOT claim the identity: image configs are not bit-reproducible
# (embedded timestamps), so re-pushing r-<rev> would point an
# immutable tag at fresh bytes — rule 145's exact prohibition. It
# publishes only its own :v… label and otherwise reuses.
if [ "$IS_TAG_PUSH" = "true" ]; then
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
else
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
fi
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
fi
- name: Download signed XPI from Forgejo release asset
# Fires on every trigger shape. dev and main each bundle the XPI their
# own sign-extension just published — that is the whole point of the
# channel work (milestone 271 step 6): the dev image carries the
# extension being developed, rather than requiring a merge to try it.
# Tag-push builds re-package the same source as the preceding main-push
# build but with an immutable version tag — they need the XPI too,
# otherwise the versioned image ships without the signed extension.
#
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
@@ -184,12 +537,21 @@ jobs:
# for up to 10min total) before giving up. Main-push's signing
# eventually wins and tag-push picks the release up on a later
# iteration.
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
# Gated on the reuse miss as well: if the image is already published it
# already contains its XPI, so this would download (and on a tag-push,
# poll up to 10 minutes for) a file nothing then reads.
if: steps.reuse.outputs.hit != 'true' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/'))
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
# Re-derived, not read from the repo: sign-extension published
# ext-<derived>, and the committed version has been inert since
# milestone 271 step 4. Both jobs run `packaging.sh version` over the
# same commit, so they agree by construction — and if they ever
# didn't, this download 404s and the build fails loudly instead of
# shipping a stale XPI.
VERSION=$(sh extension/scripts/packaging.sh version)
# Poll for the ext-<version> release. main-push's sign-extension
# step (AMO round-trip, 1-5min) needs to finish + upload before
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
@@ -238,54 +600,46 @@ jobs:
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
ls -la frontend/public/extension/
- name: Determine tag
id: tag
run: |
# Three trigger shapes:
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
# no `.N` per family release-posture rule).
# Publish ONLY the immutable version tag;
# don't touch :latest (the main-push build
# for the merge commit already did that).
# refs/heads/main → push to main: publish :main + :latest
# (floating) AND :c-<short_sha> (immutable
# per-commit rollback substrate, per family
# release-posture rule "Tags are milestones,
# not gates — commit-SHA images are the
# rollback unit"). Rollback to any commit
# becomes `docker pull …:c-<sha>` without a
# release ceremony.
# anything else → safety net; shouldn't fire given the `on:`
# config above. Tag :dev to surface the
# unexpected run in the registry.
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
fi
- name: Login to Forgejo registry
uses: docker/login-action@v3
with:
registry: git.fabledsword.com
username: ${{ github.actor }}
password: ${{ secrets.RELEASE_TOKEN }}
- name: Build and push web image
if: steps.reuse.outputs.hit != 'true'
uses: docker/build-push-action@v5
with:
context: .
file: Dockerfile
push: true
tags: ${{ steps.tag.outputs.tags }}
tags: ${{ steps.reuse.outputs.build_tags }}
# Only the web image carries a channel: it is the one that serves
# /api/extension/manifest. The ml and agent images have nothing to
# report it to.
build-args: |
FC_CHANNEL=${{ steps.tag.outputs.channel }}
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
# leaving :dev or :latest pointing at something older than the commit
# that was just pushed.
- name: Repoint the tags at the published image (reuse)
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
IDENTITY: ${{ steps.reuse.outputs.identity }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
build-ml:
runs-on: python-ci
@@ -293,6 +647,42 @@ jobs:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong,
# too-low value or finds nothing at all, and neither is a failure
# the build would otherwise notice.
fetch-depth: 0
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
# never be the reason an image does not ship.
#
# What it should say:
# * a push touching only agent/ moves the agent and leaves web and ml
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
set -u
A=ml
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
- name: Determine tag
id: tag
@@ -306,29 +696,138 @@ jobs:
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
# THIS artifact's shipped files last changed in. Day precision is
# deliberate — same-day work is not something worth pinning, so a
# second main build the same day replaces the first rather than
# accumulating a tag nobody would roll back to.
#
# Derived per artifact, so an image whose files did not change keeps
# the tag it already had: the agent reads 2026.7.17 today while web
# reads 2026.8.27 — and the reuse step below turns that into a
# skipped build rather than a rebuild of bytes that already exist.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
CALVER=$(sh scripts/artifacts.sh tag ml)
# Guarded, and computed only on this path. There is no `set -e` in
# this step, so a failed derivation would otherwise leave CALVER
# empty and publish the tag `fabledcurator-ml:` — an invalid
# name, from a green step. An empty pin must never reach the
# registry.
if [ -z "$CALVER" ]; then
echo "ERROR: could not derive a ml version tag" >&2
exit 1
fi
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-ml:${CALVER}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
echo "channel=dev" >> "$GITHUB_OUTPUT"
fi
# Shell step rather than docker/login-action — see build-web's note on
# the shared action-cache race (#3118).
- name: Login to Forgejo registry
uses: docker/login-action@v3
with:
registry: git.fabledsword.com
username: ${{ github.actor }}
password: ${{ secrets.RELEASE_TOKEN }}
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
#
# The failure direction is deliberate. An inspect that errors for ANY
# reason — network, auth, a registry hiccup — reads as a miss and the
# build runs. Only a genuine 200 skips one, so there is no path here
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator-ml:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
CHANNEL: ${{ steps.tag.outputs.channel }}
TAGS: ${{ steps.tag.outputs.tags }}
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity ml "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# A tag-push builds a revision that main already published, so it
# must NOT claim the identity: image configs are not bit-reproducible
# (embedded timestamps), so re-pushing r-<rev> would point an
# immutable tag at fresh bytes — rule 145's exact prohibition. It
# publishes only its own :v… label and otherwise reuses.
if [ "$IS_TAG_PUSH" = "true" ]; then
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
else
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
fi
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
fi
- name: Build and push ml image
if: steps.reuse.outputs.hit != 'true'
uses: docker/build-push-action@v5
with:
context: .
file: Dockerfile.ml
push: true
tags: ${{ steps.tag.outputs.tags }}
tags: ${{ steps.reuse.outputs.build_tags }}
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
# leaving :dev or :latest pointing at something older than the commit
# that was just pushed.
- name: Repoint the tags at the published image (reuse)
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
IDENTITY: ${{ steps.reuse.outputs.identity }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
# the GPU machine instead of building locally. Independent of web/ml (its own
@@ -339,31 +838,176 @@ jobs:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong,
# too-low value or finds nothing at all, and neither is a failure
# the build would otherwise notice.
fetch-depth: 0
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
# never be the reason an image does not ship.
#
# What it should say:
# * a push touching only agent/ moves the agent and leaves web and ml
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
set -u
A=agent
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
- name: Determine tag
id: tag
run: |
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
# THIS artifact's shipped files last changed in. Day precision is
# deliberate — same-day work is not something worth pinning, so a
# second main build the same day replaces the first rather than
# accumulating a tag nobody would roll back to.
#
# Derived per artifact, so an image whose files did not change keeps
# the tag it already had: the agent reads 2026.7.17 today while web
# reads 2026.8.27 — and the reuse step below turns that into a
# skipped build rather than a rebuild of bytes that already exist.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
CALVER=$(sh scripts/artifacts.sh tag agent)
# Guarded, and computed only on this path. There is no `set -e` in
# this step, so a failed derivation would otherwise leave CALVER
# empty and publish the tag `fabledcurator-agent:` — an invalid
# name, from a green step. An empty pin must never reach the
# registry.
if [ -z "$CALVER" ]; then
echo "ERROR: could not derive a agent version tag" >&2
exit 1
fi
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-agent:${CALVER}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
echo "channel=dev" >> "$GITHUB_OUTPUT"
fi
# Shell step rather than docker/login-action — see build-web's note on
# the shared action-cache race (#3118).
- name: Login to Forgejo registry
uses: docker/login-action@v3
with:
registry: git.fabledsword.com
username: ${{ github.actor }}
password: ${{ secrets.RELEASE_TOKEN }}
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
#
# The failure direction is deliberate. An inspect that errors for ANY
# reason — network, auth, a registry hiccup — reads as a miss and the
# build runs. Only a genuine 200 skips one, so there is no path here
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator-agent:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
CHANNEL: ${{ steps.tag.outputs.channel }}
TAGS: ${{ steps.tag.outputs.tags }}
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity agent "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# A tag-push builds a revision that main already published, so it
# must NOT claim the identity: image configs are not bit-reproducible
# (embedded timestamps), so re-pushing r-<rev> would point an
# immutable tag at fresh bytes — rule 145's exact prohibition. It
# publishes only its own :v… label and otherwise reuses.
if [ "$IS_TAG_PUSH" = "true" ]; then
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
else
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
fi
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
fi
- name: Build and push agent image
if: steps.reuse.outputs.hit != 'true'
uses: docker/build-push-action@v5
with:
context: agent
file: agent/Dockerfile
push: true
tags: ${{ steps.tag.outputs.tags }}
tags: ${{ steps.reuse.outputs.build_tags }}
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
# leaving :dev or :latest pointing at something older than the commit
# that was just pushed.
- name: Repoint the tags at the published image (reuse)
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
IDENTITY: ${{ steps.reuse.outputs.identity }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
+53 -99
View File
@@ -2,7 +2,7 @@ name: CI
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
# - extension-version: guards the extension publish path (see the job).
# - extension-version: the derived version resolves and MAJOR.MINOR agrees.
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
# - frontend-build: vitest unit + vite build.
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
@@ -42,18 +42,29 @@ jobs:
# catching syntax errors before the image build.
run: python -m compileall -q agent/fc_agent
# Guards the extension publish path, which has no self-correcting behavior.
# The extension version is DERIVED, not hand-maintained (milestone 271 step
# 4): build.yml computes it from the commit TIME of the newest packaged
# extension change and stamps it into manifest.json / package.json at build
# time. The guard that used to live here — "packaged files changed but nobody
# bumped the version" — was therefore checking a fact that had stopped
# existing. Worse than useless: it would have failed this lane on every real
# extension change, demanding a bump that decides nothing. Retired 2026-08-27
# rather than left running beside the new mechanism (rule 22).
#
# build.yml's sign-extension job keys its AMO-signing cache purely on the
# version string in extension/package.json: if an `ext-<version>` Forgejo
# release already carries an XPI, signing is SKIPPED and that old signed XPI
# is what build-web bakes into `:latest`. Nothing in that path inspects
# whether extension/ actually changed — so a forgotten version bump ships a
# stale extension on a fully green build, silently. (AMO can't help: it 409s
# on re-signing a version, which is exactly why the cache exists.)
# Two things are still worth asserting, and this is the only lane that can:
# the extension.yml suite runs on node:24-slim, which is exactly why
# version.spec.js sticks to packaging.sh's git-free subcommands.
# 1. the derivation actually resolves on this commit
# 2. MAJOR.MINOR agrees between the two files — the one part still hand-set,
# and packaging.sh reads it from manifest.json ALONE, so a divergence
# ships a version package.json disagrees with
#
# This job makes that case loud, on the dev push, instead of invisible at
# merge-to-main. It is pure git + text work — no deps, no services.
# Deliberately NOT checked here: that the derived value beats what has already
# been signed. That guard belongs in build.yml, where it compares against the
# real ext-* releases. Comparing against origin/main here would be wrong —
# dev legitimately derives a LOWER value whenever main is ahead on the
# extension, and a lane that fails for being behind is a lane people learn to
# ignore.
extension-version:
runs-on: python-ci
container:
@@ -61,101 +72,37 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Full history: the check diffs against the push's `before` SHA (or
# the PR base), which a depth-1 clone wouldn't contain.
# The derivation needs real history: a depth-1 clone sees one commit
# and produces a wrong, too-low value RATHER THAN FAILING. Checking
# that here is half the point of the lane.
fetch-depth: 0
- name: Extension version guard
env:
BEFORE: ${{ github.event.before }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
- name: Extension version derives cleanly
run: |
set -eu
# busybox sh on the act_runner — no bashisms (family rule).
ver() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/'; }
PKG=$(ver extension/package.json)
MAN=$(ver extension/manifest.json)
test -n "$PKG" || { echo "ERROR: no version found in extension/package.json"; exit 1; }
test -n "$MAN" || { echo "ERROR: no version found in extension/manifest.json"; exit 1; }
# (1) Unconditional: the two version strings must agree. `web-ext sign`
# reads manifest.json (package.json sits in --ignore-files and isn't
# even inside the XPI), so AMO signs MAN and Firefox installs MAN.
# build.yml keys its cache, release tag, XPI filename — and therefore
# the version /api/extension/manifest reports to the update prompt —
# on PKG. Divergence either hard-fails at AMO or ships a mislabelled
# XPI whose update prompt lies about what's installed.
VERSION=$(sh extension/scripts/packaging.sh version)
echo "derived: $VERSION"
# The shape AMO accepts, and the shape build.yml will stamp.
if ! echo "$VERSION" | grep -qE '^[0-9]+(\.[0-9]+)*$'; then
echo "ERROR: derived version '$VERSION' is not plain dotted-numeric."
echo "AMO would reject it, and build.yml stamps it verbatim."
exit 1
fi
mm() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([0-9]+\.[0-9]+).*/\1/'; }
MAN=$(mm extension/manifest.json)
PKG=$(mm extension/package.json)
test -n "$MAN" || { echo "ERROR: no parseable version in extension/manifest.json"; exit 1; }
test -n "$PKG" || { echo "ERROR: no parseable version in extension/package.json"; exit 1; }
if [ "$MAN" != "$PKG" ]; then
echo "ERROR: extension version mismatch."
echo " extension/manifest.json = $MAN <- what AMO signs / Firefox installs"
echo " extension/package.json = $PKG <- what CI caches, names, and reports"
echo "Set both to the same value."
echo "ERROR: MAJOR.MINOR disagrees between the two files."
echo " extension/manifest.json = $MAN <- packaging.sh reads MAJOR.MINOR from here"
echo " extension/package.json = $PKG"
echo "Only MAJOR.MINOR is hand-set. The patch component is derived from"
echo "commit time and overwritten at build time, so the committed patch"
echo "numbers are inert — but MAJOR.MINOR still ships. Set both the same."
exit 1
fi
# (2) If the SHIPPED extension changed, the version must have moved.
#
# Compare against MAIN, not against the previous push. The publish
# decision is made at merge-to-main against whatever ext-<version>
# already exists, so "differs from main" is the question that matters.
# Diffing against the previous dev push instead would demand a fresh
# bump on every iteration — push, tweak the extension again, and CI
# would insist on a second bump that buys nothing, inflating the
# version for no reason. On a main push there is no "main to compare
# to" yet, so fall back to that push's own before-SHA.
if [ "${GITHUB_REF##*/}" = "main" ]; then
BASE="${BEFORE:-}"
else
BASE=$(git rev-parse --verify -q origin/main 2>/dev/null || git rev-parse --verify -q main 2>/dev/null || echo "")
# PR base is the fallback when main isn't in the clone at all.
[ -n "$BASE" ] || BASE="${PR_BASE:-}"
fi
case "$BASE" in
''|0000000000000000000000000000000000000000)
echo "No usable base ref (no main in clone / first push) — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
;;
esac
if ! git cat-file -e "$BASE^{commit}" 2>/dev/null; then
echo "Base commit $BASE not in this clone — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
# Exclusions mirror --ignore-files in extension/package.json's web-ext
# scripts: these files are not packaged into the XPI, so touching them
# (e.g. Renovate bumping the web-ext devDep, or editing a spec)
# changes nothing shipped and must not demand a version bump.
# KEEP IN SYNC with --ignore-files — a file packaged into the XPI but
# excluded here is exactly the silent-stale-ship this job exists to
# prevent. test/version.spec.js pins the two lists' shared intent.
CHANGED=$(git diff --name-only "$BASE" HEAD -- extension/ \
':(exclude)extension/package.json' \
':(exclude)extension/package-lock.json' \
':(exclude)extension/README.md' \
':(exclude)extension/.gitignore' \
':(exclude)extension/vitest.config.js' \
':(exclude)extension/test/**')
if [ -z "$CHANGED" ]; then
echo "No packaged extension files changed since $BASE — nothing to guard."
echo "OK: extension version $PKG"
exit 0
fi
echo "Packaged extension files changed since $BASE:"
echo "$CHANGED" | sed 's/^/ /'
PKG_OLD=$(git show "$BASE:extension/package.json" 2>/dev/null | grep -E '"version"' | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
if [ -z "$PKG_OLD" ]; then
echo "Could not read the base version — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
if [ "$PKG_OLD" = "$PKG" ]; then
echo "ERROR: packaged extension files changed but the version is still $PKG."
echo "build.yml would find the existing ext-$PKG release, skip AMO signing,"
echo "and bake the OLD signed XPI into :latest — a green build shipping stale code."
echo "Bump the version in BOTH extension/package.json and extension/manifest.json."
exit 1
fi
echo "OK: extension version $PKG_OLD -> $PKG"
echo "OK: MAJOR.MINOR $MAN, derived version $VERSION"
backend-lint-and-test:
runs-on: python-ci
@@ -168,6 +115,13 @@ jobs:
SECRET_KEY: ci_unit_test_placeholder
steps:
- uses: actions/checkout@v4
with:
# Full history for tests/test_artifact_identity.py, which derives
# each artifact's revision to check the identity scheme. On a
# depth-1 clone that derivation either fails or returns the tip sha
# — so the lane would go green while asserting nothing, which is
# the one outcome worse than a red one.
fetch-depth: 0
# Cache step removed 2026-05-26: act_runner's cache backend has been
# broken on this homelab runner since 2026-05-15 (first as request-
+50 -3
View File
@@ -10,15 +10,20 @@ on:
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
# test/version.spec.js asserts ci.yml's extension-version guard never
# ignores a file web-ext actually packages, so a ci.yml-only edit can
# break this suite and must trigger it.
# test/version.spec.js asserts things ABOUT the other two workflows —
# that neither inlines the packaged-file set, and that build.yml derives
# the shipped version rather than reading it out of the repo. A
# workflow-only edit can therefore break this suite, so it has to trigger
# it. build.yml joined the list at milestone 271 step 5, when the spec
# started asserting against it.
- '.forgejo/workflows/ci.yml'
- '.forgejo/workflows/build.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
- '.forgejo/workflows/ci.yml'
- '.forgejo/workflows/build.yml'
workflow_dispatch:
jobs:
@@ -38,3 +43,45 @@ jobs:
# package version-consistency checks. No browser, no network.
- name: Unit tests
run: cd extension && npm run test:unit
# Everything else about packaging is asserted against our own declaration
# of what ships. This is the only check that asks web-ext what it ACTUALLY
# put in the archive. Until now that was an unverified assumption about
# glob semantics — and a fragile one: `test/**` reaches web-ext intact
# only because callers `set -f` first, so losing that quoting would
# silently start shipping dev files with no other signal.
- name: Verify XPI contents
run: |
set -eu
command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
cd extension
npm run build
ZIP=$(ls web-ext-artifacts/*.zip | head -1)
echo "=== packaged entries in $ZIP ==="
unzip -Z1 "$ZIP" | sort
echo "=== end ==="
ENTRIES=$(unzip -Z1 "$ZIP")
fail=0
# Must NOT ship: repo infrastructure with no business in a user's browser.
for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
if echo "$ENTRIES" | grep -q "^$pat"; then
echo "ERROR: '$pat' was packaged into the XPI but must not be"
fail=1
fi
done
# Must ship: if an exclusion pattern ever over-matches, the extension
# breaks at runtime rather than at build time, so assert presence too.
for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
if ! echo "$ENTRIES" | grep -q "^$req$"; then
echo "ERROR: '$req' is missing from the XPI"
fail=1
fi
done
for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
if ! echo "$ENTRIES" | grep -q "^$dir"; then
echo "ERROR: nothing from '$dir' was packaged"
fail=1
fi
done
[ "$fail" -eq 0 ] || exit 1
echo "XPI contents verified."
+17
View File
@@ -47,6 +47,23 @@ RUN chmod +x entrypoint.sh
COPY --from=frontend-builder /build/dist ./frontend/dist
# Which channel this image belongs to — `dev` or `main` (milestone 271 step 7).
# build.yml passes it; /api/extension/manifest reports it beside the version so
# an operator can tell which channel an install came from without the channel
# ever touching the version string.
#
# Empty by default, deliberately: a locally-built image then reports NO channel
# rather than claiming to be one, and the manifest omits the field entirely —
# indistinguishable from an image built before the field existed, which is
# exactly the shape every reader already has to handle.
#
# Declared LAST on purpose. An ARG/ENV invalidates every layer below it, and
# this is the one value that differs between the dev and main builds of
# identical source — put it any earlier and the two channels could never share
# a cached pip install.
ARG FC_CHANNEL=""
ENV FC_CHANNEL=${FC_CHANNEL}
EXPOSE 8080
ENTRYPOINT ["./entrypoint.sh"]
+35 -6
View File
@@ -6,7 +6,21 @@ Combines what was [ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo)
## Status
Pre-v1. Not yet functional.
In production. `main` is continuously deployed — every merge to `main` builds
and publishes `:latest` images, so whatever is on `main` is what is running.
Day-to-day work happens on `dev`, which publishes `:dev` images.
## What's in here
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
| Piece | Built from | Image | Role |
| --- | --- | --- | --- |
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. |
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
## Quick start
@@ -29,22 +43,37 @@ docker compose -f docker-compose.yml up -d
# (skips the override so containers pull registry images)
```
The GPU agent is deployed separately, on the machine with the card —
`agent/docker-compose.yml`, not this stack.
## Deployment posture
FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS.
## CI / Forgejo setup
The repo's workflows expect:
Three workflows: `ci.yml` (lint, extension-version check, backend unit tests,
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
content verification), and `build.yml` (sign + publish).
- **Runner label `python-ci`** — a Forgejo runner with Python 3.14, ruff, and Node 22 pre-installed. Both `ci.yml` and `build.yml` use this label. The runner image (`runner-base:python-ci`) is built from `CI-Runner/CI-python/` in the operator's workspace; `make push` from that directory builds and pushes a new image when toolchain pins change.
- **Repo secret `RELEASE_TOKEN`** — a Forgejo PAT with the following scopes:
**The toolchain each job runs in is its `container.image`, not its `runs-on`
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
then names the image it actually wants. `ci-requirements.md` is the current,
authoritative list of images and per-job installs — read that rather than a
copy here, so the two can't drift.
The repo expects one secret:
- **`RELEASE_TOKEN`** — a Forgejo PAT with:
- `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
- `write:release` — for future release-cutting workflows
- `write:issue` — for future issue-management automation
- `write:release` — for the `ext-<version>` releases that cache the signed XPI
- `write:issue` — for issue-management automation
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on
`main` only and is cached per version, since AMO rejects a re-signed version.
## License
Personal project; use at your own discretion.
+16
View File
@@ -459,6 +459,22 @@ async def trigger_prune_missing_files():
return _queued(async_result)
@admin_bp.route("/maintenance/reclaim-attachments", methods=["POST"])
async def trigger_reclaim_attachments():
"""Reclaim orphaned attachments (#3068). Body {"dry_run": bool}: dry_run
(the DEFAULT here) projects the orphan rows and unreferenced store blobs
without touching either; dry_run=false deletes the rows then unlinks every
blob no surviving row references. Maintenance queue; operator-triggered
only — never an unattended sweep, since the apply unlinks files. Returns the
Celery task id — poll /maintenance/task-result/<id> for the summary."""
from ..tasks.admin import reclaim_orphaned_attachments_task
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
async_result = reclaim_orphaned_attachments_task.delay(dry_run=dry_run)
return _queued(async_result)
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
async def trigger_dedup_videos():
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
+35 -2
View File
@@ -6,6 +6,8 @@ from __future__ import annotations
import asyncio
import hashlib
import hmac
import os
import re
from pathlib import Path
@@ -30,6 +32,12 @@ XPI_DIR = Path("/app/frontend/dist/extension")
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
# Which channel this image belongs to — "dev" or "main" — baked in at build
# time from the FC_CHANNEL build arg (milestone 271 step 7). Empty for a local
# build, or for any image predating the field. Tests override by monkeypatching
# this constant, same as XPI_DIR above.
FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
async def _ext_key_required(session) -> bool:
"""Unlike /api/credentials (which accepts the browser path with no
@@ -41,7 +49,15 @@ async def _ext_key_required(session) -> bool:
stored = (await session.execute(
select(AppSetting.value).where(AppSetting.key == "extension_api_key")
)).scalar_one_or_none()
return stored is not None and supplied == stored
if stored is None:
return False
# compare_digest, not `==`: the stored key is a shared secret, and a
# short-circuiting compare leaks its prefix through timing. Costs nothing
# here — it is not that this route is exposed (#3072). Compared as BYTES:
# compare_digest's str form rejects non-ASCII with TypeError, and this
# header is attacker-supplied, so a str compare would turn a junk key into
# a 500 instead of a 403.
return hmac.compare_digest(supplied.encode("utf-8"), stored.encode("utf-8"))
def _extract_version(xpi_name: str) -> str:
@@ -124,13 +140,30 @@ def _read_manifest_sync() -> dict | None:
return None
versioned.sort(key=lambda p: p.stat().st_mtime)
latest = versioned[-1]
return {
info = {
"installed": True,
"version": _extract_version(latest.name),
"xpi_url": f"/extension/{latest.name}",
"latest_url": "/extension/fabledcurator-latest.xpi",
"sha256": _sha256(latest),
}
# The channel goes BESIDE the version, never inside it. A `-dev` suffix is
# what silently disabled the dev channel in the sibling project this design
# comes from: the comparator returned nothing for a non-integer segment, so
# every dev version compared equal and "no update available" became
# indistinguishable from "I cannot read this version".
#
# Omitted rather than defaulted when unset. Absence already has a meaning
# every reader must handle — an image built before this field existed says
# exactly the same thing by not having the key — so a blank channel reuses
# that path instead of inventing a second "unknown" spelling.
#
# Reported verbatim, not validated against {"dev", "main"}: if an image
# declares something else, showing what it actually claims is more useful
# to whoever is debugging it than dropping the value on the floor.
if FC_CHANNEL:
info["channel"] = FC_CHANNEL
return info
@extension_bp.route("/manifest", methods=["GET"])
+2 -1
View File
@@ -27,7 +27,7 @@ from .patreon_seen_media import PatreonSeenMedia
from .pixiv_failed_media import PixivFailedMedia
from .pixiv_seen_media import PixivSeenMedia
from .post import Post
from .post_attachment import PostAttachment
from .post_attachment import PostAttachment, attachment_download_url
from .presentation_review import PresentationReview
from .series_chapter import SeriesChapter
from .series_page import SeriesPage
@@ -58,6 +58,7 @@ __all__ = [
"SubscribeStarSeenMedia",
"Post",
"PostAttachment",
"attachment_download_url",
"PresentationReview",
"SeriesChapter",
"SeriesPage",
+12
View File
@@ -65,3 +65,15 @@ class PostAttachment(Base):
captured_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
def attachment_download_url(attachment_id: int) -> str:
"""The path that streams this attachment's bytes.
Both serializers that expose an attachment to the frontend
(`provenance_service`, `post_feed_service`) built this literal themselves,
so changing the route in `api/attachments.py` meant two edits and only one
would be remembered (#3072). `test_attachment_download_url` pins it against
the app's registered rule, so the drift is caught rather than trusted to.
"""
return f"/api/attachments/{attachment_id}/download"
+256 -6
View File
@@ -48,6 +48,47 @@ log = logging.getLogger(__name__)
_VIDEO_DURATION_UNKNOWN = -1.0
# -- artist-cascade predicates (rule 93: ONE definition, preview + apply) ---
# project_artist_cascade (preview) and delete_artist_cascade (apply) both build
# their queries from these. The preview used to re-derive its own — which is how
# it came to count images and stay silent about posts and attachments while the
# apply destroyed both. Same failure shape as the 2026-06-08 fandom-tag
# deletion, where a re-implemented delete predicate diverged from the preview's.
# Returned as condition LISTS spread into `.where(*conds)`, matching
# _unused_tag_conditions / _bare_post_conditions below.
def _artist_images_conditions(artist_id: int) -> list:
"""Images the cascade deletes (rows AND their on-disk files)."""
return [ImageRecord.artist_id == artist_id]
def _artist_posts_conditions(artist_id: int) -> list:
"""Posts the cascade destroys. The apply never names these — post.artist_id
is ondelete=CASCADE, so Postgres takes them when the artist row goes — which
is exactly why the preview has to name them: an artist whose posts are
body-only (no images) otherwise previews as `images: 0` and reads as an
empty artist, while every captured body/description/external-link set is
destroyed."""
return [Post.artist_id == artist_id]
def _artist_attachments_conditions(artist_id: int) -> list:
"""Attachments the cascade deletes. Matched by artist_id OR by the owning
post's artist: artist_id is nullable (_capture_attachment leaves it NULL
when no artist resolved), so neither arm alone covers every row. The
sha-addressed blobs are NOT unlinked (one blob backs many rows) — these are
row counts, and the bytes are not part of this operation's footprint."""
return [
or_(
PostAttachment.artist_id == artist_id,
PostAttachment.post_id.in_(
select(Post.id).where(*_artist_posts_conditions(artist_id))
),
)
]
def project_artist_cascade(session: Session, *, slug: str) -> dict:
"""Read-only projection of what delete_artist_cascade would touch.
@@ -56,12 +97,17 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
"artist": {"id": int, "name": str, "slug": str},
"projected": {
"images": int,
"posts": int, # hard-deleted by the post.artist_id CASCADE
"attachments": int, # rows deleted; the sha-addressed blobs stay
"sources": int,
"thumbs": int, # images with a thumbnail_path set
"import_tasks": int, # ImportTask rows referencing the artist's images
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
},
}
Every count is built from the shared `_artist_*_conditions` predicates the
apply uses, so the two halves cannot drift (rule 93).
Raises LookupError if slug not found. No mutations.
"""
from ..models.import_task import ImportTask
@@ -73,36 +119,49 @@ def project_artist_cascade(session: Session, *, slug: str) -> dict:
if artist is None:
raise LookupError(f"artist slug not found: {slug!r}")
images_conds = _artist_images_conditions(artist.id)
images_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
select(func.count(ImageRecord.id)).where(*images_conds)
).scalar_one()
posts_count = session.execute(
select(func.count(Post.id))
.where(*_artist_posts_conditions(artist.id))
).scalar_one()
attachments_count = session.execute(
select(func.count(PostAttachment.id))
.where(*_artist_attachments_conditions(artist.id))
).scalar_one()
# Sources have no shared predicate: the apply never queries them either, it
# gets them from the Artist.sources ORM cascade. Counted directly here.
sources_count = session.execute(
select(func.count(Source.id))
.where(Source.artist_id == artist.id)
).scalar_one()
thumbs_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
.where(*images_conds)
.where(ImageRecord.thumbnail_path.is_not(None))
).scalar_one()
import_tasks_count = session.execute(
select(func.count(ImportTask.id))
.where(
ImportTask.result_image_id.in_(
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
select(ImageRecord.id).where(*images_conds)
)
)
).scalar_one()
bytes_on_disk = session.execute(
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
.where(ImageRecord.artist_id == artist.id)
.where(*images_conds)
).scalar_one()
return {
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"projected": {
"images": images_count,
"posts": posts_count,
"attachments": attachments_count,
"sources": sources_count,
"thumbs": thumbs_count,
"import_tasks": import_tasks_count,
@@ -277,6 +336,10 @@ def delete_artist_cascade(
series_page / tag_suggestion_rejection from ImageRecord delete,
and source / post / download_event / etc. from Artist delete
(via Artist.sources cascade="all, delete-orphan").
The artist's post_attachment rows are cleared EXPLICITLY before the
artist row goes — see the comment at that step; leaving them to the
cascade aborts the whole delete on a unique violation.
"""
artist = session.get(Artist, artist_id)
if artist is None:
@@ -287,11 +350,22 @@ def delete_artist_cascade(
"files_deleted": 0,
"thumbs_deleted": 0,
"import_tasks_nulled": 0,
"posts_deleted": 0,
"attachments_deleted": 0,
"files_failed": 0,
},
}
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
# Counted BEFORE the delete: Postgres takes these via the post.artist_id
# CASCADE when the artist row goes, so afterwards there is nothing left to
# count. Reported so the summary can be checked against the preview's
# `posts` — the parity rule 93 asks for is only testable if both halves
# actually state the number.
posts_deleted = session.execute(
select(func.count(Post.id)).where(*_artist_posts_conditions(artist.id))
).scalar_one()
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
@@ -300,7 +374,7 @@ def delete_artist_cascade(
while True:
rows = session.execute(
select(ImageRecord)
.where(ImageRecord.artist_id == artist.id)
.where(*_artist_images_conditions(artist.id))
.limit(500)
).scalars().all()
if not rows:
@@ -323,6 +397,28 @@ def delete_artist_cascade(
# source_path_prefix matching that's out of scope here.
import_tasks_nulled = 0
# Clear the artist's attachments BEFORE the artist row, or the delete below
# aborts. Deleting an artist CASCADEs to Post (post.artist_id is
# ondelete=CASCADE), which SET NULLs post_attachment.post_id — and
# `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE
# WHERE post_id IS NULL, so any two of this artist's attachments sharing a
# sha collapse onto one another and raise. That is an ORDINARY shape, not a
# corrupt one: _capture_attachment deliberately writes one row per post over
# a single sha-addressed blob (a creator who attaches the same pdf to two
# posts has two rows), and a pre-existing filesystem-import row with the same
# sha and a NULL post_id collides on its own. Migration 0043 reasoned only
# about upgrade-time safety and never about this later SET NULL.
# _repoint_post_links guards the identical collision class in the reconcile
# path; this is its artist-cascade counterpart.
#
# Which rows count as the artist's — and why the blobs are left on disk —
# is _artist_attachments_conditions, shared with the preview.
attachments_deleted = session.execute(
delete(PostAttachment)
.where(*_artist_attachments_conditions(artist.id))
).rowcount or 0
session.commit()
session.delete(artist)
session.commit()
@@ -333,6 +429,8 @@ def delete_artist_cascade(
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_nulled": import_tasks_nulled,
"posts_deleted": posts_deleted,
"attachments_deleted": attachments_deleted,
"files_failed": files_failed,
},
}
@@ -1494,3 +1592,155 @@ def purge_gated_previews(
"ledger_cleared": ledger_cleared,
"posts_deleted": posts_deleted,
}
# -- orphaned attachment reclamation ---------------------------------------
# PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or
# artist leaves the row behind rather than taking it. Nothing ever pruned those
# rows, and nothing has ever unlinked a file under the attachment store — so
# both rows and bytes accumulated permanently and were invisible to every
# existing diagnostic.
#
# Why this is a DISK->DB reconciliation rather than a row sweep: the store is
# sha-addressed and idempotent (attachment_store.store), so ONE blob backs MANY
# rows. Deleting a row therefore does not free its blob, and — since the artist
# cascade now deletes its attachment rows outright — a freed blob has no DB
# pointer left to find it by. Walking the store and asking "does any row still
# reference this sha?" catches orphans from every cause, including ones no
# future delete path will think to report.
# A blob is written by attachment_store.store BEFORE its row is inserted and
# committed, so a just-stored file legitimately has no referencing row for a
# moment. Same guard, same reasoning as ORPHAN_TEMP_MIN_AGE_HOURS in
# tasks/maintenance.py: never judge a file younger than this.
_ATTACHMENT_ORPHAN_MIN_AGE_HOURS = 6
# Wall-clock budget for the store walk (rule 89). A library with a large
# attachment store shouldn't be able to run this past its soft time limit; on
# exhaustion it reports partial=True and the operator re-runs to finish.
_ATTACHMENT_RECLAIM_BUDGET_SECONDS = 900
# The store names files `<sha256><ext>`. Parse the sha as the first 64 chars
# rather than via Path.stem: store() takes the extension straight from the
# source filename, and a URL-encoded basename yields a multi-dot "suffix"
# (see [[reference_url_encoded_basename_suffix]]) that would make stem eat part
# of the sha. Validating the 64 chars as hex also skips anything else in the
# tree that isn't a stored blob.
_SHA256_HEX_LEN = 64
def _orphan_attachment_conditions() -> list:
"""PostAttachment rows belonging to nothing: both FKs nulled by a deleted
post AND a deleted artist. A row with post_id NULL but an artist_id is the
deliberate filesystem-import case (importer._capture_attachment writes it
that way) and is NOT an orphan — it is still attributed."""
return [
PostAttachment.post_id.is_(None),
PostAttachment.artist_id.is_(None),
]
def _is_sha_named(name: str) -> bool:
"""True when `name` starts with a 64-char lowercase-hex sha256."""
if len(name) < _SHA256_HEX_LEN:
return False
head = name[:_SHA256_HEX_LEN]
return all(c in "0123456789abcdef" for c in head)
def reclaim_orphaned_attachments(
session: Session, *, images_root: Path, dry_run: bool = False,
) -> dict:
"""Prune unattributed PostAttachment rows, then unlink store blobs that no
surviving row references.
Returns (same discovery keys either way, so the UI renders one shape):
{"rows": int, # orphan rows found / deleted
"files": int, # unreferenced blobs found / unlinked
"bytes": int, # their total size
"scanned": int, # blobs examined
"skipped_recent": int, # blobs under the min-age guard
"files_failed": int, # unlink raised (apply only)
"partial": bool} # walk hit the time budget
dry_run computes exactly what the apply would do and mutates nothing — the
surviving-sha set is derived by NEGATING the same orphan predicate the
delete uses, so the preview cannot disagree with the apply (rule 93).
"""
started = time.monotonic()
orphan_conds = _orphan_attachment_conditions()
if dry_run:
rows = session.execute(
select(func.count(PostAttachment.id)).where(*orphan_conds)
).scalar_one()
else:
rows = session.execute(
delete(PostAttachment).where(*orphan_conds)
).rowcount or 0
session.commit()
# Shas that still have a home. In the apply path the orphan rows are already
# gone, so `NOT orphan` is redundant but harmless; in the dry-run path it is
# what makes the projection honest about blobs the delete would free. One
# predicate, one query, both modes.
surviving_shas = set(session.execute(
select(PostAttachment.sha256).where(~and_(*orphan_conds)).distinct()
).scalars())
root = Path(images_root) / "attachments"
cutoff = (
datetime.now(UTC).timestamp()
- _ATTACHMENT_ORPHAN_MIN_AGE_HOURS * 3600
)
files = 0
freed_bytes = 0
scanned = 0
skipped_recent = 0
files_failed = 0
partial = False
if root.is_dir():
for path in root.rglob("*"):
if time.monotonic() - started >= _ATTACHMENT_RECLAIM_BUDGET_SECONDS:
partial = True
break
# .partial staging files belong to cleanup_orphaned_temp_files —
# leave them alone rather than racing an in-flight store().
if path.suffix in (".part", ".partial") or not path.is_file():
continue
if not _is_sha_named(path.name):
continue
scanned += 1
sha = path.name[:_SHA256_HEX_LEN]
if sha in surviving_shas:
continue
try:
st = path.stat()
if st.st_mtime >= cutoff:
skipped_recent += 1
continue
size = st.st_size
if not dry_run:
path.unlink()
files += 1
freed_bytes += size
except OSError as exc:
files_failed += 1
log.warning("reclaim_orphaned_attachments: %s: %s", path, exc)
if not dry_run and (rows or files):
log.info(
"attachment reclaim: %d orphan row(s) deleted, %d blob(s) unlinked "
"(%d bytes), %d failed, partial=%s",
rows, files, freed_bytes, files_failed, partial,
)
return {
"rows": rows,
"files": files,
"bytes": freed_bytes,
"scanned": scanned,
"skipped_recent": skipped_recent,
"files_failed": files_failed,
"partial": partial,
}
+1 -1
View File
@@ -181,7 +181,7 @@ def _augment_cookies(platform: str, netscape: str) -> str:
"""Delegate to the platform's `augment_cookies` hook if one is
registered (subscribestar, hentaifoundry, etc. — see
`services/platforms/<name>.py`). No-op when the platform doesn't
register a hook (Patreon, DeviantArt). Centralizing the
register a hook (Patreon, Discord). Centralizing the
quirks-per-platform in the platforms package means adding a new
platform's cookie quirks doesn't require touching this file."""
info = PLATFORMS.get(platform)
+2 -3
View File
@@ -31,9 +31,8 @@ from .pixiv_ingester import PixivIngester
from .subscribestar_ingester import SubscribeStarIngester
# Platforms whose download + verify go through the native ingester rather than
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord,
# deviantart — the latter slated for retirement, not migration) until they
# migrate too.
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
# they migrate too.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "pixiv"})
# Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
@@ -55,12 +55,6 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
re.IGNORECASE,
)),
("deviantart", re.compile(
r"^https?://(?:www\.)?deviantart\.com/"
r"(?!home$|watch\b|tag\b|browse\b)"
r"(?P<slug>[^/?#]+)/?$",
re.IGNORECASE,
)),
("pixiv", re.compile(
r"^https?://(?:www\.)?pixiv\.net/(?:en/)?users/(?P<slug>\d+)",
re.IGNORECASE,
+3 -11
View File
@@ -299,8 +299,9 @@ class GalleryDLService:
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = {
# subscribestar removed — native-ingester platform now (#71); pixiv
# removed likewise (#129). The remaining entries are the gallery-dl
# platforms not yet migrated.
# removed likewise (#129); deviantart removed at #3069 as a dropped
# platform, not a migrated one. The remaining entries are the
# gallery-dl platforms not yet migrated.
"hentaifoundry": {
"content_types": ["all"],
"directory": [],
@@ -316,15 +317,6 @@ class GalleryDLService:
"reactions": False,
"threads": True,
},
"deviantart": {
"content_types": ["all"],
"directory": [],
"filename": "{index:>03}_{title[:50]}.{extension}",
"flat": True,
"original": True,
"mature": True,
"metadata": True,
},
}
def __init__(
+51
View File
@@ -0,0 +1,51 @@
"""Bulk, idempotent writes to the ``image_tag`` association table.
Three writers attach tags to images in bulk: the WIP-title backfill
(`wip_title.apply_wip_image_tags`), the concept-head auto-apply sweep and the
system-tag auto-apply sweep (both in `ml/heads.py`). The two sweeps used to
issue ONE INSERT PER ROW from inside their per-image loop — fine in steady
state, but a first pass over a back-catalogue is tens of thousands of
individual round-trips (#3072). All three share this one chunked multi-row
insert now.
Sync only: every caller runs on a sync ``Session`` (the Celery task path). No
async service writes image_tag in bulk, so there is no async sibling to keep in
step — unlike `db_helpers.get_or_create`, which does have one.
"""
from __future__ import annotations
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from ..models.tag import image_tag
# 5000 rows x 3 bound params = 15000, comfortably inside Postgres' 65535-param
# ceiling for a single statement. Raising this past ~21000 rows would exceed it.
INSERT_CHUNK = 5000
def insert_image_tags(
session: Session, rows: list[dict], *, chunk: int = INSERT_CHUNK
) -> None:
"""Attach ``rows`` to their images, skipping any tag already on one.
Each row is ``{"image_record_id": int, "tag_id": int, "source": str}``.
Does NOT commit — the caller owns the transaction.
ON CONFLICT DO NOTHING against the (image_record_id, tag_id) primary key,
so an existing tag keeps its ORIGINAL ``source``: re-running a sweep can
never re-stamp a tag the operator applied by hand as machine-applied.
Returns nothing on purpose. psycopg reports ``rowcount`` -1 for a multi-row
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so a count
taken from the statement would be a lie rather than an approximation.
Callers that need an accurate count derive it themselves — see
`wip_title.apply_wip_image_tags`' pre-SELECT, and the sweeps' `skip` sets.
"""
for start in range(0, len(rows), chunk):
session.execute(
pg_insert(image_tag)
.values(rows[start:start + chunk])
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
)
+20 -13
View File
@@ -41,6 +41,7 @@ from ...models import (
TagSuggestionRejection,
)
from ...models.tag import CHROME_SYSTEM_TAGS, PROCESS_SYSTEM_TAGS, image_tag
from ..image_tag_apply import insert_image_tags
from .training_data import (
_AUTO_SOURCES,
_applied_or_rejected,
@@ -757,6 +758,10 @@ def auto_apply_sweep(
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = _sigmoid(Xn @ W.T + b, np) # (N, H)
scanned += len(cids)
# Collected across every head, then written as ONE insert below. Was an
# insert per applied tag from inside this loop, which on a first sweep
# over a back-catalogue is tens of thousands of round-trips (#3072).
pending: list[dict] = []
for h in range(len(rows)):
tid = tag_ids[h]
for idx in np.where(probs[:, h] >= thr[h])[0]:
@@ -766,12 +771,12 @@ def auto_apply_sweep(
skip[tid].add(iid)
applied[h] += 1
if not dry_run:
session.execute(
pg_insert(image_tag)
.values(image_record_id=iid, tag_id=tid, source="head_auto")
.on_conflict_do_nothing()
)
pending.append({
"image_record_id": iid, "tag_id": tid,
"source": "head_auto",
})
if not dry_run:
insert_image_tags(session, pending)
session.commit()
run.last_progress_at = datetime.now(UTC)
session.commit()
@@ -913,6 +918,11 @@ def system_tag_auto_apply_sweep(
if Wc is not None:
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np) # (N,), (N,)
scanned += len(cids)
# Same batching as auto_apply_sweep (#3072): collect the chunk's rows
# and write them once, below. The PresentationReview rows stay per-row —
# they FK to image_record/tag, not to image_tag, so writing the tags
# after them is safe, and a flagged conflict is rare by construction.
pending: list[dict] = []
for p in range(len(pres)):
tid = pres_tag_ids[p]
for idx in np.where(probs[:, p] >= thr)[0]:
@@ -922,14 +932,10 @@ def system_tag_auto_apply_sweep(
skip[tid].add(iid)
applied[p] += 1
if not dry_run:
session.execute(
pg_insert(image_tag)
.values(
image_record_id=iid, tag_id=tid,
source=source,
)
.on_conflict_do_nothing()
)
pending.append({
"image_record_id": iid, "tag_id": tid,
"source": source,
})
# Guard 2: also looks like real content → still apply, but flag it
# for the review strip instead of silently marking (chrome hides,
# process stays visible — either way the operator gets a heads-up).
@@ -944,6 +950,7 @@ def system_tag_auto_apply_sweep(
mode=mode,
)
if not dry_run:
insert_image_tags(session, pending)
session.commit()
concepts = [
+3 -4
View File
@@ -8,9 +8,10 @@ PLATFORMS below. Sidecar parsing, cookie materialization, and
Lifted from GallerySubscriber's
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
and ~/.../extension/lib/platforms.js. Five platforms; auth_type and
URL patterns match GS exactly so the existing browser extension
hits FC unmodified.
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
FC downloaders are art-dedicated services only.
"""
from .base import (
@@ -18,7 +19,6 @@ from .base import (
DEFAULT_EXTERNAL_POST_ID_KEYS,
PlatformInfo,
)
from .deviantart import INFO as _DEVIANTART
from .discord import INFO as _DISCORD
from .hentaifoundry import INFO as _HENTAIFOUNDRY
from .patreon import INFO as _PATREON
@@ -33,7 +33,6 @@ PLATFORMS: dict[str, PlatformInfo] = {
_HENTAIFOUNDRY,
_DISCORD,
_PIXIV,
_DEVIANTART,
)
}
+1 -1
View File
@@ -63,7 +63,7 @@ class PlatformInfo:
# Synthesize a post permalink from sidecar data. Required when
# gallery-dl's `url` field is the file/CDN URL rather than the post
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
# `url` field (patreon, deviantart).
# `url` field (patreon).
derive_post_url: Callable[[dict], str | None] | None = None
# Post-process the materialized cookies.txt for gallery-dl. Used by
@@ -1,23 +0,0 @@
"""DeviantArt — no exercised quirks yet.
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
audit, so we don't know yet whether DA's gallery-dl sidecars are
well-behaved or have their own quirks. When DA gets exercised for the
first time, add `derive_post_url` / `augment_cookies` here as needed.
"""
from .base import GD_DEFAULTS, PlatformInfo
INFO = PlatformInfo(
key="deviantart",
name="DeviantArt",
description="Download artwork from DeviantArt artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?deviantart\.com/",
url_examples=[
"https://www.deviantart.com/example-artist",
"https://www.deviantart.com/example-artist/gallery",
],
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
)
+2 -1
View File
@@ -24,6 +24,7 @@ from ..models import (
Post,
PostAttachment,
Source,
attachment_download_url,
)
from ..utils.html_sanitize import (
extract_img_srcs,
@@ -360,7 +361,7 @@ class PostFeedService:
"ext": att.ext,
"mime": att.mime,
"size_bytes": att.size_bytes,
"download_url": f"/api/attachments/{att.id}/download",
"download_url": attachment_download_url(att.id),
})
return out
+2 -1
View File
@@ -16,6 +16,7 @@ from ..models import (
Post,
PostAttachment,
Source,
attachment_download_url,
)
from ..utils.html_sanitize import sanitize_post_html
@@ -53,7 +54,7 @@ def _attachment_dict(a: PostAttachment) -> dict:
"original_filename": a.original_filename,
"size_bytes": a.size_bytes,
"ext": a.ext,
"download_url": f"/api/attachments/{a.id}/download",
"download_url": attachment_download_url(a.id),
}
+5 -9
View File
@@ -20,10 +20,10 @@ family gains one member.
import re
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from ..models.tag import WIP_SYSTEM_TAG, Tag, image_tag
from .image_tag_apply import insert_image_tags
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other
# apply sources so provenance stays legible and a future undo can target only these.
@@ -113,13 +113,9 @@ def apply_wip_image_tags(
to_insert = [iid for iid in chunk if iid not in already]
if not to_insert:
continue
session.execute(
pg_insert(image_tag)
.values([
{"image_record_id": iid, "tag_id": tag_id, "source": source}
for iid in to_insert
])
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
)
insert_image_tags(session, [
{"image_record_id": iid, "tag_id": tag_id, "source": source}
for iid in to_insert
])
inserted += len(to_insert)
return inserted
+28
View File
@@ -409,3 +409,31 @@ def rescan_series_suggestions_task(self, after_post_id: int = 0) -> dict:
)
rescan_series_suggestions_task.delay(summary["resume_after_id"])
return summary
@celery.task(
name="backend.app.tasks.admin.reclaim_orphaned_attachments_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
# The service stops walking at its own 900s budget and reports partial, so
# these limits are the backstop for a wedged filesystem (NFS stall), not the
# expected exit. Comfortably above the budget so a normal run always returns
# its summary rather than being killed mid-walk.
soft_time_limit=1200, time_limit=1500, # 20 min / 25 min
)
def reclaim_orphaned_attachments_task(self, dry_run: bool = True) -> dict:
"""Reclaim unattributed PostAttachment rows and the store blobs nothing
references any more (#3068). dry_run (the default) returns the projection
without touching rows or files; apply deletes the orphan rows, then unlinks
every blob no surviving row references.
Defaults to the SAFE preview — unlike the other tasks here, whose apply is
reversible-ish or scoped; this one deletes files. Operator-triggered only,
never on a beat: an unattended sweep that unlinks blobs is not something to
run without someone reading the projection first."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.reclaim_orphaned_attachments(
session, images_root=IMAGES_ROOT, dry_run=dry_run,
)
+6
View File
@@ -173,6 +173,12 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
# task-name override beats the queue threshold whatever queue the row records
# (it recorded 'default' before the celery_signals fix → download). 65 = 60+5.
"backend.app.tasks.external.fetch_external_link": 65,
# Attachment reclaim walks the whole sha-addressed store; the service caps
# itself at a 900s budget and reports partial, but the task's hard limit is
# 25 min for a wedged filesystem (NFS stall). Same phantom-flag class as the
# external-fetch entry above — without an override a healthy in-flight walk
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
}
+39 -4
View File
@@ -27,6 +27,10 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
- `pip install -r requirements.txt pytest pytest-asyncio` — in `backend-lint-and-test` and `integration` jobs
- `npm install --no-audit --no-fund` — in `frontend-build` job
- `npm install --no-audit --no-fund` — in `extension.yml`'s `lint` job (web-ext + vitest)
- `unzip` — in `extension.yml`'s "Verify XPI contents" step, installed via apt
only when absent (`node:24-bookworm-slim` may or may not carry it). Debian
package, ~2s. Not worth baking into a shared image for a single consumer, per
`docs/process.md`'s ">1 project" rule.
## Notes
@@ -49,7 +53,38 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
classic script (`test/helpers/loadLib.js`) rather than adding `module.exports`
shims to production code — the libs ship as `background.scripts`, not ES
modules, so the specs exercise exactly the bytes packaged into the XPI.
- Extension test files are excluded from the XPI via `--ignore-files` in
`extension/package.json`, and the same paths are excluded from `ci.yml`'s
`extension-version` guard. Those two lists must agree — `test/version.spec.js`
asserts the guard never ignores a file web-ext actually packages.
- **`extension/scripts/packaging.sh` is the single definition of what ships
inside the XPI.** Two consumers read from it rather than keeping their own
copy: web-ext's `--ignore-files` (`extension/package.json`), and the `git log`
pathspec inside the script's own version derivation. It was three until
2026-08-27 — `ci.yml`'s `extension-version` guard held the third and went when
the manual bump it guarded did (milestone 271 step 5). Hand-kept copies of
that one fact is what allowed issue #2397, so `extension/test/version.spec.js`
asserts no workflow has reintroduced a literal `:(exclude)extension/…`.
- **The shipped extension version is derived, not committed.** It is the commit
TIME of the newest packaged-extension change (minutes since 2020-01-01, per
family rule 149 — never a commit count, which orders by branch rather than by
recency). `build.yml`'s `sign-extension` computes it and stamps it into
`extension/manifest.json` + `package.json` in the working tree before signing;
the stamp is never committed. Treat the version in the repo as a base: only
its MAJOR.MINOR is read, and its patch component is inert.
- Every job that calls `packaging.sh version` checks out with `fetch-depth: 0`
`build.yml`'s `sign-extension` and `build-web`, and `ci.yml`'s
`extension-version`. A depth-1 clone sees one commit and derives a wrong,
too-low value **rather than failing**, so the full-history checkout is
load-bearing rather than incidental.
- **`FC_CHANNEL` is a build arg, not a runtime setting.** `build.yml` passes
`dev` / `main` to the web image only (the ml and agent images have nothing to
report it to), and `/api/extension/manifest` reports it beside the version so
an install can be traced to a channel. It is declared LAST in the Dockerfile
on purpose: an ARG invalidates every layer below it, and this is the one value
that differs between the dev and main builds of identical source, so placing
it earlier would stop the two channels ever sharing a cached `pip install`.
Empty by default — a local build then reports no channel at all rather than
claiming one.
- Callers MUST `set -f` before substituting the script's output. Without it the
shell expands `test/**` against the working tree and silently narrows the
pattern to whatever files exist at that moment — a failure that looks like
nothing until dev files start appearing in the XPI. `test/version.spec.js`
asserts every `--ignore-files` consumer sets it, and that no consumer has
quietly reinstated a hardcoded list.
+62 -9
View File
@@ -1,13 +1,14 @@
# FabledCurator Firefox Extension
Self-hosted Firefox extension that pushes session cookies from supported
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv,
DeviantArt) into FabledCurator, and lets you add a creator as a Source
from their page in one click.
platforms (Patreon, SubscribeStar, Hentai-Foundry, Discord, Pixiv)
into FabledCurator, and lets you add a creator as a Source from their
page in one click.
## Install (operator)
The signed XPI is bundled into the FC Docker image. Open FC →
The signed XPI is bundled into the FC Docker image`:dev` and
`:latest` each carry their own channel's build. Open FC →
Settings → Maintenance → Browser extension → click "Install Firefox
extension". Firefox shows its native install prompt. After installing,
open the extension's options page (about:addons → FabledCurator →
@@ -20,6 +21,7 @@ same card.
cd extension/
npm install --no-save # web-ext only
npm run lint # web-ext lint
npm run test:unit # vitest — lib/ logic + packaging/version checks
npm run start # launches Firefox with extension loaded
npm run build # unsigned XPI in web-ext-artifacts/
```
@@ -36,10 +38,61 @@ npm run build # unsigned XPI in web-ext-artifacts/
- [ ] Subscriptions list: popup → "Sources" tab → list renders
- [ ] Check now: click play icon on source row → no error toast
## Versioning — don't hand-edit the patch number
The shipped version is **derived**, not committed. `scripts/packaging.sh
version` returns `MAJOR.MINOR` from `manifest.json` plus a patch component
that is the commit *time* of the newest change to a packaged extension file,
in minutes since 2020-01-01. `build.yml` computes it and stamps it into both
`manifest.json` and `package.json` at build time. The stamp is never
committed — the commit carrying it would itself be a change to the extension,
which would move the version again.
So:
- **Editing the patch number does nothing.** It is overwritten before web-ext
ever reads it. There is no bump to make, and none to forget.
- **MAJOR.MINOR is still yours.** It carries the deliberate meaning, it is read
from `manifest.json` alone, and CI fails the `extension-version` lane if the
two files disagree on it.
- `npm run build` locally produces an XPI labelled with the *committed*
version, since nothing stamped it. Fine for loading into a test profile; not
what ships.
Why commit time and not a commit count: a count is per-branch, so `dev` and
`main` count different histories of the same code and their versions end up
ordered by which branch accumulated more commits rather than by which is newer.
Commit time gives both branches the same number for the same source — which is
exactly what lets one AMO signature serve both channels (family rule 149, FC
issue #3092).
## Channels
`dev` and `main` each build and sign their own extension, and an install is
tied to whichever FC instance it points at — Firefox's static `update_url`
cannot apply here, since every FC install is a different host, so the extension
asks its configured backend. **The channel therefore IS the instance.**
Switching channel means repointing the FC URL in options and reinstalling from
that host; there is no separate channel setting, and adding one would
contradict each server build shipping its own extension.
The channel is reported *beside* the version, never inside it:
`/api/extension/manifest` answers `{"version": "...", "channel": "dev"}`. It is
optional — an instance that declares none simply omits the key, and the popup,
the toolbar tooltip and the Settings card all read exactly as they did before
the field existed. Do not be tempted to make it a `-dev` version suffix: the
comparator parses each dotted segment with `parseInt`, so a suffixed segment
reads as 0 and every dev build compares equal to every other, collapsing "no
update available" and "I cannot read this version" into one answer.
## Release
Bump `manifest.json` + `package.json` SemVer (both files) and commit
under `extension/**`. The `.forgejo/workflows/extension.yml` workflow
runs `web-ext sign` on main, commits the signed XPI to
`frontend/public/extension/`, and the next FC server build bundles it
into the Docker image.
Nothing to do by hand. Push to `dev`: `build.yml` signs the extension if this
change moved the version, caches the signed XPI as a Forgejo `ext-<version>`
release, and bundles it into `fabledcurator:dev`. Merging to `main` derives the
same version, hits that cache, and bundles the byte-identical XPI into
`:latest` with no second AMO call.
AMO refuses to re-sign a version it has already issued, so signing is one-shot
per version — which is why the cache exists and why the version must never move
backwards.
+25 -2
View File
@@ -37,7 +37,16 @@ ensureInitialized().catch(e => console.error('init failed:', e));
// configured backend for the latest published version and nudge the operator to
// reinstall the freshly-signed XPI — surfaced as a popup banner (on demand) and
// a toolbar badge (daily). /api/extension/manifest is public and returns
// {version, latest_url, sha256}; the XPI is served from the web root (not /api).
// {version, latest_url, sha256} plus an OPTIONAL {channel} naming which channel
// that instance serves ("dev"/"main", #3113); the XPI is served from the web
// root (not /api).
//
// The channel IS the instance: Firefox's static update_url cannot apply here
// because every FC install is a different host, so the extension asks its
// configured backend — which means switching channel is repointing apiUrl in
// options and reinstalling from that host. There is no separate channel
// setting to build, and building one would contradict each server build
// shipping its own extension.
function versionIsNewer(candidate, current) {
// Dotted numeric compare so 1.0.10 > 1.0.9 (a plain string compare wouldn't).
@@ -60,12 +69,22 @@ async function checkForUpdateInfo() {
}
const currentVersion = browser.runtime.getManifest().version;
const latestVersion = info && info.version ? info.version : null;
// Which channel the configured instance serves — reported ALONGSIDE the
// version, never folded into it. A `-dev` suffix would have to survive
// versionIsNewer's parseInt above, and it wouldn't: the segment would read
// as 0 and every dev build would compare equal to every other.
//
// null is a normal answer, not a failure — an instance built before the
// field existed, or one built locally with no channel declared. Nothing
// below branches on it except the label.
const channel = info && info.channel ? info.channel : null;
// latest_url is served from the web root, not the JSON API.
const base = api.webRoot();
return {
updateAvailable: !!latestVersion && versionIsNewer(latestVersion, currentVersion),
currentVersion,
latestVersion,
channel,
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null,
};
}
@@ -77,7 +96,11 @@ async function refreshUpdateBadge() {
await browser.action.setBadgeText({ text: r.updateAvailable ? '↑' : '' });
if (r.updateAvailable) {
await browser.action.setBadgeBackgroundColor({ color: '#F4BA7A' });
await browser.action.setTitle({ title: `FabledCurator — update available (v${r.latestVersion})` });
// Channel first, version second, and the channel dropped entirely when
// the instance doesn't report one — so the tooltip reads exactly as it
// did before the field existed rather than saying "(unknown ...)".
const label = r.channel ? `${r.channel} v${r.latestVersion}` : `v${r.latestVersion}`;
await browser.action.setTitle({ title: `FabledCurator — update available (${label})` });
} else {
await browser.action.setTitle({ title: 'FabledCurator' });
}
-11
View File
@@ -68,16 +68,6 @@ const PLATFORMS = {
urlPattern: /^https?:\/\/(www\.)?pixiv\.net/,
note: 'Click to authenticate via OAuth',
},
deviantart: {
name: 'DeviantArt',
domains: ['.deviantart.com', 'www.deviantart.com', 'deviantart.com'],
authType: 'cookies',
color: '#05CC47',
urlPattern: /^https?:\/\/(www\.)?deviantart\.com/,
// DA's logged-in-only endpoints sit behind their internal _napi
// namespace which shifts; skipping verify until a stable check
// surfaces. Same posture as SubscribeStar.
},
};
/**
@@ -98,7 +88,6 @@ const PLATFORM_ARTIST_PATTERNS = {
patreon: /^https?:\/\/(www\.)?patreon\.com\/(?:cw\/|c\/)?(?!(?:home|search|messages|notifications|library|settings|posts)(?:[\/?#]|$))[^/?#]+/i,
subscribestar: /^https?:\/\/(www\.)?subscribestar\.(com|adult)\/(?!feed$|messages$|library$)[^/?#]+\/?$/i,
hentaifoundry: /^https?:\/\/(www\.)?hentai-foundry\.com\/user\/[^/?#]+/i,
deviantart: /^https?:\/\/(www\.)?deviantart\.com\/(?!home$|watch\b|tag\b|browse\b)[^/?#]+\/?$/i,
pixiv: /^https?:\/\/(www\.)?pixiv\.net\/(en\/)?users\/\d+/i,
};
+1 -3
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.10",
"version": "1.0.11",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
@@ -33,7 +33,6 @@
"*://*.hentai-foundry.com/*",
"*://*.discord.com/*",
"*://*.pixiv.net/*",
"*://*.deviantart.com/*",
"*://app-api.pixiv.net/*",
"*://oauth.secure.pixiv.net/*",
"*://*/*"
@@ -61,7 +60,6 @@
"*://*.subscribestar.com/*",
"*://*.subscribestar.adult/*",
"*://*.hentai-foundry.com/*",
"*://*.deviantart.com/*",
"*://*.pixiv.net/*"
],
"js": ["lib/platforms.js", "content/content-script.js"],
+6 -5
View File
@@ -1,13 +1,14 @@
{
"name": "fabledcurator-extension",
"version": "1.0.10",
"version": "1.0.11",
"private": true,
"description": "Firefox extension for FabledCurator",
"comment_ignore_files": "The --ignore-files list comes from scripts/packaging.sh, the single source of truth shared with ci.yml's guard and the derived-version patch count. `set -f` is REQUIRED before the substitution: without it the shell globs `test/**` against the working tree and silently narrows the pattern to whatever files happen to exist.",
"scripts": {
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\"",
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --firefox=firefox",
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --overwrite-dest",
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore vitest.config.js \"test/**\" --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
"lint": "set -f; web-ext lint --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore)",
"start": "set -f; web-ext run --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --firefox=firefox",
"build": "set -f; web-ext build --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --overwrite-dest",
"sign": "set -f; web-ext sign --source-dir=. --no-config-discovery --ignore-files $(sh scripts/packaging.sh ignore) --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET",
"test:unit": "vitest run"
},
"devDependencies": {
+5 -1
View File
@@ -81,8 +81,12 @@ async function checkForUpdate() {
}
function showUpdateBanner(r) {
// The channel names itself beside the version, never inside it (#3113).
// Absent when the instance doesn't report one, and the banner then reads
// exactly as it did before the field existed.
const channel = r.channel ? ` (${r.channel})` : '';
document.getElementById('update-text').textContent =
`Update available — v${r.latestVersion} (installed v${r.currentVersion})`;
`Update available${channel} — v${r.latestVersion} (installed v${r.currentVersion})`;
// Opening the signed XPI triggers Firefox's native install prompt.
document.getElementById('update-btn').addEventListener('click', () => {
browser.tabs.create({ url: r.xpiUrl });
+157
View File
@@ -0,0 +1,157 @@
#!/bin/sh
# Single source of truth for "what ships inside the XPI", plus the version
# derived from it.
#
# Three consumers used to hand-maintain their own copy of this list, and
# keeping three copies of one fact in sync by hand is how issue #2397 happened:
#
# 1. web-ext's --ignore-files (extension/package.json's four scripts)
# 2. the :(exclude) pathspec (what moves the version — a WIDER
# set than the ignore list; see
# NOT_VERSION_RELEVANT)
# 3. the git-log pathspec (the derived version, below)
#
# They now all read from here. POSIX sh only — CI's run shell is busybox.
#
# -f (no pathname expansion) is set for the whole script and is load-bearing:
# the lists below are iterated with deliberate word-splitting, and without -f
# the shell would also GLOB them, expanding `test/**` into whatever files
# happen to exist and corrupting the output. A caller's own `set -f` does not
# help here — this runs as a separate sh process and does not inherit it.
# Callers still need their own `set -f` for the substituted result; the two
# guards protect different expansions.
set -euf
# Paths under extension/ that are NOT packaged into the XPI.
#
# Split by whether git tracks them: node_modules and web-ext-artifacts are
# build/dependency output that never appears in a commit, so they belong in
# web-ext's ignore list but would be meaningless in a git pathspec.
#
# Directories need BOTH forms. `test/**` matches the files inside, but not the
# directory entry itself — web-ext writes an entry for the directory too, so
# with only the glob the XPI ends up carrying empty `test/` and `scripts/`
# entries (caught by the XPI-content check on 2026-08-03). The bare name alone
# is not enough either: minimatch's `test` does not match `test/url.spec.js`,
# so dropping the glob would ship the contents. Keep both.
NOT_PACKAGED_TRACKED='package.json package-lock.json README.md .gitignore vitest.config.js scripts scripts/** test test/**'
NOT_PACKAGED_BUILD='web-ext-artifacts node_modules'
# Paths under extension/ that cannot change the SHIPPED BYTES, and so must not
# move the derived version.
#
# Deliberately NOT the same list as NOT_PACKAGED_TRACKED, and the whole
# difference is `scripts/`. packaging.sh is not packaged into the XPI — but it
# DECIDES the version string, and build.yml stamps that string into the
# manifest.json that is packaged. A change to how the version is computed is
# therefore a change to the shipped bytes.
#
# Excluding it was harmless only while every push rebuilt the web image.
# Milestone 313 step 4 made the rebuild conditional on the derived revision
# moving, which turned it into a silent failure: a packaging.sh change gives a
# NEW version, so sign-extension misses its ext-<version> cache and signs —
# while build-web sees an unmoved revision, reuses the published image, and
# ships the OLD XPI. An orphaned AMO signature, and an instance quietly serving
# code the registry says is current.
#
# The two directions are not symmetric, which is why this list is the narrower
# one. Too wide costs a re-sign and a rebuild for a change that ships nothing
# new. Too narrow serves stale bytes and says nothing.
NOT_VERSION_RELEVANT='package.json package-lock.json README.md .gitignore vitest.config.js test test/**'
usage() {
echo "usage: packaging.sh {ignore|pathspec|version|major-minor|patch}" >&2
exit 2
}
# web-ext --ignore-files values, space-separated.
#
# Callers MUST disable pathname expansion first (`set -f`), or the shell will
# glob `test/**` against the working tree before web-ext ever sees the pattern
# and silently narrow it to whatever happens to exist right now.
cmd_ignore() {
echo "$NOT_PACKAGED_TRACKED $NOT_PACKAGED_BUILD"
}
# git pathspec excluding the tracked files that cannot change the shipped
# bytes, e.g. :(exclude)extension/package.json :(exclude)extension/test/**
#
# This answers "what moves the version?", NOT "what goes in the XPI?" — see
# NOT_VERSION_RELEVANT for why those differ. cmd_ignore answers the other one.
# Same `set -f` requirement as above.
cmd_pathspec() {
for entry in $NOT_VERSION_RELEVANT; do
printf ':(exclude)extension/%s ' "$entry"
done
echo
}
# MAJOR.MINOR stays hand-set in manifest.json — it's the part that carries
# deliberate meaning. Only the patch component is derived.
cmd_major_minor() {
root=$(git rev-parse --show-toplevel)
grep -E '"version"' "$root/extension/manifest.json" \
| head -1 \
| sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([0-9]+)\.([0-9]+).*/\1.\2/'
}
# 2020-01-01T00:00:00Z — the anchor for the derived patch component. Fixed
# forever; moving it would renumber every version downwards.
VERSION_EPOCH=1577836800
# Minutes since VERSION_EPOCH of the LATEST commit that touched a PACKAGED
# extension file.
#
# Time-derived, per family rule 149: an artifact's ordering key must never be a
# commit count. A count is per-branch — `dev` and `main` count different
# histories of the same code — so the moment BOTH channels publish, their
# versions order by which branch accumulated more commits rather than by which
# is newer. A squash-merge makes that permanent: main gains one commit where dev
# gained five, so dev climbs away from main and a dev install can never cross
# back. That is Roundtable's 2026-08-24 incident (`versionCode` was the branch's
# commit count) in a different repo. Measured here on 2026-08-27: main=23,
# dev=24 under the old formula — one apart, which is exactly how the inversion
# stays invisible until it strands somebody.
#
# Why the commit's time and not the build's:
# * MONOTONIC — max() over a set that only ever gains members. Verified
# across all 24 extension-touching commits: zero non-monotonic steps.
# * STABLE while the extension is unchanged, so an unchanged extension keeps
# its version, the ext-<version> signature cache still hits, and AMO is
# called once per extension CHANGE rather than once per push. Build-time
# minutes would re-sign on every push and never let two channels share a
# signature.
# * SHARED ACROSS CHANNELS — after a merge, `main` sees the same commit and
# derives the same number, so `:latest` reuses the signature `:dev` already
# produced for byte-identical code. Same code, same version, one signing.
# * REPRODUCIBLE — any checkout of a commit yields that commit's version.
#
# Requires real history: a depth-1 clone sees one commit and will derive a wrong
# (too low) value. Every consumer must check out with fetch-depth: 0.
cmd_patch() {
root=$(git rev-parse --show-toplevel)
# Unquoted on purpose: the pathspec must word-split into separate args.
# Globbing is already off script-wide (set -euf above).
# shellcheck disable=SC2046
ts=$(cd "$root" && git log --format=%ct HEAD -- extension/ $(cmd_pathspec) \
| sort -n | tail -1)
if [ -z "$ts" ]; then
echo "packaging.sh: no commit touches a packaged extension file" >&2
exit 1
fi
echo $(( (ts - VERSION_EPOCH) / 60 ))
}
cmd_version() {
echo "$(cmd_major_minor).$(cmd_patch)"
}
[ $# -ge 1 ] || usage
case "$1" in
ignore) cmd_ignore ;;
pathspec) cmd_pathspec ;;
version) cmd_version ;;
major-minor) cmd_major_minor ;;
patch) cmd_patch ;;
*) usage ;;
esac
+71 -8
View File
@@ -1,6 +1,12 @@
import { describe, it, expect } from 'vitest'
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import path from 'node:path'
import { loadLib } from './helpers/loadLib.js'
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
const manifest = JSON.parse(readFileSync(path.join(EXT_DIR, 'manifest.json'), 'utf8'))
const { getPlatformFromUrl, isArtistPage, PLATFORMS, PLATFORM_ARTIST_PATTERNS } = loadLib(
'platforms.js',
['getPlatformFromUrl', 'isArtistPage', 'PLATFORMS', 'PLATFORM_ARTIST_PATTERNS']
@@ -13,7 +19,6 @@ describe('getPlatformFromUrl', () => {
expect(getPlatformFromUrl('https://www.hentai-foundry.com/user/someone')).toBe('hentaifoundry')
expect(getPlatformFromUrl('https://discord.com/channels/@me')).toBe('discord')
expect(getPlatformFromUrl('https://www.pixiv.net/en/users/123')).toBe('pixiv')
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe('deviantart')
})
it('accepts http as well as https, with or without www', () => {
@@ -26,6 +31,15 @@ describe('getPlatformFromUrl', () => {
expect(getPlatformFromUrl('https://not-patreon.com/Atole')).toBe(null)
expect(getPlatformFromUrl('')).toBe(null)
})
it('returns null for deviantart, retired at #3069', () => {
// The 2026-07-05 product decision (FC downloaders = art-dedicated services
// only) left deviantart wired for seven weeks. Asserting the negative is
// what keeps a partial retirement from being re-completed by accident.
expect(getPlatformFromUrl('https://www.deviantart.com/someone')).toBe(null)
expect(PLATFORMS.deviantart).toBeUndefined()
expect(PLATFORM_ARTIST_PATTERNS.deviantart).toBeUndefined()
})
})
describe('isArtistPage', () => {
@@ -72,12 +86,6 @@ describe('isArtistPage', () => {
expect(isArtistPage('https://www.pixiv.net/en/artworks/999', 'pixiv')).toBe(false)
})
it('excludes DeviantArt navigation roots', () => {
expect(isArtistPage('https://www.deviantart.com/someone', 'deviantart')).toBe(true)
expect(isArtistPage('https://www.deviantart.com/home', 'deviantart')).toBe(false)
expect(isArtistPage('https://www.deviantart.com/watch', 'deviantart')).toBe(false)
})
it('returns false for a platform with no artist pattern (discord)', () => {
expect(isArtistPage('https://discord.com/channels/@me', 'discord')).toBe(false)
})
@@ -116,7 +124,6 @@ describe('platform table integrity', () => {
patreon: 'https://www.patreon.com/cw/Atole',
subscribestar: 'https://subscribestar.adult/someone',
hentaifoundry: 'https://www.hentai-foundry.com/user/someone',
deviantart: 'https://www.deviantart.com/someone',
pixiv: 'https://www.pixiv.net/en/users/12345'
}
for (const [key, url] of Object.entries(samples)) {
@@ -125,3 +132,59 @@ describe('platform table integrity', () => {
}
})
})
describe('manifest.json agrees with the platform table', () => {
// #3069: deviantart was dropped from the product in July but survived in
// manifest.json until late August, because NOTHING tied the manifest's
// domain lists back to PLATFORMS. These two specs are that tie. Both
// directions matter: a stale match ships host access the product decided
// not to use, and a missing one silently kills the Add-to-FC button.
const matches = manifest.content_scripts[0].matches
// '*://*.patreon.com/*' -> '.patreon.com', the form PLATFORMS.domains uses.
const hostOf = (m) => m.replace(/^\*:\/\/\*/, '').replace(/\/\*$/, '')
it('injects the content script only on domains a platform claims', () => {
for (const m of matches) {
const host = hostOf(m)
const owner = Object.entries(PLATFORMS).find(
([, p]) => p.domains.includes(host)
)
expect(owner, `no platform claims content-script match "${m}"`).toBeTruthy()
// The content script exists to draw the Add-as-source button, so a
// platform with no artist pattern (discord) has no business here.
expect(
PLATFORM_ARTIST_PATTERNS[owner[0]],
`"${m}" injects for ${owner[0]}, which has no artist pattern`
).toBeTruthy()
}
})
it('injects on every platform that has an artist pattern', () => {
const covered = new Set(
matches
.map(hostOf)
.map((h) => Object.entries(PLATFORMS).find(([, p]) => p.domains.includes(h)))
.filter(Boolean)
.map(([key]) => key)
)
for (const key of Object.keys(PLATFORM_ARTIST_PATTERNS)) {
expect(covered, `${key} has an artist pattern but no content-script match`).toContain(key)
}
})
it('requests no host permission for a domain no platform claims', () => {
// '*://*/*' is the deliberate exception: FC is self-hosted at an arbitrary
// operator-chosen URL, so the extension cannot enumerate its own backend.
// Every OTHER entry is a platform domain and must still have an owner.
for (const h of manifest.host_permissions) {
if (h === '*://*/*') continue
const host = hostOf(h)
// pixiv's OAuth/API hosts are pixiv infrastructure, not creator pages,
// so they are matched by suffix rather than by the domains list.
const claimed = Object.values(PLATFORMS).some(
(p) => p.domains.includes(host) || p.domains.some((d) => host.endsWith(d))
)
expect(claimed, `host permission "${h}" belongs to no platform`).toBe(true)
}
})
})
+148 -42
View File
@@ -1,32 +1,162 @@
import { describe, it, expect } from 'vitest'
import { readFileSync } from 'node:fs'
import { execFileSync } from 'node:child_process'
import { fileURLToPath } from 'node:url'
import path from 'node:path'
const EXT_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), '..')
const read = (name) => JSON.parse(readFileSync(path.join(EXT_DIR, name), 'utf8'))
const readText = (...seg) => readFileSync(path.join(EXT_DIR, ...seg), 'utf8')
describe('extension version consistency', () => {
// Duplicates check (1) of ci.yml's extension-version job, deliberately.
// That job is the gate that can't be bypassed; this spec is the one that
// fails in a second on the developer's own CI lane with a readable diff.
// The two version strings feed different systems and nothing else reconciles
// them:
// manifest.json -> what `web-ext sign` signs, so what Firefox installs
// (package.json is in --ignore-files, not in the XPI)
// package.json -> build.yml's AMO cache key, the ext-<version> release
// tag, the XPI filename, and therefore the version
// /api/extension/manifest reports to the update prompt
it('keeps manifest.json and package.json in lockstep', () => {
const manifest = read('manifest.json')
const pkg = read('package.json')
expect(manifest.version).toBe(pkg.version)
// Only the git-free subcommands are exercised here: `version`/`patch` shell out
// to git, and the extension lane runs on node:24-bookworm-slim which may not
// ship it. Those two are covered where git is guaranteed — ci.yml and build.yml
// run on ci-python.
const packaging = (cmd) =>
execFileSync('sh', [path.join(EXT_DIR, 'scripts', 'packaging.sh'), cmd], {
cwd: EXT_DIR,
encoding: 'utf8'
})
.trim()
.split(/\s+/)
.filter(Boolean)
describe('packaging.sh — the single definition of what ships', () => {
it('emits an ignore list and a pathspec that agree on the tracked files', () => {
const ignore = packaging('ignore')
const pathspec = packaging('pathspec').map((e) => e.replace(':(exclude)extension/', ''))
// Every git-excluded path must also be hidden from web-ext. The reverse is
// not required: node_modules and web-ext-artifacts are build output git
// never tracks, so they appear only in the ignore list.
for (const entry of pathspec) {
expect(ignore, `pathspec has "${entry}" but --ignore-files does not`).toContain(entry)
}
expect(pathspec.length).toBeGreaterThan(0)
expect(ignore).toContain('node_modules')
})
it('emits glob patterns literally, never expanded against the working tree', () => {
// The script iterates its lists with deliberate word-splitting, so it must
// run with pathname expansion off. Without that, invoking it from a cwd
// where test/ exists (exactly how ci.yml and vitest call it) expands
// `test/**` into the individual spec files, and the pathspec silently stops
// covering anything added later.
const pathspec = packaging('pathspec')
expect(pathspec).toContain(':(exclude)extension/test/**')
expect(pathspec.some((e) => e.includes('.spec.js'))).toBe(false)
expect(pathspec.some((e) => e.includes('helpers'))).toBe(false)
const ignore = packaging('ignore')
expect(ignore).toContain('test/**')
expect(ignore).toContain('scripts/**')
expect(ignore.some((e) => e.includes('.spec.js'))).toBe(false)
})
it('lets packaging.sh move the version, though it never ships in the XPI', () => {
// The two lists answer different questions and this is the one place they
// disagree. scripts/ is ignored by web-ext — it is repo tooling, not addon
// code — but packaging.sh DECIDES the version string, and build.yml stamps
// that string into the manifest.json that does ship. So changing how the
// version is computed changes the shipped bytes.
//
// Excluding it from the pathspec was invisible while every push rebuilt the
// web image. Milestone 313 step 4 made that rebuild conditional on the
// derived revision moving, and the omission turned into a silent failure:
// a new version means sign-extension misses its ext-<version> cache and
// signs, while build-web sees an unmoved revision, reuses the published
// image and ships the OLD XPI. An orphaned signature, and an instance
// serving code the registry calls current.
const pathspec = packaging('pathspec')
expect(
pathspec.some((e) => e.startsWith(':(exclude)extension/scripts')),
'the pathspec excludes scripts/, so a change to how the version is '
+ 'derived would not move the version it derives',
).toBe(false)
// ...and it is still kept out of the package itself. Both must hold: the
// tempting "fix" for either half is to make the two lists one again.
expect(packaging('ignore')).toContain('scripts')
})
it('keeps its own scripts and specs out of the XPI', () => {
// Both are repo infrastructure. web-ext packages everything not ignored, so
// omitting either would ship dev tooling to users -- and `test/**` in
// particular only survives because callers `set -f` before substituting it.
const ignore = packaging('ignore')
expect(ignore).toContain('vitest.config.js')
// Both forms per directory. The glob covers the contents; the bare name
// covers the directory ENTRY, which web-ext writes separately — with only
// the glob, the XPI carries an empty `test/` and `scripts/`.
for (const dir of ['test', 'scripts']) {
expect(ignore, `${dir} contents`).toContain(`${dir}/**`)
expect(ignore, `${dir} directory entry`).toContain(dir)
}
})
})
describe('consumers delegate rather than keeping their own copy', () => {
// These assertions are the actual anti-regression value: it is easy for a
// future edit to "simplify" by inlining a literal list again, which silently
// reintroduces the drift that issue #2397 was about.
it('package.json derives --ignore-files from the script', () => {
for (const [name, script] of Object.entries(read('package.json').scripts)) {
if (!script.includes('--ignore-files')) continue
expect(script, `${name} should call packaging.sh`).toContain('scripts/packaging.sh ignore')
expect(script, `${name} must set -f before the substitution`).toMatch(/set -f;/)
}
})
const WORKFLOWS = ['ci.yml', 'build.yml', 'extension.yml']
it('no workflow hardcodes the packaged-file set', () => {
// ci.yml used to substitute `packaging.sh pathspec` directly, for the
// manual-bump guard that milestone 271 step 5 retired. Nothing inlines the
// set today, and nothing should start to: a literal :(exclude)extension/...
// in a workflow means someone bypassed the shared definition, which is
// exactly the drift #2397 was about. Asserted across all three rather than
// against one named consumer, so it keeps holding as consumers come and go.
for (const wf of WORKFLOWS) {
const text = readText('..', '.forgejo', 'workflows', wf)
expect(text, `${wf} inlines an :(exclude) literal`).not.toMatch(/:\(exclude\)extension\//)
}
})
it('build.yml takes the shipped version from the script, not from the repo', () => {
// The version is DERIVED from commit time (#3092, milestone 271 step 4).
// Going back to reading the committed value is not a style regression, it
// is the bug: a hand-set version makes dev and main sign the same number
// for different code, and the ext-<version> cache then serves one channel
// the other's XPI.
const build = readText('..', '.forgejo', 'workflows', 'build.yml')
expect(build).toContain('packaging.sh version')
expect(build, 'build.yml re-reads the committed version instead of deriving it')
.not.toMatch(/grep[^\n]*'"version"'[^\n]*package\.json/)
})
})
describe('extension version', () => {
const majorMinor = (v) => v.split('.').slice(0, 2).join('.')
it('keeps the hand-set MAJOR.MINOR in lockstep across both files', () => {
// Narrowed from full-string equality at milestone 271 step 5. Since step 4
// the patch component is derived from commit time and stamped into both
// files at build time, so the committed patch numbers are inert — nothing
// reads them and they are not what ships. Asserting on them would fail for
// a difference that changes nothing.
//
// MAJOR.MINOR is the opposite: still hand-set, still shipped, and
// packaging.sh reads it from manifest.json ALONE. Let the two diverge and
// the extension ships a version package.json disagrees with, with no other
// signal.
expect(majorMinor(read('manifest.json').version))
.toBe(majorMinor(read('package.json').version))
})
it('uses a plain dotted numeric version AMO will accept', () => {
// AMO rejects exotic version strings, and build.yml embeds this value in a
// release tag and a filename — so anything needing escaping breaks the
// publish path rather than the extension.
// The committed value seeds MAJOR.MINOR, so it still has to parse even
// though its patch component never ships. ci.yml asserts the same shape on
// the DERIVED value, which is the one AMO actually sees.
expect(read('package.json').version).toMatch(/^\d+(\.\d+)*$/)
})
@@ -34,30 +164,6 @@ describe('extension version consistency', () => {
expect(read('manifest.json').manifest_version).toBe(3)
})
it('never lets the CI guard ignore a file that actually ships', () => {
// ci.yml's extension-version job skips its bump check for paths it deems
// non-shipping. If it excludes something web-ext DOES package, a real
// change to shipped code passes the guard unnoticed — precisely the
// silent-stale-ship the guard exists to stop. The reverse drift (guard
// stricter than web-ext) only costs a needless bump, so it isn't asserted.
const ci = readFileSync(path.join(EXT_DIR, '..', '.forgejo', 'workflows', 'ci.yml'), 'utf8')
const lint = read('package.json').scripts.lint
const after = lint.split('--ignore-files')[1] ?? ''
const ignored = new Set(
after
.split(/\s+/)
.filter((tok) => tok && !tok.startsWith('--'))
.map((tok) => tok.replace(/^["']|["']$/g, ''))
)
expect(ignored.size, 'parsed --ignore-files from the lint script').toBeGreaterThan(0)
const guarded = [...ci.matchAll(/:\(exclude\)extension\/(\S+?)'/g)].map((m) => m[1])
expect(guarded.length, 'parsed :(exclude) entries from ci.yml').toBeGreaterThan(0)
for (const entry of guarded) {
expect(ignored, `ci.yml excludes "${entry}" but web-ext packages it`).toContain(entry)
}
})
it('lists every background script that exists, in dependency order', () => {
// url.js must load BEFORE api.js: api.js calls normalizeApiUrl at
// init()-time, and these are classic scripts sharing one scope, so a
@@ -50,14 +50,19 @@ const projected = ref(null)
const projectedCounts = computed(() => projected.value?.projected || null)
const modalDescription = computed(
() => projected.value
? `Artist “${props.artistName}” — `
+ `${projected.value.projected.images} images, `
+ `${projected.value.projected.sources} sources, `
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
// `posts` is named here, not left to the counts grid below it: an artist whose
// posts are body-only previews as `images: 0`, and a summary line that says
// only "0 images" reads as "this artist is empty" while the apply destroys
// every captured post body (#3067). Attachments stay in the grid — the grid
// renders every key, so this line carries only what changes the read.
const modalDescription = computed(() => {
const p = projectedCounts.value
return p
? `Artist “${props.artistName}” — ${p.images} images, `
+ `${p.posts} posts, ${p.sources} sources, `
+ `${Math.round(p.bytes_on_disk / 1_048_576)} MiB on disk`
: ''
})
async function onClick() {
loading.value = true
@@ -139,9 +139,9 @@ function onThumbError() { thumbError.value = true }
position: absolute; top: 8px; left: 8px;
width: 22px; height: 22px; border-radius: 4px;
border: 2px solid rgba(232, 228, 216, 0.8);
background: rgba(20, 23, 26, 0.45);
background: rgba(var(--v-theme-background), 0.45);
display: grid; place-items: center;
color: #14171A; z-index: 11;
color: rgb(var(--v-theme-background)); z-index: 11;
}
.fc-gallery-item__checkbox.on {
background: rgb(var(--v-theme-accent));
@@ -152,7 +152,7 @@ function onThumbError() { thumbError.value = true }
min-width: 22px; height: 22px; padding: 0 5px;
border-radius: 11px;
background: rgb(var(--v-theme-accent));
color: #14171A; font-size: 12px; font-weight: 700;
color: rgb(var(--v-theme-background)); font-size: 12px; font-weight: 700;
display: grid; place-items: center; z-index: 11;
pointer-events: none;
}
@@ -160,7 +160,8 @@ function onThumbError() { thumbError.value = true }
position: absolute; left: 0; right: 0; bottom: 0;
padding: 14px 8px 6px;
background: linear-gradient(
to top, rgba(20, 23, 26, 0.78), rgba(20, 23, 26, 0)
to top, rgba(var(--v-theme-background), 0.78),
rgba(var(--v-theme-background), 0)
);
font-size: 12px; line-height: 1.2;
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
@@ -0,0 +1,127 @@
<template>
<!-- #3068: attachment reclamation. PostAttachment's FKs are both SET NULL, so
a deleted post or artist leaves the row behind; and the store is
sha-addressed, so one blob backs many rows and deleting a row never freed
its file. Nothing swept either. Preview first, then apply (destructive:
unlinks files). -->
<MaintenanceTile
icon="mdi-paperclip-off"
title="Reclaim orphaned attachments"
blurb="Remove attachment records belonging to nothing, and the files nothing references."
destructive
:open="applying || previewing"
>
<p class="text-body-2 mb-3">
Attachment records survive the post and artist they belonged to, and the
files behind them are shared between records — so a deleted record never
freed its file on its own. This finds records attributed to
<strong>neither</strong> a post nor an artist, and files in the attachment
store that <strong>no remaining record</strong> points at.
<strong>Preview</strong> first; <strong>Apply</strong> deletes those
records and unlinks those files. Files written in the last few hours are
always left alone, so an in-progress download is never caught mid-write.
</p>
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
<v-btn
color="primary" variant="tonal" rounded="pill"
:loading="previewing" :disabled="applying" @click="preview"
>
<v-icon start>mdi-magnify</v-icon> Preview
</v-btn>
<v-btn
color="error" rounded="pill"
:loading="applying"
:disabled="previewing || !canApply"
@click="confirmOpen = true"
>
<v-icon start>mdi-paperclip-off</v-icon> Apply
</v-btn>
</div>
<v-alert
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
density="comfortable"
>
<span v-if="applied">
Deleted {{ summary.rows }} orphaned record(s) and unlinked
{{ summary.files }} file(s), reclaiming {{ humanBytes(summary.bytes) }}.
</span>
<span v-else-if="hasWork">
{{ summary.rows }} orphaned record(s) and {{ summary.files }}
unreferenced file(s) — {{ humanBytes(summary.bytes) }} reclaimable.
Click <strong>Apply</strong> to remove them.
</span>
<span v-else>Nothing to reclaim — every attachment is accounted for.</span>
<!-- Both of these change what the numbers MEAN, so they are stated
whenever they are non-zero rather than hidden in a tooltip. -->
<div v-if="summary.files_failed" class="mt-1 text-caption">
{{ summary.files_failed }} file(s) could not be read or removed — see
the worker log.
</div>
<div v-if="summary.partial" class="mt-1 text-caption">
Stopped early at the time limit; some of the store was not examined.
Run it again to continue.
</div>
</v-alert>
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance" />
<v-dialog v-model="confirmOpen" max-width="440">
<v-card>
<v-card-title>Reclaim orphaned attachments?</v-card-title>
<v-card-text class="text-body-2">
This permanently deletes
<strong>{{ summary?.rows ?? 0 }}</strong> attachment record(s) and
unlinks <strong>{{ summary?.files ?? 0 }}</strong> file(s)
({{ humanBytes(summary?.bytes) }}). Only files that no remaining
record points at are removed, so nothing still attached to a post
is affected.
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
<v-btn color="error" @click="apply">Reclaim</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</MaintenanceTile>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
import { humanBytes } from '../../utils/bytes.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue'
const confirmOpen = ref(false)
// Walks the whole attachment store, so it can run for minutes on a large
// library — the service caps itself at 900s and reports `partial`. 150 polls
// × 2s ≈ 5m of foreground waiting; past that the composable hands off to the
// task dashboard rather than spinning forever.
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
endpoint: '/api/admin/maintenance/reclaim-attachments',
storageKey: 'fc.maint.reclaimAttachments',
appliedToast: 'Orphaned attachments reclaimed',
maxPolls: 150,
})
const hasWork = computed(
() => !!summary.value && (summary.value.rows > 0 || summary.value.files > 0),
)
const canApply = computed(() => hasWork.value && !applied.value)
const summaryType = computed(() => {
if (applied.value) return 'success'
return hasWork.value ? 'info' : 'success'
})
// The confirm dialog gates the destructive apply; close it, then run.
function apply () {
confirmOpen.value = false
applyTask()
}
</script>
@@ -4,12 +4,22 @@
<span v-if="manifest?.installed" class="text-caption fc-muted">
· Firefox · v{{ manifest.version }}
</span>
<!-- Which channel this instance serves, so it is visible without
installing anything. Rendered only when the image declares one: a
locally-built image, or one predating the field, says nothing rather
than guessing. Never merged into the version string beside it see
the endpoint's note on why a `-dev` suffix breaks the comparator. -->
<v-chip
v-if="manifest?.channel"
size="x-small" variant="tonal" class="ml-2"
:color="manifest.channel === 'dev' ? 'warning' : 'info'"
>{{ manifest.channel }}</v-chip>
</CardHeading>
<v-card-text>
<p class="fc-muted text-body-2">
Pushes session cookies from supported platforms
(patreon, subscribestar, hentaifoundry, discord, pixiv, deviantart)
(patreon, subscribestar, hentaifoundry, discord, pixiv)
into FabledCurator, and lets you add a creator as a source from
their page in one click.
</p>
@@ -35,7 +35,7 @@
All subscription sources healthy.
</p>
<p v-else class="text-body-2 mb-0">
<b class="fc-bad">{{ failing.length }}</b> failing source(s):
<b class="fc-weak">{{ failing.length }}</b> failing source(s):
<span class="fc-muted">{{ failingNames }}</span>
</p>
</v-card-text>
@@ -72,5 +72,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
</script>
<style scoped>
.fc-bad { color: rgb(var(--v-theme-error)); }
</style>
@@ -102,6 +102,7 @@
import { computed, ref } from 'vue'
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
import { humanBytes } from '../../utils/bytes.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue'
@@ -122,14 +123,6 @@ const summaryType = computed(() => {
return summary.value && summary.value.matched > 0 ? 'info' : 'success'
})
function humanBytes (n) {
const b = Number(n || 0)
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
return b + ' B'
}
// The confirm dialog gates the destructive apply; close it, then run.
function apply () {
confirmOpen.value = false
@@ -25,7 +25,7 @@
<div class="fc-cell__l">done</div>
</div>
<div class="fc-cell">
<div class="fc-cell__n" :class="q.error ? 'fc-bad' : ''">{{ q.error }}</div>
<div class="fc-cell__n" :class="q.error ? 'fc-weak' : ''">{{ q.error }}</div>
<div class="fc-cell__l">errored</div>
</div>
</div>
@@ -104,5 +104,4 @@ onUnmounted(() => { if (pollId) clearInterval(pollId) })
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-bad { color: rgb(var(--v-theme-error)); }
</style>
@@ -78,6 +78,7 @@
import { computed, ref } from 'vue'
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
import { humanBytes } from '../../utils/bytes.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue'
@@ -98,14 +99,6 @@ const summaryType = computed(() => {
return summary.value && summary.value.redundant > 0 ? 'info' : 'success'
})
function humanBytes (n) {
const b = Number(n || 0)
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
return b + ' B'
}
// The confirm dialog gates the destructive apply; close it, then run.
function apply () {
confirmOpen.value = false
+6 -1
View File
@@ -50,7 +50,12 @@
/* Status text colours (DRY pass #161): fc-good = success, fc-weak = error,
consolidated from the GPU / heads cards. fc-ok is intentionally NOT global —
it means on-surface in HeadsCard but success in QueuesTable. */
it means on-surface in HeadsCard but success in QueuesTable.
No `.fc-bad` (#3072): it was defined locally and identically in the Downloads
and GPU activity panels, and it is fc-weak under a second name — GpuAgentCard
and GpuActivityPanel were colouring the same "errored" count with different
class names. Both now use fc-weak. Reach for fc-weak, not a new synonym. */
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
+17
View File
@@ -0,0 +1,17 @@
// Human-readable byte sizes for maintenance summaries ("2.4 GB reclaimable").
//
// Promoted out of the cleanup cards, which had grown byte-identical private
// copies (VideoDedupCard, GatedPurgeCard) and were about to grow a third for
// the attachment reclaim. Binary units (1 KB = 1024 B) — these numbers come
// from st_size / SUM(size_bytes), so they describe disk, not marketing.
//
// NOT the same shape as the `formatBytes` helpers in SystemStatsCards,
// BackupRunsTable and PostCard — those differ in units, precision and
// zero-handling. Left alone deliberately rather than force-fitted here.
export function humanBytes (n) {
const b = Number(n || 0)
if (b >= 1 << 30) return (b / (1 << 30)).toFixed(1) + ' GB'
if (b >= 1 << 20) return (b / (1 << 20)).toFixed(1) + ' MB'
if (b >= 1 << 10) return (b / (1 << 10)).toFixed(1) + ' KB'
return b + ' B'
}
+5 -6
View File
@@ -1,8 +1,10 @@
// Single source of truth for platform → color + icon mapping. Used by
// PlatformChip and any other GS-style platform-tagged surface. The six
// PlatformChip and any other GS-style platform-tagged surface. The five
// platforms FC supports map 1:1 to the GS palette; unknown platforms fall
// back to grey + mdi-web. Operator-confirmed scope 2026-05-27. The ICONS key
// set is pinned against backend known_platform_keys() by
// back to grey + mdi-web — which is deliberately what a retired platform
// hits: a pre-#3069 deviantart source row still renders, as its raw key on
// a grey chip. Operator-confirmed scope 2026-05-27. The ICONS key set is
// pinned against backend known_platform_keys() by
// tests/test_fe_be_contract.py.
const ICONS = {
@@ -11,7 +13,6 @@ const ICONS = {
hentaifoundry: 'mdi-palette',
discord: 'mdi-discord',
pixiv: 'mdi-alpha-p-box',
deviantart: 'mdi-deviantart',
}
const COLORS = {
@@ -20,7 +21,6 @@ const COLORS = {
hentaifoundry: 'purple',
discord: 'indigo',
pixiv: 'blue',
deviantart: 'green',
}
const LABELS = {
@@ -29,7 +29,6 @@ const LABELS = {
hentaifoundry: 'HentaiFoundry',
discord: 'Discord',
pixiv: 'Pixiv',
deviantart: 'DeviantArt',
}
export function platformIcon(platform) {
+5 -2
View File
@@ -19,14 +19,16 @@
</section>
<section class="fc-section">
<h3 class="fc-section__title">Duplicates &amp; posts</h3>
<h3 class="fc-section__title">Duplicates &amp; leftovers</h3>
<p class="fc-section__hint">
Tidy post records, duplicates and locked-preview leftovers.
Tidy post records, duplicates, locked-preview leftovers and attachments
that outlived what they belonged to.
</p>
<div class="fc-tile-grid">
<PostMaintenanceCard />
<VideoDedupCard />
<GatedPurgeCard />
<AttachmentReclaimCard />
</div>
</section>
@@ -60,6 +62,7 @@ import SingleColorAuditCard from '../components/cleanup/SingleColorAuditCard.vue
import PostMaintenanceCard from '../components/settings/PostMaintenanceCard.vue'
import VideoDedupCard from '../components/settings/VideoDedupCard.vue'
import GatedPurgeCard from '../components/settings/GatedPurgeCard.vue'
import AttachmentReclaimCard from '../components/settings/AttachmentReclaimCard.vue'
import TagMaintenanceCard from '../components/settings/TagMaintenanceCard.vue'
import DangerZoneCard from '../components/settings/DangerZoneCard.vue'
</script>
@@ -0,0 +1,72 @@
// @vitest-environment happy-dom
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { nextTick } from 'vue'
import BrowserExtensionCard from '../../src/components/settings/BrowserExtensionCard.vue'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
// useApi is a thin fetch wrapper, so the seam is fetch itself (same shape as
// showcase.spec.js) rather than a module mock.
function stubApi(manifest) {
globalThis.fetch = vi.fn(async (url) => {
const payload = String(url).includes('/api/extension/manifest')
? manifest
: { key: 'test-key' }
return {
ok: true, status: 200, statusText: '200',
text: async () => JSON.stringify(payload),
}
})
}
async function mountCard(manifest) {
stubApi(manifest)
const w = mountComponent(BrowserExtensionCard, { pinia: freshPinia() })
// onMounted fires two fetches (manifest + key) and each resolves through a
// chain of microtasks. Yielding to a macrotask drains the whole queue, which
// a fixed number of nextTicks would only do by luck.
await new Promise((resolve) => setTimeout(resolve, 0))
await nextTick()
return w
}
const INSTALLED = {
installed: true,
version: '1.0.3499884',
xpi_url: '/extension/fabledcurator-1.0.3499884.xpi',
latest_url: '/extension/fabledcurator-latest.xpi',
sha256: 'abc',
}
describe('BrowserExtensionCard — channel', () => {
beforeEach(() => { vi.restoreAllMocks() })
afterEach(() => { delete globalThis.fetch })
it('names the channel the instance reports', async () => {
// The point of the whole channel scheme: an operator can tell a dev
// instance from a main one without installing anything.
const w = await mountCard({ ...INSTALLED, channel: 'dev' })
expect(w.text()).toContain('dev')
})
it('shows the version and the channel as SEPARATE text, never merged', async () => {
// Regression guard with teeth: the tempting shortcut is a `-dev` version
// suffix, and that is precisely what breaks the extension's comparator —
// it parses each dotted segment with parseInt, so a suffixed segment reads
// as 0 and every dev build compares equal to every other. If someone ever
// "simplifies" by folding the channel into the version, the version text
// stops being the bare derived number and this fails.
const w = await mountCard({ ...INSTALLED, channel: 'dev' })
expect(w.text()).toContain('v1.0.3499884')
expect(w.text()).not.toContain('1.0.3499884-dev')
})
it('renders no channel when the instance declares none', async () => {
// A locally-built image, or one predating the field. The card must read
// exactly as it did before the channel existed rather than inventing an
// "unknown" badge — absence is a normal answer here, not a fault.
const w = await mountCard(INSTALLED)
expect(w.text()).toContain('v1.0.3499884')
expect(w.findAll('v-chip')).toHaveLength(0)
})
})
+206
View File
@@ -0,0 +1,206 @@
#!/bin/sh
# Single definition of WHAT EACH PUBLISHED ARTIFACT IS BUILT FROM, and the
# version derived from it. Milestone 313; generalises the shape
# extension/scripts/packaging.sh established for the extension alone.
#
# Four artifacts, four independent versions. An artifact whose shipped files
# did not change keeps its version and does not rebuild — that is the whole
# point, and it is why each path set must match its Dockerfile rather than
# being a plausible guess. Getting a set wrong is quiet in BOTH directions:
#
# too narrow -> a pin serves stale bytes, because the version did not move
# when the content did. This is the dangerous one.
# too wide -> the artifact re-versions and rebuilds for a change it does
# not ship. Merely wasteful.
#
# tests/test_artifact_paths.py asserts every COPY source in each Dockerfile is
# covered here, so adding a COPY without updating this file fails CI.
#
# POSIX sh only — CI's run shell is busybox on some paths.
#
# -f (no pathname expansion) is load-bearing for the whole script: the lists
# below are iterated with deliberate word-splitting, and without it the shell
# would glob `frontend/test/**` against the working tree and silently narrow
# the pattern. Callers substituting the output need their own `set -f` too;
# the two guards protect different expansions.
set -euf
ROOT=$(git rev-parse --show-toplevel)
# --- what each artifact ships ------------------------------------------------
#
# Each set includes its own Dockerfile and requirements: changing a base image
# or a pin changes the artifact just as surely as changing a source file.
#
# web (Dockerfile, context `.`) — the runtime stage copies backend/, alembic/,
# alembic.ini, entrypoint.sh and requirements.txt; the frontend-builder stage
# copies frontend/ and the runtime takes its `dist` output.
#
# frontend/test is excluded: `npm run build` is vite, which builds from src/,
# index.html and public/ and never reads test/. It lands in the builder layer
# but not in `dist`, so it cannot reach the shipped image.
#
# The web image ALSO bundles the signed XPI (build.yml downloads it into
# frontend/public/extension/ before the docker build), so an extension change
# changes the web image. The extension's packaged set is appended in cmd_paths
# rather than restated — one definition, per #2397.
WEB_PATHS='Dockerfile requirements.txt backend alembic alembic.ini entrypoint.sh frontend :(exclude)frontend/test :(exclude)frontend/test/**'
# ml (Dockerfile.ml, context `.`) — no frontend, no extension. Note it copies
# BOTH requirements-ml.txt and requirements.txt.
ML_PATHS='Dockerfile.ml requirements-ml.txt requirements.txt backend alembic alembic.ini entrypoint.sh'
# agent (agent/Dockerfile, context `agent`) — copies requirements.txt and
# fc_agent only. agent/README.md, agent/docker-compose.yml and agent/ruff.toml
# live in the directory but never reach the image, so they must not re-version
# it: this is deliberately NOT `agent/`.
AGENT_PATHS='agent/Dockerfile agent/requirements.txt agent/fc_agent'
# Which artifacts bake the BUILD CHANNEL into the image, and therefore cannot
# share a content identity across channels. The web image takes FC_CHANNEL as
# a build-arg and reports it from /api/extension/manifest (milestone 271 step
# 7), so `main` and `dev` builds of one revision are genuinely different
# images — reusing the dev one on main would ship an instance that names
# itself `dev` forever.
#
# ml and agent take no build-args at all: one revision, one image, and a merge
# to main can reuse exactly what dev already built. That is not a detail, it is
# most of what step 4 saves — merges would otherwise rebuild the agent's CUDA
# image to produce bytes that already exist.
#
# Extend this list if a second artifact ever gains a build-arg;
# tests/test_artifact_identity.py reads the Dockerfiles and fails if it drifts.
CHANNELLED='web'
usage() {
echo "usage: artifacts.sh {paths|revision|version|tag} {web|ml|agent|extension}" >&2
echo " artifacts.sh identity {web|ml|agent} [channel]" >&2
exit 2
}
# The extension's packaged set, read from its own definition rather than
# copied. packaging.sh emits `:(exclude)extension/...` entries, so the bare
# `extension` include has to come with them.
ext_paths() {
echo "extension $(sh "$ROOT/extension/scripts/packaging.sh" pathspec)"
}
cmd_paths() {
case "$1" in
web) echo "$WEB_PATHS $(ext_paths)" ;;
ml) echo "$ML_PATHS" ;;
agent) echo "$AGENT_PATHS" ;;
extension) ext_paths ;;
*) usage ;;
esac
}
# "<unix ts> <sha>" of the newest commit touching this artifact's shipped set.
# Unquoted on purpose: the pathspec must word-split into separate args.
# Globbing is already off script-wide.
newest() {
# shellcheck disable=SC2046
set -- "$(cd "$ROOT" && git log --format='%ct %H' HEAD -- $(cmd_paths "$1") \
| sort -n | tail -1)"
if [ -z "$1" ]; then
echo "artifacts.sh: no commit touches this artifact's shipped files" >&2
exit 1
fi
echo "$1"
}
# Formatted through git rather than date(1): busybox date does not reliably
# accept `-d @<epoch>`, and git's own --date=format-local is available wherever
# git is. TZ=UTC so the value does not depend on the runner's timezone.
fmt() {
(cd "$ROOT" && TZ=UTC git show -s --format=%cd --date="format-local:$2" "$1")
}
# Leading zeros stripped so every segment is a plain integer — some version
# validators reject `08`, and a leading zero buys nothing. `0000` (midnight)
# must survive as `0`, not as the empty string.
strip0() {
printf '%s' "$1" | sed -e 's/^0*//' -e 's/^$/0/'
}
# The IDENTITY of an artifact's content: the commit its shipped files last
# changed in. This — not the tag — is what decides whether a build can be
# skipped, because the published tag is only day-precise and two different
# builds can share it.
cmd_revision() {
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
}
# The ORDERING KEY: full precision, YYYY.M.D.HHMM. Used by the extension,
# where the value is what Firefox compares to decide whether an update exists
# — two same-day builds MUST be distinguishable or the second never reaches
# anyone.
cmd_version() {
sha=$(echo "$(newest "$1")" | cut -d' ' -f2)
printf '%s.%s.%s.%s\n' \
"$(fmt "$sha" %Y)" \
"$(strip0 "$(fmt "$sha" %m)")" \
"$(strip0 "$(fmt "$sha" %d)")" \
"$(strip0 "$(fmt "$sha" %H%M)")"
}
# The PUBLISHED IMAGE TAG: day precision, YYYY.M.D. Deliberately coarser than
# the ordering key, per the operator 2026-08-28 — same-day work is not
# something worth pinning, so a second build the same day replaces the first
# rather than accumulating a tag nobody would roll back to. Safe only because
# skip decisions key on cmd_revision, never on this.
cmd_tag() {
sha=$(echo "$(newest "$1")" | cut -d' ' -f2)
printf '%s.%s.%s\n' \
"$(fmt "$sha" %Y)" \
"$(strip0 "$(fmt "$sha" %m)")" \
"$(strip0 "$(fmt "$sha" %d)")"
}
# The CONTENT IDENTITY of a published image: an immutable tag naming exactly
# what a build of this commit would produce. build.yml asks the registry for it
# and, on a hit, skips the build entirely and repoints the channel and date
# tags at the manifest that is already there (milestone 313 step 4).
#
# It is deliberately NOT either of the other two values:
# * the date tag is day-precise and last-one-wins, so two different builds
# share it — it cannot answer "is this content published?".
# * the commit sha moves on every push, so it would never hit, which is the
# redundant rebuild this exists to remove.
#
# The revision does both jobs: it is content-unique AND stable across pushes
# that did not touch the artifact.
cmd_identity() {
_art=$1
_chan=${2:-}
case "$_art" in
web|ml|agent) ;;
extension)
echo "artifacts.sh: the extension is cached as an ext-<version> Forgejo release, not an image tag — use \`version\`" >&2
exit 2 ;;
*) usage ;;
esac
for _c in $CHANNELLED; do
if [ "$_art" = "$_c" ]; then
# Refused rather than defaulted: an unqualified identity for a
# channelled artifact would let a dev image be reused as the main one.
if [ -z "$_chan" ]; then
echo "artifacts.sh: $_art bakes the channel into the image — identity needs one" >&2
exit 2
fi
printf 'r-%s-%s\n' "$(cmd_revision "$_art")" "$_chan"
return
fi
done
printf 'r-%s\n' "$(cmd_revision "$_art")"
}
[ $# -ge 2 ] || usage
case "$1" in
paths) cmd_paths "$2" ;;
revision) cmd_revision "$2" ;;
version) cmd_version "$2" ;;
tag) cmd_tag "$2" ;;
identity) cmd_identity "$2" "${3:-}" ;;
*) usage ;;
esac
+31
View File
@@ -677,3 +677,34 @@ async def test_reset_content_tagging_apply_requires_confirm_token(client, db):
)
assert resp.status_code == 200
assert (await resp.get_json())["deleted"] == 1
@pytest.mark.asyncio
async def test_trigger_reclaim_attachments_defaults_to_preview(client, monkeypatch):
"""Unlike the other maintenance triggers, this one's apply unlinks FILES —
so an empty body must mean preview, not apply."""
from backend.app.tasks import admin as admin_tasks
calls = []
monkeypatch.setattr(
admin_tasks.reclaim_orphaned_attachments_task, "delay", _fake_delay(calls)
)
resp = await client.post("/api/admin/maintenance/reclaim-attachments", json={})
assert resp.status_code == 202
assert (await resp.get_json())["task_id"] == "task-xyz"
assert calls[0][1] == {"dry_run": True}
@pytest.mark.asyncio
async def test_trigger_reclaim_attachments_threads_apply(client, monkeypatch):
from backend.app.tasks import admin as admin_tasks
calls = []
monkeypatch.setattr(
admin_tasks.reclaim_orphaned_attachments_task, "delay", _fake_delay(calls)
)
resp = await client.post(
"/api/admin/maintenance/reclaim-attachments", json={"dry_run": False},
)
assert resp.status_code == 202
assert calls[0][1] == {"dry_run": False}
+14 -1
View File
@@ -1,6 +1,6 @@
import pytest
from backend.app.models import Artist, PostAttachment
from backend.app.models import Artist, PostAttachment, attachment_download_url
pytestmark = pytest.mark.integration
@@ -33,3 +33,16 @@ async def test_download_streams_with_disposition(client, db, tmp_path):
async def test_download_404(client):
resp = await client.get("/api/attachments/999999/download")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_attachment_download_url_routes_to_the_download_endpoint(app):
"""The two serializers no longer hand-format this path (#3072) — but a
single definition is only worth having if it still matches the route. Pin
it by MATCHING against the real URL map rather than comparing to a literal:
a string equality test would pass just as happily after someone renamed the
route, which is the exact drift the helper exists to prevent."""
built = attachment_download_url(4242)
endpoint, args = app.url_map.bind("localhost").match(built)
assert endpoint == "attachments.download"
assert args == {"attachment_id": 4242}
+64 -1
View File
@@ -129,7 +129,6 @@ async def test_resolve_artist_name_dispatches_per_platform(db, monkeypatch):
("https://www.subscribestar.com/foobar", "subscribestar", "foobar"),
("https://subscribestar.adult/foobar", "subscribestar", "foobar"),
("https://www.hentai-foundry.com/user/Foo/profile", "hentaifoundry", "Foo"),
("https://www.deviantart.com/baz", "deviantart", "baz"),
("https://www.pixiv.net/users/12345", "pixiv", "12345"),
("https://www.pixiv.net/en/users/12345", "pixiv", "12345"),
])
@@ -160,6 +159,23 @@ async def test_quick_add_source_unknown_url_400(client, ext_key):
assert "known" in body
@pytest.mark.asyncio
async def test_quick_add_source_rejects_retired_deviantart(client, ext_key):
"""#3069: a DeviantArt creator URL used to derive cleanly. Now that the
platform is retired, the extension's own gate should never offer the
button — but a stale content script on an un-updated browser still can,
so the backend has to refuse it rather than create an unusable source."""
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.deviantart.com/baz"},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "unknown_platform"
assert "deviantart" not in body["known"]
@pytest.mark.asyncio
async def test_quick_add_source_invalid_url_400(client, ext_key):
resp = await client.post(
@@ -367,6 +383,53 @@ async def test_extension_manifest_returns_metadata_when_xpi_present(client, monk
assert body["sha256"] == hashlib.sha256(b"fake-xpi-content").hexdigest()
@pytest.mark.asyncio
async def test_extension_manifest_reports_the_channel_the_image_declares(
client, monkeypatch, tmp_path
):
"""The channel travels BESIDE the version, never inside it.
Folding it in as a `1.0.3499884-dev` suffix is the failure this design
exists to avoid: the extension's comparator parses each dotted segment as
an integer, so a suffixed segment collapses to 0 and every dev build
compares equal to every other — "no update available" and "I cannot read
this version" stop being distinguishable. Asserting the two are separate
keys is what keeps a future edit from merging them.
"""
(tmp_path / "fabledcurator-1.2.3.xpi").write_bytes(b"x")
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
monkeypatch.setattr(extension_module, "FC_CHANNEL", "dev")
resp = await client.get("/api/extension/manifest")
assert resp.status_code == 200
body = await resp.get_json()
assert body["channel"] == "dev"
assert body["version"] == "1.2.3"
@pytest.mark.asyncio
async def test_extension_manifest_omits_the_channel_when_the_image_declares_none(
client, monkeypatch, tmp_path
):
"""A local build, or any image from before the field existed.
The key must be ABSENT rather than present-and-empty: absence is the state
every consumer already handles (an older image conveys it by not having the
key at all), so a blank channel reuses that path instead of introducing a
second spelling of "unknown" for each reader to special-case.
"""
(tmp_path / "fabledcurator-1.2.3.xpi").write_bytes(b"x")
monkeypatch.setattr(extension_module, "XPI_DIR", tmp_path)
monkeypatch.setattr(extension_module, "FC_CHANNEL", "")
resp = await client.get("/api/extension/manifest")
assert resp.status_code == 200
body = await resp.get_json()
assert "channel" not in body
# Everything else still answers — an image with no channel is not a
# degraded one, it just cannot say which channel it came from.
assert body["installed"] is True
assert body["latest_url"] == "/extension/fabledcurator-latest.xpi"
# --- /extension/<filename> -----------------------------------------
+3 -2
View File
@@ -6,16 +6,17 @@ pytestmark = pytest.mark.integration
@pytest.mark.asyncio
async def test_platforms_returns_gs_six(client):
async def test_platforms_returns_gs_five(client):
resp = await client.get("/api/platforms")
assert resp.status_code == 200
body = await resp.get_json()
platforms = body["platforms"]
assert set(platforms.keys()) == {
"patreon", "subscribestar", "hentaifoundry",
"discord", "pixiv", "deviantart",
"discord", "pixiv",
}
assert "fanbox" not in platforms
assert "deviantart" not in platforms # retired at #3069
@pytest.mark.asyncio
+140
View File
@@ -0,0 +1,140 @@
"""`artifacts.sh identity` is what decides whether a build gets skipped.
Milestone 313 step 4: build.yml asks the registry for `<image>:<identity>` and,
on a hit, publishes NO new bytes — it repoints the channel and date tags at the
manifest already there. So the identity has to be a true name for the content.
Both ways of getting it wrong are silent at build time and only surface in
production:
* **too coarse** — two genuinely different images share an identity, so the
second one never gets built and its tags point at the first one's bytes. The
live case is FC_CHANNEL: a `dev` and a `main` build of one revision differ,
and collapsing them ships an instance that reports the wrong channel forever.
* **too fine** — the identity moves when the content did not, nothing ever
hits, and step 4 buys nothing. A commit sha would do exactly this.
The Dockerfiles are read here rather than trusted, because the coarse direction
appears the moment someone adds a build-arg without touching `CHANNELLED`.
"""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parent.parent
# Only image artifacts have an identity — the extension is cached as an
# ext-<version> Forgejo release, not a registry tag.
IMAGE_ARTIFACTS = {
"web": "Dockerfile",
"ml": "Dockerfile.ml",
"agent": "agent/Dockerfile",
}
CHANNELS = ("main", "dev")
# docker's own tag grammar: [A-Za-z0-9_][A-Za-z0-9._-]{0,127}
_TAG = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$")
# `ARG FC_CHANNEL` in a Dockerfile means build.yml passes a per-channel value
# in, so the channel is part of what the image IS.
_ARG_CHANNEL = re.compile(r"^\s*ARG\s+FC_CHANNEL\b", re.MULTILINE)
def identity(artifact: str, channel: str | None = None) -> subprocess.CompletedProcess:
cmd = ["sh", str(ROOT / "scripts" / "artifacts.sh"), "identity", artifact]
if channel is not None:
cmd.append(channel)
return subprocess.run(cmd, capture_output=True, text=True, cwd=ROOT)
def ok(artifact: str, channel: str | None = None) -> str:
proc = identity(artifact, channel)
assert proc.returncode == 0, f"identity {artifact} {channel}: {proc.stderr}"
return proc.stdout.strip()
def bakes_the_channel(artifact: str) -> bool:
return bool(_ARG_CHANNEL.search((ROOT / IMAGE_ARTIFACTS[artifact]).read_text()))
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_channel_dependence_matches_the_dockerfile(artifact):
"""The coarse direction, caught at its source.
Whether the channel belongs in the identity is not a preference — it is
dictated by whether the Dockerfile takes it as a build-arg. Adding an
`ARG FC_CHANNEL` to another image without adding it to `CHANNELLED` would
make its dev and main builds collide, and nothing else would notice.
"""
per_channel = {c: ok(artifact, c) for c in CHANNELS}
differs = len(set(per_channel.values())) > 1
if bakes_the_channel(artifact):
assert differs, (
f"{IMAGE_ARTIFACTS[artifact]} declares ARG FC_CHANNEL, so a dev "
f"build and a main build of one revision are different images — "
f"but both derive the identity {per_channel['main']!r}. The main "
f"build would reuse the dev image and report the wrong channel. "
f"Add {artifact!r} to CHANNELLED in scripts/artifacts.sh."
)
else:
assert not differs, (
f"{IMAGE_ARTIFACTS[artifact]} takes no channel build-arg, so one "
f"revision is one image and a merge to main should reuse what dev "
f"already built — but the identity differs per channel "
f"({per_channel}), so every merge rebuilds it for nothing. Remove "
f"{artifact!r} from CHANNELLED in scripts/artifacts.sh."
)
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_identity_tracks_the_artifacts_own_revision(artifact):
"""The fine direction: the identity must be the revision, not the push.
`revision` is the commit this artifact's shipped files last changed in, so
it holds still across pushes that did not touch it. Anything derived from
HEAD instead would move every push and never hit the registry.
"""
rev = subprocess.run(
["sh", str(ROOT / "scripts" / "artifacts.sh"), "revision", artifact],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout.strip()
value = ok(artifact, "main")
assert rev and rev in value, (
f"identity {value!r} does not contain the {artifact} revision {rev!r}"
)
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_identity_is_a_legal_docker_tag(artifact):
"""It is pushed as a tag, so an illegal one fails at the registry — after
the build has already run."""
for channel in CHANNELS:
value = ok(artifact, channel)
assert _TAG.match(value), f"{value!r} is not a valid docker tag"
def test_a_channelled_artifact_refuses_an_unqualified_identity():
"""Refusing beats defaulting. If `identity web` quietly returned the
unqualified `r-<rev>`, a workflow that forgot to pass the channel would
publish one image under a name both channels then reuse — the exact
collision the CHANNELLED list exists to prevent, reintroduced by an
omission rather than by an edit."""
proc = identity("web")
assert proc.returncode != 0, (
"identity web returned a value with no channel: "
f"{proc.stdout.strip()!r}"
)
def test_the_extension_has_no_image_identity():
"""It is cached as an ext-<version> release asset, and its cache key is the
version. Answering with a plausible image tag would invite a second,
divergent cache."""
proc = identity("extension", "main")
assert proc.returncode != 0
assert "ext-" in proc.stderr
+168
View File
@@ -0,0 +1,168 @@
"""`scripts/artifacts.sh` path sets must match what the Dockerfiles copy.
Each published artifact's version derives from the newest commit touching its
own shipped file set (milestone 313). The whole scheme rests on those sets
being right, and both ways of being wrong are silent:
* **too narrow** — a file ships but is not in the set, so the version does not
move when the content does, and a pin serves stale bytes. This is the
dangerous direction and the one this module exists for.
* **too wide** — a file is in the set but never reaches the image, so the
artifact re-versions and rebuilds for a change it does not ship.
Nothing else notices either. The version still derives, CI still goes green,
and the mismatch only surfaces as "I pinned that build and got the wrong
bytes". So the Dockerfiles are read here and compared against the declaration.
"""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parent.parent
# artifact -> (dockerfile, build context relative to the repo root)
ARTIFACTS = {
"web": ("Dockerfile", ""),
"ml": ("Dockerfile.ml", ""),
"agent": ("agent/Dockerfile", "agent"),
}
# COPY --from=<stage> copies from an earlier build stage, not from the build
# context, so its source is not a repo path and cannot be in a path set.
_COPY = re.compile(r"^\s*COPY\s+(?!--from=)(?P<args>.+)$", re.MULTILINE)
def declared_paths(artifact: str) -> list[str]:
out = subprocess.run(
["sh", str(ROOT / "scripts" / "artifacts.sh"), "paths", artifact],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout
return out.split()
def includes(artifact: str) -> list[str]:
"""The set minus its `:(exclude)…` entries."""
return [p for p in declared_paths(artifact) if not p.startswith(":(exclude)")]
def copy_sources(dockerfile: str, context: str) -> list[str]:
"""Repo-relative sources of every context COPY in a Dockerfile."""
text = (ROOT / dockerfile).read_text()
sources: list[str] = []
for m in _COPY.finditer(text):
args = m.group("args").split()
# Last arg is the destination; everything before it is a source.
for src in args[:-1]:
# `frontend/package-lock.json*` — the glob is an optional-file
# idiom; the directory it sits in is what matters for coverage.
src = src.rstrip("*")
sources.append(f"{context}/{src}" if context else src)
return sources
def covered_by(path: str, include: str) -> bool:
"""`path` ships if an include names it or one of its ancestors."""
path = path.rstrip("/").lstrip("./")
include = include.rstrip("/")
return path == include or path.startswith(include + "/")
@pytest.mark.parametrize("artifact", sorted(ARTIFACTS))
def test_every_copied_path_is_in_the_artifacts_path_set(artifact):
"""The too-narrow direction — the one that serves stale bytes on a pin."""
dockerfile, context = ARTIFACTS[artifact]
inc = includes(artifact)
for src in copy_sources(dockerfile, context):
assert any(covered_by(src, i) for i in inc), (
f"{dockerfile} copies {src!r} into the {artifact} image, but no "
f"include in scripts/artifacts.sh covers it. The {artifact} "
f"version will not move when that file changes, so a pinned build "
f"will serve stale bytes. Add it to the path set.\n"
f" declared includes: {inc}"
)
@pytest.mark.parametrize("artifact", sorted(ARTIFACTS))
def test_the_dockerfile_itself_is_in_the_path_set(artifact):
"""Changing a base image or a RUN changes the artifact as surely as
changing a source file, so each set must include its own Dockerfile."""
dockerfile, _ = ARTIFACTS[artifact]
assert any(covered_by(dockerfile, i) for i in includes(artifact)), (
f"{dockerfile} is not in the {artifact} path set — a base-image bump "
f"would not move the version."
)
def test_the_web_image_versions_on_an_extension_change():
"""The web image bundles the signed XPI, so the extension's packaged files
are part of what it ships. Miss this and `:latest` serves a NEW extension
under an unchanged web version — a pin that quietly disagrees with itself.
"""
inc = includes("web")
assert any(covered_by("extension/background/background.js", i) for i in inc), (
"the web path set does not cover the extension's packaged files, but "
"build.yml downloads the signed XPI into frontend/public/extension/ "
"before the docker build"
)
def test_the_web_image_versions_on_a_version_derivation_change():
"""packaging.sh ships in no image, yet it belongs in the sets that bundle
the XPI — because it decides the version string build.yml stamps into the
packaged manifest.json. Changing the derivation changes the shipped bytes.
Left out, milestone 313 step 4 turns it silent: the new version misses the
ext-<version> cache and gets signed, while web's revision has not moved, so
the reuse path republishes the old image and the fresh signature is
orphaned. Guarded for web and the extension both, since web bundles what
the extension produces.
"""
for artifact in ("extension", "web"):
inc = includes(artifact)
excluded = [
p[len(":(exclude)"):] for p in declared_paths(artifact)
if p.startswith(":(exclude)")
]
path = "extension/scripts/packaging.sh"
assert any(covered_by(path, i) for i in inc), (
f"{path} is not in the {artifact} path set"
)
assert not any(
covered_by(path, e.rstrip("*").rstrip("/")) for e in excluded
), (
f"{path} is excluded from the {artifact} path set, so a change to "
f"how the version is derived would not move the version — and "
f"step 4 would reuse the image that carries the old one"
)
@pytest.mark.parametrize(
"artifact, path",
[
# Deliberate exclusions — the too-wide direction. Each of these lives
# beside shipped code but never reaches an image, and including it
# would re-version the artifact for a change it does not carry.
("agent", "agent/README.md"),
("agent", "agent/ruff.toml"),
("agent", "agent/docker-compose.yml"),
# vite builds from src/, index.html and public/; it never reads test/,
# so a frontend test change cannot reach `dist`.
("web", "frontend/test/gallery.spec.js"),
],
)
def test_files_that_never_reach_an_image_do_not_version_it(artifact, path):
paths = declared_paths(artifact)
excluded = [p[len(":(exclude)"):] for p in paths if p.startswith(":(exclude)")]
inc = [p for p in paths if not p.startswith(":(exclude)")]
included = any(covered_by(path, i) for i in inc)
exempted = any(covered_by(path, e.rstrip("*").rstrip("/")) for e in excluded)
assert not included or exempted, (
f"{path} is in the {artifact} path set but is not copied into the "
f"image — it would re-version and rebuild {artifact} for a change it "
f"does not ship."
)
+1 -1
View File
@@ -154,7 +154,7 @@ async def test_list_platform_filter_excludes_no_source(db):
@pytest.mark.asyncio
async def test_list_platform_filter_excludes_wrong_platform(db):
a = await _seed_artist(db, "alice-wplat")
await _seed_source(db, a.id, "deviantart", "https://d/alice-wp")
await _seed_source(db, a.id, "discord", "https://d/alice-wp")
await db.commit()
page = await ArtistDirectoryService(db).list_artists(
+339
View File
@@ -5,6 +5,7 @@ side effects use tmp_path. Assertions on mutated rows use COLUMN
SELECTS per reference_async_coredml_test_assertions — never
re-read ORM attributes after a service mutates and re-fetches.
"""
import os
from datetime import UTC, datetime
import pytest
@@ -68,6 +69,8 @@ def test_project_artist_cascade_returns_zeroes_for_empty_artist(db_sync):
assert result["artist"]["slug"] == "empty"
assert result["projected"] == {
"images": 0,
"posts": 0,
"attachments": 0,
"sources": 0,
"thumbs": 0,
"import_tasks": 0,
@@ -92,6 +95,87 @@ def test_project_artist_cascade_counts_images_and_thumbs_and_bytes(db_sync, tmp_
assert result["projected"]["bytes_on_disk"] == 3500
def test_project_artist_cascade_counts_posts_and_attachments(db_sync, tmp_path):
"""The body-only artist: zero images, but posts and attachments that the
apply destroys. Previewing this as `images: 0` alone is what made a
content-only artist read as an empty one (#3067)."""
a = _make_artist(db_sync, slug="bodyonly")
p1 = Post(artist_id=a.id, external_post_id="bo-1", description="a body")
p2 = Post(artist_id=a.id, external_post_id="bo-2", description="another")
db_sync.add_all([p1, p2])
db_sync.flush()
db_sync.add(PostAttachment(
post_id=p1.id, artist_id=a.id, sha256="b0d1".ljust(64, "0"),
path="/store/b0d1/a.pdf", original_filename="a.pdf",
ext=".pdf", size_bytes=5,
))
# artist_id NULL, reachable only through its post — the second arm of
# _artist_attachments_conditions.
db_sync.add(PostAttachment(
post_id=p2.id, artist_id=None, sha256="b0d2".ljust(64, "0"),
path="/store/b0d2/b.pdf", original_filename="b.pdf",
ext=".pdf", size_bytes=5,
))
db_sync.commit()
projected = cleanup_service.project_artist_cascade(
db_sync, slug="bodyonly",
)["projected"]
assert projected["images"] == 0
assert projected["posts"] == 2
assert projected["attachments"] == 2
def test_artist_cascade_preview_matches_apply(db_sync, tmp_path):
"""Rule 93: the preview's numbers must be what the apply actually does.
Guards the drift directly rather than trusting that both halves happen to
use the same predicate — the preview and apply are asserted against each
other on one artist carrying all three row kinds.
"""
a = _make_artist(db_sync, slug="parity")
for i in range(3):
f = tmp_path / f"par{i}.jpg"
f.write_bytes(b"x")
_make_image(
db_sync, artist=a, path=str(f), sha256=f"{i:064x}", size=10,
)
posts = [
Post(artist_id=a.id, external_post_id=f"par-{i}") for i in range(4)
]
db_sync.add_all(posts)
db_sync.flush()
for i, p in enumerate(posts[:2]):
db_sync.add(PostAttachment(
post_id=p.id, artist_id=a.id, sha256=f"par{i}".ljust(64, "0"),
path=f"/store/par{i}/f.zip", original_filename="f.zip",
ext=".zip", size_bytes=9,
))
db_sync.commit()
artist_id = a.id
projected = cleanup_service.project_artist_cascade(
db_sync, slug="parity",
)["projected"]
summary = cleanup_service.delete_artist_cascade(
db_sync, artist_id=artist_id, images_root=tmp_path,
)["summary"]
assert projected["images"] == summary["images_deleted"] == 3
assert projected["posts"] == summary["posts_deleted"] == 4
assert projected["attachments"] == summary["attachments_deleted"] == 2
# And the apply really did remove them — a matching pair of numbers is
# worth nothing if neither half touched the DB.
assert db_sync.execute(
select(func.count(Post.id)).where(Post.artist_id == artist_id)
).scalar_one() == 0
assert db_sync.execute(
select(func.count(PostAttachment.id))
.where(PostAttachment.artist_id == artist_id)
).scalar_one() == 0
def test_project_artist_cascade_raises_on_unknown_slug(db_sync):
with pytest.raises(LookupError):
cleanup_service.project_artist_cascade(db_sync, slug="nope")
@@ -294,6 +378,88 @@ def test_delete_artist_cascade_idempotent_on_missing(db_sync, tmp_path):
assert result["summary"]["images_deleted"] == 0
def test_delete_artist_cascade_survives_same_sha_on_two_posts(db_sync, tmp_path):
"""Same file attached to two of the artist's posts must not abort the delete.
Left to the cascade this raises: artist delete CASCADEs to Post, which SET
NULLs post_attachment.post_id, and `uq_post_attachment_null_post_sha`
(sha256 alone, WHERE post_id IS NULL) then rejects the second row. That's an
ordinary shape — _capture_attachment writes one row per post over one
sha-addressed blob by design. Also covers the NULL-artist_id arm of the
delete predicate: the second row has no artist_id, only a post that does.
"""
a = _make_artist(db_sync, slug="casatt")
p1 = Post(artist_id=a.id, external_post_id="att-p1")
p2 = Post(artist_id=a.id, external_post_id="att-p2")
db_sync.add_all([p1, p2])
db_sync.flush()
shared_sha = "ca5a".ljust(64, "0")
db_sync.add(PostAttachment(
post_id=p1.id, artist_id=a.id, sha256=shared_sha,
path="/store/ca5a/bundle.zip", original_filename="bundle.zip",
ext=".zip", size_bytes=7,
))
db_sync.add(PostAttachment(
post_id=p2.id, artist_id=None, sha256=shared_sha,
path="/store/ca5a/bundle.zip", original_filename="bundle.zip",
ext=".zip", size_bytes=7,
))
db_sync.commit()
artist_id = a.id
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=artist_id, images_root=tmp_path,
)
assert result["summary"]["attachments_deleted"] == 2
assert db_sync.execute(
select(func.count(Artist.id)).where(Artist.id == artist_id)
).scalar_one() == 0
assert db_sync.execute(
select(func.count(PostAttachment.id))
.where(PostAttachment.sha256 == shared_sha)
).scalar_one() == 0
def test_delete_artist_cascade_keeps_unrelated_null_post_attachment(
db_sync, tmp_path,
):
"""A filesystem-import row (post_id NULL) sharing the sha is the other way
this collides — and it must SURVIVE: it belongs to no artist, so the
cascade has no claim on it."""
a = _make_artist(db_sync, slug="casorph")
p = Post(artist_id=a.id, external_post_id="orph-p1")
db_sync.add(p)
db_sync.flush()
sha = "0rfa".ljust(64, "0")
standalone = PostAttachment(
post_id=None, artist_id=None, sha256=sha,
path="/store/0rfa/manual.pdf", original_filename="manual.pdf",
ext=".pdf", size_bytes=3,
)
db_sync.add(standalone)
db_sync.add(PostAttachment(
post_id=p.id, artist_id=a.id, sha256=sha,
path="/store/0rfa/manual.pdf", original_filename="manual.pdf",
ext=".pdf", size_bytes=3,
))
db_sync.commit()
artist_id, standalone_id = a.id, standalone.id
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=artist_id, images_root=tmp_path,
)
assert result["summary"]["attachments_deleted"] == 1
surviving = db_sync.execute(
select(PostAttachment.id, PostAttachment.post_id)
.where(PostAttachment.sha256 == sha)
).all()
assert surviving == [(standalone_id, None)]
# --- delete_images --------------------------------------------------
@@ -916,3 +1082,176 @@ def test_reconcile_preserves_from_attachment_on_provenance_collision(db_sync, tm
.where(ImageProvenance.image_record_id == img_id)
).all()
assert rows == [(native_id, att_id)]
# --- reclaim_orphaned_attachments -----------------------------------
def _store_blob(root, sha, *, ext=".pdf", age_hours=48, data=b"blob"):
"""Write a file into the sha-addressed attachment store, aged past the
min-age guard by default."""
d = root / "attachments" / sha[:3]
d.mkdir(parents=True, exist_ok=True)
p = d / f"{sha}{ext}"
p.write_bytes(data)
old = datetime.now(UTC).timestamp() - age_hours * 3600
os.utime(p, (old, old))
return p
def _attachment(db_sync, *, sha, post=None, artist=None):
att = PostAttachment(
post_id=post.id if post else None,
artist_id=artist.id if artist else None,
sha256=sha, path=f"/store/{sha[:3]}/f.pdf",
original_filename="f.pdf", ext=".pdf", size_bytes=4,
)
db_sync.add(att)
db_sync.flush()
return att
def test_reclaim_attachments_dry_run_projects_without_mutating(db_sync, tmp_path):
a = _make_artist(db_sync, slug="recl-dry")
p = Post(artist_id=a.id, external_post_id="rd-1")
db_sync.add(p)
db_sync.flush()
kept_sha, orphan_sha = "aa11".ljust(64, "0"), "bb22".ljust(64, "0")
_attachment(db_sync, sha=kept_sha, post=p, artist=a)
_attachment(db_sync, sha=orphan_sha) # both FKs NULL → orphan
db_sync.commit()
kept_blob = _store_blob(tmp_path, kept_sha)
orphan_blob = _store_blob(tmp_path, orphan_sha)
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=True,
)
assert result["rows"] == 1
assert result["files"] == 1
assert result["bytes"] == orphan_blob.stat().st_size
# Nothing actually happened.
assert kept_blob.exists() and orphan_blob.exists()
assert db_sync.execute(
select(func.count(PostAttachment.id))
).scalar_one() == 2
def test_reclaim_attachments_apply_deletes_rows_and_unlinks_blobs(db_sync, tmp_path):
a = _make_artist(db_sync, slug="recl-apply")
p = Post(artist_id=a.id, external_post_id="ra-1")
db_sync.add(p)
db_sync.flush()
kept_sha, orphan_sha = "cc33".ljust(64, "0"), "dd44".ljust(64, "0")
_attachment(db_sync, sha=kept_sha, post=p, artist=a)
_attachment(db_sync, sha=orphan_sha)
db_sync.commit()
kept_blob = _store_blob(tmp_path, kept_sha)
orphan_blob = _store_blob(tmp_path, orphan_sha)
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
assert result["rows"] == 1
assert result["files"] == 1
assert kept_blob.exists() # still referenced
assert not orphan_blob.exists() # nothing points at it any more
surviving = db_sync.execute(select(PostAttachment.sha256)).scalars().all()
assert surviving == [kept_sha]
def test_reclaim_attachments_preview_matches_apply(db_sync, tmp_path):
"""Rule 93 — the dry-run's numbers are what the apply does. The projection
has to negate the orphan predicate to be honest about blobs the delete is
about to free, so this is the assertion that catches getting that backwards.
"""
orphan_sha = "ee55".ljust(64, "0")
_attachment(db_sync, sha=orphan_sha)
db_sync.commit()
_store_blob(tmp_path, orphan_sha)
projected = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=True,
)
applied = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
for key in ("rows", "files", "bytes"):
assert projected[key] == applied[key], key
assert applied["rows"] == 1 and applied["files"] == 1
def test_reclaim_attachments_keeps_shared_blob_while_any_row_remains(db_sync, tmp_path):
"""The refcount case this whole sweep exists for: one sha-addressed blob
backs several rows, so deleting SOME of them must not free the file."""
a = _make_artist(db_sync, slug="recl-shared")
p = Post(artist_id=a.id, external_post_id="rs-1")
db_sync.add(p)
db_sync.flush()
sha = "ff66".ljust(64, "0")
_attachment(db_sync, sha=sha, post=p, artist=a) # attributed — survives
_attachment(db_sync, sha=sha) # orphan — deleted
db_sync.commit()
blob = _store_blob(tmp_path, sha)
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
assert result["rows"] == 1 # the orphan row went
assert result["files"] == 0 # the blob did NOT
assert blob.exists()
def test_reclaim_attachments_spares_filesystem_import_rows(db_sync, tmp_path):
"""post_id NULL with an artist_id is the deliberate filesystem-import shape
(importer._capture_attachment), not an orphan — it is still attributed."""
a = _make_artist(db_sync, slug="recl-fsimport")
sha = "1177".ljust(64, "0")
_attachment(db_sync, sha=sha, artist=a) # post NULL, artist set
db_sync.commit()
blob = _store_blob(tmp_path, sha)
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
assert result["rows"] == 0
assert result["files"] == 0
assert blob.exists()
assert db_sync.execute(
select(func.count(PostAttachment.id))
).scalar_one() == 1
def test_reclaim_attachments_skips_recent_and_staging_files(db_sync, tmp_path):
"""A blob is written BEFORE its row commits, so a just-stored file with no
row is in-flight, not orphaned. `.partial` staging files belong to
cleanup_orphaned_temp_files and must be left alone either way."""
fresh_sha, staged_sha = "2288".ljust(64, "0"), "3399".ljust(64, "0")
fresh = _store_blob(tmp_path, fresh_sha, age_hours=0)
staged = _store_blob(tmp_path, staged_sha, ext=".pdf.partial")
db_sync.commit()
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
assert result["files"] == 0
assert result["skipped_recent"] == 1
assert fresh.exists() and staged.exists()
def test_reclaim_attachments_ignores_non_sha_named_files(db_sync, tmp_path):
"""The walk must only judge files it can identify as store blobs — anything
else under the root is none of its business."""
d = tmp_path / "attachments" / "zzz"
d.mkdir(parents=True)
stray = d / "notes.txt"
stray.write_text("not a blob")
old = datetime.now(UTC).timestamp() - 48 * 3600
os.utime(stray, (old, old))
result = cleanup_service.reclaim_orphaned_attachments(
db_sync, images_root=tmp_path, dry_run=False,
)
assert result["files"] == 0
assert stray.exists()
+1 -1
View File
@@ -19,7 +19,7 @@ def test_native_platforms():
def test_gallery_dl_platforms_are_not_native():
# The platforms still served by gallery-dl must NOT route to the native
# ingester — guards an accidental over-broad migration.
for platform in ("hentaifoundry", "discord", "deviantart"):
for platform in ("hentaifoundry", "discord"):
assert uses_native_ingester(platform) is False
+119
View File
@@ -0,0 +1,119 @@
"""`insert_image_tags` — the shared bulk write behind the WIP-title backfill and
both auto-apply sweeps (#3072).
The sweeps previously issued one INSERT per applied tag from inside their
per-image loop; they now hand this helper a chunk's worth of rows. That makes
this function the single place three writers can be wrong at once, so it is
tested directly rather than only through its callers.
"""
import pytest
from sqlalchemy import select
from backend.app.models import ImageRecord, Tag, TagKind
from backend.app.models.tag import image_tag
from backend.app.services.image_tag_apply import insert_image_tags
pytestmark = pytest.mark.integration
_N = 0
def _img(db_sync):
global _N
_N += 1
rec = ImageRecord(
path=f"/images/ita/{_N}.jpg", sha256=f"a{_N:063d}",
size_bytes=1, mime="image/jpeg", width=1, height=1,
origin="imported_filesystem", integrity_status="unknown",
)
db_sync.add(rec)
db_sync.flush()
return rec
def _tag(db_sync, name):
t = Tag(name=name, kind=TagKind.general)
db_sync.add(t)
db_sync.flush()
return t
def _rows(db_sync, tag_id):
"""(image_record_id, source) pairs currently carrying `tag_id`."""
return dict(db_sync.execute(
select(image_tag.c.image_record_id, image_tag.c.source)
.where(image_tag.c.tag_id == tag_id)
).all())
def _row(image_record_id, tag_id, source):
return {
"image_record_id": image_record_id, "tag_id": tag_id, "source": source,
}
def test_inserts_every_row_in_one_call(db_sync):
t = _tag(db_sync, "ita-basic")
imgs = [_img(db_sync) for _ in range(3)]
insert_image_tags(
db_sync, [_row(i.id, t.id, "head_auto") for i in imgs]
)
assert _rows(db_sync, t.id) == {i.id: "head_auto" for i in imgs}
def test_spans_several_tags_in_a_single_call(db_sync):
"""The sweeps accumulate across ALL heads before flushing, so one call
carries rows for different tags. A per-tag implementation would drop all
but the first."""
t1, t2 = _tag(db_sync, "ita-multi-1"), _tag(db_sync, "ita-multi-2")
a, b = _img(db_sync), _img(db_sync)
insert_image_tags(db_sync, [
_row(a.id, t1.id, "head_auto"), _row(b.id, t1.id, "head_auto"),
_row(a.id, t2.id, "head_auto"),
])
assert _rows(db_sync, t1.id) == {a.id: "head_auto", b.id: "head_auto"}
assert _rows(db_sync, t2.id) == {a.id: "head_auto"}
def test_an_existing_tag_keeps_its_original_source(db_sync):
"""THE assertion this helper exists for. A sweep re-running over an image
the operator tagged by hand must not restamp it as machine-applied — that
would silently poison the head's own training data, which excludes the
auto sources. ON CONFLICT DO NOTHING, never DO UPDATE."""
t = _tag(db_sync, "ita-manual")
rec = _img(db_sync)
insert_image_tags(db_sync, [_row(rec.id, t.id, "manual")])
insert_image_tags(db_sync, [_row(rec.id, t.id, "head_auto")])
assert _rows(db_sync, t.id) == {rec.id: "manual"}
def test_a_repeat_within_one_call_does_not_raise(db_sync):
"""Two heads can both fire on the same (image, tag) inside one chunk. The
conflict is resolved by the statement, not by the caller de-duplicating."""
t = _tag(db_sync, "ita-dupe")
rec = _img(db_sync)
insert_image_tags(db_sync, [
_row(rec.id, t.id, "head_auto"), _row(rec.id, t.id, "head_auto"),
])
assert _rows(db_sync, t.id) == {rec.id: "head_auto"}
def test_more_rows_than_the_chunk_size_all_land(db_sync):
"""The chunk exists to stay under Postgres' 65535 bound-parameter ceiling.
Driven with a tiny chunk so the split is real rather than theoretical — at
the 5000 default no test would ever reach a second statement."""
t = _tag(db_sync, "ita-chunked")
imgs = [_img(db_sync) for _ in range(7)]
insert_image_tags(
db_sync, [_row(i.id, t.id, "head_auto") for i in imgs], chunk=2
)
assert _rows(db_sync, t.id) == {i.id: "head_auto" for i in imgs}
def test_no_rows_is_a_no_op(db_sync):
"""A dry-run chunk, or a chunk where every candidate was already skipped,
hands over an empty list. `.values([])` is a SQL error, so the empty case
must never reach the statement."""
insert_image_tags(db_sync, [])
+14
View File
@@ -778,3 +778,17 @@ def test_vacuum_analyze_runs_over_high_churn_tables():
result = vacuum_analyze.apply().get()
assert result["vacuumed"] == list(VACUUM_TABLES)
def test_reclaim_attachments_stuck_threshold_exceeds_hard_time_limit():
"""#883's invariant, applied to the attachment reclaim: a task whose stall
threshold is under its own hard limit gets phantom-flagged 'RecoverySweep'
while it is still healthily running."""
from backend.app.tasks.admin import reclaim_orphaned_attachments_task
from backend.app.tasks.maintenance import TASK_STUCK_THRESHOLD_MINUTES
hard_minutes = reclaim_orphaned_attachments_task.time_limit / 60
override = TASK_STUCK_THRESHOLD_MINUTES[
"backend.app.tasks.admin.reclaim_orphaned_attachments_task"
]
assert override >= hard_minutes
+1 -1
View File
@@ -9,8 +9,8 @@ pytestmark = pytest.mark.integration
def test_non_serialized_platform_has_no_lock():
# gallery-dl platforms aren't capped — they get no lock at all.
assert platform_lock("deviantart", ttl_seconds=60) is None
assert platform_lock("hentaifoundry", ttl_seconds=60) is None
assert platform_lock("discord", ttl_seconds=60) is None
def test_subscribestar_is_serialized():
+11 -2
View File
@@ -11,10 +11,10 @@ from backend.app.services.platforms import (
)
def test_known_platform_keys_is_gs_six():
def test_known_platform_keys_is_gs_five():
assert known_platform_keys() == frozenset({
"patreon", "subscribestar", "hentaifoundry",
"discord", "pixiv", "deviantart",
"discord", "pixiv",
})
@@ -23,6 +23,15 @@ def test_fanbox_not_in_registry():
assert "fanbox" not in PLATFORMS
def test_deviantart_is_retired():
# #3069 executed the 2026-07-05 drop decision (FC downloaders = ART-
# DEDICATED services only). The registry is what /api/platforms, the
# source validator and the credential validator all read, so its absence
# here is what actually retires the platform everywhere else.
assert "deviantart" not in PLATFORMS
assert auth_type_for("deviantart") is None
def test_auth_type_for_known_and_unknown():
assert auth_type_for("patreon") == "cookies"
assert auth_type_for("discord") == "token"
+2 -2
View File
@@ -169,7 +169,7 @@ async def test_scroll_filters_by_artist(db):
async def test_scroll_filters_by_platform(db):
artist = await _seed_artist(db, "alice-platf")
src_p = await _seed_source(db, artist.id, "patreon", "https://p/alice-pp")
src_d = await _seed_source(db, artist.id, "deviantart", "https://d/alice-dd")
src_d = await _seed_source(db, artist.id, "discord", "https://d/alice-dd")
now = datetime.now(UTC)
pp = await _seed_post(db, src_p.id, external_id="PP", post_date=now)
await _seed_post(db, src_d.id, external_id="PD", post_date=now)
@@ -234,7 +234,7 @@ async def test_scroll_combined_artist_and_platform(db):
alice = await _seed_artist(db, "alice-combo")
bob = await _seed_artist(db, "bob-combo")
src_alice_patreon = await _seed_source(db, alice.id, "patreon", "https://p/alice-c")
src_alice_da = await _seed_source(db, alice.id, "deviantart", "https://d/alice-c")
src_alice_da = await _seed_source(db, alice.id, "discord", "https://d/alice-c")
src_bob_patreon = await _seed_source(db, bob.id, "patreon", "https://p/bob-c")
now = datetime.now(UTC)
target = await _seed_post(
+324
View File
@@ -0,0 +1,324 @@
"""Layer-2 auto-refetch remediation — `services/refetch_service.py` (#3071).
This module was the only one under `backend/app/services/` with no test
file, which matters more than a coverage gap normally would: it runs
UNATTENDED off the recovery sweep (`tasks/maintenance.py`, gated by
FC_AUTO_REFETCH_CORRUPT) and it DELETES a file from disk before asking a
downloader for a fresh copy. The frontend cites it by name as the reason
the Import tab could be retired at all (`stores/import.js`: imports
"heal themselves").
`test_api_import_admin.py` already drives the happy path end-to-end
through `POST /api/import/tasks/<id>/refetch` — file deleted, task
flagged, one dispatch, second attempt a no-op. What it CANNOT reach is
the branching inside `resolve_refetch_source`, and it never proves the
negative that actually protects the operator's data: that a file whose
source does NOT resolve is still on disk afterwards. Its `no_source`
case points at a path that never existed, so nothing survives to check.
Those two things are what this module covers.
"""
import json
from pathlib import Path
import pytest
from sqlalchemy import select
from backend.app.models import Artist, ImportBatch, ImportTask, Source
from backend.app.services.refetch_service import (
attempt_refetch,
resolve_refetch_source,
)
pytestmark = pytest.mark.integration
# --- fixtures / helpers ----------------------------------------------------
@pytest.fixture
def import_root(tmp_path):
root = tmp_path / "import"
root.mkdir()
return root
def _media(import_root: Path, artist_dir: str, name: str = "post.jpg") -> Path:
"""A corrupt-import stand-in at import_root/<artist_dir>/<name>."""
d = import_root / artist_dir if artist_dir else import_root
d.mkdir(parents=True, exist_ok=True)
m = d / name
m.write_bytes(b"corrupt-bytes")
return m
def _sidecar(media: Path, payload) -> Path:
"""gallery-dl writes `<stem>.json` beside the media file."""
sc = media.with_suffix(".json")
sc.write_text(payload if isinstance(payload, str) else json.dumps(payload))
return sc
def _artist(session, name: str) -> Artist:
a = Artist(name=name, slug=name.lower())
session.add(a)
session.flush()
return a
def _source(session, artist, platform="patreon", url=None, enabled=True) -> Source:
s = Source(
artist_id=artist.id,
platform=platform,
url=url if url is not None else f"https://www.{platform}.com/{artist.slug}",
enabled=enabled,
config_overrides={},
)
session.add(s)
session.flush()
return s
def _task(session, media: Path, refetched: bool = False) -> ImportTask:
batch = ImportBatch(
triggered_by="manual", source_path=str(media.parent), scan_mode="quick",
)
session.add(batch)
session.flush()
t = ImportTask(
batch_id=batch.id, source_path=str(media), task_type="media",
status="failed", refetched=refetched,
)
session.add(t)
session.flush()
return t
@pytest.fixture
def no_dispatch(monkeypatch):
"""Capture download_source.delay instead of queueing a real re-check.
refetch_service imports the task lazily (inside attempt_refetch, to
dodge a tasks->services->tasks cycle), so patching the attribute on
the module is enough — the import resolves at call time.
"""
from backend.app.tasks import download as download_mod
calls = []
monkeypatch.setattr(download_mod.download_source, "delay", calls.append)
return calls
# --- resolve_refetch_source: what counts as re-pollable --------------------
def test_resolve_finds_enabled_source_matching_the_sidecar_platform(db_sync, import_root):
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon", "post_id": 1})
artist = _artist(db_sync, "Alice")
src = _source(db_sync, artist)
assert resolve_refetch_source(db_sync, str(m), import_root).id == src.id
def test_resolve_skips_a_disabled_source(db_sync, import_root):
"""A disabled Source is not re-pollable: the operator turned it off,
and a sweep must not reach past that to delete their file."""
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"), enabled=False)
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_rejects_a_synthetic_sidecar_anchor_url(db_sync, import_root):
"""`sidecar:<platform>:<slug>` is a bookkeeping anchor for files that
arrived on disk, not a feed. Re-polling it is impossible, so it must
not qualify — otherwise the file is deleted for a fetch that can
never happen."""
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"), url="sidecar:patreon:alice")
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_needs_a_source_on_the_sidecars_own_platform(db_sync, import_root):
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"), platform="pixiv")
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_picks_the_lowest_id_when_several_sources_qualify(db_sync, import_root):
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
artist = _artist(db_sync, "Alice")
first = _source(db_sync, artist, url="https://www.patreon.com/alice-one")
_source(db_sync, artist, url="https://www.patreon.com/alice-two")
# Deterministic choice, not "whichever the planner returned first" —
# the pick decides which downloader runs.
assert resolve_refetch_source(db_sync, str(m), import_root).id == first.id
def test_resolve_reads_the_gallery_dl_numbered_sidecar(db_sync, import_root):
"""gallery-dl prefixes media with `NN_` for in-post ordering but
writes the sidecar under the UNPREFIXED stem. Refetch resolves real
downloaded files, so it has to follow that convention."""
m = _media(import_root, "Alice", name="01_post.jpg")
(import_root / "Alice" / "post.json").write_text(json.dumps({"category": "patreon"}))
src = _source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root).id == src.id
# --- resolve_refetch_source: every way it declines -------------------------
def test_resolve_declines_without_a_sidecar(db_sync, import_root):
m = _media(import_root, "Alice")
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_declines_on_unreadable_sidecar_json(db_sync, import_root):
m = _media(import_root, "Alice")
_sidecar(m, "{not valid json")
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_declines_when_the_sidecar_is_not_an_object(db_sync, import_root):
# A bare JSON list parses fine but has no `category` to read.
m = _media(import_root, "Alice")
_sidecar(m, ["patreon"])
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_declines_when_the_sidecar_names_no_platform(db_sync, import_root):
m = _media(import_root, "Alice")
_sidecar(m, {"post_id": 1})
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_declines_for_a_file_sitting_directly_in_import_root(db_sync, import_root):
"""No artist folder means no artist bucket to resolve — the
filesystem-only drop case."""
m = _media(import_root, "")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
def test_resolve_declines_when_no_artist_row_matches_the_folder(db_sync, import_root):
m = _media(import_root, "Nobody")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"))
assert resolve_refetch_source(db_sync, str(m), import_root) is None
# --- attempt_refetch: the destructive half ---------------------------------
def test_attempt_refetch_deletes_the_file_and_queues_one_recheck(
db_sync, import_root, no_dispatch,
):
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
src = _source(db_sync, _artist(db_sync, "Alice"))
task = _task(db_sync, m)
result = attempt_refetch(db_sync, task, import_root)
assert result == {"status": "refetch_queued", "source_id": src.id}
assert not m.exists() # the bad copy is gone...
assert no_dispatch == [src.id] # ...and exactly one re-check was queued
assert task.refetched is True
def test_attempt_refetch_leaves_the_file_alone_when_nothing_resolves(
db_sync, import_root, no_dispatch,
):
"""THE assertion this module exists for. `no_source` is the common
case on a filesystem-only library, and the file on disk is then the
operator's ONLY copy — deleting it without a downloader that can
replace it destroys the thing the remediation was meant to repair.
The route-level `no_source` test cannot catch a regression here: its
path never existed, so an unconditional unlink would pass it.
"""
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"), enabled=False)
task = _task(db_sync, m)
assert attempt_refetch(db_sync, task, import_root) == {"status": "no_source"}
assert m.exists()
assert m.read_bytes() == b"corrupt-bytes"
assert no_dispatch == []
assert task.refetched is False # not consumed — a real fix can still run
def test_attempt_refetch_is_bounded_to_a_single_attempt(
db_sync, import_root, no_dispatch,
):
"""The `refetched` bound is what stops SOURCE-side corruption from
looping: re-downloading a file that is broken upstream returns the
same bytes forever. The check must come FIRST — a second call has to
leave the (re-downloaded) file untouched, not delete it again.
"""
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
_source(db_sync, _artist(db_sync, "Alice"))
task = _task(db_sync, m, refetched=True)
assert attempt_refetch(db_sync, task, import_root) == {"status": "already_refetched"}
assert m.exists()
assert no_dispatch == []
def test_attempt_refetch_proceeds_when_the_file_is_already_gone(
db_sync, import_root, no_dispatch,
):
"""`missing_ok=True`: the sweep races an operator who deleted the bad
file by hand. The re-check is still the right next move."""
m = _media(import_root, "Alice")
_sidecar(m, {"category": "patreon"})
src = _source(db_sync, _artist(db_sync, "Alice"))
task = _task(db_sync, m)
m.unlink()
assert attempt_refetch(db_sync, task, import_root)["status"] == "refetch_queued"
assert no_dispatch == [src.id]
def test_attempt_refetch_survives_an_unlink_failure(
db_sync, import_root, no_dispatch,
):
"""An unremovable path is logged and stepped over, not raised — this
runs unattended, and a raise would abort the whole recovery sweep for
every OTHER poison-pill row in the batch.
A directory standing where the media file should be produces a
genuine IsADirectoryError (an OSError) without patching pathlib, so
the handler is exercised rather than simulated.
"""
d = import_root / "Alice" / "post.jpg"
d.mkdir(parents=True)
(import_root / "Alice" / "post.json").write_text(json.dumps({"category": "patreon"}))
src = _source(db_sync, _artist(db_sync, "Alice"))
task = _task(db_sync, d)
assert attempt_refetch(db_sync, task, import_root)["status"] == "refetch_queued"
assert d.exists() # removal genuinely failed...
assert no_dispatch == [src.id] # ...and the sweep carried on anyway
assert db_sync.execute(
select(ImportTask.refetched).where(ImportTask.id == task.id)
).scalar_one() is True
+4 -2
View File
@@ -23,12 +23,14 @@ async def _artist(db, name="Alice"):
@pytest.mark.asyncio
async def test_known_platforms_is_gs_six(db):
async def test_known_platforms_is_gs_five(db):
assert KNOWN_PLATFORMS == frozenset({
"patreon", "subscribestar", "hentaifoundry",
"discord", "pixiv", "deviantart",
"discord", "pixiv",
})
assert "fanbox" not in KNOWN_PLATFORMS
# Retired at #3069 — a source can no longer be created on it.
assert "deviantart" not in KNOWN_PLATFORMS
@pytest.mark.asyncio