Compare commits

...
16 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 42a60e3e74 test: the patreon ingester tests run phase 3's seen-marking before counting post keys (#4436)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 4s
CI and images / extension-test (push) Successful in 21s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 52s
CI and images / promote (push) Successful in 1s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 11:41:16 -04:00
bvandeusenandClaude Opus 5.5 621c2b8315 fix: a post record is marked seen only after it is upserted, and an hourly sweep dates the posts killed walks left undated (#4436)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Failing after 2m25s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
ingest_core marked a post's record key seen when it wrote _post.json, but
the record reaches the database (and dates the post and its images) only
in phase 3. A walk killed before phase 3 left posts the ledger called
recorded and the database never dated; ticks early-out long before
reaching them again. Keys now join the media in mark_seen_after_import.

date_posts_from_records finds each undated native post's record under
its artist's folder and upserts it with the post's own source. Hourly on
maintenance_long; an empty query once everything is dated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 11:36:17 -04:00
bvandeusenandClaude Opus 5.5 ca802bd885 fix: a non-image file lands on the source it was downloaded for, and the undated shells it left are folded into their real posts (#4435)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 18s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
The attachment path looked the post's source up by (artist, platform),
taking the artist's lowest-id source. A Discord artist has one source per
channel, so every archive or pdf from a later channel became an undated
second post on the first channel's source. _post_for_sidecar now takes
the downloading source, as upsert_post_record and _apply_sidecar already
did. Migration 0114 re-points each shell's attachments to its dated twin
and deletes the shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 10:56:08 -04:00
bvandeusenandClaude Opus 5.5 e2eeb63115 style: sort the boot hook's imports (#4433)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m43s
CI and images / smoke-web (push) Successful in 54s
CI and images / promote (push) Successful in 1s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 10:36:32 -04:00
bvandeusenandClaude Opus 5.5 e704c70f32 fix: the download boot hook imports a model-only module, keeping the membership roster off the fetch path (#4433)
CI and images / lint (push) Failing after 4s
CI and images / extension-version (push) Successful in 4s
CI and images / extension-test (push) Successful in 18s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 10:32:41 -04:00
bvandeusenandClaude Opus 5.5 0fad744bfb fix: a restart no longer strands downloads — the download lane closes its predecessor's runs as interrupted and frees the platform locks at boot (#4433)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Failing after 31s
CI and images / integration (push) Successful in 2m23s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
A walk that outlives the 90s stop grace is SIGKILLed: its event never
finalizes and its platform lock is held for the 27-min TTL. The 30-min
sweep then errored every stranded event and bumped consecutive_failures,
backing sources off (and blocking backfills) as if the platform failed.

On worker_ready, the process consuming 'download' ends pre-boot
pending/running events as skipped with error_type 'interrupted', leaves
the source untouched so the next tick resumes it, and releases the
serialized platforms' locks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 10:29:05 -04:00
bvandeusenandClaude Opus 5.5 32874ca678 fix: the release page drops internal rule numbers and the miscounted "three" images
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-web (push) Successful in 6s
CI and images / build-agent (push) Successful in 6s
CI and images / smoke-web (push) Successful in 41s
CI and images / promote (push) Successful in 1s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 10:09:47 -04:00
bvandeusenandClaude Opus 5.5 edd2daa16a docs: the README says ML tagging ships off and fetches its weights when enabled
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 6s
CI and images / smoke-web (push) Successful in 41s
CI and images / promote (push) Successful in 1s
The first-run note still said the ML worker downloads its model weights
on first boot. Milestone 422 moved that fetch to the moment the ML lane is
enabled under Settings → System, and the lane ships at 0 slots. The
overview paragraph is also the first release's notes (release_notes.py
reads it between the overview markers), so it now says the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:58:48 -04:00
bvandeusenandClaude Opus 5.5 23e062dd4a test: the stall-sweep tests read the thresholds they check, and stay clear of the new import value (#4432)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / extension-test (push) Successful in 18s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Successful in 1s
Run 7505 failed test_recover_stalled_task_runs_ml_queue_uses_longer_threshold.
It restated the old 25-minute ml threshold as a 30-minute "stale" row,
which is now inside the 40-minute window. The test now reads the value
from QUEUE_STUCK_THRESHOLD_MINUTES.

The archive test's fast-import row was exactly 10 minutes old, which is
the new import threshold, so it passed only by the milliseconds between
seeding and sweeping. It is now 15 minutes old.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:51:42 -04:00
bvandeusenandClaude Opus 5.5 a360d69ee8 fix: task runs record the lane Celery really routes them to, and no healthy long job is swept as stalled (#4432)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Failing after 2m24s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
celery_signals._queue_for was a hand-kept copy of task_routes and had
drifted:

- backup, admin and library_audit jobs, and backfill_phash, run on
  maintenance_long but were recorded as `maintenance`;
- translation and gpu_queue jobs were recorded as `default`, where the
  5-minute stall sweep failed healthy 35-minute translation runs.

It now asks the router, cached per task name.

A new guard test checks every registered task's hard time limit against
the stall threshold the sweep would use for it. It also caught these
sweeps, which failed healthy runs mid-flight and are fixed here:

- ml's scheduled sweeps (35 min, previously swept at 25);
- train_heads and apply_head_tags (65 min);
- import_media_file (6 min, previously swept at 5);
- the long lane, which now has its own 45-minute threshold.

UI changes:

- the admin job poller follows a job by celery_task_id, via a new filter
  on /runs, instead of by lane;
- the archive re-extract and missing-file repair cards show the long
  lane's backlog, where their jobs actually wait;
- the queue table lists maintenance_long.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:46:39 -04:00
bvandeusenandClaude Opus 5.5 dfd28a0aa6 fix(ci): the base refresh's lanes test main, the branch it publishes (#4430)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 5s
CI and images / smoke-web (push) Successful in 40s
CI and images / promote (push) Successful in 1s
A refresh publishes `:latest` from `main`, but the six lanes kept the
default checkout, which is the cron's triggering commit on dev. The gate
therefore tested dev's code and passed main's.

The new LANE_REF is `main` on a refresh and empty otherwise. Empty keeps
the checkout default, so push and PR runs are unchanged, including a PR's
merge ref, which the default reaches by ref rather than by SHA.

Every lane still runs on every trigger: no condition is added and no
lane can skip, so the `needs:` gate on each publishing job is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:40:08 -04:00
bvandeusenandClaude Opus 5.5 efcb548ebf fix: natively downloaded images take their post's date, not their download time (#4431)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m22s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m40s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 2s
The native ingesters (Discord, Patreon, SubscribeStar) import a post's
media before its record. Only the record (`_post.json`) carries the
date: the Discord message timestamp, or the Patreon/SubscribeStar
published_at. So each image was linked to a post with no date yet, and
it kept its download time in both gallery date columns. The post itself
was dated correctly once the record landed, but nothing went back to
update its images.

- upsert_post_record now re-dates the images already linked to the
  post. `effective_date` becomes the primary post's date, and
  `earliest_post_date` the earliest dated post the image is in, which
  are the same rules _attach_provenance applies.
- Migration 0113 repairs the library from the posts. It writes only
  rows that differ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:30:17 -04:00
bvandeusenandClaude Opus 5.5 4fa7975963 fix(ci): every publishing job builds the commit that fired the run, not the branch tip (#4427)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 18s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / smoke-web (push) Successful in 41s
CI and images / promote (push) Successful in 1s
CI and images / build-web (push) Successful in 6s
BUILD_REF resolved to the branch name on ordinary runs, and each job's
checkout re-resolves a branch when that job starts. A push that lands
mid-run therefore moved the later jobs onto the new tip:

- run 7499 signed 423275a's extension;
- its build-web then checked out 83e1382, derived a version nobody had
  signed, and got a 404 on the download.

The guard failed closed. A job without such a guard would have published
a commit the run's lanes never tested.

The lanes already check out github.sha by default. BUILD_REF is now
github.sha on every trigger except the base refresh, which keeps `main`
and its branch guard. The publish therefore builds exactly what the
lanes passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 09:13:41 -04:00
bvandeusenandClaude Opus 5.5 7b1570f2a5 feat: retire the sketch/doodle WIP title tier, and the review strip asks "Is this a WIP?" (milestone 430)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m24s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m35s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Successful in 1s
The soft tier (#1474) tagged `wip` on any post titled sketch, doodle or
scribble: 6,096 of the library's 8,876 wip tags. Its conflict audit flagged
most of them, because finished art scores >= 0.5 on some content head,
which filled the Gallery strip with 2,086 cards. A "sketch" is usually
finished work, so the operator retired the tier. The artist's own
"WIP" / "work in progress" title rule and human wip tags stay.

- Removed: the soft matcher, source and prefilter; the importer and backfill
  soft branches; soft_wip_conflict_audit with its task and beat; the
  wip_soft_title_tagging_enabled setting, API field and toggle; and
  wip_title_soft from _AUTO_SOURCES.
- Migration 0112: a soft tag the operator confirmed, or kept from the strip,
  becomes `manual`. Every other soft tag is deleted, along with the open
  review cards whose tag is gone. The column is dropped.
- Review strip (#4424): each card asks "Is this a WIP?" (or banner, and so on),
  and the buttons read "Is a WIP" / "Is not a WIP". The content tag it also
  scored on is shown as the reason it was flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 08:20:28 -04:00
bvandeusenandClaude Opus 5.5 83e1382812 fix: extension updates install the new build — no 12h-cached "latest" XPI, and the popup's Update opens FC's install page
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m23s
CI and images / build-agent (push) Successful in 5s
CI and images / sign-extension (push) Successful in 2m29s
CI and images / build-web (push) Successful in 1m43s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 2s
Operator, 2026-09-25: "the extension update trigger from inside the
extension doesn't work and the manual update seems to not move it to the
most recent version or at least mark it the most recent."

- fabledcurator-latest.xpi was served with Quart's default
  `public, max-age=43200`: one URL whose bytes change every release, so a
  browser that had fetched it reinstalled the previous build for 12 hours
  (measured on the instance). It is now `no-cache` (the ETag keeps an
  unchanged file a 304); versioned XPIs are `immutable`.
- The web Settings card installs/downloads the VERSIONED xpi_url, which can
  only ever be that build's bytes.
- The popup's Update button did tabs.create() on the .xpi, which Firefox
  refuses (NS_ERROR_FAILURE on a 200: it only installs from a user click on
  a web page). It now opens FC's install card (/subscriptions?tab=settings).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 08:01:02 -04:00
bvandeusenandClaude Opus 5.5 423275a1e5 feat: the artist picker on Patreon and SubscribeStar too — and the Patreon name is canon (milestone 429)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m22s
CI and images / build-agent (push) Successful in 6s
CI and images / sign-extension (push) Successful in 4m48s
CI and images / build-web (push) Failing after 6s
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
Operator: "yes add the artist picker to patreon and subscribestar too. but
generally we treat the patreon name as the canon"

- Every add now goes through the panel. On Patreon/SubscribeStar it opens
  on the creator's display name (read only when the panel opens:
  probe?names=1), searches FC's artists with it, and auto-picks a match.
  An untouched URL handle sends no name, so the server still resolves it.
- Patreon is canon: joining a Patreon source to an artist known by another
  name offers "Rename “x” to the Patreon name “X”", ticked by default.
  quick-add's use_platform_name renames server-side from the name it reads
  itself; name only, the slug never moves (#130); never to a URL handle when
  the name can't be read; ignored on SubscribeStar and Discord.
- _platform_display_name returns None rather than the handle, bounded by the
  same 6s lookup budget as Discord's names.
- panelDefaults / addRequest / renameOffer replace the Discord-only helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-25 07:50:48 -04:00
54 changed files with 1761 additions and 481 deletions
+36 -7
View File
@@ -142,6 +142,16 @@ concurrency:
# Deriving it per job invites the two halves to disagree: sign-extension would # Deriving it per job invites the two halves to disagree: sign-extension would
# derive dev's extension version while build-web bundled main's, and the # derive dev's extension version while build-web bundled main's, and the
# release download would 404 on a version that exists perfectly well. # release download would 404 on a version that exists perfectly well.
#
# On every other trigger it is the COMMIT that fired (`github.sha`), never the
# branch name. A branch is re-resolved by each job's checkout when that job
# starts, so a push landing mid-run moved the later jobs onto the new tip: run
# 7499 signed 423275a's extension, then build-web checked out 83e1382 (pushed
# while 7499 ran), derived a version nobody had signed, and 404'd on the
# download (#4427). The guard failed closed that time; a job without one would
# have published a commit the run's own lanes never tested — the lanes check
# out `github.sha` by default, so pinning here makes the publish build exactly
# what they passed.
# IS THIS A BASE REFRESH? Asked in five places and previously spelled five # IS THIS A BASE REFRESH? Asked in five places and previously spelled five
# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in # ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
# one shell, an `EVENT:` env passed into another, and a bare expression on # one shell, an `EVENT:` env passed into another, and a bare expression on
@@ -178,7 +188,16 @@ concurrency:
# where a step-level `if:` needs the answer before any shell runs. # where a step-level `if:` needs the answer before any shell runs.
env: env:
IS_REFRESH: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'true' || 'false' }} IS_REFRESH: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'true' || 'false' }}
BUILD_REF: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'main' || github.ref }} BUILD_REF: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'main' || github.sha }}
# What the six LANES check out. Empty — the checkout default, the triggering
# commit (or a PR's merge ref) — on every trigger but the refresh, where it is
# `main`: the refresh publishes main, so the gate has to test main (#4430). It
# is not BUILD_REF itself because a pull_request run's `github.sha` is a merge
# commit the default checkout reaches through its ref, not by sha. Each job
# still resolves `main` when it starts, so a merge to main during the ~5 min of
# a Sunday-06:00 refresh could put the lanes and the build one commit apart;
# the build jobs' own guards assert the branch, not the commit.
LANE_REF: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'main' || '' }}
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes: # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com) # - write:package, read:package (for docker push to git.fabledsword.com)
@@ -219,6 +238,8 @@ jobs:
image: git.fabledsword.com/bvandeusen/ci-python:3.14 image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with:
ref: ${{ env.LANE_REF }}
- name: Ruff lint - name: Ruff lint
# agent/ included so the GPU-agent is linted before its image is built # agent/ included so the GPU-agent is linted before its image is built
# (build.yml only `docker build`s it — this is where it gets checked). # (build.yml only `docker build`s it — this is where it gets checked).
@@ -265,6 +286,7 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
ref: ${{ env.LANE_REF }}
# The derivation needs real history: a depth-1 clone sees one commit # The derivation needs real history: a depth-1 clone sees one commit
# and produces a wrong, too-low value RATHER THAN FAILING. Checking # and produces a wrong, too-low value RATHER THAN FAILING. Checking
# that here is half the point of the lane. # that here is half the point of the lane.
@@ -320,6 +342,7 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
ref: ${{ env.LANE_REF }}
# Full history for tests/test_artifact_identity.py, which derives # Full history for tests/test_artifact_identity.py, which derives
# each artifact's revision to check the identity scheme. On a # each artifact's revision to check the identity scheme. On a
# depth-1 clone that derivation either fails or returns the tip sha # depth-1 clone that derivation either fails or returns the tip sha
@@ -366,6 +389,8 @@ jobs:
working-directory: frontend working-directory: frontend
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with:
ref: ${{ env.LANE_REF }}
# No package-lock.json is tracked yet (we don't run npm locally per # No package-lock.json is tracked yet (we don't run npm locally per
# feedback-no-local-runs). Using `npm install` instead of `npm ci`. # feedback-no-local-runs). Using `npm install` instead of `npm ci`.
# If we want strict lockfile-based reproducibility later, commit a # If we want strict lockfile-based reproducibility later, commit a
@@ -395,6 +420,8 @@ jobs:
image: node:24-bookworm-slim image: node:24-bookworm-slim
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with:
ref: ${{ env.LANE_REF }}
# Not --no-save: vitest and web-ext are both real devDependencies now, # Not --no-save: vitest and web-ext are both real devDependencies now,
# and the suite needs vitest resolvable from node_modules. # and the suite needs vitest resolvable from node_modules.
- name: Install dev dependencies - name: Install dev dependencies
@@ -497,6 +524,8 @@ jobs:
--health-retries 10 --health-retries 10
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with:
ref: ${{ env.LANE_REF }}
- name: Integration suite (resolve service IPs, migrate, test) - name: Integration suite (resolve service IPs, migrate, test)
run: | run: |
set -eux set -eux
@@ -605,8 +634,8 @@ jobs:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a # Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref # scheduled refresh this is `main`; on everything else it is the
# that fired, so this is a no-op on every ordinary path. # commit that fired, the same one the lanes above tested.
ref: ${{ env.BUILD_REF }} ref: ${{ env.BUILD_REF }}
# Full history is load-bearing, not a convenience: the version this # Full history is load-bearing, not a convenience: the version this
# job signs is derived from the commit TIME of the newest packaged # job signs is derived from the commit TIME of the newest packaged
@@ -945,8 +974,8 @@ jobs:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a # Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref # scheduled refresh this is `main`; on everything else it is the
# that fired, so this is a no-op on every ordinary path. # commit that fired, the same one the lanes above tested.
ref: ${{ env.BUILD_REF }} ref: ${{ env.BUILD_REF }}
# Full history: this job RE-DERIVES the extension version rather than # Full history: this job RE-DERIVES the extension version rather than
# being handed it, and a depth-1 clone derives a wrong, too-low value # being handed it, and a depth-1 clone derives a wrong, too-low value
@@ -2174,8 +2203,8 @@ jobs:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a # Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref # scheduled refresh this is `main`; on everything else it is the
# that fired, so this is a no-op on every ordinary path. # commit that fired, the same one the lanes above tested.
ref: ${{ env.BUILD_REF }} ref: ${{ env.BUILD_REF }}
# Full history: this job derives its artifact's version from the # Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A # commit its shipped files last changed in (milestone 313). A
+8 -4
View File
@@ -22,7 +22,8 @@ through afterwards.
- **ML tagging.** Runs image models in-container to suggest tags, group - **ML tagging.** Runs image models in-container to suggest tags, group
characters, find near-duplicates and power similarity search. Suggestions are characters, find near-duplicates and power similarity search. Suggestions are
reviewable — it proposes, you confirm, and it learns which proposals you keep reviewable — it proposes, you confirm, and it learns which proposals you keep
rejecting. rejecting. It ships switched off: turn it on under Settings → System when
you want it, and it fetches its model weights then.
- **Deduplication and provenance.** Everything that arrives is hashed and - **Deduplication and provenance.** Everything that arrives is hashed and
deduplicated by content, metadata sidecars are read wherever the source deduplicated by content, metadata sidecars are read wherever the source
writes them, and every file keeps a record of where it came from. writes them, and every file keeps a record of where it came from.
@@ -119,9 +120,12 @@ needs every credential entered again by hand.
A few other things are worth knowing about the first few minutes: A few other things are worth knowing about the first few minutes:
- **The ML worker downloads its model weights on first boot**, several GB from - **ML tagging starts switched off, and nothing is downloaded at boot.** Give
HuggingFace into `./models`. Until that finishes, tagging is queued rather the ML lane a slot under **Settings → System** and it fetches its model
than broken. It is idempotent — a restart resumes rather than refetches. weights then — several GB from HuggingFace into `./models`, shown as a job
under **Settings → Activity** that you can watch and retry. Until it
finishes, tagging is queued rather than broken, and the fetch only takes
what is missing, so turning the lane off and on again does not refetch.
- **The gallery starts empty**, and that is the expected state. Add a creator - **The gallery starts empty**, and that is the expected state. Add a creator
under **Subscriptions** and it fills as posts come down. under **Subscriptions** and it fills as posts come down.
- **If you already have a library on disk**, there is no screen that imports - **If you already have a library on disk**, there is no screen that imports
@@ -0,0 +1,78 @@
"""Retire the sketch/doodle WIP title tier — its tags, its review flags, its toggle.
Milestone 430, #4428. The soft tier (#1474) tagged `wip` on any post titled
sketch / doodle / scribble. Measured on the operator's library it was 6,096 of
8,876 wip tags, and its conflict audit filled the Gallery's review strip with
2,086 cards, because most finished art scores >= 0.5 on some content head. A
"sketch" is usually finished work, so the operator retired it (2026-09-25).
Data:
* A soft tag the operator stood behind is kept and relabelled `manual`: one they
confirmed (tag_positive_confirmation), or one whose review flag they resolved
while leaving the tag on (the strip's "Keep tag").
* Every other `wip_title_soft` row is deleted.
* Unresolved review flags whose tag is no longer on the image are deleted: the
question they ask no longer applies. That is the audit's cards, and any older
orphan the same way.
Then `import_settings.wip_soft_title_tagging_enabled` is dropped. The downgrade
restores the column only; deleted tags are not recreated.
Revision ID: 0112
Revises: 0111
Create Date: 2026-09-25
"""
import sqlalchemy as sa
from alembic import op
revision = "0112"
down_revision = "0111"
branch_labels = None
depends_on = None
def retire_soft_wip_tags(conn) -> None:
"""The data half, on a plain connection, so a test can run it directly."""
conn.execute(sa.text("""
UPDATE image_tag it SET source = 'manual'
WHERE it.source = 'wip_title_soft'
AND (
EXISTS (
SELECT 1 FROM tag_positive_confirmation c
WHERE c.image_record_id = it.image_record_id AND c.tag_id = it.tag_id
)
OR EXISTS (
SELECT 1 FROM presentation_review pr
WHERE pr.image_record_id = it.image_record_id AND pr.tag_id = it.tag_id
AND pr.resolved_at IS NOT NULL
)
)
"""))
conn.execute(sa.text("DELETE FROM image_tag WHERE source = 'wip_title_soft'"))
conn.execute(sa.text("""
DELETE FROM presentation_review pr
WHERE pr.resolved_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM image_tag it
WHERE it.image_record_id = pr.image_record_id AND it.tag_id = pr.tag_id
)
"""))
def upgrade():
retire_soft_wip_tags(op.get_bind())
op.drop_column("import_settings", "wip_soft_title_tagging_enabled")
def downgrade():
op.add_column(
"import_settings",
sa.Column(
"wip_soft_title_tagging_enabled",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
)
@@ -0,0 +1,60 @@
"""Re-date images whose post's date arrived after they were linked.
#4431. The native ingesters import a post's media before its record, and the
date travels in the record (`_post.json`). Every natively downloaded image was
therefore linked to an undated post and kept its download time in both gallery
date columns, while the post itself was dated correctly. The importer now
re-dates a post's images when its record lands; this repairs the images that
landed before that.
Both columns get back the rules the importer keeps:
* `effective_date` is the primary post's date (left alone when that post has
none, as the importer does);
* `earliest_post_date` is the earliest dated post the image is linked to.
Only rows that differ are written. The downgrade does nothing: the old values
were download times that no one chose.
Revision ID: 0113
Revises: 0112
Create Date: 2026-09-25
"""
import sqlalchemy as sa
from alembic import op
revision = "0113"
down_revision = "0112"
branch_labels = None
depends_on = None
def redate_images(conn) -> None:
"""The data step, on a plain connection, so a test can run it directly."""
conn.execute(sa.text("""
UPDATE image_record ir SET effective_date = p.post_date
FROM post p
WHERE p.id = ir.primary_post_id
AND p.post_date IS NOT NULL
AND ir.effective_date IS DISTINCT FROM p.post_date
"""))
conn.execute(sa.text("""
UPDATE image_record ir SET earliest_post_date = m.earliest
FROM (
SELECT ip.image_record_id, MIN(p.post_date) AS earliest
FROM image_provenance ip JOIN post p ON p.id = ip.post_id
WHERE p.post_date IS NOT NULL
GROUP BY ip.image_record_id
) m
WHERE m.image_record_id = ir.id
AND ir.earliest_post_date IS DISTINCT FROM m.earliest
"""))
def upgrade():
redate_images(op.get_bind())
def downgrade():
pass
@@ -0,0 +1,79 @@
"""Fold the undated shell posts a misfiled attachment created into the real post.
#4435. A non-image file (an archive, a pdf) downloaded for one of an artist's
Discord channels was filed under the artist's FIRST Discord source: the
attachment path looked the source up by (artist, platform), which takes the
lowest id. That created an undated, url-less post there holding only the
attachment, and the message's real post record then created the dated post
under the right source. The importer now uses the source it was downloading
for; this repairs the pairs it left.
A shell is folded only when all of this holds:
* it has no date and no url, and nothing synthesized it;
* another post of the same artist, platform and external id HAS a date;
* no image is linked to the shell (it held an attachment and nothing else).
Its attachments move to the dated post, dropping any the dated post already
has (same sha256, which the per-post unique forbids twice), and the shell is
deleted. A shell with no dated twin is left alone: which channel it belongs to
is not recorded anywhere but the file name. The downgrade does nothing.
Revision ID: 0114
Revises: 0113
Create Date: 2026-09-25
"""
import sqlalchemy as sa
from alembic import op
revision = "0114"
down_revision = "0113"
branch_labels = None
depends_on = None
_PAIRS = """
SELECT DISTINCT ON (shell.id) shell.id AS shell_id, real.id AS real_id
FROM post shell
JOIN source ss ON ss.id = shell.source_id
JOIN post real
ON real.artist_id = shell.artist_id
AND real.external_post_id = shell.external_post_id
AND real.id <> shell.id
AND real.post_date IS NOT NULL
JOIN source rs ON rs.id = real.source_id AND rs.platform = ss.platform
WHERE shell.post_date IS NULL
AND shell.post_url IS NULL
AND shell.synthesized_by IS NULL
AND NOT EXISTS (SELECT 1 FROM image_provenance ip WHERE ip.post_id = shell.id)
ORDER BY shell.id, real.id
"""
def fold_misfiled_attachment_posts(conn) -> int:
"""The data step, on a plain connection, so a test can run it directly.
Returns how many shells were folded."""
pairs = conn.execute(sa.text(_PAIRS)).all()
for shell_id, real_id in pairs:
conn.execute(sa.text("""
DELETE FROM post_attachment pa
WHERE pa.post_id = :shell
AND EXISTS (
SELECT 1 FROM post_attachment keep
WHERE keep.post_id = :real AND keep.sha256 = pa.sha256
)
"""), {"shell": shell_id, "real": real_id})
conn.execute(
sa.text("UPDATE post_attachment SET post_id = :real WHERE post_id = :shell"),
{"shell": shell_id, "real": real_id},
)
conn.execute(sa.text("DELETE FROM post WHERE id = :shell"), {"shell": shell_id})
return len(pairs)
def upgrade() -> None:
fold_misfiled_attachment_posts(op.get_bind())
def downgrade() -> None:
pass
+7 -1
View File
@@ -95,7 +95,9 @@ async def probe_source():
return _bad("unauthorized", status=401) return _bad("unauthorized", status=401)
# crypto lets a Discord probe name the server and channel with the # crypto lets a Discord probe name the server and channel with the
# stored token; every other platform ignores it. # stored token; every other platform ignores it.
result = await ExtensionService(session, _get_crypto()).probe(url) result = await ExtensionService(session, _get_crypto()).probe(
url, names=request.args.get("names") in ("1", "true"),
)
return jsonify(result) return jsonify(result)
@@ -116,6 +118,9 @@ async def quick_add_source():
artist_name = body.get("artist_name") artist_name = body.get("artist_name")
if artist_name is not None and not isinstance(artist_name, str): if artist_name is not None and not isinstance(artist_name, str):
return _bad("invalid_body", detail="artist_name must be a string") return _bad("invalid_body", detail="artist_name must be a string")
# Patreon is canon: adding a Patreon source to an existing artist can take
# the creator's Patreon display name (name only; the slug never moves).
use_platform_name = body.get("use_platform_name") is True
from .credentials import _get_crypto from .credentials import _get_crypto
@@ -127,6 +132,7 @@ async def quick_add_source():
# stored credential (else it falls back to the URL handle). #130. # stored credential (else it falls back to the URL handle). #130.
result = await ExtensionService(session, _get_crypto()).quick_add_source( result = await ExtensionService(session, _get_crypto()).quick_add_source(
url, artist_id=artist_id, artist_name=artist_name, url, artist_id=artist_id, artist_name=artist_name,
use_platform_name=use_platform_name,
) )
except UnknownArtistError as exc: except UnknownArtistError as exc:
return _bad("not_found", detail=str(exc), status=404) return _bad("not_found", detail=str(exc), status=404)
-7
View File
@@ -57,7 +57,6 @@ _EDITABLE_FIELDS = (
"translation_target_lang", "translation_target_lang",
"translation_min_confidence", "translation_min_confidence",
"wip_title_tagging_enabled", "wip_title_tagging_enabled",
"wip_soft_title_tagging_enabled",
) )
# Per-host external-download toggles — all plain booleans, validated uniformly. # Per-host external-download toggles — all plain booleans, validated uniformly.
@@ -196,12 +195,6 @@ async def update_import_settings():
return jsonify( return jsonify(
{"error": "wip_title_tagging_enabled must be a boolean"} {"error": "wip_title_tagging_enabled must be a boolean"}
), 400 ), 400
if "wip_soft_title_tagging_enabled" in body and not isinstance(
body["wip_soft_title_tagging_enabled"], bool
):
return jsonify(
{"error": "wip_soft_title_tagging_enabled must be a boolean"}
), 400
async with get_session() as session: async with get_session() as session:
row = await ImportSettings.load(session) row = await ImportSettings.load(session)
+5
View File
@@ -152,6 +152,8 @@ async def list_runs():
queue=<name> filter to one queue queue=<name> filter to one queue
status=<status> filter to one status (running/ok/error/timeout/retry) status=<status> filter to one status (running/ok/error/timeout/retry)
task=<substr> case-insensitive substring match on task_name task=<substr> case-insensitive substring match on task_name
celery_task_id=<id> exactly one run — how a page follows a job it
started without having to know its lane
limit=<int> default 50, max 200 limit=<int> default 50, max 200
before_id=<int> cursor for keyset pagination before_id=<int> cursor for keyset pagination
@@ -167,6 +169,7 @@ async def list_runs():
queue = request.args.get("queue") queue = request.args.get("queue")
status = request.args.get("status") status = request.args.get("status")
task = request.args.get("task") task = request.args.get("task")
celery_task_id = request.args.get("celery_task_id")
before_id_raw = request.args.get("before_id") before_id_raw = request.args.get("before_id")
before_id = int(before_id_raw) if before_id_raw else None before_id = int(before_id_raw) if before_id_raw else None
@@ -176,6 +179,8 @@ async def list_runs():
stmt = stmt.where(TaskRun.queue == queue) stmt = stmt.where(TaskRun.queue == queue)
if status: if status:
stmt = stmt.where(TaskRun.status == status) stmt = stmt.where(TaskRun.status == status)
if celery_task_id:
stmt = stmt.where(TaskRun.celery_task_id == celery_task_id)
if task: if task:
# Task names contain literal underscores (download_source, # Task names contain literal underscores (download_source,
# vacuum_analyze) — escape LIKE wildcards so a search for # vacuum_analyze) — escape LIKE wildcards so a search for
+7 -5
View File
@@ -69,6 +69,8 @@ def make_celery() -> Celery:
# up behind it (2026-09-24: 7 waiting, "all workers busy for 18 # up behind it (2026-09-24: 7 waiting, "all workers busy for 18
# minutes"). An exact name wins over the glob above. # minutes"). An exact name wins over the glob above.
"backend.app.tasks.maintenance.backfill_phash": {"queue": "maintenance_long"}, "backend.app.tasks.maintenance.backfill_phash": {"queue": "maintenance_long"},
# Walks a folder tree per artist with undated posts (#4436).
"backend.app.tasks.maintenance.date_posts_from_records": {"queue": "maintenance_long"},
"backend.app.tasks.backup.*": {"queue": "maintenance_long"}, "backend.app.tasks.backup.*": {"queue": "maintenance_long"},
"backend.app.tasks.admin.*": {"queue": "maintenance_long"}, "backend.app.tasks.admin.*": {"queue": "maintenance_long"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance_long"}, "backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
@@ -156,6 +158,11 @@ def make_celery() -> Celery:
"schedule": 86400.0, # daily — sweep .part/.partial left by a "schedule": 86400.0, # daily — sweep .part/.partial left by a
# download/import killed mid-write (graceful-shutdown fallout) # download/import killed mid-write (graceful-shutdown fallout)
}, },
"date-posts-from-records-hourly": {
"task": "backend.app.tasks.maintenance.date_posts_from_records",
"schedule": 3600.0, # an empty query once every native post is
# dated; otherwise upserts the _post.json a killed walk left (#4436)
},
"backfill-phash-daily": { "backfill-phash-daily": {
"task": "backend.app.tasks.maintenance.backfill_phash", "task": "backend.app.tasks.maintenance.backfill_phash",
"schedule": 86400.0, # daily — NULL-only, so a no-op once the "schedule": 86400.0, # daily — NULL-only, so a no-op once the
@@ -223,11 +230,6 @@ def make_celery() -> Celery:
"schedule": 86400.0, # auto-tag wip/editor process art (#1464); "schedule": 86400.0, # auto-tag wip/editor process art (#1464);
# no-op unless process_auto_apply_enabled (opt-in) # no-op unless process_auto_apply_enabled (opt-in)
}, },
"soft-wip-conflict-audit-daily": {
"task": "backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
"schedule": 86400.0, # flag ring-loud soft-WIP (sketch/doodle) tags
# for review (#1474); no-op with no content heads
},
"prune-presentation-reviews-daily": { "prune-presentation-reviews-daily": {
"task": "backend.app.tasks.ml.prune_presentation_reviews", "task": "backend.app.tasks.ml.prune_presentation_reviews",
"schedule": 86400.0, # retention: drop resolved review flags >30d "schedule": 86400.0, # retention: drop resolved review flags >30d
+67 -33
View File
@@ -18,11 +18,18 @@ dark for that interval. Monitoring NEVER breaks the thing it's
monitoring. monitoring.
""" """
import functools
import logging import logging
from datetime import UTC, datetime from datetime import UTC, datetime
from celery.exceptions import SoftTimeLimitExceeded from celery.exceptions import SoftTimeLimitExceeded
from celery.signals import task_failure, task_postrun, task_prerun, task_retry from celery.signals import (
task_failure,
task_postrun,
task_prerun,
task_retry,
worker_ready,
)
from .models import TaskRun from .models import TaskRun
from .tasks._sync_engine import sync_session_factory from .tasks._sync_engine import sync_session_factory
@@ -53,42 +60,29 @@ _INT32_MIN = -2_147_483_648
def _queue_for(task) -> str: def _queue_for(task) -> str:
"""Reverse the task→queue routing from celery_app.task_routes. """The queue Celery routes this task to — asked of the router itself.
Keep in sync if task_routes is reordered.
Audit 2026-06-02: backup/admin/library_audit prefixes were This was a hand-kept copy of `celery_app.task_routes`, and it drifted
missing here even though task_routes sent all three to twice (the 2026-06-02 audit, then #4432). Long-lane jobs were recorded as
'maintenance'. The TaskRun.queue column then lied for those `maintenance`, and translation and gpu_queue runs as `default`, where the
rows (claimed 'default') so per-queue dashboard filters and 5-minute stall sweep failed healthy 35-minute translation runs. The router
per-queue threshold overrides silently missed them. answers from the same table the broker uses, so the two cannot disagree.
""" """
name = getattr(task, "name", "") or "" name = getattr(task, "name", "") or ""
if name.startswith("backend.app.tasks.import_file."): app = getattr(task, "app", None)
return "import" if app is None:
if name.startswith("backend.app.tasks.ml."): from .celery_app import celery as app
return "ml" return _routed_queue(app, name)
if name.startswith("backend.app.tasks.thumbnail."):
return "thumbnail"
if name.startswith(( @functools.lru_cache(maxsize=1024)
"backend.app.tasks.download.", def _routed_queue(app, name: str) -> str:
# External file-host fetches share the download lane (celery_app try:
# routes external.* → download). Mirror it here or TaskRun.queue queue = app.amqp.router.route({}, name).get("queue")
# lies 'default' for them, so per-queue dashboard filters and the except Exception: # noqa: BLE001 — monitoring never breaks the task
# per-queue threshold override miss them — the same gap the log.warning("task_run: could not resolve the queue for %s", name)
# 2026-06-02 audit fixed for backup/admin/library_audit.
"backend.app.tasks.external.",
)):
return "download"
if name.startswith("backend.app.tasks.scan."):
return "scan"
if name.startswith((
"backend.app.tasks.maintenance.",
"backend.app.tasks.backup.",
"backend.app.tasks.admin.",
"backend.app.tasks.library_audit.",
)):
return "maintenance"
return "default" return "default"
return getattr(queue, "name", None) or (queue if isinstance(queue, str) else "default")
def _target_id_from_args(args) -> int | None: def _target_id_from_args(args) -> int | None:
@@ -225,3 +219,43 @@ def _on_retry(sender=None, request=None, reason=None, einfo=None, **_):
error_message=str(reason) if reason else None, error_message=str(reason) if reason else None,
retry_count=getattr(request, "retries", 0), retry_count=getattr(request, "retries", 0),
) )
def _consumed_queues(consumer) -> set[str]:
"""The queue names this worker process consumes (its `-Q`), or empty when
they can't be read — which makes the boot hook below a no-op, not a guess."""
try:
return {q.name for q in consumer.task_consumer.queues}
except Exception: # noqa: BLE001 — shape varies across celery versions
return set()
@worker_ready.connect
def _on_worker_ready(sender=None, **_):
"""The download lane clears what its previous process left behind (#4433).
A restart SIGKILLs any walk that outlives the stop grace, so its event is
never finalized and its platform lock is never released. Both used to wait
out timers — a 30-min stall sweep that then blamed the source, and a 27-min
lock TTL that stalled every other source on the platform. Only the process
consuming `download` does this; the other lanes booting beside it must not.
Best-effort: a failure here is logged and the worker still starts.
"""
if "download" not in _consumed_queues(sender):
return
booted_at = datetime.now(UTC)
try:
from .services.download_recovery import interrupt_orphaned_download_events
from .services.platform_lock import release_all_platform_locks
with sync_session_factory()() as session:
closed = interrupt_orphaned_download_events(session, booted_at=booted_at)
session.commit()
released = release_all_platform_locks()
if closed or released:
log.info(
"download lane boot: closed %d orphaned download event(s) as "
"interrupted, released %d platform lock(s)", closed, released,
)
except Exception: # noqa: BLE001 — never block the worker from starting
log.exception("download lane boot recovery failed")
+20 -2
View File
@@ -46,6 +46,14 @@ async def serve_extension(filename: str):
The application/x-xpinstall MIME tells Firefox to show its native The application/x-xpinstall MIME tells Firefox to show its native
install prompt instead of downloading the file as a blob. install prompt instead of downloading the file as a blob.
Caching differs by name, and has to. A versioned name is one build's bytes
forever, so it can be cached for good. `fabledcurator-latest.xpi` is ONE
URL whose bytes change on every release, and Quart's default for a file
is `public, max-age=43200`: a browser that fetched it once reused those
bytes for 12 hours, so "install the latest" quietly reinstalled the
previous build (operator-flagged 2026-09-25). It is `no-cache` — the ETag
still makes an unchanged file a cheap 304.
""" """
if not _XPI_NAME_RE.fullmatch(filename): if not _XPI_NAME_RE.fullmatch(filename):
abort(404) abort(404)
@@ -56,10 +64,11 @@ async def serve_extension(filename: str):
if not xpis: if not xpis:
abort(404) abort(404)
latest = xpis[-1] latest = xpis[-1]
return await send_file( resp = await send_file(
latest, mimetype="application/x-xpinstall", latest, mimetype="application/x-xpinstall",
attachment_filename=latest.name, attachment_filename=latest.name,
) )
return _cache(resp, "no-cache")
target = (XPI_DIR / filename).resolve() target = (XPI_DIR / filename).resolve()
try: try:
target.relative_to(XPI_DIR) target.relative_to(XPI_DIR)
@@ -67,10 +76,19 @@ async def serve_extension(filename: str):
abort(404) abort(404)
if not target.is_file(): if not target.is_file():
abort(404) abort(404)
return await send_file( resp = await send_file(
target, mimetype="application/x-xpinstall", target, mimetype="application/x-xpinstall",
attachment_filename=filename, attachment_filename=filename,
) )
return _cache(resp, "public, max-age=31536000, immutable")
def _cache(resp, policy: str):
"""Set the XPI's Cache-Control, dropping the Expires send_file adds so the
two can never disagree."""
resp.headers["Cache-Control"] = policy
resp.headers.pop("Expires", None)
return resp
@frontend_bp.route("/") @frontend_bp.route("/")
-7
View File
@@ -238,13 +238,6 @@ class ImportSettings(Base):
wip_title_tagging_enabled: Mapped[bool] = mapped_column( wip_title_tagging_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True, server_default="true", Boolean, nullable=False, default=True, server_default="true",
) )
# Soft WIP title tier (#1474): also tag sketch/doodle/scribble titles, but with
# a PROVISIONAL source (`wip_title_soft`) that never trains the head, since these
# are lower-precision (a finished "sketch" isn't WIP). OFF by default — a lower-
# precision tier is opt-in (the ring-loud audit surfaces false positives).
wip_soft_title_tagging_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, server_default="false",
)
@classmethod @classmethod
async def load(cls, session) -> ImportSettings: async def load(cls, session) -> ImportSettings:
+57
View File
@@ -0,0 +1,57 @@
"""Closing the download runs a worker restart orphaned (#4433).
Its own module, importing nothing but the model, because its caller is the
worker boot hook in `celery_signals` — which every download task imports via
`celery_app`. Living in `tasks.maintenance` put the whole maintenance import
graph, the membership roster included, on the fetch path, which
`test_gated_reason` forbids.
"""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import literal, update
from sqlalchemy.dialects.postgresql import JSONB
from ..models import DownloadEvent
DOWNLOAD_INTERRUPTED_MESSAGE = (
"interrupted by a worker restart — the next check picks it up where it left off"
)
def interrupt_orphaned_download_events(session, *, booted_at: datetime) -> int:
"""Close the download events a restart orphaned, without blaming the source.
Called when the download lane comes up (#4433). Anything still
pending/running from before this boot belongs to the previous process:
a walk that outlived the 90s stop grace was SIGKILLed, and a queued or
serialize-deferred task is held unacked until Redis redelivers it about an
hour later. Left alone, the 30-min stall sweep would error each one and
bump `consecutive_failures`, backing the source off as if the platform had
failed.
Instead they end as `skipped` (terminal, not a failure) and the source is
not touched: `last_checked_at` keeps its old value, so the next tick finds
it due and the walk resumes from its checkpoint. A redelivered message
that arrives later finds no pending event and opens a fresh one.
An event promoted to running after the boot has `started_at` reset to its
real start (download_service), so it is never caught here. Does NOT commit.
"""
now = datetime.now(UTC)
result = session.execute(
update(DownloadEvent)
.where(DownloadEvent.status.in_(["pending", "running"]))
.where(DownloadEvent.started_at < booted_at)
.values(
status="skipped",
finished_at=now,
error=DOWNLOAD_INTERRUPTED_MESSAGE,
metadata_=DownloadEvent.metadata_.op("||")(
literal({"error_type": "interrupted"}, JSONB)
),
)
.returning(DownloadEvent.id)
)
return len(result.all())
+58 -30
View File
@@ -109,12 +109,20 @@ class ExtensionService:
*, *,
artist_id: int | None = None, artist_id: int | None = None,
artist_name: str | None = None, artist_name: str | None = None,
use_platform_name: bool = False,
) -> dict: ) -> dict:
"""Add `url` as a source. `artist_id` connects it to an existing """Add `url` as a source. `artist_id` connects it to an existing
artist, `artist_name` to that artist (created if new); with neither, artist, `artist_name` to that artist (created if new); with neither,
the artist is resolved from the platform as before.""" the artist is resolved from the platform as before.
`use_platform_name` applies the operator's convention that the Patreon
name is canon: a Patreon source added to an existing artist renames
that artist to the creator's Patreon display name. Name only — the
slug, and every path keyed off it, never moves (#130). Ignored on every
other platform, and when the name can't be read."""
platform, raw_slug = self._derive(url) platform, raw_slug = self._derive(url)
url = canonical_source_url(platform, url, raw_slug) url = canonical_source_url(platform, url, raw_slug)
renamed_from = None
# Identity by SOURCE handle (#130): an existing (platform, url) source # Identity by SOURCE handle (#130): an existing (platform, url) source
# keeps its artist on re-add — even if that artist was since renamed (its # keeps its artist on re-add — even if that artist was since renamed (its
# frozen slug no longer matches the current name), and even when the # frozen slug no longer matches the current name), and even when the
@@ -134,6 +142,8 @@ class ExtensionService:
if artist is None: if artist is None:
raise UnknownArtistError(f"no artist with id {artist_id}") raise UnknownArtistError(f"no artist with id {artist_id}")
created_artist = False created_artist = False
if use_platform_name and platform == "patreon":
renamed_from = await self._adopt_patreon_name(artist, raw_slug, url)
else: else:
name = (artist_name or "").strip() name = (artist_name or "").strip()
if not name: if not name:
@@ -144,7 +154,21 @@ class ExtensionService:
source, created_source = await self._find_or_create_source( source, created_source = await self._find_or_create_source(
artist_id=artist.id, platform=platform, url=url, artist_id=artist.id, platform=platform, url=url,
) )
return self._shape(source, artist, created_source, created_artist) shaped = self._shape(source, artist, created_source, created_artist)
if renamed_from is not None:
shaped["renamed_from"] = renamed_from
return shaped
async def _adopt_patreon_name(self, artist, raw_slug: str, url: str) -> str | None:
"""Rename `artist` to the Patreon display name; the old name when it
changed, else None. Unreadable name → no rename, never the handle."""
name = await self._platform_display_name("patreon", raw_slug, url)
if not name or name == artist.name:
return None
old = artist.name
artist.name = name
await self.session.commit()
return old
async def _existing_source(self, platform: str, url: str) -> Source | None: async def _existing_source(self, platform: str, url: str) -> Source | None:
"""The source this URL already is, whichever artist owns it. Discord """The source this URL already is, whichever artist owns it. Discord
@@ -193,8 +217,18 @@ class ExtensionService:
server_id = raw_slug.split("/", 1)[0] server_id = raw_slug.split("/", 1)[0]
names = await self._discord_names(server_id, None) names = await self._discord_names(server_id, None)
return names.get("server") or f"Discord {server_id}" return names.get("server") or f"Discord {server_id}"
return await self._platform_display_name(platform, raw_slug, url) or raw_slug
async def _platform_display_name(
self, platform: str, raw_slug: str, url: str
) -> str | None:
"""The creator's display name as Patreon or SubscribeStar shows it, read
with the stored cookies; None when it can't be read (no credential, a
network error, a slow answer, any other platform). None, not the handle,
so a caller can tell a real name from a fallback — a rename to the
Patreon name must never rename to a URL handle instead."""
if self._crypto is None or platform not in ("patreon", "subscribestar"): if self._crypto is None or platform not in ("patreon", "subscribestar"):
return raw_slug return None
import asyncio import asyncio
from .credential_service import CredentialService from .credential_service import CredentialService
@@ -204,7 +238,7 @@ class ExtensionService:
if platform == "patreon": if platform == "patreon":
cookies = await cred.get_cookies_path("patreon") cookies = await cred.get_cookies_path("patreon")
from .patreon_resolver import resolve_display_name from .patreon_resolver import resolve_display_name
name = await loop.run_in_executor( call = loop.run_in_executor(
None, resolve_display_name, raw_slug, None, resolve_display_name, raw_slug,
str(cookies) if cookies else None, str(cookies) if cookies else None,
) )
@@ -212,15 +246,14 @@ class ExtensionService:
cookies = await cred.get_cookies_path("subscribestar") cookies = await cred.get_cookies_path("subscribestar")
from .subscribestar_client import SubscribeStarClient from .subscribestar_client import SubscribeStarClient
client = SubscribeStarClient(str(cookies) if cookies else None) client = SubscribeStarClient(str(cookies) if cookies else None)
name = await loop.run_in_executor( call = loop.run_in_executor(None, client.resolve_display_name, url)
None, client.resolve_display_name, url name = await asyncio.wait_for(call, timeout=_NAME_LOOKUP_SECONDS)
)
except Exception as exc: # resolution is best-effort — never block the add except Exception as exc: # resolution is best-effort — never block the add
log.warning("artist display-name resolution failed (%s): %s", platform, exc) log.warning("artist display-name resolution failed (%s): %s", platform, exc)
return raw_slug return None
return name or raw_slug return (name or "").strip() or None
async def probe(self, url: str) -> dict: async def probe(self, url: str, *, names: bool = False) -> dict:
"""Read-only resolution of a creator-page URL against the FC DB. """Read-only resolution of a creator-page URL against the FC DB.
Returns one of: Returns one of:
- {state: 'unknown_platform'} — URL didn't match any - {state: 'unknown_platform'} — URL didn't match any
@@ -236,7 +269,11 @@ class ExtensionService:
— exact (artist, platform, — exact (artist, platform,
url) Source already exists url) Source already exists
Side-effect-free: two SELECTs at most. `names` (the Add panel asks, the chip does not) adds `display_name`:
the creator's name as Patreon/SubscribeStar shows it, or None. It costs
a request to the platform, so a plain page view never pays it.
Side-effect-free: two SELECTs at most, plus that one lookup.
""" """
try: try:
platform, raw_slug = self._derive(url) platform, raw_slug = self._derive(url)
@@ -246,13 +283,16 @@ class ExtensionService:
return await self._probe_discord(raw_slug) return await self._probe_discord(raw_slug)
slug = slugify(raw_slug) slug = slugify(raw_slug)
result: dict = {"platform": platform, "slug": slug}
if names:
result["display_name"] = await self._platform_display_name(
platform, raw_slug, url,
)
artist = (await self.session.execute( artist = (await self.session.execute(
select(Artist).where(Artist.slug == slug) select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none() )).scalar_one_or_none()
if artist is None: if artist is None:
return {"state": "new", "platform": platform, "slug": slug} return {"state": "new", **result}
artist_payload = {"id": artist.id, "name": artist.name, "slug": artist.slug}
source = (await self.session.execute( source = (await self.session.execute(
select(Source).where( select(Source).where(
@@ -262,25 +302,13 @@ class ExtensionService:
) )
)).scalar_one_or_none() )).scalar_one_or_none()
if source is None: if source is None:
return { return {"state": "artist_match", **result, "artist": self._artist_payload(artist)}
"state": "artist_match",
"platform": platform,
"slug": slug,
"artist": artist_payload,
}
return { return {
"state": "source_match", "state": "source_match",
"platform": platform, **result,
"slug": slug, "artist": self._artist_payload(artist),
"artist": artist_payload, "source": self._source_payload(source),
"source": {
"id": source.id,
"artist_id": source.artist_id,
"platform": source.platform,
"url": source.url,
"enabled": source.enabled,
},
} }
async def _probe_discord(self, raw_slug: str) -> dict: async def _probe_discord(self, raw_slug: str) -> dict:
+64 -19
View File
@@ -49,10 +49,8 @@ from .audits import single_color
from .link_extract import extract_external_links from .link_extract import extract_external_links
from .thumbnailer import Thumbnailer from .thumbnailer import Thumbnailer
from .wip_title import ( from .wip_title import (
WIP_TITLE_SOFT_SOURCE,
WIP_TITLE_SOURCE, WIP_TITLE_SOURCE,
apply_wip_image_tags, apply_wip_image_tags,
matches_soft_wip_title,
matches_wip_title, matches_wip_title,
resolve_wip_tag_id, resolve_wip_tag_id,
) )
@@ -428,11 +426,19 @@ class Importer:
return self._upsert_artist(name) if name else None return self._upsert_artist(name) if name else None
def _post_for_sidecar( def _post_for_sidecar(
self, source: Path, artist: Artist | None self, source: Path, artist: Artist | None,
*, source_row: Source | None = None,
) -> Post | None: ) -> Post | None:
"""If a sidecar sits next to `source`, ensure its Source+Post """If a sidecar sits next to `source`, ensure its Source+Post
exist (idempotent) and return the Post — so attachments can link exist (idempotent) and return the Post — so attachments can link
to the same Post the per-member _apply_sidecar will reuse.""" to the same Post the per-member _apply_sidecar will reuse.
`source_row` is the subscription being downloaded, when there is one,
and wins over the (artist, platform) lookup — as it does in
`upsert_post_record`. The lookup takes the artist's FIRST source on the
platform, which is right only while an artist has one: a Discord artist
has one per channel, and every non-image file from a later channel was
filed under the first as an undated second post (#4435)."""
sc = find_sidecar(source) sc = find_sidecar(source)
if sc is None or artist is None: if sc is None or artist is None:
return None return None
@@ -444,6 +450,9 @@ class Importer:
log.warning("sidecar parse failed for %s: %s", sc, exc) log.warning("sidecar parse failed for %s: %s", sc, exc)
return None return None
sd = parse_sidecar(data) sd = parse_sidecar(data)
if source_row is not None:
src = source_row
else:
platform = sd.platform or "unknown" platform = sd.platform or "unknown"
src = self._lookup_source_for_sidecar( src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform, artist_id=artist.id, platform=platform,
@@ -545,7 +554,7 @@ class Importer:
# nothing silently vanishes, matching extract_archive's # nothing silently vanishes, matching extract_archive's
# fail-soft contract. # fail-soft contract.
artist_use = artist if artist is not None else self._resolve_artist(source) artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use) post = self._post_for_sidecar(source, artist_use, source_row=source_row)
self._capture_attachment( self._capture_attachment(
source, post=post, artist=artist_use, resolved=True, source, post=post, artist=artist_use, resolved=True,
) )
@@ -554,7 +563,7 @@ class Importer:
return ImportResult(status="attached", error=reason) return ImportResult(status="attached", error=reason)
artist_use = artist if artist is not None else self._resolve_artist(source) artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use) post = self._post_for_sidecar(source, artist_use, source_row=source_row)
member_ids: list[int] = [] member_ids: list[int] = []
# Every member image touched (new + superseded + deduped), so the # Every member image touched (new + superseded + deduped), so the
# from_attachment_id stamp below covers files that already existed in the # from_attachment_id stamp below covers files that already existed in the
@@ -1040,9 +1049,7 @@ class Importer:
removal sticks. The existing catalogue is covered separately by the removal sticks. The existing catalogue is covered separately by the
operator-triggered backfill sweep. Gated by the settings toggle, and operator-triggered backfill sweep. Gated by the settings toggle, and
best-effort: any failure is logged, never allowed to fail the import.""" best-effort: any failure is logged, never allowed to fail the import."""
hard_on = self.settings.wip_title_tagging_enabled if not self.settings.wip_title_tagging_enabled:
soft_on = self.settings.wip_soft_title_tagging_enabled
if not (hard_on or soft_on):
return return
if record.primary_post_id is None: if record.primary_post_id is None:
return return
@@ -1050,21 +1057,14 @@ class Importer:
title = self.session.execute( title = self.session.execute(
select(Post.post_title).where(Post.id == record.primary_post_id) select(Post.post_title).where(Post.id == record.primary_post_id)
).scalar_one_or_none() ).scalar_one_or_none()
# HARD tier ("WIP"/"work in progress") wins — higher precision, and it if not matches_wip_title(title):
# trains the head; SOFT (sketch/doodle, #1474) is the provisional fallback
# that never trains (source wip_title_soft).
if hard_on and matches_wip_title(title):
source = WIP_TITLE_SOURCE
elif soft_on and matches_soft_wip_title(title):
source = WIP_TITLE_SOFT_SOURCE
else:
return return
if self._wip_tag_id is _UNSET: if self._wip_tag_id is _UNSET:
self._wip_tag_id = resolve_wip_tag_id(self.session) self._wip_tag_id = resolve_wip_tag_id(self.session)
if self._wip_tag_id is None: if self._wip_tag_id is None:
return return
apply_wip_image_tags( apply_wip_image_tags(
self.session, [record.id], self._wip_tag_id, source=source self.session, [record.id], self._wip_tag_id, source=WIP_TITLE_SOURCE
) )
except Exception as exc: # noqa: BLE001 — a tag must never fail an import except Exception as exc: # noqa: BLE001 — a tag must never fail an import
log.warning( log.warning(
@@ -1141,9 +1141,51 @@ class Importer:
if post.artist_id is None: if post.artist_id is None:
post.artist_id = artist.id post.artist_id = artist.id
self._apply_post_fields(post, sd) self._apply_post_fields(post, sd)
self._redate_post_images(post)
self.session.commit() self.session.commit()
return True return True
def _redate_post_images(self, post: Post) -> None:
"""Carry a post's date onto the images already linked to it (#4431).
The native ingesters import a post's media BEFORE its record: the
per-media sidecar holds only the image identity (post-first, #856), and
the date arrives with `_post.json`. So `_attach_provenance` links each
image to a post that has no date yet, and the image keeps its download
time. This runs when the record lands, and applies the same two rules
`_attach_provenance` applies: `effective_date` is the PRIMARY post's
date, and `earliest_post_date` is the earliest date across every post
the image is linked to. Only rows that differ are written."""
if post.post_date is None:
return
self.session.flush()
self.session.execute(
update(ImageRecord)
.where(ImageRecord.primary_post_id == post.id)
.where(ImageRecord.effective_date.is_distinct_from(post.post_date))
.values(effective_date=post.post_date)
.execution_options(synchronize_session=False)
)
linked = select(ImageProvenance.image_record_id).where(
ImageProvenance.post_id == post.id
)
earliest = (
select(func.min(Post.post_date))
.select_from(ImageProvenance)
.join(Post, Post.id == ImageProvenance.post_id)
.where(ImageProvenance.image_record_id == ImageRecord.id)
.where(Post.post_date.is_not(None))
.correlate(ImageRecord)
.scalar_subquery()
)
self.session.execute(
update(ImageRecord)
.where(ImageRecord.id.in_(linked))
.where(ImageRecord.earliest_post_date.is_distinct_from(earliest))
.values(earliest_post_date=earliest)
.execution_options(synchronize_session=False)
)
def attach_in_place( def attach_in_place(
self, self,
path: Path, path: Path,
@@ -1187,7 +1229,10 @@ class Importer:
path, artist=artist, source_row=source, path, artist=artist, source_row=source,
) )
if not is_supported(path): if not is_supported(path):
post = self._post_for_sidecar(path, artist) if artist else None post = (
self._post_for_sidecar(path, artist, source_row=source)
if artist else None
)
return self._capture_attachment( return self._capture_attachment(
path, post=post, artist=artist, resolved=True, path, post=post, artist=artist, resolved=True,
) )
+9 -2
View File
@@ -289,6 +289,11 @@ class Ingester:
# Media handed to phase 3 for import. Marked seen by phase 3 once the # Media handed to phase 3 for import. Marked seen by phase 3 once the
# import has run (`mark_seen_after_import`), not here — see there. # import has run (`mark_seen_after_import`), not here — see there.
fetched: list[tuple[str, str]] = [] fetched: list[tuple[str, str]] = []
# Post-record keys written this walk. Marked with the media, after
# phase 3 has upserted the records — marking them at write time left a
# walk killed before phase 3 with posts the ledger calls recorded that
# the database never dated, and no later tick walks back to them (#4436).
recorded: list[tuple[str, str]] = []
downloaded = 0 downloaded = 0
errors = 0 errors = 0
quarantined = 0 quarantined = 0
@@ -342,7 +347,9 @@ class Ingester:
written_paths=written, written_paths=written,
post_record_paths=list(post_records), post_record_paths=list(post_records),
relink_source_paths=list(relink), relink_source_paths=list(relink),
mark_seen_after_import=lambda: self._mark_seen(source_id, fetched), mark_seen_after_import=lambda: self._mark_seen(
source_id, fetched + recorded,
),
stdout="\n".join(log_lines), stdout="\n".join(log_lines),
stderr="", stderr="",
return_code=return_code, return_code=return_code,
@@ -511,7 +518,7 @@ class Ingester:
posts_with_body += 1 posts_with_body += 1
if rec.path is not None: if rec.path is not None:
post_records.append(str(rec.path)) post_records.append(str(rec.path))
self._mark_seen(source_id, [(pkey, ppid)]) recorded.append((pkey, ppid))
# Per-post handling line in the run stdout (the existing # Per-post handling line in the run stdout (the existing
# "Raw stdout" panel) — the downloader already read the # "Raw stdout" panel) — the downloader already read the
# post; we only format its outcome here. post_type beside # post; we only format its outcome here. post_type beside
+5 -68
View File
@@ -97,8 +97,8 @@ def _sigmoid(z, np):
def _conflict_scores(Xn, Wc, bc, np): def _conflict_scores(Xn, Wc, bc, np):
"""The presentation conflict signal (#141): per row, the MAX content-head """The presentation conflict signal (#141): per row, the MAX content-head
probability and WHICH head produced it. Shared by the system-tag sweep's guard-2 probability and WHICH head produced it — the system-tag sweep's guard-2 asks
and the soft-wip audit — both ask "does this ALSO look like real content?".""" "does this ALSO look like real content?"."""
cprobs = _sigmoid(Xn @ Wc.T + bc, np) cprobs = _sigmoid(Xn @ Wc.T + bc, np)
return cprobs.max(axis=1), cprobs.argmax(axis=1) return cprobs.max(axis=1), cprobs.argmax(axis=1)
@@ -106,10 +106,9 @@ def _conflict_scores(Xn, Wc, bc, np):
def _insert_presentation_review( def _insert_presentation_review(
session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode, session, *, image_record_id, tag_id, conflict_tag_id, conflict_score, mode,
): ):
"""Single-source the ring-loud PresentationReview row shape so the two writers """Single-source the ring-loud PresentationReview row shape, so every writer of
(system-tag sweep guard-2 + soft-wip audit) can't drift on columns or `mode` — the (image_record_id, tag_id) composite PK agrees on columns and `mode` — a
they share the (image_record_id, tag_id) composite PK, so a divergent `mode` divergent `mode` would be a silent first-writer-wins bug."""
would be a silent first-writer-wins bug."""
session.execute( session.execute(
pg_insert(PresentationReview) pg_insert(PresentationReview)
.values( .values(
@@ -963,68 +962,6 @@ def system_tag_auto_apply_sweep(
} }
def soft_wip_conflict_audit(session: Session, dry_run: bool = False) -> dict:
"""Ring-loud audit for the SOFT WIP-title cohort (#1474). Images auto-tagged
`wip` from a low-precision sketch/doodle title (source='wip_title_soft') that ALSO
score >= the process conflict threshold on a content head are probably FINISHED
art mis-tagged as process — flag them (PresentationReview, mode='process') so the
review strip surfaces them ("also looks like <X>", Keep tag / Remove tag). Does
NOT remove the tag; the operator decides. No-op when there are no content heads.
numpy-only. Returns {n_scanned, n_flagged}."""
import numpy as np
from ..wip_title import WIP_TITLE_SOFT_SOURCE, resolve_wip_tag_id
settings = _settings(session)
ver = settings.embedder_model_version
conflict_thr = float(settings.process_conflict_threshold)
conf = _conflict_heads(session, ver)
wip_id = resolve_wip_tag_id(session)
if not conf or wip_id is None:
return {"n_scanned": 0, "n_flagged": 0}
Wc = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in conf])
bc = np.asarray([r.bias for r in conf], dtype=np.float32)
conf_tag_ids = [r.tag_id for r in conf]
soft_ids = [iid for (iid,) in session.execute(
select(image_tag.c.image_record_id)
.where(image_tag.c.tag_id == wip_id)
.where(image_tag.c.source == WIP_TITLE_SOFT_SOURCE)
)]
# Skip images already flagged for this tag (idempotent re-runs).
flagged = {iid for (iid,) in session.execute(
select(PresentationReview.image_record_id)
.where(PresentationReview.tag_id == wip_id)
)}
soft_ids = [i for i in soft_ids if i not in flagged]
n_flagged = 0
scanned = 0
for start in range(0, len(soft_ids), _AUTO_APPLY_CHUNK):
chunk = soft_ids[start:start + _AUTO_APPLY_CHUNK]
emb = _load_embeddings(session, chunk)
cids = [i for i in chunk if i in emb]
if not cids:
continue
scanned += len(cids)
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
max_c, arg_c = _conflict_scores(Xn, Wc, bc, np)
for k in range(len(cids)):
if float(max_c[k]) >= conflict_thr:
n_flagged += 1
if not dry_run:
_insert_presentation_review(
session,
image_record_id=cids[k], tag_id=wip_id,
conflict_tag_id=conf_tag_ids[int(arg_c[k])],
conflict_score=float(max_c[k]),
mode="process",
)
if not dry_run:
session.commit()
return {"n_scanned": scanned, "n_flagged": n_flagged}
def retract_auto_applied_heads(session: Session) -> int: def retract_auto_applied_heads(session: Session) -> int:
"""Soft auto-apply (milestone 139): re-score every standing source='head_auto' """Soft auto-apply (milestone 139): re-score every standing source='head_auto'
tag against its CURRENT head and REMOVE the ones now BELOW the head's tag against its CURRENT head and REMOVE the ones now BELOW the head's
-3
View File
@@ -32,11 +32,8 @@ from ...models.tag import image_tag
# `process_auto` (#1464): wip/editor screenshot applied by the process sweep are # `process_auto` (#1464): wip/editor screenshot applied by the process sweep are
# ALSO provisional — the head must learn only from title (`wip_title`) + manual # ALSO provisional — the head must learn only from title (`wip_title`) + manual
# labels, never its own auto-applied output, or it would runaway (operator 2026-07-12). # labels, never its own auto-applied output, or it would runaway (operator 2026-07-12).
# `wip_title_soft` (#1474): the soft title tier (sketch/doodle) is LOW-precision, so
# it's provisional too — a finished piece titled "sketch" must not train the wip head.
_AUTO_SOURCES = ( _AUTO_SOURCES = (
"head_auto", "ccip_auto", "ml_auto", "presentation_auto", "process_auto", "head_auto", "ccip_auto", "ml_auto", "presentation_auto", "process_auto",
"wip_title_soft",
) )
+19
View File
@@ -60,3 +60,22 @@ def platform_lock(platform: str, *, ttl_seconds: int):
except redis.RedisError as exc: # pragma: no cover - broker outage except redis.RedisError as exc: # pragma: no cover - broker outage
log.warning("platform_lock unavailable for %s: %s", platform, exc) log.warning("platform_lock unavailable for %s: %s", platform, exc)
return None return None
def release_all_platform_locks() -> int:
"""Drop every serialized platform's lock. Returns how many were held.
Only for the download lane's boot (#4433). A worker that restarts mid-walk
is SIGKILLed past its stop grace, so its `finally` never releases the lock,
and the TTL keeps every other source on that platform bouncing for up to
27 minutes after the new worker is ready. At boot no walk of ours can be
running, so a held lock names a dead one. Assumes one download consumer —
the only shape FC deploys; a second replica booting would free a live
walk's lock (not corrupt it: the walk runs on, a second walk may overlap it).
"""
try:
client = _redis()
return int(client.delete(*(f"{_LOCK_PREFIX}{p}" for p in SERIALIZED_PLATFORMS)))
except redis.RedisError as exc: # pragma: no cover - broker outage
log.warning("could not release platform locks at boot: %s", exc)
return 0
@@ -0,0 +1,86 @@
"""Date the posts whose record reached the disk but never the database (#4436).
A native walk writes each post's record (`_post.json`, or Discord's
`<day>_<message id>_post.json`) as it goes, and phase 3 upserts those records
after the walk — which is when the post, and through it its images, get their
date. Until #4436 the walk marked a record's post key seen at write time, so a
walk killed before phase 3 (a restart, a stall) left the post undated with the
ledger saying it was done, and no later tick walked back that far to fix it.
The record files are still on disk. This finds each undated native post's
record under its artist's folder and upserts it with the post's OWN source —
never the (artist, platform) lookup, which picks the artist's first source and
would misfile a Discord channel's post (#4435). Idempotent: a post that is
already dated is never looked at, and upserting a record twice changes nothing.
"""
from __future__ import annotations
import json
import logging
from collections import defaultdict
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from ..models import Artist, Post, Source
from ..utils.sidecar import parse_sidecar
from .download_backends import NATIVE_INGESTER_PLATFORMS
log = logging.getLogger(__name__)
def _is_record(path: Path) -> bool:
return path.name == "_post.json" or path.name.endswith("_post.json")
def _record_id(path: Path) -> str | None:
try:
data = json.loads(path.read_text("utf-8"))
except (OSError, ValueError):
return None
if not isinstance(data, dict):
return None
return parse_sidecar(data).external_post_id
def date_posts_from_records(session: Session, importer, images_root: Path) -> dict:
"""Upsert the on-disk record of every undated native post. Returns counts."""
rows = session.execute(
select(Post.external_post_id, Post.source_id, Post.artist_id, Source.platform)
.join(Source, Source.id == Post.source_id)
.where(
Post.post_date.is_(None),
Post.synthesized_by.is_(None),
Source.platform.in_(NATIVE_INGESTER_PLATFORMS),
)
).all()
wanted: dict[tuple[int, str], dict[str, int]] = defaultdict(dict)
for epid, source_id, artist_id, platform in rows:
wanted[(artist_id, platform)][epid] = source_id
summary = {"undated": len(rows), "dated": 0, "no_record": 0}
for (artist_id, platform), by_epid in wanted.items():
artist = session.get(Artist, artist_id)
root = Path(images_root) / artist.slug / platform if artist else None
if root is None or not root.is_dir():
summary["no_record"] += len(by_epid)
continue
found = 0
for record in root.rglob("*post.json"):
if not _is_record(record):
continue
epid = _record_id(record)
source_id = by_epid.pop(epid, None) if epid else None
if source_id is None:
continue
source = session.get(Source, source_id)
if importer.upsert_post_record(record, artist=artist, source=source):
found += 1
if not by_epid:
break
summary["dated"] += found
summary["no_record"] += len(by_epid)
if summary["undated"]:
log.info("date_posts_from_records: %s", summary)
return summary
+6 -26
View File
@@ -27,13 +27,10 @@ from .image_tag_apply import insert_image_tags
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other # image_tag.source stamped on title-heuristic WIP tags — distinct from the other
# apply sources so provenance stays legible and a future undo can target only these. # apply sources so provenance stays legible and a future undo can target only these.
# HARD tier ("WIP"/"work in progress") is high-precision → trains the wip head. # Only the artist's own "WIP"/"work in progress" label counts — high-precision, so it
# trains the wip head. A sketch/doodle/scribble tier (#1474) was retired in milestone
# 430: a "sketch" is usually finished art, and its 6k tags flooded the review strip.
WIP_TITLE_SOURCE = "wip_title" WIP_TITLE_SOURCE = "wip_title"
# SOFT tier (sketch/doodle/scribble, #1474) is LOWER-precision — a finished "sketch"
# is often not WIP. This source is PROVISIONAL (in training_data._AUTO_SOURCES) so it
# NEVER trains the wip head; a soft-tagged image that also looks like real content is
# surfaced by the ring-loud audit for review.
WIP_TITLE_SOFT_SOURCE = "wip_title_soft"
# A standalone "WIP" / "W.I.P" token, or the phrase "work in progress" # A standalone "WIP" / "W.I.P" token, or the phrase "work in progress"
# (space/underscore/hyphen separated). The letter-boundary lookarounds are what # (space/underscore/hyphen separated). The letter-boundary lookarounds are what
@@ -45,20 +42,10 @@ _WIP_RE = re.compile(
re.IGNORECASE, re.IGNORECASE,
) )
# Soft tier: sketch / doodle / scribble (+ plurals), letter-boundary anchored so # Coarse SQL prefilter for the backfill sweep — narrows the post scan to rows that
# "sketchbook" / "kadoodle" don't trip it. Deliberately conservative — recall is
# secondary because the soft source doesn't train the head and the ring-loud audit
# catches false positives.
_SOFT_WIP_RE = re.compile(
r"(?<![A-Za-z])(?:sketch|sketches|doodle|doodles|scribble|scribbles)(?![A-Za-z])",
re.IGNORECASE,
)
# Coarse SQL prefilters for the backfill sweep — narrow the post scan to rows that
# COULD match before the precise regex confirms. Case-insensitive ILIKE patterns. # COULD match before the precise regex confirms. Case-insensitive ILIKE patterns.
# Each MUST stay a SUPERSET of its regex or the sweep would silently miss posts. # It MUST stay a SUPERSET of the regex or the sweep would silently miss posts.
WIP_TITLE_SQL_PREFILTER = ("%wip%", "%work%progress%") WIP_TITLE_SQL_PREFILTER = ("%wip%", "%work%progress%")
SOFT_WIP_TITLE_SQL_PREFILTER = ("%sketch%", "%doodle%", "%scribble%")
# Chunk bulk inserts so a large sweep can't blow past psycopg's 65535-parameter # Chunk bulk inserts so a large sweep can't blow past psycopg's 65535-parameter
# ceiling (3 params/row → ~21k rows max; 5k stays comfortably under). # ceiling (3 params/row → ~21k rows max; 5k stays comfortably under).
@@ -66,19 +53,12 @@ _INSERT_CHUNK = 5000
def matches_wip_title(title: str | None) -> bool: def matches_wip_title(title: str | None) -> bool:
"""True when a post title explicitly marks it work-in-progress (HARD tier).""" """True when a post title explicitly marks it work-in-progress."""
if not title: if not title:
return False return False
return _WIP_RE.search(title) is not None return _WIP_RE.search(title) is not None
def matches_soft_wip_title(title: str | None) -> bool:
"""True when a title carries a SOFT WIP cue (sketch/doodle/scribble, #1474)."""
if not title:
return False
return _SOFT_WIP_RE.search(title) is not None
def resolve_wip_tag_id(session: Session) -> int | None: def resolve_wip_tag_id(session: Session) -> int | None:
"""The seeded ``wip`` system tag's id (migration 0075), or None if absent.""" """The seeded ``wip`` system tag's id (migration 0075), or None if absent."""
return session.execute( return session.execute(
+47 -21
View File
@@ -137,7 +137,13 @@ IMPORT_BATCH_KEEP_DAYS = 30
# (the import queue itself stays at the 5-min default for single # (the import queue itself stays at the 5-min default for single
# files); time_limit=2100. # files); time_limit=2100.
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = { QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"ml": 25, # ml: the scheduled auto-apply sweeps and refresh_character_prototypes run
# to a 35-min hard limit (2100s); 25 swept them mid-run (#4432). The two
# 65-min jobs have their own entries below.
"ml": 40,
# import: import_media_file's hard limit is 6 min (360s), one past the
# 5-min default this queue fell to (#4432).
"import": 10,
# download_source legitimately walks 5-25 min (Patreon/gallery-dl # download_source legitimately walks 5-25 min (Patreon/gallery-dl
# deep creators); its hard time_limit is DOWNLOAD_HARD_TIME_LIMIT # deep creators); its hard time_limit is DOWNLOAD_HARD_TIME_LIMIT
# (1500s = 25m). The 5-min default flagged healthy in-flight walks as # (1500s = 25m). The 5-min default flagged healthy in-flight walks as
@@ -154,6 +160,12 @@ QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
# overrides below cover the outliers (backups, library audit). # overrides below cover the outliers (backups, library audit).
"maintenance": 75, "maintenance": 75,
"scan": 75, "scan": 75,
# The long lane (#4432). Until TaskRun.queue asked the router, nothing was
# recorded here: these runs read as `maintenance` (75) or, for
# translation, `default` (5 — which failed healthy 35-min runs). The
# longest task without its own entry below is the admin family at a
# 40-min hard limit; 45 = 40 + 5.
"maintenance_long": 45,
} }
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = { TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"backend.app.tasks.import_file.import_archive_file": 40, "backend.app.tasks.import_file.import_archive_file": 40,
@@ -179,6 +191,10 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
# external-fetch entry above — without an override a healthy in-flight walk # external-fetch entry above — without an override a healthy in-flight walk
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5. # is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30, "backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
# Head training and the manual head apply run to 65 min (3900s) — past the
# ml queue's threshold (#4432). 70 = 65 + 5.
"backend.app.tasks.ml.train_heads": 70,
"backend.app.tasks.ml.apply_head_tags": 70,
} }
@@ -715,6 +731,32 @@ def recover_stalled_download_events() -> int:
return events_recovered return events_recovered
@celery.task(
name="backend.app.tasks.maintenance.date_posts_from_records",
soft_time_limit=1500,
time_limit=1800,
)
def date_posts_from_records() -> dict:
"""Date undated native posts from the records their walk left on disk
(#4436). Hourly and self-limiting: once every post is dated it is one
empty query."""
from ..services.importer import Importer
from ..services.post_record_repair import date_posts_from_records as _repair
from ..services.thumbnailer import Thumbnailer
images_root = IMAGES_ROOT
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
importer = Importer(
session=session,
images_root=images_root,
import_root=images_root,
thumbnailer=Thumbnailer(images_root=images_root),
settings=ImportSettings.load_sync(session),
)
return _repair(session, importer, images_root)
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_backup_runs") @celery.task(name="backend.app.tasks.maintenance.recover_stalled_backup_runs")
def recover_stalled_backup_runs() -> int: def recover_stalled_backup_runs() -> int:
"""Flip BackupRun rows stuck in running/restoring past the hard limit """Flip BackupRun rows stuck in running/restoring past the hard limit
@@ -1041,8 +1083,7 @@ def cleanup_old_download_events() -> int:
def _backfill_wip_tier(session, tag_id, prefilter, matcher, source) -> int: def _backfill_wip_tier(session, tag_id, prefilter, matcher, source) -> int:
"""One keyset-paginated pass over posts whose title matches a WIP tier, applying """One keyset-paginated pass over posts whose title matches a WIP tier, applying
`tag_id` (stamped `source`) to their images. Shared by the hard + soft tiers `tag_id` (stamped `source`) to their images (#1458). Coarse `prefilter` (ILIKE superset) narrows the scan; the precise
(#1458 / #1474). Coarse `prefilter` (ILIKE superset) narrows the scan; the precise
`matcher` confirms. Idempotent-additive (ON CONFLICT DO NOTHING). Returns the row `matcher` confirms. Idempotent-additive (ON CONFLICT DO NOTHING). Returns the row
count newly applied.""" count newly applied."""
from ..models import Post from ..models import Post
@@ -1082,26 +1123,17 @@ def _backfill_wip_tier(session, tag_id, prefilter, matcher, source) -> int:
) )
def backfill_wip_title_tags() -> int: def backfill_wip_title_tags() -> int:
"""Scan EXISTING posts for WIP titles and apply the `wip` system tag to their """Scan EXISTING posts for WIP titles and apply the `wip` system tag to their
images — the operator-triggered back-catalogue catch-up (task #1458 hard tier + images — the operator-triggered back-catalogue catch-up (task #1458). New
#1474 soft tier). New imports are tagged live by the importer; this covers the imports are tagged live by the importer; this covers the existing library.
existing library. Keyset-paginated, restart-safe.
HARD tier ("WIP"/"work in progress") always runs (the operator triggered the
scan); the SOFT tier (sketch/doodle, provisional source) runs only when
wip_soft_title_tagging_enabled, AFTER hard so a title matching both keeps the
trained hard tag (ON CONFLICT DO NOTHING). Keyset-paginated, restart-safe.
Deliberately NOT scheduled as a beat: a periodic re-run would re-apply to matching Deliberately NOT scheduled as a beat: a periodic re-run would re-apply to matching
posts and silently undo a manual WIP removal, so it stays an explicit operator posts and silently undo a manual WIP removal, so it stays an explicit operator
action (Settings → "Scan existing posts for WIP titles"). Returns rows applied. action (Settings → "Scan existing posts for WIP titles"). Returns rows applied.
""" """
from ..models import ImportSettings
from ..services.wip_title import ( from ..services.wip_title import (
SOFT_WIP_TITLE_SQL_PREFILTER,
WIP_TITLE_SOFT_SOURCE,
WIP_TITLE_SOURCE, WIP_TITLE_SOURCE,
WIP_TITLE_SQL_PREFILTER, WIP_TITLE_SQL_PREFILTER,
matches_soft_wip_title,
matches_wip_title, matches_wip_title,
resolve_wip_tag_id, resolve_wip_tag_id,
) )
@@ -1114,16 +1146,10 @@ def backfill_wip_title_tags() -> int:
"backfill_wip_title_tags: no `wip` system tag present; nothing to do" "backfill_wip_title_tags: no `wip` system tag present; nothing to do"
) )
return 0 return 0
settings = ImportSettings.load_sync(session)
applied = _backfill_wip_tier( applied = _backfill_wip_tier(
session, tag_id, WIP_TITLE_SQL_PREFILTER, matches_wip_title, session, tag_id, WIP_TITLE_SQL_PREFILTER, matches_wip_title,
WIP_TITLE_SOURCE, WIP_TITLE_SOURCE,
) )
if settings.wip_soft_title_tagging_enabled:
applied += _backfill_wip_tier(
session, tag_id, SOFT_WIP_TITLE_SQL_PREFILTER, matches_soft_wip_title,
WIP_TITLE_SOFT_SOURCE,
)
if applied: if applied:
log.info("backfill_wip_title_tags: applied wip to %d image(s)", applied) log.info("backfill_wip_title_tags: applied wip to %d image(s)", applied)
return applied return applied
-18
View File
@@ -620,24 +620,6 @@ def scheduled_process_auto_apply() -> str:
return f"applied={result['n_applied']} flagged={result['n_flagged']}" return f"applied={result['n_applied']} flagged={result['n_flagged']}"
@celery.task(
name="backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
soft_time_limit=1800, time_limit=2100,
)
def scheduled_soft_wip_conflict_audit() -> str:
"""Ring-loud audit over the SOFT WIP-title cohort (#1474) — flag sketch/doodle
auto-tags that ALSO look like real content for review. No-op when there are no
content heads; idempotent (already-flagged images skipped). Runs regardless of
the process-sweep toggle, since soft-title tags come from the importer, not that
sweep. Wall-clock bounded by the task time limits."""
from ..services.ml.heads import soft_wip_conflict_audit
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
result = soft_wip_conflict_audit(session)
return f"scanned={result['n_scanned']} flagged={result['n_flagged']}"
@celery.task(name="backend.app.tasks.ml.prune_presentation_reviews") @celery.task(name="backend.app.tasks.ml.prune_presentation_reviews")
def prune_presentation_reviews() -> str: def prune_presentation_reviews() -> str:
"""Retention (rule 89): drop RESOLVED presentation-review flags older than 30 """Retention (rule 89): drop RESOLVED presentation-review flags older than 30
+4 -1
View File
@@ -180,7 +180,10 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
BUILD_REF` that every checkout in the file takes, rather than per job — BUILD_REF` that every checkout in the file takes, rather than per job —
otherwise `sign-extension` would derive dev's extension version while otherwise `sign-extension` would derive dev's extension version while
`build-web` bundled main's, and the release download would 404 on a version `build-web` bundled main's, and the release download would 404 on a version
that exists perfectly well. Every job then ASSERTS its checkout is `main` that exists perfectly well. On every other trigger `BUILD_REF` is the
triggering COMMIT (`github.sha`), not the branch: a branch is re-resolved
per job, so a push landing mid-run used to move the publishing jobs onto a
commit the run's lanes never tested (run 7499, #4427). Every job then ASSERTS its checkout is `main`
before doing anything, because `env` inside `with:` is not a context this before doing anything, because `env` inside `with:` is not a context this
runner is known to evaluate — if it silently resolved to empty, checkout runner is known to evaluate — if it silently resolved to empty, checkout
would fall back to the triggering ref and the refresh would publish dev's would fall back to the triggering ref and the refresh would publish dev's
+8 -2
View File
@@ -88,7 +88,12 @@ async function checkForUpdateInfo() {
currentVersion, currentVersion,
latestVersion, latestVersion,
channel, channel,
xpiUrl: info && info.latest_url ? `${base}${info.latest_url}` : null, // Where the Update button sends the operator: FC's own install card, not
// the XPI. Firefox refuses an add-on install whose navigation an extension
// started (tabs.create on the .xpi dies with NS_ERROR_FAILURE — operator-
// flagged 2026-09-25); it accepts one from a user click on a web page,
// which is exactly what the card's Install button is.
installPageUrl: base ? `${base}/subscriptions?tab=settings` : null,
}; };
} }
@@ -270,6 +275,7 @@ browser.runtime.onMessage.addListener(async (msg) => {
return await api.quickAddSource(msg.url, { return await api.quickAddSource(msg.url, {
artistId: msg.artistId ?? null, artistId: msg.artistId ?? null,
artistName: msg.artistName ?? null, artistName: msg.artistName ?? null,
usePlatformName: msg.usePlatformName === true,
}); });
} catch (e) { } catch (e) {
return { error: e.message }; return { error: e.message };
@@ -284,7 +290,7 @@ browser.runtime.onMessage.addListener(async (msg) => {
case 'PROBE_SOURCE': case 'PROBE_SOURCE':
try { try {
return await api.probeSource(msg.url); return await api.probeSource(msg.url, { names: msg.names === true });
} catch (e) { } catch (e) {
return { error: e.message }; return { error: e.message };
} }
+3
View File
@@ -97,3 +97,6 @@
.fc-panel__result-tag { font-size: 11px; color: rgb(140, 220, 160); } .fc-panel__result-tag { font-size: 11px; color: rgb(140, 220, 160); }
.fc-panel__empty { padding: 6px 9px; font-size: 12px; color: rgb(170, 166, 156); } .fc-panel__empty { padding: 6px 9px; font-size: 12px; color: rgb(170, 166, 156); }
.fc-panel__hint--match { color: rgb(140, 220, 160); } .fc-panel__hint--match { color: rgb(140, 220, 160); }
/* The panel's own [hidden] — its rows are display:flex, which beats the UA's. */
.fc-panel [hidden] { display: none !important; }
.fc-panel__rename { margin-top: 8px; font-size: 13px; }
+63 -19
View File
@@ -84,13 +84,14 @@
await openArtist(btn, probe.artist?.slug); await openArtist(btn, probe.artist?.slug);
return; return;
} }
// A Discord URL names a channel, not a creator — ask which artist. // Every add goes through the panel, so the operator can match the page to
if (probe?.platform === 'discord') { // an artist FabledCurator already has (the same creator is often spelled
if (document.getElementById('fc-discord-panel')) closePanel(); // differently per platform) instead of minting a duplicate.
else openDiscordPanel(probe); if (document.getElementById('fc-add-panel')) {
closePanel();
return; return;
} }
await add(btn, { url: window.location.href }); await openAddPanel(btn, probe);
} }
async function openArtist(btn, slug) { async function openArtist(btn, slug) {
@@ -121,7 +122,8 @@
return false; return false;
} }
const verb = r.created_source ? 'Added to' : 'Already a source for'; const verb = r.created_source ? 'Added to' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success'); const renamed = r.renamed_from ? ` — renamed from “${r.renamed_from}”` : '';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})${renamed}`, 'success');
// Re-probe so the chip flips green without waiting for a navigation. // Re-probe so the chip flips green without waiting for a navigation.
evaluate(); evaluate();
return true; return true;
@@ -134,10 +136,13 @@
} }
} }
// ---- Discord Add panel ---- // ---- Add panel ----
// Where: this channel or the whole server. Who: the suggested artist, one // Who: the suggested artist, one found by search, or a new one by name —
// found by search, or a new one by name. Built with createElement only — // on every platform. Where (Discord only): this channel or the whole
// server, channel and artist names are other people's text. // server. On Patreon, joining an artist known by another name offers the
// Patreon name, which the operator treats as canon. Built with
// createElement only — server, channel and artist names are other people's
// text.
function el(tag, props = {}, children = []) { function el(tag, props = {}, children = []) {
const node = document.createElement(tag); const node = document.createElement(tag);
@@ -150,13 +155,35 @@
} }
function closePanel() { function closePanel() {
document.getElementById('fc-discord-panel')?.remove(); document.getElementById('fc-add-panel')?.remove();
} }
function openDiscordPanel(probe) { async function openAddPanel(btn, probe) {
closePanel(); closePanel();
const platformName = PLATFORMS[probe.platform]?.name || probe.platform;
// Discord's probe already carries its names. Patreon/SubscribeStar read the
// creator's display name only now, when the panel needs it — a request to
// the platform the chip's own probe deliberately doesn't make.
if (probe.platform !== 'discord') {
const original = btn.textContent;
btn.disabled = true;
btn.textContent = `Reading the ${platformName} name…`;
try {
const named = await browser.runtime.sendMessage({
type: 'PROBE_SOURCE', url: window.location.href, names: true,
});
if (named && !named.error) probe = named;
} catch { /* fall back to the chip's probe: the URL handle */ }
btn.disabled = false;
btn.textContent = original;
if (probe.state === 'source_match') {
renderButton(probe);
return;
}
}
const d = probe.discord || {}; const d = probe.discord || {};
const choice = discordPanelDefaults(probe); const discord = probe.platform === 'discord';
const choice = panelDefaults(probe, window.location.href);
const scopeRow = (value, label, disabled) => { const scopeRow = (value, label, disabled) => {
const input = el('input', { const input = el('input', {
@@ -186,14 +213,27 @@
const addBtn = el('button', { class: 'fc-panel__btn fc-panel__btn--primary', text: 'Add' }); const addBtn = el('button', { class: 'fc-panel__btn fc-panel__btn--primary', text: 'Add' });
const cancelBtn = el('button', { class: 'fc-panel__btn', text: 'Cancel' }); const cancelBtn = el('button', { class: 'fc-panel__btn', text: 'Cancel' });
const panel = el('div', { id: 'fc-discord-panel', class: 'fc-panel' }, [ // Patreon is canon: joining an artist known by another name takes the
el('div', { class: 'fc-panel__title', text: 'Add Discord source' }), // Patreon name unless this is unticked. Shown only when it would rename.
el('div', { class: 'fc-panel__sub', text: serverLabel(d) }), const renameBox = el('input', { type: 'checkbox', checked: choice.adoptPlatformName });
const renameText = el('span');
const renameRow = el('label', { class: 'fc-panel__radio fc-panel__rename' }, [renameBox, renameText]);
renameBox.addEventListener('change', () => { choice.adoptPlatformName = renameBox.checked; refresh(); });
const sub = discord
? serverLabel(d)
: [probe.display_name, probe.slug].filter(Boolean).filter((v, i, a) => a.indexOf(v) === i).join(' · ');
const panel = el('div', { id: 'fc-add-panel', class: 'fc-panel' }, [
el('div', { class: 'fc-panel__title', text: `Add ${platformName} source` }),
el('div', { class: 'fc-panel__sub', text: sub }),
...(discord ? [
el('div', { class: 'fc-panel__label', text: 'Follow' }), el('div', { class: 'fc-panel__label', text: 'Follow' }),
scopeRow('channel', d.channel_id ? channelLabel(d) : 'this channel', !d.channel_id), scopeRow('channel', d.channel_id ? channelLabel(d) : 'this channel', !d.channel_id),
scopeRow('server', `Every channel in ${serverLabel(d)}`, false), scopeRow('server', `Every channel in ${serverLabel(d)}`, false),
] : []),
el('div', { class: 'fc-panel__label', text: 'Artist' }), el('div', { class: 'fc-panel__label', text: 'Artist' }),
el('div', { class: 'fc-panel__combo' }, [nameInput, results]), el('div', { class: 'fc-panel__combo' }, [nameInput, results]),
renameRow,
hint, hint,
el('div', { class: 'fc-panel__actions' }, [cancelBtn, addBtn]), el('div', { class: 'fc-panel__actions' }, [cancelBtn, addBtn]),
]); ]);
@@ -205,12 +245,16 @@
let listOpen = false; let listOpen = false;
function refresh() { function refresh() {
const req = discordAddRequest(choice); const req = addRequest(choice);
addBtn.disabled = !req; addBtn.disabled = !req;
if (!req) hint.textContent = 'Pick an artist or type a name.'; if (!req) hint.textContent = 'Pick an artist or type a name.';
else if (req.artistId != null) hint.textContent = `✓ Connects to ${choice.artist.name}, already in FabledCurator.`; else if (req.artistId != null) hint.textContent = `✓ Connects to ${choice.artist.name}, already in FabledCurator.`;
else hint.textContent = `Creates a new artist “${req.artistName}”.`; else if (req.artistName) hint.textContent = `Creates a new artist “${req.artistName}”.`;
else hint.textContent = `Creates a new artist, named from the ${platformName} page.`;
hint.classList.toggle('fc-panel__hint--match', !!req && req.artistId != null); hint.classList.toggle('fc-panel__hint--match', !!req && req.artistId != null);
const offer = renameOffer(choice);
renameRow.hidden = !offer;
if (offer) renameText.textContent = `Rename “${offer.from}” to the Patreon name “${offer.to}”`;
} }
function pick(artist) { function pick(artist) {
@@ -346,7 +390,7 @@
cancelBtn.addEventListener('click', closePanel); cancelBtn.addEventListener('click', closePanel);
addBtn.addEventListener('click', async () => { addBtn.addEventListener('click', async () => {
const req = discordAddRequest(choice); const req = addRequest(choice);
if (!req) return; if (!req) return;
const btn = document.getElementById('fc-add-source-btn'); const btn = document.getElementById('fc-add-source-btn');
addBtn.disabled = true; addBtn.disabled = true;
+11 -4
View File
@@ -104,16 +104,23 @@ class FabledCuratorAPI {
// artistId connects the source to an existing artist, artistName to the // artistId connects the source to an existing artist, artistName to the
// artist of that name (created if new); with neither the server derives the // artist of that name (created if new); with neither the server derives the
// artist from the URL. A Discord channel always sends one. // artist from the URL. A Discord channel always sends one.
quickAddSource(url, { artistId = null, artistName = null } = {}) { // usePlatformName: a Patreon source joining an existing artist renames it
// to the Patreon display name (Patreon is canon; name only, never the slug).
quickAddSource(url, { artistId = null, artistName = null, usePlatformName = false } = {}) {
const body = { url }; const body = { url };
if (artistId != null) body.artist_id = artistId; if (artistId != null) body.artist_id = artistId;
else if (artistName) body.artist_name = artistName; else if (artistName) body.artist_name = artistName;
if (usePlatformName) body.use_platform_name = true;
return this.request('POST', '/extension/quick-add-source', body); return this.request('POST', '/extension/quick-add-source', body);
} }
probeSource(url) { probeSource(url, { names = false } = {}) {
// Read-only existence check. Drives the content-script chip's // Read-only existence check. Drives the content-script chip's
// color/copy BEFORE the operator clicks Add. // color/copy BEFORE the operator clicks Add. `names` also reads the
const qs = new URLSearchParams({ url }).toString(); // creator's display name from the platform — the Add panel asks for it,
// the chip doesn't, so a plain page view never costs a platform request.
const params = { url };
if (names) params.names = '1';
const qs = new URLSearchParams(params).toString();
return this.request('GET', `/extension/probe?${qs}`); return this.request('GET', `/extension/probe?${qs}`);
} }
// Latest published extension version on this instance — drives the in-app // Latest published extension version on this instance — drives the in-app
+53 -14
View File
@@ -1,7 +1,7 @@
/** /**
* The content script's decisions, kept apart from its DOM so the specs can * The content script's decisions, kept apart from its DOM so the specs can
* load them (test/chip.spec.js): which state the chip shows, what it says, * load them (test/chip.spec.js): which state the chip shows, what it says,
* and what the Discord Add panel starts out proposing. * and what the Add panel starts out proposing and finally sends.
* *
* `probe` is /api/extension/probe's answer; `platformName` is the display * `probe` is /api/extension/probe's answer; `platformName` is the display
* name (PLATFORMS[key].name), passed in so this file needs no other lib. * name (PLATFORMS[key].name), passed in so this file needs no other lib.
@@ -46,37 +46,76 @@ function serverLabel(d) {
} }
/** /**
* What the Discord Add panel opens with. The channel is the default scope * What the Add panel opens with, on any platform.
* when there is one: a server source walks every channel the token can read, *
* which is rarely what a single art channel wants. The artist is the probe's * Discord: the channel is the default scope when there is one — a server
* suggestion (the owner of another source on this server), else a new artist * source walks every channel the token can read, which is rarely what a
* named after the server. * single art channel wants. The artist is the probe's suggestion (the owner
* of another source on this server), else a new one named after the server.
*
* Patreon / SubscribeStar: the page URL is the source. The artist is the one
* whose slug the URL already names (artist_match), else a new one under the
* creator's display name — `probe.display_name`, from the probe the panel
* makes with names=1 — falling back to the URL handle, which the server
* resolves on its own when it is left untouched (`nameIsHandle`).
*/ */
function discordPanelDefaults(probe) { function panelDefaults(probe, pageUrl) {
const d = probe?.discord || {}; const d = probe?.discord || {};
const suggested = probe?.state === 'artist_match' && probe.artist ? probe.artist : null; const suggested = probe?.state === 'artist_match' && probe.artist ? probe.artist : null;
const discord = probe?.platform === 'discord';
const shown = probe?.display_name || null;
let artistName = '';
if (suggested) artistName = suggested.name;
else if (discord) artistName = d.server_name || '';
else artistName = shown || probe?.slug || '';
return { return {
scope: d.channel_id ? 'channel' : 'server', platform: probe?.platform || null,
scope: discord ? (d.channel_id ? 'channel' : 'server') : 'page',
pageUrl: pageUrl || null,
channelUrl: d.channel_url || null, channelUrl: d.channel_url || null,
serverUrl: d.server_url || null, serverUrl: d.server_url || null,
artist: suggested ? { id: suggested.id, name: suggested.name } : null, artist: suggested ? { id: suggested.id, name: suggested.name } : null,
artistName: suggested ? suggested.name : (d.server_name || ''), artistName,
nameIsHandle: !discord && !suggested && !shown,
handle: probe?.slug || '',
displayName: shown,
// Patreon is canon: joining an artist known by another name takes the
// Patreon name, unless the operator unticks it.
adoptPlatformName: true,
}; };
} }
/**
* The rename the panel offers, or null: only on Patreon (the canon name),
* only when joining an existing artist, only with a name actually read from
* Patreon, and only when it differs from what the artist is called now.
*/
function renameOffer(choice) {
if (choice.platform !== 'patreon' || !choice.artist || !choice.displayName) return null;
if (choice.artist.name === choice.displayName) return null;
return { from: choice.artist.name, to: choice.displayName };
}
/** /**
* The quick-add body for the panel's current choice. A picked artist goes by * The quick-add body for the panel's current choice. A picked artist goes by
* id — names can collide once slugified — and a typed name creates (or * id — names can collide once slugified — and a typed name creates (or
* finds) that artist. null when there is nothing valid to send. * finds) that artist. The URL handle left untouched sends no name, so the
* server resolves the display name itself. null when there is nothing valid.
*/ */
function discordAddRequest(choice) { function addRequest(choice) {
const url = choice.scope === 'server' ? choice.serverUrl : choice.channelUrl; let url = choice.pageUrl;
if (choice.scope === 'server') url = choice.serverUrl;
else if (choice.scope === 'channel') url = choice.channelUrl;
if (!url) return null; if (!url) return null;
if (choice.artist && choice.artist.id != null && choice.artist.name === choice.artistName) { if (choice.artist && choice.artist.id != null && choice.artist.name === choice.artistName) {
return { url, artistId: choice.artist.id }; const req = { url, artistId: choice.artist.id };
if (choice.adoptPlatformName && renameOffer(choice)) req.usePlatformName = true;
return req;
} }
const name = (choice.artistName || '').trim(); const name = (choice.artistName || '').trim();
return name ? { url, artistName: name } : null; if (!name) return null;
if (choice.nameIsHandle && name === choice.handle) return { url };
return { url, artistName: name };
} }
/** /**
+8 -4
View File
@@ -76,7 +76,7 @@ function updateConnectionDot(connected) {
async function checkForUpdate() { async function checkForUpdate() {
try { try {
const r = await browser.runtime.sendMessage({ type: 'CHECK_UPDATE' }); const r = await browser.runtime.sendMessage({ type: 'CHECK_UPDATE' });
if (r && r.updateAvailable && r.xpiUrl) showUpdateBanner(r); if (r && r.updateAvailable && r.installPageUrl) showUpdateBanner(r);
} catch { /* non-fatal */ } } catch { /* non-fatal */ }
} }
@@ -86,10 +86,14 @@ function showUpdateBanner(r) {
// exactly as it did before the field existed. // exactly as it did before the field existed.
const channel = r.channel ? ` (${r.channel})` : ''; const channel = r.channel ? ` (${r.channel})` : '';
document.getElementById('update-text').textContent = document.getElementById('update-text').textContent =
`Update available${channel} — v${r.latestVersion} (installed v${r.currentVersion})`; `Update available${channel} — v${r.latestVersion} (installed v${r.currentVersion}). ` +
// Opening the signed XPI triggers Firefox's native install prompt. 'Opens FabledCurator — click “Install Firefox extension” there.';
// Opens FC's install card rather than the XPI: Firefox only installs an
// add-on from a user click on a web page, never from a tab an extension
// opened on the .xpi itself.
document.getElementById('update-btn').addEventListener('click', () => { document.getElementById('update-btn').addEventListener('click', () => {
browser.tabs.create({ url: r.xpiUrl }); browser.tabs.create({ url: r.installPageUrl });
window.close();
}); });
document.getElementById('update-banner').classList.remove('hidden'); document.getElementById('update-banner').classList.remove('hidden');
} }
+71 -14
View File
@@ -1,11 +1,12 @@
import { describe, it, expect } from 'vitest' import { describe, it, expect } from 'vitest'
import { loadLib } from './helpers/loadLib.js' import { loadLib } from './helpers/loadLib.js'
const { chipState, chipLabel, discordPanelDefaults, discordAddRequest } = loadLib('chip.js', [ const { chipState, chipLabel, panelDefaults, addRequest, renameOffer } = loadLib('chip.js', [
'chipState', 'chipState',
'chipLabel', 'chipLabel',
'discordPanelDefaults', 'panelDefaults',
'discordAddRequest' 'addRequest',
'renameOffer'
]) ])
const discord = (extra = {}) => ({ const discord = (extra = {}) => ({
@@ -63,47 +64,47 @@ describe('chip state and label', () => {
}) })
}) })
describe('Discord Add panel', () => { describe('Add panel on Discord', () => {
it('opens on the channel with the suggested artist preselected', () => { it('opens on the channel with the suggested artist preselected', () => {
const d = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } })) const d = panelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
expect(d.scope).toBe('channel') expect(d.scope).toBe('channel')
expect(d.artist).toEqual({ id: 7, name: 'Tamada' }) expect(d.artist).toEqual({ id: 7, name: 'Tamada' })
expect(d.artistName).toBe('Tamada') expect(d.artistName).toBe('Tamada')
}) })
it('proposes a new artist named after the server when nothing is suggested', () => { it('proposes a new artist named after the server when nothing is suggested', () => {
const d = discordPanelDefaults(discord({ state: 'new' })) const d = panelDefaults(discord({ state: 'new' }))
expect(d.artist).toBe(null) expect(d.artist).toBe(null)
expect(d.artistName).toBe('Studio') expect(d.artistName).toBe('Studio')
}) })
it('sends a picked artist by id', () => { it('sends a picked artist by id', () => {
const choice = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } })) const choice = panelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
expect(discordAddRequest(choice)).toEqual({ expect(addRequest(choice)).toEqual({
url: 'https://discord.com/channels/111/222', url: 'https://discord.com/channels/111/222',
artistId: 7 artistId: 7
}) })
}) })
it('sends a typed name once the picked artist has been edited away', () => { it('sends a typed name once the picked artist has been edited away', () => {
const choice = discordPanelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } })) const choice = panelDefaults(discord({ state: 'artist_match', artist: { id: 7, name: 'Tamada' } }))
choice.artistName = 'Tamada Alt' choice.artistName = 'Tamada Alt'
expect(discordAddRequest(choice)).toEqual({ expect(addRequest(choice)).toEqual({
url: 'https://discord.com/channels/111/222', url: 'https://discord.com/channels/111/222',
artistName: 'Tamada Alt' artistName: 'Tamada Alt'
}) })
}) })
it('adds the whole server when the operator picks it', () => { it('adds the whole server when the operator picks it', () => {
const choice = discordPanelDefaults(discord({ state: 'new' })) const choice = panelDefaults(discord({ state: 'new' }))
choice.scope = 'server' choice.scope = 'server'
expect(discordAddRequest(choice).url).toBe('https://discord.com/channels/111') expect(addRequest(choice).url).toBe('https://discord.com/channels/111')
}) })
it('has nothing to send without an artist', () => { it('has nothing to send without an artist', () => {
const choice = discordPanelDefaults(discord({ state: 'new' })) const choice = panelDefaults(discord({ state: 'new' }))
choice.artistName = ' ' choice.artistName = ' '
expect(discordAddRequest(choice)).toBe(null) expect(addRequest(choice)).toBe(null)
}) })
}) })
@@ -141,3 +142,59 @@ describe('artist matching for the Add panel', () => {
expect(inlineCompletion('', results)).toBe(null) expect(inlineCompletion('', results)).toBe(null)
}) })
}) })
describe('Add panel on Patreon and SubscribeStar', () => {
const PAGE = 'https://www.patreon.com/cw/tamadaheijun'
const patreon = (extra = {}) => ({ platform: 'patreon', slug: 'tamadaheijun', ...extra })
it('opens on the Patreon display name as a new artist', () => {
const c = panelDefaults(patreon({ state: 'new', display_name: 'Tamada Heijun' }), PAGE)
expect(c.scope).toBe('page')
expect(c.artistName).toBe('Tamada Heijun')
expect(addRequest(c)).toEqual({ url: PAGE, artistName: 'Tamada Heijun' })
})
it('leaves an untouched URL handle to the server to resolve', () => {
const c = panelDefaults(patreon({ state: 'new' }), PAGE)
expect(c.artistName).toBe('tamadaheijun')
expect(addRequest(c)).toEqual({ url: PAGE })
c.artistName = 'Someone Else'
expect(addRequest(c)).toEqual({ url: PAGE, artistName: 'Someone Else' })
})
it('offers the Patreon name when joining an artist known by another name', () => {
const c = panelDefaults(patreon({ state: 'new', display_name: 'Tamada Heijun' }), PAGE)
c.artist = { id: 4, name: 'tamada' }
c.artistName = 'tamada'
expect(renameOffer(c)).toEqual({ from: 'tamada', to: 'Tamada Heijun' })
expect(addRequest(c)).toEqual({ url: PAGE, artistId: 4, usePlatformName: true })
c.adoptPlatformName = false
expect(addRequest(c)).toEqual({ url: PAGE, artistId: 4 })
})
it('offers no rename when the names agree or the name was not read', () => {
const same = panelDefaults(patreon({ state: 'new', display_name: 'Tamada Heijun' }), PAGE)
same.artist = { id: 4, name: 'Tamada Heijun' }
expect(renameOffer(same)).toBe(null)
const unread = panelDefaults(patreon({ state: 'new' }), PAGE)
unread.artist = { id: 4, name: 'tamada' }
expect(renameOffer(unread)).toBe(null)
})
it('never renames from SubscribeStar or Discord — Patreon is the canon', () => {
const ss = panelDefaults(
{ platform: 'subscribestar', slug: 'tamada', state: 'new', display_name: 'SS Tamada' },
'https://subscribestar.adult/tamada'
)
ss.artist = { id: 4, name: 'Tamada Heijun' }
ss.artistName = 'Tamada Heijun'
expect(renameOffer(ss)).toBe(null)
expect(addRequest(ss)).toEqual({ url: 'https://subscribestar.adult/tamada', artistId: 4 })
})
it('preselects the artist the URL already names', () => {
const c = panelDefaults(patreon({ state: 'artist_match', artist: { id: 9, name: 'Tamada' } }), PAGE)
expect(c.artist).toEqual({ id: 9, name: 'Tamada' })
expect(c.artistName).toBe('Tamada')
})
})
@@ -2,14 +2,14 @@
<!-- System-tag auto-applies (chrome hides / process WIP tags) that ALSO looked <!-- System-tag auto-applies (chrome hides / process WIP tags) that ALSO looked
like real content — surfaced PROACTIVELY atop the gallery whenever there's like real content — surfaced PROACTIVELY atop the gallery whenever there's
something to review (NOT gated on the Show-hidden toggle, so misfires can't something to review (NOT gated on the Show-hidden toggle, so misfires can't
go unnoticed), most-concerning first, with keep / remove (#141, #1464). go unnoticed), most-concerning first (#141, #1464). Each card asks one
question — is this image a <tag>? — and its buttons answer it (#4424).
Renders nothing when there's nothing to review. --> Renders nothing when there's nothing to review. -->
<section v-if="items.length" class="fc-review" aria-label="Auto-tagged images to review"> <section v-if="items.length" class="fc-review" aria-label="Auto-tagged images to review">
<div class="fc-review__head"> <div class="fc-review__head">
<v-icon size="18" color="warning">mdi-alert-outline</v-icon> <v-icon size="18" color="warning">mdi-alert-outline</v-icon>
<span class="fc-review__title"> <span class="fc-review__title">
{{ items.length }} auto-tagged {{ items.length === 1 ? 'image' : 'images' }} {{ items.length }} {{ items.length === 1 ? 'auto-tag' : 'auto-tags' }} to check
may be real content — review
</span> </span>
</div> </div>
<div class="fc-review__cards"> <div class="fc-review__cards">
@@ -21,22 +21,23 @@
class="fc-review-card__thumb" loading="lazy" class="fc-review-card__thumb" loading="lazy"
> >
<div class="fc-review-card__body"> <div class="fc-review-card__body">
<div class="fc-review-card__question">{{ question(it) }}</div>
<div <div
class="fc-review-card__conflict" class="fc-review-card__reason"
:title="`Scored ${Math.round(it.conflict_score * 100)}% on “${it.conflict_name || 'a content tag'}”`" :title="reasonTitle(it)"
> >
also looks like <strong>{{ it.conflict_name || 'content' }}</strong> also {{ Math.round(it.conflict_score * 100) }}%
<strong>{{ it.conflict_name || 'content' }}</strong>
</div> </div>
<div class="fc-review-card__tag">{{ tagLine(it) }}</div>
<div class="fc-review-card__acts"> <div class="fc-review-card__acts">
<button <button
type="button" class="fc-review-btn fc-review-btn--keep" type="button" class="fc-review-btn fc-review-btn--keep"
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')" :disabled="busy.includes(keyOf(it))" @click="resolve(it, 'keep')"
>{{ keepLabel(it) }}</button> >Is {{ withArticle(it) }}</button>
<button <button
type="button" class="fc-review-btn fc-review-btn--unhide" type="button" class="fc-review-btn fc-review-btn--unhide"
:disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')" :disabled="busy.includes(keyOf(it))" @click="resolve(it, 'unhide')"
>{{ removeLabel(it) }}</button> >Is not {{ withArticle(it) }}</button>
</div> </div>
</div> </div>
</div> </div>
@@ -55,11 +56,18 @@ const items = ref([])
const busy = ref([]) const busy = ref([])
function keyOf(it) { return `${it.image_id}:${it.tag_id}` } function keyOf(it) { return `${it.image_id}:${it.tag_id}` }
// Chrome flags hide the image (keep-hidden / un-hide); process flags leave it // The card asks whether the image IS the auto-applied system tag, and the buttons
// visible and just tagged (keep-tag / remove-tag). Same endpoints, different words. // answer that (operator, #4424: "is a <tag>" / "is not a <tag>"). "Is" keeps the
function tagLine(it) { return (it.mode === 'process' ? 'auto-tagged ' : 'hidden as ') + it.tag_name } // tag ('keep'); "Is not" removes it, un-hiding a chrome image ('unhide'). The
function keepLabel(it) { return it.mode === 'process' ? 'Keep tag' : 'Keep hidden' } // content tag it also scored on is the reason it was flagged, not the question.
function removeLabel(it) { return it.mode === 'process' ? 'Remove tag' : 'Un-hide' } function noun(it) { return it.tag_name === 'wip' ? 'WIP' : it.tag_name }
function withArticle(it) { return (/^[aeiou]/i.test(noun(it)) ? 'an ' : 'a ') + noun(it) }
function question(it) { return `Is this ${withArticle(it)}?` }
function reasonTitle(it) {
const hidden = it.mode === 'process' ? '' : ' It is hidden from the gallery until you answer.'
return `Auto-tagged “${it.tag_name}”, but it also scored ${Math.round(it.conflict_score * 100)}% `
+ `on “${it.conflict_name || 'a content tag'}”, so it may be finished art.${hidden}`
}
async function load() { async function load() {
// Fetched unconditionally on mount — the strip prompts for pending misfires // Fetched unconditionally on mount — the strip prompts for pending misfires
@@ -77,14 +85,11 @@ async function resolve(it, action) {
await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`) await api.post(`/api/gallery/hidden-review/${it.image_id}/${it.tag_id}/${action}`)
items.value = items.value.filter((x) => keyOf(x) !== k) items.value = items.value.filter((x) => keyOf(x) !== k)
if (action === 'unhide') { if (action === 'unhide') {
const verb = it.mode === 'process' ? 'Removed' : 'Un-hidden' const shown = it.mode === 'process' ? '' : ', back in the gallery'
toast({ text: `${verb} — “${it.tag_name}” removed; it'll train the head`, type: 'success' }) toast({ text: `Not ${withArticle(it)} — “${it.tag_name}” removed${shown}; the tagger learns from it`, type: 'success' })
} }
} catch (e) { } catch (e) {
toast({ toast({ text: `Could not save your answer: ${e.message}`, type: 'error' })
text: `Could not ${action === 'keep' ? 'keep hidden' : 'un-hide'}: ${e.message}`,
type: 'error',
})
} finally { } finally {
busy.value = busy.value.filter((x) => x !== k) busy.value = busy.value.filter((x) => x !== k)
} }
@@ -112,7 +117,7 @@ onMounted(load)
display: flex; gap: 10px; overflow-x: auto; padding-bottom: 4px; display: flex; gap: 10px; overflow-x: auto; padding-bottom: 4px;
} }
.fc-review-card { .fc-review-card {
flex: 0 0 auto; width: 150px; flex: 0 0 auto; width: 170px;
display: flex; flex-direction: column; display: flex; flex-direction: column;
border: 1px solid rgb(var(--v-theme-surface-light)); border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 6px; overflow: hidden; border-radius: 6px; overflow: hidden;
@@ -123,16 +128,16 @@ onMounted(load)
background: rgb(var(--v-theme-surface-light)); background: rgb(var(--v-theme-surface-light));
} }
.fc-review-card__body { padding: 6px 8px; } .fc-review-card__body { padding: 6px 8px; }
.fc-review-card__conflict { .fc-review-card__question {
font-size: 11px; color: rgb(var(--v-theme-on-surface)); font-size: 12px; font-weight: 600; color: rgb(var(--v-theme-on-surface));
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
} }
.fc-review-card__conflict strong { color: rgb(var(--v-theme-warning)); } .fc-review-card__reason {
.fc-review-card__tag {
font-size: 10px; color: rgb(var(--v-theme-on-surface-variant)); font-size: 10px; color: rgb(var(--v-theme-on-surface-variant));
margin: 1px 0 6px; margin: 1px 0 6px;
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
} }
.fc-review-card__reason strong { color: rgb(var(--v-theme-warning)); font-weight: 600; }
.fc-review-card__acts { display: flex; gap: 4px; } .fc-review-card__acts { display: flex; gap: 4px; }
.fc-review-btn { .fc-review-btn {
flex: 1; font-size: 11px; padding: 3px 4px; border-radius: 4px; flex: 1; font-size: 11px; padding: 3px 4px; border-radius: 4px;
@@ -17,7 +17,7 @@
<v-icon start>mdi-folder-zip-outline</v-icon> Re-extract archives now <v-icon start>mdi-folder-zip-outline</v-icon> Re-extract archives now
</v-btn> </v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span> <span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" /> <QueueStatusBar queue="maintenance_long" queue-label="Long maintenance" />
</MaintenanceTile> </MaintenanceTile>
</template> </template>
@@ -49,16 +49,20 @@
sometimes triggered nothing instead of the install dialog sometimes triggered nothing instead of the install dialog
(operator-flagged 2026-05-26). No `download` attribute — (operator-flagged 2026-05-26). No `download` attribute —
that would force a save dialog instead of install. --> that would force a save dialog instead of install. -->
<!-- The VERSIONED file, not the `latest` alias: a versioned URL can
only ever be this build's bytes, so a browser cache can't hand
back the previous build (operator-flagged 2026-09-25: a cached
alias reinstalled the old version and the update never took). -->
<v-btn <v-btn
v-if="isFirefox" v-if="isFirefox"
color="accent" variant="flat" rounded="pill" color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-firefox" prepend-icon="mdi-firefox"
:href="manifest.latest_url" :href="manifest.xpi_url"
>Install Firefox extension</v-btn> >Install Firefox extension</v-btn>
<v-btn <v-btn
variant="outlined" rounded="pill" variant="outlined" rounded="pill"
:href="manifest.latest_url" download :href="manifest.xpi_url" download
prepend-icon="mdi-download" prepend-icon="mdi-download"
>Download XPI</v-btn> >Download XPI</v-btn>
@@ -103,17 +103,6 @@
the Explore browse. Applies to new imports; run the scan below to catch the Explore browse. Applies to new imports; run the scan below to catch
posts already in your library. posts already in your library.
</div> </div>
<v-switch
v-model="local.wip_soft_title_tagging_enabled"
label="Also tag “sketch” / “doodle” titles (lower precision)"
density="compact" hide-details color="primary" @change="save"
/>
<div class="fc-help mb-3">
Extends the above to softer cues (<code>sketch</code>, <code>doodle</code>,
<code>scribble</code>). These stay <strong>visible</strong> and never train
the tagging model — a daily audit flags any that actually look like finished
art for review. Off by default.
</div>
<v-btn <v-btn
variant="tonal" color="primary" size="small" variant="tonal" color="primary" size="small"
:loading="store.wipScanBusy" prepend-icon="mdi-magnify" :loading="store.wipScanBusy" prepend-icon="mdi-magnify"
@@ -156,7 +145,6 @@ const local = reactive({
skip_single_color: false, single_color_threshold: 0.95, skip_single_color: false, single_color_threshold: 0.95,
phash_threshold: 24, phash_threshold: 24,
wip_title_tagging_enabled: true, wip_title_tagging_enabled: true,
wip_soft_title_tagging_enabled: false,
}) })
watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true }) watch(() => store.settings, (s) => { if (s) Object.assign(local, s) }, { immediate: true })
@@ -18,7 +18,7 @@
<v-icon start>mdi-file-remove-outline</v-icon> Repair missing-file records <v-icon start>mdi-file-remove-outline</v-icon> Repair missing-file records
</v-btn> </v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span> <span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" /> <QueueStatusBar queue="maintenance_long" queue-label="Long maintenance" />
</MaintenanceTile> </MaintenanceTile>
</template> </template>
@@ -41,7 +41,7 @@ const props = defineProps({
const QUEUE_NAMES = [ const QUEUE_NAMES = [
'default', 'import', 'thumbnail', 'ml', 'default', 'import', 'thumbnail', 'ml',
'download', 'scan', 'maintenance', 'download', 'scan', 'maintenance', 'maintenance_long',
] ]
function formatDepth(name) { function formatDepth(name) {
+4 -2
View File
@@ -119,7 +119,7 @@ export const useAdminStore = defineStore('admin', () => {
// --- Task progress polling (taps FC-3i activity dashboard) -------- // --- Task progress polling (taps FC-3i activity dashboard) --------
/** /**
* Polls /api/system/activity/runs?queue=maintenance every 3s, * Polls /api/system/activity/runs?celery_task_id=<id> every 3s,
* resolves when a task_run row with the given celery task_id * resolves when a task_run row with the given celery task_id
* reaches a terminal status (ok / error / timeout). Returns the * reaches a terminal status (ok / error / timeout). Returns the
* row. Times out after 30 min by default. * row. Times out after 30 min by default.
@@ -129,7 +129,9 @@ export const useAdminStore = defineStore('admin', () => {
while (Date.now() < deadline) { while (Date.now() < deadline) {
const body = await api.get( const body = await api.get(
'/api/system/activity/runs', '/api/system/activity/runs',
{ params: { queue: 'maintenance', limit: 20 } }, // By id, not by lane: these jobs run on `maintenance_long`, and a
// lane filter here is one more copy of the routing table (#4432).
{ params: { celery_task_id: taskId, limit: 1 } },
) )
const row = (body.runs || []).find(r => r.celery_task_id === taskId) const row = (body.runs || []).find(r => r.celery_task_id === taskId)
if (row && ['ok', 'error', 'timeout'].includes(row.status)) { if (row && ['ok', 'error', 'timeout'].includes(row.status)) {
+8 -8
View File
@@ -210,8 +210,8 @@ def render(
) )
parts.append( parts.append(
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag " f"Built from `{short}`. To roll back to this release, pull these "
f"(rule 145) — these three move together:\n\n```\n" f"immutable `:c-` tags — the images move together:\n\n```\n"
+ "\n".join(f"{image}:c-{short}" for image in IMAGES) + "\n".join(f"{image}:c-{short}" for image in IMAGES)
+ "\n```" + "\n```"
) )
@@ -221,10 +221,10 @@ def render(
# truncated to MAX_COMMITS, which is 200 lines of internal build-out # truncated to MAX_COMMITS, which is 200 lines of internal build-out
# presented to someone who has never seen this project. # presented to someone who has never seen this project.
parts.append( parts.append(
"---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so " "---\n\n_The first release, so there is no earlier one to diff "
"there is no predecessor to diff against and no changelog to derive. " "against and no changelog to derive. The description above is "
"The description above is README.md's, quoted at publish time. Later " "README.md's, quoted at publish time. Later releases carry the "
"releases carry the commits since the previous one._" "commits since the previous one._"
) )
return "\n\n".join(parts) return "\n\n".join(parts)
@@ -257,8 +257,8 @@ def cross_checks(tag: str, sha: str) -> list[str]:
if not RULE_148.match(tag): if not RULE_148.match(tag):
notes.append( notes.append(
f"`{tag}` is not rule 148's `vYYYY.MM.DD.HHMM` shape. Published " f"`{tag}` is not the `vYYYY.MM.DD.HHMM` release-tag shape. "
f"anyway — the old `v26.*` tags predate the rule." f"Published anyway — the old `v26.*` tags predate it."
) )
else: else:
derived = artifact_version("web") derived = artifact_version("web")
+137
View File
@@ -526,6 +526,121 @@ async def test_probe_a_discord_dm_is_not_a_source(client, ext_key):
assert body["state"] == "unknown_platform" assert body["state"] == "unknown_platform"
# --- Patreon is canon: the Add panel's names and the rename (milestone 429) ---
@pytest.fixture
def platform_names(monkeypatch):
"""Stub both platforms' display-name lookups (no network in tests)."""
from backend.app.services import patreon_resolver
from backend.app.services.credential_service import CredentialService
from backend.app.services.subscribestar_client import SubscribeStarClient
async def _cookies(self, platform):
return "/tmp/cookies.txt"
names = {"patreon": "Tamada Heijun", "subscribestar": "SS Tamada"}
monkeypatch.setattr(CredentialService, "get_cookies_path", _cookies)
monkeypatch.setattr(
patreon_resolver, "resolve_display_name", lambda v, c: names["patreon"],
)
monkeypatch.setattr(
SubscribeStarClient, "resolve_display_name", lambda self, u: names["subscribestar"],
)
return names
@pytest.mark.asyncio
async def test_probe_with_names_reads_the_patreon_display_name(client, ext_key, platform_names):
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/tamadaheijun", "names": "1"},
headers={"X-Extension-Key": ext_key},
)
body = await resp.get_json()
assert body["state"] == "new"
assert body["display_name"] == "Tamada Heijun"
@pytest.mark.asyncio
async def test_a_plain_probe_does_not_look_the_name_up(client, ext_key, platform_names):
resp = await client.get(
"/api/extension/probe",
query_string={"url": "https://www.patreon.com/tamadaheijun"},
headers={"X-Extension-Key": ext_key},
)
assert "display_name" not in await resp.get_json()
async def _artist(db, name, slug):
artist = Artist(name=name, slug=slug, is_subscription=True)
db.add(artist)
await db.commit()
return artist
@pytest.mark.asyncio
async def test_a_patreon_source_renames_the_artist_it_joins_to_the_patreon_name(
client, ext_key, db, db_sync, platform_names,
):
artist = await _artist(db, "tamada", "tamada")
resp = await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/tamadaheijun",
"artist_id": artist.id, "use_platform_name": True},
headers={"X-Extension-Key": ext_key},
)
assert resp.status_code == 201
body = await resp.get_json()
assert body["artist"]["name"] == "Tamada Heijun"
assert body["renamed_from"] == "tamada"
# Name only: the slug, and every path keyed off it, stays.
row = db_sync.execute(select(Artist.name, Artist.slug).where(Artist.id == artist.id)).one()
assert tuple(row) == ("Tamada Heijun", "tamada")
@pytest.mark.asyncio
async def test_no_rename_without_the_flag(client, ext_key, db, platform_names):
artist = await _artist(db, "tamada", "tamada")
body = await (await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/tamadaheijun", "artist_id": artist.id},
headers={"X-Extension-Key": ext_key},
)).get_json()
assert body["artist"]["name"] == "tamada"
assert "renamed_from" not in body
@pytest.mark.asyncio
async def test_an_unreadable_patreon_name_never_renames_to_the_handle(
client, ext_key, db, platform_names,
):
platform_names["patreon"] = None
artist = await _artist(db, "Tamada", "tamada")
body = await (await client.post(
"/api/extension/quick-add-source",
json={"url": "https://www.patreon.com/tamadaheijun",
"artist_id": artist.id, "use_platform_name": True},
headers={"X-Extension-Key": ext_key},
)).get_json()
assert body["artist"]["name"] == "Tamada"
assert "renamed_from" not in body
@pytest.mark.asyncio
async def test_only_patreon_names_are_canon(client, ext_key, db, platform_names):
"""A SubscribeStar source joins the picked artist under the name it has."""
artist = await _artist(db, "Tamada Heijun", "tamada-heijun")
body = await (await client.post(
"/api/extension/quick-add-source",
json={"url": "https://subscribestar.adult/tamada",
"artist_id": artist.id, "use_platform_name": True},
headers={"X-Extension-Key": ext_key},
)).get_json()
assert body["artist"]["name"] == "Tamada Heijun"
assert "renamed_from" not in body
# --- /api/extension/manifest --------------------------------------- # --- /api/extension/manifest ---------------------------------------
@@ -664,6 +779,28 @@ async def test_serve_extension_latest_returns_most_recent_xpi(
assert data == b"new" assert data == b"new"
@pytest.mark.asyncio
async def test_the_latest_alias_is_never_served_from_a_stale_cache(
client, monkeypatch, tmp_path,
):
"""One URL whose bytes change every release: a cached copy reinstalls the
previous build (operator-flagged 2026-09-25, when it was max-age=43200)."""
(tmp_path / "fabledcurator-1.0.1.xpi").write_bytes(b"new")
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-latest.xpi")
assert resp.headers["Cache-Control"] == "no-cache"
assert "Expires" not in resp.headers
@pytest.mark.asyncio
async def test_a_versioned_xpi_is_cached_for_good(client, monkeypatch, tmp_path):
"""A versioned name is one build's bytes forever."""
(tmp_path / "fabledcurator-1.0.1.xpi").write_bytes(b"new")
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
resp = await client.get("/extension/fabledcurator-1.0.1.xpi")
assert "immutable" in resp.headers["Cache-Control"]
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_serve_extension_latest_404_when_dir_empty(client, monkeypatch, tmp_path): async def test_serve_extension_latest_404_when_dir_empty(client, monkeypatch, tmp_path):
monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path) monkeypatch.setattr(frontend_module, "XPI_DIR", tmp_path)
+8
View File
@@ -295,3 +295,11 @@ async def test_failures_only_within_24h_window(client, _seed_failures):
body = await resp.get_json() body = await resp.get_json()
ids = {r["error_type"] for r in body["recent"]} ids = {r["error_type"] for r in body["recent"]}
assert "OldError" not in ids assert "OldError" not in ids
@pytest.mark.asyncio
async def test_runs_filter_by_celery_task_id(client, _seed_runs):
# How a page follows a job it started, without knowing its lane (#4432).
resp = await client.get("/api/system/activity/runs?celery_task_id=tid-3")
body = await resp.get_json()
assert [r["celery_task_id"] for r in body["runs"]] == ["tid-3"]
+55 -11
View File
@@ -2,6 +2,8 @@
`maintenance_long` lane so a 30-min backup or a multi-chunk audit can't starve `maintenance_long` lane so a 30-min backup or a multi-chunk audit can't starve
the quick recovery sweeps / vacuum on the concurrency-1 `maintenance` lane the quick recovery sweeps / vacuum on the concurrency-1 `maintenance` lane
(operator-flagged 2026-06-07).""" (operator-flagged 2026-06-07)."""
import pytest
from backend.app.celery_app import celery from backend.app.celery_app import celery
@@ -20,23 +22,65 @@ def test_quick_maintenance_stays_on_maintenance():
assert routes["backend.app.tasks.maintenance.*"]["queue"] == "maintenance" assert routes["backend.app.tasks.maintenance.*"]["queue"] == "maintenance"
def test_queue_for_mirrors_external_to_download(): @pytest.mark.parametrize(("name", "queue"), [
"""celery_signals._queue_for is a hand-maintained mirror of task_routes ("backend.app.tasks.external.fetch_external_link", "download"),
that stamps TaskRun.queue. external.* routes to the download lane, so the # The rows #4432 found recorded on the wrong lane:
mirror must agree — else TaskRun.queue lies 'default' for external fetches ("backend.app.tasks.translation.translate_posts", "maintenance_long"),
and per-queue dashboard filters / threshold overrides miss them ("backend.app.tasks.gpu_queue.enqueue_gpu_backfill", "maintenance"),
(operator-flagged 2026-06-17).""" ("backend.app.tasks.maintenance.backfill_phash", "maintenance_long"),
("backend.app.tasks.maintenance.date_posts_from_records", "maintenance_long"),
("backend.app.tasks.admin.normalize_tags_task", "maintenance_long"),
("backend.app.tasks.backup.backup_db_task", "maintenance_long"),
("backend.app.tasks.maintenance.vacuum_analyze", "maintenance"),
("backend.app.tasks.not_routed.anything", "default"),
])
def test_task_run_records_the_queue_the_router_sends_to(name, queue):
"""TaskRun.queue comes from the router, not a copy of task_routes (#4432):
the stall sweep's per-queue thresholds and the System activity filters both
key off it."""
from backend.app.celery_signals import _queue_for from backend.app.celery_signals import _queue_for
class _T: class _T:
name = "backend.app.tasks.external.fetch_external_link" pass
assert _queue_for(_T()) == "download" t = _T()
assert ( t.name = name
celery.conf.task_routes["backend.app.tasks.external.*"]["queue"] t.app = celery
== "download" assert _queue_for(t) == queue
def test_no_task_outlives_its_stall_threshold():
"""A task whose hard time limit is longer than the stall sweep's threshold
for it gets failed 'RecoverySweep' while it is still healthy — the class
#4432 found on the ml, import and long-maintenance lanes. The threshold is
resolved the way recover_stalled_task_runs resolves it: task-name override,
then queue, then the default."""
from backend.app.celery_signals import _queue_for
from backend.app.tasks.maintenance import (
QUEUE_STUCK_THRESHOLD_MINUTES,
STUCK_THRESHOLD_MINUTES,
TASK_STUCK_THRESHOLD_MINUTES,
) )
celery.loader.import_default_modules()
checked = 0
too_short = []
for name, task in sorted(celery.tasks.items()):
if name.startswith("celery."):
continue
limit = getattr(task, "time_limit", None)
if not limit:
continue
threshold = TASK_STUCK_THRESHOLD_MINUTES.get(
name,
QUEUE_STUCK_THRESHOLD_MINUTES.get(_queue_for(task), STUCK_THRESHOLD_MINUTES),
)
checked += 1
if limit / 60 > threshold:
too_short.append(f"{name}: limit {limit / 60:.0f} min > sweep {threshold} min")
assert checked > 20, "the task registry did not load — this guard checked nothing"
assert not too_short, "\n".join(too_short)
def test_backfill_phash_runs_on_the_long_lane(): def test_backfill_phash_runs_on_the_long_lane():
"""It lives in maintenance.py, so the quick-lane glob matches it too — """It lives in maintenance.py, so the quick-lane glob matches it too —
+71
View File
@@ -0,0 +1,71 @@
"""#4433: only the process consuming `download` clears a restart's leftovers.
The consolidated container boots four lanes side by side. If the scheduler or
ml lane ran this too, a lane restarting on its own (supervisord restarts a
crashed program) would close the events of walks that are still running.
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from backend.app import celery_signals
def _consumer(*queues: str):
return SimpleNamespace(
task_consumer=SimpleNamespace(queues=[SimpleNamespace(name=q) for q in queues])
)
@pytest.fixture
def calls(monkeypatch):
seen: list[str] = []
class _Session:
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def commit(self):
seen.append("commit")
monkeypatch.setattr(celery_signals, "sync_session_factory", lambda: _Session)
monkeypatch.setattr(
"backend.app.services.download_recovery.interrupt_orphaned_download_events",
lambda session, *, booted_at: seen.append("interrupt") or 0,
)
monkeypatch.setattr(
"backend.app.services.platform_lock.release_all_platform_locks",
lambda: seen.append("release") or 0,
)
return seen
def test_the_download_lane_clears_orphans_and_locks(calls):
celery_signals._on_worker_ready(sender=_consumer("default", "download", "import"))
assert calls == ["interrupt", "commit", "release"]
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",), ("maintenance_long",)])
def test_other_lanes_leave_downloads_alone(calls, queues):
celery_signals._on_worker_ready(sender=_consumer(*queues))
assert calls == []
def test_unreadable_queues_do_nothing_rather_than_guess(calls):
celery_signals._on_worker_ready(sender=SimpleNamespace())
assert calls == []
def test_a_failure_does_not_stop_the_worker_starting(monkeypatch, calls):
def boom(session, *, booted_at):
raise RuntimeError("db down")
monkeypatch.setattr(
"backend.app.services.download_recovery.interrupt_orphaned_download_events", boom
)
celery_signals._on_worker_ready(sender=_consumer("download")) # must not raise
@@ -0,0 +1,106 @@
"""Migration 0114 (#4435): the undated shell post a misfiled attachment made on
the artist's first Discord source is folded into the real, dated post."""
import importlib.util
from datetime import UTC, datetime
from pathlib import Path
import pytest
from sqlalchemy import select
from backend.app.models import (
Artist,
ImageProvenance,
Post,
PostAttachment,
Source,
)
from tests.factories import make_image as _img
pytestmark = pytest.mark.integration
_MIGRATION = (
Path(__file__).resolve().parents[1]
/ "alembic" / "versions" / "0114_fold_misfiled_attachment_posts.py"
)
def _fold():
spec = importlib.util.spec_from_file_location("m0114", _MIGRATION)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod.fold_misfiled_attachment_posts
def _source(db, artist, channel):
s = Source(
artist_id=artist.id, platform="discord",
url=f"https://discord.com/channels/1/{channel}",
)
db.add(s)
db.flush()
return s
def _post(db, artist, source, epid, when=None):
p = Post(
artist_id=artist.id, source_id=source.id, external_post_id=epid,
post_date=when,
post_url=f"https://discord.com/channels/1/x/{epid}" if when else None,
)
db.add(p)
db.flush()
return p
def _attach(db, post, sha, name):
db.add(PostAttachment(
post_id=post.id, artist_id=post.artist_id, sha256=sha,
path=f"/att/{sha}", original_filename=name, ext=".rar", size_bytes=1,
))
db.flush()
def test_shells_fold_into_their_dated_twin_and_nothing_else_moves(db_sync):
sent = datetime(2024, 12, 26, 1, 42, tzinfo=UTC)
artist = Artist(name="Yellow", slug="yellow")
db_sync.add(artist)
db_sync.flush()
first = _source(db_sync, artist, 100)
second = _source(db_sync, artist, 200)
# The #4435 shape: shell on the first source, real post on the second.
shell = _post(db_sync, artist, first, "555")
real = _post(db_sync, artist, second, "555", sent)
_attach(db_sync, shell, "a" * 64, "pack.rar")
# A shell whose attachment the real post already has: dropped, not doubled.
shell2 = _post(db_sync, artist, first, "556")
real2 = _post(db_sync, artist, second, "556", sent)
_attach(db_sync, shell2, "b" * 64, "same.rar")
_attach(db_sync, real2, "b" * 64, "same.rar")
# Left alone: no dated twin, and an undated post that holds an image.
lonely = _post(db_sync, artist, first, "557")
_attach(db_sync, lonely, "c" * 64, "lonely.rar")
with_image = _post(db_sync, artist, first, "558")
_post(db_sync, artist, second, "558", sent)
img = _img(db_sync, "d" * 64)
db_sync.add(ImageProvenance(image_record_id=img.id, post_id=with_image.id))
db_sync.flush()
shell_id, shell2_id = shell.id, shell2.id
folded = _fold()(db_sync.connection())
db_sync.expire_all()
assert folded == 2
assert db_sync.get(Post, shell_id) is None
assert db_sync.get(Post, shell2_id) is None
owners = dict(db_sync.execute(
select(PostAttachment.original_filename, PostAttachment.post_id)
).all())
assert owners["pack.rar"] == real.id
assert owners["lonely.rar"] == lonely.id
kept = db_sync.execute(
select(PostAttachment.id).where(PostAttachment.post_id == real2.id)
).scalars().all()
assert len(kept) == 1
assert db_sync.get(Post, lonely.id) is not None
assert db_sync.get(Post, with_image.id) is not None
+45
View File
@@ -435,3 +435,48 @@ def test_attach_in_place_non_media_routes_to_attachment(importer, db_sync):
).scalar_one() ).scalar_one()
assert row.ext == ".txt" assert row.ext == ".txt"
assert row.artist_id == artist.id assert row.artist_id == artist.id
def test_a_non_media_file_lands_on_the_source_being_downloaded(importer, db_sync):
"""#4435: a Discord artist has one source per channel. A non-media file
downloaded for the SECOND channel was filed under the artist's first
Discord source (the (artist, platform) lookup takes the lowest id), as an
undated shell post; the real post record then made a second post under
the right source. The attachment must follow the source it was fetched for."""
from backend.app.models import Post, PostAttachment, Source
images_root = importer.images_root
artist = Artist(name="Yara", slug="yara")
db_sync.add(artist)
db_sync.flush()
first = Source(
artist_id=artist.id, platform="discord",
url="https://discord.com/channels/1/100",
)
second = Source(
artist_id=artist.id, platform="discord",
url="https://discord.com/channels/1/200",
)
db_sync.add_all([first, second])
db_sync.flush()
rar = images_root / "yara" / "discord" / "rewards" / "20241226_555_01_pack.rar"
rar.parent.mkdir(parents=True, exist_ok=True)
rar.write_bytes(b"not a real archive, so it is kept as an attachment")
rar.with_suffix(rar.suffix + ".json").write_text(
'{"category": "discord", "id": "555", "message_id": "555"}'
)
result = importer.attach_in_place(rar, artist=artist, source=second)
assert result.status == "attached"
owner = db_sync.execute(
select(Post.source_id, Post.external_post_id)
.join(PostAttachment, PostAttachment.post_id == Post.id)
.where(PostAttachment.original_filename == rar.name)
).one()
assert owner.source_id == second.id
assert owner.external_post_id == "555"
assert db_sync.execute(
select(func.count()).select_from(Post).where(Post.source_id == first.id)
).scalar_one() == 0
+70 -14
View File
@@ -335,26 +335,30 @@ def test_recover_stalled_task_runs_ml_queue_uses_longer_threshold(db_sync):
"""ml-queue tasks (embed_image video branch) legitimately run """ml-queue tasks (embed_image video branch) legitimately run
past the default 5-min threshold. The sweep must NOT flag an past the default 5-min threshold. The sweep must NOT flag an
ml-queue task that's only been running 10 min — the override ml-queue task that's only been running 10 min — the override
threshold (25 min via QUEUE_STUCK_THRESHOLD_MINUTES) protects threshold (QUEUE_STUCK_THRESHOLD_MINUTES["ml"]) protects in-flight
in-flight video tagging. Operator-flagged 2026-05-28 after video tagging. Operator-flagged 2026-05-28 after image 6288 (mp4)
image 6288 (mp4) was marked failed at the 5-min tick mid-run.""" was marked failed at the 5-min tick mid-run. Read from the table
rather than restated: the value moved 25 -> 40 in #4432."""
from sqlalchemy import select from sqlalchemy import select
from backend.app.models import TaskRun from backend.app.models import TaskRun
from backend.app.tasks.maintenance import recover_stalled_task_runs from backend.app.tasks.maintenance import (
QUEUE_STUCK_THRESHOLD_MINUTES,
recover_stalled_task_runs,
)
ml_threshold = QUEUE_STUCK_THRESHOLD_MINUTES["ml"]
now = datetime.now(UTC) now = datetime.now(UTC)
# 10-min-old ml-queue row: stale by the default 5-min rule but # 10-min-old ml-queue row: stale by the default 5-min rule but
# fresh by the 25-min ml override. Must survive the sweep. # fresh by the ml override. Must survive the sweep.
ml_fresh_id = _make_task_run( ml_fresh_id = _make_task_run(
db_sync, status="running", queue="ml", db_sync, status="running", queue="ml",
started_at=now - timedelta(minutes=10), started_at=now - timedelta(minutes=10),
) )
# 30-min-old ml-queue row: past even the ml override. Must be # Past even the ml override. Must be flagged.
# flagged.
ml_stale_id = _make_task_run( ml_stale_id = _make_task_run(
db_sync, status="running", queue="ml", db_sync, status="running", queue="ml",
started_at=now - timedelta(minutes=30), started_at=now - timedelta(minutes=ml_threshold + 5),
) )
db_sync.commit() db_sync.commit()
@@ -431,9 +435,10 @@ def test_download_stuck_threshold_exceeds_hard_time_limit():
def test_recover_stalled_task_runs_archive_task_uses_longer_threshold(db_sync): def test_recover_stalled_task_runs_archive_task_uses_longer_threshold(db_sync):
"""import_archive_file shares the 'import' queue with fast """import_archive_file shares the 'import' queue with fast
single-file import_media_file, so it gets a per-task-name override single-file import_media_file, so it gets a per-task-name override
(40 min) while the import queue stays at the 5-min default. A (40 min) while the import queue keeps its short threshold (10 min
10-min-old archive task-run must survive; a 50-min-old one is since #4432; import_media_file's hard limit is 6). A 10-min-old
flagged. Operator-flagged 2026-05-28.""" archive task-run must survive; a 50-min-old one is flagged.
Operator-flagged 2026-05-28."""
from sqlalchemy import select from sqlalchemy import select
from backend.app.models import TaskRun from backend.app.models import TaskRun
@@ -441,12 +446,12 @@ def test_recover_stalled_task_runs_archive_task_uses_longer_threshold(db_sync):
archive_name = "backend.app.tasks.import_file.import_archive_file" archive_name = "backend.app.tasks.import_file.import_archive_file"
now = datetime.now(UTC) now = datetime.now(UTC)
# Fast single-file import on the same queue, 10 min old → flagged # Fast single-file import on the same queue, 15 min old → flagged
# by the default 5-min rule. # by the import queue's 10-min threshold.
media_id = _make_task_run( media_id = _make_task_run(
db_sync, status="running", queue="import", db_sync, status="running", queue="import",
task_name="backend.app.tasks.import_file.import_media_file", task_name="backend.app.tasks.import_file.import_media_file",
started_at=now - timedelta(minutes=10), started_at=now - timedelta(minutes=15),
) )
# Archive on the same queue, 10 min old → survives (40-min override). # Archive on the same queue, 10 min old → survives (40-min override).
archive_fresh_id = _make_task_run( archive_fresh_id = _make_task_run(
@@ -713,6 +718,57 @@ def test_recover_stalled_download_skips_fresh(db_sync):
assert failures == 0 assert failures == 0
def test_download_lane_boot_closes_pre_boot_events_without_blaming_the_source(db_sync):
"""#4433: a restart SIGKILLs a walk past its stop grace and strands queued
ones. At the download lane's boot, everything pending/running from before
it ends as `skipped`/interrupted — and the source is left as it was, so it
is due on the next tick instead of backed off as a failure."""
from sqlalchemy import select
from backend.app.models import DownloadEvent, Source
from backend.app.services.download_recovery import (
DOWNLOAD_INTERRUPTED_MESSAGE,
interrupt_orphaned_download_events,
)
sid = _make_source(db_sync, slug="rebooted")
booted_at = datetime.now(UTC)
before = booted_at - timedelta(minutes=5)
walking = DownloadEvent(
source_id=sid, status="running", started_at=before,
metadata_={"live": {"downloaded": 3}},
)
queued = DownloadEvent(source_id=sid, status="pending", started_at=before)
finished = DownloadEvent(source_id=sid, status="ok", started_at=before)
# Promoted to running after the boot: download_service resets started_at.
fresh = DownloadEvent(
source_id=sid, status="running", started_at=booted_at + timedelta(seconds=5),
)
db_sync.add_all([walking, queued, finished, fresh])
db_sync.commit()
closed = interrupt_orphaned_download_events(db_sync, booted_at=booted_at)
db_sync.commit()
assert closed == 2
db_sync.expire_all()
for ev in (walking, queued):
assert ev.status == "skipped"
assert ev.error == DOWNLOAD_INTERRUPTED_MESSAGE
assert ev.finished_at is not None
assert ev.metadata_["error_type"] == "interrupted"
assert walking.metadata_["live"] == {"downloaded": 3}
assert finished.status == "ok"
assert fresh.status == "running"
src = db_sync.execute(
select(Source.consecutive_failures, Source.last_error, Source.last_checked_at)
.where(Source.id == sid)
).one()
assert src.consecutive_failures == 0
assert src.last_error is None
assert src.last_checked_at is None
def test_recover_stalled_download_flips_stale_pending(db_sync): def test_recover_stalled_download_flips_stale_pending(db_sync):
"""A 2-hour-old pending event flips to error AND the source is bumped """A 2-hour-old pending event flips to error AND the source is bumped
(consecutive_failures, last_error, last_checked_at) so the next scan (consecutive_failures, last_error, last_checked_at) so the next scan
+16 -7
View File
@@ -239,8 +239,8 @@ async def test_tick_downloads_unseen_and_marks_seen(source_id, sync_engine, tmp_
# plan #704: structured run_stats carry the real counts. # plan #704: structured run_stats carry the real counts.
assert result.run_stats["downloaded_count"] == 2 assert result.run_stats["downloaded_count"] == 2
assert result.posts_processed == 1 assert result.posts_processed == 1
# The media wait for phase 3 to import them; only the post key is in yet. # The media and the post record both wait for phase 3 to import them (#4436).
assert _count_ledger(sync_engine, source_id) == 1 assert _count_ledger(sync_engine, source_id) == 0
result.mark_seen_after_import() result.mark_seen_after_import()
# 2 media keys + 1 synthetic post key (body/links recaptured per post). # 2 media keys + 1 synthetic post key (body/links recaptured per post).
assert _count_ledger(sync_engine, source_id) == 3 assert _count_ledger(sync_engine, source_id) == 3
@@ -648,8 +648,10 @@ async def test_recovery_tier2_disk_still_skips(source_id, sync_engine, tmp_path)
assert result.files_downloaded == 0 assert result.files_downloaded == 0
assert downloader.download_calls == 0 assert downloader.download_calls == 0
assert result.written_paths == [] assert result.written_paths == []
# Disk-skip reconciles the media key + the synthetic post key (recovery # Disk-skip reconciles the media key at once; the synthetic post key
# recaptures the body/links per post) = 2. # (recovery recaptures the body/links per post) waits for phase 3 (#4436).
assert _count_ledger(sync_engine, source_id) == 1
result.mark_seen_after_import()
assert _count_ledger(sync_engine, source_id) == 2 assert _count_ledger(sync_engine, source_id) == 2
@@ -943,8 +945,10 @@ async def test_a_run_that_dies_before_import_leaves_its_media_unmarked(
_FakeDownloader(tmp_path)) _FakeDownloader(tmp_path))
ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest", ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="tick") url="https://patreon.com/ingest", mode="tick")
# Phase 3 never ran, so `mark_seen_after_import` never did: only the post key. # Phase 3 never ran, so `mark_seen_after_import` never did: neither the
assert _count_ledger(sync_engine, source_id) == 1 # media nor the post record is marked (#4436 — the record, marked at write
# time, left the post undated for good once the run died before phase 3).
assert _count_ledger(sync_engine, source_id) == 0
# The next walk finds the file on disk with no record, and imports it. # The next walk finds the file on disk with no record, and imports it.
ing2 = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1])])]), ing2 = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1])])]),
@@ -954,6 +958,7 @@ async def test_a_run_that_dies_before_import_leaves_its_media_unmarked(
assert result.written_paths == [str(tmp_path / "p1_1.jpg")] assert result.written_paths == [str(tmp_path / "p1_1.jpg")]
assert result.files_downloaded == 0 # not fetched again assert result.files_downloaded == 0 # not fetched again
assert "on disk but never imported: p1_1.jpg" in result.stdout assert "on disk but never imported: p1_1.jpg" in result.stdout
assert len(result.post_record_paths) == 1 # the record is written again
result.mark_seen_after_import() result.mark_seen_after_import()
assert _count_ledger(sync_engine, source_id) == 2 assert _count_ledger(sync_engine, source_id) == 2
@@ -1200,7 +1205,10 @@ async def test_tick_captures_media_less_post_once(source_id, sync_engine, tmp_pa
assert result.success is True assert result.success is True
assert len(result.post_record_paths) == 1 assert len(result.post_record_paths) == 1
assert downloader.post_records == 1 assert downloader.post_records == 1
# The synthetic `post:ptext` key was marked seen (gates re-capture). # The synthetic `post:ptext` key is marked once phase 3 has upserted the
# record (#4436), and then gates re-capture.
assert _count_ledger(sync_engine, source_id) == 0
result.mark_seen_after_import()
assert _count_ledger(sync_engine, source_id) == 1 assert _count_ledger(sync_engine, source_id) == 1
# Second walk: already recorded → gated, no re-write, no new ledger row. # Second walk: already recorded → gated, no re-write, no new ledger row.
@@ -1540,6 +1548,7 @@ async def test_revisits_do_not_feed_the_body_drift_canary(
url="https://patreon.com/ingest", mode="tick", revisit_days=30, url="https://patreon.com/ingest", mode="tick", revisit_days=30,
) )
assert first.success is True assert first.success is True
first.mark_seen_after_import() # phase 3 ran
# Second walk: every post is a revisit, and every body comes back empty. # Second walk: every post is a revisit, and every body comes back empty.
client2 = _FakeClient([(None, posts)], published=published, empty_body=True) client2 = _FakeClient([(None, posts)], published=published, empty_body=True)
+3 -39
View File
@@ -126,53 +126,17 @@ def test_process_sweep_flags_conflict_with_process_mode(db_sync):
def test_process_auto_source_never_trains_head(db_sync): def test_process_auto_source_never_trains_head(db_sync):
# The runaway break: provisional wip tags (process sweep 'process_auto', soft # The runaway break: provisional wip tags (process sweep 'process_auto') are NOT
# title 'wip_title_soft') are NOT training positives; a HARD title-heuristic / # training positives; a title-heuristic / manual one IS. So the head learns only
# manual one IS. So the head learns only from trusted labels, never its own # from trusted labels, never its own output (#1464).
# output or the low-precision sketch/doodle tier (#1464 + #1474).
wip = _system_tag(db_sync, "wip") wip = _system_tag(db_sync, "wip")
auto_img = _img(db_sync, "f" * 64, _emb(0)) auto_img = _img(db_sync, "f" * 64, _emb(0))
soft_img = _img(db_sync, "9" * 64, _emb(2))
title_img = _img(db_sync, "0" * 64, _emb(1)) title_img = _img(db_sync, "0" * 64, _emb(1))
db_sync.execute(image_tag.insert().values( db_sync.execute(image_tag.insert().values(
image_record_id=auto_img.id, tag_id=wip.id, source="process_auto")) image_record_id=auto_img.id, tag_id=wip.id, source="process_auto"))
db_sync.execute(image_tag.insert().values(
image_record_id=soft_img.id, tag_id=wip.id, source="wip_title_soft"))
db_sync.execute(image_tag.insert().values( db_sync.execute(image_tag.insert().values(
image_record_id=title_img.id, tag_id=wip.id, source="wip_title")) image_record_id=title_img.id, tag_id=wip.id, source="wip_title"))
db_sync.commit() db_sync.commit()
positives = set(_ids_with_tag(db_sync, wip.id)) positives = set(_ids_with_tag(db_sync, wip.id))
assert title_img.id in positives # trusted HARD label trains the head assert title_img.id in positives # trusted HARD label trains the head
assert auto_img.id not in positives # its own auto-applied output does NOT assert auto_img.id not in positives # its own auto-applied output does NOT
assert soft_img.id not in positives # low-precision soft tier does NOT
def test_soft_wip_conflict_audit_flags_ring_loud(db_sync):
# A soft-tagged image (sketch/doodle title) that ALSO scores high on a content
# head is probably finished art mis-tagged — flagged for review; a quiet one is not.
from backend.app.services.ml.heads import soft_wip_conflict_audit
s = db_sync.execute(select(MLSettings).where(MLSettings.id == 1)).scalar_one()
s.process_conflict_threshold = 0.6
wip = _system_tag(db_sync, "wip")
content = Tag(name="looksreal", kind=TagKind.general)
db_sync.add(content)
db_sync.flush()
_head(db_sync, content.id, 0, weight=1.0) # sigmoid(1)=0.73 > 0.6 conflict
ring = _img(db_sync, "1" * 64, _emb(0)) # scores on the content head
quiet = _img(db_sync, "2" * 64, _emb(5)) # orthogonal → 0.5 < 0.6
for img in (ring, quiet):
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=wip.id, source="wip_title_soft"))
db_sync.commit()
res = soft_wip_conflict_audit(db_sync)
assert res["n_flagged"] == 1
flag = db_sync.execute(
select(PresentationReview).where(PresentationReview.image_record_id == ring.id)
).scalar_one()
assert flag.mode == "process"
assert flag.conflict_tag_id == content.id
assert db_sync.execute(
select(PresentationReview).where(PresentationReview.image_record_id == quiet.id)
).scalar_one_or_none() is None
+70
View File
@@ -0,0 +1,70 @@
"""Migration 0113 (#4431): images linked to a post before its date arrived get
the post's date back. Runs the migration's data step against real rows."""
import importlib.util
from datetime import UTC, datetime
from pathlib import Path
import pytest
from backend.app.models import Artist, ImageProvenance, ImageRecord, Post
from tests.factories import make_image as _img
pytestmark = pytest.mark.integration
_MIGRATION = (
Path(__file__).resolve().parents[1]
/ "alembic" / "versions" / "0113_redate_native_images.py"
)
def _redate():
spec = importlib.util.spec_from_file_location("m0113", _MIGRATION)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod.redate_images
def _post(db, artist, epid, when):
p = Post(artist_id=artist.id, external_post_id=epid, post_date=when)
db.add(p)
db.flush()
return p
def _link(db, img, post, *, primary):
db.add(ImageProvenance(image_record_id=img.id, post_id=post.id))
if primary:
img.primary_post_id = post.id
db.flush()
def test_redate_images_from_their_posts(db_sync):
sent = datetime(2024, 3, 1, 18, 30, tzinfo=UTC)
earlier = datetime(2023, 1, 5, tzinfo=UTC)
artist = Artist(name="Alice", slug="alice")
db_sync.add(artist)
db_sync.flush()
undated = _post(db_sync, artist, "u1", None)
dated = _post(db_sync, artist, "d1", sent)
repost = _post(db_sync, artist, "r1", earlier)
stale = _img(db_sync, "a" * 64) # primary post dated, image not
reposted = _img(db_sync, "b" * 64) # also in an earlier post
orphan = _img(db_sync, "c" * 64) # only an undated post
_link(db_sync, stale, dated, primary=True)
_link(db_sync, reposted, dated, primary=True)
_link(db_sync, reposted, repost, primary=False)
_link(db_sync, orphan, undated, primary=True)
orphan_before = orphan.effective_date
_redate()(db_sync.connection())
db_sync.expire_all()
stale = db_sync.get(ImageRecord, stale.id)
reposted = db_sync.get(ImageRecord, reposted.id)
orphan = db_sync.get(ImageRecord, orphan.id)
assert stale.effective_date == sent
assert stale.earliest_post_date == sent
assert reposted.effective_date == sent # the primary post's date
assert reposted.earliest_post_date == earlier # the earliest post's date
assert orphan.effective_date == orphan_before # no date to take
+10
View File
@@ -14,6 +14,7 @@ re-implementation of it.
""" """
from __future__ import annotations from __future__ import annotations
import re
import subprocess import subprocess
from pathlib import Path from pathlib import Path
@@ -162,6 +163,15 @@ def test_the_first_release_describes_the_product_instead_of_diffing(shaped_histo
assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")] assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")]
@pytest.mark.parametrize("tag", ["v2026.08.28.2208", "v2026.08.29.1000"])
def test_the_release_page_cites_no_internal_rule_numbers(shaped_history, tag):
"""The release page is read by strangers. "rule 145" names a record in the
operator's own notes, which a reader cannot open — say what the rule means
instead. Covers the first-release overview and the changelog body."""
body = body_of(notes(tag, cwd=shaped_history))
assert not re.search(r"\brule\s+\d+", body, re.IGNORECASE), body
def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history): def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history):
"""Two hand-maintained descriptions of one product drift and nothing """Two hand-maintained descriptions of one product drift and nothing
catches it. The release page quotes README.md so there is one source.""" catches it. The release page quotes README.md so there is one source."""
+90
View File
@@ -0,0 +1,90 @@
"""Migration 0112 (milestone 430, #4428): retiring the sketch/doodle WIP title tier.
Runs the migration's data step against real rows. The soft tags the operator stood
behind (confirmed, or kept through the review strip) survive as `manual`; the rest go,
and so do the unresolved review cards whose tag went with them. Hard title tags and
their flags are untouched.
"""
import importlib.util
from datetime import UTC, datetime
from pathlib import Path
import pytest
from sqlalchemy import select
from backend.app.models import PresentationReview, TagPositiveConfirmation
from backend.app.models.tag import image_tag
from backend.app.services.wip_title import resolve_wip_tag_id
from tests.factories import make_image as _img
pytestmark = pytest.mark.integration
_MIGRATION = (
Path(__file__).resolve().parents[1]
/ "alembic" / "versions" / "0112_retire_soft_wip_title.py"
)
def _retire():
spec = importlib.util.spec_from_file_location("m0112", _MIGRATION)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod.retire_soft_wip_tags
def _source(db, image_id, tag_id):
return db.execute(
select(image_tag.c.source)
.where(image_tag.c.image_record_id == image_id)
.where(image_tag.c.tag_id == tag_id)
).scalar_one_or_none()
def _review(db, image_id, tag_id):
return db.execute(
select(PresentationReview)
.where(PresentationReview.image_record_id == image_id)
.where(PresentationReview.tag_id == tag_id)
).scalar_one_or_none()
def _flag(db, image_id, tag_id, *, resolved):
db.add(PresentationReview(
image_record_id=image_id, tag_id=tag_id, conflict_score=0.8, mode="process",
resolved_at=datetime.now(UTC) if resolved else None,
))
def test_retire_soft_wip_keeps_human_judged_and_clears_the_rest(db_sync):
wip = resolve_wip_tag_id(db_sync)
plain = _img(db_sync, "a" * 64) # soft, never looked at → removed
confirmed = _img(db_sync, "b" * 64) # soft + confirmed → kept as manual
kept = _img(db_sync, "c" * 64) # soft + "Keep tag" in the strip → manual
pending = _img(db_sync, "d" * 64) # soft + open card → tag and card removed
hard = _img(db_sync, "e" * 64) # hard title tag + open card → untouched
for img in (plain, confirmed, kept, pending):
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=wip, source="wip_title_soft"))
db_sync.execute(image_tag.insert().values(
image_record_id=hard.id, tag_id=wip, source="wip_title"))
db_sync.add(TagPositiveConfirmation(image_record_id=confirmed.id, tag_id=wip))
_flag(db_sync, kept.id, wip, resolved=True)
_flag(db_sync, pending.id, wip, resolved=False)
_flag(db_sync, hard.id, wip, resolved=False)
db_sync.flush()
_retire()(db_sync.connection())
db_sync.expire_all()
assert _source(db_sync, plain.id, wip) is None
assert _source(db_sync, confirmed.id, wip) == "manual"
assert _source(db_sync, kept.id, wip) == "manual"
assert _source(db_sync, pending.id, wip) is None
assert _source(db_sync, hard.id, wip) == "wip_title"
assert _review(db_sync, pending.id, wip) is None
assert _review(db_sync, kept.id, wip) is not None # resolved history stays
assert _review(db_sync, hard.id, wip) is not None # its tag is still on
assert db_sync.execute(
select(image_tag.c.image_record_id).where(image_tag.c.source == "wip_title_soft")
).first() is None
+81
View File
@@ -382,3 +382,84 @@ def test_external_links_not_duplicated_on_reimport(importer, import_layout):
assert importer.session.execute( assert importer.session.execute(
select(func.count()).select_from(ExternalLink) select(func.count()).select_from(ExternalLink)
).scalar_one() == 1 ).scalar_one() == 1
def test_post_record_redates_images_linked_before_it(importer, import_layout):
"""#4431: the native ingesters import a message's media before its record,
and only the record carries the date. The images start on their download
time; when the record lands they take the post's date."""
import_root, _ = import_layout
artist = Artist(name="Alice", slug="alice")
importer.session.add(artist)
importer.session.flush()
m = import_root / "Alice" / "20240301_123_01_art.jpg"
_split(m, "v")
_sidecar(m, {"category": "discord", "message_id": "123"})
r = importer.import_one(m)
assert r.status == "imported"
rec = importer.session.get(ImageRecord, r.image_id)
post = importer.session.execute(select(Post)).scalar_one()
assert post.post_date is None
download_time = rec.effective_date
sc = import_root / "Alice" / "20240301_123_post.json"
sc.write_text(json.dumps({
"category": "discord", "message_id": "123", "message": "",
"date": "2024-03-01T18:30:00.000000+00:00",
}))
assert importer.upsert_post_record(sc, artist=artist) is True
importer.session.expire_all()
rec = importer.session.get(ImageRecord, r.image_id)
post = importer.session.execute(select(Post)).scalar_one()
assert post.post_date is not None
assert post.post_date != download_time
assert rec.effective_date == post.post_date
assert rec.earliest_post_date == post.post_date
def test_an_undated_post_is_dated_from_the_record_its_walk_left_on_disk(
importer, import_layout,
):
"""#4436: a walk killed before phase 3 wrote the message's record to disk
but never upserted it, and marked it seen, so no later tick fixed it. The
repair finds the record under the artist's folder and dates the post — and
its images — under the post's own source."""
from backend.app.services.post_record_repair import date_posts_from_records
import_root, images_root = import_layout
artist = Artist(name="Alice", slug="alice")
importer.session.add(artist)
importer.session.flush()
importer.session.add(Source(
artist_id=artist.id, platform="discord",
url="https://discord.com/channels/1/100",
))
importer.session.flush()
m = import_root / "Alice" / "20240301_123_01_art.jpg"
_split(m, "v")
_sidecar(m, {"category": "discord", "message_id": "123"})
r = importer.import_one(m)
assert r.status == "imported"
post = importer.session.execute(select(Post)).scalar_one()
assert post.post_date is None and post.source_id is not None
channel = images_root / "alice" / "discord" / "rewards"
channel.mkdir(parents=True)
(channel / "20240301_123_post.json").write_text(json.dumps({
"category": "discord", "message_id": "123", "message": "",
"date": "2024-03-01T18:30:00.000000+00:00",
}))
(channel / "20240301_999_post.json").write_text("not json") # skipped, not fatal
summary = date_posts_from_records(importer.session, importer, images_root)
assert summary == {"undated": 1, "dated": 1, "no_record": 0}
importer.session.expire_all()
post = importer.session.execute(select(Post)).scalar_one()
rec = importer.session.get(ImageRecord, r.image_id)
assert post.post_date is not None
assert rec.earliest_post_date == post.post_date
# Nothing left undated: the next sweep is an empty query.
assert date_posts_from_records(importer.session, importer, images_root) == {
"undated": 0, "dated": 0, "no_record": 0,
}
+1 -27
View File
@@ -8,7 +8,7 @@ negative cases (substrings like ``swipe`` / ``wiped``) are the load-bearing ones
""" """
import pytest import pytest
from backend.app.services.wip_title import matches_soft_wip_title, matches_wip_title from backend.app.services.wip_title import matches_wip_title
@pytest.mark.parametrize("title", [ @pytest.mark.parametrize("title", [
@@ -48,29 +48,3 @@ def test_matches_positive(title):
]) ])
def test_matches_negative(title): def test_matches_negative(title):
assert matches_wip_title(title) is False assert matches_wip_title(title) is False
@pytest.mark.parametrize("title", [
"quick sketch",
"Nami sketch",
"morning doodle",
"some doodles",
"sketches from today",
"a little scribble",
"SKETCH",
])
def test_soft_matches_positive(title):
assert matches_soft_wip_title(title) is True
@pytest.mark.parametrize("title", [
None,
"",
"sketchbook tour", # 'sketch' inside sketchbook — must NOT match
"kadoodle mascot", # 'doodle' mid-word
"the final piece",
"WIP", # a HARD cue is not a SOFT cue
"prescribed colours", # 'scrib' inside prescribed — must NOT match
])
def test_soft_matches_negative(title):
assert matches_soft_wip_title(title) is False
-12
View File
@@ -10,7 +10,6 @@ from backend.app.celery_app import celery
from backend.app.models import Artist, ImageProvenance, ImageRecord, Post, Source from backend.app.models import Artist, ImageProvenance, ImageRecord, Post, Source
from backend.app.models.tag import image_tag from backend.app.models.tag import image_tag
from backend.app.services.wip_title import ( from backend.app.services.wip_title import (
WIP_TITLE_SOFT_SOURCE,
apply_wip_image_tags, apply_wip_image_tags,
resolve_wip_tag_id, resolve_wip_tag_id,
) )
@@ -108,14 +107,3 @@ def test_backfill_tags_only_wip_titled_posts(db_sync):
# Idempotent: a second sweep finds the tag already present and applies nothing. # Idempotent: a second sweep finds the tag already present and applies nothing.
assert backfill_wip_title_tags.apply().get() == 0 assert backfill_wip_title_tags.apply().get() == 0
def test_apply_soft_source_stamps_wip_title_soft(db_sync):
# The soft tier (#1474) stamps a distinct provisional source.
tag_id = resolve_wip_tag_id(db_sync)
rec = _img(db_sync)
db_sync.commit()
assert apply_wip_image_tags(
db_sync, [rec.id], tag_id, source=WIP_TITLE_SOFT_SOURCE
) == 1
assert _wip_source(db_sync, rec.id, tag_id) == "wip_title_soft"