From 84f13135ce9ebca00e59308a143a787a2b203892 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 07:48:25 -0400 Subject: [PATCH 01/94] =?UTF-8?q?feat:=20worker=20lanes=20become=20rows=20?= =?UTF-8?q?=E2=80=94=20slots,=20a=20settable=20cap,=20a=20derived=20ceilin?= =?UTF-8?q?g=20(4291)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 1. The data model the rest of the milestone reads. No behaviour change: nothing consumes these rows yet, and every lane still boots at its CELERY_CONCURRENCY env value. Three numbers, not two, per the operator's distinction — the derived value is a cap ON the cap: slots <= slots_cap <= derived_ceiling (live) (operator) (computed) They can always lower their own cap; they cannot raise it past what the container can hold. The ceiling is never stored, so a row written on a 32GB host and later run in a 4GB container is bounded by the 4GB. `services/worker_lanes.py` is the one place that knows the lane set. `models/worker_lane.py` holds only what an operator may change. Two deviations from the step as written, both deliberate: QUEUES ARE NOT A COLUMN. The step body said the row carries its `-Q` list, but a lane's queues are decided by celery_app's task_routes, not by preference — an operator cannot move a backup off maintenance_long. Storing them would create a row that can contradict the routing table, with nothing to notice until a queue had no consumer. So queues are code, slots are data. `test_every_routed_queue_has_a_lane_that_serves_it` reads the real routing table and fails if a route is ever added without a lane. ROLE_NAMES IS NOW DERIVED, not left alone. It was a hand-kept second copy of "queue set -> display name" and had already drifted: maintenance_long is a live lane with four task routes and a dedicated worker in the operator's stack, and the roster did not know its name — so the System tab labelled it `Worker (maintenance_long)`. Adding a lane table beside it would have made three copies. The ceiling honours cgroup limits rather than the host's. `os.cpu_count()` reports the HOST's cores from inside a container, so a 4-core quota on a 32-core host would otherwise offer 32 slots — and the operator's own stack sets `cpus: '4.0'` on ml-worker, so that is real configuration, not a hypothetical. Memory reads cgroup v2 then v1, and recognises v1's PAGE_SIZE-aligned LONG_MAX sentinel by magnitude rather than treating it as petabytes. Every uncertain case fails LOW. An unreadable limit yields UNKNOWN_CEILING, never unlimited — not knowing how much memory there is must not read as plenty. A box too small to hold one model beside the web process gets an ML ceiling of 0 rather than a floor of 1: offering a slot that OOMs the container the first time it is used is exactly what this exists to prevent. ML_BYTES_PER_SLOT is 4 GiB and is UNMEASURED — flagged as such in the code, with the method for replacing it with a real figure. It decides whether a stranger's server survives enabling tagging, so it errs toward refusing a slot that would have fitted. Seeded one-of-each with ml at 0 and disabled (alembic 0103). ML off is step 6's requirement arriving early: enabling the lane is what triggers the SigLIP download, and rule 164 permits a runtime fetch only for a feature that is optional and clearly off. The seed values are literals rather than an import of LANES — a migration is a statement about one moment, and importing the live defaults would silently change what this revision does on a fresh database in 2027. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- alembic/versions/0103_worker_lane_settings.py | 121 ++++++++ backend/app/models/__init__.py | 2 + backend/app/models/worker_lane.py | 86 ++++++ backend/app/services/service_roster.py | 16 +- backend/app/services/worker_lanes.py | 281 ++++++++++++++++++ tests/test_worker_lanes.py | 258 ++++++++++++++++ 6 files changed, 759 insertions(+), 5 deletions(-) create mode 100644 alembic/versions/0103_worker_lane_settings.py create mode 100644 backend/app/models/worker_lane.py create mode 100644 backend/app/services/worker_lanes.py create mode 100644 tests/test_worker_lanes.py diff --git a/alembic/versions/0103_worker_lane_settings.py b/alembic/versions/0103_worker_lane_settings.py new file mode 100644 index 0000000..5dca467 --- /dev/null +++ b/alembic/versions/0103_worker_lane_settings.py @@ -0,0 +1,121 @@ +"""worker_lane — settings-backed slots for each celery lane. + +Milestone 422 step 1. One row per lane, holding only what an operator can +change: how many slots it runs, the ceiling they have set for themselves, and +whether it consumes its queues at all. + +## What is deliberately not a column + +**The queues.** They are decided by `celery_app.py`'s `task_routes`, not by +preference, so a stored copy could contradict the routing table with nothing +to notice until a queue had no consumer. They live in +`services/worker_lanes.py`. + +**The derived ceiling.** Computed from the container's cgroup limits on every +read. A row written on a 32GB host and later run in a 4GB container must be +bounded by the 4GB; a stored ceiling would quietly authorise what the box can +no longer hold. + +## The seeded values + +Written out literally rather than imported from `worker_lanes.LANES`. A +migration is a statement about one moment in the schema's history — if it +imported the live defaults, changing them in 2027 would silently change what +this 2026 revision does on a fresh database. The two are allowed to diverge +afterwards, and that is correct: `LANES` supplies defaults for a lane added +later, this file records what was seeded today. + + lane slots cap enabled + worker 1 4 yes + scheduler 1 2 yes + maintenance_long 1 2 yes + ml 0 1 NO + +One of each, per the operator (2026-09-22: *"that starting value should be one +of each"*), and far below their own production numbers — worker 8 and ml 2 are +tuned for their hardware and are not a sane first boot for a stranger. + +**ml ships at zero and disabled**, which is milestone 422 step 6's requirement +arriving early: enabling the lane is what triggers the SigLIP download, and +rule 164 permits a runtime fetch only for a feature that is "optional and +clearly off". Seeding it on would make every fresh install reach HuggingFace. + +The caps start low on purpose. A cap that begins at the ceiling is a rubber +stamp; starting at 4/2/2/1 means raising slots within the cap is ordinary and +raising the cap is a deliberate act. + +## Existing installs + +Nothing is migrated FROM. The `CELERY_QUEUES` / `CELERY_CONCURRENCY` env vars +stay exactly as they are and remain the baseline each lane boots at; these +rows are the adjustment applied on top (step 3). So this migration changes no +behaviour on a running stack — it only makes the numbers storable. + +Revision ID: 0103 +Revises: 0102 +Create Date: 2026-09-22 + +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "0103" +down_revision: Union[str, None] = "0102" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +# (name, slots, slots_cap, enabled) — see the docstring for why these are +# literals and not an import. +_SEED = ( + ("worker", 1, 4, True), + ("scheduler", 1, 2, True), + ("maintenance_long", 1, 2, True), + ("ml", 0, 1, False), +) + + +def upgrade() -> None: + worker_lane = op.create_table( + "worker_lane", + sa.Column("name", sa.String(length=32), nullable=False), + sa.Column("slots", sa.Integer(), nullable=False), + sa.Column("slots_cap", sa.Integer(), nullable=False), + sa.Column("enabled", sa.Boolean(), nullable=False), + sa.Column( + "updated_at", sa.DateTime(timezone=True), + server_default=sa.text("now()"), nullable=False, + ), + sa.PrimaryKeyConstraint("name", name=op.f("pk_worker_lane")), + # Bare constraint names: Base.metadata's naming convention prepends + # ck_worker_lane_, and pre-prefixing doubles it — the defect alembic + # 0088 had to rename four constraints for (#3275). op.f() marks these + # as already-final so autogenerate does not propose renaming them. + sa.CheckConstraint("slots >= 0", name=op.f("ck_worker_lane_slots_non_negative")), + sa.CheckConstraint("slots_cap >= 0", name=op.f("ck_worker_lane_cap_non_negative")), + # The invariant that makes the cap mean anything, in the database + # rather than only in the service: a row violating it is not a + # rejected request, it is a lane that step 3's reconcile will drive UP + # to a number the operator capped. + sa.CheckConstraint("slots <= slots_cap", name=op.f("ck_worker_lane_slots_within_cap")), + ) + # No index beyond the primary key, deliberately — four rows, forever. Same + # reasoning as service_seen, and the lesson of #3301, which removed seven + # indexes that were write cost buying nothing. + op.bulk_insert( + worker_lane, + [ + {"name": name, "slots": slots, "slots_cap": cap, "enabled": enabled} + for name, slots, cap, enabled in _SEED + ], + ) + + +def downgrade() -> None: + # The rows go with the table. They are settings with shipped defaults, not + # operator data that predates this revision — a downgrade returns the stack + # to reading its concurrency from env, which is where it reads it from + # today anyway. + op.drop_table("worker_lane") diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index bc0cf30..08f1d3f 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -44,6 +44,7 @@ from .tag_head import TagHead from .tag_positive_confirmation import TagPositiveConfirmation from .tag_suggestion_rejection import TagSuggestionRejection from .task_run import TaskRun +from .worker_lane import WorkerLane __all__ = [ "Base", @@ -94,4 +95,5 @@ __all__ = [ "TagPositiveConfirmation", "TagSuggestionRejection", "TaskRun", + "WorkerLane", ] diff --git a/backend/app/models/worker_lane.py b/backend/app/models/worker_lane.py new file mode 100644 index 0000000..ae3cc09 --- /dev/null +++ b/backend/app/models/worker_lane.py @@ -0,0 +1,86 @@ +"""worker_lane — how many slots the operator wants each worker lane to have. + +Milestone 422 step 1. One row per lane in `services/worker_lanes.LANES`. + +## What is NOT in here + +The lane's queues and its display name. Those are decided by `celery_app.py`'s +`task_routes` — an operator cannot move a backup off `maintenance_long` — so +storing them would be a row that can contradict the routing table, with +nothing to notice until a queue had no consumer. They live in +`services/worker_lanes.py`; this table holds only what an operator may change. + +The DERIVED CEILING is also absent, and that is deliberate rather than an +omission. It is computed from the container's cgroup limits on every read, so +a row written on a 32GB host and later run in a 4GB container is bounded by +the 4GB — a stored ceiling would quietly authorise what the box can no longer +hold. + +## The three numbers + + slots <= slots_cap <= derived_ceiling + (live) (this row) (computed) + +Operator's distinction, 2026-09-22: the derived value is *a cap on the cap*. +`slots_cap` is theirs and is always lowerable; it simply may not exceed what +the container can hold. The CHECK constraint below enforces the left half, +which is a fact about the row; the right half is enforced at write, because +it depends on a value no database column holds. + +## enabled + +Whether the lane consumes its queues at all. This is how ML ships off +(milestone 422 step 6): `enabled=false` with `slots=0`, so a fresh install +never loads a model or reaches HuggingFace, and turning tagging on in +Settings is what triggers the fetch. + +Not a substitute for `slots=0`. A lane can be enabled with zero slots while +it is being resized, and the two answer different questions: `enabled` is +intent, `slots` is capacity. +""" + +from datetime import datetime + +from sqlalchemy import ( + Boolean, + CheckConstraint, + DateTime, + Integer, + String, + func, +) +from sqlalchemy.orm import Mapped, mapped_column + +from .base import Base + + +class WorkerLane(Base): + __tablename__ = "worker_lane" + __table_args__ = ( + # Bare names — Base.metadata's naming convention prepends + # ck_worker_lane_. Pre-prefixing here doubles it, which is what + # alembic 0088 had to rename four constraints for (#3275). + CheckConstraint("slots >= 0", name="slots_non_negative"), + CheckConstraint("slots_cap >= 0", name="cap_non_negative"), + # The invariant that makes the cap mean anything. Enforced in the + # database rather than only in the service, because a row that + # violates it is not a rejected request — it is a lane that will be + # reconciled UP to a value the operator capped. + CheckConstraint("slots <= slots_cap", name="slots_within_cap"), + ) + + # The lane name from services/worker_lanes.LANES — never a container + # hostname. See models/service_seen.py for why: celery's worker names here + # are `celery@`, minted fresh on every deploy. + name: Mapped[str] = mapped_column(String(32), primary_key=True) + + slots: Mapped[int] = mapped_column(Integer, nullable=False) + slots_cap: Mapped[int] = mapped_column(Integer, nullable=False) + enabled: Mapped[bool] = mapped_column(Boolean, nullable=False) + + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + nullable=False, + server_default=func.now(), + onupdate=func.now(), + ) diff --git a/backend/app/services/service_roster.py b/backend/app/services/service_roster.py index 5b16f3b..ff831b2 100644 --- a/backend/app/services/service_roster.py +++ b/backend/app/services/service_roster.py @@ -36,6 +36,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession from ..models import ServiceSeen +from .worker_lanes import LANES log = logging.getLogger(__name__) @@ -53,13 +54,18 @@ INSPECT_TIMEOUT_SECONDS = 2.0 # Queue set -> the name an operator recognises. Sorted-tuple keys, because the # order celery reports them in is not guaranteed. # -# A deployment that slices CELERY_QUEUES differently falls through to the raw -# queue list rather than being given a name this table invented for it: a +# DERIVED from `worker_lanes.LANES` (milestone 422 step 1) rather than written +# out here. It was a hand-kept second copy of the same fact, and it had already +# drifted: `maintenance_long` is a live lane with four task routes pointing at +# it and a dedicated worker in the operator's stack, and this map did not know +# it — so the System tab labelled it `Worker (maintenance_long)`. One list of +# lanes now names them everywhere. +# +# A deployment that slices CELERY_QUEUES differently still falls through to the +# raw queue list rather than being given a name this code invented for it: a # wrong-but-confident label on a status page is worse than an ugly true one. ROLE_NAMES: dict[tuple[str, ...], str] = { - ("default", "download", "import", "thumbnail"): "Worker", - ("maintenance", "scan"): "Scheduler", - ("ml",): "ML worker", + lane.queue_key: lane.display_name for lane in LANES } diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py new file mode 100644 index 0000000..a0d7444 --- /dev/null +++ b/backend/app/services/worker_lanes.py @@ -0,0 +1,281 @@ +"""The worker lanes: what they are, and how many slots each may be given. + +Milestone 422 step 1. This module is the ONE place that knows the lane set; +`models/worker_lane.py` holds only what the operator can change about them. + +## Why the queues are here and not in the table + +A lane's queue set is not a preference — it is decided by `celery_app.py`'s +`task_routes`, which is what puts a backup on `maintenance_long` and a +thumbnail on `thumbnail`. An operator cannot move a task to another lane, so +storing the queues as settings would create a row that can disagree with the +routing table, and nothing would notice until a queue had no consumer. + +So: queues and display names are code, slots and caps are data. The table +stores three numbers and a flag, and nothing that could contradict celery. + +This also collapses a duplicate rather than adding one. +`service_roster.ROLE_NAMES` was a second copy of "queue set -> the name an +operator recognises", and it had already drifted: `maintenance_long` is a +live lane with four task routes pointing at it, and the roster did not know +its name, so the System tab rendered it as `Worker (maintenance_long)`. That +map is now derived from `LANES` below, so a lane added here is named +everywhere at once. + +## Why the ceiling is derived rather than configured + +Consolidating the stack into one container (step 5) widens the OOM blast +radius: today an ml-worker that exhausts memory is killed by Docker on its +own, and web keeps serving. In one container the kernel picks a victim from +the whole cgroup, and it may pick hypercorn — so a tagging task can take the +UI down with it, on exactly the modest hardware least able to spare the +memory. + +Operator, 2026-09-22: *"ram isn't an issue for me but some users might run +this on weaker hardware and I don't want it to kill their servers."* + +So the maximum is computed from what the container actually has, and the +operator's own `slots_cap` must fit under it. Three numbers, not two, and the +ordering is the point: + + slots <= slots_cap <= derived_ceiling + (live) (operator) (this module) + +The operator can always lower their cap. They cannot raise it past what the +box can hold. The derived ceiling is never stored — a row that outlived a +change in container limits must not carry a stale one. +""" + +from __future__ import annotations + +import logging +import os +from dataclasses import dataclass +from pathlib import Path + +log = logging.getLogger(__name__) + +# --- the lanes --------------------------------------------------------------- + + +@dataclass(frozen=True) +class Lane: + """A worker lane. `name` is the stable key the settings row is keyed on. + + Keyed on a lane NAME rather than a container hostname for the reason + `models/service_seen.py` gives at length: celery's worker names here are + `celery@` and are minted fresh on every deploy, so anything + keyed on them records a death and a birth every time the stack updates. + """ + + name: str + display_name: str + queues: tuple[str, ...] + default_slots: int + # The cap a lane STARTS with, which is not the ceiling. Set low enough + # that raising slots within it is an ordinary adjustment, and raising the + # cap itself is a deliberate act — a cap that begins at the ceiling is a + # rubber stamp and protects nobody. + default_slots_cap: int + default_enabled: bool + # True when a slot costs a copy of the ML model rather than just a process. + # The only lane whose ceiling is decided by memory instead of by cores. + memory_bound: bool = False + + @property + def queue_key(self) -> tuple[str, ...]: + """The sorted queue set, which is how `service_seen` identifies a + running worker. The join between what is configured here and what + `celery inspect` reports.""" + return tuple(sorted(self.queues)) + + +# Defaults are ONE OF EACH, with ML off — operator, 2026-09-22: *"that +# starting value should be one of each."* Deliberately far below the +# operator's own production numbers (worker 8, ml 2), which are tuned for +# their hardware and are not a sane first boot for a stranger. +# +# ML ships disabled because enabling it is what triggers the SigLIP download +# (milestone 422 step 6) — rule 164 allows a feature that needs a fetch only +# when it is "optional and clearly off", and off-by-default is also what keeps +# a small box from loading a multi-GB model it was never asked to load. +LANES: tuple[Lane, ...] = ( + Lane( + name="worker", + display_name="Worker", + queues=("default", "import", "thumbnail", "download"), + default_slots=1, + default_slots_cap=4, + default_enabled=True, + ), + Lane( + name="scheduler", + display_name="Scheduler", + queues=("maintenance", "scan"), + default_slots=1, + default_slots_cap=2, + default_enabled=True, + ), + Lane( + name="maintenance_long", + display_name="Long maintenance", + queues=("maintenance_long",), + default_slots=1, + default_slots_cap=2, + default_enabled=True, + ), + Lane( + name="ml", + display_name="ML tagging", + queues=("ml",), + default_slots=0, + default_slots_cap=1, + default_enabled=False, + memory_bound=True, + ), +) + +LANES_BY_NAME: dict[str, Lane] = {lane.name: lane for lane in LANES} +LANES_BY_QUEUE_KEY: dict[tuple[str, ...], Lane] = { + lane.queue_key: lane for lane in LANES +} + + +# --- what the container actually has ----------------------------------------- + +# cgroup v2 first, then v1. A container started without an explicit memory +# limit reports "max" on v2 and a sentinel near 2**63 on v1; both mean "no +# limit", and the answer then is the host's RAM. +_CGROUP_V2_MEMORY = Path("/sys/fs/cgroup/memory.max") +_CGROUP_V1_MEMORY = Path("/sys/fs/cgroup/memory/memory.limit_in_bytes") +_CGROUP_V2_CPU = Path("/sys/fs/cgroup/cpu.max") +_CGROUP_V1_CPU_QUOTA = Path("/sys/fs/cgroup/cpu/cpu.cfs_quota_us") +_CGROUP_V1_CPU_PERIOD = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us") + +# A v1 "unlimited" is PAGE_SIZE-aligned LONG_MAX, not a round number, so it is +# recognised by magnitude rather than by equality. Anything claiming more than +# a petabyte is a sentinel, not a machine. +_UNLIMITED_ABOVE = 1 << 50 + +GIB = 1024 ** 3 + +# Memory one ML slot needs: the SigLIP so400m weights plus the runtime holding +# them. Prefork forks a child per slot and each child loads its own copy, so +# this multiplies — it is not a one-off cost. +# +# UNMEASURED AND DELIBERATELY CONSERVATIVE. This number decides whether a +# stranger's server survives enabling tagging, so it errs toward refusing a +# slot that would have fitted rather than granting one that will not. To +# replace it with a real figure: enable the lane on a container with a known +# limit, run one tagging task, and read the worker child's peak RSS +# (`grep VmHWM /proc//status`). Put the measurement in the commit +# message when you do. +ML_BYTES_PER_SLOT = 4 * GIB + +# Held back for hypercorn and the non-ML lanes before any ML slot is offered. +# In the consolidated container these share one cgroup with ML, and they are +# the processes an OOM kill must not take (see the module docstring). +RESERVED_BYTES = 2 * GIB + +# The floor a cores-derived ceiling never goes below. A single-core box still +# needs to be able to run its lanes; the ceiling exists to stop absurd values, +# not to make a small machine unusable. +MIN_CEILING = 1 + +# What an unreadable limit yields. Low rather than unlimited, on purpose: not +# knowing how much memory there is must never read as "plenty". An unswept +# absence is not a verdict. +UNKNOWN_CEILING = 1 + + +def _read_int(path: Path) -> int | None: + try: + raw = path.read_text().strip() + except OSError: + return None + if raw == "max": + return None + try: + return int(raw) + except ValueError: + return None + + +def container_memory_bytes() -> int | None: + """The memory this container may use, or None when it cannot be read. + + None means UNKNOWN, never UNLIMITED. Every caller must treat it as the + conservative case — the whole point of the ceiling is to protect a machine + whose size we are unsure of. + """ + for path in (_CGROUP_V2_MEMORY, _CGROUP_V1_MEMORY): + value = _read_int(path) + if value is not None and value < _UNLIMITED_ABOVE: + return value + if value is not None: + # A sentinel: the cgroup exists but sets no limit, so the real + # bound is the host's. + break + try: + return os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") + except (ValueError, OSError, AttributeError): + return None + + +def container_cpu_count() -> int | None: + """Effective cores, honouring a cgroup CPU quota. + + `os.cpu_count()` reports the HOST's cores from inside a container, so a + quota of 2.0 on a 32-core host would otherwise offer 32 slots. The + operator's own stack sets `cpus: '4.0'` on ml-worker, so this is a real + configuration here and not a hypothetical. + """ + quota: float | None = None + try: + raw = _CGROUP_V2_CPU.read_text().strip().split() + if raw and raw[0] != "max": + quota = int(raw[0]) / int(raw[1]) + except (OSError, ValueError, IndexError, ZeroDivisionError): + pass + if quota is None: + q = _read_int(_CGROUP_V1_CPU_QUOTA) + p = _read_int(_CGROUP_V1_CPU_PERIOD) + if q is not None and p and q > 0: + quota = q / p + if quota is not None and quota > 0: + return max(1, int(quota)) + return os.cpu_count() + + +def derived_ceiling(lane: Lane) -> int: + """The most slots `lane` may be given on this container. + + Never stored. Recomputed on every read so a container whose limits changed + is bounded by what it has NOW rather than by what it had when its row was + written. + """ + if lane.memory_bound: + total = container_memory_bytes() + if total is None: + log.warning( + "worker_lanes: cannot read a memory limit; capping %s at %d", + lane.name, UNKNOWN_CEILING, + ) + return UNKNOWN_CEILING + usable = total - RESERVED_BYTES + if usable < ML_BYTES_PER_SLOT: + # Honestly zero. A box that cannot hold one model alongside the web + # process must be told it cannot run tagging, not sold a slot that + # will OOM the container the first time it is used. + return 0 + return int(usable // ML_BYTES_PER_SLOT) + + cores = container_cpu_count() + if cores is None: + return UNKNOWN_CEILING + return max(MIN_CEILING, cores) + + +def ceilings() -> dict[str, int]: + """Every lane's ceiling, for the settings API and the UI.""" + return {lane.name: derived_ceiling(lane) for lane in LANES} diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py new file mode 100644 index 0000000..7482e4e --- /dev/null +++ b/tests/test_worker_lanes.py @@ -0,0 +1,258 @@ +"""Worker-lane definitions and the derived ceiling (milestone 422 step 1). + +Pure — no DB, no broker. The ceiling reads files under /sys/fs/cgroup, so the +tests point it at temporary files instead of at whatever the CI container +happens to have. +""" + +from __future__ import annotations + +import pytest + +from backend.app.services import worker_lanes as wl + + +# --- the lane definitions ---------------------------------------------------- + + +def test_lane_names_are_unique_and_stable(): + names = [lane.name for lane in wl.LANES] + assert len(names) == len(set(names)) + # The key every settings row and every API call uses. Renaming one orphans + # its row silently, so the set is pinned rather than merely counted. + assert set(names) == {"worker", "scheduler", "maintenance_long", "ml"} + + +def test_queue_sets_are_unique(): + """Two lanes sharing a queue set would be indistinguishable to + `celery inspect`, which groups by exactly that (see service_roster).""" + keys = [lane.queue_key for lane in wl.LANES] + assert len(keys) == len(set(keys)) + + +def test_every_routed_queue_has_a_lane_that_serves_it(): + """The check that actually matters: a queue celery routes tasks to, with + no lane consuming it, is work that queues forever and never runs. + + Read from `celery_app`'s real routing table rather than a list written + here, so adding a route without a lane fails this test. + """ + from backend.app.celery_app import celery as celery_app + + routes = celery_app.conf.task_routes or {} + routed = {spec["queue"] for spec in routes.values() if "queue" in spec} + served = {q for lane in wl.LANES for q in lane.queues} + assert routed - served == set(), ( + f"queues with no lane to consume them: {sorted(routed - served)}" + ) + + +def test_defaults_are_one_of_each_with_ml_off(): + """Operator, 2026-09-22: 'that starting value should be one of each.' + + ML at zero AND disabled is milestone 422 step 6's requirement arriving + early: enabling the lane is what triggers the SigLIP download, and rule + 164 allows a runtime fetch only for a feature that is optional and clearly + off. A default of 1 here would make every fresh install reach HuggingFace. + """ + by_name = wl.LANES_BY_NAME + for name in ("worker", "scheduler", "maintenance_long"): + assert by_name[name].default_slots == 1 + assert by_name[name].default_enabled is True + assert by_name["ml"].default_slots == 0 + assert by_name["ml"].default_enabled is False + + +def test_default_caps_leave_room_but_are_not_the_ceiling(): + """A cap that starts at the ceiling is a rubber stamp. Each lane's cap + must be at least its default slots (or the CHECK constraint rejects the + seeded row) and must leave somewhere to grow.""" + for lane in wl.LANES: + assert lane.default_slots_cap >= lane.default_slots + assert lane.default_slots_cap >= 1 + + +def test_only_ml_is_memory_bound(): + """Which lane is memory-bound decides which ceiling formula applies. If + another lane ever becomes so, it needs its own per-slot cost — the ML + figure is about a SigLIP copy and means nothing for a thumbnailer.""" + assert [lane.name for lane in wl.LANES if lane.memory_bound] == ["ml"] + + +# --- reading the container's limits ------------------------------------------ + + +def _point_memory_at(monkeypatch, tmp_path, contents: str | None, *, v2=True): + target = tmp_path / "memory.max" + if contents is not None: + target.write_text(contents) + attr = "_CGROUP_V2_MEMORY" if v2 else "_CGROUP_V1_MEMORY" + monkeypatch.setattr(wl, attr, target) + # Neutralise the other tier so the test controls exactly one source. + other = "_CGROUP_V1_MEMORY" if v2 else "_CGROUP_V2_MEMORY" + monkeypatch.setattr(wl, other, tmp_path / "absent") + + +def test_memory_read_from_cgroup_v2(monkeypatch, tmp_path): + _point_memory_at(monkeypatch, tmp_path, str(8 * wl.GIB)) + assert wl.container_memory_bytes() == 8 * wl.GIB + + +def test_memory_read_from_cgroup_v1(monkeypatch, tmp_path): + target = tmp_path / "limit_in_bytes" + target.write_text(str(6 * wl.GIB)) + monkeypatch.setattr(wl, "_CGROUP_V2_MEMORY", tmp_path / "absent") + monkeypatch.setattr(wl, "_CGROUP_V1_MEMORY", target) + assert wl.container_memory_bytes() == 6 * wl.GIB + + +def test_v2_max_means_no_limit_not_zero(monkeypatch, tmp_path): + """'max' is a container with no memory limit set. Falling through to host + RAM is right; reading it as 0 or as an error would cap a large machine at + the unknown-ceiling.""" + _point_memory_at(monkeypatch, tmp_path, "max") + value = wl.container_memory_bytes() + assert value is not None and value > wl.GIB + + +def test_v1_sentinel_is_recognised_as_unlimited(monkeypatch, tmp_path): + """cgroup v1 spells 'no limit' as a PAGE_SIZE-aligned LONG_MAX rather than + a word, so it is recognised by magnitude. Taken literally it would be + petabytes and the ML ceiling would be nonsense.""" + target = tmp_path / "limit_in_bytes" + target.write_text(str(9223372036854771712)) + monkeypatch.setattr(wl, "_CGROUP_V2_MEMORY", tmp_path / "absent") + monkeypatch.setattr(wl, "_CGROUP_V1_MEMORY", target) + value = wl.container_memory_bytes() + assert value is not None + assert value < wl._UNLIMITED_ABOVE + + +# --- the derived ceiling ----------------------------------------------------- + + +def test_ml_ceiling_is_memory_divided_by_per_slot_cost(monkeypatch, tmp_path): + # 2 GiB reserved for web and the other lanes, then 4 GiB per model copy. + _point_memory_at(monkeypatch, tmp_path, str(14 * wl.GIB)) + assert wl.derived_ceiling(wl.LANES_BY_NAME["ml"]) == 3 + + +def test_a_small_box_is_told_it_cannot_run_tagging(monkeypatch, tmp_path): + """Honestly zero rather than a floor of one. A 4GB box cannot hold a model + alongside the web process, and offering a slot that OOMs the container the + first time it is used is precisely what the ceiling exists to prevent — + the operator's 'I don't want it to kill their servers'.""" + _point_memory_at(monkeypatch, tmp_path, str(4 * wl.GIB)) + assert wl.derived_ceiling(wl.LANES_BY_NAME["ml"]) == 0 + + +def test_unreadable_memory_limit_yields_a_low_ceiling_not_an_unlimited_one( + monkeypatch, tmp_path +): + """The failure DIRECTION is the point. Not knowing how much memory there + is must never read as 'plenty' — an unswept absence is not a verdict.""" + _point_memory_at(monkeypatch, tmp_path, None) + monkeypatch.setattr(wl.os, "sysconf", lambda _: (_ for _ in ()).throw(OSError)) + assert wl.derived_ceiling(wl.LANES_BY_NAME["ml"]) == wl.UNKNOWN_CEILING + + +def test_cpu_lanes_are_bounded_by_cores_not_memory(monkeypatch, tmp_path): + """A CPU-bound lane on a memory-starved box must still get slots — its + slots are processes, not model copies.""" + _point_memory_at(monkeypatch, tmp_path, str(2 * wl.GIB)) + monkeypatch.setattr(wl, "container_cpu_count", lambda: 8) + assert wl.derived_ceiling(wl.LANES_BY_NAME["worker"]) == 8 + + +def test_cpu_quota_beats_host_core_count(monkeypatch, tmp_path): + """`os.cpu_count()` reports the HOST's cores from inside a container, so a + 4-core quota on a 32-core host would otherwise offer 32 slots. The + operator's own stack sets `cpus: '4.0'` on ml-worker, so this is real + configuration rather than a hypothetical.""" + quota = tmp_path / "cpu.max" + quota.write_text("400000 100000") + monkeypatch.setattr(wl, "_CGROUP_V2_CPU", quota) + monkeypatch.setattr(wl.os, "cpu_count", lambda: 32) + assert wl.container_cpu_count() == 4 + + +def test_a_single_core_box_still_gets_a_slot(monkeypatch): + monkeypatch.setattr(wl, "container_cpu_count", lambda: 1) + assert wl.derived_ceiling(wl.LANES_BY_NAME["worker"]) >= wl.MIN_CEILING + + +def test_the_ceiling_is_computed_not_stored(monkeypatch, tmp_path): + """The property the design rests on: the same lane yields a different + ceiling when the container's limits change, with no row edit. A stored + ceiling would keep authorising what the box no longer has.""" + lane = wl.LANES_BY_NAME["ml"] + _point_memory_at(monkeypatch, tmp_path, str(34 * wl.GIB)) + big = wl.derived_ceiling(lane) + (tmp_path / "memory.max").write_text(str(10 * wl.GIB)) + small = wl.derived_ceiling(lane) + assert big > small + + +def test_ceilings_covers_every_lane(): + assert set(wl.ceilings()) == {lane.name for lane in wl.LANES} + + +# --- the duplicate this step collapsed --------------------------------------- + + +def test_role_names_is_derived_from_the_lane_table(): + """`service_roster.ROLE_NAMES` was a hand-kept second copy of 'queue set + -> display name' and had already drifted: maintenance_long is a live lane + with four task routes and a dedicated worker, and the roster did not know + its name, so the System tab rendered `Worker (maintenance_long)`. + + Asserting the derivation rather than the contents — a test listing the + names again would be a third copy. + """ + from backend.app.services.service_roster import ROLE_NAMES, role_display_name + + assert ROLE_NAMES == {lane.queue_key: lane.display_name for lane in wl.LANES} + assert role_display_name(("maintenance_long",)) == "Long maintenance" + + +def test_an_unrecognised_queue_set_still_gets_a_true_label(): + """A deployment slicing CELERY_QUEUES differently must not be handed a + name this code invented for it.""" + from backend.app.services.service_roster import role_display_name + + assert role_display_name(("nonsense",)) == "Worker (nonsense)" + + +# --- the model's invariant --------------------------------------------------- + + +@pytest.mark.parametrize( + "slots,cap,ok", + [ + (0, 0, True), + (1, 1, True), + (1, 4, True), + (5, 4, False), # slots above its own cap + (-1, 1, False), # negative slots + (1, -1, False), # negative cap + ], +) +def test_worker_lane_check_constraints(slots, cap, ok): + """The constraints live in the database, not only in the service, because + a row violating `slots <= slots_cap` is not a rejected request — it is a + lane the reconcile (step 3) will drive UP to a number the operator + capped.""" + from backend.app.models import WorkerLane + + constraints = { + c.name: str(c.sqltext) for c in WorkerLane.__table__.constraints + if hasattr(c, "sqltext") + } + assert "slots_within_cap" in constraints + assert "slots_non_negative" in constraints + assert "cap_non_negative" in constraints + + # Evaluate the same predicates the database will, so the parametrize table + # documents what is accepted rather than restating the SQL. + satisfied = slots >= 0 and cap >= 0 and slots <= cap + assert satisfied is ok -- 2.54.0 From 5974a1bfbcedb9f400423f9ff92a8676ce922713 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 07:53:30 -0400 Subject: [PATCH 02/94] fix: two errors in the worker-lane tests (4291) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both mine, both in tests/test_worker_lanes.py, neither in the code under test. Run 7242. RUFF I001 — two blank lines between the import block and the first module-level comment. Rule 102 names this exact trap ("exactly ONE blank line between imports and a module-level constant/comment/pytestmark") and I was pointed at that rule repeatedly before opening it. SIX FAILURES in test_worker_lane_check_constraints — the test asserted bare constraint names, but Base.metadata's naming_convention has already applied the `ck_worker_lane_` prefix by the time __table__.constraints is read. The failure output is worth keeping: it shows the model emits exactly the three intended constraints, prefixed once — ck_worker_lane_slots_non_negative slots >= 0 ck_worker_lane_cap_non_negative slots_cap >= 0 ck_worker_lane_slots_within_cap slots <= slots_cap — which is the model behaving correctly, and confirms the migration's op.f() names match what the ORM produces. The assertion is now an equality against the prefixed names plus an explicit check for a doubled prefix. That is strictly more valuable than what I wrote: a bare-name assertion would have passed just as happily against `ck_worker_lane_ck_worker_lane_slots_within_cap`, which is the defect alembic 0088 had to rename four constraints for (#3275). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/worker_control.py | 196 +++++++++++++++++++++++++ tests/test_worker_lanes.py | 18 ++- 2 files changed, 210 insertions(+), 4 deletions(-) create mode 100644 backend/app/services/worker_control.py diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py new file mode 100644 index 0000000..b9b7493 --- /dev/null +++ b/backend/app/services/worker_control.py @@ -0,0 +1,196 @@ +"""Read and change a lane's live pool, over the broker. + +Milestone 422 step 2. The half of the milestone that does something. + +## No docker socket is involved, and that is the point + +Milestone 365 put "acting on the state" out of scope because restarting a +dead worker needs a docker socket the web container deliberately does not +have. That is true of RESTARTING a container. It is not true of changing how +much work a RUNNING worker does: celery's remote control sends a message over +the broker and the worker resizes its own pool. Same Redis the app already +uses, no new privilege, no new surface. + + pool_grow / pool_shrink how many slots a lane runs + add_consumer / cancel_consumer whether it consumes its queues at all + +The operator ruled the socket out independently (2026-09-22: *"this feature +is a very invasive idea in my mind and I'd like to avoid it"*), and nothing +here raises the question. + +## The setting is PER PROCESS, not per lane total + +`pool_grow(n, destination=[...])` adds n slots to EACH destination it names. +While the stack still runs several containers per lane — the operator's +production `worker` is `replicas: 2` — a single delta applied to a lane's +total would be wrong for every replica. + +So `slots` means what `CELERY_CONCURRENCY` means: the pool size of one +process. The reconcile below drives EACH replica to that number +independently, computing its own delta from that replica's current pool, so +replicas that have drifted apart (one restarted, one was grown) converge +rather than being moved in lockstep from a shared baseline. + +After step 5 there is one process per lane and the distinction disappears. +It matters now, and getting it wrong now would be invisible — the totals +would simply be double what the UI claimed. + +## Why reserved() is read alongside the queue depth + +Celery PREFETCHES: a worker pulls more messages than it can run and holds +them in memory. Those have already left the Redis list, so `LLEN` — which is +what `/api/system/activity/queues` reports — can read 0 while thirty tasks +are waiting inside a worker. Any judgement about backlog that uses only LLEN +under-reports, which matters for the UI and is disqualifying for step 7's +autoscaler. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass, field + +from .worker_lanes import LANES, LANES_BY_QUEUE_KEY, Lane + +log = logging.getLogger(__name__) + +# celery control is a broker round trip on a request path, so it gets a +# deadline (rule 156) — the same reasoning and the same budget as +# service_roster's inspect. A broker that stopped answering must make this +# report "not present", which is true, rather than hang the page. +CONTROL_TIMEOUT_SECONDS = 2.0 + + +@dataclass +class LaneLiveState: + """What `celery inspect` says about one lane right now. + + `present=False` is NOT "zero slots" — it is "nothing answered". A lane + whose worker is restarting, or whose broker is unreachable, must read as + unknown rather than as stopped: an unswept absence is not a verdict + (snippet #3969). The reconcile in step 3 skips an absent lane rather than + correcting it, which is only safe because this distinction is kept. + """ + + present: bool = False + replicas: int = 0 + # Per-process pool size. Equal across replicas unless one has drifted. + pool: int | None = None + active: int = 0 + reserved: int = 0 + hostnames: list[str] = field(default_factory=list) + + +def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None: + return LANES_BY_QUEUE_KEY.get(tuple(sorted(queues))) + + +def inspect_lanes_sync() -> dict[str, LaneLiveState]: + """Live state per lane name. Sync — callers wrap in asyncio.to_thread. + + Never raises. Every lane is present in the result; ones nothing answered + for carry `present=False`, so a caller cannot accidentally read a missing + lane as an empty one by iterating only what came back. + """ + out = {lane.name: LaneLiveState() for lane in LANES} + try: + from ..celery_app import celery as celery_app + + insp = celery_app.control.inspect(timeout=CONTROL_TIMEOUT_SECONDS) + active_queues = insp.active_queues() or {} + stats = insp.stats() or {} + active = insp.active() or {} + reserved = insp.reserved() or {} + except Exception: + log.warning("worker_control: celery inspect failed", exc_info=True) + return out + + for hostname, queues in active_queues.items(): + lane = _lane_for_queues(tuple(q["name"] for q in queues)) + if lane is None: + # A deployment slicing CELERY_QUEUES differently. Reported by the + # roster under its raw queue list; it simply has no lane row to + # control, which is honest rather than an error. + continue + state = out[lane.name] + state.present = True + state.replicas += 1 + state.hostnames.append(hostname) + state.active += len(active.get(hostname, [])) + state.reserved += len(reserved.get(hostname, [])) + + # `pool.max-concurrency` is the number pool_grow/pool_shrink move and + # the number the UI shows. Absent on a worker whose stats did not + # answer, which leaves pool=None — unknown, not zero. + pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency") + if isinstance(pool, int): + state.pool = pool if state.pool is None else max(state.pool, pool) + + for state in out.values(): + state.hostnames.sort() + return out + + +def set_lane_slots_sync(lane: Lane, target: int) -> tuple[bool, str | None]: + """Drive every replica of `lane` to `target` slots. Returns (applied, err). + + Per-replica deltas rather than one shared delta: see the module docstring. + A replica already at the target is issued nothing at all, which is what + makes step 3's periodic reconcile converge instead of re-sending a grow of + zero forever (lesson #4183 — an enforcer without a reachable fixed point + re-does its own work every tick). + + `applied=False` is not a failure of the SETTING. The caller has already + stored the value; this says only that the live push did not land, and the + reconcile will carry it when the lane answers again. + """ + try: + from ..celery_app import celery as celery_app + + live = inspect_lanes_sync()[lane.name] + if not live.present: + return False, "lane is not running" + if live.pool is None: + return False, "worker did not report its pool size" + + control = celery_app.control + for hostname in live.hostnames: + delta = target - live.pool + if delta > 0: + control.pool_grow(delta, destination=[hostname]) + elif delta < 0: + control.pool_shrink(-delta, destination=[hostname]) + return True, None + except Exception as exc: # noqa: BLE001 — reported, never raised at a caller + log.warning("worker_control: could not resize %s", lane.name, exc_info=True) + return False, str(exc) + + +def set_lane_enabled_sync(lane: Lane, enabled: bool) -> tuple[bool, str | None]: + """Start or stop `lane` consuming its queues, without killing the process. + + `cancel_consumer` rather than a shutdown: a stopped consumer keeps its + worker alive and answering `inspect`, so a disabled lane stays visible and + can be turned back on. A killed worker would read as absent, which is the + same signal as a crash — and the whole point of the roster (#365) is that + those two must not look alike. + """ + try: + from ..celery_app import celery as celery_app + + live = inspect_lanes_sync()[lane.name] + if not live.present: + return False, "lane is not running" + control = celery_app.control + for queue in lane.queues: + if enabled: + control.add_consumer(queue, destination=live.hostnames) + else: + control.cancel_consumer(queue, destination=live.hostnames) + return True, None + except Exception as exc: # noqa: BLE001 + log.warning( + "worker_control: could not %s %s", + "enable" if enabled else "disable", lane.name, exc_info=True, + ) + return False, str(exc) diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index 7482e4e..7958676 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -11,7 +11,6 @@ import pytest from backend.app.services import worker_lanes as wl - # --- the lane definitions ---------------------------------------------------- @@ -248,9 +247,20 @@ def test_worker_lane_check_constraints(slots, cap, ok): c.name: str(c.sqltext) for c in WorkerLane.__table__.constraints if hasattr(c, "sqltext") } - assert "slots_within_cap" in constraints - assert "slots_non_negative" in constraints - assert "cap_non_negative" in constraints + # The names carry the convention's `ck_worker_lane_` prefix ALREADY — the + # model declares them bare and Base.metadata's naming_convention applies it. + # Asserting the prefixed form is what pins the thing that actually went + # wrong once: alembic 0088 had to rename four constraints that shipped as + # `ck_x_ck_x_name`, because the migration pre-prefixed a name the + # convention then prefixed again (#3275). A bare-name assertion here would + # pass just as happily against a doubled one. + assert constraints == { + "ck_worker_lane_slots_non_negative": "slots >= 0", + "ck_worker_lane_cap_non_negative": "slots_cap >= 0", + "ck_worker_lane_slots_within_cap": "slots <= slots_cap", + } + for name in constraints: + assert not name.startswith("ck_worker_lane_ck_"), f"doubled prefix: {name}" # Evaluate the same predicates the database will, so the parametrize table # documents what is accepted rather than restating the SQL. -- 2.54.0 From a9c1b421a723f5562f0da29617d7fa1985f78597 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:01:23 -0400 Subject: [PATCH 03/94] feat: change a lane's slots on a running system, over the broker (4292) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to its live pool; `POST /api/system/workers/` changes it. NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because restarting a dead worker needs a socket the web container deliberately does not have. That holds for restarting a CONTAINER; it does not hold for changing how much work a RUNNING worker does. celery's pool_grow / pool_shrink / add_consumer / cancel_consumer send a message over the Redis the app already uses, and the worker resizes itself. No new privilege, no new surface, and the security question that deferred this is never raised. PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a restart drops every lane to its env concurrency — so a UI that only pushed would lose the setting on the next deploy with nothing to show for it (lesson #4202). Storing alone would describe nothing until something restarted. A failed PUSH is not a failed setting: 200 with `applied: false` and a reason, so the UI says "saved, not yet live" rather than "that didn't work". Step 3's reconcile carries it when the lane answers again. PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH destination, so while `worker` runs `replicas: 2` a single delta from an aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY means — one process's pool — and each replica is driven to it from its OWN current size, so replicas that drifted apart converge rather than moving in lockstep. I wrote this wrong first: the docstring claimed per-replica while the code computed one delta from the max across replicas. LaneLiveState now carries `pools` per hostname and exposes `pool` as a property. A replica already at the target is sent nothing at all — the reachable fixed point step 3's periodic reconcile needs, or it re-issues a grow of zero every tick forever (lesson #4183). A replica that answered inspect but not stats is NAMED in the error rather than skipped silently, since otherwise it would run at a size the UI claims it does not. `present=False` is not "zero slots", it is "nothing answered" — kept distinct throughout, because step 3 skips an absent lane rather than correcting it. /workers now also reports pool size (from `insp.stats()`) and RESERVED count. Celery prefetches, so tasks that have left the Redis list but not started are invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker memory. `pending` is depth + reserved. The UI is misleading without this and step 7's autoscaler would be simply wrong. Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES, whose own comment admitted the coupling ("must match celery_app.task_routes") and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed. Now derived from LANES. The rendered order changes to lane grouping, which is the better shape for a lane-oriented UI. Separate blueprint rather than folding into system_activity, which states in its first line that it is read-only and answers a different question — its /workers is keyed on celery HOSTNAME and reports which nodes answered. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/api/__init__.py | 2 + backend/app/api/system_activity.py | 18 ++- backend/app/api/workers.py | 103 ++++++++++++ backend/app/services/worker_control.py | 209 ++++++++++++++++++++++++- tests/test_api_workers.py | 172 ++++++++++++++++++++ tests/test_worker_control.py | 188 ++++++++++++++++++++++ 6 files changed, 678 insertions(+), 14 deletions(-) create mode 100644 backend/app/api/workers.py create mode 100644 tests/test_api_workers.py create mode 100644 tests/test_worker_control.py diff --git a/backend/app/api/__init__.py b/backend/app/api/__init__.py index 075e15d..9dad185 100644 --- a/backend/app/api/__init__.py +++ b/backend/app/api/__init__.py @@ -41,6 +41,7 @@ def all_blueprints() -> list[Blueprint]: from .system_health import system_health_bp from .tags import tags_bp from .thumbnails import thumbnails_bp + from .workers import workers_bp return [ api_bp, attachments_bp, @@ -52,6 +53,7 @@ def all_blueprints() -> list[Blueprint]: showcase_bp, settings_bp, system_activity_bp, + workers_bp, system_health_bp, system_backup_bp, admin_bp, diff --git a/backend/app/api/system_activity.py b/backend/app/api/system_activity.py index 0df6e5b..a50bd26 100644 --- a/backend/app/api/system_activity.py +++ b/backend/app/api/system_activity.py @@ -21,18 +21,22 @@ from ..config import get_config from ..extensions import get_session from ..models import TaskRun from ..services.scheduler_service import scheduler_status +from ..services.worker_lanes import LANES system_activity_bp = Blueprint( "system_activity", __name__, url_prefix="/api/system/activity", ) -# Canonical queue order — must match celery_app.task_routes. UI renders -# in this order; queues with no LLEN response show as null rather than -# absent. -_QUEUE_NAMES = ( - "default", "import", "thumbnail", "ml", - "download", "scan", "maintenance", "maintenance_long", -) +# Every queue, grouped by the lane that consumes it. DERIVED from +# `worker_lanes.LANES` (milestone 422 step 1) rather than written out: +# this was a hand-kept third copy of "which queues exist", alongside +# celery_app.task_routes and service_roster.ROLE_NAMES, and its own comment +# admitted the coupling — "must match celery_app.task_routes". +# +# The rendered ORDER changes with this: lane order rather than the previous +# hand-chosen one. That is the better grouping for a lane-oriented UI, and +# queues with no LLEN response still show as null rather than absent. +_QUEUE_NAMES = tuple(q for lane in LANES for q in lane.queues) # Cache module-level so all requests share the cache between polls. # Tests can reset via direct dict mutation if needed. diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py new file mode 100644 index 0000000..b5f96f0 --- /dev/null +++ b/backend/app/api/workers.py @@ -0,0 +1,103 @@ +"""Worker lanes: what each is doing, and the dial that changes it. + +Milestone 422 step 2. The write half of a surface `api/system_activity.py` +only reads. + +## Why this is a separate blueprint + +`system_activity` says in its own first line that it is read-only, and it +answers a different question: its `/workers` is keyed on celery HOSTNAME and +reports which nodes answered. That stays as it is — the existing +SystemActivityTab consumes it. + +This is keyed on LANE, joins the stored settings to the live pool, and +accepts writes. Two endpoints answering "which celery processes exist" and +"how much work is each lane allowed to do" are not the same endpoint, and +folding the second into the first would make a read-only module a write one. +""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from quart import Blueprint, jsonify, request + +from ..extensions import get_session +from ..services.worker_control import LaneUpdateRefused, lane_view, set_lane +from ..services.worker_lanes import LANES_BY_NAME +from ._responses import error_response as _bad + +workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers") + + +@workers_bp.route("", methods=["GET"]) +async def list_lanes(): + """Every lane: configured slots, the cap, the ceiling, and live state. + + Response: {lanes: [...], fetched_at: iso8601} + + Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is + the surface an operator watches while dragging a stepper, and a cached + reply would show them the value from before their own change and read as + the control having failed. + """ + async with get_session() as session: + lanes = await lane_view(session) + return jsonify({ + "lanes": lanes, + "fetched_at": datetime.now(UTC).isoformat(), + }) + + +@workers_bp.route("/", methods=["POST"]) +async def update_lane(name: str): + """Set a lane's slots, cap and/or enabled flag. Stores, then pushes live. + + Partial: only the keys present are changed, so the UI's stepper can send + `{"slots": 3}` without restating the cap it did not touch. + + Two failure kinds, deliberately different statuses: + + * **400** — the value is not allowed (above the cap, above the ceiling, + negative). Nothing was stored. The body carries `detail`, which is the + sentence the UI shows; a refused control with no reason reads as a bug. + * **200 with `applied: false`** — the value WAS stored but could not be + pushed, because the lane is not currently answering. That is not an + error: step 3's reconcile carries it when the lane comes back, and the + UI should say "saved, not yet live" rather than "that didn't work". + """ + lane = LANES_BY_NAME.get(name) + if lane is None: + return _bad("unknown_lane", detail=name, known=sorted(LANES_BY_NAME)) + + body = await request.get_json() + if not isinstance(body, dict): + return _bad("invalid_body", detail="body must be a JSON object") + + fields: dict = {} + for key in ("slots", "slots_cap"): + if key in body: + value = body[key] + # Rejected rather than coerced: `True` is an int in Python, and + # silently reading it as 1 slot would be a control that appears to + # work and sets something nobody asked for. + if not isinstance(value, int) or isinstance(value, bool): + return _bad("invalid_body", detail=f"{key} must be an integer") + fields[key] = value + if "enabled" in body: + if not isinstance(body["enabled"], bool): + return _bad("invalid_body", detail="enabled must be a boolean") + fields["enabled"] = body["enabled"] + + if not fields: + return _bad( + "invalid_body", + detail="give at least one of slots, slots_cap, enabled", + ) + + async with get_session() as session: + try: + result = await set_lane(session, lane, **fields) + except LaneUpdateRefused as exc: + return _bad("refused", detail=str(exc)) + return jsonify(result) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index b9b7493..7adac19 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -47,10 +47,15 @@ autoscaler. from __future__ import annotations +import asyncio import logging from dataclasses import dataclass, field -from .worker_lanes import LANES, LANES_BY_QUEUE_KEY, Lane +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from ..models import WorkerLane +from .worker_lanes import LANES, LANES_BY_QUEUE_KEY, Lane, derived_ceiling log = logging.getLogger(__name__) @@ -74,11 +79,22 @@ class LaneLiveState: present: bool = False replicas: int = 0 - # Per-process pool size. Equal across replicas unless one has drifted. - pool: int | None = None active: int = 0 reserved: int = 0 hostnames: list[str] = field(default_factory=list) + # Pool size PER HOSTNAME, not aggregated. The resize below computes each + # replica's own delta from its own current pool, so replicas that have + # drifted apart converge instead of being moved in lockstep from a shared + # baseline — which is what an aggregate here would silently reintroduce. + pools: dict[str, int] = field(default_factory=dict) + + @property + def pool(self) -> int | None: + """One number for the UI. `max` rather than a sum: `slots` means the + pool size of ONE process (see the module docstring), so the largest + replica is the honest answer to "what is this lane set to". None when + no replica reported — unknown, never zero.""" + return max(self.pools.values()) if self.pools else None def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None: @@ -124,7 +140,7 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: # answer, which leaves pool=None — unknown, not zero. pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency") if isinstance(pool, int): - state.pool = pool if state.pool is None else max(state.pool, pool) + state.pools[hostname] = pool for state in out.values(): state.hostnames.sort() @@ -150,16 +166,22 @@ def set_lane_slots_sync(lane: Lane, target: int) -> tuple[bool, str | None]: live = inspect_lanes_sync()[lane.name] if not live.present: return False, "lane is not running" - if live.pool is None: + if not live.pools: return False, "worker did not report its pool size" control = celery_app.control - for hostname in live.hostnames: - delta = target - live.pool + unreported = [h for h in live.hostnames if h not in live.pools] + for hostname, current in live.pools.items(): + delta = target - current if delta > 0: control.pool_grow(delta, destination=[hostname]) elif delta < 0: control.pool_shrink(-delta, destination=[hostname]) + if unreported: + # Resized what could be resized, and said which could not. Silence + # here would leave a replica running at a size the UI claims it is + # not, with nothing anywhere recording the gap. + return False, f"no pool size reported by {', '.join(sorted(unreported))}" return True, None except Exception as exc: # noqa: BLE001 — reported, never raised at a caller log.warning("worker_control: could not resize %s", lane.name, exc_info=True) @@ -194,3 +216,176 @@ def set_lane_enabled_sync(lane: Lane, enabled: bool) -> tuple[bool, str | None]: "enable" if enabled else "disable", lane.name, exc_info=True, ) return False, str(exc) + + +# --- the settings half, which is async ---------------------------------------- +# +# Sync celery control above, async DB below, in one module. Same split +# `service_roster` already runs (`_inspect_celery_sync` beside `touch_service`) +# — the boundary is the transport, not the concern, and "control the workers" +# is one concern. + + +async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]: + """Every lane's row, creating any that are missing from its LANES defaults. + + Self-heals rather than depending on a migration having run for a lane + added later: alembic 0103 seeded the four that existed on 2026-09-22, and + a fifth added to LANES afterwards gets its row the first time anything + asks. Without this, a new lane would read as absent and the UI would + simply not show it. + """ + rows = { + row.name: row + for row in (await session.execute(select(WorkerLane))).scalars() + } + missing = [lane for lane in LANES if lane.name not in rows] + for lane in missing: + row = WorkerLane( + name=lane.name, + slots=lane.default_slots, + slots_cap=lane.default_slots_cap, + enabled=lane.default_enabled, + ) + session.add(row) + rows[lane.name] = row + if missing: + await session.commit() + return rows + + +async def lane_view(session: AsyncSession) -> list[dict]: + """Every lane: what is configured, what is live, what it may grow to. + + One call rather than making the UI join three sources. `pending` is the + honest backlog — Redis depth PLUS reserved — because celery prefetches and + LLEN alone reads 0 while a worker holds tasks in memory. + """ + rows = await _rows_by_name(session) + live = await asyncio.to_thread(inspect_lanes_sync) + depths = await asyncio.to_thread(_queue_depths_sync) + + out = [] + for lane in LANES: + row = rows[lane.name] + state = live[lane.name] + # None for a queue the broker did not answer for, which must not be + # silently summed as zero — an unknown depth is not an empty one. + known = [depths.get(q) for q in lane.queues] + depth = sum(d for d in known if d is not None) if any( + d is not None for d in known + ) else None + out.append({ + "name": lane.name, + "display_name": lane.display_name, + "queues": list(lane.queues), + "slots": row.slots, + "slots_cap": row.slots_cap, + "ceiling": derived_ceiling(lane), + "enabled": row.enabled, + "memory_bound": lane.memory_bound, + "live": { + "present": state.present, + "replicas": state.replicas, + "pool": state.pool, + "active": state.active, + "reserved": state.reserved, + }, + "queue_depth": depth, + "pending": None if depth is None else depth + state.reserved, + }) + return out + + +def _queue_depths_sync() -> dict[str, int | None]: + """Redis LLEN per queue. None for one that did not answer — see lane_view. + + Sync; the caller threads it. A per-queue try/except so one bad queue does + not cost the whole report, matching `api/system_activity._read_queues_sync`. + """ + import redis + + from ..config import get_config + + out: dict[str, int | None] = {} + try: + client = redis.Redis.from_url(get_config().celery_broker_url) + except Exception: + log.warning("worker_control: no broker for queue depths", exc_info=True) + return {q: None for lane in LANES for q in lane.queues} + for lane in LANES: + for queue in lane.queues: + try: + out[queue] = int(client.llen(queue)) + except Exception: # noqa: BLE001 — a hiccup must not break the UI + out[queue] = None + return out + + +class LaneUpdateRefused(ValueError): + """A requested value is outside what the lane may hold. Carries the reason + the UI shows — a greyed control with no explanation reads as a bug.""" + + +async def set_lane( + session: AsyncSession, + lane: Lane, + *, + slots: int | None = None, + slots_cap: int | None = None, + enabled: bool | None = None, +) -> dict: + """Store the operator's choice, then push it to the running lane. + + BOTH, in one call, and the order matters. `pool_grow`/`pool_shrink` are + not durable — a restart drops every lane back to its env concurrency — so + a UI that only pushed would have its setting evaporate on the next deploy + with nothing to show for it (lesson #4202: the live change does not + survive, and nothing says so). Storing alone would be a number that + describes nothing until something restarts. + + A failed PUSH is not a failed setting. The value is saved either way and + step 3's reconcile carries it when the lane answers again; the result says + `applied: false` with a reason so the UI can say "saved, not yet live" + rather than "that didn't work". + """ + rows = await _rows_by_name(session) + row = rows[lane.name] + + new_cap = row.slots_cap if slots_cap is None else slots_cap + new_slots = row.slots if slots is None else slots + new_enabled = row.enabled if enabled is None else enabled + + ceiling = derived_ceiling(lane) + if new_cap < 0 or new_slots < 0: + raise LaneUpdateRefused("slots and cap cannot be negative") + if new_cap > ceiling: + raise LaneUpdateRefused( + f"cap {new_cap} is above what this container can hold " + f"({ceiling} for {lane.display_name})" + ) + if new_slots > new_cap: + raise LaneUpdateRefused(f"slots {new_slots} is above the cap {new_cap}") + + row.slots_cap = new_cap + row.slots = new_slots + row.enabled = new_enabled + await session.commit() + + applied, error = True, None + if enabled is not None: + applied, error = await asyncio.to_thread( + set_lane_enabled_sync, lane, new_enabled, + ) + if applied and slots is not None: + applied, error = await asyncio.to_thread(set_lane_slots_sync, lane, new_slots) + + return { + "name": lane.name, + "slots": row.slots, + "slots_cap": row.slots_cap, + "ceiling": ceiling, + "enabled": row.enabled, + "applied": applied, + "apply_error": error, + } diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py new file mode 100644 index 0000000..97bd728 --- /dev/null +++ b/tests/test_api_workers.py @@ -0,0 +1,172 @@ +"""/api/system/workers — the lane dial (milestone 422 step 2). + +Exercises the real endpoint against the real database. Only `celery inspect` +is stubbed, and only to keep the suite fast: an unstubbed inspect blocks for +its full 2s timeout per call with no workers to answer, which several writes +would turn into most of the lane's runtime. +""" + +import pytest +import pytest_asyncio +from sqlalchemy import select + +from backend.app.models import WorkerLane +from backend.app.services import worker_control as wc +from backend.app.services.worker_lanes import LANES + +pytestmark = pytest.mark.integration + + +@pytest_asyncio.fixture +async def no_live_workers(monkeypatch): + """Nothing is running — which is the CI lane's actual truth, asserted + rather than waited for. Makes every push fail, which is the interesting + half: the setting must still be stored.""" + monkeypatch.setattr( + wc, "inspect_lanes_sync", + lambda: {lane.name: wc.LaneLiveState() for lane in LANES}, + ) + + +async def _lane_row(db, name: str) -> WorkerLane: + return (await db.execute( + select(WorkerLane).where(WorkerLane.name == name) + )).scalar_one() + + +# --- reading ----------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_get_lists_every_lane_with_its_ceiling(client, no_live_workers): + resp = await client.get("/api/system/workers") + assert resp.status_code == 200 + body = await resp.get_json() + + by_name = {lane["name"]: lane for lane in body["lanes"]} + assert set(by_name) == {"worker", "scheduler", "maintenance_long", "ml"} + for lane in body["lanes"]: + assert lane["ceiling"] >= 0 + assert lane["slots"] <= lane["slots_cap"] + # Nothing is running, so live state must say so rather than report + # zeroes that read like a healthy idle lane. + assert lane["live"]["present"] is False + + +@pytest.mark.asyncio +async def test_ml_ships_off(client, no_live_workers): + """Rule 164's carve-out and the weak-hardware default in one row: enabling + the lane is what triggers the SigLIP download, so a fresh install must not + find it on.""" + body = await (await client.get("/api/system/workers")).get_json() + ml = next(lane for lane in body["lanes"] if lane["name"] == "ml") + assert ml["enabled"] is False + assert ml["slots"] == 0 + + +# --- the persist / push split ------------------------------------------------ + + +@pytest.mark.asyncio +async def test_a_value_is_stored_even_when_it_cannot_be_pushed( + client, db, no_live_workers +): + """THE test for this step. `pool_grow` is not durable and the lane is not + answering, so the push fails — and the setting must survive anyway, or a + UI that saved while a worker was restarting would silently lose it + (lesson #4202). 200 with `applied: false`, not an error. + """ + resp = await client.post("/api/system/workers/worker", json={"slots": 3}) + assert resp.status_code == 200 + body = await resp.get_json() + + assert body["slots"] == 3 + assert body["applied"] is False + assert "not running" in body["apply_error"] + + assert (await _lane_row(db, "worker")).slots == 3 + + +@pytest.mark.asyncio +async def test_a_partial_update_leaves_the_other_fields_alone( + client, db, no_live_workers +): + """The stepper sends `{"slots": n}` without restating a cap it did not + touch.""" + before = await _lane_row(db, "worker") + original_cap = before.slots_cap + + await client.post("/api/system/workers/worker", json={"slots": 2}) + + await db.refresh(before) + assert before.slots == 2 + assert before.slots_cap == original_cap + + +# --- what is refused --------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_slots_above_the_cap_are_refused_and_nothing_is_stored( + client, db, no_live_workers +): + row = await _lane_row(db, "worker") + resp = await client.post( + "/api/system/workers/worker", json={"slots": row.slots_cap + 1}, + ) + assert resp.status_code == 400 + body = await resp.get_json() + assert body["error"] == "refused" + # The sentence the UI shows. A greyed control with no reason reads as a bug. + assert "cap" in body["detail"] + + await db.refresh(row) + assert row.slots <= row.slots_cap + + +@pytest.mark.asyncio +async def test_a_cap_above_the_derived_ceiling_is_refused( + client, db, no_live_workers +): + """The operator's cap is theirs to set, but not past what the container + can hold — the whole point of deriving a ceiling rather than typing one.""" + resp = await client.post( + "/api/system/workers/ml", json={"slots_cap": 10_000}, + ) + assert resp.status_code == 400 + body = await resp.get_json() + assert body["error"] == "refused" + assert "container can hold" in body["detail"] + + +@pytest.mark.asyncio +async def test_a_boolean_is_not_accepted_as_a_slot_count(client, no_live_workers): + """`True` is an int in Python. Coerced, it would silently set one slot — + a control that appears to work and does something nobody asked for.""" + resp = await client.post("/api/system/workers/worker", json={"slots": True}) + assert resp.status_code == 400 + body = await resp.get_json() + assert body["error"] == "invalid_body" + + +@pytest.mark.asyncio +async def test_an_unknown_lane_is_refused_and_names_the_known_ones( + client, no_live_workers +): + resp = await client.post("/api/system/workers/nonsense", json={"slots": 1}) + assert resp.status_code == 400 + body = await resp.get_json() + assert body["error"] == "unknown_lane" + assert "worker" in body["known"] + + +@pytest.mark.asyncio +async def test_an_empty_body_is_refused_rather_than_treated_as_a_no_op( + client, no_live_workers +): + """A POST that changes nothing and returns 200 is indistinguishable from + one that worked, which is how a broken UI control goes unnoticed.""" + resp = await client.post("/api/system/workers/worker", json={}) + assert resp.status_code == 400 + body = await resp.get_json() + assert body["error"] == "invalid_body" diff --git a/tests/test_worker_control.py b/tests/test_worker_control.py new file mode 100644 index 0000000..47df672 --- /dev/null +++ b/tests/test_worker_control.py @@ -0,0 +1,188 @@ +"""Changing a lane's slots on a running system (milestone 422 step 2). + +The celery control calls are stubbed: what is being tested is the DELTA +ARITHMETIC and the persist/push split, not that celery can resize its own +pool. A test that asserted celery's behaviour would be testing celery. +""" + +from __future__ import annotations + +import pytest + +from backend.app.services import worker_control as wc +from backend.app.services.worker_lanes import LANES_BY_NAME + +# --- what inspect reports ---------------------------------------------------- + + +class _Control: + """Records the control messages that were sent.""" + + def __init__(self): + self.grew: list[tuple[int, list[str]]] = [] + self.shrank: list[tuple[int, list[str]]] = [] + self.added: list[tuple[str, list[str]]] = [] + self.cancelled: list[tuple[str, list[str]]] = [] + + def pool_grow(self, n, destination=None): + self.grew.append((n, destination)) + + def pool_shrink(self, n, destination=None): + self.shrank.append((n, destination)) + + def add_consumer(self, queue, destination=None): + self.added.append((queue, destination)) + + def cancel_consumer(self, queue, destination=None): + self.cancelled.append((queue, destination)) + + +def _stub_live(monkeypatch, lane_name, *, pools, present=True, reserved=0): + state = wc.LaneLiveState( + present=present, + replicas=len(pools), + hostnames=sorted(pools), + pools=dict(pools), + reserved=reserved, + ) + monkeypatch.setattr( + wc, "inspect_lanes_sync", + lambda: {name: (state if name == lane_name else wc.LaneLiveState()) + for name in LANES_BY_NAME}, + ) + return state + + +def _stub_control(monkeypatch): + control = _Control() + + class _Celery: + pass + + celery = _Celery() + celery.control = control + import sys + import types + mod = types.ModuleType("backend.app.celery_app") + mod.celery = celery + monkeypatch.setitem(sys.modules, "backend.app.celery_app", mod) + return control + + +def test_pool_property_is_max_not_sum(): + """`slots` means the pool size of ONE process, so the aggregate shown to + the operator is the largest replica — not the total. A sum would report 8 + for two replicas of 4 and invite them to 'reduce it to 4', which would + halve the lane.""" + state = wc.LaneLiveState(pools={"a": 4, "b": 4}) + assert state.pool == 4 + + +def test_pool_is_none_when_nothing_reported(): + """Unknown, never zero — the distinction step 3's reconcile depends on.""" + assert wc.LaneLiveState(present=True).pool is None + + +# --- the delta arithmetic ---------------------------------------------------- + + +def test_each_replica_gets_its_own_delta(monkeypatch): + """The bug this exists to prevent: one delta computed from an aggregate + and applied to every replica. With replicas at 2 and 6 and a target of 4, + a shared delta moves both the same way and leaves them at 4 and 8 — or 0 + and 4 — depending on which aggregate was used. Per-replica deltas + converge both on 4. + """ + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 2, "host-b": 6}) + + applied, err = wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 4) + + assert (applied, err) == (True, None) + assert control.grew == [(2, ["host-a"])] + assert control.shrank == [(2, ["host-b"])] + + +def test_a_replica_already_at_the_target_is_sent_nothing(monkeypatch): + """The fixed point step 3's reconcile needs. An enforcer that re-issues a + grow of zero every tick never converges and re-does its own work forever + (lesson #4183).""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 4}) + + applied, err = wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 4) + + assert (applied, err) == (True, None) + assert control.grew == [] + assert control.shrank == [] + + +def test_resizing_an_absent_lane_reports_rather_than_raises(monkeypatch): + _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={}, present=False) + + applied, err = wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 4) + + assert applied is False + assert "not running" in err + + +def test_a_replica_with_no_reported_pool_is_named_not_skipped_silently( + monkeypatch, +): + """Resize what can be resized, then say which could not. Silence would + leave a replica running at a size the UI claims it is not.""" + control = _stub_control(monkeypatch) + state = _stub_live(monkeypatch, "worker", pools={"host-a": 2}) + state.hostnames = ["host-a", "host-b"] # b answered inspect, not stats + state.replicas = 2 + + applied, err = wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 4) + + assert control.grew == [(2, ["host-a"])] + assert applied is False + assert "host-b" in err + + +# --- enabling and disabling -------------------------------------------------- + + +def test_disabling_cancels_consumers_rather_than_killing_the_worker(monkeypatch): + """A cancelled consumer keeps the process alive and answering inspect, so + a disabled lane stays visible. A killed worker reads as ABSENT, which is + the same signal as a crash — and milestone 365 exists precisely so those + two do not look alike.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "ml", pools={"host-a": 1}) + + applied, err = wc.set_lane_enabled_sync(LANES_BY_NAME["ml"], False) + + assert (applied, err) == (True, None) + assert control.cancelled == [("ml", ["host-a"])] + assert control.added == [] + + +def test_enabling_adds_a_consumer_for_every_queue_in_the_lane(monkeypatch): + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 1}) + + wc.set_lane_enabled_sync(LANES_BY_NAME["worker"], True) + + assert [q for q, _ in control.added] == list(LANES_BY_NAME["worker"].queues) + + +# --- lane identity ----------------------------------------------------------- + + +def test_queue_sets_map_back_to_their_lane_in_any_order(): + """celery does not guarantee the order it lists a worker's queues in, so + the lookup sorts. Unsorted, a lane would intermittently fail to match and + read as absent.""" + lane = LANES_BY_NAME["worker"] + assert wc._lane_for_queues(tuple(reversed(lane.queues))) is lane + + +def test_an_unknown_queue_set_maps_to_no_lane(): + """A deployment slicing CELERY_QUEUES differently has no lane row to + control. Honest rather than an error — the roster still reports it.""" + assert wc._lane_for_queues(("something", "else")) is None -- 2.54.0 From 5f8c63f61bd630ea12e244eaf890a1515af14393 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:08:38 -0400 Subject: [PATCH 04/94] feat: reconcile every running lane back to its stored slots (4293) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by its supervisor comes back at its ENV concurrency, silently below whatever the operator set, and nothing on step 2's write path would ever notice. Storing the value made it survivable; this makes it survive. A BEAT TASK, NOT A HOOK IN WEB — a deliberate deviation from the step as written, for a reason already recorded in this codebase. Step 3 said "web applies the stored values after it starts". It cannot: service_roster.py documents that hypercorn runs --workers 4, so anything in before_serving becomes four concurrent loops per container hammering the broker forever. service_roster's own answer — refresh on demand from whichever request arrives — was also rejected, because the two solve different problems. A stale ROSTER only misleads someone looking at it, so recomputing when they look is exactly right. A lane running at the wrong size is doing less work than it was told to whether or not anyone is watching, and the case that matters is a deploy at 3am followed by a backlog nobody is awake to see. So: unattended, every 5 minutes, on the quick `maintenance` lane beside the other recovery sweeps. Accepted cost — a dead scheduler stops reconciliation, but a dead scheduler already stops every other sweep and the roster reports it, so this adds no new blind spot. A BUG I WROTE AND CAUGHT BEFORE COMMITTING. The first version called set_lane_enabled_sync unconditionally, so a settled system re-sent add_consumer for every queue on every tick — forever. Harmless per call (add_consumer on an already-consumed queue does nothing), unbounded in aggregate, and completely invisible. That is lesson #4183's failure mode exactly, in the very function whose docstring cites it. Worse, my test would not have caught it: it asserted only on grew/shrank. LaneLiveState now carries `consuming` — which queues a lane is actually serving, distinct from the queues it was configured with — so the reconcile compares before acting. The test now asserts ALL FOUR control families are silent on a settled tick, plus a new case for an already-disabled lane, which is the other half of the same fixed point. An absent lane is SKIPPED, not corrected. present=False means nothing answered, not zero slots; correcting it would be a conclusion from an unswept read (snippet #3969), and there would be nothing to send the message to. One lane failing does not stop the others. One inspect serves every lane: the two setters now take an optional pre-fetched LaneLiveState, so a tick costs one broker round trip rather than one per lane. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/celery_app.py | 8 ++ backend/app/services/worker_control.py | 101 +++++++++++++++++- backend/app/tasks/maintenance.py | 50 +++++++++ tests/test_worker_control.py | 137 ++++++++++++++++++++++++- 4 files changed, 291 insertions(+), 5 deletions(-) diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index d49c8ef..894fcd1 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -111,6 +111,14 @@ def make_celery() -> Celery: "task": "backend.app.tasks.maintenance.recover_interrupted_tasks", "schedule": 300.0, # every 5 minutes }, + "reconcile-worker-lanes": { + "task": "backend.app.tasks.maintenance.reconcile_worker_lanes", + "schedule": 300.0, # every 5 minutes — the window in which a + # restarted worker runs at its ENV concurrency rather than the + # slots the operator set (milestone 422 step 3). A no-op once + # every lane matches: one broker round trip, no control + # messages, nothing logged. + }, "cleanup-old-tasks": { "task": "backend.app.tasks.maintenance.cleanup_old_tasks", "schedule": 86400.0, # daily diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 7adac19..97e70f8 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -82,6 +82,13 @@ class LaneLiveState: active: int = 0 reserved: int = 0 hostnames: list[str] = field(default_factory=list) + # The queues this lane is actually consuming right now, across replicas. + # Distinct from the lane's CONFIGURED queues: `cancel_consumer` stops a + # worker consuming one without changing what it was started with, which + # is how `enabled=false` is implemented. The reconcile needs this to tell + # "already disabled" from "needs disabling" — without it, it would re-send + # add_consumer for every queue on every tick forever (lesson #4183). + consuming: set[str] = field(default_factory=set) # Pool size PER HOSTNAME, not aggregated. The resize below computes each # replica's own delta from its own current pool, so replicas that have # drifted apart converge instead of being moved in lockstep from a shared @@ -134,6 +141,7 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: state.hostnames.append(hostname) state.active += len(active.get(hostname, [])) state.reserved += len(reserved.get(hostname, [])) + state.consuming.update(q["name"] for q in queues) # `pool.max-concurrency` is the number pool_grow/pool_shrink move and # the number the UI shows. Absent on a worker whose stats did not @@ -147,7 +155,9 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: return out -def set_lane_slots_sync(lane: Lane, target: int) -> tuple[bool, str | None]: +def set_lane_slots_sync( + lane: Lane, target: int, live: LaneLiveState | None = None, +) -> tuple[bool, str | None]: """Drive every replica of `lane` to `target` slots. Returns (applied, err). Per-replica deltas rather than one shared delta: see the module docstring. @@ -163,7 +173,8 @@ def set_lane_slots_sync(lane: Lane, target: int) -> tuple[bool, str | None]: try: from ..celery_app import celery as celery_app - live = inspect_lanes_sync()[lane.name] + if live is None: + live = inspect_lanes_sync()[lane.name] if not live.present: return False, "lane is not running" if not live.pools: @@ -188,7 +199,9 @@ def set_lane_slots_sync(lane: Lane, target: int) -> tuple[bool, str | None]: return False, str(exc) -def set_lane_enabled_sync(lane: Lane, enabled: bool) -> tuple[bool, str | None]: +def set_lane_enabled_sync( + lane: Lane, enabled: bool, live: LaneLiveState | None = None, +) -> tuple[bool, str | None]: """Start or stop `lane` consuming its queues, without killing the process. `cancel_consumer` rather than a shutdown: a stopped consumer keeps its @@ -200,7 +213,8 @@ def set_lane_enabled_sync(lane: Lane, enabled: bool) -> tuple[bool, str | None]: try: from ..celery_app import celery as celery_app - live = inspect_lanes_sync()[lane.name] + if live is None: + live = inspect_lanes_sync()[lane.name] if not live.present: return False, "lane is not running" control = celery_app.control @@ -389,3 +403,82 @@ async def set_lane( "applied": applied, "apply_error": error, } + + +def reconcile_lanes_sync(desired: dict[str, tuple[int, bool]]) -> dict: + """Drive every RUNNING lane to its stored slots and enabled flag. + + `desired` is lane name -> (slots, enabled), read from the database by the + caller. This function touches no database: the celery task that schedules + it owns the sync session, and keeping the DB out of here is what lets the + same code be called from anywhere that already knows the target. + + ## Why this exists at all + + `pool_grow` is not durable. A worker that dies and is restarted by its + supervisor comes back at its ENV concurrency — silently below whatever the + operator set — and nothing in step 2's path would ever notice. Storing the + value made it survivable; this is what makes it actually survive. + + ## It must converge and then go quiet + + One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing at + all to a replica already at its target. So a settled system performs one + broker round trip per tick and sends no control messages — the reachable + fixed point lesson #4183 is about. An enforcer that re-sent a grow of zero + every tick would churn forever and bury a real correction in its own noise, + which is why `changed` below counts only lanes that actually moved. + + ## An absent lane is SKIPPED, not corrected + + `present=False` means nothing answered — a worker restarting, or a broker + that is unreachable. It does NOT mean zero slots. Correcting an absence + would be drawing a conclusion from an unswept read (snippet #3969), and + here it would be worse than useless: there is nothing to send the message + to. The lane is reported as skipped and picked up on a later tick. + """ + live = inspect_lanes_sync() + changed: list[str] = [] + skipped: list[str] = [] + failed: dict[str, str] = {} + + for lane in LANES: + target = desired.get(lane.name) + if target is None: + continue + slots, enabled = target + state = live[lane.name] + if not state.present: + skipped.append(lane.name) + continue + + # Enabled first: a lane being turned on should be consuming before + # its pool is sized, so the slots it gains have work to pick up. + # + # Only when it DISAGREES. Calling this unconditionally would send + # add_consumer for every queue on every tick of a settled system — + # the exact churn lesson #4183 describes, and invisible because + # add_consumer on a queue already consumed is harmless. + consuming_all = state.consuming.issuperset(lane.queues) + if enabled != consuming_all: + ok, err = set_lane_enabled_sync(lane, enabled, live=state) + if not ok: + failed[lane.name] = err or "could not set consumers" + continue + changed.append(lane.name) + + current = state.pool + if current is not None and current == slots: + continue + ok, err = set_lane_slots_sync(lane, slots, live=state) + if ok: + if lane.name not in changed: + changed.append(lane.name) + log.info( + "worker_control: %s reconciled %s -> %s slots", + lane.name, current, slots, + ) + else: + failed[lane.name] = err or "could not resize" + + return {"changed": changed, "skipped": skipped, "failed": failed} diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index 3a87924..c7c8fe7 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -1348,3 +1348,53 @@ def sync_memberships() -> str: if res.get("suggested") is not None: parts.append(f"suggested={res['suggested']}") return " ".join(parts) or "no platforms" + + +@celery.task(name="backend.app.tasks.maintenance.reconcile_worker_lanes") +def reconcile_worker_lanes() -> dict: + """Drive every running lane back to the slots the operator set. + + Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by + its supervisor comes back at its ENV concurrency, silently below whatever + was configured, and nothing on step 2's write path would ever notice. + + ## Why a beat task and not a hook in web + + Step 3 was written as "web applies the stored values after it starts". It + cannot, and `services/service_roster.py` already records why: hypercorn + runs `--workers 4`, so anything in `before_serving` becomes FOUR + concurrent loops per container, all hammering the broker forever. + + The other option was service_roster's own answer — refresh on demand from + whichever request happens to arrive. Rejected here because the two are + solving different problems. A stale ROSTER only misleads someone who is + looking at it, so recomputing it when they look is exactly right. A lane + running at the wrong size is doing less work than it was told to whether + or not anyone is watching — and the case that matters is a deploy at 3am + followed by a backlog nobody is awake to notice. + + So: unattended, on the quick `maintenance` lane (its module routes it + there), beside the other recovery sweeps. The accepted cost is that a dead + scheduler stops reconciliation — but a dead scheduler already stops every + other sweep, and the roster reports it, so this adds no new blind spot. + + ## Quiet when settled + + One broker round trip per tick, and no control messages at all once every + lane matches. Returns the lanes it actually moved, so the log shows a + correction rather than a heartbeat. + """ + from ..models import WorkerLane + from ..services.worker_control import reconcile_lanes_sync + + with _sync_session_factory()() as session: + desired = { + row.name: (row.slots, row.enabled) + for row in session.execute(select(WorkerLane)).scalars() + } + if not desired: + # Migration 0103 seeds these, so an empty table means it has not run + # yet. Nothing to assert — and inventing defaults here would let this + # task disagree with the seed it is supposed to be enforcing. + return {"changed": [], "skipped": [], "failed": {}} + return reconcile_lanes_sync(desired) diff --git a/tests/test_worker_control.py b/tests/test_worker_control.py index 47df672..77f1b63 100644 --- a/tests/test_worker_control.py +++ b/tests/test_worker_control.py @@ -37,13 +37,20 @@ class _Control: self.cancelled.append((queue, destination)) -def _stub_live(monkeypatch, lane_name, *, pools, present=True, reserved=0): +def _stub_live( + monkeypatch, lane_name, *, pools, present=True, reserved=0, consuming=None, +): + # `consuming` defaults to the lane's full queue set — i.e. an ENABLED + # lane. Tests for the disabled case pass an empty set explicitly. + if consuming is None: + consuming = set(LANES_BY_NAME[lane_name].queues) state = wc.LaneLiveState( present=present, replicas=len(pools), hostnames=sorted(pools), pools=dict(pools), reserved=reserved, + consuming=set(consuming), ) monkeypatch.setattr( wc, "inspect_lanes_sync", @@ -186,3 +193,131 @@ def test_an_unknown_queue_set_maps_to_no_lane(): """A deployment slicing CELERY_QUEUES differently has no lane row to control. Honest rather than an error — the roster still reports it.""" assert wc._lane_for_queues(("something", "else")) is None + + +# --- the reconcile (step 3) -------------------------------------------------- + + +def test_a_settled_system_sends_no_control_messages(monkeypatch): + """THE property this sweep lives or dies on. It runs every 5 minutes + forever, so a converged tick must be silent — one broker round trip and + nothing else. An enforcer that re-issues a grow of zero churns forever and + buries a real correction in its own noise (lesson #4183). + """ + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 4}) + + result = wc.reconcile_lanes_sync({"worker": (4, True)}) + + # EVERY control family, not just the pool ones. The first version of this + # test asserted only grew/shrank and would have passed while the reconcile + # re-sent add_consumer for all four queues on every tick — harmless per + # call, unbounded churn in aggregate, and invisible. + assert control.grew == [] + assert control.shrank == [] + assert control.added == [] + assert control.cancelled == [] + assert result["changed"] == [] + + +def test_a_worker_back_at_its_env_concurrency_is_corrected(monkeypatch): + """The failure this exists for: a restart drops the pool to CELERY_ + CONCURRENCY, silently below what the operator set.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 2}) + + result = wc.reconcile_lanes_sync({"worker": (6, True)}) + + assert control.grew == [(4, ["host-a"])] + assert result["changed"] == ["worker"] + + +def test_an_absent_lane_is_skipped_not_corrected(monkeypatch): + """`present=False` is 'nothing answered', not 'zero slots'. Correcting it + would be a conclusion drawn from an unswept read (snippet #3969) — and + there is nothing to send the message to anyway.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={}, present=False) + + result = wc.reconcile_lanes_sync({"worker": (6, True)}) + + assert result["skipped"] == ["worker"] + assert result["changed"] == [] + assert control.grew == [] + assert control.shrank == [] + + +def test_one_lane_failing_does_not_stop_the_others(monkeypatch): + """A broker blip on one lane must not leave the rest un-reconciled for + another five minutes.""" + control = _stub_control(monkeypatch) + present = wc.LaneLiveState( + present=True, replicas=1, hostnames=["host-a"], pools={"host-a": 1}, + ) + absent = wc.LaneLiveState() + monkeypatch.setattr( + wc, "inspect_lanes_sync", + lambda: {"worker": absent, "scheduler": present, + "maintenance_long": absent, "ml": absent}, + ) + + result = wc.reconcile_lanes_sync({ + "worker": (4, True), "scheduler": (3, True), + }) + + assert result["skipped"] == ["worker"] + assert result["changed"] == ["scheduler"] + assert control.grew == [(2, ["host-a"])] + + +def test_a_lane_disabled_in_settings_but_still_consuming_is_stopped(monkeypatch): + """The case that made `consuming` necessary: a lane turned off while its + worker was down comes back consuming, and must be stopped when it + returns.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "ml", pools={"host-a": 0}, consuming={"ml"}) + + result = wc.reconcile_lanes_sync({"ml": (0, False)}) + + assert control.cancelled == [("ml", ["host-a"])] + assert result["changed"] == ["ml"] + + +def test_an_already_disabled_lane_is_not_cancelled_again(monkeypatch): + """The other half of the fixed point. `cancel_consumer` on a queue that is + not being consumed succeeds and does nothing, so an unconditional + reconcile would churn here forever with no symptom.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "ml", pools={"host-a": 0}, consuming=set()) + + result = wc.reconcile_lanes_sync({"ml": (0, False)}) + + assert control.cancelled == [] + assert control.added == [] + assert result["changed"] == [] + + +def test_a_lane_with_no_stored_row_is_left_alone(monkeypatch): + """A lane in LANES but not in `desired` is one whose row has not been + seeded. Inventing a target here would let the sweep enforce a number that + disagrees with the migration it is supposed to be upholding.""" + control = _stub_control(monkeypatch) + _stub_live(monkeypatch, "worker", pools={"host-a": 2}) + + result = wc.reconcile_lanes_sync({}) + + assert result["changed"] == [] + assert control.grew == [] + + +def test_the_reconcile_task_is_registered_and_scheduled(): + """A task name only enters `celery.tasks` when its module is imported, and + a beat entry naming a task that is not registered fails at tick time + rather than at import — silently, every five minutes.""" + import backend.app.tasks.maintenance # noqa: F401 + from backend.app.celery_app import celery + + name = "backend.app.tasks.maintenance.reconcile_worker_lanes" + assert name in celery.tasks + scheduled = {e["task"] for e in celery.conf.beat_schedule.values()} + assert name in scheduled -- 2.54.0 From da48edf7da9ea3856e8cea47b759fde368f1d37f Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:23:14 -0400 Subject: [PATCH 05/94] =?UTF-8?q?feat:=20the=20worker-lanes=20card=20?= =?UTF-8?q?=E2=80=94=20see=20each=20lane,=20change=20its=20slots=20(4294)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 4. Rule 27: no UI, no ship. This is where the previous three steps become usable. REACHABLE AT: Settings -> Activity -> Worker lanes, directly under the "Queues + workers" pane. Written from opening the view, not from memory of having built it — lesson #4282, and #3463 is the same trap landing inside milestone 365, where the System page shipped with no navigation to it. Under that pane deliberately, not in the System tab. System answers "is everything running", where every control would be about something broken. This is about something working that should work harder, and it belongs beside the backlog it reacts to: you watch a queue grow and give that lane another slot without leaving the pane. Per lane: queues, pending, busy, a stepper, an enable switch. - PENDING is depth + reserved, not LLEN. Celery prefetches, so LLEN alone reads 0 while a worker holds tasks in memory — the number that would make someone think a buried lane was idle. - NOT ANSWERING, never "stopped". present=false means nothing replied; saying stopped would send the operator looking for a crash that has not happened. - THE CEILING IS ON SCREEN, with "(memory)" on the ML lane. It is the one number here the operator cannot change, so it has to justify itself; a greyed stepper with no explanation reads as a bug. - `busy` is per lane, so adjusting one does not freeze the others. TWO OUTCOMES THAT MUST NOT COLLAPSE INTO ONE MESSAGE. A stored-but-unpushed change (applied:false — the lane is restarting) is information: the value is saved and the reconcile will carry it, so the card says so and invites waiting. A refused value (400) is an error and shows the endpoint's own sentence. Collapsing them would make one of the two invite the wrong action. A BUG CAUGHT BEFORE COMMIT: the card read `e.detail?.detail`, but ApiError puts the parsed body on `.body` and `.message` on the short `error` key. It would have shown the operator the bare word "refused" with no reason — the exact failure that line exists to prevent. The test would not have caught it either: `rejects.toThrow()` passes whether the sentence is reachable or not. It now asserts `err.body.detail` specifically. Also: queueOptions in SystemActivityTab was a fourth hand-kept copy of the queue list and had drifted — `maintenance_long` was missing, so activity on that lane could not be filtered for at all despite four task routes pointing there. Added, and DELIBERATELY left as a written-out list rather than derived like the other three were: this filters task_run HISTORY, so a derived list would hide the filter for any queue that has rows but no longer has a lane — precisely when someone is looking — and would be empty whenever the endpoint is down. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../components/settings/SystemActivityTab.vue | 18 ++ .../components/settings/WorkerLanesCard.vue | 191 ++++++++++++++++++ frontend/src/stores/systemActivity.js | 40 +++- frontend/test/workerLanes.spec.js | 141 +++++++++++++ 4 files changed, 387 insertions(+), 3 deletions(-) create mode 100644 frontend/src/components/settings/WorkerLanesCard.vue create mode 100644 frontend/test/workerLanes.spec.js diff --git a/frontend/src/components/settings/SystemActivityTab.vue b/frontend/src/components/settings/SystemActivityTab.vue index 8470352..7c8d8a0 100644 --- a/frontend/src/components/settings/SystemActivityTab.vue +++ b/frontend/src/components/settings/SystemActivityTab.vue @@ -17,6 +17,11 @@ + + + @@ -172,6 +177,7 @@ import { useSystemActivityStore } from '../../stores/systemActivity.js' import { formatRelative as fmtRelative } from '../../utils/date.js' import ErrorDetailModal from '../common/ErrorDetailModal.vue' import QueuesTable from './QueuesTable.vue' +import WorkerLanesCard from './WorkerLanesCard.vue' import CardHeading from '../common/CardHeading.vue' import GpuActivityPanel from './GpuActivityPanel.vue' import DownloadsActivityPanel from './DownloadsActivityPanel.vue' @@ -198,6 +204,16 @@ const filterErrorType = ref(null) const filterTask = ref(null) // server-side task-name search (All activity) const failureSearch = ref('') // client-side search over loaded failures +// This filters HISTORY — task_run.queue — which is why it stays a written-out +// list rather than being derived from the lanes endpoint like the other queue +// lists were in milestone 422. Two reasons, and the second is the real one: +// a derived list is empty whenever that endpoint is down, and more importantly +// it would HIDE the filter for any queue that has rows but no longer has a +// lane serving it, which is exactly when someone is looking. +// +// It had drifted regardless — `maintenance_long` was missing, so activity on +// the long-maintenance lane could not be filtered for at all despite four task +// routes pointing there. Added. const queueOptions = [ { title: 'All queues', value: null }, { title: 'import', value: 'import' }, @@ -206,6 +222,7 @@ const queueOptions = [ { title: 'download', value: 'download' }, { title: 'scan', value: 'scan' }, { title: 'maintenance', value: 'maintenance' }, + { title: 'maintenance_long', value: 'maintenance_long' }, { title: 'default', value: 'default' }, ] const statusOptions = [ @@ -225,6 +242,7 @@ function pollQueues() { store.loadQueues() store.loadWorkers() store.loadRecentRuns() + store.loadLanes() } function pollFailures() { if (document.hidden) return diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue new file mode 100644 index 0000000..ca0d09e --- /dev/null +++ b/frontend/src/components/settings/WorkerLanesCard.vue @@ -0,0 +1,191 @@ + + + + + diff --git a/frontend/src/stores/systemActivity.js b/frontend/src/stores/systemActivity.js index 89ebbab..2d72fb6 100644 --- a/frontend/src/stores/systemActivity.js +++ b/frontend/src/stores/systemActivity.js @@ -12,13 +12,21 @@ export const useSystemActivityStore = defineStore('systemActivity', () => { const recentRuns = ref([]) // last-60s rows (for Overview summary) const failures = ref(null) // { recent, count_by_type, since } + // Worker lanes (milestone 422): the configured slots joined to the live + // pool. Lives here rather than in its own store because it is the same + // domain the queues and workers above describe — a second store polling + // /api/system/* would be two things to keep in step. + const lanes = ref(null) // { lanes: [...], fetched_at } + // Paginated runs (Activity tab "All recent activity" pane). const runs = ref([]) const runsCursor = ref(null) const runsHasMore = ref(false) const runsFilter = ref({ queue: null, status: null, task: null, limit: 50 }) - const loading = ref({ queues: false, workers: false, runs: false, failures: false }) + const loading = ref({ + queues: false, workers: false, runs: false, failures: false, lanes: false, + }) const lastError = ref(null) async function loadQueues() { @@ -45,6 +53,32 @@ export const useSystemActivityStore = defineStore('systemActivity', () => { } } + async function loadLanes() { + loading.value.lanes = true + lastError.value = null + try { + lanes.value = await api.get('/api/system/workers') + } catch (e) { + lastError.value = e.message + } finally { + loading.value.lanes = false + } + } + + // Change one lane. Returns the endpoint's reply so the caller can tell a + // stored-but-not-yet-live change (`applied: false`) from a live one — the + // difference between "saved, the lane is restarting" and "that failed", + // which the UI must not collapse into one message. + // + // Deliberately NOT swallowing the error: a refused value (400) carries the + // sentence explaining why, and the card shows it. Returning null on failure + // would leave the operator with a control that silently did nothing. + async function setLane(name, fields) { + const reply = await api.post(`/api/system/workers/${name}`, { body: fields }) + await loadLanes() + return reply + } + async function loadRecentRuns() { // Used by the Overview summary card: pull last 60s of runs to compute // per-queue ok/err counts. One call covers all queues; UI groups. @@ -107,10 +141,10 @@ export const useSystemActivityStore = defineStore('systemActivity', () => { } return { - queues, workers, recentRuns, failures, summary, + queues, workers, recentRuns, failures, summary, lanes, runs, runsCursor, runsHasMore, runsFilter, loading, lastError, - loadQueues, loadWorkers, loadRecentRuns, + loadQueues, loadWorkers, loadRecentRuns, loadLanes, setLane, loadRuns, loadFailures, loadSummary, setFilter, } }) diff --git a/frontend/test/workerLanes.spec.js b/frontend/test/workerLanes.spec.js new file mode 100644 index 0000000..a0e75ae --- /dev/null +++ b/frontend/test/workerLanes.spec.js @@ -0,0 +1,141 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useSystemActivityStore } from '../src/stores/systemActivity.js' + +// Milestone 422 step 4. Covers the store half of the worker-lane dial — the +// part that decides what the card can tell the operator. +// +// The distinction being protected: a change that was STORED but not pushed +// (`applied: false`, because the lane is restarting) is not a failure, and a +// REFUSED value (400) is. Collapsing those two into one message is how a +// control stops being trustworthy — one invites waiting, the other invites +// changing what you asked for. + +function stubFetch(handler) { + globalThis.fetch = vi.fn(async (url, init) => { + const { status, body } = handler(url, init) + return { + ok: status >= 200 && status < 300, + status, + statusText: String(status), + text: async () => (body == null ? '' : JSON.stringify(body)), + } + }) +} + +const LANES_BODY = { + lanes: [ + { + name: 'worker', display_name: 'Worker', + queues: ['default', 'import', 'thumbnail', 'download'], + slots: 1, slots_cap: 4, ceiling: 8, enabled: true, memory_bound: false, + live: { present: true, replicas: 1, pool: 1, active: 0, reserved: 3 }, + queue_depth: 5, pending: 8, + }, + { + name: 'ml', display_name: 'ML tagging', queues: ['ml'], + slots: 0, slots_cap: 1, ceiling: 2, enabled: false, memory_bound: true, + live: { present: false, replicas: 0, pool: null, active: 0, reserved: 0 }, + queue_depth: null, pending: null, + }, + ], + fetched_at: '2026-09-22T12:00:00Z', +} + +describe('worker lanes store', () => { + beforeEach(() => setActivePinia(createPinia())) + afterEach(() => vi.restoreAllMocks()) + + it('loads the lanes', async () => { + stubFetch(() => ({ status: 200, body: LANES_BODY })) + const s = useSystemActivityStore() + await s.loadLanes() + expect(s.lanes.lanes.map((l) => l.name)).toEqual(['worker', 'ml']) + }) + + it('a load failure records the error rather than throwing at the caller', async () => { + // The card polls this every 3s. An unhandled rejection per tick would + // drown the console and stop the other pollers in the same function. + stubFetch(() => ({ status: 500, body: { error: 'boom' } })) + const s = useSystemActivityStore() + await expect(s.loadLanes()).resolves.toBeUndefined() + expect(s.lastError).toBeTruthy() + }) + + it('setLane posts only the fields it was given', async () => { + // Partial update: the stepper sends slots without restating a cap it did + // not touch. Sending the whole row back would make two operators editing + // different fields clobber each other. + const calls = [] + stubFetch((url, init) => { + calls.push({ url, init }) + if (init?.method === 'POST') { + return { status: 200, body: { name: 'worker', slots: 2, applied: true } } + } + return { status: 200, body: LANES_BODY } + }) + const s = useSystemActivityStore() + await s.setLane('worker', { slots: 2 }) + + const post = calls.find((c) => c.init?.method === 'POST') + expect(post.url).toContain('/api/system/workers/worker') + expect(JSON.parse(post.init.body)).toEqual({ slots: 2 }) + }) + + it('setLane refetches so the card shows the server truth, not the guess', async () => { + // The reply is one lane; the card renders all of them plus live pool and + // pending. Patching the local row from the reply would leave every other + // column stale and eventually wrong. + let gets = 0 + stubFetch((url, init) => { + if (init?.method === 'POST') return { status: 200, body: { applied: true } } + gets += 1 + return { status: 200, body: LANES_BODY } + }) + const s = useSystemActivityStore() + await s.setLane('worker', { slots: 2 }) + expect(gets).toBe(1) + }) + + it('a stored-but-unapplied change comes back as applied:false, not an error', async () => { + // The lane is restarting. The value IS saved and the reconcile will carry + // it — so this must reach the card as information, not as a failure that + // invites the operator to set it again. + stubFetch((url, init) => { + if (init?.method === 'POST') { + return { + status: 200, + body: { applied: false, apply_error: 'lane is not running', slots: 2 }, + } + } + return { status: 200, body: LANES_BODY } + }) + const s = useSystemActivityStore() + const reply = await s.setLane('worker', { slots: 2 }) + expect(reply.applied).toBe(false) + expect(reply.apply_error).toContain('not running') + }) + + it('a refused value throws so the card can show the reason', async () => { + // Deliberately NOT swallowed. The detail is written to be read by a person + // ("above what this container can hold"), and a control that silently does + // nothing is worse than one that refuses out loud. + stubFetch((url, init) => { + if (init?.method === 'POST') { + return { + status: 400, + body: { error: 'refused', detail: 'cap 10000 is above what this container can hold (2 for ML tagging)' }, + } + } + return { status: 200, body: LANES_BODY } + }) + const s = useSystemActivityStore() + // Assert the REASON is reachable, not merely that it threw. `toThrow()` + // alone passes whether the card can read the sentence or not — which is + // how the first version of the card shipped reading `e.detail` (always + // undefined) and would have shown the operator the bare word "refused". + const err = await s.setLane('ml', { slots_cap: 10000 }).catch((e) => e) + expect(err.status).toBe(400) + expect(err.body.detail).toContain('container can hold') + }) +}) -- 2.54.0 From 172e33de9afe66bf6320c518914dc51b35e18be5 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:32:27 -0400 Subject: [PATCH 06/94] feat: run web and every worker lane in one container (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d` gives three containers — FabledCurator, Postgres, Redis — where the stack previously needed seven. THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app services separately and remains the right shape for a Swarm deployment spread across hosts, where per-service rolling rollback and placement constraints matter. This adds a compose file; it deletes none. `entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES and execs it as PID 1. Generated rather than checked in because a static .conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of the queue names — after celery_app.task_routes and the three collapsed in steps 1, 2 and 4. Every one of those had already drifted when found. Generating gives a stronger guarantee than "they match today": a lane added to LANES gets a process, and a queue cannot end up with no consumer because someone missed a file. supervisord over s6-overlay: one pip dependency on an image already Python, with per-program stop timeouts and stopasgroup. The process-group part is not a detail — celery's prefork pool forks children, and a TERM reaching only the parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and zombie handling) comes from `init: true` instead. Nothing in FC talks to the supervisor, so the choice is reversible without touching product code. FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements live only in Dockerfile.ml until step 6 merges the images, so an `ml` program here would fail to import on every restart forever. `--with-ml` is the flag step 6 turns on. THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught: 1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord parses that key as a COMMA-separated list, so it reads as CELERY_QUEUES=default plus three malformed entries and the worker lane would have consumed only `default`. Silent: the worker starts, reports healthy, never picks up an import. Now quoted, and the test asserts the quoted form rather than the bare substring, which passed either way. 2. The generator emitted `entrypoint.sh `, but `maintenance_long` is not a role — compose runs it as the plain `worker` role with different queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh to assert every role a lane names actually exists. 3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY. Harmless while only compose started it and set none; with a generated value being passed, the lane would have sat at 1 until the reconcile noticed, with nothing saying why. The healthcheck asserts BOTH halves — hypercorn answers and every configured lane is answering the broker. That is the failure mode consolidation creates: docker can no longer see the lanes as separate services, so a web-only check would report a healthy container with every lane inside it dead. It deliberately ignores the `enabled` flag: a disabled lane still has a running process with its consumers cancelled, and marking the container unhealthy for turning tagging off would be wrong. stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s) rather than the average, with a test asserting no program's stopwaitsecs can exceed what compose allows. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/scripts/gen_supervisord.py | 184 ++++++++++++++++++++++++ backend/app/scripts/healthcheck_all.py | 83 +++++++++++ backend/app/services/worker_lanes.py | 10 ++ docker-compose.single.yml | 113 +++++++++++++++ entrypoint.sh | 33 ++++- requirements.txt | 11 ++ tests/test_gen_supervisord.py | 185 +++++++++++++++++++++++++ 7 files changed, 616 insertions(+), 3 deletions(-) create mode 100644 backend/app/scripts/gen_supervisord.py create mode 100644 backend/app/scripts/healthcheck_all.py create mode 100644 docker-compose.single.yml create mode 100644 tests/test_gen_supervisord.py diff --git a/backend/app/scripts/gen_supervisord.py b/backend/app/scripts/gen_supervisord.py new file mode 100644 index 0000000..2747599 --- /dev/null +++ b/backend/app/scripts/gen_supervisord.py @@ -0,0 +1,184 @@ +"""Emit a supervisord config for the single-container layout. + +Milestone 422 step 5. Writes to stdout; `entrypoint.sh all` redirects it to a +file and execs supervisord against it. + +## Why this is generated and not a checked-in .conf + +A static config would spell out each lane's `-Q` list, and that would be a +FIFTH hand-kept copy of the queue names — after `celery_app.task_routes`, and +the three collapsed in steps 1, 2 and 4 (`service_roster.ROLE_NAMES`, +`system_activity._QUEUE_NAMES`, and the Activity filter). Every one of those +had already drifted by the time it was found. + +Generating from `worker_lanes.LANES` makes a stronger guarantee than "they +match today": the processes this container runs and the lanes the application +believes in are the same list, so a lane added to `LANES` gets a process +without anyone remembering to add one, and a queue can never end up with no +consumer because a config file was missed. + +## Why supervisord + +It is one pip dependency on an image that is already Python, and it does the +four things this needs without being clever: restart a program that exits, +give each one its OWN stop timeout, signal the process GROUP rather than the +leader, and put every program's output on one stdout. + +The process-group part is not a detail. Celery's prefork pool forks children, +and a TERM delivered only to the parent leaves them running — which is how a +"graceful" shutdown turns into orphaned workers holding tasks. `stopasgroup` +and `killasgroup` are both set for every program. + +s6-overlay is the other standard answer and would work; it needs a build-time +download and a second mental model, and its advantage (correct PID-1 signal +and zombie handling) is available here from `init: true` in compose, which +puts tini in front of supervisord. Neither choice reaches the application — +nothing in FC talks to the supervisor — so this is reversible without touching +a line of product code. + +## What this does NOT start + +The `ml` lane, unless `--with-ml` is passed. Until step 6 merges the images, +torch and the ML requirements live only in `Dockerfile.ml`, so an `ml` program +in the web image would fail to import on every restart forever. Step 6 is +where that flag turns on. +""" + +from __future__ import annotations + +import argparse +import shlex +import sys + +from ..services.worker_lanes import LANES, Lane + +# One number for the whole container, and it must cover the SLOWEST lane — +# docker gives the container a single stop timeout, where compose today gives +# each service its own (90/60/180/120s). `maintenance_long` is the 180s one: +# DB backups, library audits and translation backfill. Anything less turns a +# routine restart into a SIGKILL mid-backup. +# +# Per-program values below are the old per-service ones, preserved: supervisord +# waits `stopwaitsecs` for each, and they stop in parallel, so the container's +# own timeout needs to cover the max rather than the sum. +STOP_WAIT_SECONDS: dict[str, int] = { + "worker": 90, + "scheduler": 60, + "maintenance_long": 180, + "ml": 120, +} +DEFAULT_STOP_WAIT = 60 + +# Lanes whose code is not in this image yet. See the module docstring. +_NEEDS_ML_DEPS = frozenset({"ml"}) + + +def _program(lane: Lane, *, slots: int) -> str: + """One [program:x] block. + + `stdout_logfile=/dev/fd/1` with maxbytes 0 puts the lane's output straight + on the container's stdout unbuffered, so `docker logs` shows every lane + interleaved rather than supervisord swallowing them into rotated files. + + The output is prefixed through `sed` so a line can be attributed to a lane + — four celery workers and hypercorn on one stream are otherwise + indistinguishable. The shell that the pipe requires is exactly why + `stopasgroup` matters: the signal has to reach the celery process, not the + `sh` holding the pipeline. + """ + inner = f"./entrypoint.sh {lane.entrypoint_role}" + prefixed = f"{inner} 2>&1 | sed -u 's/^/[{lane.name}] /'" + stop_wait = STOP_WAIT_SECONDS.get(lane.name, DEFAULT_STOP_WAIT) + return "\n".join([ + f"[program:{lane.name}]", + f"command=sh -c {shlex.quote(prefixed)}", + # QUOTED, and that is load-bearing. supervisord parses `environment` + # as a COMMA-separated KEY=VALUE list, so an unquoted queue list reads + # as CELERY_QUEUES=default followed by three malformed entries — and + # the lane would consume only its first queue. Silent: the worker + # starts, reports healthy, and simply never picks up `import`. + f'environment=CELERY_QUEUES="{",".join(lane.queues)}",' + f"CELERY_CONCURRENCY={slots}", + "autostart=true", + "autorestart=true", + # A lane that dies instantly and repeatedly is a broken image, not a + # transient fault. Backing off stops it burning a core in a restart + # loop while still recovering from a one-off crash. + "startretries=3", + "startsecs=5", + f"stopwaitsecs={stop_wait}", + "stopasgroup=true", + "killasgroup=true", + "stdout_logfile=/dev/fd/1", + "stdout_logfile_maxbytes=0", + "redirect_stderr=true", + "", + ]) + + +def _web_program() -> str: + """hypercorn. Started FIRST (priority) because its role runs + `alembic upgrade head`, and a worker that boots against an un-migrated + schema fails in a way that looks like application breakage.""" + prefixed = "./entrypoint.sh web 2>&1 | sed -u 's/^/[web] /'" + return "\n".join([ + "[program:web]", + f"command=sh -c {shlex.quote(prefixed)}", + "priority=1", + "autostart=true", + "autorestart=true", + "startretries=3", + "startsecs=5", + # Short: HTTP requests and the occasional file download. Matches the + # 30s the operator's production stack gives the web service. + "stopwaitsecs=30", + "stopasgroup=true", + "killasgroup=true", + "stdout_logfile=/dev/fd/1", + "stdout_logfile_maxbytes=0", + "redirect_stderr=true", + "", + ]) + + +def render(*, with_ml: bool = False) -> str: + parts = [ + "\n".join([ + "[supervisord]", + # PID 1 in the container, so it must not daemonise. + "nodaemon=true", + # supervisord's OWN log. /dev/fd/1 keeps it on the container's + # stdout beside the programs rather than in a file nobody reads. + "logfile=/dev/fd/1", + "logfile_maxbytes=0", + "loglevel=info", + "", + ]), + _web_program(), + ] + # Lanes after web, in LANES order, so the log reads in a stable sequence. + for lane in LANES: + if lane.name in _NEEDS_ML_DEPS and not with_ml: + continue + # A lane configured at zero slots still gets a PROCESS, at one slot + # with its consumers cancelled by the reconcile. Without a running + # worker there is nothing for `add_consumer` to reach, so enabling the + # lane from the UI could not work at all — the process has to exist for + # the switch to have something to switch. + parts.append(_program(lane, slots=max(1, lane.default_slots))) + return "\n".join(parts) + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--with-ml", action="store_true", + help="include the ml lane (only valid once the ML deps are in this image)", + ) + args = ap.parse_args(argv) + sys.stdout.write(render(with_ml=args.with_ml)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/scripts/healthcheck_all.py b/backend/app/scripts/healthcheck_all.py new file mode 100644 index 0000000..4b8076a --- /dev/null +++ b/backend/app/scripts/healthcheck_all.py @@ -0,0 +1,83 @@ +"""Container healthcheck for the single-container layout. + +Milestone 422 step 5. Exit 0 healthy, non-zero unhealthy. + +## Why this is not just "does :8080 answer" + +In the multi-service stack every service has its OWN healthcheck, so a dead +worker turns that service unhealthy while web stays green — docker knows which +part failed. Collapsing them into one container collapses that too: a web-only +check would report a perfectly healthy container while every lane inside it +had crashed and been abandoned by supervisord after its retries. + +So this asserts both halves: hypercorn answers, AND every lane this container +was configured to run is answering the broker. + +## What it deliberately does NOT do + +It does not read the database, and it does not consult the `enabled` flag. A +DISABLED lane still has a running process with its consumers cancelled (see +the config generator), so it answers `inspect` and is healthy. Health is +"is the process alive", and whether it should be consuming is a settings +question the reconcile owns — conflating them would make turning a lane off +in the UI mark the container unhealthy. + +It also cannot distinguish "the broker is down" from "every lane is down", +and reports unhealthy either way. That is correct: a container that cannot +reach its broker is not serving, whichever half is at fault. +""" + +from __future__ import annotations + +import sys +import urllib.error +import urllib.request + +WEB_URL = "http://localhost:8080/api/health" +WEB_TIMEOUT = 5.0 + + +def _web_ok() -> tuple[bool, str]: + try: + with urllib.request.urlopen(WEB_URL, timeout=WEB_TIMEOUT) as resp: + if resp.status == 200: + return True, "" + return False, f"web returned {resp.status}" + except (urllib.error.URLError, OSError) as exc: + return False, f"web unreachable: {exc}" + + +def _lanes_ok(*, with_ml: bool) -> tuple[bool, str]: + from ..services.worker_control import inspect_lanes_sync + from ..services.worker_lanes import LANES + + expected = { + lane.name for lane in LANES + if with_ml or lane.name != "ml" + } + live = inspect_lanes_sync() + missing = sorted(n for n in expected if not live[n].present) + if missing: + return False, "lanes not answering: " + ", ".join(missing) + return True, "" + + +def main(argv: list[str] | None = None) -> int: + argv = sys.argv[1:] if argv is None else argv + with_ml = "--with-ml" in argv + + ok, detail = _web_ok() + if not ok: + print(detail, file=sys.stderr) + return 1 + + ok, detail = _lanes_ok(with_ml=with_ml) + if not ok: + print(detail, file=sys.stderr) + return 1 + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index a0d7444..cdd639f 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -71,6 +71,12 @@ class Lane: name: str display_name: str queues: tuple[str, ...] + # Which `entrypoint.sh` role starts this lane. NOT always the lane name: + # `maintenance_long` is the plain `worker` role pointed at a different + # queue, exactly as docker-compose starts it today (`command: ["worker"]` + # with CELERY_QUEUES=maintenance_long). Recorded here so the generated + # supervisord config and the compose file cannot disagree about it. + entrypoint_role: str default_slots: int # The cap a lane STARTS with, which is not the ceiling. Set low enough # that raising slots within it is an ordinary adjustment, and raising the @@ -104,6 +110,7 @@ LANES: tuple[Lane, ...] = ( name="worker", display_name="Worker", queues=("default", "import", "thumbnail", "download"), + entrypoint_role="worker", default_slots=1, default_slots_cap=4, default_enabled=True, @@ -112,6 +119,7 @@ LANES: tuple[Lane, ...] = ( name="scheduler", display_name="Scheduler", queues=("maintenance", "scan"), + entrypoint_role="scheduler", default_slots=1, default_slots_cap=2, default_enabled=True, @@ -120,6 +128,7 @@ LANES: tuple[Lane, ...] = ( name="maintenance_long", display_name="Long maintenance", queues=("maintenance_long",), + entrypoint_role="worker", default_slots=1, default_slots_cap=2, default_enabled=True, @@ -128,6 +137,7 @@ LANES: tuple[Lane, ...] = ( name="ml", display_name="ML tagging", queues=("ml",), + entrypoint_role="ml-worker", default_slots=0, default_slots_cap=1, default_enabled=False, diff --git a/docker-compose.single.yml b/docker-compose.single.yml new file mode 100644 index 0000000..d94741c --- /dev/null +++ b/docker-compose.single.yml @@ -0,0 +1,113 @@ +# FabledCurator in three containers — the install path. +# +# docker compose -f docker-compose.single.yml up -d +# +# Milestone 422 step 5. FabledCurator runs web and every worker lane inside +# ONE container, with Postgres and Redis beside it. How much work each lane +# does is then a dial in the web UI (Settings -> Activity -> Worker lanes), +# live, with no compose edit and no restart. +# +# THE MULTI-SERVICE STACK IS NOT REPLACED. `docker-compose.yml` still runs the +# five app services separately and is the right shape for a Swarm deployment +# spread across hosts, where per-service rolling rollback and placement +# constraints matter. This file is the adopter path: one box, one command. +# +# What consolidating costs, stated here rather than discovered later: +# - Everything shares one host, so there is no spreading work across nodes. +# - Rollback is all-or-nothing; there is no rolling back `web` alone. +# - The ML lane cannot write to the library read-only any more — in the +# multi-service stack ml-worker mounts /images:ro, and one container +# cannot mount one path two ways. +# - One stop timeout for the whole container, sized to the slowest lane. +# +# FabledCurator has no authentication. Whatever can reach ${PORT} is an +# administrator, including over the stored platform session cookies. Do not +# publish this port beyond a network you trust — see "Before you expose it" +# in README.md. + +services: + redis: + image: redis:7-alpine + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + + postgres: + image: pgvector/pgvector:pg16 + environment: + POSTGRES_USER: ${DB_USER:-curator} + POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres} + POSTGRES_DB: ${DB_NAME:-curator} + volumes: + - postgres_data:/var/lib/postgresql/data + # pgvector index builds and the gallery's TABLESAMPLE reads both want more + # shared memory than docker's 64MB default. + shm_size: 512m + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-curator} -d ${DB_NAME:-curator}"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + + fabledcurator: + image: git.fabledsword.com/bvandeusen/fabledcurator:latest + # Everything: hypercorn plus one celery process per lane, under + # supervisord, whose config is generated from the application's own lane + # table so the two cannot disagree. + command: ["all"] + # tini as PID 1, in front of supervisord. supervisord reaps its own + # children, but a container's PID 1 also inherits orphans from anywhere + # below — celery's prefork pool and gallery-dl's subprocesses both make + # them. Without this they accumulate as zombies for the life of the + # container. + init: true + # Sized to the SLOWEST lane, not the average. maintenance_long runs DB + # backups, library audits and translation backfill, and gets 180s to + # finish a chunk; the lanes stop in parallel, so this covers the max + # rather than their sum. Below this, a routine restart becomes a SIGKILL + # mid-backup — which is recoverable (the work is chunked and idempotent) + # but wastes however long it had run. + stop_grace_period: 200s + # BOTH halves: hypercorn answers AND every configured lane is answering + # the broker. A web-only check would report a healthy container while + # every lane inside it had crashed — the failure mode consolidation + # creates, since docker can no longer see the lanes as separate services. + healthcheck: + test: ["CMD", "python", "-m", "backend.app.scripts.healthcheck_all"] + interval: 30s + timeout: 15s + retries: 3 + # Covers alembic + hypercorn boot + four celery workers registering. + start_period: 90s + environment: + DB_USER: ${DB_USER:-curator} + DB_PASSWORD: ${DB_PASSWORD:-postgres} + DB_HOST: postgres + DB_PORT: "5432" + DB_NAME: ${DB_NAME:-curator} + CELERY_BROKER_URL: redis://redis:6379/0 + CELERY_RESULT_BACKEND: redis://redis:6379/0 + SECRET_KEY: ${SECRET_KEY:-change-me-before-you-expose-this} + EXTENSION_API_KEY: ${EXTENSION_API_KEY:-} + LOG_LEVEL: ${LOG_LEVEL:-INFO} + ports: + - "${PORT:-8080}:8080" + volumes: + - ${IMAGES_DIR:-./images}:/images + # Read-only. The filesystem scan copies out of here and never writes to + # it, so a mistake cannot reach the source library. + - ${IMPORT_DIR:-./import}:/import:ro + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } + restart: unless-stopped + +volumes: + redis_data: + postgres_data: diff --git a/entrypoint.sh b/entrypoint.sh index 9231cce..b03b252 100755 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -35,12 +35,18 @@ case "$ROLE" in scheduler) QUEUES="${CELERY_QUEUES:-maintenance,scan}" - echo "[entrypoint] Starting Celery beat+worker queues=$QUEUES" + # Honours CELERY_CONCURRENCY like the `worker` role does. It was hardcoded + # to 1, which was harmless while only compose started this lane and set no + # concurrency for it — but the generated supervisord config (milestone 422 + # step 5) passes one, and a value silently ignored at boot would leave the + # lane at 1 until the reconcile sweep noticed, with nothing saying why. + CONCURRENCY="${CELERY_CONCURRENCY:-1}" + echo "[entrypoint] Starting Celery beat+worker queues=$QUEUES concurrency=$CONCURRENCY" exec celery -A backend.app.celery_app:celery worker \ --beat \ --loglevel=info \ -Q "$QUEUES" \ - --concurrency=1 + --concurrency="$CONCURRENCY" ;; ml-worker) @@ -53,6 +59,27 @@ case "$ROLE" in --concurrency=1 ;; + all) + # The single-container layout (milestone 422 step 5): hypercorn plus one + # celery process per lane, under supervisord, in one container beside + # Postgres and Redis. + # + # The config is GENERATED from services/worker_lanes.LANES rather than + # checked in, so the processes this container runs and the lanes the + # application believes in cannot disagree — see the generator's docstring + # for why a static .conf would have been a fifth copy of the queue names. + # + # supervisord is PID 1 here and never reads the database. Every lane boots + # at its LANES default; the reconcile sweep raises it to whatever the + # operator stored, within one tick. That ordering is deliberate: settings + # adjust a baseline that already works, and can never prevent a boot. + CONF="${SUPERVISOR_CONF:-/tmp/supervisord.conf}" + echo "[entrypoint] Generating $CONF from the lane table" + python -m backend.app.scripts.gen_supervisord ${FC_WITH_ML:+--with-ml} > "$CONF" + echo "[entrypoint] Starting supervisord (web + worker lanes)" + exec supervisord -c "$CONF" + ;; + shell|bash) exec /bin/bash "$@" ;; @@ -63,7 +90,7 @@ case "$ROLE" in *) echo "[entrypoint] Unknown role: $ROLE" >&2 - echo "[entrypoint] Valid roles: web | worker | scheduler | ml-worker | shell | alembic" >&2 + echo "[entrypoint] Valid roles: all | web | worker | scheduler | maintenance_long | ml | ml-worker | shell | alembic" >&2 exit 1 ;; esac diff --git a/requirements.txt b/requirements.txt index 1950b50..f54bc43 100644 --- a/requirements.txt +++ b/requirements.txt @@ -43,3 +43,14 @@ py7zr>=1,<2 # #830). Handles Drive's confirm-token + virus-scan interstitial. mega.nz uses # the `megatools` binary instead (Debian apt pkg in the runtime image, not pip). gdown>=6,<7 + +# Process supervisor for the single-container layout (milestone 422 step 5). +# `entrypoint.sh all` generates its config from the lane table and execs it as +# PID 1, running hypercorn plus one celery process per lane in one container. +# Unused by the multi-service compose path, where docker supervises instead. +# +# Chosen over s6-overlay because it is a pip install on an image that is +# already Python, and gives per-program stop timeouts plus stopasgroup — +# celery's prefork pool forks children, and a TERM that reaches only the +# parent leaves them orphaned holding tasks. +supervisor>=4.2,<5 diff --git a/tests/test_gen_supervisord.py b/tests/test_gen_supervisord.py new file mode 100644 index 0000000..b92cce3 --- /dev/null +++ b/tests/test_gen_supervisord.py @@ -0,0 +1,185 @@ +"""The generated supervisord config (milestone 422 step 5). + +Asserts the config against the LANE TABLE rather than against a fixture of +expected text. A fixture would have to be updated whenever a lane changes, +which is the same hand-kept coupling generating the config exists to remove — +and it would pass while describing a container that does not match the +application's own idea of what it runs. +""" + +from __future__ import annotations + +import configparser + +from backend.app.scripts import gen_supervisord as gen +from backend.app.services.worker_lanes import LANES, LANES_BY_NAME + +# --- structure --------------------------------------------------------------- + + +def _parse(**kwargs) -> configparser.ConfigParser: + """supervisord's config is ini, so parse it rather than grepping strings. + + A substring assertion passes on a line that is present but malformed — + inside a comment, in the wrong section, or with a typo'd key that + supervisord silently ignores. + """ + cp = configparser.ConfigParser() + cp.read_string(gen.render(**kwargs)) + return cp + + +def test_it_is_valid_ini_with_a_supervisord_section(): + cp = _parse() + assert cp.has_section("supervisord") + # PID 1 in a container: daemonising would exit immediately and take the + # container with it. + assert cp.get("supervisord", "nodaemon") == "true" + + +def test_web_and_every_non_ml_lane_get_a_program(): + cp = _parse() + expected = {"program:web"} | { + f"program:{lane.name}" for lane in LANES if lane.name != "ml" + } + assert set(cp.sections()) - {"supervisord"} == expected + + +def test_the_ml_lane_is_absent_until_its_deps_are_in_the_image(): + """The web image has no torch. An `ml` program here would fail to import + on every restart, forever — startretries would give up and the lane would + be permanently dead while the container reported healthy.""" + assert not _parse().has_section("program:ml") + assert _parse(with_ml=True).has_section("program:ml") + + +# --- the coupling this generator exists to guarantee ------------------------- + + +def test_each_program_serves_exactly_its_lane_s_queues(): + """The whole point: the container's processes and the application's lane + table are one list. A queue in LANES with no program means work that + queues forever with nothing consuming it.""" + cp = _parse(with_ml=True) + for lane in LANES: + env = cp.get(f"program:{lane.name}", "environment") + # The QUOTED form. supervisord splits `environment` on commas, so an + # unquoted multi-queue value silently degrades to its first queue — + # and an assertion on the bare string passes either way, which is how + # that would have shipped. + assert f'CELERY_QUEUES="{",".join(lane.queues)}"' in env + + +def test_each_program_invokes_the_lane_s_entrypoint_role_not_its_name(): + """`maintenance_long` is the plain `worker` role pointed at a different + queue — exactly as docker-compose starts it today. Invoking + `entrypoint.sh maintenance_long` would hit the unknown-role branch and + exit 1 on every restart.""" + cp = _parse(with_ml=True) + for lane in LANES: + command = cp.get(f"program:{lane.name}", "command") + assert f"entrypoint.sh {lane.entrypoint_role}" in command + + +def test_every_entrypoint_role_a_lane_names_actually_exists(): + """Reads entrypoint.sh itself. The generator can only emit a role name; + whether the script handles it is a separate fact, and getting it wrong + fails at container start rather than here.""" + from pathlib import Path + + script = Path(__file__).resolve().parents[1] / "entrypoint.sh" + text = script.read_text() + for lane in LANES: + # Roles are `case` arms: ` worker)` possibly in an alternation. + assert f" {lane.entrypoint_role})" in text or \ + f"|{lane.entrypoint_role})" in text, \ + f"{lane.name} names entrypoint role {lane.entrypoint_role!r}, which does not exist" + + +# --- shutdown ---------------------------------------------------------------- + + +def test_every_program_signals_its_whole_process_group(): + """Celery's prefork pool forks children. A TERM delivered only to the + parent leaves them running and holding tasks — a 'graceful' shutdown that + orphans workers. The `sh -c … | sed` wrapper makes this doubly necessary: + without it the signal reaches the shell holding the pipeline, not celery.""" + cp = _parse(with_ml=True) + for section in cp.sections(): + if not section.startswith("program:"): + continue + assert cp.get(section, "stopasgroup") == "true", section + assert cp.get(section, "killasgroup") == "true", section + + +def test_the_long_maintenance_lane_keeps_its_180s_drain(): + """The per-service stop_grace_period values from the multi-service stack + are preserved per program. maintenance_long runs DB backups and library + audits; cutting its drain turns a restart into a SIGKILL mid-backup.""" + cp = _parse() + assert cp.getint("program:maintenance_long", "stopwaitsecs") == 180 + + +def test_no_program_waits_longer_than_the_compose_stop_grace_period(): + """The container gets ONE timeout and the programs stop in parallel, so it + must cover the slowest. If a lane's stopwaitsecs ever exceeds what + docker-compose.single.yml allows, docker kills the container while that + lane still believes it has time to drain.""" + import re + from pathlib import Path + + compose = (Path(__file__).resolve().parents[1] / "docker-compose.single.yml").read_text() + m = re.search(r"stop_grace_period:\s*(\d+)s", compose) + assert m, "docker-compose.single.yml has no stop_grace_period" + grace = int(m.group(1)) + + cp = _parse(with_ml=True) + for section in cp.sections(): + if section.startswith("program:"): + assert cp.getint(section, "stopwaitsecs") <= grace, section + + +# --- what runs, and how much ------------------------------------------------ + + +def test_a_zero_slot_lane_still_gets_a_running_process(): + """ML ships at 0 slots and disabled — but `add_consumer` needs something to + reach. With no process there would be nothing for the UI switch to switch, + and enabling tagging could not work at all.""" + cp = _parse(with_ml=True) + assert LANES_BY_NAME["ml"].default_slots == 0 + env = cp.get("program:ml", "environment") + assert "CELERY_CONCURRENCY=1" in env + + +def test_programs_restart_but_back_off_rather_than_looping(): + """A lane that dies instantly and repeatedly is a broken image, not a + transient fault. Unbounded restarts would burn a core forever and bury the + original error under its own noise.""" + cp = _parse() + for section in cp.sections(): + if section.startswith("program:"): + assert cp.get(section, "autorestart") == "true", section + assert cp.getint(section, "startretries") >= 1, section + + +def test_web_starts_first_because_it_runs_the_migration(): + """A worker booting against an un-migrated schema fails in a way that + looks like application breakage rather than an ordering problem.""" + cp = _parse() + assert cp.getint("program:web", "priority") == 1 + + +def test_every_program_writes_to_the_container_stdout_with_its_lane_named(): + """Four celery workers and hypercorn on one stream are indistinguishable + without this. Unbuffered (`maxbytes 0`) so `docker logs` is live rather + than arriving in rotated chunks.""" + cp = _parse(with_ml=True) + for section in cp.sections(): + if not section.startswith("program:"): + continue + name = section.split(":", 1)[1] + assert cp.get(section, "stdout_logfile") == "/dev/fd/1", section + assert cp.getint(section, "stdout_logfile_maxbytes") == 0, section + assert cp.get(section, "redirect_stderr") == "true", section + assert f"[{name}] " in cp.get(section, "command"), section -- 2.54.0 From e57933345513272612171750fbd2b4f5c06b64c1 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:38:49 -0400 Subject: [PATCH 07/94] docs: record that the ml :ro loss is a ruled non-issue (4295) Operator, 2026-09-22: "I don't care about the :ro loss thank you for calling it out repeated but I don't care." Raised three times across the milestone body, this file's header and two reports. Recorded as settled at the point someone would rediscover it, so it is not raised a fourth time. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- docker-compose.single.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docker-compose.single.yml b/docker-compose.single.yml index d94741c..c6e0f38 100644 --- a/docker-compose.single.yml +++ b/docker-compose.single.yml @@ -15,11 +15,13 @@ # What consolidating costs, stated here rather than discovered later: # - Everything shares one host, so there is no spreading work across nodes. # - Rollback is all-or-nothing; there is no rolling back `web` alone. -# - The ML lane cannot write to the library read-only any more — in the -# multi-service stack ml-worker mounts /images:ro, and one container -# cannot mount one path two ways. # - One stop timeout for the whole container, sized to the slowest lane. # +# NOT a cost, recorded so it is not rediscovered and raised again: the +# multi-service stack mounts /images:ro on ml-worker and one container cannot +# mount one path two ways. Operator ruled that a non-issue (2026-09-22) — it +# is the same codebase either way. +# # FabledCurator has no authentication. Whatever can reach ${PORT} is an # administrator, including over the stored platform session cookies. Do not # publish this port beyond a network you trust — see "Before you expose it" -- 2.54.0 From ecbd3254375091aa5882341636aaa0ea253085be Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:45:22 -0400 Subject: [PATCH 08/94] feat: an optional lane says it is optional, and what enabling it costs (4296) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator, 2026-09-22: "since the ml-worker is optional it should be shown as such in the UI and have a warning about what it does and that it pulls the models and what models and their projected size and ram requirements to run." The card previously said "a few GB, once" — a number sourced from nothing, which is exactly the hand-wave I had flagged in this step's own survey log as something that should be measured rather than asserted. ONE FACT CORRECTED WHILE WRITING THE COPY. I had named the lane "ML tagging". It downloads an EMBEDDER: google/siglip-so400m-patch14-384. WD14 tagging is the GPU agent's job — celery_app.py:5 still names both, but that has been stale since B3 (#1238), when the agent took over and this lane was left as the CPU embed fallback for stacks running no agent (see MLSettings.cpu_embed_enabled). Telling someone the lane "does tagging" would have been wrong in exactly the way this request exists to prevent. The facts are structured data on the lane, not prose in a component: ModelRequirement(repo, approx_download_bytes, approx_resident_bytes, measured). The API carries them; the card renders them. Numbers come from the system, wording from the UI. ML_BYTES_PER_SLOT IS NOW DERIVED from that requirement rather than stated separately. They have to be one number: the figure quoted to the operator before they enable the lane and the figure the cap enforces. Two copies could disagree, and the UI would promise a slot the cap then refuses. `measured=False` travels with the numbers and the card renders "about". They are estimates from the checkpoint's parameter count and dtype — ~877M params at fp32 is ~3.5GB of weights — not from a build. This decides whether someone's server survives, so it is labelled rather than rounded into something that reads like a fact. A test asserts the flag is false, to be flipped in the same commit that records a real measurement. The card now shows: an "optional" chip in the row itself (someone scanning the table should not have to enable a lane to learn it was never required), and before the switch, what the lane does, that you only need it if you are NOT running the GPU agent, the repo id, the download size, the per-slot RAM, and why the ceiling is what it is — including saying plainly when a box has too little memory to run it at all. Keyed on the lane's own `optional` flag, not on the name 'ml', so a second optional lane gets the same treatment without anyone remembering to add it. A test asserts no REQUIRED lane declares a model: if one ever needs a download, it stops being required. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/worker_control.py | 14 ++++ backend/app/services/worker_lanes.py | 78 +++++++++++++++---- .../components/settings/WorkerLanesCard.vue | 69 ++++++++++++++-- tests/test_worker_lanes.py | 41 ++++++++++ 4 files changed, 181 insertions(+), 21 deletions(-) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 97e70f8..022abee 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -298,6 +298,20 @@ async def lane_view(session: AsyncSession) -> list[dict]: "ceiling": derived_ceiling(lane), "enabled": row.enabled, "memory_bound": lane.memory_bound, + "optional": lane.optional, + # What enabling this lane will download, so the UI can say WHICH + # model and how big BEFORE the switch is thrown rather than after + # a multi-GB fetch has started. `measured` travels with the + # numbers: the card must not present an estimate as a fact. + "models": [ + { + "repo": m.repo, + "download_bytes": m.approx_download_bytes, + "resident_bytes": m.approx_resident_bytes, + "measured": m.measured, + } + for m in lane.models + ], "live": { "present": state.present, "replicas": state.replicas, diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index cdd639f..66521a2 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -58,6 +58,58 @@ log = logging.getLogger(__name__) # --- the lanes --------------------------------------------------------------- +GIB = 1024 ** 3 + + +@dataclass(frozen=True) +class ModelRequirement: + """A model a lane must download before it can do anything. + + Surfaced to the UI so the operator is told WHICH model, how big, and what + it costs to hold — before they turn the lane on, not after a multi-GB + download has already started. The lane is optional and its cost is not + obvious from its name, which is the whole reason this is structured data + rather than a sentence in a component. + + `measured=False` means the numbers are ESTIMATES and the UI must say so. + They come from the checkpoint's parameter count and dtype, not from a + build — and a number presented as fact decides whether someone's server + survives, so it is labelled rather than rounded confidently. + """ + + # The Hugging Face repo id, which is the honest answer to "which model". + repo: str + # Roughly what the download costs, for the operator's bandwidth and disk. + approx_download_bytes: int + # Roughly what ONE slot holds while running. Prefork forks a child per + # slot and each loads its own copy, so this multiplies. + approx_resident_bytes: int + measured: bool = False + + +# SigLIP so400m — the only model FabledCurator itself downloads. +# +# What it is for, which is NOT obvious from the lane's name: it produces the +# image embeddings that back similarity search, duplicate grouping and the +# tag heads. WD14 tagging is the GPU AGENT's job, not this lane's — the +# comment in celery_app.py naming both is stale since B3 (#1238), when the +# agent took over and this lane was left as the CPU embed fallback for stacks +# running no agent at all (see MLSettings.cpu_embed_enabled). +# +# Both numbers are ESTIMATES, derived from the checkpoint rather than from a +# build: ~877M parameters at fp32 is ~3.5GB of weights, and holding them plus +# activations and the torch runtime is what the resident figure covers. They +# err high. Replace them with measurements — download the repo and read its +# size; run one embed and read the worker child's VmHWM — and set +# `measured=True` when you do. +SIGLIP_MODEL = ModelRequirement( + repo="google/siglip-so400m-patch14-384", + approx_download_bytes=3_500_000_000, + approx_resident_bytes=4 * GIB, + measured=False, +) + + @dataclass(frozen=True) class Lane: """A worker lane. `name` is the stable key the settings row is keyed on. @@ -87,6 +139,12 @@ class Lane: # True when a slot costs a copy of the ML model rather than just a process. # The only lane whose ceiling is decided by memory instead of by cores. memory_bound: bool = False + # Models this lane downloads the first time it is enabled. Empty for every + # lane that needs none, which is how the UI knows whether to warn at all. + models: tuple[ModelRequirement, ...] = () + # An optional lane is one the product works without. Shown as such, so + # nobody turns on a multi-GB download believing it is required. + optional: bool = False @property def queue_key(self) -> tuple[str, ...]: @@ -142,6 +200,8 @@ LANES: tuple[Lane, ...] = ( default_slots_cap=1, default_enabled=False, memory_bound=True, + models=(SIGLIP_MODEL,), + optional=True, ), ) @@ -167,20 +227,10 @@ _CGROUP_V1_CPU_PERIOD = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us") # a petabyte is a sentinel, not a machine. _UNLIMITED_ABOVE = 1 << 50 -GIB = 1024 ** 3 - -# Memory one ML slot needs: the SigLIP so400m weights plus the runtime holding -# them. Prefork forks a child per slot and each child loads its own copy, so -# this multiplies — it is not a one-off cost. -# -# UNMEASURED AND DELIBERATELY CONSERVATIVE. This number decides whether a -# stranger's server survives enabling tagging, so it errs toward refusing a -# slot that would have fitted rather than granting one that will not. To -# replace it with a real figure: enable the lane on a container with a known -# limit, run one tagging task, and read the worker child's peak RSS -# (`grep VmHWM /proc//status`). Put the measurement in the commit -# message when you do. -ML_BYTES_PER_SLOT = 4 * GIB +# DERIVED from the model requirement above, never restated. The ceiling and +# the number shown to the operator before they enable the lane have to be the +# same figure, or the UI promises something the cap will then refuse. +ML_BYTES_PER_SLOT = SIGLIP_MODEL.approx_resident_bytes # Held back for hypercorn and the non-ML lanes before any ML slot is offered. # In the consolidated container these share one cgroup with ML, and they are diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue index ca0d09e..f60a6e3 100644 --- a/frontend/src/components/settings/WorkerLanesCard.vue +++ b/frontend/src/components/settings/WorkerLanesCard.vue @@ -43,7 +43,15 @@ -
{{ lane.display_name }}
+
+ {{ lane.display_name }} + + + optional + +
+ +
+ {{ lane.display_name }} is optional and currently off +
+

+ It computes image embeddings on the CPU — what similarity search, + duplicate grouping and tag suggestions are built on. You only need it + if you are not running the GPU agent, which does the same + work faster. +

+

Turning it on downloads, once:

+
    +
  • + {{ m.repo }} — + {{ approx(m.measured) }}{{ gb(m.download_bytes) }} to download, + and about {{ gb(m.resident_bytes) }} of RAM for + each slot while it runs. +
    • +
+

+ Each slot loads its own copy, which is why this container caps the + lane at {{ lane.ceiling }}. + +

+ @@ -134,10 +174,25 @@ const busy = ref(null) const notice = ref(null) const lanesList = computed(() => store.lanes?.lanes ?? []) -const mlOff = computed(() => - lanesList.value.some((l) => l.name === 'ml' && !l.enabled), + +// Optional lanes that are OFF — the only ones whose cost the operator has not +// already accepted. Keyed on the lane's own `optional` flag rather than on the +// name 'ml', so a second optional lane gets the same treatment without anyone +// remembering to add it here. +const offOptionalLanes = computed(() => + lanesList.value.filter((l) => l.optional && !l.enabled && l.models?.length), ) +function gb(bytes) { + return `${(bytes / 1024 ** 3).toFixed(1)} GB` +} + +// An unmeasured figure says so. It decides whether someone's server survives, +// and presenting an estimate as a measurement is the failure this guards. +function approx(measured) { + return measured ? '' : 'about ' +} + async function apply(lane, fields) { busy.value = lane.name notice.value = null diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index 7958676..4627ad6 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -266,3 +266,44 @@ def test_worker_lane_check_constraints(slots, cap, ok): # documents what is accepted rather than restating the SQL. satisfied = slots >= 0 and cap >= 0 and slots <= cap assert satisfied is ok + + +# --- what an optional lane tells the operator before it is enabled ----------- + + +def test_the_ml_lane_is_marked_optional_and_declares_its_model(): + """The operator's ask, 2026-09-22: an optional lane must SAY it is + optional, and say what enabling it downloads — which model, how big, and + what it costs to hold — before the switch is thrown rather than after a + multi-GB fetch has begun.""" + ml = wl.LANES_BY_NAME["ml"] + assert ml.optional is True + assert len(ml.models) == 1 + assert ml.models[0].repo == "google/siglip-so400m-patch14-384" + assert ml.models[0].approx_download_bytes > 0 + assert ml.models[0].approx_resident_bytes > 0 + + +def test_no_required_lane_claims_a_model(): + """A lane the product cannot work without must not be gated behind a + download. If one ever needs a model, it stops being required.""" + for lane in wl.LANES: + if lane.models: + assert lane.optional, f"{lane.name} needs a model but is not optional" + + +def test_the_per_slot_ceiling_is_the_same_number_the_ui_shows(): + """DERIVED, not restated. The figure quoted to the operator before they + enable the lane and the figure the cap enforces have to be one number, or + the UI promises a slot the cap will then refuse.""" + ml = wl.LANES_BY_NAME["ml"] + assert wl.ML_BYTES_PER_SLOT == ml.models[0].approx_resident_bytes + + +def test_estimated_numbers_are_flagged_as_estimates(): + """`measured` travels with the figures so the card can say "about". An + estimate presented as a measurement is what decides whether someone's + server survives — it must not be rounded into something that reads like a + fact. Flip this to True in the same commit that records a real + measurement.""" + assert wl.SIGLIP_MODEL.measured is False -- 2.54.0 From ffcd13096ae3ad2f64ca57fb39a265295c638ab8 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:51:44 -0400 Subject: [PATCH 09/94] feat: one image for every lane, with the model fetch gated on enabling (4296) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .forgejo/workflows/build.yml | 8 +++- Dockerfile | 40 ++++++++++++++++- Dockerfile.ml | 43 ------------------- README.md | 2 +- backend/app/scripts/gen_supervisord.py | 30 ++++++------- backend/app/scripts/healthcheck_all.py | 16 +++---- backend/app/services/worker_control.py | 39 +++++++++++++++++ backend/app/tasks/ml.py | 28 ++++++++++++ docker-compose.override.yml | 2 +- entrypoint.sh | 22 +++++++--- .../components/settings/WorkerLanesCard.vue | 8 +++- requirements-ml.txt | 4 +- scripts/artifacts.sh | 8 +++- tests/test_artifact_paths.py | 2 +- tests/test_gen_supervisord.py | 33 +++++++------- 15 files changed, 183 insertions(+), 102 deletions(-) delete mode 100644 Dockerfile.ml diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index eb6aa08..533cf4e 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -1685,7 +1685,13 @@ jobs: uses: docker/build-push-action@v5 with: context: . - file: Dockerfile.ml + # The merged image (milestone 422 step 6). `fabledcurator-ml` keeps + # publishing from it — same bytes under both names — because the + # operator's Swarm stack references fabledcurator-ml:latest and + # lives outside this repo. Dropping the name here would not break + # their deploy, it would freeze it silently at the last publish. + # Retiring the NAME is its own task, gated on that stack moving. + file: Dockerfile push: true # Re-resolve the FROM references against the registry instead of # trusting whatever digest the cache was built against. This is the diff --git a/Dockerfile b/Dockerfile index f0e184d..292a087 100644 --- a/Dockerfile +++ b/Dockerfile @@ -32,13 +32,51 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libwebp7 \ libpng16-16 \ ca-certificates \ + # opencv-python-headless (via requirements-ml.txt) links these even in its + # headless build. Came from Dockerfile.ml when the images merged + # (milestone 422 step 6). + libgl1 \ + libglib2.0-0 \ && rm -rf /var/lib/apt/lists/* WORKDIR /app -COPY requirements.txt ./ +COPY requirements.txt requirements-ml.txt ./ RUN pip install -r requirements.txt +# --- ML, merged from Dockerfile.ml (milestone 422 step 6) -------------------- +# +# ONE image now serves every lane. It was two because the ML lane ran in its +# own container; with the single-container layout (step 5) running every lane +# in one process tree, a second image would mean the `ml` lane could never be +# enabled from the UI — there would be no worker in this container to enable. +# +# The COST, stated because it is real and falls on every adopter: this adds +# torch, torchvision, transformers, onnxruntime and opencv to an image that +# previously carried none of them. Everyone pulls it, including the many who +# will never turn tagging on. That is the trade the milestone accepted for +# being able to offer the lane as a switch rather than a second deployment. +# What it buys back is that nothing downloads a MODEL until the switch is +# thrown — the weights are not baked in, and rule 164 permits that only +# because the feature is optional and clearly off. +# +# CPU-only torch from the PyTorch CPU index. The default PyPI wheel bundles +# the NVIDIA CUDA runtime (~5.6GB of layer) and nothing here uses a GPU — the +# GPU agent is a separate service with its own image. `--index-url`, not +# `--extra-index-url`: the latter would let pip resolve a +cu wheel anyway. +RUN pip install --index-url https://download.pytorch.org/whl/cpu \ + "torch>=2.12,<3.0" "torchvision>=0.27,<0.28" +RUN pip install -r requirements-ml.txt + +# Where the model lands. Deliberately NOT a VOLUME instruction: that mints an +# anonymous volume when nobody mounts one, which survives `docker rm` and +# accumulates 3.5GB copies nobody can find. The compose files mount it +# explicitly instead, so an unmounted run simply re-downloads — visible, and +# recoverable. +ENV HF_HOME=/models/.huggingface \ + TRANSFORMERS_CACHE=/models/.huggingface \ + ML_MODEL_DIR=/models + COPY backend/ ./backend/ COPY alembic/ ./alembic/ COPY alembic.ini ./ diff --git a/Dockerfile.ml b/Dockerfile.ml deleted file mode 100644 index 13efe30..0000000 --- a/Dockerfile.ml +++ /dev/null @@ -1,43 +0,0 @@ -# syntax=docker/dockerfile:1.25 - -FROM python:3.14-slim -ENV PYTHONUNBUFFERED=1 \ - PYTHONDONTWRITEBYTECODE=1 \ - PIP_NO_CACHE_DIR=1 \ - PIP_DISABLE_PIP_VERSION_CHECK=1 \ - HF_HOME=/models/.huggingface \ - TRANSFORMERS_CACHE=/models/.huggingface \ - ML_MODEL_DIR=/models - -RUN apt-get update && apt-get install -y --no-install-recommends \ - ffmpeg \ - libpq5 \ - libjpeg62-turbo \ - libwebp7 \ - libpng16-16 \ - libgl1 \ - libglib2.0-0 \ - ca-certificates \ - && rm -rf /var/lib/apt/lists/* - -WORKDIR /app - -COPY requirements-ml.txt requirements.txt ./ -# CPU-only torch: the default PyPI wheel bundles the CUDA runtime (~5.6GB -# layer); this pipeline never uses a GPU. --index-url (not --extra-index-url) -# guarantees only +cpu wheels are considered, so no nvidia-*-cu12 deps. -RUN pip install --index-url https://download.pytorch.org/whl/cpu \ - "torch>=2.12,<3.0" "torchvision>=0.27,<0.28" -RUN pip install -r requirements-ml.txt - -COPY backend/ ./backend/ -COPY alembic/ ./alembic/ -COPY alembic.ini ./ -COPY entrypoint.sh ./ -RUN chmod +x entrypoint.sh - -# Models self-heal into /models on first start (FC-2 implements this) -VOLUME ["/models"] - -ENTRYPOINT ["./entrypoint.sh"] -CMD ["ml-worker"] diff --git a/README.md b/README.md index 8c94e40..0e0cf55 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ Five deployable pieces, built by `.forgejo/workflows/build.yml`: | Piece | Built from | Image | Role | | --- | --- | --- | --- | | **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. | -| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. | +| **ML worker** | `Dockerfile` | `fabledcurator-ml` | The same image as the web service since milestone 422 — one image serves every lane. Published under this name too, for stacks that still reference it. | | **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. See `agent/README.md`. | | **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. | | **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. | diff --git a/backend/app/scripts/gen_supervisord.py b/backend/app/scripts/gen_supervisord.py index 2747599..cd76171 100644 --- a/backend/app/scripts/gen_supervisord.py +++ b/backend/app/scripts/gen_supervisord.py @@ -36,12 +36,16 @@ puts tini in front of supervisord. Neither choice reaches the application — nothing in FC talks to the supervisor — so this is reversible without touching a line of product code. -## What this does NOT start +## Every lane, including ml -The `ml` lane, unless `--with-ml` is passed. Until step 6 merges the images, -torch and the ML requirements live only in `Dockerfile.ml`, so an `ml` program -in the web image would fail to import on every restart forever. Step 6 is -where that flag turns on. +Step 6 merged the images, so this one carries torch and the ML requirements +and the `ml` lane gets a program like any other. It starts at one slot with +its consumers CANCELLED — `enabled=false` in the seeded settings — so it +holds a process and no model. That matters: `add_consumer` needs a running +worker to reach, and without one the UI switch would have nothing to switch. + +Nothing is downloaded by starting it. The model fetch is enqueued when the +lane is enabled, which is what lets rule 164 permit a runtime fetch at all. """ from __future__ import annotations @@ -69,10 +73,6 @@ STOP_WAIT_SECONDS: dict[str, int] = { } DEFAULT_STOP_WAIT = 60 -# Lanes whose code is not in this image yet. See the module docstring. -_NEEDS_ML_DEPS = frozenset({"ml"}) - - def _program(lane: Lane, *, slots: int) -> str: """One [program:x] block. @@ -141,7 +141,7 @@ def _web_program() -> str: ]) -def render(*, with_ml: bool = False) -> str: +def render() -> str: parts = [ "\n".join([ "[supervisord]", @@ -158,8 +158,6 @@ def render(*, with_ml: bool = False) -> str: ] # Lanes after web, in LANES order, so the log reads in a stable sequence. for lane in LANES: - if lane.name in _NEEDS_ML_DEPS and not with_ml: - continue # A lane configured at zero slots still gets a PROCESS, at one slot # with its consumers cancelled by the reconcile. Without a running # worker there is nothing for `add_consumer` to reach, so enabling the @@ -171,12 +169,8 @@ def render(*, with_ml: bool = False) -> str: def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description=__doc__) - ap.add_argument( - "--with-ml", action="store_true", - help="include the ml lane (only valid once the ML deps are in this image)", - ) - args = ap.parse_args(argv) - sys.stdout.write(render(with_ml=args.with_ml)) + ap.parse_args(argv) + sys.stdout.write(render()) return 0 diff --git a/backend/app/scripts/healthcheck_all.py b/backend/app/scripts/healthcheck_all.py index 4b8076a..8d565e5 100644 --- a/backend/app/scripts/healthcheck_all.py +++ b/backend/app/scripts/healthcheck_all.py @@ -47,14 +47,15 @@ def _web_ok() -> tuple[bool, str]: return False, f"web unreachable: {exc}" -def _lanes_ok(*, with_ml: bool) -> tuple[bool, str]: +def _lanes_ok() -> tuple[bool, str]: from ..services.worker_control import inspect_lanes_sync from ..services.worker_lanes import LANES - expected = { - lane.name for lane in LANES - if with_ml or lane.name != "ml" - } + # Every lane, ml included: one image carries them all since step 6, and a + # disabled lane still runs a process (consumers cancelled), so it answers + # inspect and is healthy. Health is "is the process alive"; whether it + # should be consuming is the reconcile's business. + expected = {lane.name for lane in LANES} live = inspect_lanes_sync() missing = sorted(n for n in expected if not live[n].present) if missing: @@ -63,15 +64,12 @@ def _lanes_ok(*, with_ml: bool) -> tuple[bool, str]: def main(argv: list[str] | None = None) -> int: - argv = sys.argv[1:] if argv is None else argv - with_ml = "--with-ml" in argv - ok, detail = _web_ok() if not ok: print(detail, file=sys.stderr) return 1 - ok, detail = _lanes_ok(with_ml=with_ml) + ok, detail = _lanes_ok() if not ok: print(detail, file=sys.stderr) return 1 diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 022abee..5047e9e 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -408,6 +408,20 @@ async def set_lane( if applied and slots is not None: applied, error = await asyncio.to_thread(set_lane_slots_sync, lane, new_slots) + # Enabling a lane that needs models is what triggers the fetch (milestone + # 422 step 6). Never at boot: that made every start of the ML role reach + # HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a + # feature that is optional and clearly OFF. + # + # Only when the lane actually came on — `enabled is True` rather than + # `new_enabled`, so re-saving slots on an already-enabled lane does not + # re-enqueue. And only when the consumer change landed: enqueueing a task + # onto a queue nothing is consuming would leave it pending with no + # explanation until the lane returns. + fetching = False + if enabled is True and lane.models and applied: + fetching = _enqueue_model_fetch() + return { "name": lane.name, "slots": row.slots, @@ -416,9 +430,34 @@ async def set_lane( "enabled": row.enabled, "applied": applied, "apply_error": error, + # Tells the card to say a download has started rather than leaving the + # operator to wonder why a freshly enabled lane is busy. + "fetching_models": fetching, } +def _enqueue_model_fetch() -> bool: + """Queue the model download. Returns whether it was accepted. + + Import inside the function: `backend.app.tasks.ml` pulls in torch, and web + must not pay that import cost on a module that every settings request + touches. + + Never raises. A broker that will not take the task is worth reporting, but + the SETTING has already been stored and the lane is already enabled — so + failing the whole request here would roll back nothing and tell the + operator their change did not happen when it did. + """ + try: + from ..tasks.ml import ensure_models + + ensure_models.delay() + return True + except Exception: # noqa: BLE001 — reported, never raised at a caller + log.warning("worker_control: could not enqueue the model fetch", exc_info=True) + return False + + def reconcile_lanes_sync(desired: dict[str, tuple[int, bool]]) -> dict: """Drive every RUNNING lane to its stored slots and enabled flag. diff --git a/backend/app/tasks/ml.py b/backend/app/tasks/ml.py index 958b57c..4388820 100644 --- a/backend/app/tasks/ml.py +++ b/backend/app/tasks/ml.py @@ -668,3 +668,31 @@ def scheduled_retract_auto_tags() -> str: with SessionLocal() as session: n_ccip = retract_auto_applied_ccip(session) return f"head={n_head} ccip={n_ccip}" + + +@celery.task(name="backend.app.tasks.ml.ensure_models", bind=True) +def ensure_models(self) -> dict: + """Fetch the models this lane needs, if they are not already present. + + Milestone 422 step 6. This used to run in `entrypoint.sh` before celery + started, which made every boot of the ML role reach HuggingFace for + ~3.5GB — a startup dependency on a third party, for a feature the operator + may never use. Rule 164 permits a runtime fetch only for something + "optional and clearly off", so it moved here: enqueued the moment the lane + is ENABLED, never at boot. + + Being a task rather than a startup step is what makes it visible: it gets + a TaskRun row like any other, so the download shows in Activity with a + duration and a status, and a failure is something the operator can see and + retry rather than a container that quietly never became useful. + + Idempotent — `download_models` fetches only what is missing — so enabling + an already-provisioned lane costs one no-op task rather than a re-download. + That matters because the reconcile may enqueue it again. + """ + from ..scripts.download_models import main as download + + rc = download() + if rc != 0: + raise RuntimeError(f"model download failed with exit code {rc}") + return {"ok": True} diff --git a/docker-compose.override.yml b/docker-compose.override.yml index aefd81f..e397e92 100644 --- a/docker-compose.override.yml +++ b/docker-compose.override.yml @@ -47,7 +47,7 @@ services: ml-worker: build: context: . - dockerfile: Dockerfile.ml + dockerfile: Dockerfile environment: LOG_LEVEL: DEBUG volumes: diff --git a/entrypoint.sh b/entrypoint.sh index b03b252..f24f007 100755 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -50,13 +50,23 @@ case "$ROLE" in ;; ml-worker) - echo "[entrypoint] Ensuring ML models present in /models..." - python -m backend.app.scripts.download_models - echo "[entrypoint] Starting ML Celery worker (ml queue)" + # NO MODEL DOWNLOAD HERE (milestone 422 step 6). This used to run + # download_models before celery started, which made every boot of this + # role reach HuggingFace for ~3.5GB. Rule 164 permits a runtime fetch only + # for a feature that is "optional and clearly off" — so the fetch moved to + # the moment the operator ENABLES the lane, where it is visible, retryable + # and attributable, instead of being a silent precondition of starting. + # + # The worker therefore starts with no model present, which is correct: it + # is not consuming the ml queue until the lane is enabled, and enabling it + # is what enqueues ensure_models. + QUEUES="${CELERY_QUEUES:-ml}" + CONCURRENCY="${CELERY_CONCURRENCY:-1}" + echo "[entrypoint] Starting ML Celery worker queues=$QUEUES concurrency=$CONCURRENCY" exec celery -A backend.app.celery_app:celery worker \ --loglevel=info \ - -Q ml \ - --concurrency=1 + -Q "$QUEUES" \ + --concurrency="$CONCURRENCY" ;; all) @@ -75,7 +85,7 @@ case "$ROLE" in # adjust a baseline that already works, and can never prevent a boot. CONF="${SUPERVISOR_CONF:-/tmp/supervisord.conf}" echo "[entrypoint] Generating $CONF from the lane table" - python -m backend.app.scripts.gen_supervisord ${FC_WITH_ML:+--with-ml} > "$CONF" + python -m backend.app.scripts.gen_supervisord > "$CONF" echo "[entrypoint] Starting supervisord (web + worker lanes)" exec supervisord -c "$CONF" ;; diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue index f60a6e3..14999f4 100644 --- a/frontend/src/components/settings/WorkerLanesCard.vue +++ b/frontend/src/components/settings/WorkerLanesCard.vue @@ -201,7 +201,13 @@ async function apply(lane, fields) { // Saved but not pushed — the lane is restarting, or the broker blipped. // NOT an error: the reconcile carries it when the lane answers again, and // saying "failed" would invite the operator to set it a second time. - if (reply && reply.applied === false) { + if (reply && reply.fetching_models) { + notice.value = { + type: 'info', + text: `${lane.display_name} is on. Downloading its model now — ` + + 'watch progress in Queues + workers above. It only happens once.', + } + } else if (reply && reply.applied === false) { notice.value = { type: 'info', text: `Saved. ${lane.display_name} is not answering right now — ` diff --git a/requirements-ml.txt b/requirements-ml.txt index c216487..52e20e5 100644 --- a/requirements-ml.txt +++ b/requirements-ml.txt @@ -3,9 +3,9 @@ # ML stack — versions current as of 2026-05-14 with Python 3.14 wheel coverage. # torch + torchvision are NOT listed here: they are installed CPU-only from -# the PyTorch CPU index in Dockerfile.ml. The default PyPI torch wheel bundles +# the PyTorch CPU index in Dockerfile. The default PyPI torch wheel bundles # the NVIDIA CUDA runtime (a ~5.6GB image layer); this pipeline is CPU-only, -# so Dockerfile.ml uses the +cpu wheels from +# so Dockerfile uses the +cpu wheels from # https://download.pytorch.org/whl/cpu instead. # # IMPORTANT: torchvision 0.27 declares requires_python "!=3.14.1,>=3.10" — diff --git a/scripts/artifacts.sh b/scripts/artifacts.sh index e93b9eb..fcce132 100755 --- a/scripts/artifacts.sh +++ b/scripts/artifacts.sh @@ -51,9 +51,13 @@ ROOT=$(git rev-parse --show-toplevel) # rather than restated — one definition, per #2397. WEB_PATHS='Dockerfile requirements.txt backend alembic alembic.ini entrypoint.sh frontend :(exclude)frontend/test :(exclude)frontend/test/**' -# ml (Dockerfile.ml, context `.`) — no frontend, no extension. Note it copies +# ml (Dockerfile, context `.`) — no frontend, no extension. Note it copies # BOTH requirements-ml.txt and requirements.txt. -ML_PATHS='Dockerfile.ml requirements-ml.txt requirements.txt backend alembic alembic.ini entrypoint.sh' +# Dockerfile, not Dockerfile.ml: the images merged at milestone 422 step 6 +# and Dockerfile.ml is gone. A path set naming a deleted file silently +# stops contributing to the derived revision, which is what the reuse +# check and the version string both read (#3202's shape). +ML_PATHS='Dockerfile requirements-ml.txt requirements.txt backend alembic alembic.ini entrypoint.sh' # agent (agent/Dockerfile, context `agent`) — copies requirements.txt and # fc_agent only. agent/README.md, agent/docker-compose.yml and agent/ruff.toml diff --git a/tests/test_artifact_paths.py b/tests/test_artifact_paths.py index 888d4c3..22b4c51 100644 --- a/tests/test_artifact_paths.py +++ b/tests/test_artifact_paths.py @@ -31,7 +31,7 @@ ROOT = Path(__file__).resolve().parent.parent # artifact -> (dockerfile, build context relative to the repo root) ARTIFACTS = { "web": ("Dockerfile", ""), - "ml": ("Dockerfile.ml", ""), + "ml": ("Dockerfile", ""), "agent": ("agent/Dockerfile", "agent"), } diff --git a/tests/test_gen_supervisord.py b/tests/test_gen_supervisord.py index b92cce3..7f27745 100644 --- a/tests/test_gen_supervisord.py +++ b/tests/test_gen_supervisord.py @@ -37,20 +37,21 @@ def test_it_is_valid_ini_with_a_supervisord_section(): assert cp.get("supervisord", "nodaemon") == "true" -def test_web_and_every_non_ml_lane_get_a_program(): +def test_every_lane_gets_a_program(): + """One image carries every lane since step 6, so nothing is conditional. + A lane in LANES with no program is a queue with no consumer.""" cp = _parse() - expected = {"program:web"} | { - f"program:{lane.name}" for lane in LANES if lane.name != "ml" - } + expected = {"program:web"} | {f"program:{lane.name}" for lane in LANES} assert set(cp.sections()) - {"supervisord"} == expected -def test_the_ml_lane_is_absent_until_its_deps_are_in_the_image(): - """The web image has no torch. An `ml` program here would fail to import - on every restart, forever — startretries would give up and the lane would - be permanently dead while the container reported healthy.""" - assert not _parse().has_section("program:ml") - assert _parse(with_ml=True).has_section("program:ml") +def test_the_ml_lane_runs_even_though_it_ships_disabled(): + """It holds a PROCESS and no model. `add_consumer` needs a running worker + to reach, so without this the UI switch would have nothing to switch — + and nothing is downloaded by starting it, which is what lets rule 164 + permit the fetch at all.""" + assert _parse().has_section("program:ml") + assert LANES_BY_NAME["ml"].default_enabled is False # --- the coupling this generator exists to guarantee ------------------------- @@ -60,7 +61,7 @@ def test_each_program_serves_exactly_its_lane_s_queues(): """The whole point: the container's processes and the application's lane table are one list. A queue in LANES with no program means work that queues forever with nothing consuming it.""" - cp = _parse(with_ml=True) + cp = _parse() for lane in LANES: env = cp.get(f"program:{lane.name}", "environment") # The QUOTED form. supervisord splits `environment` on commas, so an @@ -75,7 +76,7 @@ def test_each_program_invokes_the_lane_s_entrypoint_role_not_its_name(): queue — exactly as docker-compose starts it today. Invoking `entrypoint.sh maintenance_long` would hit the unknown-role branch and exit 1 on every restart.""" - cp = _parse(with_ml=True) + cp = _parse() for lane in LANES: command = cp.get(f"program:{lane.name}", "command") assert f"entrypoint.sh {lane.entrypoint_role}" in command @@ -104,7 +105,7 @@ def test_every_program_signals_its_whole_process_group(): parent leaves them running and holding tasks — a 'graceful' shutdown that orphans workers. The `sh -c … | sed` wrapper makes this doubly necessary: without it the signal reaches the shell holding the pipeline, not celery.""" - cp = _parse(with_ml=True) + cp = _parse() for section in cp.sections(): if not section.startswith("program:"): continue @@ -133,7 +134,7 @@ def test_no_program_waits_longer_than_the_compose_stop_grace_period(): assert m, "docker-compose.single.yml has no stop_grace_period" grace = int(m.group(1)) - cp = _parse(with_ml=True) + cp = _parse() for section in cp.sections(): if section.startswith("program:"): assert cp.getint(section, "stopwaitsecs") <= grace, section @@ -146,7 +147,7 @@ def test_a_zero_slot_lane_still_gets_a_running_process(): """ML ships at 0 slots and disabled — but `add_consumer` needs something to reach. With no process there would be nothing for the UI switch to switch, and enabling tagging could not work at all.""" - cp = _parse(with_ml=True) + cp = _parse() assert LANES_BY_NAME["ml"].default_slots == 0 env = cp.get("program:ml", "environment") assert "CELERY_CONCURRENCY=1" in env @@ -174,7 +175,7 @@ def test_every_program_writes_to_the_container_stdout_with_its_lane_named(): """Four celery workers and hypercorn on one stream are indistinguishable without this. Unbuffered (`maxbytes 0`) so `docker logs` is live rather than arriving in rotated chunks.""" - cp = _parse(with_ml=True) + cp = _parse() for section in cp.sections(): if not section.startswith("program:"): continue -- 2.54.0 From f174981b070fa830ed27e2a8a5540d6fda1630c2 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 08:57:09 -0400 Subject: [PATCH 10/94] fix: ml and web are one image, so they are one path set (4296) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7271. `test_every_copied_path_is_in_the_artifacts_path_set` — a guard I did not know this repo had — caught the merge's real consequence within minutes, and it is precisely #3202's class: ml: Dockerfile copies 'frontend/package.json' into the ml image, but no include in scripts/artifacts.sh covers it. web: Dockerfile copies 'requirements-ml.txt' into the web image, but no include covers it. Both follow from step 6 and neither was visible by reading my own diff. The merged Dockerfile carries the frontend-builder stage, so the ml image now copies frontend files; and the web image now installs the ML requirements. Left as they were, each artifact's derived version would have stopped moving when those files changed, and a pinned build would serve stale bytes — silently, since nothing fails when a version simply does not advance. The fix is structural rather than two more entries. `fabledcurator-ml` IS web's image now: same Dockerfile, same context, same bytes, published under a second name only because the operator's Swarm stack still references it. So ML_PATHS is WEB_PATHS by assignment, and `cmd_paths` gives ml the same deriver and extension append — the XPI is in those bytes too. Two lists describing one image is the duplication this milestone has been collapsing all day. It existed for about an hour and the guard found it first, which is the argument for the guard. I also predicted this failure would be the image build. It was not; the build passed and the unit lane failed. Worth noting because the prediction was confident and wrong, and reading the log took one call. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- scripts/artifacts.sh | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/scripts/artifacts.sh b/scripts/artifacts.sh index fcce132..6a3df2f 100755 --- a/scripts/artifacts.sh +++ b/scripts/artifacts.sh @@ -49,15 +49,22 @@ ROOT=$(git rev-parse --show-toplevel) # frontend/public/extension/ before the docker build), so an extension change # changes the web image. The extension's packaged set is appended in cmd_paths # rather than restated — one definition, per #2397. -WEB_PATHS='Dockerfile requirements.txt backend alembic alembic.ini entrypoint.sh frontend :(exclude)frontend/test :(exclude)frontend/test/**' +WEB_PATHS='Dockerfile requirements.txt requirements-ml.txt backend alembic alembic.ini entrypoint.sh frontend :(exclude)frontend/test :(exclude)frontend/test/**' -# ml (Dockerfile, context `.`) — no frontend, no extension. Note it copies -# BOTH requirements-ml.txt and requirements.txt. -# Dockerfile, not Dockerfile.ml: the images merged at milestone 422 step 6 -# and Dockerfile.ml is gone. A path set naming a deleted file silently -# stops contributing to the derived revision, which is what the reuse -# check and the version string both read (#3202's shape). -ML_PATHS='Dockerfile requirements-ml.txt requirements.txt backend alembic alembic.ini entrypoint.sh' +# ml — THE SAME IMAGE as web since milestone 422 step 6, built from the same +# Dockerfile with the same context and published under a second name only +# because the operator's Swarm stack still references it. +# +# So it is the same path set, by assignment rather than by a copy that would +# drift. This was two lists describing one image for about an hour, and the +# artifact-paths guard caught it immediately: the merged Dockerfile carries +# the frontend-builder stage, so the ml image copies `frontend/package.json`, +# and the old ml list did not cover it — its version would not have moved when +# the frontend changed, and a pinned build would serve stale bytes (#3202). +# +# `requirements-ml.txt` is in WEB_PATHS for the mirror-image reason: the web +# image now installs the ML requirements, so changing them changes it. +ML_PATHS="$WEB_PATHS" # agent (agent/Dockerfile, context `agent`) — copies requirements.txt and # fc_agent only. agent/README.md, agent/docker-compose.yml and agent/ruff.toml @@ -108,7 +115,9 @@ ext_paths() { cmd_paths() { case "$1" in web) echo "$WEB_PATHS $DERIVER $(ext_paths)" ;; - ml) echo "$ML_PATHS" ;; + # Identical to web, deliberately: it IS web's image. That includes the + # deriver and the bundled extension — the XPI is in these bytes too. + ml) echo "$ML_PATHS $DERIVER $(ext_paths)" ;; agent) echo "$AGENT_PATHS" ;; extension) ext_paths ;; *) usage ;; -- 2.54.0 From 0f98e46200870c469208a3cd5a24102c098a1e7a Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 09:02:47 -0400 Subject: [PATCH 11/94] =?UTF-8?q?docs:=20the=20merged=20image's=20cost,=20?= =?UTF-8?q?measured=20=E2=80=94=20and=20it=20corrects=20my=20own=20estimat?= =?UTF-8?q?e=20(4296)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I wrote in ffcd130's Dockerfile comment that merging ML in means "everyone pulls it, including the many who will never turn tagging on", framed as a real cost the milestone accepted. My working estimate behind that was ~4GB. Measured from run 7273's build-web log: torch 2.12.1+cpu wheel 192.3 MB torchvision 0.27.1+cpu 1.8 MB transformers / onnxruntime / opencv / sklearn and friends 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB largest newly-pushed layer 222.07 MB The ML code adds a few HUNDRED MB, not gigabytes. The `--index-url` CPU resolution is what makes that true — the default PyPI torch wheel carries the CUDA runtime and is ~2GB by itself, and the log confirms 2.12.1+cpu resolved, so it is working as intended rather than as intended-but-unverified. Why this matters beyond a comment being wrong: it settles the trade this step was explicitly asked to weigh and could not, and it reverses how close the call looked. Baking the weights in adds ~3.5GB to every pull for a feature many adopters never enable; shipping the code and fetching on demand adds ~350MB. An order of magnitude, where the estimate had them within 15% of each other. Off-by-default is not a judgement call here, it is arithmetic. The gigabytes were always in the MODEL, and the model is not in the image. Two things NOT measured, still: the total image size (the push only transfers layers the registry lacks, so a push log cannot give it) and the per-slot resident RAM, which stays flagged `measured=False` in the lane table and renders as "about" in the UI. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- Dockerfile | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/Dockerfile b/Dockerfile index 292a087..1cffc96 100644 --- a/Dockerfile +++ b/Dockerfile @@ -51,19 +51,35 @@ RUN pip install -r requirements.txt # in one process tree, a second image would mean the `ml` lane could never be # enabled from the UI — there would be no worker in this container to enable. # -# The COST, stated because it is real and falls on every adopter: this adds -# torch, torchvision, transformers, onnxruntime and opencv to an image that -# previously carried none of them. Everyone pulls it, including the many who -# will never turn tagging on. That is the trade the milestone accepted for -# being able to offer the lane as a switch rather than a second deployment. -# What it buys back is that nothing downloads a MODEL until the switch is -# thrown — the weights are not baked in, and rule 164 permits that only -# because the feature is optional and clearly off. +# THE COST, MEASURED from run 7273 rather than guessed — and it is far +# smaller than the estimate this comment first carried, which said "everyone +# pulls ~4GB": # -# CPU-only torch from the PyTorch CPU index. The default PyPI wheel bundles -# the NVIDIA CUDA runtime (~5.6GB of layer) and nothing here uses a GPU — the -# GPU agent is a separate service with its own image. `--index-url`, not -# `--extra-index-url`: the latter would let pip resolve a +cu wheel anyway. +# torch 2.12.1+cpu wheel 192.3 MB +# torchvision 0.27.1+cpu 1.8 MB +# transformers / onnxruntime / opencv / sklearn and friends +# 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB +# largest newly-pushed layer 222.07 MB +# +# So the ML code adds a few hundred MB to the pull, not gigabytes. The CPU +# index is what makes that true: the default PyPI torch wheel bundles the +# NVIDIA CUDA runtime and is ~2GB on its own. +# +# The GIGABYTES are in the MODEL — ~3.5GB of SigLIP weights — and those are +# NOT in this image. They arrive only when the operator enables the lane, +# which is what lets rule 164 permit a runtime fetch at all ("optional and +# clearly off"). That also settles the trade this step was asked to weigh: +# baking the weights in would add ~3.5GB to every pull for a feature many +# adopters never enable, against ~350MB for the code that makes the switch +# available. Off-by-default wins by an order of magnitude, which was NOT +# obvious before measuring — the estimate had the two costs within 15% of +# each other. +# +# `--index-url`, not `--extra-index-url`: the latter would let pip resolve a +# +cu wheel anyway, and the whole saving above depends on it not doing that. +# +# CPU-only torch from the PyTorch CPU index. Nothing here uses a GPU — the +# GPU agent is a separate service with its own image. RUN pip install --index-url https://download.pytorch.org/whl/cpu \ "torch>=2.12,<3.0" "torchvision>=0.27,<0.28" RUN pip install -r requirements-ml.txt -- 2.54.0 From 5ca1058fb5ffb29bf84412a19c7fa5cad6c637a9 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 09:50:25 -0400 Subject: [PATCH 12/94] test: the smoke runs with egress blocked, on every build, and proves the block (4296) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 6's rule-164 half. THE PROPERTY WAS NEVER TESTED, not even weekly. smoke-web already booted the built image against real Postgres and Redis — but on the runner's default network, with the internet one hop away. It proved the image WORKS; it never proved it works OFFLINE, which is the thing rule 164 is about and the thing step 6 put at risk by moving download_models out of boot. Now on an `--internal` docker network, which is the mechanism rule 164's own verify_with names. `--network none` is explicitly the wrong check here: it would only prove the app fails without a database, which says nothing about egress. Internal blocks the default route while leaving container-to-container traffic and embedded DNS intact, so Postgres and Redis stay reachable and nothing else does. The service containers are runner-created siblings, so they are ATTACHED to the network rather than created on it, and their addresses are re-read on it — the bridge IPs discovered earlier are not routable from a container that is only on the internal network. A STEP THAT PROVES THE SANDBOX IS SEALED, before anything depends on it. It tries to reach 1.1.1.1:443 from inside the candidate and fails the job if it succeeds. Without it the rest is theatre: if `--internal` silently stopped working, or the container picked up a second network, every check below would pass with the internet available and report an offline boot that never happened. A guard that cannot fail is not a guard (rule 167). IT RUNS ON EVERY BUILD, not just the weekly refresh. The egress property is broken by a code or Dockerfile change — a push — so checking it only on the refresh would test it on the one trigger that changes no source. Addressed by the DIGEST build-web published rather than by a tag: a tag can move between the build and the smoke, and then the check reports on bytes nobody built here. A reuse hit is skipped, because those bytes were smoked when built. WHAT THIS STILL IS NOT, filed as #4310: on a push it runs AFTER build-web has written the channel tag, so it detects rather than gates. Rule 164 asks for the check BETWEEN build and push. Closing that means giving the push path the candidate-then-promote shape the refresh already has — per-channel candidate tags, promote learning its channel, and the :c- repoint moving after the gate. That is a redesign of the path that publishes production and it is not something to fold into a test change. Also filed #4311: retiring the fabledcurator-ml image NAME, gated on the operator moving their Swarm stack file. Same two-phase shape #406 used for pixiv, for the same reason — dropping it would not break their deploy, it would freeze it silently. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .forgejo/workflows/build.yml | 96 ++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 4 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 533cf4e..ce3bf76 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -502,6 +502,10 @@ jobs: # to know — a candidate was published — rather than restating why. outputs: candidate: ${{ steps.reuse.outputs.promote }} + # The manifest THIS run pushed, empty on a reuse hit. smoke-web addresses + # it by digest rather than by tag: a tag can move between the build and + # the smoke, and then the check reports on bytes nobody built here. + digest: ${{ steps.build.outputs.digest }} # A plain `needs` — no `always()`. That expression existed to let a # SKIPPED sign-extension through on a tag push while still blocking a # FAILED one. With no tag trigger, sign-extension always runs, so the @@ -1152,7 +1156,25 @@ jobs: # publish. smoke-web: needs: [build-web] - if: needs.build-web.outputs.candidate == 'true' + # Every run that actually BUILT something, not just the weekly refresh. + # The egress property (rule 164) is broken by a code or Dockerfile change, + # which is a push — checking it only on the refresh would test it on the + # one trigger that changes no source. + # + # A reuse hit is skipped deliberately: those bytes are already published + # and were smoked when they were built. Re-smoking them would burn two + # minutes to re-learn a fact. + # + # HONEST LIMIT, and it is the reason #4299 exists: on a push this runs + # AFTER build-web has written the channel tag, so it detects rather than + # gates. Rule 164's verify_with asks for the check BETWEEN build and push. + # Closing that needs the push path to adopt the candidate-then-promote + # shape the refresh already has — per-channel candidate tags, promote + # learning its channel, and the :c- repoint moving after the gate. + # That is a redesign of the production publish path and is its own task. + if: >- + needs.build-web.outputs.candidate == 'true' + || needs.build-web.outputs.digest != '' runs-on: python-ci container: image: git.fabledsword.com/bvandeusen/ci-python:3.14 @@ -1193,6 +1215,7 @@ jobs: env: TOKEN: ${{ secrets.RELEASE_TOKEN }} ACTOR: ${{ github.actor }} + BUILT_DIGEST: ${{ needs.build-web.outputs.digest }} run: | set -eux # Service discovery mirrors ci.yml's integration lane: these jobs run @@ -1222,10 +1245,59 @@ jobs: fi echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin - CANDIDATE="$IMAGE:refresh-candidate" + # A refresh publishes to the candidate tag; a push writes the + # channel tag directly and hands us its digest. Address the digest + # where we have one — it names the exact manifest this run built, + # which a tag stops doing the moment anything else moves it. + if [ -n "${BUILT_DIGEST:-}" ]; then + CANDIDATE="$IMAGE@$BUILT_DIGEST" + else + CANDIDATE="$IMAGE:refresh-candidate" + fi docker pull "$CANDIDATE" - ENVOPTS="-e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP" + # --- EGRESS BLOCKED from here (rule 164) --------------------------- + # + # Rule 164 requires a deployed instance to start and serve its full + # UI with NO outbound internet, and says to verify it by removing the + # network rather than by reading the code. Until now this job proved + # the image WORKS; it never proved it works OFFLINE, because every + # container below ran on the runner's default network with the + # internet one hop away. + # + # That gap became load-bearing at milestone 422 step 6. The ML role + # used to run `download_models` before celery started — a boot that + # reached HuggingFace for ~3.5GB — and that fetch moved to a task + # enqueued when the lane is enabled. This check is what proves it + # actually moved, rather than proving it on the machine that built it + # where the model is already cached. + # + # `--internal` is the mechanism rule 164's own verify_with names, and + # `--network none` is explicitly the WRONG check here: it would only + # prove the app fails without a database, which proves nothing about + # egress. An internal network blocks the default route while leaving + # container-to-container traffic and embedded DNS intact, so Postgres + # and Redis stay reachable and nothing else is. + # + # The service containers are SIBLINGS created by the runner, so they + # are attached to the internal network rather than created on it. + # They keep their original network too — that is fine, since what + # must be offline is the APP container, and it is created with only + # this network. + NET=smoke-noegress-$$ + docker network create --internal "$NET" + trap 'docker network rm "$NET" >/dev/null 2>&1 || true' EXIT + docker network connect "$NET" "$PG" + docker network connect "$NET" "$RD" + # Re-read the addresses ON THIS NETWORK. The IPs discovered above + # belong to the runner's default bridge and are not routable from a + # container that is only on the internal one. + PG_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$PG") + RD_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$RD") + test -n "$PG_IP" && test -n "$RD_IP" + + ENVOPTS="--network $NET" + ENVOPTS="$ENVOPTS -e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP" ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY" ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0" ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0" @@ -1238,6 +1310,22 @@ jobs: # user-facing file mentions this variable at all. ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1" + # 0. PROVE the network is actually blocking egress. Without this the + # rest is theatre: if `--internal` silently stopped working, or + # the app container picked up a second network, every check below + # would pass with the internet available and report an offline + # boot that never happened. A guard that cannot fail is not a + # guard (rule 167). + echo "smoke: confirming the sandbox has no route out" + if docker run --rm --network "$NET" "$CANDIDATE" shell -c \ + 'python3 -c "import socket,sys; s=socket.socket(); s.settimeout(5); sys.exit(0 if s.connect_ex((\"1.1.1.1\", 443)) == 0 else 1)"'; then + echo "smoke: FAILED — the sandbox reached 1.1.1.1:443." >&2 + echo "smoke: the network is NOT internal, so nothing below would" >&2 + echo "smoke: have tested the offline property (rule 164)." >&2 + exit 1 + fi + echo "smoke: no route out, as required" + # 1. The schema builds from empty, using the image's OWN libpq and # psycopg. This is the same call entrypoint.sh makes before it # serves anything, so a failure here is a failure to boot. @@ -1296,7 +1384,7 @@ jobs: curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" echo - echo "smoke: all checks passed against $CANDIDATE" + echo "smoke: all checks passed against $CANDIDATE, with egress blocked" # Move the channel tags — the whole point of the gate. # -- 2.54.0 From a01165365b182d84e37b0adc6c6961db875116de Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 22 Sep 2026 10:05:43 -0400 Subject: [PATCH 13/94] feat: a saturated lane can grow itself, within the cap the operator set (4297) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../versions/0104_worker_lane_autoscale.py | 51 ++++ backend/app/api/workers.py | 11 +- backend/app/celery_app.py | 8 + backend/app/models/worker_lane.py | 12 + backend/app/services/worker_control.py | 255 +++++++++++++++++- backend/app/services/worker_lanes.py | 5 + backend/app/tasks/maintenance.py | 65 ++++- .../components/settings/WorkerLanesCard.vue | 48 +++- frontend/src/stores/systemActivity.js | 23 ++ frontend/test/workerLanes.spec.js | 53 +++- tests/test_worker_control.py | 242 +++++++++++++++++ 11 files changed, 754 insertions(+), 19 deletions(-) create mode 100644 alembic/versions/0104_worker_lane_autoscale.py diff --git a/alembic/versions/0104_worker_lane_autoscale.py b/alembic/versions/0104_worker_lane_autoscale.py new file mode 100644 index 0000000..38e74d0 --- /dev/null +++ b/alembic/versions/0104_worker_lane_autoscale.py @@ -0,0 +1,51 @@ +"""worker_lane.autoscale — may this lane grow itself? + +Milestone 422 step 7. One boolean, defaulting FALSE on every existing row and +on every new one. + +## Why the default is false and not "sensible" + +This is the only part of the milestone that acts without anyone watching. The +manual dial (step 4) and the reconcile (step 3) both do exactly what someone +asked for; this one decides. Shipping it on would mean every install starts +with a process that changes its own resource usage based on a heuristic tuned +against nobody's workload. + +Off also makes the failure mode benign: if the signal is wrong, nothing +happens until an operator opts a lane in, and they opted in while watching. + +## Why per lane and not one global switch + +The lanes are not alike in what a slot costs. A `worker` slot is a process; +an `ml` slot is another copy of a ~3.5GB model. A global switch would enable +growth on a lane whose behaviour under load nobody has observed, and the one +it would hurt most is the one whose cost is least visible. + +Revision ID: 0104 +Revises: 0103 +Create Date: 2026-09-22 + +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "0104" +down_revision: Union[str, None] = "0103" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "worker_lane", + sa.Column( + "autoscale", sa.Boolean(), + server_default=sa.text("false"), nullable=False, + ), + ) + + +def downgrade() -> None: + op.drop_column("worker_lane", "autoscale") diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py index b5f96f0..678c061 100644 --- a/backend/app/api/workers.py +++ b/backend/app/api/workers.py @@ -84,15 +84,16 @@ async def update_lane(name: str): if not isinstance(value, int) or isinstance(value, bool): return _bad("invalid_body", detail=f"{key} must be an integer") fields[key] = value - if "enabled" in body: - if not isinstance(body["enabled"], bool): - return _bad("invalid_body", detail="enabled must be a boolean") - fields["enabled"] = body["enabled"] + for key in ("enabled", "autoscale"): + if key in body: + if not isinstance(body[key], bool): + return _bad("invalid_body", detail=f"{key} must be a boolean") + fields[key] = body[key] if not fields: return _bad( "invalid_body", - detail="give at least one of slots, slots_cap, enabled", + detail="give at least one of slots, slots_cap, enabled, autoscale", ) async with get_session() as session: diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index 894fcd1..e509f8f 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -111,6 +111,14 @@ def make_celery() -> Celery: "task": "backend.app.tasks.maintenance.recover_interrupted_tasks", "schedule": 300.0, # every 5 minutes }, + "autoscale-worker-lanes": { + "task": "backend.app.tasks.maintenance.autoscale_worker_lanes", + "schedule": 60.0, # every minute — it reacts to a BACKLOG, and + # a five-minute reaction to a queue filling up is no reaction. + # Cheap when nothing opted in (one DB read, then nothing) and + # cheap when settled (one inspect + one LLEN sweep, no control + # messages), so the short interval costs little. + }, "reconcile-worker-lanes": { "task": "backend.app.tasks.maintenance.reconcile_worker_lanes", "schedule": 300.0, # every 5 minutes — the window in which a diff --git a/backend/app/models/worker_lane.py b/backend/app/models/worker_lane.py index ae3cc09..94daf45 100644 --- a/backend/app/models/worker_lane.py +++ b/backend/app/models/worker_lane.py @@ -78,6 +78,18 @@ class WorkerLane(Base): slots_cap: Mapped[int] = mapped_column(Integer, nullable=False) enabled: Mapped[bool] = mapped_column(Boolean, nullable=False) + # Whether the autoscaler may raise this lane's slots on its own, up to + # slots_cap and never past it (milestone 422 step 7). + # + # OFF by default and per-lane rather than global. It is the one part of + # this milestone that acts unattended, so it is opt-in per lane the + # operator has actually thought about — a global switch would turn it on + # for lanes whose behaviour under load nobody has watched, including `ml`, + # where every extra slot is another copy of a multi-GB model. + autoscale: Mapped[bool] = mapped_column( + Boolean, nullable=False, server_default="false", + ) + updated_at: Mapped[datetime] = mapped_column( DateTime(timezone=True), nullable=False, diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 5047e9e..157b01b 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -50,11 +50,12 @@ from __future__ import annotations import asyncio import logging from dataclasses import dataclass, field +from datetime import UTC, datetime -from sqlalchemy import select +from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession -from ..models import WorkerLane +from ..models import TaskRun, WorkerLane from .worker_lanes import LANES, LANES_BY_QUEUE_KEY, Lane, derived_ceiling log = logging.getLogger(__name__) @@ -103,6 +104,16 @@ class LaneLiveState: no replica reported — unknown, never zero.""" return max(self.pools.values()) if self.pools else None + @property + def capacity(self) -> int: + """Total slots across replicas — how many tasks this lane can run at + once. Distinct from `pool`, and the two must not be confused: `pool` + is the DIAL (one process's size, what grow/shrink move), `capacity` is + the CAPABILITY. Asking "is this lane saturated" compares `active`, + which is summed across replicas, against this — against `pool` it + would call two half-busy replicas of 4 saturated at 4 active.""" + return sum(self.pools.values()) + def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None: return LANES_BY_QUEUE_KEY.get(tuple(sorted(queues))) @@ -260,6 +271,7 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]: slots=lane.default_slots, slots_cap=lane.default_slots_cap, enabled=lane.default_enabled, + autoscale=lane.default_autoscale, ) session.add(row) rows[lane.name] = row @@ -278,7 +290,9 @@ async def lane_view(session: AsyncSession) -> list[dict]: rows = await _rows_by_name(session) live = await asyncio.to_thread(inspect_lanes_sync) depths = await asyncio.to_thread(_queue_depths_sync) + oldest = await _oldest_running_by_queue(session) + now = datetime.now(UTC) out = [] for lane in LANES: row = rows[lane.name] @@ -297,6 +311,7 @@ async def lane_view(session: AsyncSession) -> list[dict]: "slots_cap": row.slots_cap, "ceiling": derived_ceiling(lane), "enabled": row.enabled, + "autoscale": row.autoscale, "memory_bound": lane.memory_bound, "optional": lane.optional, # What enabling this lane will download, so the UI can say WHICH @@ -321,10 +336,55 @@ async def lane_view(session: AsyncSession) -> list[dict]: }, "queue_depth": depth, "pending": None if depth is None else depth + state.reserved, + # How long the oldest still-running task on this lane has been + # going, in minutes. The operator asked for a trigger here — grow + # a lane whose tasks run past some duration — and it stayed a + # REPORT: a long task does not finish sooner because the lane + # gained a slot, so scaling on it would spend memory to change + # nothing. Shown so they can see a lane wedged on one slow job, + # which is the genuinely useful half of the idea. + "oldest_running_minutes": _minutes_since( + min( + (oldest[q] for q in lane.queues if q in oldest), + default=None, + ), + now, + ), }) return out +async def _oldest_running_by_queue(session: AsyncSession) -> dict[str, datetime]: + """When the longest-running unfinished task on each queue started. + + Read from `task_run`, which is OUR OWN table on OUR OWN wall clock, and + deliberately not from celery's `inspect active()`. Those entries carry a + `time_start` taken from the WORKER's `time.monotonic()` — a clock with an + arbitrary origin per process. Subtracting it from this process's wall + clock produces a number that looks like a duration and is meaningless, and + it would be meaningless in the direction that matters: plausible. + + `task_run` also already carries the per-queue staleness thresholds the + recovery sweep uses, so a row still `running` here is one the system + itself considers legitimately in flight rather than abandoned. + """ + result = await session.execute( + select(TaskRun.queue, func.min(TaskRun.started_at)) + .where(TaskRun.status == "running", TaskRun.finished_at.is_(None)) + .group_by(TaskRun.queue) + ) + return {queue: started for queue, started in result if started is not None} + + +def _minutes_since(started: datetime | None, now: datetime) -> int | None: + """Whole minutes, or None when nothing is running. Never negative: a row + written by a container whose clock is a few seconds ahead must read as 0 + rather than as a task that starts in the future.""" + if started is None: + return None + return max(0, int((now - started).total_seconds() // 60)) + + def _queue_depths_sync() -> dict[str, int | None]: """Redis LLEN per queue. None for one that did not answer — see lane_view. @@ -362,6 +422,7 @@ async def set_lane( slots: int | None = None, slots_cap: int | None = None, enabled: bool | None = None, + autoscale: bool | None = None, ) -> dict: """Store the operator's choice, then push it to the running lane. @@ -383,6 +444,7 @@ async def set_lane( new_cap = row.slots_cap if slots_cap is None else slots_cap new_slots = row.slots if slots is None else slots new_enabled = row.enabled if enabled is None else enabled + new_autoscale = row.autoscale if autoscale is None else autoscale ceiling = derived_ceiling(lane) if new_cap < 0 or new_slots < 0: @@ -398,6 +460,7 @@ async def set_lane( row.slots_cap = new_cap row.slots = new_slots row.enabled = new_enabled + row.autoscale = new_autoscale await session.commit() applied, error = True, None @@ -428,6 +491,7 @@ async def set_lane( "slots_cap": row.slots_cap, "ceiling": ceiling, "enabled": row.enabled, + "autoscale": row.autoscale, "applied": applied, "apply_error": error, # Tells the card to say a download has started rather than leaving the @@ -458,11 +522,28 @@ def _enqueue_model_fetch() -> bool: return False -def reconcile_lanes_sync(desired: dict[str, tuple[int, bool]]) -> dict: +def reconcile_lanes_sync( + desired: dict[str, tuple[int, bool]], + autoscaling: frozenset[str] = frozenset(), +) -> dict: """Drive every RUNNING lane to its stored slots and enabled flag. `desired` is lane name -> (slots, enabled), read from the database by the - caller. This function touches no database: the celery task that schedules + caller. `autoscaling` names the lanes the autoscaler is allowed to move. + + ## For an autoscaling lane the stored value is a FLOOR, not a target + + Step 7's autoscaler raises a saturated lane's live pool without changing + its row — the row holds what the OPERATOR set. If this pass treated that + row as an exact target it would shrink the lane back on the very next + tick, and the two sweeps would fight forever at five-minute intervals: + grow, revert, grow, revert. That is lesson #4183's failure arriving + between two enforcers rather than inside one. + + So for those lanes the target becomes `max(stored, current)` — this pass + still restores a lane that came back from a restart below what the + operator set, and never takes back what the autoscaler added. Bringing it + down is the autoscaler's job, and it does so only to that same floor. This function touches no database: the celery task that schedules it owns the sync session, and keeping the DB out of here is what lets the same code be called from anywhere that already knows the target. @@ -521,17 +602,177 @@ def reconcile_lanes_sync(desired: dict[str, tuple[int, bool]]) -> dict: changed.append(lane.name) current = state.pool - if current is not None and current == slots: + # The floor, for a lane the autoscaler manages. See the docstring. + target_slots = slots + if lane.name in autoscaling and current is not None: + target_slots = max(slots, current) + if current is not None and current == target_slots: continue - ok, err = set_lane_slots_sync(lane, slots, live=state) + ok, err = set_lane_slots_sync(lane, target_slots, live=state) if ok: if lane.name not in changed: changed.append(lane.name) log.info( "worker_control: %s reconciled %s -> %s slots", - lane.name, current, slots, + lane.name, current, target_slots, ) else: failed[lane.name] = err or "could not resize" return {"changed": changed, "skipped": skipped, "failed": failed} + + +# --- the autoscaler (step 7) -------------------------------------------------- +# +# The only part of this milestone that acts without anyone asking. Everything +# above does what an operator pressed; this decides. So it is off by default, +# opted into per lane, bounded by the cap the operator set, and it reports what +# it did rather than moving numbers silently. + +# ## Why these three numbers are not in Settings +# +# Rule 25 puts anything an operator might want to tune in the UI, and the +# knobs that decide what this does ARE there: whether a lane autoscales at +# all, its cap, and its floor — all DB-backed, all changeable without a +# restart. What is left here is the POLICY's internals, and exposing them +# would add four numbers per lane to a card whose whole value is being +# readable at a glance, to tune a decision the operator has a better lever +# for. If growth turns out to be too eager or too shy in practice, that is a +# reason to change these values for everyone, not to ask each operator to +# discover them. + +# All slots busy AND this many tasks waiting before a lane may grow. +# +# The AND is the design. Depth with free slots means nothing — celery is about +# to pick those up, and growing the pool would add idle children. Saturation +# with an empty queue means nothing either: the lane is busy with exactly as +# much work as exists. Only both together say "there is more work than this +# lane can reach". +AUTOSCALE_BACKLOG_THRESHOLD = 10 + +# Grow by one slot per tick, never to the cap in one jump. A lane that is +# saturated because of one slow burst settles a slot or two above where it +# started rather than at its ceiling, and the next tick re-measures rather +# than committing to a guess made once. +AUTOSCALE_STEP = 1 + +# Hysteresis: shrink only when the backlog is well BELOW the grow threshold, +# not merely under it. Equal thresholds flap — one task arriving and leaving +# would grow and shrink the lane forever at the tick interval, which is +# lesson #4183's churn arriving through a different door. +AUTOSCALE_SHRINK_BELOW = 2 + + +@dataclass +class AutoscaleDecision: + """What the autoscaler did to one lane, and why — in the operator's terms. + + A reason string on every outcome including "nothing", because an + autoscaler that only speaks when it acts is one nobody can debug when it + does not. + """ + + lane: str + action: str # "grew" | "shrank" | "held" + slots: int + reason: str + + +def autoscale_lanes_sync( + lanes: dict[str, tuple[int, int, bool]], +) -> list[AutoscaleDecision]: + """Decide and apply one round of autoscaling. + + `lanes` is name -> (slots_cap, configured_slots, autoscale_on), read from + the database by the caller — this function touches no database, for the + same reason `reconcile_lanes_sync` does not. + + ## What is current, and what is the floor + + The value this moves is the LIVE pool, read from `inspect`. The stored + `configured_slots` is what the operator set and is only the FLOOR: growth + goes above it and a shrink returns to it, never below. + + The two are deliberately not the same number, and reading the stored value + as "current" is the mistake that makes this function useless in a way no + unit test of a single tick would show. The autoscaler never writes the + row, so the stored value never moves; a tick that computed `stored + 1` + would propose the same target forever, cap the lane one slot above the + floor no matter the load, and — because resizing a replica already at the + target issues nothing and reports success — claim `grew` on every tick + while nothing changed. Lesson #4183's non-convergence, arriving with a + success message attached. + + So: `current = state.pool`, `configured` is the floor, and both `grew` and + `shrank` mean the live pool actually moved. + """ + live = inspect_lanes_sync() + depths = _queue_depths_sync() + out: list[AutoscaleDecision] = [] + + for lane in LANES: + target = lanes.get(lane.name) + if target is None: + continue + cap, configured, on = target + if not on: + continue + + state = live[lane.name] + if not state.present or state.pool is None: + # Nothing answered. Not "idle" — unknown, and a decision drawn + # from an unswept read is exactly what snippet #3969 warns about. + # `configured` is reported because there is no live number to + # report; it is what the lane will come back at. + out.append(AutoscaleDecision( + lane.name, "held", configured, "lane is not answering", + )) + continue + + current = state.pool + known = [depths.get(q) for q in lane.queues] + if all(d is None for d in known): + out.append(AutoscaleDecision( + lane.name, "held", current, "queue depth unavailable", + )) + continue + backlog = sum(d for d in known if d is not None) + state.reserved + # Against CAPACITY, not against the dial: `active` is summed across + # replicas, so comparing it to one replica's pool size would call two + # half-busy replicas of 4 saturated at 4 active and grow a lane that + # has idle slots. + saturated = state.active >= state.capacity > 0 + busy = backlog >= AUTOSCALE_BACKLOG_THRESHOLD + + if saturated and busy and current < cap: + new = min(cap, current + AUTOSCALE_STEP) + ok, err = set_lane_slots_sync(lane, new, live=state) + out.append(AutoscaleDecision( + lane.name, "grew" if ok else "held", new if ok else current, + f"{backlog} waiting and all {state.capacity} slots busy" + if ok else f"could not grow: {err}", + )) + elif saturated and busy: + # At the cap with work still waiting. Said out loud rather than + # held silently: this is the operator's own ceiling doing its job, + # and it is the moment they would want to know they set it. + out.append(AutoscaleDecision( + lane.name, "held", current, + f"{backlog} waiting but the cap is {cap}", + )) + elif current > configured and backlog <= AUTOSCALE_SHRINK_BELOW: + new = max(configured, current - AUTOSCALE_STEP) + ok, err = set_lane_slots_sync(lane, new, live=state) + out.append(AutoscaleDecision( + lane.name, "shrank" if ok else "held", new if ok else current, + f"backlog cleared, back toward {configured}" + if ok else f"could not shrink: {err}", + )) + else: + # The fixed point. A settled lane sends nothing and says so — + # the tick is one inspect and one LLEN sweep, no control messages. + out.append(AutoscaleDecision( + lane.name, "held", current, + f"{backlog} waiting, {state.active} busy", + )) + return out diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index 66521a2..c36ec52 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -136,6 +136,11 @@ class Lane: # rubber stamp and protects nobody. default_slots_cap: int default_enabled: bool + # Whether a lane arrives with the autoscaler on. False for every lane, and + # the field exists anyway: this module is the one place that describes a + # lane, and leaving one operator-settable default to the column's DDL + # default would make it the only setting you cannot read here. + default_autoscale: bool = False # True when a slot costs a copy of the ML model rather than just a process. # The only lane whose ceiling is decided by memory instead of by cores. memory_bound: bool = False diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index c7c8fe7..b13ac68 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -1387,14 +1387,69 @@ def reconcile_worker_lanes() -> dict: from ..models import WorkerLane from ..services.worker_control import reconcile_lanes_sync + # Both dicts built INSIDE the session. Reading a column off a detached + # instance happens to work while the attribute is still loaded and stops + # working the moment anything expires it — a failure that would appear + # long after this line, in a sweep nobody is watching. with _sync_session_factory()() as session: - desired = { - row.name: (row.slots, row.enabled) - for row in session.execute(select(WorkerLane)).scalars() - } + rows = list(session.execute(select(WorkerLane)).scalars()) + desired = {row.name: (row.slots, row.enabled) for row in rows} + # Lanes the autoscaler may move. For these the stored value is a + # FLOOR: this sweep restores a lane that fell below it and never takes + # back what the autoscaler added, or the two would fight every five + # minutes. + autoscaling = frozenset(row.name for row in rows if row.autoscale) if not desired: # Migration 0103 seeds these, so an empty table means it has not run # yet. Nothing to assert — and inventing defaults here would let this # task disagree with the seed it is supposed to be enforcing. return {"changed": [], "skipped": [], "failed": {}} - return reconcile_lanes_sync(desired) + return reconcile_lanes_sync(desired, autoscaling) + + +@celery.task(name="backend.app.tasks.maintenance.autoscale_worker_lanes") +def autoscale_worker_lanes() -> dict: + """Grow a saturated lane, within the cap the operator set. + + Milestone 422 step 7, and the only sweep in this milestone that decides + rather than obeys. Everything else applies what someone pressed. + + OFF unless a lane opts in. A no-op costs one `celery inspect` and one LLEN + sweep and sends no control messages — the same fixed point the reconcile + holds, and for the same reason: this runs forever, so a settled system has + to be silent or the real signal drowns in its own heartbeat. + + Returns every lane's decision INCLUDING the ones it held, each with a + reason. An autoscaler that only speaks when it acts cannot be debugged on + the day it does not. + """ + from ..models import WorkerLane + from ..services.worker_control import autoscale_lanes_sync + + with _sync_session_factory()() as session: + lanes = { + # The stored `slots` is passed ONLY as the floor. What the + # autoscaler moves is the live pool, which it reads itself — this + # row is never written, so using it as the current value would pin + # every decision to the same number forever. + row.name: (row.slots_cap, row.slots, True) + for row in session.execute(select(WorkerLane)).scalars() + if row.autoscale + } + if not lanes: + return {"decisions": []} + + decisions = autoscale_lanes_sync(lanes) + for d in decisions: + if d.action != "held": + log.info( + "autoscale: %s %s to %s slots — %s", + d.lane, d.action, d.slots, d.reason, + ) + return { + "decisions": [ + {"lane": d.lane, "action": d.action, "slots": d.slots, + "reason": d.reason} + for d in decisions + ], + } diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue index 14999f4..dce4fd8 100644 --- a/frontend/src/components/settings/WorkerLanesCard.vue +++ b/frontend/src/components/settings/WorkerLanesCard.vue @@ -37,6 +37,7 @@ Pending Busy Slots + Auto On @@ -62,6 +63,15 @@
{{ lane.live.replicas }} replicas
+ +
+ all slots busy for {{ laneStuckFor(lane) }} +
{{ lane.queues.join(', ') }} @@ -106,6 +116,31 @@ + + + +
+ raise the cap +
+ + import { computed, ref } from 'vue' -import { useSystemActivityStore } from '../../stores/systemActivity.js' +import { laneStuckFor, useSystemActivityStore } from '../../stores/systemActivity.js' import { formatRelative } from '../../utils/date.js' import CardHeading from '../common/CardHeading.vue' @@ -238,6 +273,17 @@ function step(lane, delta) { function toggle(lane, value) { return apply(lane, { enabled: Boolean(value) }) } + +// Room to grow into. The autoscaler moves the LIVE pool, but the floor it +// starts from is the stored value, so a lane whose floor already sits at its +// cap has nowhere to go and enabling it would do nothing at all. +function canGrow(lane) { + return lane.slots_cap > lane.slots +} + +function setAutoscale(lane, value) { + return apply(lane, { autoscale: Boolean(value) }) +} diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue index dce4fd8..247e9db 100644 --- a/frontend/src/components/settings/WorkerLanesCard.vue +++ b/frontend/src/components/settings/WorkerLanesCard.vue @@ -1,210 +1,194 @@ diff --git a/frontend/src/components/settings/WorkerLanesCard.vue b/frontend/src/components/settings/WorkerLanesCard.vue deleted file mode 100644 index 247e9db..0000000 --- a/frontend/src/components/settings/WorkerLanesCard.vue +++ /dev/null @@ -1,342 +0,0 @@ - - - - - diff --git a/frontend/src/styles/app.css b/frontend/src/styles/app.css index 93c8d7a..fbd9380 100644 --- a/frontend/src/styles/app.css +++ b/frontend/src/styles/app.css @@ -132,50 +132,3 @@ backdrop-filter: blur(2px); -webkit-backdrop-filter: blur(2px); } - -/* The System tab's row idiom (operator 2026-09-23: "improve the view to be - more inline with other UI elements"). - - The roster (milestone 365) and the worker lanes (milestone 422) sit on the - same tab and answer the two halves of one question — what is running, and - how hard. They were built months apart and looked it: one a flat list of - dotted rows, the other a bordered card wrapping a v-table. Lifting these out - of SystemHealthTab's scoped block is what lets the second pane BE the first - one's idiom rather than imitate it — a copy would drift the first time - either was touched. - - `fc-sys__` rather than a new prefix because the roster's markup already uses - these names; renaming would have been churn in the file that is not - changing. */ -.fc-sys__lede, .fc-sys__muted, .fc-sys__checked, .fc-sys__foot { - color: rgb(var(--v-theme-on-surface) / 0.66); -} -.fc-sys__checked { font-size: 0.78rem; } -.fc-sys__card { background: rgb(var(--v-theme-on-surface) / 0.04); } - -.fc-sys__row { - display: flex; align-items: center; gap: 12px; - padding: 12px 16px; - border-bottom: 1px solid rgb(var(--v-theme-on-surface) / 0.08); -} -.fc-sys__row:last-child { border-bottom: 0; } - -.fc-sys__dot { width: 9px; height: 9px; border-radius: 50%; flex: 0 0 auto; } -.fc-sys__dot--ok { background: rgb(var(--v-theme-success)); } -.fc-sys__dot--stale { background: rgb(var(--v-theme-warning)); } -.fc-sys__dot--down { background: rgb(var(--v-theme-error)); } -.fc-sys__dot--unknown { background: rgb(var(--v-theme-on-surface) / 0.35); } - -.fc-sys__body { min-width: 0; flex: 1 1 auto; } -.fc-sys__name { font-weight: 600; } -.fc-sys__kind { - margin-left: 8px; font-weight: 400; font-size: 0.72rem; text-transform: uppercase; - letter-spacing: 0.04em; color: rgb(var(--v-theme-on-surface) / 0.5); -} -.fc-sys__detail { font-size: 0.82rem; color: rgb(var(--v-theme-on-surface) / 0.72); } - -.fc-sys__meta { - text-align: right; font-size: 0.75rem; flex: 0 0 auto; - font-variant-numeric: tabular-nums; color: rgb(var(--v-theme-on-surface) / 0.6); -} -.fc-sys__queues { opacity: 0.75; } diff --git a/frontend/src/utils/systemParts.js b/frontend/src/utils/systemParts.js new file mode 100644 index 0000000..e222edf --- /dev/null +++ b/frontend/src/utils/systemParts.js @@ -0,0 +1,100 @@ +// Joining the roster to the worker lanes, for the System tab's one table. +// +// Extracted from the component rather than left inline because the failure +// this can have is SILENT and is exactly the thing the merge exists to fix: if +// a lane stops matching its roster part, nothing throws — the table simply +// grows a second row for the same worker, one with controls and one without, +// which is the duplication the operator asked to be rid of, returned by the +// code that removed it. +// +// Operator, 2026-09-23: "I feel that we can probably combine the two +// sections into a single table." + +// A learned roster part and a lane are the same thing seen from two sides, and +// the QUEUES are what identify it — `service_roster.refresh_celery_roster` +// keys a celery part on exactly `"celery:" + ",".join(sorted(queues))`. +// +// Matched on the sorted set rather than on that string so the join survives a +// change to how the key is spelled, and so neither side has to agree about +// ORDER: the lane table lists a lane's queues in the order the role reads them +// (`default, import, thumbnail, download`) while the roster sorts them +// (`default, download, import, thumbnail`). +export function queueKey(queues) { + return [...(queues || [])].sort().join(',') +} + +// Worst first. A stopped datastore is why someone opened this tab. +export const SEVERITY = { down: 3, stale: 2, unknown: 1, ok: 0 } + +export function kindLabel(kind) { + if (kind === 'celery') return 'worker lane' + if (kind === 'agent') return 'GPU agent' + if (kind === 'datastore') return 'datastore' + return kind +} + +/** + * One row per moving part, with a lane attached where there is one. + * + * @param parts the roster's parts, as /api/system/health returns them + * @param lanes the lane rows, as /api/system/workers returns them + * @param stuckFor a lane -> "40 minutes" | null reporter (laneStuckFor) + */ +export function mergeParts(parts, lanes, stuckFor = () => null) { + const unmatched = {} + for (const lane of lanes || []) unmatched[queueKey(lane.queues)] = lane + + const out = [] + for (const part of parts || []) { + const key = queueKey(part.queues) + const lane = part.kind === 'celery' ? unmatched[key] : undefined + if (lane) delete unmatched[key] + out.push({ + key: part.key, + name: part.name, + kindLabel: lane?.optional ? 'optional lane' : kindLabel(part.kind), + state: part.state, + // A lane dialled to zero is OFF, not broken. Say so, rather than let the + // roster's heartbeat sentence report the operator's own choice as a + // fault — the roster cannot know the difference, and the lane can. + detail: lane && lane.slots === 0 ? 'off — no slots' : part.detail, + queues: (part.queues || []).join(', '), + lane, + stuckFor: lane ? stuckFor(lane) : null, + severity: SEVERITY[part.state] ?? SEVERITY.unknown, + }) + } + + // A lane the roster has not learned yet. Parts appear only once they have + // checked in, while the lane table is known up front — so without this, the + // lane an operator most needs to find (an optional one, never yet started) + // would be the only one missing from the table. + for (const lane of Object.values(unmatched)) out.push(laneRow(lane, stuckFor)) + + // Severity leads; then lanes ahead of everything else, because they are the + // rows you can actually do something about; then by name. + return out.sort((a, b) => + b.severity - a.severity + || Number(Boolean(b.lane)) - Number(Boolean(a.lane)) + || a.name.localeCompare(b.name)) +} + +function laneRow(lane, stuckFor) { + const on = lane.slots > 0 + let state = 'unknown' + if (lane.live?.present) state = on ? (stuckFor(lane) ? 'stale' : 'ok') : 'unknown' + else if (on) state = 'down' + return { + key: `lane:${lane.name}`, + name: lane.display_name, + kindLabel: lane.optional ? 'optional lane' : 'worker lane', + state, + detail: lane.live?.present + ? (on ? 'running' : 'off — no slots') + : 'has not checked in yet', + queues: (lane.queues || []).join(', '), + lane, + stuckFor: stuckFor(lane), + severity: SEVERITY[state], + } +} diff --git a/frontend/test/systemParts.spec.js b/frontend/test/systemParts.spec.js new file mode 100644 index 0000000..894b09d --- /dev/null +++ b/frontend/test/systemParts.spec.js @@ -0,0 +1,128 @@ +import { describe, expect, it } from 'vitest' + +import { mergeParts, queueKey } from '../src/utils/systemParts.js' + +// The System tab's one table (operator 2026-09-23: "combine the two sections +// into a single table"). What is pinned here is the JOIN, because its failure +// is silent and is precisely the thing the merge exists to fix: a lane that +// stops matching its roster part does not throw — the table grows a SECOND row +// for the same worker, one with controls and one without. + +const PART = { + key: 'celery:default,download,import,thumbnail', + kind: 'celery', + name: 'Worker', + state: 'ok', + detail: 'Worker is running', + // The roster SORTS a celery part's queues into its key. + queues: ['default', 'download', 'import', 'thumbnail'], +} + +const LANE = { + name: 'worker', + display_name: 'Worker', + // The lane table lists them in the order the role reads them, which is NOT + // sorted. If the join ever compares these two lists directly rather than as + // sets, this fixture is what catches it. + queues: ['default', 'import', 'thumbnail', 'download'], + slots: 2, + slots_cap: 4, + autoscale: false, + live: { present: true, replicas: 1, pool: 2, active: 0 }, + pending: 0, +} + +const POSTGRES = { + key: 'postgres', kind: 'datastore', name: 'PostgreSQL', + state: 'ok', detail: 'answering', latency_ms: 2.5, +} + +describe('queueKey', () => { + it('does not care what order either side lists its queues in', () => { + expect(queueKey(LANE.queues)).toBe(queueKey(PART.queues)) + }) + + it('survives a part that has no queues at all', () => { + // A datastore, and also the stale `Worker ()` row a previous deployment + // left in the roster with an empty queue set. Neither must match a lane. + expect(queueKey(undefined)).toBe('') + expect(queueKey([])).toBe('') + }) +}) + +describe('mergeParts', () => { + it('gives a worker ONE row, carrying its controls', () => { + const rows = mergeParts([PART, POSTGRES], [LANE]) + + expect(rows).toHaveLength(2) + const worker = rows.find((r) => r.name === 'Worker') + expect(worker.lane).toBe(LANE) + expect(rows.filter((r) => r.name === 'Worker')).toHaveLength(1) + }) + + it('leaves a datastore without a lane rather than guessing one', () => { + const pg = mergeParts([PART, POSTGRES], [LANE]).find((r) => r.key === 'postgres') + expect(pg.lane).toBeUndefined() + }) + + it('still lists a lane the roster has never seen', () => { + // Parts are learned as they appear; the lane table is known up front. The + // lane an operator most needs to find — an optional one, never started — + // is exactly the one with no roster entry. + const ml = { + ...LANE, name: 'ml', display_name: 'ML tagging', queues: ['ml'], + slots: 0, optional: true, + live: { present: false, replicas: 0, pool: null, active: 0 }, + } + const rows = mergeParts([POSTGRES], [ml]) + + const row = rows.find((r) => r.name === 'ML tagging') + expect(row.lane).toBe(ml) + expect(row.kindLabel).toBe('optional lane') + }) + + it('does not call a lane at zero slots broken', () => { + // The roster only knows a heartbeat age, so it goes on saying "is running" + // for a lane the operator deliberately dialled to nothing. The lane knows + // the difference; reporting the operator's own choice as a fault is how an + // indicator stops being read. + const off = { ...LANE, slots: 0 } + const row = mergeParts([PART], [off])[0] + + expect(row.detail).toBe('off — no slots') + }) + + it('puts the broken thing first, whatever it is', () => { + const down = { ...POSTGRES, state: 'down', detail: 'not answering' } + const rows = mergeParts([PART, down], [LANE]) + + expect(rows[0].name).toBe('PostgreSQL') + }) + + it('otherwise puts the rows you can act on first', () => { + const rows = mergeParts([POSTGRES, PART], [LANE]) + + expect(rows.map((r) => r.name)).toEqual(['Worker', 'PostgreSQL']) + }) + + it('reports a wedged lane, and only through the reporter it was given', () => { + // `laneStuckFor` is passed in rather than imported, so this file does not + // re-test the store's rule — it tests that the merge asks. + const asked = [] + const rows = mergeParts([PART], [LANE], (lane) => { + asked.push(lane.name) + return '40 minutes' + }) + + expect(asked).toEqual(['worker']) + expect(rows[0].stuckFor).toBe('40 minutes') + // The roster still owns a matched row's state — `stuckFor` is a note + // beside it, not a verdict that overrides the heartbeat. + expect(rows[0].state).toBe('ok') + }) + + it('handles an empty everything without inventing rows', () => { + expect(mergeParts([], [])).toEqual([]) + expect(mergeParts(undefined, undefined)).toEqual([]) + }) +}) diff --git a/frontend/test/workerLanes.spec.js b/frontend/test/workerLanes.spec.js index 787d7f0..f553e57 100644 --- a/frontend/test/workerLanes.spec.js +++ b/frontend/test/workerLanes.spec.js @@ -54,8 +54,9 @@ describe('worker lanes store', () => { }) it('a load failure records the error rather than throwing at the caller', async () => { - // The card polls this every 3s. An unhandled rejection per tick would - // drown the console and stop the other pollers in the same function. + // The System tab polls this every 15s. An unhandled rejection per tick + // would drown the console and stop the other pollers in the same + // function. stubFetch(() => ({ status: 500, body: { error: 'boom' } })) const s = useSystemActivityStore() await expect(s.loadLanes()).resolves.toBeUndefined() @@ -83,9 +84,9 @@ describe('worker lanes store', () => { }) it('setLane refetches so the card shows the server truth, not the guess', async () => { - // The reply is one lane; the card renders all of them plus live pool and - // pending. Patching the local row from the reply would leave every other - // column stale and eventually wrong. + // The reply is one lane; the table renders all of them plus live pool + // and pending. Patching the local row from the reply would leave every + // other column stale and eventually wrong. let gets = 0 stubFetch((url, init) => { if (init?.method === 'POST') return { status: 200, body: { applied: true } } diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py index 97bd728..d36e1fd 100644 --- a/tests/test_api_workers.py +++ b/tests/test_api_workers.py @@ -170,3 +170,114 @@ async def test_an_empty_body_is_refused_rather_than_treated_as_a_no_op( assert resp.status_code == 400 body = await resp.get_json() assert body["error"] == "invalid_body" + + +# --- the dial is the switch -------------------------------------------------- +# +# Operator, 2026-09-23: *"almost all of it always needs to run there's only one +# optional piece and it is killed by moving the 'cap' to zero."* So `enabled` +# is derived from the number rather than being a second control the operator +# has to keep in agreement with it. It stays on the API — these assert that it +# still does, because it is the mechanism the reconcile and the healthcheck +# read. + + +@pytest.mark.asyncio +async def test_dialling_a_lane_to_zero_turns_it_off(client, db, no_live_workers): + await client.post("/api/system/workers/worker", json={"slots": 0}) + + row = await _lane_row(db, "worker") + assert row.slots == 0 + assert row.enabled is False + + +@pytest.mark.asyncio +async def test_dialling_it_back_up_turns_it_on(client, db, no_live_workers): + await client.post("/api/system/workers/ml", json={"slots": 1}) + + row = await _lane_row(db, "ml") + assert row.slots == 1 + assert row.enabled is True, "the lane the operator just asked for work from" + + +@pytest.mark.asyncio +async def test_an_explicit_enabled_still_wins(client, db, no_live_workers): + """The field is not removed, only derived when absent. Something that + genuinely wants a lane holding its process with consumers cancelled — a + drain before a restart — must still be able to say so without having to + destroy the operator's slot count to express it.""" + await client.post( + "/api/system/workers/worker", json={"slots": 3, "enabled": False}, + ) + + row = await _lane_row(db, "worker") + assert (row.slots, row.enabled) == (3, False) + + +@pytest.mark.asyncio +async def test_a_cap_only_write_does_not_decide_the_switch( + client, db, no_live_workers, +): + """Only the SLOTS dial derives it. A cap is a ceiling, not a request for + work, and letting it flip the lane would make raising a ceiling start + something.""" + before = await _lane_row(db, "ml") + assert before.enabled is False + + await client.post("/api/system/workers/ml", json={"slots_cap": 1}) + + await db.refresh(before) + assert (before.slots, before.enabled) == (0, False) + + +@pytest.mark.asyncio +async def test_the_model_fetch_fires_on_the_transition_not_on_the_field( + client, db, no_live_workers, monkeypatch, +): + """The trap the derivation set, caught here rather than in production. + + The fetch used to be conditioned on `enabled is True` — the FIELD having + been sent. The UI no longer sends it at all, so the download that makes + the ML lane usable would simply never have fired, and the lane would have + come on and sat there consuming a queue it had no model for. + """ + fired = [] + monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) + monkeypatch.setattr( + wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None), + ) + monkeypatch.setattr( + wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None), + ) + + body = await (await client.post( + "/api/system/workers/ml", json={"slots": 1}, + )).get_json() + + assert body["fetching_models"] is True + assert fired == [1] + + +@pytest.mark.asyncio +async def test_it_does_not_fire_again_on_a_lane_already_running( + client, db, no_live_workers, monkeypatch, +): + """The other half. A second nudge of the dial on a lane that is already on + must not re-enqueue a multi-GB download.""" + monkeypatch.setattr( + wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None), + ) + monkeypatch.setattr( + wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None), + ) + await client.post("/api/system/workers/ml", json={"slots": 1}) + + fired = [] + monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) + + body = await (await client.post( + "/api/system/workers/ml", json={"slots": 1}, + )).get_json() + + assert body["fetching_models"] is False + assert fired == [] -- 2.54.0 From 445164c852a704e07dd775e225f11afc0b48a61a Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 12:48:22 -0400 Subject: [PATCH 34/94] =?UTF-8?q?feat:=20one=20number=20per=20lane=20?= =?UTF-8?q?=E2=80=94=20the=20cap=20=E2=80=94=20and=20the=20autoscaler=20is?= =?UTF-8?q?=20the=20mechanism=20(4295)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- alembic/versions/0105_worker_lane_one_cap.py | 123 ++++ backend/app/api/workers.py | 61 +- backend/app/celery_app.py | 33 +- backend/app/models/worker_lane.py | 89 +-- backend/app/scripts/gen_supervisord.py | 4 +- backend/app/services/worker_control.py | 534 +++++--------- backend/app/services/worker_lanes.py | 82 ++- backend/app/tasks/maintenance.py | 120 +--- .../components/settings/SystemHealthTab.vue | 139 ++-- frontend/src/utils/systemParts.js | 52 +- frontend/test/systemParts.spec.js | 78 +- frontend/test/workerLanes.spec.js | 23 +- tests/test_api_workers.py | 291 ++++---- tests/test_gen_supervisord.py | 35 +- tests/test_worker_control.py | 664 +++++++----------- tests/test_worker_lanes.py | 53 +- 16 files changed, 1110 insertions(+), 1271 deletions(-) create mode 100644 alembic/versions/0105_worker_lane_one_cap.py diff --git a/alembic/versions/0105_worker_lane_one_cap.py b/alembic/versions/0105_worker_lane_one_cap.py new file mode 100644 index 0000000..01b4813 --- /dev/null +++ b/alembic/versions/0105_worker_lane_one_cap.py @@ -0,0 +1,123 @@ +"""worker_lane — one number: the cap. `slots`, `enabled` and `autoscale` go. + +Milestone 422, reshaped by the operator 2026-09-23: + + "auto should be always on, not a setting, so that idle instances quiet + down when not running. the number that is visible and something the user + can tweak and manage should be the cap itself the number of running + workers is handled by the autoscaling function which is always on." + +## What each dropped column was, and why it is not needed + +**`slots`** — how many workers the lane should run. That is a MEASUREMENT, +not a preference: the autoscaler moves the live pool between one and the cap +according to the backlog, and reads it back from the worker every minute. +Storing it made it look like something to keep in agreement with the cap, +which is exactly what the operator had to do. + +**`autoscale`** — whether the lane was allowed to size itself. It gated the +mechanism behind a per-lane opt-in, so a lane nobody enabled simply never +gave its slots back. Always on now, which is the only way "idle instances +quiet down" can be true of an install nobody has configured. + +**`enabled`** — whether the lane consumes its queues. Derived from `cap > 0`. +It and `slots = 0` were two spellings of one fact and were free to disagree; +this migration picks the one an operator can see. + +## Why the caps are rewritten rather than preserved + +The old defaults were 4 / 2 / 2 / 1, chosen when the number meant "the most +you may raise SLOTS to" — a bound on a manual control, deliberately loose +because moving within it was the ordinary act. The number now means "the most +workers this lane may actually use", which is a different promise, and +carrying the old figure over would silently quadruple the worker lane on +every existing install at the moment this deploys. + +So every row is reset to the new defaults: **one for each required lane, zero +for ML.** That loses whatever an operator had set — which is the honest +trade, because what they set was an answer to a different question. The UI +now tells a busy lane's operator to raise its cap, which is how the number +gets back up on an install that needs it. + +ML at zero also keeps rule 164's carve-out intact: no consumers, so no model +download until someone raises the cap. + +Revision ID: 0105 +Revises: 0104 +Create Date: 2026-09-23 + +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "0105" +down_revision: Union[str, None] = "0104" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +# (name, cap) — the same values `services/worker_lanes.LANES` declares. Seeded +# here as literals rather than imported: a migration must describe the schema +# at ITS point in history, and importing the live table would make this file +# change meaning every time that table does. +_CAPS = ( + ("worker", 1), + ("scheduler", 1), + ("maintenance_long", 1), + ("ml", 0), +) + + +def upgrade() -> None: + # The constraint goes first: it names `slots`, so dropping the column out + # from under it fails on Postgres. + op.drop_constraint("ck_worker_lane_slots_within_cap", "worker_lane", type_="check") + op.drop_constraint( + "ck_worker_lane_slots_non_negative", "worker_lane", type_="check", + ) + op.drop_column("worker_lane", "slots") + op.drop_column("worker_lane", "enabled") + op.drop_column("worker_lane", "autoscale") + + # Reset to the new meaning. See the docstring: the old value answered a + # different question, and carrying it over would raise every lane. + for name, cap in _CAPS: + op.execute( + sa.text("UPDATE worker_lane SET slots_cap = :cap WHERE name = :name") + .bindparams(cap=cap, name=name) + ) + + # A lane the old seed never wrote — or one an operator added by hand — is + # left alone rather than guessed at. `_rows_by_name` creates any missing + # row at the lane's default on first read. + + +def downgrade() -> None: + op.add_column( + "worker_lane", + sa.Column("slots", sa.Integer(), nullable=False, server_default="1"), + ) + op.add_column( + "worker_lane", + sa.Column( + "enabled", sa.Boolean(), nullable=False, server_default=sa.text("true"), + ), + ) + op.add_column( + "worker_lane", + sa.Column( + "autoscale", sa.Boolean(), nullable=False, server_default=sa.text("false"), + ), + ) + # Restore the pre-0105 invariants. `slots` comes back as 1 everywhere and + # the caps are 1/1/1/0, so a lane at cap 0 would violate `slots <= cap` — + # hence the clamp before the constraint is added. + op.execute(sa.text("UPDATE worker_lane SET slots = 0 WHERE slots_cap = 0")) + op.execute(sa.text("UPDATE worker_lane SET enabled = (slots_cap > 0)")) + op.create_check_constraint( + op.f("ck_worker_lane_slots_non_negative"), "worker_lane", "slots >= 0", + ) + op.create_check_constraint( + op.f("ck_worker_lane_slots_within_cap"), "worker_lane", "slots <= slots_cap", + ) diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py index 678c061..10c03ee 100644 --- a/backend/app/api/workers.py +++ b/backend/app/api/workers.py @@ -10,10 +10,10 @@ answers a different question: its `/workers` is keyed on celery HOSTNAME and reports which nodes answered. That stays as it is — the existing SystemActivityTab consumes it. -This is keyed on LANE, joins the stored settings to the live pool, and -accepts writes. Two endpoints answering "which celery processes exist" and -"how much work is each lane allowed to do" are not the same endpoint, and -folding the second into the first would make a read-only module a write one. +This is keyed on LANE, joins the stored cap to the live pool, and accepts +writes. Two endpoints answering "which celery processes exist" and "how much +work is each lane allowed to do" are not the same endpoint, and folding the +second into the first would make a read-only module a write one. """ from __future__ import annotations @@ -32,7 +32,7 @@ workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers") @workers_bp.route("", methods=["GET"]) async def list_lanes(): - """Every lane: configured slots, the cap, the ceiling, and live state. + """Every lane: its cap, the ceiling above it, and what is live. Response: {lanes: [...], fetched_at: iso8601} @@ -51,20 +51,22 @@ async def list_lanes(): @workers_bp.route("/", methods=["POST"]) async def update_lane(name: str): - """Set a lane's slots, cap and/or enabled flag. Stores, then pushes live. + """Set a lane's cap. Stores it, then makes the live lane obey it. - Partial: only the keys present are changed, so the UI's stepper can send - `{"slots": 3}` without restating the cap it did not touch. + ONE field, since 2026-09-23. It used to take `slots`, `slots_cap`, + `enabled` and `autoscale`; how many workers are running is now a + measurement the sizing pass owns, and `enabled` is `cap > 0`. Two failure kinds, deliberately different statuses: - * **400** — the value is not allowed (above the cap, above the ceiling, - negative). Nothing was stored. The body carries `detail`, which is the - sentence the UI shows; a refused control with no reason reads as a bug. + * **400** — the value is not allowed (negative, or above what this + container can hold). Nothing was stored. The body carries `detail`, + which is the sentence the UI shows; a refused control with no reason + reads as a bug. * **200 with `applied: false`** — the value WAS stored but could not be - pushed, because the lane is not currently answering. That is not an - error: step 3's reconcile carries it when the lane comes back, and the - UI should say "saved, not yet live" rather than "that didn't work". + pushed, because the lane is not currently answering. Not an error: the + sizing pass carries it within a minute, and the UI should say "saved, + not yet live" rather than "that didn't work". """ lane = LANES_BY_NAME.get(name) if lane is None: @@ -74,31 +76,18 @@ async def update_lane(name: str): if not isinstance(body, dict): return _bad("invalid_body", detail="body must be a JSON object") - fields: dict = {} - for key in ("slots", "slots_cap"): - if key in body: - value = body[key] - # Rejected rather than coerced: `True` is an int in Python, and - # silently reading it as 1 slot would be a control that appears to - # work and sets something nobody asked for. - if not isinstance(value, int) or isinstance(value, bool): - return _bad("invalid_body", detail=f"{key} must be an integer") - fields[key] = value - for key in ("enabled", "autoscale"): - if key in body: - if not isinstance(body[key], bool): - return _bad("invalid_body", detail=f"{key} must be a boolean") - fields[key] = body[key] - - if not fields: - return _bad( - "invalid_body", - detail="give at least one of slots, slots_cap, enabled, autoscale", - ) + if "slots_cap" not in body: + return _bad("invalid_body", detail="give slots_cap") + value = body["slots_cap"] + # Rejected rather than coerced: `True` is an int in Python, and silently + # reading it as a cap of 1 would be a control that appears to work and + # sets something nobody asked for. + if not isinstance(value, int) or isinstance(value, bool): + return _bad("invalid_body", detail="slots_cap must be an integer") async with get_session() as session: try: - result = await set_lane(session, lane, **fields) + result = await set_lane(session, lane, slots_cap=value) except LaneUpdateRefused as exc: return _bad("refused", detail=str(exc)) return jsonify(result) diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index e509f8f..21c98bf 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -111,21 +111,24 @@ def make_celery() -> Celery: "task": "backend.app.tasks.maintenance.recover_interrupted_tasks", "schedule": 300.0, # every 5 minutes }, - "autoscale-worker-lanes": { - "task": "backend.app.tasks.maintenance.autoscale_worker_lanes", - "schedule": 60.0, # every minute — it reacts to a BACKLOG, and - # a five-minute reaction to a queue filling up is no reaction. - # Cheap when nothing opted in (one DB read, then nothing) and - # cheap when settled (one inspect + one LLEN sweep, no control - # messages), so the short interval costs little. - }, - "reconcile-worker-lanes": { - "task": "backend.app.tasks.maintenance.reconcile_worker_lanes", - "schedule": 300.0, # every 5 minutes — the window in which a - # restarted worker runs at its ENV concurrency rather than the - # slots the operator set (milestone 422 step 3). A no-op once - # every lane matches: one broker round trip, no control - # messages, nothing logged. + "size-worker-lanes": { + "task": "backend.app.tasks.maintenance.size_worker_lanes", + "schedule": 60.0, # every minute. + # + # ONE entry, replacing `autoscale-worker-lanes` (60s) and + # `reconcile-worker-lanes` (300s) on 2026-09-23. They were two + # sweeps over one number and most of the autoscaler's design + # existed to stop the reconcile undoing its work; with the + # stored `slots` gone there is nothing to disagree about. + # + # A minute because it reacts to a BACKLOG, and a five-minute + # reaction to a queue filling up is no reaction. It also now + # carries what the reconcile was for — a worker restarted at + # its ENV concurrency is corrected on the next tick rather + # than after five. + # + # Cheap when settled: one inspect plus one LLEN sweep, and no + # control messages at all once every lane matches. }, "cleanup-old-tasks": { "task": "backend.app.tasks.maintenance.cleanup_old_tasks", diff --git a/backend/app/models/worker_lane.py b/backend/app/models/worker_lane.py index 94daf45..6504656 100644 --- a/backend/app/models/worker_lane.py +++ b/backend/app/models/worker_lane.py @@ -1,48 +1,49 @@ -"""worker_lane — how many slots the operator wants each worker lane to have. +"""worker_lane — the most workers the operator will let each lane use. -Milestone 422 step 1. One row per lane in `services/worker_lanes.LANES`. +Milestone 422 step 1, reshaped 2026-09-23. One row per lane in +`services/worker_lanes.LANES`, and ONE COLUMN an operator sets. -## What is NOT in here +## Why there is only one number now -The lane's queues and its display name. Those are decided by `celery_app.py`'s -`task_routes` — an operator cannot move a backup off `maintenance_long` — so -storing them would be a row that can contradict the routing table, with -nothing to notice until a queue had no consumer. They live in -`services/worker_lanes.py`; this table holds only what an operator may change. +There were three — `slots`, `slots_cap` and `autoscale` — because the manual +dial was built first and the autoscaler arrived last, beside a control that +already existed rather than in place of it. -The DERIVED CEILING is also absent, and that is deliberate rather than an -omission. It is computed from the container's cgroup limits on every read, so -a row written on a 32GB host and later run in a 4GB container is bounded by -the 4GB — a stored ceiling would quietly authorise what the box can no longer -hold. +Operator: *"auto should be always on, not a setting, so that idle instances +quiet down when not running. the number that is visible and something the +user can tweak and manage should be the cap itself the number of running +workers is handled by the autoscaling function which is always on."* -## The three numbers +So `slots` is gone. How many workers a lane is running right now is a +MEASUREMENT — read live from the worker, moved by the autoscaler, never +stored. Storing it made it look like a preference, which meant the operator +had to keep two numbers in agreement and the autoscaler had to be told it was +allowed to touch one of them. - slots <= slots_cap <= derived_ceiling - (live) (this row) (computed) +`autoscale` is gone for the same reason: it gated the mechanism behind a +choice, and a lane nobody opted in simply never gave its slots back. -Operator's distinction, 2026-09-22: the derived value is *a cap on the cap*. -`slots_cap` is theirs and is always lowerable; it simply may not exceed what -the container can hold. The CHECK constraint below enforces the left half, -which is a fact about the row; the right half is enforced at write, because -it depends on a value no database column holds. +`enabled` is gone too, and is now DERIVED: a cap of zero means no consumers. +"Off" and "may use no workers" were two spellings of one fact, free to +disagree. -## enabled +## What remains -Whether the lane consumes its queues at all. This is how ML ships off -(milestone 422 step 6): `enabled=false` with `slots=0`, so a fresh install -never loads a model or reaches HuggingFace, and turning tagging on in -Settings is what triggers the fetch. + 1 <= live pool <= slots_cap <= derived_ceiling + (autoscaler) (this row) (computed) -Not a substitute for `slots=0`. A lane can be enabled with zero slots while -it is being resized, and the two answer different questions: `enabled` is -intent, `slots` is capacity. +The floor is one PROCESS, not zero: billiard will not run an empty pool, and +the parked process is what `add_consumer` lands on when the cap goes back up. + +The DERIVED CEILING is deliberately absent from this table. It is computed +from the container's cgroup limits on every read, so a row written on a 32GB +host and later run in a 4GB container is bounded by the 4GB — a stored +ceiling would quietly authorise what the box can no longer hold. """ from datetime import datetime from sqlalchemy import ( - Boolean, CheckConstraint, DateTime, Integer, @@ -57,16 +58,10 @@ from .base import Base class WorkerLane(Base): __tablename__ = "worker_lane" __table_args__ = ( - # Bare names — Base.metadata's naming convention prepends + # Bare name — Base.metadata's naming convention prepends # ck_worker_lane_. Pre-prefixing here doubles it, which is what # alembic 0088 had to rename four constraints for (#3275). - CheckConstraint("slots >= 0", name="slots_non_negative"), CheckConstraint("slots_cap >= 0", name="cap_non_negative"), - # The invariant that makes the cap mean anything. Enforced in the - # database rather than only in the service, because a row that - # violates it is not a rejected request — it is a lane that will be - # reconciled UP to a value the operator capped. - CheckConstraint("slots <= slots_cap", name="slots_within_cap"), ) # The lane name from services/worker_lanes.LANES — never a container @@ -74,21 +69,13 @@ class WorkerLane(Base): # are `celery@`, minted fresh on every deploy. name: Mapped[str] = mapped_column(String(32), primary_key=True) - slots: Mapped[int] = mapped_column(Integer, nullable=False) - slots_cap: Mapped[int] = mapped_column(Integer, nullable=False) - enabled: Mapped[bool] = mapped_column(Boolean, nullable=False) - - # Whether the autoscaler may raise this lane's slots on its own, up to - # slots_cap and never past it (milestone 422 step 7). + # The most workers this lane may use. Zero means off — no consumers, so + # the lane takes no work and (for ML) downloads no model. # - # OFF by default and per-lane rather than global. It is the one part of - # this milestone that acts unattended, so it is opt-in per lane the - # operator has actually thought about — a global switch would turn it on - # for lanes whose behaviour under load nobody has watched, including `ml`, - # where every extra slot is another copy of a multi-GB model. - autoscale: Mapped[bool] = mapped_column( - Boolean, nullable=False, server_default="false", - ) + # There is no upper CHECK here, because the bound it would need is the + # derived ceiling, and no column holds that: it depends on the cgroup the + # container is running in right now. Enforced at write instead. + slots_cap: Mapped[int] = mapped_column(Integer, nullable=False) updated_at: Mapped[datetime] = mapped_column( DateTime(timezone=True), diff --git a/backend/app/scripts/gen_supervisord.py b/backend/app/scripts/gen_supervisord.py index 447e66b..e8c92fc 100644 --- a/backend/app/scripts/gen_supervisord.py +++ b/backend/app/scripts/gen_supervisord.py @@ -54,7 +54,7 @@ import argparse import shlex import sys -from ..services.worker_lanes import LANES, Lane +from ..services.worker_lanes import LANES, MIN_POOL_SLOTS, Lane # One number for the whole container, and it must cover the SLOWEST lane — # docker gives the container a single stop timeout, where compose today gives @@ -212,7 +212,7 @@ def render() -> str: # worker there is nothing for `add_consumer` to reach, so enabling the # lane from the UI could not work at all — the process has to exist for # the switch to have something to switch. - parts.append(_program(lane, slots=max(1, lane.default_slots))) + parts.append(_program(lane, slots=MIN_POOL_SLOTS)) return "\n".join(parts) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 7e31f09..641c545 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -56,7 +56,13 @@ from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession from ..models import TaskRun, WorkerLane -from .worker_lanes import LANES, LANES_BY_QUEUE_KEY, Lane, derived_ceiling +from .worker_lanes import ( + LANES, + LANES_BY_QUEUE_KEY, + MIN_POOL_SLOTS, + Lane, + derived_ceiling, +) log = logging.getLogger(__name__) @@ -66,28 +72,6 @@ log = logging.getLogger(__name__) # report "not present", which is true, rather than hang the page. CONTROL_TIMEOUT_SECONDS = 2.0 -# A celery prefork pool cannot have ZERO processes, and a lane at zero slots -# is expressed by cancelling its consumers rather than by emptying its pool. -# -# The generator already starts every lane at one process for exactly this -# reason: `add_consumer` needs something to reach, so enabling a lane from the -# UI would be impossible if no process existed. The consequence was missed -# until the operator's live deploy, 2026-09-23: -# -# [scheduler] worker_control: ml reconciled 1 -> 0 slots -# [ml] pidbox command error: ValueError("Can't shrink pool. All processes -# busy!") -# -# billiard refuses to remove the last worker, so ml could never reach 0. And -# `set_lane_slots_sync` returns True on SENDING the control message — the -# failure happens later, on the worker — so the reconcile logged success and -# reported `changed: ['ml']` on every single tick. An enforcer with no -# reachable fixed point, re-doing its own work forever and saying it worked: -# lesson #4183, in production, on the default configuration of every install. -# -# So the floor is one process. Zero slots still means zero WORK, because the -# consumers are cancelled — the idle process is the switch's landing pad. -MIN_POOL_SLOTS = 1 # The WORST case of `inspect_lanes_sync`, for callers that need a deadline. # @@ -342,13 +326,7 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]: } missing = [lane for lane in LANES if lane.name not in rows] for lane in missing: - row = WorkerLane( - name=lane.name, - slots=lane.default_slots, - slots_cap=lane.default_slots_cap, - enabled=lane.default_enabled, - autoscale=lane.default_autoscale, - ) + row = WorkerLane(name=lane.name, slots_cap=lane.default_slots_cap) session.add(row) rows[lane.name] = row if missing: @@ -398,17 +376,18 @@ async def lane_view(session: AsyncSession) -> list[dict]: "name": lane.name, "display_name": lane.display_name, "queues": list(lane.queues), - "slots": row.slots, "slots_cap": row.slots_cap, "ceiling": derived_ceiling(lane), - "enabled": row.enabled, - "autoscale": row.autoscale, + # DERIVED, never stored. A cap of zero means no consumers, so + # "off" and "may use no workers" cannot disagree. + "enabled": row.slots_cap > 0, "memory_bound": lane.memory_bound, "optional": lane.optional, - # What enabling this lane will download, so the UI can say WHICH - # model and how big BEFORE the switch is thrown rather than after - # a multi-GB fetch has started. `measured` travels with the - # numbers: the card must not present an estimate as a fact. + # What raising this lane's cap will download, so the UI can say + # WHICH model and how big BEFORE the first slot is asked for + # rather than after a multi-GB fetch has started. `measured` + # travels with the numbers: the UI must not present an estimate + # as a fact. "models": [ { "repo": m.repo, @@ -506,119 +485,89 @@ class LaneUpdateRefused(ValueError): the UI shows — a greyed control with no explanation reads as a bug.""" -async def set_lane( - session: AsyncSession, - lane: Lane, - *, - slots: int | None = None, - slots_cap: int | None = None, - enabled: bool | None = None, - autoscale: bool | None = None, -) -> dict: - """Store the operator's choice, then push it to the running lane. +async def set_lane(session: AsyncSession, lane: Lane, *, slots_cap: int) -> dict: + """Store the operator's cap for `lane`, then make the live lane obey it. - BOTH, in one call, and the order matters. `pool_grow`/`pool_shrink` are - not durable — a restart drops every lane back to its env concurrency — so - a UI that only pushed would have its setting evaporate on the next deploy - with nothing to show for it (lesson #4202: the live change does not - survive, and nothing says so). Storing alone would be a number that - describes nothing until something restarts. + ONE value, since 2026-09-23. It used to take `slots`, `slots_cap`, + `enabled` and `autoscale`, which was four ways of saying two things — and + two of them were the caller's job to keep in agreement with each other. + + ## What happens live, and what does not + + A cap CHANGE is pushed immediately in one direction only: lowering it + shrinks the pool now, because a cap the operator just lowered should not + be exceeded for up to a minute. Raising it does NOT grow the pool here — + a cap is permission, not a request, and growing on permission would put + slots on a lane with nothing to do. The sizing pass adds them on its next + tick if there is work, which is the whole point of it being always on. + + Consumers follow the cap in both directions and immediately: zero means + off, and off must take effect when it is asked for. A failed PUSH is not a failed setting. The value is saved either way and - step 3's reconcile carries it when the lane answers again; the result says - `applied: false` with a reason so the UI can say "saved, not yet live" - rather than "that didn't work". + the sizing pass carries it within a minute; the result says `applied: + false` with a reason so the UI can say "saved, not yet live" rather than + "that didn't work" (lesson #4202 — a live change that does not survive, + with nothing saying so). """ rows = await _rows_by_name(session) row = rows[lane.name] - new_cap = row.slots_cap if slots_cap is None else slots_cap - new_slots = row.slots if slots is None else slots - new_autoscale = row.autoscale if autoscale is None else autoscale - - # THE DIAL IS THE SWITCH. A lane at zero slots is a lane that is off, and - # there is no second control saying so. - # - # Operator, 2026-09-23, on the card that had both: *"there's nothing to - # describe what 'auto' means or why their needs to be or should be on/off - # toggles. almost all of it always needs to run there's only one optional - # piece and it is killed by moving the 'cap' to zero."* They are right. Of - # four lanes, three must run for the application to work at all, so a - # switch beside each of them offered a choice that was never real — and - # for the one lane that IS optional, "off" and "zero slots" were two ways - # of saying the same thing that could disagree with each other. - # - # `enabled` stays in the model and on the API. It is still the mechanism: - # a disabled lane keeps its process and cancels its consumers, which is - # what makes it visible in the roster instead of looking like a crash. It - # is now DERIVED from the number the operator actually sets, rather than - # being a second thing for them to keep in agreement with it. - was_enabled = row.enabled - if enabled is not None: - new_enabled = enabled - elif slots is not None: - new_enabled = new_slots > 0 - else: - new_enabled = row.enabled - ceiling = derived_ceiling(lane) - if new_cap < 0 or new_slots < 0: - raise LaneUpdateRefused("slots and cap cannot be negative") - if new_cap > ceiling: + if slots_cap < 0: + raise LaneUpdateRefused("a cap cannot be negative") + if slots_cap > ceiling: raise LaneUpdateRefused( - f"cap {new_cap} is above what this container can hold " + f"a cap of {slots_cap} is above what this container can hold " f"({ceiling} for {lane.display_name})" ) - if new_slots > new_cap: - raise LaneUpdateRefused(f"slots {new_slots} is above the cap {new_cap}") - row.slots_cap = new_cap - row.slots = new_slots - row.enabled = new_enabled - row.autoscale = new_autoscale + was_on = row.slots_cap > 0 + row.slots_cap = slots_cap await session.commit() + now_on = slots_cap > 0 applied, error = True, None - # On the CHANGE, not on the field being present. Now that `enabled` is - # derived, every slots write would otherwise re-send a consumer command - # that changes nothing — the churn lesson #4183 keeps producing, arriving - # here through the new derivation. - if new_enabled != was_enabled: + if now_on != was_on: + applied, error = await asyncio.to_thread(set_lane_enabled_sync, lane, now_on) + if applied and not now_on: + # Down to the floor at once. The pool cannot be emptied, so "off" is + # one parked process with its consumers cancelled. applied, error = await asyncio.to_thread( - set_lane_enabled_sync, lane, new_enabled, + set_lane_slots_sync, lane, MIN_POOL_SLOTS, ) - if applied and slots is not None: - applied, error = await asyncio.to_thread(set_lane_slots_sync, lane, new_slots) + elif applied and now_on: + # Only DOWNWARD. See the docstring: raising a cap is permission, and + # the sizing pass decides whether there is work to spend it on. + live = await asyncio.to_thread(inspect_lanes_sync) + current = live[lane.name].pool + if current is not None and current > slots_cap: + applied, error = await asyncio.to_thread( + set_lane_slots_sync, lane, slots_cap, live=live[lane.name], + ) - # Enabling a lane that needs models is what triggers the fetch (milestone + # Raising the cap off zero is what triggers the model download (milestone # 422 step 6). Never at boot: that made every start of the ML role reach # HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a # feature that is optional and clearly OFF. # - # Only on the TRANSITION from off to on, so re-saving slots on a lane that - # is already running does not re-enqueue. This used to test `enabled is - # True` — the field having been sent — which stopped meaning "came on" the - # moment the dial became the switch: the UI no longer sends `enabled` at - # all, so the fetch that makes the ML lane usable would never have fired. - # - # And only when the consumer change landed: enqueueing onto a queue - # nothing is consuming would leave the task pending with no explanation - # until the lane returns. + # On the TRANSITION, so re-saving a cap on a lane already running does not + # re-enqueue. And only when the consumer change landed: enqueueing onto a + # queue nothing is consuming would leave the task pending with no + # explanation until the lane returns. fetching = False - if new_enabled and not was_enabled and lane.models and applied: + if now_on and not was_on and lane.models and applied: fetching = _enqueue_model_fetch() return { "name": lane.name, - "slots": row.slots, "slots_cap": row.slots_cap, "ceiling": ceiling, - "enabled": row.enabled, - "autoscale": row.autoscale, + "enabled": now_on, "applied": applied, "apply_error": error, - # Tells the card to say a download has started rather than leaving the - # operator to wonder why a freshly enabled lane is busy. + # Tells the UI to say a download has started rather than leaving the + # operator to wonder why a lane they just turned on is busy. "fetching_models": fetching, } @@ -645,263 +594,166 @@ def _enqueue_model_fetch() -> bool: return False -def reconcile_lanes_sync( - desired: dict[str, tuple[int, bool]], - autoscaling: frozenset[str] = frozenset(), -) -> dict: - """Drive every RUNNING lane to its stored slots and enabled flag. - `desired` is lane name -> (slots, enabled), read from the database by the - caller. `autoscaling` names the lanes the autoscaler is allowed to move. - - ## For an autoscaling lane the stored value is a FLOOR, not a target - - Step 7's autoscaler raises a saturated lane's live pool without changing - its row — the row holds what the OPERATOR set. If this pass treated that - row as an exact target it would shrink the lane back on the very next - tick, and the two sweeps would fight forever at five-minute intervals: - grow, revert, grow, revert. That is lesson #4183's failure arriving - between two enforcers rather than inside one. - - So for those lanes the target becomes `max(stored, current)` — this pass - still restores a lane that came back from a restart below what the - operator set, and never takes back what the autoscaler added. Bringing it - down is the autoscaler's job, and it does so only to that same floor. This function touches no database: the celery task that schedules - it owns the sync session, and keeping the DB out of here is what lets the - same code be called from anywhere that already knows the target. - - ## Why this exists at all - - `pool_grow` is not durable. A worker that dies and is restarted by its - supervisor comes back at its ENV concurrency — silently below whatever the - operator set — and nothing in step 2's path would ever notice. Storing the - value made it survivable; this is what makes it actually survive. - - ## It must converge and then go quiet - - One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing at - all to a replica already at its target. So a settled system performs one - broker round trip per tick and sends no control messages — the reachable - fixed point lesson #4183 is about. An enforcer that re-sent a grow of zero - every tick would churn forever and bury a real correction in its own noise, - which is why `changed` below counts only lanes that actually moved. - - ## An absent lane is SKIPPED, not corrected - - `present=False` means nothing answered — a worker restarting, or a broker - that is unreachable. It does NOT mean zero slots. Correcting an absence - would be drawing a conclusion from an unswept read (snippet #3969), and - here it would be worse than useless: there is nothing to send the message - to. The lane is reported as skipped and picked up on a later tick. - """ - live = inspect_lanes_sync() - changed: list[str] = [] - skipped: list[str] = [] - failed: dict[str, str] = {} - - for lane in LANES: - target = desired.get(lane.name) - if target is None: - continue - slots, enabled = target - state = live[lane.name] - if not state.present: - skipped.append(lane.name) - continue - - # Enabled first: a lane being turned on should be consuming before - # its pool is sized, so the slots it gains have work to pick up. - # - # Only when it DISAGREES. Calling this unconditionally would send - # add_consumer for every queue on every tick of a settled system — - # the exact churn lesson #4183 describes, and invisible because - # add_consumer on a queue already consumed is harmless. - consuming_all = state.consuming.issuperset(lane.queues) - if enabled != consuming_all: - ok, err = set_lane_enabled_sync(lane, enabled, live=state) - if not ok: - failed[lane.name] = err or "could not set consumers" - continue - changed.append(lane.name) - - current = state.pool - # The floor, for a lane the autoscaler manages. See the docstring. - target_slots = slots - if lane.name in autoscaling and current is not None: - target_slots = max(slots, current) - # And the floor every lane has: a pool cannot be emptied. Applied to - # the COMPARISON, not just the send — against the raw 0 this sees a - # difference no control message can close and re-sends it every tick. - target_slots = effective_slots(target_slots) - if current is not None and current == target_slots: - continue - ok, err = set_lane_slots_sync(lane, target_slots, live=state) - if ok: - if lane.name not in changed: - changed.append(lane.name) - log.info( - "worker_control: %s reconciled %s -> %s slots", - lane.name, current, target_slots, - ) - else: - failed[lane.name] = err or "could not resize" - - return {"changed": changed, "skipped": skipped, "failed": failed} - - -# --- the autoscaler (step 7) -------------------------------------------------- +# --- the sizing pass: one sweep, always on ------------------------------------ # -# The only part of this milestone that acts without anyone asking. Everything -# above does what an operator pressed; this decides. So it is off by default, -# opted into per lane, bounded by the cap the operator set, and it reports what -# it did rather than moving numbers silently. - -# ## Why these three numbers are not in Settings +# This replaced BOTH `reconcile_lanes_sync` (step 3) and `autoscale_lanes_sync` +# (step 7) on 2026-09-23. They were two enforcers over one number, and the +# whole of step 7's hardest reasoning — a stored value that is a FLOOR, a +# target of `max(stored, current)` so the reconcile does not undo what the +# autoscaler added — existed only to stop them fighting. Delete one of them and +# the problem is not solved, it is absent. # -# Rule 25 puts anything an operator might want to tune in the UI, and the -# knobs that decide what this does ARE there: whether a lane autoscales at -# all, its cap, and its floor — all DB-backed, all changeable without a -# restart. What is left here is the POLICY's internals, and exposing them -# would add four numbers per lane to a card whose whole value is being -# readable at a glance, to tune a decision the operator has a better lever -# for. If growth turns out to be too eager or too shy in practice, that is a -# reason to change these values for everyone, not to ask each operator to -# discover them. - -# All slots busy AND this many tasks waiting before a lane may grow. +# Operator: *"auto should be always on, not a setting, so that idle instances +# quiet down when not running. the number that is visible and something the +# user can tweak and manage should be the cap itself the number of running +# workers is handled by the autoscaling function which is always on."* # -# The AND is the design. Depth with free slots means nothing — celery is about -# to pick those up, and growing the pool would add idle children. Saturation -# with an empty queue means nothing either: the lane is busy with exactly as -# much work as exists. Only both together say "there is more work than this -# lane can reach". -AUTOSCALE_BACKLOG_THRESHOLD = 10 +# So there is one pass, it runs every minute, it reads the live pool rather +# than any stored number, and the only thing it obeys is the cap. +# +# It also subsumes what the reconcile existed for. `pool_grow` is not durable: +# a worker restarted by its supervisor comes back at its ENV concurrency, +# silently below what the lane should be running. This pass reads the live +# pool every minute and sizes from the backlog, so that worker is corrected on +# the next tick — sooner than the five-minute reconcile managed, and without a +# second sweep that could disagree with this one. -# Grow by one slot per tick, never to the cap in one jump. A lane that is -# saturated because of one slow burst settles a slot or two above where it -# started rather than at its ceiling, and the next tick re-measures rather -# than committing to a guess made once. -AUTOSCALE_STEP = 1 - -# Hysteresis: shrink only when the backlog is well BELOW the grow threshold, -# not merely under it. Equal thresholds flap — one task arriving and leaving -# would grow and shrink the lane forever at the tick interval, which is -# lesson #4183's churn arriving through a different door. -AUTOSCALE_SHRINK_BELOW = 2 +# How much work justifies a slot. `pending` is depth + reserved, so it already +# counts what celery has prefetched into worker memory — one task, one slot. +# +# Growth is IMMEDIATE and shrink is one slot per tick, deliberately asymmetric. +# A backlog of four thousand should not take an hour to reach the cap, and a +# lane that idles for one minute should not drop every process it has: the +# cost of being one slot too large for a minute is a sleeping process, and the +# cost of being too small is work not happening. For ML the asymmetry matters +# most — every new slot reloads a multi-GB model, so the slow shrink is what +# stops a quiet patch from paying that cost again a minute later. +SHRINK_STEP = 1 @dataclass -class AutoscaleDecision: - """What the autoscaler did to one lane, and why — in the operator's terms. +class LaneSizing: + """What the pass did to one lane, and why — in the operator's terms. - A reason string on every outcome including "nothing", because an - autoscaler that only speaks when it acts is one nobody can debug when it - does not. + A reason on every outcome including "held", because a sizing pass that + only speaks when it acts is one nobody can debug when it does not. """ lane: str - action: str # "grew" | "shrank" | "held" + action: str # "grew" | "shrank" | "held" | "skipped" slots: int reason: str -def autoscale_lanes_sync( - lanes: dict[str, tuple[int, int, bool]], -) -> list[AutoscaleDecision]: - """Decide and apply one round of autoscaling. +def wanted_slots(cap: int, active: int, pending: int | None) -> int: + """How many workers this lane has work for right now, within its cap. - `lanes` is name -> (slots_cap, configured_slots, autoscale_on), read from - the database by the caller — this function touches no database, for the - same reason `reconcile_lanes_sync` does not. + One slot per task in flight or waiting, floored at one process and + ceilinged by the cap. `pending` of None means the broker did not answer + for this lane's queues — an unknown backlog is not an empty one (snippet + #3969), so it contributes nothing rather than being read as zero. - ## What is current, and what is the floor + A cap of zero still returns one: billiard cannot run an empty pool, and + the parked process is what `add_consumer` lands on when the cap goes back + up. "Off" is expressed by cancelling consumers, not by emptying the pool. + """ + if cap <= 0: + return MIN_POOL_SLOTS + return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0))) - The value this moves is the LIVE pool, read from `inspect`. The stored - `configured_slots` is what the operator set and is only the FLOOR: growth - goes above it and a shrink returns to it, never below. - The two are deliberately not the same number, and reading the stored value - as "current" is the mistake that makes this function useless in a way no - unit test of a single tick would show. The autoscaler never writes the - row, so the stored value never moves; a tick that computed `stored + 1` - would propose the same target forever, cap the lane one slot above the - floor no matter the load, and — because resizing a replica already at the - target issues nothing and reports success — claim `grew` on every tick - while nothing changed. Lesson #4183's non-convergence, arriving with a - success message attached. +def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]: + """Size every lane to its backlog, within the cap. The whole control loop. - So: `current = state.pool`, `configured` is the floor, and both `grew` and - `shrank` mean the live pool actually moved. + `caps` is lane name -> slots_cap, read from the database by the caller. + This function touches no database: the celery task that schedules it owns + the session, and keeping the DB out of here is what lets it be called from + anywhere that already knows the caps. + + ## It must converge and then go quiet + + One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a + replica already at its target. A settled system therefore performs one + broker round trip plus one LLEN sweep per tick and sends no control + messages at all — the reachable fixed point lesson #4183 is about. An + enforcer that re-sent a grow of zero every tick would churn forever and + bury a real correction in its own noise. + + ## An absent lane is SKIPPED, not corrected + + `present=False` means nothing answered — a worker restarting, or an + unreachable broker. It does NOT mean zero slots. Deciding from that would + be a verdict drawn from an unswept read, and here it is worse than + useless: there is nothing to send the message to. """ live = inspect_lanes_sync() depths = _queue_depths_sync() - out: list[AutoscaleDecision] = [] + out: list[LaneSizing] = [] for lane in LANES: - target = lanes.get(lane.name) - if target is None: + cap = caps.get(lane.name) + if cap is None: continue - cap, configured, on = target - if not on: + state = live[lane.name] + if not state.present: + out.append(LaneSizing(lane.name, "skipped", 0, "lane is not answering")) continue - state = live[lane.name] - if not state.present or state.pool is None: - # Nothing answered. Not "idle" — unknown, and a decision drawn - # from an unswept read is exactly what snippet #3969 warns about. - # `configured` is reported because there is no live number to - # report; it is what the lane will come back at. - out.append(AutoscaleDecision( - lane.name, "held", configured, "lane is not answering", - )) - continue + # Consumers first, and only when they DISAGREE. Sending add_consumer + # for every queue on every tick of a settled system is the exact churn + # above, and invisible: add_consumer on a queue already consumed is + # harmless and reports success. + should_consume = cap > 0 + if should_consume != state.consuming.issuperset(lane.queues): + ok, err = set_lane_enabled_sync(lane, should_consume, live=state) + if not ok: + out.append(LaneSizing( + lane.name, "held", state.pool or 0, + f"could not {'start' if should_consume else 'stop'} " + f"consuming: {err}", + )) + continue current = state.pool - known = [depths.get(q) for q in lane.queues] - if all(d is None for d in known): - out.append(AutoscaleDecision( - lane.name, "held", current, "queue depth unavailable", + if current is None: + out.append(LaneSizing( + lane.name, "held", 0, "worker did not report its pool size", )) continue - backlog = sum(d for d in known if d is not None) + state.reserved - # Against CAPACITY, not against the dial: `active` is summed across - # replicas, so comparing it to one replica's pool size would call two - # half-busy replicas of 4 saturated at 4 active and grow a lane that - # has idle slots. - saturated = state.active >= state.capacity > 0 - busy = backlog >= AUTOSCALE_BACKLOG_THRESHOLD - if saturated and busy and current < cap: - new = min(cap, current + AUTOSCALE_STEP) - ok, err = set_lane_slots_sync(lane, new, live=state) - out.append(AutoscaleDecision( - lane.name, "grew" if ok else "held", new if ok else current, - f"{backlog} waiting and all {state.capacity} slots busy" - if ok else f"could not grow: {err}", - )) - elif saturated and busy: - # At the cap with work still waiting. Said out loud rather than - # held silently: this is the operator's own ceiling doing its job, - # and it is the moment they would want to know they set it. - out.append(AutoscaleDecision( - lane.name, "held", current, - f"{backlog} waiting but the cap is {cap}", - )) - elif current > effective_slots(configured) and ( - backlog <= AUTOSCALE_SHRINK_BELOW - ): - new = max(effective_slots(configured), current - AUTOSCALE_STEP) - ok, err = set_lane_slots_sync(lane, new, live=state) - out.append(AutoscaleDecision( - lane.name, "shrank" if ok else "held", new if ok else current, - f"backlog cleared, back toward {configured}" - if ok else f"could not shrink: {err}", - )) + known = [depths.get(q) for q in lane.queues] + depth = ( + sum(d for d in known if d is not None) + if any(d is not None for d in known) else None + ) + pending = None if depth is None else depth + state.reserved + want = wanted_slots(cap, state.active, pending) + + if want > current: + new = want + verb = "grew" + elif want < current: + # One at a time on the way down. See SHRINK_STEP. + new = max(want, current - SHRINK_STEP) + verb = "shrank" else: - # The fixed point. A settled lane sends nothing and says so — - # the tick is one inspect and one LLEN sweep, no control messages. - out.append(AutoscaleDecision( + out.append(LaneSizing( lane.name, "held", current, - f"{backlog} waiting, {state.active} busy", + f"{pending if pending is not None else '?'} waiting, " + f"{state.active} busy, cap {cap}", )) + continue + + ok, err = set_lane_slots_sync(lane, new, live=state) + if not ok: + out.append(LaneSizing( + lane.name, "held", current, f"could not resize: {err}", + )) + continue + out.append(LaneSizing( + lane.name, verb, new, + f"{pending if pending is not None else '?'} waiting, " + f"{state.active} busy, cap {cap}", + )) return out diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index c36ec52..7694696 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -129,18 +129,15 @@ class Lane: # with CELERY_QUEUES=maintenance_long). Recorded here so the generated # supervisord config and the compose file cannot disagree about it. entrypoint_role: str - default_slots: int - # The cap a lane STARTS with, which is not the ceiling. Set low enough - # that raising slots within it is an ordinary adjustment, and raising the - # cap itself is a deliberate act — a cap that begins at the ceiling is a - # rubber stamp and protects nobody. + # THE cap a lane starts with — and, since 2026-09-23, the only number an + # operator sets for it. How many workers actually run is the autoscaler's + # job; this is the most it may use. Zero means the lane is off. + # + # One, and zero for ML. Deliberately far below the operator's own + # production numbers, which are tuned for their hardware and are not a + # sane first boot for a stranger — and low enough that a busy instance + # tells them to raise it rather than quietly consuming the machine. default_slots_cap: int - default_enabled: bool - # Whether a lane arrives with the autoscaler on. False for every lane, and - # the field exists anyway: this module is the one place that describes a - # lane, and leaving one operator-settable default to the column's DDL - # default would make it the only setting you cannot read here. - default_autoscale: bool = False # True when a slot costs a copy of the ML model rather than just a process. # The only lane whose ceiling is decided by memory instead of by cores. memory_bound: bool = False @@ -159,51 +156,58 @@ class Lane: return tuple(sorted(self.queues)) -# Defaults are ONE OF EACH, with ML off — operator, 2026-09-22: *"that -# starting value should be one of each."* Deliberately far below the -# operator's own production numbers (worker 8, ml 2), which are tuned for -# their hardware and are not a sane first boot for a stranger. +# ONE CAP PER LANE, and that is the whole of what an operator sets. # -# ML ships disabled because enabling it is what triggers the SigLIP download -# (milestone 422 step 6) — rule 164 allows a feature that needs a fetch only -# when it is "optional and clearly off", and off-by-default is also what keeps -# a small box from loading a multi-GB model it was never asked to load. +# Operator, 2026-09-23: *"auto should be always on, not a setting, so that +# idle instances quiet down when not running. the number that is visible and +# something the user can tweak and manage should be the cap itself the number +# of running workers is handled by the autoscaling function which is always +# on."* +# +# Until then a lane had THREE operator values — `slots`, `slots_cap` and +# `autoscale` — because the manual dial was built first (steps 2-4) and the +# autoscaler arrived last (step 7) as an opt-in beside a control that already +# existed. Nothing ever asked whether the dial should still exist once +# something could move it automatically. It should not: "how many are running +# right now" is a measurement, not a preference. +# +# One of each, and ML at zero. ML at zero is also rule 164's carve-out: a cap +# of zero means no consumers, so a fresh install never loads a model or +# reaches HuggingFace, and raising the cap is what triggers the fetch. +# +# These are far below the operator's own production numbers, and deliberately +# so — they are what a stranger's first boot should do, not what a tuned +# machine can. The UI is what closes that gap: a lane sitting at its cap with +# a backlog says so, and says raising the cap is the fix. Without that a +# conservative default is just a slow instance nobody knows how to speed up. LANES: tuple[Lane, ...] = ( Lane( name="worker", display_name="Worker", queues=("default", "import", "thumbnail", "download"), entrypoint_role="worker", - default_slots=1, - default_slots_cap=4, - default_enabled=True, + default_slots_cap=1, ), Lane( name="scheduler", display_name="Scheduler", queues=("maintenance", "scan"), entrypoint_role="scheduler", - default_slots=1, - default_slots_cap=2, - default_enabled=True, + default_slots_cap=1, ), Lane( name="maintenance_long", display_name="Long maintenance", queues=("maintenance_long",), entrypoint_role="worker", - default_slots=1, - default_slots_cap=2, - default_enabled=True, + default_slots_cap=1, ), Lane( name="ml", display_name="ML tagging", queues=("ml",), entrypoint_role="ml-worker", - default_slots=0, - default_slots_cap=1, - default_enabled=False, + default_slots_cap=0, memory_bound=True, models=(SIGLIP_MODEL,), optional=True, @@ -242,6 +246,22 @@ ML_BYTES_PER_SLOT = SIGLIP_MODEL.approx_resident_bytes # the processes an OOM kill must not take (see the module docstring). RESERVED_BYTES = 2 * GIB +# The smallest pool a lane can actually run: ONE process, never zero. +# +# billiard refuses to remove the last worker in a pool, so a lane asked to +# shrink to nothing gets `ValueError("Can't shrink pool. All processes +# busy!")` and the sizing pass re-sends the doomed message forever. Found on +# the operator's live deploy, 2026-09-23. +# +# It is also what makes "off" expressible: a lane at cap 0 keeps this one +# parked process with its consumers cancelled, so it still answers `inspect` +# (and so reads as present rather than crashed), and `add_consumer` has +# something to reach when the cap goes back up. +# +# Lives HERE rather than in `worker_control` because `gen_supervisord` needs +# it at container boot and must not import the models package to get it. +MIN_POOL_SLOTS = 1 + # The floor a cores-derived ceiling never goes below. A single-core box still # needs to be able to run its lanes; the ceiling exists to stop absurd values, # not to make a small machine unusable. diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index b13ac68..bfcaee3 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -1349,107 +1349,55 @@ def sync_memberships() -> str: parts.append(f"suggested={res['suggested']}") return " ".join(parts) or "no platforms" +@celery.task(name="backend.app.tasks.maintenance.size_worker_lanes") +def size_worker_lanes() -> dict: + """Size every lane to its backlog, within the cap the operator set. -@celery.task(name="backend.app.tasks.maintenance.reconcile_worker_lanes") -def reconcile_worker_lanes() -> dict: - """Drive every running lane back to the slots the operator set. + ONE sweep, replacing `reconcile_worker_lanes` and `autoscale_worker_lanes` + (2026-09-23). They were two enforcers over one number: the reconcile drove + the pool to a stored `slots`, the autoscaler moved it away from that same + value, and most of the autoscaler's design existed to keep the reconcile + from undoing its work. Deleting the stored number deletes the conflict. - Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by - its supervisor comes back at its ENV concurrency, silently below whatever - was configured, and nothing on step 2's write path would ever notice. + It still does what the reconcile existed for. `pool_grow` is not durable — + a worker restarted by its supervisor comes back at its ENV concurrency, + silently below what the lane should run — and this reads the LIVE pool + every minute, so that worker is corrected on the next tick rather than + after five. - ## Why a beat task and not a hook in web - - Step 3 was written as "web applies the stored values after it starts". It - cannot, and `services/service_roster.py` already records why: hypercorn - runs `--workers 4`, so anything in `before_serving` becomes FOUR - concurrent loops per container, all hammering the broker forever. - - The other option was service_roster's own answer — refresh on demand from - whichever request happens to arrive. Rejected here because the two are - solving different problems. A stale ROSTER only misleads someone who is - looking at it, so recomputing it when they look is exactly right. A lane - running at the wrong size is doing less work than it was told to whether - or not anyone is watching — and the case that matters is a deploy at 3am - followed by a backlog nobody is awake to notice. - - So: unattended, on the quick `maintenance` lane (its module routes it - there), beside the other recovery sweeps. The accepted cost is that a dead - scheduler stops reconciliation — but a dead scheduler already stops every - other sweep, and the roster reports it, so this adds no new blind spot. - - ## Quiet when settled - - One broker round trip per tick, and no control messages at all once every - lane matches. Returns the lanes it actually moved, so the log shows a - correction rather than a heartbeat. + Returns every lane's outcome INCLUDING the ones it held, each with a + reason. A pass that only speaks when it acts cannot be debugged on the day + it does not. """ from ..models import WorkerLane - from ..services.worker_control import reconcile_lanes_sync + from ..services.worker_control import size_lanes_sync - # Both dicts built INSIDE the session. Reading a column off a detached - # instance happens to work while the attribute is still loaded and stops - # working the moment anything expires it — a failure that would appear - # long after this line, in a sweep nobody is watching. + # Read INSIDE the session. Reading a column off a detached instance + # happens to work while the attribute is still loaded and stops working + # the moment anything expires it — a failure that would appear long after + # this line, in a sweep nobody is watching. with _sync_session_factory()() as session: - rows = list(session.execute(select(WorkerLane)).scalars()) - desired = {row.name: (row.slots, row.enabled) for row in rows} - # Lanes the autoscaler may move. For these the stored value is a - # FLOOR: this sweep restores a lane that fell below it and never takes - # back what the autoscaler added, or the two would fight every five - # minutes. - autoscaling = frozenset(row.name for row in rows if row.autoscale) - if not desired: - # Migration 0103 seeds these, so an empty table means it has not run - # yet. Nothing to assert — and inventing defaults here would let this - # task disagree with the seed it is supposed to be enforcing. - return {"changed": [], "skipped": [], "failed": {}} - return reconcile_lanes_sync(desired, autoscaling) - - -@celery.task(name="backend.app.tasks.maintenance.autoscale_worker_lanes") -def autoscale_worker_lanes() -> dict: - """Grow a saturated lane, within the cap the operator set. - - Milestone 422 step 7, and the only sweep in this milestone that decides - rather than obeys. Everything else applies what someone pressed. - - OFF unless a lane opts in. A no-op costs one `celery inspect` and one LLEN - sweep and sends no control messages — the same fixed point the reconcile - holds, and for the same reason: this runs forever, so a settled system has - to be silent or the real signal drowns in its own heartbeat. - - Returns every lane's decision INCLUDING the ones it held, each with a - reason. An autoscaler that only speaks when it acts cannot be debugged on - the day it does not. - """ - from ..models import WorkerLane - from ..services.worker_control import autoscale_lanes_sync - - with _sync_session_factory()() as session: - lanes = { - # The stored `slots` is passed ONLY as the floor. What the - # autoscaler moves is the live pool, which it reads itself — this - # row is never written, so using it as the current value would pin - # every decision to the same number forever. - row.name: (row.slots_cap, row.slots, True) + caps = { + row.name: row.slots_cap for row in session.execute(select(WorkerLane)).scalars() - if row.autoscale } - if not lanes: - return {"decisions": []} + if not caps: + # Migration 0103/0105 seed these, so an empty table means they have + # not run yet. Nothing to assert — and inventing defaults here would + # let this task disagree with the seed it is meant to be enforcing. + return {"sized": []} - decisions = autoscale_lanes_sync(lanes) - for d in decisions: - if d.action != "held": + sized = size_lanes_sync(caps) + for d in sized: + if d.action not in ("held", "skipped"): log.info( - "autoscale: %s %s to %s slots — %s", + "worker lanes: %s %s to %s slots — %s", d.lane, d.action, d.slots, d.reason, ) return { - "decisions": [ + "sized": [ {"lane": d.lane, "action": d.action, "slots": d.slots, "reason": d.reason} - for d in decisions + for d in sized ], } diff --git a/frontend/src/components/settings/SystemHealthTab.vue b/frontend/src/components/settings/SystemHealthTab.vue index d5e4364..ae24f91 100644 --- a/frontend/src/components/settings/SystemHealthTab.vue +++ b/frontend/src/components/settings/SystemHealthTab.vue @@ -55,10 +55,9 @@ Part Queues - Pending - Busy - Slots - Auto + Waiting + Workers + Max workers @@ -85,7 +84,15 @@ memory to change nothing. Seeing a lane wedged on one slow job is the useful half. -->
- all slots busy for {{ row.stuckFor }} + all workers busy for {{ row.stuckFor }} +
+ +
+ mdi-arrow-up-bold-outline + {{ row.advice }}
@@ -98,6 +105,9 @@ {{ row.lane ? (row.lane.pending ?? '—') : '—' }} + — - - - - — - - +

- Slots is how many tasks a lane runs at once. Changes - reach the running worker immediately and survive a restart. - Zero slots turns the lane off — it keeps its process - and stops taking work, so it stays listed here rather than looking like - a crash. Three of the four lanes need to be running for FabledCurator to - work at all; ML tagging is the one that is genuinely optional. The - of N beneath each dial is the most that lane may have on this - machine. + Max workers is the only thing you set. How many + workers a lane is actually running at any moment is decided for you — + it grows to meet a backlog and gives the workers back when the queue + empties, so an idle instance settles down to one of each without being + told to. The cap is the ceiling on that, never a target, so raising it + costs nothing until there is work to spend it on.

- Auto lets a lane add slots by itself when its - queue is backed up and every slot it has is busy — one at a - time, never past of N — and hand them back once the backlog - clears. Off means the lane stays at exactly the number you set. It is - off by default, per lane, because this is the only thing on this page - that acts without being asked. A lane already dialled to its maximum has - nowhere to grow, so its switch stays unavailable until you leave it some - room. + A cap of zero turns the lane off. It keeps one parked + process and stops taking work, so it stays listed here rather than + looking like a crash. Three of the four lanes need to be running for + FabledCurator to work at all; ML tagging is the one that is genuinely + optional, and it ships at zero because switching it on downloads a + model. Up to N beneath each dial is what this machine can + hold — memory for ML tagging, processor cores for the rest — and it is + recalculated from the container's real limits every time this page + loads. +

+

+ The shipped caps are one of each, which is right for a first boot and + wrong for a busy library. A lane that is running everything its cap + allows while work piles up will say so in its row, and + raising that cap is the answer when it does.

A part is called stale after @@ -204,7 +208,7 @@ you are not running the GPU agent, which does the same work faster.

-

Giving it a slot downloads, once:

+

Raising its cap above zero downloads, once:

  • {{ m.repo }} — @@ -214,8 +218,8 @@

- Each slot loads its own copy, which is why this machine allows it at - most {{ lane.ceiling }}. + Each worker loads its own copy, which is why this machine allows it + at most {{ lane.ceiling }}. @@ -230,7 +234,7 @@ import { computed, onMounted, onUnmounted, ref } from 'vue' import { laneStuckFor, useSystemActivityStore } from '../../stores/systemActivity.js' import { useSystemHealthStore } from '../../stores/systemHealth.js' import { formatRelative } from '../../utils/date.js' -import { mergeParts } from '../../utils/systemParts.js' +import { laneAdvice, mergeParts } from '../../utils/systemParts.js' const store = useSystemHealthStore() const lanesStore = useSystemActivityStore() @@ -266,11 +270,11 @@ onUnmounted(() => { clearInterval(pollId) }) // that removed it. const rows = computed(() => mergeParts( store.parts, lanesStore.lanes?.lanes ?? [], laneStuckFor, -)) +).map((row) => ({ ...row, advice: laneAdvice(row.lane) }))) const offOptionalLanes = computed(() => (lanesStore.lanes?.lanes ?? []).filter( - (l) => l.optional && l.slots === 0 && l.models?.length, + (l) => l.optional && l.slots_cap === 0 && l.models?.length, ), ) @@ -319,23 +323,12 @@ async function apply(lane, fields) { } } -// The dial IS the switch — the API derives `enabled` from the number, so -// stepping to zero turns the lane off and stepping off zero turns it on. -// Nothing here sends `enabled`, and there is no second control that could -// disagree with the number on screen. +// The cap is the only thing an operator sets, and it doubles as the switch: +// zero means no consumers. How many workers actually run is the sizing pass's +// business — always on, reading the live pool every minute — so nothing here +// sends a worker count. function step(lane, delta) { - return apply(lane, { slots: lane.slots + delta }) -} - -// Room to grow into. The autoscaler moves the LIVE pool, but the floor it -// starts from is the stored value, so a lane already dialled to its cap has -// nowhere to go and turning this on would do nothing at all. -function canGrow(lane) { - return lane.slots_cap > lane.slots -} - -function setAutoscale(lane, value) { - return apply(lane, { autoscale: Boolean(value) }) + return apply(lane, { slots_cap: lane.slots_cap + delta }) } @@ -390,6 +383,10 @@ function setAutoscale(lane, value) { font-size: 0.8rem; color: rgb(var(--v-theme-on-surface) / 0.72); padding-left: 17px; } +.fc-parts__advice { + font-size: 0.8rem; padding-left: 17px; margin-top: 2px; + color: rgb(var(--v-theme-accent)); +} .fc-parts__queues { font-size: 0.78rem; color: rgb(var(--v-theme-on-surface) / 0.6); } diff --git a/frontend/src/utils/systemParts.js b/frontend/src/utils/systemParts.js index e222edf..7eb6510 100644 --- a/frontend/src/utils/systemParts.js +++ b/frontend/src/utils/systemParts.js @@ -54,10 +54,10 @@ export function mergeParts(parts, lanes, stuckFor = () => null) { name: part.name, kindLabel: lane?.optional ? 'optional lane' : kindLabel(part.kind), state: part.state, - // A lane dialled to zero is OFF, not broken. Say so, rather than let the + // A lane capped at zero is OFF, not broken. Say so, rather than let the // roster's heartbeat sentence report the operator's own choice as a // fault — the roster cannot know the difference, and the lane can. - detail: lane && lane.slots === 0 ? 'off — no slots' : part.detail, + detail: lane && lane.slots_cap === 0 ? 'off — cap is zero' : part.detail, queues: (part.queues || []).join(', '), lane, stuckFor: lane ? stuckFor(lane) : null, @@ -80,7 +80,7 @@ export function mergeParts(parts, lanes, stuckFor = () => null) { } function laneRow(lane, stuckFor) { - const on = lane.slots > 0 + const on = lane.slots_cap > 0 let state = 'unknown' if (lane.live?.present) state = on ? (stuckFor(lane) ? 'stale' : 'ok') : 'unknown' else if (on) state = 'down' @@ -90,7 +90,7 @@ function laneRow(lane, stuckFor) { kindLabel: lane.optional ? 'optional lane' : 'worker lane', state, detail: lane.live?.present - ? (on ? 'running' : 'off — no slots') + ? (on ? 'running' : 'off — cap is zero') : 'has not checked in yet', queues: (lane.queues || []).join(', '), lane, @@ -98,3 +98,47 @@ function laneRow(lane, stuckFor) { severity: SEVERITY[state], } } + + +// How much has to be waiting before we tell someone to raise a cap. +// +// Not "anything at all". A lane at its cap with three items queued is working +// normally and will be empty in a moment; a notice there is one people learn +// to scroll past, and at that point it is worse than not having it. +export const ADVISE_BACKLOG = 10 + +/** + * The sentence that tells an operator the cap is now the limiting factor. + * + * Operator, 2026-09-23: *"there needs to be something that tells you user to + * bump those numbers to improve processing rate or they'd never know the + * controls exist."* The defaults are deliberately one-of-each, so on a busy + * instance the shipped configuration IS the bottleneck — and a conservative + * default nobody knows how to raise is just a slow product. + * + * Only fires when raising the cap would actually help: there is real work + * waiting, every worker the cap allows is already running, and the cap is + * below what this machine can hold. A lane already at its ceiling gets + * nothing, because there is nothing it could be told to do. + */ +export function laneAdvice(lane) { + if (!lane) return null + const pending = lane.pending + if (pending == null || pending < ADVISE_BACKLOG) return null + + if (lane.slots_cap === 0) { + return `${pending.toLocaleString()} waiting, and this lane is off. ` + + 'Raise its cap to start working through them.' + } + if (lane.ceiling <= lane.slots_cap) { + // At the machine's limit, not the operator's. Saying "raise the cap" + // here would be advice they cannot take. + return null + } + if (!lane.live?.present || (lane.live.pool ?? 0) < lane.slots_cap) return null + + return `${pending.toLocaleString()} waiting and all ${lane.slots_cap} ` + + `worker${lane.slots_cap === 1 ? '' : 's'} busy. ` + + `Raise the cap to run more at once — this machine allows up to ` + + `${lane.ceiling}.` +} diff --git a/frontend/test/systemParts.spec.js b/frontend/test/systemParts.spec.js index 894b09d..7879b01 100644 --- a/frontend/test/systemParts.spec.js +++ b/frontend/test/systemParts.spec.js @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest' -import { mergeParts, queueKey } from '../src/utils/systemParts.js' +import { laneAdvice, mergeParts, queueKey } from '../src/utils/systemParts.js' // The System tab's one table (operator 2026-09-23: "combine the two sections // into a single table"). What is pinned here is the JOIN, because its failure @@ -25,9 +25,8 @@ const LANE = { // sorted. If the join ever compares these two lists directly rather than as // sets, this fixture is what catches it. queues: ['default', 'import', 'thumbnail', 'download'], - slots: 2, slots_cap: 4, - autoscale: false, + ceiling: 8, live: { present: true, replicas: 1, pool: 2, active: 0 }, pending: 0, } @@ -71,7 +70,7 @@ describe('mergeParts', () => { // is exactly the one with no roster entry. const ml = { ...LANE, name: 'ml', display_name: 'ML tagging', queues: ['ml'], - slots: 0, optional: true, + slots_cap: 0, optional: true, live: { present: false, replicas: 0, pool: null, active: 0 }, } const rows = mergeParts([POSTGRES], [ml]) @@ -81,15 +80,15 @@ describe('mergeParts', () => { expect(row.kindLabel).toBe('optional lane') }) - it('does not call a lane at zero slots broken', () => { + it('does not call a lane capped at zero broken', () => { // The roster only knows a heartbeat age, so it goes on saying "is running" // for a lane the operator deliberately dialled to nothing. The lane knows // the difference; reporting the operator's own choice as a fault is how an // indicator stops being read. - const off = { ...LANE, slots: 0 } + const off = { ...LANE, slots_cap: 0 } const row = mergeParts([PART], [off])[0] - expect(row.detail).toBe('off — no slots') + expect(row.detail).toBe('off — cap is zero') }) it('puts the broken thing first, whatever it is', () => { @@ -126,3 +125,68 @@ describe('mergeParts', () => { expect(mergeParts(undefined, undefined)).toEqual([]) }) }) + + +// The nudge. Operator, 2026-09-23: *"there needs to be something that tells +// you user to bump those numbers to improve processing rate or they'd never +// know the controls exist."* +// +// The caps ship at one of each, so on a busy instance the SHIPPED +// CONFIGURATION is the bottleneck. What is pinned here is that it fires only +// when raising the cap would actually help — a notice on a lane that is +// keeping up, or one already at the machine's limit, is a notice people learn +// to scroll past, and at that point it is worse than not having it. + +describe('laneAdvice', () => { + const busyAtCap = { + slots_cap: 1, ceiling: 7, pending: 4060, + live: { present: true, pool: 1, active: 1 }, + } + + it('speaks when the cap is the limiting factor', () => { + const advice = laneAdvice(busyAtCap) + expect(advice).toContain('4,060 waiting') + expect(advice).toContain('Raise the cap') + // The headroom, so it is an instruction rather than a complaint. + expect(advice).toContain('7') + }) + + it('says something different about a lane that is switched off', () => { + const advice = laneAdvice({ ...busyAtCap, slots_cap: 0 }) + expect(advice).toContain('this lane is off') + }) + + it('stays quiet when the lane is keeping up', () => { + expect(laneAdvice({ ...busyAtCap, pending: 2 })).toBeNull() + }) + + it('stays quiet when the cap is not the thing holding it back', () => { + // Four allowed, two running — the sizing pass has room it has not taken, + // so the queue is not the cap's fault. + expect(laneAdvice({ + ...busyAtCap, slots_cap: 4, live: { present: true, pool: 2, active: 2 }, + })).toBeNull() + }) + + it('stays quiet at the machine ceiling, where the advice is untakeable', () => { + // The one case where saying "raise the cap" would send someone to a + // control that will refuse them. + expect(laneAdvice({ ...busyAtCap, ceiling: 1 })).toBeNull() + }) + + it('stays quiet about a lane that is not answering', () => { + // An unswept read is not a verdict: nothing replied, so "all workers + // busy" is a claim nobody made. + expect(laneAdvice({ + ...busyAtCap, live: { present: false, pool: null, active: 0 }, + })).toBeNull() + }) + + it('stays quiet when the backlog is unknown rather than reading it as huge', () => { + expect(laneAdvice({ ...busyAtCap, pending: null })).toBeNull() + }) + + it('says nothing about a row that is not a lane at all', () => { + expect(laneAdvice(undefined)).toBeNull() + }) +}) diff --git a/frontend/test/workerLanes.spec.js b/frontend/test/workerLanes.spec.js index f553e57..b594853 100644 --- a/frontend/test/workerLanes.spec.js +++ b/frontend/test/workerLanes.spec.js @@ -28,13 +28,13 @@ const LANES_BODY = { { name: 'worker', display_name: 'Worker', queues: ['default', 'import', 'thumbnail', 'download'], - slots: 1, slots_cap: 4, ceiling: 8, enabled: true, memory_bound: false, + slots_cap: 1, ceiling: 8, enabled: true, memory_bound: false, live: { present: true, replicas: 1, pool: 1, active: 0, reserved: 3 }, queue_depth: 5, pending: 8, }, { name: 'ml', display_name: 'ML tagging', queues: ['ml'], - slots: 0, slots_cap: 1, ceiling: 2, enabled: false, memory_bound: true, + slots_cap: 0, ceiling: 2, enabled: false, memory_bound: true, live: { present: false, replicas: 0, pool: null, active: 0, reserved: 0 }, queue_depth: null, pending: null, }, @@ -64,23 +64,22 @@ describe('worker lanes store', () => { }) it('setLane posts only the fields it was given', async () => { - // Partial update: the stepper sends slots without restating a cap it did - // not touch. Sending the whole row back would make two operators editing - // different fields clobber each other. + // One field: the cap. How many workers are running is a measurement the + // sizing pass owns, so there is nothing else for the UI to send. const calls = [] stubFetch((url, init) => { calls.push({ url, init }) if (init?.method === 'POST') { - return { status: 200, body: { name: 'worker', slots: 2, applied: true } } + return { status: 200, body: { name: 'worker', slots_cap: 2, applied: true } } } return { status: 200, body: LANES_BODY } }) const s = useSystemActivityStore() - await s.setLane('worker', { slots: 2 }) + await s.setLane('worker', { slots_cap: 2 }) const post = calls.find((c) => c.init?.method === 'POST') expect(post.url).toContain('/api/system/workers/worker') - expect(JSON.parse(post.init.body)).toEqual({ slots: 2 }) + expect(JSON.parse(post.init.body)).toEqual({ slots_cap: 2 }) }) it('setLane refetches so the card shows the server truth, not the guess', async () => { @@ -94,7 +93,7 @@ describe('worker lanes store', () => { return { status: 200, body: LANES_BODY } }) const s = useSystemActivityStore() - await s.setLane('worker', { slots: 2 }) + await s.setLane('worker', { slots_cap: 2 }) expect(gets).toBe(1) }) @@ -106,13 +105,13 @@ describe('worker lanes store', () => { if (init?.method === 'POST') { return { status: 200, - body: { applied: false, apply_error: 'lane is not running', slots: 2 }, + body: { applied: false, apply_error: 'lane is not running', slots_cap: 2 }, } } return { status: 200, body: LANES_BODY } }) const s = useSystemActivityStore() - const reply = await s.setLane('worker', { slots: 2 }) + const reply = await s.setLane('worker', { slots_cap: 2 }) expect(reply.applied).toBe(false) expect(reply.apply_error).toContain('not running') }) @@ -182,7 +181,7 @@ describe('laneStuckFor', () => { }) it('says nothing when the pool is zero', () => { - // A lane sized to zero is not "fully busy at zero" — it is off. + // A lane with no workers is not "fully busy at zero" — it is off. expect(laneStuckFor(lane({ live: { present: true, pool: 0, active: 0 } }))) .toBeNull() }) diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py index d36e1fd..d39f4fe 100644 --- a/tests/test_api_workers.py +++ b/tests/test_api_workers.py @@ -47,237 +47,186 @@ async def test_get_lists_every_lane_with_its_ceiling(client, no_live_workers): assert set(by_name) == {"worker", "scheduler", "maintenance_long", "ml"} for lane in body["lanes"]: assert lane["ceiling"] >= 0 - assert lane["slots"] <= lane["slots_cap"] + assert lane["slots_cap"] <= lane["ceiling"] # Nothing is running, so live state must say so rather than report # zeroes that read like a healthy idle lane. assert lane["live"]["present"] is False +@pytest.mark.asyncio +async def test_enabled_is_derived_from_the_cap_and_never_stored( + client, no_live_workers, +): + """The reshape of 2026-09-23. "Off" and "may use no workers" were two + spellings of one fact, stored separately and free to disagree.""" + body = await (await client.get("/api/system/workers")).get_json() + for lane in body["lanes"]: + assert lane["enabled"] == (lane["slots_cap"] > 0), lane["name"] + + @pytest.mark.asyncio async def test_ml_ships_off(client, no_live_workers): - """Rule 164's carve-out and the weak-hardware default in one row: enabling - the lane is what triggers the SigLIP download, so a fresh install must not - find it on.""" + """Rule 164's carve-out and the weak-hardware default in one row: raising + the cap is what triggers the SigLIP download, so a fresh install must not + find it above zero.""" body = await (await client.get("/api/system/workers")).get_json() ml = next(lane for lane in body["lanes"] if lane["name"] == "ml") + assert ml["slots_cap"] == 0 assert ml["enabled"] is False - assert ml["slots"] == 0 + + +@pytest.mark.asyncio +async def test_the_other_lanes_ship_at_one(client, no_live_workers): + """Operator: *"the cap defaults should be 1 and 0 for the ml-worker."*""" + body = await (await client.get("/api/system/workers")).get_json() + caps = {lane["name"]: lane["slots_cap"] for lane in body["lanes"]} + assert caps == { + "worker": 1, "scheduler": 1, "maintenance_long": 1, "ml": 0, + } # --- the persist / push split ------------------------------------------------ @pytest.mark.asyncio -async def test_a_value_is_stored_even_when_it_cannot_be_pushed( - client, db, no_live_workers +async def test_a_cap_is_stored_even_when_it_cannot_be_pushed( + client, db, no_live_workers, ): - """THE test for this step. `pool_grow` is not durable and the lane is not - answering, so the push fails — and the setting must survive anyway, or a - UI that saved while a worker was restarting would silently lose it - (lesson #4202). 200 with `applied: false`, not an error. - """ - resp = await client.post("/api/system/workers/worker", json={"slots": 3}) + """Nothing is answering, so the live push fails. That is NOT a failed + setting: the value is saved and the sizing pass carries it within a + minute (lesson #4202 — a live change that does not survive, with nothing + saying so).""" + resp = await client.post("/api/system/workers/worker", json={"slots_cap": 3}) + assert resp.status_code == 200 body = await resp.get_json() - - assert body["slots"] == 3 + assert body["slots_cap"] == 3 assert body["applied"] is False assert "not running" in body["apply_error"] - assert (await _lane_row(db, "worker")).slots == 3 + assert (await _lane_row(db, "worker")).slots_cap == 3 + + +# --- the cap is the switch --------------------------------------------------- @pytest.mark.asyncio -async def test_a_partial_update_leaves_the_other_fields_alone( - client, db, no_live_workers +async def test_a_cap_of_zero_turns_the_lane_off(client, db, no_live_workers): + await client.post("/api/system/workers/worker", json={"slots_cap": 0}) + + row = await _lane_row(db, "worker") + assert row.slots_cap == 0 + body = await (await client.get("/api/system/workers")).get_json() + worker = next(lane for lane in body["lanes"] if lane["name"] == "worker") + assert worker["enabled"] is False + + +@pytest.mark.asyncio +async def test_raising_it_off_zero_turns_the_lane_on(client, db, no_live_workers): + await client.post("/api/system/workers/ml", json={"slots_cap": 1}) + + assert (await _lane_row(db, "ml")).slots_cap == 1 + body = await (await client.get("/api/system/workers")).get_json() + ml = next(lane for lane in body["lanes"] if lane["name"] == "ml") + assert ml["enabled"] is True + + +@pytest.mark.asyncio +async def test_the_model_fetch_fires_on_the_transition_not_on_every_write( + client, db, no_live_workers, monkeypatch, ): - """The stepper sends `{"slots": n}` without restating a cap it did not - touch.""" - before = await _lane_row(db, "worker") - original_cap = before.slots_cap + """Raising the cap off zero downloads SigLIP, once. A second nudge of the + same dial must not re-enqueue a multi-GB download — and the trigger must + be the TRANSITION rather than "a field was sent", which is what it tested + before the UI stopped sending `enabled` at all.""" + monkeypatch.setattr( + wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None), + ) + monkeypatch.setattr( + wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None), + ) + fired = [] + monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) - await client.post("/api/system/workers/worker", json={"slots": 2}) + first = await (await client.post( + "/api/system/workers/ml", json={"slots_cap": 1}, + )).get_json() + second = await (await client.post( + "/api/system/workers/ml", json={"slots_cap": 2}, + )).get_json() - await db.refresh(before) - assert before.slots == 2 - assert before.slots_cap == original_cap + assert first["fetching_models"] is True + assert second["fetching_models"] is False + assert fired == [1] # --- what is refused --------------------------------------------------------- -@pytest.mark.asyncio -async def test_slots_above_the_cap_are_refused_and_nothing_is_stored( - client, db, no_live_workers -): - row = await _lane_row(db, "worker") - resp = await client.post( - "/api/system/workers/worker", json={"slots": row.slots_cap + 1}, - ) - assert resp.status_code == 400 - body = await resp.get_json() - assert body["error"] == "refused" - # The sentence the UI shows. A greyed control with no reason reads as a bug. - assert "cap" in body["detail"] - - await db.refresh(row) - assert row.slots <= row.slots_cap - - @pytest.mark.asyncio async def test_a_cap_above_the_derived_ceiling_is_refused( - client, db, no_live_workers + client, db, no_live_workers, ): - """The operator's cap is theirs to set, but not past what the container - can hold — the whole point of deriving a ceiling rather than typing one.""" + """The ceiling is the machine's, not the operator's, and it is the one + bound they cannot lower themselves past. The detail is written to be read + by a person — a refused control with no reason reads as a bug.""" + before = (await _lane_row(db, "ml")).slots_cap + resp = await client.post( "/api/system/workers/ml", json={"slots_cap": 10_000}, ) + assert resp.status_code == 400 body = await resp.get_json() - assert body["error"] == "refused" assert "container can hold" in body["detail"] + await _refreshed(db, "ml", before) @pytest.mark.asyncio -async def test_a_boolean_is_not_accepted_as_a_slot_count(client, no_live_workers): - """`True` is an int in Python. Coerced, it would silently set one slot — - a control that appears to work and does something nobody asked for.""" - resp = await client.post("/api/system/workers/worker", json={"slots": True}) +async def test_a_negative_cap_is_refused(client, db, no_live_workers): + resp = await client.post("/api/system/workers/worker", json={"slots_cap": -1}) assert resp.status_code == 400 - body = await resp.get_json() - assert body["error"] == "invalid_body" + + +@pytest.mark.asyncio +async def test_a_boolean_is_not_accepted_as_a_cap(client, no_live_workers): + """`True` is an int in Python. Reading it as a cap of 1 would be a control + that appears to work and sets something nobody asked for.""" + resp = await client.post("/api/system/workers/worker", json={"slots_cap": True}) + assert resp.status_code == 400 + + +@pytest.mark.asyncio +async def test_the_retired_fields_are_no_longer_accepted(client, no_live_workers): + """`slots`, `enabled` and `autoscale` are gone. A client still sending one + must be told, not silently ignored — a POST that returns 200 having + changed nothing is the worst of the three outcomes.""" + for field in ("slots", "enabled", "autoscale"): + resp = await client.post( + "/api/system/workers/worker", json={field: 2}, + ) + assert resp.status_code == 400, field @pytest.mark.asyncio async def test_an_unknown_lane_is_refused_and_names_the_known_ones( - client, no_live_workers + client, no_live_workers, ): - resp = await client.post("/api/system/workers/nonsense", json={"slots": 1}) + resp = await client.post("/api/system/workers/nope", json={"slots_cap": 1}) assert resp.status_code == 400 body = await resp.get_json() - assert body["error"] == "unknown_lane" assert "worker" in body["known"] @pytest.mark.asyncio async def test_an_empty_body_is_refused_rather_than_treated_as_a_no_op( - client, no_live_workers + client, no_live_workers, ): - """A POST that changes nothing and returns 200 is indistinguishable from - one that worked, which is how a broken UI control goes unnoticed.""" resp = await client.post("/api/system/workers/worker", json={}) assert resp.status_code == 400 - body = await resp.get_json() - assert body["error"] == "invalid_body" -# --- the dial is the switch -------------------------------------------------- -# -# Operator, 2026-09-23: *"almost all of it always needs to run there's only one -# optional piece and it is killed by moving the 'cap' to zero."* So `enabled` -# is derived from the number rather than being a second control the operator -# has to keep in agreement with it. It stays on the API — these assert that it -# still does, because it is the mechanism the reconcile and the healthcheck -# read. - - -@pytest.mark.asyncio -async def test_dialling_a_lane_to_zero_turns_it_off(client, db, no_live_workers): - await client.post("/api/system/workers/worker", json={"slots": 0}) - - row = await _lane_row(db, "worker") - assert row.slots == 0 - assert row.enabled is False - - -@pytest.mark.asyncio -async def test_dialling_it_back_up_turns_it_on(client, db, no_live_workers): - await client.post("/api/system/workers/ml", json={"slots": 1}) - - row = await _lane_row(db, "ml") - assert row.slots == 1 - assert row.enabled is True, "the lane the operator just asked for work from" - - -@pytest.mark.asyncio -async def test_an_explicit_enabled_still_wins(client, db, no_live_workers): - """The field is not removed, only derived when absent. Something that - genuinely wants a lane holding its process with consumers cancelled — a - drain before a restart — must still be able to say so without having to - destroy the operator's slot count to express it.""" - await client.post( - "/api/system/workers/worker", json={"slots": 3, "enabled": False}, - ) - - row = await _lane_row(db, "worker") - assert (row.slots, row.enabled) == (3, False) - - -@pytest.mark.asyncio -async def test_a_cap_only_write_does_not_decide_the_switch( - client, db, no_live_workers, -): - """Only the SLOTS dial derives it. A cap is a ceiling, not a request for - work, and letting it flip the lane would make raising a ceiling start - something.""" - before = await _lane_row(db, "ml") - assert before.enabled is False - - await client.post("/api/system/workers/ml", json={"slots_cap": 1}) - - await db.refresh(before) - assert (before.slots, before.enabled) == (0, False) - - -@pytest.mark.asyncio -async def test_the_model_fetch_fires_on_the_transition_not_on_the_field( - client, db, no_live_workers, monkeypatch, -): - """The trap the derivation set, caught here rather than in production. - - The fetch used to be conditioned on `enabled is True` — the FIELD having - been sent. The UI no longer sends it at all, so the download that makes - the ML lane usable would simply never have fired, and the lane would have - come on and sat there consuming a queue it had no model for. - """ - fired = [] - monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) - monkeypatch.setattr( - wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None), - ) - monkeypatch.setattr( - wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None), - ) - - body = await (await client.post( - "/api/system/workers/ml", json={"slots": 1}, - )).get_json() - - assert body["fetching_models"] is True - assert fired == [1] - - -@pytest.mark.asyncio -async def test_it_does_not_fire_again_on_a_lane_already_running( - client, db, no_live_workers, monkeypatch, -): - """The other half. A second nudge of the dial on a lane that is already on - must not re-enqueue a multi-GB download.""" - monkeypatch.setattr( - wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None), - ) - monkeypatch.setattr( - wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None), - ) - await client.post("/api/system/workers/ml", json={"slots": 1}) - - fired = [] - monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) - - body = await (await client.post( - "/api/system/workers/ml", json={"slots": 1}, - )).get_json() - - assert body["fetching_models"] is False - assert fired == [] +async def _refreshed(db, name: str, expected: int) -> None: + row = await _lane_row(db, name) + await db.refresh(row) + assert row.slots_cap == expected, "a refused write must store nothing" diff --git a/tests/test_gen_supervisord.py b/tests/test_gen_supervisord.py index f89812d..0eb1957 100644 --- a/tests/test_gen_supervisord.py +++ b/tests/test_gen_supervisord.py @@ -54,13 +54,13 @@ def test_every_lane_gets_a_program(): assert programs == expected -def test_the_ml_lane_runs_even_though_it_ships_disabled(): +def test_the_ml_lane_runs_even_though_it_ships_off(): """It holds a PROCESS and no model. `add_consumer` needs a running worker - to reach, so without this the UI switch would have nothing to switch — + to reach, so without this raising the cap would have nothing to reach — and nothing is downloaded by starting it, which is what lets rule 164 permit the fetch at all.""" assert _parse().has_section("program:ml") - assert LANES_BY_NAME["ml"].default_enabled is False + assert LANES_BY_NAME["ml"].default_slots_cap == 0 # --- the coupling this generator exists to guarantee ------------------------- @@ -152,12 +152,12 @@ def test_no_program_waits_longer_than_the_compose_stop_grace_period(): # --- what runs, and how much ------------------------------------------------ -def test_a_zero_slot_lane_still_gets_a_running_process(): - """ML ships at 0 slots and disabled — but `add_consumer` needs something to - reach. With no process there would be nothing for the UI switch to switch, - and enabling tagging could not work at all.""" +def test_a_lane_capped_at_zero_still_gets_a_running_process(): + """ML ships at cap 0 — but `add_consumer` needs something to reach. With + no process there would be nothing for the cap to switch back on, and + enabling tagging could not work at all.""" cp = _parse() - assert LANES_BY_NAME["ml"].default_slots == 0 + assert LANES_BY_NAME["ml"].default_slots_cap == 0 env = cp.get("program:ml", "environment") assert "CELERY_CONCURRENCY=1" in env @@ -275,29 +275,28 @@ def test_supervisorctl_can_reach_supervisord(): def test_each_program_starts_at_the_smallest_pool_the_control_path_allows(): """The two ends of the same floor, asserted together. - `gen_supervisord` starts every lane at `max(1, default_slots)` because - billiard will not run a pool of zero. `worker_control` has the same floor - for the opposite reason: it cannot SHRINK to zero either — + `gen_supervisord` starts every lane at one process because billiard will + not run a pool of zero. The sizing pass has the same floor for the + opposite reason: it cannot SHRINK to zero either — [ml] pidbox command error: ValueError("Can't shrink pool. All processes busy!") - Live, 2026-09-23. ML starts at one process and stores zero, so the + Live, 2026-09-23. ML started at one process and stored zero, so the reconcile tried 1 -> 0 on every tick, billiard refused, and `set_lane_slots_sync` — which returns True on SENDING the message — reported the lane changed forever (lesson #4183, on the default configuration of every install). - Two constants, in two files, that must agree or the container cannot - settle. Asserted through `effective_slots` rather than against a literal - 1, so raising the floor moves both ends at once. + One constant now, read from the same module by both, rather than two that + must agree. Asserted through it rather than against a literal 1, so + raising the floor moves both ends at once. """ - from backend.app.services.worker_control import effective_slots + from backend.app.services.worker_lanes import MIN_POOL_SLOTS cp = _parse() for lane in LANES: env = cp.get(f"program:{lane.name}", "environment") - want = effective_slots(lane.default_slots) - assert f"CELERY_CONCURRENCY={want}," in env, ( + assert f"CELERY_CONCURRENCY={MIN_POOL_SLOTS}," in env, ( f"{lane.name} starts at a size the control path cannot reach" ) diff --git a/tests/test_worker_control.py b/tests/test_worker_control.py index e2e31f2..c98133d 100644 --- a/tests/test_worker_control.py +++ b/tests/test_worker_control.py @@ -195,388 +195,6 @@ def test_an_unknown_queue_set_maps_to_no_lane(): assert wc._lane_for_queues(("something", "else")) is None -# --- the reconcile (step 3) -------------------------------------------------- - - -def test_a_settled_system_sends_no_control_messages(monkeypatch): - """THE property this sweep lives or dies on. It runs every 5 minutes - forever, so a converged tick must be silent — one broker round trip and - nothing else. An enforcer that re-issues a grow of zero churns forever and - buries a real correction in its own noise (lesson #4183). - """ - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 4}) - - result = wc.reconcile_lanes_sync({"worker": (4, True)}) - - # EVERY control family, not just the pool ones. The first version of this - # test asserted only grew/shrank and would have passed while the reconcile - # re-sent add_consumer for all four queues on every tick — harmless per - # call, unbounded churn in aggregate, and invisible. - assert control.grew == [] - assert control.shrank == [] - assert control.added == [] - assert control.cancelled == [] - assert result["changed"] == [] - - -def test_a_worker_back_at_its_env_concurrency_is_corrected(monkeypatch): - """The failure this exists for: a restart drops the pool to CELERY_ - CONCURRENCY, silently below what the operator set.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 2}) - - result = wc.reconcile_lanes_sync({"worker": (6, True)}) - - assert control.grew == [(4, ["host-a"])] - assert result["changed"] == ["worker"] - - -def test_an_absent_lane_is_skipped_not_corrected(monkeypatch): - """`present=False` is 'nothing answered', not 'zero slots'. Correcting it - would be a conclusion drawn from an unswept read (snippet #3969) — and - there is nothing to send the message to anyway.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={}, present=False) - - result = wc.reconcile_lanes_sync({"worker": (6, True)}) - - assert result["skipped"] == ["worker"] - assert result["changed"] == [] - assert control.grew == [] - assert control.shrank == [] - - -def test_one_lane_failing_does_not_stop_the_others(monkeypatch): - """A broker blip on one lane must not leave the rest un-reconciled for - another five minutes.""" - control = _stub_control(monkeypatch) - present = wc.LaneLiveState( - present=True, replicas=1, hostnames=["host-a"], pools={"host-a": 1}, - ) - absent = wc.LaneLiveState() - monkeypatch.setattr( - wc, "inspect_lanes_sync", - lambda: {"worker": absent, "scheduler": present, - "maintenance_long": absent, "ml": absent}, - ) - - result = wc.reconcile_lanes_sync({ - "worker": (4, True), "scheduler": (3, True), - }) - - assert result["skipped"] == ["worker"] - assert result["changed"] == ["scheduler"] - assert control.grew == [(2, ["host-a"])] - - -def test_a_lane_disabled_in_settings_but_still_consuming_is_stopped(monkeypatch): - """The case that made `consuming` necessary: a lane turned off while its - worker was down comes back consuming, and must be stopped when it - returns. - - Its live pool is ONE — zero slots means zero WORK, not an empty pool. - billiard will not run one, and the process is what the enable switch lands - on. - """ - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming={"ml"}) - - result = wc.reconcile_lanes_sync({"ml": (0, False)}) - - assert control.cancelled == [("ml", ["host-a"])] - assert result["changed"] == ["ml"] - - -def test_an_already_disabled_lane_is_not_cancelled_again(monkeypatch): - """The other half of the fixed point. `cancel_consumer` on a queue that is - not being consumed succeeds and does nothing, so an unconditional - reconcile would churn here forever with no symptom. - - The live pool is ONE, not zero. This fixture said zero until the floor - landed, and it was describing a container that cannot exist — billiard - will not run an empty pool, and the generator starts ml at one process for - exactly that reason. A fixture in an impossible state passes for the wrong - reason and then fails on the correct fix, which is what it did. - """ - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming=set()) - - result = wc.reconcile_lanes_sync({"ml": (0, False)}) - - assert control.cancelled == [] - assert control.added == [] - assert result["changed"] == [] - - -def test_a_lane_with_no_stored_row_is_left_alone(monkeypatch): - """A lane in LANES but not in `desired` is one whose row has not been - seeded. Inventing a target here would let the sweep enforce a number that - disagrees with the migration it is supposed to be upholding.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 2}) - - result = wc.reconcile_lanes_sync({}) - - assert result["changed"] == [] - assert control.grew == [] - - -def test_the_reconcile_task_is_registered_and_scheduled(): - """A task name only enters `celery.tasks` when its module is imported, and - a beat entry naming a task that is not registered fails at tick time - rather than at import — silently, every five minutes.""" - import backend.app.tasks.maintenance # noqa: F401 - from backend.app.celery_app import celery - - name = "backend.app.tasks.maintenance.reconcile_worker_lanes" - assert name in celery.tasks - scheduled = {e["task"] for e in celery.conf.beat_schedule.values()} - assert name in scheduled - - -# --- the autoscaler (step 7) -------------------------------------------------- -# -# Every case below fixes the LIVE pool and the stored floor to DIFFERENT -# numbers wherever it can. That is deliberate: the first version of this -# function read the stored value as the current one, and with the two equal -# — which is what a single tick of a settled system looks like — every test -# here still passed while the autoscaler could not grow past floor+1 or shrink -# at all. Equal fixtures cannot see that bug. - - -def _autoscale_live( - monkeypatch, *, pools, active, reserved, depth, present=True, -): - state = wc.LaneLiveState( - present=present, - replicas=len(pools), - hostnames=sorted(pools), - pools=dict(pools), - active=active, - reserved=reserved, - consuming=set(LANES_BY_NAME["worker"].queues), - ) - monkeypatch.setattr( - wc, "inspect_lanes_sync", - lambda: {name: (state if name == "worker" else wc.LaneLiveState()) - for name in LANES_BY_NAME}, - ) - monkeypatch.setattr( - wc, "_queue_depths_sync", - lambda: {q: depth for lane in LANES_BY_NAME.values() for q in lane.queues}, - ) - return state - - -def _worker(decisions): - return next(d for d in decisions if d.lane == "worker") - - -def test_a_prefetched_backlog_still_triggers_growth(monkeypatch): - """THE case an LLEN-only implementation misses, and the reason `reserved` - was plumbed through in step 2. Celery prefetches, so a lane can read queue - depth 0 while holding thirty tasks in worker memory — an autoscaler - watching LLEN alone sees an idle system and never grows.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=30, depth=0) - - d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})) - - assert d.action == "grew" - assert d.slots == 3 - assert control.grew == [(1, ["host-a"])] - - -def test_it_keeps_climbing_past_the_floor_on_later_ticks(monkeypatch): - """The regression test for reading the stored value as the current one. - - The row still says 2 — the autoscaler never writes it — but the live pool - is already 5 from earlier ticks. The target must be 6. Computing from the - stored 2 would propose 3, which resizes nothing (the replica is past it), - reports success, and pins the lane one slot above its floor forever while - claiming to grow on every tick.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 5}, active=5, reserved=40, depth=0) - - d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})) - - assert d.slots == 6 - assert control.grew == [(1, ["host-a"])] - - -def test_backlog_with_free_slots_does_nothing(monkeypatch): - """Depth alone is not a signal — celery is about to pick those up, and - growing would add children that idle.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 8}, active=1, reserved=0, depth=50) - - assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held" - assert control.grew == [] - - -def test_saturation_is_measured_against_every_replica(monkeypatch): - """`active` is summed across replicas and `pool` is one replica's size, so - comparing them calls two half-busy replicas of 4 saturated at 4 active. - Capacity is 8 here and 4 tasks are running: half the lane is idle.""" - control = _stub_control(monkeypatch) - _autoscale_live( - monkeypatch, pools={"host-a": 4, "host-b": 4}, - active=4, reserved=40, depth=0, - ) - - assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held" - assert control.grew == [] - - -def test_a_saturated_lane_with_no_backlog_does_nothing(monkeypatch): - """The long-running-task case. One slow task holding every slot with an - empty queue needs no extra slots — growing does not make it finish sooner, - which is why the operator's 'runs for x time' idea became a UI warning - rather than a trigger.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=0, depth=0) - - assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held" - assert control.grew == [] - - -def test_growth_stops_at_the_cap_and_says_so(monkeypatch): - """The autoscaler gets no authority the operator does not already have. - And it SAYS it is capped — that is the moment they would want to know they - set one.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 4}, active=4, reserved=99, depth=0) - - d = _worker(wc.autoscale_lanes_sync({"worker": (4, 2, True)})) - - assert d.action == "held" - assert "cap is 4" in d.reason - assert control.grew == [] - - -def test_a_cleared_backlog_returns_the_lane_to_the_operator_s_value(monkeypatch): - """Down toward `configured`, one step at a time.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 5}, active=0, reserved=0, depth=0) - - d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})) - - assert d.action == "shrank" - assert d.slots == 4 - assert control.shrank == [(1, ["host-a"])] - - -def test_it_never_shrinks_below_what_the_operator_set(monkeypatch): - """The stored value is the operator's and the autoscaler must not eat it — - there would be nothing left to restore to.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 2}, active=0, reserved=0, depth=0) - - assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held" - assert control.shrank == [] - - -def test_hysteresis_keeps_a_middling_backlog_from_flapping(monkeypatch): - """Between the two thresholds nothing happens. Equal thresholds would grow - and shrink the lane forever as one task arrives and leaves — lesson - #4183's churn arriving through a different door.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 5}, active=5, reserved=5, depth=0) - - assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held" - assert control.grew == [] - assert control.shrank == [] - - -def test_a_lane_that_did_not_opt_in_is_never_touched(monkeypatch): - """Off by default, per lane. This is the only sweep that decides rather - than obeys, so it acts only where someone said it may — and it does not - even report on a lane it was not given.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=99, depth=0) - - assert wc.autoscale_lanes_sync({"worker": (8, 2, False)}) == [] - assert control.grew == [] - - -def test_an_absent_lane_holds_rather_than_guessing(monkeypatch): - """`present=False` is 'nothing answered', not 'idle'. Deciding from an - unswept read is snippet #3969's shape.""" - control = _stub_control(monkeypatch) - _autoscale_live( - monkeypatch, pools={}, active=0, reserved=0, depth=0, present=False, - ) - - d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})) - - assert d.action == "held" - assert "not answering" in d.reason - assert control.grew == [] - - -def test_a_settled_lane_sends_no_control_messages(monkeypatch): - """The fixed point, asserted as a property rather than inferred from the - reported action: this runs every minute forever, so a settled system has - to be silent or a real correction drowns in the heartbeat.""" - control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 2}, active=1, reserved=0, depth=1) - - for _ in range(5): - assert _worker( - wc.autoscale_lanes_sync({"worker": (8, 2, True)}) - ).action == "held" - assert control.grew == [] - assert control.shrank == [] - - -# --- the two sweeps must not fight ------------------------------------------- - - -def test_the_reconcile_does_not_undo_what_the_autoscaler_added(monkeypatch): - """Otherwise the two would fight every five minutes: grow, revert, grow, - revert. For an autoscaling lane the stored slots are a FLOOR.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 6}) - - wc.reconcile_lanes_sync({"worker": (2, True)}, frozenset({"worker"})) - - assert control.shrank == [] - - -def test_the_reconcile_still_restores_an_autoscaling_lane_that_fell_below( - monkeypatch, -): - """A floor is still a floor. A restart drops the lane to its env value and - this must bring it back to what the operator set.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 1}) - - wc.reconcile_lanes_sync({"worker": (4, True)}, frozenset({"worker"})) - - assert control.grew == [(3, ["host-a"])] - - -def test_a_non_autoscaling_lane_is_still_driven_down_to_its_value(monkeypatch): - """The floor applies only to lanes that opted in — otherwise turning - autoscale off would leave the lane stuck at whatever it had grown to.""" - control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "worker", pools={"host-a": 6}) - - wc.reconcile_lanes_sync({"worker": (2, True)}, frozenset()) - - assert control.shrank == [(4, ["host-a"])] - - -def test_the_autoscale_task_is_registered_and_scheduled(): - import backend.app.tasks.maintenance # noqa: F401 - from backend.app.celery_app import celery - - name = "backend.app.tasks.maintenance.autoscale_worker_lanes" - assert name in celery.tasks - assert name in {e["task"] for e in celery.conf.beat_schedule.values()} - - # --- the inspect round trips ------------------------------------------------- @@ -680,49 +298,275 @@ def test_the_round_trip_bound_matches_the_reads_actually_made(): ) -# --- a pool cannot be emptied ------------------------------------------------ -def test_a_lane_at_zero_slots_is_never_shrunk_to_an_empty_pool(monkeypatch): - """The error in the operator's live log, 2026-09-23: +# --- the sizing pass --------------------------------------------------------- +# +# One sweep replaced two on 2026-09-23. The reconcile drove the pool to a +# stored `slots`; the autoscaler moved it away from that same number; and most +# of the autoscaler's design existed to stop the reconcile undoing its work. +# The stored number is gone, so there is nothing left to disagree about — and +# these tests no longer have to assert that two sweeps get along. +# +# The fixtures deliberately set the live pool and the cap to DIFFERENT numbers +# wherever the distinction matters. Lesson #4318: equal fixtures cannot tell +# "reads live" from "reads stored", and that is exactly how the old autoscaler +# shipped unable to do its job with every test green. - [scheduler] worker_control: ml reconciled 1 -> 0 slots - [ml] pidbox command error: - ValueError("Can't shrink pool. All processes busy!") - billiard refuses to remove the last worker, so a lane at zero stored slots - could never reach its target — and `set_lane_slots_sync` returns True on - SENDING the message, so the reconcile logged success and reported the lane - as `changed` on every tick, forever. Lesson #4183 in production, on the - default configuration of every install. - """ +def _sizing_live( + monkeypatch, *, pools, active, reserved, depth, consuming=None, present=True, +): + state = wc.LaneLiveState( + present=present, + replicas=len(pools), + hostnames=sorted(pools), + pools=dict(pools), + active=active, + reserved=reserved, + consuming=set(LANES_BY_NAME["worker"].queues if consuming is None else consuming), + ) + monkeypatch.setattr( + wc, "inspect_lanes_sync", + lambda: {name: (state if name == "worker" else wc.LaneLiveState()) + for name in LANES_BY_NAME}, + ) + monkeypatch.setattr( + wc, "_queue_depths_sync", + lambda: {q: depth for lane in LANES_BY_NAME.values() for q in lane.queues}, + ) + return state + + +def _worker(sized): + return next(d for d in sized if d.lane == "worker") + + +# --- what a lane has work for ------------------------------------------------ + + +def test_work_in_flight_and_waiting_each_justify_a_worker(): + # Two running plus six queued is eight tasks, so eight workers — if the + # cap allowed it. + assert wc.wanted_slots(cap=10, active=2, pending=6) == 8 + + +def test_it_never_exceeds_the_cap(): + assert wc.wanted_slots(cap=2, active=2, pending=4000) == 2 + + +def test_an_idle_lane_falls_to_one_and_not_to_zero(): + """billiard will not run an empty pool, and the parked process is what + `add_consumer` lands on when the cap goes back up.""" + assert wc.wanted_slots(cap=8, active=0, pending=0) == 1 + + +def test_a_lane_capped_at_zero_still_keeps_its_process(): + """`cap 0` is expressed by cancelling CONSUMERS, not by emptying the pool. + A lane with no process reads as absent, which is the same signal as a + crash — and the roster exists to keep those two apart.""" + assert wc.wanted_slots(cap=0, active=0, pending=0) == wc.MIN_POOL_SLOTS + + +def test_an_unknown_backlog_contributes_nothing_rather_than_zero(): + """The broker did not answer for these queues. Reading that as "empty" + would shrink a lane on the strength of a failed read (snippet #3969) — + and reading it as "huge" would grow one. It contributes nothing, and the + work actually in flight still counts.""" + assert wc.wanted_slots(cap=8, active=3, pending=None) == 3 + + +# --- growing ----------------------------------------------------------------- + + +def test_a_backlog_is_met_in_one_tick_not_one_slot_per_minute(monkeypatch): + """The operator's condition for always-on autoscaling: *"always on"* is + only pleasant if the ramp keeps up. Growing +1 per minute would take four + minutes to answer a burst, and the old autoscaler did exactly that.""" control = _stub_control(monkeypatch) - _stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming=set()) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=0, depth=4000) - wc.reconcile_lanes_sync({"ml": (0, False)}) + d = _worker(wc.size_lanes_sync({"worker": 4})) - assert control.shrank == [], "tried to empty a pool billiard will not empty" + assert (d.action, d.slots) == ("grew", 4) + assert control.grew == [(3, ["host-a"])] -def test_the_floor_applies_to_a_direct_resize_too(monkeypatch): - """Not only the reconcile — the UI dial and the autoscaler go through the - same function, so the clamp belongs there rather than at each caller.""" +def test_a_prefetched_backlog_counts(monkeypatch): + """THE case an LLEN-only implementation misses. Celery prefetches, so a + lane can read queue depth 0 while holding thirty tasks in worker memory — + a sizing pass watching LLEN alone sees an idle system and shrinks.""" control = _stub_control(monkeypatch) - state = _stub_live(monkeypatch, "worker", pools={"host-a": 3}) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=30, depth=0) - wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 0, live=state) - - assert control.shrank == [(2, ["host-a"])], "should stop at one, not zero" + assert _worker(wc.size_lanes_sync({"worker": 4})).action == "grew" + assert control.grew == [(3, ["host-a"])] -def test_the_autoscaler_will_not_shrink_into_an_empty_pool(monkeypatch): - """A lane whose operator value is 0 and whose live pool is the floor has - nowhere to shrink to — and saying `shrank` every minute is the same - non-convergence wearing a different hat.""" +def test_a_restarted_worker_is_pulled_back_up(monkeypatch): + """What the five-minute reconcile existed for, now done in one minute by + the pass that was already running. `pool_grow` is not durable: a worker + restarted by its supervisor comes back at its ENV concurrency, silently + below what the lane should be running.""" control = _stub_control(monkeypatch) - _autoscale_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=9, depth=0) - d = _worker(wc.autoscale_lanes_sync({"worker": (8, 0, True)})) + assert _worker(wc.size_lanes_sync({"worker": 4})).slots == 4 + assert control.grew == [(3, ["host-a"])] + + +# --- shrinking --------------------------------------------------------------- + + +def test_an_idle_lane_gives_a_worker_back_one_at_a_time(monkeypatch): + """Operator: *"so that idle instances quiet down when not running."* + + One per tick on the way down, against immediate growth. Being one worker + too large for a minute costs a sleeping process; being too small costs + work not happening. For ML it matters most — every new slot reloads a + multi-GB model, so a slow shrink is what stops a quiet patch from paying + that cost again a minute later.""" + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 4}, active=0, reserved=0, depth=0) + + d = _worker(wc.size_lanes_sync({"worker": 4})) + + assert (d.action, d.slots) == ("shrank", 3) + assert control.shrank == [(1, ["host-a"])] + + +def test_it_stops_shrinking_at_one(monkeypatch): + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0) + + assert _worker(wc.size_lanes_sync({"worker": 4})).action == "held" + assert control.shrank == [] + + +def test_lowering_the_cap_pulls_a_lane_down(monkeypatch): + """The cap is a ceiling on the live pool, not only on future growth.""" + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 4}, active=4, reserved=99, depth=0) + + assert _worker(wc.size_lanes_sync({"worker": 2})).slots == 3 + assert control.shrank == [(1, ["host-a"])] + + +# --- the fixed point --------------------------------------------------------- + + +def test_a_settled_lane_sends_no_control_messages(monkeypatch): + """THE property this pass lives or dies on. It runs every minute forever, + so a converged tick must be silent — one inspect, one LLEN sweep, nothing + else. An enforcer that re-issues a grow of zero churns forever and buries + a real correction in its own noise (lesson #4183).""" + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=0, depth=0) + + d = _worker(wc.size_lanes_sync({"worker": 4})) assert d.action == "held" + # EVERY control family, not just the pool ones: an unconditional + # add_consumer for four queues per tick is harmless per call and unbounded + # in aggregate, and invisible. + assert control.grew == [] assert control.shrank == [] + assert control.added == [] + assert control.cancelled == [] + + +def test_an_absent_lane_is_skipped_not_corrected(monkeypatch): + """`present=False` means nothing answered — a worker restarting, or an + unreachable broker. It is NOT zero workers, and there is nothing to send + a message to.""" + control = _stub_control(monkeypatch) + _sizing_live( + monkeypatch, pools={}, active=0, reserved=0, depth=0, present=False, + ) + + d = _worker(wc.size_lanes_sync({"worker": 4})) + + assert d.action == "skipped" + assert control.grew == [] and control.shrank == [] + + +def test_a_lane_with_no_row_is_left_alone(monkeypatch): + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=99) + + assert wc.size_lanes_sync({}) == [] + assert control.grew == [] + + +# --- the cap is also the switch ---------------------------------------------- + + +def test_a_cap_of_zero_stops_the_lane_consuming(monkeypatch): + control = _stub_control(monkeypatch) + _sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0) + + wc.size_lanes_sync({"worker": 0}) + + assert sorted(q for q, _ in control.cancelled) == sorted( + LANES_BY_NAME["worker"].queues + ) + + +def test_a_cap_above_zero_starts_it_consuming(monkeypatch): + control = _stub_control(monkeypatch) + _sizing_live( + monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0, + consuming=set(), + ) + + wc.size_lanes_sync({"worker": 2}) + + assert sorted(q for q, _ in control.added) == sorted( + LANES_BY_NAME["worker"].queues + ) + + +def test_an_already_stopped_lane_is_not_cancelled_again(monkeypatch): + """The other half of the fixed point. `cancel_consumer` on a queue that is + not being consumed succeeds and does nothing, so an unconditional pass + would churn here forever with no symptom.""" + control = _stub_control(monkeypatch) + _sizing_live( + monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0, + consuming=set(), + ) + + wc.size_lanes_sync({"worker": 0}) + + assert control.cancelled == [] + assert control.added == [] + + +def test_a_stopped_lane_still_keeps_exactly_one_process(monkeypatch): + """The live bug of 2026-09-23, from the other direction: the pass must not + try to empty a pool billiard will not empty, and must not report having + done so every tick.""" + control = _stub_control(monkeypatch) + _sizing_live( + monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0, + consuming=set(), + ) + + assert _worker(wc.size_lanes_sync({"worker": 0})).action == "held" + assert control.shrank == [] + + +def test_the_sizing_task_is_registered_and_scheduled(): + """A pass nothing schedules is a pass that never runs — and since this one + replaced two entries, a stale name in the beat schedule would leave the + lanes unmanaged with nothing red anywhere.""" + from backend.app.celery_app import celery + + name = "backend.app.tasks.maintenance.size_worker_lanes" + assert name in celery.tasks + entries = celery.conf.beat_schedule + assert any(e["task"] == name for e in entries.values()) + # The two it replaced are gone, not merely unreferenced. + tasks = {e["task"] for e in entries.values()} + assert "backend.app.tasks.maintenance.reconcile_worker_lanes" not in tasks + assert "backend.app.tasks.maintenance.autoscale_worker_lanes" not in tasks diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index 4627ad6..b3ba4fc 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -46,29 +46,50 @@ def test_every_routed_queue_has_a_lane_that_serves_it(): ) -def test_defaults_are_one_of_each_with_ml_off(): - """Operator, 2026-09-22: 'that starting value should be one of each.' +def test_the_shipped_caps_are_one_of_each_with_ml_at_zero(): + """Operator, 2026-09-22: *"that starting value should be one of each."* + Restated 2026-09-23 when the cap became the ONLY number: *"the cap + defaults should be 1 and 0 for the ml-worker."* - ML at zero AND disabled is milestone 422 step 6's requirement arriving - early: enabling the lane is what triggers the SigLIP download, and rule - 164 allows a runtime fetch only for a feature that is optional and clearly - off. A default of 1 here would make every fresh install reach HuggingFace. + ML at zero is milestone 422 step 6's requirement: a cap of zero means no + consumers, and raising it is what triggers the SigLIP download. Rule 164 + allows a runtime fetch only for a feature that is optional and clearly + off, so a default of 1 here would make every fresh install reach + HuggingFace. """ by_name = wl.LANES_BY_NAME for name in ("worker", "scheduler", "maintenance_long"): - assert by_name[name].default_slots == 1 - assert by_name[name].default_enabled is True - assert by_name["ml"].default_slots == 0 - assert by_name["ml"].default_enabled is False + assert by_name[name].default_slots_cap == 1 + assert by_name["ml"].default_slots_cap == 0 -def test_default_caps_leave_room_but_are_not_the_ceiling(): - """A cap that starts at the ceiling is a rubber stamp. Each lane's cap - must be at least its default slots (or the CHECK constraint rejects the - seeded row) and must leave somewhere to grow.""" +def test_a_lane_has_exactly_one_operator_setting(): + """The whole point of the 2026-09-23 reshape. `slots`, `enabled` and + `autoscale` are gone: how many workers run is a measurement the sizing + pass owns, and "off" is a cap of zero. + + Asserted against the dataclass's own fields rather than by name, so a + second knob added later fails here instead of quietly reappearing in the + UI beside the cap — which is exactly how three settings accumulated the + first time.""" + import dataclasses + + settable = { + f.name for f in dataclasses.fields(wl.Lane) if f.name.startswith("default_") + } + assert settable == {"default_slots_cap"}, ( + f"a lane has more than one operator default: {sorted(settable)}" + ) + + +def test_the_shipped_cap_is_never_the_ceiling_on_a_real_machine(): + """A cap that starts at the ceiling is a rubber stamp: there is nowhere to + raise it to, so the nudge that tells an operator to raise it would have + nothing to say. One-of-each leaves room on anything bigger than a + single-core box.""" for lane in wl.LANES: - assert lane.default_slots_cap >= lane.default_slots - assert lane.default_slots_cap >= 1 + assert lane.default_slots_cap >= 0 + assert lane.default_slots_cap <= 1 def test_only_ml_is_memory_bound(): -- 2.54.0 From 830394ed5ea6947686e16489855e308051c065c9 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 12:51:27 -0400 Subject: [PATCH 35/94] fix: the migration's DROP CONSTRAINT names doubled their own prefix (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7365, integration lane: (psycopg.errors.UndefinedObject) constraint "ck_worker_lane_ck_worker_lane_slots_within_cap" of relation "worker_lane" does not exist That is #3275 exactly, from the other direction. alembic 0088 had to RENAME four constraints CREATED with a doubled prefix; this one tried to DROP two with the same doubling. `op.drop_constraint` runs its name through Base.metadata's naming convention, which prepends `ck_worker_lane_` to a string that already carries it — `op.f()` is what marks a name as final, and 0103 used it on the way in. The model test also went red, correctly: `test_worker_lane_check_constraints` was parametrised over (slots, cap) pairs and asserted all three constraints, and two of them went with the `slots` column. It is one unparametrised test now, asserting the whole remaining set rather than a membership — a constraint left behind naming a dropped column is not a harmless leftover, it is a table the migration cannot have produced. Worth recording: **the gate worked.** Run 7365 skipped `sign-extension`, `build-web`, `smoke-web`, `promote` and `build-agent`, and `:dev` still names the previous digest. That is the red-direction verification #4339 owed, and it arrived by accident rather than by a forced failure — which is the better evidence. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- alembic/versions/0105_worker_lane_one_cap.py | 18 ++++++-- tests/test_worker_lanes.py | 44 ++++++++------------ 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/alembic/versions/0105_worker_lane_one_cap.py b/alembic/versions/0105_worker_lane_one_cap.py index 01b4813..d36458b 100644 --- a/alembic/versions/0105_worker_lane_one_cap.py +++ b/alembic/versions/0105_worker_lane_one_cap.py @@ -70,11 +70,21 @@ _CAPS = ( def upgrade() -> None: - # The constraint goes first: it names `slots`, so dropping the column out - # from under it fails on Postgres. - op.drop_constraint("ck_worker_lane_slots_within_cap", "worker_lane", type_="check") + # The constraints go first: they name `slots`, so dropping the column out + # from under them fails on Postgres. + # + # `op.f()` around each name, and it is load-bearing. Without it alembic + # runs the name through Base.metadata's naming convention, which prepends + # `ck_worker_lane_` to a string that already carries it — and the DROP + # goes looking for `ck_worker_lane_ck_worker_lane_slots_within_cap`, which + # no database has. That is #3275 exactly, from the other direction: + # alembic 0088 had to RENAME four constraints created with the same + # doubling. Caught here by the integration lane, run 7365. op.drop_constraint( - "ck_worker_lane_slots_non_negative", "worker_lane", type_="check", + op.f("ck_worker_lane_slots_within_cap"), "worker_lane", type_="check", + ) + op.drop_constraint( + op.f("ck_worker_lane_slots_non_negative"), "worker_lane", type_="check", ) op.drop_column("worker_lane", "slots") op.drop_column("worker_lane", "enabled") diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index b3ba4fc..14be621 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -246,22 +246,16 @@ def test_an_unrecognised_queue_set_still_gets_a_true_label(): # --- the model's invariant --------------------------------------------------- -@pytest.mark.parametrize( - "slots,cap,ok", - [ - (0, 0, True), - (1, 1, True), - (1, 4, True), - (5, 4, False), # slots above its own cap - (-1, 1, False), # negative slots - (1, -1, False), # negative cap - ], -) -def test_worker_lane_check_constraints(slots, cap, ok): - """The constraints live in the database, not only in the service, because - a row violating `slots <= slots_cap` is not a rejected request — it is a - lane the reconcile (step 3) will drive UP to a number the operator - capped.""" +def test_the_row_carries_exactly_one_constraint_now(): + """`slots >= 0` and `slots <= slots_cap` went with the `slots` column on + 2026-09-23 — there is one number left, and the only thing that can be + wrong with it is being negative. + + Asserted as the WHOLE set rather than as a membership check: a constraint + left behind naming a dropped column is not a harmless leftover, it is a + table the migration cannot have produced, and the model would then + describe a schema no database has. + """ from backend.app.models import WorkerLane constraints = { @@ -269,17 +263,13 @@ def test_worker_lane_check_constraints(slots, cap, ok): if hasattr(c, "sqltext") } # The names carry the convention's `ck_worker_lane_` prefix ALREADY — the - # model declares them bare and Base.metadata's naming_convention applies it. - # Asserting the prefixed form is what pins the thing that actually went - # wrong once: alembic 0088 had to rename four constraints that shipped as - # `ck_x_ck_x_name`, because the migration pre-prefixed a name the - # convention then prefixed again (#3275). A bare-name assertion here would - # pass just as happily against a doubled one. - assert constraints == { - "ck_worker_lane_slots_non_negative": "slots >= 0", - "ck_worker_lane_cap_non_negative": "slots_cap >= 0", - "ck_worker_lane_slots_within_cap": "slots <= slots_cap", - } + # model declares them bare and Base.metadata's naming_convention applies + # it. Asserting the prefixed form is what pins the thing that actually + # went wrong once: alembic 0088 had to rename four constraints that + # shipped as `ck_x_ck_x_name`, because the migration pre-prefixed a name + # the convention then prefixed again (#3275). A bare-name assertion here + # would pass just as happily against a doubled one. + assert constraints == {"ck_worker_lane_cap_non_negative": "slots_cap >= 0"} for name in constraints: assert not name.startswith("ck_worker_lane_ck_"), f"doubled prefix: {name}" -- 2.54.0 From 364e0506325d183f67769f1e2387022789e8ed29 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 12:53:27 -0400 Subject: [PATCH 36/94] fix: a spliced-out test left its tail behind (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7366, in both the lint and unit lanes — one bug wearing two hats: NameError: name 'slots' is not defined tests/test_worker_lanes.py:278 Rewriting `test_worker_lane_check_constraints` — parametrised over (slots, cap, ok) — I cut the parametrize decorator and the assertion but left the two lines after it, which still referenced all three names. `ast.parse` is happy with that; only running it is not. Both failures were the same fragment, which is the tell: the lint lane's F821 and the unit lane's NameError are the same fact reported twice. Checked the rest of today's splices for the same seam rather than fixing only the one CI named — a module-level undefined-name sweep over all four test files I rewrote today comes back clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- tests/test_worker_lanes.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index 14be621..160a552 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -273,11 +273,6 @@ def test_the_row_carries_exactly_one_constraint_now(): for name in constraints: assert not name.startswith("ck_worker_lane_ck_"), f"doubled prefix: {name}" - # Evaluate the same predicates the database will, so the parametrize table - # documents what is accepted rather than restating the SQL. - satisfied = slots >= 0 and cap >= 0 and slots <= cap - assert satisfied is ok - # --- what an optional lane tells the operator before it is enabled ----------- -- 2.54.0 From 61641fbba7120ae55999adecf9f81f632a33cf88 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 12:57:33 -0400 Subject: [PATCH 37/94] fix: a test still described the control the cap replaced (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7367, integration lane: FAILED test_a_cap_is_stored_even_when_it_cannot_be_pushed assert True is False The code was right. Raising a cap is PERMISSION, not a request — it deliberately does not grow the pool, because that would put workers on a lane with nothing to do, and the sizing pass spends the permission on its next tick if there is work. So nothing is pushed and `applied` is vacuously true. The test was carried over from when the number meant "run this many", where every write pushed. It asserted the old control's behaviour against the new one — lesson #4338's shape again: an assertion encoding the thing that changed, failing on the change rather than on a defect. Split into the two cases that actually exist now: - raising a cap stores it and pushes nothing, reporting applied; - turning a lane OFF does push, because consumers follow the cap immediately in both directions — off must take effect when it is asked for — so with nothing answering it reports `applied: false` with a reason, and the value is still stored for the sizing pass to carry. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- tests/test_api_workers.py | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py index d39f4fe..5fc61e2 100644 --- a/tests/test_api_workers.py +++ b/tests/test_api_workers.py @@ -89,22 +89,47 @@ async def test_the_other_lanes_ship_at_one(client, no_live_workers): @pytest.mark.asyncio -async def test_a_cap_is_stored_even_when_it_cannot_be_pushed( +async def test_raising_a_cap_stores_it_and_pushes_nothing( client, db, no_live_workers, ): - """Nothing is answering, so the live push fails. That is NOT a failed - setting: the value is saved and the sizing pass carries it within a - minute (lesson #4202 — a live change that does not survive, with nothing - saying so).""" + """A cap is PERMISSION, not a request. Raising it must not grow the pool + here — that would put workers on a lane with nothing to do — so there is + nothing to push and `applied` is vacuously true. + + This asserted `applied is False` until run 7367, carried over from when + the number meant "run this many". The code was right and the test was + describing the control it replaced. + """ resp = await client.post("/api/system/workers/worker", json={"slots_cap": 3}) assert resp.status_code == 200 body = await resp.get_json() assert body["slots_cap"] == 3 + assert body["applied"] is True + assert (await _lane_row(db, "worker")).slots_cap == 3 + + +@pytest.mark.asyncio +async def test_turning_a_lane_off_is_stored_even_when_it_cannot_be_pushed( + client, db, no_live_workers, +): + """The direction that DOES push. Consumers follow the cap immediately in + both directions — off must take effect when it is asked for — so with + nothing answering, the push fails. + + That is NOT a failed setting: the value is saved and the sizing pass + carries it within a minute (lesson #4202 — a live change that does not + survive, with nothing saying so). The UI says "saved, not yet live" + rather than "that didn't work", which is the distinction `applied` + exists to carry. + """ + resp = await client.post("/api/system/workers/worker", json={"slots_cap": 0}) + + assert resp.status_code == 200 + body = await resp.get_json() assert body["applied"] is False assert "not running" in body["apply_error"] - - assert (await _lane_row(db, "worker")).slots_cap == 3 + assert (await _lane_row(db, "worker")).slots_cap == 0 # --- the cap is the switch --------------------------------------------------- -- 2.54.0 From 895589a578033c4855ed80104fc8d3b241037f63 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 13:04:26 -0400 Subject: [PATCH 38/94] fix: first boot raced itself for the credential key (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7368's smoke, on an image whose five verification lanes were all green: WARNING Generating NEW Fernet credential key at /images/secrets/credential_key.b64 ... ValueError: Fernet key must be 32 url-safe base64-encoded bytes. Nothing to do with this batch's changes — it is a first-boot race that has been there since the key file existed, and it is a RACE rather than a certainty: the same code booted cleanly on the three runs before it. hypercorn starts several worker processes and each one builds the app, so on a first boot they all reach the bootstrap together. `write_bytes` creates the file at size zero and fills it a moment later, which gives the second process an `exists()` of True and a `read_bytes()` of `b""`. A first boot that fails one time in five is worse than one that fails every time, because it looks like the deployment rather than the code — and this is the very first thing a new install does. The key is now written to a temp file and `os.link`ed into place. `os.link` is the atomic part: it either creates the name or raises FileExistsError, and it cannot expose a half-written file. Deliberately NOT `os.replace`, which would succeed — two processes that both generated a key would each think they had won, and the loser's key would overwrite the one the winner had already handed to Fernet. The losing branch reads the winner's key back rather than returning its own, which is what keeps every worker on ONE key. Tested for AGREEMENT, not for the absence of a crash: eight threads through a barrier, and all eight must end up holding the same key. A race that left each worker with its own would pass a "did it raise" check and produce a system where a credential written by one worker cannot be read by the next. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/credential_crypto.py | 44 +++++++++++++++-- tests/test_credential_crypto.py | 59 +++++++++++++++++++++++ 2 files changed, 100 insertions(+), 3 deletions(-) diff --git a/backend/app/services/credential_crypto.py b/backend/app/services/credential_crypto.py index 273e574..65cb930 100644 --- a/backend/app/services/credential_crypto.py +++ b/backend/app/services/credential_crypto.py @@ -80,10 +80,48 @@ class CredentialCrypto: parent = self._key_path.parent parent.mkdir(parents=True, exist_ok=True) os.chmod(parent, 0o700) + + # Written to a temp file and LINKED into place, not written directly. + # + # hypercorn starts several worker processes and each one builds the + # app, so on a first boot they all reach this at once. A plain + # `write_bytes` creates the file at size zero and fills it a moment + # later, which gives a second process an `exists()` of True and a + # `read_bytes()` of b"" — and the app dies with + # + # ValueError: Fernet key must be 32 url-safe base64-encoded bytes. + # + # Seen on run 7368's smoke, and it is a race rather than a certainty: + # the same image had booted cleanly on the three runs before it. A + # first boot that fails one time in five is worse than one that fails + # every time, because it looks like the deployment rather than the code. + # + # `os.link` is the atomic part: it either creates the name or raises + # FileExistsError, and it cannot expose a half-written file. NOT + # `os.replace`, which would succeed — so two processes that both + # generated a key would each think they had won, and the loser's key + # would overwrite the one the winner had already handed to Fernet. key = Fernet.generate_key() - self._key_path.write_bytes(key) - os.chmod(self._key_path, 0o600) - return key + tmp = parent / f".{self._key_path.name}.{os.getpid()}.tmp" + try: + tmp.write_bytes(key) + os.chmod(tmp, 0o600) + try: + os.link(tmp, self._key_path) + except FileExistsError: + # Another process created it between our `exists()` check and + # here. Theirs is as good as ours, and using it is what keeps + # every worker on ONE key. + log.info( + "another process created %s first; using that key", + self._key_path, + ) + finally: + tmp.unlink(missing_ok=True) + # Read back rather than returning `key`: on the losing branch the file + # holds somebody else's, and returning ours would leave this worker + # encrypting with a key no other worker can read. + return self._key_path.read_bytes() def encrypt(self, plaintext: str) -> bytes: return self._fernet.encrypt(plaintext.encode("utf-8")) diff --git a/tests/test_credential_crypto.py b/tests/test_credential_crypto.py index 2ca31b0..689e067 100644 --- a/tests/test_credential_crypto.py +++ b/tests/test_credential_crypto.py @@ -135,3 +135,62 @@ def test_compose_passes_the_bootstrap_variable_through(): compose = (_REPO_ROOT / "docker-compose.yml").read_text() assert f"{_BOOTSTRAP_ENV_VAR}: ${{{_BOOTSTRAP_ENV_VAR}" in compose + + +# --- the first-boot race ----------------------------------------------------- + + +def test_every_process_bootstrapping_at_once_ends_up_with_the_same_key(tmp_path): + """Run 7368's smoke, and the reason the key write is a link rather than a + write. + + hypercorn starts several worker processes and each one builds the app, so + on a first boot they all reach the bootstrap together. `write_bytes` + creates the file at size zero and fills it a moment later, which gave a + second process an `exists()` of True and a `read_bytes()` of b"": + + ValueError: Fernet key must be 32 url-safe base64-encoded bytes. + + Threads rather than processes because the failure is about the ORDER of + two filesystem operations, which threads reproduce and which a subprocess + fixture would make this suite pay for on every run. + + What is asserted is AGREEMENT, not merely that nobody crashed: a race that + left each worker holding its own key would pass a "did it raise" check and + produce a system where a credential written by one worker cannot be read + by the next. + """ + import threading + + path = tmp_path / "secrets" / "credential_key.b64" + keys: list[bytes] = [] + errors: list[Exception] = [] + start = threading.Barrier(8) + + def bootstrap(): + try: + start.wait(timeout=5) + CredentialCrypto(path, bootstrap_ok=True) + keys.append(path.read_bytes()) + except Exception as exc: # noqa: BLE001 — recorded, asserted below + errors.append(exc) + + threads = [threading.Thread(target=bootstrap) for _ in range(8)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=10) + + assert errors == [] + assert len(keys) == 8 + assert len(set(keys)) == 1, "the workers disagree about the credential key" + + +def test_the_bootstrap_leaves_no_temp_file_behind(tmp_path): + """The temp file is an implementation detail of the atomic write and must + not survive it — a stray `.credential_key.b64..tmp` in the secrets + directory is a copy of the key with nothing guarding it.""" + path = tmp_path / "secrets" / "credential_key.b64" + CredentialCrypto(path, bootstrap_ok=True) + + assert sorted(p.name for p in path.parent.iterdir()) == ["credential_key.b64"] -- 2.54.0 From c09ebd663986f92e693c7e902bac950dcf1f881c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 13:11:29 -0400 Subject: [PATCH 39/94] fix: the race fix's temp name assumed one bootstrap per process (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7370: six of the eight threads in the new test died with FileNotFoundError(2, 'No such file or directory') The first cut named the temp file `.credential_key.b64..tmp`, which assumes one bootstrap per PROCESS. Eight threads share one pid, so all eight raced the same filename and six found it already unlinked by another. The assumption held for hypercorn's workers, which are processes, and would have held in production. It was still an assumption the code did not need to make, and the test found it in one run — which is the test doing its job rather than the test being wrong, so `tempfile.mkstemp` replaces the guess. Verified against the SHIPPED file, loaded by path so the package `__init__` (which wants quart) stays out of it: 25 consecutive attempts, eight threads through a barrier each time, one key, no leftover temp files. My first check of this reimplemented the logic in a scratch script, which measures a copy — rule 10 names that exact failure, and it is not evidence about the code that ships. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/credential_crypto.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/backend/app/services/credential_crypto.py b/backend/app/services/credential_crypto.py index 65cb930..5308018 100644 --- a/backend/app/services/credential_crypto.py +++ b/backend/app/services/credential_crypto.py @@ -24,6 +24,7 @@ rows undecryptable (recovery = delete the rows and re-upload). import logging import os +import tempfile from pathlib import Path from cryptography.fernet import Fernet, InvalidToken @@ -101,10 +102,21 @@ class CredentialCrypto: # `os.replace`, which would succeed — so two processes that both # generated a key would each think they had won, and the loser's key # would overwrite the one the winner had already handed to Fernet. + # `mkstemp`, not a pid-derived name. The first cut spelled the temp + # file `.credential_key.b64..tmp`, which assumes one bootstrap per + # process — and the test that exercises this with eight THREADS shares + # one pid, so all eight raced the same filename and six died with + # FileNotFoundError when another had already unlinked it. The + # assumption held for hypercorn's workers and would have held in + # production; it was still an assumption the code did not need to make. key = Fernet.generate_key() - tmp = parent / f".{self._key_path.name}.{os.getpid()}.tmp" + fd, tmp_name = tempfile.mkstemp( + dir=parent, prefix=f".{self._key_path.name}.", suffix=".tmp", + ) + tmp = Path(tmp_name) try: - tmp.write_bytes(key) + with os.fdopen(fd, "wb") as fh: + fh.write(key) os.chmod(tmp, 0o600) try: os.link(tmp, self._key_path) -- 2.54.0 From 5b6f2ba52629fa29a23772d0c1b463a90a0ef442 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 14:52:36 -0400 Subject: [PATCH 40/94] fix: a Postgres connection was held across every celery round trip (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/api/workers.py | 22 ++++- backend/app/services/worker_control.py | 131 +++++++++++++++++-------- tests/test_api_workers.py | 78 +++++++++++++++ 3 files changed, 189 insertions(+), 42 deletions(-) diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py index 10c03ee..78f7f1c 100644 --- a/backend/app/api/workers.py +++ b/backend/app/api/workers.py @@ -23,7 +23,13 @@ from datetime import UTC, datetime from quart import Blueprint, jsonify, request from ..extensions import get_session -from ..services.worker_control import LaneUpdateRefused, lane_view, set_lane +from ..services.worker_control import ( + LaneUpdateRefused, + lane_settings, + lane_view, + push_lane_cap, + store_lane_cap, +) from ..services.worker_lanes import LANES_BY_NAME from ._responses import error_response as _bad @@ -41,8 +47,14 @@ async def list_lanes(): reply would show them the value from before their own change and read as the control having failed. """ + # The session closes BEFORE the broker work. Holding a Postgres connection + # across a celery inspect is what made this page block the whole site — + # see `worker_control.LaneSettings`. This endpoint polls every 15s and the + # inspect budget is 11s, so each poll was pinning a connection for most of + # the interval. async with get_session() as session: - lanes = await lane_view(session) + settings = await lane_settings(session) + lanes = await lane_view(settings) return jsonify({ "lanes": lanes, "fetched_at": datetime.now(UTC).isoformat(), @@ -85,9 +97,13 @@ async def update_lane(name: str): if not isinstance(value, int) or isinstance(value, bool): return _bad("invalid_body", detail="slots_cap must be an integer") + # Store, close the session, THEN push. Same reason as the GET above, and + # more sharply here: a cap change could do three broker round trips, all + # of them previously with a connection held. async with get_session() as session: try: - result = await set_lane(session, lane, slots_cap=value) + was_cap = await store_lane_cap(session, lane, value) except LaneUpdateRefused as exc: return _bad("refused", detail=str(exc)) + result = await push_lane_cap(lane, value, was_cap=was_cap) return jsonify(result) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 641c545..88d599f 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -334,14 +334,53 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]: return rows -async def lane_view(session: AsyncSession) -> list[dict]: +@dataclass +class LaneSettings: + """What the DATABASE knows about the lanes — read and finished with before + anything touches the broker. + + This exists because holding a Postgres connection across a celery round + trip is what made the System tab block the whole site (operator, + 2026-09-23: *"something about changing the cap number is blocking to the + website"*). + + `lane_view` used to take the session and keep it open through an inspect + whose budget is eleven seconds — and that page polls every fifteen. With a + lane not answering, every inspect ran to nearly its full budget, so each + poll pinned a connection for ten seconds. SQLAlchemy's default pool is + five connections plus ten overflow; a couple of browser tabs, the health + endpoint doing the same thing, and a cap change adding two more inspects + exhausts that, and every OTHER request then waits on a connection. + + Nothing was slow in itself. The slowness was a scarce resource held across + it, which is why it surfaced as the whole site stalling rather than as one + slow page. + """ + + caps: dict[str, int] + oldest_by_queue: dict[str, datetime] + + +async def lane_settings(session: AsyncSession) -> LaneSettings: + """Every DB read the lane view needs, in one short-lived session.""" + rows = await _rows_by_name(session) + return LaneSettings( + caps={name: row.slots_cap for name, row in rows.items()}, + oldest_by_queue=await _oldest_running_by_queue(session), + ) + + +async def lane_view(settings: LaneSettings) -> list[dict]: """Every lane: what is configured, what is live, what it may grow to. - One call rather than making the UI join three sources. `pending` is the - honest backlog — Redis depth PLUS reserved — because celery prefetches and - LLEN alone reads 0 while a worker holds tasks in memory. + Takes the settings rather than a session ON PURPOSE — see `LaneSettings`. + Everything below this line is broker work, and no database connection is + held while it happens. + + `pending` is the honest backlog — Redis depth PLUS reserved — because + celery prefetches and LLEN alone reads 0 while a worker holds tasks in + memory. """ - rows = await _rows_by_name(session) # A deadline, because this is a request path and `to_thread` on its own is # an await with no bound (rule 156). `inspect_lanes_sync` never raises and # every inner call has its own timeout, so the only way past the budget is @@ -359,12 +398,12 @@ async def lane_view(session: AsyncSession) -> list[dict]: ) live = {lane.name: LaneLiveState() for lane in LANES} depths = await asyncio.to_thread(_queue_depths_sync) - oldest = await _oldest_running_by_queue(session) + oldest = settings.oldest_by_queue now = datetime.now(UTC) out = [] for lane in LANES: - row = rows[lane.name] + cap = settings.caps[lane.name] state = live[lane.name] # None for a queue the broker did not answer for, which must not be # silently summed as zero — an unknown depth is not an empty one. @@ -376,11 +415,11 @@ async def lane_view(session: AsyncSession) -> list[dict]: "name": lane.name, "display_name": lane.display_name, "queues": list(lane.queues), - "slots_cap": row.slots_cap, + "slots_cap": cap, "ceiling": derived_ceiling(lane), # DERIVED, never stored. A cap of zero means no consumers, so # "off" and "may use no workers" cannot disagree. - "enabled": row.slots_cap > 0, + "enabled": cap > 0, "memory_bound": lane.memory_bound, "optional": lane.optional, # What raising this lane's cap will download, so the UI can say @@ -485,30 +524,14 @@ class LaneUpdateRefused(ValueError): the UI shows — a greyed control with no explanation reads as a bug.""" -async def set_lane(session: AsyncSession, lane: Lane, *, slots_cap: int) -> dict: - """Store the operator's cap for `lane`, then make the live lane obey it. +async def store_lane_cap( + session: AsyncSession, lane: Lane, slots_cap: int, +) -> int: + """Validate and store the cap. Returns the PREVIOUS cap. DB only. - ONE value, since 2026-09-23. It used to take `slots`, `slots_cap`, - `enabled` and `autoscale`, which was four ways of saying two things — and - two of them were the caller's job to keep in agreement with each other. - - ## What happens live, and what does not - - A cap CHANGE is pushed immediately in one direction only: lowering it - shrinks the pool now, because a cap the operator just lowered should not - be exceeded for up to a minute. Raising it does NOT grow the pool here — - a cap is permission, not a request, and growing on permission would put - slots on a lane with nothing to do. The sizing pass adds them on its next - tick if there is work, which is the whole point of it being always on. - - Consumers follow the cap in both directions and immediately: zero means - off, and off must take effect when it is asked for. - - A failed PUSH is not a failed setting. The value is saved either way and - the sizing pass carries it within a minute; the result says `applied: - false` with a reason so the UI can say "saved, not yet live" rather than - "that didn't work" (lesson #4202 — a live change that does not survive, - with nothing saying so). + Split from the live push for the reason `LaneSettings` gives at length: a + Postgres connection must not be held across a celery round trip. Everything + here is fast and finished with before `push_lane_cap` starts. """ rows = await _rows_by_name(session) row = rows[lane.name] @@ -522,11 +545,40 @@ async def set_lane(session: AsyncSession, lane: Lane, *, slots_cap: int) -> dict f"({ceiling} for {lane.display_name})" ) - was_on = row.slots_cap > 0 + was_cap = row.slots_cap row.slots_cap = slots_cap await session.commit() + # The previous value, because the push needs the DIRECTION: lowering a cap + # has to reach the running lane now, and raising one has nothing to say. + return was_cap - now_on = slots_cap > 0 + +async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict: + """Make the running lane obey a cap that is already stored. NO database. + + ## What is pushed, and what is not + + Consumers follow the cap immediately in BOTH directions: zero means off, + and off must take effect when it is asked for rather than up to a minute + later. + + The pool is only ever pushed DOWNWARD. Raising a cap is permission, not a + request — growing on permission would put workers on a lane with nothing + to do — so the sizing pass spends it on its next tick if there is work. + That also makes the common case (raising a cap) free: no broker round trip + AT ALL, which is the difference between a control that answers instantly + and one that takes ten seconds. Keyed on the previous cap rather than on + "is it on" — the first cut only knew on/off, so it inspected on every + raise to find out whether the pool needed lowering, and the control it was + meant to make instant still waited out an inspect. + + A failed push is not a failed setting. The value is already stored and the + sizing pass carries it within a minute; `applied: false` with a reason + lets the UI say "saved, not yet live" rather than "that didn't work" + (lesson #4202 — a live change that does not survive, with nothing saying + so). + """ + was_on, now_on = was_cap > 0, slots_cap > 0 applied, error = True, None if now_on != was_on: applied, error = await asyncio.to_thread(set_lane_enabled_sync, lane, now_on) @@ -536,9 +588,10 @@ async def set_lane(session: AsyncSession, lane: Lane, *, slots_cap: int) -> dict applied, error = await asyncio.to_thread( set_lane_slots_sync, lane, MIN_POOL_SLOTS, ) - elif applied and now_on: - # Only DOWNWARD. See the docstring: raising a cap is permission, and - # the sizing pass decides whether there is work to spend it on. + elif applied and now_on and slots_cap < was_cap: + # LOWERED on a running lane. Only this direction needs a message, and + # only when the pool is actually above the new cap — so it reads the + # live pool rather than resizing blind. A raise never reaches here. live = await asyncio.to_thread(inspect_lanes_sync) current = live[lane.name].pool if current is not None and current > slots_cap: @@ -561,8 +614,8 @@ async def set_lane(session: AsyncSession, lane: Lane, *, slots_cap: int) -> dict return { "name": lane.name, - "slots_cap": row.slots_cap, - "ceiling": ceiling, + "slots_cap": slots_cap, + "ceiling": derived_ceiling(lane), "enabled": now_on, "applied": applied, "apply_error": error, diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py index 5fc61e2..90bcace 100644 --- a/tests/test_api_workers.py +++ b/tests/test_api_workers.py @@ -255,3 +255,81 @@ async def _refreshed(db, name: str, expected: int) -> None: row = await _lane_row(db, name) await db.refresh(row) assert row.slots_cap == expected, "a refused write must store nothing" + + +# --- no database connection is held across a broker round trip --------------- + + +@pytest.mark.asyncio +async def test_the_lane_read_holds_no_session_while_it_inspects(monkeypatch): + """Operator, 2026-09-23: *"something about changing the cap number is + blocking to the website... it shouldn't be"*. + + Nothing here was slow in itself. A Postgres connection was held across a + celery inspect whose budget is eleven seconds, on a page that polls every + fifteen — so with a lane not answering, each poll pinned a connection for + most of the interval. SQLAlchemy's default pool is five plus ten overflow; + two browser tabs, the health endpoint doing the same, and a cap change + adding more inspects exhausts it, and every OTHER request then waits on a + connection. It surfaced as the whole site stalling rather than as one slow + page, which is why it took a screenshot to find. + + Asserted STRUCTURALLY rather than by timing: `lane_view` must not accept a + session at all. A timing test would be flaky, and a mock-call-order test + would pass against a version that took the session and merely used it + early — the property that matters is that it CANNOT. + """ + import inspect as _inspect + + from backend.app.services.worker_control import lane_settings, lane_view + + assert "session" not in _inspect.signature(lane_view).parameters, ( + "lane_view takes a session again; the broker work must run with none held" + ) + # And the DB half still exists, so the split did not simply lose the reads. + assert "session" in _inspect.signature(lane_settings).parameters + + +@pytest.mark.asyncio +async def test_the_cap_write_holds_no_session_while_it_pushes(): + """The same property on the write path, where it was worse: a cap change + could make three broker round trips, each with a connection held.""" + import inspect as _inspect + + from backend.app.services.worker_control import push_lane_cap, store_lane_cap + + assert "session" in _inspect.signature(store_lane_cap).parameters + assert "session" not in _inspect.signature(push_lane_cap).parameters, ( + "push_lane_cap takes a session again; the push must run with none held" + ) + + +@pytest.mark.asyncio +async def test_raising_a_cap_costs_no_broker_round_trip_at_all( + client, db, no_live_workers, monkeypatch, +): + """The common case must be instant. Raising a cap is permission, not a + request — the sizing pass spends it — so there is nothing to tell the + broker, and the operator's `+` should answer immediately rather than + waiting out an inspect.""" + from backend.app.services import worker_control as wc + + calls = [] + monkeypatch.setattr( + wc, "inspect_lanes_sync", lambda: calls.append("inspect") or {}, + ) + monkeypatch.setattr( + wc, "set_lane_slots_sync", + lambda *a, **k: calls.append("resize") or (True, None), + ) + monkeypatch.setattr( + wc, "set_lane_enabled_sync", + lambda *a, **k: calls.append("consumers") or (True, None), + ) + + await client.post("/api/system/workers/worker", json={"slots_cap": 1}) + calls.clear() + resp = await client.post("/api/system/workers/worker", json={"slots_cap": 6}) + + assert resp.status_code == 200 + assert calls == [], f"raising a cap talked to the broker: {calls}" -- 2.54.0 From 693759f2bbef2aaff4cb8a3b22f558ffd8136992 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 14:52:36 -0400 Subject: [PATCH 41/94] fix: an idle GPU agent could not check in, so the roster called it stopped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator, 2026-09-23: *"I'm running the gpu agent on my device and it currently reads as 'offline' but it's running and has checked in recently."* It had checked in — twelve minutes ago. Two cadences that never agreed: idle lease poll ceiling 900s agent/fc_agent/worker.py (sleep mode) heartbeat while idle never gated on holding leases roster "stopped" after 300s api/system_health.py The roster records an agent check-in on `lease` and `heartbeat`. The heartbeat loop was gated on `if ids:`, so an agent holding no leases sent nothing at all — leaving the lease poll as the only check-in, and sleep mode backs that off exponentially to a 900s ceiling. 900 against 300: an IDLE agent was structurally guaranteed to read as stopped. Nothing was broken; nothing was misconfigured; the two halves simply disagreed. Not a recent regression. Sleep mode landed 2026-07-02; the roster adopted the lease as its check-in on 2026-09-02 — *"A lease IS the check-in … Recorded on the call that was already happening"* — without noticing that the call it was piggybacking on had been deliberately slowed ten weeks earlier. The heartbeat now sends whether or not it holds leases. An empty one extends nothing (`id.in_([])` matches no rows) and costs one small POST every 45s — against the 6/min lease poll sleep mode exists to avoid, that is not a cadence worth protecting, and it is what makes "is the agent alive" answerable at all. Still gated on `self._running`: a worker that has been stopped is not checking in for work, and reporting it as present would be a different lie. Two things I could NOT determine from the code, both needing the live table: whether a stale `agent:agent` row exists from an older build that omitted `agent_id` (the server defaults it), and whether changing `AGENT_ID` has ever stranded an abandoned row — nothing prunes `service_seen`, so either would sit there reading "stopped" forever. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- agent/fc_agent/worker.py | 39 ++++++++++++++++++++---- tests/test_api_gpu.py | 64 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+), 5 deletions(-) diff --git a/agent/fc_agent/worker.py b/agent/fc_agent/worker.py index e0c4cc5..8683ad2 100644 --- a/agent/fc_agent/worker.py +++ b/agent/fc_agent/worker.py @@ -342,15 +342,44 @@ class Worker: # --- background loops --------------------------------------------------- def _heartbeat_loop(self) -> None: - """Keep every held lease alive so buffered jobs waiting on the GPU aren't - reclaimed by curator's 180s TTL. Errors are swallowed by client.heartbeat; - a reclaimed lease just re-leases elsewhere — never fatal.""" + """Keep every held lease alive, and say we are here even when holding none. + + Leases: buffered jobs waiting on the GPU would otherwise be reclaimed by + curator's 180s TTL. Errors are swallowed by client.heartbeat; a reclaimed + lease just re-leases elsewhere — never fatal. + + ## Why this sends with an EMPTY list rather than skipping + + Curator's roster records a check-in on this call (and on `lease`), and + calls an agent stopped after 300s of silence. This loop used to be + gated on `if ids:` — so an agent holding no leases sent nothing at all, + and the only check-in left was the lease poll, which sleep mode backs + off exponentially to a 900s ceiling (see IDLE_POLL_MAX_SECONDS). + + 900 against 300: an IDLE agent was structurally guaranteed to read as + stopped. Operator, 2026-09-23: *"I'm running the gpu agent on my device + and it currently reads as 'offline' but it's running and has checked in + recently."* It had — twelve minutes ago, partway up the backoff ladder. + + The two halves were written ten weeks apart and never reconciled: sleep + mode landed 2026-07-02, and the roster adopted the lease as its + check-in on 2026-09-02 without noticing the call it was piggybacking on + had been deliberately slowed. + + An empty heartbeat extends nothing (`id.in_([])` matches no rows) and + costs one small POST every 45s — against the 6/min lease poll sleep + mode exists to avoid, that is not a cadence worth protecting, and it is + what makes "is the agent alive" answerable at all. + + Still gated on `self._running`: a worker that has been stopped is not + checking in for work, and reporting it as present would be a different + lie. + """ while True: if self._running: with self._held_lock: ids = list(self._held) - if ids: - self.client.heartbeat(ids) + self.client.heartbeat(ids) time.sleep(HEARTBEAT_INTERVAL) def _queue_poll_loop(self): diff --git a/tests/test_api_gpu.py b/tests/test_api_gpu.py index 25f0a7b..db51353 100644 --- a/tests/test_api_gpu.py +++ b/tests/test_api_gpu.py @@ -314,3 +314,67 @@ async def test_cpu_embed_never_blocks_gpu_crop_backfills(db): select(GpuJob.task).where(GpuJob.image_record_id == img.id) )).scalars().all()) assert tasks == {"ccip", "siglip"} + + +# --- an idle agent still checks in ------------------------------------------- + + +@pytest.mark.asyncio +async def test_a_heartbeat_with_no_jobs_still_records_the_check_in(client, db): + """Operator, 2026-09-23: *"I'm running the gpu agent on my device and it + currently reads as 'offline' but it's running and has checked in + recently."* + + It had — twelve minutes ago. The roster takes its agent check-in from the + `lease` and `heartbeat` calls, and calls an agent stopped after 300s of + silence. The agent's heartbeat loop was gated on holding leases, so an + IDLE agent sent none; the only check-in left was the lease poll, which + sleep mode backs off to a 900s ceiling. 900 against 300 — an idle agent + was structurally guaranteed to read as stopped. + + So the empty heartbeat has to be a real check-in on the server side, not + merely tolerated. Asserted on `last_seen_at` moving, because "it returned + 200" would pass against an endpoint that recorded nothing. + """ + from backend.app.models import ServiceSeen + + token = (await (await client.post("/api/gpu/token/rotate")).get_json())["token"] + hdr = {"Authorization": f"Bearer {token}"} + + resp = await client.post( + "/api/gpu/jobs/heartbeat", + json={"agent_id": "desktop-agent", "job_ids": []}, headers=hdr, + ) + + assert resp.status_code == 200 + assert (await resp.get_json())["extended"] == 0, "it extends no lease" + + row = (await db.execute( + select(ServiceSeen).where(ServiceSeen.key == "agent:desktop-agent") + )).scalar_one() + assert row.kind == "agent" + assert row.display_name == "GPU agent (desktop-agent)" + assert row.last_seen_at is not None + + +def test_the_agent_heartbeats_whether_or_not_it_holds_a_lease(): + """The other half, in the agent itself — the half that was actually wrong. + + Read from the source rather than by running the loop: it is a `while True` + with a sleep, so exercising it means threads and timing, and the property + is simply that the call is not behind a `if ids:`. + """ + from pathlib import Path + + src = ( + Path(__file__).resolve().parents[1] / "agent" / "fc_agent" / "worker.py" + ).read_text() + loop = src[src.index("def _heartbeat_loop"):] + loop = loop[:loop.index("\n def ")] + + assert "self.client.heartbeat(ids)" in loop + assert "if ids:" not in loop, ( + "the heartbeat is gated on holding leases again; an idle agent then " + "reads as stopped after 300s while sleep mode backs its lease poll " + "off to 900s" + ) -- 2.54.0 From a4c66601db02dd9ff214b6c579f99f3216760ce3 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 14:59:06 -0400 Subject: [PATCH 42/94] fix: the heartbeat guard grepped its own explanation (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7375: the test that asserts the agent's heartbeat is not gated on holding leases failed — on the docstring of the fix, which quotes the construct the fix removed, because that is what a docstring explaining a fix does. assert "if ids:" not in loop A source-TEXT assertion cannot tell code from prose about code. Parsed now: the function's AST body, unparsed with the docstring node dropped, so the guard reads only what executes. Worth stating as the general shape, since this repo writes long explanatory comments on purpose: any check that greps source for the absence of a pattern is in tension with documenting why that pattern is gone. Either it excludes the prose, or the next person to explain the fix breaks the guard that protects it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- tests/test_api_gpu.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/tests/test_api_gpu.py b/tests/test_api_gpu.py index db51353..e36c490 100644 --- a/tests/test_api_gpu.py +++ b/tests/test_api_gpu.py @@ -362,19 +362,33 @@ def test_the_agent_heartbeats_whether_or_not_it_holds_a_lease(): Read from the source rather than by running the loop: it is a `while True` with a sleep, so exercising it means threads and timing, and the property - is simply that the call is not behind a `if ids:`. + is simply that the call is not behind a guard on `ids`. + + Parsed rather than grepped, and that is not fussiness — the first cut + searched the raw function text and failed on its own explanation. The + docstring of the fix quotes the construct the fix removed, because that is + what a docstring explaining a fix DOES. A source-text assertion cannot + tell the code from the prose about the code; the AST can, so the body is + unparsed with its docstring dropped. """ + import ast from pathlib import Path src = ( Path(__file__).resolve().parents[1] / "agent" / "fc_agent" / "worker.py" ).read_text() - loop = src[src.index("def _heartbeat_loop"):] - loop = loop[:loop.index("\n def ")] + fn = next( + n for n in ast.walk(ast.parse(src)) + if isinstance(n, ast.FunctionDef) and n.name == "_heartbeat_loop" + ) + code = "\n".join( + ast.unparse(n) for n in fn.body + if not (isinstance(n, ast.Expr) and isinstance(n.value, ast.Constant)) + ) - assert "self.client.heartbeat(ids)" in loop - assert "if ids:" not in loop, ( + assert "self.client.heartbeat(ids)" in code + assert "if ids" not in code, ( "the heartbeat is gated on holding leases again; an idle agent then " "reads as stopped after 300s while sleep mode backs its lease poll " - "off to 900s" + f"off to 900s\n\n{code}" ) -- 2.54.0 From 48108a356923ba4795b37c35cd4906a46abb8745 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 15:21:21 -0400 Subject: [PATCH 43/94] fix: a lane that is OFF was not attributable to itself (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator, 2026-09-23: *"clean up the stale service_seen rows"*. **They were not stale.** They were phantoms, written on purpose, and they will come back on every install that turns a lane off — so the rows are the smaller half of this. A celery worker was attributed to its lane by the queues it was CONSUMING. A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` with an empty list, matches no lane, and is dropped. Three consequences, all on the operator's screen at once: 1. The lanes table reported the lane **not answering** — the signal for a crashed worker, not for one the operator turned off. 2. The roster grew a phantom row named **`Worker ()`** — the empty queue set rendered as a display name — shown "running" beside the real lane's row going stale, because nothing updated it any more. 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every lane present. ML ships at cap 0, so a fresh install was permanently unhealthy and Swarm restarts an unhealthy task forever. That third one is the severe one, and its docstring asserted the opposite of what the code did — *"a disabled lane still runs its process with its consumers cancelled, so it answers inspect and is healthy"*. It answers. It was not attributed. A comment can be right about the intent and wrong about the program, and this one had been wrong since the consolidated container shipped. `worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels with the process rather than with what it happens to be doing. `gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the information was there and nothing read it. Falls back to the queue set for a deployment that names no node, and `docker-compose.yml` now sets one per service so the multi-service stack gets it too. The roster keys on the LANE's queue set when the node resolves, which is the same string the row already had while it was consuming — so an existing row keeps updating rather than a second one appearing. Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately NOT a retention sweep: the roster never forgets on purpose, so a quiet row is what it is FOR, and only a row that cannot correspond to anything real is safe to remove. An `agent:agent` row, if one exists, is left alone — nothing here can tell an abandoned agent id from a second agent that is genuinely down, and hiding a dead GPU agent is the one thing the roster must not do. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../0106_prune_phantom_roster_rows.py | 70 ++++++++++++ backend/app/scripts/healthcheck.py | 16 ++- backend/app/services/service_roster.py | 17 ++- backend/app/services/worker_control.py | 9 +- backend/app/services/worker_lanes.py | 38 +++++++ docker-compose.yml | 15 +++ tests/test_service_roster.py | 77 +++++++++++++ tests/test_worker_control.py | 101 ++++++++++++++++++ 8 files changed, 336 insertions(+), 7 deletions(-) create mode 100644 alembic/versions/0106_prune_phantom_roster_rows.py diff --git a/alembic/versions/0106_prune_phantom_roster_rows.py b/alembic/versions/0106_prune_phantom_roster_rows.py new file mode 100644 index 0000000..14d24d2 --- /dev/null +++ b/alembic/versions/0106_prune_phantom_roster_rows.py @@ -0,0 +1,70 @@ +"""service_seen — delete the roster rows the fixed code can no longer write. + +Operator, 2026-09-23: *"clean up the stale service_seen rows"*. They were not +stale. They were PHANTOMS, written on purpose by code that identified a celery +worker from the queues it was consuming. + +A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` +with an empty list. The roster grouped on that empty set, wrote it under the +key `celery:` and rendered `role_display_name(())` as the display name — a row +called **`Worker ()`**, reported as running, beside the real lane's row going +stale because nothing updated it any more. + +`worker_lanes.lane_for_node` fixes the cause: a worker is attributed by its +NODE NAME, which survives having no consumers. Nothing will write `celery:` +again. + +## Why a migration and not a retention sweep + +Lesson #4202: a guard that refuses to produce a bad value does not undo the +bad value already stored. The row is the thing that has to change. + +And it must be deleted rather than aged out, because the roster deliberately +NEVER forgets — *"anything that has run at least once stays listed, that is +what lets a stopped one be noticed rather than simply vanishing"*. A row that +merely goes quiet is exactly what the roster is for. Only a row that cannot +correspond to anything real is safe to remove, and `celery:` is precisely +that: the empty queue set, which no correctly-attributed worker can produce. + +## What is deliberately NOT deleted + +**Celery rows with a real but unmatched queue set.** A deployment slicing +`CELERY_QUEUES` differently is supported and its rows are true. It is not this +migration's business to decide that somebody else's worker is obsolete. + +**Agent rows, including a possible `agent:agent` from a build that omitted +`agent_id`.** Nothing here can tell an abandoned agent id from a second agent +that is currently down, and deleting a real one would hide a genuinely dead +GPU agent — the one thing the roster exists to show. If such a row is present +it needs a person to look at it, not a migration guessing. + +Revision ID: 0106 +Revises: 0105 +Create Date: 2026-09-23 + +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "0106" +down_revision: Union[str, None] = "0105" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Exactly the one key the empty queue set produced. Matched literally + # rather than by a LIKE or a prefix: `celery:` with nothing after it is + # the phantom, and `celery:ml` is a real lane. + op.execute( + sa.text("DELETE FROM service_seen WHERE key = :key").bindparams(key="celery:") + ) + + +def downgrade() -> None: + # Nothing. The row carried no information — an empty queue set and a + # timestamp — and the roster re-learns anything real on its next refresh. + # Re-creating it would put a phantom back. + pass diff --git a/backend/app/scripts/healthcheck.py b/backend/app/scripts/healthcheck.py index 78b07e2..fcbaf52 100644 --- a/backend/app/scripts/healthcheck.py +++ b/backend/app/scripts/healthcheck.py @@ -121,11 +121,19 @@ def _this_node_ok() -> tuple[bool, str]: def _lanes_ok() -> tuple[bool, str]: """Every lane in the table is answering. - Deliberately ignores whether a lane is ENABLED: a disabled lane still runs - its process with its consumers cancelled, so it answers `inspect` and is + Deliberately ignores whether a lane is ON: a lane at cap 0 still runs its + process with its consumers cancelled, so it answers `inspect` and is healthy. Health is "is the process alive"; whether it should be consuming - is a settings question the reconcile owns, and conflating them would make - turning a lane off in the UI mark the container unhealthy. + is a settings question the sizing pass owns, and conflating them would + make turning a lane off mark the container unhealthy. + + That was not merely a risk — it was happening. Until 2026-09-23 a worker + was attributed to its lane by the queues it was CONSUMING, and a lane with + its consumers cancelled reports none, so it read as absent and this check + failed. ML ships at cap 0, so a fresh install was permanently unhealthy + and Swarm restarts an unhealthy task forever. The docstring above said the + right thing while the code did the opposite; `worker_lanes.lane_for_node` + is what makes it true. """ from ..services.worker_control import inspect_lanes_sync from ..services.worker_lanes import LANES diff --git a/backend/app/services/service_roster.py b/backend/app/services/service_roster.py index 83de7db..7a77363 100644 --- a/backend/app/services/service_roster.py +++ b/backend/app/services/service_roster.py @@ -36,7 +36,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession from ..models import ServiceSeen -from .worker_lanes import LANES +from .worker_lanes import LANES, lane_for_node log = logging.getLogger(__name__) @@ -118,7 +118,20 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]: grouped: dict[tuple[str, ...], dict] = {} for hostname, queues in active_queues.items(): - key = tuple(sorted({q["name"] for q in queues})) + # Keyed on the LANE's queue set when the node name identifies one, so + # a lane keeps the same roster row whether or not it is consuming. + # + # Grouping on the ACTIVE queues alone meant a lane at cap 0 — which + # cancels its consumers — reported an empty set, landed under the key + # `celery:`, and rendered as a phantom row named `Worker ()` while its + # real row went stale beside it. Both symptoms on the operator's + # screen, 2026-09-23, from this one line. + # + # Deriving the key from `lane.queue_key` rather than inventing a new + # one keeps every existing row: it is the same string the lane already + # had while it was running. + lane = lane_for_node(hostname) + key = lane.queue_key if lane else tuple(sorted({q["name"] for q in queues})) entry = grouped.setdefault(key, {"hostnames": [], "active": 0}) entry["hostnames"].append(hostname) entry["active"] += len(active_tasks.get(hostname, [])) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 88d599f..c4c1d7d 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -62,6 +62,7 @@ from .worker_lanes import ( MIN_POOL_SLOTS, Lane, derived_ceiling, + lane_for_node, ) log = logging.getLogger(__name__) @@ -189,7 +190,13 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: return out for hostname, queues in active_queues.items(): - lane = _lane_for_queues(tuple(q["name"] for q in queues)) + # The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its + # consumers cancelled and answers with an empty queue list, which + # matches no lane, so attributing by queues alone dropped every lane + # the operator had turned off and reported it as "not answering". + lane = lane_for_node(hostname) or _lane_for_queues( + tuple(q["name"] for q in queues) + ) if lane is None: # A deployment slicing CELERY_QUEUES differently. Reported by the # roster under its raw queue list; it simply has no lane row to diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index 7694696..8a23513 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -332,6 +332,44 @@ def container_cpu_count() -> int | None: return os.cpu_count() +def lane_for_node(hostname: str) -> Lane | None: + """`ml@7f3c9a1b` -> the ml lane. None for a node this build did not name. + + ## Why the node name, and not the queues it is consuming + + Because a lane that is OFF is consuming nothing, and "nothing" identifies + no lane at all. + + Both the roster and `inspect_lanes_sync` used to map a worker to its lane + through `active_queues()`. That is exact while the lane is running and + useless the moment it is not: a lane at cap 0 has its consumers cancelled, + so it answers the broadcast with an EMPTY queue list, matches no lane, and + is dropped. Three things followed, and the operator saw all three at once + on 2026-09-23: + + 1. The lanes table showed the lane as **not answering** — which is the + signal for a crashed worker, not for one the operator turned off. + 2. The roster grew a phantom row called **`Worker ()`**, the empty queue + set rendered as a display name, "running" beside the real lane's row + going stale. + 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires + every lane in the table to be present, and its docstring asserted the + opposite of what the code did — *"a disabled lane still runs its + process with its consumers cancelled, so it answers inspect and is + healthy"*. It answers; it is not attributed. ML ships at cap 0, so a + fresh install would have been permanently unhealthy, and Swarm + restarts an unhealthy task forever. + + The node name survives all of that: `gen_supervisord` sets + `CELERY_NODENAME={lane.name}` per program and the entrypoint passes it to + `celery -n`, so the identity travels with the PROCESS rather than with + what it happens to be doing. Falls back to the queue set for a deployment + that sets no node name — the multi-service compose stack, where every node + is `celery@`. + """ + return LANES_BY_NAME.get(hostname.split("@", 1)[0]) + + def derived_ceiling(lane: Lane) -> int: """The most slots `lane` may be given on this container. diff --git a/docker-compose.yml b/docker-compose.yml index 2248734..671dca0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -180,6 +180,10 @@ services: environment: <<: *app_env CELERY_QUEUES: default,import,thumbnail,download + # Names the celery node for the roster. A lane whose consumers are + # cancelled reports no queues, so the NODE is the only thing left that + # identifies it — see services/worker_lanes.lane_for_node. + CELERY_NODENAME: worker CELERY_CONCURRENCY: "2" # /downloads dropped — nothing in the app references it (operator-flagged # 2026-06-07: it wasn't mapped in prod and everything worked). @@ -200,6 +204,10 @@ services: environment: <<: *app_env CELERY_QUEUES: maintenance,scan + # Names the celery node for the roster. A lane whose consumers are + # cancelled reports no queues, so the NODE is the only thing left that + # identifies it — see services/worker_lanes.lane_for_node. + CELERY_NODENAME: scheduler volumes: - ./images:/images - ./import:/import @@ -223,6 +231,10 @@ services: environment: <<: *app_env CELERY_QUEUES: maintenance_long + # Names the celery node for the roster. A lane whose consumers are + # cancelled reports no queues, so the NODE is the only thing left that + # identifies it — see services/worker_lanes.lane_for_node. + CELERY_NODENAME: maintenance_long CELERY_CONCURRENCY: "1" # Only /images: backups write to /images/_backups, audits read /images, and # the admin tasks (re-extract/cascade-delete/normalize) operate on /images. @@ -241,6 +253,9 @@ services: deploy: *deploy_policy environment: <<: *app_env + # See the worker service — the node name is what identifies a lane + # whose consumers are cancelled. + CELERY_NODENAME: ml volumes: - ./images:/images:ro - ./models:/models diff --git a/tests/test_service_roster.py b/tests/test_service_roster.py index 372bee4..c837297 100644 --- a/tests/test_service_roster.py +++ b/tests/test_service_roster.py @@ -52,3 +52,80 @@ def test_the_round_trip_count_matches_the_calls_actually_made(): f"_inspect_celery_sync makes {calls} inspect calls but " f"INSPECT_ROUND_TRIPS says {sr.INSPECT_ROUND_TRIPS}" ) + + +# --- a lane that is off keeps its own row ------------------------------------ + + +def _stub_inspect(monkeypatch, active_queues): + class _Insp: + def __init__(self, **_): + pass + + def active_queues(self): + return active_queues + + def active(self): + return {} + + class _Control: + inspect = _Insp + + import sys + import types + mod = types.ModuleType("backend.app.celery_app") + + class _C: + pass + + c = _C() + c.control = _Control() + mod.celery = c + monkeypatch.setitem(sys.modules, "backend.app.celery_app", mod) + + +def test_a_lane_with_no_consumers_keeps_the_row_it_had_while_running(monkeypatch): + """The phantom, and the reason the operator had two wrong rows at once. + + A lane at cap 0 cancels its consumers, so it answers with an EMPTY queue + list. Grouped on that, it landed under the key `celery:` and rendered as a + row called `Worker ()` — reported running — while the real lane's row went + stale beside it because nothing updated it any more. + + Keyed on the LANE's queue set now, which is the same string the row + already had while the lane was consuming. So turning a lane off updates + its row instead of minting a second one. + """ + from backend.app.services.worker_lanes import LANES_BY_NAME + + _stub_inspect(monkeypatch, {"ml@abc123": []}) + + grouped = sr._inspect_celery_sync() + + assert list(grouped) == [LANES_BY_NAME["ml"].queue_key] + assert () not in grouped, "the empty queue set is the phantom `Worker ()`" + + +def test_the_row_is_the_same_one_whether_the_lane_is_consuming_or_not(monkeypatch): + """Stated as an identity rather than as two separate assertions: if these + keys ever differ, turning a lane off silently starts a second roster row + and the first goes stale — which is exactly what happened.""" + ml_queues = [{"name": "ml"}] + + _stub_inspect(monkeypatch, {"ml@abc123": ml_queues}) + on = set(sr._inspect_celery_sync()) + _stub_inspect(monkeypatch, {"ml@abc123": []}) + off = set(sr._inspect_celery_sync()) + + assert on == off + + +def test_a_worker_this_build_did_not_name_is_still_grouped_by_its_queues( + monkeypatch, +): + """The fallback, and the case the roster exists to report honestly: a + deployment slicing CELERY_QUEUES differently gets its raw queue list + rather than a name this code invented for it.""" + _stub_inspect(monkeypatch, {"celery@xyz": [{"name": "odd"}]}) + + assert list(sr._inspect_celery_sync()) == [("odd",)] diff --git a/tests/test_worker_control.py b/tests/test_worker_control.py index c98133d..8febdb9 100644 --- a/tests/test_worker_control.py +++ b/tests/test_worker_control.py @@ -570,3 +570,104 @@ def test_the_sizing_task_is_registered_and_scheduled(): tasks = {e["task"] for e in entries.values()} assert "backend.app.tasks.maintenance.reconcile_worker_lanes" not in tasks assert "backend.app.tasks.maintenance.autoscale_worker_lanes" not in tasks + + +# --- a lane that is OFF is still its own lane --------------------------------- +# +# The operator's screen, 2026-09-23: "ML tagging has not checked in for 135 +# min" beside a phantom row called "Worker ()" reported as running. One cause: +# a worker was attributed to its lane by the queues it was CONSUMING, and a +# lane at cap 0 has its consumers cancelled, so it consumes none and matched +# nothing. + + +def _stub_active_queues(monkeypatch, by_host): + """Only `active_queues` — the read that decides which lane a node IS.""" + class _Insp: + def __init__(self, **_): + pass + + def active_queues(self): + return by_host + + def stats(self): + return {h: {"pool": {"max-concurrency": 1}} for h in by_host} + + def active(self): + return {} + + def reserved(self): + return {} + + control = _stub_control(monkeypatch) + control.inspect = _Insp + import sys + import types + mod = types.ModuleType("backend.app.celery_app") + + class _C: + pass + + c = _C() + c.control = control + mod.celery = c + monkeypatch.setitem(sys.modules, "backend.app.celery_app", mod) + + +def test_a_lane_with_its_consumers_cancelled_is_still_found(monkeypatch): + """THE bug. `ml@host` consuming nothing must read as the ml lane, present, + rather than as no lane at all. + + Attributed by the NODE NAME, which survives having no consumers — + `gen_supervisord` sets `CELERY_NODENAME={lane.name}` per program for + exactly this.""" + _stub_active_queues(monkeypatch, {"ml@abc123": []}) + + live = wc.inspect_lanes_sync() + + assert live["ml"].present is True + assert live["ml"].consuming == set(), "it is present AND consuming nothing" + + +def test_that_is_what_keeps_the_container_healthy(monkeypatch): + """Why it mattered more than a cosmetic row. + + `healthcheck._lanes_ok` requires every lane to be present, and ML ships at + cap 0 — so a fresh install reported a lane not answering, the container + went permanently unhealthy, and Swarm restarts an unhealthy task forever. + """ + _stub_active_queues(monkeypatch, { + "worker@a": [{"name": q} for q in LANES_BY_NAME["worker"].queues], + "scheduler@a": [{"name": q} for q in LANES_BY_NAME["scheduler"].queues], + "maintenance_long@a": [ + {"name": q} for q in LANES_BY_NAME["maintenance_long"].queues + ], + "ml@a": [], # off, as it ships + }) + + live = wc.inspect_lanes_sync() + + missing = sorted(name for name, s in live.items() if not s.present) + assert missing == [], f"the healthcheck would fail the container: {missing}" + + +def test_a_node_this_build_did_not_name_still_matches_on_its_queues(monkeypatch): + """The fallback. The multi-service compose stack ran every worker as + `celery@`, and a deployment that sets no node name must keep + working — the node name is an improvement, not a requirement.""" + _stub_active_queues(monkeypatch, { + "celery@xyz": [{"name": q} for q in LANES_BY_NAME["worker"].queues], + }) + + assert wc.inspect_lanes_sync()["worker"].present is True + + +def test_an_unknown_worker_is_still_ignored_rather_than_guessed_at(monkeypatch): + """A deployment slicing CELERY_QUEUES differently has no lane row to + control. It belongs to the roster, not here — and must not be attributed + to whichever lane happens to be first.""" + _stub_active_queues(monkeypatch, {"celery@xyz": [{"name": "something-else"}]}) + + live = wc.inspect_lanes_sync() + + assert all(not s.present for s in live.values()) -- 2.54.0 From 1353d346b3a969999a534c4658b436775fc435d3 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 15:41:32 -0400 Subject: [PATCH 44/94] fix: the cap dial waited out a broker round trip it did not need (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator: "when the number is changed the change should be queued so that it isn't blocking of the webui or the system itself. we shouldn't have to wait for the validation live." Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection being HELD across them, which is what had been stalling the whole site, and left the press itself as slow as it was. 1. The store refetched after every write. GET /api/system/workers runs a celery inspect on an eleven-second budget, so the stepper stayed disabled through a round trip the press did not need. It now patches the row from the reply — cap, ceiling, enabled, the three fields that reply actually decides — and lets the 15s poll bring the live columns, which are measurements it must not invent. 2. The endpoint pushed to the broker before answering. Turning a lane off is four cancel_consumer messages; lowering a cap reads the live pool first. Now it stores the cap, answers `queued`, and hands the push to a Quart background task. Raising a cap was already free and stays free. Nothing is lost by not waiting: the stored cap is what the system obeys and the sizing pass re-reads it every minute. That sweep was already the backstop for a push that failed, which under `no_live_workers` is every push in the suite. Also closes a hole the move exposed: the model fetch was gated on the consumer change having landed, so raising ML off zero while the lane was restarting stored the cap, let the sizing pass start the consumers a minute later, and left the lane running with no model — nothing else ever asks for one. It now fires on the transition and waits in the ml queue. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/api/workers.py | 78 ++++++++-- backend/app/services/worker_control.py | 31 +++- .../components/settings/SystemHealthTab.vue | 15 +- frontend/src/stores/systemActivity.js | 30 +++- frontend/test/workerLanes.spec.js | 82 +++++++--- tests/test_api_workers.py | 142 +++++++++++++++--- 6 files changed, 306 insertions(+), 72 deletions(-) diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py index 78f7f1c..c0f6638 100644 --- a/backend/app/api/workers.py +++ b/backend/app/api/workers.py @@ -19,8 +19,9 @@ second into the first would make a read-only module a write one. from __future__ import annotations from datetime import UTC, datetime +from functools import partial -from quart import Blueprint, jsonify, request +from quart import Blueprint, current_app, jsonify, request from ..extensions import get_session from ..services.worker_control import ( @@ -30,7 +31,7 @@ from ..services.worker_control import ( push_lane_cap, store_lane_cap, ) -from ..services.worker_lanes import LANES_BY_NAME +from ..services.worker_lanes import LANES_BY_NAME, Lane, derived_ceiling from ._responses import error_response as _bad workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers") @@ -63,22 +64,38 @@ async def list_lanes(): @workers_bp.route("/", methods=["POST"]) async def update_lane(name: str): - """Set a lane's cap. Stores it, then makes the live lane obey it. + """Set a lane's cap. Stores it, answers, and makes the lane follow after. ONE field, since 2026-09-23. It used to take `slots`, `slots_cap`, `enabled` and `autoscale`; how many workers are running is now a measurement the sizing pass owns, and `enabled` is `cap > 0`. - Two failure kinds, deliberately different statuses: + ## The reply does not wait for the lane + + Operator, 2026-09-23: *"when the number is changed the change should be + queued so that it isn't blocking of the webui or the system itself. we + shouldn't have to wait for the validation live."* + + So the request does exactly one thing that can be slow — a row update — + and hands the broker work to a background task. Turning a lane off is + four `cancel_consumer` messages and a resize; lowering a cap is an + `inspect` on an eleven-second budget. Both used to happen between the + click and the response, with the stepper disabled the whole time. + + Nothing is lost by not waiting: the cap in the database is what the + system obeys, the sizing pass re-reads it every minute, and the table + polls, so the live columns catch up on their own. If the web process dies + before the background task runs, that sweep is the backstop — which is + the same guarantee the awaited version had, since a push could fail + there too. + + Refusals still happen inline, because they are decided from the value and + the machine's ceiling alone and never touch the broker: * **400** — the value is not allowed (negative, or above what this container can hold). Nothing was stored. The body carries `detail`, which is the sentence the UI shows; a refused control with no reason reads as a bug. - * **200 with `applied: false`** — the value WAS stored but could not be - pushed, because the lane is not currently answering. Not an error: the - sizing pass carries it within a minute, and the UI should say "saved, - not yet live" rather than "that didn't work". """ lane = LANES_BY_NAME.get(name) if lane is None: @@ -97,13 +114,48 @@ async def update_lane(name: str): if not isinstance(value, int) or isinstance(value, bool): return _bad("invalid_body", detail="slots_cap must be an integer") - # Store, close the session, THEN push. Same reason as the GET above, and - # more sharply here: a cap change could do three broker round trips, all - # of them previously with a connection held. + # Store, close the session, THEN hand off. The session must not be held + # across broker work — that is what made this page block the whole site + # (see `worker_control.LaneSettings`) — and now the request does not wait + # for that work either. async with get_session() as session: try: was_cap = await store_lane_cap(session, lane, value) except LaneUpdateRefused as exc: return _bad("refused", detail=str(exc)) - result = await push_lane_cap(lane, value, was_cap=was_cap) - return jsonify(result) + + _schedule_push(lane, value, was_cap) + + return jsonify({ + "name": lane.name, + "slots_cap": value, + "ceiling": derived_ceiling(lane), + "enabled": value > 0, + # The value is stored; the live lane is being told separately. The UI + # patches its row from this and lets the next poll bring the live + # columns, rather than refetching and paying for an inspect it just + # avoided. + "queued": True, + # Raising the cap off zero is what downloads the model (step 6), and + # the background task does it. Reported here so the UI can say a + # download has started rather than leaving the operator to wonder why + # a lane they just turned on is busy. + "fetching_models": value > 0 and was_cap == 0 and bool(lane.models), + }) + + +def _schedule_push(lane: Lane, slots_cap: int, was_cap: int) -> None: + """Run the live push after the response has gone out. + + A seam, not an abstraction: it is one call, and it exists so the tests can + hold the push still — a background task that outlived a test's patches + would reach the real broker during teardown. + + Quart tracks the task on the app and awaits it at shutdown, so an + in-flight push survives a graceful restart. `partial` rather than passing + `was_cap=` through `add_background_task`, so nothing depends on how that + forwards keyword arguments. + """ + current_app.add_background_task( + partial(push_lane_cap, lane, slots_cap, was_cap=was_cap) + ) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index c4c1d7d..1df00eb 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -563,6 +563,13 @@ async def store_lane_cap( async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict: """Make the running lane obey a cap that is already stored. NO database. + Runs OFF the request path since 2026-09-23 — the endpoint stores the cap, + answers, and hands this to a background task (operator: *"the change + should be queued so that it isn't blocking of the webui"*). Nothing here + changed as a result except who waits for it: the return value is now read + by the log rather than by a browser, and every branch below already + treated failure as "the sizing pass will carry it". + ## What is pushed, and what is not Consumers follow the cap immediately in BOTH directions: zero means off, @@ -612,13 +619,29 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict: # feature that is optional and clearly OFF. # # On the TRANSITION, so re-saving a cap on a lane already running does not - # re-enqueue. And only when the consumer change landed: enqueueing onto a - # queue nothing is consuming would leave the task pending with no - # explanation until the lane returns. + # re-enqueue. + # + # NOT gated on the consumer change having landed, which it was until + # 2026-09-23. The reasoning then was that enqueueing onto a queue nothing + # consumes leaves the task pending — true, and it is the right place for + # it to wait. Gated, a cap raised while the lane was restarting stored the + # cap, let the sizing pass start the consumers a minute later, and left + # the lane running with no model, because nothing else ever asks for one. + # A task parked on the `ml` queue is picked up the moment that happens. fetching = False - if now_on and not was_on and lane.models and applied: + if now_on and not was_on and lane.models: fetching = _enqueue_model_fetch() + # Nobody is waiting on this any more, so the log is where a push that did + # not land has to be visible. Not an error: the value is stored and the + # sizing pass carries it within a minute. + if not applied: + log.info( + "worker_control: %s cap %s stored, not pushed (%s); " + "the sizing pass will carry it", + lane.name, slots_cap, error, + ) + return { "name": lane.name, "slots_cap": slots_cap, diff --git a/frontend/src/components/settings/SystemHealthTab.vue b/frontend/src/components/settings/SystemHealthTab.vue index ae24f91..4ae4b34 100644 --- a/frontend/src/components/settings/SystemHealthTab.vue +++ b/frontend/src/components/settings/SystemHealthTab.vue @@ -299,16 +299,13 @@ async function apply(lane, fields) { text: `${lane.display_name} is on. Downloading its model now — watch ` + 'progress under Activity. It only happens once.', } - } else if (reply && reply.applied === false) { - // Saved but not pushed — the lane is restarting, or the broker blipped. - // NOT an error: the reconcile carries it when the lane answers again, - // and saying "failed" would invite setting it a second time. - notice.value = { - type: 'info', - text: `Saved. ${lane.display_name} is not answering right now — ` - + 'it will pick this up within a few minutes.', - } } + // There is no "saved but not applied" message any more, because there is + // nothing to report yet: the endpoint answers once the cap is STORED and + // tells the running lane afterwards, so the press cannot wait on a broker + // round trip (operator: "we shouldn't have to wait for the validation + // live"). Whether the lane has caught up is what its row says, and the + // row updates on the next poll a few seconds later. } catch (e) { // The endpoint's `detail` is written to be read by a person ("cap 10000 is // above what this container can hold"). Surface it rather than a status diff --git a/frontend/src/stores/systemActivity.js b/frontend/src/stores/systemActivity.js index ca9c05b..efb2617 100644 --- a/frontend/src/stores/systemActivity.js +++ b/frontend/src/stores/systemActivity.js @@ -88,17 +88,37 @@ export const useSystemActivityStore = defineStore('systemActivity', () => { } } - // Change one lane. Returns the endpoint's reply so the caller can tell a - // stored-but-not-yet-live change (`applied: false`) from a live one — the - // difference between "saved, the lane is restarting" and "that failed", - // which the UI must not collapse into one message. + // Change one lane, and PATCH the row from the reply. + // + // It refetched until 2026-09-23, on the reasoning that the server's whole + // table beats a local guess. The cost of that was the operator's: + // + // "something about changing the cap number is blocking to the website... + // it shouldn't be" / "the change should be queued so that it isn't + // blocking of the webui or the system itself. we shouldn't have to wait + // for the validation live." + // + // GET /api/system/workers costs a celery inspect — an eleven-second budget + // — so every press of `+` disabled the stepper until a broker round trip + // the press did not need had finished. The endpoint now answers as soon as + // the cap is stored and tells the lane separately. + // + // Only the three fields the reply actually decides are copied. The live + // columns (pool, active, pending) are measurements this reply does not + // carry and must not invent — the 15s poller brings them, a beat behind, + // which is what they are anyway. // // Deliberately NOT swallowing the error: a refused value (400) carries the // sentence explaining why, and the card shows it. Returning null on failure // would leave the operator with a control that silently did nothing. async function setLane(name, fields) { const reply = await api.post(`/api/system/workers/${name}`, { body: fields }) - await loadLanes() + const row = lanes.value?.lanes?.find((l) => l.name === name) + if (row && reply) { + if (reply.slots_cap !== undefined) row.slots_cap = reply.slots_cap + if (reply.ceiling !== undefined) row.ceiling = reply.ceiling + if (reply.enabled !== undefined) row.enabled = reply.enabled + } return reply } diff --git a/frontend/test/workerLanes.spec.js b/frontend/test/workerLanes.spec.js index b594853..22981de 100644 --- a/frontend/test/workerLanes.spec.js +++ b/frontend/test/workerLanes.spec.js @@ -5,11 +5,18 @@ import { laneStuckFor, useSystemActivityStore } from '../src/stores/systemActivi // Milestone 422 step 4. Covers the store half of the worker-lane dial — the // part that decides what the card can tell the operator. // -// The distinction being protected: a change that was STORED but not pushed -// (`applied: false`, because the lane is restarting) is not a failure, and a -// REFUSED value (400) is. Collapsing those two into one message is how a -// control stops being trustworthy — one invites waiting, the other invites -// changing what you asked for. +// What is protected here since 2026-09-23: pressing the dial must not wait on +// a broker round trip. Operator: *"when the number is changed the change +// should be queued so that it isn't blocking of the webui or the system +// itself. we shouldn't have to wait for the validation live."* The endpoint +// answers once the cap is STORED, so the store patches its row from that +// reply and lets the 15s poll bring the live columns — it used to refetch, +// and GET /api/system/workers costs a celery inspect on an eleven-second +// budget. +// +// And still: a REFUSED value (400) is a failure and must reach the operator +// as one. A control that silently does nothing is worse than one that +// refuses out loud. function stubFetch(handler) { globalThis.fetch = vi.fn(async (url, init) => { @@ -70,7 +77,7 @@ describe('worker lanes store', () => { stubFetch((url, init) => { calls.push({ url, init }) if (init?.method === 'POST') { - return { status: 200, body: { name: 'worker', slots_cap: 2, applied: true } } + return { status: 200, body: { name: 'worker', slots_cap: 2, queued: true } } } return { status: 200, body: LANES_BODY } }) @@ -82,38 +89,73 @@ describe('worker lanes store', () => { expect(JSON.parse(post.init.body)).toEqual({ slots_cap: 2 }) }) - it('setLane refetches so the card shows the server truth, not the guess', async () => { - // The reply is one lane; the table renders all of them plus live pool - // and pending. Patching the local row from the reply would leave every - // other column stale and eventually wrong. + it('setLane does not refetch — that refetch is what blocked the press', async () => { + // It refetched until 2026-09-23, so that the card showed server truth + // rather than a local guess. The cost was the operator's: GET + // /api/system/workers runs a celery inspect, so every press of `+` sat + // with the stepper disabled through a broker round trip the press did not + // need. let gets = 0 stubFetch((url, init) => { - if (init?.method === 'POST') return { status: 200, body: { applied: true } } + if (init?.method === 'POST') { + return { status: 200, body: { slots_cap: 2, ceiling: 8, enabled: true } } + } gets += 1 return { status: 200, body: LANES_BODY } }) const s = useSystemActivityStore() + await s.loadLanes() + gets = 0 await s.setLane('worker', { slots_cap: 2 }) - expect(gets).toBe(1) + + expect(gets).toBe(0) }) - it('a stored-but-unapplied change comes back as applied:false, not an error', async () => { - // The lane is restarting. The value IS saved and the reconcile will carry - // it — so this must reach the card as information, not as a failure that - // invites the operator to set it again. + it('patches the row from the reply, so the new number shows at once', async () => { stubFetch((url, init) => { if (init?.method === 'POST') { return { status: 200, - body: { applied: false, apply_error: 'lane is not running', slots_cap: 2 }, + body: { name: 'worker', slots_cap: 2, ceiling: 8, enabled: true, queued: true }, } } return { status: 200, body: LANES_BODY } }) const s = useSystemActivityStore() - const reply = await s.setLane('worker', { slots_cap: 2 }) - expect(reply.applied).toBe(false) - expect(reply.apply_error).toContain('not running') + await s.loadLanes() + await s.setLane('worker', { slots_cap: 2 }) + + const worker = s.lanes.lanes.find((l) => l.name === 'worker') + expect(worker.slots_cap).toBe(2) + }) + + it('does not invent the live columns the reply cannot know', async () => { + // The reply is decided from the stored cap and the machine's ceiling + // alone — it never asked a worker anything. Pool, active and pending are + // MEASUREMENTS, and the poll a few seconds later is what carries them. + // Zeroing or guessing them here would make the table lie in the direction + // that reads as "the lane stopped". + stubFetch((url, init) => { + if (init?.method === 'POST') { + return { status: 200, body: { slots_cap: 2, ceiling: 8, enabled: true } } + } + return { status: 200, body: LANES_BODY } + }) + const s = useSystemActivityStore() + await s.loadLanes() + const before = { ...s.lanes.lanes[0].live } + await s.setLane('worker', { slots_cap: 2 }) + + expect(s.lanes.lanes[0].live).toEqual(before) + expect(s.lanes.lanes[0].pending).toBe(8) + }) + + it('survives a reply arriving for a lane it has not loaded yet', async () => { + // First paint, or a lane added by a newer build. Patching must not be the + // thing that throws inside the click handler. + stubFetch(() => ({ status: 200, body: { slots_cap: 2 } })) + const s = useSystemActivityStore() + await expect(s.setLane('worker', { slots_cap: 2 })).resolves.toBeTruthy() }) it('a refused value throws so the card can show the reason', async () => { diff --git a/tests/test_api_workers.py b/tests/test_api_workers.py index 90bcace..1f247db 100644 --- a/tests/test_api_workers.py +++ b/tests/test_api_workers.py @@ -10,6 +10,7 @@ import pytest import pytest_asyncio from sqlalchemy import select +from backend.app.api import workers as workers_api from backend.app.models import WorkerLane from backend.app.services import worker_control as wc from backend.app.services.worker_lanes import LANES @@ -28,6 +29,39 @@ async def no_live_workers(monkeypatch): ) +@pytest.fixture(autouse=True) +def queued_pushes(monkeypatch): + """Hold the background push still, and hand the test the hand-off. + + Since 2026-09-23 the endpoint stores the cap, answers, and gives the live + push to a Quart background task — operator: *"the change should be queued + so that it isn't blocking of the webui or the system itself."* A task that + outlived a test's monkeypatches would reach the real broker during + teardown and wait out its timeout there, so every test in this module + captures the hand-off instead, and the ones that care about what the push + DOES run it deliberately with `_run_pushes`. + + Autouse, because a test that forgets is not a test that fails — it is a + test that leaks a 2s broker call into whichever test runs next. + """ + scheduled: list[tuple] = [] + monkeypatch.setattr( + workers_api, "_schedule_push", + lambda lane, slots_cap, was_cap: scheduled.append((lane, slots_cap, was_cap)), + ) + return scheduled + + +async def _run_pushes(scheduled: list[tuple]) -> list[dict]: + """Run what the endpoint queued, in order, and clear the queue.""" + out = [ + await wc.push_lane_cap(lane, cap, was_cap=was_cap) + for lane, cap, was_cap in scheduled + ] + scheduled.clear() + return out + + async def _lane_row(db, name: str) -> WorkerLane: return (await db.execute( select(WorkerLane).where(WorkerLane.name == name) @@ -90,11 +124,11 @@ async def test_the_other_lanes_ship_at_one(client, no_live_workers): @pytest.mark.asyncio async def test_raising_a_cap_stores_it_and_pushes_nothing( - client, db, no_live_workers, + client, db, no_live_workers, queued_pushes, ): """A cap is PERMISSION, not a request. Raising it must not grow the pool - here — that would put workers on a lane with nothing to do — so there is - nothing to push and `applied` is vacuously true. + — that would put workers on a lane with nothing to do — so the queued + push has nothing to say to the broker. This asserted `applied is False` until run 7367, carried over from when the number meant "run this many". The code was right and the test was @@ -105,32 +139,33 @@ async def test_raising_a_cap_stores_it_and_pushes_nothing( assert resp.status_code == 200 body = await resp.get_json() assert body["slots_cap"] == 3 - assert body["applied"] is True assert (await _lane_row(db, "worker")).slots_cap == 3 + assert [r["applied"] for r in await _run_pushes(queued_pushes)] == [True] @pytest.mark.asyncio async def test_turning_a_lane_off_is_stored_even_when_it_cannot_be_pushed( - client, db, no_live_workers, + client, db, no_live_workers, queued_pushes, ): """The direction that DOES push. Consumers follow the cap immediately in both directions — off must take effect when it is asked for — so with nothing answering, the push fails. - That is NOT a failed setting: the value is saved and the sizing pass - carries it within a minute (lesson #4202 — a live change that does not - survive, with nothing saying so). The UI says "saved, not yet live" - rather than "that didn't work", which is the distinction `applied` - exists to carry. + That is NOT a failed setting, and it is why the reply does not wait for + it: the value is saved and the sizing pass carries it within a minute + (lesson #4202 — a live change that does not survive, with nothing saying + so). The push reports the failure to the log, where an operator can find + it; the table shows the lane not answering either way. """ resp = await client.post("/api/system/workers/worker", json={"slots_cap": 0}) assert resp.status_code == 200 - body = await resp.get_json() - assert body["applied"] is False - assert "not running" in body["apply_error"] assert (await _lane_row(db, "worker")).slots_cap == 0 + pushed = await _run_pushes(queued_pushes) + assert pushed[0]["applied"] is False + assert "not running" in pushed[0]["apply_error"] + # --- the cap is the switch --------------------------------------------------- @@ -158,7 +193,7 @@ async def test_raising_it_off_zero_turns_the_lane_on(client, db, no_live_workers @pytest.mark.asyncio async def test_the_model_fetch_fires_on_the_transition_not_on_every_write( - client, db, no_live_workers, monkeypatch, + client, db, no_live_workers, queued_pushes, monkeypatch, ): """Raising the cap off zero downloads SigLIP, once. A second nudge of the same dial must not re-enqueue a multi-GB download — and the trigger must @@ -176,15 +211,42 @@ async def test_the_model_fetch_fires_on_the_transition_not_on_every_write( first = await (await client.post( "/api/system/workers/ml", json={"slots_cap": 1}, )).get_json() + await _run_pushes(queued_pushes) second = await (await client.post( "/api/system/workers/ml", json={"slots_cap": 2}, )).get_json() + await _run_pushes(queued_pushes) + # The reply promises it, the queued push performs it. Both halves are + # checked, because the reply is what the UI says out loud. assert first["fetching_models"] is True assert second["fetching_models"] is False assert fired == [1] +@pytest.mark.asyncio +async def test_the_model_fetch_is_enqueued_even_if_the_lane_is_not_answering( + client, no_live_workers, queued_pushes, monkeypatch, +): + """Turning ML on while it is restarting must still fetch the model. + + It was gated on the consumer change having landed until 2026-09-23, on + the reasoning that a task enqueued onto a queue nothing consumes just sits + there. It does — and that is the right place for it to wait. Gated, this + path stored the cap, let the sizing pass start the consumers a minute + later, and left the lane running with no model, because nothing else ever + asks for one. + """ + fired = [] + monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True) + + await client.post("/api/system/workers/ml", json={"slots_cap": 1}) + pushed = await _run_pushes(queued_pushes) + + assert pushed[0]["applied"] is False, "the fixture must leave the push failing" + assert fired == [1] + + # --- what is refused --------------------------------------------------------- @@ -306,14 +368,12 @@ async def test_the_cap_write_holds_no_session_while_it_pushes(): @pytest.mark.asyncio async def test_raising_a_cap_costs_no_broker_round_trip_at_all( - client, db, no_live_workers, monkeypatch, + client, db, no_live_workers, queued_pushes, monkeypatch, ): - """The common case must be instant. Raising a cap is permission, not a - request — the sizing pass spends it — so there is nothing to tell the - broker, and the operator's `+` should answer immediately rather than - waiting out an inspect.""" - from backend.app.services import worker_control as wc - + """Raising a cap is permission, not a request — the sizing pass spends it + — so there is nothing to tell the broker AT ALL. Asserted on the queued + push rather than on the request, because the request no longer waits for + it either way and would pass this vacuously.""" calls = [] monkeypatch.setattr( wc, "inspect_lanes_sync", lambda: calls.append("inspect") or {}, @@ -328,8 +388,48 @@ async def test_raising_a_cap_costs_no_broker_round_trip_at_all( ) await client.post("/api/system/workers/worker", json={"slots_cap": 1}) + await _run_pushes(queued_pushes) calls.clear() resp = await client.post("/api/system/workers/worker", json={"slots_cap": 6}) + await _run_pushes(queued_pushes) assert resp.status_code == 200 assert calls == [], f"raising a cap talked to the broker: {calls}" + + +@pytest.mark.asyncio +async def test_the_reply_never_waits_for_the_broker_in_any_direction( + client, db, no_live_workers, queued_pushes, monkeypatch, +): + """Operator, 2026-09-23: *"when the number is changed the change should be + queued so that it isn't blocking of the webui or the system itself. we + shouldn't have to wait for the validation live."* + + Raising a cap was already free. The rest were not: turning a lane off is + four `cancel_consumer` messages, and lowering a cap reads the live pool + first — an `inspect` on an eleven-second budget — all of it between the + click and the response, with the stepper disabled throughout. + + Asserted by making any broker call from the request path RAISE. A timing + assertion would be flaky, and counting calls afterwards would pass against + a version that made them and was merely quick about it. + """ + def boom(*args, **kwargs): + raise AssertionError("the request path talked to the broker") + + monkeypatch.setattr(wc, "inspect_lanes_sync", boom) + monkeypatch.setattr(wc, "set_lane_slots_sync", boom) + monkeypatch.setattr(wc, "set_lane_enabled_sync", boom) + + # Every direction: on, up, down, off. + for cap in (1, 4, 2, 0): + resp = await client.post( + "/api/system/workers/worker", json={"slots_cap": cap}, + ) + assert resp.status_code == 200, cap + assert (await resp.get_json())["queued"] is True, cap + assert (await _lane_row(db, "worker")).slots_cap == cap + # The work was handed off rather than skipped — a control that answers + # instantly by doing nothing is the failure this could become. + assert queued_pushes, f"cap {cap} queued no push" + queued_pushes.clear() -- 2.54.0 From 7f1693a40d3aeb731d1591c56c68686fc9a08fd1 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 16:46:22 -0400 Subject: [PATCH 45/94] fix: the ML dial offered slots the machine had no cores to feed (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in flight by Little's law, and the daily CCIP sweep dying on its 1800s soft limit in a numpy matmul. The billiard/pool.py frame in that traceback is the soft-timeout signal handler, not a pool fault. Two causes, both mine. 1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading "keep N_replicas x this within the cores allotted to ML" — a constraint stated where nothing could act on it. A large-memory host offered ~49 slots, the operator took what the dial offered, and the lane asked the box for ~200 torch threads. The number moves onto the lane as `threads_per_slot`, the embedder reads it rather than restating it, and the ceiling is now the smaller of the two bounds. They fail differently on purpose: too little memory is honestly zero, because the first task would OOM the container; too few cores is merely slow, so it floors at one rather than making the lane unreachable on a small box. 2. `scheduled_ccip_auto_apply` scored one image per matmul, over every image in the library, on every daily run — ~119k products each too small to pay for its own BLAS setup. `char_maxima` does the same arithmetic in blocks bounded by elements, so its memory stays flat as either axis grows. Batching changes no arithmetic: a character's score for an image is a max over that image's figures AND that character's prototypes, and max does not care how it is grouped. Pinned against the old loop written out longhand, and against itself with the blocking forced to split every row. The UI copy said the ML ceiling came from memory; it says cores or memory, whichever runs out first. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/ml/ccip.py | 52 ++++++++++++ backend/app/services/ml/embedder.py | 15 +++- backend/app/services/worker_lanes.py | 73 +++++++++++----- backend/app/tasks/ml.py | 23 +++-- .../components/settings/SystemHealthTab.vue | 11 +-- tests/test_ml_dry_helpers.py | 85 +++++++++++++++++++ tests/test_worker_lanes.py | 51 +++++++++++ 7 files changed, 277 insertions(+), 33 deletions(-) diff --git a/backend/app/services/ml/ccip.py b/backend/app/services/ml/ccip.py index 833b55c..fc80ec2 100644 --- a/backend/app/services/ml/ccip.py +++ b/backend/app/services/ml/ccip.py @@ -35,6 +35,58 @@ DEFAULT_SIM_THRESHOLD = 0.85 _FIGURE_KINDS = ("face", "figure") +# How many cosine scores to hold in memory at once, per matmul block. +# 4M float32 is 16 MB — small enough to stay in cache-friendly territory on the +# shared ml lane, large enough that the per-call overhead stops mattering. +_MAX_SCORE_ELEMS = 4_000_000 + + +def char_maxima(q_by_image, allref, seg, np, *, max_elems=_MAX_SCORE_ELEMS): + """(n_images, n_chars) — each image's best cosine to each character. + + `q_by_image` is one L2-normalised `(n_figures, dim)` array per image, in + the order the answer comes back in. `allref` is every character's + prototypes stacked, and `seg` their per-character start offsets into it. + + ## Why this is batched, and why that is safe + + `scheduled_ccip_auto_apply` did this one image at a time — a `(nq, dim) @ + (dim, total)` product per image, over every image in the library on every + run. At ~119k images that is 119k separate matmuls, each too small to pay + for its own BLAS setup, and on 2026-09-23 the daily sweep hit its 1800s + soft limit on the operator's instance. + + Batching changes no arithmetic. The score a character gets for an image is + a max over that image's figures AND over that character's prototypes, and + max does not care in what order or grouping it is taken — so reducing the + prototype axis first (per row, inside a block) and the figure axis after + (per image, across blocks) gives exactly what the per-image loop gave. + That equivalence is what `test_char_maxima_matches_the_per_image_loop` + pins, against the naive form written out longhand. + + Blocked by ROWS rather than done in one product, because the full score + matrix is (all figures in the chunk x every prototype) and that grows with + the library on both axes. The block bound is on elements, so the memory + this uses stays flat as either axis grows. + """ + counts = [len(q) for q in q_by_image] + rows = np.vstack(q_by_image) + total = max(int(allref.shape[0]), 1) + block = max(1, max_elems // total) + + per_row = np.empty((rows.shape[0], len(seg)), dtype=np.float32) + for a in range(0, rows.shape[0], block): + scores = rows[a:a + block] @ allref.T + per_row[a:a + block] = np.maximum.reduceat(scores, seg, axis=1) + + # Start offset of each image's rows. Every image has at least one figure — + # it is in `q_by_image` because a region produced it — so these strictly + # increase, which is what `reduceat` needs to reduce rather than pass a row + # through untouched. + starts = np.cumsum([0] + counts[:-1]) + return np.maximum.reduceat(per_row, starts, axis=0) + + async def _settings_threshold(session: AsyncSession) -> float: val = ( await session.execute( diff --git a/backend/app/services/ml/embedder.py b/backend/app/services/ml/embedder.py index d55646c..abb87cf 100644 --- a/backend/app/services/ml/embedder.py +++ b/backend/app/services/ml/embedder.py @@ -11,12 +11,21 @@ from pathlib import Path import numpy as np from PIL import Image, ImageFile +from ..worker_lanes import LANES_BY_NAME + ImageFile.LOAD_TRUNCATED_IMAGES = True # Cap torch's intra-op threads so each ml-worker replica is a bounded core -# consumer on a shared node (torch otherwise uses all cores). Keep -# N_replicas × this within the cores allotted to ML to avoid oversubscription. -_INTRA_OP_THREADS = 4 +# consumer on a shared node (torch otherwise uses all cores). +# +# Read from the lane rather than restated here. This was a literal 4 beside a +# comment reading "keep N_replicas x this within the cores allotted to ML" — +# a constraint written where nothing could act on it, and nothing did: the ML +# ceiling came from memory alone, offered the operator ~49 slots on a +# large-memory host, and the lane spent 2026-09-23 with ~200 torch threads on +# it. `derived_ceiling` now divides the cores by this number, which only means +# anything while the two are the same number. +_INTRA_OP_THREADS = LANES_BY_NAME["ml"].threads_per_slot DEFAULT_MODEL_NAME = os.environ.get( "SIGLIP_MODEL_NAME", "google/siglip-so400m-patch14-384" diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index 8a23513..aad8f9a 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -139,8 +139,22 @@ class Lane: # tells them to raise it rather than quietly consuming the machine. default_slots_cap: int # True when a slot costs a copy of the ML model rather than just a process. - # The only lane whose ceiling is decided by memory instead of by cores. + # Such a lane is bounded by memory AS WELL AS by cores, never instead of. memory_bound: bool = False + # CPU threads ONE slot uses. More than one for a lane whose work is an + # inference library with its own thread pool: `services/ml/embedder.py` + # calls `torch.set_num_threads` with this number, so a slot is four cores' + # worth of demand rather than one process's. + # + # It lives here because the CEILING has to know it. It was a private + # constant in the embedder with a comment saying "keep N_replicas x this + # within the cores allotted to ML" — a rule stated where nothing could + # enforce it. Nothing did: the ML ceiling was computed from memory alone, + # so a large-memory host offered ~49 slots, the operator took them, and + # 2026-09-23's log shows ~200 torch threads fighting over the box — + # embeds at 107-246s each, and the daily CCIP sweep sharing that pool + # timing out at 1800s. + threads_per_slot: int = 1 # Models this lane downloads the first time it is enabled. Empty for every # lane that needs none, which is how the UI knows whether to warn at all. models: tuple[ModelRequirement, ...] = () @@ -209,6 +223,7 @@ LANES: tuple[Lane, ...] = ( entrypoint_role="ml-worker", default_slots_cap=0, memory_bound=True, + threads_per_slot=4, models=(SIGLIP_MODEL,), optional=True, ), @@ -370,6 +385,21 @@ def lane_for_node(hostname: str) -> Lane | None: return LANES_BY_NAME.get(hostname.split("@", 1)[0]) +def _cpu_bound_slots(lane: Lane) -> int: + """How many slots this container's cores can feed, at `threads_per_slot`. + + Never zero: a machine with fewer cores than one slot wants still runs the + lane, just slowly. That is a real trade an operator may want, and refusing + to offer the lane at all on a small box would make ML unreachable there — + unlike the memory bound, where the honest answer IS zero, because the + first task would OOM the container rather than merely be slow. + """ + cores = container_cpu_count() + if cores is None: + return UNKNOWN_CEILING + return max(MIN_CEILING, cores // lane.threads_per_slot) + + def derived_ceiling(lane: Lane) -> int: """The most slots `lane` may be given on this container. @@ -377,26 +407,31 @@ def derived_ceiling(lane: Lane) -> int: is bounded by what it has NOW rather than by what it had when its row was written. """ - if lane.memory_bound: - total = container_memory_bytes() - if total is None: - log.warning( - "worker_lanes: cannot read a memory limit; capping %s at %d", - lane.name, UNKNOWN_CEILING, - ) - return UNKNOWN_CEILING - usable = total - RESERVED_BYTES - if usable < ML_BYTES_PER_SLOT: - # Honestly zero. A box that cannot hold one model alongside the web - # process must be told it cannot run tagging, not sold a slot that - # will OOM the container the first time it is used. - return 0 - return int(usable // ML_BYTES_PER_SLOT) + by_cpu = _cpu_bound_slots(lane) + if not lane.memory_bound: + return by_cpu - cores = container_cpu_count() - if cores is None: + total = container_memory_bytes() + if total is None: + log.warning( + "worker_lanes: cannot read a memory limit; capping %s at %d", + lane.name, UNKNOWN_CEILING, + ) return UNKNOWN_CEILING - return max(MIN_CEILING, cores) + usable = total - RESERVED_BYTES + if usable < ML_BYTES_PER_SLOT: + # Honestly zero. A box that cannot hold one model alongside the web + # process must be told it cannot run tagging, not sold a slot that + # will OOM the container the first time it is used. + return 0 + + # BOTH bounds, whichever binds first. Memory alone was the whole answer + # until 2026-09-23, and on a large-memory host that is the wrong one: RAM + # said ~49 slots, and each of those slots wants `threads_per_slot` cores. + # The operator raised the cap to what the dial offered and the lane + # starved itself — a control is not allowed to offer a number the machine + # cannot feed. + return min(int(usable // ML_BYTES_PER_SLOT), by_cpu) def ceilings() -> dict[str, int]: diff --git a/backend/app/tasks/ml.py b/backend/app/tasks/ml.py index 4388820..e21c1aa 100644 --- a/backend/app/tasks/ml.py +++ b/backend/app/tasks/ml.py @@ -488,7 +488,7 @@ def scheduled_ccip_auto_apply() -> str: from ..models import ImageRegion, MLSettings, Tag, TagKind from ..models.tag import image_tag - from ..services.ml.ccip import _FIGURE_KINDS + from ..services.ml.ccip import _FIGURE_KINDS, char_maxima from ..services.ml.training_data import _applied_or_rejected, _l2norm SessionLocal = _sync_session_factory() @@ -553,11 +553,22 @@ def scheduled_ccip_auto_apply() -> str: by_img: dict[int, list] = {} for iid, vec in rows: by_img.setdefault(iid, []).append(vec) - for iid, vecs in by_img.items(): - q = _l2norm(np.asarray(vecs, dtype=np.float32), np) # (nq, 768) - colmax = (q @ allref.T).max(axis=0) # (total,) - charmax = np.maximum.reduceat(colmax, seg) # (n_chars,) - for ci in np.where(charmax >= thr)[0]: + if not by_img: + continue + + # One matmul per BLOCK of figures, not one per image. This loop ran + # over every image in the library on every daily run and did a + # matmul too small to pay for itself each time; it hit the 1800s + # soft limit on the operator's instance on 2026-09-23. Same + # arithmetic — see `char_maxima`. + iids = list(by_img) + charmax = char_maxima( + [_l2norm(np.asarray(by_img[i], dtype=np.float32), np) for i in iids], + allref, seg, np, + ) # (n_img, n_chars) + + for row, iid in enumerate(iids): + for ci in np.where(charmax[row] >= thr)[0]: t = ref_tags[int(ci)] if iid in skip[t]: continue diff --git a/frontend/src/components/settings/SystemHealthTab.vue b/frontend/src/components/settings/SystemHealthTab.vue index 4ae4b34..4b96486 100644 --- a/frontend/src/components/settings/SystemHealthTab.vue +++ b/frontend/src/components/settings/SystemHealthTab.vue @@ -170,9 +170,9 @@ FabledCurator to work at all; ML tagging is the one that is genuinely optional, and it ships at zero because switching it on downloads a model. Up to N beneath each dial is what this machine can - hold — memory for ML tagging, processor cores for the rest — and it is - recalculated from the container's real limits every time this page - loads. + hold — processor cores for every lane, and for ML tagging whichever + runs out first, its cores or its memory. It is recalculated from the + container's real limits every time this page loads.

The shipped caps are one of each, which is right for a first boot and @@ -218,8 +218,9 @@

- Each worker loads its own copy, which is why this machine allows it - at most {{ lane.ceiling }}. + Each worker loads its own copy and asks for several processor cores + while it runs, which is why this machine allows it at most + {{ lane.ceiling }}. diff --git a/tests/test_ml_dry_helpers.py b/tests/test_ml_dry_helpers.py index bff8d27..67bed9d 100644 --- a/tests/test_ml_dry_helpers.py +++ b/tests/test_ml_dry_helpers.py @@ -57,3 +57,88 @@ def test_applied_or_rejected_unions_applied_any_source_and_rejected(db_sync): assert skip[b.id] == {imgs[3].id} assert imgs[4].id not in skip[a.id] assert imgs[4].id not in skip[b.id] + + +# --- the CCIP auto-apply sweep's scorer --------------------------------------- +# +# `scheduled_ccip_auto_apply` scored one image per matmul, over every image in +# the library, on every daily run — and on 2026-09-23 it hit its 1800s soft +# limit on the operator's instance. `char_maxima` does the same arithmetic in +# blocks. These pin THAT: same answer, whatever the blocking. + + +def _score_fixture(np): + """Four images with 1-3 figures each, three characters with 2/5/1 + prototypes. Deliberately ragged — equal group sizes would let a wrong + `reduceat` offset pass.""" + from backend.app.services.ml.training_data import _l2norm + + rng = np.random.default_rng(7) + dim = 16 + q_by_image = [ + _l2norm(rng.standard_normal((n, dim)).astype(np.float32), np) + for n in (1, 3, 2, 1) + ] + mats = [ + _l2norm(rng.standard_normal((k, dim)).astype(np.float32), np) + for k in (2, 5, 1) + ] + allref = np.vstack(mats) + seg = np.cumsum([0] + [len(m) for m in mats])[:-1] + return q_by_image, allref, seg + + +def _naive(q_by_image, allref, seg, np): + """The loop as it was written before batching, kept longhand. The point of + comparing against this rather than against a stored array is that it is + the OLD CODE — if the batched form ever diverges, this says so in the + terms the change was justified in.""" + return np.vstack([ + np.maximum.reduceat((q @ allref.T).max(axis=0), seg) for q in q_by_image + ]) + + +def test_char_maxima_matches_the_per_image_loop(): + import numpy as np + + from backend.app.services.ml.ccip import char_maxima + + q_by_image, allref, seg = _score_fixture(np) + got = char_maxima(q_by_image, allref, seg, np) + + assert got.shape == (len(q_by_image), len(seg)) + np.testing.assert_allclose( + got, _naive(q_by_image, allref, seg, np), rtol=1e-6, atol=1e-6, + ) + + +def test_the_answer_does_not_depend_on_where_the_blocks_fall(): + """The one thing batching could get wrong. Rows are reduced over the + PROTOTYPE axis inside a block and over the FIGURE axis afterwards, so a + block boundary may fall in the middle of an image's figures — which is + safe only because max does not care how it is grouped. `max_elems=1` + forces a boundary between every single row.""" + import numpy as np + + from backend.app.services.ml.ccip import char_maxima + + q_by_image, allref, seg = _score_fixture(np) + whole = char_maxima(q_by_image, allref, seg, np, max_elems=10_000_000) + split = char_maxima(q_by_image, allref, seg, np, max_elems=1) + + np.testing.assert_allclose(whole, split, rtol=1e-6, atol=1e-6) + + +def test_one_character_and_one_figure_still_reduces(): + """The degenerate shape `reduceat` is easiest to get wrong: a single + segment starting at 0, and a single row.""" + import numpy as np + + from backend.app.services.ml.ccip import char_maxima + + q = np.array([[1.0, 0.0]], dtype=np.float32) + allref = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32) + got = char_maxima([q], allref, np.array([0]), np) + + assert got.shape == (1, 1) + assert got[0][0] == pytest.approx(1.0) diff --git a/tests/test_worker_lanes.py b/tests/test_worker_lanes.py index 160a552..cdf0a2f 100644 --- a/tests/test_worker_lanes.py +++ b/tests/test_worker_lanes.py @@ -153,10 +153,60 @@ def test_v1_sentinel_is_recognised_as_unlimited(monkeypatch, tmp_path): def test_ml_ceiling_is_memory_divided_by_per_slot_cost(monkeypatch, tmp_path): # 2 GiB reserved for web and the other lanes, then 4 GiB per model copy. + # Cores pinned high so the memory bound is the one being read here. _point_memory_at(monkeypatch, tmp_path, str(14 * wl.GIB)) + monkeypatch.setattr(wl, "container_cpu_count", lambda: 32) assert wl.derived_ceiling(wl.LANES_BY_NAME["ml"]) == 3 +def test_ml_is_bounded_by_cores_as_well_as_memory(monkeypatch, tmp_path): + """The bug of 2026-09-23, in one assertion. + + The ML ceiling was memory ALONE. On the operator's large-memory host that + offered ~49 slots; they took them, and each slot asks torch for + `threads_per_slot` cores — so the lane ran ~200 threads over a box that + has nowhere near that many. Embeds that should be seconds took 107-246s, + and the daily CCIP sweep sharing that pool died on its 1800s soft limit. + + Memory here says 49. The cores say 8 / 4 = 2, and the smaller bound is the + only honest one: a control must not offer a number the machine cannot + feed. + """ + _point_memory_at(monkeypatch, tmp_path, str(200 * wl.GIB)) + monkeypatch.setattr(wl, "container_cpu_count", lambda: 8) + + ml = wl.LANES_BY_NAME["ml"] + assert ml.threads_per_slot == 4 + assert wl.derived_ceiling(ml) == 2 + + +def test_a_few_cores_still_offer_one_ml_slot_rather_than_none( + monkeypatch, tmp_path, +): + """The two bounds fail differently, deliberately. Too little MEMORY is + honestly zero — the first task would OOM the container. Too few CORES is + merely slow, which is a trade an operator may want, so it floors at one + rather than making the lane unreachable on a small box.""" + _point_memory_at(monkeypatch, tmp_path, str(200 * wl.GIB)) + monkeypatch.setattr(wl, "container_cpu_count", lambda: 1) + assert wl.derived_ceiling(wl.LANES_BY_NAME["ml"]) == 1 + + +def test_the_embedder_asks_for_exactly_what_the_ceiling_budgeted(monkeypatch): + """The two halves of the same number, tied. + + `threads_per_slot` is only meaningful because `embedder.load()` calls + `torch.set_num_threads` with it. It lived in the embedder as a private 4 + beside a comment saying "keep N_replicas x this within the cores allotted + to ML" — a constraint stated where nothing could enforce it, and nothing + did. If these two ever drift, the ceiling is budgeting cores for a demand + the worker does not make, and nothing else would notice. + """ + from backend.app.services.ml import embedder + + assert embedder._INTRA_OP_THREADS == wl.LANES_BY_NAME["ml"].threads_per_slot + + def test_a_small_box_is_told_it_cannot_run_tagging(monkeypatch, tmp_path): """Honestly zero rather than a floor of one. A 4GB box cannot hold a model alongside the web process, and offering a slot that OOMs the container the @@ -206,6 +256,7 @@ def test_the_ceiling_is_computed_not_stored(monkeypatch, tmp_path): ceiling when the container's limits change, with no row edit. A stored ceiling would keep authorising what the box no longer has.""" lane = wl.LANES_BY_NAME["ml"] + monkeypatch.setattr(wl, "container_cpu_count", lambda: 32) _point_memory_at(monkeypatch, tmp_path, str(34 * wl.GIB)) big = wl.derived_ceiling(lane) (tmp_path / "memory.max").write_text(str(10 * wl.GIB)) -- 2.54.0 From 45bb7044f7410ed5059acb83ce710e1e3d1edb86 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 23 Sep 2026 18:52:08 -0400 Subject: [PATCH 46/94] feat: the System tab reads a stored sample instead of inspecting per load (4295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- alembic/versions/0107_worker_lane_sample.py | 63 +++++ backend/app/api/system_health.py | 35 ++- backend/app/api/workers.py | 33 ++- backend/app/celery_app.py | 24 +- backend/app/models/__init__.py | 2 + backend/app/models/worker_lane_sample.py | 83 +++++++ backend/app/services/service_roster.py | 132 ++++++---- backend/app/services/worker_control.py | 234 ++++++++++++++---- backend/app/services/worker_lanes.py | 25 ++ backend/app/tasks/maintenance.py | 37 ++- .../components/settings/SystemHealthTab.vue | 30 ++- frontend/src/utils/systemParts.js | 22 ++ frontend/test/systemParts.spec.js | 36 ++- tests/doubles.py | 33 +++ tests/test_api_workers.py | 117 ++++++++- tests/test_service_roster.py | 62 +++++ tests/test_worker_lanes.py | 36 +++ 17 files changed, 871 insertions(+), 133 deletions(-) create mode 100644 alembic/versions/0107_worker_lane_sample.py create mode 100644 backend/app/models/worker_lane_sample.py create mode 100644 tests/doubles.py diff --git a/alembic/versions/0107_worker_lane_sample.py b/alembic/versions/0107_worker_lane_sample.py new file mode 100644 index 0000000..df3fdfa --- /dev/null +++ b/alembic/versions/0107_worker_lane_sample.py @@ -0,0 +1,63 @@ +"""worker_lane_sample — where the sizing sweep leaves what it measured. + +Operator, 2026-09-23, on the System tab: *"there is a repull every time this +page loads — is there a reason this info isn't being tracked in the +background and stored in some way?"* + +`/api/system/workers` ran a full celery inspect on every call — four +broadcasts on an eleven-second budget — and the page polls it every fifteen +seconds. `size_worker_lanes` was already inspecting on a timer to decide pool +sizes, computing exactly these numbers and discarding them. This table is +where they land instead, and the endpoint becomes a plain read. + +## Why a new table rather than columns on `worker_lane` + +`worker_lane` holds the one number an operator sets. Putting a measurement +beside it is the mistake alembic 0105 undid: `slots` sat next to `slots_cap`, +and a measurement next to a preference reads as a second preference. + +No backfill. A row appears when the sweep first runs (within its period), and +until then the lane reads as not-yet-measured, which is true. + +Revision ID: 0107 +Revises: 0106 +Create Date: 2026-09-23 + +""" +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "0107" +down_revision: Union[str, None] = "0106" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_table( + "worker_lane_sample", + sa.Column("lane", sa.String(length=32), primary_key=True), + # Nullable=False with no server_default: the sweep writes every column + # on every upsert, so a row only ever exists complete. + sa.Column("present", sa.Boolean(), nullable=False), + sa.Column("replicas", sa.Integer(), nullable=False), + # Nullable on purpose — unknown, never zero. A worker that answered + # without reporting its pool, and a queue the broker did not answer + # for, must not be summed as empty. + sa.Column("pool", sa.Integer(), nullable=True), + sa.Column("active", sa.Integer(), nullable=False), + sa.Column("reserved", sa.Integer(), nullable=False), + sa.Column("queue_depth", sa.Integer(), nullable=True), + sa.Column( + "measured_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + ) + + +def downgrade() -> None: + op.drop_table("worker_lane_sample") diff --git a/backend/app/api/system_health.py b/backend/app/api/system_health.py index e767ede..92dde1a 100644 --- a/backend/app/api/system_health.py +++ b/backend/app/api/system_health.py @@ -26,7 +26,6 @@ a true statement. from __future__ import annotations import asyncio -import logging import time from datetime import UTC, datetime @@ -36,9 +35,7 @@ from sqlalchemy import select, text from ..config import get_config from ..extensions import get_session from ..models import ServiceSeen -from ..services.service_roster import refresh_if_stale - -log = logging.getLogger(__name__) +from ..services.worker_lanes import SWEEP_PERIOD_SECONDS system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system") @@ -53,6 +50,23 @@ system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system" STALE_AFTER_SECONDS = 90 DOWN_AFTER_SECONDS = 300 +# The celery roster is written by `size_worker_lanes` and by nothing else, so +# these thresholds are only meaningful against ITS cadence. Asserted at import +# rather than left to a reader, because this is precisely the comparison that +# was never made for the GPU agent: its lease poll backed off to 900s while +# the roster called it stopped at 300s, and both numbers were individually +# correct, in different directions, in different files (lesson #4355). +# +# Two clear sweeps before a part is even called STALE. One missed tick is +# routine — the sweep rides the maintenance queue and does an inspect that can +# take eleven seconds — and must not turn the page yellow. +_SWEEPS_BEFORE_STALE = 2 +assert STALE_AFTER_SECONDS >= SWEEP_PERIOD_SECONDS * _SWEEPS_BEFORE_STALE, ( + f"a {SWEEP_PERIOD_SECONDS}s sweep cannot keep a roster fresh against a " + f"{STALE_AFTER_SECONDS}s stale threshold: raise the threshold or shorten " + f"the sweep" +) + # Probes cross a process boundary, so they carry deadlines. A hung Postgres # must make this endpoint say "postgres: down", not hang alongside it. PROBE_TIMEOUT_SECONDS = 2.0 @@ -151,14 +165,11 @@ async def system_health(): parts.append(pg) if pg["state"] == _OK: - # Rate-limited inside; see service_roster on why the web process - # is the right observer. - try: - await refresh_if_stale(session) - await session.commit() - except Exception: # noqa: BLE001 - log.warning("system health: roster refresh failed", exc_info=True) - + # A PURE READ since 2026-09-23. This used to refresh the celery + # roster here, rate-limited to once per 20s — so the roster only + # advanced while somebody had a browser open, and a broadcast rode + # on a request. `size_worker_lanes` writes it now, on a timer, and + # the assertion below is what keeps that cadence honest. rows = ( await session.execute(select(ServiceSeen).order_by(ServiceSeen.display_name)) ).scalars().all() diff --git a/backend/app/api/workers.py b/backend/app/api/workers.py index c0f6638..eb5de55 100644 --- a/backend/app/api/workers.py +++ b/backend/app/api/workers.py @@ -31,7 +31,12 @@ from ..services.worker_control import ( push_lane_cap, store_lane_cap, ) -from ..services.worker_lanes import LANES_BY_NAME, Lane, derived_ceiling +from ..services.worker_lanes import ( + LANES_BY_NAME, + SWEEP_PERIOD_SECONDS, + Lane, + derived_ceiling, +) from ._responses import error_response as _bad workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers") @@ -43,22 +48,26 @@ async def list_lanes(): Response: {lanes: [...], fetched_at: iso8601} - Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is - the surface an operator watches while dragging a stepper, and a cached - reply would show them the value from before their own change and read as - the control having failed. + One database read, and NO broker call. Operator, 2026-09-23: *"there is a + repull every time this page loads — is there a reason this info isn't + being tracked in the background and stored in some way?"* + + It used to inspect the broker here, four broadcasts on an eleven-second + budget, four times a minute per open tab — while `size_worker_lanes` was + already inspecting on a timer and discarding the same numbers. The sweep + stores them now (`worker_lane_sample`) and this reads them. + + So the live figures are up to `SWEEP_PERIOD_SECONDS` old, and each lane + carries the `measured_at` that says so. `sweep_period_seconds` is returned + alongside, so the UI can explain the age without hard-coding the cadence + in a second place. """ - # The session closes BEFORE the broker work. Holding a Postgres connection - # across a celery inspect is what made this page block the whole site — - # see `worker_control.LaneSettings`. This endpoint polls every 15s and the - # inspect budget is 11s, so each poll was pinning a connection for most of - # the interval. async with get_session() as session: settings = await lane_settings(session) - lanes = await lane_view(settings) return jsonify({ - "lanes": lanes, + "lanes": lane_view(settings), "fetched_at": datetime.now(UTC).isoformat(), + "sweep_period_seconds": SWEEP_PERIOD_SECONDS, }) diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index 21c98bf..d341530 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -14,6 +14,7 @@ Queues: from celery import Celery from .config import get_config +from .services.worker_lanes import SWEEP_PERIOD_SECONDS def make_celery() -> Celery: @@ -113,7 +114,14 @@ def make_celery() -> Celery: }, "size-worker-lanes": { "task": "backend.app.tasks.maintenance.size_worker_lanes", - "schedule": 60.0, # every minute. + "schedule": SWEEP_PERIOD_SECONDS, + # + # The number lives in `services/worker_lanes` because three + # places must agree on it: this schedule, the freshness of the + # sample the System tab reads, and the roster staleness + # thresholds in `api/system_health` — which now depend on this + # sweep rather than on a browser being open, and assert their + # headroom over it at import. # # ONE entry, replacing `autoscale-worker-lanes` (60s) and # `reconcile-worker-lanes` (300s) on 2026-09-23. They were two @@ -121,14 +129,16 @@ def make_celery() -> Celery: # existed to stop the reconcile undoing its work; with the # stored `slots` gone there is nothing to disagree about. # - # A minute because it reacts to a BACKLOG, and a five-minute - # reaction to a queue filling up is no reaction. It also now - # carries what the reconcile was for — a worker restarted at - # its ENV concurrency is corrected on the next tick rather - # than after five. + # Fast enough to react to a BACKLOG — a five-minute reaction to + # a queue filling up is no reaction. It also carries what the + # reconcile was for: a worker restarted at its ENV concurrency + # is corrected on the next tick rather than after five. # # Cheap when settled: one inspect plus one LLEN sweep, and no - # control messages at all once every lane matches. + # control messages at all once every lane matches. It is also + # now the ONLY thing that inspects — nothing on a request path + # does — so this is the whole broker cost of the System tab, + # whether nobody or ten tabs are watching. }, "cleanup-old-tasks": { "task": "backend.app.tasks.maintenance.cleanup_old_tasks", diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index 08f1d3f..b925805 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -45,6 +45,7 @@ from .tag_positive_confirmation import TagPositiveConfirmation from .tag_suggestion_rejection import TagSuggestionRejection from .task_run import TaskRun from .worker_lane import WorkerLane +from .worker_lane_sample import WorkerLaneSample __all__ = [ "Base", @@ -96,4 +97,5 @@ __all__ = [ "TagSuggestionRejection", "TaskRun", "WorkerLane", + "WorkerLaneSample", ] diff --git a/backend/app/models/worker_lane_sample.py b/backend/app/models/worker_lane_sample.py new file mode 100644 index 0000000..26503dd --- /dev/null +++ b/backend/app/models/worker_lane_sample.py @@ -0,0 +1,83 @@ +"""worker_lane_sample — the last thing the sizing sweep measured about a lane. + +Milestone 422, 2026-09-23. A MEASUREMENT table, deliberately separate from +`worker_lane`, which holds the one number an operator sets. + +## Why this exists + +Operator, 2026-09-23, looking at the System tab: *"there is a repull every +time this page loads — is there a reason this info isn't being tracked in the +background and stored in some way?"* + +There was not a good one. `/api/system/workers` ran a full celery inspect — +four broadcast round trips on an eleven-second budget — on every call, and +the page polls it every fifteen seconds. Meanwhile `size_worker_lanes` was +already inspecting on a timer to decide pool sizes, computing exactly these +numbers, using them, and throwing them away. The browser then asked the +broker for them again. + +So the sweep writes what it saw here, and the endpoint reads this table. The +request path makes no broker call at all any more. + +## Why NOT columns on `worker_lane` + +Because that is the mistake this milestone already made once and undid. That +table used to carry `slots` — how many workers were running — beside +`slots_cap`, and a measurement sitting next to a preference reads as a second +preference: the operator had to keep two numbers in agreement, and the +autoscaler had to be granted permission to move one of them. + +The distinction is the whole design, so it is a table boundary. Nothing an +operator sets lives here; nothing here is ever an input to a decision about +what they wanted. + +## Freshness is a value, not an assumption + +`measured_at` is returned to the UI, which says how old the reading is rather +than implying it is live. A sample is a fact about a moment, and a page that +presents a one-minute-old number as current is how an operator ends up +mistrusting the whole surface. +""" + +from datetime import datetime + +from sqlalchemy import Boolean, DateTime, Integer, String, func +from sqlalchemy.orm import Mapped, mapped_column + +from .base import Base + + +class WorkerLaneSample(Base): + __tablename__ = "worker_lane_sample" + + # The lane name from services/worker_lanes.LANES. One row per lane, + # overwritten in place: this is the LATEST reading, not a history. A time + # series would be a different table with a different retention problem, + # and nothing has asked for one. + lane: Mapped[str] = mapped_column(String(32), primary_key=True) + + # Whether anything answered for this lane. NOT the same as "zero workers" + # — an unswept absence is not a verdict (snippet #3969). False here means + # the inspect came back without this lane, so every count below is + # meaningless and the UI must say "not answering" rather than "0". + present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) + + replicas: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + + # Pool size of ONE process, nullable because a worker that answered + # without reporting its pool is unknown rather than empty. + pool: Mapped[int | None] = mapped_column(Integer, nullable=True) + + active: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + reserved: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + + # Redis LLEN across the lane's queues. Nullable for the same reason as + # `pool`: a queue the broker did not answer for is unknown, and summing it + # as zero would report a buried lane as idle. + queue_depth: Mapped[int | None] = mapped_column(Integer, nullable=True) + + measured_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), + nullable=False, + server_default=func.now(), + ) diff --git a/backend/app/services/service_roster.py b/backend/app/services/service_roster.py index 7a77363..cd95f1e 100644 --- a/backend/app/services/service_roster.py +++ b/backend/app/services/service_roster.py @@ -31,7 +31,7 @@ from __future__ import annotations import asyncio import logging -from sqlalchemy import func, select +from sqlalchemy import func from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession @@ -40,10 +40,11 @@ from .worker_lanes import LANES, lane_for_node log = logging.getLogger(__name__) -# How stale the roster may be before a health request refreshes it. Comfortably -# under the staleness thresholds that decide a service is missing, so the -# verdict is never limited by how often anyone looked. -REFRESH_TTL_SECONDS = 20.0 +# There is no refresh TTL any more. It existed because the HEALTH REQUEST +# refreshed the roster, rate-limited to 20s so that a page open in two tabs +# did not inspect twice as often. `size_worker_lanes` owns the refresh now, on +# `SWEEP_PERIOD_SECONDS`, so the cadence is a schedule rather than a side +# effect of someone looking. # celery inspect is a broker round trip and this sits on a request path, so it # gets a deadline (rule 156). A broker that has stopped answering must make the @@ -140,10 +141,13 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]: return grouped -async def touch_service( - session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict -) -> None: - """Record that a part checked in just now. +def touch_service_stmt(*, key: str, kind: str, display_name: str, details: dict): + """The upsert that records a check-in, as a statement. + + Built here rather than inline so the async caller (an agent lease, over + the API) and the sync one (the sizing sweep, in a celery task) run the + SAME write. Two spellings of one upsert is the kind of duplication that + stays correct right up until one of them gains a column. Upsert rather than read-modify-write: several web processes and several agents can be doing this at once, and the last writer is simply the most @@ -154,7 +158,7 @@ async def touch_service( stmt = pg_insert(ServiceSeen).values( key=key, kind=kind, display_name=display_name, details=details, ) - stmt = stmt.on_conflict_do_update( + return stmt.on_conflict_do_update( index_elements=[ServiceSeen.key], set_={ "kind": stmt.excluded.kind, @@ -163,7 +167,75 @@ async def touch_service( "last_seen_at": func.now(), }, ) - await session.execute(stmt) + + +async def touch_service( + session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict +) -> None: + """Record that a part checked in just now.""" + await session.execute(touch_service_stmt( + key=key, kind=kind, display_name=display_name, details=details, + )) + + +def _roster_rows(grouped: dict[tuple[str, ...], dict]) -> list[dict]: + """The `touch_service` arguments for everything that answered. + + Split from the write so the async and sync refreshes below share the + mapping as well as the statement — what a roster row IS should not depend + on which kind of session is writing it. + """ + return [ + { + "key": "celery:" + ",".join(queues), + "kind": "celery", + "display_name": role_display_name(queues), + "details": { + "queues": list(queues), + "hostnames": entry["hostnames"], + "replicas": len(entry["hostnames"]), + "active": entry["active"], + }, + } + for queues, entry in grouped.items() + ] + + +def refresh_celery_roster_sync(session) -> None: + """The roster refresh, from the sizing sweep's sync session. + + ## Why the sweep owns this now + + It used to run on the request path, rate-limited to once every 20s by the + newest celery row. So the roster only advanced while somebody had a + browser open — the liveness of the workers was a function of whether + anyone was looking at them, which is the observer-effect version of the + bug this roster exists to prevent. + + Operator, 2026-09-23: *"there is a repull every time this page loads — is + there a reason this info isn't being tracked in the background and stored + in some way?"* + + Now a timer writes it and the page only reads. The cadence is + `SWEEP_PERIOD_SECONDS`, and `api/system_health` asserts it leaves headroom + under the staleness thresholds — because a sweep period and a stale + threshold chosen in different files and never compared is exactly how the + idle GPU agent came to read as stopped (lesson #4355). + + Never raises. A failure means the roster does not advance, and the rows + going stale is then a TRUE report about a broker nobody can reach. + """ + try: + grouped = _inspect_celery_sync() + except Exception: + log.warning( + "service roster: celery inspect failed; roster not refreshed", + exc_info=True, + ) + return + for row in _roster_rows(grouped): + session.execute(touch_service_stmt(**row)) + session.commit() async def refresh_celery_roster(session: AsyncSession) -> None: @@ -186,39 +258,5 @@ async def refresh_celery_roster(session: AsyncSession) -> None: log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True) return - for queues, entry in grouped.items(): - await touch_service( - session, - key="celery:" + ",".join(queues), - kind="celery", - display_name=role_display_name(queues), - details={ - "queues": list(queues), - "hostnames": entry["hostnames"], - "replicas": len(entry["hostnames"]), - "active": entry["active"], - }, - ) - - -async def refresh_if_stale(session: AsyncSession) -> None: - """Refresh the celery roster if nobody has for REFRESH_TTL_SECONDS. - - Rate-limited by the data rather than by a lock: the gate is the newest - last_seen_at across the celery rows, which every web process can see. Two - processes racing through the gate costs one redundant inspect and writes - the same values twice, so the benign outcome needs no coordination to - prevent. - """ - newest = ( - await session.execute( - select(func.max(ServiceSeen.last_seen_at)).where(ServiceSeen.kind == "celery") - ) - ).scalar_one_or_none() - - if newest is not None: - age = (await session.execute(select(func.now()))).scalar_one() - newest - if age.total_seconds() < REFRESH_TTL_SECONDS: - return - - await refresh_celery_roster(session) + for row in _roster_rows(grouped): + await touch_service(session, **row) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 1df00eb..0013a57 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -53,9 +53,10 @@ from dataclasses import dataclass, field from datetime import UTC, datetime from sqlalchemy import func, select +from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession -from ..models import TaskRun, WorkerLane +from ..models import TaskRun, WorkerLane, WorkerLaneSample from .worker_lanes import ( LANES, LANES_BY_QUEUE_KEY, @@ -341,6 +342,76 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]: return rows +@dataclass(frozen=True) +class LaneSample: + """What the sizing sweep last measured about one lane. + + The same fields `LaneLiveState` carries, plus the queue depth and WHEN — + because this one is read from a table rather than from the broker, and a + reading with no timestamp invites being presented as current. + + `measured_at=None` means no sweep has written this lane yet: a fresh + install inside its first period, or a stack whose beat is not running. + Distinct from `present=False` (something asked, nothing answered), and the + UI says different things about the two. + """ + + present: bool = False + replicas: int = 0 + pool: int | None = None + active: int = 0 + reserved: int = 0 + queue_depth: int | None = None + measured_at: datetime | None = None + + +def _lane_depth(lane: Lane, depths: dict[str, int | None]) -> int | None: + """A lane's backlog across its queues — None when NOTHING answered. + + A queue the broker did not answer for must not be summed as zero: an + unknown depth is not an empty one, and reporting a buried lane as idle is + the direction that matters. + """ + known = [depths.get(q) for q in lane.queues] + if not any(d is not None for d in known): + return None + return sum(d for d in known if d is not None) + + +def store_lane_samples_sync(session, live: dict[str, LaneLiveState], depths) -> None: + """Write what the sweep just measured. SYNC — the celery task owns a sync + session, and this is the only place these rows are written. + + Upsert per lane, last writer wins, same shape as `service_roster`'s + `touch_service`: two processes sweeping at once is a benign race that + needs no coordination, because both are recording what they actually saw. + + A lane that did not answer is STILL written, with `present=False`. Skipping + it would leave the previous reading in place and let the page go on showing + a pool that is no longer there — the stale row would read as a current one + (lesson #4202: the row is the thing that has to change). + """ + now = datetime.now(UTC) + for lane in LANES: + state = live.get(lane.name) or LaneLiveState() + values = { + "lane": lane.name, + "present": state.present, + "replicas": state.replicas, + "pool": state.pool, + "active": state.active, + "reserved": state.reserved, + "queue_depth": _lane_depth(lane, depths), + "measured_at": now, + } + stmt = pg_insert(WorkerLaneSample).values(**values) + session.execute(stmt.on_conflict_do_update( + index_elements=[WorkerLaneSample.lane], + set_={k: v for k, v in values.items() if k != "lane"}, + )) + session.commit() + + @dataclass class LaneSettings: """What the DATABASE knows about the lanes — read and finished with before @@ -366,58 +437,80 @@ class LaneSettings: caps: dict[str, int] oldest_by_queue: dict[str, datetime] + # The sizing sweep's last reading per lane. Since 2026-09-23 this is where + # the live numbers come from: the endpoint no longer inspects at all. + samples: dict[str, LaneSample] = field(default_factory=dict) async def lane_settings(session: AsyncSession) -> LaneSettings: - """Every DB read the lane view needs, in one short-lived session.""" + """Every DB read the lane view needs, in one short-lived session. + + Which is now ALL of them. `lane_view` below takes what this returns and + talks to nothing. + """ rows = await _rows_by_name(session) + samples = { + row.lane: LaneSample( + present=row.present, + replicas=row.replicas, + pool=row.pool, + active=row.active, + reserved=row.reserved, + queue_depth=row.queue_depth, + measured_at=row.measured_at, + ) + for row in ( + await session.execute(select(WorkerLaneSample)) + ).scalars() + } return LaneSettings( caps={name: row.slots_cap for name, row in rows.items()}, oldest_by_queue=await _oldest_running_by_queue(session), + samples=samples, ) -async def lane_view(settings: LaneSettings) -> list[dict]: - """Every lane: what is configured, what is live, what it may grow to. +def lane_view(settings: LaneSettings) -> list[dict]: + """Every lane: what is configured, what was last measured, what it may + grow to. NO broker call, and no database — `settings` is the whole input. - Takes the settings rather than a session ON PURPOSE — see `LaneSettings`. - Everything below this line is broker work, and no database connection is - held while it happens. + ## It used to inspect, on every request - `pending` is the honest backlog — Redis depth PLUS reserved — because + Four broadcast round trips on an eleven-second budget, on a page that + polls every fifteen seconds. Operator, 2026-09-23: *"there is a repull + every time this page loads — is there a reason this info isn't being + tracked in the background and stored in some way?"* + + There was one, and it had expired. The docstring here used to say the + endpoint was deliberately uncached because *"this is the surface an + operator watches while dragging a stepper, and a cached reply would show + them the value from before their own change"*. True while a cap change + refetched the table — and that refetch is exactly what was removed in + `1353d34`, so the UI now patches its own row from the write's reply and + nothing depends on this being live. + + Meanwhile `size_worker_lanes` was already inspecting on a timer to decide + pool sizes: the same numbers, computed, used, and discarded, while the + browser asked the broker for them again four times a minute. + + So the sweep writes `worker_lane_sample` and this reads it. The reading is + up to `SWEEP_PERIOD_SECONDS` old, and `measured_at` travels with it so the + UI can say so rather than implying it is current. + + `pending` is still the honest backlog — depth PLUS reserved — because celery prefetches and LLEN alone reads 0 while a worker holds tasks in memory. """ - # A deadline, because this is a request path and `to_thread` on its own is - # an await with no bound (rule 156). `inspect_lanes_sync` never raises and - # every inner call has its own timeout, so the only way past the budget is - # the thread not being scheduled — and a page that renders "not answering" - # is a better answer than one that does not render. - try: - live = await asyncio.wait_for( - asyncio.to_thread(inspect_lanes_sync), - timeout=INSPECT_BUDGET_SECONDS, - ) - except TimeoutError: - log.warning( - "worker_control: inspect exceeded %ss; reporting every lane as " - "not answering", INSPECT_BUDGET_SECONDS, - ) - live = {lane.name: LaneLiveState() for lane in LANES} - depths = await asyncio.to_thread(_queue_depths_sync) oldest = settings.oldest_by_queue - now = datetime.now(UTC) out = [] for lane in LANES: cap = settings.caps[lane.name] - state = live[lane.name] - # None for a queue the broker did not answer for, which must not be - # silently summed as zero — an unknown depth is not an empty one. - known = [depths.get(q) for q in lane.queues] - depth = sum(d for d in known if d is not None) if any( - d is not None for d in known - ) else None + # A lane with no row yet is not-measured, which is distinct from + # measured-as-absent. The default carries `measured_at=None`, and the + # UI says "not measured yet" rather than "not answering". + sample = settings.samples.get(lane.name) or LaneSample() + depth = sample.queue_depth out.append({ "name": lane.name, "display_name": lane.display_name, @@ -444,17 +537,24 @@ async def lane_view(settings: LaneSettings) -> list[dict]: for m in lane.models ], "live": { - "present": state.present, - "replicas": state.replicas, - "pool": state.pool, - "active": state.active, - "reserved": state.reserved, + "present": sample.present, + "replicas": sample.replicas, + "pool": sample.pool, + "active": sample.active, + "reserved": sample.reserved, }, "queue_depth": depth, - "pending": None if depth is None else depth + state.reserved, + "pending": None if depth is None else depth + sample.reserved, + # When the numbers above were read. Per lane rather than one for + # the response, because a lane whose row has never been written + # has no reading at all and must not borrow another lane's. + "measured_at": ( + sample.measured_at.isoformat() if sample.measured_at else None + ), # How long the oldest still-running task on this lane has been - # going, in minutes. The operator asked for a trigger here — grow - # a lane whose tasks run past some duration — and it stayed a + # going, in minutes. Read from `task_run`, not from the sweep, so + # this one IS current. The operator asked for a trigger here — + # grow a lane whose tasks run past some duration — and it stayed a # REPORT: a long task does not finish sooner because the lane # gained a slot, so scaling on it would spend memory to change # nothing. Shown so they can see a lane wedged on one slow job, @@ -606,7 +706,22 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict: # LOWERED on a running lane. Only this direction needs a message, and # only when the pool is actually above the new cap — so it reads the # live pool rather than resizing blind. A raise never reaches here. - live = await asyncio.to_thread(inspect_lanes_sync) + # + # Bounded (rule 156): `to_thread` on its own is an await with no + # deadline, and this runs in a background task where a hang would be + # silent rather than visible as a slow page. On a timeout the lane is + # simply not resized here and the sizing sweep carries it. + try: + live = await asyncio.wait_for( + asyncio.to_thread(inspect_lanes_sync), + timeout=INSPECT_BUDGET_SECONDS, + ) + except TimeoutError: + log.warning( + "worker_control: inspect exceeded %ss lowering %s; leaving the " + "pool to the sizing pass", INSPECT_BUDGET_SECONDS, lane.name, + ) + return _cap_result(lane, slots_cap, now_on, applied, error, False) current = live[lane.name].pool if current is not None and current > slots_cap: applied, error = await asyncio.to_thread( @@ -642,6 +757,15 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict: lane.name, slots_cap, error, ) + return _cap_result(lane, slots_cap, now_on, applied, error, fetching) + + +def _cap_result( + lane: Lane, slots_cap: int, now_on: bool, applied: bool, + error: str | None, fetching: bool, +) -> dict: + """The push's outcome. One builder, because `push_lane_cap` has two exits + and a second literal would be free to disagree with the first.""" return { "name": lane.name, "slots_cap": slots_cap, @@ -746,7 +870,12 @@ def wanted_slots(cap: int, active: int, pending: int | None) -> int: return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0))) -def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]: +def size_lanes_sync( + caps: dict[str, int], + *, + live: dict[str, LaneLiveState] | None = None, + depths: dict[str, int | None] | None = None, +) -> list[LaneSizing]: """Size every lane to its backlog, within the cap. The whole control loop. `caps` is lane name -> slots_cap, read from the database by the caller. @@ -754,6 +883,13 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]: the session, and keeping the DB out of here is what lets it be called from anywhere that already knows the caps. + `live` and `depths` are the measurements. Passing them in is not an + optimisation — it is how the caller gets to KEEP them. The sweep now + stores what it measured (`worker_lane_sample`) so the System tab reads a + table instead of inspecting on every page load, and that is only possible + if the same reading serves both purposes. Measured here when not given, so + every existing caller and test is unaffected. + ## It must converge and then go quiet One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a @@ -770,8 +906,10 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]: be a verdict drawn from an unswept read, and here it is worse than useless: there is nothing to send the message to. """ - live = inspect_lanes_sync() - depths = _queue_depths_sync() + if live is None: + live = inspect_lanes_sync() + if depths is None: + depths = _queue_depths_sync() out: list[LaneSizing] = [] for lane in LANES: @@ -805,11 +943,7 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]: )) continue - known = [depths.get(q) for q in lane.queues] - depth = ( - sum(d for d in known if d is not None) - if any(d is not None for d in known) else None - ) + depth = _lane_depth(lane, depths) pending = None if depth is None else depth + state.reserved want = wanted_slots(cap, state.active, pending) diff --git a/backend/app/services/worker_lanes.py b/backend/app/services/worker_lanes.py index aad8f9a..a1b2b0c 100644 --- a/backend/app/services/worker_lanes.py +++ b/backend/app/services/worker_lanes.py @@ -282,6 +282,31 @@ MIN_POOL_SLOTS = 1 # not to make a small machine unusable. MIN_CEILING = 1 +# How often `size_worker_lanes` runs — the beat schedule, and the freshness of +# everything the System tab shows. +# +# It is here, in the import-light module, because three places have to agree +# about it and they are in different packages: the beat entry in `celery_app`, +# the sample the sweep writes (`worker_lane_sample`), and the roster's +# staleness thresholds in `api/system_health`, which now depend on this sweep +# rather than on a browser being open. +# +# 30s, down from 60s, because the sweep became the ONLY writer of the celery +# roster on 2026-09-23. A part is called stale after 90s of silence, so a +# 60-second sweep left one missed tick between "normal" and "everything is +# yellow". That is the shape of lesson #4355 — a reader's threshold and an +# emitter's cadence chosen in different files and never compared — and the +# fix is headroom plus a test that asserts it, not a number that happens to +# work today. +# +# The cost is one inspect every 30s instead of every 60s; the saving is every +# inspect that used to run on a request path, which with a single tab open +# was roughly four a minute against this two. Consequence worth knowing: the +# pass also SHRINKS an idle lane by one slot per tick, so an idle lane now +# gives its workers back twice as fast. That is the direction the operator +# asked for — *"idle instances quiet down when not running"*. +SWEEP_PERIOD_SECONDS = 30.0 + # What an unreadable limit yields. Low rather than unlimited, on purpose: not # knowing how much memory there is must never read as "plenty". An unswept # absence is not a verdict. diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index bfcaee3..d796bec 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -1368,9 +1368,31 @@ def size_worker_lanes() -> dict: Returns every lane's outcome INCLUDING the ones it held, each with a reason. A pass that only speaks when it acts cannot be debugged on the day it does not. + + ## It is also the only thing that MEASURES, since 2026-09-23 + + It always inspected the broker to decide pool sizes, and then threw the + reading away — while `/api/system/workers` ran the same inspect on every + page load and the System tab polls it four times a minute. Operator: + *"there is a repull every time this page loads — is there a reason this + info isn't being tracked in the background and stored in some way?"* + + So one inspect now feeds three things: the sizing decision, the stored + sample the System tab reads, and the celery roster. No request path + touches the broker any more. + + Order matters. The sample is stored BEFORE the roster refresh, because + that refresh does its own broadcast and a broker that has just started + failing must not cost us the reading we already have. """ from ..models import WorkerLane - from ..services.worker_control import size_lanes_sync + from ..services.service_roster import refresh_celery_roster_sync + from ..services.worker_control import ( + _queue_depths_sync, + inspect_lanes_sync, + size_lanes_sync, + store_lane_samples_sync, + ) # Read INSIDE the session. Reading a column off a detached instance # happens to work while the attribute is still loaded and stops working @@ -1387,7 +1409,18 @@ def size_worker_lanes() -> dict: # let this task disagree with the seed it is meant to be enforcing. return {"sized": []} - sized = size_lanes_sync(caps) + # Measured ONCE, here, and then used three times. Passing them down is + # what makes the reading keepable rather than an implementation detail of + # a function that returns decisions. + live = inspect_lanes_sync() + depths = _queue_depths_sync() + + sized = size_lanes_sync(caps, live=live, depths=depths) + + with _sync_session_factory()() as session: + store_lane_samples_sync(session, live, depths) + refresh_celery_roster_sync(session) + for d in sized: if d.action not in ("held", "skipped"): log.info( diff --git a/frontend/src/components/settings/SystemHealthTab.vue b/frontend/src/components/settings/SystemHealthTab.vue index 4b96486..9a365fb 100644 --- a/frontend/src/components/settings/SystemHealthTab.vue +++ b/frontend/src/components/settings/SystemHealthTab.vue @@ -150,6 +150,24 @@ + +

+ + +

+ +
+ both call it + {{ p.signals.identity_token }} + ({{ Math.round((p.signals.identity ?? 0) * 100) }}% — shared with + another post of theirs, so not conclusive on its own)
Link @@ -98,10 +127,12 @@ import SettingNumberField from '../common/SettingNumberField.vue' const store = usePostAssociationsStore() const enabled = ref(true) +const auto = ref(true) const threshold = ref(0.6) const windowHours = ref(24) watch(() => store.enabled, (v) => { enabled.value = v }, { immediate: true }) +watch(() => store.auto, (v) => { auto.value = v }, { immediate: true }) watch(() => store.threshold, (v) => { threshold.value = v }, { immediate: true }) watch(() => store.windowHours, (v) => { windowHours.value = v }, { immediate: true }) diff --git a/frontend/src/stores/postAssociations.js b/frontend/src/stores/postAssociations.js index e8a2027..b98cc55 100644 --- a/frontend/src/stores/postAssociations.js +++ b/frontend/src/stores/postAssociations.js @@ -6,14 +6,18 @@ import { useAsyncAction } from '../composables/useAsyncAction.js' import { toast } from '../utils/toast.js' // Backs the announcement review queue (#388 E5): "this Patreon post announced -// that Discord drop". Confirm-only, deliberately — a wrongly-asserted link -// tells the operator two different pieces are one, which is worse than no link -// at all, so accept is the ONLY thing that makes a link real. Mirrors -// seriesSuggestions (FC-6.3), which is the same shape for the same reason. +// that Discord drop". A wrongly-asserted link tells the operator two different +// pieces are one, which is worse than no link at all — so what reaches this +// queue is everything FC is NOT sure enough about to act on by itself. +// +// A pair the creator's own working name identifies, where that name is on +// these two posts and nowhere else in their library, is linked without asking +// (`auto`). Mirrors seriesSuggestions (FC-6.3) in shape. export const usePostAssociationsStore = defineStore('postAssociations', () => { const api = useApi() const proposals = ref([]) const enabled = ref(true) + const auto = ref(true) const threshold = ref(0.6) const windowHours = ref(24) const { loading, error, run } = useAsyncAction({ errorAs: 'message' }) @@ -28,6 +32,7 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { async function loadSettings () { const s = await api.get('/api/settings/import') enabled.value = s.discord_link_enabled + auto.value = s.discord_link_auto threshold.value = s.discord_link_threshold windowHours.value = s.discord_link_window_hours } @@ -41,6 +46,11 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { await saveSettings({ discord_link_enabled: v }) } + async function setAuto (v) { + auto.value = v + await saveSettings({ discord_link_auto: v }) + } + async function setThreshold (v) { threshold.value = v await saveSettings({ discord_link_threshold: v }) @@ -76,8 +86,8 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { } return { - proposals, enabled, threshold, windowHours, loading, error, - load, loadSettings, setEnabled, setThreshold, setWindowHours, + proposals, enabled, auto, threshold, windowHours, loading, error, + load, loadSettings, setEnabled, setAuto, setThreshold, setWindowHours, accept, dismiss, rescan } }) -- 2.54.0 From 941f1c6e07fe954954c52ce2172fef03502dc838 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 08:20:12 -0400 Subject: [PATCH 58/94] =?UTF-8?q?fix:=20run=207464's=20two=20failures=20?= =?UTF-8?q?=E2=80=94=20an=20unused=20loop=20target=20and=20a=20missed=20ca?= =?UTF-8?q?ll=20site=20(4392)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both mine, both from the same change. `identity` is unpacked in the write loop and never read there — the decision it feeds happens above it, when `conclusive` is built. flake8-bugbear is on repo-wide and B007 is exactly this. Renamed `_identity`, and `linked += status == "linked"` spelled out as the `if` it actually is. The second is worse, because it was a real assertion silently pointed at the wrong shape. `match_post` now returns `(proposed, linked)`, and when I rewrote the call sites I matched on `) == 0` — so the one comparison in the file that reads `) == 1` kept comparing a tuple to an integer. Found by walking the module's AST for every comparison against `match_post` and every use of a name assigned from it, rather than grepping again with the pattern that had already missed it once. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/post_association_service.py | 5 +++-- tests/test_post_association.py | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/backend/app/services/post_association_service.py b/backend/app/services/post_association_service.py index 399713c..ddab64f 100644 --- a/backend/app/services/post_association_service.py +++ b/backend/app/services/post_association_service.py @@ -391,9 +391,10 @@ class PostAssociationService: auto_id = conclusive[0][0].id linked = 0 - for group, score, signals, identity in scored: + for group, score, signals, _identity in scored: status = "linked" if group.id == auto_id else "pending" - linked += status == "linked" + if status == "linked": + linked += 1 self.session.add(PostAssociation( announcement_post_id=announcement.id, payload_post_id=group.id, diff --git a/tests/test_post_association.py b/tests/test_post_association.py index 2535451..c7c6bed 100644 --- a/tests/test_post_association.py +++ b/tests/test_post_association.py @@ -306,7 +306,7 @@ async def test_a_dismissed_pair_is_never_proposed_again(db): svc = PostAssociationService(db) assert await svc.match_post( teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, - ) == 1 + ) == (1, 0) await db.commit() assoc = (await db.execute(select(PostAssociation))).scalar_one() -- 2.54.0 From f7b3e1501427930c437e3656dd493fb903832f03 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 08:27:20 -0400 Subject: [PATCH 59/94] feat: the drop carrying the teaser's own image links it (4392) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Crop-to-source matching was held until the cheap signals could be shown insufficient. They can: of artist 8's 27 teasers with a drop inside a day, 11 go unlinked, and five are screenshot teasers with no working name at all. So it was tried, on exactly those pairs. Every teaser image correlated against every window of every nearby drop image at five scales, ground truth being the pairs the working name independently confirms, control being unrelated same-artist posts a month away. **It does not separate** — true pairs score as low as 0.401 while the control reaches 0.605, and no threshold divides them. The reason is the one the naive version was rejected for, which turns out to apply just as hard to the careful one: a single artist's work is stylistically homogeneous, so a whole-image comparison between two of their pieces is high whether or not it is the same piece. That is now written down in the module docstring with its numbers, so the next person to reach for it inherits the measurement instead of repeating it. What survived asks a narrower question the measurement shows IS answerable: not "is this a crop of that" but "is this the same image". Same pairs, same control, using the pHash FC already stores on every image — pairs the name confirms score 0, 0 and 20 bits of 256; the nearest unrelated pair in a 29-sample control scores 108. The threshold sits at 32, which is the number gallery_service already calls a near-duplicate, inside a 76-bit gap. It earns its place by being the only signal needing no cooperation from the creator: it works on a teaser called `Screenshot 2026-08-13`, and on a creator whose two platforms share no naming convention. It is quiet most of the time, because a teaser is usually a crop rather than a copy — but where it fires it is close to certain, and it recovers `Cute Selfie, Cute Dress` from the unreachable list. utils/phash warns the hash alone must not decide a MERGE, since variants of one piece collide at this distance. That does not invert here — it is the point. A merge destroys a file, so a variant colliding with its original is a loss; this asks whether two POSTS are about the same piece, and a variant of the drop's image is exactly that. Nothing is deleted either way. Gated on posts like the other two: an image on many of the creator's posts is a banner, not a piece. `_rarity` is public as `rarity` now that all three signals share it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../app/services/post_association_service.py | 129 +++++++++++++++-- backend/app/services/post_naming.py | 8 +- backend/app/utils/phash.py | 27 ++++ tests/test_post_association.py | 131 +++++++++++++++++- 4 files changed, 270 insertions(+), 25 deletions(-) diff --git a/backend/app/services/post_association_service.py b/backend/app/services/post_association_service.py index ddab64f..737fdc8 100644 --- a/backend/app/services/post_association_service.py +++ b/backend/app/services/post_association_service.py @@ -29,27 +29,42 @@ additive, weighted, and no single one of its signals may reach the threshold: marker is in THIS artist's posts, because a habitual emoji is punctuation. IDENTITY evidence says two things are the same thing, and it gets its own -route (see `IDENTITY_FLOOR`): +route (see `IDENTITY_FLOOR`). Two signals, and the stronger one stands rather +than them being summed — saying "the same piece" twice is not more true: 4. **A shared working name.** The creator exports the teaser and the release from one file, and the internal name survives into both platforms untouched. Measured on the operator's artist: `ConnFront` ↔ `ConnFront`. This is the only signal that reaches a pair 23.8 hours apart, which proximity scores at 0.005. +5. **The drop contains the teaser's image.** Rare, and near-certain when it + happens. It is the one signal needing no cooperation from the creator: it + works on a teaser called `Screenshot 2026-08-13`, and on a creator whose + two platforms share no naming convention. ## The one deliberately NOT built -**Crop-to-source matching is HELD, on the plan's own instruction** — it is -real work with real false-positive risk, and it is only worth building once -the cheap signals are shown to be insufficient against the operator's actual -artists. Half of this creator's recent teasers are screenshots carrying no -working name at all, and those pairs are out of reach here; that, measured, is -what would justify it. +**Crop-to-source matching stays held, and now for a measured reason rather +than a cautious one.** -Note also that a naive whole-image SigLIP similarity is NOT that signal. A -cropped teaser and its full version are exactly the pair a whole-image -comparison handles worst, so adding one as a "bonus" would mostly add noise -while looking like progress. +It was deferred until the cheap signals could be shown insufficient. They can: +of artist 8's 27 teasers with a drop inside a day, 11 still go unlinked, and +five of those are screenshot teasers carrying no working name at all. + +So it was tried, on those exact pairs. Every teaser image was correlated +against every window of every nearby drop image at five scales, with the pairs +the working name independently confirms as ground truth and unrelated +same-artist posts a month away as a control. **It does not separate.** True +pairs score as low as 0.401 while the control reaches 0.605 — the two +distributions overlap, and no threshold divides them. + +The reason is the reason the naive version was rejected in the first place, +and it turns out to apply just as hard to the sophisticated one: one artist's +work is stylistically homogeneous, so any whole-image comparison between two +of their pieces is high whether or not it is the same piece. Signal 5 above is +what survived that experiment — it asks a narrower question ("is this the same +image") that the measurement shows is answerable, instead of a broader one +("is this a crop of that") that it shows is not. ## Creator identity comes free, so E4 is not actually a prerequisite @@ -81,12 +96,15 @@ from sqlalchemy import and_, func, or_, select from sqlalchemy.ext.asyncio import AsyncSession from ..models import ImageRecord, ImportSettings, Post, PostAssociation +from ..utils.phash import hamming, hash_bits from ..utils.text import html_to_plain from .discord_grouping import DROP_GROUPER from .post_naming import ( IDENTITY_FLOOR, + MAX_TOKEN_POSTS, marker_frequencies, marker_overlap, + rarity, shared_identity, token_frequencies, working_name_tokens, @@ -143,6 +161,24 @@ MAX_RECENT_DROPS = 200 # are conclusive, 2 more propose, 2 fall short of both. AUTO_LINK_FLOOR = 1.0 +# When the drop simply CONTAINS the teaser's image — a pHash within this many +# of 256 bits. +# +# 32, the same number and unit `gallery_service._diversify_similar` already +# calls a near-duplicate. Measured on artist 8, comparing every teaser against +# every drop within a day: pairs the working name independently confirms score +# 0, 0 and 20, and the nearest unrelated same-artist pair in a 29-sample +# control scores **108**. A 76-bit gap, so the threshold is not finely tuned +# and does not need to be. +# +# `utils/phash` warns that the hash alone must not decide a MERGE, because +# variants of one piece collide at this distance. That warning does not invert +# here, it is the point: merging destroys a file, so a variant colliding with +# its original is a loss, while this is asking whether two POSTS are about the +# same piece — and a variant of the drop's image is exactly that. Nothing is +# deleted either way, so no pixel confirm is needed to accept. +DUPLICATE_MAX_DISTANCE = 32 + @dataclass(frozen=True) class _Corpus: @@ -159,6 +195,8 @@ class _Corpus: token_posts: Counter[str] text_by_post: dict[int, str] marker_posts: Counter[str] + hashes_by_post: dict[int, list[int]] + hash_posts: Counter[int] def proximity_signal(gap: timedelta, window: timedelta) -> float: @@ -195,6 +233,40 @@ def declared_signal(description: str | None) -> float: return 0.0 +def shared_image( + left: list[int], + right: list[int], + hash_posts: Counter[int], + *, + max_distance: int = DUPLICATE_MAX_DISTANCE, + max_frequency: int = MAX_TOKEN_POSTS, +) -> float: + """Strength in [0, 1] that the drop contains the teaser's own image. + + IDENTITY evidence, and the only one of the three that needs no cooperation + from the creator — it works on a teaser named `Screenshot 2026-08-13`, and + on a creator whose two platforms share no naming convention at all. Where + it fires it is close to certain; it is simply quiet most of the time, + because a teaser is usually a crop rather than a copy. + + Rarity-gated on POSTS like the other two: an image the creator puts on many + posts is a banner, not a piece. + """ + if not left or not right: + return 0.0 + best = None + for a in left: + for b in right: + d = hamming(a, b) + if d is None or d > max_distance: + continue + span = max(hash_posts.get(a, 1), hash_posts.get(b, 1), 1) + strength = rarity(span, max_frequency) + if best is None or strength > best: + best = strength + return round(best, 4) if best is not None else 0.0 + + def weighted_score(signals: dict) -> float: return round(sum(WEIGHTS[k] * signals.get(k, 0.0) for k in WEIGHTS), 4) @@ -213,14 +285,20 @@ class PostAssociationService: return self._corpora[artist_id] paths_by_post: dict[int, list[str]] = {} + hashes_by_post: dict[int, list[int]] = {} rows = await self.session.execute( - select(ImageRecord.primary_post_id, ImageRecord.path).where( + select( + ImageRecord.primary_post_id, ImageRecord.path, ImageRecord.phash + ).where( ImageRecord.artist_id == artist_id, ImageRecord.primary_post_id.is_not(None), ) ) - for post_id, path in rows: + for post_id, path, phash in rows: paths_by_post.setdefault(post_id, []).append(path) + bits = hash_bits(phash) + if bits is not None: + hashes_by_post.setdefault(post_id, []).append(bits) text_by_post: dict[int, str] = {} rows = await self.session.execute( @@ -243,6 +321,14 @@ class PostAssociationService: token_posts=token_frequencies(paths_by_post.values()), text_by_post=text_by_post, marker_posts=marker_frequencies(text_by_post.values()), + hashes_by_post=hashes_by_post, + # An image the creator puts on many posts — a banner, a watermark + # plate, a recurring title card — is a habit exactly as a character + # name is, and gets gated the same way. Counted on the EXACT hash, + # which is what a reused file produces. + hash_posts=Counter( + h for hs in hashes_by_post.values() for h in set(hs) + ), ) self._corpora[artist_id] = corpus return corpus @@ -325,17 +411,28 @@ class PostAssociationService: corpus = await self._corpus(announcement.artist_id) here = corpus.tokens_by_post.get(announcement.id, set()) here_text = corpus.text_by_post.get(announcement.id, "") + here_hashes = corpus.hashes_by_post.get(announcement.id, []) made = 0 scored: list[tuple[Post, float, dict, float]] = [] for group in await self._candidate_groups(announcement, window=window): if group.id in already: continue - identity, token = shared_identity( + named, token = shared_identity( here, corpus.tokens_by_post.get(group.id, set()), corpus.token_posts, ) + # The two identity signals answer the same question by different + # means, so the stronger one stands rather than them being summed: + # a name and a shared image both say "the same piece", and saying + # it twice is not more true. + copied = shared_image( + here_hashes, + corpus.hashes_by_post.get(group.id, []), + corpus.hash_posts, + ) + identity = max(named, copied) circumstantial = { "proximity": proximity_signal( _post_time(group) - _post_time(announcement), window, @@ -370,7 +467,9 @@ class PostAssociationService: if score < threshold: continue signals = {**circumstantial, "identity": identity} - if token: + if copied: + signals["identity_image"] = copied + if token and named >= copied: # Carried so the queue can say WHY. A review queue that cannot # explain itself is one the operator learns to click through. signals["identity_token"] = token diff --git a/backend/app/services/post_naming.py b/backend/app/services/post_naming.py index 9137cc2..e63aff4 100644 --- a/backend/app/services/post_naming.py +++ b/backend/app/services/post_naming.py @@ -214,10 +214,10 @@ def token_frequencies(posts: Iterable[Iterable[str]]) -> Counter[str]: return counts -def _rarity(freq: int, max_frequency: int) -> float: +def rarity(freq: int, max_frequency: int) -> float: """Rarity of one token within an artist's own corpus, in [0, 1]. - Shared by BOTH signals deliberately. They carried one formula each + Shared by EVERY rarity-gated signal deliberately. They carried one each until 2026-09-24, and the copies drifted: the filename signal grew a frequency gate and the marker signal never did, so a creator's habitual emoji scored the same 1.00 as a marker they had used twice. One @@ -262,7 +262,7 @@ def shared_identity( # The rarest shared token decides — one decisive token beats three vague # ones. token = min(shared, key=lambda t: (frequencies.get(t, 0), -len(t), t)) - strength = round(_rarity(max(frequencies.get(token, 1), 1), max_frequency), 4) + strength = round(rarity(max(frequencies.get(token, 1), 1), max_frequency), 4) # A token sitting exactly ON the cap decays to zero, and naming it anyway # would hand the review queue a reason that carries no weight — "matched on # loislanetb2", with nothing behind it. Measured: that token is on 6 of this @@ -372,7 +372,7 @@ def marker_overlap( if not shared: return 0.0 score = sum( - (1.0 if _SYMBOL.match(t) else 0.25) * _rarity(frequencies.get(t, 1), max_frequency) + (1.0 if _SYMBOL.match(t) else 0.25) * rarity(frequencies.get(t, 1), max_frequency) for t in shared ) return round(min(1.0, score), 4) diff --git a/backend/app/utils/phash.py b/backend/app/utils/phash.py index 4592b3a..66d127a 100644 --- a/backend/app/utils/phash.py +++ b/backend/app/utils/phash.py @@ -80,6 +80,33 @@ def _seek_first_frame(pil_image) -> None: pass +def hash_bits(hex_str: str | None) -> int | None: + """A stored pHash hex string as an integer, or None if it is missing or + unparseable. Fails CLOSED, like every other gate in this module. + + Parsed to an int rather than an imagehash object because the caller that + needs this compares one image against many: `int.bit_count()` on an XOR is + a machine instruction, where rebuilding a 16x16 boolean array per + comparison is not. + """ + if not hex_str: + return None + try: + return int(hex_str, 16) + except (TypeError, ValueError): + return None + + +def hamming(a: int | None, b: int | None) -> int | None: + """Bits differing between two parsed hashes, or None if either is absent. + + Out of 256 at HASH_SIZE 16. + """ + if a is None or b is None: + return None + return (a ^ b).bit_count() + + def compute_phash(pil_image) -> str | None: """Perceptual hash of an opened PIL image, as a hex string. None on any failure (videos/unreadable/non-image). diff --git a/tests/test_post_association.py b/tests/test_post_association.py index c7c6bed..ff9a890 100644 --- a/tests/test_post_association.py +++ b/tests/test_post_association.py @@ -7,6 +7,7 @@ of what follows pins refusals, and the central one is structural rather than behavioural: the threshold sits above every single signal weight, which is what makes "time proximity alone is never sufficient" arithmetic instead of a hope. """ +from collections import Counter from datetime import UTC, datetime, timedelta import pytest @@ -23,11 +24,13 @@ from backend.app.models import ( from backend.app.services.discord_grouping import DROP_GROUPER from backend.app.services.post_association_service import ( DECLARED_MENTION, + DUPLICATE_MAX_DISTANCE, WEIGHTS, PostAssociationService, declared_signal, proximity_signal, rescan, + shared_image, weighted_score, ) from backend.app.services.post_feed_service import PostFeedService @@ -131,13 +134,18 @@ async def _artist_with_channels(db, name: str): return artist, patreon, discord -async def _images(db, artist, post, ext, names): +async def _images(db, artist, post, ext, names, phashes=None): """Attach named files to a post. The NAME is the point — the working-name - signal reads it, so a test that cares about identity supplies one.""" + signal reads it, so a test that cares about identity supplies one. + + `phashes` aligns with `names`; a test that cares about the shared-image + signal supplies those instead (or as well). + """ for i, name in enumerate(names): db.add(ImageRecord( path=f"/images/{artist.id}/{ext}_{i}_{name}.jpg", sha256=f"{ext}{i}{name}".ljust(64, "0")[:64], + phash=(phashes or [None] * len(names))[i], size_bytes=10, mime="image/jpeg", width=10, height=10, origin="downloaded", primary_post_id=post.id, artist_id=artist.id, )) @@ -145,19 +153,19 @@ async def _images(db, artist, post, ext, names): async def _teaser(db, artist, source, *, at, body, ext="teaser", names=None, - title="New piece"): + title="New piece", phashes=None): post = Post( source_id=source.id, artist_id=artist.id, external_post_id=ext, post_date=at, post_title=title, description=body, ) db.add(post) await db.flush() - await _images(db, artist, post, ext, names or [ext]) + await _images(db, artist, post, ext, names or [ext], phashes) return post async def _drop(db, artist, source, *, at, ext="fc-drop:1", body=None, - names=()): + names=(), phashes=None): post = Post( source_id=source.id, artist_id=artist.id, external_post_id=ext, post_date=at, description=body, synthesized_by=DROP_GROUPER, @@ -166,7 +174,7 @@ async def _drop(db, artist, source, *, at, ext="fc-drop:1", body=None, db.add(post) await db.flush() if names: - await _images(db, artist, post, ext.replace(":", "-"), names) + await _images(db, artist, post, ext.replace(":", "-"), names, phashes) return post @@ -831,3 +839,114 @@ async def test_a_drop_another_post_already_claims_is_never_taken(db): ) )).scalars().all() assert [r.status for r in rows] == ["pending"] + + +# --- when the drop just contains the teaser's image ------------------------- +# +# Measured on artist 8, every teaser against every drop within a day: pairs the +# working name independently confirms score 0, 0 and 20 bits of 256, and the +# nearest unrelated same-artist pair in a 29-sample control scores 108. The +# threshold sits at 32 — the same number gallery_service already calls a +# near-duplicate — inside a 76-bit gap. + +_PIECE = "a5" * 32 # the image +_REENCODED = "a5" * 31 + "a4" # the same image, one bit different +_UNRELATED = "5a" * 32 # 256 bits away — every bit differs + + +@pytest.mark.asyncio +async def test_the_drop_carrying_the_teasers_own_image_links_it(db): + """The one signal that needs no cooperation from the creator. No shared + name, nothing said about Discord, 20 hours apart — and the drop is + carrying the same picture.""" + artist, patreon, discord = await _artist_with_channels(db, "dupartist") + now = datetime.now(UTC) + teaser = await _teaser( + db, artist, patreon, at=now - timedelta(hours=20), body="a preview", + ext="t0", names=["Alpha"], phashes=[_PIECE], + ) + await _drop(db, artist, discord, at=now, names=["Beta"], + phashes=[_REENCODED]) + await db.commit() + + made = await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, + auto_link=True, + ) + await db.commit() + + assert made == (1, 1) + assoc = (await db.execute(select(PostAssociation))).scalar_one() + assert assoc.signals["identity_image"] == 1.0 + assert "identity_token" not in assoc.signals, ( + "the names share nothing — claiming one would be a false reason" + ) + + +@pytest.mark.asyncio +async def test_a_different_picture_links_nothing(db): + """The negative the threshold exists for. 256 bits apart is two different + images, whatever else the posts have in common.""" + artist, patreon, discord = await _artist_with_channels(db, "diffartist") + now = datetime.now(UTC) + teaser = await _teaser( + db, artist, patreon, at=now - timedelta(hours=20), body="a preview", + ext="t0", names=["Alpha"], phashes=[_PIECE], + ) + await _drop(db, artist, discord, at=now, names=["Beta"], + phashes=[_UNRELATED]) + await db.commit() + + made = await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, + auto_link=True, + ) + await db.commit() + + assert made == (0, 0) + + +@pytest.mark.asyncio +async def test_an_image_the_creator_reuses_everywhere_links_nothing(db): + """The same guard the other two signals have, on the third. A banner, a + watermark plate or a recurring title card is a habit, not a piece — and it + would otherwise link every post carrying it to every drop carrying it.""" + artist, patreon, discord = await _artist_with_channels(db, "bannerartist") + now = datetime.now(UTC) + teaser = await _teaser( + db, artist, patreon, at=now - timedelta(hours=20), body="a preview", + ext="t0", names=["Alpha"], phashes=[_PIECE], + ) + await _drop(db, artist, discord, at=now, names=["Beta"], + phashes=[_REENCODED]) + for i in range(7): + await _teaser( + db, artist, patreon, at=now - timedelta(days=30 + i), body="older", + ext=f"other{i}", names=[f"Gamma{i}"], phashes=[_PIECE], + ) + await db.commit() + + svc = PostAssociationService(db) + made = await svc.match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, + auto_link=True, + ) + await db.commit() + + assert made == (0, 0) + corpus = await svc._corpus(artist.id) + assert corpus.hash_posts[int(_PIECE, 16)] == 8 + + +def test_a_missing_hash_is_not_a_match(): + """Fails CLOSED, like every gate in utils/phash. An image whose pHash was + never computed must read as "no evidence", never as "identical to the + other thing that also has none".""" + assert shared_image([], [], Counter()) == 0.0 + + +def test_the_duplicate_threshold_sits_inside_the_measured_gap(): + """Stated as a property so the number cannot drift out of the gap that + justifies it: 20 bits was the widest true pair, 108 the nearest unrelated + one.""" + assert 20 < DUPLICATE_MAX_DISTANCE < 108 -- 2.54.0 From 30a263a47a8db1743c7f755b05a61c03e812724e Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 11:25:52 -0400 Subject: [PATCH 60/94] feat: a teaser's card references the drop it announced and the piece's variants (4402, 4401) The operator's problem: a Patreon teaser is a pointer, and its card showed the censored crop plus a text link while the content it pointed at sat on another card. The fix is a REFERENCE, not an absorption: "the nested items on the unified post are a duplicate or reference of existing content". Nothing is written. Discord posts keep their own rows, dates and places in the feed. - post_unification: for each teaser with a linked association, the drop's images and text, plus its variant family: Discord images sharing the seed's gated LEADING working name, or a phash near-duplicate, within a window of the teaser. One hop only, oldest first. - Measured on artist 8 before writing it: of 121 message pairs 2-60 days apart that share a gated token, 106 share the leading name and all read as real families. Of the 15 sharing only a trailing word, 14 are sibling pieces and one is a plain collision (`bottom`, 56 days). The family cap is 8, not the pairing cap of 6, because `tentacooler` and `0-k1` (6 posts each) are real families. - The feed drops a linked drop's own card only within discord_link_fold_hours of its teaser (default 24): "only hidden from the post view they're posted the same day". Older referenced posts stay where they landed. - post_association.linked_by records whether FC or a person made the link, so the card can say so. Undo is the existing dismiss. - `image0` (gallery-dl's fallback name) becomes a stopword. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- alembic/versions/0110_unified_post_card.py | 59 +++ backend/app/api/settings.py | 8 + backend/app/models/import_settings.py | 25 + backend/app/models/post_association.py | 7 + .../app/services/post_association_service.py | 8 +- backend/app/services/post_feed_service.py | 40 +- backend/app/services/post_naming.py | 53 ++- backend/app/services/post_unification.py | 427 ++++++++++++++++++ tests/test_api_settings_downloader.py | 31 ++ tests/test_post_naming.py | 45 ++ tests/test_post_unification.py | 395 ++++++++++++++++ 11 files changed, 1087 insertions(+), 11 deletions(-) create mode 100644 alembic/versions/0110_unified_post_card.py create mode 100644 backend/app/services/post_unification.py create mode 100644 tests/test_post_unification.py diff --git a/alembic/versions/0110_unified_post_card.py b/alembic/versions/0110_unified_post_card.py new file mode 100644 index 0000000..f1724fa --- /dev/null +++ b/alembic/versions/0110_unified_post_card.py @@ -0,0 +1,59 @@ +"""The unified post card — fold window, family window, and who linked a pair. + +Milestone 388, #4402 and #4401. A Patreon teaser's card shows the Discord drop +it announced, and the rest of that piece's variants, by REFERENCE: nothing is +absorbed, nothing changes owner, and every Discord post keeps its own place. + +Three columns: + +* `import_settings.discord_link_fold_hours` — a linked drop leaves the feed + only when it is this close to its teaser (the same release, shown twice). +* `import_settings.discord_family_window_days` — how far from the teaser the + card reaches for variants. 60 is measured: named families spread up to 44 + days on artist 8, every collision found over 500. +* `post_association.linked_by` — "fc" or "operator", so a link FC made by + itself can say so on the card and offer the undo the operator asked for. + +Revision ID: 0110 +Revises: 0109 +Create Date: 2026-09-24 + +""" +import sqlalchemy as sa +from alembic import op + +revision = "0110" +down_revision = "0109" +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column( + "import_settings", + sa.Column( + "discord_link_fold_hours", + sa.Float(), + nullable=False, + server_default=sa.text("24"), + ), + ) + op.add_column( + "import_settings", + sa.Column( + "discord_family_window_days", + sa.Float(), + nullable=False, + server_default=sa.text("60"), + ), + ) + op.add_column( + "post_association", + sa.Column("linked_by", sa.String(length=16), nullable=True), + ) + + +def downgrade(): + op.drop_column("post_association", "linked_by") + op.drop_column("import_settings", "discord_family_window_days") + op.drop_column("import_settings", "discord_link_fold_hours") diff --git a/backend/app/api/settings.py b/backend/app/api/settings.py index ffcbd6f..c8bc828 100644 --- a/backend/app/api/settings.py +++ b/backend/app/api/settings.py @@ -45,6 +45,8 @@ _EDITABLE_FIELDS = ( "discord_link_threshold", "discord_link_window_hours", "discord_link_auto", + "discord_link_fold_hours", + "discord_family_window_days", "extdl_mega_enabled", "extdl_gdrive_enabled", "extdl_mediafire_enabled", @@ -182,6 +184,12 @@ async def update_import_settings(): return jsonify( {"error": "discord_link_window_hours must be a positive number"} ), 400 + # Zero is meaningful for both: fold nothing, or reference no variants. + for key in ("discord_link_fold_hours", "discord_family_window_days"): + if key in body: + v = body[key] + if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0: + return jsonify({"error": f"{key} must be a number >= 0"}), 400 if "wip_title_tagging_enabled" in body and not isinstance( body["wip_title_tagging_enabled"], bool ): diff --git a/backend/app/models/import_settings.py b/backend/app/models/import_settings.py index 35218a6..3a0bba2 100644 --- a/backend/app/models/import_settings.py +++ b/backend/app/models/import_settings.py @@ -158,6 +158,31 @@ class ImportSettings(Base): server_default="true", ) + # The unified card (#4402). A linked Discord drop is NOT absorbed into its + # teaser — it keeps its own post, date and provenance, and the teaser's card + # shows it by REFERENCE. Operator, 2026-09-24: *"discord 'posts' land as + # normal and only hidden from the post view they're posted the same day."* + # + # So the drop's own card leaves the feed only when it sits within this many + # hours of the teaser that references it — the adjacency that reads as the + # same thing twice. Hours rather than a calendar day: a teaser at 23:00 and + # its drop at 01:00 are one release, and "the same day" has no timezone + # the server can know. + discord_link_fold_hours: Mapped[float] = mapped_column( + Float, nullable=False, default=24.0, + server_default="24", + ) + # How far from the teaser the card reaches for the rest of a piece's + # variants — the wips, alts and censor passes a creator trickles out under + # one working name (#4401). Measured on artist 8: named families spread a + # median 5 days and up to 44, while every name collision found spreads + # over 500. A reference, not a regrouping, so a generous value costs one + # extra thumbnail at worst — never a post moved or hidden. + discord_family_window_days: Mapped[float] = mapped_column( + Float, nullable=False, default=60.0, + server_default="60", + ) + # #830 off-platform file-host downloads — per-host enable lever (default on, # rule #26). Column names are extdl__enabled so the worker reads them # via getattr(settings, f"extdl_{host}_enabled", True). diff --git a/backend/app/models/post_association.py b/backend/app/models/post_association.py index 96eaa3f..9e78feb 100644 --- a/backend/app/models/post_association.py +++ b/backend/app/models/post_association.py @@ -91,6 +91,13 @@ class PostAssociation(Base): status: Mapped[str] = mapped_column( String(16), nullable=False, server_default="pending", index=True ) + # WHO linked it: "fc" when the matcher linked a conclusive pair by itself + # (discord_link_auto), "operator" when a person accepted it. The card needs + # this to be honest — a link FC asserted on its own says so and offers an + # undo, which the operator chose over a silent merge (#4402). NULL on a row + # that is not linked, and on rows linked before the column existed, all of + # which an operator accepted: auto-linking shipped in the same release. + linked_by: Mapped[str | None] = mapped_column(String(16), nullable=True) created_at: Mapped[datetime] = mapped_column( DateTime(timezone=True), nullable=False, server_default=func.now() ) diff --git a/backend/app/services/post_association_service.py b/backend/app/services/post_association_service.py index 737fdc8..f5706c6 100644 --- a/backend/app/services/post_association_service.py +++ b/backend/app/services/post_association_service.py @@ -500,6 +500,7 @@ class PostAssociationService: score=score, signals=signals, status=status, + linked_by="fc" if status == "linked" else None, )) made += 1 return made, linked @@ -526,14 +527,19 @@ class PostAssociationService: if a is None: return None a.status = "linked" + a.linked_by = "operator" return {"id": a.id, "status": a.status} async def dismiss(self, association_id: int) -> dict | None: a = await self.session.get(PostAssociation, association_id) if a is None: return None - # Kept, not deleted — the row is what remembers the rejection. + # Kept, not deleted — the row is what remembers the rejection. It is + # also the undo for a link FC made itself (#4402): the unified card + # dismisses the pair, and the dismissed row stops the next sweep from + # linking it straight back. a.status = "dismissed" + a.linked_by = None return {"id": a.id, "status": a.status} async def linked_for(self, post_ids: list[int]) -> dict[int, list[dict]]: diff --git a/backend/app/services/post_feed_service.py b/backend/app/services/post_feed_service.py index 6207667..e41b5d2 100644 --- a/backend/app/services/post_feed_service.py +++ b/backend/app/services/post_feed_service.py @@ -19,6 +19,7 @@ from ..models import ( ExternalLink, ImageProvenance, ImageRecord, + ImportSettings, Post, PostAttachment, Source, @@ -118,6 +119,15 @@ class PostFeedService: # from. `around` and `get_post` deliberately do NOT apply this: reaching # a member by id is how you inspect a grouping. stmt = stmt.where(Post.absorbed_by_post_id.is_(None)) + # A linked Discord drop is shown ON its teaser's card by reference + # (#4402), so its own card sitting beside that teaser is the same + # release twice. Only then is it left out — an older drop keeps its + # place, because a reference does not take anything out of history. + fold_hours = await self._fold_hours() + if fold_hours > 0: + from .post_unification import fold_clause + + stmt = stmt.where(fold_clause(fold_hours)) if artist_id is not None: stmt = stmt.where(Post.artist_id == artist_id) if platform is not None: @@ -169,9 +179,12 @@ class PostFeedService: thumbs_map = await self._thumbnails_for(post_ids) atts_map = await self._attachments_for(post_ids) links_map = await self._links_for(post_ids) + unified_map = await self._unified_for([p for p, _, _ in rows]) items = [ - self._to_dict(post, artist, source, thumbs_map, atts_map, links_map) + self._to_dict( + post, artist, source, thumbs_map, atts_map, links_map, unified_map, + ) for post, artist, source in rows ] return {"items": items, "next_cursor": next_cursor} @@ -213,6 +226,7 @@ class PostFeedService: anchor_item = self._to_dict( anchor_post, anchor_artist, anchor_source, thumbs_map, atts_map, await self._links_for([anchor_post.id]), + await self._unified_for([anchor_post]), ) return { "items": newer["items"] + [anchor_item] + older["items"], @@ -239,6 +253,7 @@ class PostFeedService: item = self._to_dict( post, artist, source, thumbs_map, atts_map, await self._links_for([post.id]), + await self._unified_for([post]), ) item["description_full"] = html_to_plain(post.description) # Full (uncapped) translated description for the detail view (#143). @@ -385,9 +400,27 @@ class PostFeedService: return await PostAssociationService(self.session).linked_for(post_ids) + async def _unified_for(self, posts: list[Post]) -> dict[int, dict]: + """The reference set each teaser's card shows (#4402). Local import for + the same reason as `_links_for`: it reaches the association service.""" + from .post_unification import PostUnificationService + + return await PostUnificationService(self.session).unified_for(posts) + + async def _fold_hours(self) -> float: + """`discord_link_fold_hours`, read without assuming the row exists. + + The feed is the one surface that must not fail on a settings row the + caller never needed, so a missing row folds nothing rather than + raising — which is also exactly how the feed behaved before this. + """ + settings = await self.session.get(ImportSettings, 1) + return float(settings.discord_link_fold_hours) if settings is not None else 0.0 + def _to_dict( self, post: Post, artist: Artist, source: Source | None, thumbs_map: dict, atts_map: dict, links_map: dict | None = None, + unified_map: dict | None = None, ) -> dict: plain_full = html_to_plain(post.description) if post.description else None if plain_full is None: @@ -436,6 +469,11 @@ class PostFeedService: # post is the drop). Always a list so the UI never branches on # absence. "associations": (links_map or {}).get(post.id, []), + # #4402. On a teaser with a linked drop: what the card shows BY + # REFERENCE — the drop's images, the piece's older variants, and + # the text of each — plus who made each link, so one FC made by + # itself can say so and offer the undo. None on every other post. + "unified": (unified_map or {}).get(post.id), # Non-null on a chat message a synthetic post absorbed. The feed # filters these out, but `around`/`get_post` still reach them, and # the UI uses this to explain why a post it linked to is not in the diff --git a/backend/app/services/post_naming.py b/backend/app/services/post_naming.py index e63aff4..d20ab52 100644 --- a/backend/app/services/post_naming.py +++ b/backend/app/services/post_naming.py @@ -112,6 +112,10 @@ _STOPWORDS = frozenset({ # `{user[name]}` as it for ~1,600 files, so it is the single most common # "name" in the library and identifies nothing. "none", + # gallery-dl's fallback when a Discord attachment has no filename of its + # own. Measured on artist 8: four unrelated images across 1,974 days, and + # the one false family the leading-name rule admitted inside 60 days. + "image0", }) # A bare year: still needed for the TEXT signal, where words and numbers are @@ -170,19 +174,17 @@ def _strip_prefixes(stem: str) -> str: return _HASH_SUFFIX.sub("", stem) -def working_name_tokens(path: str) -> set[str]: - """The identity-bearing tokens in one image's filename. +def _ordered_tokens(path: str) -> list[str]: + """The identity-bearing tokens of one filename, in the order written. - Returns an EMPTY set for a name that carries no working title — a - screenshot, a bare number, a stopword. Empty means "no evidence", which the - caller must treat as silence rather than as a weak match; see the module - docstring for the false positive that rule exists for. + The one tokenizer both public readings share, so the set of names and the + leading name cannot disagree about what counts as a name. """ stem = _strip_prefixes(PurePosixPath(path).stem) if _SCREENSHOT.match(stem.strip()): - return set() + return [] - out: set[str] = set() + out: list[str] = [] # Hyphens are kept INSIDE tokens — `0-k` is a real working name on the live # instance, and splitting on hyphen would reduce it to a single character # and then discard it for being too short. @@ -192,10 +194,43 @@ def working_name_tokens(path: str) -> set[str]: continue if tok in _STOPWORDS or not _HAS_LETTER.search(tok): continue - out.add(tok) + if tok not in out: + out.append(tok) return out +def working_name_tokens(path: str) -> set[str]: + """The identity-bearing tokens in one image's filename. + + Returns an EMPTY set for a name that carries no working title — a + screenshot, a bare number, a stopword. Empty means "no evidence", which the + caller must treat as silence rather than as a weak match; see the module + docstring for the false positive that rule exists for. + """ + return set(_ordered_tokens(path)) + + +def leading_name(path: str) -> str | None: + """The FIRST identity token of a filename — the piece, not its decoration. + + Creators lead with what the piece is and trail with what this export of it + is: `Year_20k_wip1`, `not_sombra_21-cumpeen`, `Tentacooler_c_ins`. Content + words sit at the tail, and they span too FEW posts for any frequency cap + to catch — `nude`, `cum` and `top` are on three of artist 8's posts each. + Position is what separates them from a name. + + Measured on artist 8, Discord messages 2-60 days apart sharing a gated + token: 121 pairs. The 106 sharing the leading name all read as one piece's + trickle; of the 15 sharing only a trailing word, 14 are sibling pieces + (`Bea_Machamp_Shiny_*` / `Bea_Machoke_Shiny_*`) and one is a plain + collision (`Undyne_insert_bottom_only-C` / `Lichgalclc_Lingerie_Bottom_21`). + + None when the name carries no identity at all — see `working_name_tokens`. + """ + tokens = _ordered_tokens(path) + return tokens[0] if tokens else None + + def token_frequencies(posts: Iterable[Iterable[str]]) -> Counter[str]: """How many of ONE ARTIST's POSTS each working-name token appears in. diff --git a/backend/app/services/post_unification.py b/backend/app/services/post_unification.py new file mode 100644 index 0000000..cc1b902 --- /dev/null +++ b/backend/app/services/post_unification.py @@ -0,0 +1,427 @@ +"""The unified post card — a teaser shows what it points at (#4402, #4401). + +Milestone 388. A Patreon teaser is a POINTER: a cropped, censored fragment +whose job is to say "the full set is in Discord". Until this module the card +rendered the fragment and a text link, and the reader had to make the join FC +had already made. + +Operator, 2026-09-24: *"the teaser from the patreon post doesn't show the items +that it's supposed to reference so I'm trying to unify the teaser post with the +content it's meant to draw attention to."* + +## A reference, never an absorption + +`discord_grouping` folds chat messages into a synthetic post by transferring +ownership (`absorbed_by_post_id`). That is the wrong primitive here, and the +operator said so directly: *"the nested items on the unified post are a +duplicate or reference of existing content. that's why they can show similar +items and not erase or invalidate the way the discord items landed."* + +So nothing here writes. The Discord posts keep their own rows, dates and +places in the feed; the teaser's card DISPLAYS them. That is also what makes +reaching back for older variants safe at all: a wrong reference shows one +extra thumbnail in one place, where a wrong regrouping would move content. + +## What a teaser references + +1. The Discord drops a `linked` PostAssociation joins it to (#4392) — accepted + by the operator, or linked by FC on a conclusive name match. +2. The rest of that piece's VARIANT FAMILY (#4401): the wips, alts and censor + passes a creator trickles out under one working name, days or weeks apart. + +Families are found by the creator's LEADING working name, not by any shared +token and not by image similarity — see `post_naming.leading_name` for the +measurement, and lesson #4400 for why a whole-image comparison between two +works by one artist cannot separate "same piece" from "same artist". +""" +from __future__ import annotations + +from collections import Counter +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import datetime, timedelta + +from sqlalchemy import and_, exists, extract, func, select +from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy.orm import aliased + +from ..models import ( + ImageProvenance, + ImageRecord, + ImportSettings, + Post, + PostAssociation, + Source, +) +from ..utils.phash import hamming, hash_bits +from ..utils.text import html_to_plain, truncate_at_word +from .discord_grouping import PLATFORM as DISCORD +from .gallery_service import thumbnail_url +from .post_association_service import DUPLICATE_MAX_DISTANCE +from .post_naming import leading_name, rarity, token_frequencies + +# A leading name spanning this many of ONE ARTIST's posts is a habit — a +# character the creator returns to — not one piece's trickle. +# +# Its own value rather than post_naming.MAX_TOKEN_POSTS (6), which is +# calibrated for PAIRING two posts, and measured too tight for a family. On +# artist 8, `tentacooler` spans 6 posts over 7 days and `0-k1` 6 posts over 10: +# both real families, both gated out at 6. At 8, `anya` (7) passes the cap — +# and has no pair inside the family window, which is what the window is for. +FAMILY_MAX_POSTS = 8 + +# The text each referenced post contributes to the card, per post. The card +# clamps it again; this keeps a long Discord thread from making the feed +# payload the size of the thread. +TEXT_LIMIT = 280 + + +@dataclass(frozen=True) +class Candidate: + """One image as the family search sees it.""" + + image_id: int + post_id: int + path: str + phash: int | None + at: datetime + + +def family( + seed: Iterable[Candidate], + pool: Iterable[Candidate], + name_posts: Counter[str], + hash_posts: Counter[int], + *, + anchor: datetime, + window: timedelta, + max_posts: int = FAMILY_MAX_POSTS, +) -> list[Candidate]: + """The images in `pool` that belong to the same piece as `seed`. + + A member shares a seed image's LEADING working name, or is a perceptual + near-duplicate of one (the same file re-posted), and lies within `window` + of `anchor` — the teaser's date. Oldest first, so the card reads as the + trickle it was. + + ONE hop from the seed, never transitive. Every measured family is one hop + from any of its members, because the members share the name; chaining is + what lets a family drift from `Year_20k` to whatever `Year_20k_Base`'s + other tokens happen to touch. + + Both identity routes are rarity-gated against `max_posts`, exactly as the + matcher gates them. A leading name the creator uses across many posts is a + character, and a hash on many posts is a banner. + """ + seed = list(seed) + names = { + name for c in seed + if (name := leading_name(c.path)) is not None + and rarity(name_posts.get(name, 0), max_posts) > 0 + } + hashes = [ + c.phash for c in seed + if c.phash is not None and rarity(hash_posts.get(c.phash, 0), max_posts) > 0 + ] + taken = {c.image_id for c in seed} + + out: list[Candidate] = [] + for c in pool: + if c.image_id in taken or abs(c.at - anchor) > window: + continue + named = leading_name(c.path) in names + copied = c.phash is not None and any( + (d := hamming(c.phash, h)) is not None and d <= DUPLICATE_MAX_DISTANCE + for h in hashes + ) + if named or copied: + out.append(c) + taken.add(c.image_id) + return sorted(out, key=lambda c: (c.at, c.image_id)) + + +def _when(post: Post) -> datetime: + return post.post_date or post.downloaded_at + + +def _text(post: Post) -> str | None: + plain = html_to_plain(post.description) if post.description else None + if not plain or not plain.strip(): + return None + return truncate_at_word(plain.strip(), TEXT_LIMIT)[0] + + +@dataclass +class _Artist: + """Everything the family search needs about one artist, loaded once.""" + + rows: dict[int, tuple] # image_id -> (post_id, path, phash, sha, mime, thumb) + posts: dict[int, Post] + platform: dict[int, str | None] # post_id -> platform + name_posts: Counter[str] + hash_posts: Counter[int] + + +class PostUnificationService: + def __init__(self, session: AsyncSession): + self.session = session + self._artists: dict[int, _Artist] = {} + + async def _artist(self, artist_id: int) -> _Artist: + if artist_id in self._artists: + return self._artists[artist_id] + + posts: dict[int, Post] = {} + platform: dict[int, str | None] = {} + for post, plat in (await self.session.execute( + select(Post, Source.platform) + .outerjoin(Source, Post.source_id == Source.id) + .where(Post.artist_id == artist_id) + )).all(): + posts[post.id] = post + platform[post.id] = plat + + rows: dict[int, tuple] = {} + paths_by_post: dict[int, list[str]] = {} + hashes_by_post: dict[int, set[int]] = {} + for img_id, post_id, path, phash, sha, mime, thumb in (await self.session.execute( + select( + ImageRecord.id, ImageRecord.primary_post_id, ImageRecord.path, + ImageRecord.phash, ImageRecord.sha256, ImageRecord.mime, + ImageRecord.thumbnail_path, + ).where( + ImageRecord.artist_id == artist_id, + ImageRecord.primary_post_id.is_not(None), + ) + )).all(): + bits = hash_bits(phash) + rows[img_id] = (post_id, path, bits, sha, mime, thumb) + paths_by_post.setdefault(post_id, []).append(path) + if bits is not None: + hashes_by_post.setdefault(post_id, set()).add(bits) + + # Counted over EVERY post the artist has, exactly as the matcher's + # corpus counts them — a family is judged against the whole library, + # not against the slice inside the window, or a character name would + # look rare in any quiet month. + found = _Artist( + rows=rows, + posts=posts, + platform=platform, + name_posts=token_frequencies(paths_by_post.values()), + hash_posts=Counter(h for hs in hashes_by_post.values() for h in hs), + ) + self._artists[artist_id] = found + return found + + async def _drop_images(self, drop_ids: list[int]) -> dict[int, list[int]]: + """drop post id -> its image ids, through provenance as the feed reads them. + + A synthetic drop owns no image outright: its images belong to the + member messages, and `discord_grouping` gives the drop a provenance row + for each. The primary_post_id arm keeps any image that has one and no + row, the same union `PostFeedService._thumbnails_for` takes. + """ + out: dict[int, list[int]] = {pid: [] for pid in drop_ids} + if not drop_ids: + return out + links = ( + select( + ImageProvenance.image_record_id.label("image_id"), + ImageProvenance.post_id.label("post_id"), + ) + .where(ImageProvenance.post_id.in_(drop_ids)) + .union( + select( + ImageRecord.id.label("image_id"), + ImageRecord.primary_post_id.label("post_id"), + ).where(ImageRecord.primary_post_id.in_(drop_ids)) + ) + .subquery() + ) + for img_id, pid in (await self.session.execute( + select(links.c.image_id, links.c.post_id).order_by(links.c.image_id) + )).all(): + out[pid].append(img_id) + return out + + async def unified_for(self, posts: Iterable[Post]) -> dict[int, dict]: + """post id -> the card's reference set, for each post that HAS one. + + Only teasers get one: a post with at least one `linked` association on + the announcing side. Every other post is absent from the result, and + the card renders exactly as it did before this module existed. + """ + teasers = {p.id: p for p in posts if p.synthesized_by is None} + if not teasers: + return {} + links = (await self.session.execute( + select(PostAssociation) + .where( + PostAssociation.status == "linked", + PostAssociation.announcement_post_id.in_(list(teasers)), + ) + .order_by(PostAssociation.id) + )).scalars().all() + if not links: + return {} + + settings = await self.session.get(ImportSettings, 1) + window = timedelta(days=float( + settings.discord_family_window_days if settings is not None else 60.0 + )) + drop_images = await self._drop_images( + sorted({a.payload_post_id for a in links}) + ) + + by_teaser: dict[int, list[PostAssociation]] = {} + for a in links: + by_teaser.setdefault(a.announcement_post_id, []).append(a) + + out: dict[int, dict] = {} + for teaser_id, assocs in by_teaser.items(): + teaser = teasers[teaser_id] + artist = await self._artist(teaser.artist_id) + out[teaser_id] = self._compose(teaser, assocs, drop_images, artist, window) + return out + + def _compose( + self, + teaser: Post, + assocs: list[PostAssociation], + drop_images: dict[int, list[int]], + artist: _Artist, + window: timedelta, + ) -> dict: + def candidate(img_id: int) -> Candidate | None: + row = artist.rows.get(img_id) + if row is None: + return None + post_id, path, bits, *_ = row + post = artist.posts.get(post_id) + if post is None: + return None + return Candidate(img_id, post_id, path, bits, _when(post)) + + drop_ids = [a.payload_post_id for a in assocs] + shown = [i for d in drop_ids for i in drop_images.get(d, [])] + own = [i for i, row in artist.rows.items() if row[0] == teaser.id] + seed = [c for i in own + shown if (c := candidate(i)) is not None] + + # Variants come from Discord only. That is where a creator trickles + # them out, it is the corpus the family rule was measured on, and it + # keeps one teaser from pulling a DIFFERENT teaser's crop onto its card. + pool = [ + c for i, row in artist.rows.items() + if artist.platform.get(row[0]) == DISCORD + and (c := candidate(i)) is not None + ] + variants = family( + seed, pool, artist.name_posts, artist.hash_posts, + anchor=_when(teaser), window=window, + ) + + def thumb(img_id: int, post_id: int, role: str) -> dict | None: + row = artist.rows.get(img_id) + if row is None: + return None + _pid, _path, _bits, sha, mime, tp = row + return { + "image_id": img_id, + "thumbnail_url": thumbnail_url(tp, sha, mime), + "mime": mime, + "post_id": post_id, + "role": role, + } + + own_ids = set(own) + thumbnails: list[dict] = [] + seen: set[int] = set(own_ids) + for drop_id in drop_ids: + for img_id in drop_images.get(drop_id, []): + if img_id in seen: + continue + if (t := thumb(img_id, drop_id, "drop")) is not None: + thumbnails.append(t) + seen.add(img_id) + for c in variants: + if c.image_id in seen: + continue + if (t := thumb(c.image_id, c.post_id, "variant")) is not None: + thumbnails.append(t) + seen.add(c.image_id) + + # The text of every item the card unifies — the operator's *"the + # unified card should also contain the text for any of the items + # unified on it"*. A drop's own description already joins its member + # messages, so a variant's text is its MESSAGE, read off the member + # post that owns the image. A line said twice (`@everyone 🍈🍈` on + # every message of a drop) is shown once. + texts: list[dict] = [] + said: set[str] = set() + + def add_text(post: Post | None, role: str) -> None: + if post is None: + return + text = _text(post) + if text is None or text in said: + return + said.add(text) + texts.append({ + "post_id": post.id, + "role": role, + "date": _when(post).isoformat(), + "text": text, + }) + + for drop_id in drop_ids: + add_text(artist.posts.get(drop_id), "drop") + for post_id in dict.fromkeys(c.post_id for c in variants): + add_text(artist.posts.get(post_id), "variant") + + return { + "links": [ + { + "association_id": a.id, + "post_id": a.payload_post_id, + # "fc" | "operator" | None (linked before the column + # existed — an operator accept, every one of them). + "linked_by": a.linked_by, + "token": (a.signals or {}).get("identity_token"), + } + for a in assocs + ], + "thumbnails": thumbnails, + "variant_count": sum(1 for t in thumbnails if t["role"] == "variant"), + "texts": texts, + } + + +def fold_clause(fold_hours: float): + """WHERE clause: this post is NOT a linked drop sitting beside its teaser. + + Operator: *"discord 'posts' land as normal and only hidden from the post + view they're posted the same day."* Everything else stays — an older + variant the teaser also references is history, and a reference does not + remove it from history. + + Built on `post_date`/`downloaded_at`, not the feed's `resurfaced_at`-led + sort key: whether two posts are the same release is a question about when + they were published, not about where the feed has since moved one. + """ + teaser = aliased(Post) + # The SQL-standard EXTRACT(epoch FROM …), which every Postgres accepts. + gap = func.abs(extract( + "epoch", + func.coalesce(Post.post_date, Post.downloaded_at) + - func.coalesce(teaser.post_date, teaser.downloaded_at), + )) + return ~exists( + select(PostAssociation.id) + .join(teaser, teaser.id == PostAssociation.announcement_post_id) + .where(and_( + PostAssociation.payload_post_id == Post.id, + PostAssociation.status == "linked", + gap <= fold_hours * 3600, + )) + ) diff --git a/tests/test_api_settings_downloader.py b/tests/test_api_settings_downloader.py index dbaebde..093ca6c 100644 --- a/tests/test_api_settings_downloader.py +++ b/tests/test_api_settings_downloader.py @@ -135,3 +135,34 @@ async def test_auto_linking_refuses_a_non_boolean(client): "/api/settings/import", json={"discord_link_auto": "yes"} ) assert resp.status_code == 400 + + +# --- #4402: the unified card's two windows --------------------------------- + + +@pytest.mark.asyncio +async def test_the_unified_card_windows_ship_with_their_measured_defaults(client): + """24h is "the same day" without a timezone the server cannot know; 60 days + is the measured family boundary (named families up to 44d on artist 8, + every collision over 500).""" + resp = await client.get("/api/settings/import") + body = await resp.get_json() + assert body["discord_link_fold_hours"] == 24 + assert body["discord_family_window_days"] == 60 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("key", ["discord_link_fold_hours", "discord_family_window_days"]) +async def test_zero_turns_a_unified_card_window_off(client, key): + """Zero is a real answer for both: fold nothing, reference no variants.""" + resp = await client.patch("/api/settings/import", json={key: 0}) + assert resp.status_code == 200 + assert (await resp.get_json())[key] == 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("key", ["discord_link_fold_hours", "discord_family_window_days"]) +@pytest.mark.parametrize("bad", [-1, "24", True]) +async def test_a_unified_card_window_refuses_nonsense(client, key, bad): + resp = await client.patch("/api/settings/import", json={key: bad}) + assert resp.status_code == 400 diff --git a/tests/test_post_naming.py b/tests/test_post_naming.py index 3bc241b..0b2e2ed 100644 --- a/tests/test_post_naming.py +++ b/tests/test_post_naming.py @@ -20,6 +20,7 @@ from backend.app.services.post_naming import ( IDENTITY_FLOOR, MAX_MARKER_POSTS, MAX_TOKEN_POSTS, + leading_name, marker_frequencies, marker_overlap, shared_identity, @@ -305,3 +306,47 @@ def test_marker_overlap_cannot_be_called_without_the_frequencies(): write by accident. A default would have kept it one keyword away.""" with pytest.raises(TypeError): marker_overlap("\U0001F348", "\U0001F348") + + +# --- the leading name: what a variant family is keyed on (#4401) ------------ + + +@pytest.mark.parametrize( + "path, expected", + [ + ("20240301_1213141516171819_01_Year_20K_wip3.png", "year"), + ("20240414_1213141516171820_01_not_sombra_21-cumpeen.png", "not"), + ("20240101_1213141516171821_02_Tentacooler_c_ins.png", "tentacooler"), + ("01_((0-k.jpg", "0-k"), + # The legacy era's `0071` has no letter, so the name is the first + # token that IS one — not "whatever came first". + ("85317841_media_212565911_0071 NoHeart__c3118a69f3__c3118a69f3.jpg", "noheart"), + ], +) +def test_the_leading_name_is_the_piece_not_its_decoration(path, expected): + assert leading_name(path) == expected + + +def test_a_trailing_content_word_is_never_the_leading_name(): + """Measured: `Undyne_insert_bottom_only-C` and `Lichgalclc_Lingerie_Bottom_21` + share `bottom`, 56 days apart, and are unrelated. `bottom` spans only three + posts, so no frequency cap can refuse it — position is what does.""" + assert leading_name("Undyne_insert_bottom_only-C.png") == "undyne" + assert leading_name("Lichgalclc_Lingerie_Bottom_21.png") == "lichgalclc" + + +@pytest.mark.parametrize( + "path", + ["01_Screenshot 2026-08-13 000004.png", "20240101_1213141516171822_01_image0.png"], +) +def test_a_name_with_no_identity_has_no_leading_name(path): + """`image0` is gallery-dl's fallback for an attachment with no name — + measured on four unrelated images across 1,974 days.""" + assert leading_name(path) is None + + +def test_the_leading_name_is_always_one_of_the_names(): + """One tokenizer serves both readings, so they cannot disagree about what + counts as a name.""" + path = "20240301_1213141516171819_01_Year_20K_wip3.png" + assert leading_name(path) in working_name_tokens(path) diff --git a/tests/test_post_unification.py b/tests/test_post_unification.py new file mode 100644 index 0000000..618c1ec --- /dev/null +++ b/tests/test_post_unification.py @@ -0,0 +1,395 @@ +"""#4402 / #4401: a teaser's card shows what it points at — by reference. + +Operator, 2026-09-24: *"the teaser from the patreon post doesn't show the items +that it's supposed to reference"*, and on how: *"discord 'posts' land as normal +and only hidden from the post view they're posted the same day. the nested +items on the unified post are a duplicate or reference of existing content."* + +Two properties carry the whole design, and most of what follows pins them: + +* NOTHING MOVES. A referenced Discord post keeps its row, its date and its + place in the feed; only a drop sitting beside its own teaser is left out. +* A family is the creator's LEADING working name, one hop from the seed, inside + a window. The refusals are the measured false positives, not invented ones. +""" +from collections import Counter +from datetime import UTC, datetime, timedelta + +import pytest +from sqlalchemy import select + +from backend.app.models import ( + Artist, + ImageRecord, + ImportSettings, + Post, + PostAssociation, + Source, +) +from backend.app.services.discord_grouping import DROP_GROUPER +from backend.app.services.post_association_service import PostAssociationService +from backend.app.services.post_feed_service import PostFeedService +from backend.app.services.post_unification import ( + FAMILY_MAX_POSTS, + Candidate, + family, +) + +T0 = datetime(2026, 9, 1, 12, 0, tzinfo=UTC) +WINDOW = timedelta(days=60) + + +def _c(image_id, name, *, days=0, post_id=None, phash=None): + """One image, Discord-shaped, `days` from T0.""" + return Candidate( + image_id=image_id, + post_id=post_id if post_id is not None else image_id, + path=f"20260901_12345678901234{image_id:04d}_01_{name}.png", + phash=phash, + at=T0 + timedelta(days=days), + ) + + +def _family(seed, pool, *, names=None, hashes=None): + return family( + seed, pool, Counter(names or {}), Counter(hashes or {}), + anchor=T0, window=WINDOW, + ) + + +# --- the family rule, pure --------------------------------------------------- + + +def test_the_older_wips_of_the_same_piece_are_its_family(): + """The operator's ask exactly: *"yellowroom trickles out variants and I want + them to show in the grouped post even if they're older"*. Measured shape: + `Year_20k_wip1 -> wip3 -> Base -> Cndm` over 44 days.""" + seed = [_c(1, "Year_20k_Cndm")] + pool = [_c(2, "Year_20k_wip1", days=-44), _c(3, "Year_20K_wip3", days=-44), + _c(4, "Year_20k_Base", days=-20)] + + assert [c.image_id for c in _family(seed, pool)] == [2, 3, 4] + + +def test_a_family_reads_oldest_first(): + """So the card shows the trickle in the order it happened.""" + seed = [_c(1, "svtt_drench_b")] + pool = [_c(2, "svtt_wip5", days=-1), _c(3, "svtt_wip1", days=-3)] + + assert [c.image_id for c in _family(seed, pool)] == [3, 2] + + +def test_a_namesake_outside_the_window_is_not_family(): + """Time does most of the work. Measured on artist 8: every collision found + spreads over 500 days — `ashley` 1258, `anya` 1217, `image0` 1974.""" + seed = [_c(1, "Ashley_TAIGA")] + pool = [_c(2, "Ashley_Re4_A", days=-1258)] + + assert _family(seed, pool) == [] + + +def test_a_shared_trailing_word_is_not_family(): + """The one plain collision inside 60 days on artist 8: `bottom`, three + posts, 56 days apart. No frequency cap can refuse a word that rare — the + leading-name rule does.""" + seed = [_c(1, "Undyne_insert_bottom_only-C")] + pool = [_c(2, "Lichgalclc_Lingerie_Bottom_21", days=-56)] + + assert _family(seed, pool) == [] + + +def test_a_leading_name_the_creator_uses_everywhere_is_not_a_family(): + """A character is a habit, not a piece. At the cap the name is gated even + inside the window.""" + seed = [_c(1, "Bea_Machamp_Shiny")] + pool = [_c(2, "Bea_Machoke_Shiny", days=-5)] + + assert _family(seed, pool, names={"bea": FAMILY_MAX_POSTS}) == [] + + +def test_the_family_cap_admits_the_measured_long_families(): + """`tentacooler` spans 6 posts over 7 days and `0-k1` 6 over 10 — real + families, both lost at the pairing cap of 6. That is why the family cap is + its own number.""" + seed = [_c(1, "Tentacooler")] + pool = [_c(2, "Tentacooler_c_ins", days=-7)] + + assert [c.image_id for c in _family(seed, pool, names={"tentacooler": 6})] == [2] + + +def test_a_near_duplicate_joins_the_family_without_a_name(): + """Half of this creator's teasers are screenshots, which carry no name. The + same file re-posted is still the same file.""" + seed = [_c(1, "image0", phash=0b1011)] + pool = [_c(2, "image0", days=-10, phash=0b1010)] + + assert [c.image_id for c in _family(seed, pool)] == [2] + + +def test_a_distant_hash_is_not_a_duplicate(): + """Lesson #4400: same-artist images are similar whether or not they are the + same piece, so only a NEAR-duplicate counts — the matcher's own line.""" + seed = [_c(1, "image0", phash=0)] + pool = [_c(2, "image0", days=-10, phash=(1 << 100) - 1)] + + assert _family(seed, pool) == [] + + +def test_a_family_is_one_hop_from_the_seed(): + """Chaining is what lets a family drift: `alpha` reaches an image that also + carries `beta`, and `beta` must not then reach its own family.""" + seed = [_c(1, "alpha_final")] + pool = [_c(2, "alpha_beta", days=-3), _c(3, "beta_wip1", days=-4)] + + assert [c.image_id for c in _family(seed, pool)] == [2] + + +def test_the_seed_never_comes_back_as_its_own_family(): + seed = [_c(1, "Year_20k_Cndm")] + + assert _family(seed, seed + [_c(2, "Year_20k_Base", days=-20)])[0].image_id == 2 + assert len(_family(seed, seed)) == 0 + + +# --- the card, end to end ---------------------------------------------------- + + +async def _channels(db, name): + artist = Artist(name=name, slug=name) + db.add(artist) + await db.flush() + patreon = Source(artist_id=artist.id, platform="patreon", + url=f"https://patreon.com/{name}", enabled=True) + discord = Source(artist_id=artist.id, platform="discord", + url=f"https://discord.com/channels/1/{name}", enabled=True) + db.add_all([patreon, discord]) + await db.flush() + return artist, patreon, discord + + +_seq = iter(range(1, 10_000)) + + +async def _post(db, artist, source, *, at, names, body=None, title=None, synthetic=False): + post = Post( + source_id=source.id, artist_id=artist.id, + external_post_id=f"ext-{next(_seq)}", post_date=at, + post_title=title, description=body, + synthesized_by=DROP_GROUPER if synthetic else None, + ) + db.add(post) + await db.flush() + for name in names: + n = next(_seq) + db.add(ImageRecord( + # Discord-shaped whatever the platform: the prefix is stripped, so + # the NAME below is the leading name. + path=f"/images/{artist.slug}/20260901_1234567890{n:06d}_01_{name}.png", + sha256=f"{n:064d}", size_bytes=10, mime="image/png", + width=10, height=10, origin="downloaded", + primary_post_id=post.id, artist_id=artist.id, + )) + await db.flush() + return post + + +async def _link(db, teaser, drop, *, status="linked", linked_by="fc"): + a = PostAssociation( + announcement_post_id=teaser.id, payload_post_id=drop.id, score=1.0, + signals={"identity": 1.0, "identity_token": "year"}, + status=status, linked_by=linked_by if status == "linked" else None, + ) + db.add(a) + await db.flush() + return a + + +async def _feed(db, artist): + page = await PostFeedService(db).scroll(artist_id=artist.id, limit=100) + return {item["id"]: item for item in page["items"]} + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_a_teaser_shows_the_drop_it_announced(db): + """The complaint itself: the card now carries the drop's images and text.""" + artist, patreon, discord = await _channels(db, "unifyartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"], + title="Year 20k", body="Full set in the server") + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm", "Year_20k_Cndm_alt"], + body="@everyone 🍈🍈 the full set", synthetic=True) + assoc = await _link(db, teaser, drop) + await db.commit() + + unified = (await _feed(db, artist))[teaser.id]["unified"] + + assert [t["role"] for t in unified["thumbnails"]] == ["drop", "drop"] + assert {t["post_id"] for t in unified["thumbnails"]} == {drop.id} + assert unified["links"] == [{ + "association_id": assoc.id, "post_id": drop.id, + "linked_by": "fc", "token": "year", + }] + assert unified["texts"][0]["text"] == "@everyone 🍈🍈 the full set" + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_a_drop_beside_its_teaser_leaves_the_feed(db): + """*"only hidden from the post view they're posted the same day"*.""" + artist, patreon, discord = await _channels(db, "foldartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=2), + names=["Year_20k_Cndm"], synthetic=True) + await _link(db, teaser, drop) + await db.commit() + + feed = await _feed(db, artist) + + assert teaser.id in feed + assert drop.id not in feed + # Left out of the FEED, not out of existence: reachable by id, as every + # post is, because it is still the images' true origin. + assert (await PostFeedService(db).get_post(drop.id))["id"] == drop.id + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_a_drop_days_from_its_teaser_keeps_its_place(db): + """A reference does not take anything out of history.""" + artist, patreon, discord = await _channels(db, "keepartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(days=3), + names=["Year_20k_Cndm"], synthetic=True) + await _link(db, teaser, drop) + await db.commit() + + feed = await _feed(db, artist) + + assert drop.id in feed + assert feed[teaser.id]["unified"]["thumbnails"][0]["post_id"] == drop.id + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_a_proposal_changes_nothing_on_the_card(db): + """Only a LINKED pair unifies. A pending proposal is a question for the + review queue, and rendering it would assert a link nobody made.""" + artist, patreon, discord = await _channels(db, "pendingartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + await _link(db, teaser, drop, status="pending") + await db.commit() + + feed = await _feed(db, artist) + + assert feed[teaser.id]["unified"] is None + assert drop.id in feed + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_the_older_variants_come_along_and_nothing_else(db): + """The family reaches back 44 days for the wips — and not 90 days for a + namesake, and not at all for a message that only shares a trailing word.""" + artist, patreon, discord = await _channels(db, "familyartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + wip = await _post(db, artist, discord, at=T0 - timedelta(days=44), + names=["Year_20k_wip1"], body="wip, feedback welcome") + base = await _post(db, artist, discord, at=T0 - timedelta(days=20), + names=["Year_20K_Base"]) + too_old = await _post(db, artist, discord, at=T0 - timedelta(days=90), + names=["Year_20k_old"]) + trailing = await _post(db, artist, discord, at=T0 - timedelta(days=5), + names=["Other_piece_year"]) + await _link(db, teaser, drop) + await db.commit() + + feed = await _feed(db, artist) + unified = feed[teaser.id]["unified"] + variants = [t["post_id"] for t in unified["thumbnails"] if t["role"] == "variant"] + + assert variants == [wip.id, base.id] + assert unified["variant_count"] == 2 + assert too_old.id not in variants and trailing.id not in variants + assert "wip, feedback welcome" in [t["text"] for t in unified["texts"]] + # Referenced, not moved: every variant is still in the feed on its own. + assert {wip.id, base.id} <= set(feed) + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_the_family_window_is_the_operators_setting(db): + artist, patreon, discord = await _channels(db, "windowartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + await _post(db, artist, discord, at=T0 - timedelta(days=20), names=["Year_20k_wip1"]) + await _link(db, teaser, drop) + settings = await db.get(ImportSettings, 1) + settings.discord_family_window_days = 10 + await db.commit() + + unified = (await _feed(db, artist))[teaser.id]["unified"] + + assert unified["variant_count"] == 0 + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_undo_is_a_dismissal_and_everything_returns(db): + """The operator chose *"nest automatically, with visible undo"*. The undo + is the review queue's own dismiss: the drop comes back to the feed, the + card loses its references, and the dismissed row is what stops the next + sweep linking the pair straight back.""" + artist, patreon, discord = await _channels(db, "undoartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + assoc = await _link(db, teaser, drop) + await db.commit() + + await PostAssociationService(db).dismiss(assoc.id) + await db.commit() + feed = await _feed(db, artist) + + assert feed[teaser.id]["unified"] is None + assert drop.id in feed + row = (await db.execute(select(PostAssociation))).scalar_one() + assert (row.status, row.linked_by) == ("dismissed", None) + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_an_operator_accept_is_recorded_as_theirs(db): + """So the card does not claim FC made a link a person made.""" + artist, patreon, discord = await _channels(db, "acceptartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + assoc = await _link(db, teaser, drop, status="pending") + await db.commit() + + await PostAssociationService(db).accept(assoc.id) + await db.commit() + + unified = (await _feed(db, artist))[teaser.id]["unified"] + assert unified["links"][0]["linked_by"] == "operator" + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_a_fold_window_of_zero_hides_nothing(db): + artist, patreon, discord = await _channels(db, "nofoldartist") + teaser = await _post(db, artist, patreon, at=T0, names=["Year_20k_teaser"]) + drop = await _post(db, artist, discord, at=T0 - timedelta(hours=1), + names=["Year_20k_Cndm"], synthetic=True) + await _link(db, teaser, drop) + settings = await db.get(ImportSettings, 1) + settings.discord_link_fold_hours = 0 + await db.commit() + + assert drop.id in await _feed(db, artist) -- 2.54.0 From 7aa074d1a93ac961224c2700cc4bc17eeccf732c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 11:25:52 -0400 Subject: [PATCH 61/94] feat: the unified card shows referenced images, their text, and an undo (4402) The linked drop's images and the piece's variants join the teaser's rail, after its own images, each with a corner badge. The meta line counts them apart ("+N from Discord"), so a Patreon card never reads as though the creator posted Discord's files there. The text of each referenced post sits under the teaser's body. If FC made the link, the card says so, names the matching working name and offers Undo. If a person accepted it, it offers Unlink and makes no claim about FC. Both dismiss the pair. The kept row then stops the next sweep from linking it straight back. The announcement settings card gets the two windows: how close a drop must be before its own card is left out of the feed, and how far to reach for variants. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- frontend/src/components/posts/PostCard.vue | 152 +++++++++++++++++- .../settings/PostAssociationsCard.vue | 45 ++++++ frontend/src/stores/postAssociations.js | 20 ++- frontend/src/stores/posts.js | 31 +++- frontend/test/components/postCard.spec.js | 89 ++++++++++ 5 files changed, 328 insertions(+), 9 deletions(-) diff --git a/frontend/src/components/posts/PostCard.vue b/frontend/src/components/posts/PostCard.vue index 8a8cfdc..2d7be9f 100644 --- a/frontend/src/components/posts/PostCard.vue +++ b/frontend/src/components/posts/PostCard.vue @@ -30,6 +30,12 @@ · {{ totalImages }} image{{ totalImages === 1 ? '' : 's' }} + + + · +{{ refImages.length }} from Discord + @@ -66,9 +72,16 @@ + +
+
+ Discord · {{ shortDate(t.date) }}{{ t.role === 'variant' ? ' · variant' : '' }} +
+

{{ t.text }}

+
+ +
@@ -191,6 +235,7 @@ import { RouterLink } from 'vue-router' import { useModalStore } from '../../stores/modal.js' import { usePostsStore } from '../../stores/posts.js' import { toPlainText } from '../../utils/htmlSanitize.js' +import { toast } from '../../utils/toast.js' import PostSeriesMenu from './PostSeriesMenu.vue' import PostTranslationControl from './PostTranslationControl.vue' @@ -207,8 +252,16 @@ const modal = useModalStore() const detail = ref(null) const attachments = computed(() => props.post.attachments || []) -const images = computed(() => props.post.thumbnails || []) -const totalImages = computed(() => images.value.length + (props.post.thumbnails_more || 0)) +// #4402. A teaser's card also shows what it points at: the linked drop's +// images and the piece's other variants, each tagged with the post it really +// belongs to. They follow the post's own images so the teaser keeps its hero, +// and the post's own capped list stays a PREFIX of everything shown — which is +// what lets the "+N" tile and the modal playlist keep indexing correctly. +const unified = computed(() => props.post.unified || null) +const ownImages = computed(() => props.post.thumbnails || []) +const refImages = computed(() => unified.value?.thumbnails || []) +const images = computed(() => [...ownImages.value, ...refImages.value]) +const totalImages = computed(() => ownImages.value.length + (props.post.thumbnails_more || 0)) const plainTitle = computed(() => toPlainText(props.post.post_title)) // #388 E2. Non-null `synthesized_by` means FC authored this row by grouping a @@ -273,6 +326,44 @@ const grewAt = computed(() => (synthesized.value ? props.post.last_grew_at : nul // never beside the artwork. Defaults to [] so a post dict from before the // feature (or composed by hand) renders without a link rather than throwing. const associations = computed(() => props.post.associations || []) +// The teaser side of a link is drawn by the unified block when there is one; +// only a link it does not cover (the drop's "announced by", or a feed payload +// from before #4402) keeps the plain text link. +const plainLinks = computed(() => + unified.value + ? associations.value.filter((a) => a.role !== 'announces') + : associations.value, +) + +function linkLabel (l) { + if (l.linked_by === 'fc') { + return l.token + ? `Linked by FabledCurator — matched on “${l.token}”` + : 'Linked by FabledCurator' + } + return 'Linked to its Discord drop' +} + +function refTitle (t) { + return t.role === 'variant' ? 'a variant from Discord' : 'from the Discord drop' +} + +function shortDate (iso) { + return new Date(iso).toLocaleDateString() +} + +const unlinking = ref(null) +async function undoLink (l) { + unlinking.value = l.association_id + try { + await postsStore.unlink(props.post.id, l.association_id) + toast({ text: 'Unlinked — the Discord drop returns to the feed on the next load', type: 'success' }) + } catch (e) { + toast({ text: `Unlink failed: ${e.message}`, type: 'error' }) + } finally { + unlinking.value = null + } +} const grewRelative = computed(() => (grewAt.value ? relativeFrom(grewAt.value) : '')) const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString()) function relativeFrom (iso) { @@ -298,7 +389,8 @@ async function fullImageIds () { detail.value = await postsStore.getPostFull(props.post.id) } catch { /* fall back to the capped feed list */ } } - return (detail.value?.thumbnails || images.value).map((t) => t.image_id) + const own = detail.value?.thumbnails || ownImages.value + return [...own, ...refImages.value].map((t) => t.image_id) } async function openModal (imageId) { @@ -447,6 +539,54 @@ function formatBytes (n) { .fc-post-card__grew { color: rgb(var(--v-theme-accent)); } .fc-post-card__assoc { margin-top: 8px; } + +/* #4402 — the unified block. Quiet by design: it explains the references in + the rail, it does not compete with them. */ +.fc-post-card__unified { + margin-top: 10px; + padding-left: 10px; + border-left: 2px solid rgba(var(--v-theme-accent), 0.5); +} +.fc-post-card__unified-head { + display: flex; align-items: center; flex-wrap: wrap; gap: 6px; + font-size: 0.8125rem; + color: rgb(var(--v-theme-accent)); +} +.fc-post-card__undo { + padding: 0; border: 0; background: none; cursor: pointer; + font-size: 0.75rem; font-weight: 600; + color: rgb(var(--v-theme-on-surface-variant)); +} +.fc-post-card__undo:hover { color: rgb(var(--v-theme-accent)); text-decoration: underline; } +.fc-post-card__undo:disabled { cursor: default; text-decoration: none; } +.fc-post-card__unified-text { margin-top: 6px; } +.fc-post-card__unified-meta { + font-size: 0.7rem; font-weight: 600; text-transform: uppercase; + letter-spacing: 0.04em; + color: rgb(var(--v-theme-on-surface-variant)); +} +.fc-post-card__unified-body { + margin: 2px 0 0; + font-size: 0.85rem; line-height: 1.45; + white-space: pre-wrap; + color: rgb(var(--v-theme-on-surface)); + display: -webkit-box; + -webkit-line-clamp: 3; + -webkit-box-orient: vertical; + overflow: hidden; +} +.fc-post-card__ref-meta { color: rgb(var(--v-theme-accent)); } + +/* A referenced tile carries a corner badge: the images are Discord's, shown + here, and the card must not pass them off as the teaser's own. */ +.fc-post-card__rail-cell { position: relative; } +.fc-post-card__rail-cell--ref { outline: 1px solid rgba(var(--v-theme-accent), 0.55); outline-offset: -1px; } +.fc-post-card__ref-badge { + position: absolute; top: 4px; right: 4px; + padding: 2px; border-radius: 4px; + background: rgba(var(--v-theme-surface), 0.85); + color: rgb(var(--v-theme-accent)); +} .fc-post-card__assoc-link { display: inline-flex; align-items: center; diff --git a/frontend/src/components/settings/PostAssociationsCard.vue b/frontend/src/components/settings/PostAssociationsCard.vue index c267fde..0e9f996 100644 --- a/frontend/src/components/settings/PostAssociationsCard.vue +++ b/frontend/src/components/settings/PostAssociationsCard.vue @@ -61,6 +61,47 @@ + +
On the teaser's card
+
+ A linked teaser shows the Discord drop it announced — its images and its + message — and the other versions of the same piece: the wips, alts and + censor passes posted under the same working name. They stay where they + landed in the feed; the card only shows them together. +
+ + + +
+ A drop this close to its teaser is the same release shown twice, so + only the teaser's card stays in the feed. A drop further away keeps + its own card. 0 hides nothing. +
+ + + +
+ How far either side of the teaser to look for the rest of the piece. + Measured on real drops: a piece's versions span up to about six + weeks, while unrelated pieces that happen to share a name are years + apart. 0 shows the drop alone. +
+ + +
{{ store.proposals.length }} waiting for review @@ -130,11 +171,15 @@ const enabled = ref(true) const auto = ref(true) const threshold = ref(0.6) const windowHours = ref(24) +const foldHours = ref(24) +const familyDays = ref(60) watch(() => store.enabled, (v) => { enabled.value = v }, { immediate: true }) watch(() => store.auto, (v) => { auto.value = v }, { immediate: true }) watch(() => store.threshold, (v) => { threshold.value = v }, { immediate: true }) watch(() => store.windowHours, (v) => { windowHours.value = v }, { immediate: true }) +watch(() => store.foldHours, (v) => { foldHours.value = v }, { immediate: true }) +watch(() => store.familyDays, (v) => { familyDays.value = v }, { immediate: true }) onMounted(async () => { // Both swallow their own failures: a settings read that fails should not diff --git a/frontend/src/stores/postAssociations.js b/frontend/src/stores/postAssociations.js index b98cc55..71cb451 100644 --- a/frontend/src/stores/postAssociations.js +++ b/frontend/src/stores/postAssociations.js @@ -20,6 +20,9 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { const auto = ref(true) const threshold = ref(0.6) const windowHours = ref(24) + // #4402 — what a link shows once made. See PostAssociationsCard. + const foldHours = ref(24) + const familyDays = ref(60) const { loading, error, run } = useAsyncAction({ errorAs: 'message' }) async function load () { @@ -35,6 +38,8 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { auto.value = s.discord_link_auto threshold.value = s.discord_link_threshold windowHours.value = s.discord_link_window_hours + foldHours.value = s.discord_link_fold_hours + familyDays.value = s.discord_family_window_days } async function saveSettings (patch) { @@ -61,6 +66,16 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { await saveSettings({ discord_link_window_hours: v }) } + async function setFoldHours (v) { + foldHours.value = v + await saveSettings({ discord_link_fold_hours: v }) + } + + async function setFamilyDays (v) { + familyDays.value = v + await saveSettings({ discord_family_window_days: v }) + } + async function accept (id) { try { await api.post(`/api/posts/associations/${id}/accept`, {}) @@ -86,8 +101,9 @@ export const usePostAssociationsStore = defineStore('postAssociations', () => { } return { - proposals, enabled, auto, threshold, windowHours, loading, error, + proposals, enabled, auto, threshold, windowHours, foldHours, familyDays, + loading, error, load, loadSettings, setEnabled, setAuto, setThreshold, setWindowHours, - accept, dismiss, rescan + setFoldHours, setFamilyDays, accept, dismiss, rescan } }) diff --git a/frontend/src/stores/posts.js b/frontend/src/stores/posts.js index 5c7ff2f..4fd3d90 100644 --- a/frontend/src/stores/posts.js +++ b/frontend/src/stores/posts.js @@ -89,6 +89,35 @@ export const usePostsStore = defineStore('posts', () => { return res } + // Undo a link on a unified card (#4402). The operator chose "nest + // automatically, with visible undo" — so the undo is the review queue's own + // dismiss, whose kept row is what stops the next sweep linking the pair + // straight back. Patches the loaded teaser in place so the card drops its + // references at once; the drop's own card is not in the loaded page (it was + // folded out), so it returns on the next load rather than being guessed into + // position here. + async function unlink(postId, associationId) { + await api.post(`/api/posts/associations/${associationId}/dismiss`, {}) + const item = items.value.find((p) => p.id === postId) + if (!item) return + item.associations = (item.associations || []).filter((a) => a.id !== associationId) + const unified = item.unified + if (!unified) return + const gone = unified.links.find((l) => l.association_id === associationId) + const links = unified.links.filter((l) => l.association_id !== associationId) + // Variants are keyed on the whole seed, so with one link left they may no + // longer all belong; with none left there is nothing to show at all. Drop + // what is certainly gone and let the next load recompute the rest. + item.unified = links.length + ? { + ...unified, + links, + thumbnails: unified.thumbnails.filter((t) => t.post_id !== gone?.post_id), + texts: unified.texts.filter((t) => t.post_id !== gone?.post_id), + } + : null + } + // Filter overlay for the around/older/newer (in-context anchored) // path. Keep this distinct from `filters.value` (the down-only feed) // so a normal-feed filter change doesn't leak into an active anchored @@ -186,7 +215,7 @@ export const usePostsStore = defineStore('posts', () => { return { items, cursor, loading, done, error, filters, cursorOlder, cursorNewer, doneOlder, doneNewer, anchorId, - loadInitial, loadMore, getPostFull, applyTranslationOverride, + loadInitial, loadMore, getPostFull, applyTranslationOverride, unlink, loadAround, loadOlder, loadNewer, } }) diff --git a/frontend/test/components/postCard.spec.js b/frontend/test/components/postCard.spec.js index 9324d07..b96f779 100644 --- a/frontend/test/components/postCard.spec.js +++ b/frontend/test/components/postCard.spec.js @@ -4,6 +4,7 @@ import { flushPromises } from '@vue/test-utils' import PostCard from '../../src/components/posts/PostCard.vue' import { useModalStore } from '../../src/stores/modal.js' +import { usePostsStore } from '../../src/stores/posts.js' import { freshPinia, mountComponent } from '../support/mountComponent.js' const now = new Date().toISOString() @@ -201,3 +202,91 @@ describe('PostCard', () => { expect(openSpy).toHaveBeenCalledWith(10, { playlistIds: [10, 11] }) }) }) + +// #4402. A teaser is a pointer, so its card shows what it points at — by +// REFERENCE. These pin the honesty half as hard as the feature half: the +// referenced images are marked as Discord's, and a link FC made by itself says +// so and offers the undo the operator chose over a silent merge. +describe('the unified card', () => { + const UNIFIED = { + links: [{ association_id: 7, post_id: 42, linked_by: 'fc', token: '0-k' }], + thumbnails: [ + { image_id: 200, thumbnail_url: '/d0', post_id: 42, role: 'drop' }, + { image_id: 201, thumbnail_url: '/v0', post_id: 40, role: 'variant' }, + ], + variant_count: 1, + texts: [ + { post_id: 42, role: 'drop', date: now, text: '@everyone the full set' }, + { post_id: 40, role: 'variant', date: now, text: 'wip, feedback welcome' }, + ], + } + const TEASER = { + ...BASE, + description_plain: 'Full set in the server', + thumbnails: [{ image_id: 10, thumbnail_url: '/a' }], + associations: [{ id: 7, role: 'announces', post_id: 42 }], + unified: UNIFIED, + } + + it('shows the drop and its variants beside the teaser, marked as references', () => { + const w = mountComponent(PostCard, { props: { post: TEASER }, pinia: freshPinia() }) + const refs = w.findAll('.fc-post-card__rail-cell--ref') + expect(refs).toHaveLength(2) + // Counted apart from the teaser's own, so the card never reads as though + // the creator put Discord's files on Patreon. + expect(w.text()).toContain('1 image') + expect(w.text()).toContain('+2 from Discord') + }) + + it('carries the text of everything it unifies', () => { + const w = mountComponent(PostCard, { props: { post: TEASER }, pinia: freshPinia() }) + expect(w.text()).toContain('@everyone the full set') + expect(w.text()).toContain('wip, feedback welcome') + }) + + it('says FC made the link, on what, and offers an undo', () => { + const w = mountComponent(PostCard, { props: { post: TEASER }, pinia: freshPinia() }) + expect(w.text()).toContain('Linked by FabledCurator') + expect(w.text()).toContain('0-k') + expect(w.find('.fc-post-card__undo').text()).toBe('Undo') + // The old text link would say the same thing twice. + expect(w.text()).not.toContain('The full set is in Discord') + }) + + it('never claims FC made a link a person accepted', () => { + const post = { + ...TEASER, + unified: { ...UNIFIED, links: [{ ...UNIFIED.links[0], linked_by: 'operator' }] }, + } + const w = mountComponent(PostCard, { props: { post }, pinia: freshPinia() }) + expect(w.text()).not.toContain('Linked by FabledCurator') + expect(w.find('.fc-post-card__undo').text()).toBe('Unlink') + }) + + it('undo dismisses that one link', async () => { + const pinia = freshPinia() + const store = usePostsStore() + const spy = vi.spyOn(store, 'unlink').mockResolvedValue() + const w = mountComponent(PostCard, { props: { post: TEASER }, pinia }) + await w.find('.fc-post-card__undo').trigger('click') + await flushPromises() + expect(spy).toHaveBeenCalledWith(1, 7) + }) + + it('arrows through the teaser first, then what it references', async () => { + const pinia = freshPinia() + const openSpy = vi.spyOn(useModalStore(), 'open').mockResolvedValue() + const w = mountComponent(PostCard, { props: { post: TEASER }, pinia }) + await w.find('.fc-post-card__hero').trigger('click') + await flushPromises() + expect(openSpy).toHaveBeenCalledWith(10, { playlistIds: [10, 200, 201] }) + }) + + it('an ordinary post is untouched', () => { + const w = mountComponent(PostCard, { + props: { post: { ...BASE, unified: null } }, pinia: freshPinia(), + }) + expect(w.find('.fc-post-card__unified').exists()).toBe(false) + expect(w.text()).not.toContain('from Discord') + }) +}) -- 2.54.0 From e08401c44ab7e143d9d05268aac5c42a9ed0854d Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 13:47:25 -0400 Subject: [PATCH 62/94] feat: a creator's trickle becomes one drop instead of a card per message (4390) The operator, on a feed of "Grouped from 1 Discord message" cards: "the groups are still single image even when they can clearly be seen as group". 665 of Yellowroom's 714 drops were one message. Both join paths demand cosine <= 0.10 to a drop's FIRST image. The stages of one piece fail that: each is nearest the one before, not the first. `svtt_wip4` never joined `svtt_wip3` from the day before. Measured on artist 8 before writing it: - phash cannot see it. Stages sit 68-134 bits apart; unrelated same-artist pairs have a median of 126 and a p5 of 110 (lesson #4400). - The embedding's nearest neighbour can. Every stage of three real trickles had a sibling as its single nearest image in the artist's library. In a control over all 137 recent Discord images, a nearest neighbour that was another message within 7 days carried the same working name 53 times out of 53. Mismatches start past 7 days. A new merge pass runs last in the sweep. A later drop folds into an earlier one within discord_group_close_after_hours (168h, the measured 7 days) when they share a gated leading working name, or when one's image is the other's nearest neighbour. A drop reaching several earlier drops pulls them all together, unless two of them are named as different pieces. A merge carries teaser links across (the payload FK would otherwise cascade them away). Growth is stamped at the messages' own time, so merging history never jumps an old drop to the top of the feed. Each drop records the route it merged by, and is checked once. Offline replay over Yellowroom's 127 drops since 2025: they become 70 posts. The Marin trickle ("Very early Marin" -> 3 screenshots -> MarinaraSauce_base) becomes one post of 5 by nearest neighbour. svtt, 0-k1, cnni14 and 0adm come together by name. FAMILY_MAX_POSTS moves to post_naming, so the grouper and the teaser card share one definition. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/discord_grouping.py | 367 ++++++++++++++++++++++- backend/app/services/post_naming.py | 13 + backend/app/services/post_unification.py | 12 +- backend/app/tasks/maintenance.py | 2 +- tests/test_discord_grouping.py | 1 + tests/test_discord_trickles.py | 253 ++++++++++++++++ tests/test_post_unification.py | 7 +- 7 files changed, 633 insertions(+), 22 deletions(-) create mode 100644 tests/test_discord_trickles.py diff --git a/backend/app/services/discord_grouping.py b/backend/app/services/discord_grouping.py index f0f2b6a..44afda3 100644 --- a/backend/app/services/discord_grouping.py +++ b/backend/app/services/discord_grouping.py @@ -59,14 +59,23 @@ from __future__ import annotations import logging import math +from collections import Counter from dataclasses import dataclass, field from datetime import UTC, datetime, timedelta -from sqlalchemy import Select, func, select, update +from sqlalchemy import Select, delete, func, select, update from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession -from ..models import ImageProvenance, ImageRecord, MLSettings, Post, Source +from ..models import ( + ImageProvenance, + ImageRecord, + MLSettings, + Post, + PostAssociation, + Source, +) +from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies log = logging.getLogger(__name__) @@ -600,6 +609,344 @@ async def join_open_groups( return joined +# --------------------------------------------------------------------------- +# #4390: a trickle is one drop — later drops of the same piece merge in. +# --------------------------------------------------------------------------- +# +# Operator, 2026-09-24, on a feed of "Grouped from 1 Discord message" cards: +# *"the groups are still single image even when they can clearly be seen as +# group"*. 665 of Yellowroom's 714 drops were one message. +# +# Both paths above join on cosine distance to the group's SEED, and the stages +# of one piece fail it: `svtt_wip4` did not join `svtt_wip3` from the day +# before. A creator trickles a piece out as sketch -> wip -> wip -> release, and +# each stage is nearest to the one before it, not to the first. +# +# Measured on artist 8 before any of this was written (#4390 log): +# +# * phash cannot see it. Stages of one piece sit 68-134 bits apart; unrelated +# pieces by the same artist sit at a median of 126, p5 110. Lesson #4400. +# * The embedding's NEAREST neighbour can. Every stage of three real trickles +# had a sibling stage as its single nearest image in the artist's whole +# library, while siblings further along ranked 40-100 — which is exactly why +# seed distance fails. Negative control over all 137 recent Discord images: +# where the nearest neighbour was another message within 7 days, the two +# carried the same working name 53 times out of 53. Disagreements start past +# 7 days. +# * The working name sees it directly, when there is one. +# +# So a later drop merges into an earlier one when the two are within +# `discord_group_close_after_hours` of each other (168h — the measured 7 days) +# AND either they share a gated LEADING working name, or one's image has the +# other's image as its nearest neighbour. A drop reaching SEVERAL earlier drops +# pulls them all together — unless two of them are named as different pieces, +# in which case nothing moves (see `_compatible`): leaving a drop alone is +# recoverable, a wrong merge asserts that unrelated art belongs together. +# +# Chaining is permitted here and was forbidden above, deliberately. The seed +# rule exists because tiny steps can drift from one piece to another; the +# measured precision of nearest-neighbour inside 7 days is what bounds drift +# for this route, and each link is between neighbours in time, never across a +# quiet week. + +# How many unchecked drops one sweep examines per source. A first run over an +# established library drains over successive sweeps, oldest first, rather than +# issuing one nearest-neighbour query per image of the whole history at once. +TRICKLE_BATCH = 300 + + +@dataclass +class _Drop: + post: Post + members: set[int] + first_at: datetime + last_at: datetime + names: set[str] + images: list[int] + nearest: set[int] | None + + +async def _nearest_message( + session: AsyncSession, *, artist_id: int, image_id: int, exclude: set[int], +) -> int | None: + """The post that owns the nearest image in the artist's whole library. + + The whole LIBRARY, not this source, because that is what was measured: a + neighbour that turns out to be a Patreon re-post simply yields no Discord + drop to merge into, which errs toward leaving things alone. `exclude` is + the drop's own messages — an image is always nearest to its own siblings + in the same drop, which says nothing. + """ + embedding = (await session.execute( + select(ImageRecord.siglip_embedding).where(ImageRecord.id == image_id) + )).scalar_one_or_none() + if embedding is None: + return None + stmt = ( + select(ImageRecord.primary_post_id) + .where( + ImageRecord.artist_id == artist_id, + ImageRecord.id != image_id, + ImageRecord.siglip_embedding.is_not(None), + ImageRecord.primary_post_id.is_not(None), + ) + .order_by(ImageRecord.siglip_embedding.cosine_distance(embedding)) + .limit(1) + ) + if exclude: + stmt = stmt.where(ImageRecord.primary_post_id.not_in(exclude)) + return (await session.execute(stmt)).scalar_one_or_none() + + +async def _load_drops(session: AsyncSession, source: Source) -> list[_Drop]: + """Every live drop of this source, with what the merge rule reads, oldest first.""" + posts = (await session.execute( + select(Post).where( + Post.source_id == source.id, + Post.synthesized_by == DROP_GROUPER, + Post.absorbed_by_post_id.is_(None), + ) + )).scalars().all() + if not posts: + return [] + by_id = {p.id: p for p in posts} + + msg_at = func.coalesce(Post.post_date, Post.downloaded_at) + members: dict[int, set[int]] = {pid: set() for pid in by_id} + times: dict[int, list[datetime]] = {pid: [] for pid in by_id} + for mid, owner, at in (await session.execute( + select(Post.id, Post.absorbed_by_post_id, msg_at) + .where(Post.absorbed_by_post_id.in_(list(by_id))) + )).all(): + members[owner].add(mid) + times[owner].append(at) + + owner_of = {m: d for d, ms in members.items() for m in ms} + names: dict[int, set[str]] = {pid: set() for pid in by_id} + images: dict[int, list[int]] = {pid: [] for pid in by_id} + if owner_of: + for iid, primary, path in (await session.execute( + select(ImageRecord.id, ImageRecord.primary_post_id, ImageRecord.path) + .where(ImageRecord.primary_post_id.in_(list(owner_of))) + .order_by(ImageRecord.id) + )).all(): + drop = owner_of[primary] + images[drop].append(iid) + if (name := leading_name(path)) is not None: + names[drop].add(name) + + out = [] + for pid, post in by_id.items(): + if not times[pid]: + continue + stored = (post.synthesis_details or {}).get("nearest_message_ids") + out.append(_Drop( + post=post, members=members[pid], + first_at=min(times[pid]), last_at=max(times[pid]), + names=names[pid], images=images[pid], + nearest=set(stored) if stored is not None else None, + )) + return sorted(out, key=lambda d: (d.first_at, d.post.id)) + + +async def _name_posts(session: AsyncSession, artist_id: int) -> Counter[str]: + """Post-span counts of the artist's working names — the same corpus the + teaser card and the announcement matcher count against.""" + by_post: dict[int, list[str]] = {} + for pid, path in (await session.execute( + select(ImageRecord.primary_post_id, ImageRecord.path).where( + ImageRecord.artist_id == artist_id, + ImageRecord.primary_post_id.is_not(None), + ) + )).all(): + by_post.setdefault(pid, []).append(path) + return token_frequencies(by_post.values()) + + +async def _repoint_associations( + session: AsyncSession, *, from_id: int, to_id: int, +) -> None: + """Move announcement links from a drop about to merge onto the one it joins. + + Without this the merge would silently undo a teaser link: the association's + payload FK cascades on delete. Where the teaser already points at the + surviving drop, the stronger claim is kept — a link over a proposal over a + dismissal — and the duplicate goes. + """ + rank = {"linked": 2, "pending": 1, "dismissed": 0} + moving = (await session.execute( + select(PostAssociation).where(PostAssociation.payload_post_id == from_id) + )).scalars().all() + for a in moving: + existing = (await session.execute( + select(PostAssociation).where( + PostAssociation.announcement_post_id == a.announcement_post_id, + PostAssociation.payload_post_id == to_id, + ) + )).scalar_one_or_none() + if existing is None: + a.payload_post_id = to_id + continue + if rank.get(a.status, 0) > rank.get(existing.status, 0): + existing.status = a.status + existing.linked_by = a.linked_by + await session.delete(a) + await session.flush() + + +async def merge_trickles( + session: AsyncSession, + source: Source, + *, + gap: timedelta, + min_images: int, + cooldown: timedelta, + batch: int = TRICKLE_BATCH, +) -> int: + """Fold later drops of the same piece into the earlier one. Returns merges.""" + drops = await _load_drops(session, source) + if len(drops) < 2: + return 0 + name_posts = await _name_posts(session, source.artist_id) + + def gated(names: set[str]) -> set[str]: + return {n for n in names if rarity(name_posts.get(n, 0), FAMILY_MAX_POSTS) > 0} + + alive: list[_Drop] = [] + merged = 0 + checked = 0 + for drop in drops: + details = drop.post.synthesis_details or {} + if details.get("trickle_checked"): + alive.append(drop) + continue + if checked >= batch: + # Unchecked and out of budget: still a candidate for LATER drops' + # reverse edges, just not examined itself this run. + alive.append(drop) + continue + checked += 1 + + if drop.nearest is None: + found: set[int] = set() + for iid in drop.images: + pid = await _nearest_message( + session, artist_id=source.artist_id, image_id=iid, + exclude=drop.members, + ) + if pid is not None: + found.add(pid) + drop.nearest = found + + mine = gated(drop.names) + targets: dict[int, tuple[_Drop, str]] = {} + for earlier in alive: + if drop.first_at - earlier.last_at > gap: + continue + shared = mine & gated(earlier.names) + if shared: + targets[earlier.post.id] = (earlier, f"name:{min(shared)}") + elif drop.nearest & earlier.members or (earlier.nearest or set()) & drop.members: + targets[earlier.post.id] = (earlier, "nearest") + + record = dict(details) + record["nearest_message_ids"] = sorted(drop.nearest) + record["trickle_checked"] = True + drop.post.synthesis_details = record + + if not targets or not _compatible( + [gated(t.names) for t, _route in targets.values()] + [mine] + ): + alive.append(drop) + continue + + # Every target is the same piece as this drop, so they are the same + # piece as each other: fold them all into the earliest, then this drop. + ordered = sorted(targets.values(), key=lambda tr: (tr[0].first_at, tr[0].post.id)) + into = ordered[0][0] + for other, route in ordered[1:]: + await _merge_drop( + session, into=into, drop=other, route=route, + min_images=min_images, cooldown=cooldown, + ) + alive.remove(other) + merged += 1 + await _merge_drop( + session, into=into, drop=drop, route=ordered[0][1], + min_images=min_images, cooldown=cooldown, + ) + merged += 1 + return merged + + +def _compatible(name_sets: list[set[str]]) -> bool: + """May drops carrying these working names become one post? + + Refused only when two of them are NAMED AS DIFFERENT PIECES — both carry a + gated name, and they share none. An unnamed drop (a canvas screenshot) + fits anywhere, which is the whole of the Marin case: two early stages both + nearest to the same later one are one trickle, not an ambiguity. + + What it does NOT refuse is a drop the creator made two pieces in + themselves. Measured on artist 8: one November message carries both + `AdL01_wip4` and `Year_20k_wip_z4`, so its drop holds both names, and a + later drop of either piece joins it on its own name. That is the creator's + co-posting carried forward — Discord shows those two together too — not a + bridge FC built. + """ + named = [n for n in name_sets if n] + return all(a & b for i, a in enumerate(named) for b in named[i + 1:]) + + +async def _merge_drop( + session: AsyncSession, + *, + into: _Drop, + drop: _Drop, + route: str, + min_images: int, + cooldown: timedelta, +) -> None: + """Absorb `drop`'s messages into `into`, carry its links over, delete it. + + Growth is stamped at the merged messages' OWN time, not the wall clock. + Merging history must not drag a two-year-old drop to the top of the feed, + and the time the group actually grew is when those messages arrived. + """ + await _repoint_associations(session, from_id=drop.post.id, to_id=into.post.id) + grew_before = into.post.last_grew_at + await _absorb_into( + session, group=into.post, member_ids=sorted(drop.members), + source_id=into.post.source_id, now=drop.last_at, + min_images=min_images, cooldown=cooldown, + ) + # Never backwards: a group that already grew later than these messages + # keeps that later date. + if grew_before is not None and grew_before > drop.last_at: + into.post.last_grew_at = grew_before + details = dict(into.post.synthesis_details or {}) + if grew_before is not None and grew_before > drop.last_at: + details["last_grew_at"] = grew_before.isoformat() + # The honesty rule, extended: a grouping FC invented says what it was + # built from, and a merge says WHY — "name:svtt" or "nearest". + details["merged"] = [ + *details.get("merged", []), + {"post_id": drop.post.id, "route": route, "message_ids": sorted(drop.members)}, + ] + details["nearest_message_ids"] = sorted((into.nearest or set()) | (drop.nearest or set())) + into.post.synthesis_details = details + + into.members |= drop.members + into.names |= drop.names + into.images += drop.images + into.nearest = (into.nearest or set()) | (drop.nearest or set()) + into.last_at = max(into.last_at, drop.last_at) + + await session.execute(delete(ImageProvenance).where(ImageProvenance.post_id == drop.post.id)) + await session.delete(drop.post) + await session.flush() + + async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict: """Group every enabled Discord source. No-op when the switch is off. @@ -612,6 +959,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict: if not settings.discord_grouping_enabled: return { "enabled": False, "sources": 0, "posts_created": 0, "images_joined": 0, + "drops_merged": 0, } sources = (await session.execute( @@ -626,6 +974,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict: created = 0 joined = 0 + merged = 0 for source in sources: joined += await join_open_groups( session, source, @@ -644,12 +993,20 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict: window_minutes=window_minutes, now=now, ) + # Last, so the drops the two passes above just wrote are merged in + # the same sweep rather than showing as singletons for an hour. + merged += await merge_trickles( + session, source, + gap=timedelta(hours=float(settings.discord_group_close_after_hours)), + min_images=int(settings.discord_group_resurface_min_images), + cooldown=timedelta(hours=float(settings.discord_group_resurface_cooldown_hours)), + ) log.info( "discord drop grouping: %d source(s), %d synthetic post(s) created, " - "%d image(s) joined to open groups", - len(sources), created, joined, + "%d image(s) joined to open groups, %d trickle drop(s) merged", + len(sources), created, joined, merged, ) return { "enabled": True, "sources": len(sources), - "posts_created": created, "images_joined": joined, + "posts_created": created, "images_joined": joined, "drops_merged": merged, } diff --git a/backend/app/services/post_naming.py b/backend/app/services/post_naming.py index d20ab52..42d4c28 100644 --- a/backend/app/services/post_naming.py +++ b/backend/app/services/post_naming.py @@ -165,6 +165,19 @@ MAX_TOKEN_POSTS = 6 # four-post band, where conto's `illustration9` and `maid` also sit. IDENTITY_FLOOR = 0.75 +# A LEADING name spanning this many of ONE ARTIST's posts is a habit — a +# character the creator returns to — not one piece's trickle. Used wherever a +# name gathers a FAMILY: the teaser card's variants (#4401) and the Discord +# grouper's trickle merge (#4390), so the two cannot disagree about what a +# family is. +# +# Its own value rather than MAX_TOKEN_POSTS (6), which is calibrated for +# PAIRING two posts and measured too tight for a family. On artist 8, +# `tentacooler` spans 6 posts over 7 days and `0-k1` 6 posts over 10: both real +# families, both gated out at 6. At 8, `anya` (7) passes the cap — and has no +# pair inside any family window, which is what the window is for. +FAMILY_MAX_POSTS = 8 + def _strip_prefixes(stem: str) -> str: """Remove the framing each platform's importer adds around the real name.""" diff --git a/backend/app/services/post_unification.py b/backend/app/services/post_unification.py index cc1b902..60edf82 100644 --- a/backend/app/services/post_unification.py +++ b/backend/app/services/post_unification.py @@ -58,17 +58,7 @@ from ..utils.text import html_to_plain, truncate_at_word from .discord_grouping import PLATFORM as DISCORD from .gallery_service import thumbnail_url from .post_association_service import DUPLICATE_MAX_DISTANCE -from .post_naming import leading_name, rarity, token_frequencies - -# A leading name spanning this many of ONE ARTIST's posts is a habit — a -# character the creator returns to — not one piece's trickle. -# -# Its own value rather than post_naming.MAX_TOKEN_POSTS (6), which is -# calibrated for PAIRING two posts, and measured too tight for a family. On -# artist 8, `tentacooler` spans 6 posts over 7 days and `0-k1` 6 posts over 10: -# both real families, both gated out at 6. At 8, `anya` (7) passes the cap — -# and has no pair inside the family window, which is what the window is for. -FAMILY_MAX_POSTS = 8 +from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies # The text each referenced post contributes to the card, per post. The card # clamps it again; this keeps a long Discord thread from making the feed diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index d3993be..8e41efd 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -1175,7 +1175,7 @@ def group_discord_drops() -> str: return "disabled" return ( f"sources={res['sources']} created={res['posts_created']} " - f"joined={res['images_joined']}" + f"joined={res['images_joined']} merged={res['drops_merged']}" ) diff --git a/tests/test_discord_grouping.py b/tests/test_discord_grouping.py index 928c197..2a3ebf9 100644 --- a/tests/test_discord_grouping.py +++ b/tests/test_discord_grouping.py @@ -374,6 +374,7 @@ async def test_the_sweep_is_a_no_op_when_the_switch_is_off(db): # `images_joined` broke this assertion, which is exactly what it is for. assert result == { "enabled": False, "sources": 0, "posts_created": 0, "images_joined": 0, + "drops_merged": 0, } diff --git a/tests/test_discord_trickles.py b/tests/test_discord_trickles.py new file mode 100644 index 0000000..76b47ad --- /dev/null +++ b/tests/test_discord_trickles.py @@ -0,0 +1,253 @@ +"""#4390: a creator's trickle is one drop, not a card per message. + +Operator, 2026-09-24, on a feed of "Grouped from 1 Discord message" cards: +*"the groups are still single image even when they can clearly be seen as +group"*. 665 of Yellowroom's 714 drops were one message, because both join +paths demand closeness to a drop's FIRST image and a piece's stages drift away +from it — each is nearest the one before, not the first. + +The merge pass joins a later drop to an earlier one within 7 days when they +share a working name, or when one's image is the other's nearest neighbour in +the artist's library. Vectors here are built at stated angles (see +test_discord_grouping._vec) so each test says exactly which image is nearest +to which, rather than hoping. +""" +import math +from datetime import UTC, datetime, timedelta + +import pytest +from sqlalchemy import select + +from backend.app.models import ( + Artist, + ImageRecord, + Post, + PostAssociation, + Source, +) +from backend.app.services.discord_grouping import ( + DROP_GROUPER, + _compatible, + group_source, + merge_trickles, +) + +pytestmark = pytest.mark.integration + +DIM = 1152 +GAP = timedelta(hours=168) +T0 = datetime.now(UTC) - timedelta(days=30) + + +def _vec(angle: float) -> list[float]: + v = [0.0] * DIM + v[0] = math.cos(angle) + v[1] = math.sin(angle) + return v + + +# --- the compatibility rule, pure -------------------------------------------- + + +def test_unnamed_stages_fit_with_anything(): + """The Marin case: canvas screenshots carry no name, and two early stages + both nearest to one later stage are one trickle, not an ambiguity.""" + assert _compatible([set(), set(), set()]) + assert _compatible([set(), {"svtt"}]) + + +def test_two_differently_named_pieces_never_meet(): + assert not _compatible([{"alpha"}, {"beta"}, set()]) + + +def test_pieces_sharing_a_name_are_one_piece(): + assert _compatible([{"year", "20k"}, {"year"}]) + + +# --- end to end -------------------------------------------------------------- + + +async def _seed(db, name): + artist = Artist(name=name, slug=name) + db.add(artist) + await db.flush() + source = Source(artist_id=artist.id, platform="discord", + url=f"https://discord.com/channels/1/{name}", enabled=True) + db.add(source) + await db.flush() + return artist, source + + +_n = iter(range(1, 100_000)) + + +async def _message(db, artist, source, *, at, angle, name=None, text=None): + n = next(_n) + post = Post(source_id=source.id, artist_id=artist.id, + external_post_id=f"msg-{n}", post_date=at, description=text) + db.add(post) + await db.flush() + # gallery-dl's Discord shape, so the prefix strips and NAME is the working + # name. Unnamed messages are canvas screenshots, which carry none. + stem = name or f"Screenshot_2026-09-08_{n:06d}" + db.add(ImageRecord( + path=f"/images/{artist.slug}/20260901_{1234567890000 + n}_01_{stem}.png", + sha256=f"{n:064d}", size_bytes=10, mime="image/png", width=10, height=10, + origin="downloaded", primary_post_id=post.id, artist_id=artist.id, + siglip_embedding=_vec(angle), + )) + await db.flush() + return post + + +async def _group_then_merge(db, source): + """E2 makes the singletons exactly as it does live; then the merge pass.""" + await group_source(db, source, max_distance=0.10, window_minutes=60) + merged = await merge_trickles( + db, source, gap=GAP, min_images=2, cooldown=timedelta(hours=24), + ) + await db.commit() + return merged + + +async def _drops(db, source): + return (await db.execute( + select(Post).where( + Post.source_id == source.id, + Post.synthesized_by == DROP_GROUPER, + Post.absorbed_by_post_id.is_(None), + ).order_by(Post.post_date) + )).scalars().all() + + +@pytest.mark.asyncio +async def test_stages_a_day_apart_become_one_drop(db): + """The screenshot, measured shape: each stage 0.12 from the next and 0.46 + from the first, so seed distance splits them and nearest-neighbour joins + them.""" + artist, source = await _seed(db, "trickle-artist") + a = await _message(db, artist, source, at=T0, angle=0.0, text="Very early Marin.") + b = await _message(db, artist, source, at=T0 + timedelta(hours=17), angle=0.5, + text="Might get mirrored.") + c = await _message(db, artist, source, at=T0 + timedelta(hours=20), angle=1.0, + text="Got there eventually.") + await db.commit() + + assert await _group_then_merge(db, source) == 2 + + (drop,) = await _drops(db, source) + assert set(drop.synthesis_details["member_post_ids"]) == {a.id, b.id, c.id} + assert drop.synthesis_details["message_count"] == 3 + # The body is every message's text, in arrival order. + assert drop.description.index("Very early") < drop.description.index("Got there") + # And it says why — a grouping FC invented has to be checkable. + assert {m["route"] for m in drop.synthesis_details["merged"]} == {"nearest"} + + +@pytest.mark.asyncio +async def test_a_shared_working_name_joins_across_days(db): + """`svtt_wip3` did not join `svtt_wip4` from the day before on the live + instance. Orthogonal vectors here, so only the name can do it — and the + route it records says so.""" + artist, source = await _seed(db, "named-artist") + await _message(db, artist, source, at=T0, angle=0.0, name="svtt_wip3") + await _message(db, artist, source, at=T0 + timedelta(days=2), angle=math.pi / 2, + name="svtt_drench_b") + await db.commit() + + await _group_then_merge(db, source) + + (drop,) = await _drops(db, source) + assert drop.synthesis_details["merged"][0]["route"] == "name:svtt" + + +@pytest.mark.asyncio +async def test_nothing_joins_across_a_quiet_week(db): + """Measured: where the nearest neighbour was another message within 7 days + the names agreed 53 times of 53; past 7 days they begin to disagree.""" + artist, source = await _seed(db, "quiet-artist") + await _message(db, artist, source, at=T0, angle=0.0, name="alpha_wip1") + await _message(db, artist, source, at=T0 + timedelta(days=8), angle=0.05, + name="alpha_base") + await db.commit() + + assert await _group_then_merge(db, source) == 0 + assert len(await _drops(db, source)) == 2 + + +@pytest.mark.asyncio +async def test_two_named_pieces_meeting_through_a_third_stay_apart(db): + """C's image is nearest A's; B's image is nearest C's. A is `alpha`, B is + `beta` — so C reaches two different pieces and nothing moves.""" + artist, source = await _seed(db, "bridge-artist") + await _message(db, artist, source, at=T0, angle=0.0, name="alpha") + await _message(db, artist, source, at=T0 + timedelta(hours=3), angle=0.6, name="beta") + await _message(db, artist, source, at=T0 + timedelta(days=1), angle=0.25) + await db.commit() + + assert await _group_then_merge(db, source) == 0 + assert len(await _drops(db, source)) == 3 + + +@pytest.mark.asyncio +async def test_a_teaser_link_survives_its_drop_being_merged(db): + """The association's payload FK cascades. Merging the drop a teaser was + linked to must carry the link across, or it silently undoes #4402.""" + artist, source = await _seed(db, "linked-artist") + patreon = Source(artist_id=artist.id, platform="patreon", + url="https://patreon.com/linked-artist", enabled=True) + db.add(patreon) + await db.flush() + teaser = Post(source_id=patreon.id, artist_id=artist.id, external_post_id="teaser", + post_date=T0 + timedelta(days=1, hours=2)) + db.add(teaser) + await _message(db, artist, source, at=T0, angle=0.0, name="svtt_wip3") + await _message(db, artist, source, at=T0 + timedelta(days=1), angle=math.pi / 2, + name="svtt_drench_b") + await db.commit() + await group_source(db, source, max_distance=0.10, window_minutes=60) + later = (await _drops(db, source))[-1] + db.add(PostAssociation(announcement_post_id=teaser.id, payload_post_id=later.id, + score=1.0, status="linked", linked_by="fc")) + await db.commit() + + await merge_trickles(db, source, gap=GAP, min_images=2, cooldown=timedelta(hours=24)) + await db.commit() + + (drop,) = await _drops(db, source) + link = (await db.execute(select(PostAssociation))).scalar_one() + assert (link.payload_post_id, link.status, link.linked_by) == (drop.id, "linked", "fc") + + +@pytest.mark.asyncio +async def test_merging_history_does_not_drag_it_to_the_top_of_the_feed(db): + """Growth is stamped at the merged messages' own time. A merge the first + sweep makes over two-year-old drops must not read as news today.""" + artist, source = await _seed(db, "history-artist") + old = datetime.now(UTC) - timedelta(days=700) + await _message(db, artist, source, at=old, angle=0.0, name="svtt_wip1") + await _message(db, artist, source, at=old + timedelta(days=1), angle=0.5, name="svtt_wip2") + await _message(db, artist, source, at=old + timedelta(days=2), angle=1.0, name="svtt_base") + await db.commit() + + await _group_then_merge(db, source) + + (drop,) = await _drops(db, source) + assert drop.last_grew_at <= old + timedelta(days=2, minutes=1) + assert drop.resurfaced_at is None or drop.resurfaced_at <= old + timedelta(days=2, minutes=1) + + +@pytest.mark.asyncio +async def test_a_checked_drop_is_not_examined_again(db): + """Each drop costs one nearest-neighbour query per image, once. The flag is + what stops every hourly sweep repeating the whole history.""" + artist, source = await _seed(db, "checked-artist") + await _message(db, artist, source, at=T0, angle=0.0, name="alpha") + await _message(db, artist, source, at=T0 + timedelta(days=3), angle=math.pi / 2, + name="beta") + await db.commit() + await _group_then_merge(db, source) + + for drop in await _drops(db, source): + assert drop.synthesis_details["trickle_checked"] is True + assert "nearest_message_ids" in drop.synthesis_details diff --git a/tests/test_post_unification.py b/tests/test_post_unification.py index 618c1ec..b1aefc8 100644 --- a/tests/test_post_unification.py +++ b/tests/test_post_unification.py @@ -29,11 +29,8 @@ from backend.app.models import ( from backend.app.services.discord_grouping import DROP_GROUPER from backend.app.services.post_association_service import PostAssociationService from backend.app.services.post_feed_service import PostFeedService -from backend.app.services.post_unification import ( - FAMILY_MAX_POSTS, - Candidate, - family, -) +from backend.app.services.post_naming import FAMILY_MAX_POSTS +from backend.app.services.post_unification import Candidate, family T0 = datetime(2026, 9, 1, 12, 0, tzinfo=UTC) WINDOW = timedelta(days=60) -- 2.54.0 From 2f9e35390e723a2d5cad2df48cfdfdfeba7b019c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 14:01:07 -0400 Subject: [PATCH 63/94] fix: the matcher sees a drop's names, finds merged trickles, and catches up on its own (4390, 4392) Three defects found while answering "is the linking automatic": 1. The name and image checks never worked on a live drop. The corpus keyed every image by primary_post_id, but a drop's images belong to its member messages and the drop claims them only through provenance, so a drop looked nameless and hashless. The tests attached images to the drop itself, which discord_grouping never does. Images now count under the post a reader sees them on: a message's absorbing drop, else the post itself. 2. The trickle merge (e08401c) dates a drop by its first stage, days before the release a teaser announces, which put merged trickles outside the 24h window. Candidates are now found and timed by their closest member message. The sweep follows recently grown drops by their messages' times the same way. 3. A pair left pending was skipped forever: the matcher skipped every recorded pair, not only decided ones. Pending pairs are now re-scored in place and linked once conclusive. Linked and dismissed pairs are still never touched. Also, "Scan now" shared the sweep's 48-hour horizon, so it could not reach the history it is described as being for. It now scores every post by an artist with Discord drops (rescan(full=True)). New tests build drops the way the grouper does, with images owned by the member messages. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/api/posts.py | 4 +- .../app/services/post_association_service.py | 159 +++++++++++++++--- tests/test_post_association.py | 153 +++++++++++++++++ 3 files changed, 289 insertions(+), 27 deletions(-) diff --git a/backend/app/api/posts.py b/backend/app/api/posts.py index 2823368..49fcf93 100644 --- a/backend/app/api/posts.py +++ b/backend/app/api/posts.py @@ -207,6 +207,8 @@ async def rescan_associations(): be within the window to exist at all); this is the button for a first run over a library that predates the feature.""" async with get_session() as session: - result = await association_rescan(session) + # full=True: the button reaches the whole history, which the hourly + # sweep's 48-hour horizon never does. + result = await association_rescan(session, full=True) await session.commit() return jsonify(result) diff --git a/backend/app/services/post_association_service.py b/backend/app/services/post_association_service.py index f5706c6..2fb0567 100644 --- a/backend/app/services/post_association_service.py +++ b/backend/app/services/post_association_service.py @@ -286,10 +286,28 @@ class PostAssociationService: paths_by_post: dict[int, list[str]] = {} hashes_by_post: dict[int, list[int]] = {} + # An image is counted under the post a reader SEES it on: a Discord + # message absorbed into a drop contributes to the drop, not to itself. + # + # Keyed on `primary_post_id` alone until 2026-09-24, which on the live + # instance meant a drop never had a name or a hash at all — its images + # are owned by its member messages, and the drop only claims them + # through provenance. The identity route therefore never fired there; + # the tests missed it because they attached images to the drop itself, + # which discord_grouping never does. + # + # Counting by the drop also makes the span honest: five wips of one + # piece posted as five messages and grouped into one drop are ONE post + # as far as "how many posts carry this name" is concerned. + owner = Post.__table__.alias("owner") rows = await self.session.execute( select( - ImageRecord.primary_post_id, ImageRecord.path, ImageRecord.phash - ).where( + func.coalesce(owner.c.absorbed_by_post_id, ImageRecord.primary_post_id), + ImageRecord.path, ImageRecord.phash, + ) + .select_from(ImageRecord) + .join(owner, owner.c.id == ImageRecord.primary_post_id) + .where( ImageRecord.artist_id == artist_id, ImageRecord.primary_post_id.is_not(None), ) @@ -333,44 +351,81 @@ class PostAssociationService: self._corpora[artist_id] = corpus return corpus - async def _decided(self, announcement_id: int) -> set[int]: - """Payload posts already proposed for this announcement, in ANY status. + async def _decided(self, announcement_id: int) -> dict[int, PostAssociation]: + """Pairs already recorded for this announcement, keyed by payload. - Dismissed pairs are included deliberately: re-proposing a pair the - operator has already rejected on every subsequent scan is the single - behaviour that makes a review queue get ignored. + Linked and dismissed pairs are never touched again: re-proposing a pair + the operator has already rejected on every subsequent scan is the + single behaviour that makes a review queue get ignored. + + A PENDING pair is different — nobody has decided it — so the caller + re-scores it. Otherwise a pair queued by an older, weaker matcher sits + in the queue forever even once the evidence is conclusive, which is the + chore the operator asked FC not to hand them. """ rows = (await self.session.execute( - select(PostAssociation.payload_post_id) + select(PostAssociation) .where(PostAssociation.announcement_post_id == announcement_id) )).scalars().all() - return set(rows) + return {a.payload_post_id: a for a in rows} async def _candidate_groups( self, announcement: Post, *, window: timedelta, - ) -> list[Post]: - """Synthetic Discord groupings by the SAME artist, inside the window. + ) -> list[tuple[Post, datetime]]: + """Synthetic Discord groupings by the SAME artist with a message inside + the window — each with the time of its message CLOSEST to the teaser. Same-artist is the identity signal and it is free (see the module docstring on E4). It is also a hard filter rather than a scored one: two different creators posting minutes apart is a coincidence, not evidence, and letting it score at all would mean a busy hour across the library could out-vote everything else. + + Measured on the MESSAGES, not the drop's own date. A drop is dated by + its first message, and since #4390 merges a creator's trickle into one + drop, that can be days before the release the teaser announces — + "Very early Marin" on Sep 7, `MarinaraSauce_base` on Sep 11. Matching + on the drop's date would put every merged trickle outside the window. """ at = _post_time(announcement) - sort_key = func.coalesce(Post.post_date, Post.downloaded_at) - return (await self.session.execute( - select(Post) + member = Post.__table__.alias("member") + member_at = func.coalesce(member.c.post_date, member.c.downloaded_at) + rows = list((await self.session.execute( + select(member.c.absorbed_by_post_id, member_at) .where( + member.c.artist_id == announcement.artist_id, + member.c.absorbed_by_post_id.is_not(None), + member_at >= at - window, + member_at <= at + window, + ) + )).all()) + # The drop's own date counts too — the first message's, so it adds + # nothing for a real drop, but it keeps a drop with no member rows + # (hand-built, or one whose messages were removed) matchable. + own_at = func.coalesce(Post.post_date, Post.downloaded_at) + rows += (await self.session.execute( + select(Post.id, own_at).where( Post.artist_id == announcement.artist_id, Post.synthesized_by == DROP_GROUPER, + own_at >= at - window, + own_at <= at + window, + ) + )).all() + closest: dict[int, datetime] = {} + for group_id, when in rows: + if group_id not in closest or abs(when - at) < abs(closest[group_id] - at): + closest[group_id] = when + if not closest: + return [] + groups = (await self.session.execute( + select(Post).where( + Post.id.in_(list(closest)), + Post.synthesized_by == DROP_GROUPER, Post.id != announcement.id, - sort_key >= at - window, - sort_key <= at + window, ) - .order_by(sort_key) - .limit(MAX_CANDIDATES) )).scalars().all() + ranked = sorted(groups, key=lambda g: (abs(closest[g.id] - at), g.id)) + return [(g, closest[g.id]) for g in ranked[:MAX_CANDIDATES]] async def _claimed(self, announcement_id: int, payload_id: int) -> bool: """Is either end of this pair already spoken for by an accepted link? @@ -415,8 +470,9 @@ class PostAssociationService: made = 0 scored: list[tuple[Post, float, dict, float]] = [] - for group in await self._candidate_groups(announcement, window=window): - if group.id in already: + for group, group_at in await self._candidate_groups(announcement, window=window): + prior = already.get(group.id) + if prior is not None and prior.status != "pending": continue named, token = shared_identity( here, @@ -435,7 +491,7 @@ class PostAssociationService: identity = max(named, copied) circumstantial = { "proximity": proximity_signal( - _post_time(group) - _post_time(announcement), window, + group_at - _post_time(announcement), window, ), "declared": declared, "marker": marker_overlap( @@ -494,6 +550,16 @@ class PostAssociationService: status = "linked" if group.id == auto_id else "pending" if status == "linked": linked += 1 + prior = already.get(group.id) + if prior is not None: + # Re-scored in place: a pending pair keeps its row (and id), + # and only an upgrade to linked counts as news. + prior.score = score + prior.signals = signals + if status == "linked": + prior.status = "linked" + prior.linked_by = "fc" + continue self.session.add(PostAssociation( announcement_post_id=announcement.id, payload_post_id=group.id, @@ -573,8 +639,17 @@ class PostAssociationService: return out -async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict: - """Score every recent non-synthetic post against nearby groupings.""" +async def rescan( + session: AsyncSession, *, now: datetime | None = None, full: bool = False, +) -> dict: + """Score recent non-synthetic posts against nearby groupings. + + `full=True` scores EVERY post by an artist who has Discord drops at all — + the manual button's job, for history that predates the feature or that a + trickle merge (#4390) has just rearranged. It used to share the sweep's + 48-hour horizon, so the button described as "a first run over a library + that predates the feature" could not reach that library. + """ settings = await ImportSettings.load(session) if not settings.discord_link_enabled: return {"enabled": False, "scanned": 0, "proposed": 0, "linked": 0} @@ -586,6 +661,18 @@ async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict: # a full-library rescan is the manual button's job, not the sweep's. horizon = now - timedelta(hours=window_hours * 2) sort_key = func.coalesce(Post.post_date, Post.downloaded_at) + if full: + with_drops = select(Post.artist_id).where( + Post.synthesized_by == DROP_GROUPER + ).distinct() + ids = set((await session.execute( + select(Post.id).where( + Post.synthesized_by.is_(None), + Post.absorbed_by_post_id.is_(None), + Post.artist_id.in_(with_drops), + ) + )).scalars().all()) + return await _score(session, settings, ids, window_hours) ids = set((await session.execute( select(Post.id).where( Post.synthesized_by.is_(None), @@ -605,14 +692,28 @@ async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict: # That is #4392's third cause, and it is the one that left a measured 0.800 # pair with an empty review queue on the live instance. The other two were # about scoring; this one meant nothing was scored at all. - drop_times = (await session.execute( - select(sort_key).where( + # + # The times are the drops' MESSAGES, not the drops' own dates: a drop that + # grew today by a trickle merge (#4390) is dated by its first stage, days + # earlier, while the teaser sits beside the message that just joined. + recent = ( + select(Post.id).where( Post.synthesized_by == DROP_GROUPER, func.coalesce(Post.last_grew_at, Post.downloaded_at) >= horizon, ) .order_by(func.coalesce(Post.last_grew_at, Post.downloaded_at).desc()) .limit(MAX_RECENT_DROPS) - )).scalars().all() + ) + member = Post.__table__.alias("member") + drop_times = set((await session.execute( + select(func.coalesce(member.c.post_date, member.c.downloaded_at)) + .where(member.c.absorbed_by_post_id.in_(recent)) + )).scalars().all()) + # Plus each drop's own date — its first message's, so a duplicate for a + # real drop, but what a drop with no member rows is placed by. + drop_times |= set((await session.execute( + select(sort_key).where(Post.id.in_(recent)) + )).scalars().all()) # The interval arithmetic is done in Python rather than SQL: a handful of # literal ranges is portable, and `now - INTERVAL` is not. ranges = [ @@ -629,6 +730,12 @@ async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict: ) )).scalars().all()) + return await _score(session, settings, ids, window_hours) + + +async def _score( + session: AsyncSession, settings: ImportSettings, ids: set[int], window_hours: float, +) -> dict: svc = PostAssociationService(session) proposed = 0 linked = 0 diff --git a/tests/test_post_association.py b/tests/test_post_association.py index ff9a890..2c0a4e3 100644 --- a/tests/test_post_association.py +++ b/tests/test_post_association.py @@ -950,3 +950,156 @@ def test_the_duplicate_threshold_sits_inside_the_measured_gap(): justifies it: 20 bits was the widest true pair, 108 the nearest unrelated one.""" assert 20 < DUPLICATE_MAX_DISTANCE < 108 + + +# --- drops shaped as the grouper actually writes them ---------------------- +# +# Every test above hand-builds a drop that OWNS its images. discord_grouping +# never does that: a drop's images belong to its member messages and the drop +# claims them through provenance. Against that shape the matcher saw no names +# and no hashes at all, so the identity route could not fire on the live +# instance — and nothing here could have noticed. + + +async def _real_drop(db, artist, discord, *, stages): + """Messages, each owning one named image, grouped by the real grouper and + merged by the real trickle pass. `stages` is [(at, name), ...].""" + from backend.app.services.discord_grouping import group_source, merge_trickles + + for i, (at, name) in enumerate(stages): + msg = Post(source_id=discord.id, artist_id=artist.id, + external_post_id=f"{artist.slug}-msg-{i}", post_date=at) + db.add(msg) + await db.flush() + vec = [0.0] * 1152 + vec[i % 1152] = 1.0 + db.add(ImageRecord( + path=f"/images/{artist.slug}/20260901_{1234567890000 + i}_01_{name}.png", + sha256=f"{artist.id:08d}{i:056d}", size_bytes=10, mime="image/png", + width=10, height=10, origin="downloaded", primary_post_id=msg.id, + artist_id=artist.id, siglip_embedding=vec, + )) + await db.flush() + await group_source(db, discord, max_distance=0.10, window_minutes=60) + await merge_trickles(db, discord, gap=timedelta(hours=168), min_images=2, + cooldown=timedelta(hours=24)) + await db.commit() + return (await db.execute( + select(Post).where(Post.source_id == discord.id, + Post.synthesized_by == DROP_GROUPER, + Post.absorbed_by_post_id.is_(None)) + )).scalars().all() + + +@pytest.mark.asyncio +async def test_a_drop_whose_images_belong_to_its_messages_still_links_on_its_name(db): + """The live shape. `0-k` on the teaser and on the drop's MESSAGE, nowhere + else — conclusive, so FC links it without asking.""" + artist, patreon, discord = await _artist_with_channels(db, "liveshape") + now = datetime.now(UTC) - timedelta(days=5) + (drop,) = await _real_drop(db, artist, discord, stages=[(now, "0-k_base")]) + teaser = await _teaser(db, artist, patreon, at=now + timedelta(hours=20), + body="new one", names=["0-k"]) + await db.commit() + + made = await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, auto_link=True, + ) + await db.commit() + + assert made == (1, 1) + assoc = (await db.execute(select(PostAssociation))).scalar_one() + assert (assoc.payload_post_id, assoc.signals["identity_token"]) == (drop.id, "0-k") + + +@pytest.mark.asyncio +async def test_a_merged_trickle_is_found_by_its_latest_stage(db): + """A trickle merged by #4390 is dated by its FIRST stage — four days before + the release the teaser announces. Matching on the drop's own date would put + it outside the 24h window.""" + artist, patreon, discord = await _artist_with_channels(db, "trickleshape") + start = datetime.now(UTC) - timedelta(days=10) + (drop,) = await _real_drop(db, artist, discord, stages=[ + (start, "svtt_wip1"), (start + timedelta(days=2), "svtt_wip3"), + (start + timedelta(days=4), "svtt_drench_b"), + ]) + teaser = await _teaser(db, artist, patreon, at=start + timedelta(days=4, hours=1), + body="new one", names=["svtt_teaser"]) + await db.commit() + + made = await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, auto_link=True, + ) + await db.commit() + + assert made[0] == 1 + assoc = (await db.execute(select(PostAssociation))).scalar_one() + assert assoc.payload_post_id == drop.id + # Timed against the closest message, an hour away — not the first, four + # days away. + assert assoc.signals["proximity"] > 0.9 + + +@pytest.mark.asyncio +async def test_a_pair_left_pending_is_linked_once_the_evidence_is_conclusive(db): + """Nobody decided a pending pair, so the matcher re-scores it. Otherwise a + pair queued by an older, weaker matcher waits for a click forever.""" + artist, patreon, discord = await _artist_with_channels(db, "pendingshape") + now = datetime.now(UTC) - timedelta(days=5) + (drop,) = await _real_drop(db, artist, discord, stages=[(now, "0-k_base")]) + teaser = await _teaser(db, artist, patreon, at=now + timedelta(hours=2), + body="new one", names=["0-k"]) + db.add(PostAssociation(announcement_post_id=teaser.id, payload_post_id=drop.id, + score=0.61, signals={"proximity": 0.9}, status="pending")) + await db.commit() + + await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, auto_link=True, + ) + await db.commit() + + assoc = (await db.execute(select(PostAssociation))).scalar_one() + assert (assoc.status, assoc.linked_by) == ("linked", "fc") + + +@pytest.mark.asyncio +async def test_a_dismissed_pair_is_never_rescored(db): + artist, patreon, discord = await _artist_with_channels(db, "dismissedshape") + now = datetime.now(UTC) - timedelta(days=5) + (drop,) = await _real_drop(db, artist, discord, stages=[(now, "0-k_base")]) + teaser = await _teaser(db, artist, patreon, at=now + timedelta(hours=2), + body="new one", names=["0-k"]) + db.add(PostAssociation(announcement_post_id=teaser.id, payload_post_id=drop.id, + score=0.61, signals={}, status="dismissed")) + await db.commit() + + assert await PostAssociationService(db).match_post( + teaser.id, threshold=DEFAULT_THRESHOLD, window_hours=WINDOW, auto_link=True, + ) == (0, 0) + assoc = (await db.execute(select(PostAssociation))).scalar_one() + assert assoc.status == "dismissed" + + +@pytest.mark.asyncio +async def test_the_full_rescan_reaches_history_the_sweep_does_not(db): + """The "Scan now" button. It used to share the sweep's 48-hour horizon, so + it could not reach the library it was described as being for.""" + artist, patreon, discord = await _artist_with_channels(db, "historyshape") + long_ago = datetime.now(UTC) - timedelta(days=400) + (drop,) = await _real_drop(db, artist, discord, stages=[(long_ago, "0-k_base")]) + # Authored long ago too. A drop FC wrote TODAY is exactly what the sweep + # follows back to its teaser (#4392's third cause), so it would find this + # pair without the button. + drop.downloaded_at = long_ago + await _teaser(db, artist, patreon, at=long_ago + timedelta(hours=2), + body="new one", names=["0-k"]) + settings = await ImportSettings.load(db) + settings.discord_link_enabled = True + await db.commit() + + assert (await rescan(db))["proposed"] == 0 + await db.commit() + result = await rescan(db, full=True) + await db.commit() + + assert (result["proposed"], result["linked"]) == (1, 1) -- 2.54.0 From 49d18c1757027620685f64403be398f4402d9583 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 14:13:12 -0400 Subject: [PATCH 64/94] fix: adding a found subscription sends its membership id (adopt 400 invalid_body) membershipReconcile.adopt passed { membership_id } as request options, so no body was sent. useApi now refuses any option other than body, params or signal, so a payload in the wrong place fails loudly in the browser instead of as a 400 from the server. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- frontend/src/composables/useApi.js | 17 +++++++++- frontend/src/stores/membershipReconcile.js | 2 +- frontend/test/composables/useApi.spec.js | 37 ++++++++++++++++++++++ 3 files changed, 54 insertions(+), 2 deletions(-) create mode 100644 frontend/test/composables/useApi.spec.js diff --git a/frontend/src/composables/useApi.js b/frontend/src/composables/useApi.js index 76cb4f7..ffeba82 100644 --- a/frontend/src/composables/useApi.js +++ b/frontend/src/composables/useApi.js @@ -10,7 +10,22 @@ export class ApiError extends Error { } } -async function request(method, url, { body, params, signal } = {}) { +const OPTIONS = new Set(['body', 'params', 'signal']) + +async function request(method, url, opts = {}) { + // Refuse an option this wrapper does not know. `api.post(url, { id: 1 })` + // reads naturally and sends NO body — the payload lands in the options bag + // and is dropped — so the server answers `invalid_body` and the caller + // looks broken server-side. It shipped exactly that way in the Subscriptions + // "Add subscription" button (2026-09-24). Failing here names the mistake. + const unknown = Object.keys(opts).filter((k) => !OPTIONS.has(k)) + if (unknown.length) { + throw new TypeError( + `useApi ${method} ${url}: unknown option(s) ${unknown.join(', ')} — ` + + 'a request payload goes under `body`, a query under `params`' + ) + } + const { body, params, signal } = opts let fullUrl = url if (params) { const search = new URLSearchParams() diff --git a/frontend/src/stores/membershipReconcile.js b/frontend/src/stores/membershipReconcile.js index 66d6a90..8298224 100644 --- a/frontend/src/stores/membershipReconcile.js +++ b/frontend/src/stores/membershipReconcile.js @@ -28,7 +28,7 @@ export const useMembershipReconcileStore = defineStore('membershipReconcile', () async function adopt (membershipId) { try { const res = await api.post('/api/sources/reconciliation/adopt', { - membership_id: membershipId + body: { membership_id: membershipId } }) toast({ text: res.already_tracked diff --git a/frontend/test/composables/useApi.spec.js b/frontend/test/composables/useApi.spec.js new file mode 100644 index 0000000..010c39c --- /dev/null +++ b/frontend/test/composables/useApi.spec.js @@ -0,0 +1,37 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' + +import { useApi } from '../../src/composables/useApi.js' + +// 2026-09-24: `api.post(url, { membership_id })` shipped in the Subscriptions +// "Add subscription" button. It reads naturally, sends NO body, and the +// server's `invalid_body` made the bug look like the backend's. The wrapper now +// refuses an option it does not know, so the mistake names itself. +describe('useApi', () => { + afterEach(() => { vi.unstubAllGlobals() }) + + function stubFetch () { + const fetch = vi.fn().mockResolvedValue({ + ok: true, status: 200, statusText: 'OK', text: () => Promise.resolve('{}'), + }) + vi.stubGlobal('fetch', fetch) + return fetch + } + + it('refuses a payload passed as options instead of under body', async () => { + const fetch = stubFetch() + await expect(useApi().post('/api/x', { membership_id: 1 })) + .rejects.toThrow(/membership_id.*body/) + expect(fetch).not.toHaveBeenCalled() + }) + + it('sends a body passed under body', async () => { + const fetch = stubFetch() + await useApi().post('/api/x', { body: { membership_id: 1 } }) + expect(JSON.parse(fetch.mock.calls[0][1].body)).toEqual({ membership_id: 1 }) + }) + + it('still accepts no options at all', async () => { + stubFetch() + await expect(useApi().get('/api/x')).resolves.toEqual({}) + }) +}) -- 2.54.0 From da2091a875ed22ea5ed908157922fd473922d559 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 14:13:12 -0400 Subject: [PATCH 65/94] fix: a download is marked seen only after it is imported A run killed between download and import left its files on disk and in the seen-ledger but never in the library, and every later walk trusted the ledger. TamadaHeijun's 12PCG post lost 7 of 13 images this way (stranded run 90402), which read as the duplicates filter. The ingester now hands phase 3 a mark_seen_after_import hook, called after the import loop. A file on disk with no ImageRecord at its path is fed to import, not reconciled into the ledger. Recapture reaches files already orphaned, because it looks past the ledger to the disk. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/download_service.py | 7 ++ backend/app/services/gallery_dl.py | 8 +++ backend/app/services/ingest_core.py | 44 +++++++++++- .../subscriptions/SourceActions.vue | 5 +- tests/test_download_service.py | 13 +++- tests/test_patreon_ingester.py | 70 +++++++++++++++++++ 6 files changed, 141 insertions(+), 6 deletions(-) diff --git a/backend/app/services/download_service.py b/backend/app/services/download_service.py index 825bb82..979676d 100644 --- a/backend/app/services/download_service.py +++ b/backend/app/services/download_service.py @@ -423,6 +423,13 @@ class DownloadService: await loop.run_in_executor(None, _upsert) + # Only now is it safe to call this walk's media seen: every file above + # has been through the importer. Had the run died before here they stay + # unmarked, and the next walk imports them from disk (ingest_core). + mark_seen = getattr(dl_result, "mark_seen_after_import", None) + if mark_seen is not None: + await loop.run_in_executor(None, mark_seen) + # #830 recapture: backfill source_filehash on EXISTING on-disk images so # their post-body inline remaps to the local copy. A # SEPARATE non-deleting channel (NOT the import list — that would unlink diff --git a/backend/app/services/gallery_dl.py b/backend/app/services/gallery_dl.py index b07fb1b..90933bf 100644 --- a/backend/app/services/gallery_dl.py +++ b/backend/app/services/gallery_dl.py @@ -17,6 +17,7 @@ import subprocess import sys import tempfile import time +from collections.abc import Callable from dataclasses import dataclass, field from datetime import UTC, datetime from enum import StrEnum @@ -230,6 +231,13 @@ class DownloadResult: # the platform cooldown matches the hint instead of a flat default. None when # unknown (no header, or not a rate-limit failure). retry_after_seconds: float | None = None + # Native ingester only: marks this walk's fetched media seen in its ledger. + # Phase 3 calls it AFTER the import loop, never before — a file marked seen + # but not yet imported is invisible to every later walk, so a run killed in + # between orphaned it for good (TamadaHeijun's 12PCG post lost 7 of 13 + # images to a stranded run, 2026-09-24). Unmarked, the next walk finds the + # file on disk with no ImageRecord and imports it. None on gallery-dl. + mark_seen_after_import: Callable[[], None] | None = None def extract_errors_warnings(stderr: str) -> str: diff --git a/backend/app/services/ingest_core.py b/backend/app/services/ingest_core.py index 8105baf..8bb1e3d 100644 --- a/backend/app/services/ingest_core.py +++ b/backend/app/services/ingest_core.py @@ -36,6 +36,7 @@ from datetime import UTC, datetime, timedelta from sqlalchemy import delete, func, select, text from sqlalchemy.dialects.postgresql import insert as pg_insert +from ..models import ImageRecord from .gallery_dl import ( DownloadResult, ErrorType, @@ -261,6 +262,9 @@ class Ingester: # source_filehash and (b) link the on-disk image to its Post (#1288) — # WITHOUT re-downloading or unlinking the file. Empty outside recapture. relink: list[tuple[str, str, str]] = [] + # Media handed to phase 3 for import. Marked seen by phase 3 once the + # import has run (`mark_seen_after_import`), not here — see there. + fetched: list[tuple[str, str]] = [] downloaded = 0 errors = 0 quarantined = 0 @@ -313,6 +317,7 @@ class Ingester: written_paths=written, post_record_paths=list(post_records), relink_source_paths=list(relink), + mark_seen_after_import=lambda: self._mark_seen(source_id, fetched), stdout="\n".join(log_lines), stderr="", return_code=return_code, @@ -512,6 +517,13 @@ class Ingester: recapture=recapture, ) + # An on-disk file is only "done" if something imported it. One + # with no ImageRecord at its path was written by a run that died + # before phase 3 — it goes to import, not to the ledger. + imported_paths = self._recorded_paths([ + str(o.path) for o in outcomes + if o.status == "skipped_disk" and o.path is not None + ]) to_mark: list[tuple[str, str]] = [] to_clear: list[str] = [] # recovered → drop any dead-letter row to_fail: list[tuple[str, str, str]] = [] # (key, post_id, error) @@ -523,11 +535,29 @@ class Ingester: downloaded += 1 if outcome.path is not None: written.append(str(outcome.path)) - to_mark.append((key, media_item.post_id)) + fetched.append((key, media_item.post_id)) to_clear.append(key) consecutive_seen = 0 + elif ( + outcome.status == "skipped_disk" + and outcome.path is not None + and str(outcome.path) not in imported_paths + ): + # On disk, never imported: a prior run wrote it and died + # before phase 3. Import it now. Safe to feed to + # attach_in_place because no record owns this path — + # the unlink below is about a file that IS the record. + written.append(str(outcome.path)) + fetched.append((key, media_item.post_id)) + to_clear.append(key) + skipped_count += 1 + consecutive_seen += 1 + log_lines.append( + f" post {media_item.post_id} — on disk but never " + f"imported: {outcome.path.name}" + ) elif outcome.status == "skipped_disk": - # Already on disk (a prior run). Reconcile the ledger so a + # Already on disk and imported. Reconcile the ledger so a # later tick skips it at tier-1 without a disk stat, but # do NOT re-feed it to phase 3 — attach_in_place would see # the duplicate sha256 and unlink the on-disk copy. @@ -887,6 +917,16 @@ class Ingester: ) session.commit() + def _recorded_paths(self, paths: list[str]) -> set[str]: + """Which of `paths` an ImageRecord already points at.""" + if not paths: + return set() + with self.session_factory() as session: + rows = session.execute( + select(ImageRecord.path).where(ImageRecord.path.in_(paths)) + ).scalars().all() + return set(rows) + def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None: """Idempotent upsert of (filehash, post_id) seen-ledger rows for a page. diff --git a/frontend/src/components/subscriptions/SourceActions.vue b/frontend/src/components/subscriptions/SourceActions.vue index e026aed..34e484f 100644 --- a/frontend/src/components/subscriptions/SourceActions.vue +++ b/frontend/src/components/subscriptions/SourceActions.vue @@ -45,8 +45,9 @@ > Recapture post text & links - Re-grab every post's body + external links and localize inline images - already on disk — without re-downloading media + Re-grab every post's body + external links, localize inline images + already on disk, and import any downloaded file that never reached the + library — without re-downloading media diff --git a/tests/test_download_service.py b/tests/test_download_service.py index 7533dcb..a347646 100644 --- a/tests/test_download_service.py +++ b/tests/test_download_service.py @@ -9,9 +9,9 @@ from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock import pytest -from sqlalchemy import select +from sqlalchemy import func, select -from backend.app.models import Artist, DownloadEvent, ImportSettings, Source +from backend.app.models import Artist, DownloadEvent, ImageRecord, ImportSettings, Source from backend.app.services.credential_crypto import CredentialCrypto from backend.app.services.credential_service import CredentialService from backend.app.services.thumbnailer import Thumbnailer @@ -152,6 +152,14 @@ async def test_download_source_attaches_written_files( files_downloaded=2, stdout=f"{f1}\n{f2}\n", ) + # The ledger is marked only once the files are in: a run killed before + # import must leave them unmarked so the next walk imports them. + imported_when_marked = [] + result.mark_seen_after_import = lambda: imported_when_marked.append( + db_sync.execute( + select(func.count(ImageRecord.id)).where(ImageRecord.path.in_([str(f1), str(f2)])) + ).scalar_one() + ) fake_gdl = _fake_gdl_with_result(result) sync_settings = db_sync.execute( @@ -182,6 +190,7 @@ async def test_download_source_attaches_written_files( assert ev.files_count == 2 assert ev.metadata_["import_summary"]["attached"] == 2 assert ev.metadata_["run_stats"]["downloaded_count"] == 2 + assert imported_when_marked == [2] @pytest.mark.asyncio diff --git a/tests/test_patreon_ingester.py b/tests/test_patreon_ingester.py index 774722b..7f56a96 100644 --- a/tests/test_patreon_ingester.py +++ b/tests/test_patreon_ingester.py @@ -7,6 +7,7 @@ real CDN. The ledger is real (a sync sessionmaker bound to the test engine), so the tier-1 skip and the idempotent mark-seen run against actual rows. """ +import hashlib from datetime import UTC, datetime, timedelta import pytest @@ -15,6 +16,7 @@ from sqlalchemy.orm import sessionmaker from backend.app.models import ( Artist, + ImageRecord, PatreonFailedMedia, PatreonSeenMedia, Source, @@ -237,6 +239,9 @@ async def test_tick_downloads_unseen_and_marks_seen(source_id, sync_engine, tmp_ # plan #704: structured run_stats carry the real counts. assert result.run_stats["downloaded_count"] == 2 assert result.posts_processed == 1 + # The media wait for phase 3 to import them; only the post key is in yet. + assert _count_ledger(sync_engine, source_id) == 1 + result.mark_seen_after_import() # 2 media keys + 1 synthetic post key (body/links recaptured per post). assert _count_ledger(sync_engine, source_id) == 3 # The post body + links are captured for media posts too (rides the walk). @@ -263,6 +268,7 @@ async def test_quarantined_media_surfaced_in_result(source_id, sync_engine, tmp_ assert result.run_stats["quarantined_count"] == 1 assert result.run_stats["downloaded_count"] == 1 assert len(result.written_paths) == 1 # quarantined NOT written + result.mark_seen_after_import() # Quarantined media is NOT marked seen (a fixed file may be re-fetched); # m1 + the synthetic post key (body/links captured per post) = 2. assert _count_ledger(sync_engine, source_id) == 2 @@ -573,6 +579,7 @@ async def test_recovery_tier2_disk_still_skips(source_id, sync_engine, tmp_path) m1 = _media("p1", 1) client = _FakeClient([(None, [("p1", [m1])])]) # File still on disk (a kept image) → tier-2 spares it even under recovery. + _seed_record(sync_engine, tmp_path / "p1_1.jpg") downloader = _FakeDownloader(tmp_path, on_disk={_ledger_key(m1)}) ing = _ingester(sync_engine, tmp_path, client, downloader) @@ -582,6 +589,7 @@ async def test_recovery_tier2_disk_still_skips(source_id, sync_engine, tmp_path) ) assert result.files_downloaded == 0 assert downloader.download_calls == 0 + assert result.written_paths == [] # Disk-skip reconciles the media key + the synthetic post key (recovery # recaptures the body/links per post) = 2. assert _count_ledger(sync_engine, source_id) == 2 @@ -609,6 +617,15 @@ async def test_backfill_recaptures_body_for_already_downloaded_post( assert downloader.post_records == 1 +def _seed_record(sync_engine, path): + """An ImageRecord at `path` — the file was imported, not just downloaded.""" + factory = sessionmaker(sync_engine, expire_on_commit=False) + with factory() as s: + s.add(ImageRecord(path=str(path), sha256=hashlib.sha256(str(path).encode()).hexdigest(), + size_bytes=1, mime="image/jpeg", origin="downloaded")) + s.commit() + + def _seed_seen(sync_engine, source_id, key, post_id=None): factory = sessionmaker(sync_engine, expire_on_commit=False) with factory() as s: @@ -629,6 +646,7 @@ async def test_backfill_skips_already_captured_post_but_recapture_forces_it( # Pre-seed BOTH the media key and the synthetic post key as already seen. _seed_seen(sync_engine, source_id, _ledger_key(m1), post_id="p1") _seed_seen(sync_engine, source_id, "post:p1", post_id="p1") + _seed_record(sync_engine, tmp_path / "p1_1.jpg") # 1) Plain backfill: post key is seen → gate skips body recapture. client = _FakeClient([(None, [("p1", [m1])])]) @@ -696,6 +714,7 @@ async def test_gated_post_skipped_entirely_no_media_no_record( assert downloader.download_calls == 1 assert len(result.post_record_paths) == 1 # only the open post assert downloader.post_records == 1 + result.mark_seen_after_import() # The gated post left NO trace in the seen-ledger (no media key, no post key): # only the open post's media key + synthetic post key are recorded. assert _count_ledger(sync_engine, source_id) == 2 @@ -827,6 +846,57 @@ async def test_recapture_does_not_refetch_seen_media_missing_from_disk( assert result.relink_source_paths == [] +# --- a run that dies between download and import --------------------------- +# TamadaHeijun's 【12PCG】 post, 2026-09-24: 13 files on disk, 5 in the library. +# A run wrote 01–08, marked them seen, and was killed before phase 3 imported +# them; every later walk trusted the ledger and never looked again. + + +@pytest.mark.asyncio +async def test_a_run_that_dies_before_import_leaves_its_media_unmarked( + source_id, sync_engine, tmp_path, +): + m1 = _media("p1", 1) + ing = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1])])]), + _FakeDownloader(tmp_path)) + ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest", + url="https://patreon.com/ingest", mode="tick") + # Phase 3 never ran, so `mark_seen_after_import` never did: only the post key. + assert _count_ledger(sync_engine, source_id) == 1 + + # The next walk finds the file on disk with no record, and imports it. + ing2 = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1])])]), + _FakeDownloader(tmp_path, on_disk={_ledger_key(m1)})) + result = ing2.run(source_id=source_id, campaign_id="c1", artist_slug="ingest", + url="https://patreon.com/ingest", mode="tick") + assert result.written_paths == [str(tmp_path / "p1_1.jpg")] + assert result.files_downloaded == 0 # not fetched again + assert "on disk but never imported: p1_1.jpg" in result.stdout + result.mark_seen_after_import() + assert _count_ledger(sync_engine, source_id) == 2 + + +@pytest.mark.asyncio +async def test_recapture_imports_a_seen_file_nothing_imported( + source_id, sync_engine, tmp_path, +): + """The repair for files already orphaned: they are in the ledger, so only + a walk that looks past it — recapture — reaches them.""" + m1, m2 = _media("p1", 1), _media("p1", 2) + for m in (m1, m2): + _seed_seen(sync_engine, source_id, _ledger_key(m), post_id="p1") + _seed_record(sync_engine, tmp_path / "p1_2.jpg") # m2 made it in; m1 did not + downloader = _FakeDownloader(tmp_path, on_disk={_ledger_key(m1), _ledger_key(m2)}) + ing = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1, m2])])]), + downloader) + result = ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest", + url="https://patreon.com/ingest", mode="recapture") + + assert result.written_paths == [str(tmp_path / "p1_1.jpg")] + assert [r[0] for r in result.relink_source_paths] == [str(tmp_path / "p1_2.jpg")] + assert downloader.download_calls == 0 + + # --- dead-letter ledger (plan #705 #7) ------------------------------------ -- 2.54.0 From 3aa899ad29f05e5151ffe2903cbb7831dff45b5e Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 14:16:07 -0400 Subject: [PATCH 66/94] test: revisit tests' first walk completes phase 3 before the second Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- tests/test_patreon_ingester.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_patreon_ingester.py b/tests/test_patreon_ingester.py index 7f56a96..47cf97f 100644 --- a/tests/test_patreon_ingester.py +++ b/tests/test_patreon_ingester.py @@ -1366,7 +1366,7 @@ async def test_an_edited_post_inside_the_window_downloads_its_new_attachment( ing1.run( source_id=source_id, campaign_id="c1", artist_slug="ingest", url="https://patreon.com/ingest", mode="tick", revisit_days=30, - ) + ).mark_seen_after_import() # phase 3 ran hotfix = _media("p1", 2) client2 = _FakeClient( @@ -1398,7 +1398,7 @@ async def test_a_post_below_the_horizon_keeps_its_capture_gate( ing1.run( source_id=source_id, campaign_id="c1", artist_slug="ingest", url="https://patreon.com/ingest", mode="tick", revisit_days=30, - ) + ).mark_seen_after_import() # phase 3 ran client2 = _FakeClient([(None, [("pold", [old])])], published={"pold": _iso(90)}) downloader2 = _FakeDownloader(tmp_path) @@ -1424,7 +1424,7 @@ async def test_a_revisit_that_finds_nothing_new_is_not_reported_as_an_update( ing1.run( source_id=source_id, campaign_id="c1", artist_slug="ingest", url="https://patreon.com/ingest", mode="tick", revisit_days=30, - ) + ).mark_seen_after_import() # phase 3 ran client2 = _FakeClient([(None, [("p1", [m])])], published={"p1": _iso(3)}) downloader2 = _FakeDownloader(tmp_path) -- 2.54.0 From 20cb8138097ad7daf6f396d12c3a1809dbe2886c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 15:16:41 -0400 Subject: [PATCH 67/94] fix: a one-message Discord drop is labelled a Discord message, not a grouping (4390) It stays synthetic, since teaser matching looks only at drops, but the card no longer claims "Grouped from 1 Discord message" / "grouped by FabledCurator" over a single message. The marker chip stays and reads "from Discord". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- frontend/src/components/posts/PostCard.vue | 11 +++++++++-- frontend/test/components/postCard.spec.js | 9 ++++++--- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/frontend/src/components/posts/PostCard.vue b/frontend/src/components/posts/PostCard.vue index 2d7be9f..d8a7981 100644 --- a/frontend/src/components/posts/PostCard.vue +++ b/frontend/src/components/posts/PostCard.vue @@ -20,7 +20,7 @@ class="fc-post-card__synthetic" :title="synthesisTitle" > - grouped by FabledCurator + {{ synthesisChip }} toPlainText(props.post.post_title)) // post dict by hand, must degrade to "not synthetic" rather than throw. const synthesized = computed(() => Boolean(props.post.synthesized_by)) const messageCount = computed(() => props.post.synthesis?.message_count ?? 0) +// A drop of one message stays a synthetic post (teaser matching only looks at +// drops), but there is nothing grouped in it — #4390: "Grouped from 1 Discord +// message" described a wrapper, not a grouping. Say what it is instead. const synthesisTitle = computed(() => { const n = messageCount.value if (!n) return 'Grouped from Discord' - return `Grouped from ${n} Discord message${n === 1 ? '' : 's'}` + if (n === 1) return 'Discord message' + return `Grouped from ${n} Discord messages` }) +const synthesisChip = computed(() => + messageCount.value === 1 ? 'from Discord' : 'grouped by FabledCurator' +) const hero = computed(() => images.value[0]) diff --git a/frontend/test/components/postCard.spec.js b/frontend/test/components/postCard.spec.js index b96f779..5bb2224 100644 --- a/frontend/test/components/postCard.spec.js +++ b/frontend/test/components/postCard.spec.js @@ -149,13 +149,16 @@ describe('PostCard', () => { expect(w.find('.fc-post-card__assoc').exists()).toBe(false) }) - it('singularises a one-message drop', () => { + it('does not call a one-message drop a grouping (#4390)', () => { const w = mountComponent(PostCard, { props: { post: { ...SYNTH, synthesis: { message_count: 1 } } }, pinia: freshPinia(), }) - expect(w.text()).toContain('Grouped from 1 Discord message') - expect(w.text()).not.toContain('1 Discord messages') + expect(w.text()).toContain('Discord message') + expect(w.text()).not.toContain('Grouped from') + // The marker stays — FC still wrote this post — but names the source. + expect(w.find('.fc-post-card__synthetic').text()).toContain('from Discord') + expect(w.text()).not.toContain('grouped by FabledCurator') }) }) -- 2.54.0 From 7cc34b57244eb8957b9289fac73230bb94fbb3ad Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 15:21:05 -0400 Subject: [PATCH 68/94] fix: a Discord message that re-posts a file is grouped, and joins the drop it repeats Grouping read a message's images through primary_post_id alone, which only the first message imported with a file holds. The backfill runs newest-first, so the original message usually owned nothing: 101 of Yellowroom's messages (mostly 2018-2020) could never be grouped. Every image they carried also sat in another message (296 links, measured on the live instance). _message_images unions ownership with provenance for the candidate query, the drop seed, the member image links, and the merge pass. Nothing is re-owned. A later drop carrying the very file an earlier one carries now merges by a new same_image route, which nearest-neighbour could not see because it skips the image itself. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/discord_grouping.py | 60 ++++++++++++++++++----- tests/test_discord_trickles.py | 61 ++++++++++++++++++++++++ 2 files changed, 110 insertions(+), 11 deletions(-) diff --git a/backend/app/services/discord_grouping.py b/backend/app/services/discord_grouping.py index 44afda3..b8e68ab 100644 --- a/backend/app/services/discord_grouping.py +++ b/backend/app/services/discord_grouping.py @@ -63,7 +63,7 @@ from collections import Counter from dataclasses import dataclass, field from datetime import UTC, datetime, timedelta -from sqlalchemy import Select, delete, func, select, update +from sqlalchemy import Select, delete, func, select, union, update from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession @@ -126,6 +126,31 @@ def cosine_distance(a, b) -> float: return 1.0 - (dot / (na * nb)) +def _message_images(posts): + """(post_id, image_id) for every image a message carries — owned AND re-posted. + + A message owns an image through `primary_post_id`, but only the FIRST + message imported with a given file does. The same file posted again is a + provenance link, and the backfill runs newest-first, so it is usually the + ORIGINAL message that ends up owning nothing. Reading ownership alone left + 101 of Yellowroom's messages (2018–2020 mostly) ungroupable: every image + they carried also sat in another message. + + `posts` is a list of ids or a select of them; filtering both branches by it + keeps the union to the messages in hand rather than the whole library. + Callers pass MESSAGE posts only — a drop's own provenance rows would read + as images it carries. + """ + owned = select( + ImageRecord.primary_post_id.label("post_id"), ImageRecord.id.label("image_id"), + ).where(ImageRecord.primary_post_id.in_(posts)) + reposted = select( + ImageProvenance.post_id.label("post_id"), + ImageProvenance.image_record_id.label("image_id"), + ).where(ImageProvenance.post_id.in_(posts)) + return union(owned, reposted).subquery() + + def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select: """Ungrouped Discord message-posts, one representative image each, OLDEST FIRST — which is the order `build_groups` requires. @@ -143,13 +168,15 @@ def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select: take the OLDEST candidates instead of the lowest-numbered ones. """ sort_key = func.coalesce(Post.post_date, Post.downloaded_at) + carried = _message_images(select(Post.id).where(Post.source_id == source_id)) inner = ( select( Post.id.label("post_id"), sort_key.label("occurred_at"), ImageRecord.siglip_embedding.label("embedding"), ) - .join(ImageRecord, ImageRecord.primary_post_id == Post.id) + .join(carried, carried.c.post_id == Post.id) + .join(ImageRecord, ImageRecord.id == carried.c.image_id) .where( Post.source_id == source_id, # Never absorb a post FC wrote, and never re-absorb one already @@ -295,9 +322,10 @@ async def _link_member_images( """ if not member_ids: return 0 - image_rows = (await session.execute( - select(ImageRecord.id).where(ImageRecord.primary_post_id.in_(member_ids)) - )).scalars().all() + carried = _message_images(member_ids) + image_rows = sorted(set((await session.execute( + select(carried.c.image_id) + )).scalars().all())) if not image_rows: return 0 await session.execute( @@ -421,9 +449,11 @@ async def _group_seed(session: AsyncSession, post_id: int) -> list[float] | None free to disagree; this way there is one. """ sort_key = func.coalesce(Post.post_date, Post.downloaded_at) + carried = _message_images(select(Post.id).where(Post.absorbed_by_post_id == post_id)) return (await session.execute( select(ImageRecord.siglip_embedding) - .join(Post, ImageRecord.primary_post_id == Post.id) + .join(carried, carried.c.image_id == ImageRecord.id) + .join(Post, Post.id == carried.c.post_id) .where( Post.absorbed_by_post_id == post_id, ImageRecord.siglip_embedding.is_not(None), @@ -725,12 +755,15 @@ async def _load_drops(session: AsyncSession, source: Source) -> list[_Drop]: names: dict[int, set[str]] = {pid: set() for pid in by_id} images: dict[int, list[int]] = {pid: [] for pid in by_id} if owner_of: - for iid, primary, path in (await session.execute( - select(ImageRecord.id, ImageRecord.primary_post_id, ImageRecord.path) - .where(ImageRecord.primary_post_id.in_(list(owner_of))) + carried = _message_images(list(owner_of)) + for iid, message, path in (await session.execute( + select(ImageRecord.id, carried.c.post_id, ImageRecord.path) + .join(carried, carried.c.image_id == ImageRecord.id) .order_by(ImageRecord.id) )).all(): - drop = owner_of[primary] + drop = owner_of[message] + if iid in images[drop]: + continue # one file carried by two of the drop's messages images[drop].append(iid) if (name := leading_name(path)) is not None: names[drop].add(name) @@ -844,7 +877,12 @@ async def merge_trickles( if drop.first_at - earlier.last_at > gap: continue shared = mine & gated(earlier.names) - if shared: + if set(drop.images) & set(earlier.images): + # The creator posted the very same file again — the strongest + # evidence there is, and one nearest-neighbour cannot see: it + # skips the image itself, which is the one they share. + targets[earlier.post.id] = (earlier, "same_image") + elif shared: targets[earlier.post.id] = (earlier, f"name:{min(shared)}") elif drop.nearest & earlier.members or (earlier.nearest or set()) & drop.members: targets[earlier.post.id] = (earlier, "nearest") diff --git a/tests/test_discord_trickles.py b/tests/test_discord_trickles.py index 76b47ad..513bce6 100644 --- a/tests/test_discord_trickles.py +++ b/tests/test_discord_trickles.py @@ -20,6 +20,7 @@ from sqlalchemy import select from backend.app.models import ( Artist, + ImageProvenance, ImageRecord, Post, PostAssociation, @@ -251,3 +252,63 @@ async def test_a_checked_drop_is_not_examined_again(db): for drop in await _drops(db, source): assert drop.synthesis_details["trickle_checked"] is True assert "nearest_message_ids" in drop.synthesis_details + + +# --- the same file posted twice ----------------------------------------------- +# 101 of Yellowroom's messages were never grouped: every image they carried also +# sat in another message, which the newest-first backfill had made its owner. + + +async def _repost(db, artist, source, *, at, of, text=None): + """A message carrying a file another message already owns.""" + n = next(_n) + post = Post(source_id=source.id, artist_id=artist.id, + external_post_id=f"msg-{n}", post_date=at, description=text) + db.add(post) + await db.flush() + image_id = (await db.execute( + select(ImageRecord.id).where(ImageRecord.primary_post_id == of.id) + )).scalar_one() + db.add(ImageProvenance(image_record_id=image_id, post_id=post.id, source_id=source.id)) + await db.flush() + return post + + +@pytest.mark.asyncio +async def test_a_message_that_only_reposts_an_image_is_still_grouped(db): + artist, source = await _seed(db, "repost-artist") + later = await _message(db, artist, source, at=T0, angle=0.0) + original = await _repost(db, artist, source, at=T0 - timedelta(days=400), of=later, + text="first posted here") + await db.commit() + + await group_source(db, source, max_distance=0.10, window_minutes=60) + await db.commit() + + await db.refresh(original) + assert original.absorbed_by_post_id is not None + # Grouping links, it never re-owns: the image keeps its primary post. + owner = (await db.execute( + select(ImageRecord.primary_post_id).join( + ImageProvenance, ImageProvenance.image_record_id == ImageRecord.id, + ).where(ImageProvenance.post_id == original.absorbed_by_post_id) + )).scalar_one() + assert owner == later.id + + +@pytest.mark.asyncio +async def test_the_same_file_posted_a_day_apart_is_one_drop(db): + """The live pair: 32554 (Aug 25) and 32553 (Aug 26) carry one file. + Nearest-neighbour cannot see it — it skips the image itself.""" + artist, source = await _seed(db, "twice-artist") + second = await _message(db, artist, source, at=T0 + timedelta(days=1), angle=0.0) + first = await _repost(db, artist, source, at=T0, of=second) + await db.commit() + + await _group_then_merge(db, source) + + (drop,) = await _drops(db, source) + assert set(drop.synthesis_details["member_post_ids"]) | { + m for r in drop.synthesis_details.get("merged", []) for m in r["message_ids"] + } >= {first.id, second.id} + assert drop.synthesis_details["merged"][0]["route"] == "same_image" -- 2.54.0 From ee59133b42adcca0f9f991c5f8d9e1d06e82b30e Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:03:38 -0400 Subject: [PATCH 69/94] fix: a new view opens at its top instead of the last view's scroll offset The router had no scrollBehavior, so Settings entered from a scrolled feed opened scrolled too, its heading and tab strip under the nav. Back/forward restores the saved position, and a new query on the same view keeps its place. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- frontend/src/router.js | 18 +++++++++++++++++- frontend/test/router.spec.js | 24 +++++++++++++++++++++++- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/frontend/src/router.js b/frontend/src/router.js index 00bd714..559af07 100644 --- a/frontend/src/router.js +++ b/frontend/src/router.js @@ -94,9 +94,25 @@ const routes = [ const history = typeof window !== 'undefined' ? createWebHistory() : createMemoryHistory() +// Where the window lands after a navigation. Without this the router keeps the +// window's scroll offset, so a view entered from halfway down another one opened +// halfway down itself, its heading and tab strip under the nav (operator-flagged +// 2026-09-24, Settings entered from a scrolled feed). +// - back/forward returns to where that entry was left; +// - the SAME view with a new query (a gallery filter, a Browse or +// Subscriptions tab) keeps its place — that is not a new page; +// - a new view opens at its top. +export function scrollBehavior(to, from, savedPosition) { + if (savedPosition) return savedPosition + if (from.matched.length && to.path === from.path) return false + if (to.hash) return { el: to.hash } + return { top: 0 } +} + const router = createRouter({ history, - routes + routes, + scrollBehavior }) const DEFAULT_TITLE = 'FabledCurator' diff --git a/frontend/test/router.spec.js b/frontend/test/router.spec.js index 44f3b28..fc0356a 100644 --- a/frontend/test/router.spec.js +++ b/frontend/test/router.spec.js @@ -1,5 +1,5 @@ import { describe, it, expect } from 'vitest' -import router, { FRONT_DOOR } from '../src/router.js' +import router, { FRONT_DOOR, scrollBehavior } from '../src/router.js' describe('router', () => { it('FRONT_DOOR is the post feed', () => { @@ -74,4 +74,26 @@ describe('router', () => { expect(r.name).toBe('series-read') expect(r.meta.immersive).toBe(true) }) + + describe('scrollBehavior', () => { + const at = (path) => ({ path, hash: '', matched: [{}] }) + + it('opens a new view at its top, not at the last view\'s offset', () => { + expect(scrollBehavior(at('/settings'), at('/latest'), null)).toEqual({ top: 0 }) + }) + + it('back and forward return to where the entry was left', () => { + const saved = { left: 0, top: 1234 } + expect(scrollBehavior(at('/latest'), at('/settings'), saved)).toBe(saved) + }) + + it('a new query on the same view keeps its place', () => { + expect(scrollBehavior(at('/gallery'), at('/gallery'), null)).toBe(false) + }) + + it('the first navigation of a page load opens at the top', () => { + const initial = { path: '/', hash: '', matched: [] } + expect(scrollBehavior(at('/latest'), initial, null)).toEqual({ top: 0 }) + }) + }) }) -- 2.54.0 From 31020d9395eca25f6ec291c5eec3be7cae96cce0 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:13:47 -0400 Subject: [PATCH 70/94] fix: a native chunk stops walking when its import work would overrun the task The walk's time budget covered the walk alone, but phase 3 runs in the same Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked for about two minutes and handed phase 3 431 orphan imports and ~3000 relinks. Phase 3 ran for 20 minutes and died at the soft limit (event 90808). This predates the worker consolidation; the limits are unchanged since June. The walk now also stops when elapsed time plus phase 3's estimated cost (2.5s per import, 0.25s per relink, measured on the live instance) passes CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so such a stop is a PARTIAL chunk boundary and the next chunk resumes the page. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/ingest_core.py | 36 +++++++++++++++++++++++++++-- tests/test_download_source_task.py | 9 ++++++++ tests/test_patreon_ingester.py | 24 +++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/backend/app/services/ingest_core.py b/backend/app/services/ingest_core.py index 8bb1e3d..6030cad 100644 --- a/backend/app/services/ingest_core.py +++ b/backend/app/services/ingest_core.py @@ -106,6 +106,22 @@ _LIVE_PROGRESS_INTERVAL = 5.0 # recapture (the operator's schema-test flow) reaches the sample. _CANARY_MIN_SAMPLE = 30 +# The walk's time budget covers only the walk, but phase 3 runs in the SAME +# Celery task, under the same soft limit (tasks/download.py: 1350s). A walk that +# finds a lot of work for phase 3 must stop early and leave it to the next chunk, +# or phase 3 is killed mid-import: TamadaHeijun's recapture, 2026-09-24, walked +# for ~2 min and then spent 20 min importing 431 orphans and relinking ~3000 +# on-disk files, and died at the soft limit. +# +# So the walk also stops when its elapsed time PLUS phase 3's estimated cost +# would pass CHUNK_TOTAL_SECONDS. Costs measured on the live instance: 431 +# imports took 976s (~2.3s each: hash, pHash, sidecar, provenance); a relink is +# a sha256 over NFS, 0.15s for the 8.8 MB average file, plus a lookup. +# test_download_source_task pins CHUNK_TOTAL_SECONDS under the soft limit. +CHUNK_TOTAL_SECONDS = 1200.0 +PHASE3_IMPORT_SECONDS = 2.5 +PHASE3_RELINK_SECONDS = 0.25 + def _parse_published(raw: object) -> datetime | None: """An ISO-8601 post date from either native client, as aware UTC. @@ -393,7 +409,17 @@ class Ingester: # Time-box check at the post boundary (coarse, like a gallery-dl # chunk). Backfill/recovery resume from emitted_cursor next chunk. - if time.monotonic() - start >= time_budget_seconds: + # The second half is phase 3's share of the task — see + # CHUNK_TOTAL_SECONDS. A mid-page stop resumes the same page. + elapsed = time.monotonic() - start + phase3 = ( + len(written) * PHASE3_IMPORT_SECONDS + + len(relink) * PHASE3_RELINK_SECONDS + ) + if ( + elapsed >= time_budget_seconds + or elapsed + phase3 >= CHUNK_TOTAL_SECONDS + ): budget_hit = True break @@ -711,7 +737,13 @@ class Ingester: # next chunk resumes from the emitted cursor. No progress → TIMEOUT, # which feeds download_service's backfill stall-guard. rc<0 mirrors # subprocess TimeoutExpired so completion detection stays false. - made_progress = downloaded > 0 or emitted_cursor != resume_cursor + # Work handed to phase 3 is progress too: a recapture chunk that + # stopped for its imports downloaded nothing, and may not have left + # its first page. + made_progress = ( + downloaded > 0 or bool(written) or bool(relink) + or emitted_cursor != resume_cursor + ) if made_progress: return _result( success=False, return_code=-1, diff --git a/tests/test_download_source_task.py b/tests/test_download_source_task.py index 22d59dd..1f50dd0 100644 --- a/tests/test_download_source_task.py +++ b/tests/test_download_source_task.py @@ -46,6 +46,15 @@ def test_timeout_ladder_keeps_subprocess_budgets_under_soft_limit(): assert DOWNLOAD_SOFT_TIME_LIMIT < DOWNLOAD_HARD_TIME_LIMIT +def test_the_native_chunk_leaves_room_to_tear_down_before_the_soft_limit(): + """The native walk sizes itself against CHUNK_TOTAL_SECONDS, walk plus + phase 3; that total has to leave the task time to finalize its event.""" + from backend.app.services.ingest_core import CHUNK_TOTAL_SECONDS + from backend.app.tasks.download import DOWNLOAD_SOFT_TIME_LIMIT + + assert CHUNK_TOTAL_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120 + + def test_decorated_limits_match_module_constants(): """The @celery.task decorator must use the audited constants, not drifted literals.""" diff --git a/tests/test_patreon_ingester.py b/tests/test_patreon_ingester.py index 47cf97f..1e344d7 100644 --- a/tests/test_patreon_ingester.py +++ b/tests/test_patreon_ingester.py @@ -846,6 +846,30 @@ async def test_recapture_does_not_refetch_seen_media_missing_from_disk( assert result.relink_source_paths == [] +@pytest.mark.asyncio +async def test_a_chunk_stops_walking_when_phase_3_would_overrun_the_task( + source_id, sync_engine, tmp_path, monkeypatch, +): + """Phase 3 shares the task's soft limit with the walk. TamadaHeijun's + recapture walked for two minutes and then died importing what it found. + With room for about one import, the walk takes two posts and stops.""" + import backend.app.services.ingest_core as core + monkeypatch.setattr(core, "CHUNK_TOTAL_SECONDS", core.PHASE3_IMPORT_SECONDS + 0.5) + + pages = [("CUR1", [(f"p{i}", [_media(f"p{i}", 1)]) for i in range(1, 4)])] + downloader = _FakeDownloader(tmp_path) + ing = _ingester(sync_engine, tmp_path, _FakeClient(pages), downloader) + result = ing.run( + source_id=source_id, campaign_id="c1", artist_slug="ingest", + url="https://patreon.com/ingest", mode="backfill", time_budget_seconds=600.0, + ) + + assert result.files_downloaded == 2 + assert downloader.download_calls == 2 # p3 left for the next chunk + assert result.error_type == ErrorType.PARTIAL # a chunk boundary, not a failure + assert result.cursor == "CUR1" # resumes the page it was cut on + + # --- a run that dies between download and import --------------------------- # TamadaHeijun's 【12PCG】 post, 2026-09-24: 13 files on disk, 5 in the library. # A run wrote 01–08, marked them seen, and was killed before phase 3 imported -- 2.54.0 From f2e4ee4d17c02d1e987453ed10f80fac4f7166eb Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:36:58 -0400 Subject: [PATCH 71/94] fix: beat takes each job's last run from task_run, so a redeploy no longer resets the schedule (4408) Beat's default scheduler kept its memory in a shelve file nothing persists, and a scheduler that remembers nothing waits a full interval before any job. Since the one-container image, every redeploy restarts beat, and no daily or weekly job had run since 2026-09-21 (cleanup, backup and download-event pruning, membership sync, thumbnail backfill, integrity check, vacuum). TaskRunScheduler seeds each entry's last_run_at at startup from task_run's newest start for that task: an overdue job runs at once, one not yet due waits the remainder, and one never recorded is due now. prune_task_runs now keeps each task's newest row however old, or a weekly job would look never-run a day after it ran and fire on every restart. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/beat_scheduler.py | 77 ++++++++++++++++++++++++++++++++ backend/app/celery_app.py | 3 ++ backend/app/tasks/maintenance.py | 7 +++ tests/test_beat_scheduler.py | 63 ++++++++++++++++++++++++++ tests/test_maintenance.py | 37 +++++++++++++++ 5 files changed, 187 insertions(+) create mode 100644 backend/app/beat_scheduler.py create mode 100644 tests/test_beat_scheduler.py diff --git a/backend/app/beat_scheduler.py b/backend/app/beat_scheduler.py new file mode 100644 index 0000000..5193163 --- /dev/null +++ b/backend/app/beat_scheduler.py @@ -0,0 +1,77 @@ +"""Celery beat that remembers when each job last ran — from task_run, not a file. + +Celery's default PersistentScheduler keeps its memory in a shelve file in the +working directory. Nothing mounts that directory, so every container recreate +forgets it, and a scheduler that remembers nothing seeds every entry with +`last_run_at = now`: each job waits a FULL interval after startup. A daily job +therefore needs 24 hours without a redeploy to fire. Since the one-container +image (172e33d) every redeploy restarts beat, and on 2026-09-24 no daily or +weekly job had run since the 21st (#4408). + +task_run already records every task that starts (celery_signals), indexed on +(task_name, started_at DESC), and prune_task_runs keeps the newest row of each +task however old it is. So on startup each entry takes its last_run_at from +there: + - a job that is overdue runs at once; + - a job that is not due waits only the remainder of its interval; + - a job that has never run is due now. + +Beat keeps last_run_at in memory from then on, as the default scheduler does; +only the startup seed changes. If the database cannot be read, the entries keep +Celery's own default rather than beat failing to start. +""" + +from __future__ import annotations + +import logging +from datetime import UTC, datetime + +from celery.beat import Scheduler +from sqlalchemy import func, select + +log = logging.getLogger(__name__) + +# Seed for a job with no recorded run: far enough back that any interval or +# crontab reads as due. +NEVER = datetime(2000, 1, 1, tzinfo=UTC) + + +def last_runs(session, task_names: list[str]) -> dict[str, datetime]: + """The latest recorded start of each task, by task name.""" + from .models import TaskRun + + if not task_names: + return {} + rows = session.execute( + select(TaskRun.task_name, func.max(TaskRun.started_at)) + .where(TaskRun.task_name.in_(sorted(set(task_names)))) + .group_by(TaskRun.task_name) + ).all() + return dict(rows) + + +def seed(entries, last: dict[str, datetime]) -> None: + """Set each entry's last_run_at from `last`; a task with none is due now.""" + for entry in entries: + entry.last_run_at = last.get(entry.task, NEVER) + + +class TaskRunScheduler(Scheduler): + """An in-memory beat seeded from task_run history at startup.""" + + def setup_schedule(self): + super().setup_schedule() + try: + from .tasks._sync_engine import sync_session_factory + + with sync_session_factory()() as session: + last = last_runs(session, [e.task for e in self.schedule.values()]) + except Exception: + log.exception("beat: could not read task_run; every job waits a full interval") + return + seed(self.schedule.values(), last) + due = sum(1 for e in self.schedule.values() if e.is_due()[0]) + log.info( + "beat: seeded %d job(s) from task_run, %d due now", + len(self.schedule), due, + ) diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index d341530..5f45567 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -353,6 +353,9 @@ def make_celery() -> Celery: }, }, timezone="UTC", + # Beat's memory of when each job last ran comes from task_run, not a + # shelve file nothing persists — see beat_scheduler (#4408). + beat_scheduler="backend.app.beat_scheduler:TaskRunScheduler", ) # FC-3i: register task_run signal handlers (side-effect import). from . import celery_signals # noqa: F401 diff --git a/backend/app/tasks/maintenance.py b/backend/app/tasks/maintenance.py index 8e41efd..8b653de 100644 --- a/backend/app/tasks/maintenance.py +++ b/backend/app/tasks/maintenance.py @@ -473,6 +473,10 @@ def prune_task_runs() -> dict: (recover_stalled_task_runs) is the mechanism that flips them to terminal state; prune doesn't touch in-flight state. - 'retry' rows: treated as failures (>7d). + - The NEWEST row of each task is never deleted, whatever its age: it is + what the beat scheduler reads to know when a job last ran (#4408). + Without it a weekly job's last run would be pruned after a day, and beat + would think it had never run and fire it on every restart. Returns dict of how many rows were deleted in each bucket. """ @@ -480,16 +484,19 @@ def prune_task_runs() -> dict: now = datetime.now(UTC) ok_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_OK_SECONDS) fail_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_FAILURE_SECONDS) + newest = select(func.max(TaskRun.id)).group_by(TaskRun.task_name) with SessionLocal() as session: ok_deleted = session.execute( delete(TaskRun) .where(TaskRun.status == "ok") .where(TaskRun.finished_at < ok_cutoff) + .where(TaskRun.id.not_in(newest)) ).rowcount or 0 fail_deleted = session.execute( delete(TaskRun) .where(TaskRun.status.in_(["error", "timeout", "retry"])) .where(TaskRun.finished_at < fail_cutoff) + .where(TaskRun.id.not_in(newest)) ).rowcount or 0 session.commit() return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted} diff --git a/tests/test_beat_scheduler.py b/tests/test_beat_scheduler.py new file mode 100644 index 0000000..5224585 --- /dev/null +++ b/tests/test_beat_scheduler.py @@ -0,0 +1,63 @@ +"""#4408: beat takes each job's last run from task_run, not a shelve file. + +The default scheduler forgot everything on each container recreate and waited +a full interval before any job, so with several redeploys a day no daily or +weekly job had run since 2026-09-21. +""" +from datetime import UTC, datetime, timedelta + +import pytest +from celery.beat import ScheduleEntry +from celery.schedules import schedule + +from backend.app.beat_scheduler import NEVER, last_runs, seed +from backend.app.celery_app import celery +from backend.app.models import TaskRun + +DAY = 86400.0 + + +def _entry(task, every=DAY): + return ScheduleEntry(name=task, task=task, schedule=schedule(every, app=celery), app=celery) + + +def test_a_job_that_ran_recently_waits_only_the_remainder(): + entry = _entry("t.daily") + seed([entry], {"t.daily": datetime.now(UTC) - timedelta(hours=20)}) + due, next_in = entry.is_due() + assert not due + assert 3 * 3600 < next_in <= 4 * 3600 + 5 + + +def test_an_overdue_job_is_due_at_once(): + """The live case: daily jobs last ran three days before the restart.""" + entry = _entry("t.daily") + seed([entry], {"t.daily": datetime.now(UTC) - timedelta(days=3)}) + assert entry.is_due()[0] + + +def test_a_job_with_no_recorded_run_is_due_now(): + entry = _entry("t.never") + seed([entry], {}) + assert entry.last_run_at == NEVER + assert entry.is_due()[0] + + +def test_the_scheduler_is_the_one_celery_uses(): + assert celery.conf.beat_scheduler == "backend.app.beat_scheduler:TaskRunScheduler" + + +@pytest.mark.integration +def test_last_runs_reads_the_newest_start_per_task(db_sync): + now = datetime.now(UTC) + for name, ago in [("t.a", 30), ("t.a", 2), ("t.b", 5)]: + db_sync.add(TaskRun( + celery_task_id="x", queue="maintenance", task_name=name, + started_at=now - timedelta(hours=ago), status="ok", + )) + db_sync.flush() + + got = last_runs(db_sync, ["t.a", "t.b", "t.c"]) + + assert set(got) == {"t.a", "t.b"} + assert abs((got["t.a"] - (now - timedelta(hours=2))).total_seconds()) < 1 diff --git a/tests/test_maintenance.py b/tests/test_maintenance.py index 5f7e3ed..74030af 100644 --- a/tests/test_maintenance.py +++ b/tests/test_maintenance.py @@ -569,6 +569,12 @@ def test_prune_task_runs_deletes_failures_older_than_7d(db_sync): started_at=now - timedelta(days=10), finished_at=now - timedelta(days=9), ) + # A later run of the same task, so the old failure is not its newest row + # (the newest is kept for beat — see the test below). + _make_task_run( + db_sync, status="ok", + started_at=now - timedelta(hours=2), finished_at=now - timedelta(hours=1), + ) db_sync.commit() result = prune_task_runs.apply().get() @@ -581,6 +587,37 @@ def test_prune_task_runs_deletes_failures_older_than_7d(db_sync): assert surviving is None +def test_prune_task_runs_keeps_each_tasks_newest_row_however_old(db_sync): + """#4408: beat reads a job's last run from task_run. A weekly job's only + row is older than the 24h ok-retention, and pruning it would make beat + think the job never ran and fire it on every restart.""" + from sqlalchemy import select + + from backend.app.models import TaskRun + from backend.app.tasks.maintenance import prune_task_runs + + now = datetime.now(UTC) + weekly = "backend.app.tasks.fake.weekly" + older = _make_task_run( + db_sync, status="ok", task_name=weekly, + started_at=now - timedelta(days=14), finished_at=now - timedelta(days=14), + ) + newest = _make_task_run( + db_sync, status="ok", task_name=weekly, + started_at=now - timedelta(days=7), finished_at=now - timedelta(days=7), + ) + db_sync.commit() + + prune_task_runs.apply().get() + + db_sync.expire_all() + surviving = set(db_sync.execute( + select(TaskRun.id).where(TaskRun.task_name == weekly) + ).scalars().all()) + assert surviving == {newest} + assert older not in surviving + + def test_prune_task_runs_keeps_recent_failures(db_sync): from sqlalchemy import select -- 2.54.0 From 0842df46e92b1b9ab8ecafd50d6d447386e2fc18 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:37:49 -0400 Subject: [PATCH 72/94] refactor: components read the obsidian surface from its token, not a typed rgba (3108) Thirteen rgba(20, 23, 26, ...) across six components (the task counted eleven). They now use --v-theme-background, which Vuetify maps to obsidian; ArtistHeader's banner fade uses --fc-chrome-rgb, since it is nav-style chrome. A spec fails if the literal reappears outside a comment. Values are unchanged, so nothing should render differently. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../src/components/artist/ArtistHeader.vue | 6 +-- .../components/gallery/GalleryFilterBar.vue | 2 +- frontend/src/components/modal/ImageViewer.vue | 6 +-- frontend/src/views/ExploreView.vue | 2 +- frontend/src/views/SeriesReaderView.vue | 4 +- frontend/src/views/SeriesView.vue | 6 +-- frontend/test/paletteLiterals.spec.js | 40 +++++++++++++++++++ 7 files changed, 53 insertions(+), 13 deletions(-) create mode 100644 frontend/test/paletteLiterals.spec.js diff --git a/frontend/src/components/artist/ArtistHeader.vue b/frontend/src/components/artist/ArtistHeader.vue index 2c63210..992622a 100644 --- a/frontend/src/components/artist/ArtistHeader.vue +++ b/frontend/src/components/artist/ArtistHeader.vue @@ -103,9 +103,9 @@ const stats = computed(() => { padding: 0.5rem 1rem; background: linear-gradient( to bottom, - rgba(20, 23, 26, 0.92) 0%, - rgba(20, 23, 26, 0.65) 60%, - rgba(20, 23, 26, 0) 100% + rgba(var(--fc-chrome-rgb), 0.92) 0%, + rgba(var(--fc-chrome-rgb), 0.65) 60%, + rgba(var(--fc-chrome-rgb), 0) 100% ); backdrop-filter: blur(2px); -webkit-backdrop-filter: blur(2px); diff --git a/frontend/src/components/gallery/GalleryFilterBar.vue b/frontend/src/components/gallery/GalleryFilterBar.vue index 0c5dc39..7217b2b 100644 --- a/frontend/src/components/gallery/GalleryFilterBar.vue +++ b/frontend/src/components/gallery/GalleryFilterBar.vue @@ -329,7 +329,7 @@ function pushFilter(mutate) { opaque than the bar/nav so the controls stay legible. */ .fc-filterbar-wrap :deep(.v-field), .fc-filterbar-wrap :deep(.v-btn-group) { - background-color: rgba(20, 23, 26, 0.72); + background-color: rgba(var(--v-theme-background), 0.72); } /* Media toggle (All / Images / Videos) as ONE cohesive segmented control. FC's global VBtn { rounded: 'pill' } default made Vuetify 4 pill-round each diff --git a/frontend/src/components/modal/ImageViewer.vue b/frontend/src/components/modal/ImageViewer.vue index 1ff8d69..f3db735 100644 --- a/frontend/src/components/modal/ImageViewer.vue +++ b/frontend/src/components/modal/ImageViewer.vue @@ -213,7 +213,7 @@ function nextFrame() { --fc-side-w: 320px; /* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav, mid-opacity + blur so the page behind shows through faintly. */ - background: rgba(20, 23, 26, 0.65); + background: rgba(var(--v-theme-background), 0.65); backdrop-filter: blur(8px); -webkit-backdrop-filter: blur(8px); outline: none; @@ -221,7 +221,7 @@ function nextFrame() { } .fc-viewer__close, .fc-viewer__nav { position: absolute; top: 50%; - background: rgba(20, 23, 26, 0.7); + background: rgba(var(--v-theme-background), 0.7); color: rgb(var(--v-theme-parchment, 232 228 216)); border: 1px solid rgb(var(--v-theme-surface-light)); border-radius: 50%; @@ -359,7 +359,7 @@ function nextFrame() { z-index: 1; /* Opaque obsidian so the scrolling panel never bleeds through the haze behind the pinned image. */ - background: rgb(20, 23, 26); + background: rgb(var(--v-theme-background)); } .fc-viewer__side { width: 100%; diff --git a/frontend/src/views/ExploreView.vue b/frontend/src/views/ExploreView.vue index 611c95d..0e99410 100644 --- a/frontend/src/views/ExploreView.vue +++ b/frontend/src/views/ExploreView.vue @@ -369,7 +369,7 @@ onUnmounted(() => { /* Center viewer. */ .fc-ex__viewer { - background: rgb(20, 23, 26); + background: rgb(var(--v-theme-background)); display: flex; flex-direction: column; min-width: 0; min-height: 0; } .fc-ex__canvas { flex: 1 1 auto; display: flex; min-height: 0; min-width: 0; } diff --git a/frontend/src/views/SeriesReaderView.vue b/frontend/src/views/SeriesReaderView.vue index a00d1d0..67d2d53 100644 --- a/frontend/src/views/SeriesReaderView.vue +++ b/frontend/src/views/SeriesReaderView.vue @@ -276,7 +276,7 @@ onUnmounted(() => { .fc-reader__thumb img { width: 100%; height: auto; display: block; } .fc-reader__thumbnum { position: absolute; bottom: 4px; right: 4px; - background: rgba(20, 23, 26, 0.8); color: rgb(var(--v-theme-on-surface)); + background: rgba(var(--v-theme-background), 0.8); color: rgb(var(--v-theme-on-surface)); padding: 1px 5px; border-radius: 4px; font-size: 0.7rem; } .fc-reader__content { @@ -302,7 +302,7 @@ onUnmounted(() => { .fc-reader__indicator { position: fixed; bottom: 1.5rem; right: 1.5rem; padding: 4px 10px; border-radius: 999px; - background: rgba(20, 23, 26, 0.6); + background: rgba(var(--v-theme-background), 0.6); border: 1px solid rgb(var(--v-theme-surface-light)); font-size: 0.8rem; color: rgb(var(--v-theme-on-surface-variant)); z-index: 100; pointer-events: none; diff --git a/frontend/src/views/SeriesView.vue b/frontend/src/views/SeriesView.vue index f69f74b..dd814cb 100644 --- a/frontend/src/views/SeriesView.vue +++ b/frontend/src/views/SeriesView.vue @@ -338,18 +338,18 @@ onMounted(() => { position: absolute; top: 6px; left: 6px; display: inline-flex; align-items: center; gap: 2px; padding: 1px 6px; border-radius: 999px; font-size: 11px; - background: rgba(20, 23, 26, 0.75); + background: rgba(var(--v-theme-background), 0.75); color: rgb(var(--v-theme-warning, var(--v-theme-accent))); } /* Kebab sits top-right of the cover, opposite the gap badge. Tinted backing so it stays legible over any cover image. */ .fc-sbcard__kebab { position: absolute; top: 4px; right: 4px; - background: rgba(20, 23, 26, 0.6) !important; + background: rgba(var(--v-theme-background), 0.6) !important; border-radius: 50%; color: rgb(var(--v-theme-on-surface)); } -.fc-sbcard__kebab:hover { background: rgba(20, 23, 26, 0.85) !important; } +.fc-sbcard__kebab:hover { background: rgba(var(--v-theme-background), 0.85) !important; } .fc-sbcard__body { padding: 8px 10px; } .fc-sbcard__name { font-family: 'Fraunces', Georgia, serif; font-size: 15px; font-weight: 600; diff --git a/frontend/test/paletteLiterals.spec.js b/frontend/test/paletteLiterals.spec.js new file mode 100644 index 0000000..8b80eb2 --- /dev/null +++ b/frontend/test/paletteLiterals.spec.js @@ -0,0 +1,40 @@ +// The obsidian surface written out as a number instead of read from its token. +// +// fabled-tokens.js is the palette's single copy ("do not hand-edit hexes; +// re-mirror here"). Thirteen hand-typed `rgba(20, 23, 26, …)` across six +// components meant a changed obsidian would miss most of the UI (#3108). +// Components read `--v-theme-background` (Vuetify maps background → obsidian) +// or, for nav-style chrome, `--fc-chrome-rgb`. Comments may still name it. + +import { readdirSync, readFileSync, statSync } from 'node:fs' +import { join, relative } from 'node:path' +import { fileURLToPath } from 'node:url' + +import { describe, expect, it } from 'vitest' + +const SRC = fileURLToPath(new URL('../src', import.meta.url)) +const LITERAL = /rgba?\(\s*20\s*,\s*23\s*,\s*26\b/ + +function vueFiles (dir) { + return readdirSync(dir).flatMap((name) => { + const path = join(dir, name) + if (statSync(path).isDirectory()) return vueFiles(path) + return name.endsWith('.vue') ? [path] : [] + }) +} + +function stripComments (text) { + return text.replace(/\/\*[\s\S]*?\*\//g, '').replace(//g, '') +} + +describe('palette literals', () => { + it('no component hand-types the obsidian surface', () => { + const offenders = vueFiles(SRC).flatMap((path) => + stripComments(readFileSync(path, 'utf8')) + .split('\n') + .filter((line) => LITERAL.test(line)) + .map((line) => `${relative(SRC, path)}: ${line.trim()}`) + ) + expect(offenders).toEqual([]) + }) +}) -- 2.54.0 From ff70f837d0e79cbb6285749171bb54152c94a638 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:39:38 -0400 Subject: [PATCH 73/94] refactor: test row factories and the fetch stub have one copy each (3109) tests/factories.py holds image_row/make_image/make_image_async/make_tag. The 17 byte-identical _img/_tag helpers (15 modules) now import them under their old names, so no call site changed. frontend/test/support/stubFetch.js replaces 15 copies that differed only in formatting. Copies whose bodies differ (other defaults, other columns, a url-only stub) are left as they are; folding those needs a look at each caller. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- frontend/test/adminStore.spec.js | 13 +------ frontend/test/credentials.spec.js | 12 +------ frontend/test/dbMaintenance.spec.js | 13 +------ frontend/test/downloads.spec.js | 12 +------ frontend/test/gallery.spec.js | 13 +------ frontend/test/gallerySelection.spec.js | 13 +------ frontend/test/platforms.spec.js | 12 +------ frontend/test/provenance.spec.js | 13 +------ frontend/test/seriesManage.spec.js | 12 +------ frontend/test/seriesReader.spec.js | 12 +------ frontend/test/sources.spec.js | 12 +------ frontend/test/suggestions.spec.js | 13 +------ frontend/test/support/stubFetch.js | 16 +++++++++ frontend/test/tagDirectory.spec.js | 13 +------ frontend/test/tags.spec.js | 13 +------ frontend/test/workerLanes.spec.js | 13 +------ tests/factories.py | 47 ++++++++++++++++++++++++++ tests/test_api_ccip.py | 11 +----- tests/test_api_gpu.py | 11 +----- tests/test_api_tag_stats.py | 11 +----- tests/test_ccip.py | 11 +----- tests/test_character_prototypes.py | 12 +------ tests/test_gpu_jobs.py | 11 +----- tests/test_head_auto_apply.py | 12 +------ tests/test_head_incremental.py | 20 ++--------- tests/test_head_metrics.py | 11 +----- tests/test_head_positive_sources.py | 20 ++--------- tests/test_ml_suggestions.py | 12 +------ tests/test_presentation_auto_apply.py | 12 +------ tests/test_process_auto_apply.py | 12 +------ tests/test_regions.py | 11 +----- tests/test_suggestions_bulk.py | 12 +------ 32 files changed, 95 insertions(+), 346 deletions(-) create mode 100644 frontend/test/support/stubFetch.js create mode 100644 tests/factories.py diff --git a/frontend/test/adminStore.spec.js b/frontend/test/adminStore.spec.js index 2dcbce8..dcfb0c4 100644 --- a/frontend/test/adminStore.spec.js +++ b/frontend/test/adminStore.spec.js @@ -1,23 +1,12 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useAdminStore } from '../src/stores/admin.js' +import { stubFetch } from './support/stubFetch.js' // Covers the two helpers the admin store actions route through (DRY Finding C, // #753): _dryRunPost (URL + dry_run body, sourceId→source_id) and _guard // (lastError capture + rethrow). The store had no frontend test before. -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} - function lastCallBody(calls) { return JSON.parse(calls.at(-1).init.body) } diff --git a/frontend/test/credentials.spec.js b/frontend/test/credentials.spec.js index 5b8d8fb..b49974d 100644 --- a/frontend/test/credentials.spec.js +++ b/frontend/test/credentials.spec.js @@ -1,17 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useCredentialsStore } from '../src/stores/credentials.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('credentials store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/dbMaintenance.spec.js b/frontend/test/dbMaintenance.spec.js index f6b36e9..55e871e 100644 --- a/frontend/test/dbMaintenance.spec.js +++ b/frontend/test/dbMaintenance.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useDbMaintenanceStore } from '../src/stores/dbMaintenance.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('dbMaintenance store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/downloads.spec.js b/frontend/test/downloads.spec.js index a76fa90..62f870d 100644 --- a/frontend/test/downloads.spec.js +++ b/frontend/test/downloads.spec.js @@ -1,17 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useDownloadsStore } from '../src/stores/downloads.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('downloads store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/gallery.spec.js b/frontend/test/gallery.spec.js index 1946e8e..3c73bcd 100644 --- a/frontend/test/gallery.spec.js +++ b/frontend/test/gallery.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { cloneFilter, filterToQuery, useGalleryStore } from '../src/stores/gallery.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' const EMPTY = { images: [], date_groups: [], next_cursor: null } diff --git a/frontend/test/gallerySelection.spec.js b/frontend/test/gallerySelection.spec.js index 7641692..6139698 100644 --- a/frontend/test/gallerySelection.spec.js +++ b/frontend/test/gallerySelection.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useGallerySelectionStore } from '../src/stores/gallerySelection.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)) - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('gallerySelection store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/platforms.spec.js b/frontend/test/platforms.spec.js index 6667673..6b77869 100644 --- a/frontend/test/platforms.spec.js +++ b/frontend/test/platforms.spec.js @@ -1,17 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { usePlatformsStore } from '../src/stores/platforms.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('platforms store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/provenance.spec.js b/frontend/test/provenance.spec.js index c5142d6..ad738e1 100644 --- a/frontend/test/provenance.spec.js +++ b/frontend/test/provenance.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useProvenanceStore } from '../src/stores/provenance.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)) - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('provenance store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/seriesManage.spec.js b/frontend/test/seriesManage.spec.js index 8a19744..a9f05da 100644 --- a/frontend/test/seriesManage.spec.js +++ b/frontend/test/seriesManage.spec.js @@ -1,17 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useSeriesManageStore, moveItem } from '../src/stores/seriesManage.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)) - } - }) -} +import { stubFetch } from './support/stubFetch.js' // FC-6.x: a flat page run + cosmetic chapter dividers. const SERIES_BODY = { diff --git a/frontend/test/seriesReader.spec.js b/frontend/test/seriesReader.spec.js index de4d61e..cce2434 100644 --- a/frontend/test/seriesReader.spec.js +++ b/frontend/test/seriesReader.spec.js @@ -6,17 +6,7 @@ import { progressPct, clampPage } from '../src/stores/seriesReader.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)) - } - }) -} +import { stubFetch } from './support/stubFetch.js' const M = [ { page_number: 1, top: 0, height: 100 }, diff --git a/frontend/test/sources.spec.js b/frontend/test/sources.spec.js index 2570c76..ae87f87 100644 --- a/frontend/test/sources.spec.js +++ b/frontend/test/sources.spec.js @@ -1,17 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useSourcesStore } from '../src/stores/sources.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('sources store', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/suggestions.spec.js b/frontend/test/suggestions.spec.js index 69e3dc2..8bdb10f 100644 --- a/frontend/test/suggestions.spec.js +++ b/frontend/test/suggestions.spec.js @@ -1,21 +1,10 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useSuggestionsStore } from '../src/stores/suggestions.js' +import { stubFetch } from './support/stubFetch.js' vi.mock('../src/utils/toast.js', () => ({ toast: vi.fn() })) -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} - // Every suggestion is a canonical DB tag now (tagging-v2): a real id, flagged // above/below its head's suggest threshold. No raw / creates-new / alias cases. const sugg = (over = {}) => ({ diff --git a/frontend/test/support/stubFetch.js b/frontend/test/support/stubFetch.js new file mode 100644 index 0000000..272b56a --- /dev/null +++ b/frontend/test/support/stubFetch.js @@ -0,0 +1,16 @@ +// Replace globalThis.fetch with a stub answering from `handler(url, init)`, +// which returns { status, body }. Fifteen specs carried their own copy of this +// (#3109); a response shape change now has one place to go. +import { vi } from 'vitest' + +export function stubFetch (handler) { + globalThis.fetch = vi.fn(async (url, init) => { + const { status, body } = handler(url, init) + return { + ok: status >= 200 && status < 300, + status, + statusText: String(status), + text: async () => (body == null ? '' : JSON.stringify(body)) + } + }) +} diff --git a/frontend/test/tagDirectory.spec.js b/frontend/test/tagDirectory.spec.js index 8fc44e2..e9487d3 100644 --- a/frontend/test/tagDirectory.spec.js +++ b/frontend/test/tagDirectory.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useTagDirectoryStore } from '../src/stores/tagDirectory.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)) - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('tagDirectory store: rename / merge', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/tags.spec.js b/frontend/test/tags.spec.js index acb7658..1d1d6f3 100644 --- a/frontend/test/tags.spec.js +++ b/frontend/test/tags.spec.js @@ -1,18 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { useTagStore } from '../src/stores/tags.js' - -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} +import { stubFetch } from './support/stubFetch.js' describe('tags store: setFandom', () => { beforeEach(() => setActivePinia(createPinia())) diff --git a/frontend/test/workerLanes.spec.js b/frontend/test/workerLanes.spec.js index 22981de..2d4c74c 100644 --- a/frontend/test/workerLanes.spec.js +++ b/frontend/test/workerLanes.spec.js @@ -1,6 +1,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { setActivePinia, createPinia } from 'pinia' import { laneStuckFor, useSystemActivityStore } from '../src/stores/systemActivity.js' +import { stubFetch } from './support/stubFetch.js' // Milestone 422 step 4. Covers the store half of the worker-lane dial — the // part that decides what the card can tell the operator. @@ -18,18 +19,6 @@ import { laneStuckFor, useSystemActivityStore } from '../src/stores/systemActivi // as one. A control that silently does nothing is worse than one that // refuses out loud. -function stubFetch(handler) { - globalThis.fetch = vi.fn(async (url, init) => { - const { status, body } = handler(url, init) - return { - ok: status >= 200 && status < 300, - status, - statusText: String(status), - text: async () => (body == null ? '' : JSON.stringify(body)), - } - }) -} - const LANES_BODY = { lanes: [ { diff --git a/tests/factories.py b/tests/factories.py new file mode 100644 index 0000000..35e958d --- /dev/null +++ b/tests/factories.py @@ -0,0 +1,47 @@ +"""Throwaway rows for tests: one copy of the placeholder columns (#3109). + +`ImageRecord` needs a unique path and sha256 plus six NOT NULL columns no test +cares about. Fifteen modules re-typed the same builder, so a new NOT NULL column +meant fifteen separate patches. The tests that need these rows import them from +here, usually under their old local name (`make_image as _img`), so their call +sites did not change. +""" + +from backend.app.models import ImageRecord, Tag, TagKind + + +def image_row(sha: str, emb=None, **overrides) -> ImageRecord: + """An unsaved ImageRecord keyed by `sha`; `emb` is its SigLIP embedding.""" + fields = { + "path": f"/images/{sha}.jpg", "sha256": sha, "size_bytes": 1, + "mime": "image/jpeg", "width": 1, "height": 1, + "origin": "imported_filesystem", "integrity_status": "unknown", + } + if emb is not None: + fields["siglip_embedding"] = emb + fields.update(overrides) + return ImageRecord(**fields) + + +def make_image(db, sha: str, emb=None, **overrides) -> ImageRecord: + """image_row, added and flushed on a sync session.""" + img = image_row(sha, emb, **overrides) + db.add(img) + db.flush() + return img + + +async def make_image_async(db, sha: str, emb=None, **overrides) -> ImageRecord: + """image_row, added and flushed on an async session.""" + img = image_row(sha, emb, **overrides) + db.add(img) + await db.flush() + return img + + +def make_tag(db, name: str, **overrides) -> Tag: + """A general tag, added and flushed on a sync session.""" + tag = Tag(**{"name": name, "kind": TagKind.general, **overrides}) + db.add(tag) + db.flush() + return tag diff --git a/tests/test_api_ccip.py b/tests/test_api_ccip.py index d41167a..af6aad5 100644 --- a/tests/test_api_ccip.py +++ b/tests/test_api_ccip.py @@ -4,6 +4,7 @@ import pytest from backend.app.models import ImageRecord, ImageRegion, TagKind from backend.app.models.tag import image_tag from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration @@ -14,16 +15,6 @@ def _ccip(slot: int) -> list[float]: return v -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - async def _figure(db, image_id, ccip): db.add(ImageRegion( image_record_id=image_id, kind="figure", rx=0.0, ry=0.0, rw=1.0, rh=1.0, diff --git a/tests/test_api_gpu.py b/tests/test_api_gpu.py index e36c490..166f743 100644 --- a/tests/test_api_gpu.py +++ b/tests/test_api_gpu.py @@ -7,20 +7,11 @@ from sqlalchemy import func, select from backend.app.models import GpuJob, ImageRecord from backend.app.services.ml.gpu_jobs import GpuJobService from backend.app.services.ml.regions import RegionService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - @pytest.mark.asyncio async def test_agent_endpoints_require_bearer(client, db): resp = await client.post("/api/gpu/jobs/lease", json={"agent_id": "a1"}) diff --git a/tests/test_api_tag_stats.py b/tests/test_api_tag_stats.py index 67900b6..81004dc 100644 --- a/tests/test_api_tag_stats.py +++ b/tests/test_api_tag_stats.py @@ -5,20 +5,11 @@ from backend.app.models import ImageRecord, TagHead, TagKind from backend.app.models.tag import image_tag from backend.app.models.tag_suggestion_rejection import TagSuggestionRejection from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - async def _apply(db, image_id, tag_id, source): await db.execute(image_tag.insert().values( image_record_id=image_id, tag_id=tag_id, source=source, diff --git a/tests/test_ccip.py b/tests/test_ccip.py index df52378..52d7b40 100644 --- a/tests/test_ccip.py +++ b/tests/test_ccip.py @@ -13,6 +13,7 @@ from backend.app.models import ( from backend.app.models.tag import image_tag from backend.app.services.ml.ccip import match_image from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration @@ -23,16 +24,6 @@ def _ccip(slot: int) -> list[float]: return v -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - async def _figure(db, image_id, ccip): db.add(ImageRegion( image_record_id=image_id, kind="figure", diff --git a/tests/test_character_prototypes.py b/tests/test_character_prototypes.py index be5bd04..3e32763 100644 --- a/tests/test_character_prototypes.py +++ b/tests/test_character_prototypes.py @@ -18,6 +18,7 @@ from backend.app.models.tag import image_tag from backend.app.services.ml.character_prototypes import ( refresh_character_prototypes, ) +from tests.factories import make_image as _img pytestmark = pytest.mark.integration @@ -28,17 +29,6 @@ def _ccip(slot: int = 0) -> list[float]: return v -def _img(db, sha: str) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", - ) - db.add(img) - db.flush() - return img - - def _figure(db, image_id: int, ccip=None) -> None: db.add(ImageRegion( image_record_id=image_id, kind="figure", diff --git a/tests/test_gpu_jobs.py b/tests/test_gpu_jobs.py index 55bcfd6..09f7f41 100644 --- a/tests/test_gpu_jobs.py +++ b/tests/test_gpu_jobs.py @@ -10,20 +10,11 @@ from backend.app.services.ml.gpu_jobs import ( PENDING_POISON_CAP, GpuJobService, ) +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - @pytest.mark.asyncio async def test_enqueue_siglip_backfill_gates_on_concept_region(db): # 'siglip' backfill enqueues images that lack a concept region (the diff --git a/tests/test_head_auto_apply.py b/tests/test_head_auto_apply.py index 0d345e0..b9b5b67 100644 --- a/tests/test_head_auto_apply.py +++ b/tests/test_head_auto_apply.py @@ -14,6 +14,7 @@ from backend.app.models import ( ) from backend.app.models.tag import image_tag from backend.app.services.ml.heads import auto_apply_sweep +from tests.factories import make_image as _img pytestmark = pytest.mark.integration @@ -24,17 +25,6 @@ def _emb(slot: int) -> list[float]: return v -def _img(db, sha: str, emb) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", siglip_embedding=emb, - ) - db.add(img) - db.flush() - return img - - def _head(db, tag_id: int, slot: int, *, threshold=0.5, n_pos=60): s = db.execute(select(MLSettings).where(MLSettings.id == 1)).scalar_one() w = [0.0] * 1152 diff --git a/tests/test_head_incremental.py b/tests/test_head_incremental.py index b16b9dd..b1294a5 100644 --- a/tests/test_head_incremental.py +++ b/tests/test_head_incremental.py @@ -17,28 +17,12 @@ from backend.app.services.ml.heads import ( _head_fingerprints, _heads_needing_retrain, ) +from tests.factories import make_image as _img +from tests.factories import make_tag as _tag pytestmark = pytest.mark.integration -def _img(db, sha: str) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", - ) - db.add(img) - db.flush() - return img - - -def _tag(db, name: str) -> Tag: - t = Tag(name=name, kind=TagKind.general) - db.add(t) - db.flush() - return t - - def _apply(db, image_id: int, tag_id: int) -> None: db.execute(image_tag.insert().values( image_record_id=image_id, tag_id=tag_id, source="manual", diff --git a/tests/test_head_metrics.py b/tests/test_head_metrics.py index 189d197..e7956a2 100644 --- a/tests/test_head_metrics.py +++ b/tests/test_head_metrics.py @@ -6,20 +6,11 @@ from sqlalchemy import select from backend.app.models import HeadMetric, HeadMetricsSnapshot, ImageRecord, TagHead, TagKind from backend.app.models.tag import image_tag from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - def _head(tag_id): return TagHead( tag_id=tag_id, embedding_version="siglip-test", weights=[0.0] * 1152, diff --git a/tests/test_head_positive_sources.py b/tests/test_head_positive_sources.py index c26039e..f864705 100644 --- a/tests/test_head_positive_sources.py +++ b/tests/test_head_positive_sources.py @@ -8,28 +8,12 @@ from backend.app.models import ImageRecord, Tag, TagKind, TagPositiveConfirmatio from backend.app.models.tag import image_tag from backend.app.services.ml.heads import _eligible_tag_ids from backend.app.services.ml.training_data import _ids_with_tag +from tests.factories import make_image as _img +from tests.factories import make_tag as _tag pytestmark = pytest.mark.integration -def _img(db, sha: str) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", - ) - db.add(img) - db.flush() - return img - - -def _tag(db, name: str) -> Tag: - t = Tag(name=name, kind=TagKind.general) - db.add(t) - db.flush() - return t - - def _apply(db, image_id: int, tag_id: int, source: str) -> None: db.execute(image_tag.insert().values( image_record_id=image_id, tag_id=tag_id, source=source, diff --git a/tests/test_ml_suggestions.py b/tests/test_ml_suggestions.py index 1091518..9a44d7c 100644 --- a/tests/test_ml_suggestions.py +++ b/tests/test_ml_suggestions.py @@ -10,6 +10,7 @@ from backend.app.services.ml.allowlist import AllowlistService from backend.app.services.ml.heads import ground_applied_tag from backend.app.services.ml.suggestions import SuggestionService from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration @@ -22,17 +23,6 @@ def _emb(slot: int, val: float = 3.0) -> list[float]: return v -async def _img(db, sha: str, emb=None) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", siglip_embedding=emb, - ) - db.add(img) - await db.flush() - return img - - async def _embver(db) -> str: s = (await db.execute(select(MLSettings).where(MLSettings.id == 1))).scalar_one() return s.embedder_model_version diff --git a/tests/test_presentation_auto_apply.py b/tests/test_presentation_auto_apply.py index c549fbb..26775b0 100644 --- a/tests/test_presentation_auto_apply.py +++ b/tests/test_presentation_auto_apply.py @@ -17,6 +17,7 @@ from backend.app.services.ml.heads import ( auto_apply_sweep, system_tag_auto_apply_sweep, ) +from tests.factories import make_image as _img pytestmark = pytest.mark.integration @@ -27,17 +28,6 @@ def _emb(slot: int) -> list[float]: return v -def _img(db, sha: str, emb) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", siglip_embedding=emb, - ) - db.add(img) - db.flush() - return img - - def _head(db, tag_id: int, slot: int, *, weight=1.0): # weight 3.0 → score sigmoid(3)=0.95 clears the 0.90 presentation floor; # weight 1.0 → sigmoid(1)=0.73 clears the 0.50 conflict floor. diff --git a/tests/test_process_auto_apply.py b/tests/test_process_auto_apply.py index 1ed5609..b8db16c 100644 --- a/tests/test_process_auto_apply.py +++ b/tests/test_process_auto_apply.py @@ -16,6 +16,7 @@ from backend.app.models import ( from backend.app.models.tag import image_tag from backend.app.services.ml.heads import system_tag_auto_apply_sweep from backend.app.services.ml.training_data import _ids_with_tag +from tests.factories import make_image as _img pytestmark = pytest.mark.integration @@ -26,17 +27,6 @@ def _emb(slot: int) -> list[float]: return v -def _img(db, sha: str, emb) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", siglip_embedding=emb, - ) - db.add(img) - db.flush() - return img - - def _head(db, tag_id: int, slot: int, *, weight=1.0): s = db.execute(select(MLSettings).where(MLSettings.id == 1)).scalar_one() w = [0.0] * 1152 diff --git a/tests/test_regions.py b/tests/test_regions.py index f060d77..a058628 100644 --- a/tests/test_regions.py +++ b/tests/test_regions.py @@ -3,20 +3,11 @@ import pytest from backend.app.models import ImageRecord from backend.app.services.ml.regions import RegionService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration -async def _img(db, sha) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", integrity_status="unknown", - ) - db.add(img) - await db.flush() - return img - - @pytest.mark.asyncio async def test_replace_and_get_regions(db): img = await _img(db, "a" * 64) diff --git a/tests/test_suggestions_bulk.py b/tests/test_suggestions_bulk.py index 5df13f2..431b367 100644 --- a/tests/test_suggestions_bulk.py +++ b/tests/test_suggestions_bulk.py @@ -7,6 +7,7 @@ from backend.app.models import ImageRecord, MLSettings, TagHead, TagKind from backend.app.models.tag import image_tag from backend.app.services.ml.suggestions import SuggestionService from backend.app.services.tag_service import TagService +from tests.factories import make_image_async as _img pytestmark = pytest.mark.integration @@ -17,17 +18,6 @@ def _emb(slot: int) -> list[float]: return v -async def _img(db, sha: str, emb=None) -> ImageRecord: - img = ImageRecord( - path=f"/images/{sha}.jpg", sha256=sha, size_bytes=1, mime="image/jpeg", - width=1, height=1, origin="imported_filesystem", - integrity_status="unknown", siglip_embedding=emb, - ) - db.add(img) - await db.flush() - return img - - async def _head(db, tag_id: int, slot: int = 0): s = (await db.execute(select(MLSettings).where(MLSettings.id == 1))).scalar_one() weights = [0.0] * 1152 -- 2.54.0 From 2587421f5bc0ecdbd9b5de3a8ea3139b0e4f6f42 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 16:51:29 -0400 Subject: [PATCH 74/94] ci: a push publishes through a candidate tag and promotes only after the smoke (4310) On a push, build-web and build-agent wrote :dev / :latest straight from the build, and :c- right after it. smoke-web then booted the image, so it could detect a broken image but not stop one reaching the tag deployments follow. Rule 164's verify_with puts the check between build and push. Both image jobs now build to :-candidate (the refresh keeps :refresh-candidate). Their repoint step acts only on a reuse hit. promote needs build-web, build-agent and smoke-web on every trigger, and writes each built image's full tag list (channel, plus :c- on main) by manifest PUT from that job's digest, reading each tag back. A failed or skipped smoke leaves every tag on the last build that worked. The refresh path now also promotes by digest rather than by the candidate tag's name. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .forgejo/workflows/build.yml | 192 ++++++++++++++++++++++------------- 1 file changed, 124 insertions(+), 68 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index db5e219..3a6d09c 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -850,6 +850,9 @@ jobs: # reuse hit — deliberately NOT folded into `digest`, which the :c- # repoint reads and which must keep meaning "what this run built" (#4290). published_digest: ${{ steps.reuse.outputs.published_digest }} + # Every tag this commit should end up under (channel, plus :c- on + # main). `promote` writes them from `digest` once the smoke has passed. + tags: ${{ steps.tag.outputs.tags }} # A plain `needs` — no `always()`. That expression existed to let a # SKIPPED sign-extension through on a tag push while still blocking a # FAILED one. With no tag trigger, sign-extension always runs, so the @@ -1108,11 +1111,13 @@ jobs: # WHERE THE BUILD PUBLISHES, which is not always the channel — and # whether the channel then has to be written separately. # - # On a push the build writes the channel tag directly: the bytes came - # from a commit, and a commit is the thing CI tests. Nothing to hold - # it behind. + # Every build writes a CANDIDATE tag, never the channel (#4310): + # :dev-candidate / :latest-candidate on a push, :refresh-candidate on + # the refresh. `promote` moves the channel only after smoke-web has + # booted the bytes. The lanes prove the SOURCE; only the smoke proves + # the BYTES, and the two are not the same claim. # - # On the scheduled refresh it writes a CANDIDATE tag instead. A + # The refresh is the case that made this obvious. A # refresh rebuilds against freshly resolved base images, and the web # image's runtime is a line of UNPINNED Debian packages (ffmpeg, # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build. @@ -1142,7 +1147,10 @@ jobs: echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" echo "promote=true" >> "$GITHUB_OUTPUT" else - echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + # A push builds to a per-channel candidate too (#4310). The channel + # tag moves in `promote`, after smoke-web has booted these bytes — + # so a broken image never reaches the tag deployments follow. + echo "build_ref=$IMAGE:$T-candidate" >> "$GITHUB_OUTPUT" echo "promote=false" >> "$GITHUB_OUTPUT" fi @@ -1418,6 +1426,16 @@ jobs: TAGS: ${{ steps.tag.outputs.tags }} run: | set -euf + # A BUILD publishes nothing from here (#4310). Its bytes sit on the + # candidate tag until smoke-web has booted them; `promote` then + # writes the channel tag AND :c- from this run's digest. Writing + # :c- here would publish an immutable rollback tag for bytes + # that might then fail the smoke. + if [ -n "${BUILT_DIGEST:-}" ]; then + echo "repoint: built $BUILT_DIGEST this run — promote publishes it" + echo "repoint: after the smoke; nothing to write here." + exit 0 + fi # WHAT WE COPY FROM, which is not what we EXCLUDE (#4290). # # This step used to copy from the channel tag by NAME. Nothing @@ -1511,14 +1529,9 @@ jobs: # source tree, and not a static inspection: `ffmpeg -version` exiting 0 would # pass while a codec removal broke every thumbnail in the library. # - # Refresh-only. On a push the bytes came from a commit, and a commit is what - # the lanes above already test — and since 2026-09-23 they gate the build, so - # those bytes could not exist without them having passed. - # - # Reports a verdict; it does not yet gate the promote (milestone 362 step 4). - # Landing the gate and the thing it gates in one change would mean the first - # time anyone saw this job run would also be the first time it could stop a - # publish. + # It gates `promote` on every trigger (#4310). The lanes above prove the + # source; they cannot see the bytes, and a Dockerfile or base change breaks + # the bytes without touching the source. smoke-web: needs: [build-web] # Every run that actually BUILT something, not just the weekly refresh. @@ -1530,13 +1543,10 @@ jobs: # and were smoked when they were built. Re-smoking them would burn two # minutes to re-learn a fact. # - # HONEST LIMIT, and it is the reason #4299 exists: on a push this runs - # AFTER build-web has written the channel tag, so it detects rather than - # gates. Rule 164's verify_with asks for the check BETWEEN build and push. - # Closing that needs the push path to adopt the candidate-then-promote - # shape the refresh already has — per-channel candidate tags, promote - # learning its channel, and the :c- repoint moving after the gate. - # That is a redesign of the production publish path and is its own task. + # This GATES the publish on every trigger (#4310): builds land on a + # candidate tag, and `promote` needs this job, so :dev / :latest and a + # built :c- move only after it passes. Rule 164's verify_with — the + # check BETWEEN build and push. # # NO `if:` — this job always runs (#4323). It used to be gated on the # build having published something, which skipped it on a reuse hit. That @@ -1954,88 +1964,113 @@ jobs: # smoked it, and it should not be read as if it were. promote: needs: [build-web, build-agent, smoke-web] - # Only a refresh publishes through a candidate; a push writes its channel - # tag directly from the build. Reads the same reuse-step decision the build - # took, via a job output — a job's `if:` cannot see the `env` context. - if: needs.build-web.outputs.candidate == 'true' + # THE PUBLISH (#4310). Both image jobs build to a candidate tag — a push to + # :dev-candidate / :latest-candidate, the weekly refresh to + # :refresh-candidate — and nothing names those bytes under a tag anybody + # deploys until this job runs. It runs only when every job in `needs` + # SUCCEEDED (a job-level `if:` without a status function implies + # success()), so a failed smoke — or a skipped one, which is not the same + # as a passing one (run 5290) — leaves :dev / :latest on the last build + # that worked. Rule 164's verify_with: the check sits BETWEEN build and + # push. + # + # Per image, from the DIGEST that image's job built (#4290): a tag can + # move between the build and this job, a digest cannot. An image whose job + # hit reuse built nothing and has nothing to promote — its build job + # already wrote its :c- from the channel tag, which "hit" proved + # carries this commit. + if: github.event_name != 'pull_request' runs-on: python-ci container: image: git.fabledsword.com/bvandeusen/ci-python:3.14 steps: - - name: Point the channel tags at the smoked candidates + - name: Point this commit's tags at the smoked builds env: TOKEN: ${{ secrets.RELEASE_TOKEN }} ACTOR: ${{ github.actor }} + WEB_DIGEST: ${{ needs.build-web.outputs.digest }} + WEB_TAGS: ${{ needs.build-web.outputs.tags }} + AGENT_DIGEST: ${{ needs.build-agent.outputs.digest }} + AGENT_TAGS: ${{ needs.build-agent.outputs.tags }} run: | set -eu - # `latest` is not a guess: a refresh always builds `main` (BUILD_REF), - # and the "must have checked out main" guard in every build job fails - # the run if that did not hold. So the channel is main's. - TAG=latest FAILED="" + MOVED=0 - for NAME in fabledcurator fabledcurator-agent; do + promote() { + NAME="$1"; DIGEST="$2"; TAGS="$3" REPO="bvandeusen/$NAME" - echo "promote: $REPO" + if [ -z "$DIGEST" ]; then + echo "promote: $NAME — no build this run (reuse hit); nothing to publish" + return 0 + fi + echo "promote: $NAME $DIGEST" # Registry auth is its own token exchange — `docker login` # authenticates the docker client, not curl. Deadline on every call # (rule 156): a registry that stops answering must fail this step, - # not hang the weekly refresh until the job times out. + # not hang the run until the job times out. BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') # Ask for the IMAGE manifest media types only. Offering the index - # types too would let the registry hand back an index if one ever - # existed at this tag, and we would faithfully copy the thing this - # whole approach exists to avoid creating. + # types too would let the registry hand back an index, and we would + # faithfully copy the thing this approach exists to avoid creating. ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$DIGEST" \ && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') test -n "$CT" - SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - echo "promote: candidate $SRC ($CT)" # NOT `imagetools create`. That wraps its source in an INDEX, and # `.Image.Config.Labels` does not resolve through one — the # fc.revision the reuse check reads off the channel tag would come # back empty, every later push would miss and rebuild, and nothing # would go red (#3183, run 4751). A manifest PUT is what "make this - # tag name that image" means at the registry: same bytes, same media - # type, same digest, no layer transfer. - curl -fsS --max-time 120 -X PUT \ - -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ - --data-binary @manifest.json \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" + # tag name that image" means at the registry: same bytes, same + # media type, same digest, no layer transfer. (It also makes a + # built :c- a plain image rather than the index the old + # repoint left.) + IFS=, + for REF in $TAGS; do + TAG="${REF##*:}" + curl -fsS --max-time 120 -X PUT \ + -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ + --data-binary @manifest.json \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" - # Read it back. A PUT that returned 2xx but landed something else is - # exactly the silent-and-plausible failure this pipeline keeps - # producing, and the check costs one request. - NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ - | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - if [ "$NOW" != "$SRC" ]; then - echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2 - FAILED="$FAILED $NAME" - continue - fi - echo "promote: $NAME:$TAG now names $NOW" - done + # Read it back. A PUT that returned 2xx but landed something else + # is exactly the silent-and-plausible failure this pipeline keeps + # producing, and the check costs one request. + NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ + | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + if [ "$NOW" != "$DIGEST" ]; then + echo "promote: FAILED — $NAME:$TAG is $NOW, expected $DIGEST" >&2 + FAILED="$FAILED $NAME:$TAG" + continue + fi + echo "promote: $NAME:$TAG now names $NOW" + MOVED=$((MOVED + 1)) + done + unset IFS + } + + promote fabledcurator "$WEB_DIGEST" "$WEB_TAGS" + promote fabledcurator-agent "$AGENT_DIGEST" "$AGENT_TAGS" if [ -n "$FAILED" ]; then echo "" >&2 echo "promote: FAILED for:$FAILED" >&2 - echo "promote: the channel tags are now INCONSISTENT — some images" >&2 - echo "promote: moved and some did not. Re-run this refresh; the" >&2 - echo "promote: candidates are still published and the promote is" >&2 - echo "promote: idempotent." >&2 + echo "promote: the tags are now INCONSISTENT — some moved and some" >&2 + echo "promote: did not. Re-run this workflow; the builds are still" >&2 + echo "promote: published by digest and the promote is idempotent." >&2 exit 1 fi - echo "promote: both channel tags moved" + echo "promote: $MOVED tag(s) written" build-agent: # THE GATE (2026-09-23). Every lane above must have PASSED before this job # exists at all — so a red suite does not produce an image, let alone push @@ -2049,6 +2084,12 @@ jobs: # where only a FAILED gate blocks would have published unverified images # while reporting success. needs: [lint, extension-version, backend-lint-and-test, frontend-build, integration] + # What `promote` needs to publish this image once the smoke has passed: + # the manifest this run built (empty on a reuse hit) and every tag it + # belongs under. Same meaning as build-web's outputs of the same names. + outputs: + digest: ${{ steps.build.outputs.digest }} + tags: ${{ steps.tag.outputs.tags }} # A pull_request run is the lanes and nothing else. This is the ONLY thing # separating "validate a Renovate bump" from "publish a Renovate bump", so # it is stated on each publishing job rather than inferred from a `needs` @@ -2240,11 +2281,13 @@ jobs: # WHERE THE BUILD PUBLISHES, which is not always the channel — and # whether the channel then has to be written separately. # - # On a push the build writes the channel tag directly: the bytes came - # from a commit, and a commit is the thing CI tests. Nothing to hold - # it behind. + # Every build writes a CANDIDATE tag, never the channel (#4310): + # :dev-candidate / :latest-candidate on a push, :refresh-candidate on + # the refresh. `promote` moves the channel only after smoke-web has + # booted the bytes. The lanes prove the SOURCE; only the smoke proves + # the BYTES, and the two are not the same claim. # - # On the scheduled refresh it writes a CANDIDATE tag instead. A + # The refresh is the case that made this obvious. A # refresh rebuilds against freshly resolved base images, and the web # image's runtime is a line of UNPINNED Debian packages (ffmpeg, # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build. @@ -2273,7 +2316,10 @@ jobs: if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" else - echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + # A push builds to a per-channel candidate too (#4310). The channel + # tag moves in `promote`, after smoke-web has booted these bytes — + # so a broken image never reaches the tag deployments follow. + echo "build_ref=$IMAGE:$T-candidate" >> "$GITHUB_OUTPUT" fi # Compare VALUES, never exit codes. Measured on buildx v0.36.1 @@ -2464,6 +2510,16 @@ jobs: TAGS: ${{ steps.tag.outputs.tags }} run: | set -euf + # A BUILD publishes nothing from here (#4310). Its bytes sit on the + # candidate tag until smoke-web has booted them; `promote` then + # writes the channel tag AND :c- from this run's digest. Writing + # :c- here would publish an immutable rollback tag for bytes + # that might then fail the smoke. + if [ -n "${BUILT_DIGEST:-}" ]; then + echo "repoint: built $BUILT_DIGEST this run — promote publishes it" + echo "repoint: after the smoke; nothing to write here." + exit 0 + fi # WHAT WE COPY FROM, which is not what we EXCLUDE (#4290). # # This step used to copy from the channel tag by NAME. Nothing -- 2.54.0 From 7a09dc3cdadbee2b9a178fb6cd903afd2c28b723 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 17:45:39 -0400 Subject: [PATCH 75/94] build: the agent installs one CUDA-13 stack instead of two, the web image drops ML packages it never imported, and Redis moves to 8 (1451, 1452) Agent: - The image ran PyPI's CUDA-13 torch 2.14 and onnxruntime-gpu 1.30 on a CUDA 12.9 cudnn-runtime base. requirements.txt had silently replaced the Dockerfile's torch 2.6+cu124, because ultralytics pulls torchvision, which pulls its own torch. That left ~3 GB of base libraries and a ~3 GB torch nothing loaded: 10 GB compressed. - Now: an nvidia/cuda 13.0.3 `base` image, with torch and torchvision installed together from cu130. CUDA and cuDNN come from the nvidia-* pip packages; onnxruntime-gpu declares its [cuda,cudnn] extras. - fc_agent/accel.py preloads those libraries for onnxruntime. It then logs, and reports in /status, whether torch and the ONNX CUDA provider actually got the GPU, since both fall back to the CPU silently. Web image: - Drop opencv-python-headless and onnxruntime, plus the opencv-only apt libs. Both have been listed since the scaffold and nothing in backend/ imports them. - torch/torchvision move to 2.14/0.29, and the unexplained caps are lifted (rule 154). Redis: 8-alpine in both compose files and both CI service containers. That gives an AGPLv3 licence option, where 7.4 was RSAL/SSPL only. The client moves to >=8.1. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .forgejo/workflows/build.yml | 8 ++-- Dockerfile | 10 ++--- agent/Dockerfile | 33 +++++++++----- agent/README.md | 3 +- agent/fc_agent/accel.py | 86 ++++++++++++++++++++++++++++++++++++ agent/fc_agent/app.py | 6 ++- agent/requirements.txt | 10 +++-- docker-compose.single.yml | 2 +- docker-compose.yml | 2 +- requirements-ml.txt | 22 ++++----- requirements.txt | 4 +- tests/test_agent_accel.py | 75 +++++++++++++++++++++++++++++++ 12 files changed, 220 insertions(+), 41 deletions(-) create mode 100644 agent/fc_agent/accel.py create mode 100644 tests/test_agent_accel.py diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 3a6d09c..0b1aed6 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -419,7 +419,7 @@ jobs: --health-timeout 5s --health-retries 10 redis: - image: redis:7-alpine + image: redis:8-alpine options: >- --health-cmd "redis-cli ping" --health-interval 10s @@ -434,7 +434,7 @@ jobs: docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}' echo "=== end landscape ===" PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1) - RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1) + RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:8-alpine" -q | head -n1) test -n "$PG" && test -n "$RD" PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG") RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD") @@ -1593,7 +1593,7 @@ jobs: --health-timeout 5s --health-retries 10 redis: - image: redis:7-alpine + image: redis:8-alpine options: >- --health-cmd "redis-cli ping" --health-interval 10s @@ -1619,7 +1619,7 @@ jobs: # in a container against a mounted docker socket, so the services are # SIBLINGS reachable by IP, not by hostname. PG=$(docker ps --filter "name=smoke" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1) - RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:7-alpine" -q | head -n1) + RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:8-alpine" -q | head -n1) test -n "$PG" && test -n "$RD" PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG") RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD") diff --git a/Dockerfile b/Dockerfile index 633c34b..36eb7f7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -36,11 +36,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libwebp7 \ libpng16-16 \ ca-certificates \ - # opencv-python-headless (via requirements-ml.txt) links these even in its - # headless build. Came from Dockerfile.ml when the images merged - # (milestone 422 step 6). - libgl1 \ - libglib2.0-0 \ && rm -rf /var/lib/apt/lists/* WORKDIR /app @@ -61,7 +56,8 @@ RUN pip install -r requirements.txt # # torch 2.12.1+cpu wheel 192.3 MB # torchvision 0.27.1+cpu 1.8 MB -# transformers / onnxruntime / opencv / sklearn and friends +# transformers / onnxruntime / opencv / sklearn and friends (opencv and +# onnxruntime since dropped, #1451 — nothing here imported them) # 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB # largest newly-pushed layer 222.07 MB # @@ -85,7 +81,7 @@ RUN pip install -r requirements.txt # CPU-only torch from the PyTorch CPU index. Nothing here uses a GPU — the # GPU agent is a separate service with its own image. RUN pip install --index-url https://download.pytorch.org/whl/cpu \ - "torch>=2.12,<3.0" "torchvision>=0.27,<0.28" + "torch>=2.14" "torchvision>=0.29" RUN pip install -r requirements-ml.txt # Where the model lands. Deliberately NOT a VOLUME instruction: that mints an diff --git a/agent/Dockerfile b/agent/Dockerfile index 1f95b02..2d0323f 100644 --- a/agent/Dockerfile +++ b/agent/Dockerfile @@ -1,10 +1,21 @@ # FabledCurator GPU agent — runs on the desktop with the GPU. -# CUDA 12.9 + cuDNN 9 runtime so onnxruntime-gpu can use the card (it needs -# cuDNN 9 — the plain -runtime image lacks it: "libcudnn.so.9: cannot open -# shared object file"); ffmpeg for video frames. Ubuntu 24.04 → Python 3.12. -# Stays on the CUDA-12 / cuDNN-9 line the default onnxruntime-gpu + torch are -# built against (CUDA 13 has only nascent ONNX Runtime support). -FROM nvidia/cuda:12.9.2-cudnn-runtime-ubuntu24.04 +# +# The `base` flavour, not `cudnn-runtime`: CUDA and cuDNN arrive as the +# `nvidia-*` pip packages torch and onnxruntime-gpu depend on, so the base only +# has to hand the container the driver (it sets NVIDIA_VISIBLE_DEVICES / +# NVIDIA_DRIVER_CAPABILITIES for the Container Toolkit). Until #1451 this was +# `12.9.2-cudnn-runtime` under a `torch==2.6.0+cu124` — and requirements.txt then +# REPLACED that torch with PyPI's CUDA-13 build (ultralytics pulls torchvision, +# which pulls its matching torch), beside a CUDA-13 onnxruntime-gpu. The image +# ran CUDA 13 on a CUDA-12 base, carrying ~3 GB of base libraries and a ~3 GB +# torch nothing loaded: 10 GB compressed. +# +# 13.0 because that is the line both wheels are built for (torch's cu130 index, +# onnxruntime-gpu's `nvidia-cuda-runtime~=13.0`). Needs an NVIDIA driver that +# supports CUDA 13 (580+); fc_agent/accel.py logs at startup whether torch and +# onnxruntime actually got the GPU, since both fall back to the CPU silently. +# ffmpeg for video frames. Ubuntu 24.04 → Python 3.12. +FROM nvidia/cuda:13.0.3-base-ubuntu24.04 # PIP_BREAK_SYSTEM_PACKAGES: Ubuntu 24.04 marks its system Python as externally # managed (PEP 668), so a global `pip install` errors without this. It's a @@ -16,10 +27,12 @@ RUN apt-get update \ && rm -rf /var/lib/apt/lists/* WORKDIR /app -# torch from the CUDA-12.4 wheel index; its wheels bundle their own CUDA + cuDNN -# so they run on the 12.9 base and coexist with onnxruntime-gpu. Installed first -# + separately so the GPU build of torch is deterministic and layer-cached. -RUN pip3 install --no-cache-dir torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124 +# torch AND torchvision from the cu130 index, together and first. Installing +# torch alone is what let the next step swap it out: ultralytics needs +# torchvision, PyPI's torchvision pins its own torch, and pip replaced ours to +# match. With both present, requirements.txt finds them satisfied. +RUN pip3 install --no-cache-dir --index-url https://download.pytorch.org/whl/cu130 \ + torch torchvision COPY requirements.txt . RUN pip3 install --no-cache-dir -r requirements.txt COPY fc_agent ./fc_agent diff --git a/agent/README.md b/agent/README.md index ec0119b..8084e23 100644 --- a/agent/README.md +++ b/agent/README.md @@ -15,7 +15,8 @@ sudo pacman -S nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker # verify: -docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi +docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi +# the header's CUDA version must be 13.0 or later (driver 580+) ``` ## 1. Get a token diff --git a/agent/fc_agent/accel.py b/agent/fc_agent/accel.py new file mode 100644 index 0000000..3bf309d --- /dev/null +++ b/agent/fc_agent/accel.py @@ -0,0 +1,86 @@ +"""Which accelerator each runtime actually got — reported once, at startup. + +The agent has two GPU runtimes and both fall back to the CPU without raising: +torch when the driver is too old for its CUDA build, and onnxruntime (the imgutils +detector + CCIP models) when its CUDA provider cannot load its libraries. A +fallback shows up only as slower work, and nothing reported it. On 2026-09-24 the +image turned out to be running a CUDA-13 torch and onnxruntime on a CUDA-12 base +(#1451), and whether the ONNX half was on the GPU could not be answered from +anything the agent had ever logged. + +Also the fix for the likeliest way the ONNX half misses: onnxruntime-gpu's CUDA +provider finds libcudart/cuBLAS/cuDNN only on the loader path, and in this image +they live in the `nvidia-*` pip packages torch installs. `preload_dlls()` (ORT +1.21+) loads them from there, so the provider resolves them by soname. + +Stdlib-only at import, so the unit suite can import it — torch and onnxruntime +are imported inside the functions. +""" + +from __future__ import annotations + +import ctypes +import importlib +import logging +from pathlib import Path + +log = logging.getLogger("fc_agent.accel") + +# Filled by report(); /status carries it so the page can show it too. +LAST: dict = {} + + +def torch_status(imp=importlib.import_module) -> dict: + try: + torch = imp("torch") + except Exception as e: + return {"device": "unavailable", "error": str(e)} + out = {"version": torch.__version__, "cuda_build": torch.version.cuda} + if torch.cuda.is_available(): + out["device"] = "cuda" + out["gpu"] = torch.cuda.get_device_name(0) + else: + out["device"] = "cpu" + return out + + +def onnx_status(imp=importlib.import_module, load=ctypes.CDLL) -> dict: + try: + ort = imp("onnxruntime") + except Exception as e: + return {"device": "unavailable", "error": str(e)} + out = {"version": ort.__version__, "providers": list(ort.get_available_providers())} + if "CUDAExecutionProvider" not in out["providers"]: + out["device"] = "cpu" + return out + preload = getattr(ort, "preload_dlls", None) + if preload is not None: + try: + preload() + except Exception as e: + out["preload_error"] = str(e) + # "Available" only means the build HAS the provider. Loading its library is + # what resolves libcudart/cuBLAS/cuDNN — the step that fails when they are + # missing, and the one a session would otherwise fail silently on. + capi = Path(ort.__file__).parent / "capi" + try: + load(str(capi / "libonnxruntime_providers_shared.so"), mode=ctypes.RTLD_GLOBAL) + load(str(capi / "libonnxruntime_providers_cuda.so")) + except OSError as e: + out["device"] = "cpu" + out["error"] = str(e) + else: + out["device"] = "cuda" + return out + + +def report() -> dict: + """Check both runtimes, log the result, and keep it for /status.""" + LAST.clear() + LAST.update(torch=torch_status(), onnx=onnx_status()) + for name, s in LAST.items(): + if s.get("device") == "cuda": + log.info("accel: %s on GPU (%s)", name, s) + else: + log.warning("accel: %s is NOT on the GPU — work runs on the CPU (%s)", name, s) + return dict(LAST) diff --git a/agent/fc_agent/app.py b/agent/fc_agent/app.py index f9a5d8a..877bea5 100644 --- a/agent/fc_agent/app.py +++ b/agent/fc_agent/app.py @@ -11,7 +11,7 @@ import logging from fastapi import FastAPI, Request from fastapi.responses import HTMLResponse, JSONResponse -from . import logbuf +from . import accel, logbuf from .build_info import FC_CHANNEL, FC_REVISION, FC_VERSION, build_id, display_version from .config import Config from .gpu import read_gpu @@ -47,6 +47,9 @@ async def _no_store(request, call_next): @app.on_event("startup") def _maybe_autostart() -> None: + # Before the worker: the report also preloads the CUDA libraries the ONNX + # models need, and it says in the log which runtimes landed on the GPU. + accel.report() # With AUTO_START set, a container restart (host reboot, or `restart: # unless-stopped` after a crash) resumes the worker on its own — the slots # then ride out a still-down curator via lease backoff. Lets the agent @@ -137,6 +140,7 @@ def status(): s["version"] = FC_VERSION or None s["channel"] = FC_CHANNEL or None s["revision"] = FC_REVISION or None + s["accel"] = accel.LAST or None return JSONResponse(s) diff --git a/agent/requirements.txt b/agent/requirements.txt index b71fdb0..934de2e 100644 --- a/agent/requirements.txt +++ b/agent/requirements.txt @@ -1,10 +1,12 @@ # CCIP + figure detection (ONNX models, auto-downloaded from HuggingFace). dghs-imgutils>=0.4 # GPU inference for the ONNX models. Swap to onnxruntime (CPU) for a slow -# server-side fallback run. -onnxruntime-gpu -# The crop EMBEDDER (concept bag). torch is installed separately in the -# Dockerfile from the CUDA-12.4 wheel index so the GPU build is deterministic; +# server-side fallback run. The extras declare the CUDA/cuDNN pip packages its +# CUDA provider loads (fc_agent/accel.py preloads them) rather than relying on +# torch happening to install the same ones. +onnxruntime-gpu[cuda,cudnn] +# The crop EMBEDDER (concept bag). torch + torchvision are installed separately +# in the Dockerfile from the cu130 wheel index, so pip never swaps them out; # transformers loads whatever SigLIP-family model the server announces. transformers>=4.45 # Crop PROPOSERS — small YOLO detectors (booru_yolo anatomy, COCO person, comic diff --git a/docker-compose.single.yml b/docker-compose.single.yml index 20a6c23..7654447 100644 --- a/docker-compose.single.yml +++ b/docker-compose.single.yml @@ -29,7 +29,7 @@ services: redis: - image: redis:7-alpine + image: redis:8-alpine volumes: - redis_data:/data healthcheck: diff --git a/docker-compose.yml b/docker-compose.yml index 671dca0..ab8f520 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -48,7 +48,7 @@ x-celery-healthcheck: &celery_healthcheck services: redis: - image: redis:7-alpine + image: redis:8-alpine volumes: - redis_data:/data healthcheck: diff --git a/requirements-ml.txt b/requirements-ml.txt index 52e20e5..fe6965d 100644 --- a/requirements-ml.txt +++ b/requirements-ml.txt @@ -8,21 +8,21 @@ # so Dockerfile uses the +cpu wheels from # https://download.pytorch.org/whl/cpu instead. # -# IMPORTANT: torchvision 0.27 declares requires_python "!=3.14.1,>=3.10" — -# Python 3.14.1 specifically is excluded due to a known incompatibility. -# The python-ci runner pulls python:3.14-bookworm (latest patch); if that -# resolves to 3.14.1 the install will fail. Pin a specific Python patch in -# the runner image (CI-Runner/CI-python/Dockerfile) if this becomes a -# blocker. 3.14.0 and 3.14.2+ are fine. +# torchvision declares requires_python "!=3.14.1" (0.27 through 0.29). The +# image's python:3.14-slim is past that patch, so it only bites a build pinned +# to exactly 3.14.1. +# +# No caps below: rule 154 wants a named breakage for one, and none of the +# `=5.8,<6.0 -onnxruntime>=1.26,<2.0 -huggingface-hub>=1.14,<2.0 -opencv-python-headless>=4.13,<5.0 +transformers>=5.8 +huggingface-hub>=1.14 # scikit-learn powers the tag-eval (#1130) head-vs-centroid comparison: logistic # regression + cross-validated precision/recall/AP. Battle-tested metrics matter # because that eval's whole purpose is producing trustworthy numbers. numpy is # left to resolve transitively (torch/transformers/sklearn all pull it) to avoid # pinning against their constraints. -scikit-learn>=1.7,<2.0 +scikit-learn>=1.7 diff --git a/requirements.txt b/requirements.txt index f54bc43..60ab750 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,7 +11,9 @@ pgvector>=0.5,<0.6 # Task queue celery>=5.6,<5.7 -redis>=7.4,<8.0 +# Uncapped (rule 154). 8.0 only changed type hints. kombu's `redis` extra +# still says <6.5, but celery is installed without that extra, so it never applies. +redis>=8.1 # Crypto for credential storage (lands in FC-3, but pinned now for stability) cryptography>=49,<50 diff --git a/tests/test_agent_accel.py b/tests/test_agent_accel.py new file mode 100644 index 0000000..cc509a2 --- /dev/null +++ b/tests/test_agent_accel.py @@ -0,0 +1,75 @@ +"""The agent's startup report of which runtime landed on the GPU (#1451). + +Both runtimes fall back to the CPU without raising, so the report is the only +thing that says so. These pin the distinction it exists for: onnxruntime +listing the CUDA provider is not the same as the provider being able to load +its libraries. +""" + +from __future__ import annotations + +import types + +from agent.fc_agent import accel + + +def _ort(providers, preload=None): + mod = types.SimpleNamespace( + __version__="1.30.0", + __file__="/site/onnxruntime/__init__.py", + get_available_providers=lambda: providers, + ) + if preload is not None: + mod.preload_dlls = preload + return mod + + +def _imp(mod): + return lambda name: mod + + +def test_onnx_on_gpu_when_the_cuda_provider_loads(): + calls = [] + s = accel.onnx_status( + _imp(_ort(["CUDAExecutionProvider", "CPUExecutionProvider"], lambda: calls.append("preload"))), + load=lambda path, mode=0: calls.append(path), + ) + assert s["device"] == "cuda" + assert calls[0] == "preload" + assert calls[-1].endswith("capi/libonnxruntime_providers_cuda.so") + + +def test_onnx_listed_but_unloadable_reports_cpu_with_the_reason(): + def load(path, mode=0): + if path.endswith("providers_cuda.so"): + raise OSError("libcudart.so.13: cannot open shared object file") + + s = accel.onnx_status(_imp(_ort(["CUDAExecutionProvider", "CPUExecutionProvider"])), load=load) + assert s["device"] == "cpu" + assert "libcudart.so.13" in s["error"] + + +def test_onnx_cpu_build_never_tries_the_cuda_library(): + def load(path, mode=0): + raise AssertionError("a CPU build has no CUDA provider to load") + + s = accel.onnx_status(_imp(_ort(["CPUExecutionProvider"])), load=load) + assert s["device"] == "cpu" + + +def test_torch_reports_cpu_when_cuda_is_unavailable(): + torch = types.SimpleNamespace( + __version__="2.14.0+cu130", + version=types.SimpleNamespace(cuda="13.0"), + cuda=types.SimpleNamespace(is_available=lambda: False), + ) + s = accel.torch_status(_imp(torch)) + assert s == {"version": "2.14.0+cu130", "cuda_build": "13.0", "device": "cpu"} + + +def test_a_missing_runtime_is_reported_not_raised(): + def imp(name): + raise ImportError(f"No module named {name!r}") + + assert accel.torch_status(imp)["device"] == "unavailable" + assert accel.onnx_status(imp)["device"] == "unavailable" -- 2.54.0 From f127c50207f954149810f0c02644476273166d01 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 18:21:11 -0400 Subject: [PATCH 76/94] fix: the lane sizer reads the live pool, so it stops growing a lane past its cap (4409) It read celery's `max-concurrency` as the pool size, but prefork reports that as the limit the pool booted with. pool_grow and pool_shrink never update it. A lane booted at 1 therefore read 1 forever. The sweep sent `target - 1` on every tick while work waited, and could never shrink, since 1 - 1 is 0. The System tab showed Scheduler "6 / 1" at a cap of 2. pool_size() counts `processes` and falls back to the limit only when there is no process list. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/worker_control.py | 30 +++++++++++++++++++---- tests/test_worker_control.py | 33 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/backend/app/services/worker_control.py b/backend/app/services/worker_control.py index 0013a57..744c532 100644 --- a/backend/app/services/worker_control.py +++ b/backend/app/services/worker_control.py @@ -211,11 +211,10 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: state.reserved += len(reserved.get(hostname, [])) state.consuming.update(q["name"] for q in queues) - # `pool.max-concurrency` is the number pool_grow/pool_shrink move and - # the number the UI shows. Absent on a worker whose stats did not - # answer, which leaves pool=None — unknown, not zero. - pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency") - if isinstance(pool, int): + # Absent on a worker whose stats did not answer, which leaves + # pool=None — unknown, not zero. + pool = pool_size((stats.get(hostname) or {}).get("pool") or {}) + if pool is not None: state.pools[hostname] = pool for state in out.values(): @@ -223,6 +222,27 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]: return out +def pool_size(pool_stats: dict) -> int | None: + """How many processes a prefork pool is running NOW, from `inspect stats`. + + The length of `processes`, not `max-concurrency`. Celery reports + `max-concurrency` as the pool's `limit`, set once at boot; `pool_grow` and + `pool_shrink` hand straight to billiard and never touch it (celery 5.6 + `concurrency/prefork.py`: `self.grow = P.grow`). So it read 1 forever on a + lane booted at 1, and the sizing sweep, computing `target - 1` on every + tick, grew a scheduler capped at 2 to six processes while the System tab + showed "6 / 1" (2026-09-24) — and could never shrink one, since 1 - 1 is 0. + + `max-concurrency` is the fallback only for a pool that lists no processes + (a non-prefork pool), where it is the best number there is. + """ + procs = pool_stats.get("processes") + if isinstance(procs, list): + return len(procs) + limit = pool_stats.get("max-concurrency") + return limit if isinstance(limit, int) else None + + def effective_slots(target: int) -> int: """What a pool can actually be set to. Never below one process. diff --git a/tests/test_worker_control.py b/tests/test_worker_control.py index 8febdb9..b2f005f 100644 --- a/tests/test_worker_control.py +++ b/tests/test_worker_control.py @@ -671,3 +671,36 @@ def test_an_unknown_worker_is_still_ignored_rather_than_guessed_at(monkeypatch): live = wc.inspect_lanes_sync() assert all(not s.present for s in live.values()) + + +# --- the pool size is the processes running, not the boot limit --------------- +# +# The operator's System tab, 2026-09-24: Scheduler "6 / 1" at a cap of 2. Celery +# reports `max-concurrency` as the limit the pool BOOTED with; pool_grow and +# pool_shrink never update it. The sweep read 1 forever, sent `target - 1` on +# every tick, and grew the lane past its cap with no way to shrink it back. + + +def test_pool_size_counts_the_live_processes_not_the_boot_limit(): + """After two pool_grow calls on a pool booted at 1.""" + assert wc.pool_size({"max-concurrency": 1, "processes": [11, 12, 13]}) == 3 + + +def test_pool_size_falls_back_to_the_limit_without_a_process_list(): + assert wc.pool_size({"max-concurrency": 4}) == 4 + assert wc.pool_size({}) is None + + +def test_a_grown_pool_reads_at_its_real_size(monkeypatch): + """The read the sweep sizes from: a pool at its target must read AT its + target, or `target - current` never reaches zero and every tick grows it.""" + _stub_active_queues(monkeypatch, {"scheduler@h": []}) + import sys + insp = sys.modules["backend.app.celery_app"].celery.control.inspect + insp.stats = lambda self: { + "scheduler@h": {"pool": {"max-concurrency": 1, "processes": [1, 2]}}, + } + + live = wc.inspect_lanes_sync() + + assert live["scheduler"].pool == 2 -- 2.54.0 From 42a40d71a4e1e350f747cffa52180a8679c7a209 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 18:28:26 -0400 Subject: [PATCH 77/94] fix: the agent's ONNX check asks CUDA for a device instead of trusting that the libraries loaded (1451) It reported "onnx on GPU" beside torch failing cuInit with "CUDA unknown error". Every library resolved, but no device could be used. The check now calls cudaGetDeviceCount and reports the CUDA error when there is one. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- agent/fc_agent/accel.py | 25 ++++++++++++++++++++++-- tests/test_agent_accel.py | 40 ++++++++++++++++++++++++++++++++++----- 2 files changed, 58 insertions(+), 7 deletions(-) diff --git a/agent/fc_agent/accel.py b/agent/fc_agent/accel.py index 3bf309d..5f29fc5 100644 --- a/agent/fc_agent/accel.py +++ b/agent/fc_agent/accel.py @@ -69,11 +69,32 @@ def onnx_status(imp=importlib.import_module, load=ctypes.CDLL) -> dict: except OSError as e: out["device"] = "cpu" out["error"] = str(e) - else: - out["device"] = "cuda" + return out + # Loading proves the libraries resolve, NOT that a GPU can be used: on + # 2026-09-24 this reported "onnx on GPU" beside torch failing cuInit with + # "CUDA unknown error" (a driver update awaiting a reboot). Asking the CUDA + # runtime for a device initialises the driver the provider would use. + error = _cuda_device_error(load) + out["device"] = "cpu" if error else "cuda" + if error: + out["error"] = error return out +def _cuda_device_error(load=ctypes.CDLL) -> str | None: + """None when the CUDA runtime can reach a device, else why it cannot.""" + try: + cudart = load("libcudart.so.13") + except OSError as e: + return str(e) + count = ctypes.c_int(0) + rc = cudart.cudaGetDeviceCount(ctypes.byref(count)) + if rc != 0: + cudart.cudaGetErrorString.restype = ctypes.c_char_p + return f"cudaGetDeviceCount: {cudart.cudaGetErrorString(rc).decode()} ({rc})" + return None if count.value > 0 else "no CUDA device visible" + + def report() -> dict: """Check both runtimes, log the result, and keep it for /status.""" LAST.clear() diff --git a/tests/test_agent_accel.py b/tests/test_agent_accel.py index cc509a2..af7721e 100644 --- a/tests/test_agent_accel.py +++ b/tests/test_agent_accel.py @@ -28,15 +28,45 @@ def _imp(mod): return lambda name: mod -def test_onnx_on_gpu_when_the_cuda_provider_loads(): +class _Cudart: + """libcudart as ctypes sees it: cudaGetDeviceCount writes through a pointer.""" + + def __init__(self, rc=0, count=1): + self.rc = rc + self.count = count + self.cudaGetErrorString = lambda rc: b"unknown error" + + def cudaGetDeviceCount(self, ref): + ref._obj.value = self.count + return self.rc + + +def _loader(calls, cudart): + def load(path, mode=0): + calls.append(path) + return cudart if path == "libcudart.so.13" else None + return load + + +GPU_BUILD = ["CUDAExecutionProvider", "CPUExecutionProvider"] + + +def test_onnx_on_gpu_when_the_provider_loads_and_a_device_answers(): calls = [] s = accel.onnx_status( - _imp(_ort(["CUDAExecutionProvider", "CPUExecutionProvider"], lambda: calls.append("preload"))), - load=lambda path, mode=0: calls.append(path), + _imp(_ort(GPU_BUILD, lambda: calls.append("preload"))), + load=_loader(calls, _Cudart()), ) assert s["device"] == "cuda" assert calls[0] == "preload" - assert calls[-1].endswith("capi/libonnxruntime_providers_cuda.so") + assert any(c.endswith("capi/libonnxruntime_providers_cuda.so") for c in calls) + + +def test_onnx_libraries_loading_is_not_a_gpu_when_cuda_cannot_initialise(): + """The 2026-09-24 case: every library resolved, cuInit failed.""" + s = accel.onnx_status(_imp(_ort(GPU_BUILD)), load=_loader([], _Cudart(rc=999))) + assert s["device"] == "cpu" + assert "unknown error" in s["error"] def test_onnx_listed_but_unloadable_reports_cpu_with_the_reason(): @@ -44,7 +74,7 @@ def test_onnx_listed_but_unloadable_reports_cpu_with_the_reason(): if path.endswith("providers_cuda.so"): raise OSError("libcudart.so.13: cannot open shared object file") - s = accel.onnx_status(_imp(_ort(["CUDAExecutionProvider", "CPUExecutionProvider"])), load=load) + s = accel.onnx_status(_imp(_ort(GPU_BUILD)), load=load) assert s["device"] == "cpu" assert "libcudart.so.13" in s["error"] -- 2.54.0 From e39ec1c550a07a2d25efeaf3f2398688838d2efb Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 19:04:03 -0400 Subject: [PATCH 78/94] =?UTF-8?q?docs(agent):=20a=20driver=20update=20leav?= =?UTF-8?q?es=20the=20CDI=20spec=20naming=20a=20stale=20nvidia-uvm=20devic?= =?UTF-8?q?e=20=E2=80=94=20how=20to=20spot=20and=20regenerate=20it=20(1451?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- agent/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/agent/README.md b/agent/README.md index 8084e23..7d51eee 100644 --- a/agent/README.md +++ b/agent/README.md @@ -19,6 +19,20 @@ docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi # the header's CUDA version must be 13.0 or later (driver 580+) ``` +### After a driver update: regenerate the CDI spec +If the agent's first log lines say `accel: torch is NOT on the GPU` or report +`cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the +toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The +`nvidia-uvm` device number changes between driver versions, and a spec +generated before the update hands the container a device node that no longer +exists (2026-09-24: host `511,0`, container `235,0`). Compare +`ls -l /dev/nvidia-uvm` on the host with the same inside the container, then: +```sh +sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml +# if your toolkit ships it, this keeps it current on every driver update: +sudo systemctl enable --now nvidia-cdi-refresh.path +``` + ## 1. Get a token In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it. -- 2.54.0 From cc53d8db7b192a3600a38eb9e5cb6c39e50b36ec Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 19:09:07 -0400 Subject: [PATCH 79/94] feat: a GPU agent on the CPU shows as degraded, in the System view and on its own page (4410) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit torch and onnxruntime both fall back to the CPU without raising, so the agent that ran CPU-bound for weeks after a driver update leased and checked in like a healthy one. - The agent sends its startup accel report on every lease and heartbeat. - The server keeps a bounded copy on the roster row. A running agent with a runtime off the GPU becomes `degraded`, with a sentence naming the runtime and the reason. - The top nav shows it amber. - The agent page carries a banner, and its pill reads "CPU only". Also: the bandwidth field gets the page's − / + stepper, and both number fields drop the browser's spin arrows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- agent/fc_agent/accel.py | 17 ++++++ agent/fc_agent/app.py | 25 +++++++- agent/fc_agent/client.py | 11 +++- backend/app/api/gpu.py | 33 ++++++++++- backend/app/api/system_health.py | 46 ++++++++++++++- frontend/src/components/TopNav.vue | 8 +++ .../components/settings/SystemHealthTab.vue | 1 + frontend/src/utils/systemParts.js | 3 +- tests/test_agent_accel.py | 17 ++++++ tests/test_agent_degraded.py | 59 +++++++++++++++++++ 10 files changed, 210 insertions(+), 10 deletions(-) create mode 100644 tests/test_agent_degraded.py diff --git a/agent/fc_agent/accel.py b/agent/fc_agent/accel.py index 5f29fc5..601bcad 100644 --- a/agent/fc_agent/accel.py +++ b/agent/fc_agent/accel.py @@ -95,6 +95,23 @@ def _cuda_device_error(load=ctypes.CDLL) -> str | None: return None if count.value > 0 else "no CUDA device visible" +def summary() -> dict | None: + """The report as FabledCurator stores it: each runtime's device, and why + when it is not the GPU. Sent on every lease and heartbeat, so the System + view can call a running agent that fell back to the CPU "degraded" rather + than "running" — the 2026-09-24 fallback went unseen for weeks because + only this agent's own log said so. None before report() has run.""" + if not LAST: + return None + out = {} + for name, s in LAST.items(): + entry = {"device": s.get("device")} + if s.get("error"): + entry["error"] = str(s["error"])[:200] + out[name] = entry + return out + + def report() -> dict: """Check both runtimes, log the result, and keep it for /status.""" LAST.clear() diff --git a/agent/fc_agent/app.py b/agent/fc_agent/app.py index 877bea5..cc8a040 100644 --- a/agent/fc_agent/app.py +++ b/agent/fc_agent/app.py @@ -192,7 +192,11 @@ _PAGE = """ width:30px;height:32px;font:700 16px system-ui;cursor:pointer} .step:hover{border-color:var(--acc)} #conc,#bw{width:3.4rem;height:32px;text-align:center;font:700 16px system-ui;background:#11151a; - color:var(--fg);border:1px solid var(--bd);border-radius:8px} + color:var(--fg);border:1px solid var(--bd);border-radius:8px;appearance:textfield;-moz-appearance:textfield} + /* The browser's own spin arrows, hidden: the − / + beside each field are the + control, styled like the rest of the page (operator, 2026-09-24). */ + #conc::-webkit-inner-spin-button,#conc::-webkit-outer-spin-button, + #bw::-webkit-inner-spin-button,#bw::-webkit-outer-spin-button{-webkit-appearance:none;margin:0} .unit{color:var(--mut);font-size:12px;font-weight:600} .hint{color:var(--mut);font-size:12px;margin-top:12px} .tiles{display:grid;grid-template-columns:repeat(6,1fr);gap:8px;margin-bottom:16px} @@ -231,6 +235,7 @@ _PAGE = """ + @@ -248,7 +253,9 @@ _PAGE = """
+ + MB/s
@@ -316,6 +323,14 @@ _PAGE = """ await fetch('/auto',{method:'POST',headers:{'Content-Type':'application/json'}, body:JSON.stringify({value:on})});refresh() } + function stepbw(d){ setbw((parseFloat(bw.value)||0)+d) } + // Runtimes that did NOT get the GPU, from the startup report. Both fall back + // to the CPU without raising, so this banner and the pill are the only place + // on this page a slow, CPU-bound agent announces itself. + function cpuOnly(s){ + const a=s.accel||{} + return Object.keys(a).filter(k=>a[k] && a[k].device!=='cuda') + } async function setbw(v){ v=Math.max(0,parseFloat(v)||0); bw.value=v await fetch('/bandwidth',{method:'POST',headers:{'Content-Type':'application/json'}, @@ -386,11 +401,17 @@ _PAGE = """ // unreachable curator; grey when stopped; red with no token. let dc='dot', lbl='stopped' if(!ok){ dc='dot red'; lbl='no token' } - else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable' } + else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable' + if(s.queue && cpuOnly(s).length){ dc='dot amber'; lbl='running · CPU only (degraded)' } } else if(st==='starting'){ dc='dot amber'; lbl='starting…' } else if(st==='stopping'){ dc='dot amber'; lbl='stopping…' } dot.className=dc; connlbl.textContent=lbl banner.style.display=(st==='running' && !s.queue)?'block':'none' + const slow=cpuOnly(s) + accelbanner.style.display=slow.length?'block':'none' + accelbanner.textContent=slow.length?('degraded — '+slow.join(' + ')+' not on the GPU, so that work runs on the CPU: ' + +slow.map(k=>k+': '+(s.accel[k].error||s.accel[k].device)).join(' · ') + +'. After a driver update, regenerate the CDI spec (agent README).'):'' queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable' } } diff --git a/agent/fc_agent/client.py b/agent/fc_agent/client.py index b2b8cea..582290a 100644 --- a/agent/fc_agent/client.py +++ b/agent/fc_agent/client.py @@ -7,6 +7,8 @@ import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry +from . import accel + class FcClient: def __init__(self, base_url: str, token: str, agent_id: str): @@ -72,7 +74,10 @@ class FcClient: def lease(self, batch_size: int) -> list[dict]: r = self.s.post( f"{self.base}/api/gpu/jobs/lease", - json={"agent_id": self.agent_id, "batch_size": batch_size}, + json={ + "agent_id": self.agent_id, "batch_size": batch_size, + "accel": accel.summary(), + }, timeout=30, ) r.raise_for_status() @@ -90,7 +95,9 @@ class FcClient: }) def heartbeat(self, job_ids: list[int]) -> None: - self._post_quiet("/api/gpu/jobs/heartbeat", {"job_ids": job_ids}) + self._post_quiet( + "/api/gpu/jobs/heartbeat", {"job_ids": job_ids, "accel": accel.summary()}, + ) def fail(self, job_id: int, error: str) -> None: self._post_quiet("/api/gpu/jobs/fail", {"job_id": job_id, "error": error}) diff --git a/backend/app/api/gpu.py b/backend/app/api/gpu.py index f0723fb..7ae69fe 100644 --- a/backend/app/api/gpu.py +++ b/backend/app/api/gpu.py @@ -245,6 +245,29 @@ async def errors_recover(image_id: int): # --- Agent (bearer token): lease / submit / heartbeat / fail ------------ + +def _accel_detail(body: dict) -> dict: + """The agent's own report of which runtime got the GPU, kept on its roster + row so the System view can call a CPU-bound agent degraded (#4410). + + Only a dict of {runtime: {device, error?}} is kept, and each value is + reduced to those two short strings: this is written on every lease, by a + client the server does not control. An agent that sends nothing (an + older build) simply has no `accel`, which reads as not-yet-reported. + """ + raw = body.get("accel") + if not isinstance(raw, dict): + return {} + accel = {} + for name, entry in list(raw.items())[:4]: + if not isinstance(entry, dict): + continue + clean = {"device": str(entry.get("device") or "")[:16]} + if entry.get("error"): + clean["error"] = str(entry["error"])[:200] + accel[str(name)[:16]] = clean + return {"accel": accel} if accel else {} + @gpu_bp.route("/jobs/lease", methods=["POST"]) async def lease(): body = await request.get_json(silent=True) or {} @@ -267,7 +290,10 @@ async def lease(): key=f"agent:{agent_id}", kind="agent", display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})", - details={"agent_id": agent_id, "last_call": "lease", "leased": len(jobs)}, + details={ + "agent_id": agent_id, "last_call": "lease", "leased": len(jobs), + **_accel_detail(body), + }, ) ml = await MLSettings.load(session) # image rows for url/mime in one shot @@ -347,7 +373,10 @@ async def heartbeat(): key=f"agent:{agent_id}", kind="agent", display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})", - details={"agent_id": agent_id, "last_call": "heartbeat", "extended": n}, + details={ + "agent_id": agent_id, "last_call": "heartbeat", "extended": n, + **_accel_detail(body), + }, ) await session.commit() return jsonify({"extended": n}) diff --git a/backend/app/api/system_health.py b/backend/app/api/system_health.py index 92dde1a..6fffe84 100644 --- a/backend/app/api/system_health.py +++ b/backend/app/api/system_health.py @@ -72,9 +72,14 @@ assert STALE_AFTER_SECONDS >= SWEEP_PERIOD_SECONDS * _SWEEPS_BEFORE_STALE, ( PROBE_TIMEOUT_SECONDS = 2.0 _OK, _STALE, _DOWN, _UNKNOWN = "ok", "stale", "down", "unknown" +# Checking in, but working at a fraction of its speed: a GPU agent whose +# runtimes fell back to the CPU (#4410). Below stale — a part that may have +# stopped is the more urgent question — and above unknown, because this one +# IS known to be wrong. +_DEGRADED = "degraded" # Worst-first, so an overall verdict is just the max. -_SEVERITY = {_OK: 0, _UNKNOWN: 1, _STALE: 2, _DOWN: 3} +_SEVERITY = {_OK: 0, _UNKNOWN: 1, _DEGRADED: 2, _STALE: 3, _DOWN: 4} def _age_state(age_seconds: float) -> str: @@ -100,6 +105,39 @@ def _describe_learned(name: str, state: str, age: float, details: dict) -> str: return f"{name} has not checked in for {ago} — treat it as stopped" +def _cpu_runtimes(details: dict) -> list[str]: + """The runtimes an agent reported as NOT on the GPU, with why. + + Both torch and onnxruntime fall back to the CPU without raising, so an + agent in that state leases, works and checks in exactly like a healthy + one. On 2026-09-24 one had been doing so since a driver update left a + stale CDI spec; the only sign was a line in the agent's own log. + """ + accel = details.get("accel") + if not isinstance(accel, dict): + return [] + out = [] + for name, entry in sorted(accel.items()): + if not isinstance(entry, dict) or entry.get("device") == "cuda": + continue + why = entry.get("error") or entry.get("device") or "unknown" + out.append(f"{name} ({why})") + return out + + +def _learned_state(name: str, state: str, age: float, details: dict) -> tuple[str, str]: + """A roster row's state and its sentence, degraded included.""" + if state == _OK: + cpu = _cpu_runtimes(details) + if cpu: + return _DEGRADED, ( + f"{name} is running on the CPU — not on the GPU: {'; '.join(cpu)}. " + "After a driver update, regenerate the agent host's CDI spec " + "(agent README)." + ) + return state, _describe_learned(name, state, age, details) + + async def _probe_postgres(session) -> dict: started = time.monotonic() try: @@ -175,13 +213,15 @@ async def system_health(): ).scalars().all() for row in rows: age = (now - row.last_seen_at).total_seconds() - state = _age_state(age) + state, detail = _learned_state( + row.display_name, _age_state(age), age, row.details or {}, + ) parts.append({ "key": row.key, "kind": row.kind, "name": row.display_name, "state": state, - "detail": _describe_learned(row.display_name, state, age, row.details or {}), + "detail": detail, "last_seen_at": row.last_seen_at.isoformat(), "first_seen_at": row.first_seen_at.isoformat(), **{k: v for k, v in (row.details or {}).items() if k != "agent_id"}, diff --git a/frontend/src/components/TopNav.vue b/frontend/src/components/TopNav.vue index 31a9645..e1249ca 100644 --- a/frontend/src/components/TopNav.vue +++ b/frontend/src/components/TopNav.vue @@ -153,6 +153,14 @@ const health = computed(() => { label: (worst?.detail || 'A part has stopped') + suffix, } } + // Running but slow — a GPU agent that fell back to the CPU (#4410). Worth + // the amber dot: nothing else anywhere says so. + if (overall === 'degraded') { + return { + icon: 'mdi-speedometer-slow', color: 'warning', + label: (worst?.detail || 'A part is running degraded') + suffix, + } + } if (overall === 'stale') { return { icon: 'mdi-alert', color: 'warning', diff --git a/frontend/src/components/settings/SystemHealthTab.vue b/frontend/src/components/settings/SystemHealthTab.vue index 9a365fb..e1b30c4 100644 --- a/frontend/src/components/settings/SystemHealthTab.vue +++ b/frontend/src/components/settings/SystemHealthTab.vue @@ -368,6 +368,7 @@ function step(lane, delta) { .fc-sys__dot { width: 9px; height: 9px; border-radius: 50%; flex: 0 0 auto; } .fc-sys__dot--ok { background: rgb(var(--v-theme-success)); } .fc-sys__dot--stale { background: rgb(var(--v-theme-warning)); } +.fc-sys__dot--degraded { background: rgb(var(--v-theme-warning)); } .fc-sys__dot--down { background: rgb(var(--v-theme-error)); } .fc-sys__dot--unknown { background: rgb(var(--v-theme-on-surface) / 0.35); } diff --git a/frontend/src/utils/systemParts.js b/frontend/src/utils/systemParts.js index 61656c1..8df1dc5 100644 --- a/frontend/src/utils/systemParts.js +++ b/frontend/src/utils/systemParts.js @@ -24,7 +24,8 @@ export function queueKey(queues) { } // Worst first. A stopped datastore is why someone opened this tab. -export const SEVERITY = { down: 3, stale: 2, unknown: 1, ok: 0 } +// `degraded`: checking in, but slow — a GPU agent on the CPU (#4410). +export const SEVERITY = { down: 4, stale: 3, degraded: 2, unknown: 1, ok: 0 } export function kindLabel(kind) { if (kind === 'celery') return 'worker lane' diff --git a/tests/test_agent_accel.py b/tests/test_agent_accel.py index af7721e..46629cd 100644 --- a/tests/test_agent_accel.py +++ b/tests/test_agent_accel.py @@ -103,3 +103,20 @@ def test_a_missing_runtime_is_reported_not_raised(): assert accel.torch_status(imp)["device"] == "unavailable" assert accel.onnx_status(imp)["device"] == "unavailable" + + +def test_summary_is_what_the_server_stores(monkeypatch): + """Device plus a bounded reason, per runtime — sent on every lease.""" + monkeypatch.setattr(accel, "LAST", { + "torch": {"version": "2.14.0", "device": "cuda", "gpu": "RTX"}, + "onnx": {"version": "1.30.0", "device": "cpu", "error": "e" * 500}, + }) + assert accel.summary() == { + "torch": {"device": "cuda"}, + "onnx": {"device": "cpu", "error": "e" * 200}, + } + + +def test_summary_before_the_report_is_none(monkeypatch): + monkeypatch.setattr(accel, "LAST", {}) + assert accel.summary() is None diff --git a/tests/test_agent_degraded.py b/tests/test_agent_degraded.py new file mode 100644 index 0000000..0bb33d8 --- /dev/null +++ b/tests/test_agent_degraded.py @@ -0,0 +1,59 @@ +"""A GPU agent that fell back to the CPU reads as DEGRADED, not running (#4410). + +Both runtimes fall back without raising, so a CPU-bound agent leases, works +and checks in exactly like a healthy one. On 2026-09-24 one had been doing so +since a driver update left a stale CDI spec; the only sign was a line in the +agent's own log. The agent now sends its startup report on every lease and +heartbeat, and the System view derives the state from it. +""" + +from __future__ import annotations + +from backend.app.api.gpu import _accel_detail +from backend.app.api.system_health import _SEVERITY, _learned_state + +GPU = {"torch": {"device": "cuda"}, "onnx": {"device": "cuda"}} +CPU = { + "torch": {"device": "cpu"}, + "onnx": {"device": "cpu", "error": "cudaGetDeviceCount: unknown error (999)"}, +} + + +def test_a_running_agent_on_the_gpu_is_ok(): + state, detail = _learned_state("GPU agent", "ok", 5, {"accel": GPU}) + assert state == "ok" + assert detail == "GPU agent is running" + + +def test_a_running_agent_on_the_cpu_is_degraded_and_says_why(): + state, detail = _learned_state("GPU agent", "ok", 5, {"accel": CPU}) + assert state == "degraded" + assert "onnx (cudaGetDeviceCount: unknown error (999))" in detail + assert "torch (cpu)" in detail + + +def test_an_agent_that_never_reported_is_not_called_degraded(): + """An older agent build sends no `accel`: that is not-yet-known, not slow.""" + assert _learned_state("GPU agent", "ok", 5, {})[0] == "ok" + + +def test_stopped_outranks_degraded(): + """Whether a quiet agent is still running is the more urgent question.""" + state, _ = _learned_state("GPU agent", "down", 900, {"accel": CPU}) + assert state == "down" + assert _SEVERITY["stale"] > _SEVERITY["degraded"] > _SEVERITY["unknown"] + + +def test_the_lease_keeps_only_a_bounded_report(): + """Written on every lease by a client the server does not control.""" + body = {"accel": { + "torch": {"device": "cpu", "error": "x" * 5000, "extra": "dropped"}, + "onnx": "not a dict", + }} + kept = _accel_detail(body)["accel"] + assert kept == {"torch": {"device": "cpu", "error": "x" * 200}} + + +def test_a_lease_without_a_report_adds_nothing(): + assert _accel_detail({}) == {} + assert _accel_detail({"accel": None}) == {} -- 2.54.0 From 058fa8560674e39e25cf7dd461753877e433c2df Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 19:26:48 -0400 Subject: [PATCH 80/94] fix: backfill_phash runs on the long maintenance lane, not the scheduler's quick one (4411) A whole-library rehash with a 35-minute limit was matched by the maintenance.* glob. It held a scheduler process for its whole run, and the minute ticks queued behind it. An exact-name route now sends it to maintenance_long. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/celery_app.py | 7 +++++++ tests/test_celery_routing.py | 14 ++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index 5f45567..bd0ad2c 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -62,6 +62,13 @@ def make_celery() -> Celery: # can never starve the quick self-healing sweeps (operator-flagged # 2026-06-07: a 2h audit blocked vacuum/backup/normalize for hours). "backend.app.tasks.maintenance.*": {"queue": "maintenance"}, + # The one long job in maintenance.py: a whole-library phash + # recompute (35 min hard limit; the library was cleared for + # re-hashing by migration 0098). On the quick lane it held a + # scheduler process for its whole run, and the minute ticks queued + # up behind it (2026-09-24: 7 waiting, "all workers busy for 18 + # minutes"). An exact name wins over the glob above. + "backend.app.tasks.maintenance.backfill_phash": {"queue": "maintenance_long"}, "backend.app.tasks.backup.*": {"queue": "maintenance_long"}, "backend.app.tasks.admin.*": {"queue": "maintenance_long"}, "backend.app.tasks.library_audit.*": {"queue": "maintenance_long"}, diff --git a/tests/test_celery_routing.py b/tests/test_celery_routing.py index 8a7cb67..e7a24e6 100644 --- a/tests/test_celery_routing.py +++ b/tests/test_celery_routing.py @@ -36,3 +36,17 @@ def test_queue_for_mirrors_external_to_download(): celery.conf.task_routes["backend.app.tasks.external.*"]["queue"] == "download" ) + + +def test_backfill_phash_runs_on_the_long_lane(): + """It lives in maintenance.py, so the quick-lane glob matches it too — + the router must pick the exact name. A 35-minute rehash on the scheduler + lane blocked the minute ticks behind it (2026-09-24).""" + route = celery.amqp.router.route( + {}, "backend.app.tasks.maintenance.backfill_phash", + ) + assert route["queue"].name == "maintenance_long" + quick = celery.amqp.router.route( + {}, "backend.app.tasks.maintenance.recover_stalled_task_runs", + ) + assert quick["queue"].name == "maintenance" -- 2.54.0 From 84e54489418658d8508dfd3a89b0c5c50bdfd186 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 19:42:00 -0400 Subject: [PATCH 81/94] =?UTF-8?q?feat:=20Discord=20on=20the=20native=20cor?= =?UTF-8?q?e=20ingester=20=E2=80=94=20client,=20downloader,=20ledgers,=20w?= =?UTF-8?q?iring=20(milestone=20428)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Discord was the last focus platform still on gallery-dl. This adds the native path, mirrored from gallery-dl 1.32.13's discord extractor: - discord_client: API v10 with the user token and gallery-dl's request profile (dated Firefox UA, Referer). Walks a server, category, forum, channel or thread in gallery-dl's order and pages each channel newest-first. Files are attachments, then embeds, then forwards, numbered across the message. The resume cursor is :. Text-only messages are not posts, since gallery-dl never made them. - discord_downloader: gallery-dl's on-disk layout, cleaned the way it cleans names on Linux (only `/` and control characters change), so existing files are skipped_disk rather than fetched again. Sidecars carry identity only. The message record keeps gallery-dl's keys, so parse_sidecar, derive_post_url and the drop grouping read it unchanged. - The ledger keys on the attachment id (or a hash of an embed's URL path), not the file's position, which an edit can renumber. Migration 0111. - DiscordIngester: token auth, body canary off (files-only drops are normal). Registered as native, verified by token, and serialised per-platform, since every source shares one user token. - ingest_core: optional `skip_feed` client seam (#4413). A tick's early-out on a multi-channel source now ends the quiet channel, not the whole walk. Clients without the seam behave as before. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- .../versions/0111_discord_native_ledger.py | 75 +++ backend/app/models/__init__.py | 4 + backend/app/models/discord_failed_media.py | 36 ++ backend/app/models/discord_seen_media.py | 34 ++ backend/app/services/discord_client.py | 544 ++++++++++++++++++ backend/app/services/discord_downloader.py | 210 +++++++ backend/app/services/discord_ingester.py | 85 +++ backend/app/services/download_backends.py | 24 +- backend/app/services/ingest_core.py | 20 +- backend/app/services/platform_lock.py | 6 +- tests/test_discord_client.py | 339 +++++++++++ tests/test_discord_downloader.py | 135 +++++ tests/test_download_backends.py | 18 +- tests/test_patreon_ingester.py | 58 ++ tests/test_platform_lock.py | 6 +- 15 files changed, 1579 insertions(+), 15 deletions(-) create mode 100644 alembic/versions/0111_discord_native_ledger.py create mode 100644 backend/app/models/discord_failed_media.py create mode 100644 backend/app/models/discord_seen_media.py create mode 100644 backend/app/services/discord_client.py create mode 100644 backend/app/services/discord_downloader.py create mode 100644 backend/app/services/discord_ingester.py create mode 100644 tests/test_discord_client.py create mode 100644 tests/test_discord_downloader.py diff --git a/alembic/versions/0111_discord_native_ledger.py b/alembic/versions/0111_discord_native_ledger.py new file mode 100644 index 0000000..2637d2d --- /dev/null +++ b/alembic/versions/0111_discord_native_ledger.py @@ -0,0 +1,75 @@ +"""Discord native ingester ledgers — seen and dead-letter, per source. + +Milestone 428, #4415. Discord moves off gallery-dl onto the native core, which +keeps its memory of what a source has already fetched in these two tables +instead of gallery-dl's archive. Same shape as the SubscribeStar pair. + +Revision ID: 0111 +Revises: 0110 +Create Date: 2026-09-24 + +""" +import sqlalchemy as sa +from alembic import op + +revision = "0111" +down_revision = "0110" +branch_labels = None +depends_on = None + + +def upgrade(): + op.create_table( + "discord_seen_media", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("source_id", sa.Integer(), nullable=False), + sa.Column("filehash", sa.String(length=128), nullable=False), + sa.Column("post_id", sa.String(length=64), nullable=True), + sa.Column( + "seen_at", sa.DateTime(timezone=True), + server_default=sa.text("now()"), nullable=False, + ), + sa.ForeignKeyConstraint( + ["source_id"], ["source.id"], + name=op.f("fk_discord_seen_media_source_id_source"), ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_seen_media")), + sa.UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"), + ) + op.create_index( + op.f("ix_discord_seen_media_source_id"), "discord_seen_media", ["source_id"], + ) + op.create_table( + "discord_failed_media", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("source_id", sa.Integer(), nullable=False), + sa.Column("filehash", sa.String(length=128), nullable=False), + sa.Column("attempts", sa.Integer(), server_default="1", nullable=False), + sa.Column("last_error", sa.Text(), nullable=True), + sa.Column( + "first_failed_at", sa.DateTime(timezone=True), + server_default=sa.text("now()"), nullable=False, + ), + sa.Column( + "last_failed_at", sa.DateTime(timezone=True), + server_default=sa.text("now()"), nullable=False, + ), + sa.ForeignKeyConstraint( + ["source_id"], ["source.id"], + name=op.f("fk_discord_failed_media_source_id_source"), ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_failed_media")), + sa.UniqueConstraint( + "source_id", "filehash", name="uq_discord_failed_media_source_id", + ), + ) + op.create_index( + op.f("ix_discord_failed_media_source_id"), "discord_failed_media", ["source_id"], + ) + + +def downgrade(): + op.drop_index(op.f("ix_discord_failed_media_source_id"), table_name="discord_failed_media") + op.drop_table("discord_failed_media") + op.drop_index(op.f("ix_discord_seen_media_source_id"), table_name="discord_seen_media") + op.drop_table("discord_seen_media") diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index b925805..b890bfd 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -8,6 +8,8 @@ from .backup_run import BackupRun from .base import Base from .character_prototype import CcipPrototypeState, CharacterPrototype from .credential import Credential +from .discord_failed_media import DiscordFailedMedia +from .discord_seen_media import DiscordSeenMedia from .download_event import DownloadEvent from .external_link import ExternalLink from .gpu_job import GpuJob @@ -56,6 +58,8 @@ __all__ = [ "BackupRun", "Source", "Credential", + "DiscordFailedMedia", + "DiscordSeenMedia", "PatreonFailedMedia", "PatreonSeenMedia", "SubscribeStarFailedMedia", diff --git a/backend/app/models/discord_failed_media.py b/backend/app/models/discord_failed_media.py new file mode 100644 index 0000000..ad0b2d5 --- /dev/null +++ b/backend/app/models/discord_failed_media.py @@ -0,0 +1,36 @@ +"""DiscordFailedMedia — per-source dead-letter ledger of Discord files that +keep failing to download or validate. + +Mirror of SubscribeStarFailedMedia. After `attempts` reaches the dead-letter +threshold a routine walk skips the file (recovery still retries it); a later +clean download clears the row. `filehash` is the seen-ledger's key. +""" + +from datetime import datetime + +from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func +from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy.types import DateTime + +from .base import Base + + +class DiscordFailedMedia(Base): + __tablename__ = "discord_failed_media" + __table_args__ = ( + UniqueConstraint("source_id", "filehash", name="uq_discord_failed_media_source_id"), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + source_id: Mapped[int] = mapped_column( + ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True + ) + filehash: Mapped[str] = mapped_column(String(128), nullable=False) + attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1") + last_error: Mapped[str | None] = mapped_column(Text, nullable=True) + first_failed_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, server_default=func.now() + ) + last_failed_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, server_default=func.now() + ) diff --git a/backend/app/models/discord_seen_media.py b/backend/app/models/discord_seen_media.py new file mode 100644 index 0000000..2757463 --- /dev/null +++ b/backend/app/models/discord_seen_media.py @@ -0,0 +1,34 @@ +"""DiscordSeenMedia — per-source ledger of Discord files already downloaded. + +Mirror of SubscribeStarSeenMedia for the native Discord ingester (milestone +428). `filehash` holds the ingester's per-file key, `:`: +the attachment id, or for an embed a hash of its URL path. Not the file's +position in the message — an edit that removes a file renumbers the rest +(see `discord_client.MediaItem`). The message record's own gate is the +synthetic `message:` key in the same column. +""" + +from datetime import datetime + +from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func +from sqlalchemy.orm import Mapped, mapped_column +from sqlalchemy.types import DateTime + +from .base import Base + + +class DiscordSeenMedia(Base): + __tablename__ = "discord_seen_media" + __table_args__ = ( + UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + source_id: Mapped[int] = mapped_column( + ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True + ) + filehash: Mapped[str] = mapped_column(String(128), nullable=False) + post_id: Mapped[str | None] = mapped_column(String(64), nullable=True) + seen_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, server_default=func.now() + ) diff --git a/backend/app/services/discord_client.py b/backend/app/services/discord_client.py new file mode 100644 index 0000000..dfa5d50 --- /dev/null +++ b/backend/app/services/discord_client.py @@ -0,0 +1,544 @@ +"""Native Discord read client — the Discord counterpart to subscribestar_client. + +Mirrors gallery-dl 1.32.13's `extractor/discord.py` (rule 130: gallery-dl is the +known-working base), adapted to the native core's client contract +(`ingest_core` module docstring): `iter_posts` / `extract_media`, plus the +post-first `post_record_key` and the `post_meta` date the revisit window reads. + +What is mirrored exactly, because drift in any of it changes what we fetch or +where it lands on disk: + + - API v10, `Authorization: ` (a USER token, not a bot token). + - gallery-dl's request profile: its date-derived Firefox User-Agent, + `Accept: */*`, `Accept-Language`, `Referer: https://discord.com/`. + - `GET /channels/{id}/messages?limit=100&before=`, newest first, + stopping on a short page. Message types {0, 19, 21} only. + - The walk: a text/news channel's own messages then its threads, a forum's + threads, a category's children, a server's text/news/forum channels. + - Files: attachments, then embeds of type image/gifv/video (FC configures + `embeds: all`, which for files is the same three plus rich/link embeds that + carry an image), then forwarded `message_snapshots`, numbered from 1 across + the lot — the `num` in `{date}_{message_id}_{num}_{filename}`. + - Text: `content`, rich-embed author/title/description/fields/footer, poll. + +Two deliberate departures, both about the walk order, neither about content: + + - Threads are walked newest-CREATED first (by id), not by last-message time. + A backfill resumes from a checkpointed channel; last-message order shifts + between chunks whenever someone posts, which can move an unwalked thread + above the resume point and skip it. Creation order only ever grows at the + front, where the next tick finds it. + - A 403 on a thread or a nested channel skips that feed instead of failing + the walk. gallery-dl skips only nested channels; one private thread the + token cannot read would otherwise stop every channel after it. + +FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API. +""" + +from __future__ import annotations + +import hashlib +import logging +import re +import time +from collections.abc import Iterator +from dataclasses import dataclass +from datetime import date +from urllib.parse import unquote + +import requests + +from .native_ingest_common import ( + NativeAuthError, + NativeDriftError, + NativeIngestError, + retry_after_seconds, +) + +log = logging.getLogger(__name__) + +API_ROOT = "https://discord.com/api/v10" +_ROOT = "https://discord.com" + +_TIMEOUT_SECONDS = 60.0 +_MESSAGES_BATCH = 100 +_THREADS_BATCH = 25 +# gallery-dl retries a 429 up to its default 4 retries, waiting +# `request_interval_429` (60s) between them. Discord's Retry-After is exact, so +# it is honoured when present; 60s is the fallback and the cap. +_MAX_429_RETRIES = 4 +_429_WAIT_SECONDS = 60.0 + +# https://discord.com/developers/docs/resources/message#message-object-message-types +# DEFAULT, REPLY, CHAT_INPUT_COMMAND — the ones that carry user content. +MESSAGE_TYPES = frozenset({0, 19, 21}) +# https://discord.com/developers/docs/resources/channel#channel-object-channel-types +_TEXT = frozenset({0, 5}) # text, announcement: messages + threads +_DIRECT = frozenset({1, 3, 10, 11, 12}) # DMs and threads: messages only +_FORUM = frozenset({15, 16}) # forum, media: threads only +_CATEGORY = 4 +_SERVER_WALK = _TEXT | _FORUM +_EMBED_TYPES = frozenset({"image", "gifv", "video"}) + +_URL_RE = re.compile( + r"^(?:https?://)?(?:www\.|ptb\.|canary\.)?discord(?:app)?\.com/channels/" + r"(?P@me|\d+)(?:/(?:\d+/threads/)?(?P\d+))?(?P/.*)?/?$" +) + + +class DiscordAPIError(NativeIngestError): + """Base for native Discord client failures.""" + + +class DiscordAuthError(DiscordAPIError, NativeAuthError): + """401 (the token is invalid or expired) or a 403 on the channel the + source names. The fix is a new token, not a new client.""" + + +class DiscordDriftError(DiscordAPIError, NativeDriftError): + """A response did not have the shape the walk depends on.""" + + +def firefox_user_agent(today: date | None = None) -> str: + """gallery-dl's default User-Agent: a Firefox whose version advances every + four weeks (`util._ff_ver`, "147 on 2026-01-13"). Computed the same way so + the profile keeps matching the gallery-dl this replaced.""" + ver = ((today or date.today()).toordinal() - 735_513) // 28 + return ( + f"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:{ver}.0) " + f"Gecko/20100101 Firefox/{ver}.0" + ) + + +def nameext_from_url(url: str) -> tuple[str, str]: + """gallery-dl's `text.nameext_from_url`: the URL's last path segment, + unquoted, split at the last dot when the extension is at most 16 chars + (lowercased); otherwise the whole name and no extension.""" + filename = unquote(url.partition("?")[0].rpartition("/")[2]) + name, _, ext = filename.rpartition(".") + if name and len(ext) <= 16: + return name, ext.lower() + return filename, "" + + +def parse_source_url(url: str) -> tuple[str | None, str | None]: + """`(server_id, channel_id)` from a Discord channel/server URL. `server_id` + is None for a DM (`@me`); `channel_id` is None for a whole server. Raises + DiscordAPIError for anything else, including a link to a single message — + a message is not something a source can subscribe to.""" + m = _URL_RE.match((url or "").strip()) + if not m or (m.group("rest") or "").strip("/"): + raise DiscordAPIError( + f"Not a Discord channel or server link: {url!r} " + "(expected https://discord.com/channels/[/])" + ) + server = m.group("server") + channel = m.group("channel") + if server == "@me": + if not channel: + raise DiscordAPIError(f"A DM link needs a channel id: {url!r}") + return None, channel + return server, channel + + +def message_text(message: dict) -> str: + """gallery-dl's `extract_message_text`: the body plus the text of rich + embeds and polls, newline-joined, empties dropped.""" + parts = [message.get("content") or ""] + for embed in message.get("embeds") or []: + if embed.get("type") != "rich": + continue + parts.append((embed.get("author") or {}).get("name") or "") + parts.append(embed.get("title") or "") + parts.append(embed.get("description") or "") + for fld in embed.get("fields") or []: + parts.append(fld.get("name") or "") + parts.append(fld.get("value") or "") + parts.append((embed.get("footer") or {}).get("text") or "") + poll = message.get("poll") + if poll: + parts.append(((poll.get("question") or {}).get("text")) or "") + for answer in poll.get("answers") or []: + parts.append(((answer.get("poll_media") or {}).get("text")) or "") + return "\n".join(p for p in parts if p) + + +@dataclass +class MediaItem: + """One file of a Discord message. `filename`/`extension` are gallery-dl's + split of the URL; `num` is its 1-based position across the message's files, + which is what names it on disk. + + `media_id` is what the seen-ledger keys on, and it is deliberately NOT + `num`: an edit that removes a file renumbers the ones after it, and a + positional key would then call a different file seen. It is the + attachment's id, or for an embed (which has none) a hash of its URL path — + the query string is a signature that changes on every fetch. `filehash` is + always None; nothing in a signed CDN URL is a content hash.""" + + url: str + filename: str + extension: str + kind: str + post_id: str + num: int + media_id: str + filehash: str | None = None + + +class DiscordClient: + """Synchronous Discord API v10 read client for one user token.""" + + def __init__( + self, + token: str | None, + *, + request_sleep: float = 0.0, + max_retries: int = _MAX_429_RETRIES, + session: requests.Session | None = None, + ): + self._session = session or requests.Session() + self._session.headers.update({ + "User-Agent": firefox_user_agent(), + "Accept": "*/*", + "Accept-Language": "en-US,en;q=0.5", + "Referer": _ROOT + "/", + }) + if token: + self._session.headers["Authorization"] = token + self._token = token + self._request_sleep = request_sleep or 0.0 + self._max_retries = max_retries + self._server: dict = {} + self._channels: dict[str, dict] = {} + self._skip_feed = False + + # -- request ----------------------------------------------------------- + + def _get(self, endpoint: str, params: dict | None = None): + if not self._token: + raise DiscordAuthError("No Discord token is configured for this source") + if self._request_sleep > 0: + time.sleep(self._request_sleep) + url = API_ROOT + endpoint + attempt = 0 + while True: + try: + resp = self._session.get(url, params=params, timeout=_TIMEOUT_SECONDS) + except requests.RequestException as exc: + raise DiscordAPIError(f"Discord request failed ({endpoint}): {exc}") from exc + if resp.status_code == 429 and attempt < self._max_retries: + attempt += 1 + delay = retry_after_seconds( + resp, attempt, base=_429_WAIT_SECONDS, cap=_429_WAIT_SECONDS, + ) + log.warning( + "Discord 429 (%s) — waiting %.1fs (retry %d/%d)", + endpoint, delay, attempt, self._max_retries, + ) + time.sleep(delay) + continue + break + if resp.status_code == 401: + raise DiscordAuthError( + "Discord rejected the token (HTTP 401) — it is invalid or has " + "expired; copy a fresh one from the browser", + status_code=401, + ) + if resp.status_code != 200: + raise DiscordAPIError( + f"Discord returned HTTP {resp.status_code} ({endpoint})", + status_code=resp.status_code, + retry_after=_retry_after(resp), + ) + try: + return resp.json() + except ValueError as exc: + raise DiscordDriftError( + f"Discord returned non-JSON for {endpoint} ({len(resp.content)} bytes)" + ) from exc + + # -- metadata (gallery-dl parse_server / parse_channel) ----------------- + + def _load_server(self, server_id: str) -> None: + server = self._get(f"/guilds/{server_id}") + if not isinstance(server, dict) or "id" not in server: + raise DiscordDriftError(f"Discord server {server_id} came back without an id") + self._server = { + "server": server.get("name") or "", + "server_id": str(server["id"]), + "owner_id": server.get("owner_id"), + } + channels = self._get(f"/guilds/{server_id}/channels") + if not isinstance(channels, list): + raise DiscordDriftError(f"Discord server {server_id} channel list is not a list") + # Categories first, so every child can name its parent. + for channel in sorted(channels, key=lambda ch: ch.get("type") != _CATEGORY): + self._parse_channel(channel) + + def _parse_channel(self, channel: dict) -> dict: + parent_id = channel.get("parent_id") + meta = { + "channel": channel.get("name") or "", + "channel_id": str(channel.get("id")), + "channel_type": channel.get("type"), + "channel_topic": channel.get("topic") or "", + "parent_id": parent_id, + "is_thread": "thread_metadata" in channel, + } + parent = self._channels.get(parent_id) if parent_id else None + if parent: + meta["parent"] = parent["channel"] + meta["parent_type"] = parent["channel_type"] + if meta["channel_type"] in {1, 3}: + recipients = channel.get("recipients") or [] + meta["channel"] = "DMs" + meta["recipients"] = [u.get("username") for u in recipients] + meta["recipients_id"] = [u.get("id") for u in recipients] + self._channels[meta["channel_id"]] = meta + return meta + + def _channel_meta(self, channel_id: str) -> dict: + if channel_id not in self._channels: + self._parse_channel(self._get(f"/channels/{channel_id}")) + return self._channels[channel_id] + + def _threads(self, channel_id: str) -> list[dict]: + """Every thread of a channel or forum, newest-created first (see the + module docstring for why not last-message order).""" + threads: list[dict] = [] + offset = 0 + while True: + data = self._get(f"/channels/{channel_id}/threads/search", { + "sort_by": "last_message_time", + "sort_order": "desc", + "limit": _THREADS_BATCH, + "offset": offset, + }) + batch = (data.get("threads") or []) if isinstance(data, dict) else [] + threads.extend(batch) + if len(batch) < _THREADS_BATCH: + break + offset += len(batch) + threads.sort(key=lambda t: int(t.get("id") or 0), reverse=True) + return threads + + # -- the walk ------------------------------------------------------------ + + def _feeds(self, channel_id: str, *, safe: bool) -> Iterator[tuple[str, bool]]: + """`(channel_id, safe)` for every message feed under `channel_id`, in + gallery-dl's order. `safe` feeds are skipped on a 403.""" + try: + ctype = self._channel_meta(channel_id)["channel_type"] + except DiscordAPIError as exc: + if exc.status_code != 403: + raise + if not safe: + raise DiscordAuthError( + f"The Discord token cannot see channel {channel_id} (HTTP 403)", + status_code=403, + ) from exc + log.info("Discord: no access to channel %s — skipped", channel_id) + return + if ctype in _TEXT or ctype in _DIRECT: + yield channel_id, safe + if ctype in _TEXT or ctype in _FORUM: + try: + threads = self._threads(channel_id) + except DiscordAPIError as exc: + if exc.status_code != 403: + raise + log.info("Discord: cannot list threads of %s — skipped", channel_id) + threads = [] + for thread in threads: + yield self._parse_channel(thread)["channel_id"], True + elif ctype == _CATEGORY: + for child in list(self._channels.values()): + if child.get("parent_id") == channel_id: + yield from self._feeds(child["channel_id"], safe=True) + elif ctype not in _DIRECT and not safe: + raise DiscordAPIError( + f"Discord channel {channel_id} is of type {ctype}, which has no messages" + ) + + def _source_feeds(self, url: str) -> Iterator[tuple[str, bool]]: + server_id, channel_id = parse_source_url(url) + self._server, self._channels = {}, {} + if server_id is not None: + self._load_server(server_id) + if channel_id is not None: + yield from self._feeds(channel_id, safe=False) + return + for meta in list(self._channels.values()): + if meta["channel_type"] in _SERVER_WALK: + yield from self._feeds(meta["channel_id"], safe=True) + + def skip_feed(self) -> None: + """Optional core seam (#4413): end the current channel and go on to the + next one. A tick's early-out means THIS channel has nothing new, not + that the server has nothing new.""" + self._skip_feed = True + + def iter_posts( + self, campaign_id: str, cursor: str | None = None + ) -> Iterator[tuple[dict, dict, str | None]]: + """Yield `(message, channel_meta, page_cursor)` for every content + message the source reaches, channel by channel, each newest first. + + `campaign_id` is the source URL. The cursor is `:` + — the channel and the `before` id that fetched the page (empty for a + channel's first page) — so a backfill resumes inside the right channel + and re-fetches the page it was cut in. A cursor naming a channel the + walk no longer reaches (a deleted thread) restarts from the top rather + than walking nothing. + """ + resume_channel, _, resume_before = (cursor or "").partition(":") + resuming = bool(resume_channel) + feeds = list(self._source_feeds(campaign_id)) if resuming else None + if feeds is not None and resume_channel not in {cid for cid, _ in feeds}: + log.warning( + "Discord: resume channel %s is no longer in %s — restarting", + resume_channel, campaign_id, + ) + resuming = False + for channel_id, safe in feeds if feeds is not None else self._source_feeds(campaign_id): + before = None + if resuming: + if channel_id != resume_channel: + continue + resuming = False + before = resume_before or None + yield from self._iter_channel(channel_id, before, safe=safe) + + def _iter_channel( + self, channel_id: str, before: str | None, *, safe: bool + ) -> Iterator[tuple[dict, dict, str | None]]: + self._skip_feed = False + meta = {**self._server, **self._channels.get(channel_id, {})} + while True: + page_cursor = f"{channel_id}:{before or ''}" + try: + messages = self._get( + f"/channels/{channel_id}/messages", + {"limit": _MESSAGES_BATCH, "before": before}, + ) + except DiscordAPIError as exc: + if exc.status_code != 403: + raise + if not safe: + raise DiscordAuthError( + f"The Discord token cannot read channel {channel_id} (HTTP 403)", + status_code=403, + ) from exc + log.info("Discord: no access to messages of %s — skipped", channel_id) + return + if not isinstance(messages, list): + raise DiscordDriftError( + f"Discord messages of {channel_id} came back as " + f"{type(messages).__name__}, not a list" + ) + for message in messages: + if message.get("type") not in MESSAGE_TYPES: + continue + message["_meta"] = meta + yield message, meta, page_cursor + if self._skip_feed: + return + if len(messages) < _MESSAGES_BATCH: + return + before = str(messages[-1]["id"]) + + # -- per-message ------------------------------------------------------- + + @staticmethod + def extract_media(post: dict, included: dict | None = None) -> list[MediaItem]: + """gallery-dl's file list for one message: attachments, then the first + of video/image/thumbnail `proxy_url` of each file-bearing embed, then + the same for every forwarded snapshot; numbered from 1 across them.""" + mid = str(post.get("id") or "") + snapshots = [post] + [ + (s or {}).get("message") or {} + for s in post.get("message_snapshots") or [] + if ((s or {}).get("message") or {}).get("type", 0) in MESSAGE_TYPES + ] + found: list[tuple[str, str, str | None]] = [] + for snap in snapshots: + for att in snap.get("attachments") or []: + if att.get("url"): + aid = att.get("id") + found.append((att["url"], "attachment", str(aid) if aid else None)) + for embed in snap.get("embeds") or []: + if embed.get("type") not in _EMBED_TYPES: + continue + for fld in ("video", "image", "thumbnail"): + url = (embed.get(fld) or {}).get("proxy_url") + if url: + found.append((url, "embed", None)) + break + items = [] + for num, (url, kind, fid) in enumerate(found, start=1): + name, ext = nameext_from_url(url) + if fid is None: + path = url.partition("?")[0].encode() + fid = "u" + hashlib.sha1(path, usedforsecurity=False).hexdigest()[:32] + items.append(MediaItem( + url=url, filename=name, extension=ext, kind=kind, post_id=mid, + num=num, media_id=fid, + )) + return items + + @staticmethod + def post_meta(post: dict) -> dict: + """No title (Discord has none); `date` is the message timestamp, ISO + with an offset — what the core's revisit window reads.""" + return {"title": None, "date": post.get("timestamp")} + + @classmethod + def post_record_key(cls, post: dict) -> tuple[str, str] | None: + """`(message:, )` — gates the message record through the seen + ledger, like `post:` on the other platforms. + + None for a message with no files. gallery-dl wrote a sidecar only + beside a file, so a text-only chat line never became a post, and the + drop grouping (discord_grouping) is built on that: a channel's chatter + recorded as posts would bury the drops it exists to surface.""" + mid = post.get("id") + mid = str(mid) if mid is not None else "" + if not mid or not cls.extract_media(post): + return None + return (f"message:{mid}", mid) + + # -- verify ------------------------------------------------------------ + + def verify_auth(self, url: str) -> tuple[bool | None, str]: + """Is the token valid, and can it see what the source names?""" + try: + server_id, channel_id = parse_source_url(url) + except DiscordAPIError as exc: + return None, str(exc) + try: + me = self._get("/users/@me") + if channel_id is not None: + self._get(f"/channels/{channel_id}") + elif server_id is not None: + self._get(f"/guilds/{server_id}") + except DiscordAuthError as exc: + return False, f"Discord rejected the token — {exc}" + except DiscordAPIError as exc: + if exc.status_code in (403, 404): + return False, ( + "The token is valid, but its account cannot see " + f"{'this channel' if channel_id else 'this server'} " + f"(HTTP {exc.status_code})" + ) + return None, f"Couldn't verify (network/HTTP issue): {exc}" + who = (me or {}).get("username") if isinstance(me, dict) else None + return True, f"Token valid{f' ({who})' if who else ''} — the source is readable." + + +def _retry_after(resp: requests.Response) -> float | None: + hdr = resp.headers.get("Retry-After") + try: + return float(hdr) if hdr else None + except (TypeError, ValueError): + return None diff --git a/backend/app/services/discord_downloader.py b/backend/app/services/discord_downloader.py new file mode 100644 index 0000000..ffd8734 --- /dev/null +++ b/backend/app/services/discord_downloader.py @@ -0,0 +1,210 @@ +"""Native Discord media downloader — the Discord counterpart to +subscribestar_downloader. + +Writes files exactly where gallery-dl wrote them, so a cutover finds every +existing file on disk (`skipped_disk`) instead of fetching it again: + + //discord//___. + +That is FC's gallery-dl config (`gallery_dl.DISCORD_DIRECTORY` / +`DISCORD_FILENAME`) under the per-source base directory +`//`. The name is cleaned the way gallery-dl cleans it +on Linux — `/` becomes `_` and control characters are removed, nothing else +(`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`, +whose Windows set would turn a `:` in a channel or file name into `_` and miss +the file gallery-dl wrote. + +Post-first (rule 120): each file gets a minimal sidecar named like it minus the +extension (what `find_sidecar` pairs first), and the message itself gets one +record, `__post.json`, carrying gallery-dl's metadata keys +— `message_id` for the post id, `server_id`/`channel_id` for the permalink, +`message` for the body, `date` — so `parse_sidecar` reads it exactly as it read +the gallery-dl sidecars. Neither file carries an `id` or `post_id` key: both +outrank `message_id` in the post-id chain (`platforms.base`). + +PURE: no DB; the seen-skip is an injected predicate. +""" + +from __future__ import annotations + +import json +import logging +import re +import time +from collections.abc import Callable +from datetime import UTC, datetime +from pathlib import Path + +import requests + +from .discord_client import firefox_user_agent, message_text +from .native_ingest_common import ( + BaseNativeDownloader, + MediaOutcome, + PostRecordOutcome, + make_session, +) + +log = logging.getLogger(__name__) + +PLATFORM = "discord" +_CONTROL = re.compile("[\x00-\x1f\x7f]") +# gallery-dl falls back to the response's type for a URL with no extension; +# we never see the response before naming, and such URLs do not occur for +# Discord attachments or embed proxies in practice. +_NO_EXTENSION = "bin" + + +def gdl_clean(segment: str) -> str: + """One path segment as gallery-dl writes it on Linux.""" + return _CONTROL.sub("", segment.replace("/", "_")) + + +def message_date(post: dict) -> datetime | None: + raw = post.get("timestamp") + if not isinstance(raw, str) or not raw: + return None + try: + dt = datetime.fromisoformat(raw.replace("Z", "+00:00")) + except ValueError: + return None + return (dt if dt.tzinfo else dt.replace(tzinfo=UTC)).astimezone(UTC) + + +def channel_dir(images_root: Path, artist_slug: str, post: dict) -> Path: + """gallery-dl's `{channel}` directory; an empty name adds no segment.""" + base = Path(images_root) / artist_slug / PLATFORM + channel = gdl_clean(((post.get("_meta") or {}).get("channel") or "").strip()) + return base / channel if channel else base + + +def media_stem(post: dict, media) -> str: + """`___` — the file's name minus `.`.""" + when = message_date(post) + day = f"{when:%Y%m%d}" if when else "None" + return gdl_clean(f"{day}_{post.get('id')}_{media.num:>02}_{media.filename}") + + +class DiscordDownloader(BaseNativeDownloader): + """Download a message's files to gallery-dl's layout. The CDN gets + gallery-dl's browser profile and no token — gallery-dl sends the token only + to the API, and the CDN URLs are pre-signed.""" + + def __init__( + self, + images_root: Path, + cookies_path: str | None = None, + *, + validate: bool = True, + rate_limit: float = 0.0, + session: requests.Session | None = None, + ): + super().__init__( + images_root, None, platform=PLATFORM, + validate=validate, rate_limit=rate_limit, + session=session if session is not None else make_session(None, extra_headers={ + "User-Agent": firefox_user_agent(), + "Accept-Language": "en-US,en;q=0.5", + "Referer": "https://discord.com/", + }), + ) + + def download_post( + self, + post: dict, + media_items: list, + artist_slug: str, + *, + is_seen: Callable[[object], bool] = lambda m: False, + should_stop: Callable[[], bool] = lambda: False, + recapture: bool = False, + ) -> list[MediaOutcome]: + """Every file of one message; per-file outcomes, one failure isolated.""" + folder = channel_dir(self.images_root, artist_slug, post) + outcomes: list[MediaOutcome] = [] + for media in media_items: + if should_stop(): + break + try: + outcomes.append(self._download_one( + post, media, folder, artist_slug, is_seen, recapture=recapture, + )) + except Exception as exc: # resilient: isolate one item's failure + log.warning( + "Discord media failed (message %s, file %d): %s", + post.get("id"), media.num, exc, + ) + outcomes.append( + MediaOutcome(media=media, status="error", path=None, error=str(exc)) + ) + return outcomes + + def _download_one( + self, + post: dict, + media, + folder: Path, + artist_slug: str, + is_seen: Callable[[object], bool], + *, + recapture: bool = False, + ) -> MediaOutcome: + seen = is_seen(media) + if seen and not recapture: + return MediaOutcome(media=media, status="skipped_seen", path=None, error=None) + stem = media_stem(post, media) + path = folder / f"{stem}.{media.extension or _NO_EXTENSION}" + if path.exists(): # tier-2: gallery-dl (or an earlier walk) wrote it + return MediaOutcome(media=media, status="skipped_disk", path=path, error=None) + if seen: # recapture never re-fetches a seen file that is gone + return MediaOutcome(media=media, status="skipped_seen", path=None, error=None) + + folder.mkdir(parents=True, exist_ok=True) + if self._rate_limit > 0: + time.sleep(self._rate_limit) + out = self._fetch_get(media.url, path) + reason, quarantined = self._validate_path(out, artist_slug, media.url) + if reason is not None: + return MediaOutcome(media=media, status="quarantined", path=quarantined, error=reason) + sidecar = {"category": PLATFORM, "message_id": str(post.get("id") or "")} + sidecar["source_url"] = media.url + (folder / f"{stem}.json").write_text(json.dumps(sidecar, indent=2)) + return MediaOutcome(media=media, status="downloaded", path=out, error=None) + + def write_post_record( + self, post: dict, artist_slug: str, *, revisit: bool = False, + ) -> PostRecordOutcome: + """The message record — the one writer of a Discord post's body, date + and permalink ids. `revisit` re-reads a message already captured (an + edit); an empty re-read writes nothing, so it never blanks a body.""" + mid = str(post.get("id") or "") + body = message_text(post) + if not mid or (revisit and not body.strip()): + return PostRecordOutcome(path=None, post_type=None, title=None, body_chars=0) + meta = post.get("_meta") or {} + author = post.get("author") or {} + record = { + "category": PLATFORM, + "message_id": mid, + "server": meta.get("server"), + "server_id": meta.get("server_id"), + "channel": meta.get("channel"), + "channel_id": meta.get("channel_id") or post.get("channel_id"), + "parent": meta.get("parent"), + "is_thread": meta.get("is_thread"), + "author": author.get("username"), + "author_id": author.get("id"), + "message": body, + "date": post.get("timestamp"), + } + folder = channel_dir(self.images_root, artist_slug, post) + folder.mkdir(parents=True, exist_ok=True) + when = message_date(post) + day = f"{when:%Y%m%d}" if when else "None" + path = folder / f"{day}_{mid}_post.json" + path.write_text(json.dumps( + {k: v for k, v in record.items() if v is not None}, indent=2, + )) + return PostRecordOutcome( + path=path, post_type=None, title=None, body_chars=len(body), + ) diff --git a/backend/app/services/discord_ingester.py b/backend/app/services/discord_ingester.py new file mode 100644 index 0000000..f83721f --- /dev/null +++ b/backend/app/services/discord_ingester.py @@ -0,0 +1,85 @@ +"""Native Discord ingester — the Discord ADAPTER over `ingest_core.Ingester`. + +Thin counterpart to subscribestar_ingester (milestone 428). The walk's modes, +both ledgers, cursor checkpointing and the post-first capture live in the core; +this wires in the Discord client, downloader, ledger models and key. + +Two things differ from the cookie platforms: + + - Discord authenticates with a user TOKEN, so `auth_token` is the credential + here rather than an argument accepted and ignored. + - The body canary is off. It fails a walk whose first 30+ captured posts all + came back without text, on the theory that a creator nearly always writes + something; a Discord drop is routinely files and nothing else, so on + Discord that is an ordinary backfill, not a broken parser. + +`campaign_id` is the source URL (a server, channel, thread or category link). +FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable +from pathlib import Path + +from ..models import DiscordFailedMedia, DiscordSeenMedia +from .discord_client import DiscordAPIError, DiscordClient, MediaItem +from .discord_downloader import DiscordDownloader +from .ingest_core import Ingester + +_LEDGER_KEY_MAX = 128 + + +def _ledger_key(media: MediaItem) -> str: + """`:` — stable across edits (see MediaItem).""" + return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX] + + +class DiscordIngester(Ingester): + """Walk a Discord source's channels, download unseen files, return a + `DownloadResult`. `client` / `downloader` are injectable for tests.""" + + def __init__( + self, + images_root: Path, + cookies_path: str | None, + session_factory: Callable[[], object], + *, + validate: bool = True, + rate_limit: float = 0.0, + request_sleep: float = 0.0, + auth_token: str | None = None, + client: DiscordClient | None = None, + downloader: DiscordDownloader | None = None, + ): + del cookies_path # Discord authenticates by token (uniform signature) + self.images_root = Path(images_root) + super().__init__( + client=client if client is not None else DiscordClient( + auth_token, request_sleep=request_sleep, + ), + downloader=downloader if downloader is not None else DiscordDownloader( + self.images_root, validate=validate, rate_limit=rate_limit, + ), + session_factory=session_factory, + seen_model=DiscordSeenMedia, + failed_model=DiscordFailedMedia, + seen_constraint="uq_discord_seen_media_source_id", + failed_constraint="uq_discord_failed_media_source_id", + ledger_key=_ledger_key, + platform="discord", + error_base=DiscordAPIError, + drift_label="Discord API", + body_canary=False, + ) + + +async def verify_discord_credential(url: str, auth_token: str | None) -> tuple[bool | None, str]: + """The uniform `(ok, message)` probe: is the token valid, and can its + account see the channel or server the source names?""" + if not auth_token: + return False, "No Discord token is saved — add one under Credentials." + client = DiscordClient(auth_token) + loop = asyncio.get_running_loop() + return await loop.run_in_executor(None, client.verify_auth, url) diff --git a/backend/app/services/download_backends.py b/backend/app/services/download_backends.py index e2382f7..74fc88b 100644 --- a/backend/app/services/download_backends.py +++ b/backend/app/services/download_backends.py @@ -23,6 +23,7 @@ from __future__ import annotations import asyncio from pathlib import Path +from .discord_ingester import DiscordIngester from .gallery_dl import DownloadResult, ErrorType from .ingest_core import DEFAULT_REVISIT_DAYS from .patreon_ingester import PatreonIngester @@ -31,9 +32,13 @@ from .platforms import known_platform_keys from .subscribestar_ingester import SubscribeStarIngester # Platforms whose download + verify go through the native ingester rather than -# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until -# they migrate too. -NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"}) +# gallery-dl. gallery-dl still serves the rest (hentaifoundry) until it +# migrates too. Discord joined in milestone 428. +NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"}) + +# Native platforms whose feed id IS the source URL, so there is nothing to +# resolve: SubscribeStar's creator page, Discord's server/channel link. +_URL_IS_FEED = frozenset({"subscribestar", "discord"}) def _unsupported_platform_message(platform: str) -> str | None: @@ -67,6 +72,7 @@ def _native_ingester_cls(platform: str): return { "patreon": PatreonIngester, "subscribestar": SubscribeStarIngester, + "discord": DiscordIngester, }[platform] @@ -126,10 +132,10 @@ async def _resolve_native_campaign_id( platform: str, url: str, cookies_path: str | None, overrides: dict, ) -> tuple[str | None, str | None]: """`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's - feed id IS the creator URL (no lookup → resolved None). Patreon resolves the - campaign id from the vanity URL (resolved non-None when a lookup actually ran, - so phase 3 caches it).""" - if platform == "subscribestar": + and Discord's feed id IS the source URL (no lookup → resolved None). Patreon + resolves the campaign id from the vanity URL (resolved non-None when a lookup + actually ran, so phase 3 caches it).""" + if platform in _URL_IS_FEED: return url, None return await resolve_campaign_id_for_source(url, cookies_path, overrides) @@ -252,6 +258,10 @@ async def verify_source_credential( from .subscribestar_ingester import verify_subscribestar_credential return await verify_subscribestar_credential(url, cookies_path, config_overrides) + if platform == "discord": + from .discord_ingester import verify_discord_credential + + return await verify_discord_credential(url, auth_token) from .patreon_ingester import verify_patreon_credential return await verify_patreon_credential(url, cookies_path, config_overrides) diff --git a/backend/app/services/ingest_core.py b/backend/app/services/ingest_core.py index 6030cad..ffc461a 100644 --- a/backend/app/services/ingest_core.py +++ b/backend/app/services/ingest_core.py @@ -264,6 +264,14 @@ class Ingester: # operator-driven "re-read every body" pass; a horizon there would be a # third overlapping answer to a question that has two. post_meta = getattr(self.client, "post_meta", None) + # #4413: optional client seam for a source that is several feeds walked + # one after another (a Discord server: every channel and thread). The + # tick early-out means "this feed has nothing new", and without the + # seam it ends the WHOLE walk — so the first quiet channel would hide + # every channel after it. With it, the early-out asks the client to + # move on and the walk continues. Absent → the early-out ends the walk, + # exactly as before (Patreon and SubscribeStar are one feed each). + skip_feed = getattr(self.client, "skip_feed", None) horizon: datetime | None = None if mode == "tick" and revisit_days > 0 and post_meta is not None: horizon = datetime.now(UTC) - timedelta(days=revisit_days) @@ -312,6 +320,7 @@ class Ingester: reached_bottom = False budget_hit = False early_out = False + feeds_caught_up = 0 # #4413: feeds a tick left early via skip_feed stopped = False # plan #708 B4: operator hit Stop mid-walk cancel_armed = False # latched once we observe a live "running" state @@ -674,9 +683,15 @@ class Ingester: }) if early_out: - break + if skip_feed is None: + break + skip_feed() + feeds_caught_up += 1 + early_out = False + consecutive_seen = 0 else: - reached_bottom = True + # A walk that left feeds early did not read to their ends. + reached_bottom = not feeds_caught_up except self._error_base as exc: # The platform's client-error base — _failure_result (adapter) # maps it to a typed error. @@ -724,6 +739,7 @@ class Ingester: f", {revisited} post(s) updated ({revisit_downloads} new file(s))" if revisited else "" ) + + (f", {feeds_caught_up} feed(s) caught up" if feeds_caught_up else "") + (", reached end" if reached_bottom else "") + (", time-boxed" if budget_hit else "") ) diff --git a/backend/app/services/platform_lock.py b/backend/app/services/platform_lock.py index 27d029e..d910016 100644 --- a/backend/app/services/platform_lock.py +++ b/backend/app/services/platform_lock.py @@ -23,8 +23,10 @@ log = logging.getLogger(__name__) # Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT # here: each runs as a self-pacing subprocess and they're lower-volume. The # native-ingester platforms are serialized (one paced scrape/API walk at a time). -# Add a platform here to cap it to a single concurrent walk. -SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar"}) +# Add a platform here to cap it to a single concurrent walk. Discord most of +# all: every source walks on the operator's ONE user token, and parallel walks +# on a user account are both how its rate limit trips and what gets it flagged. +SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"}) _LOCK_PREFIX = "fc:download_lock:" diff --git a/tests/test_discord_client.py b/tests/test_discord_client.py new file mode 100644 index 0000000..230afde --- /dev/null +++ b/tests/test_discord_client.py @@ -0,0 +1,339 @@ +"""The native Discord client walks what gallery-dl walked, in its order (#4412). + +No network: a fake session answers by endpoint. What these pin is the part of +gallery-dl's behaviour that decides WHICH files exist and WHAT they are called — +the channel walk, the file list and its numbering, the name split — because a +difference there is a re-download or a missed file at cutover, not a style +choice. +""" + +from __future__ import annotations + +from datetime import date + +import pytest + +from backend.app.services import discord_client as dc +from backend.app.services.discord_client import ( + DiscordAPIError, + DiscordAuthError, + DiscordClient, + firefox_user_agent, + message_text, + nameext_from_url, + parse_source_url, +) + + +class _Resp: + def __init__(self, status, body=None, headers=None): + self.status_code = status + self._body = body + self.headers = headers or {} + self.content = b"x" + + def json(self): + return self._body + + +class _Session: + """Answers `GET API_ROOT + endpoint` from a dict. A value may be a list of + responses, served in order; a messages entry is keyed by (endpoint, before).""" + + def __init__(self, routes): + self.routes = routes + self.headers = {} + self.calls = [] + + def get(self, url, params=None, timeout=None): + endpoint = url[len(dc.API_ROOT):] + self.calls.append((endpoint, dict(params or {}))) + key = endpoint + if endpoint.endswith("/messages"): + key = (endpoint, (params or {}).get("before")) + elif endpoint.endswith("/threads/search"): + key = (endpoint, (params or {}).get("offset")) + answer = self.routes.get(key, _Resp(404, {})) + if isinstance(answer, list): + return answer.pop(0) + return answer + + +def _ok(body): + return _Resp(200, body) + + +def _msg(mid, *, content="", attachments=(), embeds=(), **extra): + return { + "id": str(mid), "type": 0, "content": content, + "timestamp": "2026-09-20T12:00:00.000000+00:00", + "author": {"id": "7", "username": "artist"}, + "attachments": list(attachments), "embeds": list(embeds), **extra, + } + + +def _client(routes, **kw): + return DiscordClient("tok", session=_Session(routes), **kw) + + +def _ids(client, url, cursor=None): + return [(m["id"], cur) for m, _meta, cur in client.iter_posts(url, cursor)] + + +# -- pure helpers -------------------------------------------------------------- + +def test_source_urls(): + assert parse_source_url("https://discord.com/channels/1/2") == ("1", "2") + assert parse_source_url("https://discord.com/channels/1") == ("1", None) + assert parse_source_url("https://discord.com/channels/1/2/threads/3") == ("1", "3") + assert parse_source_url("https://discord.com/channels/@me/5") == (None, "5") + assert parse_source_url("discord.com/channels/1/2/") == ("1", "2") + + +def test_a_message_link_is_not_a_source(): + with pytest.raises(DiscordAPIError): + parse_source_url("https://discord.com/channels/1/2/3") + with pytest.raises(DiscordAPIError): + parse_source_url("https://example.com/channels/1/2") + + +def test_name_split_matches_gallery_dl(): + url = "https://cdn.discordapp.com/attachments/1/2/My%20Pic.final.PNG?ex=a&hm=b" + assert nameext_from_url(url) == ("My Pic.final", "png") + assert nameext_from_url("https://x/y/noext") == ("noext", "") + assert nameext_from_url("https://x/y/a." + "b" * 17) == ("a." + "b" * 17, "") + + +def test_user_agent_is_gallery_dls_dated_firefox(): + """gallery-dl's own comment: "147 on 2026-01-13".""" + assert "Firefox/147.0" in firefox_user_agent(date(2026, 1, 13)) + + +def test_message_text_takes_rich_embeds_and_polls(): + m = _msg(1, content="hello", embeds=[ + {"type": "rich", "author": {"name": "A"}, "title": "T", + "fields": [{"name": "f", "value": "v"}], "footer": {"text": "ft"}}, + {"type": "image", "title": "not text"}, + ], poll={"question": {"text": "Q?"}, "answers": [{"poll_media": {"text": "yes"}}]}) + assert message_text(m) == "hello\nA\nT\nf\nv\nft\nQ?\nyes" + + +def test_files_are_attachments_then_embeds_then_snapshots_numbered_across(): + m = _msg(9, attachments=[{"url": "https://cdn/a/1.png"}], embeds=[ + {"type": "video", "video": {"proxy_url": "https://media/v.mp4"}, + "thumbnail": {"proxy_url": "https://media/t.jpg"}}, + {"type": "image", "thumbnail": {"proxy_url": "https://media/i.webp"}}, + {"type": "rich", "image": {"proxy_url": "https://media/rich.png"}}, + ], message_snapshots=[ + {"message": {"type": 0, "attachments": [{"url": "https://cdn/a/fwd.gif"}], + "embeds": []}}, + {"message": {"type": 7, "attachments": [{"url": "https://cdn/a/join.png"}]}}, + ]) + items = DiscordClient.extract_media(m) + assert [(i.num, i.filename, i.extension, i.kind) for i in items] == [ + (1, "1", "png", "attachment"), + (2, "v", "mp4", "embed"), + (3, "i", "webp", "embed"), + (4, "fwd", "gif", "attachment"), + ] + assert {i.post_id for i in items} == {"9"} + + +def test_the_ledger_identity_survives_a_renumbering_edit(): + """Removing the first file renumbers the second; its identity must not move.""" + a = {"id": "100", "url": "https://cdn/a/1.png?ex=1"} + b = {"id": "200", "url": "https://cdn/a/2.png?ex=1"} + before = DiscordClient.extract_media(_msg(9, attachments=[a, b])) + after = DiscordClient.extract_media(_msg(9, attachments=[b])) + assert (before[1].num, after[0].num) == (2, 1) + assert before[1].media_id == after[0].media_id == "200" + + +def test_an_embeds_identity_ignores_its_signature(): + def embed(sig): + return {"type": "image", "image": {"proxy_url": f"https://media/p/x.png?ex={sig}"}} + + one = DiscordClient.extract_media(_msg(9, embeds=[embed("a")]))[0].media_id + two = DiscordClient.extract_media(_msg(9, embeds=[embed("b")]))[0].media_id + assert one == two and len(one) <= 33 + + +def test_post_seams(): + with_file = _msg(5, attachments=[{"url": "https://cdn/a/1.png"}]) + assert DiscordClient.post_record_key(with_file) == ("message:5", "5") + assert DiscordClient.post_record_key({}) is None + + +def test_a_text_only_message_is_not_a_post(): + """gallery-dl never made one: chat lines would bury the drops.""" + assert DiscordClient.post_record_key(_msg(6, content="brb")) is None + assert DiscordClient.post_meta(_msg(1))["date"].startswith("2026-09-20") + + +# -- the walk -------------------------------------------------------------------- + +def test_a_channel_pages_newest_first_and_skips_system_messages(): + page1 = [_msg(i) for i in range(300, 200, -1)] + page1[3]["type"] = 7 # a member-join line: not content + routes = { + "/guilds/1": _ok({"id": "1", "name": "S"}), + "/guilds/1/channels": _ok([{"id": "2", "type": 0, "name": "art"}]), + ("/channels/2/messages", None): _ok(page1), + ("/channels/2/messages", "201"): _ok([_msg(150)]), + ("/channels/2/threads/search", 0): _ok({"threads": []}), + } + got = _ids(_client(routes), "https://discord.com/channels/1/2") + assert len(got) == 100 # 99 of page 1 + 1 of page 2 + assert "297" not in [mid for mid, _ in got] + assert got[0] == ("300", "2:") + assert got[-1] == ("150", "2:201") + + +def test_messages_carry_server_and_channel_metadata(): + routes = { + "/guilds/1": _ok({"id": "1", "name": "Studio", "owner_id": "9"}), + "/guilds/1/channels": _ok([ + {"id": "4", "type": 4, "name": "Art"}, + {"id": "2", "type": 0, "name": "drops", "parent_id": "4"}, + ]), + ("/channels/2/messages", None): _ok([_msg(10)]), + ("/channels/2/threads/search", 0): _ok({"threads": []}), + } + [(msg, meta, _)] = list(_client(routes).iter_posts("https://discord.com/channels/1/2")) + assert msg["_meta"] is meta + assert meta["server"] == "Studio" and meta["server_id"] == "1" + assert meta["channel"] == "drops" and meta["channel_id"] == "2" + assert meta["parent"] == "Art" + + +def test_a_server_walks_text_then_threads_newest_created_first_and_skips_private(): + routes = { + "/guilds/1": _ok({"id": "1", "name": "S"}), + "/guilds/1/channels": _ok([ + {"id": "2", "type": 0, "name": "text"}, + {"id": "3", "type": 2, "name": "voice"}, + {"id": "5", "type": 15, "name": "forum"}, + {"id": "6", "type": 0, "name": "private"}, + ]), + ("/channels/2/messages", None): _ok([_msg(20)]), + ("/channels/2/threads/search", 0): _ok({"threads": [ + {"id": "21", "type": 11, "name": "old", "parent_id": "2", "thread_metadata": {}}, + {"id": "22", "type": 11, "name": "new", "parent_id": "2", "thread_metadata": {}}, + ]}), + ("/channels/22/messages", None): _ok([_msg(220)]), + ("/channels/21/messages", None): _Resp(403, {}), # a private thread + ("/channels/5/threads/search", 0): _ok({"threads": [ + {"id": "51", "type": 11, "name": "post", "parent_id": "5", "thread_metadata": {}}, + ]}), + ("/channels/51/messages", None): _ok([_msg(510)]), + ("/channels/6/messages", None): _Resp(403, {}), + ("/channels/6/threads/search", 0): _Resp(403, {}), + } + got = [mid for mid, _ in _ids(_client(routes), "https://discord.com/channels/1")] + assert got == ["20", "220", "510"] + + +def test_a_resume_cursor_reenters_its_channel_at_its_page(): + routes = { + "/guilds/1": _ok({"id": "1", "name": "S"}), + "/guilds/1/channels": _ok([ + {"id": "2", "type": 0, "name": "a"}, + {"id": "3", "type": 0, "name": "b"}, + ]), + ("/channels/2/threads/search", 0): _ok({"threads": []}), + ("/channels/3/threads/search", 0): _ok({"threads": []}), + ("/channels/3/messages", "77"): _ok([_msg(70)]), + } + client = _client(routes) + assert _ids(client, "https://discord.com/channels/1", "3:77") == [("70", "3:77")] + fetched = [c for c in client._session.calls if c[0].endswith("/messages")] + assert fetched == [("/channels/3/messages", {"limit": 100, "before": "77"})] + + +def test_skip_feed_ends_the_channel_not_the_walk(): + routes = { + "/guilds/1": _ok({"id": "1", "name": "S"}), + "/guilds/1/channels": _ok([ + {"id": "2", "type": 0, "name": "a"}, + {"id": "3", "type": 0, "name": "b"}, + ]), + ("/channels/2/messages", None): _ok([_msg(29), _msg(28)]), + ("/channels/2/threads/search", 0): _ok({"threads": []}), + ("/channels/3/messages", None): _ok([_msg(39)]), + ("/channels/3/threads/search", 0): _ok({"threads": []}), + } + client = _client(routes) + seen = [] + for msg, _meta, _cur in client.iter_posts("https://discord.com/channels/1"): + seen.append(msg["id"]) + if msg["id"] == "29": + client.skip_feed() + assert seen == ["29", "39"] + + +def test_the_named_channel_refusing_the_token_is_an_auth_failure(): + routes = { + "/guilds/1": _ok({"id": "1", "name": "S"}), + "/guilds/1/channels": _ok([{"id": "2", "type": 0, "name": "a"}]), + ("/channels/2/messages", None): _Resp(403, {}), + } + with pytest.raises(DiscordAuthError): + list(_client(routes).iter_posts("https://discord.com/channels/1/2")) + + +def test_401_is_an_invalid_token(): + with pytest.raises(DiscordAuthError): + list(_client({"/guilds/1": _Resp(401, {})}).iter_posts( + "https://discord.com/channels/1")) + + +def test_no_token_fails_before_any_request(): + client = DiscordClient(None, session=_Session({})) + with pytest.raises(DiscordAuthError): + list(client.iter_posts("https://discord.com/channels/1")) + assert client._session.calls == [] + + +def test_429_waits_and_retries(monkeypatch): + waits = [] + monkeypatch.setattr(dc.time, "sleep", waits.append) + routes = {"/users/@me": [ + _Resp(429, {}, {"Retry-After": "1.5"}), _ok({"username": "me"}), + ], "/channels/2": _ok({"id": "2", "type": 0})} + ok, msg = _client(routes).verify_auth("https://discord.com/channels/1/2") + assert ok is True and "me" in msg + assert waits == [1.5] + + +def test_verify_tells_a_bad_token_from_a_hidden_channel(): + bad = _client({"/users/@me": _Resp(401, {})}) + assert bad.verify_auth("https://discord.com/channels/1/2")[0] is False + hidden = _client({"/users/@me": _ok({"username": "me"}), + "/channels/2": _Resp(403, {})}) + ok, msg = hidden.verify_auth("https://discord.com/channels/1/2") + assert ok is False and "cannot see this channel" in msg + assert _client({}).verify_auth("https://discord.com/channels/1/2/3")[0] is None + + +def test_the_request_profile_is_gallery_dls(): + client = _client({}) + h = client._session.headers + assert h["Authorization"] == "tok" + assert h["Referer"] == "https://discord.com/" + assert h["Accept"] == "*/*" + assert h["User-Agent"].startswith("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:") + + +# -- the adapter --------------------------------------------------------------- + +def test_the_adapter_authenticates_with_the_token_and_keys_by_identity(tmp_path): + from backend.app.services.discord_ingester import DiscordIngester, _ledger_key + + ing = DiscordIngester(tmp_path, None, session_factory=None, auth_token="tok") + assert ing.client._session.headers["Authorization"] == "tok" + # A files-only drop is ordinary on Discord, not a broken parser. + assert ing._body_canary is False + [media] = DiscordClient.extract_media( + _msg(9, attachments=[{"id": "300", "url": "https://cdn/a/1.png"}]) + ) + assert _ledger_key(media) == "9:300" diff --git a/tests/test_discord_downloader.py b/tests/test_discord_downloader.py new file mode 100644 index 0000000..711f4ca --- /dev/null +++ b/tests/test_discord_downloader.py @@ -0,0 +1,135 @@ +"""The native Discord downloader lands files where gallery-dl did (#4414). + +A cutover that names one file differently re-downloads it and imports a +duplicate, so these pin the path, the name cleaning and the sidecar pairing +against what gallery-dl's config produces — and that the records it writes read +back through `parse_sidecar` as the same post the gallery-dl sidecars made. +""" + +from __future__ import annotations + +import json + +from backend.app.services.discord_client import DiscordClient +from backend.app.services.discord_downloader import ( + DiscordDownloader, + channel_dir, + gdl_clean, + media_stem, +) +from backend.app.utils.sidecar import find_sidecar, parse_sidecar + +_PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 16 + b"\x00\x00\x00\x00IEND\xaeB`\x82" + + +class _Resp: + status_code = 200 + headers: dict = {} + + def raise_for_status(self): + pass + + def iter_content(self, chunk_size=None): + yield _PNG + + +class _Media: + def __init__(self): + self.calls = [] + + def get(self, url, stream=None, timeout=None, headers=None): + self.calls.append(url) + return _Resp() + + +def _message(**extra): + return { + "id": "1234", "type": 0, "timestamp": "2026-09-20T23:30:00.000000+00:00", + "content": "new set!", "channel_id": "22", + "author": {"id": "7", "username": "artist"}, + "attachments": [{"url": "https://cdn.discordapp.com/attachments/2/3/Red%3AAlt.PNG?ex=1"}], + "embeds": [], + "_meta": {"server": "Studio", "server_id": "11", "channel": "drops/nsfw", + "channel_id": "22", "is_thread": False}, + **extra, + } + + +def _downloader(tmp_path, session=None): + return DiscordDownloader(tmp_path, validate=False, session=session or _Media()) + + +def test_names_follow_gallery_dls_pattern_and_linux_cleaning(tmp_path): + msg = _message() + [media] = DiscordClient.extract_media(msg) + assert channel_dir(tmp_path, "art", msg) == tmp_path / "art" / "discord" / "drops_nsfw" + # `:` survives: gallery-dl on Linux only replaces `/`. + assert media_stem(msg, media) == "20260920_1234_01_Red:Alt" + assert gdl_clean("a\x07b/c") == "ab_c" + + +def test_a_channel_with_no_name_adds_no_directory(tmp_path): + msg = _message(_meta={"channel": " "}) + assert channel_dir(tmp_path, "art", msg) == tmp_path / "art" / "discord" + + +def test_a_file_gallery_dl_already_wrote_is_not_fetched_again(tmp_path): + msg = _message() + media = DiscordClient.extract_media(msg) + existing = tmp_path / "art" / "discord" / "drops_nsfw" / "20260920_1234_01_Red:Alt.png" + existing.parent.mkdir(parents=True) + existing.write_bytes(_PNG) + session = _Media() + [out] = _downloader(tmp_path, session).download_post(msg, media, "art") + assert out.status == "skipped_disk" and out.path == existing + assert session.calls == [] + + +def test_a_new_file_gets_a_sidecar_the_importer_pairs_to_its_message(tmp_path): + msg = _message() + [out] = _downloader(tmp_path).download_post(msg, DiscordClient.extract_media(msg), "art") + assert out.status == "downloaded" + assert out.path.name == "20260920_1234_01_Red:Alt.png" + sidecar = find_sidecar(out.path) + assert sidecar is not None + data = json.loads(sidecar.read_text()) + # No `id`/`post_id`: either would outrank message_id as the post id. + assert "id" not in data and "post_id" not in data + sd = parse_sidecar(data) + assert sd.external_post_id == "1234" + assert sd.source_url.startswith("https://cdn.discordapp.com/") + + +def test_a_seen_file_is_skipped_without_a_request(tmp_path): + msg = _message() + session = _Media() + [out] = _downloader(tmp_path, session).download_post( + msg, DiscordClient.extract_media(msg), "art", is_seen=lambda m: True, + ) + assert out.status == "skipped_seen" and session.calls == [] + + +def test_the_message_record_reads_back_as_the_gallery_dl_post(tmp_path): + rec = _downloader(tmp_path).write_post_record(_message(), "art") + assert rec.path.name == "20260920_1234_post.json" + assert rec.body_chars == len("new set!") + sd = parse_sidecar(json.loads(rec.path.read_text())) + assert sd.platform == "discord" + assert sd.external_post_id == "1234" + assert sd.post_url == "https://discord.com/channels/11/22/1234" + assert sd.description == "new set!" + assert sd.post_date.isoformat().startswith("2026-09-20T23:30") + + +def test_the_record_is_not_a_media_sidecar(tmp_path): + """It must not pair with any file of the message.""" + msg = _message() + dl = _downloader(tmp_path) + [out] = dl.download_post(msg, DiscordClient.extract_media(msg), "art") + rec = dl.write_post_record(msg, "art") + assert find_sidecar(out.path) != rec.path + + +def test_an_empty_re_read_never_blanks_a_stored_body(tmp_path): + rec = _downloader(tmp_path).write_post_record(_message(content=""), "art", revisit=True) + assert rec.path is None diff --git a/tests/test_download_backends.py b/tests/test_download_backends.py index 5ee3f6b..932a2f6 100644 --- a/tests/test_download_backends.py +++ b/tests/test_download_backends.py @@ -17,7 +17,7 @@ from backend.app.services.gallery_dl import ErrorType def test_native_platforms(): - for platform in ("patreon", "subscribestar"): + for platform in ("patreon", "subscribestar", "discord"): assert uses_native_ingester(platform) is True assert platform in NATIVE_INGESTER_PLATFORMS @@ -104,8 +104,20 @@ async def test_verifying_a_retired_platform_is_inconclusive_not_rejected(): def test_gallery_dl_platforms_are_not_native(): # The platforms still served by gallery-dl must NOT route to the native # ingester — guards an accidental over-broad migration. - for platform in ("hentaifoundry", "discord"): - assert uses_native_ingester(platform) is False + assert uses_native_ingester("hentaifoundry") is False + + +@pytest.mark.asyncio +async def test_discord_verify_without_a_token_is_a_rejection_not_a_request(): + """Discord authenticates by token (milestone 428); with none saved there is + nothing to send, and saying so beats an HTTP 401 from Discord.""" + ok, message = await verify_source_credential( + platform="discord", url="https://discord.com/channels/1/2", + artist_slug="someone", config_overrides=None, cookies_path=None, + auth_token=None, images_root=Path("/nonexistent"), + ) + assert ok is False + assert "token" in message.lower() def test_unknown_platform_is_not_native(): diff --git a/tests/test_patreon_ingester.py b/tests/test_patreon_ingester.py index 1e344d7..8d158bc 100644 --- a/tests/test_patreon_ingester.py +++ b/tests/test_patreon_ingester.py @@ -319,6 +319,64 @@ async def test_tick_early_out_after_threshold(source_id, sync_engine, tmp_path): assert client.consumed_posts == 2 +class _FeedsClient(_FakeClient): + """A source that is several feeds walked in turn (a Discord server's + channels), with the optional `skip_feed` seam (#4413). `feeds` is a list of + `pages` lists, one per feed.""" + + def __init__(self, feeds): + super().__init__([page for pages in feeds for page in pages]) + self._feeds = feeds + self._skip = False + self.skips = 0 + + def skip_feed(self): + self._skip = True + self.skips += 1 + + def iter_posts(self, campaign_id, cursor=None): + for pages in self._feeds: + self._skip = False + self._pages = pages + for item in super().iter_posts(campaign_id, cursor): + yield item + if self._skip: + break + + +def _seed_seen_media(sync_engine, source_id, media): + factory = sessionmaker(sync_engine, expire_on_commit=False) + with factory() as s: + for m in media: + s.add(PatreonSeenMedia(source_id=source_id, filehash=_ledger_key(m), post_id=m.post_id)) + s.commit() + + +@pytest.mark.asyncio +async def test_a_quiet_feed_ends_itself_not_the_walk(source_id, sync_engine, tmp_path): + """#4413: a Discord server's first channel is all seen; the tick must still + reach the second channel's new file, instead of stopping at the first.""" + quiet = [_media(f"a{i}", 1) for i in range(1, 5)] + _seed_seen_media(sync_engine, source_id, quiet) + fresh = _media("b1", 1) + client = _FeedsClient([ + [(None, [(m.post_id, [m]) for m in quiet])], + [(None, [("b1", [fresh])])], + ]) + downloader = _FakeDownloader(tmp_path) + result = _ingester(sync_engine, tmp_path, client, downloader).run( + source_id=source_id, campaign_id="c1", artist_slug="ingest", + url="https://patreon.com/ingest", mode="tick", seen_threshold=2, + ) + assert result.success is True + assert client.skips == 1 + # The quiet feed stopped after its 2nd seen post; the next feed was walked. + assert client.consumed_posts == 3 + assert downloader.download_calls == 1 + assert "1 feed(s) caught up" in result.stdout + assert "reached end" not in result.stdout + + # --- backfill ------------------------------------------------------------- diff --git a/tests/test_platform_lock.py b/tests/test_platform_lock.py index b677e1e..c4def8b 100644 --- a/tests/test_platform_lock.py +++ b/tests/test_platform_lock.py @@ -10,7 +10,11 @@ pytestmark = pytest.mark.integration def test_non_serialized_platform_has_no_lock(): # gallery-dl platforms aren't capped — they get no lock at all. assert platform_lock("hentaifoundry", ttl_seconds=60) is None - assert platform_lock("discord", ttl_seconds=60) is None + + +def test_discord_is_serialized(): + # Native since milestone 428, and every source shares one user token. + assert platform_lock("discord", ttl_seconds=60) is not None def test_subscribestar_is_serialized(): -- 2.54.0 From e5bdcd25968a05369f319201b59be8c8fc4b3d86 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Thu, 24 Sep 2026 23:14:48 -0400 Subject: [PATCH 82/94] =?UTF-8?q?feat:=20finish=20the=20Discord=20switchov?= =?UTF-8?q?er=20=E2=80=94=20recapture=20on=20every=20native=20source,=20ba?= =?UTF-8?q?ckfills=20that=20run,=20gallery-dl's=20Discord=20config=20retir?= =?UTF-8?q?ed=20(milestone=20428)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Recover and Recapture show on every native source. The menu gated them on a copied platform list ('patreon', 'subscribestar') that went stale when Discord moved over. Sources now carry `native_ingester` from the backend's own predicate. - A running backfill is due on every scheduler tick. Nothing queued a backfill's next chunk: each one waited for the source's regular interval, so an armed backfill sat idle until the next check (8h at the default) and a five-chunk walk took most of two days. The in-flight guard and the platform lock keep one chunk at a time. A failing source falls back to its backoff, and a stalled or out-of-budget walk stops being due. It also runs when the artist has auto-check off, since the operator started it by hand. - gallery-dl no longer carries Discord: its naming constants, platform defaults, sidecar-mirroring postprocessor and token injection are gone. The naming test moves to the native downloader and still renders against the real gallery-dl sidecar fixture. That is the guard that the files gallery-dl wrote are found on disk rather than fetched again. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR --- backend/app/services/discord_downloader.py | 8 +- backend/app/services/gallery_dl.py | 77 +----------- backend/app/services/scheduler_service.py | 37 +++++- backend/app/services/source_service.py | 10 +- .../subscriptions/SourceActions.vue | 8 +- .../subscriptions/SubscriptionsTab.vue | 2 +- frontend/src/stores/sources.js | 2 +- tests/test_discord_naming.py | 69 +++++++++++ tests/test_gallery_dl_naming.py | 117 ------------------ tests/test_scheduler_service.py | 62 ++++++++++ tests/test_source_service.py | 18 +++ 11 files changed, 208 insertions(+), 202 deletions(-) create mode 100644 tests/test_discord_naming.py delete mode 100644 tests/test_gallery_dl_naming.py diff --git a/backend/app/services/discord_downloader.py b/backend/app/services/discord_downloader.py index ffd8734..d83e354 100644 --- a/backend/app/services/discord_downloader.py +++ b/backend/app/services/discord_downloader.py @@ -6,9 +6,11 @@ existing file on disk (`skipped_disk`) instead of fetching it again: //discord//___. -That is FC's gallery-dl config (`gallery_dl.DISCORD_DIRECTORY` / -`DISCORD_FILENAME`) under the per-source base directory -`//`. The name is cleaned the way gallery-dl cleans it +That is what FC's gallery-dl config produced (directory `{channel}`, filename +`{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, under the +per-source base directory `//`), retired from that +config once Discord moved here; tests/test_discord_naming.py pins the match +against a real gallery-dl sidecar. The name is cleaned the way gallery-dl cleans it on Linux — `/` becomes `_` and control characters are removed, nothing else (`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`, whose Windows set would turn a `:` in a channel or file name into `_` and miss diff --git a/backend/app/services/gallery_dl.py b/backend/app/services/gallery_dl.py index 90933bf..4ee68ae 100644 --- a/backend/app/services/gallery_dl.py +++ b/backend/app/services/gallery_dl.py @@ -95,48 +95,6 @@ BACKFILL_CHUNK_SECONDS = 600 _DEFAULT_GDL_TIMEOUT_SECONDS = 870 -# --- Discord naming --------------------------------------------------------- -# -# Derived from a REAL sidecar (operator's instance, 2026-09-13), not from memory -# of gallery-dl's extractor. What gallery-dl's discord extractor actually emits -# for an attachment: `channel` is a plain STRING (the channel's name), the -# message is `message_id`, the attachment's position in it is `num`, and there -# is NO `id` key at all. -# -# The previous patterns asked for `{channel[name]}` and `{id}`. Both render as -# "None", so every Discord download since the platform was added landed in a -# directory called `None` as `_None_`. Worse, the sidecar was -# named `{filename}.json` — the attachment's ORIGINAL name — which (a) `find_ -# sidecar` can never pair with `_None_.png`, so no Discord file ever -# got a Post or a post date, and (b) collides: every `image.png` in a channel -# overwrote the same `image.json`, so the one sidecar that survived described -# whichever message happened to be written last. -# -# The fix names the sidecar EXACTLY like the media minus its extension, so -# `find_sidecar`'s first candidate (`media.with_suffix(".json")`) is the match -# and the name is unique per attachment. tests/test_gallery_dl_naming.py renders -# these patterns against a sanitized copy of the real sidecar, so a key that -# does not exist fails CI instead of silently becoming "None". -DISCORD_FILENAME = "{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}" -DISCORD_DIRECTORY = ["{channel}"] - - -def sidecar_name_for(media_pattern: str) -> str | None: - """The metadata filename pattern that names a sidecar exactly like its media. - - Returns None for a pattern that does not end in `.{extension}`, since then - there is no media stem to mirror and the caller must fall back. - """ - suffix = ".{extension}" - if not media_pattern.endswith(suffix): - return None - return media_pattern[: -len(suffix)] + ".json" - - -def metadata_postprocessor(filename: str) -> dict: - return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename} - - def archive_path(images_root: Path) -> Path: """gallery-dl's download archive: the record of what it has already fetched. @@ -425,27 +383,16 @@ class GalleryDLService: # (services/patreon_ingester.py), not gallery-dl. PLATFORM_DEFAULTS = { # subscribestar removed — native-ingester platform now (#71); pixiv - # removed likewise (#129); deviantart removed at #3069 as a dropped - # platform, not a migrated one. The remaining entries are the - # gallery-dl platforms not yet migrated. + # removed likewise (#129); discord likewise (milestone 428, whose + # downloader keeps this config's on-disk naming); deviantart removed at + # #3069 as a dropped platform, not a migrated one. HentaiFoundry is the + # one platform left here, by the operator's choice not to migrate it. "hentaifoundry": { "content_types": ["all"], "directory": [], "filename": "{category}_{index:>03}_{title[:50]}.{extension}", "include": "all", }, - "discord": { - "content_types": ["all"], - "directory": DISCORD_DIRECTORY, - "filename": DISCORD_FILENAME, - # Overrides the global `{filename}.json` sidecar for this extractor - # only — see the Discord naming note above. - "postprocessors": [metadata_postprocessor(sidecar_name_for(DISCORD_FILENAME))], - "embeds": "all", - "stickers": True, - "reactions": False, - "threads": True, - }, } def __init__( @@ -560,17 +507,6 @@ class GalleryDLService: if source_config.filename_pattern: platform_section["filename"] = source_config.filename_pattern - # A platform that names its sidecar after its media must keep doing so - # under a per-source filename override, or the pairing breaks exactly the - # way Discord's did. No metadata wanted means no platform postprocessor - # either — the global list was already dropped above. - if "postprocessors" in platform_section: - mirrored = sidecar_name_for(platform_section.get("filename") or "") - if not source_config.save_metadata or mirrored is None: - platform_section.pop("postprocessors") - else: - platform_section["postprocessors"] = [metadata_postprocessor(mirrored)] - platform_section["metadata"] = source_config.save_metadata return config @@ -818,9 +754,6 @@ class GalleryDLService: if cookies_path: config["extractor"]["cookies"] = cookies_path - if auth_token and platform == "discord": - config["extractor"].setdefault("discord", {}) - config["extractor"]["discord"]["token"] = auth_token with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False, dir=str(self._config_dir), @@ -1004,8 +937,6 @@ class GalleryDLService: config = self._build_config_for_source(platform, source_config, artist_slug) if cookies_path: config["extractor"]["cookies"] = cookies_path - if auth_token and platform == "discord": - config["extractor"].setdefault("discord", {})["token"] = auth_token with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False, dir=str(self._config_dir), diff --git a/backend/app/services/scheduler_service.py b/backend/app/services/scheduler_service.py index 43187cf..b24a6fe 100644 --- a/backend/app/services/scheduler_service.py +++ b/backend/app/services/scheduler_service.py @@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime return active +def backfill_ready(source: Source) -> bool: + """A deep walk the operator started, with budget left and no failure + backing it off — due NOW rather than at its next scheduled check. + + A backfill runs one time-boxed chunk per download (plan #693), and nothing + queued the next chunk: each waited for the source's regular interval. At + the 8-hour default a freshly armed backfill sat untouched until the next + check (the operator armed one on 2026-09-25 and saw nothing happen) and a + five-chunk walk took most of two days. The tick's in-flight guard keeps + one chunk at a time per source and the platform lock one walk per + platform, so "due every tick" means "next chunk as soon as the last one + ends". + + The failure gate is what keeps a broken source from retrying every + minute: any failed chunk raises `consecutive_failures`, which drops the + source back onto its backed-off interval. A chunk that fails to progress + twice marks the walk stalled (download_service), which ends it here too. + """ + co = source.config_overrides or {} + return ( + co.get("_backfill_state") == "running" + and (source.backfill_runs_remaining or 0) > 0 + and not (source.consecutive_failures or 0) + ) + + async def select_due_sources(session: AsyncSession) -> list[Source]: """Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval. @@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]: cooldown is the preventive half of the burst-prevention pair (per-source consecutive_failures backoff handles the offending source itself). + A running backfill (`backfill_ready`) is due on every tick, and whether + or not its artist is on auto-check — the operator started it by hand. + Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked sources go first, then the longest-since-checked, so the most overdue sources hit Celery's FIFO download queue first. Anti-starvation: if @@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]: .options(selectinload(Source.artist)) .join(Artist, Source.artist_id == Artist.id) .where(Source.enabled.is_(True)) - .where(Artist.auto_check.is_(True)) .order_by(Source.last_checked_at.asc().nulls_first(), Source.id) )).scalars().all() @@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]: for s in rows: if s.platform in cooldowns: continue + if backfill_ready(s): + due.append(s) + continue + if not s.artist.auto_check: + continue interval = compute_effective_interval(s, s.artist, settings) if s.last_checked_at is None: due.append(s) @@ -161,6 +194,8 @@ def compute_next_check_at( source: Source, artist: Artist, settings: ImportSettings, ) -> datetime | None: """Return the projected datetime of the next check, or None if never checked.""" + if backfill_ready(source): + return datetime.now(UTC) if source.last_checked_at is None: return None interval = compute_effective_interval(source, artist, settings) diff --git a/backend/app/services/source_service.py b/backend/app/services/source_service.py index 439d566..42201a7 100644 --- a/backend/app/services/source_service.py +++ b/backend/app/services/source_service.py @@ -18,6 +18,7 @@ from ..models import ( Source, ) from .db_helpers import failing_sources_clause +from .download_backends import uses_native_ingester from .gallery_dl import ErrorType from .membership_reconcile import KEPT_KEY, STOPPED_KEY from .membership_roster import gated_reasons_for_sources @@ -125,6 +126,11 @@ class SourceRecord: "backfill_posts": self.backfill_posts, "tier_gated_count": self.tier_gated_count, "gated_reason": self.gated_reason, + # Recover / recapture exist only on the native ingester. Sent so the + # UI asks the backend's own predicate instead of keeping a copy of + # the platform list — the copy said "patreon, subscribestar" for a + # day after Discord went native (milestone 428). + "native_ingester": uses_native_ingester(self.platform), } @@ -551,8 +557,8 @@ class SourceService: whole source); the two flags are mutually exclusive, so arming recapture clears bypass_seen. Clears prior cursor/chunk/stall state so it walks fresh from the top. The flag is cleared on completion (download_service) - and on stop. Recapture is Patreon-only (the native ingester's post-record - capture); inert elsewhere. The UI gates the action to Patreon sources.""" + and on stop. Recapture needs the native ingester's post-record capture, + so the UI offers it on native sources only (`native_ingester`).""" source = (await self.session.execute( select(Source).where(Source.id == source_id) )).scalar_one_or_none() diff --git a/frontend/src/components/subscriptions/SourceActions.vue b/frontend/src/components/subscriptions/SourceActions.vue index 34e484f..3512b70 100644 --- a/frontend/src/components/subscriptions/SourceActions.vue +++ b/frontend/src/components/subscriptions/SourceActions.vue @@ -76,10 +76,10 @@ const running = computed(() => props.source.backfill_state === 'running') const recovering = computed(() => !!props.source.backfill_bypass_seen) const recapturing = computed(() => !!props.source.backfill_recapture) // Recover / recapture are native-ingester features (ledger-bypass re-walk and -// post-text re-grab), available to every native platform — not just Patreon. -// Mirrors backend download_backends.NATIVE_INGESTER_PLATFORMS. -const NATIVE_PLATFORMS = ['patreon', 'subscribestar'] -const isNative = computed(() => NATIVE_PLATFORMS.includes(props.source.platform)) +// post-text re-grab), available on every native platform. The backend says +// which those are (`native_ingester`); a copied list here went stale when +// Discord moved over. +const isNative = computed(() => !!props.source.native_ingester)