CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m12s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m16s
Ebi77 sat in the "1 source is failing" banner for six days with no action available, reading `stranded by recovery sweep (no terminal status after time_limit)`. Four things lined up: 1. The membership sweep did its job — saw `former_patron`, disabled the source, cleared its failure state. Clean at 02:50. 2. Twenty minutes later a deep scan was armed on it. `/backfill` had a credential pre-flight but NO `enabled` guard, while `/check` has carried one all along. The two trigger endpoints disagreed, and the ungated one is the one that arms the long walk. 3. Without a membership the walk cannot finish, never reaches a terminal status, and the recovery sweep strands it with consecutive_failures = 1. 4. Nothing could clear that. A disabled source is never scheduled, so no successful run resets the count; `SourceService.update` clears only on an explicit disable and it was already disabled; and the banner's Retry routes to `/check`, which refuses a disabled source. The card offered a button structurally incapable of acting on the only source it was showing. `failing_sources_clause()` now means "enabled AND erroring". That also settles a disagreement its two callers already had: the scheduler's count paired it with `enabled.is_(True)` and `SourceService.list(failing=True)` did not, so one counted Ebi77 and the other did not — exactly the drift the note above that function warns about, which is why the test belongs IN the predicate rather than beside it. The scheduler's now-duplicate clause is dropped so one place decides. `/backfill` gains the guard for start/recover/recapture. `stop` stays open on a disabled source, or arming becomes a one-way door. Migration 0101 clears failure state on sources that are already disabled — the predicate fixes what the surfaces report, not what the rows carry, and the rows are why the operator had no way out (lesson #4202). It matches what `update` already does on an explicit disable, so rows disabled by any other path come into line. Enabled sources are untouched: a real failure on a live source must keep showing, which the second new test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
98 lines
4.4 KiB
Python
98 lines
4.4 KiB
Python
"""Shared DB-access helpers for the async services.
|
|
|
|
`get_or_create` centralizes the race-safe find-or-create dance — SELECT, then on
|
|
a miss a savepoint INSERT that recovers (NOT a full rollback) when a concurrent
|
|
worker inserted the same row first. It was hand-rolled identically in
|
|
ArtistService, TagService and ExtensionService; divergent copies are exactly how
|
|
the duplicate-row / race bugs in [[reference_scalar_one_or_none_duplicates]] crept
|
|
in, so it lives in one place now (DRY pattern sweep 2026-06-09).
|
|
|
|
Note: this is the ASYNC sibling of `Importer._get_or_create` (sync, used by the
|
|
filesystem-import path). The two can't share an implementation across the
|
|
sync/async boundary; the importer one stays as the lone sync consumer.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Awaitable, Callable
|
|
|
|
from sqlalchemy import Select, and_
|
|
from sqlalchemy.exc import IntegrityError
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
|
|
from ..models import Source
|
|
from .gallery_dl import ErrorType
|
|
|
|
|
|
async def get_or_create[T](
|
|
session: AsyncSession,
|
|
select_stmt: Select,
|
|
factory: Callable[[], Awaitable[T]],
|
|
) -> tuple[T, bool]:
|
|
"""Race-safe find-or-create. Returns ``(row, created)``.
|
|
|
|
Run ``select_stmt`` (scalar_one_or_none); if a row exists, return it with
|
|
``created=False``. Otherwise open a SAVEPOINT and ``await factory()`` — which
|
|
must add its row(s), flush, and return the primary row. On ``IntegrityError``
|
|
(a concurrent worker inserted the same row first) roll back the SAVEPOINT —
|
|
NOT the outer transaction, which would lose the caller's surrounding work —
|
|
and re-run ``select_stmt`` (scalar_one) to return the row the other worker
|
|
created. The caller owns the outer commit.
|
|
|
|
A UNIQUE/partial-unique constraint matching ``select_stmt``'s predicate is
|
|
required for the recovery to trip; without it a duplicate slips through.
|
|
"""
|
|
existing = (await session.execute(select_stmt)).scalar_one_or_none()
|
|
if existing is not None:
|
|
return existing, False
|
|
sp = await session.begin_nested()
|
|
try:
|
|
row = await factory()
|
|
await sp.commit()
|
|
return row, True
|
|
except IntegrityError:
|
|
await sp.rollback()
|
|
return (await session.execute(select_stmt)).scalar_one(), False
|
|
|
|
|
|
# --- shared Source health predicates ----------------------------------------
|
|
#
|
|
# The subscriptions rollup, the front-door status ribbon and the list endpoint
|
|
# all have to agree on what "failing" and "no access" MEAN, or the ribbon says
|
|
# 3 and the card it links to shows 4. Same reasoning as get_or_create above:
|
|
# divergent copies of one predicate are how the drift creeps in. Defined here
|
|
# rather than in source_service because scheduler_service needs them too, and
|
|
# source_service already imports scheduler_service (the other direction would
|
|
# be a cycle).
|
|
|
|
|
|
def failing_sources_clause():
|
|
"""A source is FAILING when it is ENABLED and its runs are erroring.
|
|
|
|
Deliberately not `last_error IS NOT NULL` — a tier-limited source clears
|
|
last_error and keeps a chip, and must never be counted as broken.
|
|
|
|
The `enabled` half was folded in 2026-09-21 (#4279). A disabled source is
|
|
one FC deliberately stopped — most often because the membership sweep saw
|
|
`former_patron` — and "stopped because you no longer subscribe" is not
|
|
"failing". Worse, it is a failure nobody can clear: a disabled source is
|
|
never scheduled, so no successful run ever resets the counter, and the
|
|
card's Retry button routes to `/check`, which refuses a disabled source
|
|
outright. Ebi77 sat in the banner for six days with no action available.
|
|
|
|
This also settles a disagreement the two callers already had. The
|
|
scheduler's status count paired this clause with `enabled.is_(True)`;
|
|
`SourceService.list(failing=True)` did not. One counted Ebi77, the other
|
|
did not — the exact drift the note above this function warns about, which
|
|
is why the `enabled` test belongs IN the predicate rather than beside it.
|
|
"""
|
|
return and_(Source.enabled.is_(True), Source.consecutive_failures > 0)
|
|
|
|
|
|
def no_access_sources_clause():
|
|
"""A source we can't see the content of: the walk works, the tier doesn't
|
|
grant it (#874 / milestone #387 phase A). Not a failure — kept separate
|
|
from failing_sources_clause on purpose, and the two are disjoint because
|
|
an informational class only ever rides an otherwise-OK run."""
|
|
return Source.error_type == ErrorType.TIER_LIMITED
|