fix: a stopped source is not a failing one, and cannot be deep-scanned (4279)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m12s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m16s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m12s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m16s
Ebi77 sat in the "1 source is failing" banner for six days with no action available, reading `stranded by recovery sweep (no terminal status after time_limit)`. Four things lined up: 1. The membership sweep did its job — saw `former_patron`, disabled the source, cleared its failure state. Clean at 02:50. 2. Twenty minutes later a deep scan was armed on it. `/backfill` had a credential pre-flight but NO `enabled` guard, while `/check` has carried one all along. The two trigger endpoints disagreed, and the ungated one is the one that arms the long walk. 3. Without a membership the walk cannot finish, never reaches a terminal status, and the recovery sweep strands it with consecutive_failures = 1. 4. Nothing could clear that. A disabled source is never scheduled, so no successful run resets the count; `SourceService.update` clears only on an explicit disable and it was already disabled; and the banner's Retry routes to `/check`, which refuses a disabled source. The card offered a button structurally incapable of acting on the only source it was showing. `failing_sources_clause()` now means "enabled AND erroring". That also settles a disagreement its two callers already had: the scheduler's count paired it with `enabled.is_(True)` and `SourceService.list(failing=True)` did not, so one counted Ebi77 and the other did not — exactly the drift the note above that function warns about, which is why the test belongs IN the predicate rather than beside it. The scheduler's now-duplicate clause is dropped so one place decides. `/backfill` gains the guard for start/recover/recapture. `stop` stays open on a disabled source, or arming becomes a one-way door. Migration 0101 clears failure state on sources that are already disabled — the predicate fixes what the surfaces report, not what the rows carry, and the rows are why the operator had no way out (lesson #4202). It matches what `update` already does on an explicit disable, so rows disabled by any other path come into line. Enabled sources are untouched: a real failure on a live source must keep showing, which the second new test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -201,6 +201,16 @@ async def set_backfill(source_id: int):
|
||||
rec = await SourceService(session).get(source_id)
|
||||
if rec is None:
|
||||
return _bad("not_found", status=404)
|
||||
# A disabled source must not be armable for a deep walk — the same
|
||||
# rule /check has carried all along (see `source_disabled` below).
|
||||
# Arming one anyway is how #4279 happened: the membership sweep had
|
||||
# stopped Ebi77 as `former_patron`, a deep scan was armed twenty
|
||||
# minutes later, the walk could not complete without access, and
|
||||
# the recovery sweep stranded it with a failure count no surface
|
||||
# could clear — a disabled source is never scheduled again, and
|
||||
# Retry routes to /check, which refuses it.
|
||||
if not rec.enabled:
|
||||
return _bad("source_disabled", detail="enable the source first")
|
||||
native = uses_native_ingester(rec.platform)
|
||||
if native:
|
||||
cred = CredentialService(session, _get_crypto())
|
||||
|
||||
@@ -16,7 +16,7 @@ from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from sqlalchemy import Select
|
||||
from sqlalchemy import Select, and_
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -67,12 +67,26 @@ async def get_or_create[T](
|
||||
|
||||
|
||||
def failing_sources_clause():
|
||||
"""A source is FAILING when its runs are actually erroring.
|
||||
"""A source is FAILING when it is ENABLED and its runs are erroring.
|
||||
|
||||
Deliberately not `last_error IS NOT NULL` — a tier-limited source clears
|
||||
last_error and keeps a chip, and must never be counted as broken.
|
||||
|
||||
The `enabled` half was folded in 2026-09-21 (#4279). A disabled source is
|
||||
one FC deliberately stopped — most often because the membership sweep saw
|
||||
`former_patron` — and "stopped because you no longer subscribe" is not
|
||||
"failing". Worse, it is a failure nobody can clear: a disabled source is
|
||||
never scheduled, so no successful run ever resets the counter, and the
|
||||
card's Retry button routes to `/check`, which refuses a disabled source
|
||||
outright. Ebi77 sat in the banner for six days with no action available.
|
||||
|
||||
This also settles a disagreement the two callers already had. The
|
||||
scheduler's status count paired this clause with `enabled.is_(True)`;
|
||||
`SourceService.list(failing=True)` did not. One counted Ebi77, the other
|
||||
did not — the exact drift the note above this function warns about, which
|
||||
is why the `enabled` test belongs IN the predicate rather than beside it.
|
||||
"""
|
||||
return Source.consecutive_failures > 0
|
||||
return and_(Source.enabled.is_(True), Source.consecutive_failures > 0)
|
||||
|
||||
|
||||
def no_access_sources_clause():
|
||||
|
||||
@@ -229,7 +229,7 @@ async def scheduler_status(session: AsyncSession) -> dict:
|
||||
# links to cannot disagree about what they are counting.
|
||||
failing_sources = (await session.execute(
|
||||
select(func.count()).select_from(Source)
|
||||
.where(Source.enabled.is_(True), failing_sources_clause())
|
||||
.where(failing_sources_clause())
|
||||
)).scalar_one()
|
||||
no_access_sources = (await session.execute(
|
||||
select(func.count()).select_from(Source)
|
||||
|
||||
Reference in New Issue
Block a user