fix: a stopped source is not a failing one, and cannot be deep-scanned (4279)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m12s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m16s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m12s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m16s
Ebi77 sat in the "1 source is failing" banner for six days with no action available, reading `stranded by recovery sweep (no terminal status after time_limit)`. Four things lined up: 1. The membership sweep did its job — saw `former_patron`, disabled the source, cleared its failure state. Clean at 02:50. 2. Twenty minutes later a deep scan was armed on it. `/backfill` had a credential pre-flight but NO `enabled` guard, while `/check` has carried one all along. The two trigger endpoints disagreed, and the ungated one is the one that arms the long walk. 3. Without a membership the walk cannot finish, never reaches a terminal status, and the recovery sweep strands it with consecutive_failures = 1. 4. Nothing could clear that. A disabled source is never scheduled, so no successful run resets the count; `SourceService.update` clears only on an explicit disable and it was already disabled; and the banner's Retry routes to `/check`, which refuses a disabled source. The card offered a button structurally incapable of acting on the only source it was showing. `failing_sources_clause()` now means "enabled AND erroring". That also settles a disagreement its two callers already had: the scheduler's count paired it with `enabled.is_(True)` and `SourceService.list(failing=True)` did not, so one counted Ebi77 and the other did not — exactly the drift the note above that function warns about, which is why the test belongs IN the predicate rather than beside it. The scheduler's now-duplicate clause is dropped so one place decides. `/backfill` gains the guard for start/recover/recapture. `stop` stays open on a disabled source, or arming becomes a one-way door. Migration 0101 clears failure state on sources that are already disabled — the predicate fixes what the surfaces report, not what the rows carry, and the rows are why the operator had no way out (lesson #4202). It matches what `update` already does on an explicit disable, so rows disabled by any other path come into line. Enabled sources are untouched: a real failure on a live source must keep showing, which the second new test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
"""Clear failure state on sources that are disabled (#4279).
|
||||
|
||||
`failing_sources_clause()` now means "enabled AND erroring", so a disabled
|
||||
source no longer counts as failing. That fixes what the surfaces REPORT; it
|
||||
does not touch what the rows already CARRY, and the rows are the reason the
|
||||
operator saw a banner for six days with no way to act on it (lesson #4202 —
|
||||
a guard does not undo the value already stored).
|
||||
|
||||
## The row this exists for
|
||||
|
||||
Ebi77 (source 19): the membership sweep stopped it as `former_patron` at
|
||||
02:50 on 2026-09-15 and correctly cleared its failure state. A deep scan was
|
||||
armed twenty minutes later — `/backfill` had no `enabled` guard, which this
|
||||
release also fixes — and could not complete without access, so the recovery
|
||||
sweep stranded it:
|
||||
|
||||
consecutive_failures = 1
|
||||
last_error = "stranded by recovery sweep (no terminal status after time_limit)"
|
||||
|
||||
Nothing could clear that. A disabled source is never scheduled, so no
|
||||
successful run resets the counter; `SourceService.update` clears failure
|
||||
state only on an explicit disable, and the source was already disabled; and
|
||||
the card's Retry routes to `/check`, which refuses a disabled source.
|
||||
|
||||
## Why every disabled source, not just that one
|
||||
|
||||
The clear matches what `SourceService.update` already does when a source is
|
||||
disabled through the app — "disable the subs you're not paying for without
|
||||
them lingering as failing" — so this brings rows disabled by any OTHER path
|
||||
(the membership sweep, a retired platform in 0097) into line with the rows
|
||||
disabled by hand. Same shape as 0097: a repair migration reaches the live
|
||||
instance on deploy rather than waiting for someone to find the row.
|
||||
|
||||
Enabled sources are untouched — a real failure on a live source must keep
|
||||
showing.
|
||||
|
||||
Revision ID: 0101
|
||||
Revises: 0100
|
||||
Create Date: 2026-09-21
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0101"
|
||||
down_revision: Union[str, None] = "0100"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"UPDATE source SET last_error = NULL, error_type = NULL, "
|
||||
"consecutive_failures = 0 "
|
||||
"WHERE NOT enabled "
|
||||
"AND (last_error IS NOT NULL OR error_type IS NOT NULL "
|
||||
" OR consecutive_failures <> 0)"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible by design: the cleared strings and counts are not recorded
|
||||
# anywhere, and restoring a failure state nobody can act on would only
|
||||
# re-create the banner this removes. Rule #22 owes no story backwards.
|
||||
pass
|
||||
Reference in New Issue
Block a user