Files
FabledCurator/backend/app/services/db_helpers.py
T
bvandeusenandClaude Opus 5.5 da98195ada
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
refactor: the native downloaders share one download loop, file record and seen-ledger key; a clean walk marks its credential verified
Patreon, SubscribeStar and Discord each carried their own copy of the same
per-post download loop, file record and seen-ledger key, and every copy was
another place a fix could be missed.

- BaseNativeDownloader runs one post's files: the seen and on-disk skips,
  the mid-post time-box, pacing, validation and the per-file sidecar. Each
  platform overrides only where its layout differs: Discord's channel folder,
  file names and message-id sidecar, and Patreon's Mux video through yt-dlp.
- One MediaItem and one ledger_key in native_ingest_common replace the three
  per-client copies. Keys are unchanged, except that a SubscribeStar file with
  no gallery id is now keyed by its filename instead of an empty id.
- DownloadResult, ErrorType and the run-stats and tier-gated helpers move
  from gallery_dl.py, which now serves only HentaiFoundry, to
  download_result.py. The unused delegator methods on GalleryDLService go.
- The download lock's platform list is the native-ingester list, not a copy.
- A walk that used a credential and finished with no error type stamps that
  credential's last-verified time, so the credentials page shows when the
  account last worked. A tier-limited walk does not count: an expired session
  hides paid posts the same way.
- The extension no longer clears a retired pixiv token on startup.
- Two comments that still described every download as gallery-dl now name
  both backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:03:32 -05:00

98 lines
4.4 KiB
Python

"""Shared DB-access helpers for the async services.
`get_or_create` centralizes the race-safe find-or-create dance — SELECT, then on
a miss a savepoint INSERT that recovers (NOT a full rollback) when a concurrent
worker inserted the same row first. It was hand-rolled identically in
ArtistService, TagService and ExtensionService; divergent copies are exactly how
the duplicate-row / race bugs in [[reference_scalar_one_or_none_duplicates]] crept
in, so it lives in one place now (DRY pattern sweep 2026-06-09).
Note: this is the ASYNC sibling of `Importer._get_or_create` (sync, used by the
filesystem-import path). The two can't share an implementation across the
sync/async boundary; the importer one stays as the lone sync consumer.
"""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from sqlalchemy import Select, and_
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Source
from .download_result import ErrorType
async def get_or_create[T](
session: AsyncSession,
select_stmt: Select,
factory: Callable[[], Awaitable[T]],
) -> tuple[T, bool]:
"""Race-safe find-or-create. Returns ``(row, created)``.
Run ``select_stmt`` (scalar_one_or_none); if a row exists, return it with
``created=False``. Otherwise open a SAVEPOINT and ``await factory()`` — which
must add its row(s), flush, and return the primary row. On ``IntegrityError``
(a concurrent worker inserted the same row first) roll back the SAVEPOINT —
NOT the outer transaction, which would lose the caller's surrounding work —
and re-run ``select_stmt`` (scalar_one) to return the row the other worker
created. The caller owns the outer commit.
A UNIQUE/partial-unique constraint matching ``select_stmt``'s predicate is
required for the recovery to trip; without it a duplicate slips through.
"""
existing = (await session.execute(select_stmt)).scalar_one_or_none()
if existing is not None:
return existing, False
sp = await session.begin_nested()
try:
row = await factory()
await sp.commit()
return row, True
except IntegrityError:
await sp.rollback()
return (await session.execute(select_stmt)).scalar_one(), False
# --- shared Source health predicates ----------------------------------------
#
# The subscriptions rollup, the front-door status ribbon and the list endpoint
# all have to agree on what "failing" and "no access" MEAN, or the ribbon says
# 3 and the card it links to shows 4. Same reasoning as get_or_create above:
# divergent copies of one predicate are how the drift creeps in. Defined here
# rather than in source_service because scheduler_service needs them too, and
# source_service already imports scheduler_service (the other direction would
# be a cycle).
def failing_sources_clause():
"""A source is FAILING when it is ENABLED and its runs are erroring.
Deliberately not `last_error IS NOT NULL` — a tier-limited source clears
last_error and keeps a chip, and must never be counted as broken.
The `enabled` half was folded in 2026-09-21 (#4279). A disabled source is
one FC deliberately stopped — most often because the membership sweep saw
`former_patron` — and "stopped because you no longer subscribe" is not
"failing". Worse, it is a failure nobody can clear: a disabled source is
never scheduled, so no successful run ever resets the counter, and the
card's Retry button routes to `/check`, which refuses a disabled source
outright. Ebi77 sat in the banner for six days with no action available.
This also settles a disagreement the two callers already had. The
scheduler's status count paired this clause with `enabled.is_(True)`;
`SourceService.list(failing=True)` did not. One counted Ebi77, the other
did not — the exact drift the note above this function warns about, which
is why the `enabled` test belongs IN the predicate rather than beside it.
"""
return and_(Source.enabled.is_(True), Source.consecutive_failures > 0)
def no_access_sources_clause():
"""A source we can't see the content of: the walk works, the tier doesn't
grant it (#874 / milestone #387 phase A). Not a failure — kept separate
from failing_sources_clause on purpose, and the two are disjoint because
an informational class only ever rides an otherwise-OK run."""
return Source.error_type == ErrorType.TIER_LIMITED