feat: a paywalled creator is no longer indistinguishable from a silent one (milestone 387 step A2)
CI / extension-version (push) Successful in 5s
Build images / sign-extension (push) Successful in 5s
CI / lint (push) Successful in 6s
Build images / build-agent (push) Successful in 12s
CI / frontend-build (push) Successful in 36s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-web (push) Successful in 1m24s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m30s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m34s
CI / extension-version (push) Successful in 5s
Build images / sign-extension (push) Successful in 5s
CI / lint (push) Successful in 6s
Build images / build-agent (push) Successful in 12s
CI / frontend-build (push) Successful in 36s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-web (push) Successful in 1m24s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m30s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m34s
A2 of milestone 387. A1 made the gated-post count true; this makes it mean something. A native walk that reached the bottom returned `error_type=None` whether the creator had posted nothing or every post sat behind a tier we don't hold. `source.error_type` stayed NULL and the source read healthy and quiet. gallery-dl has classified this as TIER_LIMITED since the paywall-as-"needs attention" complaint; the native path never did. Two things had to move that the plan didn't foresee, both found by reading the consumers rather than by testing afterwards: The backfill lifecycle's completion test required `error_type is None`. Returning TIER_LIMITED naively would have dropped a fully-paywalled backfill into the not-finished branch — zero downloads means no progress, two strikes marks it "stalled" — so the creator we can see least would become the one we re-walk most. `walk_completed` now admits informational classes. `_update_source_health` only stamps `error_type` on status "error" and CLEARS it on "ok". Since TIER_LIMITED is a success, the chip was wiped by the very run that produced it — which is why FailingSourcesCard's `tier_limited` palette entry has never been reachable. An "ok" run now keeps an informational class while failures stay 0 and last_error stays clear: the run did not fail and must not earn a backoff. Deviation from the plan, deliberate: the filed step said classify only when `downloaded == 0`. gallery-dl doesn't condition on that, and diverging the two backends over the same concept is what rule 169 forbids — so the native path mirrors it. "There is content here you aren't paying for" is equally true in a week we also got the cheap posts. Pinned by a test, since the stricter rule looks more correct. The predicate, the wording and the completion test are defined once in gallery_dl.py and spread into both backends (snippet 3087), rather than re-derived per half — which is exactly how they drifted apart before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
This commit is contained in:
@@ -261,6 +261,72 @@ def make_run_stats(
|
||||
}
|
||||
|
||||
|
||||
# --- tier-gated classification, shared by BOTH backends ---------------------
|
||||
#
|
||||
# These three live together because the native ingester and the gallery-dl
|
||||
# subprocess must reach the same verdict from the same number. They did not:
|
||||
# gallery-dl classified TIER_LIMITED while ingest_core counted gated posts and
|
||||
# threw the count away, so the platforms FC owns reported a paywalled creator as
|
||||
# a silent one (#874 follow-up). One predicate, spread into both, rather than
|
||||
# the condition re-derived per backend.
|
||||
|
||||
|
||||
def classify_tier_gated(tier_gated_count: int) -> ErrorType | None:
|
||||
"""TIER_LIMITED when a walk saw tier-gated posts and nothing else failed.
|
||||
|
||||
Deliberately NOT conditioned on `downloaded == 0`. A creator whose top-tier
|
||||
posts we cannot see is tier-limited even in a week we did get their cheaper
|
||||
ones — the fact the operator needs ("there is content here you are not
|
||||
paying for") is true either way. gallery-dl has classified it this way since
|
||||
the paywall-as-"needs attention" complaint (see `_categorize_error`), and the
|
||||
native path now matches rather than inventing a stricter rule.
|
||||
|
||||
Callers must apply this only AFTER the real error categories (auth, rate
|
||||
limit, drift, …) have had their turn; tier-gating is the weakest signal and
|
||||
must never mask a genuine failure.
|
||||
"""
|
||||
return ErrorType.TIER_LIMITED if tier_gated_count else None
|
||||
|
||||
|
||||
def tier_gated_message(count: int) -> str:
|
||||
"""The one wording for the tier-gated verdict, so the two backends can't
|
||||
describe the same state differently in the Logs UI."""
|
||||
return (
|
||||
f"Subscription tier does not grant access to "
|
||||
f"{count} post{'s' if count != 1 else ''}"
|
||||
)
|
||||
|
||||
|
||||
# `Source.error_type` doubles as the failure-class chip, and a status of "ok"
|
||||
# CLEARS it (alembic 0032). TIER_LIMITED breaks that assumption: it rides an
|
||||
# otherwise-successful run, so without an exemption the chip is wiped the moment
|
||||
# it is set and `FailingSourcesCard`'s `tier_limited` palette entry can never
|
||||
# render. Informational classes are the exemption — they describe the source,
|
||||
# not a failure of the run.
|
||||
INFORMATIONAL_ERROR_TYPES = frozenset({ErrorType.TIER_LIMITED.value})
|
||||
|
||||
|
||||
def is_informational(error_type) -> bool:
|
||||
"""True for a class that reports a state rather than a failure. Accepts an
|
||||
ErrorType or the plain string persisted on Source.error_type."""
|
||||
return error_type is not None and str(error_type) in INFORMATIONAL_ERROR_TYPES
|
||||
|
||||
|
||||
def walk_completed(result: DownloadResult) -> bool:
|
||||
"""Did this walk reach the bottom cleanly?
|
||||
|
||||
The backfill lifecycle's completion test. An informational error_type still
|
||||
counts as complete: a fully-paywalled creator's backfill DID finish, and
|
||||
treating it as unfinished re-walks the same wall until the stall counter
|
||||
trips — the creator we can see least becoming the one we fetch most.
|
||||
"""
|
||||
return (
|
||||
result.success
|
||||
and result.return_code == 0
|
||||
and (result.error_type is None or is_informational(result.error_type))
|
||||
)
|
||||
|
||||
|
||||
class GalleryDLService:
|
||||
"""Service for executing gallery-dl downloads."""
|
||||
|
||||
@@ -531,12 +597,12 @@ class GalleryDLService:
|
||||
line for line in combined.split("\n")
|
||||
if "][warning]" in line and "not allowed to view post" in line
|
||||
]
|
||||
if tier_gated_lines:
|
||||
count = len(tier_gated_lines)
|
||||
return (
|
||||
ErrorType.TIER_LIMITED,
|
||||
f"Subscription tier does not grant access to {count} post{'s' if count != 1 else ''}",
|
||||
)
|
||||
# Same predicate + wording the native path uses, so the two backends
|
||||
# can't drift on what counts as tier-gated or how it reads.
|
||||
count = len(tier_gated_lines)
|
||||
gated = classify_tier_gated(count)
|
||||
if gated is not None:
|
||||
return (gated, tier_gated_message(count))
|
||||
|
||||
# Partial-success: the subprocess exited non-zero (typically because
|
||||
# the wall-clock timeout fired mid-walk), but it had downloaded ≥1
|
||||
|
||||
Reference in New Issue
Block a user