feat: a paywalled creator is no longer indistinguishable from a silent one (milestone 387 step A2)
CI / extension-version (push) Successful in 5s
Build images / sign-extension (push) Successful in 5s
CI / lint (push) Successful in 6s
Build images / build-agent (push) Successful in 12s
CI / frontend-build (push) Successful in 36s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-web (push) Successful in 1m24s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m30s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m34s

A2 of milestone 387. A1 made the gated-post count true; this makes it
mean something.

A native walk that reached the bottom returned `error_type=None`
whether the creator had posted nothing or every post sat behind a tier
we don't hold. `source.error_type` stayed NULL and the source read
healthy and quiet. gallery-dl has classified this as TIER_LIMITED since
the paywall-as-"needs attention" complaint; the native path never did.

Two things had to move that the plan didn't foresee, both found by
reading the consumers rather than by testing afterwards:

The backfill lifecycle's completion test required `error_type is None`.
Returning TIER_LIMITED naively would have dropped a fully-paywalled
backfill into the not-finished branch — zero downloads means no
progress, two strikes marks it "stalled" — so the creator we can see
least would become the one we re-walk most. `walk_completed` now admits
informational classes.

`_update_source_health` only stamps `error_type` on status "error" and
CLEARS it on "ok". Since TIER_LIMITED is a success, the chip was wiped
by the very run that produced it — which is why FailingSourcesCard's
`tier_limited` palette entry has never been reachable. An "ok" run now
keeps an informational class while failures stay 0 and last_error stays
clear: the run did not fail and must not earn a backoff.

Deviation from the plan, deliberate: the filed step said classify only
when `downloaded == 0`. gallery-dl doesn't condition on that, and
diverging the two backends over the same concept is what rule 169
forbids — so the native path mirrors it. "There is content here you
aren't paying for" is equally true in a week we also got the cheap
posts. Pinned by a test, since the stricter rule looks more correct.

The predicate, the wording and the completion test are defined once in
gallery_dl.py and spread into both backends (snippet 3087), rather than
re-derived per half — which is exactly how they drifted apart before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
This commit is contained in:
2026-09-09 15:40:19 -04:00
co-authored by Claude Opus 5
parent 173f4b00aa
commit 7751715b83
5 changed files with 281 additions and 24 deletions
+72 -6
View File
@@ -261,6 +261,72 @@ def make_run_stats(
}
# --- tier-gated classification, shared by BOTH backends ---------------------
#
# These three live together because the native ingester and the gallery-dl
# subprocess must reach the same verdict from the same number. They did not:
# gallery-dl classified TIER_LIMITED while ingest_core counted gated posts and
# threw the count away, so the platforms FC owns reported a paywalled creator as
# a silent one (#874 follow-up). One predicate, spread into both, rather than
# the condition re-derived per backend.
def classify_tier_gated(tier_gated_count: int) -> ErrorType | None:
"""TIER_LIMITED when a walk saw tier-gated posts and nothing else failed.
Deliberately NOT conditioned on `downloaded == 0`. A creator whose top-tier
posts we cannot see is tier-limited even in a week we did get their cheaper
ones — the fact the operator needs ("there is content here you are not
paying for") is true either way. gallery-dl has classified it this way since
the paywall-as-"needs attention" complaint (see `_categorize_error`), and the
native path now matches rather than inventing a stricter rule.
Callers must apply this only AFTER the real error categories (auth, rate
limit, drift, …) have had their turn; tier-gating is the weakest signal and
must never mask a genuine failure.
"""
return ErrorType.TIER_LIMITED if tier_gated_count else None
def tier_gated_message(count: int) -> str:
"""The one wording for the tier-gated verdict, so the two backends can't
describe the same state differently in the Logs UI."""
return (
f"Subscription tier does not grant access to "
f"{count} post{'s' if count != 1 else ''}"
)
# `Source.error_type` doubles as the failure-class chip, and a status of "ok"
# CLEARS it (alembic 0032). TIER_LIMITED breaks that assumption: it rides an
# otherwise-successful run, so without an exemption the chip is wiped the moment
# it is set and `FailingSourcesCard`'s `tier_limited` palette entry can never
# render. Informational classes are the exemption — they describe the source,
# not a failure of the run.
INFORMATIONAL_ERROR_TYPES = frozenset({ErrorType.TIER_LIMITED.value})
def is_informational(error_type) -> bool:
"""True for a class that reports a state rather than a failure. Accepts an
ErrorType or the plain string persisted on Source.error_type."""
return error_type is not None and str(error_type) in INFORMATIONAL_ERROR_TYPES
def walk_completed(result: DownloadResult) -> bool:
"""Did this walk reach the bottom cleanly?
The backfill lifecycle's completion test. An informational error_type still
counts as complete: a fully-paywalled creator's backfill DID finish, and
treating it as unfinished re-walks the same wall until the stall counter
trips — the creator we can see least becoming the one we fetch most.
"""
return (
result.success
and result.return_code == 0
and (result.error_type is None or is_informational(result.error_type))
)
class GalleryDLService:
"""Service for executing gallery-dl downloads."""
@@ -531,12 +597,12 @@ class GalleryDLService:
line for line in combined.split("\n")
if "][warning]" in line and "not allowed to view post" in line
]
if tier_gated_lines:
count = len(tier_gated_lines)
return (
ErrorType.TIER_LIMITED,
f"Subscription tier does not grant access to {count} post{'s' if count != 1 else ''}",
)
# Same predicate + wording the native path uses, so the two backends
# can't drift on what counts as tier-gated or how it reads.
count = len(tier_gated_lines)
gated = classify_tier_gated(count)
if gated is not None:
return (gated, tier_gated_message(count))
# Partial-success: the subprocess exited non-zero (typically because
# the wall-clock timeout fired mid-walk), but it had downloaded ≥1