Today's platform-cooldown commit (61ce1ce) only filtered the scan tick
— manual /api/sources/<id>/check still bypassed it. Operator-flagged
2026-05-30: clicked "Retry failed" on a Patreon failure pile and saw
every one go 'queued' without realising the cooldown wasn't in the
loop. Bulk retry with N sources on a cooled-down platform bowls right
back into the rate limit the cooldown is trying to prevent.
**Backend (`/api/sources/<id>/check`):**
- Reads optional `?force=true` query flag.
- Without force: queries `active_platform_cooldowns` (renamed from the
private `_platforms_in_cooldown` since it's now a cross-module API).
If the source's platform is in cooldown, returns **202** with
`{status: 'deferred', platform, cooldown_until}` — no event created,
no dispatch.
- With force: cooldown skipped entirely.
- In-flight guard always applies (no point creating duplicate pendings).
**Frontend (`sourcesStore.checkNow(id, {force=false})`):** new optional
`force` flag → adds `?force=true` to the URL.
**Frontend (`DownloadsTab`):**
- `onRetrySource` (single-source RETRY click): passes `force: true` →
explicit operator override, useful for rapid auth-fix testing.
- `onRetryAll` (RETRY ALL + MaintenanceMenu "Retry failed"): no force →
cooldown respected. Tallies `deferred` alongside `queued` /
`already_running`; toast reads e.g. *"5 queued, 12 deferred
(cooldown), 3 already running"*. That count is the operator's
diagnostic answer for "is rate-limit the cause of most failures?"
(12-of-20 deferred → yes; 0 deferred → no).
**Auto-resume:** no new sweep needed. Deferred sources still have stale
`last_checked_at`, so the next scan tick after the cooldown AppSetting
expires picks them up via `select_due_sources` (which already filters
on `active_platform_cooldowns`).
Tests: two new — deferred-on-cooldown returns 202 with the right body
and no dispatch; force=true overrides the cooldown and creates the
event normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:
> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.
Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.
Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.
When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.
INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.
Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.
Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick +
GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources)
+ SchedulerStatusBar on the Subscriptions tab (re-polled every 30s).
D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard
on Downloads with per-source and bulk "retry" (re-runs the feed via /check).
D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar
chart by the stat chips (failures stacked in error color); refreshes on live poll.
D4 credential staleness: surface last_verified age + "re-verify recommended"
warning past 30d; also fixes the dead last_verified_at field-name mismatch so
the verification row renders at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen