Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
attach_in_place mirrored only the media flow, so gallery-dl-downloaded
zips/PDFs/audio bounced back as `skipped+invalid_image`, which
download_service counted as an ingest error and flipped runs to
status="error" despite N successful image attaches. Lustria patreon
event #38998 (21 images + 1 OST zip) went red for exactly this reason.
Now attach_in_place dispatches the same way as import_one: archives →
_import_archive (extracts media members, captures archive as
PostAttachment), non-media → _capture_attachment. Download_service
accepts the new `attached` result and treats non-duplicate skips as
soft skips, not ingest errors.
Also: ShowcaseView always loadInitial() on mount, not just when the
store is empty — Pinia persists across navigations and operator wants
a fresh shuffle every time the showcase loads.
Operator-flagged 2026-06-01: downloaded images stayed at
thumbnail_path=NULL until a periodic backfill sweep picked them up,
surfacing as broken-thumbnail tiles in the gallery for hours after
the download landed.
Importer.attach_in_place deliberately skips inline thumbnail
generation (importer.py:591-592) so the import queue stays moving —
the CALLING task is responsible for the enqueue. tasks/import_file.py
already did this (line 228-239). tasks/download.py / download_service
did not — every gallery-dl-attached image landed un-thumbnailed.
Fix in download_service._phase3_persist: after each
`attach_in_place` returning status in (imported, superseded), fan out
`generate_thumbnail.delay()` + `tag_and_embed.delay()` for each
image_id. Lazy import avoids circular-import risk between
download_service and the celery task modules that depend on it.
Mirrors the existing pattern verbatim — single source of truth for
"what fires after a successful attach" remains a comment in two
places (filesystem-import task, download orchestrator) rather than
a shared helper, because the contexts differ enough (sync session vs
async orchestrator) that abstracting would obscure more than it'd
share.
Test covers the happy-path with two attached files: both get the
thumbnail enqueue AND the ML enqueue, with image_ids drawn from
ImportResult (so future supersede-on-attach paths stay covered).
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.
Two coupled changes, both operator-flagged 2026-06-01:
* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
asks gallery-dl to exit after 20 contiguous archived items. Fresh
subscriptions + new-content cases still walk normally; established
subscription with zero new content exits in ~30s of HEAD requests
instead of pegging the timeout. 20 (not 5) gives headroom against
paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
downloads use `skip: True` + 1800s timeout. Auto-decrements per run
with early-reset to 0 when a clean run finds zero files (queue
drained). N defaults to 3 — multiple runs give the system enough
budget to finish a deep walk across timeout boundaries. New
`POST /api/sources/{id}/backfill` arms the source; "Deep scan"
button on each SourceRow (chip shows remaining count) wires it.
Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."
Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).
Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.
When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.
INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.
Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.
Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen