scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).
It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.
Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM
ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat
readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats.
- dbMaintenance store (loadStats / runVacuum) + test.
- Fix ruff I001: combine the two _sync_engine imports onto one line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.
- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
(VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
autocommit connection.
- GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.
Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).
Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
subprocess budgets (870 tick / 1170 backfill) sit below the soft
limit with phase-3 persist headroom. Promote to module constants and
guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
in-flight event with a real reason, mirror phase-3 source-health, and
decrement backfill so a chronically-slow source self-heals to tick
mode. The existing celery_signals handler only covered TaskRun, not
DownloadEvent — that was the gap.
Updates stale 900/1200 references in gallery_dl.py + maintenance.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
TimeoutExpired (matched the tick 870s/900s rationale). Backfill
runs that hit the cap let the next tick continue via the archive.
- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
without it cleanup_old_tasks' WHERE finished_at<cutoff never
reaped orphan-swept rows. recover_stalled_task_runs also now sets
duration_ms (matches celery_signals.finalize's millisecond math).
- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
on Source creation, mirroring SourceService.create. Without it,
Firefox quick-add on a creator with >20 unsynced posts walked the
full feed until subprocess timeout. Renamed the constant from
_NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.
- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
NO_NEW_CONTENT — the download path (line 712) already does, and
TIER_LIMITED proves auth reached the post and was told it was
tier-gated. Verify endpoint previously showed red on this and
prompted operators to rotate working cookies.
- prune_unused_tags now runs a single DELETE with the NOT-IN
predicate find_unused_tags uses, instead of SELECT-ids →
DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
would have surfaced on a tag explosion (>65k unused tags).
- credentials.upload() reflects the returned record into the store
cache (`.set(platform, rec)`) instead of evicting it; previously
the card briefly rendered "no credential" between upload and
loadAll().
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.
cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).
New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
download_source 1200s hard kill, so legitimately-running tasks are
never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
backoff is 2^N on that counter so per-event bumps would needlessly
inflate the next interval — sets last_error, stamps last_checked_at.
UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.
Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.
New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
derive the artist from the import path, find an ENABLED Source with a
real feed URL for (artist, platform). Returns None for filesystem-only
imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
set ImportTask.refetched=True, and trigger ONE download_source
re-check. Bounded by `refetched` so source-side corruption can't loop.
Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
tasks). Returns refetch_queued / no_source / already_refetched /
not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
manual button is the primary path; auto is opt-in since re-fetch
deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
→ stores.import.refetchTask → toast keyed on the result status.
Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.
Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.
**Split archive into its own task**
- New `import_archive_file` task: same body as import_media_file
(dispatch is by file-kind inside Importer.import_one) but
soft=30min / hard=35min. Shared `_run_import_task` helper holds the
flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
truth for media-vs-archive dispatch. Used by all three enqueue sites:
scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
is_archive(entry) (the model field already existed, anticipating
this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
task_routes `import_file.*` wildcard — no worker config change.
**Archive-aware recovery sweeps**
Both sweeps would otherwise preempt a legitimately-running archive:
- recover_interrupted_tasks (ImportTask 'processing' sweep): now
task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
35-min hard limit). Single UPDATE with an OR predicate over the two
(task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
the per-queue overrides added for ml. import_archive_file gets 40 min
while the 'import' queue stays at the 5-min default for single-file
imports. Precedence: task_name → queue → default, each pass excluding
rows claimed by a higher-precedence pass so every row is touched once.
**Tests**
- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
pins that a 10-min archive task-run survives, a 50-min one is flagged,
and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.
After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.
Root cause is two interacting limits, both undersized for video work:
tag_and_embed: soft_time_limit=300, time_limit=420
(sized for the image branch, ≈2 GPU ops)
recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues
The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.
**Two-part fix**
1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
(20 min). Sized for the video path's worst case; image runs return
in seconds and don't care.
2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
Queues with legitimately-long-running tasks (currently just `ml` at
25 min — 5-min buffer past the new hard kill) get their own
threshold; queues not in the dict use the default 5 min. The sweep
now issues one UPDATE per distinct threshold value, with
`queue.notin_(override_queues)` on the default pass so each row is
touched at most once.
Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
pins both directions: a 10-min-old ml row survives (fresh by 25-min
override), a 30-min-old ml row gets flagged.
After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff:
The remaining I001 errors came from ruff treating `alembic` as a first-
party module (because the alembic/ directory exists in the repo root)
rather than third-party. Ran `ruff check --fix` locally — auto-sorted
import groupings to put alembic/sqlalchemy alongside backend.* as first-
party, and trimmed redundant blank lines after a few import blocks.
Frontend:
`npm run check` (vue-tsc --noEmit) was failing because vue-tsc has no
tsconfig.json to read against, and the frontend is pure JS without
JSDoc annotations — vue-tsc had nothing to do. Skipping the step until
we add a tsconfig + convert to TS or add JSDoc annotations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff lint surfaced 23 violations across three rules; all addressed:
UP017 (Use datetime.UTC alias):
Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC),
also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for
timezone.utc that ruff's pyupgrade rules prefer.
UP042 (StrEnum):
Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)`
with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and
is the modern idiom. Behavior is equivalent for our usage (the .value
attribute, str(member) semantics).
I001 (Import sorting):
Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so
ruff groups `backend.*` imports correctly. Without it, ruff treated
them as third-party and demanded a different grouping. The existing
import order is stdlib → third-party → first-party → local relative,
which ruff now accepts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
recover_interrupted_tasks runs every 5 minutes, finds ImportTask rows
stuck in 'processing' for >30 minutes (well above any legitimate import
duration), and re-queues them. cleanup_old_tasks runs daily and deletes
finished tasks older than 7 days so the task table stays an operational
view rather than an archive.
Both thresholds match ImageRepo's precedent. The 30-min stuck threshold
is documented inline so a future reader can adjust it intentionally
rather than mistaking it for a 'magic number'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen