Compare commits

...

176 Commits

Author SHA1 Message Date
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
Build images / build-ml (push) Successful in 3m28s
CI / intimp (push) Successful in 4m26s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen d3245f0c22 feat(ext): verify cookies in-browser before uploading (1.0.7)
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 40s
extension / lint (push) Successful in 36s
CI / intimp (push) Successful in 4m4s
CI / intapi (push) Successful in 8m2s
CI / intcore (push) Successful in 8m45s
extension / lint (pull_request) Successful in 14s
Pre-upload verify request: after capturing the live browser cookies,
hit a known authenticated endpoint with credentials:'include' from the
extension's background context. If the platform reports we're not
logged in, abort the upload so we don't overwrite FC-side credentials
with stale data.

- platforms.js: add `verify` config per cookie-auth platform
  - hentaifoundry: HEAD /?enterAgree=1 (mirrors gallery-dl's HF
    _init_site_filters; same 401 path the operator hit 2026-06-03)
  - patreon: GET /api/current_user (clean 401 when logged out)
  - subscribestar, deviantart: no stable auth endpoint, skip verify
- cookies.js: verifyCookiesForPlatform() returns {ok, status, reason}.
  ok=true/false/null tri-state — null = verify not configured, caller
  treats as "proceed".
- background.js EXPORT_COOKIES + EXPORT_ALL_COOKIES: verify gates the
  upload; failures bubble up with the platform's name + reason.
- popup.js: success message now appends "(verified ✓)" when applicable.
- manifest + package.json: 1.0.6 → 1.0.7.
2026-06-03 14:04:45 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
2026-06-03 13:18:44 -04:00
bvandeusen e450145304 fix(ml): preserve digit-only tag names in normalize (year tags)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m42s
CI / intcore (push) Successful in 8m12s
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
2026-06-03 13:09:35 -04:00
bvandeusen a6e8d4b52e feat(ml): normalize Camie suggestion names to human-readable
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Failing after 24s
CI / frontend-build (push) Successful in 28s
CI / intimp (push) Successful in 3m57s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m22s
Camie's booru-style vocab strings (`uchiha_sasuke_(naruto)`,
`#unicus_(idolmaster)`, `1000-nen_ikiteru_(vocaloid)`, `:/`) were
surfacing raw in SuggestionsPanel — and worse, the SAME raw string was
written to tag.name on Accept, polluting the DB with `underscored_lowercase`
names that don't match the operator's "Title Case" tag convention.

Add backend/app/services/ml/tag_name.py with a single normalize()
applying nine rules (strip leading junk #/./+/;/~/_/ws, drop trailing
_(disambiguator) blocks iteratively, strip wrapping quotes, underscores
to spaces, space after colon, title-case each word's first char,
preserve hyphens/apostrophes/digits, drop entries with no letters).

Wire into SuggestionService.for_image:
- raw Camie key kept for alias_map lookup (alias rows are hand-curated
  against raw keys; don't disturb)
- display_name = normalize(raw); None means drop the candidate
- existing-tag lookup widened to case-insensitive match against BOTH
  raw and normalized forms so legacy underscore-named Tag rows accepted
  before this change still surface as "existing" not "+ new"
2026-06-03 13:00:08 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen f1860866de chore(modal): drop the ?image=N soft-compat from G5.4
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m38s
CI / intcore (push) Successful in 9m31s
Operator confirmed they have no existing ?image=N bookmarks to
preserve, so the soft-compat read on initial mount + router.replace
strip is dead weight. The modal is now purely a Pinia overlay — no
URL involvement on open OR initial mount.

Drops 33 lines plus the now-unused vue-router imports in both
ArtistGalleryTab (entire onMounted gone) and GalleryView (just the
?image=N block; the post_id/tag_id filter handling stays).
2026-06-02 19:24:09 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen b181d779fe fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 8m13s
CI / intcore (push) Successful in 7m6s
2026-06-02 18:49:24 -04:00
bvandeusen 0fbb19dc24 fix(modal): TagPanel kebab — apply the wrapping-span fix that the prior commit missed
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 37s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m59s
CI / intapi (push) Failing after 8m56s
CI / intcore (push) Successful in 10m0s
The previous commit (8326e54) updated the CSS but the structural
edit to the v-menu wrapper didn't take. Re-applying so the chip
kebab actually opens.
2026-06-02 18:48:53 -04:00
bvandeusen 8326e5447a fix(modal): kebab menus weren't opening (chip + suggestion rows)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Failing after 8m41s
CI / intcore (push) Successful in 10m0s
Operator-flagged 2026-06-02. Both kebab activators were broken:

- TagPanel chip kebab had `@click.stop` directly on the v-icon
  activator. In Vue 3, an explicit @click on the same element as
  `v-bind="props"` overrides the spread onClick — so Vuetify's
  activator handler never fired. Menu never opened.

- SuggestionItem kebab didn't have @click.stop, but for consistency
  and to make both kebabs follow the same shape, wrap it too.

The fix: each kebab is now wrapped in a `<span @click.stop>`. The
v-icon / v-btn receives Vuetify's onClick cleanly and opens the menu;
the bubbling click then reaches the span where stopPropagation
absorbs it before it can affect a parent (the v-chip's close button
in TagPanel's case).
2026-06-02 18:48:32 -04:00
bvandeusen 1fd594baaf chore(ml): suggestion_threshold default 0.50 → 0.70
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 28s
CI / frontend-build (push) Successful in 29s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Failing after 8m16s
CI / intcore (push) Successful in 8m43s
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01)
surfaces too many low-confidence picks in the modal's Suggestions
rail. 0.70 keeps the rail signal-rich while still showing more than
the original 0.95 (which hid almost everything).

Alembic 0033 updates the singleton row conditionally — only rows
still at the old 0.50 default flip to 0.70. Operators who tuned to
some other value via Settings → ML keep their pick.

Settings UI already exposes both sliders (MLThresholdSliders.vue),
so further tuning continues to work without a deploy.
2026-06-02 18:38:12 -04:00
bvandeusen ecac6c4bda fix(audit-g5): centroid version DB-as-truth + modal as overlay
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 24s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 7m57s
CI / intcore (push) Successful in 8m46s
Closes the last two findings from the 2026-06-02 audit (G5.1 + G5.4).

G5.1 — Centroid version no longer drifts:

CentroidService now reads MLSettings.embedder_model_version (the DB
row tag_and_embed already writes from) for both the centroid model-
version stamp and the drift-detection comparison. Previously the
centroid sites imported MODEL_VERSION from env, so the version stamped
on centroids could disagree with the version stamped on the embeddings
they were built from. By construction those now match, so list_drifted
won't silently miss the env-vs-DB drift case.

embedder.py keeps MODEL_VERSION as an env-driven constant for the
actual model loader — that's a different concern (which weights are
loaded) from the version-stamp that gets persisted alongside data.

G5.4 — Modal is a Pinia-only overlay:

The previous URL↔modal sync in GalleryView and ArtistGalleryTab
leaked the modal across route changes (RouterLink to /artist/<slug>
left the modal mounted on top of the new route) and re-opened it
on history back/forward with stale ?image=N entries.

Now: openImage() just calls modal.open(id) — no URL push.
GalleryView's dead closeImage helper is deleted. A route.name
watcher in App.vue closes the modal whenever the route changes,
which auto-fixes RouterLink-in-modal and back/forward.

Backward-compat: ?image=N is still honored on initial mount as a
one-shot deep-link opener, then router.replace strips the query so
the URL doesn't re-trigger and no extra history entry is added.
Existing bookmarks / shared URLs keep working; new opens stay
Pinia-only.
2026-06-02 18:28:57 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 9f7261b9c0 fix(audit-g5c): set CURATOR_BOOTSTRAP_NEW_KEY=1 in conftest
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m50s
CI / intapi (push) Successful in 8m33s
CI / intcore (push) Successful in 9m7s
CredentialCrypto's safety check fires on create_app() instantiation
because the test environment has no pre-seeded Fernet key file. Set
the bootstrap env var before any test imports so the auto-create path
is allowed during tests.
2026-06-02 17:55:25 -04:00
bvandeusen f05aaa707b fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Failing after 26s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Failing after 8m10s
CI / intcore (push) Failing after 9m42s
Alembic 0032 adds Source.error_type (varchar(32), indexed).
_update_source_health stamps it alongside last_error on status='error'
and clears it on 'ok'. SourceRecord/to_dict exposes it.

FailingSourcesCard renders a colored chip next to the consecutive-
failures count, with a tooltip explaining the suggested operator
action. Color reflects intent:
  - warning (yellow) — operator action needed (auth_error)
  - info (blue)      — backend-paced (rate_limited / timeout /
                       network_error / partial / tier_limited)
  - error (red)      — likely terminal without intervention
                       (not_found / access_denied / validation_failed /
                        unsupported_url / http_error / unknown_error)

Audit 2026-06-02: the backend computed 13 ErrorType categories but
only the free-text last_error reached the operator. Bulk-triage by
class ("all auth_error → rotate cookies", "12 rate_limited → just
wait") required opening Logs per row.
2026-06-02 17:54:44 -04:00
bvandeusen 4df98171ab fix(audit-g5c): refuse silent Fernet key regeneration on partial restore
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Failing after 25s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Failing after 8m13s
CI / intcore (push) Failing after 8m48s
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.

Two opt-ins now needed for auto-creation:
  1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
  2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)

Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.

Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.

Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
2026-06-02 17:04:27 -04:00
bvandeusen 8d75ade1d5 fix(audit-g5b): race-poisoning in three find-or-create sites
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m29s
Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the
race under concurrent writers / recovery-sweep replays, poisoning the
outer transaction with an unrecoverable IntegrityError and crashing
the calling task. Same shape as the banked 2026-05-26 ImageProvenance
incident (see reference_scalar_one_or_none_duplicates memory).

Mirrors the importer._get_or_create pattern in all three: savepoint
via begin_nested + IntegrityError rollback to that savepoint + re-
SELECT to grab the row the other worker committed first.

- importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store
  is sha-addressed so both workers race to write the same on-disk path
  (shutil.copy2 + rename is idempotent), so no extra cleanup needed.

- TagService.find_or_create: partial uniqueness index on
  (name, kind, COALESCE(fandom_id, -1)). The previous docstring
  claimed "INSERT ... ON CONFLICT" but the implementation was
  SELECT-then-INSERT with no recovery.

- ArtistService.find_or_create: already had IntegrityError handling
  but did session.rollback() (unwinds the WHOLE transaction); now
  uses begin_nested + sp.rollback() so the surrounding request's
  progress isn't lost.
2026-06-02 17:02:12 -04:00
bvandeusen 75c63e1511 fix(audit-g5a): ruff isort — platforms after patreon_resolver
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 16:55:56 -04:00
bvandeusen 98673d4dca fix(audit-g5a): small architectural cleanups bundle
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m23s
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.

- download_service: replace hardcoded ('discord','pixiv') tuple with
  auth_type_for(platform) == 'token'. A 7th token-platform now picks
  up the right credential path without touching this site.

- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
  merge so the target's centroid reflects its new image set
  immediately. Daily list_drifted catches it within 24h, but eager
  recompute closes the suggestion-quality dip in the meantime.

- backfill_thumbnails added to beat_schedule (daily). The task
  docstring claimed periodic Beat but the entry was never registered,
  so the library got no self-healing thumbnail repair; only the
  manual admin-UI button fired it.

- modal.createAndAdd pushes a kind='fandom' tag into
  tagsStore.fandomCache so FandomPicker sees the new fandom on next
  open. Was: cache-gated load (length===0) skipped refetch, new
  fandom invisible until full page reload.

- cleanup cluster:
  - Drop .webp from cleanup_service.unlink — thumbnailer only writes
    .jpg/.png; the third tuple member was dead code.
  - Drop effective_date from /api/gallery/scroll response — no FE
    consumer reads it. Service still computes the attribute for
    timeline ordering; this just trims the JSON.
  - Rename store.recentMinute → store.recentRuns across the
    systemActivity store + three consumers (SystemActivitySummary,
    QueuesTable, SystemActivityTab). The data is the last 200 runs
    (not actually "last minute"), so the name lied.

NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
2026-06-02 16:46:46 -04:00
bvandeusen 89b48f8f35 Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 35s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m10s
2026-06-02 16:15:00 -04:00
bvandeusen 4bff1d8558 fix(audit-g4): status-enum miss batch
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m37s
CI / intcore (push) Successful in 8m18s
Five extension-miss findings from the 2026-06-02 audit, where a status
value was added on one side but a downstream consumer didn't pick it up.

- download_service._phase3_persist: explicit branches for
  ImportResult.status in ('failed','refreshed'). For 'failed' (archive
  probe crash from _import_archive), unlink the source file so the
  filesystem scanner doesn't re-import and re-crash on the same
  archive forever. 'refreshed' is currently unreachable from the
  download path (no deep=True) but matches the importer's documented
  contract; treat as 'attached'.

- gallery-dl backfill auto-complete now gates on dl_result.success +
  no error_type, not just return_code==0 + files_downloaded==0.
  VALIDATION_FAILED exits the subprocess with returncode=0 and
  files_downloaded=0 when every file was quarantined, matching the
  prior predicate exactly and zeroing the operator's armed backfill
  budget on the FIRST quarantine run instead of decrementing.

- attach_in_place archive dispatch now threads artist + source_row
  through _import_archive (and _import_media for archive members)
  and _supersede. The path-walk fallback (_resolve_artist) is still
  used by filesystem-import; the download path now binds
  ImageProvenance to the explicit subscription Source instead of
  rediscovering by (artist_id, platform).

- Three FE handlers now recognize status:'deferred' from
  /api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting
  "event #undefined"), SubscriptionsTab.checkAll (was counting
  deferred as queued), DownloadEventRow.onRetry (was saying
  "re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll
  which already had it.

- celery_signals._queue_for now maps backup/admin/library_audit
  prefixes to 'maintenance' (matching task_routes). TaskRun.queue
  was returning 'default' for those rows, so per-queue dashboard
  filters and per-queue threshold overrides (added in G3) silently
  missed them.
2026-06-02 16:04:59 -04:00
bvandeusen d60e0b9494 Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 36s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Successful in 7m39s
CI / intcore (push) Successful in 8m10s
2026-06-02 14:49:28 -04:00
bvandeusen e30f50e6fe fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m30s
CI / intapi (push) Successful in 7m30s
CI / intcore (push) Successful in 8m8s
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.

Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
  running/restoring past 7h (covers the 6.5h images-backup hard
  limit) to error. prune_backups docstring corrected — the FC-3i
  TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
  past 135 min (10-min buffer above scan_library_for_rule's 2h5m
  hard limit) to error. Previously a SIGKILL'd row blocked all
  future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
  stuck running >2h whose child tasks are all terminal (orphan case
  where the orchestrator crashed before the closing UPDATE). Uses
  the same EXISTS predicate /api/system/stats already had.

Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
  matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
  the model relationship.

time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).

start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.

/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.

DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
2026-06-02 14:30:46 -04:00
bvandeusen 9c27a2d3c7 Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 37s
Build images / build-web (push) Successful in 13s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 8m50s
2026-06-02 14:17:12 -04:00
bvandeusen e66987f092 fix(audit-g2): async race / state-leak across eight stores
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 34s
CI / intimp (push) Successful in 3m31s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m2s
Extracts gallery.js's hand-rolled inflightId pattern into a new
useInflightToken composable; adopts in every store that previously
had no guard against late-response overwrites or wrong-image URL
interpolation.

Two operator-impacting bugs the audit (workflow wf_bbe3fdb1-e62)
flagged:

- modal.removeTag rolled back the chip rail unconditionally even
  when only the secondary dismiss POST had failed — UI lied until
  refresh. And all tag-mutation URLs interpolated currentImageId
  AFTER an await, so a fast prev/next could route DELETE/POST to
  the wrong image. Both fixed: split try/catch (dismiss failure
  surfaces a warning, doesn't roll back the delete); imageId
  captured at call-time and used in URLs throughout.

- suggestions.accept dereferenced currentImageId after the awaited
  POST /api/tags, so the subsequent /suggestions/accept could
  apply A's chosen tag to image B AND push it to B's allowlist.
  Fixed by capturing imageId at click-time + inflight guard on
  load().

Same shape across artist / downloads / artistDirectory /
tagDirectory / posts stores: rapid filter/nav changes used to
interleave responses (last-writer-wins). Now the late response is
discarded and the most-recent request wins. Filter-change-during-
search no longer drops the second fetch because the loading flag
was still true from the first.

gallery.js's inflightId removed in favor of the shared composable
so the pattern stays consistent.
2026-06-02 14:07:58 -04:00
bvandeusen 93e37681b7 Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 10s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m28s
CI / intcore (push) Successful in 8m3s
2026-06-02 13:29:17 -04:00
bvandeusen 80ef9bce48 fix(test): credentials.upload reflects record into cache (not invalidate)
CI / lint (push) Successful in 6s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 35s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 8m12s
Stale assertion pinned the buggy pre-G1.6 behavior (the test name
"upload invalidates the cache" literally describes the bug). The
audit's correction makes upload mirror the returned record into the
store so the card updates immediately — update the assertion to
match the corrected behavior.
2026-06-02 13:11:39 -04:00
bvandeusen 3898ce7be4 fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit
CI / lint (push) Successful in 5s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Failing after 27s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 8m6s
CI / intcore (push) Successful in 9m3s
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
  30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
  TimeoutExpired (matched the tick 870s/900s rationale). Backfill
  runs that hit the cap let the next tick continue via the archive.

- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
  without it cleanup_old_tasks' WHERE finished_at<cutoff never
  reaped orphan-swept rows. recover_stalled_task_runs also now sets
  duration_ms (matches celery_signals.finalize's millisecond math).

- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
  on Source creation, mirroring SourceService.create. Without it,
  Firefox quick-add on a creator with >20 unsynced posts walked the
  full feed until subprocess timeout. Renamed the constant from
  _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.

- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
  NO_NEW_CONTENT — the download path (line 712) already does, and
  TIER_LIMITED proves auth reached the post and was told it was
  tier-gated. Verify endpoint previously showed red on this and
  prompted operators to rotate working cookies.

- prune_unused_tags now runs a single DELETE with the NOT-IN
  predicate find_unused_tags uses, instead of SELECT-ids →
  DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
  would have surfaced on a tag explosion (>65k unused tags).

- credentials.upload() reflects the returned record into the store
  cache (`.set(platform, rec)`) instead of evicting it; previously
  the card briefly rendered "no credential" between upload and
  loadAll().
2026-06-02 11:24:05 -04:00
bvandeusen 64ca858574 Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 08:26:58 -04:00
bvandeusen 91be9df671 fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m43s
CI / intcore (push) Successful in 8m14s
attach_in_place mirrored only the media flow, so gallery-dl-downloaded
zips/PDFs/audio bounced back as `skipped+invalid_image`, which
download_service counted as an ingest error and flipped runs to
status="error" despite N successful image attaches. Lustria patreon
event #38998 (21 images + 1 OST zip) went red for exactly this reason.

Now attach_in_place dispatches the same way as import_one: archives →
_import_archive (extracts media members, captures archive as
PostAttachment), non-media → _capture_attachment. Download_service
accepts the new `attached` result and treats non-duplicate skips as
soft skips, not ingest errors.

Also: ShowcaseView always loadInitial() on mount, not just when the
store is empty — Pinia persists across navigations and operator wants
a fresh shuffle every time the showcase loads.
2026-06-02 08:16:13 -04:00
bvandeusen 412edec028 fix(showcase): don't exhaust on all-dupe batches; retry up to 8x
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 23s
CI / intimp (push) Successful in 3m25s
CI / intapi (push) Successful in 7m41s
CI / intcore (push) Successful in 8m14s
/api/showcase returns a random sample; after enough scrolling, an
unlucky 3-item batch can be entirely in the `seen` Set, which was
flipping `exhausted=true` and surfacing "End." mid-scroll. The
showcase is endless by intent — only a genuinely empty API response
(library has 0 images) should mark it exhausted. Tiny-library
fallback: cap retries at 8 to avoid spinning forever.
2026-06-01 23:37:21 -04:00
bvandeusen 937421485d fix(modal): refresh tag chips after accepting a suggestion
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 25s
CI / intimp (push) Successful in 3m37s
CI / intapi (push) Successful in 7m33s
CI / intcore (push) Successful in 8m13s
Operator-flagged 2026-06-01: clicking Accept on a suggestion added
the tag to the database but the modal's chip rail kept showing the
old set. Suggestion would disappear from the suggestions list
(suggestionsStore drops it from byCategory) and the toast would
confirm "Tagged: …", but visually the modal looked unchanged
until you closed and reopened it.

Root cause: useSuggestionsStore.accept() POSTs to the backend and
updates its own state, but never tells useModalStore that the
backing image's tag set has changed. The addExistingTag and
createAndAdd flows in TagPanel already call modal.reloadTags()
after applying — the suggestions path needed the same refresh.

Two-line fix in SuggestionsPanel: after store.accept() and
store.aliasAccept(), call modal.reloadTags() so TagPanel's
`modal.current?.tags` rebinds.
2026-06-01 23:02:57 -04:00
bvandeusen 9d0c0b7da8 Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 18s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m4s
2026-06-01 22:34:38 -04:00
bvandeusen 43b778aa04 fix(test): include 'scanned' in all backfill result assertions
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m11s
Same 'change shared shape, miss pinned tests' trap as cfa4fb4 — the
prior commit updated only test_backfill_mixed_aggregate; three sibling
tests in the same file also pinned the old {enqueued,ok,regenerated}
shape without 'scanned' and CI bounced.
2026-06-01 22:09:02 -04:00
bvandeusen 9cbdb70e13 fix(thumbnails): surface backfill results + tighten validity check
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 24s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m54s
CI / intcore (push) Failing after 8m18s
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."

1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
   returned just `{celery_task_id}` and the admin card said
   "Enqueued." with no counts. There was no way to tell whether the
   scan found 0 candidates, 5000 candidates, or whether the worker
   even picked up the task. "Found nothing" was indistinguishable
   from a broken queue.

   Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
   shared by the Celery task and the API endpoint. The API now runs
   the scan in an executor and returns `{scanned, enqueued, ok,
   regenerated}`. The actual thumbnail generation work still goes
   to the thumbnail Celery queue per row via
   `generate_thumbnail.delay()` — the scan itself is fast
   (SELECT id+thumbnail_path + a file.stat() per row).

2. **`_thumb_is_valid` accepted header-only corrupt files.** The
   magic-byte check passed for any 8-byte file starting with a JPEG
   or PNG header, including empty/truncated/zero-pad files that
   browsers render as broken. Backfill counted these as `ok` and
   never regenerated.

   Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
   thumbnails are minimum ~2KB even on solid-color sources; header-
   only corrupt files top out around 12 bytes. 256 is well above
   the corrupt floor and well below any legitimate thumbnail.

Plus the admin card now shows the per-run counts instead of
"Enqueued.":
  Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
2026-06-01 21:57:29 -04:00
bvandeusen 8e4d252ae4 Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m46s
2026-06-01 21:52:42 -04:00
bvandeusen bd06794647 fix(downloads): enqueue thumbnail + ML tasks per attached image
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 21s
CI / intimp (push) Successful in 3m43s
CI / intapi (push) Successful in 7m53s
CI / intcore (push) Successful in 8m23s
Operator-flagged 2026-06-01: downloaded images stayed at
thumbnail_path=NULL until a periodic backfill sweep picked them up,
surfacing as broken-thumbnail tiles in the gallery for hours after
the download landed.

Importer.attach_in_place deliberately skips inline thumbnail
generation (importer.py:591-592) so the import queue stays moving —
the CALLING task is responsible for the enqueue. tasks/import_file.py
already did this (line 228-239). tasks/download.py / download_service
did not — every gallery-dl-attached image landed un-thumbnailed.

Fix in download_service._phase3_persist: after each
`attach_in_place` returning status in (imported, superseded), fan out
`generate_thumbnail.delay()` + `tag_and_embed.delay()` for each
image_id. Lazy import avoids circular-import risk between
download_service and the celery task modules that depend on it.

Mirrors the existing pattern verbatim — single source of truth for
"what fires after a successful attach" remains a comment in two
places (filesystem-import task, download orchestrator) rather than
a shared helper, because the contexts differ enough (sync session vs
async orchestrator) that abstracting would obscure more than it'd
share.

Test covers the happy-path with two attached files: both get the
thumbnail enqueue AND the ML enqueue, with image_ids drawn from
ImportResult (so future supersede-on-attach paths stay covered).
2026-06-01 21:39:17 -04:00
bvandeusen fdd3e01f56 Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 10s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Successful in 8m16s
2026-06-01 21:09:28 -04:00
bvandeusen f575cfb93b ux(failing-sources): visible row separators + clearer hover
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m15s
The zebra striping in 717b601 was too subtle — inside the tonal error
card, surface-tint contrast washed out and rows still looked uniform.
Operator-flagged the same issue twice.

Replace the surface-tint approach with hard visual structure:
- Bottom border on each row (1px white/0.08) — clearly delineated rows
  regardless of card background
- Hover darkens to black/0.25 — much more obvious than the previous
  error-tint hover, gives the eye a clear horizontal track
- Slightly more vertical padding (8px vs 6px) for tap-target comfort
- Drop the gap; borders are the separator now
2026-06-01 20:57:41 -04:00
bvandeusen c82fb308b6 Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main
Build images / build-ml (push) Failing after 2s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-web (push) Successful in 12s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m47s
CI / intapi (push) Successful in 7m52s
CI / intcore (push) Successful in 8m16s
2026-06-01 20:44:23 -04:00
bvandeusen 717b601c81 ux(failing-sources): zebra striping + hover highlight on rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 8m29s
CI / intcore (push) Successful in 8m58s
Operator-flagged 2026-06-01: with the failing-sources panel expanded,
all rows have the same flat low-opacity background, no visual track
from the artist name on the left to the Logs/Retry buttons on the
right. Hard to tell which row a button belongs to when scanning down
a list of 17.

Three pure-CSS changes (no DOM, no logic):
- 3px gap between rows (was 2px) — slightly more breathing room
- Even rows get a darker surface tint (zebra striping)
- Hover paints the whole row in a low-opacity error tint, so moving
  the cursor toward Logs/Retry lights up the whole horizontal band
  and the eye traces back to the artist name automatically
2026-06-01 20:14:48 -04:00
bvandeusen cfa4fb4084 fix(test): drop stale 'starts at 0' assertion after auto-arm-on-create
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 28s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m59s
CI / intcore (push) Successful in 9m32s
`test_set_backfill_runs_arms_source` was pinning the pre-auto-arm initial
value (0) when checking that the override works. Now that create() pre-arms
enabled sources to 3, the assertion is stale — the test was already
verifying the override path, the pre-assertion just decorated it.

Drop the pre-assertion; keep the override check. Other tests use raw
Source(...) constructors (default to 0 via the column server_default),
not SourceService.create(), so they're unaffected.
2026-06-01 20:12:55 -04:00
bvandeusen 2aa2002f22 feat(subscriptions): newly added enabled sources start in backfill mode
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Failing after 8m23s
A freshly created subscription has no gallery-dl archive yet, so the
first poll in tick mode would walk forever — exit:20 doesn't trip
until 20 contiguous archive-hits, and there are none. The wall-clock
cap kicks in mid-walk, and the partial-success classifier (plan #544)
gracefully labels it status=ok, but the operator still wonders why
their new subscription isn't grabbing the back-catalog.

Pre-arm `backfill_runs_remaining = 3` on create() when `enabled=True`.
Same default as the manual "Deep scan" button, same auto-decrement
and auto-reset rules — once the queue drains (clean exit + zero new
files) or the budget runs out, tick mode resumes naturally.

Disabled sources (sidecar synthetics with `url='sidecar:<platform>:<slug>'`
that arrive disabled = False, or operator-added sources that start
disabled deliberately) skip the pre-arm — they're never polled, no
budget to waste.
2026-06-01 20:02:19 -04:00
bvandeusen 66ff671f09 fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m58s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m19s
Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.

gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.

Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.

Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.
2026-06-01 19:00:50 -04:00
bvandeusen 19aece1fc4 feat(download): tick/backfill modes + partial-success classifier (plan #544)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 27s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m20s
Routine subscription polls walked the entire post history every tick
even when nothing had changed, because gallery-dl's default `skip: True`
continues iterating archived posts. A creator with ~550 archived posts
(Knuxy patreon) saturates the 870s wall-clock cap before completing,
even with zero downloads needed. Plus, a tier-limited run that
downloaded hundreds of files but ran out the clock should be a
warning, not an error.

Two coupled changes, both operator-flagged 2026-06-01:

* **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"`
  asks gallery-dl to exit after 20 contiguous archived items. Fresh
  subscriptions + new-content cases still walk normally; established
  subscription with zero new content exits in ~30s of HEAD requests
  instead of pegging the timeout. 20 (not 5) gives headroom against
  paywall warnings interleaving with archived items.
* **Backfill mode (explicit, operator-triggered).** Sticky for N runs
  via new `Source.backfill_runs_remaining` (alembic 0031). While > 0,
  downloads use `skip: True` + 1800s timeout. Auto-decrements per run
  with early-reset to 0 when a clean run finds zero files (queue
  drained). N defaults to 3 — multiple runs give the system enough
  budget to finish a deep walk across timeout boundaries. New
  `POST /api/sources/{id}/backfill` arms the source; "Deep scan"
  button on each SourceRow (chip shows remaining count) wires it.

Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file
downloaded + no source-level error fires `ErrorType.PARTIAL`, which
download_service maps to `status=\"ok\"`. The run did real work; the
next tick continues via gallery-dl's archive. No more red events for
"timed out mid-walk after downloading 300 files."

Retires `SourceConfig.skip_existing` — skip value is now derived from
the source state and passed as a separate `skip_value` parameter
through download() / _build_config_for_source(). `GD_DEFAULTS` drops
the now-dead key (was inert data after this refactor).

Tests cover:
* tick + backfill skip-value emission in _build_config_for_source
* PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering
* SourceService.set_backfill_runs validation + persistence
* /api/sources/{id}/backfill 200/400/404 paths
* download_service auto-decrement / auto-reset / tick-mode-no-touch
* PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)
2026-06-01 18:23:28 -04:00
bvandeusen c9089b1d03 fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 8m18s
CI / intcore (push) Successful in 8m48s
gallery_service._provenance_clause artist branch was correlating its bare
Post reference to the outer query's primary_post_id outer-join, so the
artist filter silently matched zero rows for images with no primary post.
Alias Post inside the EXISTS subquery so SQLAlchemy adds it to the inner
FROM rather than treating it as a correlated outer table.

Five sidecar/import tests still asserted that a synthetic Source row
appears after a filesystem import. Alembic 0030 retired that behavior;
the Post sits null-source and the artist linkage lives on Post.artist_id.
Updated test_sidecar_creates_provenance, test_reimport_same_post_idempotent,
test_sidecar_artist_used_when_no_folder_artist, test_supersede_applies_new_file_sidecar,
and test_apply_sidecar_recovers_from_integrity_error to assert
post.source_id IS NULL + post.artist_id linkage instead.
2026-06-01 14:40:28 -04:00
bvandeusen 644d538bab fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 3m46s
CI / intapi (push) Successful in 8m0s
CI / intcore (push) Failing after 8m45s
2026-06-01 14:23:18 -04:00
bvandeusen ff35da4743 fix(lint): drop unused Source import after Post.artist_id refactor
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 21s
CI / intapi (push) Failing after 2m16s
CI / intimp (push) Failing after 2m16s
CI / intcore (push) Failing after 2m20s
2026-06-01 14:19:37 -04:00
bvandeusen 2f66de2928 feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 2m15s
CI / intapi (push) Failing after 2m18s
CI / intcore (push) Failing after 2m17s
Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.
2026-06-01 14:17:52 -04:00
bvandeusen 8cf8d2ca4d Merge pull request 'Modal Esc/overflow polish, artist-scoped post scroll, failing-sources Logs button' (#43) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 2m45s
Build images / build-ml (push) Successful in 3m31s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m47s
2026-06-01 12:10:11 -04:00
bvandeusen 94e7d20792 feat(downloads): Logs button on failing-sources rollup rows
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 16s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m43s
CI / intcore (push) Successful in 8m10s
Operator-asked 2026-06-01: each row in the "X sources are failing"
rollup needs a button to surface the last run's stdout/stderr/error
without leaving the Downloads tab to find the matching event row.

Wired:
- `downloadsStore.loadLastForSource(sourceId)` two-steps via
  `GET /api/downloads?source_id=N&limit=1` (most recent event) then
  the existing `loadOne(id)` for the full detail payload. Returns
  null if no events exist (toast warns).
- `FailingSourcesCard` row: new text button `Logs` with
  `mdi-text-box-search-outline`, per-row spinner via `logLoadingIds`,
  emits `view-logs` with the source record.
- `DownloadsTab.onViewFailingLogs` is the handler — same
  `DownloadDetailModal` instance the event-row clicks use.
2026-06-01 10:04:22 -04:00
bvandeusen fb605af959 fix(posts): anchored view scroll inherits artist_id/platform filters
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
Operator-flagged 2026-06-01: clicking a Provenance post title from the
view modal opens the post in the posts feed scoped to that artist
(`/posts?post_id=N&artist_id=A`), but the older/newer infinite-scroll
loaded UNFILTERED global posts instead of staying in the artist's
stream. Root cause: the posts store's loadAround/loadOlder/loadNewer
sent only `{around, cursor, direction}` — never the `artist_id` /
`platform` filters. Backend `/api/posts` accepts them on every path
(post_feed_service.scroll filters via `Source.artist_id`); the
frontend just wasn't passing them.

Fix:
- `posts.js` `loadAround(postId, filters)` now takes the filter
  snapshot and stores it. `_aroundParams(extra)` mixes the active
  filters into around/cursor calls.
- `PostsView.vue` `loadAroundAndAnchor` passes `artist_id` +
  `platform` from the route query.
2026-06-01 08:24:43 -04:00
bvandeusen 4c56cf121f fix(modal): Esc closes from tag input; Provenance cards scroll at ~2.5
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 29s
CI / intimp (push) Successful in 3m54s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m16s
Two operator-flagged UX gaps from 2026-06-01:

1. **Esc trapped inside the tag-entry field**
   The autofocused TagAutocomplete input made Esc-to-close unreachable
   because ImageViewer's keydown handler bailed early on
   `isTextEntry(ev.target)` for every key including Escape. Now Escape
   closes the modal from text inputs too — except when a Vuetify
   overlay is open (FandomPicker, autocomplete dropdown, suggestion
   3-dot menu), in which case that overlay's own Esc-handling fires
   instead of closing the whole modal mid-interaction. Detected via
   `.v-overlay--active`. Arrow keys still gate on isTextEntry so the
   tag input handles typing without navigating images.

2. **Provenance cards expanded the side panel unbounded**
   When an image had many ImageProvenance entries the cards stack
   pushed the Tags section below the fold. Wrapped the cards in a
   `.fc-prov__cards` container with max-height 270px (≈2.5 cards) and
   thin overflow scrollbar. Title stays anchored at top; Tags panel
   sits at a consistent position below regardless of provenance count.
2026-06-01 08:17:54 -04:00
bvandeusen b1d58bc3b8 Merge pull request 'fix(ci): POSIX-safe SHORT_SHA in build.yml' (#42) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 3m24s
CI / intimp (push) Successful in 3m22s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m5s
2026-06-01 08:03:17 -04:00
bvandeusen 9564d073b9 fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m26s
CI / intcore (push) Successful in 8m3s
`${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner
executes the step via dash/BusyBox sh and errored with
`Bad substitution` (act_runner workflow shell, observed on the
65386f0 main-push run). Switched to
`SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in
both shells. Both build-web and build-ml tag-determination steps
updated.
2026-06-01 07:53:33 -04:00
bvandeusen 65386f02a0 Merge pull request 'View modal batch: autofocus, suggestions UX, post-title click, retire copyright/artist' (#41) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 4s
Build images / build-web (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m17s
CI / intcore (push) Successful in 8m2s
2026-06-01 07:01:42 -04:00
bvandeusen f87a06a6bd feat(modal): post title is the primary click (artist-scoped) + suggestion rows look like buttons
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 16s
CI / intimp (push) Successful in 4m16s
CI / intapi (push) Successful in 8m16s
CI / intcore (push) Successful in 8m41s
Two operator-asked modal UX changes (Scribe plan #514):

1. **Post title click → artist-scoped posts feed**
   - The bold post title in ProvenancePanel is now the primary click
     target. Click navigates to /posts?post_id=N&artist_id=A; the
     PostsView already filters on `artist_id`, so the user lands in
     that creator's stream, not the global one.
   - The redundant "View post" link is removed. "Show description"
     stays as the only action link below the meta line.
   - Title is styled as a button-link: accent color, hover underline,
     focus ring for keyboard nav (family rule 24 — UI quality bar).

2. **Suggestion rows look like buttons**
   - SuggestionItem becomes a chip-card: visible border, hover/focus
     background, unified container for name + score + actions
     (operator's "nothing visual to unify the buttons to their object"
     complaint).
   - Accept is an explicit tonal pill button labeled "Accept"
     (operator's pick over whole-row-clickable). 3-dot menu retains
     alias/dismiss, now `variant="outlined"` so it reads as a button.
   - Score is a fixed-width monospace pill on the right.
   - "+ new" badge upgraded to a pill chip with accent border so
     it's visibly an annotation, not part of the tag name.
2026-06-01 02:21:59 -04:00
bvandeusen 5d284aae9f fix(test): unpin general-threshold test from old 0.95 default (alembic 0029)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m55s
CI / intcore (push) Successful in 8m46s
2026-06-01 02:18:54 -04:00
bvandeusen af7b5c95e9 feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Failing after 8m18s
Four coupled operator-asked changes to the view modal (Scribe plan #509):

1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus
   in onMounted/nextTick so the caret is in the input the moment the
   modal renders. No click needed to start typing.

2. **General suggestions expanded by default** — SuggestionsPanel's
   general-category group now mounts with `:default-open="true"`.
   Operator can collapse if too noisy, but the v1 frame shows them.

3. **Lower general threshold default 0.95 → 0.50** — MLSettings.
   suggestion_threshold_general default matches character. Alembic
   0029 also bumps the existing singleton row's value if it's still
   at the old 0.95. Operator can re-tune from Settings → ML.

4. **Retire `copyright` + `artist` as ML suggestion categories** —
   neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never
   really existed as a copyright tag-kind). They were surfaced in the
   suggestions pipeline + threshold settings UI but had no follow-
   through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for,
   ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029
   drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS,
   SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and
   MLThresholdSliders rows.

Out of scope (intentional): `tag_kind` Postgres enum still includes
`artist` for historic Tag row queryability (per the model comment);
no operator pain reported, no enum-shrink needed.

Tests:
- test_surfaced_categories asserts {character, general}, excludes
  artist + copyright.
- test_threshold_for_artist_is_unsurfaced extended to cover copyright.
- test_get_and_patch_settings asserts new 0.50 default and the absent
  artist + copyright keys in the GET payload.
2026-06-01 02:08:10 -04:00
bvandeusen 667b05f14e Merge pull request 'Extension probe-and-add (v1.0.6) + per-commit image tags' (#40) from dev into main
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 22s
extension / lint (push) Successful in 8s
Build images / sign-extension (push) Successful in 1m51s
Build images / build-web (push) Failing after 4s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
2026-06-01 01:44:03 -04:00
bvandeusen 8de7ccd07d build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m40s
CI / intcore (push) Successful in 8m15s
extension / lint (pull_request) Successful in 16s
Adds an immutable per-commit docker tag to every main-push build:
`git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`,
alongside the existing floating `:main` + `:latest`. Implements the
new family release-posture rule "Tags are milestones, not gates —
commit-SHA images are the rollback unit" so rollback to any commit
on main is `docker pull …:c-<sha>` with no release ceremony required.

Behavior change summary:
- main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>}
- tag-push (opt-in vYY.MM.DD only, no .N): unchanged
- safety-net dev: unchanged

No code changes; the rule is about how the tag list is constructed.
Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still
fire them when the operator wants a labeled checkpoint.
2026-06-01 01:28:55 -04:00
bvandeusen d65f0b2091 feat(extension): probe shows current state before click; v1.0.6
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 20s
extension / lint (push) Successful in 13s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m41s
Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup):
"the add source/subscription button idea to the firefox extension so
it can tell me if a source/artist is added or not and offer an option
to add it if it isn't." Plan tracked in Scribe task #507.

## Backend

- `ExtensionService.probe(url)` — read-only resolution. Reuses
  `_derive` for platform+slug, then 2 SELECTs. Returns one of:
  - `source_match` (exact (artist, platform, url) Source exists)
  - `artist_match` (artist exists, this URL isn't a Source yet;
     collapses the sidecar-synthetic-only case from v26.06.01.0)
  - `new` (neither exists)
  - `unknown_platform` (URL didn't match any artist-page regex)
- `GET /api/extension/probe?url=...` route with `X-Extension-Key`
  auth posture matching `/quick-add-source`. Read-only, side-effect
  free.
- 6 backend tests in tests/test_api_extension.py covering each state
  + auth + invalid URL.

## Extension

- `api.js`: `probeSource(url)` mirroring `quickAddSource` shape.
- `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The
  latter strips the `/api` suffix from configured `apiUrl` (placeholder
  format per options.html) and opens `${base}/artist/{slug}` in a new
  tab via `browser.tabs.create`.
- `content-script.js`: probe-first render — on page-load and SPA
  navigation, asks the backend for the URL's state and renders the
  chip in the matching color/copy on FIRST paint instead of flashing
  generic "Add" and updating after. Click handler branches:
  `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing
  ADD_AS_SOURCE flow (then re-probes so the chip flips green
  immediately, no wait for next nav).
- `content-script.css`: three state-color modifiers
  (--new, --artist-match, --source-match) on the FC parchment-on-slate
  palette. Sage for already-added, amber for artist-exists, accent
  orange for new.

## Versioning

- `extension/manifest.json` + `extension/package.json` → 1.0.6.
  build.yml's sign-extension job will fire on push to main since no
  `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the
  regenerated AMO keys end-to-end.

## Behavior on the sidecar-synthetic case

Filesystem-imported "Dymkens"-style artist with only a sidecar
synthetic Source: probe returns `artist_match` (not `new`), so the
chip reads "+ Add Patreon source to Dymkens" rather than offering to
recreate the artist. Clicking adds the real Source; existing
`_source_for_sidecar` preference logic (v26.06.01.0) routes future
gallery-dl Posts to the real one.
2026-06-01 00:41:17 -04:00
bvandeusen 856e9104b4 Merge pull request 'Sidecar synthetic anchor cleanup + tier-gated classifier fix' (#39) from dev into main
CI / lint (push) Successful in 5s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m54s
Build images / sign-extension (push) Has been skipped
Build images / build-web (push) Successful in 1m1s
Build images / build-ml (push) Successful in 1m21s
CI / intapi (push) Successful in 7m31s
CI / intcore (push) Successful in 8m0s
2026-06-01 00:16:58 -04:00
bvandeusen 66f19d67f5 fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 6m56s
CI / intcore (push) Successful in 7m35s
Three coupled operator-reported pains from the 2026-05-31 download
event audit:

1. `[patreon][warning] Not allowed to view post N` was bubbling up as
   an error event, bumping consecutive_failures and parking the source
   in "needs attention." The classifier's tier-gated branch was gated
   on `return_code in (1, 4)`. Gallery-dl returns a different exit
   code for mixed-failure runs (e.g. paywall warnings + a missing
   yt-dlp dep flipping the exit bits), so the branch never fired and
   the path fell through to UNKNOWN_ERROR. Widen the gate: when no
   source-level error fired AND tier-gated warnings are present,
   classify as TIER_LIMITED regardless of return code.

2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
   "stranded by recovery sweep (no terminal status after time_limit)"
   + empty stdout/stderr. Root cause: subprocess.run timeout (900s)
   and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
   wiped the in-memory captured output and the DownloadEvent ended up
   empty-logged 18 minutes later when the sweep finalized it. Drop
   gallery-dl's default subprocess timeout to 870s — a 30s margin
   shy of Celery's soft limit — so subprocess.TimeoutExpired always
   wins the race and captures the partial stdout/stderr via the
   existing handler.

3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
   firing on every video attachment, causing per-item download
   failures that masked legitimate tier-gated classification.
   Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
   video posts download normally and the per-item failure noise
   disappears.

Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
2026-05-31 23:30:39 -04:00
bvandeusen 6fc8ae3106 fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 8m20s
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:

1. `SourceService.list()` returned every Source, no filter on URL.
   alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
   one canonical row per (artist, platform); when no real campaign URL
   was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
   enabled=false as a disabled anchor. The UI then listed those
   anchors as if they were polls — disabled, but visible. Fix: `list()`
   excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
   opts back in for admin tooling.

2. `importer._source_for_sidecar` picked the lowest-id Source for
   (artist, platform). When alembic 0022 had rewritten a per-post row
   into a synthetic anchor (lower id) AND the operator later added the
   real subscription (higher id), every gallery-dl download silently
   attached its Post to the SYNTHETIC instead of the real Source. Fix:
   prefer a non-`sidecar:%` URL when one exists; fall back to the
   synthetic; only create a new synthetic when nothing exists for
   (artist, platform).

alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.
2026-05-31 23:08:38 -04:00
bvandeusen 0397642b21 Merge pull request 'Showcase cadence tuning + cooldown-aware bulk retry' (#38) from dev into main 2026-05-30 23:50:36 -04:00
bvandeusen a5101494b6 feat(downloads): bulk retry respects cooldown; single-source RETRY overrides
Today's platform-cooldown commit (61ce1ce) only filtered the scan tick
— manual /api/sources/<id>/check still bypassed it. Operator-flagged
2026-05-30: clicked "Retry failed" on a Patreon failure pile and saw
every one go 'queued' without realising the cooldown wasn't in the
loop. Bulk retry with N sources on a cooled-down platform bowls right
back into the rate limit the cooldown is trying to prevent.

**Backend (`/api/sources/<id>/check`):**
- Reads optional `?force=true` query flag.
- Without force: queries `active_platform_cooldowns` (renamed from the
  private `_platforms_in_cooldown` since it's now a cross-module API).
  If the source's platform is in cooldown, returns **202** with
  `{status: 'deferred', platform, cooldown_until}` — no event created,
  no dispatch.
- With force: cooldown skipped entirely.
- In-flight guard always applies (no point creating duplicate pendings).

**Frontend (`sourcesStore.checkNow(id, {force=false})`):** new optional
`force` flag → adds `?force=true` to the URL.

**Frontend (`DownloadsTab`):**
- `onRetrySource` (single-source RETRY click): passes `force: true` →
  explicit operator override, useful for rapid auth-fix testing.
- `onRetryAll` (RETRY ALL + MaintenanceMenu "Retry failed"): no force →
  cooldown respected. Tallies `deferred` alongside `queued` /
  `already_running`; toast reads e.g. *"5 queued, 12 deferred
  (cooldown), 3 already running"*. That count is the operator's
  diagnostic answer for "is rate-limit the cause of most failures?"
  (12-of-20 deferred → yes; 0 deferred → no).

**Auto-resume:** no new sweep needed. Deferred sources still have stale
`last_checked_at`, so the next scan tick after the cooldown AppSetting
expires picks them up via `select_due_sources` (which already filters
on `active_platform_cooldowns`).

Tests: two new — deferred-on-cooldown returns 202 with the right body
and no dispatch; force=true overrides the cooldown and creates the
event normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 23:15:07 -04:00
bvandeusen e3a7aff7a3 ux(showcase): pipeline fetches in-order, chunk size 3, keep the trickle
Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast
but could let later chunks arrive ahead of earlier ones — even when
each chunk is a random sample, that "later chunk loads first" risk made
the load order non-deterministic. And the original goal behind asking
for batching was a faster first-image-on-screen, which neither sequen-
tial nor parallel really addressed cleanly.

Switched the loadInitial flow to a PIPELINE:
- Only one fetch in flight at any moment (in-order arrival, no race).
- The NEXT fetch kicks off as soon as the current one resolves (NOT
  after its trickle finishes), so the next RTT overlaps the visible
  trickle window — round-trips are hidden behind the animation cadence.
- PAGE 5 → 3 + INITIAL_BATCHES 12 → 20 (total still 60). Smaller chunk
  → first chunk's items appear sooner (a chunk of 3 trickles in 240ms,
  well within one RTT, so by the time chunk 2 is in-hand the first
  trickle is just finishing).

Trickle, sequence-token guard, and infinite-scroll behaviour unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 22:52:55 -04:00
bvandeusen 9cd6d09e60 ux(showcase): smooth one-at-a-time cadence + earlier infinite-scroll trigger
Operator-flagged 2026-05-30 (round 2): the batched-loads change improved
consistency but still felt "chunky" — 5 items appeared together, then a
~200 ms API round-trip pause, then another 5. And infinite-scroll fired
too late when a single tall image (long manga page) made one masonry
column much taller than its siblings.

**Cadence (showcase store):**
- Fire all INITIAL_BATCHES fetches in PARALLEL — collapses the per-
  batch round-trip gap so all the data arrives in ~1 RTT instead of 12
  sequential RTTs.
- Trickle each response's items into images.value one at a time with
  APPEND_DELAY_MS = 80ms between each (≈ the MasonryGrid stagger
  animation, 70ms). User sees a smooth steady stream.
- fetchPage (the infinite-scroll path) uses the same trickle so its
  5-item appends also cascade one-by-one instead of popping together.
- Sequence token guards against a fast shuffle / mount-then-shuffle
  interleaving two trickles into the same images.value.
- Dropped useAsyncAction here — the parallel-fetch-then-trickle flow
  doesn't fit its single-wrap-call shape cleanly; inline loading/error
  state is clearer.

**Infinite-scroll trigger (MasonryGrid):**
- Pass `rootMargin: '2400px'` to useInfiniteScroll (was the 600px
  default). The masonry sentinel sits at the bottom of the container,
  whose height = MAX(column heights). A tall image in one column pushes
  the sentinel ~2× viewport below where the user is actually reading
  (the bottom of the SHORTER columns). 2400px ≈ 2-3 screen-heights of
  pre-emptive trigger, comfortable for typical tall manga heights.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 22:48:58 -04:00
bvandeusen 237575447d Merge pull request 'Thumbnail URL fix + archive daemon fix + batched initial loads' (#37) from dev into main 2026-05-30 22:01:43 -04:00
bvandeusen 810baf63ac fix(import): archive probe → subprocess (was multiprocessing.Process)
Every archive import was failing immediately with "AssertionError:
daemonic processes are not allowed to have children" (operator-flagged
2026-05-30 — import_archive_file crashes in 49ms with the assertion).
Celery's prefork pool runs tasks in daemon processes; Python's
multiprocessing module refuses to let daemons spawn children, which is
exactly what probe_archive was doing via mp.get_context("spawn")
.Process. The Layer-3 crash-isolation feature added 2026-05-28 was
effectively a hard-blocker on the very import path it was meant to
protect.

Switched probe_archive to subprocess.run — no daemon restriction, still
isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill
the worker). The probe body lifted to a tiny runner module
(_archive_probe_runner) that imports the unchanged _run_probe helper
and prints a single JSON line; parent parses stdout, returns the
ProbeResult exactly as before (timeout, signal, OOM, clean-rejection,
ok — all preserved).

cwd for the subprocess is the repo root derived from __file__ parents
so `python -m backend.app.utils._archive_probe_runner` resolves both in
the Celery container and pytest, regardless of where the worker was
launched.

Test refactor: test_archive_probe_target_bomb_guard →
test_run_probe_bomb_guard. Same in-process call to the (renamed) probe
body so the monkeypatched cap still takes effect; the real subprocess
path is exercised by the existing test_probe_archive_valid_zip /
test_probe_archive_corrupt_zip_clean_rejection tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 21:48:51 -04:00
bvandeusen 44bb12a93d fix(thumbnails): derive URL from stored thumbnail_path, not (sha256, mime)
The showcase/gallery/artist/series/post-feed APIs were constructing
thumbnail URLs from (sha256, mime). The MIME-based extension predicate
("png if image/png or image/gif else jpg") DISAGREED with the
thumbnailer's actual on-disk extension predicate ("png if alpha else
jpg"). Result: every PNG source without transparency 404'd (URL asked
.png, disk had .jpg); every WebP/AVIF source with transparency 404'd
(URL asked .jpg, disk had .png) — despite the thumbnail file existing
on disk.

The backfill task couldn't catch these because backfill checks the
ACTUAL thumbnail_path stored on the record (correct), not the URL the
browser fetches (broken derivation). So records with valid on-disk
thumbnails kept showing as broken in the UI no matter how many times
backfill ran.

Operator-flagged 2026-05-30: "the generate thumbnails function appears
to not catch all of the failed thumbnail cases" — turned out to not be
a backfill bug at all.

Fix: thumbnail_url now takes (thumbnail_path, sha256, mime) and returns
the stored path verbatim — Quart serves /images/* 1:1 from the volume
(frontend.py:20-36), so the URL IS the disk path. Falls back to the old
sha256+mime derivation only when thumbnail_path is NULL (thumbnailer
hasn't run yet); that URL will 404 in the browser until backfill catches
it, same as before the path was tracked.

All 8 callers updated: showcase_service, gallery_service (2 sites),
artist_service, series_service, post_feed_service, tag_directory_service,
artist_directory_service. The four sites whose query was raw-tuple now
also SELECT ImageRecord.thumbnail_path.

Net effect: every record that has a valid on-disk thumbnail will now
render correctly, regardless of which extension the thumbnailer chose,
without any DB migration or backfill rerun needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 16:55:38 -04:00
bvandeusen 1eefed9ab3 fix(ui): finish showcase store batching (PAGE 60 → 5, INITIAL_BATCHES 12)
The showcase-store change in adeee64 didn't actually land — only
gallery.js + ShowcaseView.vue made it into the commit. Without this,
ShowcaseView mounts loadInitial() which doesn't exist yet, so the
showcase blows up. Landing the matching store edit now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:53:24 -04:00
bvandeusen adeee64a2d feat(ui): batch initial showcase/gallery loads into smaller chunks
Showcase and gallery were fetching the full initial batch (60 / 50
items) in a single API call. Operator-flagged 2026-05-30: items should
stream in as small batches so they render progressively rather than
blocking on one big response.

- showcase store: PAGE 60 → 5, INITIAL_BATCHES = 12 (5 × 12 = 60, same
  total). loadInitial() fetches PAGE chunks in sequence; shuffle()
  delegates. fetchPage() still returns one PAGE-sized chunk so
  infinite-scroll also pulls 5 per trigger.
- gallery store: loadMore() limit 50 → PAGE (5), INITIAL_BATCHES = 10.
  loadInitial() loops loadMore() up to INITIAL_BATCHES times, stopping
  early when nextCursor becomes null (end of data).
- ShowcaseView: onMounted calls loadInitial() instead of fetchPage().
  Gallery/setTagFilter/setPostFilter already routed through loadInitial.

Net visual effect: each ~5-item batch lands and renders independently;
combined with MasonryGrid's animateFromIndex logic, the showcase now
cascades batches in as they arrive instead of one big pop-in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:51:58 -04:00
bvandeusen ed358757dc Merge pull request 'Most-overdue-first scheduling + rich timeout diagnostics' (#36) from dev into main 2026-05-30 14:30:41 -04:00
bvandeusen 99b66aa85f fix(download): preserve partial output + classify timeouts richer
Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show
anything other than that the task timedout and was cleaned up. I can't
tell why it ran over or if it was stuck failed or there was just that
much to get."

The TimeoutExpired branch was returning a DownloadResult with no stdout,
no stderr, no files_downloaded, and a generic "Download timed out after
N seconds" message — even though subprocess.TimeoutExpired carries the
partial output gallery-dl emitted before being killed.

Now:
- Capture e.stdout / e.stderr (coerced str if bytes; "" if None).
- Count files_downloaded from partial stdout via _count_downloaded_files.
- Surface a tail-of-stderr hint in error_message so the UI summary tells
  the operator at a glance whether it was "lots of content" (high count,
  clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung
  silent" (zero count, "no stderr output").
- Promote error_type to RATE_LIMITED when the partial stderr matches
  RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole
  900s window is the timeout-shaped tail of a real rate limit, and the
  platform cooldown should kick in for the same reason.

Existing test_download_timeout strengthened to also assert empty-partial
case stays correctly TIMEOUT-classified with no preserved output.
New test_download_timeout_preserves_partial_output_and_classifies covers
the rich-partial-output → RATE_LIMITED promotion path.

DownloadEvent.metadata already flows stdout/stderr/run_stats from
DownloadResult via _phase3_persist — no UI change needed; the existing
DownloadDetailModal will surface the captured output automatically once
the build redeploys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 14:16:54 -04:00
bvandeusen 77f7a23410 feat(scheduler): order due sources by last_checked_at — most overdue first
select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:

> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.

Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.

Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 12:08:43 -04:00
bvandeusen d181f4afb8 Merge pull request 'Downloads burst-prevention + maintenance-menu fix + gdl timeout' (#35) from dev into main 2026-05-30 11:43:18 -04:00
bvandeusen ff9e96e0e2 fix(tests): update scheduler-status shape pins for platform_cooldowns
61ce1ce added platform_cooldowns to scheduler_status(), and two pinned
key-set assertions tied to the old shape failed in CI. Updated:

- tests/test_api_sources.py::test_schedule_status_shape — direct
  /api/sources/schedule-status response.
- tests/test_api_system_activity.py::test_summary_returns_rollup_shape
  — nested under body["scheduler"] in the summary endpoint.

[[feedback-plan-grep-pinned-tests]] miss — should have grepped tests/
for `last_tick_at` / `auto_sources` before adding the new key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 11:25:36 -04:00
bvandeusen 61ce1ce13c feat(scheduler): platform-wide cooldown on RATE_LIMITED — burst prevention
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.

When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.

INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.

Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.

Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 11:01:40 -04:00
bvandeusen d28db32012 fix(download): align gallery-dl subprocess timeout to Celery soft limit
SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's
Celery task has soft_time_limit=900s and hard time_limit=1200s. So
gallery-dl never hit its own subprocess.run timeout — Celery always
killed it first. The hard SIGKILL leaves no terminal flip on the
DownloadEvent, which then sat pending/running until the recovery sweep
flipped it to error at 30 min from start. From the operator's seat that
was a ~30–40 min "hang" on every retry of a broken source.

Pinning the default to 900s (matching Celery's soft_time_limit) lets
subprocess.run raise TimeoutExpired cleanly inside Celery's window, and
the existing `except subprocess.TimeoutExpired` branch
(gallery_dl.py:635) captures it as a clean error_type='timeout' with a
real message. The DownloadEvent flips to error in ~15 min instead of
waiting on the recovery sweep at 30.

Per-source bumps still live in source.config_overrides for legitimately
long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries
the rationale and the Celery-soft-limit dependency in its comment.

Operator-flagged 2026-05-30 on 59-source strand pile retries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 02:25:04 -04:00
bvandeusen 77e9859da3 fix(downloads): rewire MaintenanceMenu to the downloads pipeline
The maintenance dropdown in Subscriptions → Downloads was wired to the
filesystem-import pipeline (POST /api/import/retry-failed +
POST /api/import/clear-stuck) — the subtitles even said so ("Re-enqueue
every failed import task"), but it was contextually misplaced. From the
Downloads view "Retry failed" queued nothing the operator could see
because the action operated on import_task rows, not download_event
rows. Import-pipeline maintenance is already reachable from Settings →
Imports (ImportTaskList.vue), so removing the import wiring loses
nothing.

Rewired:
- "Retry failed" → bulk-retries the failing-sources list, same loop as
  FailingSourcesCard's RETRY ALL (sourcesStore.checkNow per source).
  Subtitle now matches: "Re-queue every currently failing source".
- "Force recovery sweep" → triggers recover_stalled_download_events on
  demand via a new POST /api/downloads/recover-stalled endpoint. The
  sweep also runs every 5 min on Beat; this is the manual fallback so
  the operator doesn't have to wait for the next tick to clear newly
  stranded events.

MaintenanceMenu is now stateless — emits retry-failed and recover-
stalled. DownloadsTab owns the handlers (reuses the existing
onRetryAll; new onRecoverStalled with a delayed refresh so swept rows
land in the failing rollup).

Operator-flagged 2026-05-29 — "the retry failed button in the
maintenance dropdown doesn't appear to queue anything but manual
requeues works."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 00:59:05 -04:00
bvandeusen 2886fa4997 Merge pull request 'Tooltip !important fix — 104cac5 follow-up after Vite CSS reorder' (#34) from dev into main 2026-05-30 00:02:24 -04:00
bvandeusen 36f8ec80fd fix(ui): tooltip override needs !important — Vite reorders CSS chunks
104cac5's override sits at the same specificity as Vuetify's default
(`.v-tooltip > .v-overlay__content` on both sides). The fix relied on
source order — app.css imported after vuetify/styles in main.js — but
Vite's production bundler reorders node_modules CSS into the final
stylesheet unpredictably, so source order isn't a reliable winner.

!important on the two contrast properties (background + color) forces
the slate-on-parchment pair regardless of stylesheet load order. The
cosmetic border + shadow don't need it (Vuetify doesn't set them, so
nothing's competing).

Operator re-flagged 2026-05-29: tooltips still rendered light-on-light
on the deployed :latest after PR #33104cac5 was in the bundle but
not winning. Also updated the file header so the source-order claim
isn't carried forward as gospel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 23:51:39 -04:00
bvandeusen f256f587ee Merge pull request 'UI batch + I1–I6 service passes + download-event recovery sweep' (#33) from dev into main 2026-05-29 22:46:16 -04:00
bvandeusen e35fb1edf7 fix(scan): recovery sweep for stranded download events
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.

cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).

New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
  download_source 1200s hard kill, so legitimately-running tasks are
  never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
  backoff is 2^N on that counter so per-event bumps would needlessly
  inflate the next interval — sets last_error, stamps last_checked_at.

UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.

Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 21:40:52 -04:00
bvandeusen 08420cd619 feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon
The plain-JS frontend re-declares two backend enum value-sets by hand:
download-event statuses (downloadStatus.js) and platform keys
(platformColor.js). With no TS codegen, drift is silent — the same class
as the last_verified_at field mismatch.

tests/test_fe_be_contract.py parses both JS mirrors and asserts they equal
the backend canon (downloads._ALLOWED_STATUSES, platforms.known_platform_keys())
so a change on either side fails CI on whichever moved. Backend is the
source of truth. Documented the invariant in both JS file headers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 15:34:20 -04:00
bvandeusen 8649a13118 refactor(I5): remove one-and-done GS/IR migration tooling
The GS/IR migration cutover is complete, so the runbook tooling is dead
weight. Removed:
- services/migrators/ (gs_ingest, ir_ingest, tag_apply, ml_queue, verify,
  cleanup), tasks/migration.py, api/migrate.py (+ blueprint registration)
- MigrationRun model; alembic 0027 drops the migration_run table
- frontend LegacyMigrationCard + migration store (+ MaintenancePanel ref)
- celery include + task route + celery_signals queue mapping for migration.*
- the 1 GB MAX_CONTENT_LENGTH / MAX_FORM_MEMORY override (added solely for
  the ir_ingest upload)
- migration-surface tests (test_api_migrate, test_migration_verify,
  test_ir_ingest, test_gs_ingest, test_tag_apply)

Kept: the alembic schema-migration tests (test_migration_00XX — unrelated)
and cleanup_service.py (the permanent artist-cascade/unlink home).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 14:38:59 -04:00
bvandeusen 8979e0e377 refactor(I4): extract useInfiniteScroll composable; migrate 6 consumers
The IntersectionObserver-on-a-sentinel infinite-scroll pattern was copy-
pasted in 7 places. New composables/useInfiniteScroll(sentinelRef, cb)
owns the observer lifecycle (attach on mount, re-attach when the ref
changes, disconnect on unmount). Migrated GalleryGrid, MasonryGrid
(gallery+showcase), ArtistPostsTab, ArtistsView, TagsView, SeriesManageView.
PostsView left manual on purpose — its anchored mode does bidirectional
scroll-position preservation that doesn't fit the simple composable.

I4(a) (timestamps) is effectively already done: the UTC displays were
converted earlier; the remaining toLocaleString uses already render local
time, so they're left as-is rather than churned for format-only consistency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 14:03:49 -04:00
bvandeusen 00e2608ba1 feat(I3): always-on pipeline status indicator in the top nav
New /api/system/activity/summary aggregates scheduler health + per-queue
pending depths + running count + 24h failure count in one cached call (safe
to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a
compact running/queued/failing chip with a scheduler-health dot that expands
to a popover (scheduler, busy queues, counts, link to downloads). Polls the
summary every 8s, paused when backgrounded. Reuses the queue-cache read via
_queues_cached(). + API test for the summary shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:24:11 -04:00
bvandeusen 9ab5d709c8 fix(test): DownloadEventRow row shows platform, not artist name
The row template renders status + platform (PlatformChip) + time + counts;
the artist isn't displayed there. Assert on 'Completed'/'Patreon' instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:16:33 -04:00
bvandeusen 44410db492 test(I2): vitest component smoke tests for high-risk UI
Adds @vue/test-utils + happy-dom and mounts CredentialCard, DownloadEventRow,
PostCard, ActiveDownloadsPanel, QueueStatusBar, asserting they render without
throwing and surface key content — catching the dead-binding / render-error
class (e.g. the last_verified_at regression, guarded explicitly). Vuetify
components are left unresolved (Vue renders unknown elements + slots), so no
Vuetify-plugin setup is needed; only RouterLink is stubbed. Per-file
happy-dom env via docblock keeps the existing node-env specs untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 12:34:58 -04:00
bvandeusen e76aa36a29 ci(I1): dedicated fast-fail ruff lint lane
ruff is pre-installed in the ci-python image, so a new `lint` job runs it
with no dependency install and fails in seconds — surfacing the common lint
bounce class without waiting on the backend job's ~30-60s wheel install.
Dropped the now-redundant ruff step from backend-lint-and-test (same job
name, required-checks unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 12:22:33 -04:00
bvandeusen c95b760294 feat(posts): in-context anchored feed with bidirectional infinite scroll
Provenance "View post" deep-links to /posts?post_id=X, which now opens the
feed centered on that post with infinite load in BOTH directions.

Backend: PostFeedService.scroll gains a direction (older|newer); new
around(post_id) returns a window of newer + the post + older with a cursor
for each end. /api/posts accepts ?around= and ?direction=. + API tests.

Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends,
newer prepends) with per-end cursors; PostsView's anchored mode scrolls to
the post, observes top + bottom sentinels, and preserves scroll position on
upward prepend so the page doesn't jump. Normal feed mode unchanged.

Closes the remaining half of the post-navigation work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:12:18 -04:00
bvandeusen 35fe420701 feat(maintenance): live queue-depth bar under the backfill buttons
Each maintenance button (ML backfill, centroid recompute → ml queue;
thumbnail backfill → thumbnail queue) now shows a status bar with the live
pending count for its Celery queue, so the operator can see work is already
queued/running before re-triggering and piling on. MaintenancePanel polls
/api/system/activity/queues every 4s; QueueStatusBar reads the depth and
turns warning-colored when the queue is busy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 20:03:32 -04:00
bvandeusen 9e74c80e2f feat(downloads): front-and-center "active now" panel with live elapsed timers
New ActiveDownloadsPanel pinned at the top of the Downloads tab shows what's
running (pulsing dot + live mm:ss timer counting from started_at) and what's
queued, so the operator can see activity at a glance without the running
filter. Backed by store.loadActive() which fetches running + pending
independent of the feed filter; refreshed every 4s by the existing live poll.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:36:49 -04:00
bvandeusen c87e8e0932 fix(ui): render absolute timestamps in the viewer's local timezone
Download event times showed raw UTC wall-clock (iso.slice). Added
formatDateTime()/formatLocalDate() (local tz, robust to naive vs tz-aware
ISO) and applied them to the download row + detail modal datetimes and the
credential/artist date displays. formatPostDate stays UTC (date-only,
locale-stable, unit-tested).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:36:42 -04:00
bvandeusen 8c3900b998 feat(showcase): dramatic 3D cascade entry — tiles tip back then settle into place
Replaced the subtle 12px fade with a perspective rotateX(-28deg) tilt that
flips up and settles flat with a slight overshoot, staggered 70ms so tiles
cascade in one at a time. Makes the showcase read as an experience rather
than a quiet fade. Tunable (tilt/stagger/duration); reduced-motion safe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 19:01:04 -04:00
bvandeusen 972d9014ce feat(showcase): IR-parity hover animation on masonry thumbnails
FC already matched IR's staggered entry fade-in; this adds the missing
piece — hover zoom (scale 1.03) + brighten (1.1) on each thumbnail, with
overflow:hidden so the scaled image clips to the rounded card. Disabled
under prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:58:05 -04:00
bvandeusen 75b6b8056e feat(posts): plain bold titles; reserve View-original to the post card; provenance links to the posts feed
- Post titles arrived as stored HTML (e.g. <strong>…</strong>) rendering as
  literal markup. New toPlainText() strips tags; titles render plain + bold
  (ProvenancePanel and PostCard).
- Removed "View original post" from the provenance panel (modal) — the
  open-original button lives on the PostCard (the post view).
- Provenance "View post" now navigates to the /posts feed (post_id query),
  not the gallery image grid. (Feed in-context landing lands next.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:18:34 -04:00
bvandeusen 42ddac9996 fix(subscriptions): fixed-height hub so only the tab content scrolls
The whole view scrolled instead of just the subscription list. Made the
hub a viewport-height flex column (tabs stay fixed) with the v-window as
the single internal scroll container; the per-tab sticky control bars now
pin to the window top (top:48px -> 0). Operator-flagged 2026-05-28.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:13:19 -04:00
bvandeusen 384d8d5e50 Merge pull request 'Dashboard insights + project-wide DRY pass' (#32) from dev into main 2026-05-28 15:38:26 -04:00
bvandeusen 1322056b22 refactor(dry-F5): useTabQuery composable for ?tab= query-param routing
ArtistView and SubscriptionsView both two-way-bound a tab ref with the
?tab= query param via identical tab->URL and URL->tab watchers. Extracted
useTabQuery(validTabs, defaultTab) — defaultTab accepts a string or a
function (ArtistView's default is postCount-dependent), and resolve() is
exposed so ArtistView can re-apply it after the artist loads. Both views
drop their now-orphaned ref/useRouter imports.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:20:30 -04:00
bvandeusen 32bdde049f refactor(dry-B3): extract _get_or_create race-safe find-or-create in Importer
_find_or_create_source, _source_for_sidecar, and _find_or_create_post each
repeated the SELECT → savepoint-INSERT → on-IntegrityError rollback+re-SELECT
pattern. Extracted _get_or_create(stmt, factory): the statement is reused for
the scalar_one_or_none lookup and the scalar_one post-conflict re-fetch, so
all three are reproduced exactly. Centralizing the race-safe pattern in one
place also reduces the risk of the copies drifting (the bug class banked
2026-05-26).

Left _upsert_artist (no savepoint by design) and the ImageProvenance void
ensure-exists block (no return / no re-select) alone — they don't fit.

The rest of the ingest pipeline was already DRY: sidecar parsing lives in
utils/sidecar.py, per-platform quirks in the platforms package, and
_safe_ext/_categorize_error/_build_config are each single-instance.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:02:55 -04:00
bvandeusen 9d18dacbe8 fix(lint): UP037 — drop quotes from ImportSettings.load return annotations (py3.14 deferred annotations)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 14:08:24 -04:00
bvandeusen 171c486939 refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.

Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 14:03:26 -04:00
bvandeusen 21c1b0a81c refactor(dry-B4): extract shared async_session_factory for Celery tasks
download/migration/scan each defined an identical _async_session_factory()
(fresh per-invocation async engine — async connections are event-loop-bound
so each asyncio.run() task needs its own engine, unlike the process-wide
_sync_engine). Moved it to tasks/_async_session.py; the 3 files import it
and drop their now-orphaned sqlalchemy.ext.asyncio / get_config imports
(migration keeps AsyncSession for a type hint). Call-site try/finally
dispose left as-is to avoid re-indenting the critical task bodies.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:15:04 -04:00
bvandeusen 597c6d48d3 refactor(dry-B1): consolidate duplicated _bad error helper into api/_responses.py
8 blueprints each defined an identical _bad() (two variants: with/without
detail). Extracted error_response() into api/_responses.py; each blueprint
now imports it `as _bad` so call sites are unchanged. The detail-aware
canonical subsumes both variants. Left settings.py's distinct _bad_int and
the inline jsonify error sites (not duplicated helpers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:10:04 -04:00
bvandeusen a37dad33c7 refactor(dry-F1): shared useAsyncAction lifecycle helper for stores
New composables/useAsyncAction.js owns the loading/error/try-finally
lifecycle. Migrated 11 stores: credentials, downloads, sources, posts
(error=raw) + ml, artistDirectory, tagDirectory, showcase, suggestions,
seriesReader, modal (error=message). The errorAs option preserves each
store's existing error shape so store.error keeps the same type for
components (~50 consumption sites unchanged). Stores whose catch also
reset data (suggestions/seriesReader/modal) clear it upfront instead.

Deliberately NOT migrated (special control flow, would change behavior):
artist (conditional 404 catch + dual loading states), migration (rethrows),
gallery (inflight-id stale-response guard), and the Shape-B no-catch /
loaded-guard / keyed-cache stores.

Net -77 lines.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 12:53:57 -04:00
bvandeusen cabd73287a fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk
Removing the create_app import left 2 blank lines before pytestmark in 9
files where ruff isort wants 1. Also stripped two pre-existing
whitespace-only blank lines surfaced by the file change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 12:43:08 -04:00
bvandeusen def967a1a8 refactor(dry-S1): hoist app/client test fixtures into conftest
Removed the app/client fixtures duplicated across 36 test files (two
variants: separate app + client(app), and a self-contained client() that
called create_app inline) and the now-unused create_app imports. Both
fixtures now live once in conftest.py. test_suggestions_bulk keeps its
import (builds the app inline in two tests); test_health drops its local
client + unused pytest_asyncio.

Net -415 lines.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 11:33:05 -04:00
bvandeusen eebc8e2413 refactor(dry-F2): centralize shared UI primitives (relative-time, toast, download-status)
- utils/date.js: add formatRelative(iso, {future,nullText}); migrate 6 sites
  (SourceRow, SubscriptionsTab, SourceHealthDot, SchedulerStatusBar +
  thin adapters in BackupRunsTable/SystemActivityTab for their '—' null text).
  PostCard (30d->absolute) and CredentialCard (mo/y buckets) intentionally
  keep bespoke formatters.
- utils/toast.js: toast(opts) wraps the globalThis.window?.__fcToast?.(...)
  incantation; migrate 63 call sites across 24 files.
- utils/downloadStatus.js: single source for the download-event status enum
  -> label/color/icon; collapse the 3 duplicate maps (DownloadStatChips,
  DownloadsFilterPopover, DownloadEventRow).

Net -33 lines. Platform metadata was already centralized in platformColor.js.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 11:20:07 -04:00
bvandeusen 2358cedf3e feat(dashboards): scheduler health strip, failing-source rollup, 24h activity sparkline, credential staleness nudge
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick +
GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources)
+ SchedulerStatusBar on the Subscriptions tab (re-polled every 30s).

D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard
on Downloads with per-source and bulk "retry" (re-runs the feed via /check).

D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar
chart by the stat chips (failures stacked in error color); refreshes on live poll.

D4 credential staleness: surface last_verified age + "re-verify recommended"
warning past 30d; also fixes the dead last_verified_at field-name mismatch so
the verification row renders at all.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 08:30:14 -04:00
bvandeusen 215a8993a1 fix(lint): noqa ASYNC109 on gallery_dl.verify timeout (subprocess.run deadline, not coroutine)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 08:18:33 -04:00
bvandeusen a459d21a65 feat(dashboards): 1600px max-width, richer Downloads filters, needs-attention + sticky headers
Operator-flagged 2026-05-28: the Subscriptions/Downloads dashboards were
full-bleed and thin on filtering. Chosen via AskUserQuestion.

**Layout**
- SubscriptionsView capped at a centered max-width 1600px (covers all
  three subtabs) so rows aren't a mile wide on ultrawide monitors.
- Sticky control headers on both tabs (top: 48px, below the top nav,
  opaque bg) so filters/stat-chips stay reachable while scrolling a
  long list.

**Downloads filters (all four requested)**
- Stat chips are now clickable filters: click Queued/Running/Completed/
  Failed/Skipped to filter the list to that status; the active chip is
  outlined + elevated; re-click clears.
- Free-text search box over the loaded events (artist / platform /
  error substring).
- Artist filter: the filter popover's numeric "Source ID" field is
  replaced with an artist autocomplete (sources.autocompleteArtist →
  artist_id, which /api/downloads already supports). Pill shows the
  artist name.
- "Show no-change scans" toggle (default OFF): hides status=ok/skipped
  rows with 0 files (the scheduled scans that found nothing) so real
  downloads + failures stand out.

**Subscriptions**
- "Needs attention" quick-filter chip: one click to show only artists
  with sources that have errors OR have never been checked; chip shows
  the count and disables the status dropdown while active.

Frontend-only — backend filter params (status/artist_id/date) and the
/api/downloads endpoint already supported everything.
2026-05-28 08:10:04 -04:00
bvandeusen 73520b7cc3 feat(downloads): copy buttons in the download detail modal
Operator-flagged 2026-05-28: the download event detail modal had no way
to copy the error / stdout / stderr text for researching a failure
(parity with the ErrorDetailModal Copy button shipped in v26.05.26.0).

Added per-block Copy buttons (Error, Errors & warnings, Raw stdout, Raw
stderr — the stdout/stderr ones sit in the expansion-panel title with
@click.stop so they don't toggle the panel) plus a "Copy all
diagnostics" button in the footer that assembles a single block: header
line (event id / platform / artist / status / timestamps) + error +
errors&warnings + full stdout + stderr, ready to paste into an issue.

All routed through utils/clipboard.js copyText() — the navigator.clipboard
→ execCommand fallback that works on the plain-HTTP homelab origin
(per feedback_no_secure_context_apis). Each copy shows a confirmation
toast.
2026-05-28 07:54:40 -04:00
bvandeusen 56970fb66d feat(credentials+downloads): real credential Verify button + live download-activity polling
Operator-flagged 2026-05-28, two asks.

**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):

- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
  gallery-dl in `--simulate --range 1-1` mode (no download) against the
  URL with the materialized credentials, then reuses _categorize_error:
  returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
  inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
  the platform to probe, runs verify, and on success stamps
  credential.last_verified (new CredentialService.mark_verified).
  Returns {valid: bool|null, reason, last_verified?}. valid=null means
  untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
  result chip (Verified ✓ / Failed / Untestable) and a toast with the
  reason. SettingsTab reloads on @verified so last_verified refreshes.

**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.

Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
2026-05-28 07:52:20 -04:00
bvandeusen bf8eb4468f feat(tags): legacy-tag purge also catches source:* general tags
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.

Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
  now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
  (LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
  by_prefix (source:* counts once under the prefix bucket regardless of
  its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
  returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
  button reads "Preview/Delete legacy tags".

Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
2026-05-28 01:20:41 -04:00
bvandeusen e1fc65bd1b feat(provenance+tags): collapse provenance descriptions; purge retired-kind tags
Operator-flagged 2026-05-28, two asks.

**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.

**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)

- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
  (dry_run) or deletes tags whose kind ∈ (archive, post, artist).
  CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
  returns per-kind counts + sample names — the preview IS the
  verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
  tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".

Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).

NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
2026-05-28 01:12:20 -04:00
bvandeusen 104cac5dca fix(ui): tooltip readability — dark bg + parchment text (was light-on-light)
Operator-flagged 2026-05-28: tooltips (e.g. the action buttons in
Subscriptions → Subscriptions) render near-white-on-near-white,
unreadable.

Cause: Vuetify's default v-tooltip pairs `on-surface-variant` text with
an `surface-variant` background. FC's theme deliberately maps
`on-surface-variant` to vellum (#C2BFB4 — a light cream, the correct
muted-text token for captions/hints on the dark page) but never defines
`surface-variant`, so Vuetify auto-generates a light-ish tooltip
background. Light text on light bg.

Fix is tooltip-specific so it doesn't disturb the (correctly light)
muted-text token elsewhere. New app-global stylesheet
frontend/src/styles/app.css, imported in main.js AFTER vuetify/styles
(equal-specificity rule wins by source order), overrides
`.v-tooltip > .v-overlay__content` to a dark elevated panel
(surface-bright = slate #2C313A) with high-contrast parchment text
(#E8E4D8) + a subtle border + shadow. Applies to every tooltip in the
app, so the fix is consistent rather than per-component.
2026-05-28 00:56:47 -04:00
bvandeusen 319e8c1d18 Merge pull request 'v26.05.28.0: downloads dashboard + task-resilience overhaul (timeouts, archive split, 3-layer poison-pill defense)' (#31) from dev into main 2026-05-28 00:45:00 -04:00
bvandeusen dcfe55d731 feat(import-resilience L2): one-shot re-download for corrupt downloaded files
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.

New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
  derive the artist from the import path, find an ENABLED Source with a
  real feed URL for (artist, platform). Returns None for filesystem-only
  imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
  anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
  delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
  set ImportTask.refetched=True, and trigger ONE download_source
  re-check. Bounded by `refetched` so source-side corruption can't loop.

Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
  tasks). Returns refetch_queued / no_source / already_refetched /
  not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
  env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
  manual button is the primary path; auto is opt-in since re-fetch
  deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
  → stores.import.refetchTask → toast keyed on the result status.

Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.

Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
2026-05-28 00:08:03 -04:00
bvandeusen e3cdd0f92b feat(import-resilience L3): subprocess-isolated probes for video + archive
Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.

New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):

- probe_video(path): validates the container + first video stream via
  ffprobe (a separate binary — a decoder crash kills only ffprobe, not
  the worker). Returns width/height, which the importer didn't capture
  for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
  (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
  test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
  spawned child process. A decompression-bomb OOM or native-lib
  segfault on a malformed archive shows up as a non-zero child exit
  code → crashed=True, never a dead worker.

ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).

Wired:
- importer._import_media video branch: probe_video before the pipeline;
  crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
  → failed, clean reject → still preserve the archive as a
  PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
  so a corrupt video is rejected (status='bad_video') instead of
  crashing the ml-worker on frame decode.

Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
2026-05-28 00:01:32 -04:00
bvandeusen e77afe8295 feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.

- alembic 0026: import_task.recovery_count (int, default 0) +
  import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
  rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
  are marked 'failed' with a diagnostic ("crashed or stalled the worker
  N times … likely a corrupt or oversized input … inspect/replace the
  file, then retry via /api/import/retry-failed") instead of re-queued.
  The re-queue pass then handles the remaining stuck rows and bumps
  recovery_count. Shared stuck_predicate (and_/or_) keeps the
  media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).

The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."

Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
2026-05-27 23:54:35 -04:00
bvandeusen 57a22d6098 fix(tests): repair test_maintenance — skips_fresh_running tail was orphaned at module scope by the inserted ml/archive sweep tests (F841/F821) 2026-05-27 23:06:00 -04:00
bvandeusen a85880f965 fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.

**Split archive into its own task**

- New `import_archive_file` task: same body as import_media_file
  (dispatch is by file-kind inside Importer.import_one) but
  soft=30min / hard=35min. Shared `_run_import_task` helper holds the
  flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
  truth for media-vs-archive dispatch. Used by all three enqueue sites:
  scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
  is_archive(entry) (the model field already existed, anticipating
  this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
  task_routes `import_file.*` wildcard — no worker config change.

**Archive-aware recovery sweeps**

Both sweeps would otherwise preempt a legitimately-running archive:

- recover_interrupted_tasks (ImportTask 'processing' sweep): now
  task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
  get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
  35-min hard limit). Single UPDATE with an OR predicate over the two
  (task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
  per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
  the per-queue overrides added for ml. import_archive_file gets 40 min
  while the 'import' queue stays at the 5-min default for single-file
  imports. Precedence: task_name → queue → default, each pass excluding
  rows claimed by a higher-precedence pass so every row is touched once.

**Tests**

- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
  pins that a 10-min archive task-run survives, a 50-min one is flagged,
  and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.

After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
2026-05-27 22:45:11 -04:00
bvandeusen 407de18ff6 fix(ml): video branch needs longer time limits; recovery sweep is now per-queue
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.

Root cause is two interacting limits, both undersized for video work:

  tag_and_embed: soft_time_limit=300, time_limit=420
                 (sized for the image branch, ≈2 GPU ops)
  recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues

The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.

**Two-part fix**

1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
   (20 min). Sized for the video path's worst case; image runs return
   in seconds and don't care.

2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
   Queues with legitimately-long-running tasks (currently just `ml` at
   25 min — 5-min buffer past the new hard kill) get their own
   threshold; queues not in the dict use the default 5 min. The sweep
   now issues one UPDATE per distinct threshold value, with
   `queue.notin_(override_queues)` on the default pass so each row is
   touched at most once.

Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
  existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
  pins both directions: a 10-min-old ml row survives (fresh by 25-min
  override), a 30-min-old ml row gets flagged.

After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
2026-05-27 22:23:35 -04:00
bvandeusen b1b129ce9f feat(downloads-tab): A+B dashboard improvements — row restyle + date-grouped sections with failed-pinned
Operator-flagged 2026-05-27: the Downloads subtab "doesn't feel like a
dashboard" — status was a tiny mdi icon at the far left, platform chip
was neutral-tonal, errors were plain orange text floating on the right,
and all 28 rows from the same hour visually had the same priority.

**Row restyle (A):**
- 4px colored left-edge bar by status (success/error/info/warning/grey)
  — visually scannable at the edge without parsing the chip text
- Status chip with text label (Completed/Failed/Running/Queued/Skipped)
  + leading icon, tonal-colored. Replaces the bare mdi-icon.
- Platform chip swapped to the color-coded subscriptions/PlatformChip
  (Patreon=red mdi-patreon, SubscribeStar=amber, HentaiFoundry=purple,
  Discord=indigo, Pixiv=blue, DeviantArt=green).
- File count: tonal info chip when > 0, dim middle-dot when 0 (so
  scheduled "no-change" scans don't dominate the column visually).
- Error: red tonal pill chip with leading icon, truncated to 60 chars
  with full text in the title tooltip. Replaces plain text.
- Per-row actions (hidden at 50% opacity, fade to full on row hover):
  Retry (only when status=error AND source_id known — hits
  POST /api/sources/<id>/check via the existing sources.checkNow),
  Details (opens the detail modal), Open artist (navigates to the
  artist page). Clicks stop-propagation so they don't bubble to the
  row click.

**Date-grouped sections (B):**
- Events are bucketed into four sections: Today / Yesterday /
  Last 7 days / Earlier. Empty buckets are skipped. Buckets boundaries
  are computed against the operator's local-time start-of-day so
  "Today" matches their intuition.
- Each section has a collapsible header with a row-count chip + a
  red "failed in this section" chip when any failures are in scope.
- Within each section, status='error' rows are pinned to the top
  (operator's eye lands on failures first; successful scans flow
  below).
- Collapsed state persists across refresh within the SubscriptionsView
  lifetime (reactive object, default all-expanded).

DownloadEventRow grid widened to accommodate the status chip + actions
column. PolyMasonry-style ellipsis on the artist link prevents long
names from breaking the layout.

No new endpoints; the Retry path reuses the existing /api/sources/<id>/check
flow (the source-check endpoint was already in place, just not wired
into a per-row button).
2026-05-27 22:18:02 -04:00
bvandeusen 9075d8eadd Merge pull request 'v26.05.27.2: subscribestar + HF cookie quirks, platforms package refactor, showcase IR-parity, secure-context audit' (#30) from dev into main 2026-05-27 21:34:02 -04:00
bvandeusen df6d89cb59 fix(secure-context): full audit — DestructiveConfirmModal.expectedTokenOverride + bulk-delete + min-dim use backend-computed tokens
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.

**Audit results across `frontend/src/`:**

  crypto.subtle.digest        — 2 sites:
    - MinDimensionCard (fixed 2026-05-27)
    - BulkEditorPanel (THIS FIX)
  navigator.clipboard         — 1 site, already guarded:
    - utils/clipboard.js writeText with execCommand fallback
  serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
  cookieStore / queryLocalFonts / WebAuthn / geolocation
                              — NOT USED, nothing to fix

  Extension scripts (background.js) use crypto.subtle but run from
  moz-extension:// which IS a Secure Context — left as-is.

**BulkEditorPanel double bug**

The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:

1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
   never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
   `delete-images-selection-<sha8>` while the backend expected
   `delete-images-<sha8>`. The two would never match.

Fix:

- Backend `/api/admin/images/bulk-delete` dry-run response now returns
  `confirm_token` (the canonical `delete-images-<sha8>` string).
  Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
  assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
  set, it bypasses the `${action}-${kind}-${runId}` formula and uses
  the explicit string. This decouples the UI label (`kind`) from the
  wire-format token (server-provided), so future endpoints can use a
  kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
  — no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
  slicing the 8-char suffix off the backend's token and passing it
  through `runId`; now passes the full backend token via
  `expected-token-override` directly). Cleaner; one source of truth.

**Banked memory**

`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.

No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
2026-05-27 21:17:40 -04:00
bvandeusen 12be188ada feat(showcase): IR-parity R-key shuffle + stagger entry animation; fix(cleanup): min-dim Delete swallowed crypto.subtle TypeError on plain HTTP
**showcase R-key + entry animation**

Restores two behaviors lost during the FC-2 IR→Vue port. Operator-flagged
2026-05-27.

- ShowcaseView listens for keydown 'r'/'R' on window. Triggers
  `store.shuffle()`. Skips when an input/textarea/contenteditable is
  focused or a Vuetify overlay is open (the dialog/menu sets
  `.v-overlay--active` on the body).
- MasonryGrid gains an opt-in `animateFromIndex` prop (default
  `Number.POSITIVE_INFINITY` = off). When set, items with index ≥ the
  threshold animate in with a stagger fade-in: 12px translateY,
  0.25s ease, 60ms per item, capped by `prefers-reduced-motion`.
  Stagger uses original-items-array index (resolved via an `idxById`
  Map) so the reading order is preserved even after the masonry
  distributes items across columns.
- ShowcaseView watches `store.images.length`: shrink-or-zero baseline
  ⇒ `animateFromIndex=0` (animate everything on initial load /
  shuffle); grow ⇒ baseline=prevCount (animate only the appended
  tail on infinite-scroll). Other MasonryGrid consumers (ArtistView's
  Gallery tab) don't pass the prop, so they keep their current
  no-animation behavior.

Direct port of IR's `app/static/js/showcase.js` keyboard handler +
`app/static/style.css` itemFadeIn keyframe.

**min-dim Delete: crypto.subtle TypeError fix**

The Delete button on the Cleanup → Minimum Dimensions card was
silently no-op'ing. Root cause: `crypto.subtle` is Secure-Context-gated
(undefined on plain-HTTP origins per the homelab posture). The card's
`onDeleteClick` computed the Tier-C confirm token via
`crypto.subtle.digest('SHA-256', ...)`, which threw TypeError before
`showModal.value = true`. The promise rejected, the click handler had
no `.catch`, the modal never opened — exactly the operator's reported
symptom.

Same shape as the v26.05.26.0 `navigator.clipboard` fix on the
ErrorDetailModal Copy button.

Fix: backend `/api/cleanup/min-dimension/preview` now returns
`confirm_token` (the canonical `delete-min-dim-<sha8>` string) in its
response. Frontend reads it from the preview response and feeds the
8-char suffix to DestructiveConfirmModal's `runId` prop — no
client-side crypto needed. Single source of truth.

Integration test `test_min_dimension_preview_returns_count` pinned to
also assert `body["confirm_token"]` matches the server-side compute.
2026-05-27 20:59:58 -04:00
bvandeusen 6d7116c090 fix(platforms): ruff I001 in base.py — one blank line between imports and module-level constant (was two) 2026-05-27 20:37:06 -04:00
bvandeusen b447c42853 fix(platforms): ruff I001 — drop unused __future__ import; switch __init__ to per-module imports for clean isort ordering 2026-05-27 19:52:50 -04:00
bvandeusen abafc3265e refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."

**Layout**

  services/platforms/
    base.py            PlatformInfo dataclass + module-default key
                       chains + shared helpers (str_id_value, str_field)
    __init__.py        PLATFORMS dict + public API (auth_type_for,
                       known_platform_keys, to_dict,
                       external_post_id_keys_for, description_keys_for)
    patreon.py         metadata only — the reference platform, no quirks
    subscribestar.py   metadata + augment_cookies (18+ agreement) +
                       derive_post_url (synthetic /posts/<post_id>)
    hentaifoundry.py   metadata + augment_cookies (host-only PHPSESSID
                       duplicate) + derive_post_url (/pictures/user/...)
    pixiv.py           metadata + derive_post_url (/artworks/<id>)
    discord.py         metadata + derive_post_url
                       (channels/<server>/<channel>/<message>)
    deviantart.py      metadata only — un-audited; quirks to be added
                       when an operator first exercises DA

**PlatformInfo extensions**

Existing fields preserved. Four new optional fields:

  external_post_id_keys: tuple[str, ...] | None
      Override the sidecar external_post_id lookup chain. None falls
      back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
      ("post_id", "id", "index", "message_id") — covers every current
      platform.

  description_keys: tuple[str, ...] | None
      Override the description body lookup chain. None falls back to
      DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
      "message") — Discord's "message" body field is covered by the
      default's trailing entry.

  derive_post_url: Callable[[dict], str | None] | None
      Synthesize the post permalink from sidecar metadata. None = trust
      the bare `url` / `post_url` field (patreon, deviantart).
      subscribestar/pixiv/hf/discord override this because their `url`
      is the file CDN URL.

  augment_cookies: Callable[[str], str] | None
      Post-process the materialized cookies.txt before gallery-dl
      consumes it. None = no-op. Used by subscribestar (age cookie) and
      hentaifoundry (host-only PHPSESSID duplicate).

**Consumer changes**

- credential_service._augment_cookies(platform, netscape) shrunk from a
  per-platform-conditional dispatcher (~80 lines of inlined helpers) to
  a 5-line lookup: `info.augment_cookies(netscape) if info and
  info.augment_cookies else netscape`. The platform-specific helper
  bodies moved verbatim into the per-platform modules.

- sidecar.parse_sidecar similarly delegates: external_post_id chain via
  external_post_id_keys_for(category), description chain via
  description_keys_for(category), post_url via
  PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
  and inline _derive_post_url body both gone. Added a shared `_first_id`
  helper for bool-safe id coercion.

**Public API preserved**

PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.

**Adding a new platform**

1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
   and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
   tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
   pick it up automatically.
2026-05-27 19:46:05 -04:00
bvandeusen 2394e47370 fix(hentaifoundry): inject host-only PHPSESSID/CSRF duplicates + extension preserves browser hostOnly
Operator-flagged 2026-05-27: HF source check 401'd on
`HEAD /?enterAgree=1` even with valid login cookies. Root cause is the
combination of (1) gallery-dl's HF extractor checking
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching, and (2) the extension's cookies.js
forcibly rewriting every captured cookie to a leading-dot subdomain-wide
form. HF's PHPSESSID is browser-stored as host-only on
`www.hentai-foundry.com`; the rewrite re-anchored it to
`.hentai-foundry.com`, which `cookies.get(...)` no longer matches even
though the cookie is still sent on actual HTTP requests (RFC 6265
subdomain rules). The extractor falls into its unauthenticated
`?enterAgree=1` fallback, which 401s (Cloudflare or HF's anti-bot HEAD
gating).

Two-part fix, no operator action required for existing stored cookies:

1. **Backend** (`credential_service._augment_cookies`) — refactored from
   the subscribestar-only single function into a per-platform dispatcher.
   New `_augment_hentaifoundry` parses the materialized netscape file
   and, for each `.hentai-foundry.com` entry whose name is PHPSESSID or
   YII_CSRF_TOKEN, appends a host-only duplicate
   (`www.hentai-foundry.com\tFALSE\t...`). Originals preserved. Three
   new tests pin: injection fires + originals preserved; idempotent
   when host-only already exists; doesn't touch unrelated cookies
   (e.g. `_ga`).

2. **Extension** (`cookies.js`) — `toNetscapeFormat` now respects
   `c.hostOnly` from the browser instead of blindly forcing a
   leading-dot subdomain-wide form. Host-only cookies are written with
   the bare host + FALSE flag; non-host-only cookies retain the
   leading-dot + TRUE form. Forward-compat — fresh captures from
   v1.0.5+ no longer need the backend's host-only duplication.
   Extension bumped 1.0.4 → 1.0.5; manifest + package.json in lockstep.

After deploy: the next HF source check on the operator's already-stored
cookies will succeed because the materialized cookies.txt now contains
host-only PHPSESSID. No browser re-export needed.
2026-05-27 19:12:51 -04:00
bvandeusen 8243740a04 fix(subscribestar): inject 18_plus_agreement_generic age cookie to bypass server gate
Operator-flagged 2026-05-27: subscribestar source check aborted with
`AbortExtraction: HTTP redirect to .../age_confirmation_warning`. The
captured `_personalization_id` cookie in the browser-stored file had
expired (annual rotation), and the user could not realistically refresh
it: SubscribeStar's frontend JS uses localStorage to suppress the
age-confirmation popup once dismissed, so a logged-in revisit doesn't
re-show the popup and the server-side cookie is never re-issued.

gallery-dl's own login flow (which FC doesn't exercise — cookies come
from the extension instead) sidesteps this by manually setting
`18_plus_agreement_generic=true` on `.subscribestar.adult`. The server
accepts that as the age-confirmation marker.

`credential_service._augment_cookies(platform, netscape)` mirrors that
behavior: when the materialized cookies file is for subscribestar and
the age cookie isn't already present, append a synthetic line for
`.subscribestar.adult` with name=`18_plus_agreement_generic` value=`true`
and a far-future expiry. No-op for other platforms; no-op if the cookie
is already present (idempotent for manual pastes / extension captures
that happen to include it).

Three new tests pin: (a) injection fires for subscribestar, preserves
existing cookies; (b) idempotent when already present (no double
injection); (c) does NOT fire for non-subscribestar platforms (Patreon
etc. don't get a foreign-domain cookie).

Not a curator handling bug per se — the extension faithfully captured
what the browser had. This is mirroring a documented gallery-dl
workaround so the cookies-via-extension auth path doesn't degrade as the
server-side cookie expires.
2026-05-27 18:18:04 -04:00
bvandeusen 88e53e5b86 Merge pull request 'v26.05.27.1: subscriptions hub + post-card merge + sidecar audit' (#29) from dev into main 2026-05-27 17:12:48 -04:00
bvandeusen aa28bddeab fix(alembic 0025): qualify ambiguous post.id / post.source_id in fragment-group SELECT (post JOIN source — both have id) 2026-05-27 15:45:42 -04:00
bvandeusen b7b313cc05 fix(alembic 0025): include HF + Discord post_url backfill (no longer 'deferred to deep-scan')
Operator-flagged: the claim that 'a future deep-scan via the new parser
will fix HF and Discord post_url' was conditional on the operator
actually running a deep-scan, which they might not do for ages. Until
then HF posts stay at post_url=NULL (HF sidecars have no `url` field)
and Discord posts stay pointing at cdn.discordapp.com/attachments/...
(the file URL, not the message permalink).

The migration was already opening sidecar files for SubscribeStar.
Generalizing the loop to also handle HF and Discord is a tiny addition
that closes the gap without operator intervention.

Per-platform Part 1 logic now:
  subscribestar — read sidecar.post_id, overwrite external_post_id +
    post_url with the derived /posts/<post_id> permalink.
  hentaifoundry — read sidecar.user + .index, overwrite post_url with
    /pictures/user/<u>/<i>. external_post_id (= index) unchanged.
  discord — read sidecar.server_id + .channel_id + .message_id,
    overwrite post_url with the discord.com/channels/.../<m> triple.
    external_post_id (= message_id) unchanged.

Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv
post_url backfill) unchanged.

Posts whose related ImageRecord paths don't resolve on disk (orphan
filesystem state) are reported per-platform in the migration output —
those still need a future deep-scan, but the in-DB-with-on-disk-files
common case is now fully covered by the migration alone.
2026-05-27 15:38:18 -04:00
bvandeusen bd3f996582 fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:

1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
   per-attachment id in `id` (e.g. 711509) and the actual post id in
   `post_id` (e.g. 360360). FC's external_post_id chain had `id`
   winning, so every multi-image SubscribeStar post was fragmented into
   N Post rows in the database. Reorder the chain to
   `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
   `post_id`), HF (uses `index`), Discord (uses `message_id`) all
   unaffected.

2. Discord `message` field not captured. Discord posts put the body in
   `message`, not `content`. Append it to the description fallback chain
   `("content", "description", "caption", "message")`.

3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
   `_derive_post_url(platform, data)` helper synthesizes proper
   permalinks from per-platform fields:
     subscribestar → https://www.subscribestar.com/posts/<post_id>
     pixiv         → https://www.pixiv.net/artworks/<id>
     hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
     discord       → https://discord.com/channels/<server>/<channel>/<message>
   Patreon's bare `url` IS a real permalink and is used as-is. For the
   four file-URL platforms, the bare `url` is NEVER trusted: derive or
   return None rather than persist a CDN URL.

Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
  wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
  shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
  bodies.
- Five new tests cover per-platform post_url derivation
  (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
  None).

Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
  its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
  post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
  same NEW external_post_id is a fragment-set — merge to a canonical
  row using the same ImageProvenance pre-delete + repoint dance as
  alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
  url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
  field (not stored on Post), Discord needs server/channel triple.
  Both will be corrected by a deep-scan re-applying sidecars through
  the new parser.

Idempotent: re-running on already-corrected data is a no-op.
2026-05-27 15:35:25 -04:00
bvandeusen ae8c78ae09 fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar)
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. Confirmed against the operator's
/mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every
post's JSON has `title: ""` and a content like
`<div>Lets say hello to you guys with my Belle <br><br><br></div>`.
FC's sidecar parser, treating empty strings as missing, had been leaving
post_title NULL on every subscribestar post since FC-3 shipped.

Fix at two layers:

1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)`
   helper strips HTML tags, collapses whitespace, returns the first
   non-empty line truncated to 120 chars with ellipsis. `parse_sidecar`
   now falls back to this when `title` resolves to None and a
   `content`/`description`/`caption` value is present. Patreon's
   non-empty titles short-circuit the fallback so existing behavior is
   unchanged. Four new tests in test_sidecar_util.py pin: derivation
   from content, truncation at 120 chars, explicit-title precedence,
   no-content no-fallback.

2. `alembic 0024_backfill_post_title_from_description` — backfills the
   same logic across existing Post rows where `post_title IS NULL OR
   post_title = ''` AND description is present. Idempotent (re-running
   is a no-op once titles are populated). Downgrade is a no-op since
   there's no safe way to tell derived rows from genuine ones.

After deploy + migration: subscribestar posts will surface a meaningful
title in PostCard, post feed search, etc.
2026-05-27 14:44:09 -04:00
bvandeusen 4d2c464045 feat(post-card): absorb PostModal into PostCard with click-to-expand
PostCard and PostModal competed for the same data and rendered redundant
chrome (header twice, image grid twice, attachment list twice). The wider
PostCard layout we shipped 2026-05-27 has enough real estate to be the
canonical post surface, so collapse the two into one.

Compact (default) state is unchanged: hero + 3-cell rail + truncated
title + 3/5-line description + attachment count badge. Whole-card click
expands in place. Expanded state shows: full title, mosaic of ALL post
images via PostImageGrid (uncapped, lazy-loaded via getPostFull), full
sanitized-HTML description with paragraph wrapping, attachments as
downloadable pill links. Click the chevron in the header to collapse;
mosaic image clicks open ImageViewer scoped to the post (modalStore's
postImageIds path is preserved — only the comment changed).

Per-card local state — no global modal store. Each PostCard owns its
own expanded ref and lazy-loaded detail; collapsing a card discards
neither (so re-expand is instant after the first fetch).

Deleted: PostModal.vue, postModal.js store. Removed the App.vue mount.
2026-05-27 14:30:04 -04:00
bvandeusen b8ad17c68d fix(build): poll for ext-<version> release in tag-push build-web (race fix)
Cutting a release fires BOTH the push-to-main workflow AND the push-to-tag
workflow in parallel. main-push runs sign-extension (AMO round-trip 1-5min)
then publishes the ext-<version> Forgejo release; tag-push skips
sign-extension (gated to main) and races straight to build-web's Download
XPI step. Tag-push lost every time — got 404 from
releases/tags/ext-<version> before main-push had finished signing.

v26.05.27.0 hit this: tag-push build-web died on exit 22 because the
ext-1.0.4 release wasn't published yet (it arrived ~4min later).

Fix: wrap the release lookup in a 20-iteration sleep+retry loop, 30s
between attempts (10min total upper bound, generous for AMO). main-push's
signing eventually publishes the release; tag-push picks it up on a later
poll. No more manual rerun of the failed job after every release cut.

Banked the trap as reference_tag_push_main_push_race.md — same shape will
recur any time a tag-push workflow consumes a main-push-produced artifact.
2026-05-27 13:25:58 -04:00
bvandeusen 1fd54897d8 fix(api): ruff UP017 — use datetime.UTC alias in /api/downloads/stats 2026-05-27 13:11:44 -04:00
bvandeusen 9322c984fd feat(subs-hub): collapse /credentials + /downloads into /subscriptions hub with three GS-style subtabs
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.

**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.

**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.

**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.

**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.

**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.

Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
2026-05-27 13:02:24 -04:00
bvandeusen 37e8b796a1 Merge pull request 'v26.05.27.0: PostCard redesign + IR-style tag suffix + drop meta/rating + extension v1.0.4 CSP fix' (#28) from dev into main 2026-05-27 11:31:18 -04:00
bvandeusen 8675f105ad fix(tests): test_api_tags prefix tests use character: not artist: (KNOWN_KINDS dropped artist)
The two prefix-parsing tests were pinned to `artist:Eric`, but `artist`
was removed from KNOWN_KINDS in commit 4cad07a (provenance is a separate
axis from tags). The parser now keeps `artist:` literal, so the assertion
`body["name"] == "Eric"` failed.

Repointed to `character:Saber` (still in KNOWN_KINDS). Also updated the
stale `artist:` docstring example in parse_kind_prefix to `fandom:`.

Caught by [[reference-grep-pinned-tests-in-plans]] — should have grep'd
tests/ for `artist:` when shrinking KNOWN_KINDS. Banking the miss.
2026-05-27 11:09:04 -04:00
bvandeusen 74dac6b960 fix(extension+migration): MV3 CSP opt-out from upgrade-insecure-requests (v1.0.4) + alembic 0023 drops the ck_tag_fandom_requires_character check before the type swap
extension/manifest.json: add content_security_policy.extension_pages = "script-src 'self'; object-src 'self';" — explicitly omits the upgrade-insecure-requests directive that MV3 inherits by default. Without this, every fetch(http://curator.../...) silently upgrades to https:// at the browser layer (Sec-Fetch-Site=same-origin, NS_ERROR_GENERATE_FAILURE), regardless of about:config. Bump XPI version 1.0.3 → 1.0.4 so a fresh signed build replaces the cached one. Operator-troubleshot 2026-05-26 via Inspect-the-extension dev tools showing the silent scheme upgrade.

alembic 0023: drop ck_tag_fandom_requires_character before the tag_kind type swap and recreate after. Postgres can't resolve `kind = 'character'` across the rename (column on tag_kind_old, literal binds to new tag_kind → "operator does not exist"). Same dance on downgrade. Banked under reference_tag_kind_enum_swap_check_drop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:57:38 -04:00
bvandeusen 9e19c081b0 fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:23:12 -04:00
bvandeusen 3838f04c16 feat(tag-kinds): drop meta + rating entirely — alembic 0023 deletes existing meta/rating tags (CASCADE clears related image_tag / alias / allowlist / suggestion_rejection / reference_embedding / series_page rows) then recreates the tag_kind ENUM without those values. Python TagKind enum trimmed; KIND_OPTIONS + KIND_COLOR + KIND_ICONS maps + TagsView KINDS array all updated. Operator confirmed they have no use for the data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:07:31 -04:00
bvandeusen 42b1340324 fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:00:12 -04:00
bvandeusen 3b1e2f1ceb feat(tag-input): IR-style kind:name suffix — drop the kind dropdown from TagAutocomplete; client-side parser mirrors backend's parse_kind_prefix (KNOWN_KINDS = artist/character/fandom/series/meta/rating); autocomplete searches across all kinds and shows kind chip in results; Create label uses parsed kind; character flow still goes through FandomPicker
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:54:11 -04:00
bvandeusen 8cdf0af0e1 feat(tags-api): IR-style kind:name parsing at POST /api/tags — when caller doesn't supply explicit kind, parse_kind_prefix runs on the name (artist:Eric → kind=artist, name='Eric'); explicit kind always wins for backward-compat; falls back to general when no recognized prefix is present. Updates the old "missing required" test that assumed kind was mandatory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:53:27 -04:00
bvandeusen ccee344099 feat(tag-prefix): parse_kind_prefix util — IR-style \kind:name\ parser at the input boundary; KNOWN_KINDS = artist/character/fandom/series/meta/rating (excludes default \general\ and system-managed archive/post)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:49 -04:00
bvandeusen 0316f92e8b feat(artist-posts-tab): bump max-width 900 → 1600 so the new wide-layout PostCard has room and the artist Posts feed doesn't leave most of an ultra-wide screen empty
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:21 -04:00
bvandeusen 6df74683b3 feat(post-card): responsive redesign — container-query split (stack <800px / side-by-side ≥800px), hero + thumb rail, +N overflow chip, line-clamp body (3 narrow / 5 wide), title/desc fallbacks for sparse data, click→postModal.open
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:10 -04:00
bvandeusen 243e536225 feat(app): mount PostModal at app root next to ImageViewer — single instance driven by usePostModalStore so PostCard can open from anywhere
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:50:34 -04:00
bvandeusen 2f16699971 feat(post-modal): PostModal — full Patreon-style v-dialog (header + image grid + sanitized body + attachments); reads from usePostModalStore
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:59 -04:00
bvandeusen a5cb684d34 feat(post-modal): PostImageGrid — fixed-cell grid (auto-fill 220px+, 4:3 aspect-cover) inside PostModal; click opens ImageViewer scoped to the post's images via modalStore.open(id, { postImageIds })
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:20 -04:00
bvandeusen 965a953b2e feat(post-card): PostEmptyThumbs — dashed-border placeholder shown in PostCard's hero slot when post has zero linked images
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:02 -04:00
bvandeusen 90c176b195 feat(postmodal-store): Pinia store driving the app-level PostModal — open(post) fetches full detail via posts store; close() clears
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:49 -04:00
bvandeusen b8d89b9f2a feat(modal-store): post-scoped cycle — open(id, { postImageIds }) pins prev/next to the array; canPrev/canNext + goPrev/goNext check the array index instead of current.value.neighbors when set. Gallery-context callers unchanged (default args clear scope)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:30 -04:00
bvandeusen 07344e0843 feat(util): htmlSanitize — whitelist-based DOM scrubber for PostModal's description v-html (Patreon ships HTML; sanitize before render)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:36 -04:00
bvandeusen 42c33e44f9 feat(post-api): get_post returns uncapped thumbnails — PostModal masonry needs full image list; feed query unchanged (still capped at 6 for previews). _thumbnails_for gains a limit kwarg; get_post passes limit=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:18 -04:00
bvandeusen 4e82208926 Merge pull request 'v26.05.26.5 — extension CORS unblock + UI gap closes + CI workflow cleanup' (#27) from dev into main 2026-05-26 20:15:07 -04:00
bvandeusen 85b640f32e fix(views): close the 24-32px gap below TopNav across all views — every v-container had py-6 (or py-8 on PlaceholderView) which pushed the first content item well below where the TopNav's fade-to-transparent gradient bottoms out. Switch to pt-2 pb-6 (8px top, 24px bottom) so content sits comfortably right below the nav, matches the ArtistHeader's 'continuous with TopNav' feel. PlaceholderView uses pt-3 pb-8 keeping its larger bottom padding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:47:23 -04:00
bvandeusen c7001f4aed fix(extension): CORS preflight for moz-extension:// + chrome-extension:// origins — operator-flagged 2026-05-26 that the extension's Test connection returned NetworkError because /api/credentials POSTs with X-Extension-Key trigger a browser preflight OPTIONS that hit a 405 (no OPTIONS method registered) with no Access-Control-Allow-* headers. Adds two app-level hooks: before_request short-circuits OPTIONS from extension origins with 204, after_request stamps the necessary ACL headers on responses to extension-origin requests. Whitelist is intentionally narrow (extension schemes only) so normal browser usage doesn't get permissive CORS. Five integration tests pin the contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:43:31 -04:00
bvandeusen f827612930 fix(artist-header): close gap below TopNav (top:64px → 48px to match TopNav's actual ~48px height) + center the tab strip via 1fr|auto|1fr layout with a right-side spacer cell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:36:19 -04:00
bvandeusen 3f0153cba5 ci(workflows): dedupe + versioned image tags
ci.yml: drop pull_request: trigger — push: branches: [dev, main] already covers it; pull_request was duplicating ci.yml runs on every dev push with an open PR. (No fork PRs in this repo.)

build.yml: drop dev from push triggers — operator doesn't use the :dev image. Add tags: ['v*'] trigger + tag-push branch in the Determine-tag logic so cutting a release tag publishes an immutable :v26.05.26.X image (rollback story) without re-publishing :latest. Extend the XPI-download step to fire on tag pushes too so the versioned image carries the signed extension.

Net per hotfix cycle: 5 runs → 3 (no tag) / 4 (with tag).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:26:56 -04:00
bvandeusen 52fff00353 Merge pull request 'v26.05.26.4 — hotfix: migration 0022 pre-DELETE colliding ImageProvenance before UPDATE' (#26) from dev into main 2026-05-26 18:06:20 -04:00
bvandeusen f3e8f30a8f fix(migration-0022): pre-DELETE colliding image_provenance rows before the UPDATE post_id — same row-by-row UNIQUE pattern as the post-collision case, just one level deeper. When image X has provenance under both keep and drop, UPDATE drop→keep would fire uq_image_provenance_image_post on the row that'd collide with the existing (X, keep). Pre-delete those rows (their info is already represented by the keep-side provenance) before the UPDATE moves the rest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:05:49 -04:00
bvandeusen eee107766e fix(migration-0022): rename unused _epid loop var (ruff B007)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:03:11 -04:00
bvandeusen c14338cbce Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main 2026-05-26 17:52:59 -04:00
bvandeusen 7a64730bd2 fix(migration-0022): pre-merge ALL duplicate-external_post_id Posts across the (canonical+others) group, not just canonical-vs-others — operator's v26.05.26.2 deploy still tripped uq_post_source_external_id because two non-canonical Sources both had Posts with epid=6166997. Bulk UPDATE moved the first cleanly then collided on the second. New pre-merge groups all Posts in the (artist, platform) by external_post_id; for any group with count>1, picks the keep (prefer one under canonical; else lowest id) and merges the rest before the bulk reparent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:52:29 -04:00
bvandeusen 1803a09306 ci(workflow): remove the 4 Cache pip wheels steps entirely — act_runner's cache backend has been broken for 11+ days and the cached path (~/.cache/pip) wasn't even the primary install tool's cache anyway (uv uses ~/.cache/uv). Net cost ~30s/job of wheel downloads. Long-term: mount ~/.cache/uv as a docker volume at the runner level (skips actions/cache entirely) or fix the runner-side cache backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:54:05 -04:00
289 changed files with 13122 additions and 5244 deletions
+97 -19
View File
@@ -2,7 +2,17 @@ name: Build images
on:
push:
branches: [dev, main]
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
# merge-to-main, not from the dev branch image. Saves one full docker
# build per dev push.
branches: [main]
# Tag-push triggers an immutable per-version image build (e.g.
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
# negligible per tag. Doesn't overlap with the push-to-main build (that
# one publishes `:main` + `:latest`; the tag-push build publishes only
# `:<tag>`).
tags: ['v*']
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com)
@@ -158,25 +168,56 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Download signed XPI from Forgejo release asset (main only)
if: github.ref == 'refs/heads/main'
- name: Download signed XPI from Forgejo release asset (main + tags)
# Fires on main-push AND on tag-push. Tag-push builds re-package the
# same source code as the preceding main-push build but with an
# immutable version tag — they need the XPI too, otherwise the
# versioned image ships without the signed extension.
#
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
# simultaneously. Main-push runs sign-extension (1-5min AMO round
# trip) before publishing the ext-<version> release; tag-push
# skips sign-extension (gated to main) and races straight to
# this download step. Tag-push lost every time. Fix: poll the
# ext-<version> release endpoint with a sleep+retry loop (30s
# for up to 10min total) before giving up. Main-push's signing
# eventually wins and tag-push picks the release up on a later
# iteration.
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
# Look up the ext-<version> release; extract the .xpi asset's
# browser_download_url (Forgejo's /releases/assets/<id> endpoint
# returns ASSET METADATA, not the binary blob — operator-flagged
# 2026-05-26: my prior code curl'd the metadata endpoint without
# -f and wrote the resulting 404-page-not-found text into
# fabledcurator-*.xpi, which Firefox then rejected as "corrupt").
# browser_download_url is the canonical binary endpoint and is
# also publicly accessible (no token needed) but we pass the
# token anyway for symmetry with private-repo support.
curl -sf -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" \
-o release.json
# Poll for the ext-<version> release. main-push's sign-extension
# step (AMO round-trip, 1-5min) needs to finish + upload before
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
for attempt in $(seq 1 20); do
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
if [ "$STATUS" = "200" ]; then
echo "Found ext-$VERSION release on attempt $attempt"
break
fi
if [ "$attempt" = "20" ]; then
echo "ERROR: ext-$VERSION release not available after 10min of polling"
echo "Last HTTP status: $STATUS"
exit 1
fi
echo "Attempt $attempt: ext-$VERSION not yet published (HTTP $STATUS); sleeping 30s"
sleep 30
done
# Extract the .xpi asset's browser_download_url (Forgejo's
# /releases/assets/<id> endpoint returns ASSET METADATA, not
# the binary blob — operator-flagged 2026-05-26: my prior
# code curl'd the metadata endpoint without -f and wrote the
# resulting 404-page-not-found text into fabledcurator-*.xpi,
# which Firefox then rejected as "corrupt").
# browser_download_url is the canonical binary endpoint and
# is also publicly accessible (no token needed) but we pass
# the token anyway for symmetry with private-repo support.
DOWNLOAD_URL=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['browser_download_url'])")
test -n "$DOWNLOAD_URL"
echo "Downloading XPI from: $DOWNLOAD_URL"
@@ -200,8 +241,33 @@ jobs:
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
# Three trigger shapes:
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
# no `.N` per family release-posture rule).
# Publish ONLY the immutable version tag;
# don't touch :latest (the main-push build
# for the merge commit already did that).
# refs/heads/main → push to main: publish :main + :latest
# (floating) AND :c-<short_sha> (immutable
# per-commit rollback substrate, per family
# release-posture rule "Tags are milestones,
# not gates — commit-SHA images are the
# rollback unit"). Rollback to any commit
# becomes `docker pull …:c-<sha>` without a
# release ceremony.
# anything else → safety net; shouldn't fire given the `on:`
# config above. Tag :dev to surface the
# unexpected run in the registry.
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
fi
@@ -231,8 +297,20 @@ jobs:
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
# Mirrors build-web's three-shape logic (tag-push / main-push /
# safety-net dev) including the per-commit :c-<short_sha> tag
# on main-push per the family release-posture rule. The -ml
# image follows the same release cadence as the web image.
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
fi
+33 -70
View File
@@ -1,17 +1,34 @@
name: CI
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
# - backend-lint-and-test: ruff + `pytest -m "not integration"`, no service containers.
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
# - frontend-build: vitest unit + vite build.
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
on:
push:
branches: [dev, main]
pull_request:
branches: [main]
# pull_request trigger intentionally absent — with branches: [dev, main]
# above, every PR commit already fires CI via the push event on dev. Adding
# pull_request would duplicate runs on dev→main PRs. FC has no fork PRs
# (single-operator Forgejo repo) so push coverage is complete.
jobs:
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
# this runs with NO dependency install and surfaces the most common bounce
# class (lint: I001 / UP037 / ASYNC109 / W293 …) in seconds — instead of
# after the backend job's ~30-60s wheel install. ruff is static analysis,
# so no DB/secret env is needed.
lint:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Ruff lint
run: ruff check backend/ tests/ alembic/
backend-lint-and-test:
runs-on: python-ci
container:
@@ -24,22 +41,17 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Cache pip wheels
# continue-on-error: act_runner's cache backend has been broken on
# this homelab setup since 2026-05-15 — bolt.db + cache/ dir exist
# but the action's JS bundle now fails to load at all
# ("Cannot find module .../dist/restore/index.js"), hard-failing
# the whole job. The install step that follows handles cold caches
# natively (uv pip / pip install both work without it), so net
# cost of disabling the cache here is ~30s of wheel downloads per
# job. Re-enable once the runner-side cache backend is fixed.
continue-on-error: true
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
restore-keys: |
pip-${{ runner.os }}-py314-
# Cache step removed 2026-05-26: act_runner's cache backend has been
# broken on this homelab runner since 2026-05-15 (first as request-
# timeout warnings, then as hard "Cannot find module .../dist/restore/
# index.js" failures that tank the whole job). The cache step targeted
# ~/.cache/pip but the install below uses `uv pip install` primarily,
# whose own cache lives at ~/.cache/uv — so the cache step's real
# benefit was marginal even when working. Cost of removal: ~30s of
# wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
# docker volume at the runner level (skips actions/cache entirely),
# or fix the runner-side cache backend (clear /var/run/act/actions/*,
# pin act_runner version, etc.).
- name: Install Python deps
# ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
@@ -54,9 +66,8 @@ jobs:
pip install -r requirements.txt pytest pytest-asyncio
fi
- name: Ruff lint
run: ruff check backend/ tests/ alembic/
# Ruff moved to the dedicated fast `lint` job above (fails in seconds,
# no dep install). This job is now unit tests only.
- name: Pytest (unit only — integration runs in the integration job)
run: pytest tests/ -v -m "not integration"
@@ -133,22 +144,6 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Cache pip wheels
# continue-on-error: act_runner's cache backend has been broken on
# this homelab setup since 2026-05-15 — bolt.db + cache/ dir exist
# but the action's JS bundle now fails to load at all
# ("Cannot find module .../dist/restore/index.js"), hard-failing
# the whole job. The install step that follows handles cold caches
# natively (uv pip / pip install both work without it), so net
# cost of disabling the cache here is ~30s of wheel downloads per
# job. Re-enable once the runner-side cache backend is fixed.
continue-on-error: true
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
restore-keys: |
pip-${{ runner.os }}-py314-
- name: API integration shard (resolve service IPs, migrate, test)
run: |
set -eux
@@ -207,22 +202,6 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Cache pip wheels
# continue-on-error: act_runner's cache backend has been broken on
# this homelab setup since 2026-05-15 — bolt.db + cache/ dir exist
# but the action's JS bundle now fails to load at all
# ("Cannot find module .../dist/restore/index.js"), hard-failing
# the whole job. The install step that follows handles cold caches
# natively (uv pip / pip install both work without it), so net
# cost of disabling the cache here is ~30s of wheel downloads per
# job. Re-enable once the runner-side cache backend is fixed.
continue-on-error: true
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
restore-keys: |
pip-${{ runner.os }}-py314-
- name: Importer integration shard (resolve service IPs, migrate, test)
run: |
set -eux
@@ -281,22 +260,6 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Cache pip wheels
# continue-on-error: act_runner's cache backend has been broken on
# this homelab setup since 2026-05-15 — bolt.db + cache/ dir exist
# but the action's JS bundle now fails to load at all
# ("Cannot find module .../dist/restore/index.js"), hard-failing
# the whole job. The install step that follows handles cold caches
# natively (uv pip / pip install both work without it), so net
# cost of disabling the cache here is ~30s of wheel downloads per
# job. Re-enable once the runner-side cache backend is fixed.
continue-on-error: true
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
restore-keys: |
pip-${{ runner.os }}-py314-
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
run: |
set -eux
@@ -83,55 +83,80 @@ def upgrade() -> None:
if not other_ids:
continue
# STEP 2: PRE-merge colliding Posts BEFORE the bulk reparent.
# Find pairs (keep, drop) where:
# keep = a Post under the canonical source
# drop = a Post under one of the non-canonical sources whose
# external_post_id equals keep.external_post_id
# If we let the bulk UPDATE try to repoint drop onto canonical,
# uq_post_source_external_id fires row-by-row and aborts the
# whole migration.
colliding = conn.execute(
# STEP 2: PRE-merge ALL Posts with duplicate external_post_id
# across the entire (canonical + others) group, BEFORE the bulk
# reparent. Two cases must both be handled:
# (A) canonical has Post X with epid=N; an "other" source has
# Post Y with epid=N → after bulk UPDATE, (canonical, N)
# collides with itself.
# (B) two different "other" sources each have a Post with
# epid=N; canonical has none → after bulk UPDATE, both
# are repointed to (canonical, N) and the second collides.
# The earlier version of this migration only handled (A); the
# operator's deploy 2026-05-26 tripped (B) at line 139.
# Fix: group ALL Posts in the (artist, platform) by epid; for
# any group with count>1, pick the keep (prefer one already
# under canonical; else lowest id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT keep.id AS keep_id, drop_.id AS drop_id
FROM post AS keep
JOIN post AS drop_
ON drop_.external_post_id = keep.external_post_id
AND drop_.id != keep.id
WHERE keep.source_id = :canonical
AND drop_.source_id = ANY(:others)
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:others)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "others": other_ids},
).fetchall()
for keep_id, drop_id in colliding:
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
# Repointed provenance may now collide on
# uq_image_provenance_image_post (alembic 0021). Dedupe:
# keep min(id) per (image_record_id, post_id).
conn.execute(text("""
DELETE FROM image_provenance ip1
USING image_provenance ip2
WHERE ip1.image_record_id = ip2.image_record_id
AND ip1.post_id = ip2.post_id
AND ip1.id > ip2.id
"""))
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
# Prefer a Post already under canonical as the keep.
canonical_posts = [p for p in posts if p[1] == canonical_id]
if canonical_posts:
keep_id = canonical_posts[0][0]
else:
keep_id = posts[0][0] # already sorted by id ASC
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id ALREADY has a provenance under keep —
# the UPDATE below would otherwise repoint them and
# trip uq_image_provenance_image_post (alembic 0021)
# row-by-row before any after-the-fact dedupe could
# run. Operator's v26.05.26.3 deploy 2026-05-26 tripped
# this at line 123.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
# Now safe to repoint the survivors.
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP 3: Bulk reparent the remaining Posts off the other
# Sources. After step 2, no collisions on
@@ -0,0 +1,99 @@
"""drop meta + rating tag kinds — operator-retired 2026-05-26
Revision ID: 0023
Revises: 0022
Create Date: 2026-05-26
Operator decided meta + rating aren't valid tag kinds for FC. Per-row
behavior: DELETE existing rows (operator chose "clean break" over
"convert to general"). All cascading FKs (image_tag, tag_alias,
tag_allowlist, tag_reference_embedding, tag_suggestion_rejection,
series_page) use ondelete="CASCADE" so a single DELETE on tag cleans
the related rows in one go.
After the data cleanup, recreate the tag_kind ENUM without 'meta' /
'rating' (Postgres has no `ALTER TYPE ... DROP VALUE`; standard
rename-create-cast-drop dance). The server default 'general' is
dropped before the type swap and restored after.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0023"
down_revision: Union[str, None] = "0022"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Delete tags of the retired kinds. CASCADE handles related tables.
op.execute("DELETE FROM tag WHERE kind IN ('meta', 'rating')")
# 2. Drop the CHECK constraint that references the enum's literal
# values. Postgres can't resolve `kind = 'character'` across the
# type swap below — the literal would bind to the new tag_kind
# but the column is on tag_kind_old, producing
# "operator does not exist: tag_kind = tag_kind_old".
# (Operator-hit during the v26.05.26.5 deploy attempt; ck was
# originally added by alembic 0002.) Recreated post-swap.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
# 3. Drop the server default — ALTER COLUMN TYPE can't carry it
# across the type swap below.
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
# 4. Recreate the tag_kind enum without meta/rating.
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
# 5. Restore the server default.
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
# 6. Restore the CHECK constraint (now bound to the new tag_kind).
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
def downgrade() -> None:
# Add the values back to the enum so old code can boot. The deleted
# tag rows are gone permanently — no safe restore.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post', 'meta', 'rating'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
@@ -0,0 +1,80 @@
"""backfill post.post_title from description first-line — 2026-05-27
Revision ID: 0024
Revises: 0023
Create Date: 2026-05-27
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. FC's sidecar parser was leaving
post_title NULL for every SubscribeStar post since FC-3 shipped. The
parser fix (sidecar._first_line_text fallback) now synthesizes a title
at parse time; this migration applies the same logic retroactively to
existing rows.
Operator-flagged 2026-05-27 after inspecting
/mnt/Data/Patreon/Cheunart/subscribestar/ sidecars.
Idempotent: only touches rows where post_title IS NULL or empty AND
description IS NOT NULL. Re-running the migration is a no-op.
"""
from __future__ import annotations
import re
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0024"
down_revision: Union[str, None] = "0023"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_TAG_RE = re.compile(r"<[^>]+>")
_WS_RE = re.compile(r"\s+")
def _first_line_text(body: str, limit: int = 120) -> str | None:
"""Mirror of sidecar._first_line_text. Kept inline so the migration
doesn't carry a runtime import dependency from app code that may
have moved by the time the migration is replayed years from now."""
if not body:
return None
text_ = _TAG_RE.sub(" ", body)
text_ = text_.replace("\xa0", " ")
for line in text_.splitlines():
line = _WS_RE.sub(" ", line).strip()
if line:
if len(line) > limit:
return line[: limit - 1].rstrip() + ""
return line
return None
def upgrade() -> None:
bind = op.get_bind()
rows = bind.execute(
text(
"SELECT id, description FROM post "
"WHERE (post_title IS NULL OR post_title = '') "
"AND description IS NOT NULL AND description <> ''"
)
).fetchall()
updated = 0
for row in rows:
derived = _first_line_text(row.description)
if not derived:
continue
bind.execute(
text("UPDATE post SET post_title = :t WHERE id = :id"),
{"t": derived, "id": row.id},
)
updated += 1
print(f"0024: backfilled post_title on {updated} row(s)")
def downgrade() -> None:
# No safe restore — we can't tell which post_titles were derived vs
# genuinely present. Leave the column alone on rollback.
pass
@@ -0,0 +1,288 @@
"""sidecar-audit followup: correct external_post_id + post_url across all platforms
Revision ID: 0025
Revises: 0024
Create Date: 2026-05-27
Closes the operator-flagged 2026-05-27 sidecar audit findings. Three
data-correctness bugs across non-Patreon platforms had been silently
corrupting Posts since FC-3 shipped; the parser fix (sidecar.py, same
commit) addresses new imports. This migration cleans up existing rows.
Per-platform actions:
subscribestar — gallery-dl wrote the per-attachment id in `id` and
the actual post id in `post_id`. FC's parser picked `id`, so every
multi-image SubscribeStar post was fragmented into N Post rows.
1. For each SubscribeStar Post, read its sidecar (via the related
ImageRecord's on-disk path), pull `post_id`, overwrite
external_post_id and post_url.
2. Merge groups of Posts under one source that now share an
external_post_id (fragments of the same actual post). Same
ImageProvenance pre-delete + repoint dance as alembic 0022.
hentaifoundry — sidecars have NO `url` field; `src` is the image
URL. FC's parser stored post_url=NULL. Read each HF Post's sidecar
for `user` + `index`, derive the canonical /pictures/user/<u>/<i>
permalink. external_post_id (= `index`) was already correct.
discord — gallery-dl wrote the CDN attachment URL in `url`. FC's
parser stored that as post_url. Read each Discord Post's sidecar
for the server/channel/message triple, derive the proper
discord.com/channels/.../<message> permalink. external_post_id (=
`message_id`) was already correct.
pixiv — pure-SQL backfill: replace any `i.pximg.net`-style URL on
Post.post_url with the derived `/artworks/<id>` permalink. Pixiv
external_post_id (= `id`) was already correct; no sidecar IO
needed.
Idempotent: re-running on already-corrected data is a no-op (skips
rows whose derived value matches what's already stored).
Posts whose related ImageRecord paths don't resolve on disk (orphaned
filesystem state) are skipped with a count in the migration output —
those will be picked up by a future deep-scan.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0025"
down_revision: Union[str, None] = "0024"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# Mirror of sidecar._NUMBERING_PREFIX. Kept inline so the migration is
# self-contained (the operator's banked rule:
# reference_postgres_enum_swap_drop_checks.md says migrations shouldn't
# import from runtime app code).
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
def _find_sidecar(media_path: Path) -> Path | None:
"""gallery-dl writes the sidecar under the unprefixed stem
(`HOLLOW-ICHIGO.json`) while the media file gets a NN_ ordering
prefix (`01_HOLLOW-ICHIGO.png`). Try in order:
1. <stem>.json next to the media
2. <media>.json next to the media (full-name variant)
3. strip the NN_ prefix from the stem, then <stripped>.json
"""
if not media_path:
return None
cand = media_path.with_suffix(".json")
if cand.is_file():
return cand
cand = media_path.parent / f"{media_path.name}.json"
if cand.is_file():
return cand
m = _NUMBERING_PREFIX.match(media_path.stem)
if m:
cand = media_path.parent / f"{m.group(1)}.json"
if cand.is_file():
return cand
return None
def _str_id(v) -> str | None:
"""str() a JSON scalar id; reject bool (JSON booleans are ints in
Python's eyes but they aren't valid sidecar ids)."""
if isinstance(v, bool):
return None
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
def _str_field(v) -> str | None:
if isinstance(v, str) and v.strip():
return v.strip()
return None
def upgrade() -> None:
conn = op.get_bind()
# ── PART 1: Per-platform corrections requiring filesystem IO ─────
# SubscribeStar, HentaiFoundry, Discord all need fields from the
# sidecar to construct the right post_url. We walk each Post's
# related ImageRecord.path to find the sidecar, read it, derive,
# and update.
targets = conn.execute(text("""
SELECT p.id, p.external_post_id, p.post_url, s.platform
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform IN ('subscribestar', 'hentaifoundry', 'discord')
""")).fetchall()
stats: dict[str, dict[str, int]] = {
plat: {"read": 0, "updated": 0, "no_sidecar": 0}
for plat in ("subscribestar", "hentaifoundry", "discord")
}
for post_row in targets:
plat = post_row.platform
path = _first_attachment_path(conn, post_row.id)
if not path:
stats[plat]["no_sidecar"] += 1
continue
sidecar = _find_sidecar(Path(path))
if sidecar is None:
stats[plat]["no_sidecar"] += 1
continue
try:
data = json.loads(sidecar.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
stats[plat]["no_sidecar"] += 1
continue
stats[plat]["read"] += 1
new_epid = post_row.external_post_id
new_url = None
if plat == "subscribestar":
pid = _str_id(data.get("post_id"))
if pid:
new_epid = pid
new_url = f"https://www.subscribestar.com/posts/{pid}"
elif plat == "hentaifoundry":
user = _str_field(data.get("user")) or _str_field(data.get("artist"))
idx = _str_id(data.get("index"))
if user and idx:
new_url = f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
elif plat == "discord":
sid = _str_id(data.get("server_id"))
cid = _str_id(data.get("channel_id"))
mid = _str_id(data.get("message_id"))
if sid and cid and mid:
new_url = f"https://discord.com/channels/{sid}/{cid}/{mid}"
# Idempotent: skip if nothing changed.
if new_epid == post_row.external_post_id and new_url == post_row.post_url:
continue
conn.execute(
text("""
UPDATE post
SET external_post_id = :epid, post_url = :url
WHERE id = :id
"""),
{"epid": new_epid, "url": new_url, "id": post_row.id},
)
stats[plat]["updated"] += 1
for plat, s in stats.items():
print(
f"0025: {plat} — read {s['read']} sidecars, "
f"updated {s['updated']} Posts, "
f"{s['no_sidecar']} Posts had no resolvable sidecar"
)
# ── PART 2: Merge SubscribeStar fragments now sharing epid ───────
# After Part 1, each group of Posts under one source with the SAME
# new external_post_id is a fragment-set of the same actual post.
# Merge to one canonical row. Pre-handle the same ImageProvenance
# collision pattern as alembic 0022 (uq_image_provenance_image_post).
fragment_groups = conn.execute(text("""
SELECT p.source_id, p.external_post_id,
ARRAY_AGG(p.id ORDER BY p.id ASC) AS post_ids
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform = 'subscribestar'
AND p.external_post_id IS NOT NULL
GROUP BY p.source_id, p.external_post_id
HAVING COUNT(*) > 1
""")).fetchall()
merged = 0
for grp in fragment_groups:
post_ids = list(grp.post_ids)
keep_id, *drop_ids = post_ids
for drop_id in drop_ids:
# Pre-DELETE colliding ImageProvenance under drop_ that
# already exist under keep (alembic 0022 banked the pattern).
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE post_attachment SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
merged += 1
print(f"0025: subscribestar — merged {merged} duplicate Post fragments")
# ── PART 3: Pixiv post_url backfill (pure SQL) ───────────────────
# Pixiv's external_post_id is already correct (gallery-dl's `id` is
# the post id). Only post_url needs derivation: replace anything
# under i.pximg.net (the file URL) with the /artworks/<id> permalink.
pixiv_updated = conn.execute(text("""
UPDATE post p
SET post_url = 'https://www.pixiv.net/artworks/' || p.external_post_id
FROM source s
WHERE p.source_id = s.id
AND s.platform = 'pixiv'
AND p.external_post_id IS NOT NULL
AND (p.post_url IS NULL
OR p.post_url LIKE 'https://i.pximg.net/%'
OR p.post_url LIKE 'http://i.pximg.net/%')
""")).rowcount
print(f"0025: pixiv — backfilled post_url on {pixiv_updated} Posts")
def _first_attachment_path(conn, post_id: int) -> str | None:
"""Return any ImageRecord.path attached to this post (via
ImageProvenance). Lowest-id row keeps the migration deterministic
so re-running on the same DB picks the same sidecar."""
row = conn.execute(
text("""
SELECT ir.path
FROM image_provenance ip
JOIN image_record ir ON ir.id = ip.image_record_id
WHERE ip.post_id = :pid
ORDER BY ip.id ASC
LIMIT 1
"""),
{"pid": post_id},
).first()
return row[0] if row else None
def downgrade() -> None:
# Lossy: external_post_id values were overwritten with the correct
# post_id; original per-attachment ids weren't preserved. Post-merge
# also deleted drop rows. No safe restore. To roll back the schema
# invariant, fork from 0024 and re-run sidecar imports.
pass
@@ -0,0 +1,53 @@
"""import_task.recovery_count + refetched — poison-pill circuit breaker
Revision ID: 0026
Revises: 0025
Create Date: 2026-05-28
Backs the import-task resilience work (operator-flagged 2026-05-28):
- recovery_count: how many times recover_interrupted_tasks has
re-queued this row from a stuck 'processing' state. A row that
hard-crashes the worker (OOM / segfault on a corrupt or oversized
input) leaves no terminal flip, so the sweep re-queues it — and
without a cap it would loop forever, re-crashing the worker each
time. After MAX_RECOVERY_ATTEMPTS the sweep marks it 'failed' with a
diagnostic instead.
- refetched: whether a one-shot re-download has already been attempted
for this task's file. Bounds the Layer-2 re-fetch remediation to a
single attempt so source-side corruption doesn't loop.
Both default to 0 / false; additive, no backfill needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0026"
down_revision: Union[str, None] = "0025"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_task",
sa.Column(
"recovery_count", sa.Integer(), nullable=False,
server_default="0",
),
)
op.add_column(
"import_task",
sa.Column(
"refetched", sa.Boolean(), nullable=False,
server_default=sa.false(),
),
)
def downgrade() -> None:
op.drop_column("import_task", "refetched")
op.drop_column("import_task", "recovery_count")
@@ -0,0 +1,50 @@
"""drop migration_run — one-and-done GS/IR migration tooling removed
Revision ID: 0027
Revises: 0026
Create Date: 2026-05-29
The GS/IR migration tooling (services/migrators, /api/migrate, the
run_migration task, LegacyMigrationCard, and the MigrationRun model) was
removed after the migration cutover completed. This drops its now-orphaned
run-log table. Downgrade recreates the table (mirrors the old model) so the
migration is reversible.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0027"
down_revision: Union[str, None] = "0026"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_table("migration_run")
def downgrade() -> None:
op.create_table(
"migration_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=32), nullable=False),
sa.Column("status", sa.String(length=32), nullable=False),
sa.Column("dry_run", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"counts", JSONB(), nullable=False, server_default=sa.text("'{}'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"metadata", JSONB(), nullable=False, server_default=sa.text("'{}'::jsonb"),
),
)
op.create_index("ix_migration_run_kind", "migration_run", ["kind"])
op.create_index("ix_migration_run_status", "migration_run", ["status"])
@@ -0,0 +1,190 @@
"""collapse-sidecar-synthetic: repoint Posts/ImageProvenance/DownloadEvents
from `sidecar:<platform>:<slug>` synthetic Source anchors onto the real
Source for the same (artist, platform) when one exists, then delete the
synthetic.
Revision ID: 0028
Revises: 0027
Create Date: 2026-05-31
Background: alembic 0022 (2026-05-26) consolidated the old per-post-URL
Source rows into one canonical Source per (artist, platform). When NO
real campaign URL was salvageable among the candidates, it rewrote the
canonical row to url='sidecar:<platform>:<slug>' enabled=false as a
disabled anchor for any Posts already attached.
That was fine while it was the only Source for that artist+platform.
But: the unique constraint on Source is (artist_id, platform, url), not
(artist_id, platform). When the operator later added the real
subscription via the UI / extension / etc., a SECOND row landed —
the real one — with id > the synthetic. Both coexisted.
Two follow-on problems surfaced 2026-05-31:
1. The Subscriptions UI listed both rows. The synthetic was disabled
so the scheduler never polled it, but it looked like a phantom
subscription. (Fixed in same commit by SourceService.list filter.)
2. importer._source_for_sidecar picked Source by `ORDER BY id ASC
LIMIT 1`, so EVERY gallery-dl download since the real Source was
added attached its Post to the SYNTHETIC anchor, not the real
Source. (Fixed in same commit by preferring non-sidecar URLs.)
This migration is the data half of the cleanup: for every (artist,
platform) with both a synthetic AND a real Source, repoint the
synthetic's children (Posts, ImageProvenance, DownloadEvents) onto the
real Source and delete the synthetic. Reuses the same epid/provenance
collision dance from alembic 0022 because the same uniqueness
constraints fire row-by-row during bulk UPDATEs.
Lone synthetic anchors — those where no real Source for the same
(artist, platform) exists (e.g., filesystem-imported artist with no
subscription added) — are LEFT INTACT. They anchor real imported
content; deleting them would CASCADE-delete the Posts the operator
imported. The SourceService.list filter hides them from the UI; the
operator can delete them by hand if they want the underlying imports
gone.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0028"
down_revision: Union[str, None] = "0027"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Find (artist_id, platform) groups where BOTH a sidecar synthetic
# and at least one real Source exist.
groups = conn.execute(text("""
SELECT artist_id, platform
FROM source
GROUP BY artist_id, platform
HAVING bool_or(url LIKE 'sidecar:%')
AND bool_or(url NOT LIKE 'sidecar:%')
""")).fetchall()
for artist_id, platform in groups:
rows = conn.execute(
text("""
SELECT id, url FROM source
WHERE artist_id = :a AND platform = :p
ORDER BY id ASC
"""),
{"a": artist_id, "p": platform},
).fetchall()
synthetic_ids = [sid for sid, url in rows if url.startswith("sidecar:")]
real_rows = [(sid, url) for sid, url in rows if not url.startswith("sidecar:")]
if not synthetic_ids or not real_rows:
continue # belt+suspenders; the GROUP BY already filtered
# Canonical real: lowest-id non-sidecar Source.
canonical_id = real_rows[0][0]
# STEP A: PRE-merge Post collisions on (canonical, external_post_id).
# Mirror alembic 0022's pre-merge logic — when synth has Post X
# epid=N and real has Post Y epid=N, the bulk UPDATE below would
# trip uq_post_source_external_id row-by-row. Group all Posts
# under (canonical + synthetics) by epid; for any group >1,
# pick a keep (prefer one already under canonical, else lowest
# id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:synths)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
).fetchall()
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
canonical_side = [p for p in posts if p[1] == canonical_id]
keep_id = canonical_side[0][0] if canonical_side else posts[0][0]
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id already has provenance under keep —
# avoids tripping uq_image_provenance_image_post (0021)
# row-by-row during the repoint UPDATE.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP B: Bulk reparent the remaining Posts off the synthetics.
conn.execute(
text("""
UPDATE post SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP C: Reparent ImageProvenance.source_id (denormalized FK;
# no UNIQUE on source_id, safe bulk).
conn.execute(
text("""
UPDATE image_provenance SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP D: Reparent any DownloadEvent.source_id. Synthetics are
# enabled=false so the scheduler never created events for them;
# this is belt+suspenders for any rows planted by manual force
# or older code paths.
conn.execute(
text("""
UPDATE download_event SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP E: Drop the now-empty synthetics.
conn.execute(
text("DELETE FROM source WHERE id = ANY(:synths)"),
{"synths": synthetic_ids},
)
def downgrade() -> None:
# Lossy migration — synthetic Sources deleted, Posts repointed and
# potentially merged. No safe downgrade.
pass
@@ -0,0 +1,71 @@
"""drop artist + copyright ml thresholds; lower general default to 0.50
Revision ID: 0029
Revises: 0028
Create Date: 2026-06-01
Operator-flagged 2026-06-01: the view modal's Suggestions panel hides
most general-category predictions because the default threshold is
0.95. Lowering the default to 0.50 (matches character) so general
suggestions surface more aggressively; the value remains tunable in
Settings → ML.
Same change retires two ML suggestion categories whose Tag.kind
surfaces are unused:
- `artist`: retired in FC-2d-vii-c — artist identity is acquisition-
derived (image_record.artist_id), never ML-inferred. The threshold
column was a leftover from before that retirement.
- `copyright`: retired 2026-06-01 — the app uses `fandom` for the
franchise/copyright concept (per TagsView.vue's doc comment); no
Tag rows of kind=copyright exist, and the threshold column never
fed anything user-visible.
Both columns are dropped from ml_settings; the existing row's
suggestion_threshold_general value is bumped from 0.95 to 0.50 iff
it's still at the old default, so deployed installs pick up the new
UX without overriding any operator tuning.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0029"
down_revision: Union[str, None] = "0028"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Bump the general threshold for installs still at the old default.
op.execute(text(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.95"
))
op.drop_column("ml_settings", "suggestion_threshold_artist")
op.drop_column("ml_settings", "suggestion_threshold_copyright")
def downgrade() -> None:
# Restore the columns with their prior defaults. The bump from
# 0.95 → 0.50 isn't reversible without remembering whether the
# operator had explicitly set 0.95 (unlikely — that was just the
# default) so we leave the current general value as-is.
from sqlalchemy import Column, Float
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_artist",
Float, nullable=False, server_default="0.30",
),
)
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_copyright",
Float, nullable=False, server_default="0.50",
),
)
@@ -0,0 +1,145 @@
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
Revision ID: 0030
Revises: 0029
Create Date: 2026-06-01
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
with enabled=false) was technically correct but misled the operator
into thinking they had phantom subscriptions. The synthetics existed
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
content with no real subscription.
This migration makes the data model honest:
1. **Post gets a denormalized `artist_id` column** so artist filters
work without traversing `Post → Source.artist_id`. Backfilled from
the existing Source linkage, then NOT NULL'd.
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
NULL`. Deleting a Source detaches its Posts instead of destroying
imported content (semantically: subscription ends, archive stays).
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
semantic change.
4. **Sidecar synthetic Sources are deleted** — first NULL out the
FKs from Post + ImageProvenance pointing at them (so the implicit
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
so no events exist for them.
Uniqueness handling: the existing `uq_post_source_external_id`
(source_id, external_post_id) keeps working for source-bound Posts
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
it). A second partial unique index covers the NULL-source case on
(artist_id, external_post_id) so filesystem-imported posts still
dedupe within an artist.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
revision: str = "0030"
down_revision: Union[str, None] = "0029"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Step 1: add Post.artist_id, initially nullable for backfill.
# FK naming follows the Base.metadata naming_convention
# (fk_<table>_<column>_<referred_table>) — alembic 0001 set this up.
op.add_column(
"post",
sa.Column("artist_id", sa.Integer, nullable=True),
)
op.create_foreign_key(
"fk_post_artist_id_artist", "post", "artist",
["artist_id"], ["id"], ondelete="CASCADE",
)
# Step 2: backfill from Source.artist_id (every existing Post has a
# Source today, so every row gets populated).
conn.execute(text("""
UPDATE post p
SET artist_id = s.artist_id
FROM source s
WHERE p.source_id = s.id AND p.artist_id IS NULL
"""))
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
remaining = conn.execute(text(
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
)).scalar_one()
if remaining:
raise RuntimeError(
f"alembic 0030: {remaining} post rows have no resolvable "
f"artist_id after backfill. Investigate before continuing."
)
# Step 3: enforce NOT NULL + add index for artist-filter queries.
op.alter_column("post", "artist_id", nullable=False)
op.create_index("ix_post_artist_id", "post", ["artist_id"])
# Step 4: relax post.source_id + flip FK to SET NULL. The original FK
# name from alembic 0001 is `fk_post_source_id_source` per the
# NAMING_CONVENTION in models/base.py.
op.alter_column("post", "source_id", nullable=True)
op.drop_constraint("fk_post_source_id_source", "post", type_="foreignkey")
op.create_foreign_key(
"fk_post_source_id_source", "post", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
op.alter_column("image_provenance", "source_id", nullable=True)
op.drop_constraint(
"fk_image_provenance_source_id_source", "image_provenance",
type_="foreignkey",
)
op.create_foreign_key(
"fk_image_provenance_source_id_source", "image_provenance", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 6: partial unique index on (artist_id, external_post_id) for
# NULL-source Posts. The existing uq_post_source_external_id keeps
# guarding source-bound rows; NULL-source rows now dedupe within
# an artist.
op.execute(
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
"ON post (artist_id, external_post_id) "
"WHERE source_id IS NULL"
)
# Step 7: retire sidecar synthetic Sources. NULL out the references
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
# being explicit makes the intent clear). Then delete the synthetic
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
# with the source — synthetics have enabled=false so there shouldn't
# be any in practice.
conn.execute(text("""
UPDATE post
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
conn.execute(text("""
UPDATE image_provenance
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
deleted = conn.execute(text(
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
)).rowcount
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
def downgrade() -> None:
# Lossy migration — the deleted sidecar synthetics can't be
# restored from the orphan post.source_id / image_provenance.source_id
# values, and the partial unique index encodes a constraint that
# NULL-source Posts may now exist. No safe downgrade.
pass
@@ -0,0 +1,45 @@
"""source.backfill_runs_remaining: sticky deep-scan mode
Revision ID: 0031
Revises: 0030
Create Date: 2026-06-01
Tick vs backfill mode for subscription downloads. When
`backfill_runs_remaining > 0`, the next N download runs use
`skip: True` + 30-min timeout (walk full history). When 0, runs use
`skip: "exit:20"` + 14.5-min timeout (catch-up mode, exits early once
20 contiguous archived items are seen).
Operator-flagged 2026-06-01 (Knuxy run #38887): a creator with ~550
archived posts saturates the 870s catch-up timeout even when there is
no new content, because gallery-dl's default `skip: True` keeps walking.
Tick mode short-circuits that; backfill mode is the explicit opt-in for
deep history scans.
Default 0 (all existing subscriptions start in tick mode).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0031"
down_revision: Union[str, None] = "0030"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column(
"backfill_runs_remaining",
sa.Integer,
nullable=False,
server_default="0",
),
)
def downgrade() -> None:
op.drop_column("source", "backfill_runs_remaining")
@@ -0,0 +1,41 @@
"""source.error_type: surface ErrorType taxonomy in FailingSourcesCard
Revision ID: 0032
Revises: 0031
Create Date: 2026-06-02
Audit 2026-06-02: the backend computes 13 ErrorType categories (auth_error,
rate_limited, not_found, access_denied, validation_failed, etc.) and
stamps each one on DownloadEvent.metadata, but the Source row only carried
the free-text last_error. Operators couldn't bulk-triage failing sources
("all auth_error → rotate cookies, all rate_limited → just wait") without
opening Logs per row.
This column receives the last error_type from _update_source_health
and gets cleared on a successful run. Nullable + indexed so the failing-
sources rollup can filter/group cheaply.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0032"
down_revision: Union[str, None] = "0031"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column("error_type", sa.String(length=32), nullable=True),
)
op.create_index(
"ix_source_error_type", "source", ["error_type"],
)
def downgrade() -> None:
op.drop_index("ix_source_error_type", table_name="source")
op.drop_column("source", "error_type")
@@ -0,0 +1,48 @@
"""suggestion_threshold default 0.50 → 0.70
Revision ID: 0033
Revises: 0032
Create Date: 2026-06-02
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) is
too noisy in practice; raise to 0.70 for both suggestion categories.
Only conditionally updates singletons whose current value is still the
2026-06-01 default (0.50). Operators who deliberately tuned their row
to some other value (0.55, 0.65, 0.80, etc. via the Settings UI) keep
their pick — the migration only catches the unchanged-default case.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0033"
down_revision: Union[str, None] = "0032"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.70 "
"WHERE id = 1 AND suggestion_threshold_character = 0.50"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.70 "
"WHERE id = 1 AND suggestion_threshold_general = 0.50"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.50 "
"WHERE id = 1 AND suggestion_threshold_character = 0.70"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.70"
)
+38 -7
View File
@@ -3,13 +3,23 @@
import logging
from pathlib import Path
from quart import Quart
from quart import Quart, request
from .api import all_blueprints
from .config import get_config
from .frontend import frontend_bp
from .services.credential_crypto import CredentialCrypto
# Browser-extension origins. The FabledCurator extension fetches from
# moz-extension://<uuid>/ on Firefox and chrome-extension://<uuid>/ on
# Chromium-based browsers. Operator-flagged 2026-05-26: extension's
# 'Test connection' returned `NetworkError` because the X-Extension-Key
# header on /api/credentials triggers a CORS preflight that our routes
# don't handle. Whitelisting only these two schemes (not opening CORS
# up generally) lets the extension talk to a plain-HTTP self-hosted FC
# without weakening the no-CORS posture for normal browser usage.
_EXTENSION_ORIGIN_SCHEMES = ("moz-extension://", "chrome-extension://")
_CREDENTIAL_KEY_PATH = Path("/images/secrets/credential_key.b64")
@@ -23,18 +33,39 @@ def create_app() -> Quart:
app = Quart(__name__)
app.secret_key = cfg.secret_key
# FC-5: legacy IR ingest JSON can run to tens of MB (hundreds of
# thousands of image_tag_associations). Werkzeug's default form
# memory cap is 500KB; raise both ceilings so the multipart upload
# for /api/migrate/ir_ingest doesn't 413.
app.config["MAX_CONTENT_LENGTH"] = 1024 * 1024 * 1024 # 1 GB
app.config["MAX_FORM_MEMORY_SIZE"] = 1024 * 1024 * 1024 # 1 GB
for bp in all_blueprints():
app.register_blueprint(bp)
# Registered last so /api/* routes win over the SPA catch-all.
app.register_blueprint(frontend_bp)
@app.before_request
async def _extension_cors_preflight():
# Short-circuit OPTIONS preflight from the browser extension with a
# 204 + CORS headers (the after_request hook below adds them).
# Without this, OPTIONS lands on routes that only declared POST/GET
# methods and 405s before the after_request gets a chance.
if request.method != "OPTIONS":
return None
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
return "", 204
return None
@app.after_request
async def _extension_cors_headers(response):
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
response.headers["Access-Control-Allow-Origin"] = origin
response.headers["Access-Control-Allow-Methods"] = (
"GET, POST, PATCH, DELETE, OPTIONS"
)
response.headers["Access-Control-Allow-Headers"] = (
"Content-Type, X-Extension-Key"
)
response.headers["Access-Control-Max-Age"] = "86400"
return response
@app.after_serving
async def _dispose_db_engine() -> None:
from .extensions import dispose_engine
-2
View File
@@ -26,7 +26,6 @@ def all_blueprints() -> list[Blueprint]:
from .extension import extension_bp
from .gallery import gallery_bp
from .import_admin import import_admin_bp
from .migrate import migrate_bp
from .ml_admin import ml_admin_bp
from .platforms import platforms_bp
from .posts import posts_bp
@@ -54,7 +53,6 @@ def all_blueprints() -> list[Blueprint]:
admin_bp,
cleanup_bp,
import_admin_bp,
migrate_bp,
suggestions_bp,
allowlist_bp,
aliases_bp,
+16
View File
@@ -0,0 +1,16 @@
"""Shared API response helpers."""
from quart import jsonify
def error_response(
error: str, *, status: int = 400, detail: str | None = None, **extra,
):
"""JSON error body + HTTP status. `detail` is included only when given;
`extra` keys are merged into the body. Returns the (response, status)
tuple Quart expects. Imported as `_bad` by the blueprints."""
body = {"error": error}
if detail is not None:
body["detail"] = detail
body.update(extra)
return jsonify(body), status
+33 -9
View File
@@ -6,6 +6,7 @@ Five action surfaces:
DELETE /api/admin/tags/<int:tag_id> (Tier B)
POST /api/admin/tags/<int:dest_id>/merge (Tier B)
POST /api/admin/tags/prune-unused (Tier A)
POST /api/admin/tags/purge-legacy (Tier A)
GET /api/admin/tags/<int:tag_id>/usage-count (helper)
Tier-C ops take a dry_run body flag (returns projection inline,
@@ -23,16 +24,11 @@ from sqlalchemy import select
from ..extensions import get_session
from ..models import Artist
from ..services.cleanup_service import project_artist_cascade, project_bulk_image_delete
from ._responses import error_response as _bad
admin_bp = Blueprint("admin", __name__, url_prefix="/api/admin")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _bulk_image_confirm_token(image_ids: list[int]) -> str:
"""Stable 8-hex token derived from the sorted id list. Mutates
when the selection changes; stays the same across modal opens of
@@ -97,11 +93,19 @@ async def images_bulk_delete():
)
)
if dry_run:
return jsonify(projected)
sha8 = _bulk_image_confirm_token(image_ids)
expected = f"delete-images-{sha8}"
if dry_run:
# Hand the canonical Tier-C confirm token back with the
# projection so the frontend doesn't have to recompute SHA-256
# client-side via crypto.subtle (Secure-Context-gated,
# undefined on plain-HTTP origins per the homelab posture).
# Operator-flagged 2026-05-27.
projected["confirm_token"] = expected
return jsonify(projected)
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
@@ -198,3 +202,23 @@ async def tags_prune_unused():
)
)
return jsonify(result)
@admin_bp.route("/tags/purge-legacy", methods=["POST"])
async def tags_purge_legacy():
"""Tier-A: delete legacy IR-migration tags — archive/post/artist
kinds (e.g. `BlenderKnight:Hannah_BJ_Loops`) PLUS general tags with
a legacy name prefix (`source:*`, from IR's source kind that fell
back to general). dry-run preview returns per-kind + per-prefix
counts + a sample so the UI shows exactly what'll go before the
operator confirms with dry_run=false."""
from ..services.cleanup_service import purge_legacy_tags
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
)
return jsonify(result)
+8 -6
View File
@@ -31,18 +31,13 @@ from sqlalchemy import select
from ..extensions import get_session
from ..models import LibraryAuditRun
from ..services import cleanup_service
from ._responses import error_response as _bad
cleanup_bp = Blueprint("cleanup", __name__, url_prefix="/api/cleanup")
IMAGES_ROOT = Path("/images")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _min_dim_token(min_w: int, min_h: int) -> str:
# SHA-256 (not MD5) — Web Crypto's subtle.digest rejects MD5; both
# sides use SHA-256 truncated to 8 hex chars.
@@ -81,6 +76,13 @@ async def min_dim_preview():
s, min_width=min_w, min_height=min_h,
)
)
# Hand the canonical Tier-C delete token back with the preview so
# the frontend doesn't have to recompute SHA-256 client-side.
# window.crypto.subtle is Secure-Context-gated and undefined on
# plain-HTTP origins (homelab posture); without this the Delete
# button silently swallowed the TypeError and never opened the
# confirm modal. Operator-flagged 2026-05-27.
projection["confirm_token"] = _min_dim_token(min_w, min_h)
return jsonify(projection)
+54 -8
View File
@@ -20,6 +20,7 @@ from ..services.credential_service import (
UnknownPlatformError,
WrongAuthTypeError,
)
from ._responses import error_response as _bad
credentials_bp = Blueprint("credentials", __name__, url_prefix="/api/credentials")
@@ -38,14 +39,6 @@ def _get_crypto() -> CredentialCrypto:
return _crypto
def _bad(error: str, *, status: int = 400, detail: str | None = None, **extra):
body = {"error": error}
if detail is not None:
body["detail"] = detail
body.update(extra)
return jsonify(body), status
async def _ext_key_ok(session) -> bool:
"""If X-Extension-Key is supplied, it must match the stored value.
Missing header → True (browser path; accepted per homelab posture).
@@ -124,3 +117,56 @@ async def delete_credential(platform: str):
except LookupError:
return _bad("not_found", status=404)
return "", 204
@credentials_bp.route("/<platform>/verify", methods=["POST"])
async def verify_credential(platform: str):
"""Test the stored credential by running gallery-dl --simulate
against one of the platform's enabled sources. On success stamps
last_verified. Returns {valid: bool|null, reason, last_verified?}.
valid=null means "couldn't test" (no credential, or no enabled
source to point at)."""
from ..models import Artist, Source
from ..services.gallery_dl import GalleryDLService, SourceConfig
async with get_session() as session:
if not await _ext_key_ok(session):
return _bad("unauthorized", status=401)
svc = CredentialService(session, _get_crypto())
record = await svc.get(platform)
if record is None:
return jsonify({"valid": None, "reason": "No credential stored for this platform."})
# Pick an enabled source for this platform to point the probe at.
row = (await session.execute(
select(Source, Artist)
.join(Artist, Artist.id == Source.artist_id)
.where(Source.platform == platform, Source.enabled.is_(True))
.order_by(Source.id.asc())
)).first()
if row is None:
return jsonify({
"valid": None,
"reason": "No enabled source for this platform to verify against — add a subscription first.",
})
source, artist = row
cookies_path = await svc.get_cookies_path(platform)
auth_token = await svc.get_token(platform)
gdl = GalleryDLService(images_root=Path("/images"))
ok, message = await gdl.verify(
url=source.url,
artist_slug=artist.slug,
platform=platform,
source_config=SourceConfig.from_dict(source.config_overrides or {}),
cookies_path=str(cookies_path) if cookies_path else None,
auth_token=auth_token,
)
last_verified = None
if ok:
async with get_session() as session:
ts = await CredentialService(session, _get_crypto()).mark_verified(platform)
last_verified = ts.isoformat() if ts else None
return jsonify({"valid": ok, "reason": message, "last_verified": last_verified})
+97 -1
View File
@@ -5,8 +5,10 @@ status/source/artist. Returns slim records.
Detail view: full DownloadEvent including the metadata JSONB.
"""
from datetime import UTC, datetime, timedelta
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from sqlalchemy import func, select
from ..extensions import get_session
from ..models import Artist, DownloadEvent, Source
@@ -95,6 +97,83 @@ async def list_downloads():
return jsonify([_list_record(e, s, a) for e, s, a in rows])
@downloads_bp.route("/stats", methods=["GET"])
async def downloads_stats():
"""Status-grouped count over download_event for the dashboard stat chips.
`?window_hours=` (default 24) bounds by `started_at`. The full set of
statuses is always present in the response (zero for missing) so the
UI doesn't have to fill in defaults.
"""
try:
window_hours = int(request.args.get("window_hours", "24"))
except ValueError:
return jsonify({"error": "invalid_window_hours"}), 400
if window_hours < 1 or window_hours > 24 * 365:
return jsonify({"error": "invalid_window_hours"}), 400
since = datetime.now(UTC) - timedelta(hours=window_hours)
out = {"pending": 0, "running": 0, "ok": 0, "error": 0, "skipped": 0}
async with get_session() as session:
stmt = (
select(DownloadEvent.status, func.count())
.where(DownloadEvent.started_at >= since)
.group_by(DownloadEvent.status)
)
for status, n in (await session.execute(stmt)).all():
if status in out:
out[status] = int(n)
return jsonify(out)
@downloads_bp.route("/activity", methods=["GET"])
async def downloads_activity():
"""Hourly download-event counts over the last `?hours=` (default 24).
Returns a fixed-length, oldest-first bucket array so the UI can render
a sparkline directly. Bucketing is done in Python against UTC to dodge
session-timezone ambiguity in SQL date_trunc.
"""
try:
hours = int(request.args.get("hours", "24"))
except ValueError:
return jsonify({"error": "invalid_hours"}), 400
hours = max(1, min(168, hours))
now = datetime.now(UTC)
end = now.replace(minute=0, second=0, microsecond=0)
start = end - timedelta(hours=hours - 1)
buckets = [
{"hour": (start + timedelta(hours=i)).isoformat(),
"ok": 0, "error": 0, "other": 0, "total": 0}
for i in range(hours)
]
async with get_session() as session:
rows = (await session.execute(
select(DownloadEvent.started_at, DownloadEvent.status)
.where(DownloadEvent.started_at >= start)
)).all()
for started_at, status in rows:
if started_at is None:
continue
sa = started_at if started_at.tzinfo else started_at.replace(tzinfo=UTC)
idx = int((sa - start).total_seconds() // 3600)
if not (0 <= idx < hours):
continue
b = buckets[idx]
if status == "ok":
b["ok"] += 1
elif status == "error":
b["error"] += 1
else:
b["other"] += 1
b["total"] += 1
return jsonify({"hours": hours, "buckets": buckets})
@downloads_bp.route("/<int:event_id>", methods=["GET"])
async def get_download(event_id: int):
async with get_session() as session:
@@ -108,3 +187,20 @@ async def get_download(event_id: int):
return jsonify({"error": "not_found"}), 404
event, source, artist = row
return jsonify(_detail_record(event, source, artist))
@downloads_bp.route("/recover-stalled", methods=["POST"])
async def recover_stalled():
"""Trigger the recover_stalled_download_events sweep on demand.
The same sweep runs every 5 min via Beat (see celery_app.beat_schedule);
this endpoint exists so the operator can force-clear stuck pending/
running download_events from the Subscriptions → Downloads maintenance
menu without waiting for the next scheduled tick.
"""
# Local import: avoids registering maintenance tasks during blueprint
# import (Celery task discovery races with the API import otherwise).
from ..tasks.maintenance import recover_stalled_download_events
recover_stalled_download_events.delay()
return jsonify({"queued": True}), 202
+19 -6
View File
@@ -20,6 +20,7 @@ from ..services.extension_service import (
UnknownPlatformError,
)
from ..services.source_service import KNOWN_PLATFORMS
from ._responses import error_response as _bad
extension_bp = Blueprint("extension", __name__, url_prefix="/api/extension")
@@ -30,12 +31,6 @@ XPI_DIR = Path("/app/frontend/dist/extension")
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
async def _ext_key_required(session) -> bool:
"""Unlike /api/credentials (which accepts the browser path with no
header), quick-add-source writes server state and must be explicitly
@@ -62,6 +57,24 @@ def _sha256(path: Path) -> str:
return h.hexdigest()
@extension_bp.route("/probe", methods=["GET"])
async def probe_source():
"""Read-only resolution of a creator-page URL: tells the extension
whether this URL is already a Source, is for an Artist that exists
but with a different URL, is brand new, or doesn't match any known
platform pattern. Drives the content-script chip's color/copy
BEFORE the operator clicks, so the button can show 'already added'
without requiring an add-attempt."""
url = (request.args.get("url") or "").strip()
if not url:
return _bad("invalid_body", detail="url query parameter is required")
async with get_session() as session:
if not await _ext_key_required(session):
return _bad("unauthorized", status=401)
result = await ExtensionService(session).probe(url)
return jsonify(result)
@extension_bp.route("/quick-add-source", methods=["POST"])
async def quick_add_source():
body = await request.get_json(silent=True)
-1
View File
@@ -43,7 +43,6 @@ async def scroll():
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+59 -8
View File
@@ -35,10 +35,26 @@ async def trigger_scan():
@import_admin_bp.route("/status", methods=["GET"])
async def status():
async with get_session() as session:
# Active batch = running batch that still has outstanding work.
# Plain "most recent running" picks freshly-created scans that
# enqueued zero new files and hides the older batch that's
# actually being processed. Mirrors the EXISTS predicate
# /api/system/stats already uses (api/settings.py:145-160).
# Audit 2026-06-02 — /api/import/status and /api/system/stats
# used to disagree on the active-batch predicate; the UI banner
# said "Scanning…" indefinitely while the stats card said idle.
active = (
await session.execute(
select(ImportBatch)
.where(ImportBatch.status == "running")
.where(
ImportBatch.status == "running",
select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists(),
)
.order_by(ImportBatch.started_at.desc())
.limit(1)
)
@@ -114,18 +130,53 @@ async def retry_failed():
status="queued", error=None,
started_at=None, finished_at=None,
)
.returning(ImportTask.id)
.returning(ImportTask.id, ImportTask.task_type)
)
failed_ids = [row[0] for row in result.all()]
if not failed_ids:
failed = result.all()
if not failed:
return jsonify({"retried": 0})
await session.commit()
from ..tasks.import_file import import_media_file
for tid in failed_ids:
import_media_file.delay(tid)
from ..tasks.import_file import enqueue_import
for tid, task_type in failed:
enqueue_import(tid, task_type)
return jsonify({"retried": len(failed_ids)})
return jsonify({"retried": len(failed)})
@import_admin_bp.route("/tasks/<int:task_id>/refetch", methods=["POST"])
async def refetch_task(task_id: int):
"""Layer-2 one-shot re-download: delete the (corrupt) file behind a
failed import task and re-run its source's downloader to fetch a
fresh copy. Only works for files that resolve to an enabled,
real-URL subscription Source; filesystem-only imports return
no_source.
Returns one of: refetch_queued (+source_id) / no_source /
already_refetched / not_found / not_failed.
"""
async with get_session() as session:
result = await session.run_sync(_refetch_task_sync, task_id)
if result["status"] == "not_found":
return jsonify(result), 404
if result["status"] == "not_failed":
return jsonify(result), 400
return jsonify(result)
def _refetch_task_sync(session, task_id: int) -> dict:
from pathlib import Path
from ..models import ImportSettings
from ..services.refetch_service import attempt_refetch
task = session.get(ImportTask, task_id)
if task is None:
return {"status": "not_found"}
if task.status != "failed":
return {"status": "not_failed"}
settings = ImportSettings.load_sync(session)
return attempt_refetch(session, task, Path(settings.import_scan_path))
@import_admin_bp.route("/clear-stuck", methods=["POST"])
-112
View File
@@ -1,112 +0,0 @@
"""FC-5: /api/migrate — trigger and poll migration runs.
Ingest kinds (gs_ingest, ir_ingest) accept multipart/form-data with an
`export_file` field. All other kinds accept JSON. Backup + rollback
were retired in FC-3h (2026-05-24); use /api/system/backup/* instead.
"""
import json
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import MigrationRun
from ..tasks.migration import run_migration
migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate")
# 'backup' + 'rollback' retired 2026-05-24 (FC-3h); see /api/system/backup/*.
_VALID_KINDS = frozenset({
"gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "cleanup",
})
_INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"})
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _run_to_dict(run: MigrationRun) -> dict:
return {
"id": run.id,
"kind": run.kind,
"status": run.status,
"dry_run": run.dry_run,
"started_at": run.started_at.isoformat(),
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"counts": run.counts or {},
"error": run.error,
"metadata": run.metadata_ or {},
}
@migrate_bp.route("/<kind>", methods=["POST"])
async def create_run(kind: str):
if kind not in _VALID_KINDS:
return _bad("unknown_kind", detail=f"kind must be one of {sorted(_VALID_KINDS)}")
# Ingest kinds accept multipart/form-data; everything else takes JSON.
if kind in _INGEST_KINDS:
form = await request.form
files = await request.files
if "export_file" not in files:
return _bad("missing_export_file", detail="multipart export_file required")
export_file = files["export_file"]
try:
raw = export_file.read()
data = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
return _bad("invalid_export_file", detail=str(exc))
dry_run = str(form.get("dry_run", "false")).lower() in ("true", "1", "yes")
params: dict = {"data": data, "dry_run": dry_run}
else:
body = await request.get_json()
if body is None:
body = {}
if not isinstance(body, dict):
return _bad("invalid_body")
dry_run = bool(body.get("dry_run", False))
params = dict(body)
async with get_session() as session:
run = MigrationRun(kind=kind, status="pending", dry_run=dry_run)
session.add(run)
await session.commit()
await session.refresh(run)
run_id = run.id
run_migration.delay(run_id, kind, params)
return jsonify({"run_id": run_id, "status": "pending"}), 202
@migrate_bp.route("/runs/<int:run_id>", methods=["GET"])
async def get_run(run_id: int):
async with get_session() as session:
run = (await session.execute(
select(MigrationRun).where(MigrationRun.id == run_id)
)).scalar_one_or_none()
if run is None:
return _bad("not_found", status=404)
return jsonify(_run_to_dict(run))
@migrate_bp.route("/runs", methods=["GET"])
async def list_runs():
try:
limit = int(request.args.get("limit", "10"))
except ValueError:
return _bad("invalid_limit")
if limit < 1 or limit > 100:
return _bad("invalid_limit")
async with get_session() as session:
rows = (await session.execute(
select(MigrationRun)
.order_by(MigrationRun.id.desc())
.limit(limit)
)).scalars().all()
return jsonify([_run_to_dict(r) for r in rows])
-4
View File
@@ -9,9 +9,7 @@ ml_admin_bp = Blueprint("ml_admin", __name__, url_prefix="/api/ml")
_EDITABLE = (
"suggestion_threshold_artist",
"suggestion_threshold_character",
"suggestion_threshold_copyright",
"suggestion_threshold_general",
"centroid_similarity_threshold",
"min_reference_images",
@@ -28,9 +26,7 @@ async def get_settings():
).scalar_one()
return jsonify(
{
"suggestion_threshold_artist": s.suggestion_threshold_artist,
"suggestion_threshold_character": s.suggestion_threshold_character,
"suggestion_threshold_copyright": s.suggestion_threshold_copyright,
"suggestion_threshold_general": s.suggestion_threshold_general,
"centroid_similarity_threshold": s.centroid_similarity_threshold,
"min_reference_images": s.min_reference_images,
+25 -11
View File
@@ -5,18 +5,11 @@ from quart import Blueprint, jsonify, request
from ..extensions import get_session
from ..services.post_feed_service import PostFeedService
from ..services.source_service import KNOWN_PLATFORMS
from ._responses import error_response as _bad
posts_bp = Blueprint("posts", __name__, url_prefix="/api/posts")
def _bad(error: str, *, status: int = 400, detail: str | None = None, **extra):
body = {"error": error}
if detail is not None:
body["detail"] = detail
body.update(extra)
return jsonify(body), status
@posts_bp.route("", methods=["GET"])
async def list_posts():
args = request.args
@@ -25,6 +18,8 @@ async def list_posts():
artist_id_raw = args.get("artist_id")
platform = args.get("platform") or None
limit_raw = args.get("limit", "24")
direction = args.get("direction", "older")
around_raw = args.get("around")
try:
limit = int(limit_raw)
@@ -33,6 +28,16 @@ async def list_posts():
if limit < 1 or limit > 100:
return _bad("invalid_limit", detail="limit must be between 1 and 100")
if direction not in ("older", "newer"):
return _bad("invalid_direction", detail="direction must be 'older' or 'newer'")
around_id = None
if around_raw is not None:
try:
around_id = int(around_raw)
except ValueError:
return _bad("invalid_around", detail="around must be an integer post id")
artist_id = None
if artist_id_raw is not None:
try:
@@ -47,11 +52,20 @@ async def list_posts():
)
async with get_session() as session:
try:
page = await PostFeedService(session).scroll(
cursor=cursor, artist_id=artist_id,
svc = PostFeedService(session)
if around_id is not None:
result = await svc.around(
post_id=around_id, artist_id=artist_id,
platform=platform, limit=limit,
)
if result is None:
return _bad("not_found", status=404, detail=f"post id={around_id}")
return jsonify(result)
try:
page = await svc.scroll(
cursor=cursor, artist_id=artist_id,
platform=platform, limit=limit, direction=direction,
)
except ValueError as exc:
# Service raises ValueError for malformed cursors only;
# limit bounds are validated above.
+2 -6
View File
@@ -31,9 +31,7 @@ _EDITABLE_FIELDS = (
@settings_bp.route("/settings/import", methods=["GET"])
async def get_import_settings():
async with get_session() as session:
row = (
await session.execute(select(ImportSettings).where(ImportSettings.id == 1))
).scalar_one()
row = await ImportSettings.load(session)
return jsonify({
"min_width": row.min_width,
"min_height": row.min_height,
@@ -99,9 +97,7 @@ async def update_import_settings():
return _bad_int("download_failure_warning_threshold", 1, 100)
async with get_session() as session:
row = (
await session.execute(select(ImportSettings).where(ImportSettings.id == 1))
).scalar_one()
row = await ImportSettings.load(session)
for field in _EDITABLE_FIELDS:
if field in body:
setattr(row, field, body[field])
+59 -10
View File
@@ -5,6 +5,7 @@ from sqlalchemy import select
from ..extensions import get_session
from ..models import DownloadEvent, Source
from ..services.scheduler_service import active_platform_cooldowns, scheduler_status
from ..services.source_service import (
KNOWN_PLATFORMS,
ArtistNotFoundError,
@@ -14,18 +15,11 @@ from ..services.source_service import (
SourceService,
UnknownPlatformError,
)
from ._responses import error_response as _bad
sources_bp = Blueprint("sources", __name__, url_prefix="/api/sources")
def _bad(error: str, *, status: int = 400, detail: str | None = None, **extra):
body = {"error": error}
if detail is not None:
body["detail"] = detail
body.update(extra)
return jsonify(body), status
@sources_bp.route("", methods=["GET"])
async def list_sources():
artist_id_raw = request.args.get("artist_id")
@@ -35,11 +29,19 @@ async def list_sources():
artist_id = int(artist_id_raw)
except ValueError:
return _bad("invalid_artist_id", detail="artist_id must be an integer")
failing = request.args.get("failing", "").lower() in ("1", "true", "yes")
async with get_session() as session:
records = await SourceService(session).list(artist_id=artist_id)
records = await SourceService(session).list(artist_id=artist_id, failing=failing)
return jsonify([r.to_dict() for r in records])
@sources_bp.route("/schedule-status", methods=["GET"])
async def schedule_status():
"""FC-dashboards: scheduler health for the Subscriptions hub."""
async with get_session() as session:
return jsonify(await scheduler_status(session))
@sources_bp.route("/<int:source_id>", methods=["GET"])
async def get_source(source_id: int):
async with get_session() as session:
@@ -118,12 +120,46 @@ async def delete_source(source_id: int):
return "", 204
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
async def set_backfill(source_id: int):
"""Plan #544: arm a source for backfill mode for the next N download
runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
source dict. While backfill_runs_remaining > 0, downloads use
gallery-dl's full-walk config (skip: True + 30-min timeout) instead
of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
payload = await request.get_json(silent=True) or {}
runs = payload.get("runs", 3)
try:
runs = int(runs)
except (TypeError, ValueError):
return _bad("invalid_runs", detail="runs must be an integer")
async with get_session() as session:
try:
record = await SourceService(session).set_backfill_runs(
source_id, runs,
)
except LookupError:
return _bad("not_found", status=404)
except ValueError as exc:
return _bad("invalid_runs", detail=str(exc))
return jsonify(record.to_dict())
@sources_bp.route("/<int:source_id>/check", methods=["POST"])
async def check_source(source_id: int):
"""FC-3c: enqueue a download for this source.
Returns 202 with the new DownloadEvent id. If a pending/running
event already exists for this source, returns 409 with that id."""
event already exists for this source, returns 409 with that id. If
the source's platform is currently in a rate-limit cooldown, returns
**202 with `{status: "deferred", cooldown_until, platform}`** and
does NOT create an event or dispatch — the bulk retry path uses this
to avoid bowling N sources right back into the rate limit the
cooldown is preventing. Single-click "retry this one source" passes
`?force=true` to override the cooldown (operator-explicit, useful
for rapid auth-fix testing). The in-flight guard always applies.
"""
force = (request.args.get("force") or "").lower() in ("1", "true", "yes")
async with get_session() as session:
source = (await session.execute(
select(Source).where(Source.id == source_id)
@@ -133,6 +169,19 @@ async def check_source(source_id: int):
if not source.enabled:
return _bad("source_disabled", detail="enable the source first")
# Cooldown gate (unless explicitly overridden). Checked before the
# in-flight guard because a deferred retry doesn't need to create
# or check for an event at all.
if not force:
cooldowns = await active_platform_cooldowns(session)
expires_at = cooldowns.get(source.platform)
if expires_at is not None:
return jsonify({
"status": "deferred",
"platform": source.platform,
"cooldown_until": expires_at.isoformat(),
}), 202
in_flight = (await session.execute(
select(DownloadEvent.id).where(
DownloadEvent.source_id == source_id,
+40 -5
View File
@@ -20,6 +20,7 @@ from sqlalchemy import desc, func, select
from ..config import get_config
from ..extensions import get_session
from ..models import TaskRun
from ..services.scheduler_service import scheduler_status
system_activity_bp = Blueprint(
"system_activity", __name__, url_prefix="/api/system/activity",
@@ -81,17 +82,22 @@ def _read_workers_sync() -> dict:
}
async def _queues_cached() -> dict:
"""Per-queue Redis LLEN, cached 2s. Shared by /queues and /summary."""
now = time.time()
if _QUEUE_CACHE["data"] is None or (now - _QUEUE_CACHE["ts"]) > _QUEUE_CACHE_TTL:
_QUEUE_CACHE["data"] = await asyncio.to_thread(_read_queues_sync)
_QUEUE_CACHE["ts"] = now
return _QUEUE_CACHE["data"]
@system_activity_bp.route("/queues", methods=["GET"])
async def get_queues():
"""Per-queue Redis LLEN. Cached 2s.
Response: {queues: {name: depth_or_null}, fetched_at: iso8601}
"""
now = time.time()
if _QUEUE_CACHE["data"] is None or (now - _QUEUE_CACHE["ts"]) > _QUEUE_CACHE_TTL:
_QUEUE_CACHE["data"] = await asyncio.to_thread(_read_queues_sync)
_QUEUE_CACHE["ts"] = now
return jsonify(_QUEUE_CACHE["data"])
return jsonify(await _queues_cached())
@system_activity_bp.route("/workers", methods=["GET"])
@@ -107,6 +113,35 @@ async def get_workers():
return jsonify(_WORKER_CACHE["data"])
@system_activity_bp.route("/summary", methods=["GET"])
async def get_summary():
"""One-call rollup for the always-on TopNav pipeline indicator:
scheduler health, per-queue pending depths, currently-running count, and
recent (24h) failure count. Cheap — cached queue LLENs + two TaskRun
counts — so it's safe to poll app-wide."""
queues_data = await _queues_cached()
depths = queues_data.get("queues", {})
queued_total = sum(v for v in depths.values() if isinstance(v, int))
since = datetime.now(UTC) - timedelta(hours=24)
async with get_session() as session:
scheduler = await scheduler_status(session)
running = (await session.execute(
select(func.count(TaskRun.id)).where(TaskRun.status == "running")
)).scalar_one()
failing = (await session.execute(
select(func.count(TaskRun.id))
.where(TaskRun.status.in_(["error", "timeout"]))
.where(TaskRun.finished_at >= since)
)).scalar_one()
return jsonify({
"scheduler": scheduler,
"queues": depths,
"queued_total": queued_total,
"running": int(running),
"failing": int(failing),
})
@system_activity_bp.route("/runs", methods=["GET"])
async def list_runs():
"""Paginated task_run history. Query params:
+3 -12
View File
@@ -14,6 +14,7 @@ from sqlalchemy import desc, select
from ..extensions import get_session
from ..models import BackupRun, ImportSettings
from ._responses import error_response as _bad
system_backup_bp = Blueprint(
"system_backup", __name__, url_prefix="/api/system/backup",
@@ -29,12 +30,6 @@ _BACKUP_SETTINGS_FIELDS = (
)
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _row_to_dict(r: BackupRun) -> dict:
return {
"id": r.id,
@@ -232,9 +227,7 @@ async def delete_run(run_id: int):
@system_backup_bp.route("/settings", methods=["GET"])
async def get_settings():
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
row = await ImportSettings.load(session)
return jsonify({
"backup_db_nightly_enabled": row.backup_db_nightly_enabled,
"backup_db_nightly_hour_utc": row.backup_db_nightly_hour_utc,
@@ -254,9 +247,7 @@ async def patch_settings():
return err
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
row = await ImportSettings.load(session)
for field in _BACKUP_SETTINGS_FIELDS:
if field in body:
setattr(row, field, body[field])
+38 -5
View File
@@ -15,6 +15,7 @@ from ..services.tag_service import (
TagService,
TagValidationError,
)
from ..utils.tag_prefix import parse_kind_prefix
tags_bp = Blueprint("tags", __name__, url_prefix="/api")
@@ -105,13 +106,39 @@ async def directory():
@tags_bp.route("/tags", methods=["POST"])
async def create_tag():
"""Create a tag. Two input shapes accepted:
1. Explicit: {name, kind, fandom_id?} — caller already split, kind wins.
2. IR-suffix: {name} where name = "kind:Name" (e.g. "artist:Eric").
The server runs parse_kind_prefix(name) to derive kind; the colon
and prefix are stripped from the stored tag name. If no recognized
prefix is present, the kind defaults to `general`.
Explicit kind ALWAYS wins (backward-compat for existing callers).
"""
body = await request.get_json()
if not body or "name" not in body or "kind" not in body:
return jsonify({"error": "name and kind required"}), 400
if not body or "name" not in body:
return jsonify({"error": "name required"}), 400
name = body["name"]
kind = _coerce_kind(body["kind"])
if kind is None:
return jsonify({"error": f"invalid kind {body['kind']!r}"}), 400
explicit_kind_raw = body.get("kind")
if explicit_kind_raw is not None:
# Caller provided kind — honor it; don't re-parse.
kind = _coerce_kind(explicit_kind_raw)
if kind is None:
return jsonify({"error": f"invalid kind {explicit_kind_raw!r}"}), 400
else:
# IR-style: parse "kind:Name" from the raw name.
parsed_kind, parsed_name = parse_kind_prefix(name)
if parsed_kind is not None:
name = parsed_name
kind = _coerce_kind(parsed_kind)
# parse_kind_prefix only returns kinds from KNOWN_KINDS which
# are all valid TagKind members, so _coerce_kind can't return
# None here — but defensive.
if kind is None:
return jsonify({"error": f"invalid kind {parsed_kind!r}"}), 400
else:
kind = TagKind.general
fandom_id = body.get("fandom_id")
async with get_session() as session:
@@ -218,6 +245,12 @@ async def merge_tag(source_id: int):
from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=result.target_id)
# Tag merge invalidates the target's centroid (the merged-in source
# tag's images now contribute to it). Daily list_drifted catches it
# within 24h, but eager recompute closes the suggestion-quality dip
# in the meantime. Audit 2026-06-02.
from ..tasks.ml import recompute_centroid
recompute_centroid.delay(result.target_id)
return jsonify(
{
"target": {
+18 -3
View File
@@ -1,5 +1,7 @@
"""Thumbnail admin API: backfill trigger."""
import asyncio
from quart import Blueprint, jsonify
thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@@ -7,7 +9,20 @@ thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@thumbnails_bp.route("/backfill", methods=["POST"])
async def trigger_backfill():
from ..tasks.thumbnail import backfill_thumbnails
"""Run the backfill scan synchronously, return the counts. The actual
thumbnail generation work is still off-loaded to the thumbnail Celery
queue via `generate_thumbnail.delay()` per missing row — so this
handler is fast even on a 100k-image library (a scan is just SELECT
id, thumbnail_path + a file.stat() per row, no heavy work).
r = backfill_thumbnails.delay()
return jsonify({"celery_task_id": r.id}), 202
Operator-flagged 2026-06-01: the previous fire-and-forget shape
returned `{celery_task_id}` only, so the admin UI had no idea whether
backfill found 0 or 5000 candidates — \"found nothing\" was
indistinguishable from \"the worker isn't picking up the task.\""""
from ..tasks.thumbnail import _run_backfill_scan
# Sync scan inside an executor so we don't block the event loop.
counts = await asyncio.get_running_loop().run_in_executor(
None, _run_backfill_scan,
)
return jsonify(counts), 200
+39 -2
View File
@@ -28,7 +28,6 @@ def make_celery() -> Celery:
"backend.app.tasks.import_file",
"backend.app.tasks.thumbnail",
"backend.app.tasks.maintenance",
"backend.app.tasks.migration",
"backend.app.tasks.ml",
"backend.app.tasks.download",
"backend.app.tasks.backup",
@@ -45,7 +44,6 @@ def make_celery() -> Celery:
"backend.app.tasks.download.*": {"queue": "download"},
"backend.app.tasks.scan.*": {"queue": "scan"},
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
"backend.app.tasks.migration.*": {"queue": "maintenance"},
"backend.app.tasks.backup.*": {"queue": "maintenance"},
"backend.app.tasks.admin.*": {"queue": "maintenance"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance"},
@@ -87,6 +85,10 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.cleanup_old_download_events",
"schedule": 86400.0, # daily
},
"recover-stalled-download-events": {
"task": "backend.app.tasks.maintenance.recover_stalled_download_events",
"schedule": 300.0, # every 5 min, matches recover-interrupted-tasks
},
"recover-stalled-task-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_task_runs",
"schedule": 300.0, # every 5 min, matches recover-interrupted-tasks
@@ -103,6 +105,41 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
# Audit 2026-06-02 — three new per-entity recovery sweeps.
# Each runs every 5 min like the other recover_stalled_*
# sweeps; each is a no-op when nothing is stuck.
"recover-stalled-backup-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_backup_runs",
"schedule": 300.0,
},
"recover-stalled-library-audit-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
"schedule": 300.0,
},
"recover-stalled-import-batches": {
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
"schedule": 300.0,
},
# Audit 2026-06-02 — daily retention for two entities
# whose terminal rows otherwise accumulate forever.
"prune-library-audit-runs": {
"task": "backend.app.tasks.maintenance.prune_library_audit_runs",
"schedule": 86400.0,
},
"prune-import-batches": {
"task": "backend.app.tasks.maintenance.prune_import_batches",
"schedule": 86400.0,
},
# Audit 2026-06-02 — backfill_thumbnails's docstring claimed
# "periodic Beat" but the entry was never registered, so the
# library got no self-healing thumbnail repair; only the
# manual admin-UI button fired it. Daily cadence is gentle
# (the task is idempotent and only enqueues regen for rows
# whose stored thumbnails are missing or corrupt).
"backfill-thumbnails-daily": {
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
"schedule": 86400.0,
},
},
timezone="UTC",
)
+11 -2
View File
@@ -54,7 +54,14 @@ _INT32_MIN = -2_147_483_648
def _queue_for(task) -> str:
"""Reverse the task→queue routing from celery_app.task_routes.
Keep in sync if task_routes is reordered."""
Keep in sync if task_routes is reordered.
Audit 2026-06-02: backup/admin/library_audit prefixes were
missing here even though task_routes sent all three to
'maintenance'. The TaskRun.queue column then lied for those
rows (claimed 'default') so per-queue dashboard filters and
per-queue threshold overrides silently missed them.
"""
name = getattr(task, "name", "") or ""
if name.startswith("backend.app.tasks.import_file."):
return "import"
@@ -68,7 +75,9 @@ def _queue_for(task) -> str:
return "scan"
if name.startswith((
"backend.app.tasks.maintenance.",
"backend.app.tasks.migration.",
"backend.app.tasks.backup.",
"backend.app.tasks.admin.",
"backend.app.tasks.library_audit.",
)):
return "maintenance"
return "default"
-2
View File
@@ -12,7 +12,6 @@ from .import_batch import ImportBatch
from .import_settings import ImportSettings
from .import_task import ImportTask
from .library_audit_run import LibraryAuditRun
from .migration_run import MigrationRun
from .ml_settings import MLSettings
from .post import Post
from .post_attachment import PostAttachment
@@ -46,7 +45,6 @@ __all__ = [
"ImportSettings",
"LibraryAuditRun",
"MLSettings",
"MigrationRun",
"TagAlias",
"TagAllowlist",
"TagReferenceEmbedding",
+6 -2
View File
@@ -34,8 +34,12 @@ class ImageProvenance(Base):
post_id: Mapped[int] = mapped_column(
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
# Nullable since alembic 0030 — provenance rows for filesystem-imported
# content with no subscription have NULL source_id. FK ondelete SET
# NULL so deleting a Source detaches its provenance rows instead of
# destroying the linkage between image and post.
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
captured_at: Mapped[datetime] = mapped_column(
+11 -1
View File
@@ -4,7 +4,7 @@ Enforced as a single row via a CHECK (id = 1) constraint. The application
always SELECTs id=1 and never inserts/deletes after the initial migration.
"""
from sqlalchemy import Boolean, CheckConstraint, Float, Integer, Text
from sqlalchemy import Boolean, CheckConstraint, Float, Integer, Text, select
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
@@ -63,3 +63,13 @@ class ImportSettings(Base):
backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
@classmethod
async def load(cls, session) -> ImportSettings:
"""The singleton settings row (id=1), via an async session."""
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
@classmethod
def load_sync(cls, session) -> ImportSettings:
"""The singleton settings row (id=1), via a sync session."""
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
+17 -1
View File
@@ -8,7 +8,16 @@ been processing longer than the stuck-task threshold.
from datetime import datetime
from sqlalchemy import BigInteger, DateTime, ForeignKey, Integer, String, Text, func
from sqlalchemy import (
BigInteger,
Boolean,
DateTime,
ForeignKey,
Integer,
String,
Text,
func,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base
@@ -26,6 +35,13 @@ class ImportTask(Base):
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True)
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks
# how many times the stuck-task sweep has re-queued this row; after
# the cap it's failed with a diagnostic instead of looping. refetched
# bounds the one-shot re-download remediation to a single attempt.
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
result_image_id: Mapped[int | None] = mapped_column(
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
)
-37
View File
@@ -1,37 +0,0 @@
"""MigrationRun — tracks each FC-5 migration invocation (backup/gs/ir/etc).
kind/status are String(32) not Postgres ENUM so adding kinds later
doesn't need a schema migration. The API layer validates values.
"""
from datetime import datetime
import sqlalchemy as sa
from sqlalchemy import Boolean, DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class MigrationRun(Base):
__tablename__ = "migration_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
status: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(),
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
counts: Mapped[dict] = mapped_column(
JSONB, nullable=False, default=dict, server_default=sa.text("'{}'::jsonb"),
)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
metadata_: Mapped[dict] = mapped_column(
"metadata", JSONB, nullable=False, default=dict,
server_default=sa.text("'{}'::jsonb"),
)
+6 -8
View File
@@ -15,17 +15,15 @@ class MLSettings(Base):
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
suggestion_threshold_artist: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30
)
suggestion_threshold_character: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
)
suggestion_threshold_copyright: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50
Float, nullable=False, default=0.70
)
# Default raised 0.50 → 0.70 on 2026-06-02 — operator-flagged 0.50
# surfaced too many low-confidence picks; 0.70 keeps the rail
# signal-rich while still surfacing more than the original 0.95
# which hid almost everything. Operator-tunable via Settings → ML.
suggestion_threshold_general: Mapped[float] = mapped_column(
Float, nullable=False, default=0.95
Float, nullable=False, default=0.70
)
centroid_similarity_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.55
+20 -4
View File
@@ -1,6 +1,9 @@
"""Post — provenance anchor for content downloaded from a Source.
"""Post — provenance anchor for one creator post (may contain many images).
A Post is one creator post; it may contain many images/videos.
`source_id` is nullable since alembic 0030 — filesystem-imported posts
with no live subscription have NULL source_id. `artist_id` is the
denormalized always-present link to the creator (added in 0030 so
artist-filter queries don't depend on the Source detour).
"""
from datetime import datetime
@@ -14,12 +17,25 @@ from .base import Base
class Post(Base):
__tablename__ = "post"
__table_args__ = (
# Source-bound dedup. Postgres treats NULL != NULL so rows
# with source_id IS NULL aren't deduped by this constraint;
# the partial unique index `uq_post_artist_external_id_null_source`
# (created in alembic 0030) covers that case via
# (artist_id, external_post_id).
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
# Denormalized; always equals source.artist_id when source_id is set
# (the importer is responsible for keeping them consistent on insert).
# Filter queries (artist detail, artist-scoped posts feed) use this
# directly instead of joining through Source.
artist_id: Mapped[int] = mapped_column(
ForeignKey("artist.id", ondelete="CASCADE"),
nullable=False, index=True,
)
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
+14
View File
@@ -26,7 +26,21 @@ class Source(Base):
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
# alembic 0032: last ErrorType category (auth_error, rate_limited,
# not_found, ...). Lets FailingSourcesCard surface the taxonomy as
# a colored chip so operators can bulk-triage by error class. Set
# by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# alembic 0031: sticky deep-scan budget. When > 0, the next N download
# runs use gallery-dl's full-walk config (skip: True + 1800s timeout);
# when 0, runs use tick mode (skip: "exit:20" + 870s, exits early once
# 20 contiguous archived items are seen). Auto-decrements per run, with
# an auto-reset to 0 on clean exit + zero downloads (queue drained).
backfill_runs_remaining: Mapped[int] = mapped_column(
Integer, nullable=False, default=0, server_default="0",
)
artist = relationship("Artist", back_populates="sources")
+4 -2
View File
@@ -35,8 +35,10 @@ class TagKind(StrEnum):
series = "series"
archive = "archive"
post = "post"
meta = "meta"
rating = "rating"
# `meta` and `rating` retired by operator 2026-05-26 (alembic 0023).
# `artist` retired in FC-2d-vii-c — artists are first-class entities
# via Artist/Source rows now, not tags — but the enum value stays
# to keep historic tag rows queryable.
image_tag = Table(
@@ -127,17 +127,20 @@ class ArtistDirectoryService:
ImageRecord.artist_id.label("artist_id"),
ImageRecord.sha256.label("sha256"),
ImageRecord.mime.label("mime"),
ImageRecord.thumbnail_path.label("thumbnail_path"),
rn,
)
.where(ImageRecord.artist_id.in_(artist_ids))
.subquery()
)
stmt = (
select(sub.c.artist_id, sub.c.sha256, sub.c.mime)
select(
sub.c.artist_id, sub.c.sha256, sub.c.mime, sub.c.thumbnail_path,
)
.where(sub.c.rn <= _PREVIEW_COUNT)
.order_by(sub.c.artist_id, sub.c.rn)
)
out: dict[int, list[str]] = {}
for aid, sha, mime in (await self.session.execute(stmt)).all():
out.setdefault(aid, []).append(thumbnail_url(sha, mime))
for aid, sha, mime, tp in (await self.session.execute(stmt)).all():
out.setdefault(aid, []).append(thumbnail_url(tp, sha, mime))
return out
+25 -18
View File
@@ -58,13 +58,17 @@ class ArtistService:
)
).scalar_one()
# Posts under this artist that have at least one image attached.
# Use Post.artist_id (alembic 0030) for the artist filter; keep
# the ImageProvenance JOIN so date bounds reflect only image-
# bearing posts (matches the original semantic). NULL-source
# posts now surface too.
date_row = (
await self.session.execute(
select(func.min(Post.post_date), func.max(Post.post_date))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).first()
dmin, dmax = date_row if date_row else (None, None)
@@ -98,14 +102,14 @@ class ArtistService:
)
).all()
# Same Post.artist_id direct filter — counts NULL-source posts too.
month = func.date_trunc("month", Post.post_date).label("m")
activity = (
await self.session.execute(
select(month, func.count(func.distinct(ImageProvenance.image_record_id)))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(and_(Source.artist_id == aid, Post.post_date.isnot(None)))
.where(and_(Post.artist_id == aid, Post.post_date.isnot(None)))
.group_by(month)
.order_by(month)
)
@@ -114,9 +118,7 @@ class ArtistService:
post_count = (
await self.session.execute(
select(func.count(func.distinct(Post.id)))
.select_from(Post)
.join(Source, Source.id == Post.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).scalar_one()
@@ -198,7 +200,7 @@ class ArtistService:
"mime": r.mime,
"width": r.width,
"height": r.height,
"thumbnail_url": thumbnail_url(r.sha256, r.mime),
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
}
for r in rows
],
@@ -206,27 +208,32 @@ class ArtistService:
)
async def find_or_create(self, name: str) -> tuple[Artist, bool]:
"""Return (artist, created). Slug-keyed; idempotent under races."""
"""Return (artist, created). Slug-keyed; idempotent under races.
Audit 2026-06-02: switched from session.rollback() to a
begin_nested savepoint + IntegrityError recovery so a lost
race doesn't unwind the calling request's surrounding work.
Mirrors importer._get_or_create.
"""
cleaned = (name or "").strip()
if not cleaned:
raise ValueError("artist name must not be empty")
slug = slugify(cleaned)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
select_existing = select(Artist).where(Artist.slug == slug)
existing = (await self.session.execute(select_existing)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
sp = await self.session.begin_nested()
try:
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
await self.session.flush()
await sp.commit()
except IntegrityError:
await self.session.rollback()
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
await sp.rollback()
existing = (await self.session.execute(select_existing)).scalar_one()
return existing, False
await self.session.commit()
return artist, True
+116 -17
View File
@@ -6,17 +6,16 @@ HTTP handlers (small ops) and from Celery tasks in
backend.app.tasks.admin (long ops).
This module is the PERMANENT home of artist-cascade + image-unlink
logic. The legacy copy at backend/app/services/migrators/cleanup.py
stays in place until FC-3j; FC-3j will replace its body with thin
re-exports from this module and then delete the wrapper.
logic. (The legacy migrators/cleanup.py copy was removed with the rest of
the one-and-done GS/IR migration tooling.)
"""
from __future__ import annotations
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
from sqlalchemy import func, select, update
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
@@ -188,10 +187,14 @@ def unlink_image_files(
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
# Convention thumbs dir — try both extensions thumbnailer writes
# (.jpg for opaque, .png for alpha). `.webp` used to be in this
# tuple but the thumbnailer never writes it (operator-flagged in
# the 2026-06-02 audit) — keep the tuple aligned with what
# actually lands on disk.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
for ext in ("jpg", "png"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
@@ -355,18 +358,101 @@ def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
Implementation note: the previous SELECT-ids → DELETE-WHERE-IN
pattern was vulnerable to the psycopg 65535-parameter ceiling on
libraries with tag explosions. The live delete now runs a single
DELETE with the same NOT-IN predicate find_unused_tags uses, so
the row count scales without binding every id as a parameter.
Audit 2026-06-02.
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
sample_rows = find_unused_tags(session, limit=50)
sample = [t.name for t in sample_rows]
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
count = session.execute(
select(func.count())
.select_from(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
).scalar_one()
return {"count": count, "sample_names": sample}
result = session.execute(
Tag.__table__.delete()
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
)
session.commit()
return {"deleted": result.rowcount or 0, "sample_names": sample}
# Legacy tags FC no longer uses, in two shapes:
# (1) kinds the tag input never produces — archive/post/artist.
# provenance (post grouping) + archive membership are their own
# systems now, and artists are first-class Artist/Source rows.
# meta/rating were already hard-deleted by alembic 0023.
# (2) name prefixes from IR kinds FC never adopted — `source:*`.
# ImageRepo had a `source` kind; FC's enum doesn't, so ir_ingest
# fell those back to `general` (kind=general, name="source:patreon"
# etc.). They can't be caught by kind, so we match the name prefix.
PURGEABLE_TAG_KINDS = ("archive", "post", "artist")
LEGACY_NAME_PREFIXES = ("source:",)
def _legacy_tag_predicate():
name_clauses = [Tag.name.like(f"{p}%") for p in LEGACY_NAME_PREFIXES]
return or_(Tag.kind.in_(PURGEABLE_TAG_KINDS), *name_clauses)
def purge_legacy_tags(session: Session, *, dry_run: bool = False) -> dict:
"""Count (dry_run) or delete legacy IR-migration tags: archive/post/
artist-kind tags PLUS general tags whose name matches a legacy
prefix (source:*).
CASCADE on image_tag / tag_alias / tag_allowlist /
tag_reference_embedding / tag_suggestion_rejection / series_page
clears the related rows on the parent DELETE.
Returns:
{"by_kind": {kind: count, ...}, # kind-matched rows
"by_prefix": {"source:*": count}, # name-prefix-matched rows
"count": total, "sample_names": [first 50],
and on live runs "deleted": total}
"""
predicate = _legacy_tag_predicate()
rows = session.execute(
select(Tag.id, Tag.name, Tag.kind).where(predicate)
).all()
by_kind: dict[str, int] = {}
by_prefix: dict[str, int] = {}
for _id, name, kind in rows:
# Classify by name-prefix first so a source:* row counts once,
# under the prefix bucket, regardless of its (general) kind.
matched_prefix = next(
(p for p in LEGACY_NAME_PREFIXES if name.startswith(p)), None,
)
if matched_prefix is not None:
label = f"{matched_prefix}*"
by_prefix[label] = by_prefix.get(label, 0) + 1
else:
key = kind.value if hasattr(kind, "value") else str(kind)
by_kind[key] = by_kind.get(key, 0) + 1
sample = [name for _id, name, _kind in rows[:50]]
total = len(rows)
result = {
"by_kind": by_kind, "by_prefix": by_prefix,
"count": total, "sample_names": sample,
}
if dry_run:
return result
if total:
session.execute(Tag.__table__.delete().where(predicate))
session.commit()
return {"deleted": len(ids), "sample_names": sample}
result["deleted"] = total
return result
# ---------------------------------------------------------------------------
@@ -435,6 +521,9 @@ class ConfirmTokenMismatch(Exception):
_VALID_RULES = ("transparency", "single_color")
_AUDIT_GUARD_THRESHOLD_MINUTES = 135 # matches LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES
def start_audit_run(
session: Session, *, rule: str, params: dict[str, Any],
) -> int:
@@ -442,11 +531,21 @@ def start_audit_run(
scan_library_for_rule Celery task. Returns the new audit_id.
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
has status='running'. Operator must cancel or wait."""
has status='running' AND started recently. Audit 2026-06-02 made
the guard age-aware: a SIGKILL'd run leaves a row in 'running'
that the recovery sweep flips on its next pass (~5 min), but a
fresh start_audit_run between the SIGKILL and the sweep would
previously block forever. Past the threshold, treat the running
row as stale and let the sweep clean it up — the new run still
gets to start.
"""
if rule not in _VALID_RULES:
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
cutoff = datetime.now(UTC) - timedelta(minutes=_AUDIT_GUARD_THRESHOLD_MINUTES)
existing = session.execute(
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
select(LibraryAuditRun.id)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at >= cutoff)
).scalar_one_or_none()
if existing is not None:
raise AuditAlreadyRunning(existing)
+53 -11
View File
@@ -1,40 +1,82 @@
"""Fernet-based encryption for credential blobs.
The key is a single 32-byte value (urlsafe-base64-encoded; what
Fernet.generate_key produces) stored at a fixed path inside the
images/data root. Created on first boot if absent; mode 0600. No KDF
needed — the file contents are already maximum-entropy random bytes.
Fernet.generate_key produces) stored at /images/secrets/credential_key.b64
(mode 0600, parent dir 0700). The 2026-06-02 audit caught a silent
key-regeneration path: on a partial disaster restore where the DB was
restored but the secrets dir was lost, the old `_load_or_create_key`
would mint a fresh key with no log, producing a working-looking system
where every authenticated download failed AUTH_ERROR until the operator
re-uploaded every credential by hand. Now the constructor refuses to
auto-generate unless either:
Operator backup procedure must include this file alongside the rest
of /images/ — losing it makes existing encrypted_blob rows
undecryptable (recovery = delete the rows and re-upload).
* the caller explicitly passes `bootstrap_ok=True` (tests, scripts), or
* the env var `CURATOR_BOOTSTRAP_NEW_KEY=1` is set (operator opt-in
during first-time setup).
Otherwise it raises `MissingCredentialKey` so the app fails fast at
startup and the operator can restore the key file from backup.
Operator backup procedure must include /images/secrets/ alongside the
rest of /images/ — losing the key file makes existing encrypted_blob
rows undecryptable (recovery = delete the rows and re-upload).
"""
import logging
import os
from pathlib import Path
from cryptography.fernet import Fernet, InvalidToken
log = logging.getLogger(__name__)
_BOOTSTRAP_ENV_VAR = "CURATOR_BOOTSTRAP_NEW_KEY"
class InvalidCredentialBlob(Exception):
"""Raised when decryption fails (wrong key, tampered blob, …)."""
class MissingCredentialKey(Exception):
"""The Fernet key file is missing AND the caller hasn't opted in to
generating a new one. Audit 2026-06-02: prevents silent key
regeneration on partial DB-restored / secrets-lost deployments.
Set CURATOR_BOOTSTRAP_NEW_KEY=1 for first-time setup, or restore the
key file from backup."""
class CredentialCrypto:
"""Fernet encrypt/decrypt with an on-disk key file.
Instantiate with a path; the file is created on first access and
reused thereafter. Tests pass a tmp_path; production calls with
Instantiate with a path; the file is loaded if present, or created
if absent AND the caller has opted in (bootstrap_ok=True or
CURATOR_BOOTSTRAP_NEW_KEY=1 env var). Production sites:
`IMAGES_ROOT / "secrets" / "credential_key.b64"`.
"""
def __init__(self, key_path: Path):
def __init__(self, key_path: Path, *, bootstrap_ok: bool | None = None):
self._key_path = Path(key_path)
self._fernet = Fernet(self._load_or_create_key())
if bootstrap_ok is None:
bootstrap_ok = os.environ.get(_BOOTSTRAP_ENV_VAR) == "1"
self._fernet = Fernet(self._load_or_create_key(bootstrap_ok))
def _load_or_create_key(self) -> bytes:
def _load_or_create_key(self, bootstrap_ok: bool) -> bytes:
if self._key_path.exists():
return self._key_path.read_bytes()
if not bootstrap_ok:
raise MissingCredentialKey(
f"Fernet key file not found at {self._key_path}. "
f"For first-time setup, set {_BOOTSTRAP_ENV_VAR}=1. "
f"If this is a restored instance, restore the key file "
f"from backup — generating a new one would make every "
f"existing Credential row undecryptable."
)
log.warning(
"Generating NEW Fernet credential key at %s. Any existing "
"encrypted_blob rows in the DB will be undecryptable — "
"re-upload each credential after this completes.",
self._key_path,
)
parent = self._key_path.parent
parent.mkdir(parents=True, exist_ok=True)
os.chmod(parent, 0o700)
+28 -1
View File
@@ -7,7 +7,7 @@ from __future__ import annotations
import json
import os
from dataclasses import dataclass
from datetime import datetime
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select
@@ -148,6 +148,7 @@ class CredentialService:
return None
plaintext = self.crypto.decrypt(row.encrypted_blob)
netscape = _to_netscape(plaintext)
netscape = _augment_cookies(platform, netscape)
self.cookies_dir.mkdir(parents=True, exist_ok=True)
out = self.cookies_dir / f"{platform}_cookies.txt"
out.write_text(netscape)
@@ -162,6 +163,32 @@ class CredentialService:
return None
return self.crypto.decrypt(row.encrypted_blob)
async def mark_verified(self, platform: str) -> datetime | None:
"""Stamp last_verified=now after a successful verify. Returns the
timestamp, or None if the credential is gone."""
row = (await self.session.execute(
select(Credential).where(Credential.platform == platform)
)).scalar_one_or_none()
if row is None:
return None
ts = datetime.now(UTC)
row.last_verified = ts
await self.session.commit()
return ts
def _augment_cookies(platform: str, netscape: str) -> str:
"""Delegate to the platform's `augment_cookies` hook if one is
registered (subscribestar, hentaifoundry, etc. — see
`services/platforms/<name>.py`). No-op when the platform doesn't
register a hook (Patreon, DeviantArt). Centralizing the
quirks-per-platform in the platforms package means adding a new
platform's cookie quirks doesn't require touching this file."""
info = PLATFORMS.get(platform)
if info is None or info.augment_cookies is None:
return netscape
return info.augment_cookies(netscape)
def _to_netscape(plaintext: str) -> str:
"""Accept either Netscape-format text (the extension's output) or a
+150 -4
View File
@@ -25,9 +25,18 @@ from sqlalchemy.orm import joinedload
from ..models import Artist, DownloadEvent, Source
from .credential_service import CredentialService
from .gallery_dl import GalleryDLService, SourceConfig
from .gallery_dl import (
BACKFILL_SKIP_VALUE,
BACKFILL_TIMEOUT_SECONDS,
TICK_SKIP_VALUE,
ErrorType,
GalleryDLService,
SourceConfig,
)
from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__)
@@ -83,6 +92,18 @@ class DownloadService:
ctx = setup
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
# alembic 0031 / plan #544: derive skip_value + timeout from the
# source's backfill_runs_remaining counter. When > 0, walk the full
# post history (skip: True + 1800s); when 0, exit gallery-dl after
# 20 contiguous archived items (skip: "exit:20" + the default
# 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
skip_value: bool | str = BACKFILL_SKIP_VALUE
source_config.timeout = BACKFILL_TIMEOUT_SECONDS
else:
skip_value = TICK_SKIP_VALUE
effective_url = _effective_url(
ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
)
@@ -94,6 +115,7 @@ class DownloadService:
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
resolved_campaign_id: str | None = None
@@ -120,6 +142,7 @@ class DownloadService:
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
return await self._phase3_persist(
@@ -155,6 +178,13 @@ class DownloadService:
return {"status": "in_flight", "event_id": existing.id}
if existing and existing.status == "pending":
existing.status = "running"
# Reset started_at on the pending→running transition so the
# recovery sweep (DOWNLOAD_STALL_THRESHOLD_MINUTES, 30 min)
# measures from real start, not from enqueue. On heavy-queue
# days a freshly-promoted event whose original started_at
# predated the cutoff would otherwise get swept mid-flight,
# racing phase3's commit. Audit 2026-06-02.
existing.started_at = datetime.now(UTC)
await self.async_session.commit()
event_id = existing.id
else:
@@ -165,7 +195,12 @@ class DownloadService:
event_id = ev.id
artist = source.artist
if source.platform in ("discord", "pixiv"):
# Drive cookies-vs-token selection from the platform registry's
# auth_type so a new 7th token-platform automatically picks the
# right credential path. The hardcoded tuple here used to drift
# out of sync with credential_service's auth_type_for(). Audit
# 2026-06-02.
if auth_type_for(source.platform) == "token":
cookies_path = None
auth_token = await self.cred_service.get_token(source.platform)
else:
@@ -184,6 +219,7 @@ class DownloadService:
"config_overrides": dict(source.config_overrides or {}),
"cookies_path": cookies_path,
"auth_token": auth_token,
"backfill_runs_remaining": source.backfill_runs_remaining or 0,
}
async def _phase3_persist(
@@ -237,6 +273,40 @@ class DownloadService:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError:
pass
# Enqueue thumbnail + ML for newly-attached images, matching
# the filesystem-import path (tasks/import_file.py:228-239).
# Importer.attach_in_place deliberately skips inline thumb
# generation to keep the import queue moving; the calling
# task is responsible for the enqueue. Operator-flagged
# 2026-06-01: without this, every downloaded image stayed
# at thumbnail_path=NULL until a periodic backfill swept
# it up, surfacing as broken-thumbnail tiles in the gallery
# for hours after a download landed. Lazy import to avoid
# circular-import risk between this service and the
# tasks/* modules that import it.
from ..tasks.ml import tag_and_embed
from ..tasks.thumbnail import generate_thumbnail
ids = list(result.member_image_ids)
if result.image_id is not None and result.image_id not in ids:
ids.append(result.image_id)
for img_id in ids:
generate_thumbnail.delay(img_id)
tag_and_embed.delay(img_id)
elif result.status == "attached":
# Non-media or extracted archive captured as PostAttachment
# (FC-2d-iii). The canonical copy lives in the attachments
# store; the original download path is now redundant —
# mirror duplicate_hash cleanup so we don't keep two copies.
# Operator-flagged 2026-06-02 (Lustria OST zip).
import_summary["attached"] += 1
try:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError:
pass
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped" and result.skip_reason and result.skip_reason.value in (
"duplicate_hash", "duplicate_phash",
):
@@ -245,6 +315,29 @@ class DownloadService:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped":
# Soft skip (too_small, too_transparent, invalid_image) —
# the file just didn't qualify, not a download/ingest
# failure. Don't flag the run as error; the file stays
# on disk for operator inspection.
import_summary["skipped"] += 1
elif result.status == "failed":
# Hard failure (today only: archive probe crash/timeout).
# The original archive sits in /images/ as an orphan; the
# filesystem scanner would re-import and re-crash on the
# same file, so delete the source file and surface the
# error in import_summary. Audit 2026-06-02.
import_summary["errors"] += 1
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "refreshed":
# Currently unreachable from attach_in_place (the download
# path never runs in deep=True mode), but the importer's
# ImportResult contract enumerates it. Treat the same as
# 'attached' — work happened, no error. Audit 2026-06-02.
import_summary["attached"] += 1
else:
import_summary["errors"] += 1
@@ -258,12 +351,21 @@ class DownloadService:
run_stats["quarantined_count"] = dl_result.files_quarantined
stderr_summary = self.gdl._extract_errors_warnings(dl_result.stderr)
status = "ok" if (dl_result.success and import_summary["errors"] == 0) else "error"
# Plan #544: PARTIAL means the run downloaded ≥1 file but the
# subprocess didn't finish in budget (typically wall-clock timeout
# mid-walk). Real work happened; the next tick continues via
# gallery-dl's archive. NOT a failure for status purposes.
if dl_result.success and import_summary["errors"] == 0:
status = "ok"
elif dl_result.error_type == ErrorType.PARTIAL and import_summary["errors"] == 0:
status = "ok"
else:
status = "error"
ev.status = status
ev.finished_at = datetime.now(UTC)
ev.files_count = import_summary["attached"]
ev.bytes_downloaded = bytes_downloaded
ev.error = dl_result.error_message if not dl_result.success else None
ev.error = dl_result.error_message if status == "error" else None
ev.metadata_ = {
"run_stats": run_stats,
"error_type": dl_result.error_type.value if dl_result.error_type else None,
@@ -276,18 +378,54 @@ class DownloadService:
}
await self._update_source_health(
source_id=ctx["source_id"], status=status, error_message=ev.error,
error_type=dl_result.error_type.value if dl_result.error_type else None,
)
# Plan #544: backfill lifecycle — auto-complete when a clean
# backfill run drained the queue (gallery-dl exited 0 + zero files
# downloaded means there was nothing to fetch); otherwise decrement
# the counter. Next tick falls back to tick mode once it hits 0.
#
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
# subprocess with return_code=0 and files_downloaded=0 (every
# file was quarantined), which used to match the auto-complete
# predicate exactly — zeroing the operator's armed budget on
# the FIRST quarantine run instead of decrementing. Require
# dl_result.success + no error_type so only genuinely-empty
# successful runs drain the counter.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
queue_drained = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
and dl_result.files_downloaded == 0
)
if queue_drained:
src.backfill_runs_remaining = 0
else:
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
await self.async_session.commit()
return event_id
async def _update_source_health(
self, *, source_id: int, status: str, error_message: str | None,
error_type: str | None = None,
) -> None:
"""FC-3d: update Source.{consecutive_failures, last_error, last_checked_at}.
ok -> failures = 0, error = None, checked_at = now
error -> failures += 1, error = error_message, checked_at = now
skipped -> failures unchanged, error = None, checked_at = now
When error_type == 'rate_limited', also stamps a platform-wide
cooldown via scheduler_service.set_platform_cooldown so the next
scan tick skips every source on this platform until the cooldown
expires. Preventive half of the burst-prevention pair —
consecutive_failures still backs the offending source off across
ticks.
"""
source = (await self.async_session.execute(
select(Source).where(Source.id == source_id)
@@ -296,9 +434,17 @@ class DownloadService:
if status == "ok":
source.consecutive_failures = 0
source.last_error = None
# alembic 0032 — clear the failure-class chip on success.
source.error_type = None
elif status == "error":
source.consecutive_failures = (source.consecutive_failures or 0) + 1
source.last_error = error_message
# alembic 0032 — stamp the failure-class so FailingSourcesCard
# can render a colored chip and operators can bulk-triage
# by error class without opening Logs per row.
source.error_type = error_type
if error_type == "rate_limited":
await set_platform_cooldown(self.async_session, source.platform)
elif status == "skipped":
source.last_error = None
source.last_checked_at = now
+69
View File
@@ -16,6 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
from ..utils.slug import slugify
from .source_service import NEW_SOURCE_BACKFILL_RUNS
class UnknownPlatformError(Exception):
@@ -86,6 +87,67 @@ class ExtensionService:
"created_artist": created_artist,
}
async def probe(self, url: str) -> dict:
"""Read-only resolution of a creator-page URL against the FC DB.
Returns one of:
- {state: 'unknown_platform'} — URL didn't match any
platform's strict artist-page pattern
- {state: 'new', platform, slug} — would create both
artist and source on quick-add
- {state: 'artist_match', platform, slug, artist}
— artist exists, this
exact URL isn't a Source yet (collapses the sidecar-synthetic
case too — the synthetic anchor counts as an existing artist
row but not as a pollable Source for this URL)
- {state: 'source_match', platform, slug, artist, source}
— exact (artist, platform,
url) Source already exists
Side-effect-free: two SELECTs at most.
"""
try:
platform, raw_slug = self._derive(url)
except (UnknownPlatformError, InvalidUrlError):
return {"state": "unknown_platform"}
slug = slugify(raw_slug)
artist = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
return {"state": "new", "platform": platform, "slug": slug}
artist_payload = {"id": artist.id, "name": artist.name, "slug": artist.slug}
source = (await self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one_or_none()
if source is None:
return {
"state": "artist_match",
"platform": platform,
"slug": slug,
"artist": artist_payload,
}
return {
"state": "source_match",
"platform": platform,
"slug": slug,
"artist": artist_payload,
"source": {
"id": source.id,
"artist_id": source.artist_id,
"platform": source.platform,
"url": source.url,
"enabled": source.enabled,
},
}
def _derive(self, url: str) -> tuple[str, str]:
if not isinstance(url, str) or not url.strip():
raise InvalidUrlError("url is empty")
@@ -143,9 +205,16 @@ class ExtensionService:
return existing, False
sp = await self.session.begin_nested()
try:
# New subscription sources arm a few backfill runs so the
# first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built).
# Mirrors SourceService.create — without it, Firefox quick-
# add on a creator with >20 unsynced posts would surface
# as "check failed" with no diagnosis. Audit 2026-06-02.
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
)
self.session.add(src)
await self.session.flush()
+230 -11
View File
@@ -39,19 +39,69 @@ class ErrorType(StrEnum):
HTTP_ERROR = "http_error"
UNSUPPORTED_URL = "unsupported_url"
VALIDATION_FAILED = "validation_failed"
# Run made real progress (downloaded ≥1 file) but did not finish in the
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
# mapping classifies this as "ok" because the next tick continues.
PARTIAL = "partial"
UNKNOWN_ERROR = "unknown_error"
# Tick mode (routine cron polls): skip ≤20 contiguous already-archived
# items, then exit gallery-dl. Established subscription with zero new
# content exits in ~30s of HEAD requests instead of walking to the bottom
# of the post history (which can be hours for prolific creators). 20 (not
# 5) is operator-set headroom against any edge case where paywalled or
# otherwise-non-downloadable items might interleave with archived ones —
# 20 contiguous HEADs is still negligible.
TICK_SKIP_VALUE = "exit:20"
# Backfill mode (operator-triggered deep scan): walk the full history.
# Source.backfill_runs_remaining > 0 selects this mode; the longer
# timeout below absorbs creators with thousands of posts.
#
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# before Celery SIGKILLs the worker — same rationale as the tick
# default at line 74. The audit (2026-06-02) caught this at 1800,
# guaranteeing SIGKILL on any backfill that ran to its subprocess
# budget: stdout/stderr lost, backfill_runs_remaining never
# decrements, recovery sweep stamps generic "stranded" 30 min later.
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
# was added to prevent. backfill_runs_remaining=3 still gives ~58
# minutes of cumulative walk across three runs for prolific creators.
BACKFILL_SKIP_VALUE = True
BACKFILL_TIMEOUT_SECONDS = 1170
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
# The 30s buffer absorbs scheduler jitter / GC pauses without making
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
# still live in source.config_overrides for legitimately long syncs.
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
@dataclass
class SourceConfig:
"""Per-source overrides loaded from Source.config_overrides JSON.
Note: the gallery-dl `skip` value (tick vs backfill, see TICK_SKIP_VALUE /
BACKFILL_SKIP_VALUE) is NOT carried here — it derives from the
Source.backfill_runs_remaining column at the download_service layer
and is passed to _build_config_for_source as `skip_value`. Same for
the per-run subprocess timeout.
"""
content_types: list[str] = field(default_factory=lambda: ["all"])
sleep: float | None = None
sleep_request: float | None = None
directory_pattern: str | None = None
filename_pattern: str | None = None
skip_existing: bool = True
save_metadata: bool = True
timeout: int = 3600
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
@classmethod
def from_dict(cls, data: dict) -> SourceConfig:
@@ -61,9 +111,8 @@ class SourceConfig:
sleep_request=data.get("sleep_request"),
directory_pattern=data.get("directory_pattern"),
filename_pattern=data.get("filename_pattern"),
skip_existing=data.get("skip_existing", True),
save_metadata=data.get("save_metadata", True),
timeout=data.get("timeout", 3600),
timeout=data.get("timeout", _DEFAULT_GDL_TIMEOUT_SECONDS),
)
@@ -219,6 +268,25 @@ class GalleryDLService:
"part-directory": str(self._config_dir / "temp"),
"retries": 3,
"timeout": 120.0,
# Forward Patreon as Referer/Origin to yt-dlp when it
# fetches video manifests. Operator-flagged 2026-06-01
# (DaferQ patreon): video posts hosted on Mux carry a JWT
# playback restriction that checks Referer/Origin on every
# request — not just the token signature. gallery-dl's
# HEAD probe to stream.mux.com returns 200 (the token is
# valid), but yt-dlp's actual GET-with-Range to fetch the
# m3u8 manifest 403s because yt-dlp sends its own default
# Referer, which Mux's policy rejects. Forcing the right
# headers fixes the headers-only case; Mux IP-range
# restrictions are unfixable from here.
"ytdl": {
"raw-options": {
"http_headers": {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
},
},
},
},
"output": {"progress": True},
}
@@ -233,7 +301,18 @@ class GalleryDLService:
platform: str,
source_config: SourceConfig,
artist_slug: str,
skip_value: bool | str = BACKFILL_SKIP_VALUE,
) -> dict:
"""`skip_value` controls gallery-dl's archive-walk behavior:
- True (BACKFILL_SKIP_VALUE): walk full post history, skipping
archived items but continuing past them. Used in backfill mode.
- "exit:20" (TICK_SKIP_VALUE): exit gallery-dl after 20
contiguous archived items. Used in tick (routine catch-up)
mode for fast no-op syncs on creators with deep history.
- False: don't skip — redownload everything (not used in FC).
The caller (download_service) chooses based on
Source.backfill_runs_remaining.
"""
config = json.loads(json.dumps(self._get_default_config())) # deep copy
destination = str(self.images_root / artist_slug / platform)
@@ -243,7 +322,7 @@ class GalleryDLService:
config["extractor"]["sleep"] = source_config.sleep
if source_config.sleep_request is not None:
config["extractor"]["sleep-request"] = source_config.sleep_request
config["extractor"]["skip"] = source_config.skip_existing
config["extractor"]["skip"] = skip_value
if source_config.save_metadata:
config["extractor"]["postprocessors"] = [
@@ -360,7 +439,17 @@ class GalleryDLService:
if return_code in (1, 4) and (skip_line_count > 0 or has_skip_text) and not has_actual_error:
return ErrorType.NO_NEW_CONTENT, "No new content to download"
if return_code in (1, 4) and not has_actual_error:
# Tier-gated classification used to require `return_code in (1, 4)`,
# which silently fell through to UNKNOWN_ERROR when gallery-dl
# returned a different exit code for mixed-failure runs (e.g.
# paywall warnings + a missing yt-dlp dep flipping the exit bits).
# The artist then surfaced as "needs attention" purely because a
# paywall blocked posts the operator wasn't paying to see —
# operator-flagged 2026-05-31. Now: if no source-level error
# category fired AND tier-gated warnings are present, classify
# as TIER_LIMITED regardless of return code. Same priority order
# as before (auth/rate/access/not_found/network/http still win).
if not has_actual_error:
tier_gated_lines = [
line for line in combined.split("\n")
if "][warning]" in line and "not allowed to view post" in line
@@ -372,6 +461,22 @@ class GalleryDLService:
f"Subscription tier does not grant access to {count} post{'s' if count != 1 else ''}",
)
# Partial-success: the subprocess exited non-zero (typically because
# the wall-clock timeout fired mid-walk), but it had downloaded ≥1
# file by then and no source-level error category fired. The work
# the run DID do is real; gallery-dl's archive will pick up where
# it left off on the next tick. Mapped to status="ok" downstream
# (download_service.py) so this doesn't flag the source as
# "needs attention." Operator-flagged 2026-06-01 after a Knuxy
# patreon run downloaded hundreds of files then ran red on timeout.
files_downloaded = self._count_downloaded_files(stdout)
if not has_actual_error and files_downloaded > 0:
return (
ErrorType.PARTIAL,
f"Downloaded {files_downloaded} file{'s' if files_downloaded != 1 else ''}; "
"run did not complete in budget — next tick will continue",
)
return ErrorType.UNKNOWN_ERROR, f"Unknown error (return code: {return_code})"
def _count_downloaded_files(self, stdout: str) -> int:
@@ -523,6 +628,7 @@ class GalleryDLService:
source_config: SourceConfig | None = None,
cookies_path: str | None = None,
auth_token: str | None = None,
skip_value: bool | str = BACKFILL_SKIP_VALUE,
) -> DownloadResult:
start_time = time.time()
started_at = datetime.now(UTC).isoformat()
@@ -530,7 +636,9 @@ class GalleryDLService:
if source_config is None:
source_config = SourceConfig()
config = self._build_config_for_source(platform, source_config, artist_slug)
config = self._build_config_for_source(
platform, source_config, artist_slug, skip_value=skip_value,
)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
@@ -632,13 +740,57 @@ class GalleryDLService:
started_at=started_at, completed_at=completed_at,
)
except subprocess.TimeoutExpired:
except subprocess.TimeoutExpired as e:
duration = time.time() - start_time
log.error("Download timeout for %s/%s after %.1fs", artist_slug, platform, duration)
# subprocess.run(text=True) makes these str if non-None, but the
# caller may have raised TimeoutExpired manually with None or
# bytes (tests do); coerce both cases to str.
partial_stdout = e.stdout or ""
partial_stderr = e.stderr or ""
if isinstance(partial_stdout, bytes):
partial_stdout = partial_stdout.decode("utf-8", "replace")
if isinstance(partial_stderr, bytes):
partial_stderr = partial_stderr.decode("utf-8", "replace")
files_so_far = self._count_downloaded_files(partial_stdout)
written_so_far = [str(p) for p in self._written_paths(partial_stdout)]
stderr_lines = partial_stderr.strip().splitlines()
tail_hint = stderr_lines[-1] if stderr_lines else "no stderr output"
# If the partial output already shows a rate-limit pattern, the
# timeout was almost certainly gallery-dl spinning on retries —
# promote to RATE_LIMITED so _update_source_health stamps the
# platform cooldown (same code path as a clean-exit rate limit).
# Otherwise stay TIMEOUT and let the captured stdout/stderr +
# files_so_far tell the operator whether it was "lots of
# content" vs "stuck retrying" vs "hung silent".
combined = (partial_stdout + "\n" + partial_stderr).lower()
if any(p in combined for p in self.RATE_LIMIT_PATTERNS):
error_type = ErrorType.RATE_LIMITED
error_message = (
f"Rate-limited and never completed within "
f"{source_config.timeout}s ({files_so_far} files written)"
)
else:
error_type = ErrorType.TIMEOUT
error_message = (
f"Download timed out after {source_config.timeout}s — "
f"{files_so_far} file(s) written; last stderr: {tail_hint}"
)
log.error(
"Download timeout for %s/%s after %.1fs (%d files written, "
"last stderr: %s)",
artist_slug, platform, duration, files_so_far, tail_hint,
)
return DownloadResult(
success=False, url=url, artist_slug=artist_slug, platform=platform,
error_type=ErrorType.TIMEOUT,
error_message=f"Download timed out after {source_config.timeout} seconds",
files_downloaded=files_so_far,
written_paths=written_so_far,
stdout=partial_stdout, stderr=partial_stderr,
return_code=-1, # killed by timeout, no real exit code
error_type=error_type, error_message=error_message,
duration_seconds=duration,
started_at=started_at,
completed_at=datetime.now(UTC).isoformat(),
@@ -658,3 +810,70 @@ class GalleryDLService:
Path(temp_config_path).unlink() # noqa: ASYNC240
except Exception:
pass
async def verify(
self,
url: str,
artist_slug: str,
platform: str,
source_config: SourceConfig | None = None,
cookies_path: str | None = None,
auth_token: str | None = None,
timeout: float = 45.0, # noqa: ASYNC109 — subprocess.run timeout, not a coroutine deadline
) -> tuple[bool, str]:
"""Test that credentials authenticate against `url` WITHOUT
downloading anything. Runs gallery-dl in --simulate mode limited
to the first item; if auth is bad the extractor errors before it
can list, which _categorize_error flags as AUTH_ERROR. Returns
(ok, message). Used by the credential Verify button."""
if source_config is None:
source_config = SourceConfig()
config = self._build_config_for_source(platform, source_config, artist_slug)
if cookies_path:
config["extractor"]["cookies"] = cookies_path
if auth_token and platform == "discord":
config["extractor"].setdefault("discord", {})["token"] = auth_token
if auth_token and platform == "pixiv":
config["extractor"].setdefault("pixiv", {})["refresh-token"] = auth_token
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
) as fh:
json.dump(config, fh, indent=2)
temp_config_path = fh.name
try:
cmd = [
sys.executable, "-m", "gallery_dl",
"--config", temp_config_path,
"--simulate", "--range", "1-1", "--verbose", url,
]
loop = asyncio.get_running_loop()
proc = await loop.run_in_executor(
None,
lambda: subprocess.run(
cmd, capture_output=True, text=True, timeout=timeout,
),
)
etype, msg = self._categorize_error(proc.returncode, proc.stdout, proc.stderr)
# TIER_LIMITED proves auth worked — gallery-dl reached the
# post, was told it's tier-gated. The download path treats
# this as success (line 712); verify must too, or operators
# rotate working cookies for no reason. Audit 2026-06-02.
if proc.returncode == 0 or etype in (
ErrorType.NO_NEW_CONTENT, ErrorType.TIER_LIMITED,
):
return True, "Credentials valid — the feed authenticated."
if etype == ErrorType.AUTH_ERROR:
return False, msg
# Network / not-found / rate-limit / unknown: inconclusive,
# not a definitive credential failure. Surface the reason.
return False, f"Could not confirm ({etype.value}): {msg}"
except subprocess.TimeoutExpired:
return False, f"Verification timed out after {timeout:.0f}s"
except Exception as exc: # noqa: BLE001
return False, f"Verification error: {exc}"
finally:
try:
Path(temp_config_path).unlink() # noqa: ASYNC240
except Exception:
pass
+40 -8
View File
@@ -20,8 +20,9 @@ from datetime import datetime
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import aliased
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
@@ -90,9 +91,27 @@ class TimelineBucket:
count: int
def thumbnail_url(sha256_hex: str, mime: str) -> str:
# Quart serves /images/* via the frontend blueprint (FC-1); thumbnails go
# under /images/thumbs/. The MIME determines the extension.
def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str:
"""Return the URL to fetch a thumbnail.
Prefers the stored thumbnail_path verbatim — Quart serves /images/*
1:1 from the volume (frontend.py:20-36), so the URL IS the disk
path. Falls back to deriving from (sha256, mime) only when the
record's thumbnail_path is NULL (thumbnailer hasn't run yet); that
URL will 404 until backfill catches it, same as before the path
was tracked.
Pre-2026-05-30 this was derived only from (sha256, mime), which
disagreed with the actual on-disk extension when the thumbnailer
chose its format from transparency rather than MIME — every PNG
source without alpha (extension was .jpg on disk) and every WebP
source with alpha (extension was .png on disk) silently 404'd
despite the thumbnail file existing.
"""
if thumbnail_path:
return thumbnail_path
# Fallback for records with no thumbnail recorded yet — preserves
# prior behavior (URL exists but 404s until backfill regenerates).
ext = ".png" if mime in ("image/png", "image/gif") else ".jpg"
bucket = sha256_hex[:3]
return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
@@ -115,10 +134,23 @@ def _provenance_clause(post_id, artist_id):
ImageProvenance.post_id == post_id,
)
if artist_id is not None:
# Use Post.artist_id (alembic 0030 denormalized column) instead
# of joining through ImageProvenance.source_id → Source.artist_id.
# The denormalization is the always-present linkage; the source
# path now drops NULL-source provenance rows (filesystem-imported
# content) which would otherwise vanish from artist-filtered
# gallery views.
# ALIAS Post: the gallery query outer-joins Post on
# ImageRecord.primary_post_id (`_outer_join_primary_post`).
# SQLAlchemy would otherwise correlate a bare `Post` reference
# in this EXISTS subquery to that outer Post (which is NULL for
# images with no primary post), and the filter would silently
# match nothing.
post_inner = aliased(Post)
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == Source.id,
Source.artist_id == artist_id,
ImageProvenance.post_id == post_inner.id,
post_inner.artist_id == artist_id,
)
return None
@@ -198,7 +230,7 @@ class GalleryService:
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.sha256, record.mime),
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
artist=artists.get(record.id),
)
for record, posted_at, eff_date in rows
@@ -306,7 +338,7 @@ class GalleryService:
"integrity_status": record.integrity_status,
"created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.sha256, record.mime),
"thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
"image_url": f"/images/{record.path.split('/images/', 1)[-1]}",
"artist": (
{"id": artist.id, "name": artist.name, "slug": artist.slug}
+215 -118
View File
@@ -30,6 +30,7 @@ from ..models import (
PostAttachment,
Source,
)
from ..utils import safe_probe
from ..utils.paths import derive_subdir, derive_top_level_artist, hash_suffixed_name
from ..utils.phash import compute_phash, find_similar
from ..utils.sidecar import find_sidecar, parse_sidecar
@@ -203,6 +204,32 @@ class Importer:
(phash, width or 0, height or 0, image_id)
)
def _get_or_create(self, stmt, factory):
"""Race-safe find-or-create. Run `stmt` (scalar_one_or_none); if a
row exists, return it. Otherwise open a savepoint and INSERT
``factory()``; on IntegrityError (a concurrent worker inserted the
same row first) roll the savepoint back — NOT the outer transaction,
which would lose the surrounding scan's progress — and re-run `stmt`
(scalar_one) to return the row the other worker created.
Centralizes the pattern shared by _find_or_create_source and
_find_or_create_post. The plain SELECT-then-INSERT version lost
races under the 5-min recovery sweep (operator-flagged
2026-05-26)."""
existing = self.session.execute(stmt).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = factory()
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(stmt).scalar_one()
def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> Source:
@@ -221,53 +248,35 @@ class Importer:
and re-select — the concurrent op just created the row we
wanted, so the second select will find it.
"""
existing = self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Source(artist_id=artist_id, platform=platform, url=url)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one()
stmt = select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
return self._get_or_create(
stmt,
lambda: Source(artist_id=artist_id, platform=platform, url=url),
)
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Filesystem-import sidecar Source resolver.
def _lookup_source_for_sidecar(
self, *, artist_id: int, platform: str,
) -> Source | None:
"""Find the real subscription Source for (artist, platform), or
None if no subscription exists.
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). The filesystem importer
used to call _find_or_create_source(url=sd.post_url), which created
one Source row per post URL — 100s of junk Sources per artist, all
with enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26.
New behaviour: if any Source row exists for (artist_id, platform),
reuse it regardless of its URL — the artist's real subscription Source
(created by the downloader / extension / UI) is the canonical
attachment point for filesystem-imported posts. If none exists, create
ONE synthetic anchor with url='sidecar:<platform>:<artist_slug>' and
enabled=False (so the subscription checker doesn't poll it).
Pre-alembic-0030 this method would CREATE a synthetic
`sidecar:<platform>:<slug>` Source when no real one existed —
because `Post.source_id` was NOT NULL and the importer needed
something to attach Posts to. Alembic 0030 relaxed both
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
synthetic anchors are obsolete; the importer now leaves
source_id as None when no subscription exists for the (artist,
platform). Operator-asked 2026-06-01: synthetic Sources had
leaked into the Subscriptions UI as phantom subscriptions and
the operator wanted the data model to truthfully say "this
content has no live subscription."
"""
existing = self.session.execute(
stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
@@ -275,63 +284,39 @@ class Importer:
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one_or_none()
if existing is not None:
return existing
synthetic_url = f"sidecar:{platform}:{artist_slug}"
sp = self.session.begin_nested()
try:
row = Source(
artist_id=artist_id,
platform=platform,
url=synthetic_url,
enabled=False,
)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one()
)
return self.session.execute(stmt).scalar_one_or_none()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
self, *, source_id: int | None, external_post_id: str,
artist_id: int,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
existing = self.session.execute(
select(Post).where(
"""Race-safe find-or-create on `post`. Keyed by
(source_id, external_post_id) when source_id is set — the
`uq_post_source_external_id` constraint guards. For NULL-source
posts the existence check matches on (artist_id, external_post_id),
which the partial unique index `uq_post_artist_external_id_null_source`
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
pattern as the rest of the helpers."""
if source_id is not None:
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Post(source_id=source_id, external_post_id=external_post_id)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one()
else:
stmt = select(Post).where(
Post.source_id.is_(None),
Post.artist_id == artist_id,
Post.external_post_id == external_post_id,
)
return self._get_or_create(
stmt,
lambda: Post(
source_id=source_id,
artist_id=artist_id,
external_post_id=external_post_id,
),
)
def import_one(self, source: Path) -> ImportResult:
"""Dispatch by kind. Media → normal pipeline. Archive → extract
@@ -371,12 +356,14 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
)
epid = sd.external_post_id or sc.stem
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
def _capture_attachment(
@@ -387,10 +374,19 @@ class Importer:
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
sha = _sha256_of(source)
existing = self.session.execute(
select(PostAttachment).where(PostAttachment.sha256 == sha)
).scalar_one_or_none()
if existing is None:
select_existing = select(PostAttachment).where(PostAttachment.sha256 == sha)
existing = self.session.execute(select_existing).scalar_one_or_none()
if existing is not None:
self.session.commit()
return ImportResult(status="attached")
# Savepoint + IntegrityError recovery — PostAttachment.sha256 is
# UNIQUE, so two workers can both pass the SELECT and only the
# second INSERT fails. Without savepoint, the outer transaction
# poisons and the calling task crashes. attachments.store is
# sha-addressed so both workers race to write the same target
# path; shutil.copy2 + rename is idempotent. Audit 2026-06-02.
sp = self.session.begin_nested()
try:
stored = self.attachments.store(source, sha)
self.session.add(PostAttachment(
post_id=post.id if post else None,
@@ -403,23 +399,67 @@ class Importer:
size_bytes=source.stat().st_size,
))
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
# Lost the race — the other worker's row is canonical.
self.session.execute(select_existing).scalar_one()
self.session.commit()
return ImportResult(status="attached")
def _import_archive(self, source: Path) -> ImportResult:
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
def _import_archive(
self, source: Path, *,
artist: Artist | None = None,
source_row: Source | None = None,
) -> ImportResult:
# Layer-3 isolation: bomb-size guard + integrity test in a
# spawned child BEFORE extracting in this process. A
# decompression bomb or a native-lib crash on a malformed
# archive is contained to the child; we reject the file cleanly
# instead of OOMing/segfaulting the import worker. extract_archive
# is already fail-soft for plain exceptions, so this only adds
# the hard-crash protection.
#
# Audit 2026-06-02: optional artist/source_row kwargs let the
# download path thread its explicit subscription context
# through instead of having _resolve_artist re-derive from
# path-walk (which works by coincidence today because gallery-dl
# lays files out under /images/<artist_slug>/...). Filesystem
# import still calls bare _import_archive(source) and falls
# back to the path-walk derivation as before.
probe = safe_probe.probe_archive(source)
if not probe.ok:
if probe.crashed:
return ImportResult(
status="failed",
error=f"archive probe crashed/timed out: {probe.reason}",
)
# Clean rejection (bomb cap exceeded, integrity mismatch):
# still preserve the archive file itself as an attachment so
# nothing silently vanishes, matching extract_archive's
# fail-soft contract.
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
self._capture_attachment(
source, post=post, artist=artist_use, resolved=True,
)
return ImportResult(status="attached")
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = []
with extract_archive(source) as members:
for _name, member_path in members:
if not is_supported(member_path):
continue # non-media preserved via the stored archive
res = self._import_media(member_path, source)
res = self._import_media(
member_path, source, explicit_source=source_row,
)
if res.status in ("imported", "superseded") and res.image_id:
member_ids.append(res.image_id)
# Preserve the archive itself (links to the same Post/Artist).
self._capture_attachment(
source, post=post, artist=artist, resolved=True
source, post=post, artist=artist_use, resolved=True
)
if member_ids:
return ImportResult(
@@ -429,7 +469,8 @@ class Importer:
return ImportResult(status="attached")
def _import_media(
self, source: Path, attribution_path: Path
self, source: Path, attribution_path: Path,
*, explicit_source: Source | None = None,
) -> ImportResult:
"""The media import pipeline (filters, dedup, copy, provenance).
@@ -446,7 +487,25 @@ class Importer:
# Compute file dimensions (images only) and apply filters.
width = height = None
has_alpha = False
if not is_video(source):
if is_video(source):
# Layer-3 isolation: validate the container via ffprobe (a
# separate process) before the rest of the pipeline touches
# it. A corrupt video that would crash a decoder is rejected
# cleanly here, and we capture width/height for free (the
# importer didn't previously record video dimensions).
probe = safe_probe.probe_video(source)
if not probe.ok:
if probe.crashed:
return ImportResult(
status="failed",
error=f"video probe crashed/timed out: {probe.reason}",
)
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=probe.reason,
)
width, height = probe.width, probe.height
else:
try:
with Image.open(source) as im:
im.verify()
@@ -558,7 +617,15 @@ class Importer:
artist = self._attach_artist(record, artist_name)
# Sidecar provenance (best-effort; never fails the import).
self._apply_sidecar(record, attribution_path, artist)
# explicit_source lets the FC-3c download path bind the new
# ImageProvenance row to its subscription Source instead of
# having _apply_sidecar re-derive via _lookup_source_for_sidecar.
# Audit 2026-06-02 — archive members extracted from a
# subscription-downloaded zip previously lost subscription
# linkage if the on-disk layout didn't match assumptions.
self._apply_sidecar(
record, attribution_path, artist, explicit_source=explicit_source,
)
# Thumbnail is queued separately by the calling task; the importer
# does not generate thumbnails inline so the import queue stays moving.
@@ -622,16 +689,36 @@ class Importer:
them through. The sidecar JSON gallery-dl emits next to each
downloaded file is read by `_apply_sidecar` via `find_sidecar`.
File-type dispatch parity with `import_one` (FC-2d-iii): zips,
PDFs, audio etc. become PostAttachments; archives are extracted.
Without this dispatch, gallery-dl-downloaded non-media bounced
back as `skipped+invalid_image`, which DownloadService counted
as an ingest error and flipped otherwise-successful runs to
status="error". Operator-flagged 2026-06-02 after a Lustria
patreon run with a 94MB OST zip went red despite 21 successful
image attaches.
Caller's responsibilities after this returns:
- duplicate_hash / duplicate_phash skip → delete the on-disk file
- superseded → file stays where it is (now canonical)
- imported → file stays where it is
- attached → the file's been copied into the attachments store;
caller may delete the on-disk original (mirrors duplicate_hash)
- failed → file untouched; caller decides
"""
if not is_supported(path):
if path.suffix.lower() == ".json":
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"unsupported extension {path.suffix}",
error="sidecar json is metadata, not content",
)
if is_archive(path):
return self._import_archive(
path, artist=artist, source_row=source,
)
if not is_supported(path):
post = self._post_for_sidecar(path, artist) if artist else None
return self._capture_attachment(
path, post=post, artist=artist, resolved=True,
)
# Format / dimension / transparency filters (mirror _import_media).
@@ -703,7 +790,8 @@ class Importer:
if rel == "smaller_exists":
target = self.session.get(ImageRecord, match_id)
self._supersede(
target, path, sha, phash, width, height, new_path=path
target, path, sha, phash, width, height,
new_path=path, artist=artist, source_row=source,
)
return ImportResult(status="superseded", image_id=match_id)
@@ -818,14 +906,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
if sd.post_url is not None:
post.post_url = sd.post_url
@@ -861,7 +950,7 @@ class Importer:
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
source_id=src.id if src else None,
captured_metadata=sd.raw,
)
)
@@ -897,6 +986,8 @@ class Importer:
self, existing: ImageRecord, source: Path, sha: str,
phash: str, width: int | None, height: int | None,
*, new_path: Path | None = None,
artist: Artist | None = None,
source_row: Source | None = None,
) -> None:
"""Replace `existing`'s file with the larger `source`, keeping the
row id (so tags/series/curation stay attached). ML is cleared so
@@ -948,8 +1039,14 @@ class Importer:
# _apply_sidecar resolves artist from the sidecar itself if the
# existing row has none, and is internally guarded against
# missing-or-malformed sidecars (silent return).
# Audit 2026-06-02: thread artist/source_row from the
# download-path caller (attach_in_place smaller_exists branch)
# so the supersede preserves explicit subscription linkage
# instead of re-deriving via path-walk.
try:
self._apply_sidecar(existing, source, None)
self._apply_sidecar(
existing, source, artist, explicit_source=source_row,
)
except Exception as exc:
# Don't unwind the supersede DB swap if sidecar parsing
# blows up unexpectedly — the file replacement is the
@@ -1,10 +0,0 @@
"""FC-5 migration tooling.
One module per concern (gs/ir/overlap/ml_queue/verify/cleanup).
Each migrator returns a counts dict; the run_migration task wires
that dict into MigrationRun.counts so the UI polling shows progress.
backup + rollback were retired in FC-3h (2026-05-24); first-class
backup lives at backend/app/services/backup_service.py and exposes
its own /api/system/backup/* surface.
"""
-182
View File
@@ -1,182 +0,0 @@
"""Targeted cleanup migrator: delete every image attributed to one Artist.
Built for the IR-migration rescue case where the filesystem scan derived
a bogus 'imagerepo' artist from a mismatched bind-mount layout. Every
image attributed to that artist (40k+ rows) needs to be removed — DB
rows, original files under `/images/<bucket>/...`, and thumbnails under
`/images/thumbs/...` — before the operator remounts and re-scans.
CASCADE handles image_tag, image_provenance, series_page, and
tag_suggestion_rejection child rows; import_task.result_image_id is
SET NULL by FK. We also delete ImportTask rows whose source_path starts
with the (still-existing) IR scan prefix so the next scan isn't fooled
by them.
"""
from __future__ import annotations
import logging
from pathlib import Path
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Artist, ImageRecord, ImportBatch, ImportTask
log = logging.getLogger(__name__)
_BATCH_SIZE = 500
def _zero_counts() -> dict:
return {
"rows_processed": 0, "rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0, "conflicts": 0,
}
def _thumb_path(images_root: Path, sha256_hex: str) -> tuple[Path, Path]:
"""Return both possible thumbnail paths (.jpg and .png). We try both
because the extension is chosen at generate-time based on the source
image's mode (alpha → .png, otherwise → .jpg)."""
bucket = sha256_hex[:3]
base = images_root / "thumbs" / bucket / sha256_hex
return base.with_suffix(".jpg"), base.with_suffix(".png")
def _delete_file(path: Path) -> bool:
"""Best-effort unlink; True if the file was actually removed."""
try:
path.unlink(missing_ok=True)
return True
except OSError as exc:
log.warning("cleanup: failed to unlink %s: %s", path, exc)
return False
async def cleanup_artist_async(
db: AsyncSession,
*,
slug: str,
images_root: Path | None = None,
dry_run: bool = False,
source_path_prefix: str | None = None,
) -> dict:
"""Delete every image attributed to the Artist with this slug,
along with the artist row itself and any associated import tasks.
Args:
slug: artist.slug to target (e.g. 'imagerepo').
images_root: defaults to /images.
dry_run: skip filesystem + DB writes; still walk rows for counts.
source_path_prefix: if set, ImportTask rows whose source_path
starts with this string are deleted too (use the IR scan
mount prefix, e.g. '/import/imagerepo').
"""
root = images_root if images_root is not None else Path("/images")
artist = (await db.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
raise ValueError(f"no Artist with slug={slug!r}")
artist_id = artist.id
artist_name = artist.name
total_images = (await db.execute(
select(func.count(ImageRecord.id)).where(ImageRecord.artist_id == artist_id)
)).scalar_one()
counts = _zero_counts()
files_deleted = 0
thumbs_deleted = 0
images_deleted = 0
# Batched delete loop. CASCADE handles image_tag, image_provenance,
# series_page, tag_suggestion_rejection. import_task.result_image_id
# is SET NULL by FK.
while True:
rows = (await db.execute(
select(ImageRecord.id, ImageRecord.path, ImageRecord.sha256)
.where(ImageRecord.artist_id == artist_id)
.limit(_BATCH_SIZE)
)).all()
if not rows:
break
ids = [r.id for r in rows]
counts["rows_processed"] += len(ids)
if not dry_run:
for r in rows:
if r.path:
if _delete_file(Path(r.path)):
files_deleted += 1
if r.sha256:
jpg, png = _thumb_path(root, r.sha256)
if _delete_file(jpg):
thumbs_deleted += 1
if _delete_file(png):
thumbs_deleted += 1
await db.execute(
delete(ImageRecord).where(ImageRecord.id.in_(ids))
)
await db.commit()
images_deleted += len(ids)
if dry_run:
# Nothing was actually deleted from the DB; bail after one
# pass so we don't loop forever.
break
import_tasks_deleted = 0
if source_path_prefix and not dry_run:
# Delete ImportTask rows whose source_path is under the bad mount
# prefix. These are mostly orphaned now (result_image_id was set
# NULL by CASCADE) but their presence still blocks the
# idempotency check in scan_directory if the operator remounts
# the same prefix.
like_pattern = source_path_prefix.rstrip("/") + "/%"
result = await db.execute(
delete(ImportTask).where(ImportTask.source_path.like(like_pattern))
)
import_tasks_deleted = result.rowcount or 0
await db.commit()
# Sweep ImportBatch rows that are now empty.
empty_batches_deleted = 0
if not dry_run:
empty_batch_ids = (await db.execute(
select(ImportBatch.id).where(
~select(ImportTask.id)
.where(ImportTask.batch_id == ImportBatch.id)
.exists()
)
)).scalars().all()
if empty_batch_ids:
result = await db.execute(
delete(ImportBatch).where(ImportBatch.id.in_(empty_batch_ids))
)
empty_batches_deleted = result.rowcount or 0
await db.commit()
# Finally, the artist row.
if not dry_run:
await db.execute(delete(Artist).where(Artist.id == artist_id))
await db.commit()
return {
"counts": counts,
"artist": {"id": artist_id, "name": artist_name, "slug": slug},
"summary": {
"images_targeted": total_images,
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_deleted": import_tasks_deleted,
"empty_batches_deleted": empty_batches_deleted,
"dry_run": dry_run,
},
}
-126
View File
@@ -1,126 +0,0 @@
"""GallerySubscriber export → FabledCurator ingest.
Reads a parsed gallerysubscriber-export-v1.json dict (no DB connection
to GS). Creates Artist (from subscriptions) + Source (nested under each
subscription) + Credential (re-encrypted with FC's key). Idempotent on
natural keys: Artist.slug, (artist_id, platform, url), Credential.platform.
Credentials arrive plaintext in the export — GS's export script
decrypts using GS's Fernet key in GS's own process. FC re-encrypts
with FC's CredentialCrypto.
"""
from __future__ import annotations
import json
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Artist, Credential, Source
from ...utils.slug import slugify
from ..credential_crypto import CredentialCrypto
def _zero_counts() -> dict:
return {
"rows_processed": 0, "rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0, "conflicts": 0,
}
async def migrate_async(
db: AsyncSession,
*,
data: dict,
fc_crypto: CredentialCrypto | None = None,
dry_run: bool = False,
) -> dict:
"""Ingest a parsed gallerysubscriber-export-v1.json dict."""
if data.get("source_app") != "gallerysubscriber":
raise ValueError("export source_app must be 'gallerysubscriber'")
if data.get("schema_version") != 1:
raise ValueError(f"unsupported schema_version: {data.get('schema_version')}")
counts = _zero_counts()
# Phase 1: subscriptions → Artist; nested sources within each.
for sub in data.get("subscriptions", []):
counts["rows_processed"] += 1
slug = slugify(sub["name"])
artist = (await db.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
if dry_run:
counts["rows_inserted"] += 1
# Continue to nested sources, but they can't link without an artist row.
continue
notes = json.dumps(sub.get("metadata"), indent=2) if sub.get("metadata") else None
artist = Artist(
name=sub["name"], slug=slug,
is_subscription=True,
auto_check=bool(sub.get("enabled", True)),
notes=notes,
)
db.add(artist)
await db.flush()
counts["rows_inserted"] += 1
else:
counts["rows_skipped"] += 1
# Nested sources under this subscription.
for src in sub.get("sources", []):
counts["rows_processed"] += 1
existing = (await db.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == src["platform"],
Source.url == src["url"],
)
)).scalar_one_or_none()
if existing is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
db.add(Source(
artist_id=artist.id,
platform=src["platform"],
url=src["url"],
enabled=bool(src.get("enabled", True)),
check_interval_override=src.get("check_interval"),
config_overrides=src.get("metadata") or {},
))
counts["rows_inserted"] += 1
# Phase 2: credentials.
for cred in data.get("credentials", []):
counts["rows_processed"] += 1
existing = (await db.execute(
select(Credential).where(Credential.platform == cred["platform"])
)).scalar_one_or_none()
if existing is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
if fc_crypto is None:
# Without a crypto helper we can't encrypt — skip rather than
# store plaintext.
counts["rows_skipped"] += 1
counts["conflicts"] += 1
continue
encrypted = fc_crypto.encrypt(cred["plaintext"])
db.add(Credential(
platform=cred["platform"],
credential_type=cred.get("credential_type") or "cookies",
encrypted_blob=encrypted,
expires_at=cred.get("expires_at"),
))
counts["rows_inserted"] += 1
if not dry_run:
await db.commit()
return counts
-146
View File
@@ -1,146 +0,0 @@
"""ImageRepo export → FabledCurator ingest.
Reads a parsed imagerepo-export-v1.json dict (no DB connection to IR).
Creates Tag rows (skipping artist/post kinds, resolving fandom_name to
FK). Writes the per-image-sha256 artist assignments + tag associations
+ series page assignments to /images/_migration_state/ir_tag_manifest.json
so tag_apply.py can join them to ImageRecord rows AFTER the operator
runs FC's filesystem scan.
"""
from __future__ import annotations
import json
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Tag, TagKind
_SKIP_KINDS = frozenset({"artist", "post"})
_MIGRATION_STATE_DIRNAME = "_migration_state"
_IR_MANIFEST_FILENAME = "ir_tag_manifest.json"
def _zero_counts() -> dict:
return {
"rows_processed": 0, "rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0, "conflicts": 0,
}
def manifest_path(images_root: Path | None = None) -> Path:
root = images_root if images_root is not None else Path("/images")
p = root / _MIGRATION_STATE_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p / _IR_MANIFEST_FILENAME
async def _resolve_fandom_id(
db: AsyncSession, fandom_name: str | None, dry_run: bool,
) -> int | None:
"""Find-or-create a fandom-kind Tag by name."""
if not fandom_name:
return None
existing = (await db.execute(
select(Tag).where(Tag.name == fandom_name, Tag.kind == "fandom")
)).scalar_one_or_none()
if existing is not None:
return existing.id
if dry_run:
return None
t = Tag(name=fandom_name, kind=TagKind.fandom)
db.add(t)
await db.flush()
return t.id
async def migrate_async(
db: AsyncSession,
*,
data: dict,
images_root: Path | None = None,
dry_run: bool = False,
) -> dict:
"""Ingest a parsed imagerepo-export-v1.json dict.
Creates Tag rows + writes the IR tag manifest file. Tag-to-image
binding happens later in tag_apply.py (after FC's filesystem scan
populates image_record.sha256 → id).
"""
if data.get("source_app") != "imagerepo":
raise ValueError("export source_app must be 'imagerepo'")
if data.get("schema_version") not in (1, 2):
raise ValueError(f"unsupported schema_version: {data.get('schema_version')}")
counts = _zero_counts()
# Phase 1: tags (skip artist + post kinds; resolve fandom_name → fandom_id).
# First pass: create all fandom-kind tags so they're available for FK resolution.
for tag in data.get("tags", []):
kind = tag.get("kind") or "general"
if kind != "fandom":
continue
counts["rows_processed"] += 1
existing = (await db.execute(
select(Tag).where(Tag.name == tag["name"], Tag.kind == "fandom")
)).scalar_one_or_none()
if existing is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
db.add(Tag(name=tag["name"], kind=TagKind.fandom))
counts["rows_inserted"] += 1
if not dry_run:
await db.flush()
# Second pass: every other kind.
for tag in data.get("tags", []):
kind_str = tag.get("kind") or "general"
if kind_str in _SKIP_KINDS:
counts["rows_skipped"] += 1
continue
if kind_str == "fandom":
continue # handled above
counts["rows_processed"] += 1
try:
kind = TagKind(kind_str)
except ValueError:
kind = TagKind.general
fandom_id = await _resolve_fandom_id(db, tag.get("fandom_name"), dry_run)
existing = (await db.execute(
select(Tag).where(Tag.name == tag["name"], Tag.kind == kind)
)).scalar_one_or_none()
if existing is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
db.add(Tag(name=tag["name"], kind=kind, fandom_id=fandom_id))
counts["rows_inserted"] += 1
if not dry_run:
await db.commit()
# Phase 2: write the per-image manifest for tag_apply.py to consume later.
# schema_version 2 (added 2026-05-24) carries `image_posts` for
# Post + Source + ImageProvenance restore; schema 1 manifests
# without it stay valid (tag_apply treats the missing field as []).
manifest = {
"schema_version": data.get("schema_version", 1),
"image_artist_assignments": data.get("image_artist_assignments", []),
"image_tag_associations": data.get("image_tag_associations", []),
"series_pages": data.get("series_pages", []),
"image_posts": data.get("image_posts", []),
}
counts["rows_processed"] += len(manifest["image_artist_assignments"])
counts["rows_processed"] += len(manifest["image_tag_associations"])
counts["rows_processed"] += len(manifest["series_pages"])
counts["rows_processed"] += len(manifest["image_posts"])
if not dry_run:
manifest_path(images_root).write_text(json.dumps(manifest, indent=2))
return counts
@@ -1,21 +0,0 @@
"""Queue every migrated image_record with no embedding for ML re-processing."""
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import ImageRecord
async def queue_all_unprocessed_async(db: AsyncSession) -> int:
"""Find every ImageRecord with siglip_embedding IS NULL, fire
tag_and_embed.delay(id) for each. Returns count queued.
"""
from ...tasks.ml import tag_and_embed
rows = (await db.execute(
select(ImageRecord.id).where(ImageRecord.siglip_embedding.is_(None))
)).scalars().all()
for image_id in rows:
tag_and_embed.delay(image_id)
return len(rows)
-368
View File
@@ -1,368 +0,0 @@
"""Apply the IR tag manifest after FC's filesystem scan.
Reads /images/_migration_state/ir_tag_manifest.json and joins each entry
to an ImageRecord row by sha256 (which exists after the operator runs
FC's filesystem scan over the mounted IR images dir).
- image_artist_assignments → ImageRecord.artist_id (find_or_create Artist by slug).
- image_tag_associations → image_tag insert (idempotent).
- series_pages → series_page insert (idempotent on image_id unique).
- image_posts (schema v2) → Source + Post + ImageProvenance restore.
Unmatched sha256s are logged into the result's `unmatched` list so the
Celery task can drop them into MigrationRun.metadata for the operator
to inspect.
"""
from __future__ import annotations
import json
from datetime import datetime
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import (
Artist,
ImageProvenance,
ImageRecord,
Post,
SeriesPage,
Source,
Tag,
TagKind,
image_tag,
)
from ...utils.slug import slugify
from .ir_ingest import manifest_path
# Per-platform artist-profile URL — used as Source.url when restoring
# IR PostMetadata into FC. Must cover every platform that
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS
# recognizes; an entry missing here silently drops ALL PostMetadata for
# that platform during phase 4 (operator hit this 2026-05-25:
# DeviantArt + Pixiv posts in the IR migration produced empty
# ImageProvenance because they fell through this table).
#
# Pixiv caveat: the real profile URL takes a numeric user_id
# (https://www.pixiv.net/users/12345), but IR's PostMetadata.artist
# stores the display name not the id. We use the slugified name here
# so we preserve the artist→post→image linkage; the resulting Source.url
# won't resolve in a browser and the operator may want to manually fix
# it via Settings → Subscriptions once the migration lands.
_PLATFORM_PROFILE_URL = {
"patreon": "https://www.patreon.com/{slug}",
"subscribestar": "https://www.subscribestar.com/{slug}",
"hentaifoundry": "https://www.hentai-foundry.com/user/{slug}",
"deviantart": "https://www.deviantart.com/{slug}",
"pixiv": "https://www.pixiv.net/users/{slug}",
}
def _profile_url(platform: str, artist_slug: str) -> str | None:
fmt = _PLATFORM_PROFILE_URL.get(platform)
return fmt.format(slug=artist_slug) if fmt else None
async def _find_or_create_source(
db: AsyncSession, *, artist_id: int, platform: str, url: str, dry_run: bool,
) -> int | None:
existing = (await db.execute(
select(Source.id).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one_or_none()
if existing is not None:
return existing
if dry_run:
return None
s = Source(artist_id=artist_id, platform=platform, url=url, enabled=False)
db.add(s)
await db.flush()
return s.id
async def _find_or_create_post(
db: AsyncSession, *,
source_id: int, external_post_id: str,
title: str | None, description: str | None, post_url: str | None,
post_date_iso: str | None, attachment_count: int, dry_run: bool,
) -> int | None:
existing = (await db.execute(
select(Post.id).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
)).scalar_one_or_none()
if existing is not None:
return existing
if dry_run:
return None
post_date = None
if post_date_iso:
post_date = datetime.fromisoformat(post_date_iso)
p = Post(
source_id=source_id,
external_post_id=external_post_id,
post_title=title,
description=description,
post_url=post_url,
post_date=post_date,
attachment_count=attachment_count,
raw_metadata={"migrated_from": "imagerepo"},
)
db.add(p)
await db.flush()
return p.id
async def _ensure_provenance(
db: AsyncSession, *,
image_id: int, post_id: int, source_id: int, dry_run: bool,
) -> bool:
"""Returns True if a new ImageProvenance row was inserted.
Also sets ImageRecord.primary_post_id to this post if the image
doesn't already have one — preserves any primary_post_id already
assigned at download time by the importer (don't clobber). This is
the linkage gallery_service.py uses to surface Post.post_date as
the image's effective date for sort/group/jump/neighbor nav.
"""
existing = (await db.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == image_id,
ImageProvenance.post_id == post_id,
ImageProvenance.source_id == source_id,
)
)).scalar_one_or_none()
# Whether-or-not the provenance row already exists, ensure the
# image's primary_post_id is set so the gallery date-coalesce works.
# Idempotent: only writes when currently NULL.
if not dry_run:
await db.execute(
ImageRecord.__table__.update()
.where(ImageRecord.id == image_id)
.where(ImageRecord.primary_post_id.is_(None))
.values(primary_post_id=post_id)
)
if existing is not None:
return False
if dry_run:
return True
db.add(ImageProvenance(
image_record_id=image_id, post_id=post_id, source_id=source_id,
))
await db.flush()
return True
def _zero_counts() -> dict:
return {
"rows_processed": 0, "rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0, "conflicts": 0,
}
async def _ensure_artist_id(
db: AsyncSession, artist_name: str, dry_run: bool,
) -> int | None:
if not artist_name or not artist_name.strip():
return None
slug = slugify(artist_name)
existing = (await db.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if existing is not None:
return existing.id
if dry_run:
return None
a = Artist(name=artist_name, slug=slug, is_subscription=False)
db.add(a)
await db.flush()
return a.id
async def _resolve_tag_id(
db: AsyncSession, tag_name: str, tag_kind: str,
) -> int | None:
try:
kind = TagKind(tag_kind)
except ValueError:
kind = TagKind.general
row = (await db.execute(
select(Tag.id).where(Tag.name == tag_name, Tag.kind == kind)
)).scalar_one_or_none()
return row
async def _sha_to_image_id(db: AsyncSession, sha: str) -> int | None:
return (await db.execute(
select(ImageRecord.id).where(ImageRecord.sha256 == sha)
)).scalar_one_or_none()
async def apply_async(
db: AsyncSession,
*,
images_root: Path | None = None,
dry_run: bool = False,
) -> dict:
"""Apply the manifest. Returns counts + an `unmatched` list of sha256s."""
mf_path = manifest_path(images_root)
if not mf_path.exists():
raise FileNotFoundError(f"no IR tag manifest at {mf_path}")
manifest = json.loads(mf_path.read_text())
counts = _zero_counts()
unmatched: list[dict] = []
# 1. Artist assignments.
for entry in manifest.get("image_artist_assignments", []):
counts["rows_processed"] += 1
img_id = await _sha_to_image_id(db, entry["sha256"])
if img_id is None:
unmatched.append({"kind": "artist", **entry})
counts["rows_skipped"] += 1
continue
aid = await _ensure_artist_id(db, entry["artist_name"], dry_run)
if aid is None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
img = await db.get(ImageRecord, img_id)
if img is not None and img.artist_id != aid:
img.artist_id = aid
counts["rows_inserted"] += 1
else:
counts["rows_skipped"] += 1
# 2. Tag associations.
for entry in manifest.get("image_tag_associations", []):
counts["rows_processed"] += 1
img_id = await _sha_to_image_id(db, entry["sha256"])
if img_id is None:
unmatched.append({"kind": "tag", **entry})
counts["rows_skipped"] += 1
continue
tag_id = await _resolve_tag_id(
db, entry["tag_name"], entry.get("tag_kind") or "general",
)
if tag_id is None:
counts["rows_skipped"] += 1
continue
# Skip if association already exists.
already = (await db.execute(
select(image_tag.c.image_record_id).where(
image_tag.c.image_record_id == img_id,
image_tag.c.tag_id == tag_id,
)
)).first()
if already is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
await db.execute(image_tag.insert().values(
image_record_id=img_id, tag_id=tag_id, source="manual",
))
counts["rows_inserted"] += 1
# 3. Series pages.
for entry in manifest.get("series_pages", []):
counts["rows_processed"] += 1
img_id = await _sha_to_image_id(db, entry["sha256"])
if img_id is None:
unmatched.append({"kind": "series", **entry})
counts["rows_skipped"] += 1
continue
series_tag_id = await _resolve_tag_id(db, entry["series_tag_name"], "series")
if series_tag_id is None:
counts["rows_skipped"] += 1
continue
existing = (await db.execute(
select(SeriesPage).where(SeriesPage.image_id == img_id)
)).scalar_one_or_none()
if existing is not None:
counts["rows_skipped"] += 1
continue
if dry_run:
counts["rows_inserted"] += 1
continue
db.add(SeriesPage(
series_tag_id=series_tag_id,
image_id=img_id,
page_number=entry["page_number"],
))
counts["rows_inserted"] += 1
# 4. Image posts (schema v2) → Source + Post + ImageProvenance.
# Restores IR PostMetadata as FC's downloader-track provenance,
# so the modal's ProvenancePanel surfaces title/description/
# source URL/publish date the same way it does for live
# gallery-dl downloads.
for entry in manifest.get("image_posts", []):
counts["rows_processed"] += 1
platform = entry.get("platform")
artist_name = entry.get("artist")
if not platform or not artist_name:
counts["rows_skipped"] += 1
continue
aid = await _ensure_artist_id(db, artist_name, dry_run)
if aid is None:
counts["rows_skipped"] += 1
continue
url = _profile_url(platform, slugify(artist_name))
if url is None:
counts["rows_skipped"] += 1
continue
source_id = await _find_or_create_source(
db, artist_id=aid, platform=platform, url=url, dry_run=dry_run,
)
if source_id is None:
counts["rows_skipped"] += 1
continue
post_id = await _find_or_create_post(
db, source_id=source_id,
external_post_id=entry.get("post_id") or "",
title=entry.get("title"),
description=entry.get("description"),
post_url=entry.get("source_url"),
post_date_iso=entry.get("published_at"),
attachment_count=entry.get("attachment_count") or 0,
dry_run=dry_run,
)
if post_id is None:
counts["rows_skipped"] += 1
continue
for sha in entry.get("image_sha256s", []):
img_id = await _sha_to_image_id(db, sha)
if img_id is None:
unmatched.append({
"kind": "post", "sha256": sha,
"post_id": entry.get("post_id"),
})
continue
inserted = await _ensure_provenance(
db, image_id=img_id, post_id=post_id,
source_id=source_id, dry_run=dry_run,
)
if inserted:
counts["rows_inserted"] += 1
else:
counts["rows_skipped"] += 1
if not dry_run:
await db.commit()
return {"counts": counts, "unmatched": unmatched}
-80
View File
@@ -1,80 +0,0 @@
"""Post-migration verification: row counts + sha256 sampling."""
from __future__ import annotations
import hashlib
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Artist, Credential, ImageRecord, Source, Tag
async def verify_async(db: AsyncSession, *, expected: dict | None = None) -> dict:
"""Return per-check status dicts. `expected` is optional row-count
assertions; checks default to status='ok' when no expected provided."""
expected = expected or {}
results: dict[str, dict] = {}
checks = {
"artist_subscriptions": (
select(func.count(Artist.id)).where(Artist.is_subscription.is_(True))
),
"source_count": select(func.count(Source.id)),
"credential_count": select(func.count(Credential.id)),
"tag_count": select(func.count(Tag.id)),
"image_record_imported_or_downloaded": (
select(func.count(ImageRecord.id))
.where(ImageRecord.origin.in_(["imported_filesystem", "downloaded"]))
),
}
for name, stmt in checks.items():
actual = (await db.execute(stmt)).scalar_one()
exp = expected.get(name)
status = "ok" if exp is None or exp == actual else "mismatch"
results[name] = {"status": status, "actual": int(actual), "expected": exp}
return results
async def verify_sha256_sample(
db: AsyncSession, *, sample_size: int = 20,
) -> dict:
"""Sample N image_records; verify file exists + sha256 matches."""
rows = (await db.execute(
select(ImageRecord.id, ImageRecord.path, ImageRecord.sha256)
.order_by(func.random()).limit(sample_size)
)).all()
matched = 0
mismatched = 0
missing = 0
samples: list[dict] = []
for img_id, path, expected_sha in rows:
p = Path(path)
# Sync stdlib filesystem ops are intentional: this verify pass runs
# inside a Celery task under asyncio.run; no other awaitables compete
# for the loop. Same pattern as download_service.py.
if not p.exists(): # noqa: ASYNC240
missing += 1
samples.append({"id": img_id, "path": path, "result": "missing"})
continue
h = hashlib.sha256()
with p.open("rb") as f: # noqa: ASYNC230
for chunk in iter(lambda: f.read(65536), b""):
h.update(chunk)
if h.hexdigest() == expected_sha:
matched += 1
samples.append({"id": img_id, "result": "ok"})
else:
mismatched += 1
samples.append({
"id": img_id, "path": path, "result": "mismatch",
"expected_sha": expected_sha, "actual_sha": h.hexdigest(),
})
return {
"sample_size": len(rows),
"matched": matched,
"mismatched": mismatched,
"missing": missing,
"samples": samples,
}
+20 -4
View File
@@ -19,7 +19,6 @@ from ...models import (
TagReferenceEmbedding,
)
from ...models.tag import image_tag
from .embedder import MODEL_VERSION as SIGLIP_VERSION
ELIGIBLE_KINDS = {
TagKind.character,
@@ -46,6 +45,21 @@ class CentroidService:
)
).scalar_one()
async def _model_version(self) -> str:
"""Audit 2026-06-02: SigLIP model-version stamp comes from the
DB row, not the env constant. tag_and_embed (tasks/ml.py:110)
already reads from MLSettings.embedder_model_version, so by
sourcing centroid stamps + drift checks from the same row, we
eliminate the silent-drift case the audit flagged. env
SIGLIP_MODEL_VERSION still drives which model embedder.py
loads at runtime; the version stamp is purely the operator-
controlled identifier."""
return (
await self.session.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
)
).scalar_one()
async def recompute_for_tag(self, tag_id: int) -> bool:
"""Recompute one tag's centroid. Returns True if a centroid was
written, False if skipped (ineligible kind or too few members)."""
@@ -69,19 +83,20 @@ class CentroidService:
return False
centroid = np.mean(np.stack(embeddings), axis=0).astype(np.float32)
model_version = await self._model_version()
stmt = insert(TagReferenceEmbedding).values(
tag_id=tag_id,
embedding=centroid.tolist(),
reference_count=len(embeddings),
model_version=SIGLIP_VERSION,
model_version=model_version,
)
stmt = stmt.on_conflict_do_update(
index_elements=["tag_id"],
set_={
"embedding": centroid.tolist(),
"reference_count": len(embeddings),
"model_version": SIGLIP_VERSION,
"model_version": model_version,
"updated_at": func.now(),
},
)
@@ -92,6 +107,7 @@ class CentroidService:
"""Tag ids whose centroid is stale: member count != reference_count,
OR no centroid row, OR centroid built on a different SigLIP version.
Only considers eligible-kind tags with embeddings present."""
current_model_version = await self._model_version()
member_counts = (
select(
image_tag.c.tag_id.label("tag_id"),
@@ -116,7 +132,7 @@ class CentroidService:
TagReferenceEmbedding.reference_count
!= member_counts.c.members
)
| (TagReferenceEmbedding.model_version != SIGLIP_VERSION)
| (TagReferenceEmbedding.model_version != current_model_version)
)
)
return list((await self.session.execute(stmt)).scalars().all())
+30 -12
View File
@@ -4,7 +4,7 @@ threshold-filtered, category-grouped, ranked suggestions for one image.
from dataclasses import dataclass, field
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import (
@@ -16,6 +16,7 @@ from ...models import (
from ...models.tag import image_tag
from .aliases import AliasService
from .centroids import CentroidService
from .tag_name import normalize as normalize_tag_name
from .tagger import SURFACED_CATEGORIES
@@ -48,11 +49,11 @@ class SuggestionService:
).scalar_one()
def _threshold_for(self, s: MLSettings, category: str) -> float:
# 'artist' intentionally absent (FC-2d-vii-c) — falls through to
# the 1.01 "never surfaces" default like any unsurfaced category.
# 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired;
# both fall through to the 1.01 "never surfaces" default like any
# unsurfaced category.
return {
"character": s.suggestion_threshold_character,
"copyright": s.suggestion_threshold_copyright,
"general": s.suggestion_threshold_general,
}.get(category, 1.01)
@@ -84,7 +85,12 @@ class SuggestionService:
)
# --- Camie predictions ---
candidates: list[tuple[str, str, float]] = []
# candidates carry (raw_name, display_name, category, confidence).
# raw_name = the booru-formatted vocab key, kept for alias_map
# lookup since alias rows are hand-curated against raw keys.
# display_name = normalize_tag_name(raw_name) — what the operator
# sees AND what gets written to tag.name on Accept.
candidates: list[tuple[str, str, str, float]] = []
for name, p in predictions.items():
category = p.get("category", "general")
if category not in SURFACED_CATEGORIES:
@@ -92,10 +98,14 @@ class SuggestionService:
conf = float(p.get("confidence", 0.0))
if conf < self._threshold_for(settings, category):
continue
candidates.append((name, category, conf))
display = normalize_tag_name(name)
if display is None:
# emoticon / pure-punctuation vocab entry — drop entirely
continue
candidates.append((name, display, category, conf))
alias_map = await self.aliases.resolve_many(
[(n, c) for n, c, _ in candidates]
[(raw, c) for raw, _disp, c, _conf in candidates]
)
merged: dict[object, Suggestion] = {}
@@ -116,8 +126,8 @@ class SuggestionService:
creates_new_tag=existing.creates_new_tag,
)
for name, category, conf in candidates:
canonical = alias_map.get((name, category))
for raw, display, category, conf in candidates:
canonical = alias_map.get((raw, category))
if canonical is not None:
if canonical.id in applied or canonical.id in rejected:
continue
@@ -133,9 +143,17 @@ class SuggestionService:
),
)
else:
# Case-insensitive match on BOTH the raw camie key AND
# the normalized form — covers legacy underscore-named
# Tag rows accepted before normalization shipped, AND
# any tag the operator created with the human form.
existing_tag = (
await self.session.execute(
select(Tag).where(Tag.name == name)
select(Tag).where(
func.lower(Tag.name).in_(
[raw.lower(), display.lower()]
)
)
)
).scalars().first()
if existing_tag is not None:
@@ -157,10 +175,10 @@ class SuggestionService:
)
else:
_merge(
f"raw:{name}:{category}",
f"raw:{display}:{category}",
Suggestion(
canonical_tag_id=None,
display_name=name,
display_name=display,
category=category,
score=conf,
source="tagger",
+62
View File
@@ -0,0 +1,62 @@
"""Camie vocabulary -> human-readable tag-name normalization.
Camie v2's ~57k tag vocabulary is booru-derived and arrives as raw
strings like `uchiha_sasuke_(naruto)`, `#unicus_(idolmaster)`,
`1000-nen_ikiteru_(vocaloid)`, or `:/`. We want the operator to see
"Uchiha Sasuke", "Unicus", "1000-Nen Ikiteru", or to never see the
emoticon at all — and we want the same clean string to be what lands
in `tag.name` when the suggestion is accepted, so Accept matches the
existing-tag convention (`tag_service.find_or_create`).
Rules (operator-approved 2026-06-03):
1. Strip leading junk chars (#, ., +, ;, ~, _, whitespace)
2. Drop trailing `_(disambiguator)` block(s), iteratively
3. Strip wrapping single/double quotes (after disambig removal so
`"foo_em_up"_(series)` -> `"foo_em_up"` -> `foo_em_up`)
4. Replace remaining `_` with space; collapse runs of whitespace
5. Add a space after any `:` (namespace:tag -> namespace: tag)
6. Preserve hyphens (booru hyphens often carry meaning)
7. Title-case each space-separated word (first character only —
apostrophes, digits, hyphens stay)
8. If no letters AND no digits remain, return None (drops emoticons
like `:/` or `^_^`; preserves bare digit tags like `2005`)
9. No surname/givenname swap — no reliable signal in the vocab
"""
import re
_LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
_MULTISPACE = re.compile(r"\s+")
_COLON_NOSPACE = re.compile(r":(?=\S)")
_HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
def _strip_wrapping_quotes(s: str) -> str:
if len(s) >= 2 and s[0] == s[-1] and s[0] in ('"', "'"):
return s[1:-1]
return s
def _title_word(w: str) -> str:
return w[:1].upper() + w[1:] if w else w
def normalize(raw: str) -> str | None:
"""Return the human-readable form of a raw Camie tag, or None if the
string is junk (emoticon, empty after stripping)."""
if not raw:
return None
s = _LEADING_JUNK.sub("", raw)
while True:
new = _TRAILING_DISAMBIG.sub("", s)
if new == s:
break
s = new
s = _strip_wrapping_quotes(s)
s = s.replace("_", " ")
s = _COLON_NOSPACE.sub(": ", s)
s = _MULTISPACE.sub(" ", s).strip()
if not s or not _HAS_ALPHANUMERIC.search(s):
return None
return " ".join(_title_word(w) for w in s.split(" "))
+7 -4
View File
@@ -38,10 +38,13 @@ STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
# The categories FC-2b surfaces in the UI. Others (meta/rating/year) are
# still stored but the suggestion service filters them out.
# FC-2d-vii-c: 'artist' retired — artist identity is acquisition-derived
# (image_record.artist_id), never ML-inferred. Raw predictions are still
# stored at STORE_FLOOR but artist never surfaces.
SURFACED_CATEGORIES = {"character", "copyright", "general"}
# 'artist' retired in FC-2d-vii-c — artist identity is acquisition-derived
# (image_record.artist_id), never ML-inferred. 'copyright' retired
# 2026-06-01 — operator doesn't use the copyright tag-kind; fandom is
# this app's franchise/series concept (per TagsView.vue's doc comment).
# Raw predictions for both categories still get stored at STORE_FLOOR but
# don't surface in suggestions.
SURFACED_CATEGORIES = {"character", "general"}
# ImageNet preprocessing constants (per Camie v2 onnx_inference.py).
_IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
-140
View File
@@ -1,140 +0,0 @@
"""FC-3b platforms registry — the single source of truth for what
FabledCurator supports.
Lifted from GallerySubscriber's
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
URL patterns match GS exactly so the existing browser extension
hits FC unmodified.
"""
from dataclasses import dataclass
from typing import Literal
@dataclass(frozen=True)
class PlatformInfo:
key: str
name: str
description: str
auth_type: Literal["cookies", "token"]
requires_auth: bool
url_pattern: str
url_examples: list[str]
default_config: dict
notes: str | None = None
# Common defaults used across most platforms; embedded per-platform
# below so per-platform overrides remain explicit.
_DEFAULTS = {
"sleep": 3.0,
"sleep_request": 1.5,
"skip_existing": True,
"save_metadata": True,
"timeout": 3600,
}
PLATFORMS: dict[str, PlatformInfo] = {
"patreon": PlatformInfo(
key="patreon",
name="Patreon",
description="Download posts from Patreon creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?patreon\.com/",
url_examples=[
"https://www.patreon.com/example_artist",
"https://www.patreon.com/user?u=12345678",
],
default_config={**_DEFAULTS, "content_types": ["images", "attachments"]},
),
"subscribestar": PlatformInfo(
key="subscribestar",
name="SubscribeStar",
description="Download posts from SubscribeStar creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?subscribestar\.(com|adult)/",
url_examples=[
"https://subscribestar.adult/example_artist",
"https://www.subscribestar.com/example_artist",
],
default_config={**_DEFAULTS, "content_types": ["all"]},
),
"hentaifoundry": PlatformInfo(
key="hentaifoundry",
name="Hentai Foundry",
description="Download artwork from Hentai Foundry artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?hentai-foundry\.com/",
url_examples=[
"https://www.hentai-foundry.com/user/example_artist",
"https://www.hentai-foundry.com/pictures/user/example_artist",
],
default_config={**_DEFAULTS, "content_types": ["pictures"]},
),
"discord": PlatformInfo(
key="discord",
name="Discord",
description="Download attachments from Discord channels",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?discord\.com/channels/",
url_examples=["https://discord.com/channels/123456789/987654321"],
default_config={**_DEFAULTS, "content_types": ["all"]},
notes="Requires Discord user token (not bot token).",
),
"pixiv": PlatformInfo(
key="pixiv",
name="Pixiv",
description="Download artwork from Pixiv artists",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?pixiv\.net/",
url_examples=[
"https://www.pixiv.net/users/12345678",
"https://www.pixiv.net/en/users/12345678",
],
default_config={**_DEFAULTS, "content_types": ["all"]},
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
),
"deviantart": PlatformInfo(
key="deviantart",
name="DeviantArt",
description="Download artwork from DeviantArt artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?deviantart\.com/",
url_examples=[
"https://www.deviantart.com/example-artist",
"https://www.deviantart.com/example-artist/gallery",
],
default_config={**_DEFAULTS, "content_types": ["gallery"]},
),
}
def known_platform_keys() -> frozenset[str]:
return frozenset(PLATFORMS.keys())
def auth_type_for(platform: str) -> str | None:
info = PLATFORMS.get(platform)
return info.auth_type if info else None
def to_dict(info: PlatformInfo) -> dict:
return {
"key": info.key,
"name": info.name,
"description": info.description,
"auth_type": info.auth_type,
"requires_auth": info.requires_auth,
"url_pattern": info.url_pattern,
"url_examples": info.url_examples,
"default_config": info.default_config,
"notes": info.notes,
}
@@ -0,0 +1,95 @@
"""FC-3b platforms registry — single source of truth for what
FabledCurator supports + where each platform's quirks live.
Adding a new platform: drop a new module `<platform>.py` next to this
one, declare an `INFO = PlatformInfo(...)`, add the import + entry in
PLATFORMS below. Sidecar parsing, cookie materialization, and
`/api/platforms` pick it up automatically.
Lifted from GallerySubscriber's
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
URL patterns match GS exactly so the existing browser extension
hits FC unmodified.
"""
from .base import (
DEFAULT_DESCRIPTION_KEYS,
DEFAULT_EXTERNAL_POST_ID_KEYS,
PlatformInfo,
)
from .deviantart import INFO as _DEVIANTART
from .discord import INFO as _DISCORD
from .hentaifoundry import INFO as _HENTAIFOUNDRY
from .patreon import INFO as _PATREON
from .pixiv import INFO as _PIXIV
from .subscribestar import INFO as _SUBSCRIBESTAR
PLATFORMS: dict[str, PlatformInfo] = {
info.key: info
for info in (
_PATREON,
_SUBSCRIBESTAR,
_HENTAIFOUNDRY,
_DISCORD,
_PIXIV,
_DEVIANTART,
)
}
def known_platform_keys() -> frozenset[str]:
return frozenset(PLATFORMS.keys())
def auth_type_for(platform: str) -> str | None:
info = PLATFORMS.get(platform)
return info.auth_type if info else None
def to_dict(info: PlatformInfo) -> dict:
"""Serialize a PlatformInfo to a JSON-safe dict for /api/platforms.
Behavioral fields (callables, sidecar-chain overrides) are
intentionally omitted — they aren't useful to API consumers.
"""
return {
"key": info.key,
"name": info.name,
"description": info.description,
"auth_type": info.auth_type,
"requires_auth": info.requires_auth,
"url_pattern": info.url_pattern,
"url_examples": info.url_examples,
"default_config": info.default_config,
"notes": info.notes,
}
def external_post_id_keys_for(platform: str | None) -> tuple[str, ...]:
"""Resolve the external_post_id lookup chain for a given platform,
falling back to the module default when the platform isn't
registered or hasn't overridden the chain."""
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.external_post_id_keys is not None:
return info.external_post_id_keys
return DEFAULT_EXTERNAL_POST_ID_KEYS
def description_keys_for(platform: str | None) -> tuple[str, ...]:
"""Resolve the description body lookup chain for a given platform."""
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.description_keys is not None:
return info.description_keys
return DEFAULT_DESCRIPTION_KEYS
__all__ = [
"PLATFORMS",
"PlatformInfo",
"auth_type_for",
"description_keys_for",
"external_post_id_keys_for",
"known_platform_keys",
"to_dict",
]
+107
View File
@@ -0,0 +1,107 @@
"""PlatformInfo dataclass + shared defaults + small helpers.
Per-platform modules import from here, register their PlatformInfo via
INFO, optionally attaching `derive_post_url` and/or `augment_cookies`
callables for behavior that diverges from gallery-dl's mainline shape
(Patreon).
Adding a new platform: drop a new module under `services/platforms/`,
declare an INFO, and add it to the import list in
`services/platforms/__init__.py`. Sidecar parsing, cookie
materialization, and the /api/platforms response pick it up
automatically.
"""
from collections.abc import Callable
from dataclasses import dataclass
from typing import Literal
# Sidecar parsing defaults. Per-platform PlatformInfo entries can
# override these by setting `external_post_id_keys=` /
# `description_keys=`. Most don't need to — the defaults already cover
# every platform FC supports.
#
# external_post_id chain: `post_id` MUST come before `id` because
# SubscribeStar gallery-dl puts the per-attachment id in `id` and the
# actual post id in `post_id`; picking `id` first fragments
# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
# no `post_id` so `id` still wins for them; HF uses `index`, Discord
# uses `message_id` — all reached via the remaining chain entries.
# (Banked 2026-05-27 during the sidecar audit.)
DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
"post_id", "id", "index", "message_id",
)
# Description body chain: Discord's gallery-dl extractor uses `message`
# (no `content`); appended to the chain so Discord posts surface body
# text.
DEFAULT_DESCRIPTION_KEYS: tuple[str, ...] = (
"content", "description", "caption", "message",
)
@dataclass(frozen=True)
class PlatformInfo:
# --- Identity / metadata ---
key: str
name: str
description: str
auth_type: Literal["cookies", "token"]
requires_auth: bool
url_pattern: str
url_examples: list[str]
default_config: dict
notes: str | None = None
# --- Sidecar parsing overrides ---
# Each is None to mean "use the module default above"; a platform
# only sets one of these when its sidecar shape genuinely differs.
external_post_id_keys: tuple[str, ...] | None = None
description_keys: tuple[str, ...] | None = None
# --- Behavioral hooks ---
# Synthesize a post permalink from sidecar data. Required when
# gallery-dl's `url` field is the file/CDN URL rather than the post
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
# `url` field (patreon, deviantart).
derive_post_url: Callable[[dict], str | None] | None = None
# Post-process the materialized cookies.txt for gallery-dl. Used by
# platforms whose server gates or extractor quirks need synthetic
# cookies the extension can't capture (subscribestar age cookie, HF
# host-only PHPSESSID duplicate). None = no-op.
augment_cookies: Callable[[str], str] | None = None
def str_id_value(v) -> str | None:
"""Coerce a JSON scalar id into a non-empty string, rejecting bool
(Python's bool is an int subclass so `isinstance(True, int)` is
True; without this guard a sidecar with `"id": true` would produce
external_post_id="True")."""
if isinstance(v, bool):
return None
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
def str_field(v) -> str | None:
"""Same idea as str_id_value but for plain string fields (no int
coercion)."""
if isinstance(v, str) and v.strip():
return v.strip()
return None
# Shared gallery-dl invocation defaults. Embedded in each platform's
# default_config (with platform-specific overrides) so per-platform
# choices stay explicit.
# Note: the gallery-dl `skip` value (tick "exit:20" vs backfill True) is
# NOT here — it's derived from Source.backfill_runs_remaining at download
# time. See plan #544 / gallery_dl.TICK_SKIP_VALUE,BACKFILL_SKIP_VALUE.
GD_DEFAULTS = {
"sleep": 3.0,
"sleep_request": 1.5,
"save_metadata": True,
"timeout": 3600,
}
@@ -0,0 +1,23 @@
"""DeviantArt — no exercised quirks yet.
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
audit, so we don't know yet whether DA's gallery-dl sidecars are
well-behaved or have their own quirks. When DA gets exercised for the
first time, add `derive_post_url` / `augment_cookies` here as needed.
"""
from .base import GD_DEFAULTS, PlatformInfo
INFO = PlatformInfo(
key="deviantart",
name="DeviantArt",
description="Download artwork from DeviantArt artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?deviantart\.com/",
url_examples=[
"https://www.deviantart.com/example-artist",
"https://www.deviantart.com/example-artist/gallery",
],
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
)
+38
View File
@@ -0,0 +1,38 @@
"""Discord — one quirk + one already-default.
post_url: gallery-dl's `url` is the CDN attachment URL. The "permalink"
for a Discord message uses the (server, channel, message) triple via
`discord.com/channels/<server>/<channel>/<message>`. Note that
permalinks are only resolvable for users in the same server — public
access doesn't work — but the URL is still useful to the operator
in-app.
Description body is in `message` not `content`. That's already covered
by the default description chain in base.py (DEFAULT_DESCRIPTION_KEYS
ends with `message`). No description_keys override needed.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
sid = str_id_value(data.get("server_id"))
cid = str_id_value(data.get("channel_id"))
mid = str_id_value(data.get("message_id"))
if sid and cid and mid:
return f"https://discord.com/channels/{sid}/{cid}/{mid}"
return None
INFO = PlatformInfo(
key="discord",
name="Discord",
description="Download attachments from Discord channels",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?discord\.com/channels/",
url_examples=["https://discord.com/channels/123456789/987654321"],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
notes="Requires Discord user token (not bot token).",
derive_post_url=derive_post_url,
)
@@ -0,0 +1,83 @@
"""HentaiFoundry — two quirks colocated.
1. post_url: HF sidecars omit `url` entirely; `src` is the image URL.
Synthesize the permalink from `user` + `index`
(/pictures/user/<user>/<index>).
2. augment_cookies: gallery-dl's HF extractor checks
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching. The extension's pre-v1.0.5
`cookies.js` aggressively rewrote every captured cookie to the
leading-dot subdomain-wide form (`.hentai-foundry.com`), which fails
the exact lookup even though the cookie IS sent on actual HTTP
requests (RFC 6265 subdomain matching). The extractor falls into
an unauthenticated `?enterAgree=1` HEAD that 401s. Inject host-only
duplicates of PHPSESSID + YII_CSRF_TOKEN so the lookup succeeds.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_field, str_id_value
_HOST_ONLY_NAMES = ("PHPSESSID", "YII_CSRF_TOKEN")
def derive_post_url(data: dict) -> str | None:
user = str_field(data.get("user")) or str_field(data.get("artist"))
idx = str_id_value(data.get("index"))
if user and idx:
return f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
return None
def augment_cookies(netscape: str) -> str:
body = netscape.rstrip("\n")
if not body:
return netscape
lines = body.split("\n")
existing_host_only: set[str] = set()
by_name: dict[str, list[str]] = {}
for raw in lines:
if not raw or raw.startswith("#"):
continue
parts = raw.split("\t")
if len(parts) < 7:
continue
domain, _flag, _path, _secure, _exp, name, _value = parts[:7]
if name not in _HOST_ONLY_NAMES:
continue
if domain == "www.hentai-foundry.com":
existing_host_only.add(name)
elif domain in (".hentai-foundry.com", "hentai-foundry.com"):
by_name.setdefault(name, []).append(raw)
appended: list[str] = []
for name in _HOST_ONLY_NAMES:
if name in existing_host_only or name not in by_name:
continue
# Duplicate the first subdomain-wide line as host-only on
# www.hentai-foundry.com. Same value + expiry; flag=FALSE marks
# the entry host-only in netscape format.
parts = by_name[name][0].split("\t")
parts[0] = "www.hentai-foundry.com"
parts[1] = "FALSE"
appended.append("\t".join(parts[:7]))
if not appended:
return netscape
return body + "\n" + "\n".join(appended) + "\n"
INFO = PlatformInfo(
key="hentaifoundry",
name="Hentai Foundry",
description="Download artwork from Hentai Foundry artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?hentai-foundry\.com/",
url_examples=[
"https://www.hentai-foundry.com/user/example_artist",
"https://www.hentai-foundry.com/pictures/user/example_artist",
],
default_config={**GD_DEFAULTS, "content_types": ["pictures"]},
derive_post_url=derive_post_url,
augment_cookies=augment_cookies,
)
+23
View File
@@ -0,0 +1,23 @@
"""Patreon — no quirks. The reference platform.
Patreon's gallery-dl sidecars are the well-behaved baseline: `url` is a
real permalink, `id` is the post id, `title` and `content` are
populated. No cookie quirks (session cookies are domain-wide). No
derivation overrides.
"""
from .base import GD_DEFAULTS, PlatformInfo
INFO = PlatformInfo(
key="patreon",
name="Patreon",
description="Download posts from Patreon creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?patreon\.com/",
url_examples=[
"https://www.patreon.com/example_artist",
"https://www.patreon.com/user?u=12345678",
],
default_config={**GD_DEFAULTS, "content_types": ["images", "attachments"]},
)
+32
View File
@@ -0,0 +1,32 @@
"""Pixiv — one quirk.
post_url: gallery-dl's `url` is the image URL on `i.pximg.net`. The
post permalink follows /artworks/<id>. external_post_id (= `id`) was
already correct, so no override there.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
pid = str_id_value(data.get("id"))
if pid:
return f"https://www.pixiv.net/artworks/{pid}"
return None
INFO = PlatformInfo(
key="pixiv",
name="Pixiv",
description="Download artwork from Pixiv artists",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?pixiv\.net/",
url_examples=[
"https://www.pixiv.net/users/12345678",
"https://www.pixiv.net/en/users/12345678",
],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
derive_post_url=derive_post_url,
)
@@ -0,0 +1,62 @@
"""SubscribeStar — three quirks colocated.
1. external_post_id: gallery-dl puts the per-attachment id in `id`
(e.g. 711509) and the actual post id in `post_id` (e.g. 360360).
The default chain in base.py already prefers `post_id`; this module
doesn't need to override it but the comment lives here too so a
future reader knows the chain's order was driven by this platform.
2. post_url: gallery-dl's `url` is the file CDN URL
(`/post_uploads?payload=...`). Synthesize the post permalink from
`post_id`.
3. augment_cookies: the server gates artist pages behind a
`_personalization_id` age-confirmation cookie that the user can't
easily refresh — SubscribeStar's frontend JS uses localStorage to
suppress the age popup once dismissed. gallery-dl's own login flow
sidesteps this by setting `18_plus_agreement_generic=true` on
`.subscribestar.adult`; we mirror that for cookies captured via the
extension.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
pid = str_id_value(data.get("post_id"))
if pid:
return f"https://www.subscribestar.com/posts/{pid}"
return None
def augment_cookies(netscape: str) -> str:
if "18_plus_agreement_generic" in netscape:
return netscape
# Far-future expiry — gallery-dl's own login flow sets this with no
# explicit expiry; the server only checks presence/value.
expiry = 4102444800 # 2100-01-01 UTC
line = "\t".join([
".subscribestar.adult", "TRUE", "/", "TRUE",
str(expiry), "18_plus_agreement_generic", "true",
])
body = netscape.rstrip("\n")
if not body:
body = "# Netscape HTTP Cookie File"
return body + "\n" + line + "\n"
INFO = PlatformInfo(
key="subscribestar",
name="SubscribeStar",
description="Download posts from SubscribeStar creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?subscribestar\.(com|adult)/",
url_examples=[
"https://subscribestar.adult/example_artist",
"https://www.subscribestar.com/example_artist",
],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
derive_post_url=derive_post_url,
augment_cookies=augment_cookies,
)
+120 -31
View File
@@ -56,38 +56,67 @@ class PostFeedService:
artist_id: int | None = None,
platform: str | None = None,
limit: int = 24,
direction: str = "older",
) -> dict:
"""Paginate the feed from `cursor`. direction='older' walks back in
time (default, infinite-scroll down); direction='newer' walks forward
(scroll up in an anchored view). Items are always returned in feed
(descending) order; `next_cursor` points to the far edge in the
requested direction (null when exhausted)."""
if limit < 1 or limit > 100:
raise ValueError("limit must be between 1 and 100")
if direction not in ("older", "newer"):
raise ValueError("direction must be 'older' or 'newer'")
sort_key = _sort_key()
# Artist via the denormalized Post.artist_id (alembic 0030);
# Source via LEFT JOIN since post.source_id can now be NULL for
# filesystem-imported posts with no live subscription. A
# platform= filter implicitly excludes NULL-source posts (they
# have no platform); an artist_id= filter still surfaces them
# because Post.artist_id is always set.
stmt = (
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
)
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
stmt = stmt.where(Post.artist_id == artist_id)
if platform is not None:
stmt = stmt.where(Source.platform == platform)
if cursor:
cur_ts, cur_id = decode_cursor(cursor)
stmt = stmt.where(
or_(
if direction == "older":
stmt = stmt.where(or_(
sort_key < cur_ts,
and_(sort_key == cur_ts, Post.id < cur_id),
)
)
))
else:
stmt = stmt.where(or_(
sort_key > cur_ts,
and_(sort_key == cur_ts, Post.id > cur_id),
))
stmt = stmt.order_by(sort_key.desc(), Post.id.desc()).limit(limit + 1)
if direction == "older":
stmt = stmt.order_by(sort_key.desc(), Post.id.desc())
else:
stmt = stmt.order_by(sort_key.asc(), Post.id.asc())
stmt = stmt.limit(limit + 1)
rows = (await self.session.execute(stmt)).all()
has_more = len(rows) > limit
rows = rows[:limit]
if direction == "newer":
# Fetched ascending (closest-newer first); flip to feed order.
rows = list(reversed(rows))
next_cursor: str | None = None
if len(rows) > limit:
last_post, _, _ = rows[limit - 1]
last_key = last_post.post_date or last_post.downloaded_at
next_cursor = encode_cursor(last_key, last_post.id)
rows = rows[:limit]
if has_more and rows:
# Far edge in the travel direction: oldest row going older,
# newest row going newer (rows is descending for display).
edge_post = rows[-1][0] if direction == "older" else rows[0][0]
edge_key = edge_post.post_date or edge_post.downloaded_at
next_cursor = encode_cursor(edge_key, edge_post.id)
post_ids = [p.id for p, _, _ in rows]
thumbs_map = await self._thumbnails_for(post_ids)
@@ -99,17 +128,63 @@ class PostFeedService:
]
return {"items": items, "next_cursor": next_cursor}
async def around(
self,
*,
post_id: int,
artist_id: int | None = None,
platform: str | None = None,
limit: int = 12,
) -> dict | None:
"""A window centered on `post_id`: up to `limit` newer posts + the
post + up to `limit` older posts, in feed (descending) order, with a
cursor for each end. Returns None if the post doesn't exist."""
anchor = (await self.session.execute(
select(Post, Artist, Source)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if anchor is None:
return None
anchor_post, anchor_artist, anchor_source = anchor
anchor_key = anchor_post.post_date or anchor_post.downloaded_at
anchor_cursor = encode_cursor(anchor_key, anchor_post.id)
older = await self.scroll(
cursor=anchor_cursor, artist_id=artist_id, platform=platform,
limit=limit, direction="older",
)
newer = await self.scroll(
cursor=anchor_cursor, artist_id=artist_id, platform=platform,
limit=limit, direction="newer",
)
thumbs_map = await self._thumbnails_for([anchor_post.id])
atts_map = await self._attachments_for([anchor_post.id])
anchor_item = self._to_dict(
anchor_post, anchor_artist, anchor_source, thumbs_map, atts_map,
)
return {
"items": newer["items"] + [anchor_item] + older["items"],
"cursor_older": older["next_cursor"],
"cursor_newer": newer["next_cursor"],
"anchor_id": anchor_post.id,
}
async def get_post(self, post_id: int) -> dict | None:
row = (await self.session.execute(
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if row is None:
return None
post, artist, source = row
thumbs_map = await self._thumbnails_for([post.id])
# Detail endpoint returns the FULL image list for PostModal's
# masonry grid — feed query still caps at THUMBNAIL_LIMIT via
# the default arg.
thumbs_map = await self._thumbnails_for([post.id], limit=None)
atts_map = await self._attachments_for([post.id])
item = self._to_dict(post, artist, source, thumbs_map, atts_map)
item["description_full"] = html_to_plain(post.description)
@@ -117,21 +192,28 @@ class PostFeedService:
# --- composition helpers ---------------------------------------------
async def _thumbnails_for(self, post_ids: list[int]) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to 6], "more": int}.
async def _thumbnails_for(
self, post_ids: list[int], *, limit: int | None = THUMBNAIL_LIMIT,
) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to limit], "more": int}.
Selects THUMBNAIL_LIMIT+1 images per post via window function so we
can detect overflow in a single query.
Selects up to `limit` images per post via window function so we
can detect overflow in a single query. Pass `limit=None` to
return ALL thumbnails per post (used by `get_post` for PostModal's
masonry grid; the feed pass keeps the default cap so payloads
stay small).
"""
if not post_ids:
return {}
# Rank images within each post and fetch only the top THUMBNAIL_LIMIT+1.
# Rank images within each post; cap at `limit` rows per post when
# limit is set, return all when limit is None.
ranked = (
select(
ImageRecord.id,
ImageRecord.primary_post_id,
ImageRecord.sha256,
ImageRecord.mime,
ImageRecord.thumbnail_path,
func.row_number().over(
partition_by=ImageRecord.primary_post_id,
order_by=ImageRecord.id.asc(),
@@ -143,19 +225,20 @@ class PostFeedService:
.where(ImageRecord.primary_post_id.in_(post_ids))
.subquery()
)
rows = (await self.session.execute(
select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.total,
).where(ranked.c.rn <= THUMBNAIL_LIMIT)
)).all()
stmt = select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.thumbnail_path, ranked.c.total,
)
if limit is not None:
stmt = stmt.where(ranked.c.rn <= limit)
rows = (await self.session.execute(stmt)).all()
out: dict[int, dict] = {pid: {"thumbs": [], "more": 0} for pid in post_ids}
for img_id, pid, sha, mime, total in rows:
for img_id, pid, sha, mime, tp, total in rows:
entry = out.setdefault(pid, {"thumbs": [], "more": 0})
entry["thumbs"].append({
"image_id": img_id,
"thumbnail_url": thumbnail_url(sha, mime),
"thumbnail_url": thumbnail_url(tp, sha, mime),
"mime": mime,
})
# `total` is constant per partition; overflow = total - THUMBNAIL_LIMIT.
@@ -183,7 +266,7 @@ class PostFeedService:
return out
def _to_dict(
self, post: Post, artist: Artist, source: Source,
self, post: Post, artist: Artist, source: Source | None,
thumbs_map: dict, atts_map: dict,
) -> dict:
plain_full = html_to_plain(post.description) if post.description else None
@@ -192,6 +275,9 @@ class PostFeedService:
else:
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
# `source` is null for filesystem-imported posts with no live
# subscription (alembic 0030). Frontend renders that as a
# "filesystem import" affordance instead of a platform chip.
return {
"id": post.id,
"external_post_id": post.external_post_id,
@@ -202,7 +288,10 @@ class PostFeedService:
"description_plain": description_plain,
"description_truncated": truncated,
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"source": {"id": source.id, "platform": source.platform},
"source": (
{"id": source.id, "platform": source.platform}
if source is not None else None
),
"thumbnails": thumbs_entry["thumbs"],
"thumbnails_more": thumbs_entry["more"],
"attachments": atts_map.get(post.id, []),
+12 -6
View File
@@ -70,11 +70,15 @@ class ProvenanceService:
rec = await self.session.get(ImageRecord, image_id)
if rec is None:
return None
# Artist via Post.artist_id (alembic 0030); Source via LEFT JOIN
# since both Post.source_id and ImageProvenance.source_id can be
# NULL for filesystem-imported content. Frontend renders source=
# null as "filesystem import."
stmt = (
select(ImageProvenance, Post, Source, Artist)
.join(Post, Post.id == ImageProvenance.post_id)
.join(Source, Source.id == ImageProvenance.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == ImageProvenance.source_id)
.where(ImageProvenance.image_record_id == image_id)
.order_by(ImageProvenance.captured_at.asc(),
ImageProvenance.id.asc())
@@ -90,7 +94,7 @@ class ProvenanceService:
"captured_at": ip.captured_at.isoformat()
if ip.captured_at else None,
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
}
for ip, post, src, art in rows
@@ -99,10 +103,12 @@ class ProvenanceService:
}
async def for_post(self, post_id: int) -> dict | None:
# Same LEFT JOIN to Source — get_post must succeed for a
# NULL-source post.
stmt = (
select(Post, Source, Artist)
.join(Source, Source.id == Post.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == Post.source_id)
.where(Post.id == post_id)
)
row = (await self.session.execute(stmt)).first()
@@ -111,7 +117,7 @@ class ProvenanceService:
post, src, art = row
return {
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
"attachments": await self._attachments_for_posts([post.id]),
}
+106
View File
@@ -0,0 +1,106 @@
"""Layer-2 one-shot re-download remediation for corrupt imported files.
When an import fails on a file that came from a known, pollable
subscription Source, deleting the bad copy and re-running the source's
downloader can fetch a fresh, unblemished copy. This only helps when:
- the corruption is in transit / on disk (not at the source), AND
- the file resolves to an ENABLED Source with a real feed URL
(a `sidecar:<platform>:<slug>` synthetic anchor is not pollable),
AND
- we haven't already re-fetched this task once (bounded by
ImportTask.refetched so source-side corruption can't loop).
Filesystem-only imports with no resolvable Source return 'no_source'
the operator's only remediation there is to replace the file on disk.
Operator-requested 2026-05-28 (Layer 2).
"""
import json
import logging
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from ..models import Artist, ImportTask, Source
from ..utils.paths import derive_top_level_artist
from ..utils.sidecar import find_sidecar, parse_sidecar
from ..utils.slug import slugify
log = logging.getLogger(__name__)
def resolve_refetch_source(
session: Session, source_path: str, import_root: Path,
) -> Source | None:
"""Find an enabled, real-URL Source for the file's (artist, platform),
or None when nothing re-pollable resolves."""
path = Path(source_path)
sc = find_sidecar(path)
if sc is None:
return None
try:
data = json.loads(sc.read_text("utf-8"))
except (OSError, json.JSONDecodeError):
return None
if not isinstance(data, dict):
return None
sd = parse_sidecar(data)
if not sd.platform:
return None
artist_name = derive_top_level_artist(path, import_root)
if not artist_name:
return None
artist = session.execute(
select(Artist).where(Artist.slug == slugify(artist_name))
).scalar_one_or_none()
if artist is None:
return None
src = session.execute(
select(Source)
.where(
Source.artist_id == artist.id,
Source.platform == sd.platform,
Source.enabled.is_(True),
)
.order_by(Source.id.asc())
).scalars().first()
if src is None:
return None
if (src.url or "").startswith("sidecar:"):
return None # synthetic anchor — not a pollable feed
return src
def attempt_refetch(
session: Session, task: ImportTask, import_root: Path,
) -> dict:
"""Delete the corrupt file, mark the task refetched, and trigger ONE
source re-check. Idempotent/bounded: a task already refetched (or
with no resolvable Source) is a no-op. Commits."""
if task.refetched:
return {"status": "already_refetched"}
src = resolve_refetch_source(session, task.source_path, import_root)
if src is None:
return {"status": "no_source"}
# Remove the bad copy so gallery-dl's archive-skip re-fetches it on
# the source re-check instead of skipping the still-present corrupt
# file.
try:
Path(task.source_path).unlink(missing_ok=True)
except OSError as exc:
log.warning("refetch unlink failed for %s: %s", task.source_path, exc)
task.refetched = True
session.add(task)
session.commit()
# Lazy import to avoid a tasks→services→tasks import cycle at module
# load. download_source.delay() is sync-safe in any context.
from ..tasks.download import download_source
download_source.delay(src.id)
return {"status": "refetch_queued", "source_id": src.id}
+153 -5
View File
@@ -9,15 +9,33 @@ from __future__ import annotations
from datetime import UTC, datetime, timedelta
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from ..models import Artist, ImportSettings, Source
from ..models import AppSetting, Artist, ImportSettings, Source
MIN_INTERVAL_SECONDS = 60
MAX_INTERVAL_SECONDS = 86400
MAX_BACKOFF_EXPONENT = 6
# AppSetting key stamped every time the Beat tick fires (see scan.py). The
# tick runs every 60s; the UI flags the scheduler as stalled if the last
# stamp is older than a few minutes.
SCHEDULER_LAST_TICK_KEY = "scheduler_last_tick_at"
# AppSetting key prefix for per-platform rate-limit cooldowns. When a
# download surfaces ErrorType.RATE_LIMITED, every other source on the same
# platform is deferred for PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS so the next
# scan tick doesn't fire a burst of due same-platform sources back into the
# same limit. Per-source consecutive_failures backoff still applies on top
# of this — but this is PREVENTIVE (kills the same-tick burst from N due
# sources hammering the platform at once), while consecutive_failures is
# REACTIVE (slows the offender down over many cycles). Operator-confirmed
# 2026-05-30.
PLATFORM_COOLDOWN_KEY_PREFIX = "platform_cooldown:"
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS = 900 # 15 min
def compute_effective_interval(
source: Source, artist: Artist, settings: ImportSettings,
@@ -40,10 +58,76 @@ def compute_effective_interval(
return max(MIN_INTERVAL_SECONDS, min(MAX_INTERVAL_SECONDS, raw))
async def set_platform_cooldown(
session: AsyncSession, platform: str,
seconds: int = PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS,
) -> None:
"""Stamp a cooldown expiry on the given platform so select_due_sources
skips every source on that platform until it expires.
Called when a download surfaces ErrorType.RATE_LIMITED so the other
sources on the same platform don't all retry into the same rate limit.
Caller is responsible for committing the session.
Uses INSERT...ON CONFLICT DO UPDATE so two concurrent workers hitting
the same platform's rate limit don't race: a SELECT-then-INSERT pattern
would let the loser's whole transaction (including the source-health
update + event finalize) roll back on a unique-violation, stranding
that event. Atomic upsert avoids that.
"""
now = datetime.now(UTC)
expires_at = (now + timedelta(seconds=seconds)).isoformat()
key = f"{PLATFORM_COOLDOWN_KEY_PREFIX}{platform}"
stmt = pg_insert(AppSetting.__table__).values(
key=key, value=expires_at, updated_at=now,
).on_conflict_do_update(
index_elements=["key"],
set_={"value": expires_at, "updated_at": now},
)
await session.execute(stmt)
async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime]:
"""Return {platform: expires_at} for platforms whose cooldown is still
in the future. Expired rows are ignored (a future maintenance sweep can
delete them; they don't affect routing decisions on their own).
Exposed beyond scheduler_service so the manual check endpoint
(`/api/sources/<id>/check`) can defer bulk retries that would bowl
into the same rate limit the cooldown is preventing.
"""
rows = (await session.execute(
select(AppSetting.key, AppSetting.value)
.where(AppSetting.key.startswith(PLATFORM_COOLDOWN_KEY_PREFIX))
)).all()
if not rows:
return {}
now = datetime.now(UTC)
active: dict[str, datetime] = {}
for key, value in rows:
try:
expires_at = datetime.fromisoformat(value)
except (ValueError, TypeError):
continue
if expires_at > now:
active[key[len(PLATFORM_COOLDOWN_KEY_PREFIX):]] = expires_at
return active
async def select_due_sources(session: AsyncSession) -> list[Source]:
"""Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
Never-checked sources (last_checked_at IS NULL) are always due.
Never-checked sources (last_checked_at IS NULL) are always due. Sources
whose platform is currently in a rate-limit cooldown are excluded — the
cooldown is the preventive half of the burst-prevention pair (per-source
consecutive_failures backoff handles the offending source itself).
Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
sources go first, then the longest-since-checked, so the most overdue
sources hit Celery's FIFO download queue first. Anti-starvation: if
queue throughput ever falls below the tick rate, a freshly-rerun source
can't keep cutting in line ahead of one that hasn't been checked at all.
Operator-confirmed 2026-05-30.
"""
rows = (await session.execute(
select(Source)
@@ -51,15 +135,17 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
.join(Artist, Source.artist_id == Artist.id)
.where(Source.enabled.is_(True))
.where(Artist.auto_check.is_(True))
.order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
)).scalars().all()
settings = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
cooldowns = await active_platform_cooldowns(session)
settings = await ImportSettings.load(session)
now = datetime.now(UTC)
due: list[Source] = []
for s in rows:
if s.platform in cooldowns:
continue
interval = compute_effective_interval(s, s.artist, settings)
if s.last_checked_at is None:
due.append(s)
@@ -78,3 +164,65 @@ def compute_next_check_at(
return None
interval = compute_effective_interval(source, artist, settings)
return source.last_checked_at + timedelta(seconds=interval)
async def record_tick(session: AsyncSession) -> None:
"""Stamp the current time on the SCHEDULER_LAST_TICK_KEY AppSetting.
Called once per Beat tick so the UI can prove the scheduler is alive.
Commits its own write so the stamp survives even if the rest of the
tick errors out.
"""
now_iso = datetime.now(UTC).isoformat()
row = (await session.execute(
select(AppSetting).where(AppSetting.key == SCHEDULER_LAST_TICK_KEY)
)).scalar_one_or_none()
if row is None:
session.add(AppSetting(key=SCHEDULER_LAST_TICK_KEY, value=now_iso))
else:
row.value = now_iso
await session.commit()
async def scheduler_status(session: AsyncSession) -> dict:
"""Summarise scheduler health for the dashboard.
Returns last_tick_at (when Beat last fired), next_due_at (earliest
upcoming scheduled check across enabled auto-check sources), due_now
(how many are due right now), and auto_sources (total under schedule).
"""
last_tick_at = (await session.execute(
select(AppSetting.value).where(AppSetting.key == SCHEDULER_LAST_TICK_KEY)
)).scalar_one_or_none()
rows = (await session.execute(
select(Source)
.options(selectinload(Source.artist))
.join(Artist, Source.artist_id == Artist.id)
.where(Source.enabled.is_(True))
.where(Artist.auto_check.is_(True))
)).scalars().all()
settings = await ImportSettings.load(session)
now = datetime.now(UTC)
due_now = 0
next_due_at: datetime | None = None
for s in rows:
if s.last_checked_at is None:
due_now += 1
continue
nca = compute_next_check_at(s, s.artist, settings)
if nca is None or nca <= now:
due_now += 1
elif next_due_at is None or nca < next_due_at:
next_due_at = nca
cooldowns = await active_platform_cooldowns(session)
return {
"last_tick_at": last_tick_at,
"next_due_at": next_due_at.isoformat() if next_due_at else None,
"due_now": due_now,
"auto_sources": len(rows),
"platform_cooldowns": {p: dt.isoformat() for p, dt in cooldowns.items()},
}
+2 -1
View File
@@ -63,6 +63,7 @@ class SeriesService:
ImageRecord.sha256,
ImageRecord.mime,
ImageRecord.path,
ImageRecord.thumbnail_path,
)
.join(ImageRecord, ImageRecord.id == SeriesPage.image_id)
.where(SeriesPage.series_tag_id == series_tag_id)
@@ -75,7 +76,7 @@ class SeriesService:
{
"image_id": r.image_id,
"page_number": r.page_number,
"thumbnail_url": thumbnail_url(r.sha256, r.mime),
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
"image_url": f"/images/{r.path.split('/images/', 1)[-1]}",
}
for r in rows
+1 -1
View File
@@ -34,7 +34,7 @@ class ShowcaseService:
"mime": r.mime,
"width": r.width,
"height": r.height,
"thumbnail_url": thumbnail_url(r.sha256, r.mime),
"thumbnail_url": thumbnail_url(r.thumbnail_path, r.sha256, r.mime),
}
for r in rows
]
+61 -9
View File
@@ -59,9 +59,11 @@ class SourceRecord:
config_overrides: dict | None
last_checked_at: str | None
last_error: str | None
error_type: str | None
check_interval_override: int | None
consecutive_failures: int
next_check_at: str | None
backfill_runs_remaining: int
def to_dict(self) -> dict:
return {
@@ -75,9 +77,11 @@ class SourceRecord:
"config_overrides": self.config_overrides,
"last_checked_at": self.last_checked_at,
"last_error": self.last_error,
"error_type": self.error_type,
"check_interval_override": self.check_interval_override,
"consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at,
"backfill_runs_remaining": self.backfill_runs_remaining,
}
@@ -85,6 +89,11 @@ class SourceRecord:
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
# Plan #544 follow-up: newly created enabled sources pre-arm backfill so
# their first N polls walk gallery-dl's full post history with the longer
# timeout (matches the manual "Deep scan" button's default).
NEW_SOURCE_BACKFILL_RUNS = 3
class SourceService:
def __init__(self, session: AsyncSession):
@@ -120,9 +129,7 @@ class SourceService:
return config
async def _load_settings(self) -> ImportSettings:
return (await self.session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
return await ImportSettings.load(self.session)
def _build_record(
self, source: Source, artist: Artist, settings: ImportSettings,
@@ -139,9 +146,11 @@ class SourceService:
config_overrides=source.config_overrides,
last_checked_at=source.last_checked_at.isoformat() if source.last_checked_at else None,
last_error=source.last_error,
error_type=source.error_type,
check_interval_override=source.check_interval_override,
consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None,
backfill_runs_remaining=source.backfill_runs_remaining or 0,
)
async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -151,14 +160,27 @@ class SourceService:
settings = await self._load_settings()
return self._build_record(source, artist, settings)
async def list(self, artist_id: int | None = None) -> list[SourceRecord]:
stmt = (
select(Source, Artist)
.join(Artist, Artist.id == Source.artist_id)
.order_by(Artist.name.asc(), Source.id.asc())
)
async def list(
self, artist_id: int | None = None, failing: bool = False,
include_synthetic: bool = False,
) -> list[SourceRecord]:
stmt = select(Source, Artist).join(Artist, Artist.id == Source.artist_id)
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
if not include_synthetic:
# Pre-alembic-0030 sidecar synthetic anchors
# have url='sidecar:<platform>:<slug>' and exist only to give
# imported Posts a NOT-NULL Source FK. They aren't pollable
# feeds; the Subscriptions UI used to render them as phantom
# subscriptions. Hide by default.
stmt = stmt.where(~Source.url.like("sidecar:%"))
if failing:
# Worst-first so the rollup card surfaces the loudest failures.
stmt = stmt.where(Source.consecutive_failures > 0).order_by(
Source.consecutive_failures.desc(), Artist.name.asc(),
)
else:
stmt = stmt.order_by(Artist.name.asc(), Source.id.asc())
rows = (await self.session.execute(stmt)).all()
settings = await self._load_settings()
return [self._build_record(s, a, settings) for s, a in rows]
@@ -190,10 +212,21 @@ class SourceService:
select(func.count(Source.id)).where(Source.artist_id == artist_id)
)).scalar_one()
# Plan #544 follow-up: a freshly added subscription has no archive
# yet, so the first few polls would walk the full post history in
# tick mode and trip exit:20 after ~20 contiguous archive hits —
# except there are none yet, so tick mode would walk forever and
# blow the wall-clock cap. Pre-arm backfill so the initial syncs
# use the longer timeout + skip:True walk. Tick mode resumes once
# the budget is spent or the queue drains.
# Disabled sources (incl. sidecar synthetics, url='sidecar:...')
# are never polled, so leave their counter at 0.
backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
source = Source(
artist_id=artist_id, platform=platform, url=url,
enabled=enabled, config_overrides=config_overrides,
check_interval_override=check_interval_override,
backfill_runs_remaining=backfill_runs,
)
self.session.add(source)
try:
@@ -253,6 +286,25 @@ class SourceService:
await self.session.commit()
return await self._row_to_record(source)
async def set_backfill_runs(
self, source_id: int, runs: int,
) -> SourceRecord:
"""Plan #544: arm a source for backfill mode. The next `runs`
download runs will use gallery-dl's full-walk config (skip: True
+ 30-min timeout) instead of the catch-up default. Runs must be
1..10 — bigger is rejected to keep the operator from accidentally
setting a runaway budget."""
if not isinstance(runs, int) or runs < 1 or runs > 10:
raise ValueError("runs must be an integer in [1, 10]")
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
source.backfill_runs_remaining = runs
await self.session.commit()
return await self._row_to_record(source)
async def delete(self, source_id: int) -> None:
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
@@ -115,12 +115,17 @@ class TagDirectoryService:
.subquery()
)
stmt = (
select(sub.c.tag_id, ImageRecord.sha256, ImageRecord.mime)
select(
sub.c.tag_id,
ImageRecord.sha256,
ImageRecord.mime,
ImageRecord.thumbnail_path,
)
.join(ImageRecord, ImageRecord.id == sub.c.image_record_id)
.where(sub.c.rn <= 3)
.order_by(sub.c.tag_id, sub.c.rn)
)
out: dict[int, list[str]] = {}
for tag_id, sha, mime in (await self.session.execute(stmt)).all():
out.setdefault(tag_id, []).append(thumbnail_url(sha, mime))
for tag_id, sha, mime, tp in (await self.session.execute(stmt)).all():
out.setdefault(tag_id, []).append(thumbnail_url(tp, sha, mime))
return out
+17 -7
View File
@@ -5,6 +5,7 @@ from dataclasses import dataclass
from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Tag, TagKind, image_tag
@@ -86,9 +87,12 @@ class TagService:
f"fandom_id {fandom_id} does not reference a fandom tag"
)
# Upsert via INSERT ... ON CONFLICT DO NOTHING. We can't use the
# uniqueness index name directly (it's a partial coalesce-based
# expression), so we re-select after insert.
# Audit 2026-06-02: race-safe upsert via savepoint +
# IntegrityError recovery. The partial uniqueness index on
# (name, kind, COALESCE(fandom_id, -1)) catches concurrent
# inserts; without the savepoint the outer transaction would
# poison and the calling request crashes. Mirrors
# importer._get_or_create.
stmt = (
select(Tag)
.where(Tag.name == name)
@@ -101,10 +105,16 @@ class TagService:
if existing:
return existing
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
return new_tag
sp = await self.session.begin_nested()
try:
new_tag = Tag(name=name, kind=kind, fandom_id=fandom_id)
self.session.add(new_tag)
await self.session.flush()
await sp.commit()
return new_tag
except IntegrityError:
await sp.rollback()
return (await self.session.execute(stmt)).scalar_one()
async def autocomplete(
self,
+21
View File
@@ -0,0 +1,21 @@
"""Per-invocation async session factory for Celery task modules.
Async engine connections are bound to the event loop. Each Celery task
runs its async body under a fresh ``asyncio.run()`` loop, so it needs its
own engine created (and disposed) within that loop — a process-wide async
engine would reuse loop-bound connections across tasks and raise "attached
to a different loop". So unlike the process-wide sync engine in
``_sync_engine.py``, this returns a fresh engine per call; the caller
disposes it (``await engine.dispose()``) when its loop ends.
"""
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from ..config import get_config
def async_session_factory():
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
cfg = get_config()
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
+7 -8
View File
@@ -240,15 +240,16 @@ def prune_backups() -> dict:
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
Status='running' / 'restoring' rows are never pruned — the
recover_stalled_backup_runs sweep flips truly-stuck ones to
'error' first. (Earlier docstring claimed the FC-3i TaskRun sweep
handled those, but TaskRun cleanup never touched BackupRun rows.
Audit 2026-06-02 added the dedicated sweep.)
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s = ImportSettings.load_sync(session)
for kind, keep in (
("db", s.backup_db_keep_last_n),
("images", s.backup_images_keep_last_n),
@@ -286,9 +287,7 @@ def backup_db_nightly() -> dict:
either {'skipped': '<reason>'} or {'dispatched': '<task_id>'}."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s = ImportSettings.load_sync(session)
nightly_enabled = s.backup_db_nightly_enabled
configured_hour = s.backup_db_nightly_hour_utc
if not nightly_enabled:
+4 -16
View File
@@ -3,12 +3,9 @@
import asyncio
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from ..celery_app import celery
from ..config import get_config
from ..models import ImportSettings
from ..services.credential_crypto import CredentialCrypto
from ..services.credential_service import CredentialService
@@ -16,18 +13,13 @@ from ..services.download_service import DownloadService
from ..services.gallery_dl import GalleryDLService
from ..services.importer import Importer
from ..services.thumbnailer import Thumbnailer
from ._async_session import async_session_factory
from .import_file import _sync_session_factory
IMAGES_ROOT = Path("/images")
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
def _async_session_factory():
cfg = get_config()
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
@celery.task(
name="backend.app.tasks.download.download_source",
bind=True,
@@ -44,13 +36,11 @@ def download_source(self, source_id: int) -> int:
"""Returns the DownloadEvent.id."""
async def _run():
async_factory, async_engine = _async_session_factory()
async_factory, async_engine = async_session_factory()
SyncFactory = _sync_session_factory()
try:
with SyncFactory() as sync_session:
settings = sync_session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
settings = ImportSettings.load_sync(sync_session)
rate_limit = settings.download_rate_limit_seconds
validate_files = settings.download_validate_files
@@ -64,9 +54,7 @@ def download_source(self, source_id: int) -> int:
async with async_factory() as async_session:
cred_service = CredentialService(async_session, crypto)
with SyncFactory() as sync_session:
sync_settings = sync_session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
sync_settings = ImportSettings.load_sync(sync_session)
importer = Importer(
session=sync_session,
images_root=IMAGES_ROOT,
+76 -29
View File
@@ -64,30 +64,13 @@ def _mark_failed(session, task, error_msg: str) -> None:
pass
@celery.task(
name="backend.app.tasks.import_file.import_media_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=360,
)
def import_media_file(self, import_task_id: int) -> dict:
"""Returns a dict so the eager-mode tests can assert without DB.
Decorator notes:
- autoretry_for: transient DB / filesystem errors retry with
exponential backoff (5s base, jitter, max 3 attempts). On final
give-up the task raises and acks_late=True (set globally on the
Celery app) does NOT redeliver — the recovery sweep catches the
row instead.
- soft_time_limit (300s) raises SoftTimeLimitExceeded in this
process so the task can mark its row failed before being killed.
- time_limit (360s) is the hard cap; SIGKILL if the soft signal
was swallowed.
def _run_import_task(import_task_id: int) -> dict:
"""Shared body for import_media_file + import_archive_file. The two
tasks differ ONLY in their Celery time limits (a single media file
is sub-second; an archive runs the full per-member pipeline inline
for every member and can take many minutes). Both flip the row to
'processing', dispatch to `_do_import`, and honor the
flip-to-terminal resilience contract.
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
@@ -103,24 +86,88 @@ def import_media_file(self, import_task_id: int) -> dict:
try:
return _do_import(session, task, import_task_id)
except SoftTimeLimitExceeded:
_mark_failed(session, task, "soft_time_limit exceeded (>300s)")
_mark_failed(session, task, "soft_time_limit exceeded")
raise
except (OperationalError, DBAPIError, OSError):
# Retryable per the decorator; do NOT mark failed (let
# autoretry have a clean go at it). If autoretry exhausts,
# the row stays 'processing' and the maintenance sweep
# flips it within 5 min.
# flips it.
raise
except Exception as exc: # noqa: BLE001 — pipeline crash, mark + re-raise
_mark_failed(session, task, f"{type(exc).__name__}: {exc}")
raise
@celery.task(
name="backend.app.tasks.import_file.import_media_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=360,
)
def import_media_file(self, import_task_id: int) -> dict:
"""Import ONE media file (or non-media → PostAttachment). Sub-second
for the common case; the tight 5-min soft limit keeps a genuinely
stuck single-file import detectable fast.
Decorator notes:
- autoretry_for: transient DB / filesystem errors retry with
exponential backoff (5s base, jitter, max 3 attempts). On final
give-up the task raises and acks_late=True (set globally on the
Celery app) does NOT redeliver — the recovery sweep catches the
row instead.
- soft_time_limit (300s) raises SoftTimeLimitExceeded in-process
so the task can mark its row failed before being killed.
- time_limit (360s) is the hard SIGKILL cap.
"""
return _run_import_task(import_task_id)
@celery.task(
name="backend.app.tasks.import_file.import_archive_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
# Archives run the full per-member pipeline (sha256 + pHash + dedup
# query + copy + provenance) for EVERY media member inline, under a
# single task budget. A multi-hundred-member archive blows the
# 5-min media limit. soft=30min / hard=35min sizes for a large
# archive. Operator-flagged 2026-05-28 (target 1645019 hit the old
# shared 300s soft limit). The recovery sweep gives this task its
# own 40-min threshold via maintenance.TASK_STUCK_THRESHOLD_MINUTES
# so it isn't preempted while legitimately grinding through members.
soft_time_limit=1800,
time_limit=2100,
)
def import_archive_file(self, import_task_id: int) -> dict:
"""Import an archive: extract + run the per-member media pipeline for
every member inline, then preserve the archive as a PostAttachment.
Same body as import_media_file (dispatch is by file kind inside
Importer.import_one); split out purely for the larger time budget."""
return _run_import_task(import_task_id)
def enqueue_import(task_id: int, task_type: str) -> None:
"""Route an ImportTask to the right Celery task by its task_type.
Single source of truth for the media-vs-archive dispatch so the
scan, retry, and recovery-requeue paths stay in sync."""
if task_type == "archive":
import_archive_file.delay(task_id)
else:
import_media_file.delay(task_id)
def _do_import(session, task, import_task_id: int) -> dict:
"""Actual work, called from inside the resilience wrapper."""
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
settings = ImportSettings.load_sync(session)
import_root = Path(settings.import_scan_path)
batch = session.get(ImportBatch, task.batch_id)
deep = bool(batch and batch.scan_mode == "deep")
+474 -39
View File
@@ -1,22 +1,56 @@
"""Periodic maintenance: recover stuck import tasks, garbage-collect old finished tasks."""
import logging
import os
import subprocess
from datetime import UTC, datetime, timedelta
from pathlib import Path
from PIL import Image
from sqlalchemy import delete, select, update
from sqlalchemy import Integer, and_, cast, delete, func, or_, select, update
from ..celery_app import celery
from ..models import DownloadEvent, ImageRecord, ImportSettings, ImportTask, TaskRun
from ..models import (
BackupRun,
DownloadEvent,
ImageRecord,
ImportBatch,
ImportSettings,
ImportTask,
LibraryAuditRun,
Source,
TaskRun,
)
from ..utils.phash import compute_phash
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
STUCK_THRESHOLD_MINUTES = 5
# Archive ImportTasks run the per-member pipeline inline for every
# member (import_archive_file: soft=30min/hard=35min). The ImportTask
# 'processing' recovery sweep must give them a longer threshold or it
# re-queues a legitimately-running archive mid-import (double-process).
# 40 min = 5-min buffer past the archive task's hard kill.
# Operator-flagged 2026-05-28 (target 1645019, a big archive).
ARCHIVE_STUCK_THRESHOLD_MINUTES = 40
# Poison-pill cap. After being recovered (re-queued from a stuck
# 'processing' state) MAX_RECOVERY_ATTEMPTS-1 times, the next sweep
# marks the row 'failed' instead of looping. 3 = two recoveries then
# give up. A row reaches this only if it leaves NO terminal flip each
# run — i.e. it hard-crashes the worker (OOM/segfault/SIGKILL), the
# signature of a corrupt or oversized input. Caught exceptions already
# flip to terminal 'failed' and never enter this loop.
MAX_RECOVERY_ATTEMPTS = 3
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
# DownloadEvent (pending|running) recovery threshold. download_source has
# time_limit=1200s (20 min); 30 min is 10 min past that, so a legitimately-
# running task is never killed by the sweep. Operator-confirmed 2026-05-29
# after 43 sources stranded at "last check never" by the in-flight guard.
DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7
PHASH_PAGE = 500
VERIFY_PAGE = 200
@@ -24,17 +58,78 @@ FFPROBE_TIMEOUT_SECONDS = 10
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# Audit 2026-06-02: per-entity recovery sweep thresholds. Each must be
# > the entity's longest legitimate runtime (its task's time_limit + a
# small buffer) so the sweep never flags in-flight work.
#
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
# 2h15m gives a 10-min buffer.
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
# Import batches finalize only after every child ImportTask hits a
# terminal state. The recovery sweep targets the case where every
# task is done but the batch never got its closing UPDATE
# (orchestrator crashed at the wrong instant). 2h is well past any
# realistic single-batch import.
IMPORT_BATCH_STALL_THRESHOLD_MINUTES = 120
# Retention windows (terminal rows older than these get deleted by
# the daily prune sweeps). 30 days = operator-flagged "useful for
# triage for a few weeks, then noise."
LIBRARY_AUDIT_KEEP_DAYS = 30
IMPORT_BATCH_KEEP_DAYS = 30
# Overrides for recover_stalled_task_runs (the TaskRun 'running' sweep).
# Tasks/queues that legitimately run longer than the default 5-min
# threshold need their own larger value, else the sweep marks in-flight
# work 'error' before it finishes. Each value MUST be ≥ the relevant
# task.time_limit + a small buffer. task_name overrides take precedence
# over queue overrides.
#
# ml queue: tag_and_embed video branch (≈20 GPU ops); time_limit=1200.
# import_archive_file: shares the 'import' queue with the fast
# single-file import_media_file, so it needs a task-name override
# (the import queue itself stays at the 5-min default for single
# files); time_limit=2100.
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"ml": 25,
# Audit 2026-06-02 — maintenance/scan queues run tasks that
# legitimately exceed the 5-min default (verify_integrity at 70m
# hard, scan_directory at 70m hard, apply_allowlist_tags /
# recompute_centroids / backfill_phash at 35m hard). 75 min lives
# above the longest of those and the per-task overrides below
# cover the outliers (backups, library audit).
"maintenance": 75,
"scan": 75,
}
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"backend.app.tasks.import_file.import_archive_file": 40,
# Backup images runs hours, not minutes (6.5h hard limit). The
# task-name override beats the queue's 75-min default so a
# legitimately-running backup isn't flagged.
"backend.app.tasks.backup.backup_images_task": 420,
"backend.app.tasks.backup.restore_images_task": 420,
# Library audit scans the full library — 2h hard limit.
"backend.app.tasks.library_audit.scan_library_for_rule": 130,
}
@celery.task(name="backend.app.tasks.maintenance.recover_interrupted_tasks")
def recover_interrupted_tasks() -> int:
"""Recover stuck ImportTask rows. Two distinct stuck states:
1. 'processing' > 5 min — worker crash mid-import. Re-queue via
.delay() and let the import retry. Was 30 min historically;
tightened 2026-05-24 after operator hit a 2224-row zombie pile.
import_media_file is sub-second for the vast majority of files and
capped at the per-task soft_time_limit (5 min), so anything still
'processing' after that window is a confirmed crash.
1. 'processing' too long — worker crash mid-import. Re-queue via
enqueue_import (routing media vs archive) and let the import
retry. Threshold is task-type-aware: media files are sub-second
and capped at the 5-min soft limit, so STUCK_THRESHOLD_MINUTES
(5) means a confirmed crash; archives run the per-member
pipeline inline (import_archive_file, 35-min hard limit) so they
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40) to avoid re-queueing a
still-running archive. (Media was tightened from 30 min to 5
2026-05-24 after a 2224-row zombie pile; archive split out
2026-05-28.)
2. 'pending' or 'queued' > 30 min — enqueue-phase crash. scan_directory
creates rows with status='pending' (commit), then in a second pass
@@ -51,7 +146,8 @@ def recover_interrupted_tasks() -> int:
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
processing_cutoff = now - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
media_cutoff = now - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
archive_cutoff = now - timedelta(minutes=ARCHIVE_STUCK_THRESHOLD_MINUTES)
orphan_cutoff = now - timedelta(minutes=ORPHAN_PENDING_THRESHOLD_MINUTES)
with SessionLocal() as session:
# Both sweeps used to be SELECT ids → UPDATE WHERE id IN (...) which
@@ -59,20 +155,67 @@ def recover_interrupted_tasks() -> int:
# tens of thousands of rows (operator hit it 2026-05-26 after the
# /import deep scan piled up orphans). Folding the SELECT into the
# UPDATE eliminates the IN-list entirely. RETURNING gives us back
# exactly the ids that flipped so the stuck sweep can still
# .delay() each one.
stuck_result = session.execute(
# exactly the (id, task_type) pairs that flipped so the requeue
# can route media vs archive correctly.
#
# Media + archive get separate cutoffs: a single media file is
# sub-second so 5 min means crash; an archive runs the per-member
# pipeline inline and can legitimately take up to its 35-min hard
# limit, so it gets ARCHIVE_STUCK_THRESHOLD_MINUTES (40) to avoid
# re-queueing a still-running archive.
stuck_predicate = and_(
ImportTask.status == "processing",
or_(
and_(ImportTask.task_type != "archive",
ImportTask.started_at < media_cutoff),
and_(ImportTask.task_type == "archive",
ImportTask.started_at < archive_cutoff),
),
)
# POISON-PILL CIRCUIT BREAKER (Layer 1, 2026-05-28). A row that
# leaves no terminal flip (hard worker crash: OOM/segfault/SIGKILL
# on a corrupt or oversized input) gets re-queued by this sweep —
# and would loop forever, re-crashing the worker each pass,
# without a cap. Once a row has already been recovered
# MAX_RECOVERY_ATTEMPTS-1 times, stop re-queueing it and mark it
# 'failed' with a diagnostic so the operator can find + replace
# the offending file. This UPDATE runs FIRST so the rows it
# claims drop out of 'processing' before the re-queue pass.
poison_result = session.execute(
update(ImportTask)
.where(ImportTask.status == "processing")
.where(ImportTask.started_at < processing_cutoff)
.where(stuck_predicate)
.where(ImportTask.recovery_count >= MAX_RECOVERY_ATTEMPTS - 1)
.values(
status="queued",
started_at=None,
error="recovered from stuck state",
status="failed",
finished_at=now,
error=(
f"crashed or stalled the worker {MAX_RECOVERY_ATTEMPTS} "
f"times without completing — likely a corrupt or "
f"oversized input. Not re-queued. Inspect/replace the "
f"file, then retry via /api/import/retry-failed."
),
)
.returning(ImportTask.id)
)
stuck_ids = [row[0] for row in stuck_result.all()]
poison_ids = [r[0] for r in poison_result.all()]
# Re-queue the remaining stuck rows (under the cap) and bump
# their recovery_count. RETURNING (id, task_type) so the requeue
# routes media vs archive correctly.
stuck_result = session.execute(
update(ImportTask)
.where(stuck_predicate)
.where(ImportTask.recovery_count < MAX_RECOVERY_ATTEMPTS - 1)
.values(
status="queued",
started_at=None,
recovery_count=ImportTask.recovery_count + 1,
error="recovered from stuck state",
)
.returning(ImportTask.id, ImportTask.task_type)
)
stuck = stuck_result.all()
orphan_result = session.execute(
update(ImportTask)
@@ -80,6 +223,11 @@ def recover_interrupted_tasks() -> int:
.where(ImportTask.created_at < orphan_cutoff)
.values(
status="failed",
# Without finished_at, cleanup_old_tasks (`WHERE
# finished_at < cutoff`) never reaps these rows —
# orphan-swept rows would become permanent table
# tenants. Audit 2026-06-02.
finished_at=now,
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
@@ -91,12 +239,34 @@ def recover_interrupted_tasks() -> int:
session.commit()
if stuck_ids:
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
if stuck:
from .import_file import enqueue_import
for tid, task_type in stuck:
enqueue_import(tid, task_type)
return len(stuck_ids) + orphan_count
# Layer-2 auto re-download (env-gated, default OFF). For each
# poison-pill row that resolves to a pollable Source, delete the
# bad file and trigger ONE source re-check to fetch a fresh
# copy. Bounded by ImportTask.refetched so source-side
# corruption can't loop. The 'failed' row stays as history; the
# re-downloaded file re-imports as a fresh task on the next scan.
if poison_ids and os.environ.get("FC_AUTO_REFETCH_CORRUPT", "0") == "1":
from ..models import ImportSettings
from ..services.refetch_service import attempt_refetch
import_root = Path(session.execute(
select(ImportSettings.import_scan_path)
.where(ImportSettings.id == 1)
).scalar_one())
for pid in poison_ids:
ptask = session.get(ImportTask, pid)
if ptask is None:
continue
try:
attempt_refetch(session, ptask, import_root)
except Exception as exc: # noqa: BLE001 — best-effort
log.warning("auto-refetch failed for task %s: %s", pid, exc)
return len(stuck) + len(poison_ids) + orphan_count
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_tasks")
@@ -121,18 +291,29 @@ def cleanup_old_tasks() -> int:
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_task_runs")
def recover_stalled_task_runs() -> int:
"""Flip task_run rows stuck in 'running' for >STUCK_THRESHOLD_MINUTES
to 'error'. FC-3i.
"""Flip task_run rows stuck in 'running' past their queue-specific
threshold to 'error'. FC-3i.
A row gets stuck when the worker dies without emitting
task_postrun / task_failure (e.g. OOM, container restart between
signals, signal handler raised+logged). Shares the 5-min threshold
with recover_interrupted_tasks for consistency.
signals, signal handler raised+logged). The default 5-min threshold
fits short-lived queues (import/thumbnail/download); queues that
legitimately run longer tasks (ml-video, deep scans) get their
own larger threshold via QUEUE_STUCK_THRESHOLD_MINUTES so the
sweep doesn't preempt them.
Runs once per distinct threshold value: each pass updates rows
whose queue maps to that threshold.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
now = datetime.now(UTC)
override_tasks = set(TASK_STUCK_THRESHOLD_MINUTES.keys())
override_queues = set(QUEUE_STUCK_THRESHOLD_MINUTES.keys())
total = 0
def _flag(minutes, *extra_where):
cutoff = now - timedelta(minutes=minutes)
stmt = (
update(TaskRun)
.where(TaskRun.status == "running")
.where(TaskRun.started_at < cutoff)
@@ -140,14 +321,50 @@ def recover_stalled_task_runs() -> int:
status="error",
error_type="RecoverySweep",
error_message=(
f"no completion signal received within "
f"{STUCK_THRESHOLD_MINUTES} min"
f"no completion signal received within {minutes} min"
),
finished_at=now,
# Matches celery_signals.finalize's
# int((now - started_at).total_seconds() * 1000)
# — sweep-closed rows now carry duration like
# normally-finalized rows. Audit 2026-06-02.
duration_ms=cast(
func.extract("epoch", now - TaskRun.started_at) * 1000,
Integer,
),
finished_at=datetime.now(UTC),
)
)
for w in extra_where:
stmt = stmt.where(w)
return session.execute(stmt).rowcount or 0
with SessionLocal() as session:
# Precedence: task_name override → queue override → default.
# Each pass excludes rows claimed by a higher-precedence pass so
# every row is touched at most once.
# 1. Per-task-name overrides (e.g. import_archive_file, which
# shares the 'import' queue with fast single-file imports).
for task_name, minutes in TASK_STUCK_THRESHOLD_MINUTES.items():
total += _flag(minutes, TaskRun.task_name == task_name)
# 2. Per-queue overrides, excluding the override task-names.
for queue, minutes in QUEUE_STUCK_THRESHOLD_MINUTES.items():
wheres = [TaskRun.queue == queue]
if override_tasks:
wheres.append(TaskRun.task_name.notin_(override_tasks))
total += _flag(minutes, *wheres)
# 3. Default — everything not claimed above.
default_wheres = []
if override_queues:
default_wheres.append(TaskRun.queue.notin_(override_queues))
if override_tasks:
default_wheres.append(TaskRun.task_name.notin_(override_tasks))
total += _flag(STUCK_THRESHOLD_MINUTES, *default_wheres)
session.commit()
return result.rowcount or 0
return total
@celery.task(name="backend.app.tasks.maintenance.prune_task_runs")
@@ -184,7 +401,12 @@ def prune_task_runs() -> dict:
return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@celery.task(name="backend.app.tasks.maintenance.backfill_phash")
@celery.task(
name="backend.app.tasks.maintenance.backfill_phash",
# Audit 2026-06-02 — keyset-paginated phash recompute over the whole
# library; legitimately runs >5 min on large libraries.
soft_time_limit=1800, time_limit=2100,
)
def backfill_phash() -> int:
"""Recompute phash for stored images that have none (imported before
FC-2d-i+ii). Keyset-paginated by id (restart-safe), NULL-only fill,
@@ -262,7 +484,13 @@ def _verify_one(path: Path, expected_sha: str, mime: str, sha_fn) -> str:
return "failed_verification"
@celery.task(name="backend.app.tasks.maintenance.verify_integrity")
@celery.task(
name="backend.app.tasks.maintenance.verify_integrity",
# Audit 2026-06-02 — full library sha256 + decode probe; on 100k-image
# libraries this runs an hour or more. Match the maintenance queue's
# recovery threshold (75 min) with 30s buffer below.
soft_time_limit=3600, time_limit=4200,
)
def verify_integrity() -> int:
"""Verify every ImageRecord file: sha256 recompute + decode/probe
(PIL for images; ffprobe for videos). Writes integrity_status
@@ -299,6 +527,215 @@ def verify_integrity() -> int:
return total
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_download_events")
def recover_stalled_download_events() -> int:
"""Recover DownloadEvent rows stuck pending/running past the worker hard kill.
The scan tick (scheduler_service.select_due_sources →
tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending')
and fires download_source.delay(). If that task dies before finalizing the
event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind
on the 1200s hard time_limit — the event stays in-flight forever. The next
tick then skips that source because of the in-flight guard (scan.py:168)
and Source.last_checked_at never updates; the operator sees "last check
never" in the Subscriptions health column, permanently.
This sweep flips matching events to 'error', stamps each affected Source's
last_checked_at + last_error and bumps consecutive_failures (once per
source, not per event — backoff is exponential on that count so an N-event
bump would inflate the next interval by 2^N for no reason). The source
becomes re-queueable on the next tick and the health dot goes amber.
Operator-confirmed 2026-05-29 (43-row strand pile in production).
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=DOWNLOAD_STALL_THRESHOLD_MINUTES)
msg = "stranded by recovery sweep (no terminal status after time_limit)"
with SessionLocal() as session:
# UPDATE...RETURNING the source_ids in one round trip — keeps us off
# the psycopg 65535-param ceiling that SELECT-then-UPDATE-WHERE-IN
# would hit on a large strand pile.
result = session.execute(
update(DownloadEvent)
.where(DownloadEvent.status.in_(["pending", "running"]))
.where(DownloadEvent.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
.returning(DownloadEvent.source_id)
)
returned = result.all()
if not returned:
session.commit()
return 0
events_recovered = len(returned)
source_ids = list({row.source_id for row in returned})
session.execute(
update(Source)
.where(Source.id.in_(source_ids))
.values(
consecutive_failures=Source.consecutive_failures + 1,
last_error=msg,
last_checked_at=now,
)
)
session.commit()
log.info(
"recover_stalled_download_events: recovered %d events across %d sources",
events_recovered, len(source_ids),
)
return events_recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_backup_runs")
def recover_stalled_backup_runs() -> int:
"""Flip BackupRun rows stuck in running/restoring past the hard limit
to error. Audit 2026-06-02.
prune_backups (FC-3h) used to claim the FC-3i task_run sweep handled
these — but that sweep only flips TaskRun rows, not the BackupRun
artifact rows. A SIGKILL'd backup left BackupRun stuck forever
(dashboard showed phantom in-flight backups, keep_last_n offset
arithmetic skewed because zombies sat outside the ok/error window).
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
)
with SessionLocal() as session:
result = session.execute(
update(BackupRun)
.where(BackupRun.status.in_(["running", "restoring"]))
.where(BackupRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("recover_stalled_backup_runs: recovered %d rows", recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_library_audit_runs")
def recover_stalled_library_audit_runs() -> int:
"""Flip LibraryAuditRun rows stuck in running past the hard limit
to error. Audit 2026-06-02.
LibraryAuditRun.status='running' was protected by an exclusive
guard in start_audit_run — a SIGKILL'd run would block all future
audits until manual DB surgery. (The guard is now age-aware, but
this sweep is what makes that work in practice.)
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
)
with SessionLocal() as session:
result = session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at < cutoff)
.values(status="error", finished_at=now, error=msg)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_library_audit_runs: recovered %d rows", recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
def recover_stalled_import_batches() -> int:
"""Finalize ImportBatch rows stuck in running past the hard limit
when NO outstanding ImportTask remains. Audit 2026-06-02.
A batch row finalizes only after every child task hits a terminal
state. The orphan case: scanner crashed between the last task's
completion and the batch's closing UPDATE. The
`/api/import/status` route then surfaces the batch as 'active'
indefinitely while `/api/system/stats` (which uses the same
EXISTS predicate we apply below) correctly returns null.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=IMPORT_BATCH_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
# Batches still 'running' past the cutoff whose tasks are all
# terminal — there's no outstanding work, so flip the batch
# too. Mirrors the EXISTS predicate the active-batch surfaces use.
result = session.execute(
update(ImportBatch)
.where(ImportBatch.status == "running")
.where(ImportBatch.started_at < cutoff)
.where(
~select(ImportTask.id)
.where(
ImportTask.batch_id == ImportBatch.id,
ImportTask.status.in_(["pending", "queued", "processing"]),
)
.exists()
)
.values(status="complete", finished_at=now)
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_import_batches: finalized %d zombie batches",
recovered,
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.prune_library_audit_runs")
def prune_library_audit_runs() -> int:
"""Daily retention: delete terminal LibraryAuditRun rows older than
LIBRARY_AUDIT_KEEP_DAYS. Never touches 'running'. Audit 2026-06-02.
Audit rows carry matched_ids JSONB blobs that can hold tens of
thousands of ids; without retention these accumulate.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=LIBRARY_AUDIT_KEEP_DAYS)
with SessionLocal() as session:
result = session.execute(
delete(LibraryAuditRun)
.where(LibraryAuditRun.status.in_(["ready", "applied", "cancelled", "error"]))
.where(LibraryAuditRun.finished_at < cutoff)
)
session.commit()
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.prune_import_batches")
def prune_import_batches() -> int:
"""Daily retention: delete terminal ImportBatch rows older than
IMPORT_BATCH_KEEP_DAYS. Cascade-deletes child ImportTask rows via
the model relationship. Never touches 'running'. Audit 2026-06-02.
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(days=IMPORT_BATCH_KEEP_DAYS)
with SessionLocal() as session:
# ORM-level delete here (not Core delete) so the
# ImportBatch->tasks cascade fires; Core delete would skip it.
old_batches = session.execute(
select(ImportBatch)
.where(ImportBatch.status.in_(["complete", "cancelled"]))
.where(ImportBatch.finished_at < cutoff)
).scalars().all()
for batch in old_batches:
session.delete(batch)
session.commit()
return len(old_batches)
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_download_events")
def cleanup_old_download_events() -> int:
"""FC-3d: delete terminal DownloadEvent rows older than the configured
@@ -311,9 +748,7 @@ def cleanup_old_download_events() -> int:
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
settings = ImportSettings.load_sync(session)
retention_days = settings.download_event_retention_days
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
result = session.execute(
-175
View File
@@ -1,175 +0,0 @@
"""FC-5 run_migration Celery task.
Dispatches to the right migrator based on `kind`. Updates MigrationRun
row's status/counts/finished_at as it runs. Failures set status='error'
with the error message preserved.
kinds: gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup
(backup + rollback retired 2026-05-24 → see /api/system/backup/*)
"""
from __future__ import annotations
import asyncio
import logging
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from ..celery_app import celery
from ..config import get_config
from ..models import MigrationRun
from ..services.credential_crypto import CredentialCrypto
from ..services.migrators import cleanup as cleanup_mod
from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
def _async_session_factory():
cfg = get_config()
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
async def _update_run(
db: AsyncSession, run_id: int, *,
status: str | None = None, counts: dict | None = None,
error: str | None = None, finished_at: datetime | None = None,
metadata_patch: dict | None = None,
) -> None:
run = (await db.execute(
select(MigrationRun).where(MigrationRun.id == run_id)
)).scalar_one()
if status is not None:
run.status = status
if counts is not None:
run.counts = counts
if error is not None:
run.error = error
if finished_at is not None:
run.finished_at = finished_at
if metadata_patch:
run.metadata_ = {**(run.metadata_ or {}), **metadata_patch}
await db.commit()
async def _run_async(run_id: int, kind: str, params: dict) -> dict:
factory, engine = _async_session_factory()
try:
async with factory() as db:
await _update_run(db, run_id, status="running")
try:
if kind in ("backup", "rollback"):
raise ValueError(
f"kind {kind!r} retired in FC-3h; "
"use /api/system/backup/* instead"
)
elif kind == "gs_ingest":
fc_crypto = CredentialCrypto(_KEY_PATH)
counts = await gs_ingest.migrate_async(
db, data=params["data"],
fc_crypto=fc_crypto,
dry_run=params.get("dry_run", False),
)
await _update_run(
db, run_id, status="ok", counts=counts,
finished_at=datetime.now(UTC),
)
return counts
elif kind == "ir_ingest":
counts = await ir_ingest.migrate_async(
db, data=params["data"],
images_root=IMAGES_ROOT,
dry_run=params.get("dry_run", False),
)
await _update_run(
db, run_id, status="ok", counts=counts,
finished_at=datetime.now(UTC),
)
return counts
elif kind == "tag_apply":
result = await tag_apply.apply_async(
db, images_root=IMAGES_ROOT,
dry_run=params.get("dry_run", False),
)
await _update_run(
db, run_id, status="ok",
counts=result["counts"],
finished_at=datetime.now(UTC),
metadata_patch={"unmatched": result["unmatched"]},
)
return result
elif kind == "ml_queue":
count = await ml_queue.queue_all_unprocessed_async(db)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": count, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
)
return {"queued": count}
elif kind == "verify":
checks = await verify.verify_async(db, expected=params.get("expected"))
sample = await verify.verify_sha256_sample(
db, sample_size=params.get("sample_size", 20),
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": sample["sample_size"],
"rows_inserted": 0, "rows_skipped": 0,
"files_copied": 0, "bytes_copied": 0,
"conflicts": sample["mismatched"] + sample["missing"]},
finished_at=datetime.now(UTC),
metadata_patch={"checks": checks, "sample": sample},
)
return {"checks": checks, "sample": sample}
elif kind == "cleanup":
slug = params.get("slug")
if not slug:
raise ValueError("cleanup requires params.slug")
result = await cleanup_mod.cleanup_artist_async(
db, slug=slug, images_root=IMAGES_ROOT,
dry_run=params.get("dry_run", False),
source_path_prefix=params.get("source_path_prefix"),
)
await _update_run(
db, run_id, status="ok",
counts=result["counts"],
finished_at=datetime.now(UTC),
metadata_patch={
"artist": result["artist"],
"summary": result["summary"],
},
)
return result
else:
raise ValueError(f"unknown kind: {kind}")
except Exception as exc:
log.exception("migration kind=%s failed", kind)
await _update_run(
db, run_id, status="error", error=str(exc),
finished_at=datetime.now(UTC),
)
raise
finally:
await engine.dispose()
@celery.task(name="backend.app.tasks.migration.run_migration", bind=True, acks_late=True)
def run_migration(self, run_id: int, kind: str, params: dict) -> dict:
"""FC-5: dispatch a migration kind. Updates MigrationRun row as it goes."""
return asyncio.run(_run_async(run_id, kind, params))
+36 -4
View File
@@ -31,8 +31,15 @@ def _is_video(path: Path) -> bool:
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=420,
# Sized for the video branch: sample 10 frames, run tagger +
# embedder on each (≈20 GPU ops vs 2 for an image). A loaded
# ml-worker can take 5-10 min on a long video; bumped from
# 5min/7min on 2026-05-28 after operator-flagged image 6288 (a
# .mp4) hit the recovery sweep at 5 min while still legitimately
# processing. Image runs return in seconds; the bump doesn't
# affect their UX.
soft_time_limit=900, # 15 min
time_limit=1200, # 20 min hard
)
def tag_and_embed(self, image_id: int) -> dict:
"""Run Camie + SigLIP on one image; store predictions + embedding;
@@ -64,6 +71,18 @@ def tag_and_embed(self, image_id: int) -> dict:
embedder = get_embedder()
if _is_video(src):
# Layer-3 isolation: ffprobe (a separate process) validates
# the container before we burn ~20 GPU ops sampling frames
# from it. A corrupt video that would crash the frame
# decoder is rejected cleanly here instead of taking down
# the ml-worker. Operator-flagged 2026-05-28.
from ..utils import safe_probe
vprobe = safe_probe.probe_video(src)
if not vprobe.ok:
return {
"status": "bad_video", "image_id": image_id,
"reason": vprobe.reason,
}
frames = _sample_video_frames(
src, int(os.environ.get("VIDEO_ML_FRAMES", "10"))
)
@@ -193,7 +212,14 @@ def backfill(self) -> int:
return enqueued
@celery.task(name="backend.app.tasks.ml.apply_allowlist_tags", bind=True)
@celery.task(
name="backend.app.tasks.ml.apply_allowlist_tags",
bind=True,
# Audit 2026-06-02 — the full-sweep mode (neither tag_id nor image_id)
# is O(images × allowlist) and legitimately runs >5 min on large
# libraries. Cap matches the maintenance queue's recovery threshold.
soft_time_limit=1800, time_limit=2100,
)
def apply_allowlist_tags(self, tag_id: int | None = None,
image_id: int | None = None) -> int:
"""Retroactively apply allowlisted tags.
@@ -322,7 +348,13 @@ def recompute_centroid(self, tag_id: int) -> bool:
return asyncio.run(_run())
@celery.task(name="backend.app.tasks.ml.recompute_centroids", bind=True)
@celery.task(
name="backend.app.tasks.ml.recompute_centroids",
bind=True,
# Audit 2026-06-02 — drifted-centroid rebuild over potentially
# hundreds of tags.
soft_time_limit=1800, time_limit=2100,
)
def recompute_centroids(self) -> int:
"""Daily: find drifted centroids, enqueue recompute_centroid for each."""
import asyncio
+24 -19
View File
@@ -12,12 +12,12 @@ from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from ..celery_app import celery
from ..config import get_config
from ..models import DownloadEvent, ImportBatch, ImportSettings, ImportTask
from ..services.scheduler_service import select_due_sources
from ..services.archive_extractor import is_archive
from ..services.scheduler_service import record_tick, select_due_sources
from ._async_session import async_session_factory
from ._sync_engine import sync_session_factory as _sync_session_factory
@@ -35,7 +35,15 @@ def _iter_import_files(import_root: Path):
yield entry
@celery.task(name="backend.app.tasks.scan.scan_directory", bind=True)
@celery.task(
name="backend.app.tasks.scan.scan_directory",
bind=True,
# Audit 2026-06-02 — large libraries make the scan legitimately long.
# Hard cap at 70 min so the corresponding QUEUE_STUCK_THRESHOLD_MINUTES
# ("scan") of 75 min always wins; soft limit gives the task a clean
# exit window before SIGKILL.
soft_time_limit=3600, time_limit=4200,
)
def scan_directory(self, triggered_by: str = "manual",
mode: str = "quick") -> int:
"""Walks the import root and creates ImportTasks. `mode` is 'quick'
@@ -44,9 +52,7 @@ def scan_directory(self, triggered_by: str = "manual",
batch id."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
settings = ImportSettings.load_sync(session)
import_root = Path(settings.import_scan_path)
batch = ImportBatch(
@@ -96,7 +102,9 @@ def scan_directory(self, triggered_by: str = "manual",
task = ImportTask(
batch_id=batch_id,
source_path=entry_str,
task_type="media",
# Archives route to import_archive_file (larger time
# budget) — they run the per-member pipeline inline.
task_type="archive" if is_archive(entry) else "media",
status="pending",
size_bytes=size,
)
@@ -115,15 +123,16 @@ def scan_directory(self, triggered_by: str = "manual",
batch.finished_at = datetime.now(UTC)
session.commit()
# Now enqueue import_media_file for each pending task.
# Now enqueue each pending task on the right Celery task
# (media vs archive) via the shared router.
from .import_file import enqueue_import
for task in session.execute(
select(ImportTask).where(ImportTask.batch_id == batch_id)
).scalars():
task.status = "queued"
session.add(task)
from .import_file import import_media_file
import_media_file.delay(task.id)
enqueue_import(task.id, task.task_type)
session.commit()
if mode == "deep":
@@ -137,16 +146,12 @@ def scan_directory(self, triggered_by: str = "manual",
# --- FC-3d: periodic source-check tick ------------------------------------
def _async_session_factory():
cfg = get_config()
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
async def _tick_due_sources_async() -> dict:
factory, engine = _async_session_factory()
factory, engine = async_session_factory()
try:
async with factory() as session:
# Prove the scheduler is alive even on empty ticks (UI reads this).
await record_tick(session)
due = await select_due_sources(session)
if not due:
return {
+85 -47
View File
@@ -20,14 +20,32 @@ IMAGES_ROOT = Path("/images")
THUMB_MAGIC_JPEG = b"\xff\xd8\xff"
THUMB_MAGIC_PNG = b"\x89PNG\r\n\x1a\n"
# Minimum file size for a thumbnail to count as valid. Anything smaller
# is almost certainly truncated/corrupt — a legitimate 400×400 JPEG@85
# bottoms out around 2KB even on a solid-color image; 400×400 PNG starts
# around 1KB. 256 bytes is well below any real thumbnail and well above
# header-only corrupt files (~8-12 bytes). Operator-flagged 2026-06-01:
# header-only corrupt files were silently passing the magic-byte check
# and backfill counted them as "ok" — so broken-image tiles in the UI
# never got regenerated even after running backfill.
MIN_THUMB_BYTES = 256
def _thumb_is_valid(path: Path) -> bool:
"""Return True iff `path` exists and starts with a JPEG or PNG magic header.
"""Return True iff `path` exists, starts with a JPEG or PNG magic
header, AND is at least MIN_THUMB_BYTES on disk.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG for
opaque sources, PNG for alpha sources. Anything else (missing file, OSError,
truncated, wrong magic) is invalid.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG
for opaque sources, PNG for alpha sources. Anything else (missing
file, OSError, truncated below the size floor, wrong magic) is
invalid and gets re-enqueued.
"""
try:
size = path.stat().st_size
except OSError:
return False
if size < MIN_THUMB_BYTES:
return False
try:
with path.open("rb") as f:
head = f.read(12)
@@ -42,6 +60,59 @@ def _thumb_is_valid(path: Path) -> bool:
return False
def _run_backfill_scan() -> dict:
"""Synchronous scan logic shared by the Celery task and the API
endpoint. Returns {enqueued, ok, regenerated, scanned}.
Operator-flagged 2026-06-01: the original task was fire-and-forget,
so the admin UI couldn't show what backfill actually found —
operator saw \"Enqueued.\" with no counts and assumed nothing was
happening. Now the API runs this synchronously and returns the
real numbers; the periodic Celery task wraps it too."""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
scanned = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
scanned += len(rows)
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {
"scanned": scanned,
"enqueued": enqueued,
"ok": ok,
"regenerated": regenerated,
}
@celery.task(
name="backend.app.tasks.thumbnail.generate_thumbnail",
bind=True,
@@ -84,48 +155,15 @@ def backfill_thumbnails(self) -> dict:
"""Scan ImageRecord and enqueue generate_thumbnail for rows whose
thumbnail is missing, gone from disk, or has wrong magic bytes.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path for
rows that point at a missing or corrupt file before enqueueing — keeps
the DB self-consistent on partial runs and makes re-runs safe.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path
for rows that point at a missing or corrupt file before enqueueing —
keeps the DB self-consistent on partial runs and makes re-runs safe.
Returns {"enqueued": N, "ok": M, "regenerated": K} where:
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL thumbnail_path
cleared (i.e. missing + corrupt)
Returns {scanned, enqueued, ok, regenerated} where:
- scanned = total rows examined
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL
thumbnail_path cleared (i.e. missing + corrupt)
"""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {"enqueued": enqueued, "ok": ok, "regenerated": regenerated}
return _run_backfill_scan()
@@ -0,0 +1,33 @@
"""Subprocess entrypoint for safe_probe.probe_archive — see safe_probe.py.
probe_archive spawns this via subprocess (not multiprocessing.Process)
because Celery's prefork worker pool runs tasks in DAEMON processes and
Python's multiprocessing forbids daemon processes from spawning children
("AssertionError: daemonic processes are not allowed to have children",
operator-flagged 2026-05-30 — every archive import failed at task
startup). subprocess has no such restriction; we still get crash-
isolation because a probe segfault/OOM exits non-zero rather than
killing the worker.
Prints a single JSON line on stdout: {"status": "ok"|"error",
"detail": "..."?}. Exit code 0 for clean outcomes; non-zero exit
(signal / OOM-kill / unhandled exception) is the poison-pill signature
the parent maps to ProbeResult(crashed=True).
"""
import json
import sys
from .safe_probe import _run_probe
def main() -> int:
if len(sys.argv) != 2:
print(json.dumps({"status": "error", "detail": "usage: <path>"}))
return 2
status, detail = _run_probe(sys.argv[1])
print(json.dumps({"status": status, "detail": detail}))
return 0
if __name__ == "__main__":
sys.exit(main())
+191
View File
@@ -0,0 +1,191 @@
"""Subprocess-isolated media probes (Layer 3 of import resilience).
A malformed video or archive can hard-crash the worker process — a
decoder OOM, a native-lib segfault, or a decompression bomb. A hard
crash leaves no terminal flip, so the recovery sweep re-queues the row
and it crashes again: a poison-pill loop (the Layer-1 cap is the
backstop, but isolating the crash is better — the file gets a clean
terminal failure and the worker never dies).
These probes run the risky read in a way that contains the blast:
- Video: `ffprobe` is a separate binary, so a crash decoding the
container kills only ffprobe (non-zero exit), never the worker. Also
returns width/height, which the importer didn't previously capture
for videos.
- Archive: an uncompressed-size guard (catches decompression bombs
before they OOM anything) plus an integrity test in a spawned child
(catches native-lib crashes on a malformed archive). A child segfault
/ OOM shows up as a non-zero exit code, not a dead worker.
Images are intentionally NOT probed here: Pillow raises (it doesn't
segfault) on the realistic corrupt-image cases, the importer already
catches that as an invalid_image skip, and a subprocess per image would
wreck deep-scan throughput on a large library. Add an image branch only
if a real image-induced worker crash is ever observed.
Operator-requested 2026-05-28 (Layer 3).
"""
import json
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
VIDEO_PROBE_TIMEOUT_SECONDS = 60
ARCHIVE_PROBE_TIMEOUT_SECONDS = 120
# Refuse archives whose total UNCOMPRESSED size exceeds this — the
# classic decompression-bomb guard (a 4 GB cap comfortably clears real
# art-pack archives while stopping a few-KB zip that expands to TB).
MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 * 1024 * 1024 * 1024
# Repo root for the subprocess cwd so `python -m backend.app.utils.*`
# resolves regardless of where Celery / pytest started. backend/app/utils
# = parents[0]; backend/app = parents[1]; backend = parents[2]; repo root
# = parents[3].
_REPO_ROOT = Path(__file__).resolve().parents[3]
_PROBE_RUNNER_MODULE = "backend.app.utils._archive_probe_runner"
@dataclass(frozen=True)
class ProbeResult:
ok: bool
# crashed=True means the probe HARD-FAILED (subprocess killed by a
# signal, OOM, or timeout) — the poison-pill signature. crashed=False
# with ok=False means a clean rejection (corrupt-but-handled,
# bomb-size-exceeded, integrity mismatch). Callers map crashed → a
# terminal 'failed', clean → a 'skipped'/'failed' of their choosing.
crashed: bool = False
reason: str | None = None
width: int | None = None
height: int | None = None
def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
"""Validate a video container + first video stream via ffprobe."""
try:
out = subprocess.run(
[
"ffprobe", "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "json", str(path),
],
capture_output=True, text=True, timeout=timeout,
)
except subprocess.TimeoutExpired:
return ProbeResult(ok=False, crashed=True, reason="ffprobe timed out")
except OSError as exc:
# ffprobe missing / not executable — environmental, not the
# file's fault. Treat as a clean non-crash failure so the import
# path can decide (it currently proceeds without dims).
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe unavailable: {exc}")
if out.returncode != 0:
return ProbeResult(
ok=False, crashed=False,
reason=f"ffprobe rejected the file: {out.stderr.strip()[:200]}",
)
try:
streams = (json.loads(out.stdout) or {}).get("streams") or []
except json.JSONDecodeError as exc:
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe output parse failed: {exc}")
if not streams:
return ProbeResult(ok=False, crashed=False, reason="no decodable video stream")
return ProbeResult(
ok=True, width=streams[0].get("width"), height=streams[0].get("height"),
)
def probe_archive(path: Path, *, timeout: float = ARCHIVE_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
"""Bomb-size guard + isolated integrity test for an archive.
Runs via subprocess (not multiprocessing.Process) because Celery's
prefork worker pool is daemon-mode and Python's multiprocessing
forbids daemon processes from spawning children ("AssertionError:
daemonic processes are not allowed to have children"). subprocess
has no such restriction and still gives the crash isolation: a probe
segfault/OOM exits non-zero rather than killing the worker.
"""
try:
result = subprocess.run(
[sys.executable, "-m", _PROBE_RUNNER_MODULE, str(path)],
capture_output=True, text=True, timeout=timeout,
cwd=str(_REPO_ROOT),
)
except subprocess.TimeoutExpired:
return ProbeResult(ok=False, crashed=True, reason="archive probe timed out")
if result.returncode != 0:
# Negative = killed by signal (segfault); positive = unhandled
# exception or OOM-kill. Either way: poison-pill signature.
return ProbeResult(
ok=False, crashed=True,
reason=f"archive probe crashed (exit {result.returncode})",
)
last_line = result.stdout.strip().splitlines()[-1:] or [""]
try:
outcome = json.loads(last_line[0])
except json.JSONDecodeError as exc:
return ProbeResult(
ok=False, crashed=True,
reason=f"archive probe produced no parseable result: {exc}",
)
if outcome.get("status") == "ok":
return ProbeResult(ok=True)
return ProbeResult(
ok=False, crashed=False,
reason=outcome.get("detail") or "archive probe rejected",
)
def _run_probe(path_str: str) -> tuple[str, str | None]:
"""Pure-Python body of the archive probe — bomb-guard + integrity test.
Returns ('ok', None) or ('error', reason). Caught exceptions become
clean 'error' rejections; uncaught crashes in the subprocess become
non-zero exit codes (poison-pill signature) handled by probe_archive.
Exposed at the module level so the subprocess runner and tests both
call the same code path.
"""
path = Path(path_str)
ext = path.suffix.lower()
try:
total, test_bad = _inspect_archive(path, ext)
except Exception as exc: # noqa: BLE001 — clean rejection
return ("error", f"{type(exc).__name__}: {exc}")
if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES:
gib = total / (1024 ** 3)
return ("error", f"uncompressed size {gib:.1f} GiB exceeds the bomb-guard cap")
if test_bad is not None:
return ("error", f"integrity test failed at member {test_bad!r}")
return ("ok", None)
def _inspect_archive(path: Path, ext: str):
"""Return (total_uncompressed_bytes | None, first_bad_member | None)
for the archive. Format-specific; raises on a structurally-broken
container (caught by the child as a clean rejection)."""
if ext in (".zip", ".cbz"):
import zipfile
with zipfile.ZipFile(path) as zf:
total = sum(zi.file_size for zi in zf.infolist())
return total, zf.testzip()
if ext == ".rar":
import rarfile
with rarfile.RarFile(path) as rf:
total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist())
rf.testrar()
return total, None
if ext == ".7z":
import py7zr
with py7zr.SevenZipFile(path, "r") as zf:
info = zf.archiveinfo()
total = getattr(info, "uncompressed", None)
ok = zf.test() # True / None when all members pass
return total, (None if ok in (True, None) else "7z test reported corruption")
# Unknown extension — nothing to test; treat as clean.
return None, None
+74 -13
View File
@@ -1,7 +1,9 @@
"""Minimal gallery-dl sidecar parsing (one-time filesystem-import aid).
No per-platform branching: a small common key set with fallbacks; the
full JSON is kept in raw so anything unmapped is recoverable later.
Per-platform quirks (post_url synthesis, key-chain overrides) live in
the platforms registry — `backend/app/services/platforms/`. This module
is platform-agnostic: it looks up `category` in the sidecar and asks
the registry for the right behavior.
"""
import re
@@ -9,6 +11,12 @@ from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from ..services.platforms import (
PLATFORMS,
description_keys_for,
external_post_id_keys_for,
)
@dataclass(frozen=True)
class SidecarData:
@@ -55,6 +63,46 @@ def _first_str(data: dict, keys: tuple[str, ...]) -> str | None:
return None
def _first_id(data: dict, keys: tuple[str, ...]) -> str | None:
"""Like `_first_str` but accepts ints and rejects bool (Python's
bool subclasses int, so a literal `"id": true` would otherwise
yield external_post_id="True")."""
for k in keys:
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
# Strip HTML tags + collapse whitespace + take the first non-empty line.
# Used to derive a display title from a body when the platform doesn't
# expose a separate title field (subscribestar posts always write
# `title: ""` and put the leading sentence inside `content` as HTML).
# Truncated to 120 chars with an ellipsis if longer — long enough to be
# meaningful in a feed, short enough to fit a row.
_TAG_RE = re.compile(r"<[^>]+>")
_WS_RE = re.compile(r"\s+")
def _first_line_text(body: str, limit: int = 120) -> str | None:
if not body:
return None
text = _TAG_RE.sub(" ", body)
text = text.replace("\xa0", " ")
# Split on hard line breaks first; the body-stripped HTML often
# collapses to one logical line, in which case the first sentence
# split is the next-best heuristic.
for line in text.splitlines():
line = _WS_RE.sub(" ", line).strip()
if line:
if len(line) > limit:
return line[: limit - 1].rstrip() + ""
return line
return None
def _parse_date(v) -> datetime | None:
if isinstance(v, bool):
return None
@@ -84,14 +132,7 @@ def parse_sidecar(data: dict) -> SidecarData:
cat = data.get("category")
platform = cat if isinstance(cat, str) and cat.strip() else None
external_post_id = None
for k in ("id", "post_id", "index", "message_id"):
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
external_post_id = str(v)
break
external_post_id = _first_id(data, external_post_id_keys_for(platform))
pc = data.get("page_count")
if isinstance(pc, bool):
@@ -111,12 +152,32 @@ def parse_sidecar(data: dict) -> SidecarData:
if post_date is not None:
break
description = _first_str(data, description_keys_for(platform))
# When `title` is empty (subscribestar always; sometimes elsewhere),
# synthesize from the description body's first non-empty text line.
# Patreon's explicit titles short-circuit the fallback.
post_title = _first_str(data, ("title",))
if post_title is None and description:
post_title = _first_line_text(description)
# post_url: ask the platform module to synthesize a permalink.
# When the platform registers a `derive_post_url`, it owns the
# field (the bare `url`/`post_url` value is a file CDN URL and
# must NEVER be persisted). When it doesn't register one, trust
# the sidecar's `url` (Patreon's case — real permalink).
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.derive_post_url is not None:
post_url = info.derive_post_url(data)
else:
post_url = _first_str(data, ("url", "post_url"))
return SidecarData(
platform=platform,
external_post_id=external_post_id,
post_url=_first_str(data, ("url", "post_url")),
post_title=_first_str(data, ("title",)),
description=_first_str(data, ("content", "description", "caption")),
post_url=post_url,
post_title=post_title,
description=description,
attachment_count=attachment_count,
post_date=post_date,
raw=data,
+46
View File
@@ -0,0 +1,46 @@
"""Parse the user-facing `kind:name` shortcut used by the add-tag input.
Mirrors IR's app/utils/tag_prefix.py. Tag.name in FC is stored bare;
the `kind:` prefix only exists as an input convention at user-facing
places (image-modal add-tag input, future bulk-add forms). The parser
is the single owner of the kind-string list — anything not in
KNOWN_KINDS keeps its colon as literal text.
"""
from __future__ import annotations
# Kinds the user can type as a prefix at the input boundary.
# Exclusions:
# - `general` is the default for un-prefixed input (never typed as prefix)
# - `archive`, `post` are system-managed
# - `artist` was retired in FC-2d-vii-c — artists are first-class
# entities (Artist row + ImageRecord.artist_id), browsed via the
# provenance axis rather than as tags. See project_provenance_separation.
# - `meta`, `rating` retired as user-typeable per operator 2026-05-26 —
# content classification only needs character/fandom/series.
KNOWN_KINDS: frozenset[str] = frozenset({
"character",
"fandom",
"series",
})
def parse_kind_prefix(raw: str) -> tuple[str | None, str]:
"""Split a raw user-typed tag string into (kind, name).
Returns (kind, name) where kind is lowercase canonical and in
KNOWN_KINDS, or (None, raw.strip()) if no recognized prefix is
present. `name` is always whitespace-stripped.
Examples:
parse_kind_prefix("character:Saber") -> ("character", "Saber")
parse_kind_prefix("Character:Saber") -> ("character", "Saber")
parse_kind_prefix("sunset") -> (None, "sunset")
parse_kind_prefix("http://example") -> (None, "http://example")
parse_kind_prefix("fandom: FSN ") -> ("fandom", "FSN")
"""
if ":" in raw:
prefix, rest = raw.split(":", 1)
if prefix.lower() in KNOWN_KINDS:
return prefix.lower(), rest.strip()
return None, raw.strip()
+41 -2
View File
@@ -194,9 +194,20 @@ browser.runtime.onMessage.addListener(async (msg) => {
if (platform.authType === 'cookies') {
const cookies = await extractCookiesForPlatform(key);
if (cookies.length === 0) return { error: 'No cookies found — log in first.' };
// Verify the captured cookies are actually live BEFORE
// uploading. Skips upload on confirmed-stale sessions so we
// don't overwrite FC-side credentials with garbage. Platforms
// without a verify config (verify.ok === null) fall through
// to upload as before.
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
return {
error: `Captured ${cookies.length} ${platform.name} cookies but they don't appear authenticated (${v.reason}). Log in again in this browser, then retry.`,
};
}
const data = toNetscapeFormat(cookies);
await api.uploadCredentials(key, 'cookies', data);
return { success: true, cookieCount: cookies.length };
return { success: true, cookieCount: cookies.length, verified: v.ok === true };
}
if (key === 'discord') {
if (!discordToken) return { error: 'Open discord.com to capture a token first.' };
@@ -229,8 +240,13 @@ browser.runtime.onMessage.addListener(async (msg) => {
results[key] = { skipped: true, reason: 'no cookies' };
continue;
}
const v = await verifyCookiesForPlatform(key);
if (v.ok === false) {
results[key] = { error: `verify failed: ${v.reason}` };
continue;
}
await api.uploadCredentials(key, 'cookies', toNetscapeFormat(cookies));
results[key] = { success: true, cookieCount: cookies.length };
results[key] = { success: true, cookieCount: cookies.length, verified: v.ok === true };
} catch (e) {
results[key] = { error: e.message };
}
@@ -259,6 +275,29 @@ browser.runtime.onMessage.addListener(async (msg) => {
return { error: e.message };
}
case 'PROBE_SOURCE':
try {
return await api.probeSource(msg.url);
} catch (e) {
return { error: e.message };
}
case 'OPEN_ARTIST_PAGE': {
// apiUrl is configured with the /api suffix (see
// options/options.html placeholder); the SPA artist route is
// /artist/:slug, served from the same origin. Strip /api so the
// browser-level URL hits the Vue router, not the JSON API.
const base = (api.baseUrl || '').replace(/\/+$/, '').replace(/\/api$/, '');
const slug = encodeURIComponent(msg.slug || '');
if (!base || !slug) return { error: 'apiUrl or slug missing' };
try {
await browser.tabs.create({ url: `${base}/artist/${slug}` });
return { success: true };
} catch (e) {
return { error: e.message };
}
}
default:
return { error: `Unknown message type: ${msg.type}` };
}
+16 -1
View File
@@ -5,11 +5,26 @@
background: rgb(20, 23, 26); color: rgb(244, 186, 122);
font: 500 14px/1.2 system-ui, sans-serif;
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.4); cursor: pointer;
transition: transform 100ms ease;
transition: transform 100ms ease, background 150ms ease, color 150ms ease;
}
.fc-add-source-btn:hover { transform: translateY(-1px); }
.fc-add-source-btn:disabled { opacity: 0.6; cursor: wait; }
/* state colors map to the FC palette: parchment-on-slate base,
accent-orange for new, sage for already-subscribed, amber-warning for
artist-exists-but-source-missing. All readable on the dark base. */
.fc-add-source-btn--new {
background: rgb(20, 23, 26); color: rgb(244, 186, 122);
}
.fc-add-source-btn--artist-match {
background: rgb(28, 23, 16); color: rgb(255, 200, 120);
border: 1px solid rgb(180, 130, 60);
}
.fc-add-source-btn--source-match {
background: rgb(18, 28, 20); color: rgb(140, 220, 160);
border: 1px solid rgb(80, 160, 100);
}
.fc-toast {
all: revert;
position: fixed; bottom: 84px; right: 24px; z-index: 2147483647;
+95 -13
View File
@@ -2,6 +2,10 @@
if (window.__fc_addsource_injected) return;
window.__fc_addsource_injected = true;
// Cached probe result for the current URL so click-handlers know which
// action to dispatch without round-tripping again.
let currentProbe = null;
evaluate();
const reEval = () => evaluate();
@@ -9,38 +13,116 @@
const origPush = history.pushState;
history.pushState = function () { origPush.apply(this, arguments); reEval(); };
function evaluate() {
const platform = getPlatformFromUrl(window.location.href);
const onArtist = platform && isArtistPage(window.location.href, platform);
let btn = document.getElementById('fc-add-source-btn');
if (onArtist && !btn) injectButton();
else if (!onArtist && btn) btn.remove();
async function evaluate() {
const url = window.location.href;
const platform = getPlatformFromUrl(url);
const onArtist = platform && isArtistPage(url, platform);
const btn = document.getElementById('fc-add-source-btn');
if (!onArtist) {
if (btn) btn.remove();
currentProbe = null;
return;
}
// On artist pages, ask the backend what state the URL is in BEFORE
// injecting the button — so the chip can render the right state on
// first paint instead of flashing the generic "Add" copy and
// updating afterwards.
let probe;
try {
probe = await browser.runtime.sendMessage({ type: 'PROBE_SOURCE', url });
} catch (e) {
probe = { error: e?.message || 'probe failed' };
}
currentProbe = probe;
if (probe?.state === 'unknown_platform') {
if (btn) btn.remove();
return;
}
renderButton(probe);
}
function injectButton() {
const btn = document.createElement('button');
btn.id = 'fc-add-source-btn';
function renderButton(probe) {
let btn = document.getElementById('fc-add-source-btn');
if (!btn) {
btn = document.createElement('button');
btn.id = 'fc-add-source-btn';
btn.addEventListener('click', onClick);
document.body.appendChild(btn);
}
// Reset state classes so re-renders (SPA navigation) don't stack.
btn.className = 'fc-add-source-btn';
btn.textContent = '+ Add to FabledCurator';
btn.addEventListener('click', onClick);
document.body.appendChild(btn);
btn.classList.add(`fc-add-source-btn--${stateModifier(probe)}`);
btn.textContent = labelFor(probe);
btn.disabled = false;
}
function stateModifier(probe) {
if (!probe || probe.error) return 'new';
return ({
source_match: 'source-match',
artist_match: 'artist-match',
new: 'new',
})[probe.state] || 'new';
}
function labelFor(probe) {
if (!probe || probe.error) return '+ Add to FabledCurator';
const platformName = platformDisplayName(probe.platform);
const artistName = probe.artist?.name;
switch (probe.state) {
case 'source_match':
return `✓ In FabledCurator · ${platformName}`;
case 'artist_match':
return `+ Add ${platformName} source to ${artistName || 'artist'}`;
case 'new':
default:
return '+ Add to FabledCurator';
}
}
function platformDisplayName(key) {
return PLATFORMS[key]?.name || key || '';
}
async function onClick() {
const btn = document.getElementById('fc-add-source-btn');
if (!btn) return;
btn.disabled = true;
const original = btn.textContent;
const probe = currentProbe;
if (probe?.state === 'source_match') {
btn.textContent = 'Opening…';
try {
const r = await browser.runtime.sendMessage({
type: 'OPEN_ARTIST_PAGE',
slug: probe.artist?.slug,
});
if (r?.error) showToast(`Error: ${r.error}`, 'error');
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
} finally {
btn.disabled = false;
btn.textContent = original;
}
return;
}
btn.textContent = 'Adding…';
try {
const r = await browser.runtime.sendMessage({
type: 'ADD_AS_SOURCE',
url: window.location.href,
});
if (r.error) {
if (r?.error) {
showToast(`Error: ${r.error}`, 'error');
} else {
const verb = r.created_source ? 'Added' : 'Already a source for';
showToast(`${verb} ${r.artist?.name || 'artist'} (${r.source?.platform || ''})`, 'success');
// Re-probe so the chip flips green without waiting for the next
// navigation.
evaluate();
return;
}
} catch (e) {
showToast(`Error: ${e.message}`, 'error');
+6
View File
@@ -83,6 +83,12 @@ class FabledCuratorAPI {
quickAddSource(url) {
return this.request('POST', '/extension/quick-add-source', { url });
}
probeSource(url) {
// Read-only existence check. Drives the content-script chip's
// color/copy BEFORE the operator clicks Add.
const qs = new URLSearchParams({ url }).toString();
return this.request('GET', `/extension/probe?${qs}`);
}
// Connection test = the cheapest read with auth.
testConnection() {

Some files were not shown because too many files have changed in this diff Show More