Compare commits

...

198 Commits

Author SHA1 Message Date
bvandeusen 319e8c1d18 Merge pull request 'v26.05.28.0: downloads dashboard + task-resilience overhaul (timeouts, archive split, 3-layer poison-pill defense)' (#31) from dev into main 2026-05-28 00:45:00 -04:00
bvandeusen dcfe55d731 feat(import-resilience L2): one-shot re-download for corrupt downloaded files
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.

New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
  derive the artist from the import path, find an ENABLED Source with a
  real feed URL for (artist, platform). Returns None for filesystem-only
  imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
  anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
  delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
  set ImportTask.refetched=True, and trigger ONE download_source
  re-check. Bounded by `refetched` so source-side corruption can't loop.

Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
  tasks). Returns refetch_queued / no_source / already_refetched /
  not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
  env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
  manual button is the primary path; auto is opt-in since re-fetch
  deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
  → stores.import.refetchTask → toast keyed on the result status.

Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.

Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
2026-05-28 00:08:03 -04:00
bvandeusen e3cdd0f92b feat(import-resilience L3): subprocess-isolated probes for video + archive
Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.

New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):

- probe_video(path): validates the container + first video stream via
  ffprobe (a separate binary — a decoder crash kills only ffprobe, not
  the worker). Returns width/height, which the importer didn't capture
  for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
  (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
  test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
  spawned child process. A decompression-bomb OOM or native-lib
  segfault on a malformed archive shows up as a non-zero child exit
  code → crashed=True, never a dead worker.

ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).

Wired:
- importer._import_media video branch: probe_video before the pipeline;
  crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
  → failed, clean reject → still preserve the archive as a
  PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
  so a corrupt video is rejected (status='bad_video') instead of
  crashing the ml-worker on frame decode.

Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
2026-05-28 00:01:32 -04:00
bvandeusen e77afe8295 feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.

- alembic 0026: import_task.recovery_count (int, default 0) +
  import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
  rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
  are marked 'failed' with a diagnostic ("crashed or stalled the worker
  N times … likely a corrupt or oversized input … inspect/replace the
  file, then retry via /api/import/retry-failed") instead of re-queued.
  The re-queue pass then handles the remaining stuck rows and bumps
  recovery_count. Shared stuck_predicate (and_/or_) keeps the
  media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).

The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."

Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
2026-05-27 23:54:35 -04:00
bvandeusen 57a22d6098 fix(tests): repair test_maintenance — skips_fresh_running tail was orphaned at module scope by the inserted ml/archive sweep tests (F841/F821) 2026-05-27 23:06:00 -04:00
bvandeusen a85880f965 fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps
Operator-flagged 2026-05-28: import_media_file on target 1645019 hit
SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct —
the timeout covered the WHOLE archive, not per object. Importer._import_archive
(importer.py:409) runs the full per-member pipeline (sha256 + pHash +
dedup query + copy + provenance) for EVERY media member inline, all
under import_media_file's single 300s soft limit. A single media file
is sub-second; a multi-hundred-member archive blows the budget. They
shared one task name and one timeout.

**Split archive into its own task**

- New `import_archive_file` task: same body as import_media_file
  (dispatch is by file-kind inside Importer.import_one) but
  soft=30min / hard=35min. Shared `_run_import_task` helper holds the
  flip-to-processing + resilience-contract wrapper; both tasks call it.
- New `enqueue_import(task_id, task_type)` router — single source of
  truth for media-vs-archive dispatch. Used by all three enqueue sites:
  scan_directory, /api/import/retry-failed, recover_interrupted_tasks.
- scan_directory now sets ImportTask.task_type = "archive" when
  is_archive(entry) (the model field already existed, anticipating
  this; scan was hardcoding "media").
- import_archive_file routes to the existing 'import' queue via the
  task_routes `import_file.*` wildcard — no worker config change.

**Archive-aware recovery sweeps**

Both sweeps would otherwise preempt a legitimately-running archive:

- recover_interrupted_tasks (ImportTask 'processing' sweep): now
  task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives
  get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the
  35-min hard limit). Single UPDATE with an OR predicate over the two
  (task_type, cutoff) pairs; requeue routes via enqueue_import.
- recover_stalled_task_runs (TaskRun 'running' sweep): now supports
  per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above
  the per-queue overrides added for ml. import_archive_file gets 40 min
  while the 'import' queue stays at the 5-min default for single-file
  imports. Precedence: task_name → queue → default, each pass excluding
  rows claimed by a higher-precedence pass so every row is touched once.

**Tests**

- test_import_archive_file_registered
- test_recover_stalled_task_runs_archive_task_uses_longer_threshold —
  pins that a 10-min archive task-run survives, a 50-min one is flagged,
  and a same-queue 10-min media import is flagged at the default.
- _make_task_run gains queue= + task_name= params.

After deploy: archive imports get a 30-min budget and aren't preempted
by either sweep; single-file imports keep their tight 5-min detection.
2026-05-27 22:45:11 -04:00
bvandeusen 407de18ff6 fix(ml): video branch needs longer time limits; recovery sweep is now per-queue
Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was
marked failed by recover_stalled_task_runs at the 5-min sweep tick
while still legitimately running. The error_type='RecoverySweep' /
"no completion signal received within 5 min" message was misleading
— the worker was busy, not stuck.

Root cause is two interacting limits, both undersized for video work:

  tag_and_embed: soft_time_limit=300, time_limit=420
                 (sized for the image branch, ≈2 GPU ops)
  recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues

The video branch samples 10 frames via ffmpeg, then runs tagger +
embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded
ml-worker can take 5-10 min on a long video, which trips both
limits well before the task naturally finishes.

**Two-part fix**

1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200
   (20 min). Sized for the video path's worst case; image runs return
   in seconds and don't care.

2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py.
   Queues with legitimately-long-running tasks (currently just `ml` at
   25 min — 5-min buffer past the new hard kill) get their own
   threshold; queues not in the dict use the default 5 min. The sweep
   now issues one UPDATE per distinct threshold value, with
   `queue.notin_(override_queues)` on the default pass so each row is
   touched at most once.

Tests:
- _make_task_run helper accepts `queue=` (defaults to "default") so
  existing tests use the default-threshold path.
- New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold`
  pins both directions: a 10-min-old ml row survives (fresh by 25-min
  override), a 30-min-old ml row gets flagged.

After deploy, operator's mp4 ML jobs run to completion without
spurious RecoverySweep failures.
2026-05-27 22:23:35 -04:00
bvandeusen b1b129ce9f feat(downloads-tab): A+B dashboard improvements — row restyle + date-grouped sections with failed-pinned
Operator-flagged 2026-05-27: the Downloads subtab "doesn't feel like a
dashboard" — status was a tiny mdi icon at the far left, platform chip
was neutral-tonal, errors were plain orange text floating on the right,
and all 28 rows from the same hour visually had the same priority.

**Row restyle (A):**
- 4px colored left-edge bar by status (success/error/info/warning/grey)
  — visually scannable at the edge without parsing the chip text
- Status chip with text label (Completed/Failed/Running/Queued/Skipped)
  + leading icon, tonal-colored. Replaces the bare mdi-icon.
- Platform chip swapped to the color-coded subscriptions/PlatformChip
  (Patreon=red mdi-patreon, SubscribeStar=amber, HentaiFoundry=purple,
  Discord=indigo, Pixiv=blue, DeviantArt=green).
- File count: tonal info chip when > 0, dim middle-dot when 0 (so
  scheduled "no-change" scans don't dominate the column visually).
- Error: red tonal pill chip with leading icon, truncated to 60 chars
  with full text in the title tooltip. Replaces plain text.
- Per-row actions (hidden at 50% opacity, fade to full on row hover):
  Retry (only when status=error AND source_id known — hits
  POST /api/sources/<id>/check via the existing sources.checkNow),
  Details (opens the detail modal), Open artist (navigates to the
  artist page). Clicks stop-propagation so they don't bubble to the
  row click.

**Date-grouped sections (B):**
- Events are bucketed into four sections: Today / Yesterday /
  Last 7 days / Earlier. Empty buckets are skipped. Buckets boundaries
  are computed against the operator's local-time start-of-day so
  "Today" matches their intuition.
- Each section has a collapsible header with a row-count chip + a
  red "failed in this section" chip when any failures are in scope.
- Within each section, status='error' rows are pinned to the top
  (operator's eye lands on failures first; successful scans flow
  below).
- Collapsed state persists across refresh within the SubscriptionsView
  lifetime (reactive object, default all-expanded).

DownloadEventRow grid widened to accommodate the status chip + actions
column. PolyMasonry-style ellipsis on the artist link prevents long
names from breaking the layout.

No new endpoints; the Retry path reuses the existing /api/sources/<id>/check
flow (the source-check endpoint was already in place, just not wired
into a per-row button).
2026-05-27 22:18:02 -04:00
bvandeusen 9075d8eadd Merge pull request 'v26.05.27.2: subscribestar + HF cookie quirks, platforms package refactor, showcase IR-parity, secure-context audit' (#30) from dev into main 2026-05-27 21:34:02 -04:00
bvandeusen df6d89cb59 fix(secure-context): full audit — DestructiveConfirmModal.expectedTokenOverride + bulk-delete + min-dim use backend-computed tokens
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.

**Audit results across `frontend/src/`:**

  crypto.subtle.digest        — 2 sites:
    - MinDimensionCard (fixed 2026-05-27)
    - BulkEditorPanel (THIS FIX)
  navigator.clipboard         — 1 site, already guarded:
    - utils/clipboard.js writeText with execCommand fallback
  serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
  cookieStore / queryLocalFonts / WebAuthn / geolocation
                              — NOT USED, nothing to fix

  Extension scripts (background.js) use crypto.subtle but run from
  moz-extension:// which IS a Secure Context — left as-is.

**BulkEditorPanel double bug**

The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:

1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
   never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
   `delete-images-selection-<sha8>` while the backend expected
   `delete-images-<sha8>`. The two would never match.

Fix:

- Backend `/api/admin/images/bulk-delete` dry-run response now returns
  `confirm_token` (the canonical `delete-images-<sha8>` string).
  Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
  assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
  set, it bypasses the `${action}-${kind}-${runId}` formula and uses
  the explicit string. This decouples the UI label (`kind`) from the
  wire-format token (server-provided), so future endpoints can use a
  kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
  — no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
  slicing the 8-char suffix off the backend's token and passing it
  through `runId`; now passes the full backend token via
  `expected-token-override` directly). Cleaner; one source of truth.

**Banked memory**

`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.

No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
2026-05-27 21:17:40 -04:00
bvandeusen 12be188ada feat(showcase): IR-parity R-key shuffle + stagger entry animation; fix(cleanup): min-dim Delete swallowed crypto.subtle TypeError on plain HTTP
**showcase R-key + entry animation**

Restores two behaviors lost during the FC-2 IR→Vue port. Operator-flagged
2026-05-27.

- ShowcaseView listens for keydown 'r'/'R' on window. Triggers
  `store.shuffle()`. Skips when an input/textarea/contenteditable is
  focused or a Vuetify overlay is open (the dialog/menu sets
  `.v-overlay--active` on the body).
- MasonryGrid gains an opt-in `animateFromIndex` prop (default
  `Number.POSITIVE_INFINITY` = off). When set, items with index ≥ the
  threshold animate in with a stagger fade-in: 12px translateY,
  0.25s ease, 60ms per item, capped by `prefers-reduced-motion`.
  Stagger uses original-items-array index (resolved via an `idxById`
  Map) so the reading order is preserved even after the masonry
  distributes items across columns.
- ShowcaseView watches `store.images.length`: shrink-or-zero baseline
  ⇒ `animateFromIndex=0` (animate everything on initial load /
  shuffle); grow ⇒ baseline=prevCount (animate only the appended
  tail on infinite-scroll). Other MasonryGrid consumers (ArtistView's
  Gallery tab) don't pass the prop, so they keep their current
  no-animation behavior.

Direct port of IR's `app/static/js/showcase.js` keyboard handler +
`app/static/style.css` itemFadeIn keyframe.

**min-dim Delete: crypto.subtle TypeError fix**

The Delete button on the Cleanup → Minimum Dimensions card was
silently no-op'ing. Root cause: `crypto.subtle` is Secure-Context-gated
(undefined on plain-HTTP origins per the homelab posture). The card's
`onDeleteClick` computed the Tier-C confirm token via
`crypto.subtle.digest('SHA-256', ...)`, which threw TypeError before
`showModal.value = true`. The promise rejected, the click handler had
no `.catch`, the modal never opened — exactly the operator's reported
symptom.

Same shape as the v26.05.26.0 `navigator.clipboard` fix on the
ErrorDetailModal Copy button.

Fix: backend `/api/cleanup/min-dimension/preview` now returns
`confirm_token` (the canonical `delete-min-dim-<sha8>` string) in its
response. Frontend reads it from the preview response and feeds the
8-char suffix to DestructiveConfirmModal's `runId` prop — no
client-side crypto needed. Single source of truth.

Integration test `test_min_dimension_preview_returns_count` pinned to
also assert `body["confirm_token"]` matches the server-side compute.
2026-05-27 20:59:58 -04:00
bvandeusen 6d7116c090 fix(platforms): ruff I001 in base.py — one blank line between imports and module-level constant (was two) 2026-05-27 20:37:06 -04:00
bvandeusen b447c42853 fix(platforms): ruff I001 — drop unused __future__ import; switch __init__ to per-module imports for clean isort ordering 2026-05-27 19:52:50 -04:00
bvandeusen abafc3265e refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."

**Layout**

  services/platforms/
    base.py            PlatformInfo dataclass + module-default key
                       chains + shared helpers (str_id_value, str_field)
    __init__.py        PLATFORMS dict + public API (auth_type_for,
                       known_platform_keys, to_dict,
                       external_post_id_keys_for, description_keys_for)
    patreon.py         metadata only — the reference platform, no quirks
    subscribestar.py   metadata + augment_cookies (18+ agreement) +
                       derive_post_url (synthetic /posts/<post_id>)
    hentaifoundry.py   metadata + augment_cookies (host-only PHPSESSID
                       duplicate) + derive_post_url (/pictures/user/...)
    pixiv.py           metadata + derive_post_url (/artworks/<id>)
    discord.py         metadata + derive_post_url
                       (channels/<server>/<channel>/<message>)
    deviantart.py      metadata only — un-audited; quirks to be added
                       when an operator first exercises DA

**PlatformInfo extensions**

Existing fields preserved. Four new optional fields:

  external_post_id_keys: tuple[str, ...] | None
      Override the sidecar external_post_id lookup chain. None falls
      back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
      ("post_id", "id", "index", "message_id") — covers every current
      platform.

  description_keys: tuple[str, ...] | None
      Override the description body lookup chain. None falls back to
      DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
      "message") — Discord's "message" body field is covered by the
      default's trailing entry.

  derive_post_url: Callable[[dict], str | None] | None
      Synthesize the post permalink from sidecar metadata. None = trust
      the bare `url` / `post_url` field (patreon, deviantart).
      subscribestar/pixiv/hf/discord override this because their `url`
      is the file CDN URL.

  augment_cookies: Callable[[str], str] | None
      Post-process the materialized cookies.txt before gallery-dl
      consumes it. None = no-op. Used by subscribestar (age cookie) and
      hentaifoundry (host-only PHPSESSID duplicate).

**Consumer changes**

- credential_service._augment_cookies(platform, netscape) shrunk from a
  per-platform-conditional dispatcher (~80 lines of inlined helpers) to
  a 5-line lookup: `info.augment_cookies(netscape) if info and
  info.augment_cookies else netscape`. The platform-specific helper
  bodies moved verbatim into the per-platform modules.

- sidecar.parse_sidecar similarly delegates: external_post_id chain via
  external_post_id_keys_for(category), description chain via
  description_keys_for(category), post_url via
  PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
  and inline _derive_post_url body both gone. Added a shared `_first_id`
  helper for bool-safe id coercion.

**Public API preserved**

PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.

**Adding a new platform**

1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
   and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
   tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
   pick it up automatically.
2026-05-27 19:46:05 -04:00
bvandeusen 2394e47370 fix(hentaifoundry): inject host-only PHPSESSID/CSRF duplicates + extension preserves browser hostOnly
Operator-flagged 2026-05-27: HF source check 401'd on
`HEAD /?enterAgree=1` even with valid login cookies. Root cause is the
combination of (1) gallery-dl's HF extractor checking
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching, and (2) the extension's cookies.js
forcibly rewriting every captured cookie to a leading-dot subdomain-wide
form. HF's PHPSESSID is browser-stored as host-only on
`www.hentai-foundry.com`; the rewrite re-anchored it to
`.hentai-foundry.com`, which `cookies.get(...)` no longer matches even
though the cookie is still sent on actual HTTP requests (RFC 6265
subdomain rules). The extractor falls into its unauthenticated
`?enterAgree=1` fallback, which 401s (Cloudflare or HF's anti-bot HEAD
gating).

Two-part fix, no operator action required for existing stored cookies:

1. **Backend** (`credential_service._augment_cookies`) — refactored from
   the subscribestar-only single function into a per-platform dispatcher.
   New `_augment_hentaifoundry` parses the materialized netscape file
   and, for each `.hentai-foundry.com` entry whose name is PHPSESSID or
   YII_CSRF_TOKEN, appends a host-only duplicate
   (`www.hentai-foundry.com\tFALSE\t...`). Originals preserved. Three
   new tests pin: injection fires + originals preserved; idempotent
   when host-only already exists; doesn't touch unrelated cookies
   (e.g. `_ga`).

2. **Extension** (`cookies.js`) — `toNetscapeFormat` now respects
   `c.hostOnly` from the browser instead of blindly forcing a
   leading-dot subdomain-wide form. Host-only cookies are written with
   the bare host + FALSE flag; non-host-only cookies retain the
   leading-dot + TRUE form. Forward-compat — fresh captures from
   v1.0.5+ no longer need the backend's host-only duplication.
   Extension bumped 1.0.4 → 1.0.5; manifest + package.json in lockstep.

After deploy: the next HF source check on the operator's already-stored
cookies will succeed because the materialized cookies.txt now contains
host-only PHPSESSID. No browser re-export needed.
2026-05-27 19:12:51 -04:00
bvandeusen 8243740a04 fix(subscribestar): inject 18_plus_agreement_generic age cookie to bypass server gate
Operator-flagged 2026-05-27: subscribestar source check aborted with
`AbortExtraction: HTTP redirect to .../age_confirmation_warning`. The
captured `_personalization_id` cookie in the browser-stored file had
expired (annual rotation), and the user could not realistically refresh
it: SubscribeStar's frontend JS uses localStorage to suppress the
age-confirmation popup once dismissed, so a logged-in revisit doesn't
re-show the popup and the server-side cookie is never re-issued.

gallery-dl's own login flow (which FC doesn't exercise — cookies come
from the extension instead) sidesteps this by manually setting
`18_plus_agreement_generic=true` on `.subscribestar.adult`. The server
accepts that as the age-confirmation marker.

`credential_service._augment_cookies(platform, netscape)` mirrors that
behavior: when the materialized cookies file is for subscribestar and
the age cookie isn't already present, append a synthetic line for
`.subscribestar.adult` with name=`18_plus_agreement_generic` value=`true`
and a far-future expiry. No-op for other platforms; no-op if the cookie
is already present (idempotent for manual pastes / extension captures
that happen to include it).

Three new tests pin: (a) injection fires for subscribestar, preserves
existing cookies; (b) idempotent when already present (no double
injection); (c) does NOT fire for non-subscribestar platforms (Patreon
etc. don't get a foreign-domain cookie).

Not a curator handling bug per se — the extension faithfully captured
what the browser had. This is mirroring a documented gallery-dl
workaround so the cookies-via-extension auth path doesn't degrade as the
server-side cookie expires.
2026-05-27 18:18:04 -04:00
bvandeusen 88e53e5b86 Merge pull request 'v26.05.27.1: subscriptions hub + post-card merge + sidecar audit' (#29) from dev into main 2026-05-27 17:12:48 -04:00
bvandeusen aa28bddeab fix(alembic 0025): qualify ambiguous post.id / post.source_id in fragment-group SELECT (post JOIN source — both have id) 2026-05-27 15:45:42 -04:00
bvandeusen b7b313cc05 fix(alembic 0025): include HF + Discord post_url backfill (no longer 'deferred to deep-scan')
Operator-flagged: the claim that 'a future deep-scan via the new parser
will fix HF and Discord post_url' was conditional on the operator
actually running a deep-scan, which they might not do for ages. Until
then HF posts stay at post_url=NULL (HF sidecars have no `url` field)
and Discord posts stay pointing at cdn.discordapp.com/attachments/...
(the file URL, not the message permalink).

The migration was already opening sidecar files for SubscribeStar.
Generalizing the loop to also handle HF and Discord is a tiny addition
that closes the gap without operator intervention.

Per-platform Part 1 logic now:
  subscribestar — read sidecar.post_id, overwrite external_post_id +
    post_url with the derived /posts/<post_id> permalink.
  hentaifoundry — read sidecar.user + .index, overwrite post_url with
    /pictures/user/<u>/<i>. external_post_id (= index) unchanged.
  discord — read sidecar.server_id + .channel_id + .message_id,
    overwrite post_url with the discord.com/channels/.../<m> triple.
    external_post_id (= message_id) unchanged.

Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv
post_url backfill) unchanged.

Posts whose related ImageRecord paths don't resolve on disk (orphan
filesystem state) are reported per-platform in the migration output —
those still need a future deep-scan, but the in-DB-with-on-disk-files
common case is now fully covered by the migration alone.
2026-05-27 15:38:18 -04:00
bvandeusen bd3f996582 fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:

1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
   per-attachment id in `id` (e.g. 711509) and the actual post id in
   `post_id` (e.g. 360360). FC's external_post_id chain had `id`
   winning, so every multi-image SubscribeStar post was fragmented into
   N Post rows in the database. Reorder the chain to
   `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
   `post_id`), HF (uses `index`), Discord (uses `message_id`) all
   unaffected.

2. Discord `message` field not captured. Discord posts put the body in
   `message`, not `content`. Append it to the description fallback chain
   `("content", "description", "caption", "message")`.

3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
   `_derive_post_url(platform, data)` helper synthesizes proper
   permalinks from per-platform fields:
     subscribestar → https://www.subscribestar.com/posts/<post_id>
     pixiv         → https://www.pixiv.net/artworks/<id>
     hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
     discord       → https://discord.com/channels/<server>/<channel>/<message>
   Patreon's bare `url` IS a real permalink and is used as-is. For the
   four file-URL platforms, the bare `url` is NEVER trusted: derive or
   return None rather than persist a CDN URL.

Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
  wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
  shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
  bodies.
- Five new tests cover per-platform post_url derivation
  (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
  None).

Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
  its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
  post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
  same NEW external_post_id is a fragment-set — merge to a canonical
  row using the same ImageProvenance pre-delete + repoint dance as
  alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
  url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
  field (not stored on Post), Discord needs server/channel triple.
  Both will be corrected by a deep-scan re-applying sidecars through
  the new parser.

Idempotent: re-running on already-corrected data is a no-op.
2026-05-27 15:35:25 -04:00
bvandeusen ae8c78ae09 fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar)
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. Confirmed against the operator's
/mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every
post's JSON has `title: ""` and a content like
`<div>Lets say hello to you guys with my Belle <br><br><br></div>`.
FC's sidecar parser, treating empty strings as missing, had been leaving
post_title NULL on every subscribestar post since FC-3 shipped.

Fix at two layers:

1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)`
   helper strips HTML tags, collapses whitespace, returns the first
   non-empty line truncated to 120 chars with ellipsis. `parse_sidecar`
   now falls back to this when `title` resolves to None and a
   `content`/`description`/`caption` value is present. Patreon's
   non-empty titles short-circuit the fallback so existing behavior is
   unchanged. Four new tests in test_sidecar_util.py pin: derivation
   from content, truncation at 120 chars, explicit-title precedence,
   no-content no-fallback.

2. `alembic 0024_backfill_post_title_from_description` — backfills the
   same logic across existing Post rows where `post_title IS NULL OR
   post_title = ''` AND description is present. Idempotent (re-running
   is a no-op once titles are populated). Downgrade is a no-op since
   there's no safe way to tell derived rows from genuine ones.

After deploy + migration: subscribestar posts will surface a meaningful
title in PostCard, post feed search, etc.
2026-05-27 14:44:09 -04:00
bvandeusen 4d2c464045 feat(post-card): absorb PostModal into PostCard with click-to-expand
PostCard and PostModal competed for the same data and rendered redundant
chrome (header twice, image grid twice, attachment list twice). The wider
PostCard layout we shipped 2026-05-27 has enough real estate to be the
canonical post surface, so collapse the two into one.

Compact (default) state is unchanged: hero + 3-cell rail + truncated
title + 3/5-line description + attachment count badge. Whole-card click
expands in place. Expanded state shows: full title, mosaic of ALL post
images via PostImageGrid (uncapped, lazy-loaded via getPostFull), full
sanitized-HTML description with paragraph wrapping, attachments as
downloadable pill links. Click the chevron in the header to collapse;
mosaic image clicks open ImageViewer scoped to the post (modalStore's
postImageIds path is preserved — only the comment changed).

Per-card local state — no global modal store. Each PostCard owns its
own expanded ref and lazy-loaded detail; collapsing a card discards
neither (so re-expand is instant after the first fetch).

Deleted: PostModal.vue, postModal.js store. Removed the App.vue mount.
2026-05-27 14:30:04 -04:00
bvandeusen b8ad17c68d fix(build): poll for ext-<version> release in tag-push build-web (race fix)
Cutting a release fires BOTH the push-to-main workflow AND the push-to-tag
workflow in parallel. main-push runs sign-extension (AMO round-trip 1-5min)
then publishes the ext-<version> Forgejo release; tag-push skips
sign-extension (gated to main) and races straight to build-web's Download
XPI step. Tag-push lost every time — got 404 from
releases/tags/ext-<version> before main-push had finished signing.

v26.05.27.0 hit this: tag-push build-web died on exit 22 because the
ext-1.0.4 release wasn't published yet (it arrived ~4min later).

Fix: wrap the release lookup in a 20-iteration sleep+retry loop, 30s
between attempts (10min total upper bound, generous for AMO). main-push's
signing eventually publishes the release; tag-push picks it up on a later
poll. No more manual rerun of the failed job after every release cut.

Banked the trap as reference_tag_push_main_push_race.md — same shape will
recur any time a tag-push workflow consumes a main-push-produced artifact.
2026-05-27 13:25:58 -04:00
bvandeusen 1fd54897d8 fix(api): ruff UP017 — use datetime.UTC alias in /api/downloads/stats 2026-05-27 13:11:44 -04:00
bvandeusen 9322c984fd feat(subs-hub): collapse /credentials + /downloads into /subscriptions hub with three GS-style subtabs
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.

**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.

**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.

**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.

**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.

**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.

Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
2026-05-27 13:02:24 -04:00
bvandeusen 37e8b796a1 Merge pull request 'v26.05.27.0: PostCard redesign + IR-style tag suffix + drop meta/rating + extension v1.0.4 CSP fix' (#28) from dev into main 2026-05-27 11:31:18 -04:00
bvandeusen 8675f105ad fix(tests): test_api_tags prefix tests use character: not artist: (KNOWN_KINDS dropped artist)
The two prefix-parsing tests were pinned to `artist:Eric`, but `artist`
was removed from KNOWN_KINDS in commit 4cad07a (provenance is a separate
axis from tags). The parser now keeps `artist:` literal, so the assertion
`body["name"] == "Eric"` failed.

Repointed to `character:Saber` (still in KNOWN_KINDS). Also updated the
stale `artist:` docstring example in parse_kind_prefix to `fandom:`.

Caught by [[reference-grep-pinned-tests-in-plans]] — should have grep'd
tests/ for `artist:` when shrinking KNOWN_KINDS. Banking the miss.
2026-05-27 11:09:04 -04:00
bvandeusen 74dac6b960 fix(extension+migration): MV3 CSP opt-out from upgrade-insecure-requests (v1.0.4) + alembic 0023 drops the ck_tag_fandom_requires_character check before the type swap
extension/manifest.json: add content_security_policy.extension_pages = "script-src 'self'; object-src 'self';" — explicitly omits the upgrade-insecure-requests directive that MV3 inherits by default. Without this, every fetch(http://curator.../...) silently upgrades to https:// at the browser layer (Sec-Fetch-Site=same-origin, NS_ERROR_GENERATE_FAILURE), regardless of about:config. Bump XPI version 1.0.3 → 1.0.4 so a fresh signed build replaces the cached one. Operator-troubleshot 2026-05-26 via Inspect-the-extension dev tools showing the silent scheme upgrade.

alembic 0023: drop ck_tag_fandom_requires_character before the tag_kind type swap and recreate after. Postgres can't resolve `kind = 'character'` across the rename (column on tag_kind_old, literal binds to new tag_kind → "operator does not exist"). Same dance on downgrade. Banked under reference_tag_kind_enum_swap_check_drop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:57:38 -04:00
bvandeusen 9e19c081b0 fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:23:12 -04:00
bvandeusen 3838f04c16 feat(tag-kinds): drop meta + rating entirely — alembic 0023 deletes existing meta/rating tags (CASCADE clears related image_tag / alias / allowlist / suggestion_rejection / reference_embedding / series_page rows) then recreates the tag_kind ENUM without those values. Python TagKind enum trimmed; KIND_OPTIONS + KIND_COLOR + KIND_ICONS maps + TagsView KINDS array all updated. Operator confirmed they have no use for the data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:07:31 -04:00
bvandeusen 42b1340324 fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:00:12 -04:00
bvandeusen 3b1e2f1ceb feat(tag-input): IR-style kind:name suffix — drop the kind dropdown from TagAutocomplete; client-side parser mirrors backend's parse_kind_prefix (KNOWN_KINDS = artist/character/fandom/series/meta/rating); autocomplete searches across all kinds and shows kind chip in results; Create label uses parsed kind; character flow still goes through FandomPicker
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:54:11 -04:00
bvandeusen 8cdf0af0e1 feat(tags-api): IR-style kind:name parsing at POST /api/tags — when caller doesn't supply explicit kind, parse_kind_prefix runs on the name (artist:Eric → kind=artist, name='Eric'); explicit kind always wins for backward-compat; falls back to general when no recognized prefix is present. Updates the old "missing required" test that assumed kind was mandatory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:53:27 -04:00
bvandeusen ccee344099 feat(tag-prefix): parse_kind_prefix util — IR-style \kind:name\ parser at the input boundary; KNOWN_KINDS = artist/character/fandom/series/meta/rating (excludes default \general\ and system-managed archive/post)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:49 -04:00
bvandeusen 0316f92e8b feat(artist-posts-tab): bump max-width 900 → 1600 so the new wide-layout PostCard has room and the artist Posts feed doesn't leave most of an ultra-wide screen empty
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:21 -04:00
bvandeusen 6df74683b3 feat(post-card): responsive redesign — container-query split (stack <800px / side-by-side ≥800px), hero + thumb rail, +N overflow chip, line-clamp body (3 narrow / 5 wide), title/desc fallbacks for sparse data, click→postModal.open
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:10 -04:00
bvandeusen 243e536225 feat(app): mount PostModal at app root next to ImageViewer — single instance driven by usePostModalStore so PostCard can open from anywhere
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:50:34 -04:00
bvandeusen 2f16699971 feat(post-modal): PostModal — full Patreon-style v-dialog (header + image grid + sanitized body + attachments); reads from usePostModalStore
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:59 -04:00
bvandeusen a5cb684d34 feat(post-modal): PostImageGrid — fixed-cell grid (auto-fill 220px+, 4:3 aspect-cover) inside PostModal; click opens ImageViewer scoped to the post's images via modalStore.open(id, { postImageIds })
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:20 -04:00
bvandeusen 965a953b2e feat(post-card): PostEmptyThumbs — dashed-border placeholder shown in PostCard's hero slot when post has zero linked images
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:02 -04:00
bvandeusen 90c176b195 feat(postmodal-store): Pinia store driving the app-level PostModal — open(post) fetches full detail via posts store; close() clears
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:49 -04:00
bvandeusen b8d89b9f2a feat(modal-store): post-scoped cycle — open(id, { postImageIds }) pins prev/next to the array; canPrev/canNext + goPrev/goNext check the array index instead of current.value.neighbors when set. Gallery-context callers unchanged (default args clear scope)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:30 -04:00
bvandeusen 07344e0843 feat(util): htmlSanitize — whitelist-based DOM scrubber for PostModal's description v-html (Patreon ships HTML; sanitize before render)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:36 -04:00
bvandeusen 42c33e44f9 feat(post-api): get_post returns uncapped thumbnails — PostModal masonry needs full image list; feed query unchanged (still capped at 6 for previews). _thumbnails_for gains a limit kwarg; get_post passes limit=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:18 -04:00
bvandeusen 4e82208926 Merge pull request 'v26.05.26.5 — extension CORS unblock + UI gap closes + CI workflow cleanup' (#27) from dev into main 2026-05-26 20:15:07 -04:00
bvandeusen 85b640f32e fix(views): close the 24-32px gap below TopNav across all views — every v-container had py-6 (or py-8 on PlaceholderView) which pushed the first content item well below where the TopNav's fade-to-transparent gradient bottoms out. Switch to pt-2 pb-6 (8px top, 24px bottom) so content sits comfortably right below the nav, matches the ArtistHeader's 'continuous with TopNav' feel. PlaceholderView uses pt-3 pb-8 keeping its larger bottom padding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:47:23 -04:00
bvandeusen c7001f4aed fix(extension): CORS preflight for moz-extension:// + chrome-extension:// origins — operator-flagged 2026-05-26 that the extension's Test connection returned NetworkError because /api/credentials POSTs with X-Extension-Key trigger a browser preflight OPTIONS that hit a 405 (no OPTIONS method registered) with no Access-Control-Allow-* headers. Adds two app-level hooks: before_request short-circuits OPTIONS from extension origins with 204, after_request stamps the necessary ACL headers on responses to extension-origin requests. Whitelist is intentionally narrow (extension schemes only) so normal browser usage doesn't get permissive CORS. Five integration tests pin the contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:43:31 -04:00
bvandeusen f827612930 fix(artist-header): close gap below TopNav (top:64px → 48px to match TopNav's actual ~48px height) + center the tab strip via 1fr|auto|1fr layout with a right-side spacer cell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:36:19 -04:00
bvandeusen 3f0153cba5 ci(workflows): dedupe + versioned image tags
ci.yml: drop pull_request: trigger — push: branches: [dev, main] already covers it; pull_request was duplicating ci.yml runs on every dev push with an open PR. (No fork PRs in this repo.)

build.yml: drop dev from push triggers — operator doesn't use the :dev image. Add tags: ['v*'] trigger + tag-push branch in the Determine-tag logic so cutting a release tag publishes an immutable :v26.05.26.X image (rollback story) without re-publishing :latest. Extend the XPI-download step to fire on tag pushes too so the versioned image carries the signed extension.

Net per hotfix cycle: 5 runs → 3 (no tag) / 4 (with tag).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:26:56 -04:00
bvandeusen 52fff00353 Merge pull request 'v26.05.26.4 — hotfix: migration 0022 pre-DELETE colliding ImageProvenance before UPDATE' (#26) from dev into main 2026-05-26 18:06:20 -04:00
bvandeusen f3e8f30a8f fix(migration-0022): pre-DELETE colliding image_provenance rows before the UPDATE post_id — same row-by-row UNIQUE pattern as the post-collision case, just one level deeper. When image X has provenance under both keep and drop, UPDATE drop→keep would fire uq_image_provenance_image_post on the row that'd collide with the existing (X, keep). Pre-delete those rows (their info is already represented by the keep-side provenance) before the UPDATE moves the rest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:05:49 -04:00
bvandeusen eee107766e fix(migration-0022): rename unused _epid loop var (ruff B007)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:03:11 -04:00
bvandeusen c14338cbce Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main 2026-05-26 17:52:59 -04:00
bvandeusen 7a64730bd2 fix(migration-0022): pre-merge ALL duplicate-external_post_id Posts across the (canonical+others) group, not just canonical-vs-others — operator's v26.05.26.2 deploy still tripped uq_post_source_external_id because two non-canonical Sources both had Posts with epid=6166997. Bulk UPDATE moved the first cleanly then collided on the second. New pre-merge groups all Posts in the (artist, platform) by external_post_id; for any group with count>1, picks the keep (prefer one under canonical; else lowest id) and merges the rest before the bulk reparent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:52:29 -04:00
bvandeusen 1803a09306 ci(workflow): remove the 4 Cache pip wheels steps entirely — act_runner's cache backend has been broken for 11+ days and the cached path (~/.cache/pip) wasn't even the primary install tool's cache anyway (uv uses ~/.cache/uv). Net cost ~30s/job of wheel downloads. Long-term: mount ~/.cache/uv as a docker volume at the runner level (skips actions/cache entirely) or fix the runner-side cache backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:54:05 -04:00
bvandeusen 8c36dd28b0 Merge pull request 'v26.05.26.2 — hotfix: alembic 0022 Post-collision pre-merge + ci.yml cache continue-on-error' (#24) from dev into main 2026-05-26 16:50:43 -04:00
bvandeusen 0f7cd3cb76 fix(migration-0022): pre-merge colliding Posts before the bulk reparent — Postgres fires uq_post_source_external_id row-by-row during UPDATE, so the post-reparent merge-collisions step never ran (operator's v26.05.26.1 deploy hit it: 'duplicate key (source_id, external_post_id)=(42, 6166997)'). Detect (keep, drop) Post pairs whose external_post_id already exists under canonical, merge the drop into keep, then bulk-reparent the rest cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:48:49 -04:00
bvandeusen 7b0dd4182c ci(workflow): continue-on-error on Cache pip wheels — act_runner's cache backend has been broken since 2026-05-15 and now hard-fails ('Cannot find module .../dist/restore/index.js') instead of warning. Install step handles cold caches natively; ~30s wheel-download cost per job until the runner-side cache backend is fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:35:26 -04:00
bvandeusen 88cfb3dd02 Merge pull request 'v26.05.26.1 — thumb backfill, modal redesign, recovery sweep race-safety, artist view redesign, extension fixes' (#23) from dev into main 2026-05-26 16:32:00 -04:00
bvandeusen fb41b90110 fix(extension): _find_or_create_artist + _find_or_create_source race-safe via savepoint + IntegrityError recovery — same pattern as importer's helpers. Two concurrent quick-add-source calls on the same artist/url would have 500'd on uq_artist_slug / uq_source_artist_platform_url; now the second one rolls the savepoint back and returns the existing row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:27:51 -04:00
bvandeusen 7d84990f6d feat(artist-view): ArtistView rewrite — sticky frosted ArtistHeader (name + stats + tabs) replaces the in-body h1; three lazy tabs (Posts default, Gallery fallback, Management); ?tab= URL state; cross-artist store reset; document.title set on slug change
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:01:14 -04:00
bvandeusen ca55d92c68 feat(artist-view): ArtistManagementTab — Overview chips (Subscription badge + subscription count) + Frequent tags + Activity sparkline + Subscriptions table + Danger zone. 'View posts' chip and 'Credential health · FC-3b' placeholder chip dropped; 'Sources' section renamed to 'Subscriptions'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:47 -04:00
bvandeusen cce014be3a feat(artist-view): ArtistGalleryTab — MasonryGrid wired to the artist store's existing loadMoreImages; modal-open preserves ?tab=. No global gallery-store coupling (avoids cross-pollution into /gallery)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:14 -04:00
bvandeusen c07effb593 feat(artist-view): ArtistPostsTab — PostCard infinite-scroll list, artist_id pinned, no filter bar; mirrors PostsView mechanics
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:02 -04:00
bvandeusen a36f72b383 feat(artist-view): ArtistHeader — sticky frosted bar (top:64px) matching TopNav recipe, hosts name + image/post stats + tab strip
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:59:46 -04:00
bvandeusen 2e8d7c960c feat(artist): post_count on the artist overview response — drives the Posts/Gallery default-tab fallback in the upcoming ArtistView redesign
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:59:28 -04:00
bvandeusen 992f38ec20 fix(test): drop unused Post binding in test_importer_provenance_race (ruff F841)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:39:57 -04:00
bvandeusen 0bc5767a2b fix(importer): Source = one per (artist, platform), not one per post — filesystem importer's sidecar paths now reuse the artist's existing subscription Source (or create one synthetic anchor with enabled=False) instead of fabricating a new Source per post URL. Alembic 0022 consolidates existing per-post Sources to canonical (prefers campaign URL; falls back to sidecar:<platform>:<slug>) and re-parents Posts + ImageProvenance, merging Post collisions.
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:19:29 -04:00
bvandeusen 397021dcbd fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 13:47:55 -04:00
bvandeusen b0bfbc585a fix(import-admin): retry-failed + clear-stuck — same UPDATE…WHERE pattern as the maintenance sweep, so neither endpoint can hit psycopg's 65535-parameter ceiling once accumulated row counts exceed ~65k
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:47:40 -04:00
bvandeusen 110c1c0e51 fix(maintenance): recover_interrupted_tasks — fold SELECT into UPDATE…WHERE…RETURNING so the IN-list no longer blows past psycopg's 65535-parameter ceiling (operator-hit 2026-05-26 after deep scan orphan pile)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:46:06 -04:00
bvandeusen 6de84d0d60 feat(ui): ErrorDetailModal — context panel (task/queue/target/duration/started/retries/worker/celery-id/args) + contrast fix (background-token bg vs surface-variant pale-on-pale) + copyText helper for Copy button
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:16:37 -04:00
bvandeusen 4e1f208a9f fix(ui-copy): copyText utility with execCommand fallback — navigator.clipboard is gated by Secure Context (HTTPS-only) and is undefined on plain-HTTP self-hosted deployments. Apply to ExtensionKeyBar + BrowserExtensionCard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:16:32 -04:00
bvandeusen 06913eba8e feat(thumb-backfill): MaintenancePanel — wire ThumbnailBackfillCard into grid, broaden intro to cover non-ML backfills
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:37:09 -04:00
bvandeusen b7f693b15e feat(thumb-backfill): ThumbnailBackfillCard — 'Run backfill now' button, mirrors MLBackfillCard pattern
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:55 -04:00
bvandeusen ecd0199799 feat(thumb-backfill): Pinia store — triggerBackfill() POSTs /api/thumbnails/backfill
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:45 -04:00
bvandeusen 983da9e5b1 feat(thumb-backfill): /api/thumbnails/backfill endpoint — POST → 202 + celery_task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:38 -04:00
bvandeusen a41eddae3f feat(thumb-backfill): backfill_thumbnails planner task — keyset-paginates ImageRecord, NULLs bad thumb paths, enqueues generate_thumbnail for NULL/missing/corrupt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:21 -04:00
bvandeusen 7aa7f5a3d6 feat(thumb-backfill): _thumb_is_valid helper — JPEG/PNG magic-byte check on the on-disk thumbnail file
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:35:06 -04:00
bvandeusen 5d4f223b71 Merge pull request 'Release v26.05.25.7 — FC-Cleanup tab + UniqueViolation fix + error modal + extension install fix' (#22) from dev into main 2026-05-26 08:26:46 -04:00
bvandeusen 2505b197ae feat(fc-cleanup): Pinia store + 3 cards + CleanupView + SettingsView tab + TagMaintenanceCard moved from Maintenance + ruff lint fixes — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:16:46 -04:00
bvandeusen 0d0b236ac3 feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:13:53 -04:00
bvandeusen a06ada4c9b fix(ext-ui): direct :href install button (Firefox needs anchor click, not programmatic navigation) + manifest version detection ignores -latest.xpi alias — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:07:22 -04:00
bvandeusen ebd985990c feat(ui): ErrorDetailModal — click error → flat-text modal with copy button (replaces unusable :title tooltip for multi-line SQLAlchemy tracebacks) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:53:14 -04:00
bvandeusen 4da8d1d774 fix(importer): race-safe savepoint-based find-or-create for Source + Post (uq_source_artist_platform_url UniqueViolation operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:51:12 -04:00
bvandeusen 05090c6e85 Merge pull request 'Release v26.05.25.7 — animated-WebP worker fix + FC-Cleanup backend' (#21) from dev into main 2026-05-26 01:48:13 -04:00
bvandeusen 2d4bfa4375 fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:32:50 -04:00
bvandeusen 6ed2021ad6 feat(fc-cleanup): scan_library_for_rule Celery task + maintenance-queue registration — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:21:37 -04:00
bvandeusen 4f2ceaaf31 feat(fc-cleanup): audit lifecycle service functions (start/apply/cancel) + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:41 -04:00
bvandeusen 8a5b337a53 feat(fc-cleanup): min-dimension service functions + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:04 -04:00
bvandeusen 900d878d27 feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:38 -04:00
bvandeusen fd80d40a34 feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:07 -04:00
bvandeusen 929d3fc092 feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:17:42 -04:00
bvandeusen c0c9e56fb9 fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:45:08 -04:00
bvandeusen 3a577d5ade Merge pull request 'fix(ext-ci): use browser_download_url + curl -f + ZIP magic check (XPI silently corrupt)' (#20) from dev into main 2026-05-26 00:43:02 -04:00
bvandeusen 06a2f60c08 fix(ext-ci): use browser_download_url not /releases/assets/<id> + add -f to curl + magic-byte sanity check (operator-flagged 2026-05-26: prior build silently wrote '404 page not found' into the XPI file, Firefox rejected as corrupt) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:42:35 -04:00
bvandeusen 0978fbac66 fix(sidecar): strip gallery-dl 'NN_' numbering prefix when locating sidecars — fixes 'deep scan refresh count high but 0 Posts created' (operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:27:43 -04:00
bvandeusen f4fe02e346 Merge pull request 'fix(ext-ci): drop actions/upload-artifact (Forgejo doesn't support v4+ GHES)' (#19) from dev into main 2026-05-25 23:33:40 -04:00
bvandeusen efb142239d fix(ext-ci): drop actions/upload-artifact (Forgejo Actions doesn't support v4+ GHES) — build-web reads XPI directly from the ext-<version> Forgejo release asset — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:32:58 -04:00
bvandeusen e766197d99 Merge pull request 'fix(ext-ci): jq→python + bump ext to 1.0.3 + rollback-on-upload-failure' (#18) from dev into main 2026-05-25 23:14:51 -04:00
bvandeusen 5587a76606 fix(ext-ci): replace jq with python3 (jq not in ci-python image) + bump ext 1.0.2→1.0.3 (escape AMO 'version already exists' from prior partial-failure run) + add rollback to prevent empty cache-release tombstones — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:14:11 -04:00
bvandeusen 3872e1dda9 Merge pull request 'fix(ext-ci): web-ext v8 .cjs config workaround' (#17) from dev into main 2026-05-25 22:49:14 -04:00
bvandeusen 17e19081a2 fix(ext-ci): drop web-ext-config.cjs (v8 mis-parses .cjs configs as if module.exports were a config option) — inline ignore-files on CLI + --no-config-discovery — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:46:58 -04:00
bvandeusen 9814f3dbaf Merge pull request 'Release v26.05.25.5 — Extension publish refactor, deep-scan IR-parity, archive-import perf, artist Settings tab' (#16) from dev into main 2026-05-25 22:44:59 -04:00
bvandeusen 770bcf3aa6 feat(artist): tab split (Overview/Settings) so DangerZone is reachable without exhausting the infinite-scroll image grid — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:32:07 -04:00
bvandeusen 52d7905c43 perf(importer): cache phash candidates on Importer to fix archive-import soft-timeout (was O(M×N) per-member SELECTs) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:25:32 -04:00
bvandeusen e6ededbe8e feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:16:26 -04:00
bvandeusen c06cbc0abe feat(ci): inline extension sign into build.yml + Forgejo Release Assets as XPI cache (v26.05.25.5) — bump ext to 1.0.2 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 21:30:11 -04:00
bvandeusen b214460fdb Merge pull request 'Release v26.05.25.4 — importer ext sanitize fix, CI shard split, BrowserExtensionCard on Overview' (#15) from dev into main 2026-05-25 21:11:50 -04:00
bvandeusen ac39509a74 fix(ci): rename shard jobs to no-separator names (intapi/intimp/intcore) + add diagnostic docker ps dump so next bounce surfaces the real act_runner naming convention — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:59:17 -04:00
bvandeusen 3531f373ee feat(settings): move BrowserExtensionCard from Maintenance to Overview tab — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen 36cc0622cb fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen e50f92d900 perf(ci): shard integration suite into 3 parallel jobs (int_api, int_imp, int_core) — newly feasible after act_runner capacity 2→6 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen ba8d9b112d fix(ext-ci): add diagnostic tracing to commit step to surface why run #309 reported success without producing the XPI side-commit — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen c451061ca5 fix(ext-ci): self-retrigger workflow on its own edits (path filter includes workflow file) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:24:31 -04:00
bvandeusen ac55d0e8d8 Merge pull request 'fix(ext-ci): match AMO-renamed signed XPI' (#14) from dev into main 2026-05-25 18:22:50 -04:00
bvandeusen 47d760550d fix(ext-ci): glob AMO-renamed signed XPI + canonicalize to fabledcurator-<version>.xpi — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:22:01 -04:00
bvandeusen 89a89e0ded Merge pull request 'Release v26.05.25.3 — ML embedder SigLIP fix, import-UX, extension publish' (#13) from dev into main 2026-05-25 17:56:50 -04:00
bvandeusen dc3bce7fc1 chore(ext): bump to 1.0.1 to trigger initial sign-and-publish — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen f657582f30 feat(import-ui): deep scan button, sticky settings tabs, tasks-above-filters, fix Scanning-undefined source_path — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen 111b952535 fix(ml): load SigLIP image-only processor to avoid SentencePiece dep — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:31:06 -04:00
bvandeusen 4e9aac2c05 Merge pull request 'v26.05.25.2: supersede + sidecar enrichment, scan toast feedback, CI uv + pip cache + durations' (#12) from dev into main 2026-05-25 14:30:25 -04:00
bvandeusen a0470b5f60 feat(importer): _supersede() now applies the new (larger) file's sidecar — operator wanted to scan GS download dir to supersede smaller IR-migrated images AND wire up gallery-dl Post metadata, but supersede was file-only and silently dropped the sidecar.
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.

Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:51:51 -04:00
bvandeusen b0bb7ae6cc ci: pip wheel cache (actions/cache on requirements.txt hash) + uv-when-available — ~2 min saved on warm runs, no risk
uv falls back to pip install on runners without uv binary, so this
change is forward-compatible with the current ci-python image. When
the runner image gets uv pre-installed in a future bump, the warm
install path drops from ~2 min to ~10 seconds.

pytest-xdist parallelization is OUT OF SCOPE for this commit:
tests/conftest.py uses a TRUNCATE ALL TABLES RESTART IDENTITY CASCADE
fixture after every integration test against a single shared
database; xdist workers running in parallel would nuke each other's
mid-test state. A future refactor to per-worker databases or
per-worker schema isolation is the prerequisite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:47:27 -04:00
bvandeusen 1bbe478fd0 ci: report slowest 25 integration tests via pytest --durations=25 — instrumentation pass before deciding parallelization vs targeted slow-test fixes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:09:25 -04:00
bvandeusen 5666fd5ca5 fix(ui): scan trigger immediate-feedback toast + delayed status re-poll — operator-flagged 'click does nothing' was actually scan_directory's skip-set finalizing the batch in <100ms when every file already had an ImportTask row, before refreshStatus could ever see the active state. Now the click always produces visible feedback (immediate 'Scan triggered' + 2s 'no new files' if it quick-finalizes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:06:11 -04:00
bvandeusen 2879ac6f2b Merge pull request 'v26.05.25.1: maintenance sweep + Camie v2 + corrupt-file handling + post-date gallery + clear-stuck escape hatch' (#11) from dev into main 2026-05-25 12:57:46 -04:00
bvandeusen 3a359f6c5e fix(ui): Quick scan button always visible (disabled when active batch present) + inline Clear stuck action — operator was clicking a spinner area thinking it was the button because activeBatch hid the Quick scan button entirely
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:43:37 -04:00
bvandeusen b6a917ac81 feat(import): /api/import/clear-stuck endpoint + Clear stuck UI button — escape hatch for the autoretry-loop case the automatic sweep can't break
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.

/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
  marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
  underlying cause is resolved

UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.

Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:37:07 -04:00
bvandeusen c361032554 feat(gallery): sort/group/jump by COALESCE(post.post_date, image_record.created_at) — surface migrated content at its original publish date, not FC scan date
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.

Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
  ImageProvenance (only if currently NULL — preserves the canonical
  download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
  COALESCE(post.post_date, image_record.created_at), used in:
    * scroll() ORDER BY + cursor WHERE clauses
    * timeline() year/month group-by
    * jump_cursor() year/month filter
    * _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
  works for images without a post (NULL on the Post side, fall back
  to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
  /scroll exposes both alongside the existing created_at so the UI
  can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.

Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.

To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:30:18 -04:00
bvandeusen 9f54efdedf fix(migrators): tag_apply phase 4 now covers deviantart + pixiv (was silently dropping IR PostMetadata from those platforms)
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.

Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.

To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:25:45 -04:00
bvandeusen 6b1bb87647 fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:12:55 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:34:36 -04:00
bvandeusen 52445eb501 fix(ui): double directory card width (220px -> 440px) + bound preview slot height (min 150 / max 220 / overflow hidden / explicit display+object-position) so tall source images can't escape on browsers that don't compute aspect-ratio
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:32:05 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output)
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:25:30 -04:00
bvandeusen 9d5abb09f6 fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:00:29 -04:00
bvandeusen b8dce6c483 Merge pull request 'FC-3h + FC-3k: backup first-class + admin destructive actions' (#10) from dev into main 2026-05-25 01:41:53 -04:00
bvandeusen 832345a245 fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:28:54 -04:00
bvandeusen a0136fa30d fix(fc3k): add mime=image/jpeg to test ImageRecord ctors (NOT NULL) + reorder admin import after stdlib/3rd-party (ruff I001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:12:52 -04:00
bvandeusen de1a4b64b7 fc3k(ui): TagMaintenanceCard — preview-then-commit prune-unused under Settings → Maintenance
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:52:30 -04:00
bvandeusen 3f500e592e fc3k(ui): per-tag dots-menu with Merge + Delete actions; Tier-B count-surfacing modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:51:51 -04:00
bvandeusen d97e3f9b59 fc3k(ui): bulk-delete action in BulkEditorPanel with sha8 confirm token + projected counts modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:49:45 -04:00
bvandeusen 035c49f675 fc3k(ui): ArtistDangerZone card + slot at bottom of ArtistView with cascade-delete flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:48:17 -04:00
bvandeusen e41ab1cca5 fc3k(ui): Pinia admin store — six endpoints + task_run polling for Tier-C dispatched ops
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:47:09 -04:00
bvandeusen 42c6b642c2 fc3k(ui): rename + relocate BackupConfirmModal → modal/DestructiveConfirmModal; add tier + projectedCounts props
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:46:37 -04:00
bvandeusen ad3d34a1fc fc3k: /api/admin endpoint integration tests — dry-run, confirm-mismatch, dispatch, Tier-A/B/C paths
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:45:31 -04:00
bvandeusen b5289ed372 fc3k: admin Celery task tests — registration, success, failure, missing-id idempotency
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:44:33 -04:00
bvandeusen f6aa805725 fc3k: cleanup_service unit tests — projections + mutations + file unlinks against real Postgres
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:43:51 -04:00
bvandeusen f096c9a5fb fc3k: register admin_bp in api/__init__ all_blueprints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:40 -04:00
bvandeusen 676a86b514 fc3k: /api/admin tags prune-unused endpoint — Tier-A preview-then-commit flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:20 -04:00
bvandeusen 44cc625d4a fc3k: /api/admin tag endpoints — Tier-B delete + merge + usage-count helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:03 -04:00
bvandeusen f7ee122243 fc3k: /api/admin blueprint — Tier-C artist cascade + bulk image delete with sha8-keyed confirm tokens
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:53 -04:00
bvandeusen 7c6f11964a fc3k: celery_app — register admin tasks on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:23 -04:00
bvandeusen 94c60c0af2 fc3k: admin Celery tasks — delete_artist_cascade_task + bulk_delete_images_task on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:03 -04:00
bvandeusen 6df102b83d fc3k: cleanup_service mutations — unlink primitive, artist cascade, bulk image delete, tag delete, prune unused
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:39:45 -04:00
bvandeusen 2ae01d27e3 fc3k: cleanup_service projections — artist cascade, bulk delete, tag usage, unused tags
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:38:57 -04:00
bvandeusen 718cc79905 fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:37:06 -04:00
bvandeusen e78a35d333 fc3h: collapse multi-line sqlalchemy import in backup_run.py — fits under line-length=100, ruff I001 would bounce
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:12:06 -04:00
bvandeusen 83bd3b4b2d fc3h(ui): slot BackupCard into Maintenance panel above migration card
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:22 -04:00
bvandeusen aecedd9fe4 fc3h(ui): BackupCard.vue + BackupRunsTable.vue — combined card with DB + Images sub-sections
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:00 -04:00
bvandeusen 1e34b1b428 fc3h(ui): BackupConfirmModal.vue — typed-token confirmation for restore + delete
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:05:04 -04:00
bvandeusen 102c21feaa fc3h(ui): Pinia backup store — runs, triggers, restore, delete, tag, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:04:37 -04:00
bvandeusen 57a338f7e6 fc3h(tests): drop pinned migration-backup tests (retired surface; coverage moved to test_backup_service.py + test_api_system_backup.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:03:57 -04:00
bvandeusen d04983138a fc3h: /api/system/backup endpoint integration tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:02:32 -04:00
bvandeusen 9ec6fdb596 fc3h: Celery task integration tests (backup, restore, prune, nightly)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:01:39 -04:00
bvandeusen 86ad9b80e9 fc3h: backup_service unit tests (subprocess monkeypatched)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:40 -04:00
bvandeusen 2b05f147f4 fc3h: migrators docstring — note backup/rollback retired to backup_service.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:01 -04:00
bvandeusen 70e1e010d1 fc3h: remove backend/app/services/migrators/backup.py + rollback.py (relocated to backup_service.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:18 -04:00
bvandeusen d3d4320ed5 fc3h: retire backup + rollback from migrate API/task — moved to /api/system/backup/* per FC-3h
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:11 -04:00
bvandeusen 7d42cddb11 fc3h: /api/system/backup blueprint — trigger, list, get, patch, restore, delete, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:57:25 -04:00
bvandeusen 1f01c4819a fc3h: celery_app — register backup tasks (include + maintenance route + Beat hourly tick + daily prune)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:56:06 -04:00
bvandeusen 06d527cb92 fc3h: prune_backups (daily retention) + backup_db_nightly (hourly tick, settings-gated)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:55:19 -04:00
bvandeusen e9ea376aed fc3h: restore_db_task + restore_images_task — restore creates 'restoring' marker row linked via restored_from_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:50 -04:00
bvandeusen 882cb491ba fc3h: backup_db_task + backup_images_task Celery tasks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:26 -04:00
bvandeusen 319e7de547 fc3h: backup_service.py — DB + images backup/restore + unlink helpers (relocated, split per-kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:35 -04:00
bvandeusen e43312a129 fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:00 -04:00
bvandeusen 8f2732a56f fc3h: alembic 0017 — backup_run table with indexes + partial-tag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:51:22 -04:00
bvandeusen c3e855bd9b fc3h: BackupRun model — artifact record for backup/restore runs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:50:59 -04:00
bvandeusen d1c0b82a22 Merge pull request 'v26.05.24.3: FC-3i System Activity dashboard + migration backup-gate retired + modal Escape' (#9) from dev into main 2026-05-24 21:47:53 -04:00
bvandeusen 37fcc74954 fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:22:42 -04:00
bvandeusen 6a532c1497 fc3i(ui): Settings → Activity tab + Overview summary card
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:10:14 -04:00
bvandeusen 368613068a fc3i(ui): SystemActivityTab.vue — queues + failures + all-activity panes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:09:20 -04:00
bvandeusen ddbb84d8aa fc3i(ui): SystemActivitySummary.vue — Overview-tab quick summary card (5s poll)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:08:30 -04:00
bvandeusen 9b252948f9 fc3i(ui): QueuesTable.vue — shared queue/worker/recent table (compact + detailed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:07:57 -04:00
bvandeusen 89d0cb2124 fc3i(ui): systemActivity Pinia store — queues, workers, runs, failures
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:07:28 -04:00
bvandeusen 36611cbe00 fc3i: tests for recover_stalled_task_runs + prune_task_runs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:07:01 -04:00
bvandeusen 48ef22445a fc3i: integration tests for /api/system/activity/* endpoints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:06:20 -04:00
bvandeusen e523d0ac94 fc3i: integration tests for Celery signal → task_run lifecycle
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:05:32 -04:00
bvandeusen 541e2bfe6a fc3i: /api/system/activity blueprint — queues, workers, runs, failures
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:04:34 -04:00
bvandeusen 7782672a51 fc3i: recover_stalled_task_runs + prune_task_runs maintenance tasks + Beat entries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:03:33 -04:00
bvandeusen d12b51f6b7 fc3i: Celery signal handlers populate task_run on every task lifecycle event
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:01:47 -04:00
bvandeusen 0cda46fcdb fc3i: alembic 0016 — task_run table with single + composite indexes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:00:46 -04:00
bvandeusen 79fee98db4 fc3i: TaskRun model — per-Celery-task lifecycle audit row
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 21:00:14 -04:00
bvandeusen b1d68929c5 fix(tests): drop test_post_apply_without_backup_rejected (gate retired in 5535677)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:15:10 -04:00
bvandeusen 553567738e fix: drop migration backup-gate (FC-3h supersedes) + modal Escape via document-level listener so video focus can't swallow it
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 14:38:33 -04:00
bvandeusen 5526b8dc78 Merge pull request 'v26.05.24.2: IR Post/Provenance restore + modal artist fallback' (#8) from dev into main 2026-05-24 14:30:06 -04:00
bvandeusen c9a3f12847 fix(lint): tag_apply.py — one blank line between imports and module-level comment (ruff I001, third bounce on this rule)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 14:22:05 -04:00
bvandeusen 538c1591e8 fc-3g-ext: IR Post/Provenance restore (tag_apply phase 4) + modal artist fallback
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 14:08:18 -04:00
195 changed files with 17949 additions and 1982 deletions
+244 -5
View File
@@ -2,26 +2,259 @@ name: Build images
on:
push:
branches: [dev, main]
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
# merge-to-main, not from the dev branch image. Saves one full docker
# build per dev push.
branches: [main]
# Tag-push triggers an immutable per-version image build (e.g.
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
# negligible per tag. Doesn't overlap with the push-to-main build (that
# one publishes `:main` + `:latest`; the tag-push build publishes only
# `:<tag>`).
tags: ['v*']
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com)
# - write:release (for future release-cutting workflows)
# - write:release (for ext-<version> release asset cache)
# - write:issue (for future issue-management automation)
# The injected GITHUB_TOKEN cannot be used — it lacks write:package.
jobs:
build-web:
# Sign-or-fetch-from-cache: signs the extension via AMO if no ext-<version>
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
# Result is uploaded as an Actions artifact for build-web to consume.
#
# Why this lives in build.yml (not a separate workflow): the merge-commit's
# docker image tagged `:latest` MUST carry the XPI. A separate sign workflow
# racing build.yml leaves `:latest` without the XPI for ~5min (until the
# commit-back triggers another build). Inline ordering eliminates the race.
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
# branch to manage). AMO blocks re-signing the same version (returns 409),
# so signing is intentionally one-shot per version bump.
sign-extension:
if: github.ref == 'refs/heads/main'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Resolve extension version
id: extver
run: |
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Resolved extension version: $VERSION"
- name: Check Forgejo release-asset cache
id: cache
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eu
VERSION=${{ steps.extver.outputs.version }}
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
echo "Tag lookup HTTP status: $STATUS"
# JSON parsing via python (ci-python:3.14 has stdlib json; jq is
# not in the image and adding it per ci-requirements.md is not
# warranted for a single consumer — operator-flagged 2026-05-26
# after a sign job failed with `jq: not found`).
if [ "$STATUS" = "200" ]; then
ASSET_ID=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['id'] if xpis else '')")
if [ -n "$ASSET_ID" ]; then
echo "cached=true" >> "$GITHUB_OUTPUT"
echo "asset_id=$ASSET_ID" >> "$GITHUB_OUTPUT"
echo "Cached XPI exists at ext-$VERSION (asset id $ASSET_ID); skipping AMO sign"
else
echo "cached=false" >> "$GITHUB_OUTPUT"
echo "Release ext-$VERSION exists but has no .xpi asset; will re-sign + re-upload"
fi
else
echo "cached=false" >> "$GITHUB_OUTPUT"
echo "No release named ext-$VERSION; will sign via AMO and upload"
fi
# No "download cached XPI in sign-extension" step: build-web
# fetches directly from the Forgejo ext-<version> release asset
# (removed 2026-05-26 alongside the actions/upload-artifact
# removal — sign-extension's job is just to ensure the cache
# exists on Forgejo; the build-web side reads it independently).
- name: Sign via AMO (cache miss)
if: steps.cache.outputs.cached != 'true'
run: |
cd extension && npm install --no-save --no-audit --no-fund && npm run sign
env:
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
- name: Upload signed XPI to ext-<version> release (cache miss)
if: steps.cache.outputs.cached != 'true'
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=${{ steps.extver.outputs.version }}
# AMO renames signed XPIs with its internal addon-id-safe-string;
# canonicalize to fabledcurator-<version>.xpi so the FC server's
# whitelist (backend/app/frontend.py expects 'fabledcurator-*.xpi')
# keeps working.
SIGNED=$(ls extension/web-ext-artifacts/*.xpi | head -1)
XPI="extension/web-ext-artifacts/fabledcurator-$VERSION.xpi"
cp "$SIGNED" "$XPI"
# Find-or-create the ext-<version> release. Track whether WE
# created it so an upload failure below can roll back (don't
# leave an empty release tombstone that the next run's
# cache-check mistakes for a partial-failure state).
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
if [ "$STATUS" = "200" ]; then
CREATED_BY_US=false
else
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"main\"}" \
-o release.json \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
CREATED_BY_US=true
fi
RELEASE_ID=$(python3 -c "import json; print(json.load(open('release.json'))['id'])")
test -n "$RELEASE_ID"
# Rollback-on-failure: if the asset upload fails AND we just
# created the release in this run, delete it. Prevents an empty
# ext-<version> release from poisoning the next workflow run
# (operator-flagged 2026-05-26 — without rollback the next run
# saw 'release exists, no asset → cache miss → sign' which AMO
# then rejected with 409 'Version already exists').
rollback_if_we_created() {
if [ "$CREATED_BY_US" = "true" ]; then
echo "Rolling back: deleting just-created release $RELEASE_ID"
curl -s -X DELETE -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID" || true
curl -s -X DELETE -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/tags/ext-$VERSION" || true
fi
}
trap 'rollback_if_we_created' EXIT
HTTP_CODE=$(curl -s -X POST -H "Authorization: token $TOKEN" \
-F "attachment=@$XPI" \
-o /dev/null -w "%{http_code}" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID/assets?name=fabledcurator-$VERSION.xpi")
if [ "$HTTP_CODE" != "201" ] && [ "$HTTP_CODE" != "200" ]; then
echo "Asset upload failed with HTTP $HTTP_CODE"
exit 1
fi
# Upload succeeded — clear the rollback trap.
trap - EXIT
echo "Uploaded fabledcurator-$VERSION.xpi to ext-$VERSION release"
# No actions/upload-artifact step: Forgejo Actions (and our
# act_runner) doesn't support upload-artifact@v4+ (GHES limitation
# surfaced 2026-05-26). Instead build-web reads the signed XPI
# straight from the ext-<version> Forgejo release we just uploaded
# to. Same source of truth; no double-store.
build-web:
needs: [sign-extension]
# sign-extension is main-only; on dev it's skipped, build-web still runs.
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Download signed XPI from Forgejo release asset (main + tags)
# Fires on main-push AND on tag-push. Tag-push builds re-package the
# same source code as the preceding main-push build but with an
# immutable version tag — they need the XPI too, otherwise the
# versioned image ships without the signed extension.
#
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
# simultaneously. Main-push runs sign-extension (1-5min AMO round
# trip) before publishing the ext-<version> release; tag-push
# skips sign-extension (gated to main) and races straight to
# this download step. Tag-push lost every time. Fix: poll the
# ext-<version> release endpoint with a sleep+retry loop (30s
# for up to 10min total) before giving up. Main-push's signing
# eventually wins and tag-push picks the release up on a later
# iteration.
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
# Poll for the ext-<version> release. main-push's sign-extension
# step (AMO round-trip, 1-5min) needs to finish + upload before
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
for attempt in $(seq 1 20); do
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
if [ "$STATUS" = "200" ]; then
echo "Found ext-$VERSION release on attempt $attempt"
break
fi
if [ "$attempt" = "20" ]; then
echo "ERROR: ext-$VERSION release not available after 10min of polling"
echo "Last HTTP status: $STATUS"
exit 1
fi
echo "Attempt $attempt: ext-$VERSION not yet published (HTTP $STATUS); sleeping 30s"
sleep 30
done
# Extract the .xpi asset's browser_download_url (Forgejo's
# /releases/assets/<id> endpoint returns ASSET METADATA, not
# the binary blob — operator-flagged 2026-05-26: my prior
# code curl'd the metadata endpoint without -f and wrote the
# resulting 404-page-not-found text into fabledcurator-*.xpi,
# which Firefox then rejected as "corrupt").
# browser_download_url is the canonical binary endpoint and
# is also publicly accessible (no token needed) but we pass
# the token anyway for symmetry with private-repo support.
DOWNLOAD_URL=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['browser_download_url'])")
test -n "$DOWNLOAD_URL"
echo "Downloading XPI from: $DOWNLOAD_URL"
mkdir -p frontend/public/extension
DEST="frontend/public/extension/fabledcurator-$VERSION.xpi"
# -f = fail on HTTP error (prevents silent corruption like the
# 2026-05-26 incident); -L = follow redirects.
curl -sfL -H "Authorization: token $TOKEN" -o "$DEST" "$DOWNLOAD_URL"
# Sanity check: the binary should start with the ZIP magic (PK\x03\x04).
# If it's anything else, the next docker build will ship a corrupt XPI.
MAGIC=$(head -c 2 "$DEST" | od -An -c | tr -d ' \n')
if [ "$MAGIC" != "PK" ]; then
echo "ERROR: downloaded XPI does not start with ZIP magic 'PK' (got '$MAGIC')"
echo "File contents preview:"
head -c 200 "$DEST"
exit 1
fi
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
ls -la frontend/public/extension/
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Three trigger shapes:
# refs/tags/v… → tag-push: publish ONLY the immutable version
# tag (e.g. :v26.05.26.5). Don't touch :latest;
# that already got published by the main-push
# build for the merge commit.
# refs/heads/main → push to main (incl. PR merge commits):
# publish :main + :latest (floating).
# anything else → safety net; shouldn't fire given the `on:`
# config above (dev was dropped). Tag :dev to
# surface the unexpected run in the registry.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
@@ -52,7 +285,13 @@ jobs:
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Mirrors build-web's three-shape logic (tag-push / main-push /
# safety-net dev). The -ml image follows the same release cadence
# as the web image.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
+183 -24
View File
@@ -8,8 +8,10 @@ name: CI
on:
push:
branches: [dev, main]
pull_request:
branches: [main]
# pull_request trigger intentionally absent — with branches: [dev, main]
# above, every PR commit already fires CI via the push event on dev. Adding
# pull_request would duplicate runs on dev→main PRs. FC has no fork PRs
# (single-operator Forgejo repo) so push coverage is complete.
jobs:
backend-lint-and-test:
@@ -24,11 +26,30 @@ jobs:
steps:
- uses: actions/checkout@v4
# Cache step removed 2026-05-26: act_runner's cache backend has been
# broken on this homelab runner since 2026-05-15 (first as request-
# timeout warnings, then as hard "Cannot find module .../dist/restore/
# index.js" failures that tank the whole job). The cache step targeted
# ~/.cache/pip but the install below uses `uv pip install` primarily,
# whose own cache lives at ~/.cache/uv — so the cache step's real
# benefit was marginal even when working. Cost of removal: ~30s of
# wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
# docker volume at the runner level (skips actions/cache entirely),
# or fix the runner-side cache backend (clear /var/run/act/actions/*,
# pin act_runner version, etc.).
- name: Install Python deps
# ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
# Dockerfile's RUFF_VERSION). Per FabledRulebook ci-runners.md, toolchain
# versions live on the runner image, not here.
run: pip install -r requirements.txt pytest pytest-asyncio
# uv: 5-10x faster wheel resolve than pip for cold caches.
# Falls back to pip install on uv-missing runners (older images).
run: |
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
- name: Ruff lint
run: ruff check backend/ tests/ alembic/
@@ -57,17 +78,28 @@ jobs:
- run: npm run test:unit
- run: npm run build
integration:
# This act_runner (swarm-runner v0.6.1) puts service containers on the
# default bridge with NO service-name DNS, and publishing fixed host
# ports collides with the operator's running docker-compose dev stack on
# the same shared daemon. Workaround: publish NO host ports, and reach
# each service by its bridge IP — discovered at runtime via the mounted
# docker socket (the ci-python image ships /usr/bin/docker). Default-bridge
# containers can talk by IP (only embedded DNS is missing), so IP
# addressing is reliable here. Everything runs in ONE step so resolved
# values don't depend on cross-step env passing. Pattern documented in
# FabledRulebook/forgejo.md "CI philosophy".
# Integration suite split into THREE parallel shards (2026-05-25, runner
# capacity bumped 2→6). Each shard gets its own Postgres + Redis service
# set and runs alembic + a disjoint subset of integration tests. Shards
# share no DB state, so the autouse TRUNCATE fixture in tests/conftest.py
# stays single-threaded per shard but multiple shards run in parallel
# wall-clock. Approximate split — rebalance once --durations=15 output
# reveals which shard is the long pole.
#
# Each shard's docker-ps filter uses its own unique job name to scope
# service-container resolution. act_runner appears to strip underscores
# from job names when building container labels — `int_api` yielded
# zero matches on 2026-05-25 — so shards use no-separator names
# (`intapi`, `intimp`, `intcore`) instead. Each step prints
# `docker ps -a` first so a future naming-convention shift surfaces in
# the log without another guess-and-push cycle.
#
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT
# done — per ci-requirements.md, FC is the only Python consumer of that
# image and the CI-Runner project's "add deps to image when used by >1
# project" rule keeps the install per-job.
intapi:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
@@ -98,15 +130,14 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Integration suite (resolve service IPs, migrate, test)
- name: API integration shard (resolve service IPs, migrate, test)
run: |
set -eux
# Scope to THIS job's service containers (act_runner names them
# ...JOB-integration...); the operator's compose stack uses the
# same images but different names, so it won't match.
PG=$(docker ps --filter "name=JOB-integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=JOB-integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intapi" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intapi" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -114,11 +145,139 @@ jobs:
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
# Wait for Postgres to accept TCP (bash /dev/tcp; no extra tools).
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
pip install -r requirements.txt pytest pytest-asyncio
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration
pytest tests/test_api_*.py -v -m integration --durations=15
intimp:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Importer integration shard (resolve service IPs, migrate, test)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intimp" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intimp" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/test_importer*.py tests/test_import_*.py tests/test_migration_*.py tests/test_phash_*.py tests/test_sidecar_*.py tests/test_scan_*.py tests/test_archive_extractor.py tests/test_backfill_phash.py -v -m integration --durations=15
intcore:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intcore" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intcore" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration --durations=15 \
--ignore-glob='tests/test_api_*.py' \
--ignore-glob='tests/test_importer*.py' \
--ignore-glob='tests/test_import_*.py' \
--ignore-glob='tests/test_migration_*.py' \
--ignore-glob='tests/test_phash_*.py' \
--ignore-glob='tests/test_sidecar_*.py' \
--ignore-glob='tests/test_scan_*.py' \
--ignore-glob='tests/test_archive_extractor.py' \
--ignore-glob='tests/test_backfill_phash.py'
+12 -42
View File
@@ -1,8 +1,19 @@
name: extension
# Lint-only workflow. The sign-and-publish dance moved into build.yml's
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
# because sign-extension runs as a build-web dependency in the SAME workflow,
# eliminating the prior race between build.yml and a separate extension.yml.
# Signed XPIs are cached in Forgejo Release Assets named `ext-<version>`.
on:
push:
branches: [dev, main]
paths: ['extension/**']
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
workflow_dispatch:
jobs:
@@ -16,44 +27,3 @@ jobs:
run: cd extension && npm install --no-save --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
sign-and-publish:
needs: lint
if: github.ref == 'refs/heads/main'
runs-on: python-ci
container:
image: node:22-bookworm-slim
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.RELEASE_TOKEN }}
- name: Install web-ext + git
run: |
apt-get update && apt-get install -y --no-install-recommends git ca-certificates
cd extension && npm install --no-save --no-audit --no-fund
- name: Sign XPI
run: cd extension && npm run sign
env:
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
- name: Commit signed XPI to frontend/public/extension/
run: |
set -e
XPI=$(ls extension/web-ext-artifacts/fabledcurator-*.xpi | head -1)
if [ -z "$XPI" ]; then
echo "No XPI produced by web-ext sign — exiting"
exit 1
fi
mkdir -p frontend/public/extension
cp "$XPI" frontend/public/extension/
# Also copy as -latest.xpi so the FC server can serve a stable URL.
cp "$XPI" "frontend/public/extension/fabledcurator-latest.xpi"
git config user.name "FC extension CI"
git config user.email "noreply@fabledsword.com"
git add frontend/public/extension/
if git diff --cached --quiet; then
echo "No changes to commit"
else
git commit -m "ext: publish signed XPI $(basename $XPI)"
git push origin HEAD:main
fi
+86
View File
@@ -0,0 +1,86 @@
"""fc3i: task_run table
Revision ID: 0016
Revises: 0015
Create Date: 2026-05-24
Additive only. New table records every Celery task attempt via signal
handlers (backend.app.celery_signals). Status is plain String(16) not
Postgres ENUM (per feedback_check_existing_enums: ENUM columns hard-
fail at INSERT, String columns extend cleanly).
Composite indexes anticipate the three dashboard panes:
- (queue, started_at desc) — per-lane recent activity
- (status, started_at desc) — recent failures pane
- (task_name, started_at desc) — drill-down by task
Indexed columns get individual indexes via `index=True` on the model;
the composites below cover the multi-column lookups.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0016"
down_revision: Union[str, None] = "0015"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"task_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("celery_task_id", sa.String(length=64), nullable=False),
sa.Column("queue", sa.String(length=32), nullable=False),
sa.Column("task_name", sa.String(length=128), nullable=False),
sa.Column("target_id", sa.Integer(), nullable=True),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("duration_ms", sa.Integer(), nullable=True),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column("error_type", sa.String(length=128), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("retry_count", sa.Integer(), nullable=True),
sa.Column("worker_hostname", sa.String(length=128), nullable=True),
sa.Column("args_summary", sa.String(length=255), nullable=True),
)
# Single-column indexes (matches Mapped[...].index=True on model).
op.create_index("ix_task_run_celery_task_id", "task_run", ["celery_task_id"])
op.create_index("ix_task_run_queue", "task_run", ["queue"])
op.create_index("ix_task_run_task_name", "task_run", ["task_name"])
op.create_index("ix_task_run_started_at", "task_run", ["started_at"])
op.create_index("ix_task_run_finished_at", "task_run", ["finished_at"])
op.create_index("ix_task_run_status", "task_run", ["status"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_task_run_queue_started",
"task_run", ["queue", sa.text("started_at DESC")],
)
op.create_index(
"ix_task_run_status_started",
"task_run", ["status", sa.text("started_at DESC")],
)
op.create_index(
"ix_task_run_name_started",
"task_run", ["task_name", sa.text("started_at DESC")],
)
def downgrade() -> None:
op.drop_index("ix_task_run_name_started", table_name="task_run")
op.drop_index("ix_task_run_status_started", table_name="task_run")
op.drop_index("ix_task_run_queue_started", table_name="task_run")
op.drop_index("ix_task_run_status", table_name="task_run")
op.drop_index("ix_task_run_finished_at", table_name="task_run")
op.drop_index("ix_task_run_started_at", table_name="task_run")
op.drop_index("ix_task_run_task_name", table_name="task_run")
op.drop_index("ix_task_run_queue", table_name="task_run")
op.drop_index("ix_task_run_celery_task_id", table_name="task_run")
op.drop_table("task_run")
+82
View File
@@ -0,0 +1,82 @@
"""fc3h: backup_run table
Revision ID: 0017
Revises: 0016
Create Date: 2026-05-24
Additive. New table records every backup/restore attempt with artifact
metadata. Lifecycle tracking lives in task_run from FC-3i; this is
artifact-only (paths, sizes, tag, restore lineage).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0017"
down_revision: Union[str, None] = "0016"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"backup_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("tag", sa.String(length=64), nullable=True),
sa.Column("triggered_by", sa.String(length=32), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("sql_path", sa.Text(), nullable=True),
sa.Column("tar_path", sa.Text(), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"manifest", sa.JSON(), nullable=False, server_default="{}",
),
sa.Column(
"restored_from_id", sa.Integer(),
sa.ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
),
)
# Single-column indexes (matches Mapped[...].index=True).
op.create_index("ix_backup_run_kind", "backup_run", ["kind"])
op.create_index("ix_backup_run_status", "backup_run", ["status"])
op.create_index("ix_backup_run_tag", "backup_run", ["tag"])
op.create_index("ix_backup_run_started_at", "backup_run", ["started_at"])
op.create_index("ix_backup_run_finished_at", "backup_run", ["finished_at"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_backup_run_kind_started",
"backup_run", ["kind", sa.text("started_at DESC")],
)
op.create_index(
"ix_backup_run_status_finished",
"backup_run", ["status", sa.text("finished_at DESC")],
)
# Partial index: only tagged rows participate in retention-exempt query.
op.create_index(
"ix_backup_run_tag_partial",
"backup_run", ["tag"],
postgresql_where=sa.text("tag IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_backup_run_tag_partial", table_name="backup_run")
op.drop_index("ix_backup_run_status_finished", table_name="backup_run")
op.drop_index("ix_backup_run_kind_started", table_name="backup_run")
op.drop_index("ix_backup_run_finished_at", table_name="backup_run")
op.drop_index("ix_backup_run_started_at", table_name="backup_run")
op.drop_index("ix_backup_run_tag", table_name="backup_run")
op.drop_index("ix_backup_run_status", table_name="backup_run")
op.drop_index("ix_backup_run_kind", table_name="backup_run")
op.drop_table("backup_run")
@@ -0,0 +1,62 @@
"""fc3h: backup_* knobs on import_settings
Revision ID: 0018
Revises: 0017
Create Date: 2026-05-24
Adds four columns to the singleton import_settings row:
- backup_db_nightly_enabled (default False — opt-in)
- backup_db_nightly_hour_utc (default 3)
- backup_db_keep_last_n (default 14)
- backup_images_keep_last_n (default 3)
server_default ensures the singleton row is backfilled in place
without an UPDATE statement.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0018"
down_revision: Union[str, None] = "0017"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_enabled", sa.Boolean(),
nullable=False, server_default=sa.false(),
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_hour_utc", sa.Integer(),
nullable=False, server_default="3",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_keep_last_n", sa.Integer(),
nullable=False, server_default="14",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_images_keep_last_n", sa.Integer(),
nullable=False, server_default="3",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "backup_images_keep_last_n")
op.drop_column("import_settings", "backup_db_keep_last_n")
op.drop_column("import_settings", "backup_db_nightly_hour_utc")
op.drop_column("import_settings", "backup_db_nightly_enabled")
@@ -0,0 +1,38 @@
"""import_batch.refreshed counter for deep-scan sidecar re-application
Revision ID: 0019
Revises: 0018
Create Date: 2026-05-25
Adds a `refreshed` counter to `import_batch`, mirroring the existing
`imported`/`skipped`/`failed`/`attachments` columns. Deep scan now
re-applies sidecar metadata to already-imported files (the IR feature
that didn't make the FC port the first time); a "refreshed" outcome
increments this counter so the UI can surface "X new, Y refreshed"
instead of the misleading "Scan complete — no new files" message.
server_default=0 backfills existing rows in place — no UPDATE needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0019"
down_revision: Union[str, None] = "0018"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_batch",
sa.Column(
"refreshed", sa.Integer(),
nullable=False, server_default=sa.text("0"),
),
)
def downgrade() -> None:
op.drop_column("import_batch", "refreshed")
@@ -0,0 +1,65 @@
"""fc-cleanup: library_audit_run table for async transparency/single_color audits
Revision ID: 0020
Revises: 0019
Create Date: 2026-05-26
The table backs the async audit lifecycle: rule + params snapshot, status
state machine ('running''ready''applied'/'cancelled'/'error'), and
the matched_ids JSONB array that the apply step deletes. Capped at 50k IDs
per row by the scan task (oversize = rule too aggressive, operator narrows
before re-running).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0020"
down_revision: Union[str, None] = "0019"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"library_audit_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rule", sa.String(32), nullable=False),
sa.Column("params", postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column(
"status", sa.String(16),
nullable=False, server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"scanned_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_ids", postgresql.JSONB(astext_type=sa.Text()),
nullable=False, server_default=sa.text("'[]'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
)
op.create_index(
"ix_library_audit_run_rule", "library_audit_run", ["rule"],
)
op.create_index(
"ix_library_audit_run_status", "library_audit_run", ["status"],
)
def downgrade() -> None:
op.drop_index("ix_library_audit_run_status", table_name="library_audit_run")
op.drop_index("ix_library_audit_run_rule", table_name="library_audit_run")
op.drop_table("library_audit_run")
@@ -0,0 +1,54 @@
"""provenance-race: dedupe + UNIQUE(image_record_id, post_id) on image_provenance
Revision ID: 0021
Revises: 0020
Create Date: 2026-05-26
Closes the race in Importer._apply_sidecar's existence-check + INSERT pattern.
Two workers writing for the same (image, post) pair both saw no existing row
and both inserted, leaving duplicates that then broke .scalar_one_or_none()
on every subsequent deep-scan rederive against those images
(MultipleResultsFound). Most plausibly seeded when the 5-min recovery sweep
re-enqueued a still-running long-import task and the second worker collided
with the first inside _apply_sidecar.
Migration steps:
1. DELETE all but min(id) per (image_record_id, post_id) pair. Operator's
DB had 2 affected pairs at write-time; harmless no-op if zero.
2. Add UNIQUE constraint so the importer's new savepoint+IntegrityError
recovery path can trip on collision and re-select, mirroring
uq_source_artist_platform_url and uq_post_source_external_id.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0021"
down_revision: Union[str, None] = "0020"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"""
DELETE FROM image_provenance ip1
USING image_provenance ip2
WHERE ip1.image_record_id = ip2.image_record_id
AND ip1.post_id = ip2.post_id
AND ip1.id > ip2.id
"""
)
op.create_unique_constraint(
"uq_image_provenance_image_post",
"image_provenance",
["image_record_id", "post_id"],
)
def downgrade() -> None:
op.drop_constraint(
"uq_image_provenance_image_post",
"image_provenance",
type_="unique",
)
@@ -0,0 +1,223 @@
"""source-collapse: one Source per (artist, platform) — consolidate junk per-post Sources
Revision ID: 0022
Revises: 0021
Create Date: 2026-05-26
Closes the operator-flagged 2026-05-26 issue where the filesystem importer
called _find_or_create_source(url=sd.post_url), creating one Source row per
imported post URL. Operator's Atole artist had 406 Source rows where there
should have been 1 (the /cw/Atole subscription Source).
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). Posts hang off it. The
filesystem importer was misusing Source as a per-post key.
Migration steps per (artist_id, platform) group with >1 Source:
1. Pick canonical — prefer a URL NOT matching '/posts/<id>$' (real
campaign URL like /cw/Atole); else min(id).
2. PRE-merge any Posts under non-canonical sources whose
external_post_id ALREADY exists under the canonical source. (Same
gallery-dl post imported via two different sidecar paths can plant
two Post rows with identical external_post_id under different
Sources for the same artist.) Repoint ImageProvenance +
ImageRecord.primary_post_id to the canonical-side Post, dedupe
ImageProvenance against alembic 0021's uq, then delete the
non-canonical-side Post. This MUST happen before step 3 — Postgres
fires uq_post_source_external_id row-by-row during the bulk UPDATE
and the merge-after-reparent ordering 500s on first collision
(operator-hit during v26.05.26.1 deploy, 2026-05-26).
3. Reparent remaining Posts onto canonical (no collisions possible now).
4. Reparent ImageProvenance.source_id off the non-canonical sources.
5. Delete the orphan Source rows.
6. If the canonical Source's URL still looks like a per-post URL (no
campaign URL existed among candidates), rewrite it to
'sidecar:<platform>:<artist_slug>' so the artist detail page shows
something readable.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0022"
down_revision: Union[str, None] = "0021"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_POST_URL_RE = r"/posts/[^/]+$"
def upgrade() -> None:
conn = op.get_bind()
# Find (artist_id, platform) groups with > 1 Source row.
groups = conn.execute(text("""
SELECT artist_id, platform
FROM source
GROUP BY artist_id, platform
HAVING COUNT(*) > 1
""")).fetchall()
for artist_id, platform in groups:
rows = conn.execute(
text("""
SELECT id, url FROM source
WHERE artist_id = :a AND platform = :p
ORDER BY id ASC
"""),
{"a": artist_id, "p": platform},
).fetchall()
# Canonical: first row whose URL doesn't look like a per-post URL;
# else min(id).
canonical_id = None
for sid, url in rows:
if not _matches_post_url(url):
canonical_id = sid
break
if canonical_id is None:
canonical_id = rows[0][0]
other_ids = [sid for sid, _ in rows if sid != canonical_id]
if not other_ids:
continue
# STEP 2: PRE-merge ALL Posts with duplicate external_post_id
# across the entire (canonical + others) group, BEFORE the bulk
# reparent. Two cases must both be handled:
# (A) canonical has Post X with epid=N; an "other" source has
# Post Y with epid=N → after bulk UPDATE, (canonical, N)
# collides with itself.
# (B) two different "other" sources each have a Post with
# epid=N; canonical has none → after bulk UPDATE, both
# are repointed to (canonical, N) and the second collides.
# The earlier version of this migration only handled (A); the
# operator's deploy 2026-05-26 tripped (B) at line 139.
# Fix: group ALL Posts in the (artist, platform) by epid; for
# any group with count>1, pick the keep (prefer one already
# under canonical; else lowest id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:others)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "others": other_ids},
).fetchall()
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
# Prefer a Post already under canonical as the keep.
canonical_posts = [p for p in posts if p[1] == canonical_id]
if canonical_posts:
keep_id = canonical_posts[0][0]
else:
keep_id = posts[0][0] # already sorted by id ASC
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id ALREADY has a provenance under keep —
# the UPDATE below would otherwise repoint them and
# trip uq_image_provenance_image_post (alembic 0021)
# row-by-row before any after-the-fact dedupe could
# run. Operator's v26.05.26.3 deploy 2026-05-26 tripped
# this at line 123.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
# Now safe to repoint the survivors.
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP 3: Bulk reparent the remaining Posts off the other
# Sources. After step 2, no collisions on
# (canonical, external_post_id) are possible.
conn.execute(
text("""
UPDATE post SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 4: Reparent ImageProvenance.source_id (denormalized FK).
# No UNIQUE on source_id; safe bulk update.
conn.execute(
text("""
UPDATE image_provenance SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 5: Drop the orphan Sources.
conn.execute(
text("DELETE FROM source WHERE id = ANY(:others)"),
{"others": other_ids},
)
# If the canonical's URL still looks per-post (no campaign URL
# existed among the candidates), rewrite to a synthetic anchor so
# the artist detail page renders something readable.
canonical_url = conn.execute(
text("SELECT url FROM source WHERE id = :id"),
{"id": canonical_id},
).scalar_one()
if _matches_post_url(canonical_url):
slug = conn.execute(
text("SELECT slug FROM artist WHERE id = :id"),
{"id": artist_id},
).scalar_one()
conn.execute(
text("""
UPDATE source
SET url = :new_url, enabled = false
WHERE id = :id
"""),
{
"id": canonical_id,
"new_url": f"sidecar:{platform}:{slug}",
},
)
def downgrade() -> None:
# Lossy migration — orphan Sources deleted, Posts reparented, Posts
# merged. No safe downgrade. If you need to roll back the schema
# invariant, fork from 0021 and re-run filesystem imports.
pass
def _matches_post_url(url: str) -> bool:
"""True if url ends with /posts/<token> (gallery-dl-style per-post URL)."""
import re
return bool(re.search(_POST_URL_RE, url or ""))
@@ -0,0 +1,99 @@
"""drop meta + rating tag kinds — operator-retired 2026-05-26
Revision ID: 0023
Revises: 0022
Create Date: 2026-05-26
Operator decided meta + rating aren't valid tag kinds for FC. Per-row
behavior: DELETE existing rows (operator chose "clean break" over
"convert to general"). All cascading FKs (image_tag, tag_alias,
tag_allowlist, tag_reference_embedding, tag_suggestion_rejection,
series_page) use ondelete="CASCADE" so a single DELETE on tag cleans
the related rows in one go.
After the data cleanup, recreate the tag_kind ENUM without 'meta' /
'rating' (Postgres has no `ALTER TYPE ... DROP VALUE`; standard
rename-create-cast-drop dance). The server default 'general' is
dropped before the type swap and restored after.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0023"
down_revision: Union[str, None] = "0022"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Delete tags of the retired kinds. CASCADE handles related tables.
op.execute("DELETE FROM tag WHERE kind IN ('meta', 'rating')")
# 2. Drop the CHECK constraint that references the enum's literal
# values. Postgres can't resolve `kind = 'character'` across the
# type swap below — the literal would bind to the new tag_kind
# but the column is on tag_kind_old, producing
# "operator does not exist: tag_kind = tag_kind_old".
# (Operator-hit during the v26.05.26.5 deploy attempt; ck was
# originally added by alembic 0002.) Recreated post-swap.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
# 3. Drop the server default — ALTER COLUMN TYPE can't carry it
# across the type swap below.
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
# 4. Recreate the tag_kind enum without meta/rating.
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
# 5. Restore the server default.
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
# 6. Restore the CHECK constraint (now bound to the new tag_kind).
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
def downgrade() -> None:
# Add the values back to the enum so old code can boot. The deleted
# tag rows are gone permanently — no safe restore.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post', 'meta', 'rating'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
@@ -0,0 +1,80 @@
"""backfill post.post_title from description first-line — 2026-05-27
Revision ID: 0024
Revises: 0023
Create Date: 2026-05-27
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. FC's sidecar parser was leaving
post_title NULL for every SubscribeStar post since FC-3 shipped. The
parser fix (sidecar._first_line_text fallback) now synthesizes a title
at parse time; this migration applies the same logic retroactively to
existing rows.
Operator-flagged 2026-05-27 after inspecting
/mnt/Data/Patreon/Cheunart/subscribestar/ sidecars.
Idempotent: only touches rows where post_title IS NULL or empty AND
description IS NOT NULL. Re-running the migration is a no-op.
"""
from __future__ import annotations
import re
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0024"
down_revision: Union[str, None] = "0023"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_TAG_RE = re.compile(r"<[^>]+>")
_WS_RE = re.compile(r"\s+")
def _first_line_text(body: str, limit: int = 120) -> str | None:
"""Mirror of sidecar._first_line_text. Kept inline so the migration
doesn't carry a runtime import dependency from app code that may
have moved by the time the migration is replayed years from now."""
if not body:
return None
text_ = _TAG_RE.sub(" ", body)
text_ = text_.replace("\xa0", " ")
for line in text_.splitlines():
line = _WS_RE.sub(" ", line).strip()
if line:
if len(line) > limit:
return line[: limit - 1].rstrip() + ""
return line
return None
def upgrade() -> None:
bind = op.get_bind()
rows = bind.execute(
text(
"SELECT id, description FROM post "
"WHERE (post_title IS NULL OR post_title = '') "
"AND description IS NOT NULL AND description <> ''"
)
).fetchall()
updated = 0
for row in rows:
derived = _first_line_text(row.description)
if not derived:
continue
bind.execute(
text("UPDATE post SET post_title = :t WHERE id = :id"),
{"t": derived, "id": row.id},
)
updated += 1
print(f"0024: backfilled post_title on {updated} row(s)")
def downgrade() -> None:
# No safe restore — we can't tell which post_titles were derived vs
# genuinely present. Leave the column alone on rollback.
pass
@@ -0,0 +1,288 @@
"""sidecar-audit followup: correct external_post_id + post_url across all platforms
Revision ID: 0025
Revises: 0024
Create Date: 2026-05-27
Closes the operator-flagged 2026-05-27 sidecar audit findings. Three
data-correctness bugs across non-Patreon platforms had been silently
corrupting Posts since FC-3 shipped; the parser fix (sidecar.py, same
commit) addresses new imports. This migration cleans up existing rows.
Per-platform actions:
subscribestar — gallery-dl wrote the per-attachment id in `id` and
the actual post id in `post_id`. FC's parser picked `id`, so every
multi-image SubscribeStar post was fragmented into N Post rows.
1. For each SubscribeStar Post, read its sidecar (via the related
ImageRecord's on-disk path), pull `post_id`, overwrite
external_post_id and post_url.
2. Merge groups of Posts under one source that now share an
external_post_id (fragments of the same actual post). Same
ImageProvenance pre-delete + repoint dance as alembic 0022.
hentaifoundry — sidecars have NO `url` field; `src` is the image
URL. FC's parser stored post_url=NULL. Read each HF Post's sidecar
for `user` + `index`, derive the canonical /pictures/user/<u>/<i>
permalink. external_post_id (= `index`) was already correct.
discord — gallery-dl wrote the CDN attachment URL in `url`. FC's
parser stored that as post_url. Read each Discord Post's sidecar
for the server/channel/message triple, derive the proper
discord.com/channels/.../<message> permalink. external_post_id (=
`message_id`) was already correct.
pixiv — pure-SQL backfill: replace any `i.pximg.net`-style URL on
Post.post_url with the derived `/artworks/<id>` permalink. Pixiv
external_post_id (= `id`) was already correct; no sidecar IO
needed.
Idempotent: re-running on already-corrected data is a no-op (skips
rows whose derived value matches what's already stored).
Posts whose related ImageRecord paths don't resolve on disk (orphaned
filesystem state) are skipped with a count in the migration output —
those will be picked up by a future deep-scan.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0025"
down_revision: Union[str, None] = "0024"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# Mirror of sidecar._NUMBERING_PREFIX. Kept inline so the migration is
# self-contained (the operator's banked rule:
# reference_postgres_enum_swap_drop_checks.md says migrations shouldn't
# import from runtime app code).
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
def _find_sidecar(media_path: Path) -> Path | None:
"""gallery-dl writes the sidecar under the unprefixed stem
(`HOLLOW-ICHIGO.json`) while the media file gets a NN_ ordering
prefix (`01_HOLLOW-ICHIGO.png`). Try in order:
1. <stem>.json next to the media
2. <media>.json next to the media (full-name variant)
3. strip the NN_ prefix from the stem, then <stripped>.json
"""
if not media_path:
return None
cand = media_path.with_suffix(".json")
if cand.is_file():
return cand
cand = media_path.parent / f"{media_path.name}.json"
if cand.is_file():
return cand
m = _NUMBERING_PREFIX.match(media_path.stem)
if m:
cand = media_path.parent / f"{m.group(1)}.json"
if cand.is_file():
return cand
return None
def _str_id(v) -> str | None:
"""str() a JSON scalar id; reject bool (JSON booleans are ints in
Python's eyes but they aren't valid sidecar ids)."""
if isinstance(v, bool):
return None
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
def _str_field(v) -> str | None:
if isinstance(v, str) and v.strip():
return v.strip()
return None
def upgrade() -> None:
conn = op.get_bind()
# ── PART 1: Per-platform corrections requiring filesystem IO ─────
# SubscribeStar, HentaiFoundry, Discord all need fields from the
# sidecar to construct the right post_url. We walk each Post's
# related ImageRecord.path to find the sidecar, read it, derive,
# and update.
targets = conn.execute(text("""
SELECT p.id, p.external_post_id, p.post_url, s.platform
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform IN ('subscribestar', 'hentaifoundry', 'discord')
""")).fetchall()
stats: dict[str, dict[str, int]] = {
plat: {"read": 0, "updated": 0, "no_sidecar": 0}
for plat in ("subscribestar", "hentaifoundry", "discord")
}
for post_row in targets:
plat = post_row.platform
path = _first_attachment_path(conn, post_row.id)
if not path:
stats[plat]["no_sidecar"] += 1
continue
sidecar = _find_sidecar(Path(path))
if sidecar is None:
stats[plat]["no_sidecar"] += 1
continue
try:
data = json.loads(sidecar.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
stats[plat]["no_sidecar"] += 1
continue
stats[plat]["read"] += 1
new_epid = post_row.external_post_id
new_url = None
if plat == "subscribestar":
pid = _str_id(data.get("post_id"))
if pid:
new_epid = pid
new_url = f"https://www.subscribestar.com/posts/{pid}"
elif plat == "hentaifoundry":
user = _str_field(data.get("user")) or _str_field(data.get("artist"))
idx = _str_id(data.get("index"))
if user and idx:
new_url = f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
elif plat == "discord":
sid = _str_id(data.get("server_id"))
cid = _str_id(data.get("channel_id"))
mid = _str_id(data.get("message_id"))
if sid and cid and mid:
new_url = f"https://discord.com/channels/{sid}/{cid}/{mid}"
# Idempotent: skip if nothing changed.
if new_epid == post_row.external_post_id and new_url == post_row.post_url:
continue
conn.execute(
text("""
UPDATE post
SET external_post_id = :epid, post_url = :url
WHERE id = :id
"""),
{"epid": new_epid, "url": new_url, "id": post_row.id},
)
stats[plat]["updated"] += 1
for plat, s in stats.items():
print(
f"0025: {plat} — read {s['read']} sidecars, "
f"updated {s['updated']} Posts, "
f"{s['no_sidecar']} Posts had no resolvable sidecar"
)
# ── PART 2: Merge SubscribeStar fragments now sharing epid ───────
# After Part 1, each group of Posts under one source with the SAME
# new external_post_id is a fragment-set of the same actual post.
# Merge to one canonical row. Pre-handle the same ImageProvenance
# collision pattern as alembic 0022 (uq_image_provenance_image_post).
fragment_groups = conn.execute(text("""
SELECT p.source_id, p.external_post_id,
ARRAY_AGG(p.id ORDER BY p.id ASC) AS post_ids
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform = 'subscribestar'
AND p.external_post_id IS NOT NULL
GROUP BY p.source_id, p.external_post_id
HAVING COUNT(*) > 1
""")).fetchall()
merged = 0
for grp in fragment_groups:
post_ids = list(grp.post_ids)
keep_id, *drop_ids = post_ids
for drop_id in drop_ids:
# Pre-DELETE colliding ImageProvenance under drop_ that
# already exist under keep (alembic 0022 banked the pattern).
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE post_attachment SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
merged += 1
print(f"0025: subscribestar — merged {merged} duplicate Post fragments")
# ── PART 3: Pixiv post_url backfill (pure SQL) ───────────────────
# Pixiv's external_post_id is already correct (gallery-dl's `id` is
# the post id). Only post_url needs derivation: replace anything
# under i.pximg.net (the file URL) with the /artworks/<id> permalink.
pixiv_updated = conn.execute(text("""
UPDATE post p
SET post_url = 'https://www.pixiv.net/artworks/' || p.external_post_id
FROM source s
WHERE p.source_id = s.id
AND s.platform = 'pixiv'
AND p.external_post_id IS NOT NULL
AND (p.post_url IS NULL
OR p.post_url LIKE 'https://i.pximg.net/%'
OR p.post_url LIKE 'http://i.pximg.net/%')
""")).rowcount
print(f"0025: pixiv — backfilled post_url on {pixiv_updated} Posts")
def _first_attachment_path(conn, post_id: int) -> str | None:
"""Return any ImageRecord.path attached to this post (via
ImageProvenance). Lowest-id row keeps the migration deterministic
so re-running on the same DB picks the same sidecar."""
row = conn.execute(
text("""
SELECT ir.path
FROM image_provenance ip
JOIN image_record ir ON ir.id = ip.image_record_id
WHERE ip.post_id = :pid
ORDER BY ip.id ASC
LIMIT 1
"""),
{"pid": post_id},
).first()
return row[0] if row else None
def downgrade() -> None:
# Lossy: external_post_id values were overwritten with the correct
# post_id; original per-attachment ids weren't preserved. Post-merge
# also deleted drop rows. No safe restore. To roll back the schema
# invariant, fork from 0024 and re-run sidecar imports.
pass
@@ -0,0 +1,53 @@
"""import_task.recovery_count + refetched — poison-pill circuit breaker
Revision ID: 0026
Revises: 0025
Create Date: 2026-05-28
Backs the import-task resilience work (operator-flagged 2026-05-28):
- recovery_count: how many times recover_interrupted_tasks has
re-queued this row from a stuck 'processing' state. A row that
hard-crashes the worker (OOM / segfault on a corrupt or oversized
input) leaves no terminal flip, so the sweep re-queues it — and
without a cap it would loop forever, re-crashing the worker each
time. After MAX_RECOVERY_ATTEMPTS the sweep marks it 'failed' with a
diagnostic instead.
- refetched: whether a one-shot re-download has already been attempted
for this task's file. Bounds the Layer-2 re-fetch remediation to a
single attempt so source-side corruption doesn't loop.
Both default to 0 / false; additive, no backfill needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0026"
down_revision: Union[str, None] = "0025"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_task",
sa.Column(
"recovery_count", sa.Integer(), nullable=False,
server_default="0",
),
)
op.add_column(
"import_task",
sa.Column(
"refetched", sa.Boolean(), nullable=False,
server_default=sa.false(),
),
)
def downgrade() -> None:
op.drop_column("import_task", "refetched")
op.drop_column("import_task", "recovery_count")
+38 -1
View File
@@ -3,13 +3,23 @@
import logging
from pathlib import Path
from quart import Quart
from quart import Quart, request
from .api import all_blueprints
from .config import get_config
from .frontend import frontend_bp
from .services.credential_crypto import CredentialCrypto
# Browser-extension origins. The FabledCurator extension fetches from
# moz-extension://<uuid>/ on Firefox and chrome-extension://<uuid>/ on
# Chromium-based browsers. Operator-flagged 2026-05-26: extension's
# 'Test connection' returned `NetworkError` because the X-Extension-Key
# header on /api/credentials triggers a CORS preflight that our routes
# don't handle. Whitelisting only these two schemes (not opening CORS
# up generally) lets the extension talk to a plain-HTTP self-hosted FC
# without weakening the no-CORS posture for normal browser usage.
_EXTENSION_ORIGIN_SCHEMES = ("moz-extension://", "chrome-extension://")
_CREDENTIAL_KEY_PATH = Path("/images/secrets/credential_key.b64")
@@ -35,6 +45,33 @@ def create_app() -> Quart:
# Registered last so /api/* routes win over the SPA catch-all.
app.register_blueprint(frontend_bp)
@app.before_request
async def _extension_cors_preflight():
# Short-circuit OPTIONS preflight from the browser extension with a
# 204 + CORS headers (the after_request hook below adds them).
# Without this, OPTIONS lands on routes that only declared POST/GET
# methods and 405s before the after_request gets a chance.
if request.method != "OPTIONS":
return None
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
return "", 204
return None
@app.after_request
async def _extension_cors_headers(response):
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
response.headers["Access-Control-Allow-Origin"] = origin
response.headers["Access-Control-Allow-Methods"] = (
"GET, POST, PATCH, DELETE, OPTIONS"
)
response.headers["Access-Control-Allow-Headers"] = (
"Content-Type, X-Extension-Key"
)
response.headers["Access-Control-Max-Age"] = "86400"
return response
@app.after_serving
async def _dispose_db_engine() -> None:
from .extensions import dispose_engine
+10
View File
@@ -14,11 +14,13 @@ api_bp.add_url_rule("/health", view_func=health.get_health, methods=["GET"])
def all_blueprints() -> list[Blueprint]:
from .admin import admin_bp
from .aliases import aliases_bp
from .allowlist import allowlist_bp
from .artist import artist_bp
from .artists import artists_bp
from .attachments import attachments_bp
from .cleanup import cleanup_bp
from .credentials import credentials_bp
from .downloads import downloads_bp
from .extension import extension_bp
@@ -33,7 +35,10 @@ def all_blueprints() -> list[Blueprint]:
from .showcase import showcase_bp
from .sources import sources_bp
from .suggestions import suggestions_bp
from .system_activity import system_activity_bp
from .system_backup import system_backup_bp
from .tags import tags_bp
from .thumbnails import thumbnails_bp
return [
api_bp,
attachments_bp,
@@ -44,12 +49,17 @@ def all_blueprints() -> list[Blueprint]:
artists_bp,
showcase_bp,
settings_bp,
system_activity_bp,
system_backup_bp,
admin_bp,
cleanup_bp,
import_admin_bp,
migrate_bp,
suggestions_bp,
allowlist_bp,
aliases_bp,
ml_admin_bp,
thumbnails_bp,
sources_bp,
platforms_bp,
posts_bp,
+208
View File
@@ -0,0 +1,208 @@
"""FC-3k: /api/admin — destructive admin actions.
Five action surfaces:
POST /api/admin/artists/<slug>/cascade-delete (Tier C)
POST /api/admin/images/bulk-delete (Tier C)
DELETE /api/admin/tags/<int:tag_id> (Tier B)
POST /api/admin/tags/<int:dest_id>/merge (Tier B)
POST /api/admin/tags/prune-unused (Tier A)
GET /api/admin/tags/<int:tag_id>/usage-count (helper)
Tier-C ops take a dry_run body flag (returns projection inline,
no dispatch) and a confirm body field (server-recomputed token).
Long-running ops dispatch a maintenance-queue Celery task; the UI
tails FC-3i's /api/system/activity/runs to surface progress.
"""
from __future__ import annotations
import hashlib
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import Artist
from ..services.cleanup_service import project_artist_cascade, project_bulk_image_delete
admin_bp = Blueprint("admin", __name__, url_prefix="/api/admin")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _bulk_image_confirm_token(image_ids: list[int]) -> str:
"""Stable 8-hex token derived from the sorted id list. Mutates
when the selection changes; stays the same across modal opens of
the same selection so the operator can paste without confusion."""
canon = ",".join(str(i) for i in sorted(image_ids))
digest = hashlib.sha256(canon.encode("utf-8")).hexdigest()
return digest[:8]
@admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"])
async def artist_cascade_delete(slug: str):
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
artist = (await session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
return _bad("not_found", status=404)
artist_id = artist.id
projected = await session.run_sync(
lambda sync_sess: project_artist_cascade(sync_sess, slug=slug)
)
if dry_run:
return jsonify(projected)
expected = f"delete-artist-{artist_id}"
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import delete_artist_cascade_task
async_result = delete_artist_cascade_task.delay(artist_id=artist_id)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/images/bulk-delete", methods=["POST"])
async def images_bulk_delete():
body = await request.get_json(silent=True) or {}
image_ids = body.get("image_ids")
if not isinstance(image_ids, list) or not image_ids:
return _bad("invalid_image_ids", detail="image_ids must be non-empty list of int")
try:
image_ids = [int(i) for i in image_ids]
except (TypeError, ValueError):
return _bad("invalid_image_ids", detail="image_ids must contain only ints")
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
projected = await session.run_sync(
lambda sync_sess: project_bulk_image_delete(
sync_sess, image_ids=image_ids,
)
)
sha8 = _bulk_image_confirm_token(image_ids)
expected = f"delete-images-{sha8}"
if dry_run:
# Hand the canonical Tier-C confirm token back with the
# projection so the frontend doesn't have to recompute SHA-256
# client-side via crypto.subtle (Secure-Context-gated,
# undefined on plain-HTTP origins per the homelab posture).
# Operator-flagged 2026-05-27.
projected["confirm_token"] = expected
return jsonify(projected)
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import bulk_delete_images_task
async_result = bulk_delete_images_task.delay(image_ids=image_ids)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/tags/<int:tag_id>", methods=["DELETE"])
async def tag_delete(tag_id: int):
"""Tier-B sync delete. UI yes/no modal is the only confirmation."""
from ..services.cleanup_service import delete_tag
async with get_session() as session:
try:
result = await session.run_sync(
lambda sync_sess: delete_tag(sync_sess, tag_id=tag_id)
)
except LookupError:
return _bad("not_found", status=404)
return jsonify(result)
@admin_bp.route("/tags/<int:dest_id>/merge", methods=["POST"])
async def tag_merge(dest_id: int):
"""Wraps TagService.merge. Source repoints to dest, dest survives,
source row deleted, protective alias auto-created if source was
ML-applied or allowlisted."""
from ..services.tag_service import TagMergeConflict, TagService, TagValidationError
body = await request.get_json(silent=True) or {}
source_id = body.get("source_id")
if not isinstance(source_id, int) or source_id == dest_id:
return _bad("invalid_source_id", detail="source_id must be int and differ from dest")
async with get_session() as session:
try:
result = await TagService(session).merge(
source_id=source_id, target_id=dest_id,
)
except TagMergeConflict as exc:
return _bad("merge_conflict", status=409, detail=str(exc))
except TagValidationError as exc:
return _bad("tag_kind_mismatch", detail=str(exc))
except LookupError:
return _bad("not_found", status=404)
# MergeResult is a frozen dataclass — flatten to dict.
return jsonify({
"result": {
"target_id": result.target_id,
"target_name": result.target_name,
"target_kind": result.target_kind,
"merged_count": result.merged_count,
"alias_created": result.alias_created,
"source_deleted": result.source_deleted,
},
})
@admin_bp.route("/tags/<int:tag_id>/usage-count", methods=["GET"])
async def tag_usage_count(tag_id: int):
"""Helper for the Tier-B yes/no prompt; surfaces "N associations"
in the dialog so the operator knows what they're nuking."""
from ..services.cleanup_service import count_tag_associations
async with get_session() as session:
count = await session.run_sync(
lambda sync_sess: count_tag_associations(
sync_sess, tag_id=tag_id,
)
)
return jsonify({"count": count})
@admin_bp.route("/tags/prune-unused", methods=["POST"])
async def tags_prune_unused():
"""Tier-A: dry-run preview list IS the prompt. UI calls with
dry_run=true first, shows the list, operator clicks button to
re-call with dry_run=false."""
from ..services.cleanup_service import prune_unused_tags
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: prune_unused_tags(
sync_sess, dry_run=dry_run,
)
)
return jsonify(result)
+200
View File
@@ -0,0 +1,200 @@
"""FC-Cleanup: /api/cleanup/* — retroactive enforcement of import filters.
Endpoints:
POST /min-dimension/preview synchronous SQL audit
POST /min-dimension/delete synchronous SQL delete (Tier-C token)
POST /audit async transparency / single_color start
GET /audit list recent audit_run rows
GET /audit/<id> single audit_run row
POST /audit/<id>/apply apply matched_ids deletes (Tier-C token)
POST /audit/<id>/cancel flip running audit to cancelled
Unused-tags retroactive prune intentionally NOT in this namespace —
TagMaintenanceCard (Maintenance tab → moved to Cleanup tab in v26.05.25.7)
uses the existing /api/admin/tags/prune-unused endpoint via the admin
store. No duplicate route here.
Confirm-token format matches modal/DestructiveConfirmModal.vue convention:
`delete-min-dim-<sha8(w,h)>` for min-dim delete
`delete-audit-<id>` for audit apply
(Modal hardcodes action ∈ {'restore', 'delete'}; "apply audit" is semantically a delete of the matched images, so we use `delete-audit-<id>`.)
"""
from __future__ import annotations
import hashlib
from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import LibraryAuditRun
from ..services import cleanup_service
cleanup_bp = Blueprint("cleanup", __name__, url_prefix="/api/cleanup")
IMAGES_ROOT = Path("/images")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _min_dim_token(min_w: int, min_h: int) -> str:
# SHA-256 (not MD5) — Web Crypto's subtle.digest rejects MD5; both
# sides use SHA-256 truncated to 8 hex chars.
canon = f"{min_w}x{min_h}"
return f"delete-min-dim-{hashlib.sha256(canon.encode()).hexdigest()[:8]}"
def _serialize_audit_run(audit: LibraryAuditRun) -> dict:
return {
"id": audit.id,
"rule": audit.rule,
"params": audit.params,
"status": audit.status,
"started_at": audit.started_at.isoformat() if audit.started_at else None,
"finished_at": audit.finished_at.isoformat() if audit.finished_at else None,
"scanned_count": audit.scanned_count,
"matched_count": audit.matched_count,
"matched_ids": audit.matched_ids,
"error": audit.error,
}
@cleanup_bp.route("/min-dimension/preview", methods=["POST"])
async def min_dim_preview():
body = await request.get_json(silent=True) or {}
try:
min_w = int(body.get("min_width", 0))
min_h = int(body.get("min_height", 0))
except (TypeError, ValueError):
return _bad("invalid_dimensions")
if min_w < 0 or min_h < 0:
return _bad("invalid_dimensions")
async with get_session() as session:
projection = await session.run_sync(
lambda s: cleanup_service.project_min_dimension_violations(
s, min_width=min_w, min_height=min_h,
)
)
# Hand the canonical Tier-C delete token back with the preview so
# the frontend doesn't have to recompute SHA-256 client-side.
# window.crypto.subtle is Secure-Context-gated and undefined on
# plain-HTTP origins (homelab posture); without this the Delete
# button silently swallowed the TypeError and never opened the
# confirm modal. Operator-flagged 2026-05-27.
projection["confirm_token"] = _min_dim_token(min_w, min_h)
return jsonify(projection)
@cleanup_bp.route("/min-dimension/delete", methods=["POST"])
async def min_dim_delete():
body = await request.get_json(silent=True) or {}
try:
min_w = int(body.get("min_width", 0))
min_h = int(body.get("min_height", 0))
except (TypeError, ValueError):
return _bad("invalid_dimensions")
if min_w < 0 or min_h < 0:
return _bad("invalid_dimensions")
supplied = body.get("confirm", "")
expected = _min_dim_token(min_w, min_h)
if supplied != expected:
return _bad("confirm_mismatch", expected=expected)
async with get_session() as session:
deleted = await session.run_sync(
lambda s: cleanup_service.delete_min_dimension_violations(
s, min_width=min_w, min_height=min_h, images_root=IMAGES_ROOT,
)
)
await session.commit()
return jsonify({"deleted": deleted})
@cleanup_bp.route("/audit", methods=["POST"])
async def audit_create():
body = await request.get_json(silent=True) or {}
rule = body.get("rule")
params = body.get("params") or {}
if rule not in ("transparency", "single_color"):
return _bad("invalid_rule")
if not isinstance(params, dict):
return _bad("invalid_params")
async with get_session() as session:
try:
audit_id = await session.run_sync(
lambda s: cleanup_service.start_audit_run(
s, rule=rule, params=params,
)
)
except cleanup_service.AuditAlreadyRunning as running_id:
return _bad(
"audit_already_running", status=409,
running_id=int(str(running_id)),
)
except ValueError as exc:
return _bad(str(exc))
await session.commit()
return jsonify({"audit_id": audit_id, "status": "running"}), 202
@cleanup_bp.route("/audit/<int:audit_id>", methods=["GET"])
async def audit_get(audit_id: int):
async with get_session() as session:
audit = (await session.execute(
select(LibraryAuditRun).where(LibraryAuditRun.id == audit_id)
)).scalar_one_or_none()
if audit is None:
return _bad("not_found", status=404)
return jsonify(_serialize_audit_run(audit))
@cleanup_bp.route("/audit", methods=["GET"])
async def audit_history():
try:
limit = min(int(request.args.get("limit", "20")), 100)
except ValueError:
return _bad("invalid_limit")
async with get_session() as session:
rows = (await session.execute(
select(LibraryAuditRun)
.order_by(LibraryAuditRun.id.desc())
.limit(limit)
)).scalars().all()
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
@cleanup_bp.route("/audit/<int:audit_id>/apply", methods=["POST"])
async def audit_apply(audit_id: int):
body = await request.get_json(silent=True) or {}
confirm = body.get("confirm", "")
async with get_session() as session:
try:
deleted = await session.run_sync(
lambda s: cleanup_service.apply_audit_run(
s, audit_id=audit_id, confirm_token=confirm,
images_root=IMAGES_ROOT,
)
)
except cleanup_service.AuditNotReady as exc:
return _bad("audit_not_ready", current_status=str(exc))
except cleanup_service.ConfirmTokenMismatch as exc:
return _bad("confirm_mismatch", expected=str(exc))
except ValueError as exc:
return _bad("not_found", status=404, detail=str(exc))
await session.commit()
return jsonify({"deleted": deleted})
@cleanup_bp.route("/audit/<int:audit_id>/cancel", methods=["POST"])
async def audit_cancel(audit_id: int):
async with get_session() as session:
await session.run_sync(
lambda s: cleanup_service.cancel_audit_run(s, audit_id=audit_id)
)
await session.commit()
return jsonify({"cancelled": True})
+32 -1
View File
@@ -5,8 +5,10 @@ status/source/artist. Returns slim records.
Detail view: full DownloadEvent including the metadata JSONB.
"""
from datetime import UTC, datetime, timedelta
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from sqlalchemy import func, select
from ..extensions import get_session
from ..models import Artist, DownloadEvent, Source
@@ -95,6 +97,35 @@ async def list_downloads():
return jsonify([_list_record(e, s, a) for e, s, a in rows])
@downloads_bp.route("/stats", methods=["GET"])
async def downloads_stats():
"""Status-grouped count over download_event for the dashboard stat chips.
`?window_hours=` (default 24) bounds by `started_at`. The full set of
statuses is always present in the response (zero for missing) so the
UI doesn't have to fill in defaults.
"""
try:
window_hours = int(request.args.get("window_hours", "24"))
except ValueError:
return jsonify({"error": "invalid_window_hours"}), 400
if window_hours < 1 or window_hours > 24 * 365:
return jsonify({"error": "invalid_window_hours"}), 400
since = datetime.now(UTC) - timedelta(hours=window_hours)
out = {"pending": 0, "running": 0, "ok": 0, "error": 0, "skipped": 0}
async with get_session() as session:
stmt = (
select(DownloadEvent.status, func.count())
.where(DownloadEvent.started_at >= since)
.group_by(DownloadEvent.status)
)
for status, n in (await session.execute(stmt)).all():
if status in out:
out[status] = int(n)
return jsonify(out)
@downloads_bp.route("/<int:event_id>", methods=["GET"])
async def get_download(event_id: int):
async with get_session() as session:
+13 -3
View File
@@ -93,10 +93,20 @@ def _read_manifest_sync() -> dict | None:
asyncio.to_thread (ASYNC240: no pathlib I/O in async functions)."""
if not XPI_DIR.is_dir():
return None
xpis = sorted(XPI_DIR.glob("fabledcurator-*.xpi"), key=lambda p: p.stat().st_mtime)
if not xpis:
# Exclude the `fabledcurator-latest.xpi` alias when picking the file to
# extract a version from — it's a copy of the latest versioned XPI,
# written at the same mtime by build.yml, and would otherwise tie or
# win the sort (operator-flagged 2026-05-26: UI displayed "v latest"
# because `_extract_version("fabledcurator-latest.xpi")` returns
# the literal "latest"). The alias still serves as `latest_url`.
versioned = [
p for p in XPI_DIR.glob("fabledcurator-*.xpi")
if p.name != "fabledcurator-latest.xpi"
]
if not versioned:
return None
latest = xpis[-1]
versioned.sort(key=lambda p: p.stat().st_mtime)
latest = versioned[-1]
return {
"installed": True,
"version": _extract_version(latest.name),
+2
View File
@@ -42,6 +42,8 @@ async def scroll():
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+132 -12
View File
@@ -47,10 +47,13 @@ async def status():
if active:
payload["active_batch"] = {
"id": active.id,
"source_path": active.source_path,
"scan_mode": active.scan_mode,
"total_files": active.total_files,
"imported": active.imported,
"skipped": active.skipped,
"failed": active.failed,
"refreshed": active.refreshed,
"started_at": active.started_at.isoformat(),
}
return jsonify(payload)
@@ -100,24 +103,141 @@ async def list_tasks():
@import_admin_bp.route("/retry-failed", methods=["POST"])
async def retry_failed():
# Fold SELECT into UPDATE…WHERE…RETURNING — the prior SELECT-then-
# UPDATE-WHERE-id-IN pattern blew past psycopg's 65535-parameter
# ceiling once failed_ids exceeded ~65k rows.
async with get_session() as session:
failed_ids = (
await session.execute(select(ImportTask.id).where(ImportTask.status == "failed"))
).scalars().all()
if not failed_ids:
return jsonify({"retried": 0})
await session.execute(
result = await session.execute(
update(ImportTask)
.where(ImportTask.id.in_(failed_ids))
.values(status="queued", error=None, started_at=None, finished_at=None)
.where(ImportTask.status == "failed")
.values(
status="queued", error=None,
started_at=None, finished_at=None,
)
.returning(ImportTask.id, ImportTask.task_type)
)
failed = result.all()
if not failed:
return jsonify({"retried": 0})
await session.commit()
from ..tasks.import_file import import_media_file
for tid in failed_ids:
import_media_file.delay(tid)
from ..tasks.import_file import enqueue_import
for tid, task_type in failed:
enqueue_import(tid, task_type)
return jsonify({"retried": len(failed_ids)})
return jsonify({"retried": len(failed)})
@import_admin_bp.route("/tasks/<int:task_id>/refetch", methods=["POST"])
async def refetch_task(task_id: int):
"""Layer-2 one-shot re-download: delete the (corrupt) file behind a
failed import task and re-run its source's downloader to fetch a
fresh copy. Only works for files that resolve to an enabled,
real-URL subscription Source; filesystem-only imports return
no_source.
Returns one of: refetch_queued (+source_id) / no_source /
already_refetched / not_found / not_failed.
"""
async with get_session() as session:
result = await session.run_sync(_refetch_task_sync, task_id)
if result["status"] == "not_found":
return jsonify(result), 404
if result["status"] == "not_failed":
return jsonify(result), 400
return jsonify(result)
def _refetch_task_sync(session, task_id: int) -> dict:
from pathlib import Path
from ..models import ImportSettings
from ..services.refetch_service import attempt_refetch
task = session.get(ImportTask, task_id)
if task is None:
return {"status": "not_found"}
if task.status != "failed":
return {"status": "not_failed"}
settings = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
return attempt_refetch(session, task, Path(settings.import_scan_path))
@import_admin_bp.route("/clear-stuck", methods=["POST"])
async def clear_stuck():
"""Force any non-terminal ImportTask (status in pending/queued/
processing) to 'failed' AND finalize any ImportBatch that ends up
with no active children. Escape hatch for the operator when the
automatic recover_interrupted_tasks sweep keeps re-queueing the
same stuck row forever (e.g., underlying file is genuinely broken
and the import keeps OSError-looping at PIL load).
Idempotent + non-destructive: rows survive as 'failed' so the
Retry-Failed button can re-attempt them once whatever was broken
is fixed. Banked 2026-05-25 — operator hit 3 large PNGs that
autoretry-looped for 2 days after a corrupt-data PIL OSError.
"""
async with get_session() as session:
# Fold SELECT into UPDATE…WHERE — see /retry-failed for the
# 65535-parameter ceiling rationale. rowcount is enough here
# because we don't need the ids afterward (no .delay()).
clear_result = await session.execute(
update(ImportTask)
.where(
ImportTask.status.in_(["pending", "queued", "processing"])
)
.values(
status="failed",
finished_at=datetime.now(UTC),
error=(
"manually cleared via /api/import/clear-stuck "
"— stuck in non-terminal state; retry once "
"underlying cause (corrupt file, missing model, "
"etc.) is resolved"
),
)
)
tasks_failed = clear_result.rowcount or 0
# Finalize any 'running' ImportBatch that no longer has any
# active children. The "Scanning..." banner is driven by
# /api/import/status finding a running batch; left untouched,
# it would persist forever after the stuck-task clear.
running_batches = (
await session.execute(
select(ImportBatch.id).where(ImportBatch.status == "running")
)
).scalars().all()
finalized_batches = 0
for batch_id in running_batches:
still_active = (
await session.execute(
select(ImportTask.id)
.where(ImportTask.batch_id == batch_id)
.where(ImportTask.status.in_(
["pending", "queued", "processing"]
))
.limit(1)
)
).scalar_one_or_none()
if still_active is None:
await session.execute(
update(ImportBatch)
.where(ImportBatch.id == batch_id)
.values(
status="complete",
finished_at=datetime.now(UTC),
)
)
finalized_batches += 1
await session.commit()
return jsonify({
"tasks_failed": tasks_failed,
"batches_finalized": finalized_batches,
})
@import_admin_bp.route("/clear-completed", methods=["POST"])
+5 -34
View File
@@ -1,14 +1,11 @@
"""FC-5: /api/migrate — trigger and poll migration runs.
Ingest kinds (gs_ingest, ir_ingest) accept multipart/form-data with an
`export_file` field. All other kinds accept JSON. Apply-without-backup
guard rejects non-dry-run ingests unless a pre_migration-tagged backup
exists in the last 24h (override with body.force=true).
`export_file` field. All other kinds accept JSON. Backup + rollback
were retired in FC-3h (2026-05-24); use /api/system/backup/* instead.
"""
import json
from datetime import UTC, datetime, timedelta
from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
@@ -19,12 +16,12 @@ from ..tasks.migration import run_migration
migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate")
# 'backup' + 'rollback' retired 2026-05-24 (FC-3h); see /api/system/backup/*.
_VALID_KINDS = frozenset({
"backup", "gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "rollback", "cleanup",
"gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "cleanup",
})
_INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"})
_APPLY_KINDS = frozenset({"gs_ingest", "ir_ingest", "tag_apply", "rollback", "cleanup"})
def _bad(error: str, *, status: int = 400, **extra):
@@ -33,19 +30,6 @@ def _bad(error: str, *, status: int = 400, **extra):
return jsonify(body), status
def _has_recent_pre_migration_backup() -> bool:
from ..services.migrators import backup as backup_mod
images_root = Path("/images")
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
return False
created_at_str = manifest.get("created_at")
if not created_at_str:
return False
created_at = datetime.fromisoformat(created_at_str)
return (datetime.now(UTC) - created_at) < timedelta(hours=24)
def _run_to_dict(run: MigrationRun) -> dict:
return {
"id": run.id,
@@ -78,7 +62,6 @@ async def create_run(kind: str):
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
return _bad("invalid_export_file", detail=str(exc))
dry_run = str(form.get("dry_run", "false")).lower() in ("true", "1", "yes")
force = str(form.get("force", "false")).lower() in ("true", "1", "yes")
params: dict = {"data": data, "dry_run": dry_run}
else:
body = await request.get_json()
@@ -87,20 +70,8 @@ async def create_run(kind: str):
if not isinstance(body, dict):
return _bad("invalid_body")
dry_run = bool(body.get("dry_run", False))
force = bool(body.get("force", False))
params = dict(body)
is_apply = (kind in _APPLY_KINDS) and not dry_run
if is_apply and not force and not _has_recent_pre_migration_backup():
return _bad(
"no_backup",
detail="apply action requires a pre_migration-tagged backup "
"in the last 24h (or force=true).",
)
if kind == "backup":
params.setdefault("tag", "pre_migration")
async with get_session() as session:
run = MigrationRun(kind=kind, status="pending", dry_run=dry_run)
session.add(run)
+209
View File
@@ -0,0 +1,209 @@
"""FC-3i: system activity dashboard endpoints.
Read-only. Combines Redis-broker queue depths (LLEN per queue),
Celery worker introspection (celery inspect), and the task_run DB
history into the surfaces the SystemActivityTab UI consumes.
All filesystem/sync-client work goes through asyncio.to_thread per
ASYNC230/240 (mirrors backend.app.api.extension's pattern).
"""
from __future__ import annotations
import asyncio
import time
from datetime import UTC, datetime, timedelta
from quart import Blueprint, jsonify, request
from sqlalchemy import desc, func, select
from ..config import get_config
from ..extensions import get_session
from ..models import TaskRun
system_activity_bp = Blueprint(
"system_activity", __name__, url_prefix="/api/system/activity",
)
# Canonical queue order — must match celery_app.task_routes. UI renders
# in this order; queues with no LLEN response show as null rather than
# absent.
_QUEUE_NAMES = (
"default", "import", "thumbnail", "ml",
"download", "scan", "maintenance",
)
# Cache module-level so all requests share the cache between polls.
# Tests can reset via direct dict mutation if needed.
_QUEUE_CACHE: dict = {"ts": 0.0, "data": None}
_WORKER_CACHE: dict = {"ts": 0.0, "data": None}
_QUEUE_CACHE_TTL = 2.0
_WORKER_CACHE_TTL = 5.0
def _read_queues_sync() -> dict:
"""Reads each queue's LLEN from the broker. Sync — caller wraps in
asyncio.to_thread. Per-queue try/except returns None on failure so
one bad queue doesn't break the whole response."""
import redis # local import; only this endpoint needs it
cfg = get_config()
client = redis.Redis.from_url(cfg.celery_broker_url)
out: dict = {}
for name in _QUEUE_NAMES:
try:
out[name] = int(client.llen(name))
except Exception: # noqa: BLE001 — broker hiccup shouldn't break UI
out[name] = None
return {
"queues": out,
"fetched_at": datetime.now(UTC).isoformat(),
}
def _read_workers_sync() -> dict:
"""celery inspect active_queues + active. Returns per-worker info."""
from ..celery_app import celery as celery_app
insp = celery_app.control.inspect(timeout=2.0)
active_queues = insp.active_queues() or {}
active_tasks = insp.active() or {}
workers: dict = {}
for hostname, queues in active_queues.items():
workers[hostname] = {
"queues": sorted({q["name"] for q in queues}),
"active_count": len(active_tasks.get(hostname, [])),
}
return {
"workers": workers,
"fetched_at": datetime.now(UTC).isoformat(),
}
@system_activity_bp.route("/queues", methods=["GET"])
async def get_queues():
"""Per-queue Redis LLEN. Cached 2s.
Response: {queues: {name: depth_or_null}, fetched_at: iso8601}
"""
now = time.time()
if _QUEUE_CACHE["data"] is None or (now - _QUEUE_CACHE["ts"]) > _QUEUE_CACHE_TTL:
_QUEUE_CACHE["data"] = await asyncio.to_thread(_read_queues_sync)
_QUEUE_CACHE["ts"] = now
return jsonify(_QUEUE_CACHE["data"])
@system_activity_bp.route("/workers", methods=["GET"])
async def get_workers():
"""Live celery inspect. Cached 5s.
Response: {workers: {hostname: {queues, active_count}}, fetched_at}
"""
now = time.time()
if _WORKER_CACHE["data"] is None or (now - _WORKER_CACHE["ts"]) > _WORKER_CACHE_TTL:
_WORKER_CACHE["data"] = await asyncio.to_thread(_read_workers_sync)
_WORKER_CACHE["ts"] = now
return jsonify(_WORKER_CACHE["data"])
@system_activity_bp.route("/runs", methods=["GET"])
async def list_runs():
"""Paginated task_run history. Query params:
queue=<name> filter to one queue
status=<status> filter to one status (running/ok/error/timeout/retry)
limit=<int> default 50, max 200
before_id=<int> cursor for keyset pagination
Response: {runs: [...], next_cursor: id|null}
"""
try:
limit = min(int(request.args.get("limit", "50")), 200)
except ValueError:
return jsonify({"error": "invalid_limit"}), 400
if limit < 1:
return jsonify({"error": "invalid_limit"}), 400
queue = request.args.get("queue")
status = request.args.get("status")
before_id_raw = request.args.get("before_id")
before_id = int(before_id_raw) if before_id_raw else None
async with get_session() as session:
stmt = select(TaskRun).order_by(desc(TaskRun.id))
if queue:
stmt = stmt.where(TaskRun.queue == queue)
if status:
stmt = stmt.where(TaskRun.status == status)
if before_id is not None:
stmt = stmt.where(TaskRun.id < before_id)
stmt = stmt.limit(limit + 1)
rows = (await session.execute(stmt)).scalars().all()
has_more = len(rows) > limit
rows = rows[:limit]
return jsonify({
"runs": [_row_to_dict(r) for r in rows],
"next_cursor": rows[-1].id if has_more and rows else None,
})
@system_activity_bp.route("/failures", methods=["GET"])
async def list_failures():
"""Recent failures across all lanes (24h window).
Response: {recent: [...], count_by_type: {ErrorClass: n}, since}
"""
try:
limit = min(int(request.args.get("limit", "50")), 200)
except ValueError:
return jsonify({"error": "invalid_limit"}), 400
since = datetime.now(UTC) - timedelta(hours=24)
async with get_session() as session:
recent_stmt = (
select(TaskRun)
.where(TaskRun.status.in_(["error", "timeout"]))
.where(TaskRun.finished_at >= since)
.order_by(desc(TaskRun.finished_at))
.limit(limit)
)
recent = (await session.execute(recent_stmt)).scalars().all()
count_stmt = (
select(TaskRun.error_type, func.count(TaskRun.id))
.where(TaskRun.status.in_(["error", "timeout"]))
.where(TaskRun.finished_at >= since)
.group_by(TaskRun.error_type)
.order_by(desc(func.count(TaskRun.id)))
)
counts = (await session.execute(count_stmt)).all()
return jsonify({
"recent": [_row_to_dict(r) for r in recent],
"count_by_type": {
(row[0] or "Unknown"): row[1]
for row in counts
},
"since": since.isoformat(),
})
def _row_to_dict(r: TaskRun) -> dict:
return {
"id": r.id,
"queue": r.queue,
"task_name": r.task_name,
"target_id": r.target_id,
"celery_task_id": r.celery_task_id,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_ms": r.duration_ms,
"status": r.status,
"error_type": r.error_type,
"error_message": r.error_message,
"retry_count": r.retry_count,
"worker_hostname": r.worker_hostname,
"args_summary": r.args_summary,
}
+264
View File
@@ -0,0 +1,264 @@
"""FC-3h: /api/system/backup — create/list/restore/delete/tag for
DB + image backups.
Read endpoints are public on FC (operator-facing internal API; same
posture as /api/system/activity). Write endpoints take a typed
`confirm` body field that must match a server-generated token for
that backup row, to prevent click-to-destroy by stale browser tabs
or accidental cURL.
"""
from __future__ import annotations
from quart import Blueprint, jsonify, request
from sqlalchemy import desc, select
from ..extensions import get_session
from ..models import BackupRun, ImportSettings
system_backup_bp = Blueprint(
"system_backup", __name__, url_prefix="/api/system/backup",
)
_KINDS = frozenset({"db", "images"})
_TAG_MAX_LEN = 64
_BACKUP_SETTINGS_FIELDS = (
"backup_db_nightly_enabled",
"backup_db_nightly_hour_utc",
"backup_db_keep_last_n",
"backup_images_keep_last_n",
)
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _row_to_dict(r: BackupRun) -> dict:
return {
"id": r.id,
"kind": r.kind,
"status": r.status,
"tag": r.tag,
"triggered_by": r.triggered_by,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": (
int((r.finished_at - r.started_at).total_seconds())
if r.finished_at and r.started_at else None
),
"sql_path": r.sql_path,
"tar_path": r.tar_path,
"size_bytes": r.size_bytes,
"error": r.error,
"restored_from_id": r.restored_from_id,
"manifest": r.manifest or {},
}
def _validate_tag(tag):
if tag is None:
return None
if not isinstance(tag, str):
return _bad("invalid_tag", detail="tag must be string or null")
tag = tag.strip()
if not tag:
return None
if len(tag) > _TAG_MAX_LEN:
return _bad("invalid_tag", detail=f"tag too long (max {_TAG_MAX_LEN})")
return tag
def _validate_backup_settings_patch(body: dict):
if "backup_db_nightly_enabled" in body and not isinstance(
body["backup_db_nightly_enabled"], bool,
):
return _bad("invalid_value", detail="backup_db_nightly_enabled must be bool")
if "backup_db_nightly_hour_utc" in body:
v = body["backup_db_nightly_hour_utc"]
if not isinstance(v, int) or isinstance(v, bool) or not (0 <= v <= 23):
return _bad("invalid_value", detail="backup_db_nightly_hour_utc must be 0..23")
if "backup_db_keep_last_n" in body:
v = body["backup_db_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 365):
return _bad("invalid_value", detail="backup_db_keep_last_n must be 1..365")
if "backup_images_keep_last_n" in body:
v = body["backup_images_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 100):
return _bad("invalid_value", detail="backup_images_keep_last_n must be 1..100")
return None
@system_backup_bp.route("/db", methods=["POST"])
async def trigger_db_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_db_task
backup_db_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/images", methods=["POST"])
async def trigger_images_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_images_task
backup_images_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/runs", methods=["GET"])
async def list_runs():
try:
limit = min(int(request.args.get("limit", "50")), 200)
except ValueError:
return _bad("invalid_limit")
if limit < 1:
return _bad("invalid_limit")
kind = request.args.get("kind")
if kind is not None and kind not in _KINDS:
return _bad("invalid_kind", detail=f"kind must be one of {sorted(_KINDS)}")
before_id_raw = request.args.get("before_id")
before_id = int(before_id_raw) if before_id_raw else None
async with get_session() as session:
stmt = select(BackupRun).order_by(desc(BackupRun.id))
if kind:
stmt = stmt.where(BackupRun.kind == kind)
if before_id is not None:
stmt = stmt.where(BackupRun.id < before_id)
stmt = stmt.limit(limit + 1)
rows = (await session.execute(stmt)).scalars().all()
has_more = len(rows) > limit
rows = rows[:limit]
return jsonify({
"runs": [_row_to_dict(r) for r in rows],
"next_cursor": rows[-1].id if has_more and rows else None,
})
@system_backup_bp.route("/runs/<int:run_id>", methods=["GET"])
async def get_run(run_id: int):
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>", methods=["PATCH"])
async def patch_run(run_id: int):
body = await request.get_json(silent=True) or {}
if "tag" not in body:
return _bad("invalid_body", detail="tag required")
tag = _validate_tag(body["tag"])
if isinstance(tag, tuple):
return tag
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
row.tag = tag
await session.commit()
await session.refresh(row)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>/restore", methods=["POST"])
async def trigger_restore(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
if row.status != "ok":
return _bad(
"not_restorable",
detail=f"source backup status={row.status!r}; only 'ok' rows are restorable",
)
expected = f"restore-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
kind = row.kind
if kind == "db":
from ..tasks.backup import restore_db_task
restore_db_task.delay(source_backup_run_id=run_id)
else: # 'images' (the only other value _KINDS allows via the trigger path)
from ..tasks.backup import restore_images_task
restore_images_task.delay(source_backup_run_id=run_id)
return jsonify({"status": "dispatched", "kind": kind}), 202
@system_backup_bp.route("/runs/<int:run_id>", methods=["DELETE"])
async def delete_run(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
expected = f"delete-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..services import backup_service
backup_service.unlink_artifact_files(
sql_path=row.sql_path, tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
await session.delete(row)
await session.commit()
return "", 204
@system_backup_bp.route("/settings", methods=["GET"])
async def get_settings():
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
return jsonify({
"backup_db_nightly_enabled": row.backup_db_nightly_enabled,
"backup_db_nightly_hour_utc": row.backup_db_nightly_hour_utc,
"backup_db_keep_last_n": row.backup_db_keep_last_n,
"backup_images_keep_last_n": row.backup_images_keep_last_n,
})
@system_backup_bp.route("/settings", methods=["PATCH"])
async def patch_settings():
body = await request.get_json(silent=True)
if not isinstance(body, dict):
return _bad("invalid_body", detail="body must be a JSON object")
err = _validate_backup_settings_patch(body)
if err is not None:
return err
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
for field in _BACKUP_SETTINGS_FIELDS:
if field in body:
setattr(row, field, body[field])
await session.commit()
return await get_settings()
+32 -5
View File
@@ -15,6 +15,7 @@ from ..services.tag_service import (
TagService,
TagValidationError,
)
from ..utils.tag_prefix import parse_kind_prefix
tags_bp = Blueprint("tags", __name__, url_prefix="/api")
@@ -105,13 +106,39 @@ async def directory():
@tags_bp.route("/tags", methods=["POST"])
async def create_tag():
"""Create a tag. Two input shapes accepted:
1. Explicit: {name, kind, fandom_id?} — caller already split, kind wins.
2. IR-suffix: {name} where name = "kind:Name" (e.g. "artist:Eric").
The server runs parse_kind_prefix(name) to derive kind; the colon
and prefix are stripped from the stored tag name. If no recognized
prefix is present, the kind defaults to `general`.
Explicit kind ALWAYS wins (backward-compat for existing callers).
"""
body = await request.get_json()
if not body or "name" not in body or "kind" not in body:
return jsonify({"error": "name and kind required"}), 400
if not body or "name" not in body:
return jsonify({"error": "name required"}), 400
name = body["name"]
kind = _coerce_kind(body["kind"])
if kind is None:
return jsonify({"error": f"invalid kind {body['kind']!r}"}), 400
explicit_kind_raw = body.get("kind")
if explicit_kind_raw is not None:
# Caller provided kind — honor it; don't re-parse.
kind = _coerce_kind(explicit_kind_raw)
if kind is None:
return jsonify({"error": f"invalid kind {explicit_kind_raw!r}"}), 400
else:
# IR-style: parse "kind:Name" from the raw name.
parsed_kind, parsed_name = parse_kind_prefix(name)
if parsed_kind is not None:
name = parsed_name
kind = _coerce_kind(parsed_kind)
# parse_kind_prefix only returns kinds from KNOWN_KINDS which
# are all valid TagKind members, so _coerce_kind can't return
# None here — but defensive.
if kind is None:
return jsonify({"error": f"invalid kind {parsed_kind!r}"}), 400
else:
kind = TagKind.general
fandom_id = body.get("fandom_id")
async with get_session() as session:
+13
View File
@@ -0,0 +1,13 @@
"""Thumbnail admin API: backfill trigger."""
from quart import Blueprint, jsonify
thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@thumbnails_bp.route("/backfill", methods=["POST"])
async def trigger_backfill():
from ..tasks.thumbnail import backfill_thumbnails
r = backfill_thumbnails.delay()
return jsonify({"celery_task_id": r.id}), 202
+25
View File
@@ -31,6 +31,9 @@ def make_celery() -> Celery:
"backend.app.tasks.migration",
"backend.app.tasks.ml",
"backend.app.tasks.download",
"backend.app.tasks.backup",
"backend.app.tasks.admin",
"backend.app.tasks.library_audit",
],
)
app.conf.update(
@@ -43,6 +46,9 @@ def make_celery() -> Celery:
"backend.app.tasks.scan.*": {"queue": "scan"},
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
"backend.app.tasks.migration.*": {"queue": "maintenance"},
"backend.app.tasks.backup.*": {"queue": "maintenance"},
"backend.app.tasks.admin.*": {"queue": "maintenance"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance"},
},
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True,
@@ -81,9 +87,28 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.cleanup_old_download_events",
"schedule": 86400.0, # daily
},
"recover-stalled-task-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_task_runs",
"schedule": 300.0, # every 5 min, matches recover-interrupted-tasks
},
"prune-task-runs": {
"task": "backend.app.tasks.maintenance.prune_task_runs",
"schedule": 86400.0, # daily
},
"fc3h-backup-db-nightly": {
"task": "backend.app.tasks.backup.backup_db_nightly",
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
},
"fc3h-prune-backups": {
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
},
timezone="UTC",
)
# FC-3i: register task_run signal handlers (side-effect import).
from . import celery_signals # noqa: F401
return app
+210
View File
@@ -0,0 +1,210 @@
"""FC-3i: task_run lifecycle via Celery signals.
Subscribes to task_prerun / task_postrun / task_failure / task_retry
and persists one task_run row per task attempt. Drop-in for every
existing and future Celery task — no per-task instrumentation.
Signal handlers run inside the worker process (sync context); DB
writes go through the existing shared sync engine
(backend.app.tasks._sync_engine.sync_session_factory) — one engine
per worker process, not per-task, so we don't blow Postgres
max_connections under load (the reason FC-3g shared-engine fix
existed).
Failure-mode discipline (operator-pressed point): every handler is
wrapped in try/except that swallows + logs. If the DB is down or the
handler has a bug, the real task still runs — the dashboard goes
dark for that interval. Monitoring NEVER breaks the thing it's
monitoring.
"""
import logging
from datetime import UTC, datetime
from celery.exceptions import SoftTimeLimitExceeded
from celery.signals import task_failure, task_postrun, task_prerun, task_retry
from .models import TaskRun
from .tasks._sync_engine import sync_session_factory
log = logging.getLogger(__name__)
# Celery-internal tasks that would generate dashboard noise without
# operational value. Conservative list; extend only when a specific
# task proves noisy.
_UNTRACKED_TASK_NAMES = frozenset({
"celery.chord_unlock",
"celery.backend_cleanup",
"celery.chunks",
})
_MAX_ERROR_MESSAGE_LEN = 2000
_MAX_ARGS_SUMMARY_LEN = 255
_MAX_WORKER_HOSTNAME_LEN = 128
# PostgreSQL Integer is signed 32-bit. Tasks called with a first-arg
# int outside this range (e.g. an absurdly large mock value, or a
# string-of-digits coercible to int but bigger than 2^31-1) would crash
# the INSERT with NumericValueOutOfRange. Bound the recorded value to
# the column's range; values outside become None (target_id is
# nullable, so this is safe).
_INT32_MAX = 2_147_483_647
_INT32_MIN = -2_147_483_648
def _queue_for(task) -> str:
"""Reverse the task→queue routing from celery_app.task_routes.
Keep in sync if task_routes is reordered."""
name = getattr(task, "name", "") or ""
if name.startswith("backend.app.tasks.import_file."):
return "import"
if name.startswith("backend.app.tasks.ml."):
return "ml"
if name.startswith("backend.app.tasks.thumbnail."):
return "thumbnail"
if name.startswith("backend.app.tasks.download."):
return "download"
if name.startswith("backend.app.tasks.scan."):
return "scan"
if name.startswith((
"backend.app.tasks.maintenance.",
"backend.app.tasks.migration.",
)):
return "maintenance"
return "default"
def _target_id_from_args(args) -> int | None:
"""Best-effort: if the first positional arg parses as int AND fits
in the column's signed-32-bit range, record it as target_id
(image_id, source_id, etc.). Never raises."""
if not args:
return None
try:
value = int(args[0])
except (TypeError, ValueError):
return None
if value < _INT32_MIN or value > _INT32_MAX:
return None
return value
def _truncate(s, limit: int) -> str | None:
if s is None:
return None
text = str(s)
return text if len(text) <= limit else text[:limit]
def _is_tracked(task_name: str | None) -> bool:
return bool(task_name) and task_name not in _UNTRACKED_TASK_NAMES
@task_prerun.connect
def _on_prerun(sender=None, task_id=None, task=None, args=None,
kwargs=None, **_):
if not _is_tracked(getattr(task, "name", None)):
return
try:
Session = sync_session_factory()
with Session() as session:
session.add(TaskRun(
celery_task_id=task_id or "",
queue=_queue_for(task),
task_name=task.name,
target_id=_target_id_from_args(args),
started_at=datetime.now(UTC),
status="running",
args_summary=_truncate(repr(args), _MAX_ARGS_SUMMARY_LEN),
worker_hostname=_truncate(
getattr(sender, "hostname", None),
_MAX_WORKER_HOSTNAME_LEN,
),
))
session.commit()
except Exception: # noqa: BLE001 — never break the worker
log.exception("task_run prerun insert failed (task=%s)",
getattr(task, "name", "?"))
def _finalize(task_id: str, *, status: str,
error_type: str | None = None,
error_message: str | None = None,
retry_count: int | None = None) -> None:
"""Shared write path for postrun/failure/retry. Picks the most-
recent task_run row for this celery_task_id that's still 'running'
(retries reuse the same celery_task_id; each new attempt's prerun
inserts a fresh row, so finalize targets the latest running row)."""
try:
from sqlalchemy import select
Session = sync_session_factory()
now = datetime.now(UTC)
with Session() as session:
row = session.execute(
select(TaskRun)
.where(TaskRun.celery_task_id == task_id)
.where(TaskRun.status == "running")
.order_by(TaskRun.id.desc())
.limit(1)
).scalar_one_or_none()
if row is None:
return # no prerun row (untracked, insert failed, or already finalized)
row.finished_at = now
row.duration_ms = int(
(now - row.started_at).total_seconds() * 1000
)
row.status = status
if error_type is not None:
row.error_type = _truncate(error_type, 128)
if error_message is not None:
row.error_message = _truncate(error_message, _MAX_ERROR_MESSAGE_LEN)
if retry_count is not None:
row.retry_count = retry_count
session.commit()
except Exception: # noqa: BLE001
log.exception("task_run finalize failed (task_id=%s)", task_id)
@task_postrun.connect
def _on_postrun(sender=None, task_id=None, task=None, args=None,
kwargs=None, retval=None, state=None, **_):
if not _is_tracked(getattr(task, "name", None)):
return
# state is one of SUCCESS/FAILURE/RETRY/etc. Only handle SUCCESS;
# task_failure handles FAILURE explicitly (with the exception).
if state != "SUCCESS":
return
_finalize(task_id, status="ok")
@task_failure.connect
def _on_failure(sender=None, task_id=None, exception=None,
args=None, kwargs=None, einfo=None, **_):
if not _is_tracked(getattr(sender, "name", None)):
return
status = ("timeout"
if isinstance(exception, SoftTimeLimitExceeded)
else "error")
_finalize(
task_id, status=status,
error_type=type(exception).__name__ if exception else "Unknown",
error_message=str(exception) if exception else None,
)
@task_retry.connect
def _on_retry(sender=None, request=None, reason=None, einfo=None, **_):
if not _is_tracked(getattr(sender, "name", None)):
return
task_id = getattr(request, "id", None)
if not task_id:
return
# Mark current attempt's row as 'retry' (terminal for this row).
# The next attempt's task_prerun inserts a fresh row.
_finalize(
task_id, status="retry",
error_type=type(reason).__name__ if reason else "Retry",
error_message=str(reason) if reason else None,
retry_count=getattr(request, "retries", 0),
)
+6
View File
@@ -2,6 +2,7 @@
from .app_setting import AppSetting
from .artist import Artist
from .backup_run import BackupRun
from .base import Base
from .credential import Credential
from .download_event import DownloadEvent
@@ -10,6 +11,7 @@ from .image_record import ImageRecord
from .import_batch import ImportBatch
from .import_settings import ImportSettings
from .import_task import ImportTask
from .library_audit_run import LibraryAuditRun
from .migration_run import MigrationRun
from .ml_settings import MLSettings
from .post import Post
@@ -21,11 +23,13 @@ from .tag_alias import TagAlias
from .tag_allowlist import TagAllowlist
from .tag_reference_embedding import TagReferenceEmbedding
from .tag_suggestion_rejection import TagSuggestionRejection
from .task_run import TaskRun
__all__ = [
"Base",
"AppSetting",
"Artist",
"BackupRun",
"Source",
"Credential",
"Post",
@@ -40,10 +44,12 @@ __all__ = [
"ImportBatch",
"ImportTask",
"ImportSettings",
"LibraryAuditRun",
"MLSettings",
"MigrationRun",
"TagAlias",
"TagAllowlist",
"TagReferenceEmbedding",
"TagSuggestionRejection",
"TaskRun",
]
+55
View File
@@ -0,0 +1,55 @@
"""FC-3h: backup_run — operator-facing artifact record for a backup run.
One row per backup attempt (kind='db' or 'images'). Lifecycle
tracking (started_at/finished_at/duration_ms/exception text) lives
in task_run from FC-3i — this row records artifact metadata: file
paths, sizes, tag (retention protection), and restore lineage via
restored_from_id.
Status values (String, not Postgres ENUM — per
feedback_check_existing_enums):
pending — created but task hasn't started yet (rare; usually
status starts as 'running' from the task body).
running — backup task is in flight.
ok — artifact successfully written.
error — task raised; error column populated.
restoring — this row represents a restore attempt (kind = restored
kind); linked to source via restored_from_id.
restored — restore completed successfully.
"""
from datetime import datetime
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class BackupRun(Base):
__tablename__ = "backup_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True,
)
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True,
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True,
)
sql_path: Mapped[str | None] = mapped_column(Text, nullable=True)
tar_path: Mapped[str | None] = mapped_column(Text, nullable=True)
size_bytes: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
manifest: Mapped[dict] = mapped_column(
JSON, nullable=False, default=dict, server_default="{}",
)
restored_from_id: Mapped[int | None] = mapped_column(
ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
)
+15 -5
View File
@@ -1,14 +1,18 @@
"""ImageProvenance — links an ImageRecord to a Post.
Many-to-one (one image, many provenance rows) enables the enrich-on-duplicate
rule (spec §3): when a downloaded image is a pHash dupe of an existing
record, we append a new provenance row to the existing record rather than
dropping the metadata.
One image can have many provenance rows — different posts each contribute
metadata (enrich-on-duplicate rule, spec §3: a downloaded image that is a
pHash dupe of an existing record gets a NEW provenance row for the new post
appended, rather than the metadata being dropped). But the (image, post)
pair is unique — alembic 0021 enforces uq_image_provenance_image_post
after operator-flagged 2026-05-26 saw _apply_sidecar's existence-check +
INSERT race plant duplicates that then broke .scalar_one_or_none() on
every later deep-scan rederive (MultipleResultsFound).
"""
from datetime import datetime
from sqlalchemy import JSON, DateTime, ForeignKey, Integer, func
from sqlalchemy import JSON, DateTime, ForeignKey, Integer, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
@@ -16,6 +20,12 @@ from .base import Base
class ImageProvenance(Base):
__tablename__ = "image_provenance"
__table_args__ = (
UniqueConstraint(
"image_record_id", "post_id",
name="uq_image_provenance_image_post",
),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
image_record_id: Mapped[int] = mapped_column(
+4
View File
@@ -26,6 +26,10 @@ class ImportBatch(Base):
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Deep-scan only: count of already-imported files whose sidecar metadata
# got re-applied this run (post/source/provenance upsert). Stays 0 on
# quick-scan batches. See `Importer.import_one(deep_scan=True)`.
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True)
# running | complete | cancelled
+14
View File
@@ -49,3 +49,17 @@ class ImportSettings(Base):
download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5
)
# FC-3h backup knobs.
backup_db_nightly_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False,
)
backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
backup_db_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=14,
)
backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
+17 -1
View File
@@ -8,7 +8,16 @@ been processing longer than the stuck-task threshold.
from datetime import datetime
from sqlalchemy import BigInteger, DateTime, ForeignKey, Integer, String, Text, func
from sqlalchemy import (
BigInteger,
Boolean,
DateTime,
ForeignKey,
Integer,
String,
Text,
func,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base
@@ -26,6 +35,13 @@ class ImportTask(Base):
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True)
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks
# how many times the stuck-task sweep has re-queued this row; after
# the cap it's failed with a diagnostic instead of looping. refetched
# bounds the one-shot re-download remediation to a single attempt.
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
result_image_id: Mapped[int | None] = mapped_column(
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
)
+37
View File
@@ -0,0 +1,37 @@
"""LibraryAuditRun — async transparency / single_color audit lifecycle.
State machine: running → ready → applied / cancelled / error.
matched_ids JSONB is appended-to by scan_library_for_rule; apply_audit_run
reads it and routes through cleanup_service.delete_images.
"""
from datetime import datetime
from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class LibraryAuditRun(Base):
__tablename__ = "library_audit_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
rule: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True,
)
# running | ready | applied | cancelled | error
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(),
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
+4 -2
View File
@@ -35,8 +35,10 @@ class TagKind(StrEnum):
series = "series"
archive = "archive"
post = "post"
meta = "meta"
rating = "rating"
# `meta` and `rating` retired by operator 2026-05-26 (alembic 0023).
# `artist` retired in FC-2d-vii-c — artists are first-class entities
# via Artist/Source rows now, not tags — but the enum value stays
# to keep historic tag rows queryable.
image_tag = Table(
+48
View File
@@ -0,0 +1,48 @@
"""FC-3i: task_run — per-Celery-task lifecycle audit row.
One row inserted by the task_prerun signal at task start, updated by
task_postrun / task_failure / task_retry. The shape supports the
SystemActivity dashboard's three panes: per-queue queue+worker summary,
recent failures (24h, grouped by error_type), and full paginated
activity history.
Retention: ok rows pruned after 24h, error/timeout after 7d (see
backend.app.tasks.maintenance.prune_task_runs).
Recovery: rows stuck in 'running' for >5 min flipped to 'error' by
backend.app.tasks.maintenance.recover_stalled_task_runs (Beat 5 min).
"""
from datetime import datetime
from sqlalchemy import DateTime, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class TaskRun(Base):
__tablename__ = "task_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
celery_task_id: Mapped[str] = mapped_column(
String(64), nullable=False, index=True,
)
queue: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
task_name: Mapped[str] = mapped_column(String(128), nullable=False, index=True)
target_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True,
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True,
)
duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True,
)
error_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
retry_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
worker_hostname: Mapped[str | None] = mapped_column(String(128), nullable=True)
args_summary: Mapped[str | None] = mapped_column(String(255), nullable=True)
+21 -2
View File
@@ -25,12 +25,31 @@ def _snapshot(repo_id: str, dest: Path, allow_patterns: list[str] | None) -> Non
def ensure_camie() -> None:
"""Fetch Camie v2 weights + metadata.
v2 layout (HuggingFace Camais03/camie-tagger-v2): the ONNX file is
named camie-tagger-v2.onnx (not model.onnx) and tags ship inside
camie-tagger-v2-metadata.json (not selected_tags.csv). Both at root.
The repo also contains app/, game/, training/, images/ subdirs full
of setup/demo files we don't need — allow_patterns scopes the fetch
to just the inference essentials (~790 MB instead of ~2 GB).
"""
dest = MODEL_ROOT / "camie"
if (dest / "model.onnx").is_file() and (dest / "selected_tags.csv").is_file():
model_file = dest / "camie-tagger-v2.onnx"
meta_file = dest / "camie-tagger-v2-metadata.json"
if model_file.is_file() and meta_file.is_file():
print(f"[download_models] Camie present at {dest}")
return
print(f"[download_models] Fetching {CAMIE_REPO} -> {dest}")
_snapshot(CAMIE_REPO, dest, ["model.onnx", "selected_tags.csv", "*.json"])
_snapshot(
CAMIE_REPO, dest,
[
"camie-tagger-v2.onnx",
"camie-tagger-v2-metadata.json",
"config.json",
"config.yaml",
],
)
def ensure_siglip() -> None:
+10
View File
@@ -111,12 +111,22 @@ class ArtistService:
)
).all()
post_count = (
await self.session.execute(
select(func.count(func.distinct(Post.id)))
.select_from(Post)
.join(Source, Source.id == Post.source_id)
.where(Source.artist_id == aid)
)
).scalar_one()
return {
"id": artist.id,
"name": artist.name,
"slug": artist.slug,
"is_subscription": bool(artist.is_subscription),
"image_count": int(image_count),
"post_count": int(post_count),
"date_range": {
"min": dmin.isoformat() if dmin else None,
"max": dmax.isoformat() if dmax else None,
+6
View File
@@ -0,0 +1,6 @@
"""Audit rule modules. Each module exposes evaluate(pil_image, **params) -> bool.
The retroactive library-cleanup tab and (future) import-time filter logic
both consume these. Importers should NOT inline rule logic going forward;
add the rule here and call from both sides.
"""
@@ -0,0 +1,53 @@
"""Single-color audit: matches images where one color dominates beyond
the threshold (within the given Euclidean RGB tolerance). The first
canonical implementation — the import-side filter (SkipReason.single_color)
was never wired; FC-Cleanup's audit module is the source of truth and a
future spec can adopt it on the import path too.
"""
from PIL import Image
_THUMB_SIZE = (64, 64)
def evaluate(
pil_image,
*,
threshold: float,
tolerance: int,
) -> bool:
"""True iff the fraction of pixels within `tolerance` (Euclidean RGB
distance) of the dominant color exceeds `threshold`.
Downsamples to 64x64 for speed (~4ms regardless of source size).
Alpha channels are stripped; only RGB is considered. Animated images
use frame 0 (PIL's default after Image.open without seek).
"""
im = pil_image
if im.mode == "RGBA":
im = im.convert("RGB")
elif im.mode not in ("RGB", "L"):
im = im.convert("RGB")
if im.size != _THUMB_SIZE:
im = im.resize(_THUMB_SIZE, Image.Resampling.BILINEAR)
pixels = list(im.getdata())
if not pixels:
return False
# Normalize L-mode pixels to RGB tuples for distance math.
if isinstance(pixels[0], int):
pixels = [(p, p, p) for p in pixels]
# Dominant color = mean RGB.
n = len(pixels)
sum_r = sum(p[0] for p in pixels)
sum_g = sum(p[1] for p in pixels)
sum_b = sum(p[2] for p in pixels)
dom = (sum_r / n, sum_g / n, sum_b / n)
tol_sq = tolerance * tolerance
within = 0
for r, g, b in pixels:
dr = r - dom[0]
dg = g - dom[1]
db = b - dom[2]
if dr * dr + dg * dg + db * db <= tol_sq:
within += 1
return (within / n) > threshold
@@ -0,0 +1,27 @@
"""Transparency audit: matches images whose transparent-pixel fraction
exceeds the threshold. Animated images short-circuit (skipped) to avoid
the multi-frame PIL decode that hits Celery's hard time limit."""
def evaluate(pil_image, *, threshold: float) -> bool:
"""True iff the image's transparent-pixel fraction exceeds threshold.
False for non-alpha modes and animated images. Mirrors the import-side
Importer._transparency_pct logic so retroactive enforcement matches
prospective filtering.
"""
if getattr(pil_image, "is_animated", False):
return False
if pil_image.mode not in ("RGBA", "LA") and not (
pil_image.mode == "P" and "transparency" in pil_image.info
):
return False
im = pil_image
if im.mode != "RGBA":
im = im.convert("RGBA")
alpha = im.getchannel("A")
histogram = alpha.histogram()
transparent = histogram[0]
total = sum(histogram)
pct = transparent / total if total else 0.0
return pct > threshold
+196
View File
@@ -0,0 +1,196 @@
"""FC-3h: first-class backup/restore service for FC.
Two independent backup kinds:
- 'db' — pg_dump only; fast; nightly via Beat (settings-gated)
- 'images' — tar+zstd of /images; slow; manual trigger only
Files live under <images_root>/_backups/. Each backup writes:
fc_<kind>_<ts>.{sql|tar.zst} — the artifact
fc_<kind>_<ts>.json — manifest (kind/tag/triggered_by)
Service functions are sync (subprocess-bound). Celery tasks in
backend.app.tasks.backup wrap each one with task_run-tracked
lifecycle + soft/hard time limits + retention bookkeeping.
"""
from __future__ import annotations
import json
import subprocess
from datetime import UTC, datetime
from pathlib import Path
_BACKUPS_DIRNAME = "_backups"
# Subprocess-level guardrails BEYOND the Celery soft_time_limit. The
# Celery soft limit signals the Python process; subprocess.Popen in a
# blocking syscall ignores that signal. These bound the worst case.
_DB_SUBPROCESS_TIMEOUT_S = 12 * 60 # 12 min (Celery soft is 10 min)
_IMAGES_SUBPROCESS_TIMEOUT_S = 7 * 60 * 60 # 7 hr (Celery soft is 6 hr)
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy +psycopg/+asyncpg driver suffix for pg_dump/psql."""
for driver in (
"postgresql+psycopg",
"postgresql+asyncpg",
"postgresql+psycopg2",
):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path) -> Path:
p = images_root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
def _now_ts() -> str:
return datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
def _file_size_or_none(path: Path) -> int | None:
try:
return path.stat().st_size
except OSError:
return None
def _write_manifest(
out_dir: Path, *, kind: str, ts: str,
tag: str | None, triggered_by: str,
artifact_path: Path,
) -> Path:
manifest = {
"kind": kind,
"backup_id": f"fc_{kind}_{ts}",
"tag": tag,
"triggered_by": triggered_by,
"created_at": datetime.now(UTC).isoformat(),
"artifact_path": str(artifact_path),
}
mf = out_dir / f"fc_{kind}_{ts}.json"
mf.write_text(json.dumps(manifest, indent=2))
return mf
def backup_db(
*, db_url: str, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run pg_dump; write .sql + manifest; return dict for the caller
to persist into BackupRun. Raises on subprocess failure."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_db_{ts}.sql"
subprocess.run(
[
"pg_dump", "--no-owner", "--no-acl",
"-f", str(sql_path), _libpq_url(db_url),
],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="db", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=sql_path,
)
return {
"kind": "db",
"ts": ts,
"sql_path": str(sql_path),
"tar_path": None,
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(sql_path),
}
def backup_images(
*, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run tar --zstd over images_root; write .tar.zst + manifest."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
subprocess.run(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=tar_path,
)
return {
"kind": "images",
"ts": ts,
"sql_path": None,
"tar_path": str(tar_path),
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(tar_path),
}
def restore_db(*, db_url: str, sql_path: Path) -> None:
"""Wipe public schema, then load from .sql. Raises on subprocess
failure; partial-restore state is the caller's concern."""
libpq = _libpq_url(db_url)
subprocess.run(
[
"psql", libpq, "-c",
"DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public;",
],
capture_output=True, check=True, timeout=120,
)
subprocess.run(
["psql", libpq, "-f", str(sql_path)],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
def restore_images(*, images_root: Path, tar_path: Path) -> None:
"""Untar over images_root.parent. Additive — files NOT in the
tarball are NOT removed. Caller wipes first if a clean restore
is needed."""
subprocess.run(
[
"tar", "--zstd", "-xf", str(tar_path),
"-C", str(images_root.parent),
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
def unlink_artifact_files(
*,
sql_path: str | None,
tar_path: str | None,
manifest_path: str | None,
) -> dict:
"""Best-effort unlink of all on-disk files for a BackupRun row.
Returns dict keyed by label with True/False per file. Missing
files count as success (missing_ok semantics)."""
deleted: dict = {}
for label, p in (
("sql", sql_path),
("tar", tar_path),
("manifest", manifest_path),
):
if not p:
continue
path = Path(p)
try:
path.unlink(missing_ok=True)
deleted[label] = True
except OSError:
deleted[label] = False
return deleted
+512
View File
@@ -0,0 +1,512 @@
"""FC-3k: first-class admin destructive operations.
Projections are pure SELECTs used by both dry-run preview endpoints
and Tier-B count prompts. Mutations (Task 2) are called from sync
HTTP handlers (small ops) and from Celery tasks in
backend.app.tasks.admin (long ops).
This module is the PERMANENT home of artist-cascade + image-unlink
logic. The legacy copy at backend/app/services/migrators/cleanup.py
stays in place until FC-3j; FC-3j will replace its body with thin
re-exports from this module and then delete the wrapper.
"""
from __future__ import annotations
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session
from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
from ..models.series_page import SeriesPage
from ..models.tag import image_tag
def project_artist_cascade(session: Session, *, slug: str) -> dict:
"""Read-only projection of what delete_artist_cascade would touch.
Returns:
{
"artist": {"id": int, "name": str, "slug": str},
"projected": {
"images": int,
"sources": int,
"thumbs": int, # images with a thumbnail_path set
"import_tasks": int, # ImportTask rows referencing the artist's images
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
},
}
Raises LookupError if slug not found. No mutations.
"""
from ..models.import_task import ImportTask
from ..models.source import Source
artist = session.execute(
select(Artist).where(Artist.slug == slug)
).scalar_one_or_none()
if artist is None:
raise LookupError(f"artist slug not found: {slug!r}")
images_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
sources_count = session.execute(
select(func.count(Source.id))
.where(Source.artist_id == artist.id)
).scalar_one()
thumbs_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
.where(ImageRecord.thumbnail_path.is_not(None))
).scalar_one()
import_tasks_count = session.execute(
select(func.count(ImportTask.id))
.where(
ImportTask.result_image_id.in_(
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
)
)
).scalar_one()
bytes_on_disk = session.execute(
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
return {
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"projected": {
"images": images_count,
"sources": sources_count,
"thumbs": thumbs_count,
"import_tasks": import_tasks_count,
"bytes_on_disk": int(bytes_on_disk),
},
}
def project_bulk_image_delete(
session: Session, *, image_ids: list[int],
) -> dict:
"""Read-only projection of what delete_images would touch.
Returns:
{
"images_found": int,
"thumbs_to_unlink": int,
"bytes_on_disk": int,
"missing_ids": list[int], # ids passed in that don't exist
}
No mutations.
"""
if not image_ids:
return {
"images_found": 0,
"thumbs_to_unlink": 0,
"bytes_on_disk": 0,
"missing_ids": [],
}
rows = session.execute(
select(
ImageRecord.id,
ImageRecord.thumbnail_path,
ImageRecord.size_bytes,
).where(ImageRecord.id.in_(image_ids))
).all()
found_ids = {r.id for r in rows}
missing = sorted(set(image_ids) - found_ids)
return {
"images_found": len(rows),
"thumbs_to_unlink": sum(1 for r in rows if r.thumbnail_path),
"bytes_on_disk": sum(r.size_bytes for r in rows),
"missing_ids": missing,
}
def count_tag_associations(session: Session, *, tag_id: int) -> int:
"""COUNT(*) FROM image_tag WHERE tag_id=?. For Tier-B prompt."""
return session.execute(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id == tag_id)
).scalar_one()
def find_unused_tags(
session: Session, *, limit: int | None = None,
) -> list[Tag]:
"""Tags with no image_tag rows AND no series_page rows.
Sorted by name. Used by both dry-run preview and the live prune.
A tag is "unused" iff it has zero rows in image_tag AND zero rows
in series_page (so we don't accidentally prune a series tag that
happens to have no images yet).
"""
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
stmt = (
select(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
.order_by(Tag.name)
)
if limit is not None:
stmt = stmt.limit(limit)
return list(session.execute(stmt).scalars().all())
def unlink_image_files(
image: ImageRecord, images_root: Path,
) -> dict:
"""Best-effort unlink of all on-disk files for an ImageRecord.
Targets: image.path (original), image.thumbnail_path (cached
thumbnail), and the computed thumbs path at
/images/thumbs/<sha256[:3]>/<sha256>.(jpg|png|webp) (tries all
three extensions; missing extension is silently OK).
Returns {"original": bool, "thumbnail": bool}. Missing files
count as success (missing_ok semantics). OSErrors are swallowed
and reported as False so the calling DB delete still proceeds.
"""
out = {"original": False, "thumbnail": False}
if image.path:
try:
Path(image.path).unlink(missing_ok=True)
out["original"] = True
except OSError:
out["original"] = False
# Custom thumbnail_path (when set) — try it first.
if image.thumbnail_path:
try:
Path(image.thumbnail_path).unlink(missing_ok=True)
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
)
except OSError:
pass
return out
def delete_artist_cascade(
session: Session, *, artist_id: int, images_root: Path,
) -> dict:
"""Batched delete of an artist's images + the artist row.
Mirrors the cleanup_artist_async pattern: 500-row batches,
commit between batches so partial progress survives a worker
kill. Idempotent on missing artist (returns zeroed counts).
Postgres cascades handle image_tag / image_provenance /
series_page / tag_suggestion_rejection from ImageRecord delete,
and source / post / download_event / etc. from Artist delete
(via Artist.sources cascade="all, delete-orphan").
"""
artist = session.get(Artist, artist_id)
if artist is None:
return {
"artist": None,
"summary": {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"import_tasks_nulled": 0,
"files_failed": 0,
},
}
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
while True:
rows = session.execute(
select(ImageRecord)
.where(ImageRecord.artist_id == artist.id)
.limit(500)
).scalars().all()
if not rows:
break
for img in rows:
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
# ImportTask.result_image_id FK is SET NULL on image delete (Postgres
# handles this in the cascade above). We don't separately count those
# in FC-3k — the legacy cleanup_artist_async did it via
# source_path_prefix matching that's out of scope here.
import_tasks_nulled = 0
session.delete(artist)
session.commit()
return {
"artist": artist_info,
"summary": {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_nulled": import_tasks_nulled,
"files_failed": files_failed,
},
}
def delete_images(
session: Session, *, image_ids: list[int], images_root: Path,
) -> dict:
"""Delete a list of images in 500-row batches with commit between.
Postgres CASCADE on image_tag / image_provenance / series_page /
tag_suggestion_rejection / post_attachment(FK SET NULL) handles
the DB side; this function handles file unlinks first then row
deletes. Idempotent on missing IDs (returned as missing_ids;
no error). On partial OSError, the row is still deleted and
files_failed is incremented.
"""
if not image_ids:
return {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"files_failed": 0,
"missing_ids": [],
}
seen_ids: set[int] = set()
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
pending = list(image_ids)
while pending:
batch_ids = pending[:500]
pending = pending[500:]
rows = session.execute(
select(ImageRecord).where(ImageRecord.id.in_(batch_ids))
).scalars().all()
for img in rows:
seen_ids.add(img.id)
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
missing = sorted(set(image_ids) - seen_ids)
return {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"files_failed": files_failed,
"missing_ids": missing,
}
def delete_tag(session: Session, *, tag_id: int) -> dict:
"""Simple DELETE FROM tag WHERE id=?.
Postgres cascades the rest (image_tag, tag_alias, tag_allowlist,
tag_reference_embedding, tag_suggestion_rejection, series_page).
Returns counts BEFORE delete so the caller can surface them.
Raises LookupError if tag_id not found.
"""
tag = session.get(Tag, tag_id)
if tag is None:
raise LookupError(f"tag id not found: {tag_id}")
associations_count = count_tag_associations(session, tag_id=tag_id)
info = {"id": tag.id, "name": tag.name, "kind": tag.kind.value}
session.delete(tag)
session.commit()
return {"deleted": info, "associations_removed": associations_count}
def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
"""Find tags with zero references and (unless dry_run) delete them.
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
# ---------------------------------------------------------------------------
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
# ---------------------------------------------------------------------------
_MIN_DIM_SAMPLE_CAP = 50
def project_min_dimension_violations(
session: Session, *, min_width: int, min_height: int,
) -> dict:
"""Return {count, sample_ids} for image_record rows with width or
height below the thresholds. Synchronous SQL — no PIL inspection
needed since width/height are stored columns."""
base = select(ImageRecord.id).where(
(ImageRecord.width < min_width) | (ImageRecord.height < min_height)
)
count = session.execute(
select(func.count()).select_from(base.subquery())
).scalar_one()
sample_ids = session.execute(
base.order_by(ImageRecord.id).limit(_MIN_DIM_SAMPLE_CAP)
).scalars().all()
return {"count": count, "sample_ids": list(sample_ids)}
def delete_min_dimension_violations(
session: Session, *, min_width: int, min_height: int, images_root: Path,
) -> int:
"""Delete every image_record where width<min_w OR height<min_h.
Routes through delete_images so file-unlink + cascading FKs
(image_tag / image_provenance / etc.) are handled uniformly."""
ids = session.execute(
select(ImageRecord.id).where(
(ImageRecord.width < min_width) | (ImageRecord.height < min_height)
)
).scalars().all()
if not ids:
return 0
result = delete_images(
session, image_ids=list(ids), images_root=images_root,
)
return result["images_deleted"]
# ---------------------------------------------------------------------------
# Audit lifecycle (transparency + single_color async scans).
# ---------------------------------------------------------------------------
class AuditAlreadyRunning(Exception):
"""Another audit_run is currently in status='running' — wait or
cancel it before starting a new one. Surfaces as HTTP 409 in the
/api/cleanup/audit POST endpoint."""
class AuditNotReady(Exception):
"""apply_audit_run called on an audit whose status is not 'ready'."""
class ConfirmTokenMismatch(Exception):
"""Operator-supplied confirm token did not match server-recomputed token."""
_VALID_RULES = ("transparency", "single_color")
def start_audit_run(
session: Session, *, rule: str, params: dict[str, Any],
) -> int:
"""Create a LibraryAuditRun row in status='running' and dispatch the
scan_library_for_rule Celery task. Returns the new audit_id.
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
has status='running'. Operator must cancel or wait."""
if rule not in _VALID_RULES:
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
existing = session.execute(
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
).scalar_one_or_none()
if existing is not None:
raise AuditAlreadyRunning(existing)
audit = LibraryAuditRun(
rule=rule,
params=params,
status="running",
scanned_count=0,
matched_count=0,
matched_ids=[],
)
session.add(audit)
session.flush()
audit_id = audit.id
# Dispatch after flush so audit_id is populated; commit happens in
# the API handler so the audit row + dispatch are visible together.
from ..tasks.library_audit import scan_library_for_rule
scan_library_for_rule.delay(audit_id)
return audit_id
def apply_audit_run(
session: Session, *, audit_id: int, confirm_token: str, images_root: Path,
) -> int:
"""Delete all images in audit_run.matched_ids after confirming token.
Marks audit status='applied'. Routes through delete_images so files
+ cascading FK rows are handled uniformly."""
audit = session.execute(
select(LibraryAuditRun).where(LibraryAuditRun.id == audit_id)
).scalar_one_or_none()
if audit is None:
raise ValueError(f"audit_run {audit_id} not found")
if audit.status != "ready":
raise AuditNotReady(audit.status)
# Token format matches modal/DestructiveConfirmModal.vue convention:
# ${action}-${kind}-${runId}. The modal hardcodes action ∈ {'restore',
# 'delete'}; "apply audit" is semantically a delete of the matched
# images, so we use 'delete-audit-<id>' (not 'apply-audit-<id>').
expected = f"delete-audit-{audit_id}"
if confirm_token != expected:
raise ConfirmTokenMismatch(expected)
ids = list(audit.matched_ids or [])
deleted = 0
if ids:
result = delete_images(session, image_ids=ids, images_root=images_root)
deleted = result["images_deleted"]
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(status="applied", finished_at=datetime.now(UTC))
)
return deleted
def cancel_audit_run(session: Session, *, audit_id: int) -> None:
"""Flip a running audit_run to 'cancelled'. The scan task checks
for status=='cancelled' between batches and exits cleanly."""
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.where(LibraryAuditRun.status == "running")
.values(status="cancelled", finished_at=datetime.now(UTC))
)
@@ -148,6 +148,7 @@ class CredentialService:
return None
plaintext = self.crypto.decrypt(row.encrypted_blob)
netscape = _to_netscape(plaintext)
netscape = _augment_cookies(platform, netscape)
self.cookies_dir.mkdir(parents=True, exist_ok=True)
out = self.cookies_dir / f"{platform}_cookies.txt"
out.write_text(netscape)
@@ -163,6 +164,19 @@ class CredentialService:
return self.crypto.decrypt(row.encrypted_blob)
def _augment_cookies(platform: str, netscape: str) -> str:
"""Delegate to the platform's `augment_cookies` hook if one is
registered (subscribestar, hentaifoundry, etc. — see
`services/platforms/<name>.py`). No-op when the platform doesn't
register a hook (Patreon, DeviantArt). Centralizing the
quirks-per-platform in the platforms package means adding a new
platform's cookie quirks doesn't require touching this file."""
info = PLATFORMS.get(platform)
if info is None or info.augment_cookies is None:
return netscape
return info.augment_cookies(netscape)
def _to_netscape(plaintext: str) -> str:
"""Accept either Netscape-format text (the extension's output) or a
JSON array of cookie dicts (a manual-paste edge case); produce
+44 -7
View File
@@ -11,6 +11,7 @@ from __future__ import annotations
import re
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
@@ -97,20 +98,40 @@ class ExtensionService:
raise UnknownPlatformError(f"no platform pattern matched {url!r}")
async def _find_or_create_artist(self, raw_name: str) -> tuple[Artist, bool]:
"""Race-safe find-or-create on Artist by slug. Mirrors the
savepoint + IntegrityError recovery pattern used in
Importer._find_or_create_source/post (see
reference_scalar_one_or_none_duplicates memory). Without this,
two concurrent quick-add-source calls hitting the same artist
would both miss the existence check and the second INSERT would
500 against uq_artist_slug.
"""
slug = slugify(raw_name)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=raw_name, slug=slug, is_subscription=True)
self.session.add(artist)
await self.session.flush()
return artist, True
sp = await self.session.begin_nested()
try:
artist = Artist(name=raw_name, slug=slug, is_subscription=True)
self.session.add(artist)
await self.session.flush()
await sp.commit()
return artist, True
except IntegrityError:
await sp.rollback()
recovered = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
return recovered, False
async def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> tuple[Source, bool]:
"""Race-safe — same pattern as _find_or_create_artist above. The
uq_source_artist_platform_url constraint catches the duplicate
insert; we roll the savepoint back and re-select."""
existing = (await self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
@@ -120,8 +141,24 @@ class ExtensionService:
)).scalar_one_or_none()
if existing is not None:
return existing, False
src = Source(artist_id=artist_id, platform=platform, url=url, enabled=True)
self.session.add(src)
await self.session.flush()
sp = await self.session.begin_nested()
try:
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
)
self.session.add(src)
await self.session.flush()
await sp.commit()
except IntegrityError:
await sp.rollback()
recovered = (await self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one()
return recovered, False
await self.session.commit()
return src, True
+126 -54
View File
@@ -1,26 +1,34 @@
"""Cursor-paginated gallery queries.
Cursor format: opaque base64-encoded "<iso8601_created_at>:<image_id>".
Pagination key is (created_at DESC, id DESC) so we don't drift when new
imports arrive between page loads. Decoding rejects malformed cursors with
a ValueError; the API layer translates that to HTTP 400.
Cursor format: opaque base64-encoded "<iso8601_effective_date>:<image_id>".
Pagination key is (effective_date DESC, id DESC) where effective_date is
COALESCE(post.post_date, image_record.created_at) so the gallery surfaces
images by ORIGINAL publish date when known, falling back to FC's scan
date. Important for migrated content: ~57k IR images scanned in a single
week would otherwise all share the same created_at and pile up in one
month bucket. The effective_date spreads them across the years they
were originally published.
Decoding rejects malformed cursors with a ValueError; the API layer
translates that to HTTP 400.
"""
import base64
from dataclasses import dataclass
from datetime import datetime
from sqlalchemy import and_, exists, func, or_, select
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ImageProvenance, ImageRecord, Source, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
def encode_cursor(created_at: datetime, image_id: int) -> str:
raw = f"{created_at.isoformat()}{CURSOR_SEPARATOR}{image_id}"
def encode_cursor(effective_date: datetime, image_id: int) -> str:
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
return base64.urlsafe_b64encode(raw.encode()).decode()
@@ -33,6 +41,26 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
raise ValueError(f"invalid cursor: {cursor!r}") from exc
def _effective_date_col():
"""SQL expression: COALESCE(post.post_date, image_record.created_at).
Used as the canonical sort/group/filter key across the gallery so
images backfilled with primary_post_id (e.g. via tag_apply phase 4)
surface at their original publish date, not their FC import date.
Images without a Post (or with Post.post_date NULL) fall back to
image_record.created_at and still order coherently against
post-attached ones.
"""
return func.coalesce(Post.post_date, ImageRecord.created_at)
def _outer_join_primary_post(stmt: Select) -> Select:
"""LEFT JOIN Post on ImageRecord.primary_post_id so the COALESCE
above sees Post.post_date when available. Images without a post
survive the join as NULL on the Post side; COALESCE handles it."""
return stmt.outerjoin(Post, Post.id == ImageRecord.primary_post_id)
@dataclass(frozen=True)
class GalleryImage:
id: int
@@ -41,7 +69,9 @@ class GalleryImage:
mime: str
width: int | None
height: int | None
created_at: datetime
created_at: datetime # FC's row-insert time
effective_date: datetime # COALESCE(post.post_date, created_at)
posted_at: datetime | None # post.post_date if known, else None
thumbnail_url: str
artist: dict | None = None
@@ -78,7 +108,7 @@ def _require_single_filter(tag_id, post_id, artist_id) -> None:
def _provenance_clause(post_id, artist_id):
"""Correlated EXISTS clause (NOT a join) so an image with multiple
matching provenance rows is returned exactly once and the
(created_at DESC, id DESC) cursor ordering is unaffected."""
(effective_date DESC, id DESC) cursor ordering is unaffected."""
if post_id is not None:
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
@@ -125,7 +155,9 @@ class GalleryService:
raise ValueError("limit must be between 1 and 200")
_require_single_filter(tag_id, post_id, artist_id)
stmt = select(ImageRecord)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
@@ -138,34 +170,38 @@ class GalleryService:
cur_ts, cur_id = decode_cursor(cursor)
stmt = stmt.where(
or_(
ImageRecord.created_at < cur_ts,
and_(ImageRecord.created_at == cur_ts, ImageRecord.id < cur_id),
eff < cur_ts,
and_(eff == cur_ts, ImageRecord.id < cur_id),
)
)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).scalars().all()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).all()
next_cursor = None
if len(rows) > limit:
last = rows[limit - 1]
next_cursor = encode_cursor(last.created_at, last.id)
last_record, _last_posted_at, last_eff = rows[limit - 1]
next_cursor = encode_cursor(last_eff, last_record.id)
rows = rows[:limit]
artists = await _artists_for(self.session, [r.id for r in rows])
artists = await _artists_for(
self.session, [r[0].id for r in rows]
)
images = [
GalleryImage(
id=r.id,
path=r.path,
sha256=r.sha256,
mime=r.mime,
width=r.width,
height=r.height,
created_at=r.created_at,
thumbnail_url=thumbnail_url(r.sha256, r.mime),
artist=artists.get(r.id),
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.sha256, record.mime),
artist=artists.get(record.id),
)
for r in rows
for record, posted_at, eff_date in rows
]
return GalleryPage(
images=images,
@@ -179,11 +215,13 @@ class GalleryService:
post_id: int | None = None,
artist_id: int | None = None,
) -> list[TimelineBucket]:
year_col = func.date_part("year", ImageRecord.created_at).label("yr")
month_col = func.date_part("month", ImageRecord.created_at).label("mo")
eff = _effective_date_col()
year_col = func.date_part("year", eff).label("yr")
month_col = func.date_part("month", eff).label("mo")
stmt = select(
year_col, month_col, func.count(ImageRecord.id).label("cnt")
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -201,14 +239,17 @@ class GalleryService:
post_id: int | None = None, artist_id: int | None = None,
) -> str | None:
"""Returns a cursor that, when passed to scroll(), positions at the
first image of the given year-month. None if the bucket is empty.
first image of the given year-month (by effective_date, not
created_at). None if the bucket is empty.
"""
from sqlalchemy import extract
stmt = select(ImageRecord).where(
extract("year", ImageRecord.created_at) == year,
extract("month", ImageRecord.created_at) == month,
eff = _effective_date_col()
stmt = select(ImageRecord, eff.label("eff")).where(
extract("year", eff) == year,
extract("month", eff) == month,
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -217,13 +258,14 @@ class GalleryService:
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).scalar_one_or_none()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).first()
if first is None:
return None
record, eff_date = first
# Cursor is exclusive; we encode a cursor with id+1 so the row itself
# is the first result in the next scroll().
return encode_cursor(first.created_at, first.id + 1)
return encode_cursor(eff_date, record.id + 1)
async def get_image_with_tags(self, image_id: int) -> dict | None:
record = await self.session.get(ImageRecord, image_id)
@@ -236,7 +278,23 @@ class GalleryService:
.order_by(Tag.kind.asc(), Tag.name.asc())
)
tags = (await self.session.execute(tag_stmt)).scalars().all()
# Fetch the canonical post.post_date for this image (if any) so
# the modal can show "Posted on <date>" alongside import date.
posted_at = None
if record.primary_post_id is not None:
posted_at = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
neighbors = await self._neighbors(record)
# Direct artist FK — used by the modal's ProvenancePanel as a
# fallback when ImageProvenance is empty (i.e., filesystem-
# imported images without a post-track provenance row). The
# source of truth for richer post-level data is still
# ImageProvenance/Post; this is just the "we at least know who
# made it" line.
artist = None
if record.artist_id is not None:
artist = await self.session.get(Artist, record.artist_id)
return {
"id": record.id,
"path": record.path,
@@ -247,8 +305,13 @@ class GalleryService:
"size_bytes": record.size_bytes,
"integrity_status": record.integrity_status,
"created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.sha256, record.mime),
"image_url": f"/images/{record.path.split('/images/', 1)[-1]}",
"artist": (
{"id": artist.id, "name": artist.name, "slug": artist.slug}
if artist is not None else None
),
"tags": [
{
"id": t.id,
@@ -262,34 +325,41 @@ class GalleryService:
}
async def _neighbors(self, record: ImageRecord) -> dict:
prev_stmt = (
select(ImageRecord.id)
.where(
# Compute the boundary image's effective_date in Python (one query
# below + the SELECT we already have on `record`) and use it for
# the neighbor comparison. Cheaper than re-deriving in SQL via
# correlated subquery.
boundary_eff = record.created_at
if record.primary_post_id is not None:
post_date = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
if post_date is not None:
boundary_eff = post_date
eff = _effective_date_col()
prev_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at > record.created_at,
eff > boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id > record.id,
),
)
)
.order_by(ImageRecord.created_at.asc(), ImageRecord.id.asc())
.limit(1)
)
next_stmt = (
select(ImageRecord.id)
.where(
).order_by(eff.asc(), ImageRecord.id.asc()).limit(1)
next_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at < record.created_at,
eff < boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id < record.id,
),
)
)
.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc())
.limit(1)
)
).order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
prev_id = (await self.session.execute(prev_stmt)).scalar_one_or_none()
next_id = (await self.session.execute(next_stmt)).scalar_one_or_none()
return {"prev_id": prev_id, "next_id": next_id}
@@ -298,9 +368,11 @@ class GalleryService:
def _group_by_year_month(
images: list[GalleryImage],
) -> list[tuple[int, int, list[int]]]:
"""Group by effective_date's year/month so migrated content surfaces
in the publish-date buckets, not the FC-scan-date bucket."""
groups: list[tuple[int, int, list[int]]] = []
for img in images:
y, m = img.created_at.year, img.created_at.month
y, m = img.effective_date.year, img.effective_date.month
if groups and groups[-1][0] == y and groups[-1][1] == m:
groups[-1][2].append(img.id)
else:
+376 -88
View File
@@ -18,6 +18,7 @@ from pathlib import Path
from PIL import Image
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from ..models import (
@@ -29,6 +30,7 @@ from ..models import (
PostAttachment,
Source,
)
from ..utils import safe_probe
from ..utils.paths import derive_subdir, derive_top_level_artist, hash_suffixed_name
from ..utils.phash import compute_phash, find_similar
from ..utils.sidecar import find_sidecar, parse_sidecar
@@ -51,7 +53,14 @@ class SkipReason(StrEnum):
@dataclass(frozen=True)
class ImportResult:
status: str # 'imported'|'skipped'|'failed'|'superseded'|'attached'
# 'imported' — new ImageRecord row created
# 'superseded' — existing ImageRecord row got the new file (larger) + sidecar
# 'attached' — non-media saved as PostAttachment
# 'refreshed' — deep scan re-applied sidecar / filled NULL phash / NULL
# artist on an already-imported row (no new ImageRecord)
# 'skipped' — no work done (true duplicate, too small, etc.)
# 'failed' — pipeline error
status: str
image_id: int | None = None
skip_reason: SkipReason | None = None
error: str | None = None
@@ -71,6 +80,31 @@ def is_video(path: Path) -> bool:
return path.suffix.lower() in VIDEO_EXTS
def _safe_ext(path: Path) -> str:
"""Conservatively extract a file extension for PostAttachment.ext
(varchar(32)).
gallery-dl produces some filenames with URL-encoded query-string
artifacts embedded into the basename (e.g.
`79507046_media_..._https___www.patreon.com_media-u_Z0FBQUFBQm5q...`).
`Path.suffix` finds the LAST dot and returns everything after, which
in those cases yields a 50+ char "extension" of mostly base64-ish
junk. That blows the column. Operator-flagged 2026-05-25.
Real extensions are short and alphanumeric. We accept anything ≤ 16
chars where every post-dot character is alphanumeric; anything else
means the input wasn't a real extension and we return the empty
string. ext is nullable-ish (empty string still satisfies NOT NULL)
and consumers should treat "" as "no known extension".
"""
suffix = path.suffix.lower()
if not suffix or len(suffix) > 16:
return ""
if not all(c.isalnum() for c in suffix[1:]):
return ""
return suffix
def _mime_for(path: Path) -> str:
suffix = path.suffix.lower()
image_mimes = {
@@ -126,6 +160,179 @@ class Importer:
self.settings = settings
self.deep = deep
self.attachments = AttachmentStore(images_root)
# phash near-dup candidate cache. Archive imports call _import_media
# per-member; without this cache the per-member SELECT *FROM
# image_record WHERE phash IS NOT NULL fetch repeats N times and a
# large library × many-member archive blew past soft_time_limit
# (300s) — operator-flagged 2026-05-25. Loaded lazily on first
# need, appended to on every imported/superseded outcome, never
# invalidated mid-Importer (Importer instances are per-task /
# per-archive-import so cross-instance staleness is harmless).
self._phash_candidates: list[tuple] | None = None
def _phash_candidates_cache(self) -> list[tuple]:
"""Cached `(phash, width, height, id)` rows from image_record.
Loaded on first call, appended-to on subsequent imported/
superseded outcomes. Soft-timeout pattern: an archive with N
members + a library of M existing rows used to do N × M-row
fetches (operator-flagged 2026-05-25); now it's exactly one.
The per-task lifecycle of Importer (instantiated fresh by
import_media_file) bounds the cache's staleness window: cross-
process changes (other workers importing concurrently) won't
be reflected, but that's the same race the un-cached version
had — `find_similar` is best-effort anyway."""
if self._phash_candidates is None:
rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
self._phash_candidates = [
(r.phash, r.width or 0, r.height or 0, r.id) for r in rows
]
return self._phash_candidates
def _phash_cache_append(self, phash, width, height, image_id) -> None:
"""Append a freshly-imported row to the cache so subsequent
members of the same archive can match against it."""
if self._phash_candidates is not None and phash is not None:
self._phash_candidates.append(
(phash, width or 0, height or 0, image_id)
)
def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> Source:
"""Race-safe find-or-create on `source` keyed by
(artist_id, platform, url) — the same key as the
`uq_source_artist_platform_url` constraint.
Two concurrent workers processing different files in the same
post can both find no existing Source row then both INSERT,
which trips the unique constraint and poisons the session with
`psycopg.errors.UniqueViolation`. Operator-flagged 2026-05-26.
Pattern: select; if absent, open a savepoint and INSERT.
On IntegrityError, roll the savepoint back (NOT the outer
transaction, which would lose the surrounding scan's progress)
and re-select — the concurrent op just created the row we
wanted, so the second select will find it.
"""
existing = self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Source(artist_id=artist_id, platform=platform, url=url)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one()
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Filesystem-import sidecar Source resolver.
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). The filesystem importer
used to call _find_or_create_source(url=sd.post_url), which created
one Source row per post URL — 100s of junk Sources per artist, all
with enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26.
New behaviour: if any Source row exists for (artist_id, platform),
reuse it regardless of its URL — the artist's real subscription Source
(created by the downloader / extension / UI) is the canonical
attachment point for filesystem-imported posts. If none exists, create
ONE synthetic anchor with url='sidecar:<platform>:<artist_slug>' and
enabled=False (so the subscription checker doesn't poll it).
"""
existing = self.session.execute(
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one_or_none()
if existing is not None:
return existing
synthetic_url = f"sidecar:{platform}:{artist_slug}"
sp = self.session.begin_nested()
try:
row = Source(
artist_id=artist_id,
platform=platform,
url=synthetic_url,
enabled=False,
)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
existing = self.session.execute(
select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Post(source_id=source_id, external_post_id=external_post_id)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one()
def import_one(self, source: Path) -> ImportResult:
"""Dispatch by kind. Media → normal pipeline. Archive → extract
@@ -165,30 +372,13 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
url = sd.post_url or f"sidecar:{platform}"
src = self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if src is None:
src = Source(artist_id=artist.id, platform=platform, url=url)
self.session.add(src)
self.session.flush()
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self.session.execute(
select(Post).where(
Post.source_id == src.id,
Post.external_post_id == epid,
)
).scalar_one_or_none()
if post is None:
post = Post(source_id=src.id, external_post_id=epid)
self.session.add(post)
self.session.flush()
return post
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
)
def _capture_attachment(
self, source: Path, *, post: Post | None = None,
@@ -209,7 +399,7 @@ class Importer:
sha256=sha,
path=stored,
original_filename=source.name,
ext=source.suffix.lower(),
ext=_safe_ext(source),
mime=_mime_for(source),
size_bytes=source.stat().st_size,
))
@@ -218,6 +408,29 @@ class Importer:
return ImportResult(status="attached")
def _import_archive(self, source: Path) -> ImportResult:
# Layer-3 isolation: bomb-size guard + integrity test in a
# spawned child BEFORE extracting in this process. A
# decompression bomb or a native-lib crash on a malformed
# archive is contained to the child; we reject the file cleanly
# instead of OOMing/segfaulting the import worker. extract_archive
# is already fail-soft for plain exceptions, so this only adds
# the hard-crash protection.
probe = safe_probe.probe_archive(source)
if not probe.ok:
if probe.crashed:
return ImportResult(
status="failed",
error=f"archive probe crashed/timed out: {probe.reason}",
)
# Clean rejection (bomb cap exceeded, integrity mismatch):
# still preserve the archive file itself as an attachment so
# nothing silently vanishes, matching extract_archive's
# fail-soft contract.
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
self._capture_attachment(source, post=post, artist=artist, resolved=True)
return ImportResult(status="attached")
artist = self._resolve_artist(source)
post = self._post_for_sidecar(source, artist)
member_ids: list[int] = []
@@ -257,7 +470,25 @@ class Importer:
# Compute file dimensions (images only) and apply filters.
width = height = None
has_alpha = False
if not is_video(source):
if is_video(source):
# Layer-3 isolation: validate the container via ffprobe (a
# separate process) before the rest of the pipeline touches
# it. A corrupt video that would crash a decoder is rejected
# cleanly here, and we capture width/height for free (the
# importer didn't previously record video dimensions).
probe = safe_probe.probe_video(source)
if not probe.ok:
if probe.crashed:
return ImportResult(
status="failed",
error=f"video probe crashed/timed out: {probe.reason}",
)
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=probe.reason,
)
width, height = probe.width, probe.height
else:
try:
with Image.open(source) as im:
im.verify()
@@ -280,7 +511,18 @@ class Importer:
)
if self.settings.skip_transparent and has_alpha:
pct = self._transparency_pct(source)
try:
pct = self._transparency_pct(source)
except OSError as exc:
# PIL.verify() at line 263 only validates header structure;
# truncated/corrupt pixel data only surfaces when load()
# actually decodes (here via getchannel('A')). Convert to
# invalid_image skip so the Celery autoretry loop doesn't
# bounce the same broken file forever.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during transparency check: {exc}",
)
if pct >= self.settings.transparency_threshold:
return ImportResult(
status="skipped", skip_reason=SkipReason.too_transparent,
@@ -302,21 +544,18 @@ class Importer:
# Perceptual near-dup (images only; videos keep phash NULL).
phash = None
if not is_video(source):
with Image.open(source) as im:
phash = compute_phash(im)
try:
with Image.open(source) as im:
phash = compute_phash(im)
except OSError as exc:
# Same rationale as the transparency-check guard above:
# broken-pixel-data files pass verify() but blow up here.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during phash compute: {exc}",
)
if phash is not None:
cand_rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
candidates = [
(c.phash, c.width or 0, c.height or 0, c.id)
for c in cand_rows
]
candidates = self._phash_candidates_cache()
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
@@ -350,6 +589,7 @@ class Importer:
)
self.session.add(record)
self.session.flush()
self._phash_cache_append(phash, width, height, record.id)
# Folder→artist (anchored to attribution_path).
artist = None
@@ -373,13 +613,27 @@ class Importer:
) -> ImportResult:
"""Deep scan: backfill phash/provenance/artist on an
already-imported record. METADATA ONLY — never re-runs the pHash
near-dup / supersede path. NULL-only, idempotent."""
near-dup / supersede path. NULL-only on phash/artist, additive on
sidecar Post/Source/ImageProvenance (via _apply_sidecar).
Idempotent: a second deep-scan over the same file finds nothing
to refresh and is a no-op.
Returns status="refreshed" so the UI can surface the work done
instead of the prior misleading "skipped/duplicate_hash" reading.
Operator-flagged 2026-05-25 — IR has had this; FC inherited it
as a no-op skip during the original port and the UI showed deep
scan as "completed with no changes" even when sidecar metadata
actually got re-applied to N existing rows.
"""
if existing.phash is None and not is_video(source):
try:
with Image.open(source) as im:
ph = compute_phash(im)
if ph is not None:
existing.phash = ph
# Promoted from NULL to non-NULL → cache is now stale
# (this row would newly qualify for the candidates set).
self._phash_candidates = None
except Exception as exc:
log.warning("deep rephash failed for %s: %s", source, exc)
@@ -392,10 +646,7 @@ class Importer:
self._apply_sidecar(existing, attribution_path, artist)
self.session.commit()
return ImportResult(
status="skipped", skip_reason=SkipReason.duplicate_hash,
image_id=existing.id, error="deep: re-derived",
)
return ImportResult(status="refreshed", image_id=existing.id)
def attach_in_place(
self,
@@ -479,18 +730,7 @@ class Importer:
except Exception:
phash = None
if phash is not None:
cand_rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
candidates = [
(c.phash, c.width or 0, c.height or 0, c.id)
for c in cand_rows
]
candidates = self._phash_candidates_cache()
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
@@ -525,6 +765,7 @@ class Importer:
record.artist_id = artist.id
self.session.add(record)
self.session.flush()
self._phash_cache_append(phash, width, height, record.id)
# Sidecar provenance (best-effort). When `source` is passed, link
# the post to that subscription Source instead of creating a new
@@ -619,30 +860,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
url = sd.post_url or f"sidecar:{platform}"
src = self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if src is None:
src = Source(artist_id=artist.id, platform=platform, url=url)
self.session.add(src)
self.session.flush()
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self.session.execute(
select(Post).where(
Post.source_id == src.id,
Post.external_post_id == epid,
)
).scalar_one_or_none()
if post is None:
post = Post(source_id=src.id, external_post_id=epid)
self.session.add(post)
self.session.flush()
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
)
if sd.post_url is not None:
post.post_url = sd.post_url
if sd.post_title is not None:
@@ -655,6 +881,15 @@ class Importer:
post.attachment_count = sd.attachment_count
post.raw_metadata = sd.raw
# Race-safe (image_record_id, post_id) upsert — mirrors the
# _find_or_create_source/post savepoint pattern. The plain
# SELECT-then-INSERT pattern lost a race when two workers ran
# _apply_sidecar on the same (image, post) pair (e.g. the 5-min
# recovery sweep re-enqueued a still-running long import), planting
# duplicates that then broke .scalar_one_or_none() on every later
# deep-scan rederive (MultipleResultsFound). Alembic 0021 adds the
# uq_image_provenance_image_post UNIQUE so this savepoint actually
# trips on collision.
exists = self.session.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == record.id,
@@ -662,14 +897,20 @@ class Importer:
)
).scalar_one_or_none()
if exists is None:
self.session.add(
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
captured_metadata=sd.raw,
sp = self.session.begin_nested()
try:
self.session.add(
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
captured_metadata=sd.raw,
)
)
)
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
if record.primary_post_id is None:
record.primary_post_id = post.id
self.session.flush()
@@ -703,6 +944,15 @@ class Importer:
row id (so tags/series/curation stay attached). ML is cleared so
the import task re-derives it on the new pixels.
After the file swap, the new file's adjacent gallery-dl sidecar
(if any) is applied via _apply_sidecar — operator-flagged
2026-05-25: scanning a GS download dir with smaller IR-migrated
images on the receiving end used to swap files but lose the GS
sidecar's post metadata entirely. _apply_sidecar is additive
(find-or-create Post / Source / ImageProvenance, NULL-only
primary_post_id update) so any pre-existing Post linkage
survives untouched.
If `new_path` is provided, `source` is assumed to ALREADY be at
that path (FC-3c attach_in_place case) — skip the copy step.
Otherwise the file is copied via _copy_to_library."""
@@ -731,6 +981,26 @@ class Importer:
# created_at intentionally preserved; updated_at auto-bumps.
self.session.flush()
self.session.commit()
# The phash candidate cache (used to avoid N+1 selects during
# archive imports) is now stale for `existing.id` — the row's
# phash/dimensions changed. Invalidate; the next call re-fetches.
self._phash_candidates = None
# Sidecar enrichment from the new (larger) file's location.
# _apply_sidecar resolves artist from the sidecar itself if the
# existing row has none, and is internally guarded against
# missing-or-malformed sidecars (silent return).
try:
self._apply_sidecar(existing, source, None)
except Exception as exc:
# Don't unwind the supersede DB swap if sidecar parsing
# blows up unexpectedly — the file replacement is the
# critical operation, sidecar is enrichment.
log.warning(
"sidecar enrichment failed during supersede of "
"image_record.id=%s from %s: %s",
existing.id, source, exc,
)
for stale in (old_path, old_thumb):
if not stale or stale == str(dest):
@@ -746,8 +1016,26 @@ class Importer:
pass
def _transparency_pct(self, source: Path) -> float:
"""Fraction of fully-transparent pixels in the image. 0.0 if no alpha."""
"""Fraction of fully-transparent pixels in the image. 0.0 if no alpha.
For animated formats (multi-frame WebP / GIF / APNG), short-circuit
to 0.0 instead of decoding every frame. PIL's `getchannel("A")`
forces a full decode of all frames in an animated image, which for
a large animated WebP takes 5+ minutes and blows past the Celery
soft+hard time limits (300s/360s → SIGKILL). Operator-flagged
2026-05-26. Transparency analysis on a multi-frame image isn't
meaningful for art-curation purposes anyway — different frames
have different alpha — so the existing too_transparent skip rule
is bypassed entirely for animated content.
"""
with Image.open(source) as im:
if getattr(im, "is_animated", False):
log.info(
"skipping transparency check for animated image %s "
"(n_frames=%d) — avoids multi-frame decode timeout",
source, getattr(im, "n_frames", 0),
)
return 0.0
if im.mode not in ("RGBA", "LA") and not (
im.mode == "P" and "transparency" in im.info
):
+5 -1
View File
@@ -1,6 +1,10 @@
"""FC-5 migration tooling.
One module per concern (backup/rollback/gs/ir/overlap/ml_queue/verify).
One module per concern (gs/ir/overlap/ml_queue/verify/cleanup).
Each migrator returns a counts dict; the run_migration task wires
that dict into MigrationRun.counts so the UI polling shows progress.
backup + rollback were retired in FC-3h (2026-05-24); first-class
backup lives at backend/app/services/backup_service.py and exposes
its own /api/system/backup/* surface.
"""
-146
View File
@@ -1,146 +0,0 @@
"""pg_dump + tar.zst-based backup, restorable via pair of subprocess calls.
Backups live under <images_root>/_backups/. Each backup is two files
(SQL + tarball) plus a manifest JSON. Tagged backups (e.g. tag='pre_migration')
are how rollback.py finds the most recent restorable snapshot.
"""
from __future__ import annotations
import json
import shutil
import subprocess
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
_BACKUPS_DIRNAME = "_backups"
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy driver suffix so pg_dump/psql accept the URL.
SQLAlchemy uses URLs like `postgresql+psycopg://...` or
`postgresql+asyncpg://...`. libpq tools (pg_dump, psql) only know
the plain `postgresql://` scheme.
"""
for driver in ("postgresql+psycopg", "postgresql+asyncpg", "postgresql+psycopg2"):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path | None = None) -> Path:
# Overridable for tests via monkeypatch.
root = images_root if images_root is not None else Path("/images")
p = root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
_DEFAULT_SUBPROCESS_TIMEOUT_S = 30 * 60 # 30 minutes
def _run_subprocess(cmd: list[str], **kwargs: Any):
# Overridable for tests via monkeypatch. Hard wall-clock timeout
# guards against pg_dump / tar / zstd hangs on NFS — without it the
# task pretends to be 'running' forever (operator hit this 2026-05-
# 23 with two backups stuck in MigrationRun). On timeout
# subprocess.run raises TimeoutExpired which the caller surfaces as
# a task error.
return subprocess.run(
cmd,
capture_output=True,
check=True,
timeout=_DEFAULT_SUBPROCESS_TIMEOUT_S,
**{k: v for k, v in kwargs.items() if not k.startswith("_")},
)
def create_backup(
*, db_url: str, images_root: Path, tag: str = "manual",
) -> dict:
"""Create a backup: pg_dump SQL + tar.zst of images.
Returns a manifest dict. Writes <ts>.sql, <ts>.tar.zst, <ts>.json into
<images_root>/_backups/.
"""
ts = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_{ts}.sql"
tar_path = out_dir / f"fc_{ts}.tar.zst"
manifest_path = out_dir / f"fc_{ts}.json"
_run_subprocess(
["pg_dump", "--no-owner", "--no-acl", "-f", str(sql_path), _libpq_url(db_url)],
_test_ts=ts,
)
_run_subprocess(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
_test_ts=ts,
)
manifest = {
"backup_id": ts,
"tag": tag,
"created_at": datetime.now(UTC).isoformat(),
"sql_path": str(sql_path),
"tar_path": str(tar_path),
}
manifest_path.write_text(json.dumps(manifest, indent=2))
return manifest
def list_backups(images_root: Path) -> list[dict]:
out_dir = _backups_dir(images_root)
items = []
for mf in sorted(out_dir.glob("fc_*.json"), reverse=True):
try:
items.append(json.loads(mf.read_text()))
except Exception:
continue
return items
def find_latest_backup(images_root: Path, *, tag: str) -> dict | None:
for mf in list_backups(images_root):
if mf.get("tag") == tag:
return mf
return None
def restore_backup(
*, manifest: dict, db_url: str, images_root: Path,
) -> dict:
"""Restore from a backup manifest.
1. Replay the .sql via psql.
2. Wipe /images/ contents (except _backups/, which holds the file we're using).
3. Untar the .tar.zst into /images/.
"""
sql_path = Path(manifest["sql_path"])
tar_path = Path(manifest["tar_path"])
_run_subprocess(
["psql", "-d", _libpq_url(db_url), "-f", str(sql_path)],
)
# Wipe everything in images_root EXCEPT _backups/ (we'd delete the backup
# we're restoring from!).
for entry in images_root.iterdir():
if entry.name == _BACKUPS_DIRNAME:
continue
if entry.is_dir():
shutil.rmtree(entry)
else:
entry.unlink()
_run_subprocess(
["tar", "--zstd", "-xf", str(tar_path), "-C", str(images_root.parent)],
)
return {"restored_from": manifest["backup_id"]}
+7 -2
View File
@@ -70,7 +70,7 @@ async def migrate_async(
"""
if data.get("source_app") != "imagerepo":
raise ValueError("export source_app must be 'imagerepo'")
if data.get("schema_version") != 1:
if data.get("schema_version") not in (1, 2):
raise ValueError(f"unsupported schema_version: {data.get('schema_version')}")
counts = _zero_counts()
@@ -126,15 +126,20 @@ async def migrate_async(
await db.commit()
# Phase 2: write the per-image manifest for tag_apply.py to consume later.
# schema_version 2 (added 2026-05-24) carries `image_posts` for
# Post + Source + ImageProvenance restore; schema 1 manifests
# without it stay valid (tag_apply treats the missing field as []).
manifest = {
"schema_version": 1,
"schema_version": data.get("schema_version", 1),
"image_artist_assignments": data.get("image_artist_assignments", []),
"image_tag_associations": data.get("image_tag_associations", []),
"series_pages": data.get("series_pages", []),
"image_posts": data.get("image_posts", []),
}
counts["rows_processed"] += len(manifest["image_artist_assignments"])
counts["rows_processed"] += len(manifest["image_tag_associations"])
counts["rows_processed"] += len(manifest["series_pages"])
counts["rows_processed"] += len(manifest["image_posts"])
if not dry_run:
manifest_path(images_root).write_text(json.dumps(manifest, indent=2))
@@ -1,21 +0,0 @@
"""Restore from the most recent 'pre_migration'-tagged backup."""
from __future__ import annotations
from pathlib import Path
from . import backup as backup_mod
class NoBackupFoundError(Exception):
"""Raised when rollback() is called with no pre_migration backup on disk."""
def rollback_to_pre_migration(*, db_url: str, images_root: Path) -> dict:
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
raise NoBackupFoundError(
"no pre_migration-tagged backup found under <images_root>/_backups/"
)
return backup_mod.restore_backup(
manifest=manifest, db_url=db_url, images_root=images_root,
)
+197 -1
View File
@@ -7,6 +7,7 @@ FC's filesystem scan over the mounted IR images dir).
- image_artist_assignments → ImageRecord.artist_id (find_or_create Artist by slug).
- image_tag_associations → image_tag insert (idempotent).
- series_pages → series_page insert (idempotent on image_id unique).
- image_posts (schema v2) → Source + Post + ImageProvenance restore.
Unmatched sha256s are logged into the result's `unmatched` list so the
Celery task can drop them into MigrationRun.metadata for the operator
@@ -15,15 +16,149 @@ to inspect.
from __future__ import annotations
import json
from datetime import datetime
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Artist, ImageRecord, SeriesPage, Tag, TagKind, image_tag
from ...models import (
Artist,
ImageProvenance,
ImageRecord,
Post,
SeriesPage,
Source,
Tag,
TagKind,
image_tag,
)
from ...utils.slug import slugify
from .ir_ingest import manifest_path
# Per-platform artist-profile URL — used as Source.url when restoring
# IR PostMetadata into FC. Must cover every platform that
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS
# recognizes; an entry missing here silently drops ALL PostMetadata for
# that platform during phase 4 (operator hit this 2026-05-25:
# DeviantArt + Pixiv posts in the IR migration produced empty
# ImageProvenance because they fell through this table).
#
# Pixiv caveat: the real profile URL takes a numeric user_id
# (https://www.pixiv.net/users/12345), but IR's PostMetadata.artist
# stores the display name not the id. We use the slugified name here
# so we preserve the artist→post→image linkage; the resulting Source.url
# won't resolve in a browser and the operator may want to manually fix
# it via Settings → Subscriptions once the migration lands.
_PLATFORM_PROFILE_URL = {
"patreon": "https://www.patreon.com/{slug}",
"subscribestar": "https://www.subscribestar.com/{slug}",
"hentaifoundry": "https://www.hentai-foundry.com/user/{slug}",
"deviantart": "https://www.deviantart.com/{slug}",
"pixiv": "https://www.pixiv.net/users/{slug}",
}
def _profile_url(platform: str, artist_slug: str) -> str | None:
fmt = _PLATFORM_PROFILE_URL.get(platform)
return fmt.format(slug=artist_slug) if fmt else None
async def _find_or_create_source(
db: AsyncSession, *, artist_id: int, platform: str, url: str, dry_run: bool,
) -> int | None:
existing = (await db.execute(
select(Source.id).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one_or_none()
if existing is not None:
return existing
if dry_run:
return None
s = Source(artist_id=artist_id, platform=platform, url=url, enabled=False)
db.add(s)
await db.flush()
return s.id
async def _find_or_create_post(
db: AsyncSession, *,
source_id: int, external_post_id: str,
title: str | None, description: str | None, post_url: str | None,
post_date_iso: str | None, attachment_count: int, dry_run: bool,
) -> int | None:
existing = (await db.execute(
select(Post.id).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
)).scalar_one_or_none()
if existing is not None:
return existing
if dry_run:
return None
post_date = None
if post_date_iso:
post_date = datetime.fromisoformat(post_date_iso)
p = Post(
source_id=source_id,
external_post_id=external_post_id,
post_title=title,
description=description,
post_url=post_url,
post_date=post_date,
attachment_count=attachment_count,
raw_metadata={"migrated_from": "imagerepo"},
)
db.add(p)
await db.flush()
return p.id
async def _ensure_provenance(
db: AsyncSession, *,
image_id: int, post_id: int, source_id: int, dry_run: bool,
) -> bool:
"""Returns True if a new ImageProvenance row was inserted.
Also sets ImageRecord.primary_post_id to this post if the image
doesn't already have one — preserves any primary_post_id already
assigned at download time by the importer (don't clobber). This is
the linkage gallery_service.py uses to surface Post.post_date as
the image's effective date for sort/group/jump/neighbor nav.
"""
existing = (await db.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == image_id,
ImageProvenance.post_id == post_id,
ImageProvenance.source_id == source_id,
)
)).scalar_one_or_none()
# Whether-or-not the provenance row already exists, ensure the
# image's primary_post_id is set so the gallery date-coalesce works.
# Idempotent: only writes when currently NULL.
if not dry_run:
await db.execute(
ImageRecord.__table__.update()
.where(ImageRecord.id == image_id)
.where(ImageRecord.primary_post_id.is_(None))
.values(primary_post_id=post_id)
)
if existing is not None:
return False
if dry_run:
return True
db.add(ImageProvenance(
image_record_id=image_id, post_id=post_id, source_id=source_id,
))
await db.flush()
return True
def _zero_counts() -> dict:
return {
@@ -167,6 +302,67 @@ async def apply_async(
))
counts["rows_inserted"] += 1
# 4. Image posts (schema v2) → Source + Post + ImageProvenance.
# Restores IR PostMetadata as FC's downloader-track provenance,
# so the modal's ProvenancePanel surfaces title/description/
# source URL/publish date the same way it does for live
# gallery-dl downloads.
for entry in manifest.get("image_posts", []):
counts["rows_processed"] += 1
platform = entry.get("platform")
artist_name = entry.get("artist")
if not platform or not artist_name:
counts["rows_skipped"] += 1
continue
aid = await _ensure_artist_id(db, artist_name, dry_run)
if aid is None:
counts["rows_skipped"] += 1
continue
url = _profile_url(platform, slugify(artist_name))
if url is None:
counts["rows_skipped"] += 1
continue
source_id = await _find_or_create_source(
db, artist_id=aid, platform=platform, url=url, dry_run=dry_run,
)
if source_id is None:
counts["rows_skipped"] += 1
continue
post_id = await _find_or_create_post(
db, source_id=source_id,
external_post_id=entry.get("post_id") or "",
title=entry.get("title"),
description=entry.get("description"),
post_url=entry.get("source_url"),
post_date_iso=entry.get("published_at"),
attachment_count=entry.get("attachment_count") or 0,
dry_run=dry_run,
)
if post_id is None:
counts["rows_skipped"] += 1
continue
for sha in entry.get("image_sha256s", []):
img_id = await _sha_to_image_id(db, sha)
if img_id is None:
unmatched.append({
"kind": "post", "sha256": sha,
"post_id": entry.get("post_id"),
})
continue
inserted = await _ensure_provenance(
db, image_id=img_id, post_id=post_id,
source_id=source_id, dry_run=dry_run,
)
if inserted:
counts["rows_inserted"] += 1
else:
counts["rows_skipped"] += 1
if not dry_run:
await db.commit()
return {"counts": counts, "unmatched": unmatched}
+13 -2
View File
@@ -34,10 +34,21 @@ class Embedder:
if self._model is not None:
return
import torch
from transformers import AutoModel, AutoProcessor
from transformers import AutoModel, SiglipImageProcessor
self._torch = torch
self._processor = AutoProcessor.from_pretrained(str(self._model_dir))
# FC's embedder only does IMAGE inference — never text. AutoProcessor
# loads the full processor including SiglipTokenizer, which requires
# the sentencepiece library at import time even if we never call it.
# SiglipImageProcessor loads ONLY preprocessor_config.json (image
# side) and skips the tokenizer config entirely. Operator hit the
# ImportError 2026-05-25 once the ml-worker started actually running
# tag_and_embed; switching to the image-only loader avoids the
# tokenizer dep without adding ~30 MB of unused C++ build to the
# lean ml-worker image.
self._processor = SiglipImageProcessor.from_pretrained(
str(self._model_dir)
)
self._model = AutoModel.from_pretrained(str(self._model_dir))
self._model.eval()
+82 -44
View File
@@ -4,12 +4,14 @@ CPU-only, single-image at a time. Loaded lazily inside the ml-worker
process; NOT thread-safe — the ml queue worker must run --concurrency=1
(set by the FC-1 entrypoint).
Camie's selected_tags.csv columns: tag_id,name,category,count
where category is a string: general|character|copyright|artist|meta|rating|year
(unlike WD14's integer Danbooru category ids).
v2 layout reference: HuggingFace Camais03/camie-tagger-v2 root has
camie-tagger-v2.onnx (789 MB) + camie-tagger-v2-metadata.json (7.77 MB)
+ config.json. Tags ship as nested JSON, not CSV. Preprocessing and
output handling follow the published onnx_inference.py reference:
ImageNet normalize, NCHW layout, sigmoid on refined logits (output[1]).
"""
import csv
import json
import os
from dataclasses import dataclass
from pathlib import Path
@@ -28,6 +30,8 @@ ImageFile.LOAD_TRUNCATED_IMAGES = True
MODEL_NAME = os.environ.get("CAMIE_MODEL_NAME", "camie-tagger-v2")
_MODEL_DIR = Path(os.environ.get("ML_MODEL_DIR", "/models")) / "camie"
_MODEL_FILE = f"{MODEL_NAME}.onnx"
_METADATA_FILE = f"{MODEL_NAME}-metadata.json"
# Below this confidence, predictions aren't stored (keeps the JSON compact).
STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
@@ -39,6 +43,12 @@ STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
# stored at STORE_FLOOR but artist never surfaces.
SURFACED_CATEGORIES = {"character", "copyright", "general"}
# ImageNet preprocessing constants (per Camie v2 onnx_inference.py).
_IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
_IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
# Square-pad color ≈ ImageNet mean × 255 (matches reference inference).
_PAD_COLOR = (124, 116, 104)
@dataclass(frozen=True)
class TagPrediction:
@@ -51,34 +61,48 @@ class Tagger:
def __init__(self, model_dir: Path | None = None):
self._model_dir = model_dir or _MODEL_DIR
self._session = None # onnxruntime.InferenceSession once load()ed
self._tag_meta: list[dict] | None = None
self._tag_names: list[str] | None = None
self._tag_categories: list[str] | None = None
self._input_name: str | None = None
self._output_name: str | None = None
self._input_size: int = 448
self._input_size: int = 512
def load(self) -> None:
if self._session is not None:
return
model_path = self._model_dir / "model.onnx"
tags_path = self._model_dir / "selected_tags.csv"
model_path = self._model_dir / _MODEL_FILE
meta_path = self._model_dir / _METADATA_FILE
if not model_path.is_file():
raise RuntimeError(
f"Camie model.onnx missing at {model_path}. "
f"Camie {_MODEL_FILE} missing at {model_path}. "
f"Populate /models via the ml-worker downloader."
)
if not tags_path.is_file():
if not meta_path.is_file():
raise RuntimeError(
f"Camie selected_tags.csv missing at {tags_path}. "
f"Camie {_METADATA_FILE} missing at {meta_path}. "
f"Populate /models via the ml-worker downloader."
)
tag_meta: list[dict] = []
with open(tags_path, newline="") as f:
reader = csv.DictReader(f)
for row in reader:
tag_meta.append(
{"name": row["name"], "category": row["category"]}
)
with open(meta_path) as f:
metadata = json.load(f)
# Per Camie v2 onnx_inference.py: idx_to_tag is keyed by str(idx);
# tag_to_category maps tag_name -> category. Project to two parallel
# lists indexed by output position for O(1) lookup in the hot path.
ds = metadata["dataset_info"]
idx_to_tag = ds["tag_mapping"]["idx_to_tag"]
tag_to_category = ds["tag_mapping"]["tag_to_category"]
total = ds["total_tags"]
names: list[str] = []
cats: list[str] = []
for i in range(total):
name = idx_to_tag.get(str(i), f"unknown-{i}")
names.append(name)
cats.append(tag_to_category.get(name, "general"))
# Input size from metadata; fall back to 512 (the v2 default).
self._input_size = int(
metadata.get("model_info", {}).get("img_size", 512)
)
# Lazy import — kept after the file-existence checks so the
# missing-model RuntimeError still fires first in environments
@@ -89,51 +113,65 @@ class Tagger:
str(model_path), providers=["CPUExecutionProvider"]
)
self._input_name = session.get_inputs()[0].name
self._output_name = session.get_outputs()[0].name
input_shape = session.get_inputs()[0].shape
for dim in input_shape:
if isinstance(dim, int) and dim > 1:
self._input_size = dim
break
# Assign sentinels last so a partial load isn't observable.
self._tag_meta = tag_meta
self._tag_names = names
self._tag_categories = cats
self._session = session
def _preprocess(self, image_path: Path) -> np.ndarray:
img = Image.open(image_path)
# Camie handles RGBA natively but we still composite onto white so
# transparency doesn't bias the model (same as IR's WD14 path).
if img.mode != "RGBA":
img = img.convert("RGBA")
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
# Composite RGBA onto neutral so transparency doesn't bias the model.
if img.mode == "RGBA":
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
elif img.mode != "RGB":
img = img.convert("RGB")
# Pad to square with ImageNet-mean color, then bicubic resize.
w, h = img.size
side = max(w, h)
square = Image.new("RGB", (side, side), (255, 255, 255))
square = Image.new("RGB", (side, side), _PAD_COLOR)
square.paste(img, ((side - w) // 2, (side - h) // 2))
square = square.resize(
(self._input_size, self._input_size), Image.BICUBIC
)
arr = np.array(square, dtype=np.float32)
return arr[np.newaxis, :, :, :] # NHWC
arr = np.array(square, dtype=np.float32) / 255.0 # HWC, [0,1]
arr = (arr - _IMAGENET_MEAN) / _IMAGENET_STD # ImageNet normalize
arr = arr.transpose(2, 0, 1) # HWC -> CHW
return arr[np.newaxis, :, :, :] # NCHW
def infer(self, image_path: Path) -> dict[str, TagPrediction]:
"""Run Camie on one image. Returns {name: TagPrediction}, only
entries with confidence >= STORE_FLOOR (across all categories —
the suggestion service does category filtering later)."""
"""Run Camie v2 on one image. Returns {name: TagPrediction} with
confidence >= STORE_FLOOR (across all categories — the suggestion
service does category filtering later).
v2 emits multiple outputs; we use the refined predictions
(output[1] per onnx_inference.py). Sigmoid is applied to raw
logits to produce [0,1] confidence scores.
"""
self.load()
x = self._preprocess(image_path)
out = self._session.run([self._output_name], {self._input_name: x})[0][0]
outputs = self._session.run(None, {self._input_name: x})
# Refined predictions if present (v2 emits initial + refined),
# fall back to initial for single-output forks.
logits = outputs[1] if len(outputs) > 1 else outputs[0]
# Squeeze batch dim, apply sigmoid.
probs = 1.0 / (1.0 + np.exp(-logits[0]))
results: dict[str, TagPrediction] = {}
for idx, score in enumerate(out):
names = self._tag_names
cats = self._tag_categories
for idx, score in enumerate(probs):
conf = float(score)
if conf < STORE_FLOOR:
continue
meta = self._tag_meta[idx]
results[meta["name"]] = TagPrediction(
name=meta["name"], category=meta["category"], confidence=conf
if idx >= len(names):
# Output longer than metadata declared — shouldn't happen but
# don't crash the import pipeline if v2 metadata desynchronizes.
continue
results[names[idx]] = TagPrediction(
name=names[idx], category=cats[idx], confidence=conf
)
return results
-140
View File
@@ -1,140 +0,0 @@
"""FC-3b platforms registry — the single source of truth for what
FabledCurator supports.
Lifted from GallerySubscriber's
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
URL patterns match GS exactly so the existing browser extension
hits FC unmodified.
"""
from dataclasses import dataclass
from typing import Literal
@dataclass(frozen=True)
class PlatformInfo:
key: str
name: str
description: str
auth_type: Literal["cookies", "token"]
requires_auth: bool
url_pattern: str
url_examples: list[str]
default_config: dict
notes: str | None = None
# Common defaults used across most platforms; embedded per-platform
# below so per-platform overrides remain explicit.
_DEFAULTS = {
"sleep": 3.0,
"sleep_request": 1.5,
"skip_existing": True,
"save_metadata": True,
"timeout": 3600,
}
PLATFORMS: dict[str, PlatformInfo] = {
"patreon": PlatformInfo(
key="patreon",
name="Patreon",
description="Download posts from Patreon creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?patreon\.com/",
url_examples=[
"https://www.patreon.com/example_artist",
"https://www.patreon.com/user?u=12345678",
],
default_config={**_DEFAULTS, "content_types": ["images", "attachments"]},
),
"subscribestar": PlatformInfo(
key="subscribestar",
name="SubscribeStar",
description="Download posts from SubscribeStar creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?subscribestar\.(com|adult)/",
url_examples=[
"https://subscribestar.adult/example_artist",
"https://www.subscribestar.com/example_artist",
],
default_config={**_DEFAULTS, "content_types": ["all"]},
),
"hentaifoundry": PlatformInfo(
key="hentaifoundry",
name="Hentai Foundry",
description="Download artwork from Hentai Foundry artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?hentai-foundry\.com/",
url_examples=[
"https://www.hentai-foundry.com/user/example_artist",
"https://www.hentai-foundry.com/pictures/user/example_artist",
],
default_config={**_DEFAULTS, "content_types": ["pictures"]},
),
"discord": PlatformInfo(
key="discord",
name="Discord",
description="Download attachments from Discord channels",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?discord\.com/channels/",
url_examples=["https://discord.com/channels/123456789/987654321"],
default_config={**_DEFAULTS, "content_types": ["all"]},
notes="Requires Discord user token (not bot token).",
),
"pixiv": PlatformInfo(
key="pixiv",
name="Pixiv",
description="Download artwork from Pixiv artists",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?pixiv\.net/",
url_examples=[
"https://www.pixiv.net/users/12345678",
"https://www.pixiv.net/en/users/12345678",
],
default_config={**_DEFAULTS, "content_types": ["all"]},
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
),
"deviantart": PlatformInfo(
key="deviantart",
name="DeviantArt",
description="Download artwork from DeviantArt artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?deviantart\.com/",
url_examples=[
"https://www.deviantart.com/example-artist",
"https://www.deviantart.com/example-artist/gallery",
],
default_config={**_DEFAULTS, "content_types": ["gallery"]},
),
}
def known_platform_keys() -> frozenset[str]:
return frozenset(PLATFORMS.keys())
def auth_type_for(platform: str) -> str | None:
info = PLATFORMS.get(platform)
return info.auth_type if info else None
def to_dict(info: PlatformInfo) -> dict:
return {
"key": info.key,
"name": info.name,
"description": info.description,
"auth_type": info.auth_type,
"requires_auth": info.requires_auth,
"url_pattern": info.url_pattern,
"url_examples": info.url_examples,
"default_config": info.default_config,
"notes": info.notes,
}
@@ -0,0 +1,95 @@
"""FC-3b platforms registry — single source of truth for what
FabledCurator supports + where each platform's quirks live.
Adding a new platform: drop a new module `<platform>.py` next to this
one, declare an `INFO = PlatformInfo(...)`, add the import + entry in
PLATFORMS below. Sidecar parsing, cookie materialization, and
`/api/platforms` pick it up automatically.
Lifted from GallerySubscriber's
~/Nextcloud/Projects/GallerySubscriber/backend/app/api/platforms.py
and ~/.../extension/lib/platforms.js. Six platforms; auth_type and
URL patterns match GS exactly so the existing browser extension
hits FC unmodified.
"""
from .base import (
DEFAULT_DESCRIPTION_KEYS,
DEFAULT_EXTERNAL_POST_ID_KEYS,
PlatformInfo,
)
from .deviantart import INFO as _DEVIANTART
from .discord import INFO as _DISCORD
from .hentaifoundry import INFO as _HENTAIFOUNDRY
from .patreon import INFO as _PATREON
from .pixiv import INFO as _PIXIV
from .subscribestar import INFO as _SUBSCRIBESTAR
PLATFORMS: dict[str, PlatformInfo] = {
info.key: info
for info in (
_PATREON,
_SUBSCRIBESTAR,
_HENTAIFOUNDRY,
_DISCORD,
_PIXIV,
_DEVIANTART,
)
}
def known_platform_keys() -> frozenset[str]:
return frozenset(PLATFORMS.keys())
def auth_type_for(platform: str) -> str | None:
info = PLATFORMS.get(platform)
return info.auth_type if info else None
def to_dict(info: PlatformInfo) -> dict:
"""Serialize a PlatformInfo to a JSON-safe dict for /api/platforms.
Behavioral fields (callables, sidecar-chain overrides) are
intentionally omitted — they aren't useful to API consumers.
"""
return {
"key": info.key,
"name": info.name,
"description": info.description,
"auth_type": info.auth_type,
"requires_auth": info.requires_auth,
"url_pattern": info.url_pattern,
"url_examples": info.url_examples,
"default_config": info.default_config,
"notes": info.notes,
}
def external_post_id_keys_for(platform: str | None) -> tuple[str, ...]:
"""Resolve the external_post_id lookup chain for a given platform,
falling back to the module default when the platform isn't
registered or hasn't overridden the chain."""
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.external_post_id_keys is not None:
return info.external_post_id_keys
return DEFAULT_EXTERNAL_POST_ID_KEYS
def description_keys_for(platform: str | None) -> tuple[str, ...]:
"""Resolve the description body lookup chain for a given platform."""
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.description_keys is not None:
return info.description_keys
return DEFAULT_DESCRIPTION_KEYS
__all__ = [
"PLATFORMS",
"PlatformInfo",
"auth_type_for",
"description_keys_for",
"external_post_id_keys_for",
"known_platform_keys",
"to_dict",
]
+105
View File
@@ -0,0 +1,105 @@
"""PlatformInfo dataclass + shared defaults + small helpers.
Per-platform modules import from here, register their PlatformInfo via
INFO, optionally attaching `derive_post_url` and/or `augment_cookies`
callables for behavior that diverges from gallery-dl's mainline shape
(Patreon).
Adding a new platform: drop a new module under `services/platforms/`,
declare an INFO, and add it to the import list in
`services/platforms/__init__.py`. Sidecar parsing, cookie
materialization, and the /api/platforms response pick it up
automatically.
"""
from collections.abc import Callable
from dataclasses import dataclass
from typing import Literal
# Sidecar parsing defaults. Per-platform PlatformInfo entries can
# override these by setting `external_post_id_keys=` /
# `description_keys=`. Most don't need to — the defaults already cover
# every platform FC supports.
#
# external_post_id chain: `post_id` MUST come before `id` because
# SubscribeStar gallery-dl puts the per-attachment id in `id` and the
# actual post id in `post_id`; picking `id` first fragments
# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
# no `post_id` so `id` still wins for them; HF uses `index`, Discord
# uses `message_id` — all reached via the remaining chain entries.
# (Banked 2026-05-27 during the sidecar audit.)
DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
"post_id", "id", "index", "message_id",
)
# Description body chain: Discord's gallery-dl extractor uses `message`
# (no `content`); appended to the chain so Discord posts surface body
# text.
DEFAULT_DESCRIPTION_KEYS: tuple[str, ...] = (
"content", "description", "caption", "message",
)
@dataclass(frozen=True)
class PlatformInfo:
# --- Identity / metadata ---
key: str
name: str
description: str
auth_type: Literal["cookies", "token"]
requires_auth: bool
url_pattern: str
url_examples: list[str]
default_config: dict
notes: str | None = None
# --- Sidecar parsing overrides ---
# Each is None to mean "use the module default above"; a platform
# only sets one of these when its sidecar shape genuinely differs.
external_post_id_keys: tuple[str, ...] | None = None
description_keys: tuple[str, ...] | None = None
# --- Behavioral hooks ---
# Synthesize a post permalink from sidecar data. Required when
# gallery-dl's `url` field is the file/CDN URL rather than the post
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
# `url` field (patreon, deviantart).
derive_post_url: Callable[[dict], str | None] | None = None
# Post-process the materialized cookies.txt for gallery-dl. Used by
# platforms whose server gates or extractor quirks need synthetic
# cookies the extension can't capture (subscribestar age cookie, HF
# host-only PHPSESSID duplicate). None = no-op.
augment_cookies: Callable[[str], str] | None = None
def str_id_value(v) -> str | None:
"""Coerce a JSON scalar id into a non-empty string, rejecting bool
(Python's bool is an int subclass so `isinstance(True, int)` is
True; without this guard a sidecar with `"id": true` would produce
external_post_id="True")."""
if isinstance(v, bool):
return None
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
def str_field(v) -> str | None:
"""Same idea as str_id_value but for plain string fields (no int
coercion)."""
if isinstance(v, str) and v.strip():
return v.strip()
return None
# Shared gallery-dl invocation defaults. Embedded in each platform's
# default_config (with platform-specific overrides) so per-platform
# choices stay explicit.
GD_DEFAULTS = {
"sleep": 3.0,
"sleep_request": 1.5,
"skip_existing": True,
"save_metadata": True,
"timeout": 3600,
}
@@ -0,0 +1,23 @@
"""DeviantArt — no exercised quirks yet.
No operator-owned DeviantArt archive existed at the 2026-05-27 sidecar
audit, so we don't know yet whether DA's gallery-dl sidecars are
well-behaved or have their own quirks. When DA gets exercised for the
first time, add `derive_post_url` / `augment_cookies` here as needed.
"""
from .base import GD_DEFAULTS, PlatformInfo
INFO = PlatformInfo(
key="deviantart",
name="DeviantArt",
description="Download artwork from DeviantArt artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?deviantart\.com/",
url_examples=[
"https://www.deviantart.com/example-artist",
"https://www.deviantart.com/example-artist/gallery",
],
default_config={**GD_DEFAULTS, "content_types": ["gallery"]},
)
+38
View File
@@ -0,0 +1,38 @@
"""Discord — one quirk + one already-default.
post_url: gallery-dl's `url` is the CDN attachment URL. The "permalink"
for a Discord message uses the (server, channel, message) triple via
`discord.com/channels/<server>/<channel>/<message>`. Note that
permalinks are only resolvable for users in the same server — public
access doesn't work — but the URL is still useful to the operator
in-app.
Description body is in `message` not `content`. That's already covered
by the default description chain in base.py (DEFAULT_DESCRIPTION_KEYS
ends with `message`). No description_keys override needed.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
sid = str_id_value(data.get("server_id"))
cid = str_id_value(data.get("channel_id"))
mid = str_id_value(data.get("message_id"))
if sid and cid and mid:
return f"https://discord.com/channels/{sid}/{cid}/{mid}"
return None
INFO = PlatformInfo(
key="discord",
name="Discord",
description="Download attachments from Discord channels",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?discord\.com/channels/",
url_examples=["https://discord.com/channels/123456789/987654321"],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
notes="Requires Discord user token (not bot token).",
derive_post_url=derive_post_url,
)
@@ -0,0 +1,83 @@
"""HentaiFoundry — two quirks colocated.
1. post_url: HF sidecars omit `url` entirely; `src` is the image URL.
Synthesize the permalink from `user` + `index`
(/pictures/user/<user>/<index>).
2. augment_cookies: gallery-dl's HF extractor checks
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching. The extension's pre-v1.0.5
`cookies.js` aggressively rewrote every captured cookie to the
leading-dot subdomain-wide form (`.hentai-foundry.com`), which fails
the exact lookup even though the cookie IS sent on actual HTTP
requests (RFC 6265 subdomain matching). The extractor falls into
an unauthenticated `?enterAgree=1` HEAD that 401s. Inject host-only
duplicates of PHPSESSID + YII_CSRF_TOKEN so the lookup succeeds.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_field, str_id_value
_HOST_ONLY_NAMES = ("PHPSESSID", "YII_CSRF_TOKEN")
def derive_post_url(data: dict) -> str | None:
user = str_field(data.get("user")) or str_field(data.get("artist"))
idx = str_id_value(data.get("index"))
if user and idx:
return f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
return None
def augment_cookies(netscape: str) -> str:
body = netscape.rstrip("\n")
if not body:
return netscape
lines = body.split("\n")
existing_host_only: set[str] = set()
by_name: dict[str, list[str]] = {}
for raw in lines:
if not raw or raw.startswith("#"):
continue
parts = raw.split("\t")
if len(parts) < 7:
continue
domain, _flag, _path, _secure, _exp, name, _value = parts[:7]
if name not in _HOST_ONLY_NAMES:
continue
if domain == "www.hentai-foundry.com":
existing_host_only.add(name)
elif domain in (".hentai-foundry.com", "hentai-foundry.com"):
by_name.setdefault(name, []).append(raw)
appended: list[str] = []
for name in _HOST_ONLY_NAMES:
if name in existing_host_only or name not in by_name:
continue
# Duplicate the first subdomain-wide line as host-only on
# www.hentai-foundry.com. Same value + expiry; flag=FALSE marks
# the entry host-only in netscape format.
parts = by_name[name][0].split("\t")
parts[0] = "www.hentai-foundry.com"
parts[1] = "FALSE"
appended.append("\t".join(parts[:7]))
if not appended:
return netscape
return body + "\n" + "\n".join(appended) + "\n"
INFO = PlatformInfo(
key="hentaifoundry",
name="Hentai Foundry",
description="Download artwork from Hentai Foundry artists",
auth_type="cookies",
requires_auth=False,
url_pattern=r"^https?://(www\.)?hentai-foundry\.com/",
url_examples=[
"https://www.hentai-foundry.com/user/example_artist",
"https://www.hentai-foundry.com/pictures/user/example_artist",
],
default_config={**GD_DEFAULTS, "content_types": ["pictures"]},
derive_post_url=derive_post_url,
augment_cookies=augment_cookies,
)
+23
View File
@@ -0,0 +1,23 @@
"""Patreon — no quirks. The reference platform.
Patreon's gallery-dl sidecars are the well-behaved baseline: `url` is a
real permalink, `id` is the post id, `title` and `content` are
populated. No cookie quirks (session cookies are domain-wide). No
derivation overrides.
"""
from .base import GD_DEFAULTS, PlatformInfo
INFO = PlatformInfo(
key="patreon",
name="Patreon",
description="Download posts from Patreon creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?patreon\.com/",
url_examples=[
"https://www.patreon.com/example_artist",
"https://www.patreon.com/user?u=12345678",
],
default_config={**GD_DEFAULTS, "content_types": ["images", "attachments"]},
)
+32
View File
@@ -0,0 +1,32 @@
"""Pixiv — one quirk.
post_url: gallery-dl's `url` is the image URL on `i.pximg.net`. The
post permalink follows /artworks/<id>. external_post_id (= `id`) was
already correct, so no override there.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
pid = str_id_value(data.get("id"))
if pid:
return f"https://www.pixiv.net/artworks/{pid}"
return None
INFO = PlatformInfo(
key="pixiv",
name="Pixiv",
description="Download artwork from Pixiv artists",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?pixiv\.net/",
url_examples=[
"https://www.pixiv.net/users/12345678",
"https://www.pixiv.net/en/users/12345678",
],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
derive_post_url=derive_post_url,
)
@@ -0,0 +1,62 @@
"""SubscribeStar — three quirks colocated.
1. external_post_id: gallery-dl puts the per-attachment id in `id`
(e.g. 711509) and the actual post id in `post_id` (e.g. 360360).
The default chain in base.py already prefers `post_id`; this module
doesn't need to override it but the comment lives here too so a
future reader knows the chain's order was driven by this platform.
2. post_url: gallery-dl's `url` is the file CDN URL
(`/post_uploads?payload=...`). Synthesize the post permalink from
`post_id`.
3. augment_cookies: the server gates artist pages behind a
`_personalization_id` age-confirmation cookie that the user can't
easily refresh — SubscribeStar's frontend JS uses localStorage to
suppress the age popup once dismissed. gallery-dl's own login flow
sidesteps this by setting `18_plus_agreement_generic=true` on
`.subscribestar.adult`; we mirror that for cookies captured via the
extension.
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
pid = str_id_value(data.get("post_id"))
if pid:
return f"https://www.subscribestar.com/posts/{pid}"
return None
def augment_cookies(netscape: str) -> str:
if "18_plus_agreement_generic" in netscape:
return netscape
# Far-future expiry — gallery-dl's own login flow sets this with no
# explicit expiry; the server only checks presence/value.
expiry = 4102444800 # 2100-01-01 UTC
line = "\t".join([
".subscribestar.adult", "TRUE", "/", "TRUE",
str(expiry), "18_plus_agreement_generic", "true",
])
body = netscape.rstrip("\n")
if not body:
body = "# Netscape HTTP Cookie File"
return body + "\n" + line + "\n"
INFO = PlatformInfo(
key="subscribestar",
name="SubscribeStar",
description="Download posts from SubscribeStar creators",
auth_type="cookies",
requires_auth=True,
url_pattern=r"^https?://(www\.)?subscribestar\.(com|adult)/",
url_examples=[
"https://subscribestar.adult/example_artist",
"https://www.subscribestar.com/example_artist",
],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
derive_post_url=derive_post_url,
augment_cookies=augment_cookies,
)
+22 -12
View File
@@ -109,7 +109,10 @@ class PostFeedService:
if row is None:
return None
post, artist, source = row
thumbs_map = await self._thumbnails_for([post.id])
# Detail endpoint returns the FULL image list for PostModal's
# masonry grid — feed query still caps at THUMBNAIL_LIMIT via
# the default arg.
thumbs_map = await self._thumbnails_for([post.id], limit=None)
atts_map = await self._attachments_for([post.id])
item = self._to_dict(post, artist, source, thumbs_map, atts_map)
item["description_full"] = html_to_plain(post.description)
@@ -117,15 +120,21 @@ class PostFeedService:
# --- composition helpers ---------------------------------------------
async def _thumbnails_for(self, post_ids: list[int]) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to 6], "more": int}.
async def _thumbnails_for(
self, post_ids: list[int], *, limit: int | None = THUMBNAIL_LIMIT,
) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to limit], "more": int}.
Selects THUMBNAIL_LIMIT+1 images per post via window function so we
can detect overflow in a single query.
Selects up to `limit` images per post via window function so we
can detect overflow in a single query. Pass `limit=None` to
return ALL thumbnails per post (used by `get_post` for PostModal's
masonry grid; the feed pass keeps the default cap so payloads
stay small).
"""
if not post_ids:
return {}
# Rank images within each post and fetch only the top THUMBNAIL_LIMIT+1.
# Rank images within each post; cap at `limit` rows per post when
# limit is set, return all when limit is None.
ranked = (
select(
ImageRecord.id,
@@ -143,12 +152,13 @@ class PostFeedService:
.where(ImageRecord.primary_post_id.in_(post_ids))
.subquery()
)
rows = (await self.session.execute(
select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.total,
).where(ranked.c.rn <= THUMBNAIL_LIMIT)
)).all()
stmt = select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.total,
)
if limit is not None:
stmt = stmt.where(ranked.c.rn <= limit)
rows = (await self.session.execute(stmt)).all()
out: dict[int, dict] = {pid: {"thumbs": [], "more": 0} for pid in post_ids}
for img_id, pid, sha, mime, total in rows:
+106
View File
@@ -0,0 +1,106 @@
"""Layer-2 one-shot re-download remediation for corrupt imported files.
When an import fails on a file that came from a known, pollable
subscription Source, deleting the bad copy and re-running the source's
downloader can fetch a fresh, unblemished copy. This only helps when:
- the corruption is in transit / on disk (not at the source), AND
- the file resolves to an ENABLED Source with a real feed URL
(a `sidecar:<platform>:<slug>` synthetic anchor is not pollable),
AND
- we haven't already re-fetched this task once (bounded by
ImportTask.refetched so source-side corruption can't loop).
Filesystem-only imports with no resolvable Source return 'no_source'
the operator's only remediation there is to replace the file on disk.
Operator-requested 2026-05-28 (Layer 2).
"""
import json
import logging
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from ..models import Artist, ImportTask, Source
from ..utils.paths import derive_top_level_artist
from ..utils.sidecar import find_sidecar, parse_sidecar
from ..utils.slug import slugify
log = logging.getLogger(__name__)
def resolve_refetch_source(
session: Session, source_path: str, import_root: Path,
) -> Source | None:
"""Find an enabled, real-URL Source for the file's (artist, platform),
or None when nothing re-pollable resolves."""
path = Path(source_path)
sc = find_sidecar(path)
if sc is None:
return None
try:
data = json.loads(sc.read_text("utf-8"))
except (OSError, json.JSONDecodeError):
return None
if not isinstance(data, dict):
return None
sd = parse_sidecar(data)
if not sd.platform:
return None
artist_name = derive_top_level_artist(path, import_root)
if not artist_name:
return None
artist = session.execute(
select(Artist).where(Artist.slug == slugify(artist_name))
).scalar_one_or_none()
if artist is None:
return None
src = session.execute(
select(Source)
.where(
Source.artist_id == artist.id,
Source.platform == sd.platform,
Source.enabled.is_(True),
)
.order_by(Source.id.asc())
).scalars().first()
if src is None:
return None
if (src.url or "").startswith("sidecar:"):
return None # synthetic anchor — not a pollable feed
return src
def attempt_refetch(
session: Session, task: ImportTask, import_root: Path,
) -> dict:
"""Delete the corrupt file, mark the task refetched, and trigger ONE
source re-check. Idempotent/bounded: a task already refetched (or
with no resolvable Source) is a no-op. Commits."""
if task.refetched:
return {"status": "already_refetched"}
src = resolve_refetch_source(session, task.source_path, import_root)
if src is None:
return {"status": "no_source"}
# Remove the bad copy so gallery-dl (skip_existing) re-fetches it on
# the source re-check instead of skipping the still-present corrupt
# file.
try:
Path(task.source_path).unlink(missing_ok=True)
except OSError as exc:
log.warning("refetch unlink failed for %s: %s", task.source_path, exc)
task.refetched = True
session.add(task)
session.commit()
# Lazy import to avoid a tasks→services→tasks import cycle at module
# load. download_source.delay() is sync-safe in any context.
from ..tasks.download import download_source
download_source.delay(src.id)
return {"status": "refetch_queued", "source_id": src.id}
+57
View File
@@ -0,0 +1,57 @@
"""FC-3k: admin destructive Celery tasks.
Two long-running ops on the maintenance queue. task_run lifecycle is
captured automatically by FC-3i signals — these tasks just return
their summary dict so it lands in task_run.metadata (via Celery's
result backend) for the dashboard to surface.
Soft/hard time limits inherit the FC-3i recovery sweep: a runaway
task gets killed and flipped to status='timeout' by
recover_stalled_task_runs.
"""
from __future__ import annotations
import logging
from pathlib import Path
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..services import cleanup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
@celery.task(
name="backend.app.tasks.admin.delete_artist_cascade_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
def delete_artist_cascade_task(self, *, artist_id: int) -> dict:
"""Wraps cleanup_service.delete_artist_cascade. Returns the
service's summary dict for FC-3i task_run.metadata capture."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_artist_cascade(
session, artist_id=artist_id, images_root=IMAGES_ROOT,
)
@celery.task(
name="backend.app.tasks.admin.bulk_delete_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=900, time_limit=1200, # 15 min / 20 min
)
def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
"""Wraps cleanup_service.delete_images."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_images(
session, image_ids=image_ids, images_root=IMAGES_ROOT,
)
+300
View File
@@ -0,0 +1,300 @@
"""FC-3h: backup/restore Celery tasks.
All tasks live on the maintenance queue (per celery_app.task_routes).
task_run lifecycle tracking is automatic via FC-3i signals — these
tasks just record the operator-facing artifact metadata into
BackupRun.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from pathlib import Path
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import BackupRun, ImportSettings
from ..services import backup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
"""Flip a BackupRun row from running/restoring to error with a
truncated error message and finished_at. Caller already holds the
session open."""
row.status = "error"
row.error = f"{type(exc).__name__}: {exc}"[:2000]
row.finished_at = datetime.now(UTC)
session.add(row)
session.commit()
@celery.task(
name="backend.app.tasks.backup.backup_db_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=10, retry_backoff_max=120, max_retries=2,
soft_time_limit=600, time_limit=720,
)
def backup_db_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one DB backup. Returns {'backup_run_id': N}."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="db", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_db(
db_url=cfg.database_url_sync, images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.sql_path = result["sql_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.backup_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=30, retry_backoff_max=300, max_retries=1,
soft_time_limit=21600, time_limit=23400,
)
def backup_images_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one images backup. Same shape as backup_db_task; uses
tar_path instead of sql_path."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="images", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_images(
images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.tar_path = result["tar_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.restore_db_task",
bind=True,
max_retries=0, # NEVER auto-retry a half-applied restore.
soft_time_limit=1200, time_limit=1800,
)
def restore_db_task(self, *, source_backup_run_id: int) -> dict:
"""Restore from a previous DB backup. Inserts a NEW BackupRun row
(kind='db', status='restoring') linked to the source via
restored_from_id; flips to 'restored' on success or 'error' on
failure. Operator sees the restore as a row in the dashboard."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "db" or not src.sql_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid DB backup"
)
marker = BackupRun(
kind="db", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_sql_path": src.sql_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
sql_path = src.sql_path
try:
backup_service.restore_db(
db_url=cfg.database_url_sync, sql_path=Path(sql_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.restore_images_task",
bind=True, max_retries=0,
soft_time_limit=21600, time_limit=23400,
)
def restore_images_task(self, *, source_backup_run_id: int) -> dict:
"""Mirrors restore_db_task; uses backup_service.restore_images."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "images" or not src.tar_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid images backup"
)
marker = BackupRun(
kind="images", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_tar_path": src.tar_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
tar_path = src.tar_path
try:
backup_service.restore_images(
images_root=IMAGES_ROOT, tar_path=Path(tar_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.prune_backups",
soft_time_limit=300, time_limit=600,
)
def prune_backups() -> dict:
"""Daily Beat. Per-kind retention from ImportSettings.
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
for kind, keep in (
("db", s.backup_db_keep_last_n),
("images", s.backup_images_keep_last_n),
):
candidates = session.execute(
select(BackupRun)
.where(BackupRun.kind == kind)
.where(BackupRun.tag.is_(None))
.where(BackupRun.status.in_(["ok", "error"]))
.order_by(BackupRun.started_at.desc())
.offset(keep)
).scalars().all()
for row in candidates:
result = backup_service.unlink_artifact_files(
sql_path=row.sql_path,
tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
counts["files_unlinked"] += sum(
1 for v in result.values() if v
)
session.delete(row)
counts[f"{kind}_deleted"] += 1
session.commit()
return counts
@celery.task(
name="backend.app.tasks.backup.backup_db_nightly",
soft_time_limit=60, time_limit=120,
)
def backup_db_nightly() -> dict:
"""Hourly tick. Dispatches a real backup ONLY if the configured
UTC hour matches and the nightly setting is enabled. Returns
either {'skipped': '<reason>'} or {'dispatched': '<task_id>'}."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
nightly_enabled = s.backup_db_nightly_enabled
configured_hour = s.backup_db_nightly_hour_utc
if not nightly_enabled:
return {"skipped": "nightly disabled"}
now_hour = datetime.now(UTC).hour
if now_hour != configured_hour:
return {"skipped": f"hour={now_hour} != configured={configured_hour}"}
res = backup_db_task.delay(triggered_by="nightly")
return {"dispatched": res.id}
+88 -28
View File
@@ -29,10 +29,12 @@ IMAGES_ROOT = Path("/images")
def _map_result_to_status(result):
"""(ImportTask.status, should_requeue_ml_and_thumb) for an ImportResult.
'superseded' = the kept row's file/ML changed → complete + re-derive.
'attached' = a non-art file preserved → complete, no ML/thumb."""
'attached' = a non-art file preserved → complete, no ML/thumb.
'refreshed' = deep scan refreshed sidecar/phash on an existing row →
complete, no ML/thumb re-derive (file/pixels unchanged)."""
if result.status in ("imported", "superseded"):
return ("complete", True)
if result.status == "attached":
if result.status in ("attached", "refreshed"):
return ("complete", False)
if result.status == "skipped":
return ("skipped", False)
@@ -62,30 +64,13 @@ def _mark_failed(session, task, error_msg: str) -> None:
pass
@celery.task(
name="backend.app.tasks.import_file.import_media_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=360,
)
def import_media_file(self, import_task_id: int) -> dict:
"""Returns a dict so the eager-mode tests can assert without DB.
Decorator notes:
- autoretry_for: transient DB / filesystem errors retry with
exponential backoff (5s base, jitter, max 3 attempts). On final
give-up the task raises and acks_late=True (set globally on the
Celery app) does NOT redeliver — the recovery sweep catches the
row instead.
- soft_time_limit (300s) raises SoftTimeLimitExceeded in this
process so the task can mark its row failed before being killed.
- time_limit (360s) is the hard cap; SIGKILL if the soft signal
was swallowed.
def _run_import_task(import_task_id: int) -> dict:
"""Shared body for import_media_file + import_archive_file. The two
tasks differ ONLY in their Celery time limits (a single media file
is sub-second; an archive runs the full per-member pipeline inline
for every member and can take many minutes). Both flip the row to
'processing', dispatch to `_do_import`, and honor the
flip-to-terminal resilience contract.
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
@@ -101,19 +86,85 @@ def import_media_file(self, import_task_id: int) -> dict:
try:
return _do_import(session, task, import_task_id)
except SoftTimeLimitExceeded:
_mark_failed(session, task, "soft_time_limit exceeded (>300s)")
_mark_failed(session, task, "soft_time_limit exceeded")
raise
except (OperationalError, DBAPIError, OSError):
# Retryable per the decorator; do NOT mark failed (let
# autoretry have a clean go at it). If autoretry exhausts,
# the row stays 'processing' and the maintenance sweep
# flips it within 5 min.
# flips it.
raise
except Exception as exc: # noqa: BLE001 — pipeline crash, mark + re-raise
_mark_failed(session, task, f"{type(exc).__name__}: {exc}")
raise
@celery.task(
name="backend.app.tasks.import_file.import_media_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=360,
)
def import_media_file(self, import_task_id: int) -> dict:
"""Import ONE media file (or non-media → PostAttachment). Sub-second
for the common case; the tight 5-min soft limit keeps a genuinely
stuck single-file import detectable fast.
Decorator notes:
- autoretry_for: transient DB / filesystem errors retry with
exponential backoff (5s base, jitter, max 3 attempts). On final
give-up the task raises and acks_late=True (set globally on the
Celery app) does NOT redeliver — the recovery sweep catches the
row instead.
- soft_time_limit (300s) raises SoftTimeLimitExceeded in-process
so the task can mark its row failed before being killed.
- time_limit (360s) is the hard SIGKILL cap.
"""
return _run_import_task(import_task_id)
@celery.task(
name="backend.app.tasks.import_file.import_archive_file",
bind=True,
autoretry_for=(OperationalError, DBAPIError, OSError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
# Archives run the full per-member pipeline (sha256 + pHash + dedup
# query + copy + provenance) for EVERY media member inline, under a
# single task budget. A multi-hundred-member archive blows the
# 5-min media limit. soft=30min / hard=35min sizes for a large
# archive. Operator-flagged 2026-05-28 (target 1645019 hit the old
# shared 300s soft limit). The recovery sweep gives this task its
# own 40-min threshold via maintenance.TASK_STUCK_THRESHOLD_MINUTES
# so it isn't preempted while legitimately grinding through members.
soft_time_limit=1800,
time_limit=2100,
)
def import_archive_file(self, import_task_id: int) -> dict:
"""Import an archive: extract + run the per-member media pipeline for
every member inline, then preserve the archive as a PostAttachment.
Same body as import_media_file (dispatch is by file kind inside
Importer.import_one); split out purely for the larger time budget."""
return _run_import_task(import_task_id)
def enqueue_import(task_id: int, task_type: str) -> None:
"""Route an ImportTask to the right Celery task by its task_type.
Single source of truth for the media-vs-archive dispatch so the
scan, retry, and recovery-requeue paths stay in sync."""
if task_type == "archive":
import_archive_file.delay(task_id)
else:
import_media_file.delay(task_id)
def _do_import(session, task, import_task_id: int) -> dict:
"""Actual work, called from inside the resilience wrapper."""
settings = session.execute(
@@ -138,6 +189,15 @@ def _do_import(session, task, import_task_id: int) -> dict:
task.result_image_id = result.image_id
counter_col_name = "imported"
counter_col = ImportBatch.imported
elif result.status == "refreshed":
# Deep-scan rederive: existing row got phash/artist/sidecar
# refreshed. Task is complete (no further work), but counted in
# `refreshed` not `imported` so the UI can surface the actual
# work done. operator-flagged 2026-05-25.
task.status = "complete"
task.result_image_id = result.image_id
counter_col_name = "refreshed"
counter_col = ImportBatch.refreshed
elif result.status == "attached":
task.status = "complete"
counter_col_name = "attachments"
+167
View File
@@ -0,0 +1,167 @@
"""scan_library_for_rule Celery task — iterates image_record in keyset-
paginated batches, evaluates the audit rule per image, populates
LibraryAuditRun.matched_ids. Runs on the maintenance queue with a 2h soft
time limit (plenty of margin for 100k+ image libraries at ~100ms PIL
decode + histogram per image).
State machine:
start: status='running'
end success: status='ready'
end error: status='error', error=traceback
oversize: status='error', error='matched too many images; tighten threshold'
external cancel: scan sees status='cancelled' between batches, exits.
"""
import logging
import traceback
from datetime import UTC, datetime
from celery.exceptions import SoftTimeLimitExceeded
from PIL import Image
from sqlalchemy import select, update
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..models import ImageRecord, LibraryAuditRun
from ..services.audits import single_color, transparency
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
_BATCH = 500
_PROGRESS_TICK = 100
_MAX_MATCHED = 50_000
_RULES = {
"transparency": transparency.evaluate,
"single_color": single_color.evaluate,
}
@celery.task(
name="backend.app.tasks.library_audit.scan_library_for_rule",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=7200,
time_limit=7500,
)
def scan_library_for_rule(self, audit_id: int) -> dict:
"""See module docstring. Returns a small summary dict for eager-mode
test assertions (real workers ignore the return value)."""
SessionLocal = _sync_session_factory()
try:
with SessionLocal() as session:
audit = session.get(LibraryAuditRun, audit_id)
if audit is None:
return {"audit_id": audit_id, "status": "missing"}
evaluate = _RULES.get(audit.rule)
if evaluate is None:
_mark_error(session, audit_id, f"unknown rule {audit.rule!r}")
return {"audit_id": audit_id, "status": "error"}
params = dict(audit.params or {})
matched: list[int] = []
scanned = 0
last_id = 0
while True:
# Cancellation check between batches.
current_status = session.execute(
select(LibraryAuditRun.status)
.where(LibraryAuditRun.id == audit_id)
).scalar_one()
if current_status == "cancelled":
return {"audit_id": audit_id, "status": "cancelled"}
rows = session.execute(
select(ImageRecord.id, ImageRecord.path)
.where(ImageRecord.id > last_id)
.where(ImageRecord.mime.like("image/%"))
.order_by(ImageRecord.id.asc())
.limit(_BATCH)
).all()
if not rows:
break
for image_id, image_path in rows:
last_id = image_id
scanned += 1
try:
with Image.open(image_path) as im:
try:
if evaluate(im, **params):
matched.append(image_id)
except Exception as exc: # noqa: BLE001
log.warning(
"audit %s: rule evaluate failed on %s: %s",
audit_id, image_path, exc,
)
except FileNotFoundError:
log.warning(
"audit %s: image_record %s file missing at %s; skipping",
audit_id, image_id, image_path,
)
except OSError as exc:
log.warning(
"audit %s: PIL load failed for %s: %s",
audit_id, image_path, exc,
)
if len(matched) > _MAX_MATCHED:
_mark_error(
session, audit_id,
f"matched > {_MAX_MATCHED} images; "
"tighten threshold and re-run",
)
return {"audit_id": audit_id, "status": "error"}
if scanned % _PROGRESS_TICK == 0:
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(scanned_count=scanned)
)
session.commit()
# Final state.
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(
scanned_count=scanned,
matched_count=len(matched),
matched_ids=matched,
status="ready",
finished_at=datetime.now(UTC),
)
)
session.commit()
return {
"audit_id": audit_id,
"status": "ready",
"scanned": scanned,
"matched": len(matched),
}
except SoftTimeLimitExceeded:
with SessionLocal() as session:
_mark_error(session, audit_id, "soft_time_limit exceeded (>7200s)")
raise
except (OperationalError, DBAPIError):
# Retryable per the decorator; leave row in 'running' and let
# autoretry try again. Recovery sweep catches if all retries fail.
raise
except Exception: # noqa: BLE001
tb = traceback.format_exc()
with SessionLocal() as session:
_mark_error(session, audit_id, tb)
raise
def _mark_error(session, audit_id: int, error_msg: str) -> None:
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(
status="error",
error=error_msg,
finished_at=datetime.now(UTC),
)
)
session.commit()
+281 -28
View File
@@ -1,63 +1,212 @@
"""Periodic maintenance: recover stuck import tasks, garbage-collect old finished tasks."""
import logging
import os
import subprocess
from datetime import UTC, datetime, timedelta
from pathlib import Path
from PIL import Image
from sqlalchemy import delete, select, update
from sqlalchemy import and_, delete, or_, select, update
from ..celery_app import celery
from ..models import DownloadEvent, ImageRecord, ImportSettings, ImportTask
from ..models import DownloadEvent, ImageRecord, ImportSettings, ImportTask, TaskRun
from ..utils.phash import compute_phash
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
STUCK_THRESHOLD_MINUTES = 5
# Archive ImportTasks run the per-member pipeline inline for every
# member (import_archive_file: soft=30min/hard=35min). The ImportTask
# 'processing' recovery sweep must give them a longer threshold or it
# re-queues a legitimately-running archive mid-import (double-process).
# 40 min = 5-min buffer past the archive task's hard kill.
# Operator-flagged 2026-05-28 (target 1645019, a big archive).
ARCHIVE_STUCK_THRESHOLD_MINUTES = 40
# Poison-pill cap. After being recovered (re-queued from a stuck
# 'processing' state) MAX_RECOVERY_ATTEMPTS-1 times, the next sweep
# marks the row 'failed' instead of looping. 3 = two recoveries then
# give up. A row reaches this only if it leaves NO terminal flip each
# run — i.e. it hard-crashes the worker (OOM/segfault/SIGKILL), the
# signature of a corrupt or oversized input. Caught exceptions already
# flip to terminal 'failed' and never enter this loop.
MAX_RECOVERY_ATTEMPTS = 3
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7
PHASH_PAGE = 500
VERIFY_PAGE = 200
FFPROBE_TIMEOUT_SECONDS = 10
TASK_RUN_KEEP_OK_SECONDS = 24 * 3600 # 24 h
TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# Overrides for recover_stalled_task_runs (the TaskRun 'running' sweep).
# Tasks/queues that legitimately run longer than the default 5-min
# threshold need their own larger value, else the sweep marks in-flight
# work 'error' before it finishes. Each value MUST be ≥ the relevant
# task.time_limit + a small buffer. task_name overrides take precedence
# over queue overrides.
#
# ml queue: tag_and_embed video branch (≈20 GPU ops); time_limit=1200.
# import_archive_file: shares the 'import' queue with the fast
# single-file import_media_file, so it needs a task-name override
# (the import queue itself stays at the 5-min default for single
# files); time_limit=2100.
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"ml": 25,
}
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
"backend.app.tasks.import_file.import_archive_file": 40,
}
@celery.task(name="backend.app.tasks.maintenance.recover_interrupted_tasks")
def recover_interrupted_tasks() -> int:
"""Find ImportTask rows stuck in 'processing' for >5 min and re-queue them.
"""Recover stuck ImportTask rows. Two distinct stuck states:
Why 5 min: import_media_file is sub-second for the vast majority of
files; even a large-video transcode caps at the per-task soft_time_limit
(5 min) defined on the task itself. Anything still 'processing' after
that window is a confirmed crash (worker died, DB disconnect mid-flush,
OOM) and must be recycled. Was 30 min historically; tightened
2026-05-24 after operator hit a 2224-row zombie pile during the IR
migration scan.
1. 'processing' too long — worker crash mid-import. Re-queue via
enqueue_import (routing media vs archive) and let the import
retry. Threshold is task-type-aware: media files are sub-second
and capped at the 5-min soft limit, so STUCK_THRESHOLD_MINUTES
(5) means a confirmed crash; archives run the per-member
pipeline inline (import_archive_file, 35-min hard limit) so they
get ARCHIVE_STUCK_THRESHOLD_MINUTES (40) to avoid re-queueing a
still-running archive. (Media was tightened from 30 min to 5
2026-05-24 after a 2224-row zombie pile; archive split out
2026-05-28.)
2. 'pending' or 'queued' > 30 min — enqueue-phase crash. scan_directory
creates rows with status='pending' (commit), then in a second pass
transitions to 'queued' and calls .delay() (commit). If the scanner
crashes between those two commits, rows are orphaned in 'pending'
(never enqueued) with no recovery path — invisible to the
'processing' sweep above. Flagged 2026-05-25 by operator hitting a
5490-row orphan pile. Flip these to 'failed' (not re-enqueue) so
the operator drains them via /api/import/retry-failed at their own
pace; bulk-re-enqueueing 5000+ rows would thundering-herd the
import worker.
Returns total rows touched (recovered + marked failed).
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
now = datetime.now(UTC)
media_cutoff = now - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
archive_cutoff = now - timedelta(minutes=ARCHIVE_STUCK_THRESHOLD_MINUTES)
orphan_cutoff = now - timedelta(minutes=ORPHAN_PENDING_THRESHOLD_MINUTES)
with SessionLocal() as session:
stuck_ids = session.execute(
select(ImportTask.id)
.where(ImportTask.status == "processing")
.where(ImportTask.started_at < cutoff)
).scalars().all()
if not stuck_ids:
return 0
session.execute(
update(ImportTask)
.where(ImportTask.id.in_(stuck_ids))
.values(status="queued", started_at=None, error="recovered from stuck state")
# Both sweeps used to be SELECT ids → UPDATE WHERE id IN (...) which
# blew past psycopg's 65535-parameter ceiling once a sweep covered
# tens of thousands of rows (operator hit it 2026-05-26 after the
# /import deep scan piled up orphans). Folding the SELECT into the
# UPDATE eliminates the IN-list entirely. RETURNING gives us back
# exactly the (id, task_type) pairs that flipped so the requeue
# can route media vs archive correctly.
#
# Media + archive get separate cutoffs: a single media file is
# sub-second so 5 min means crash; an archive runs the per-member
# pipeline inline and can legitimately take up to its 35-min hard
# limit, so it gets ARCHIVE_STUCK_THRESHOLD_MINUTES (40) to avoid
# re-queueing a still-running archive.
stuck_predicate = and_(
ImportTask.status == "processing",
or_(
and_(ImportTask.task_type != "archive",
ImportTask.started_at < media_cutoff),
and_(ImportTask.task_type == "archive",
ImportTask.started_at < archive_cutoff),
),
)
# POISON-PILL CIRCUIT BREAKER (Layer 1, 2026-05-28). A row that
# leaves no terminal flip (hard worker crash: OOM/segfault/SIGKILL
# on a corrupt or oversized input) gets re-queued by this sweep —
# and would loop forever, re-crashing the worker each pass,
# without a cap. Once a row has already been recovered
# MAX_RECOVERY_ATTEMPTS-1 times, stop re-queueing it and mark it
# 'failed' with a diagnostic so the operator can find + replace
# the offending file. This UPDATE runs FIRST so the rows it
# claims drop out of 'processing' before the re-queue pass.
poison_result = session.execute(
update(ImportTask)
.where(stuck_predicate)
.where(ImportTask.recovery_count >= MAX_RECOVERY_ATTEMPTS - 1)
.values(
status="failed",
finished_at=now,
error=(
f"crashed or stalled the worker {MAX_RECOVERY_ATTEMPTS} "
f"times without completing — likely a corrupt or "
f"oversized input. Not re-queued. Inspect/replace the "
f"file, then retry via /api/import/retry-failed."
),
)
.returning(ImportTask.id)
)
poison_ids = [r[0] for r in poison_result.all()]
# Re-queue the remaining stuck rows (under the cap) and bump
# their recovery_count. RETURNING (id, task_type) so the requeue
# routes media vs archive correctly.
stuck_result = session.execute(
update(ImportTask)
.where(stuck_predicate)
.where(ImportTask.recovery_count < MAX_RECOVERY_ATTEMPTS - 1)
.values(
status="queued",
started_at=None,
recovery_count=ImportTask.recovery_count + 1,
error="recovered from stuck state",
)
.returning(ImportTask.id, ImportTask.task_type)
)
stuck = stuck_result.all()
orphan_result = session.execute(
update(ImportTask)
.where(ImportTask.status.in_(["pending", "queued"]))
.where(ImportTask.created_at < orphan_cutoff)
.values(
status="failed",
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
"/api/import/retry-failed"
),
)
)
orphan_count = orphan_result.rowcount or 0
session.commit()
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
if stuck:
from .import_file import enqueue_import
for tid, task_type in stuck:
enqueue_import(tid, task_type)
return len(stuck_ids)
# Layer-2 auto re-download (env-gated, default OFF). For each
# poison-pill row that resolves to a pollable Source, delete the
# bad file and trigger ONE source re-check to fetch a fresh
# copy. Bounded by ImportTask.refetched so source-side
# corruption can't loop. The 'failed' row stays as history; the
# re-downloaded file re-imports as a fresh task on the next scan.
if poison_ids and os.environ.get("FC_AUTO_REFETCH_CORRUPT", "0") == "1":
from ..models import ImportSettings
from ..services.refetch_service import attempt_refetch
import_root = Path(session.execute(
select(ImportSettings.import_scan_path)
.where(ImportSettings.id == 1)
).scalar_one())
for pid in poison_ids:
ptask = session.get(ImportTask, pid)
if ptask is None:
continue
try:
attempt_refetch(session, ptask, import_root)
except Exception as exc: # noqa: BLE001 — best-effort
log.warning("auto-refetch failed for task %s: %s", pid, exc)
return len(stuck) + len(poison_ids) + orphan_count
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_tasks")
@@ -80,6 +229,110 @@ def cleanup_old_tasks() -> int:
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_task_runs")
def recover_stalled_task_runs() -> int:
"""Flip task_run rows stuck in 'running' past their queue-specific
threshold to 'error'. FC-3i.
A row gets stuck when the worker dies without emitting
task_postrun / task_failure (e.g. OOM, container restart between
signals, signal handler raised+logged). The default 5-min threshold
fits short-lived queues (import/thumbnail/download); queues that
legitimately run longer tasks (ml-video, deep scans) get their
own larger threshold via QUEUE_STUCK_THRESHOLD_MINUTES so the
sweep doesn't preempt them.
Runs once per distinct threshold value: each pass updates rows
whose queue maps to that threshold.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
override_tasks = set(TASK_STUCK_THRESHOLD_MINUTES.keys())
override_queues = set(QUEUE_STUCK_THRESHOLD_MINUTES.keys())
total = 0
def _flag(minutes, *extra_where):
cutoff = now - timedelta(minutes=minutes)
stmt = (
update(TaskRun)
.where(TaskRun.status == "running")
.where(TaskRun.started_at < cutoff)
.values(
status="error",
error_type="RecoverySweep",
error_message=(
f"no completion signal received within {minutes} min"
),
finished_at=now,
)
)
for w in extra_where:
stmt = stmt.where(w)
return session.execute(stmt).rowcount or 0
with SessionLocal() as session:
# Precedence: task_name override → queue override → default.
# Each pass excludes rows claimed by a higher-precedence pass so
# every row is touched at most once.
# 1. Per-task-name overrides (e.g. import_archive_file, which
# shares the 'import' queue with fast single-file imports).
for task_name, minutes in TASK_STUCK_THRESHOLD_MINUTES.items():
total += _flag(minutes, TaskRun.task_name == task_name)
# 2. Per-queue overrides, excluding the override task-names.
for queue, minutes in QUEUE_STUCK_THRESHOLD_MINUTES.items():
wheres = [TaskRun.queue == queue]
if override_tasks:
wheres.append(TaskRun.task_name.notin_(override_tasks))
total += _flag(minutes, *wheres)
# 3. Default — everything not claimed above.
default_wheres = []
if override_queues:
default_wheres.append(TaskRun.queue.notin_(override_queues))
if override_tasks:
default_wheres.append(TaskRun.task_name.notin_(override_tasks))
total += _flag(STUCK_THRESHOLD_MINUTES, *default_wheres)
session.commit()
return total
@celery.task(name="backend.app.tasks.maintenance.prune_task_runs")
def prune_task_runs() -> dict:
"""Daily retention for task_run rows. FC-3i.
- 'ok' rows: deleted after TASK_RUN_KEEP_OK_SECONDS (24h default).
Success is high-volume, not interesting after a day.
- 'error' / 'timeout' rows: deleted after TASK_RUN_KEEP_FAILURE_SECONDS
(7 days default). Failures are operationally interesting longer.
- 'running' rows: NEVER deleted by this task. The recovery sweep
(recover_stalled_task_runs) is the mechanism that flips them to
terminal state; prune doesn't touch in-flight state.
- 'retry' rows: treated as failures (>7d).
Returns dict of how many rows were deleted in each bucket.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
ok_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_OK_SECONDS)
fail_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_FAILURE_SECONDS)
with SessionLocal() as session:
ok_deleted = session.execute(
delete(TaskRun)
.where(TaskRun.status == "ok")
.where(TaskRun.finished_at < ok_cutoff)
).rowcount or 0
fail_deleted = session.execute(
delete(TaskRun)
.where(TaskRun.status.in_(["error", "timeout", "retry"]))
.where(TaskRun.finished_at < fail_cutoff)
).rowcount or 0
session.commit()
return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@celery.task(name="backend.app.tasks.maintenance.backfill_phash")
def backfill_phash() -> int:
"""Recompute phash for stored images that have none (imported before
+6 -32
View File
@@ -4,7 +4,8 @@ Dispatches to the right migrator based on `kind`. Updates MigrationRun
row's status/counts/finished_at as it runs. Failures set status='error'
with the error message preserved.
kinds: backup, gs_ingest, ir_ingest, tag_apply, ml_queue, verify, rollback, cleanup
kinds: gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup
(backup + rollback retired 2026-05-24 → see /api/system/backup/*)
"""
from __future__ import annotations
@@ -20,10 +21,8 @@ from ..celery_app import celery
from ..config import get_config
from ..models import MigrationRun
from ..services.credential_crypto import CredentialCrypto
from ..services.migrators import backup as backup_mod
from ..services.migrators import cleanup as cleanup_mod
from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify
from ..services.migrators import rollback as rollback_mod
log = logging.getLogger(__name__)
@@ -65,21 +64,11 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
async with factory() as db:
await _update_run(db, run_id, status="running")
try:
if kind == "backup":
manifest = backup_mod.create_backup(
db_url=get_config().database_url_sync,
images_root=IMAGES_ROOT,
tag=params.get("tag", "manual"),
if kind in ("backup", "rollback"):
raise ValueError(
f"kind {kind!r} retired in FC-3h; "
"use /api/system/backup/* instead"
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"manifest": manifest},
)
return manifest
elif kind == "gs_ingest":
fc_crypto = CredentialCrypto(_KEY_PATH)
@@ -166,21 +155,6 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
)
return result
elif kind == "rollback":
result = rollback_mod.rollback_to_pre_migration(
db_url=get_config().database_url_sync,
images_root=IMAGES_ROOT,
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"rollback_result": result},
)
return result
else:
raise ValueError(f"unknown kind: {kind}")
+21 -2
View File
@@ -31,8 +31,15 @@ def _is_video(path: Path) -> bool:
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=300,
time_limit=420,
# Sized for the video branch: sample 10 frames, run tagger +
# embedder on each (≈20 GPU ops vs 2 for an image). A loaded
# ml-worker can take 5-10 min on a long video; bumped from
# 5min/7min on 2026-05-28 after operator-flagged image 6288 (a
# .mp4) hit the recovery sweep at 5 min while still legitimately
# processing. Image runs return in seconds; the bump doesn't
# affect their UX.
soft_time_limit=900, # 15 min
time_limit=1200, # 20 min hard
)
def tag_and_embed(self, image_id: int) -> dict:
"""Run Camie + SigLIP on one image; store predictions + embedding;
@@ -64,6 +71,18 @@ def tag_and_embed(self, image_id: int) -> dict:
embedder = get_embedder()
if _is_video(src):
# Layer-3 isolation: ffprobe (a separate process) validates
# the container before we burn ~20 GPU ops sampling frames
# from it. A corrupt video that would crash the frame
# decoder is rejected cleanly here instead of taking down
# the ml-worker. Operator-flagged 2026-05-28.
from ..utils import safe_probe
vprobe = safe_probe.probe_video(src)
if not vprobe.ok:
return {
"status": "bad_video", "image_id": image_id,
"reason": vprobe.reason,
}
frames = _sample_video_frames(
src, int(os.environ.get("VIDEO_ML_FRAMES", "10"))
)
+26 -13
View File
@@ -17,6 +17,7 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_asyn
from ..celery_app import celery
from ..config import get_config
from ..models import DownloadEvent, ImportBatch, ImportSettings, ImportTask
from ..services.archive_extractor import is_archive
from ..services.scheduler_service import select_due_sources
from ._sync_engine import sync_session_factory as _sync_session_factory
@@ -59,16 +60,25 @@ def scan_directory(self, triggered_by: str = "manual",
session.flush()
batch_id = batch.id
# Skip-set: any source_path that already has a non-failed ImportTask
# row. Re-running scan_directory must not re-enqueue files the
# importer has already handled (or is currently handling); doing so
# creates duplicate work and inflates the queue. Failed prior tasks
# are eligible for retry.
# Skip-set behavior splits by mode (operator-flagged 2026-05-25):
#
# quick: any non-failed prior ImportTask (active OR finished) is
# skipped — quick scan only does new-file enqueue, so re-touching
# already-imported files is wasted work.
#
# deep: ONLY currently-in-flight tasks (pending/queued/processing)
# are skipped. Completed and skipped tasks ARE re-queued because
# deep scan exists precisely to re-touch already-imported files
# (refresh sidecar metadata, fill NULL phash, fill NULL artist
# via Importer._deep_rederive). Matches IR's deep-scan behavior.
active_statuses = ["pending", "queued", "processing"]
if mode == "deep":
skip_statuses = active_statuses
else:
skip_statuses = active_statuses + ["complete", "skipped"]
non_failed_existing = set(session.execute(
select(ImportTask.source_path).where(
ImportTask.status.in_(
["pending", "queued", "processing", "complete", "skipped"]
),
ImportTask.status.in_(skip_statuses),
)
).scalars().all())
@@ -87,7 +97,9 @@ def scan_directory(self, triggered_by: str = "manual",
task = ImportTask(
batch_id=batch_id,
source_path=entry_str,
task_type="media",
# Archives route to import_archive_file (larger time
# budget) — they run the per-member pipeline inline.
task_type="archive" if is_archive(entry) else "media",
status="pending",
size_bytes=size,
)
@@ -106,15 +118,16 @@ def scan_directory(self, triggered_by: str = "manual",
batch.finished_at = datetime.now(UTC)
session.commit()
# Now enqueue import_media_file for each pending task.
# Now enqueue each pending task on the right Celery task
# (media vs archive) via the shared router.
from .import_file import enqueue_import
for task in session.execute(
select(ImportTask).where(ImportTask.batch_id == batch_id)
).scalars():
task.status = "queued"
session.add(task)
from .import_file import import_media_file
import_media_file.delay(task.id)
enqueue_import(task.id, task.task_type)
session.commit()
if mode == "deep":
+79
View File
@@ -17,6 +17,30 @@ from ._sync_engine import sync_session_factory as _sync_session_factory
IMAGES_ROOT = Path("/images")
THUMB_MAGIC_JPEG = b"\xff\xd8\xff"
THUMB_MAGIC_PNG = b"\x89PNG\r\n\x1a\n"
def _thumb_is_valid(path: Path) -> bool:
"""Return True iff `path` exists and starts with a JPEG or PNG magic header.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG for
opaque sources, PNG for alpha sources. Anything else (missing file, OSError,
truncated, wrong magic) is invalid.
"""
try:
with path.open("rb") as f:
head = f.read(12)
except OSError:
return False
if len(head) < 8:
return False
if head[:3] == THUMB_MAGIC_JPEG:
return True
if head[:8] == THUMB_MAGIC_PNG:
return True
return False
@celery.task(
name="backend.app.tasks.thumbnail.generate_thumbnail",
@@ -50,3 +74,58 @@ def generate_thumbnail(self, image_id: int) -> dict:
session.add(record)
session.commit()
return {"status": "ok", "image_id": image_id, "path": str(result.path)}
@celery.task(
name="backend.app.tasks.thumbnail.backfill_thumbnails",
bind=True,
)
def backfill_thumbnails(self) -> dict:
"""Scan ImageRecord and enqueue generate_thumbnail for rows whose
thumbnail is missing, gone from disk, or has wrong magic bytes.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path for
rows that point at a missing or corrupt file before enqueueing — keeps
the DB self-consistent on partial runs and makes re-runs safe.
Returns {"enqueued": N, "ok": M, "regenerated": K} where:
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL thumbnail_path
cleared (i.e. missing + corrupt)
"""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {"enqueued": enqueued, "ok": ok, "regenerated": regenerated}
+14 -1
View File
@@ -13,8 +13,21 @@ HASH_SIZE = 8
def compute_phash(pil_image) -> str | None:
"""Perceptual hash of an opened PIL image, as a hex string. None on any
failure (videos/unreadable/non-image)."""
failure (videos/unreadable/non-image).
For animated images (multi-frame WebP/GIF/APNG), explicitly seek to
frame 0 first. Without this, some PIL operations downstream of
imagehash.phash (convert("L"), resize) can iterate all frames and
blow past Celery's hard time limit on large animations
(operator-flagged 2026-05-26 against animated WebPs). The pHash of
frame 0 is the conventional choice for animated content.
"""
try:
if getattr(pil_image, "is_animated", False):
try:
pil_image.seek(0)
except Exception:
pass
return str(imagehash.phash(pil_image, hash_size=HASH_SIZE))
except Exception:
return None
+170
View File
@@ -0,0 +1,170 @@
"""Subprocess-isolated media probes (Layer 3 of import resilience).
A malformed video or archive can hard-crash the worker process — a
decoder OOM, a native-lib segfault, or a decompression bomb. A hard
crash leaves no terminal flip, so the recovery sweep re-queues the row
and it crashes again: a poison-pill loop (the Layer-1 cap is the
backstop, but isolating the crash is better — the file gets a clean
terminal failure and the worker never dies).
These probes run the risky read in a way that contains the blast:
- Video: `ffprobe` is a separate binary, so a crash decoding the
container kills only ffprobe (non-zero exit), never the worker. Also
returns width/height, which the importer didn't previously capture
for videos.
- Archive: an uncompressed-size guard (catches decompression bombs
before they OOM anything) plus an integrity test in a spawned child
(catches native-lib crashes on a malformed archive). A child segfault
/ OOM shows up as a non-zero exit code, not a dead worker.
Images are intentionally NOT probed here: Pillow raises (it doesn't
segfault) on the realistic corrupt-image cases, the importer already
catches that as an invalid_image skip, and a subprocess per image would
wreck deep-scan throughput on a large library. Add an image branch only
if a real image-induced worker crash is ever observed.
Operator-requested 2026-05-28 (Layer 3).
"""
import json
import multiprocessing as mp
import subprocess
from dataclasses import dataclass
from pathlib import Path
VIDEO_PROBE_TIMEOUT_SECONDS = 60
ARCHIVE_PROBE_TIMEOUT_SECONDS = 120
# Refuse archives whose total UNCOMPRESSED size exceeds this — the
# classic decompression-bomb guard (a 4 GB cap comfortably clears real
# art-pack archives while stopping a few-KB zip that expands to TB).
MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 * 1024 * 1024 * 1024
@dataclass(frozen=True)
class ProbeResult:
ok: bool
# crashed=True means the probe HARD-FAILED (subprocess killed by a
# signal, OOM, or timeout) — the poison-pill signature. crashed=False
# with ok=False means a clean rejection (corrupt-but-handled,
# bomb-size-exceeded, integrity mismatch). Callers map crashed → a
# terminal 'failed', clean → a 'skipped'/'failed' of their choosing.
crashed: bool = False
reason: str | None = None
width: int | None = None
height: int | None = None
def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
"""Validate a video container + first video stream via ffprobe."""
try:
out = subprocess.run(
[
"ffprobe", "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "json", str(path),
],
capture_output=True, text=True, timeout=timeout,
)
except subprocess.TimeoutExpired:
return ProbeResult(ok=False, crashed=True, reason="ffprobe timed out")
except OSError as exc:
# ffprobe missing / not executable — environmental, not the
# file's fault. Treat as a clean non-crash failure so the import
# path can decide (it currently proceeds without dims).
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe unavailable: {exc}")
if out.returncode != 0:
return ProbeResult(
ok=False, crashed=False,
reason=f"ffprobe rejected the file: {out.stderr.strip()[:200]}",
)
try:
streams = (json.loads(out.stdout) or {}).get("streams") or []
except json.JSONDecodeError as exc:
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe output parse failed: {exc}")
if not streams:
return ProbeResult(ok=False, crashed=False, reason="no decodable video stream")
return ProbeResult(
ok=True, width=streams[0].get("width"), height=streams[0].get("height"),
)
def probe_archive(path: Path, *, timeout: float = ARCHIVE_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
"""Bomb-size guard + isolated integrity test for an archive."""
ctx = mp.get_context("spawn")
q = ctx.Queue()
proc = ctx.Process(target=_archive_probe_target, args=(str(path), q))
proc.start()
proc.join(timeout)
if proc.is_alive():
proc.terminate()
proc.join(5)
return ProbeResult(ok=False, crashed=True, reason="archive probe timed out")
if proc.exitcode != 0:
# Negative exitcode = killed by signal (segfault); positive =
# the child os._exit'd or was OOM-killed. Either way the file
# hard-crashed the probe — the poison-pill signature.
return ProbeResult(
ok=False, crashed=True,
reason=f"archive probe crashed (exit {proc.exitcode})",
)
try:
outcome = q.get(timeout=5)
except Exception: # noqa: BLE001 — empty queue / broken pipe
return ProbeResult(ok=False, crashed=True, reason="archive probe produced no result")
status, detail = outcome
if status == "ok":
return ProbeResult(ok=True)
return ProbeResult(ok=False, crashed=False, reason=detail)
def _archive_probe_target(path_str: str, q) -> None:
"""Runs in the spawned child. Reads member sizes (bomb guard) then
runs the format's integrity test. Puts ('ok', None) or
('error', reason). A crash/OOM here never reaches the queue — the
parent reads the non-zero exit code instead."""
path = Path(path_str)
ext = path.suffix.lower()
try:
total, test_bad = _inspect_archive(path, ext)
except Exception as exc: # noqa: BLE001 — clean rejection
q.put(("error", f"{type(exc).__name__}: {exc}"))
return
if total is not None and total > MAX_ARCHIVE_UNCOMPRESSED_BYTES:
gib = total / (1024 ** 3)
q.put(("error", f"uncompressed size {gib:.1f} GiB exceeds the bomb-guard cap"))
return
if test_bad is not None:
q.put(("error", f"integrity test failed at member {test_bad!r}"))
return
q.put(("ok", None))
def _inspect_archive(path: Path, ext: str):
"""Return (total_uncompressed_bytes | None, first_bad_member | None)
for the archive. Format-specific; raises on a structurally-broken
container (caught by the child as a clean rejection)."""
if ext in (".zip", ".cbz"):
import zipfile
with zipfile.ZipFile(path) as zf:
total = sum(zi.file_size for zi in zf.infolist())
return total, zf.testzip()
if ext == ".rar":
import rarfile
with rarfile.RarFile(path) as rf:
total = sum(getattr(ri, "file_size", 0) for ri in rf.infolist())
rf.testrar()
return total, None
if ext == ".7z":
import py7zr
with py7zr.SevenZipFile(path, "r") as zf:
info = zf.archiveinfo()
total = getattr(info, "uncompressed", None)
ok = zf.test() # True / None when all members pass
return total, (None if ok in (True, None) else "7z test reported corruption")
# Unknown extension — nothing to test; treat as clean.
return None, None
+93 -13
View File
@@ -1,13 +1,22 @@
"""Minimal gallery-dl sidecar parsing (one-time filesystem-import aid).
No per-platform branching: a small common key set with fallbacks; the
full JSON is kept in raw so anything unmapped is recoverable later.
Per-platform quirks (post_url synthesis, key-chain overrides) live in
the platforms registry — `backend/app/services/platforms/`. This module
is platform-agnostic: it looks up `category` in the sidecar and asks
the registry for the right behavior.
"""
import re
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from ..services.platforms import (
PLATFORMS,
description_keys_for,
external_post_id_keys_for,
)
@dataclass(frozen=True)
class SidecarData:
@@ -21,10 +30,28 @@ class SidecarData:
raw: dict
# gallery-dl prefixes media filenames with `NN_` for in-post ordering
# (`01_HOLLOW-ICHIGO.png`, `02_HOLOW ICHIGO.zip`) but writes the sidecar
# under the attachment's stem WITHOUT that ordering prefix
# (`HOLLOW-ICHIGO.json`). Strip the prefix when looking for sidecars.
# Confirmed against real Patreon downloads 2026-05-26 — without this
# strip, every gallery-dl post-level sidecar was invisible to FC since
# FC-3 shipped (24 deep-refresh calls produced 0 Posts in operator's DB).
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
def find_sidecar(media: Path) -> Path | None:
# Attachment-level sidecars (image.jpg.json, image.json).
for cand in (media.with_suffix(".json"), Path(str(media) + ".json")):
if cand.is_file():
return cand
# gallery-dl post-numbered convention: strip the `NN_` prefix from
# the stem and look for that.json in the same directory.
m = _NUMBERING_PREFIX.match(media.stem)
if m:
cand = media.parent / f"{m.group(1)}.json"
if cand.is_file():
return cand
return None
@@ -36,6 +63,46 @@ def _first_str(data: dict, keys: tuple[str, ...]) -> str | None:
return None
def _first_id(data: dict, keys: tuple[str, ...]) -> str | None:
"""Like `_first_str` but accepts ints and rejects bool (Python's
bool subclasses int, so a literal `"id": true` would otherwise
yield external_post_id="True")."""
for k in keys:
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
# Strip HTML tags + collapse whitespace + take the first non-empty line.
# Used to derive a display title from a body when the platform doesn't
# expose a separate title field (subscribestar posts always write
# `title: ""` and put the leading sentence inside `content` as HTML).
# Truncated to 120 chars with an ellipsis if longer — long enough to be
# meaningful in a feed, short enough to fit a row.
_TAG_RE = re.compile(r"<[^>]+>")
_WS_RE = re.compile(r"\s+")
def _first_line_text(body: str, limit: int = 120) -> str | None:
if not body:
return None
text = _TAG_RE.sub(" ", body)
text = text.replace("\xa0", " ")
# Split on hard line breaks first; the body-stripped HTML often
# collapses to one logical line, in which case the first sentence
# split is the next-best heuristic.
for line in text.splitlines():
line = _WS_RE.sub(" ", line).strip()
if line:
if len(line) > limit:
return line[: limit - 1].rstrip() + ""
return line
return None
def _parse_date(v) -> datetime | None:
if isinstance(v, bool):
return None
@@ -65,14 +132,7 @@ def parse_sidecar(data: dict) -> SidecarData:
cat = data.get("category")
platform = cat if isinstance(cat, str) and cat.strip() else None
external_post_id = None
for k in ("id", "post_id", "index", "message_id"):
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
external_post_id = str(v)
break
external_post_id = _first_id(data, external_post_id_keys_for(platform))
pc = data.get("page_count")
if isinstance(pc, bool):
@@ -92,12 +152,32 @@ def parse_sidecar(data: dict) -> SidecarData:
if post_date is not None:
break
description = _first_str(data, description_keys_for(platform))
# When `title` is empty (subscribestar always; sometimes elsewhere),
# synthesize from the description body's first non-empty text line.
# Patreon's explicit titles short-circuit the fallback.
post_title = _first_str(data, ("title",))
if post_title is None and description:
post_title = _first_line_text(description)
# post_url: ask the platform module to synthesize a permalink.
# When the platform registers a `derive_post_url`, it owns the
# field (the bare `url`/`post_url` value is a file CDN URL and
# must NEVER be persisted). When it doesn't register one, trust
# the sidecar's `url` (Patreon's case — real permalink).
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.derive_post_url is not None:
post_url = info.derive_post_url(data)
else:
post_url = _first_str(data, ("url", "post_url"))
return SidecarData(
platform=platform,
external_post_id=external_post_id,
post_url=_first_str(data, ("url", "post_url")),
post_title=_first_str(data, ("title",)),
description=_first_str(data, ("content", "description", "caption")),
post_url=post_url,
post_title=post_title,
description=description,
attachment_count=attachment_count,
post_date=post_date,
raw=data,
+46
View File
@@ -0,0 +1,46 @@
"""Parse the user-facing `kind:name` shortcut used by the add-tag input.
Mirrors IR's app/utils/tag_prefix.py. Tag.name in FC is stored bare;
the `kind:` prefix only exists as an input convention at user-facing
places (image-modal add-tag input, future bulk-add forms). The parser
is the single owner of the kind-string list — anything not in
KNOWN_KINDS keeps its colon as literal text.
"""
from __future__ import annotations
# Kinds the user can type as a prefix at the input boundary.
# Exclusions:
# - `general` is the default for un-prefixed input (never typed as prefix)
# - `archive`, `post` are system-managed
# - `artist` was retired in FC-2d-vii-c — artists are first-class
# entities (Artist row + ImageRecord.artist_id), browsed via the
# provenance axis rather than as tags. See project_provenance_separation.
# - `meta`, `rating` retired as user-typeable per operator 2026-05-26 —
# content classification only needs character/fandom/series.
KNOWN_KINDS: frozenset[str] = frozenset({
"character",
"fandom",
"series",
})
def parse_kind_prefix(raw: str) -> tuple[str | None, str]:
"""Split a raw user-typed tag string into (kind, name).
Returns (kind, name) where kind is lowercase canonical and in
KNOWN_KINDS, or (None, raw.strip()) if no recognized prefix is
present. `name` is always whitespace-stripped.
Examples:
parse_kind_prefix("character:Saber") -> ("character", "Saber")
parse_kind_prefix("Character:Saber") -> ("character", "Saber")
parse_kind_prefix("sunset") -> (None, "sunset")
parse_kind_prefix("http://example") -> (None, "http://example")
parse_kind_prefix("fandom: FSN ") -> ("fandom", "FSN")
"""
if ":" in raw:
prefix, rest = raw.split(":", 1)
if prefix.lower() in KNOWN_KINDS:
return prefix.lower(), rest.strip()
return None, raw.strip()
+25 -3
View File
@@ -38,11 +38,33 @@ function deduplicateCookies(cookies) {
function toNetscapeFormat(cookies) {
const lines = ['# Netscape HTTP Cookie File'];
for (const c of cookies) {
let domain = c.domain.replace(/^\.?www\./, '.');
if (!domain.startsWith('.')) domain = '.' + domain;
// Preserve the browser's actual scope. Earlier versions rewrote
// every cookie to a leading-dot subdomain-wide form, which broke
// gallery-dl's HF extractor: its `cookies.get(name,
// domain="www.hentai-foundry.com")` does EXACT domain matching and
// missed host-only PHPSESSID rewritten to `.hentai-foundry.com`.
// Operator-flagged 2026-05-27. Backend `_augment_cookies` covers
// the already-stored cookies; this fix is forward-compat for fresh
// captures.
//
// Cookie storage semantics (Firefox):
// c.hostOnly === true → cookie set without a Domain= attribute;
// applies to the exact host only.
// c.hostOnly === false → cookie set with Domain=X; applies to
// that domain and its subdomains.
//
// Netscape format:
// leading-dot domain + TRUE flag → subdomain-wide
// bare-host domain + FALSE flag → host-only
const hostOnly = c.hostOnly === true;
let domain = c.domain;
if (!hostOnly && !domain.startsWith('.')) {
domain = '.' + domain;
}
const subdomainFlag = hostOnly ? 'FALSE' : 'TRUE';
const secure = c.secure ? 'TRUE' : 'FALSE';
const expiration = c.expirationDate ? Math.floor(c.expirationDate) : 0;
lines.push([domain, 'TRUE', c.path || '/', secure, String(expiration), c.name, c.value].join('\t'));
lines.push([domain, subdomainFlag, c.path || '/', secure, String(expiration), c.name, c.value].join('\t'));
}
return lines.join('\n');
}
+6 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.0",
"version": "1.0.5",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
@@ -11,6 +11,11 @@
}
},
"content_security_policy": {
"_comment": "Override the MV3 default CSP to OMIT upgrade-insecure-requests. FC runs over plain HTTP per the homelab posture (feedback_homelab_http), and the default MV3 CSP would silently upgrade every fetch(http://curator.../...) to https:// and fail with NS_ERROR_GENERATE_FAILURE. Operator-flagged 2026-05-26 after the 'Test connection' button errored despite a working CORS preflight on the backend.",
"extension_pages": "script-src 'self'; object-src 'self';"
},
"permissions": [
"cookies",
"storage",
+5 -5
View File
@@ -1,13 +1,13 @@
{
"name": "fabledcurator-extension",
"version": "1.0.0",
"version": "1.0.5",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
"lint": "web-ext lint --source-dir=.",
"start": "web-ext run --source-dir=. --firefox=firefox",
"build": "web-ext build --source-dir=. --overwrite-dest",
"sign": "web-ext sign --source-dir=. --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore",
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --firefox=firefox",
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --overwrite-dest",
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
},
"devDependencies": {
"web-ext": "^8.0.0"
-13
View File
@@ -1,13 +0,0 @@
module.exports = {
sourceDir: '.',
artifactsDir: './web-ext-artifacts',
ignoreFiles: [
'package.json',
'package-lock.json',
'web-ext-config.cjs',
'web-ext-artifacts',
'node_modules',
'README.md',
'.gitignore',
],
};
@@ -0,0 +1,95 @@
<template>
<v-card class="fc-danger-zone mt-8" variant="outlined">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-alert-octagon" color="error" size="small" />
<span>Danger zone</span>
</v-card-title>
<v-card-text>
<p class="text-body-2 fc-muted mb-4">
Cascade-delete this artist and every image, source, post, and
attachment associated with them. This cannot be undone.
Recoverable only from an FC-3h backup.
</p>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-forever"
:loading="loading"
@click="onClick"
>Delete artist &amp; cascade</v-btn>
<DestructiveConfirmModal
v-model="modalOpen"
action="delete"
kind="artist"
:run-id="artistId"
tier="C"
:projected-counts="projectedCounts"
:description="modalDescription"
@confirm="onConfirm"
/>
</v-card-text>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useRouter } from 'vue-router'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useAdminStore } from '../../stores/admin.js'
const props = defineProps({
slug: { type: String, required: true },
artistId: { type: Number, required: true },
artistName: { type: String, required: true },
})
const router = useRouter()
const store = useAdminStore()
const loading = ref(false)
const modalOpen = ref(false)
const projected = ref(null)
const projectedCounts = computed(() => projected.value?.projected || null)
const modalDescription = computed(
() => projected.value
? `Artist “${props.artistName}” — `
+ `${projected.value.projected.images} images, `
+ `${projected.value.projected.sources} sources, `
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
async function onClick() {
loading.value = true
try {
projected.value = await store.projectArtistCascade(props.slug)
modalOpen.value = true
} finally {
loading.value = false
}
}
async function onConfirm(token) {
loading.value = true
try {
const result = await store.dispatchArtistCascade(props.slug, token)
const taskId = result.task_id
router.push('/artists')
if (taskId) {
store.pollTaskUntilDone(taskId).catch(() => {})
}
} finally {
loading.value = false
}
}
</script>
<style scoped>
.fc-danger-zone {
border-color: rgb(var(--v-theme-error));
border-radius: 8px;
}
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,48 @@
<template>
<div class="fc-artist-gallery">
<MasonryGrid
:items="store.images"
:loading="store.imagesLoading"
:has-more="store.hasMoreImages"
@load-more="store.loadMoreImages(props.slug)"
@open="openImage"
/>
</div>
</template>
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../../stores/artist.js'
import { useModalStore } from '../../stores/modal.js'
import MasonryGrid from '../discovery/MasonryGrid.vue'
const props = defineProps({
slug: { type: String, required: true },
})
const store = useArtistStore()
const modal = useModalStore()
const route = useRoute()
const router = useRouter()
onMounted(() => {
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage (id) {
router.push({ query: { ...route.query, image: id } })
}
</script>
<style scoped>
.fc-artist-gallery { min-width: 0; }
</style>
@@ -0,0 +1,128 @@
<template>
<header class="fc-artist-header">
<div class="fc-artist-header__left">
<h1 class="fc-artist-header__name">{{ name }}</h1>
<span v-if="stats" class="fc-artist-header__stats">{{ stats }}</span>
</div>
<v-tabs
:model-value="modelValue"
color="accent"
density="compact"
class="fc-artist-header__tabs"
@update:model-value="$emit('update:modelValue', $event)"
>
<v-tab value="posts">
Posts
<span v-if="postCount != null" class="fc-artist-header__tab-count">
({{ postCount }})
</span>
</v-tab>
<v-tab value="gallery">
Gallery
<span v-if="imageCount != null" class="fc-artist-header__tab-count">
({{ imageCount }})
</span>
</v-tab>
<v-tab value="management">Management</v-tab>
</v-tabs>
<!-- Right-side spacer: balances the left cell's flex weight so the
centered tabs stay geometrically centered regardless of the
artist-name length. Mirrors TopNav's 1fr | auto | 1fr layout. -->
<div class="fc-artist-header__right" />
</header>
</template>
<script setup>
import { computed } from 'vue'
const props = defineProps({
name: { type: String, required: true },
imageCount: { type: Number, default: null },
postCount: { type: Number, default: null },
lastAdded: { type: String, default: null },
modelValue: { type: String, required: true },
})
defineEmits(['update:modelValue'])
const stats = computed(() => {
const parts = []
if (props.imageCount != null) {
parts.push(`${props.imageCount} image${props.imageCount === 1 ? '' : 's'}`)
}
if (props.lastAdded) {
parts.push(`last added ${props.lastAdded.slice(0, 10)}`)
}
return parts.join(' · ')
})
</script>
<style scoped>
/* Matches TopNav.vue's frosted recipe exactly. top:48px parks it flush
against TopNav's bottom edge (TopNav is 0.75rem padding + ~24px content
= ~48px tall; operator-flagged 2026-05-26 that top:64px left a visible
gap). The two bars now read as one continuous frosted strip. */
.fc-artist-header {
position: sticky;
top: 48px;
z-index: 4;
display: flex;
align-items: center;
gap: 1rem;
padding: 0.5rem 1rem;
background: linear-gradient(
to bottom,
rgba(20, 23, 26, 0.92) 0%,
rgba(20, 23, 26, 0.65) 60%,
rgba(20, 23, 26, 0) 100%
);
backdrop-filter: blur(2px);
-webkit-backdrop-filter: blur(2px);
}
.fc-artist-header__left {
flex: 1 1 0;
min-width: 0;
display: flex;
align-items: baseline;
gap: 12px;
overflow: hidden;
}
.fc-artist-header__name {
font-family: 'Fraunces', Georgia, serif;
font-size: 24px;
font-weight: 500;
margin: 0;
color: rgb(var(--v-theme-on-surface));
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.fc-artist-header__stats {
font-size: 13px;
color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.fc-artist-header__tabs {
flex: 0 0 auto;
}
.fc-artist-header__right {
flex: 1 1 0;
min-width: 0;
}
.fc-artist-header__tab-count {
margin-left: 4px;
font-size: 12px;
color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
</style>
@@ -0,0 +1,133 @@
<template>
<div class="fc-artist-mgmt">
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Overview</h2>
<div class="fc-artist-mgmt__chips">
<v-chip
size="small"
:variant="overview.is_subscription ? 'flat' : 'outlined'"
:color="overview.is_subscription ? 'accent' : undefined"
prepend-icon="mdi-rss"
>{{ overview.is_subscription ? 'Subscription' : 'One-off' }}</v-chip>
<v-chip
size="small" variant="outlined" prepend-icon="mdi-link-variant"
:to="`/subscriptions?artist_id=${overview.id}`"
>{{ overview.sources.length }} subscription{{ overview.sources.length === 1 ? '' : 's' }}</v-chip>
</div>
</section>
<section v-if="overview.cooccurring_tags.length" class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Frequent tags</h2>
<div class="fc-artist-mgmt__tags">
<v-chip
v-for="t in overview.cooccurring_tags" :key="t.id"
size="small" @click="openTag(t.id)"
>{{ t.name }} <span class="fc-artist-mgmt__tagc">{{ t.count }}</span></v-chip>
</div>
</section>
<section v-if="overview.activity.length" class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Activity</h2>
<svg
class="fc-artist-mgmt__spark" :viewBox="`0 0 ${sparkW} ${sparkH}`"
preserveAspectRatio="none" role="img" aria-label="posts over time"
>
<polyline :points="sparkPoints" fill="none"
stroke="rgb(var(--v-theme-accent))" stroke-width="2" />
</svg>
</section>
<section v-if="overview.sources.length" class="fc-artist-mgmt__sec">
<div class="fc-artist-mgmt__sec-head">
<h2 class="fc-h2">Subscriptions</h2>
<RouterLink
:to="`/subscriptions?artist_id=${overview.id}`"
class="fc-artist-mgmt__manage"
>Manage subscriptions </RouterLink>
</div>
<v-table density="compact">
<thead>
<tr><th>Platform</th><th>URL</th><th class="text-right">Images</th></tr>
</thead>
<tbody>
<tr v-for="s in overview.sources" :key="s.id">
<td>{{ s.platform }}</td>
<td class="fc-artist-mgmt__url">{{ s.url }}</td>
<td class="text-right">{{ s.image_count }}</td>
</tr>
</tbody>
</v-table>
</section>
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Danger zone</h2>
<ArtistDangerZone
:slug="overview.slug"
:artist-id="overview.id"
:artist-name="overview.name"
/>
</section>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { useRouter, RouterLink } from 'vue-router'
import ArtistDangerZone from './ArtistDangerZone.vue'
const props = defineProps({
overview: { type: Object, required: true },
})
const router = useRouter()
const sparkW = 600
const sparkH = 80
const sparkPoints = computed(() => {
const a = props.overview.activity ?? []
if (a.length === 0) return ''
const max = Math.max(...a.map(p => p.count), 1)
const stepX = a.length > 1 ? sparkW / (a.length - 1) : 0
return a.map((p, i) => {
const x = i * stepX
const y = sparkH - (p.count / max) * (sparkH - 4) - 2
return `${x.toFixed(1)},${y.toFixed(1)}`
}).join(' ')
})
function openTag (tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } })
}
</script>
<style scoped>
.fc-artist-mgmt { padding-top: 1rem; }
.fc-h2 {
font-family: 'Fraunces', Georgia, serif;
font-size: 20px; font-weight: 500; margin-bottom: 8px;
}
.fc-artist-mgmt__sec { margin-bottom: 28px; }
.fc-artist-mgmt__chips { display: flex; gap: 8px; flex-wrap: wrap; }
.fc-artist-mgmt__tags { display: flex; flex-wrap: wrap; gap: 6px; }
.fc-artist-mgmt__tagc {
color: rgb(var(--v-theme-on-surface-variant));
margin-left: 4px;
font-variant-numeric: tabular-nums;
}
.fc-artist-mgmt__spark { width: 100%; height: 80px; }
.fc-artist-mgmt__url {
max-width: 380px; overflow: hidden; text-overflow: ellipsis;
white-space: nowrap;
}
.fc-artist-mgmt__sec-head {
display: flex; align-items: baseline; justify-content: space-between;
margin-bottom: 0.25rem;
}
.fc-artist-mgmt__manage {
font-size: 0.85rem;
color: rgb(var(--v-theme-accent));
text-decoration: none;
}
.fc-artist-mgmt__manage:hover { text-decoration: underline; }
</style>
@@ -0,0 +1,90 @@
<template>
<div class="fc-artist-posts">
<v-alert v-if="store.error" type="error" variant="tonal" closable class="mb-3">
{{ String(store.error) }}
</v-alert>
<div v-if="store.loading && store.items.length === 0" class="fc-artist-posts__loading">
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<div v-else-if="store.items.length === 0 && store.done" class="fc-artist-posts__empty">
<p>No posts for this artist yet. Switch to
<a href="#" @click.prevent="$emit('switch-tab', 'gallery')">Gallery</a>
to see imported images, or visit
<RouterLink to="/subscriptions">Subscriptions</RouterLink>
to start capturing posts.
</p>
</div>
<div v-else>
<PostCard v-for="p in store.items" :key="p.id" :post="p" />
<div ref="sentinel" class="fc-artist-posts__sentinel">
<v-progress-circular v-if="store.loading" indeterminate color="accent" size="24" />
<span v-else-if="store.done" class="fc-artist-posts__end">End of stream</span>
</div>
</div>
</div>
</template>
<script setup>
import { onMounted, onUnmounted, ref, watch } from 'vue'
import { RouterLink } from 'vue-router'
import { usePostsStore } from '../../stores/posts.js'
import PostCard from '../posts/PostCard.vue'
const props = defineProps({
artistId: { type: Number, required: true },
})
defineEmits(['switch-tab'])
const store = usePostsStore()
const sentinel = ref(null)
let observer = null
async function reload () {
await store.loadInitial({ artist_id: props.artistId, platform: null })
}
watch(() => props.artistId, reload)
onMounted(async () => {
await reload()
observer = new IntersectionObserver((entries) => {
if (entries.some(e => e.isIntersecting)) {
store.loadMore()
}
}, { rootMargin: '400px 0px' })
if (sentinel.value) observer.observe(sentinel.value)
})
onUnmounted(() => {
if (observer) observer.disconnect()
})
</script>
<style scoped>
.fc-artist-posts {
max-width: 1600px;
margin: 0 auto;
}
.fc-artist-posts__loading,
.fc-artist-posts__empty {
display: flex;
justify-content: center;
padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-artist-posts__sentinel {
display: flex;
justify-content: center;
padding: 1.5rem 0;
min-height: 2rem;
}
.fc-artist-posts__end {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem;
}
</style>
@@ -0,0 +1,124 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-image-size-select-small" size="small" />
<span>Minimum dimensions</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Find and delete images smaller than the threshold. Mirrors the
import-time <code>min_width</code> / <code>min_height</code>
filter, applied retroactively to the existing library.
</p>
<v-row dense>
<v-col cols="6">
<v-text-field
v-model.number="minW" label="Min width (px)" type="number"
min="0" density="compact" hide-details
/>
</v-col>
<v-col cols="6">
<v-text-field
v-model.number="minH" label="Min height (px)" type="number"
min="0" density="compact" hide-details
/>
</v-col>
</v-row>
<div class="d-flex align-center mt-3" style="gap: 10px;">
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="busy"
@click="onPreview"
>Preview</v-btn>
<span v-if="preview" class="text-body-2">
<strong>{{ preview.count }}</strong> image(s) would be deleted.
</span>
</div>
<v-btn
v-if="preview && preview.count > 0"
class="mt-3"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onDeleteClick"
>Delete {{ preview.count }} matching...</v-btn>
</v-card-text>
<DestructiveConfirmModal
v-model="showModal"
action="delete"
kind="min-dim"
tier="C"
:expected-token-override="preview?.confirm_token || ''"
:projected-counts="projectedCounts"
:description="`Width < ${minW} OR height < ${minH}`"
@confirm="onConfirmedDelete"
/>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
// Backend's preview response hands the full Tier-C confirm token back
// as `confirm_token` (e.g. `delete-min-dim-1a2b3c4d`); passed straight
// to the modal via `expected-token-override`. We previously
// reconstructed via Web Crypto's SHA-256, but `crypto.subtle` is
// Secure-Context-gated and undefined on plain-HTTP origins, so the
// Delete button silently swallowed the TypeError. Operator-flagged
// 2026-05-27.
const store = useCleanupStore()
const minW = ref(0)
const minH = ref(0)
const preview = ref(null)
const busy = ref(false)
const showModal = ref(false)
const projectedCounts = ref({})
onMounted(async () => {
await store.loadDefaults()
minW.value = store.defaults.min_width
minH.value = store.defaults.min_height
})
async function onPreview() {
busy.value = true
try {
preview.value = await store.previewMinDim(minW.value, minH.value)
} catch (e) {
window.__fcToast?.({ text: `Preview failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
function onDeleteClick() {
projectedCounts.value = { 'Images to delete': preview.value.count }
showModal.value = true
}
async function onConfirmedDelete(token) {
try {
const res = await store.deleteMinDim(minW.value, minH.value, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
preview.value = null
} catch (e) {
window.__fcToast?.({ text: `Delete failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,183 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-palette-swatch" size="small" />
<span>Single-color audit</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Scan library for images dominated by one color within the
tolerance. Catches placeholder / solid-fill / error-page images
that slipped through the import filter. Same background-scan
cadence as the transparency audit.
</p>
<v-row dense>
<v-col cols="6">
<v-text-field
v-model.number="threshold" label="Threshold (01)"
type="number" min="0" max="1" step="0.01"
density="compact" hide-details
:disabled="audit && audit.status === 'running'"
/>
</v-col>
<v-col cols="6">
<v-text-field
v-model.number="tolerance" label="Color tolerance (0441)"
type="number" min="0" max="441"
density="compact" hide-details
:disabled="audit && audit.status === 'running'"
/>
</v-col>
</v-row>
<v-btn
v-if="!audit || audit.status !== 'running'"
class="mt-3"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify-scan"
:loading="busy"
@click="onStart"
>Scan library</v-btn>
<div v-if="audit && audit.status === 'running'" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 d-flex align-center" style="gap: 10px;">
<span>
Scanning {{ audit.scanned_count }} checked,
{{ audit.matched_count }} matched
</span>
<v-btn
variant="text" size="small" color="warning" rounded="pill"
@click="onCancel"
>Cancel</v-btn>
</div>
</div>
<div v-if="audit && audit.status === 'ready'" class="mt-3">
<p class="text-body-2 mb-2">
Scan complete. <strong>{{ audit.matched_count }}</strong>
image(s) match.
</p>
<v-btn
v-if="audit.matched_count > 0"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onApplyClick"
>Delete {{ audit.matched_count }} matching...</v-btn>
</div>
<v-alert
v-if="audit && audit.status === 'error'"
type="error" variant="tonal" density="compact" class="mt-3"
>Scan failed: {{ audit.error }}</v-alert>
<v-alert
v-if="audit && audit.status === 'applied'"
type="success" variant="tonal" density="compact" class="mt-3"
>Applied matched images deleted.</v-alert>
</v-card-text>
<DestructiveConfirmModal
v-if="audit"
v-model="showModal"
action="delete"
kind="audit"
:run-id="audit.id"
tier="C"
:projected-counts="projectedCounts"
description="Permanently deletes images matched by the single-color scan."
@confirm="onConfirmedApply"
/>
</v-card>
</template>
<script setup>
import { onMounted, onUnmounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
const store = useCleanupStore()
const threshold = ref(0.95)
const tolerance = ref(30)
const audit = ref(null)
const busy = ref(false)
const showModal = ref(false)
const projectedCounts = ref({})
let pollTimer = null
onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.single_color_threshold
tolerance.value = store.defaults.single_color_tolerance
})
onUnmounted(() => stopPoll())
function startPoll(id) {
stopPoll()
pollTimer = setInterval(async () => {
try {
const fresh = await store.getAudit(id)
audit.value = fresh
if (fresh.status !== 'running') stopPoll()
} catch (e) {
stopPoll()
window.__fcToast?.({ text: `Audit poll failed: ${e.message}`, type: 'error' })
}
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onStart() {
busy.value = true
try {
const res = await store.startAudit('single_color', {
threshold: threshold.value, tolerance: tolerance.value,
})
audit.value = await store.getAudit(res.audit_id)
startPoll(res.audit_id)
} catch (e) {
window.__fcToast?.({ text: `Scan start failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
async function onCancel() {
if (!audit.value) return
try {
await store.cancelAudit(audit.value.id)
audit.value = await store.getAudit(audit.value.id)
stopPoll()
} catch (e) {
window.__fcToast?.({ text: `Cancel failed: ${e.message}`, type: 'error' })
}
}
function onApplyClick() {
projectedCounts.value = { 'Images to delete': audit.value.matched_count }
showModal.value = true
}
async function onConfirmedApply(token) {
try {
const res = await store.applyAudit(audit.value.id, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
audit.value = await store.getAudit(audit.value.id)
} catch (e) {
window.__fcToast?.({ text: `Apply failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,166 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-checkerboard" size="small" />
<span>Transparency audit</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Scan library for images whose transparent-pixel fraction exceeds
the threshold. Animated WebPs / GIFs are skipped (the import-side
rule does the same). Runs as a background task ~50ms per image,
so a 57k library takes ~50 minutes.
</p>
<v-text-field
v-model.number="threshold" label="Transparency threshold (01)"
type="number" min="0" max="1" step="0.01" density="compact" hide-details
:disabled="audit && audit.status === 'running'"
class="mb-3"
/>
<v-btn
v-if="!audit || audit.status !== 'running'"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify-scan"
:loading="busy"
@click="onStart"
>Scan library</v-btn>
<div v-if="audit && audit.status === 'running'" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 d-flex align-center" style="gap: 10px;">
<span>
Scanning {{ audit.scanned_count }} checked,
{{ audit.matched_count }} matched
</span>
<v-btn
variant="text" size="small" color="warning" rounded="pill"
@click="onCancel"
>Cancel</v-btn>
</div>
</div>
<div v-if="audit && audit.status === 'ready'" class="mt-3">
<p class="text-body-2 mb-2">
Scan complete. <strong>{{ audit.matched_count }}</strong>
image(s) match.
</p>
<v-btn
v-if="audit.matched_count > 0"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onApplyClick"
>Delete {{ audit.matched_count }} matching...</v-btn>
</div>
<v-alert
v-if="audit && audit.status === 'error'"
type="error" variant="tonal" density="compact" class="mt-3"
>Scan failed: {{ audit.error }}</v-alert>
<v-alert
v-if="audit && audit.status === 'applied'"
type="success" variant="tonal" density="compact" class="mt-3"
>Applied matched images deleted.</v-alert>
</v-card-text>
<DestructiveConfirmModal
v-if="audit"
v-model="showModal"
action="delete"
kind="audit"
:run-id="audit.id"
tier="C"
:projected-counts="projectedCounts"
description="Permanently deletes images matched by the transparency scan."
@confirm="onConfirmedApply"
/>
</v-card>
</template>
<script setup>
import { onMounted, onUnmounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
const store = useCleanupStore()
const threshold = ref(0.9)
const audit = ref(null)
const busy = ref(false)
const showModal = ref(false)
const projectedCounts = ref({})
let pollTimer = null
onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.transparency_threshold
})
onUnmounted(() => stopPoll())
function startPoll(id) {
stopPoll()
pollTimer = setInterval(async () => {
try {
const fresh = await store.getAudit(id)
audit.value = fresh
if (fresh.status !== 'running') stopPoll()
} catch (e) {
stopPoll()
window.__fcToast?.({ text: `Audit poll failed: ${e.message}`, type: 'error' })
}
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onStart() {
busy.value = true
try {
const res = await store.startAudit('transparency', { threshold: threshold.value })
audit.value = await store.getAudit(res.audit_id)
startPoll(res.audit_id)
} catch (e) {
window.__fcToast?.({ text: `Scan start failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
async function onCancel() {
if (!audit.value) return
try {
await store.cancelAudit(audit.value.id)
audit.value = await store.getAudit(audit.value.id)
stopPoll()
} catch (e) {
window.__fcToast?.({ text: `Cancel failed: ${e.message}`, type: 'error' })
}
}
function onApplyClick() {
projectedCounts.value = { 'Images to delete': audit.value.matched_count }
showModal.value = true
}
async function onConfirmedApply(token) {
try {
const res = await store.applyAudit(audit.value.id, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
audit.value = await store.getAudit(audit.value.id)
} catch (e) {
window.__fcToast?.({ text: `Apply failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,172 @@
<template>
<v-dialog :model-value="modelValue" max-width="900"
@update:model-value="$emit('update:modelValue', $event)">
<v-card>
<v-card-title class="d-flex align-center" style="gap: 12px;">
<v-icon icon="mdi-alert-circle-outline" color="error" />
<span>{{ displayTitle }}</span>
<v-spacer />
<v-btn icon variant="text" size="small" @click="close">
<v-icon>mdi-close</v-icon>
</v-btn>
</v-card-title>
<v-card-text>
<dl v-if="contextRows.length" class="fc-err-context">
<template v-for="(row, idx) in contextRows" :key="idx">
<dt>{{ row[0] }}</dt>
<dd>{{ row[1] }}</dd>
</template>
</dl>
<pre class="fc-err-pre">{{ displayMessage || '(no error message)' }}</pre>
</v-card-text>
<v-card-actions>
<v-btn
variant="text" rounded="pill" size="small"
:prepend-icon="copied ? 'mdi-check' : 'mdi-content-copy'"
@click="onCopy"
>{{ copied ? 'Copied' : 'Copy' }}</v-btn>
<v-spacer />
<v-btn variant="text" rounded="pill" @click="close">Close</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { copyText } from '../../utils/clipboard.js'
const props = defineProps({
modelValue: { type: Boolean, default: false },
// Legacy mode: pass title + message strings. Used by callers whose row
// shape lacks structured context (e.g. ImportTaskList where `error` is
// a plain string on the import_task row, not a TaskRun).
title: { type: String, default: 'Error details' },
message: { type: String, default: '' },
// Row mode: pass the full TaskRun-shaped dict from /api/system_activity.
// When set, displayTitle/displayMessage derive from the row and a context
// panel of task_name/queue/target/duration/etc. renders above the error.
row: { type: Object, default: null },
})
const emit = defineEmits(['update:modelValue'])
const copied = ref(false)
let copiedTimer = null
const displayTitle = computed(() => {
if (props.row) return props.row.error_type || 'Error details'
return props.title
})
const displayMessage = computed(() => {
if (props.row) return props.row.error_message || ''
return props.message
})
function _shortTaskName (name) {
if (!name) return ''
const parts = String(name).split('.')
return parts[parts.length - 1]
}
function _formatDuration (ms) {
if (ms == null) return null
if (ms < 1000) return `${ms} ms`
if (ms < 60_000) return `${(ms / 1000).toFixed(1)} s`
return `${(ms / 60_000).toFixed(1)} min`
}
const contextRows = computed(() => {
const r = props.row
if (!r) return []
const rows = []
if (r.task_name) rows.push(['Task', _shortTaskName(r.task_name)])
if (r.queue) rows.push(['Queue', r.queue])
if (r.target_id != null) rows.push(['Target', r.target_id])
const dur = _formatDuration(r.duration_ms)
if (dur != null) rows.push(['Duration', dur])
if (r.started_at) rows.push(['Started', r.started_at])
if (r.finished_at) rows.push(['Finished', r.finished_at])
if (r.retry_count) rows.push(['Retries', r.retry_count])
if (r.worker_hostname) rows.push(['Worker', r.worker_hostname])
if (r.celery_task_id) rows.push(['Celery ID', r.celery_task_id])
if (r.args_summary) rows.push(['Args', r.args_summary])
return rows
})
watch(() => props.modelValue, (open) => {
if (!open) {
copied.value = false
if (copiedTimer) { clearTimeout(copiedTimer); copiedTimer = null }
}
})
function close () {
emit('update:modelValue', false)
}
async function onCopy () {
let text = displayMessage.value || ''
if (contextRows.value.length) {
const header = contextRows.value.map(([k, v]) => `${k}: ${v}`).join('\n')
text = `${header}\n\nError: ${displayTitle.value}\n${text}`
}
try {
await copyText(text)
copied.value = true
if (copiedTimer) clearTimeout(copiedTimer)
copiedTimer = setTimeout(() => { copied.value = false }, 1500)
} catch (e) {
window.__fcToast?.({ text: `Copy failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
/* Context panel: muted labels in vellum, crisp values in parchment. The
2-column key/value grid keeps rows visually scannable when there are
many fields (Task, Queue, Target, Duration, Started, Finished, Retries,
Worker, Celery ID, Args). */
.fc-err-context {
display: grid;
grid-template-columns: max-content 1fr;
column-gap: 14px;
row-gap: 4px;
margin: 0 0 14px 0;
font-size: 13px;
}
.fc-err-context dt {
color: rgb(var(--v-theme-on-surface-variant));
font-weight: 500;
white-space: nowrap;
}
.fc-err-context dd {
color: rgb(var(--v-theme-on-surface));
margin: 0;
font-variant-numeric: tabular-nums;
word-break: break-all;
}
/* Error pre block: high-contrast pairing. The page's `background` token
(obsidian #14171A) is darker than the modal card's `surface` (iron
#1E2228), so parchment text reads crisply against it. The prior pairing
used `surface-variant` which Vuetify auto-derives to a near-parchment
light value in this theme — pale-on-pale and unreadable. Operator-
flagged 2026-05-26 ("ui contrast is poor"). */
.fc-err-pre {
font-family: 'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 12px;
line-height: 1.5;
white-space: pre-wrap;
word-break: break-word;
background: rgb(var(--v-theme-background));
color: rgb(var(--v-theme-on-surface));
padding: 12px 14px;
border-radius: 6px;
max-height: 60vh;
overflow: auto;
margin: 0;
}
</style>
@@ -28,6 +28,7 @@
<script setup>
import { onMounted, ref } from 'vue'
import { useCredentialsStore } from '../../stores/credentials.js'
import { copyText } from '../../utils/clipboard.js'
const store = useCredentialsStore()
const showRotateConfirm = ref(false)
@@ -37,7 +38,7 @@ onMounted(() => store.loadKey())
async function copyKey() {
if (!store.extensionKey) return
try {
await navigator.clipboard.writeText(store.extensionKey)
await copyText(store.extensionKey)
globalThis.window?.__fcToast?.({ text: 'Copied', type: 'success' })
} catch {
globalThis.window?.__fcToast?.({ text: 'Copy failed', type: 'error' })
@@ -38,9 +38,18 @@ function onCardClick() {
.fc-artistcard { cursor: pointer; }
.fc-artistcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute. */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-artistcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-artistcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-artistcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
@@ -4,7 +4,10 @@
<div v-for="(col, ci) in columns" :key="ci" class="fc-masonry__col">
<button
v-for="item in col" :key="item.id"
class="fc-masonry__item" type="button"
class="fc-masonry__item"
:class="{ 'fc-masonry__item--anim': shouldAnimate(item) }"
:style="itemStyle(item)"
type="button"
@click="$emit('open', item.id)"
>
<img
@@ -33,7 +36,13 @@ import { usePolyMasonry } from '../../composables/usePolyMasonry.js'
const props = defineProps({
items: { type: Array, default: () => [] },
loading: { type: Boolean, default: false },
hasMore: { type: Boolean, default: false }
hasMore: { type: Boolean, default: false },
// Items at indices >= animateFromIndex get the stagger fade-in. Opt-in
// — defaults to Infinity (no animation) so views that don't want it
// (ArtistView, etc.) don't pay the layout-shift cost. ShowcaseView
// uses 0 on initial load / shuffle and prevCount on infinite-scroll
// appends.
animateFromIndex: { type: Number, default: Number.POSITIVE_INFINITY },
})
const emit = defineEmits(['load-more', 'open'])
@@ -43,6 +52,25 @@ const { columnCount, distribute } = usePolyMasonry(containerEl)
const columns = computed(() => distribute(props.items, columnCount.value))
// id → index lookup so we can derive the stagger from natural reading
// order even after the masonry distributes items across columns.
const idxById = computed(() => {
const m = new Map()
props.items.forEach((it, i) => m.set(it.id, i))
return m
})
function shouldAnimate(item) {
const idx = idxById.value.get(item.id)
return idx !== undefined && idx >= props.animateFromIndex
}
function itemStyle(item) {
if (!shouldAnimate(item)) return {}
const idx = idxById.value.get(item.id) - props.animateFromIndex
return { '--stagger-index': idx }
}
function aspectStyle(item) {
const w = Number(item.width)
const h = Number(item.height)
@@ -81,4 +109,23 @@ onUnmounted(() => observer && observer.disconnect())
display: flex; justify-content: center; padding: 32px 0; min-height: 60px;
}
.fc-masonry__end { text-align: center; padding: 32px 0; }
/* IR-parity stagger fade-in for showcase entry / shuffle. 60ms between
items, 250ms each — matches IR's `itemFadeIn` keyframe (style.css
~line 1834). Honors prefers-reduced-motion. */
@keyframes fc-masonry-item-in {
from { opacity: 0; transform: translateY(12px); }
to { opacity: 1; transform: translateY(0); }
}
.fc-masonry__item--anim {
animation: fc-masonry-item-in 0.25s ease forwards;
animation-delay: calc(var(--stagger-index, 0) * 60ms);
opacity: 0;
}
@media (prefers-reduced-motion: reduce) {
.fc-masonry__item--anim {
animation: none;
opacity: 1;
}
}
</style>
+47 -4
View File
@@ -44,7 +44,31 @@
</div>
<div class="fc-tagcard__meta">
<v-chip size="x-small" label>{{ card.kind }}</v-chip>
<span class="fc-tagcard__count">{{ card.image_count }}</span>
<div class="fc-tagcard__meta-right">
<span class="fc-tagcard__count">{{ card.image_count }}</span>
<v-menu>
<template #activator="{ props: act }">
<v-btn
class="fc-tagcard__menu"
icon="mdi-dots-vertical" size="x-small" variant="text"
v-bind="act" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item
title="Merge with…"
prepend-icon="mdi-call-merge"
@click="$emit('merge-with', card)"
/>
<v-list-item
title="Delete tag"
prepend-icon="mdi-delete"
base-color="error"
@click="$emit('delete', card)"
/>
</v-list>
</v-menu>
</div>
</div>
</v-card-text>
</v-card>
@@ -54,7 +78,7 @@
import { ref } from 'vue'
const props = defineProps({ card: { type: Object, required: true } })
const emit = defineEmits(['open', 'rename', 'manage', 'read'])
const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
const editing = ref(false)
const draft = ref('')
@@ -82,9 +106,19 @@ function submit() {
.fc-tagcard { cursor: pointer; }
.fc-tagcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute
(older Safari, embedded webviews). */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-tagcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-tagcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-tagcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
@@ -106,4 +140,13 @@ function submit() {
}
.fc-tagcard:hover .fc-tagcard__edit { opacity: .6; }
.fc-tagcard__edit:hover { opacity: 1; }
.fc-tagcard__meta-right {
display: flex; align-items: center; gap: 4px;
}
.fc-tagcard__menu {
opacity: 0;
transition: opacity .15s ease;
}
.fc-tagcard:hover .fc-tagcard__menu { opacity: .6; }
.fc-tagcard__menu:hover { opacity: 1; }
</style>
@@ -1,47 +1,118 @@
<template>
<div class="fc-dl-row" @click="$emit('open', event.id)">
<v-icon :icon="statusIcon" :color="statusColor" size="small" />
<div
class="fc-dl-row"
:class="[`fc-dl-row--${event.status || 'unknown'}`]"
@click="$emit('open', event.id)"
>
<!-- Colored left edge marks the run's status; matches the row's
status-chip color but reads at a glance without needing to
parse the chip text. -->
<div class="fc-dl-row__bar" />
<v-chip
:color="statusColor"
size="small"
variant="tonal"
:prepend-icon="statusIcon"
class="fc-dl-row__status"
>{{ statusLabel }}</v-chip>
<RouterLink
v-if="event.artist_slug"
:to="`/artist/${event.artist_slug}`"
class="fc-dl-row__artist"
@click.stop
>{{ event.artist_name }}</RouterLink>
<span v-else class="fc-dl-row__artist"></span>
<v-chip size="x-small" variant="tonal">{{ event.platform || '—' }}</v-chip>
<span class="fc-dl-row__time">{{ fmtTime(event.started_at) }}</span>
<span class="fc-dl-row__files">{{ event.files_count }} files</span>
<span class="fc-dl-row__duration">{{ fmtDuration(event.summary?.duration_seconds) }}</span>
<span v-if="event.error" class="fc-dl-row__error">{{ event.error }}</span>
<span v-else class="fc-dl-row__artist fc-dl-row__artist--missing"></span>
<PlatformChip
v-if="event.platform"
:platform="event.platform"
size="x-small"
class="fc-dl-row__platform"
/>
<span v-else class="fc-dl-row__platform-missing"></span>
<span class="fc-dl-row__time" :title="event.started_at">
{{ fmtTime(event.started_at) }}
</span>
<v-chip
v-if="event.files_count > 0"
size="x-small" variant="tonal" color="info"
prepend-icon="mdi-image-multiple"
class="fc-dl-row__files"
>{{ event.files_count }}</v-chip>
<span v-else class="fc-dl-row__no-files" aria-label="no new files">·</span>
<span class="fc-dl-row__duration">
{{ fmtDuration(event.summary?.duration_seconds) }}
</span>
<v-chip
v-if="event.error"
color="error" size="x-small" variant="tonal"
prepend-icon="mdi-alert-octagon"
class="fc-dl-row__error"
:title="event.error"
>{{ truncateError(event.error) }}</v-chip>
<span v-else class="fc-dl-row__error-spacer" />
<div class="fc-dl-row__actions" @click.stop>
<v-btn
v-if="event.status === 'error' && event.source_id"
icon size="x-small" variant="text" color="warning"
:loading="retrying"
@click.stop="onRetry"
>
<v-icon size="small">mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Retry source check</v-tooltip>
</v-btn>
<v-btn icon size="x-small" variant="text" @click.stop="$emit('open', event.id)">
<v-icon size="small">mdi-information-outline</v-icon>
<v-tooltip activator="parent" location="top">Details</v-tooltip>
</v-btn>
<v-btn
v-if="event.artist_slug"
icon size="x-small" variant="text"
:to="`/artist/${event.artist_slug}`"
@click.stop
>
<v-icon size="small">mdi-account-circle</v-icon>
<v-tooltip activator="parent" location="top">Open artist</v-tooltip>
</v-btn>
</div>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { computed, ref } from 'vue'
import { RouterLink } from 'vue-router'
import PlatformChip from '../subscriptions/PlatformChip.vue'
import { useSourcesStore } from '../../stores/sources.js'
const props = defineProps({ event: { type: Object, required: true } })
defineEmits(['open'])
const statusIcon = computed(() => ({
ok: 'mdi-check-circle',
error: 'mdi-alert-circle',
running: 'mdi-progress-clock',
pending: 'mdi-clock-outline',
skipped: 'mdi-minus-circle',
}[props.event.status] || 'mdi-help-circle'))
const sourcesStore = useSourcesStore()
const retrying = ref(false)
const statusColor = computed(() => ({
ok: 'success',
error: 'error',
running: 'info',
pending: 'secondary',
skipped: 'warning',
}[props.event.status] || undefined))
const _STATUS = {
ok: { color: 'success', icon: 'mdi-check-circle', label: 'Completed' },
error: { color: 'error', icon: 'mdi-alert-circle', label: 'Failed' },
running: { color: 'info', icon: 'mdi-progress-clock', label: 'Running' },
pending: { color: 'grey', icon: 'mdi-clock-outline', label: 'Queued' },
skipped: { color: 'warning', icon: 'mdi-skip-next', label: 'Skipped' },
}
const statusColor = computed(() => _STATUS[props.event.status]?.color || 'grey')
const statusIcon = computed(() => _STATUS[props.event.status]?.icon || 'mdi-help-circle')
const statusLabel = computed(() => _STATUS[props.event.status]?.label || props.event.status)
function fmtTime(iso) {
if (!iso) return '—'
return iso.slice(0, 19).replace('T', ' ')
// 2026-05-27 23:36 — second granularity is in the row's title attr
return iso.slice(0, 16).replace('T', ' ')
}
function fmtDuration(sec) {
if (sec == null) return '—'
@@ -49,34 +120,107 @@ function fmtDuration(sec) {
const m = Math.floor(sec / 60), s = Math.floor(sec % 60)
return `${m}m ${s}s`
}
function truncateError(msg) {
const s = String(msg || '')
if (s.length <= 60) return s
return s.slice(0, 57) + '…'
}
async function onRetry() {
if (!props.event.source_id) return
retrying.value = true
try {
await sourcesStore.checkNow(props.event.source_id)
globalThis.window?.__fcToast?.({
text: `Source check re-queued`, type: 'success',
})
} catch (e) {
const isInFlight = !!e?.body?.download_event_id
globalThis.window?.__fcToast?.({
text: isInFlight ? 'Already running' : `Retry failed: ${e?.detail || e?.message || e}`,
type: isInFlight ? 'info' : 'error',
})
} finally {
retrying.value = false
}
}
</script>
<style scoped>
.fc-dl-row {
position: relative;
display: grid;
grid-template-columns: 24px 1fr 96px 160px 80px 80px 1fr;
gap: 0.75rem;
grid-template-columns:
/* bar */ 4px
/* status */ 120px
/* artist */ minmax(120px, 1.2fr)
/* plat */ 140px
/* time */ 140px
/* files */ 60px
/* dur */ 70px
/* error */ minmax(0, 1.5fr)
/* actions*/ 120px;
gap: 0.6rem;
align-items: center;
padding: 0.5rem 0.75rem;
border-bottom: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.18);
padding: 0.55rem 0.75rem 0.55rem 0;
border-bottom: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.15);
cursor: pointer;
transition: background 0.12s ease;
}
.fc-dl-row:hover { background: rgb(var(--v-theme-surface) / 0.5); }
.fc-dl-row:hover {
background: rgb(var(--v-theme-on-surface) / 0.04);
}
.fc-dl-row__bar {
width: 4px;
align-self: stretch;
border-radius: 0 2px 2px 0;
}
.fc-dl-row--ok .fc-dl-row__bar { background: rgb(var(--v-theme-success)); }
.fc-dl-row--error .fc-dl-row__bar { background: rgb(var(--v-theme-error)); }
.fc-dl-row--running .fc-dl-row__bar { background: rgb(var(--v-theme-info)); }
.fc-dl-row--skipped .fc-dl-row__bar { background: rgb(var(--v-theme-warning)); }
.fc-dl-row--pending .fc-dl-row__bar { background: rgb(var(--v-theme-on-surface-variant) / 0.4); }
.fc-dl-row__status { justify-self: start; }
.fc-dl-row__artist {
color: rgb(var(--v-theme-on-surface));
text-decoration: none;
font-weight: 500;
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
}
.fc-dl-row__artist--missing,
.fc-dl-row__platform-missing,
.fc-dl-row__no-files {
color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.5;
}
.fc-dl-row__artist:hover { color: rgb(var(--v-theme-accent)); }
.fc-dl-row__time, .fc-dl-row__files, .fc-dl-row__duration {
.fc-dl-row__platform { justify-self: start; }
.fc-dl-row__time,
.fc-dl-row__duration {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem;
font-variant-numeric: tabular-nums;
}
.fc-dl-row__error {
color: rgb(var(--v-theme-error));
font-size: 0.85rem;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.fc-dl-row__no-files {
text-align: center;
font-size: 1.1rem;
}
.fc-dl-row__error {
justify-self: start;
max-width: 100%;
}
.fc-dl-row__error :deep(.v-chip__content) {
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
}
.fc-dl-row__error-spacer { /* keeps the grid column reserved */ }
.fc-dl-row__actions {
display: flex; gap: 2px;
justify-self: end;
opacity: 0.5;
transition: opacity 0.12s ease;
}
.fc-dl-row:hover .fc-dl-row__actions { opacity: 1; }
</style>
@@ -55,19 +55,44 @@
</template>
</div>
<div class="fc-bulk-panel__section">
<h4>Destructive</h4>
<v-btn
color="error" variant="flat" rounded="pill" block
prepend-icon="mdi-delete-forever"
:disabled="!sel.count"
:loading="deleting"
@click="onDeleteClick"
>Delete {{ sel.count }} selected</v-btn>
</div>
<div class="fc-bulk-panel__foot">
<v-btn variant="text" block @click="sel.clear()">Clear selection</v-btn>
</div>
<DestructiveConfirmModal
v-model="deleteModalOpen"
action="delete"
kind="images-selection"
:expected-token-override="bulkProjected?.confirm_token || ''"
tier="C"
:projected-counts="bulkProjectedCounts"
:description="bulkDescription"
@confirm="onDeleteConfirm"
/>
</aside>
</template>
<script setup>
import { ref, watch } from 'vue'
import { computed, ref, watch } from 'vue'
import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
import { useAdminStore } from '../../stores/admin.js'
import { useApi } from '../../composables/useApi.js'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
const sel = useGallerySelectionStore()
const api = useApi()
const adminStore = useAdminStore()
const addModel = ref(null)
const addHits = ref([])
@@ -99,6 +124,63 @@ async function onAddPick(id) {
watch(() => sel.order.length, () => {
if (sel.isSelectMode) sel.refresh()
})
// --- FC-3k bulk delete -----------------------------------------------
const deleting = ref(false)
const deleteModalOpen = ref(false)
const bulkProjected = ref(null)
const bulkProjectedCounts = computed(() => bulkProjected.value
? {
images: bulkProjected.value.images_found,
thumbnails: bulkProjected.value.thumbs_to_unlink,
bytes: bulkProjected.value.bytes_on_disk,
}
: null,
)
const bulkDescription = computed(
() => bulkProjected.value
? `${bulkProjected.value.images_found} images, `
+ `${Math.round(bulkProjected.value.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
// The dry-run response hands the canonical Tier-C confirm token back
// as `confirm_token` (e.g. `delete-images-1a2b3c4d`), passed straight
// to the modal via `expected-token-override`. We used to compute the
// hash client-side via `crypto.subtle.digest`, but (1) that's
// Secure-Context-gated and undefined on plain-HTTP origins
// (homelab posture), so the click silently threw TypeError and the
// modal never opened, and (2) the modal's `kind="images-selection"`
// produced `delete-images-selection-<sha8>` while the backend
// expected `delete-images-<sha8>` — so it never would have worked
// even on HTTPS. Operator-flagged 2026-05-27.
async function onDeleteClick() {
if (!sel.order.length) return
deleting.value = true
try {
bulkProjected.value = await adminStore.projectBulkImageDelete(sel.order)
deleteModalOpen.value = true
} finally {
deleting.value = false
}
}
async function onDeleteConfirm(token) {
deleting.value = true
try {
const result = await adminStore.dispatchBulkImageDelete(sel.order, token)
const taskId = result.task_id
if (taskId) {
adminStore.pollTaskUntilDone(taskId).catch(() => {})
}
sel.clear()
} finally {
deleting.value = false
}
}
</script>
<style scoped>
@@ -0,0 +1,138 @@
<template>
<v-dialog
:model-value="modelValue"
max-width="520" persistent
@update:model-value="$emit('update:modelValue', $event)"
>
<v-card>
<v-card-title>
{{ titleVerb }}
{{ kindLabel }}<span v-if="runId"> #{{ runId }}</span>
</v-card-title>
<v-card-text>
<v-alert
:type="alertType"
variant="tonal" density="compact" class="mb-3"
>
<strong>{{ warningText }}</strong>
<div v-if="description" class="mt-1">{{ description }}</div>
</v-alert>
<div v-if="projectedCounts" class="fc-counts mb-3">
<div v-for="(v, k) in projectedCounts" :key="k">
<span class="fc-counts-key">{{ k }}:</span>
<span class="fc-counts-val">{{ v }}</span>
</div>
</div>
<template v-if="tier === 'C'">
<div class="text-body-2 mb-2">Type the following to confirm:</div>
<div class="fc-token mb-3">{{ expectedToken }}</div>
<v-text-field
v-model="typed"
variant="outlined" density="compact" hide-details
autofocus
placeholder="paste the token above"
/>
</template>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="onCancel">Cancel</v-btn>
<v-btn
:color="confirmColor"
variant="flat" rounded="pill"
:disabled="!canConfirm"
@click="onConfirm"
>
{{ titleVerb }}
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
const props = defineProps({
modelValue: { type: Boolean, required: true },
action: { type: String, required: true }, // 'restore' | 'delete'
kind: { type: String, required: true }, // 'db' | 'images' | 'artist' | 'tag' | 'images-selection' | 'audit' | 'min-dim'
runId: { type: [Number, String], default: '' }, // numeric id or sha8 string
description: { type: String, default: '' },
tier: { type: String, default: 'C' }, // 'B' | 'C'
projectedCounts: { type: Object, default: null },
// Override the `${action}-${kind}-${runId}` token formula. Use when
// the backend computes the canonical confirm token (e.g. bulk-delete
// and min-dim cleanup both return `confirm_token` from their dry-run
// endpoints) and the UI's kind/runId would otherwise produce a
// mismatched string. Operator-flagged 2026-05-27 after the
// BulkEditor's kind="images-selection" produced
// `delete-images-selection-<sha8>` while the backend expected
// `delete-images-<sha8>`.
expectedTokenOverride: { type: String, default: '' },
})
const emit = defineEmits(['update:modelValue', 'confirm'])
const typed = ref('')
const expectedToken = computed(
() => props.expectedTokenOverride
|| `${props.action}-${props.kind}-${props.runId}`,
)
const titleVerb = computed(
() => props.action === 'restore' ? 'Restore' : 'Delete',
)
const kindLabel = computed(() => ({
db: 'database backup',
images: 'images backup',
artist: 'artist',
tag: 'tag',
'images-selection': 'image selection',
}[props.kind] || props.kind))
const alertType = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const confirmColor = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const warningText = computed(() => (
props.action === 'restore'
? 'This replaces current state with the backup. There is no undo.'
: 'This permanently deletes the listed items. Cannot be recovered.'
))
const canConfirm = computed(
() => props.tier === 'B'
? true
: typed.value === expectedToken.value,
)
watch(() => props.modelValue, (open) => {
if (open) typed.value = ''
})
function onCancel() {
emit('update:modelValue', false)
}
function onConfirm() {
emit('confirm', expectedToken.value)
emit('update:modelValue', false)
}
</script>
<style scoped>
.fc-token {
font-family: 'JetBrains Mono', monospace;
background: rgb(var(--v-theme-surface-light));
padding: 6px 10px; border-radius: 4px;
font-size: 14px; word-break: break-all;
}
.fc-counts {
display: grid; grid-template-columns: max-content auto;
gap: 4px 12px;
font-size: 13px;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-counts-key { font-weight: 500; text-transform: capitalize; }
.fc-counts-val { font-variant-numeric: tabular-nums; }
</style>

Some files were not shown because too many files have changed in this diff Show More