Compare commits

...

52 Commits

Author SHA1 Message Date
bvandeusen 2879ac6f2b Merge pull request 'v26.05.25.1: maintenance sweep + Camie v2 + corrupt-file handling + post-date gallery + clear-stuck escape hatch' (#11) from dev into main 2026-05-25 12:57:46 -04:00
bvandeusen 3a359f6c5e fix(ui): Quick scan button always visible (disabled when active batch present) + inline Clear stuck action — operator was clicking a spinner area thinking it was the button because activeBatch hid the Quick scan button entirely
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:43:37 -04:00
bvandeusen b6a917ac81 feat(import): /api/import/clear-stuck endpoint + Clear stuck UI button — escape hatch for the autoretry-loop case the automatic sweep can't break
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.

/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
  marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
  underlying cause is resolved

UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.

Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:37:07 -04:00
bvandeusen c361032554 feat(gallery): sort/group/jump by COALESCE(post.post_date, image_record.created_at) — surface migrated content at its original publish date, not FC scan date
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.

Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
  ImageProvenance (only if currently NULL — preserves the canonical
  download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
  COALESCE(post.post_date, image_record.created_at), used in:
    * scroll() ORDER BY + cursor WHERE clauses
    * timeline() year/month group-by
    * jump_cursor() year/month filter
    * _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
  works for images without a post (NULL on the Post side, fall back
  to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
  /scroll exposes both alongside the existing created_at so the UI
  can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.

Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.

To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:30:18 -04:00
bvandeusen 9f54efdedf fix(migrators): tag_apply phase 4 now covers deviantart + pixiv (was silently dropping IR PostMetadata from those platforms)
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.

Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.

To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:25:45 -04:00
bvandeusen 6b1bb87647 fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:12:55 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:34:36 -04:00
bvandeusen 52445eb501 fix(ui): double directory card width (220px -> 440px) + bound preview slot height (min 150 / max 220 / overflow hidden / explicit display+object-position) so tall source images can't escape on browsers that don't compute aspect-ratio
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:32:05 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output)
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:25:30 -04:00
bvandeusen 9d5abb09f6 fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:00:29 -04:00
bvandeusen b8dce6c483 Merge pull request 'FC-3h + FC-3k: backup first-class + admin destructive actions' (#10) from dev into main 2026-05-25 01:41:53 -04:00
bvandeusen 832345a245 fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:28:54 -04:00
bvandeusen a0136fa30d fix(fc3k): add mime=image/jpeg to test ImageRecord ctors (NOT NULL) + reorder admin import after stdlib/3rd-party (ruff I001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:12:52 -04:00
bvandeusen de1a4b64b7 fc3k(ui): TagMaintenanceCard — preview-then-commit prune-unused under Settings → Maintenance
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:52:30 -04:00
bvandeusen 3f500e592e fc3k(ui): per-tag dots-menu with Merge + Delete actions; Tier-B count-surfacing modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:51:51 -04:00
bvandeusen d97e3f9b59 fc3k(ui): bulk-delete action in BulkEditorPanel with sha8 confirm token + projected counts modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:49:45 -04:00
bvandeusen 035c49f675 fc3k(ui): ArtistDangerZone card + slot at bottom of ArtistView with cascade-delete flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:48:17 -04:00
bvandeusen e41ab1cca5 fc3k(ui): Pinia admin store — six endpoints + task_run polling for Tier-C dispatched ops
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:47:09 -04:00
bvandeusen 42c6b642c2 fc3k(ui): rename + relocate BackupConfirmModal → modal/DestructiveConfirmModal; add tier + projectedCounts props
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:46:37 -04:00
bvandeusen ad3d34a1fc fc3k: /api/admin endpoint integration tests — dry-run, confirm-mismatch, dispatch, Tier-A/B/C paths
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:45:31 -04:00
bvandeusen b5289ed372 fc3k: admin Celery task tests — registration, success, failure, missing-id idempotency
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:44:33 -04:00
bvandeusen f6aa805725 fc3k: cleanup_service unit tests — projections + mutations + file unlinks against real Postgres
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:43:51 -04:00
bvandeusen f096c9a5fb fc3k: register admin_bp in api/__init__ all_blueprints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:40 -04:00
bvandeusen 676a86b514 fc3k: /api/admin tags prune-unused endpoint — Tier-A preview-then-commit flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:20 -04:00
bvandeusen 44cc625d4a fc3k: /api/admin tag endpoints — Tier-B delete + merge + usage-count helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:03 -04:00
bvandeusen f7ee122243 fc3k: /api/admin blueprint — Tier-C artist cascade + bulk image delete with sha8-keyed confirm tokens
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:53 -04:00
bvandeusen 7c6f11964a fc3k: celery_app — register admin tasks on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:23 -04:00
bvandeusen 94c60c0af2 fc3k: admin Celery tasks — delete_artist_cascade_task + bulk_delete_images_task on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:03 -04:00
bvandeusen 6df102b83d fc3k: cleanup_service mutations — unlink primitive, artist cascade, bulk image delete, tag delete, prune unused
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:39:45 -04:00
bvandeusen 2ae01d27e3 fc3k: cleanup_service projections — artist cascade, bulk delete, tag usage, unused tags
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:38:57 -04:00
bvandeusen 718cc79905 fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:37:06 -04:00
bvandeusen e78a35d333 fc3h: collapse multi-line sqlalchemy import in backup_run.py — fits under line-length=100, ruff I001 would bounce
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:12:06 -04:00
bvandeusen 83bd3b4b2d fc3h(ui): slot BackupCard into Maintenance panel above migration card
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:22 -04:00
bvandeusen aecedd9fe4 fc3h(ui): BackupCard.vue + BackupRunsTable.vue — combined card with DB + Images sub-sections
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:00 -04:00
bvandeusen 1e34b1b428 fc3h(ui): BackupConfirmModal.vue — typed-token confirmation for restore + delete
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:05:04 -04:00
bvandeusen 102c21feaa fc3h(ui): Pinia backup store — runs, triggers, restore, delete, tag, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:04:37 -04:00
bvandeusen 57a338f7e6 fc3h(tests): drop pinned migration-backup tests (retired surface; coverage moved to test_backup_service.py + test_api_system_backup.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:03:57 -04:00
bvandeusen d04983138a fc3h: /api/system/backup endpoint integration tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:02:32 -04:00
bvandeusen 9ec6fdb596 fc3h: Celery task integration tests (backup, restore, prune, nightly)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:01:39 -04:00
bvandeusen 86ad9b80e9 fc3h: backup_service unit tests (subprocess monkeypatched)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:40 -04:00
bvandeusen 2b05f147f4 fc3h: migrators docstring — note backup/rollback retired to backup_service.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:01 -04:00
bvandeusen 70e1e010d1 fc3h: remove backend/app/services/migrators/backup.py + rollback.py (relocated to backup_service.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:18 -04:00
bvandeusen d3d4320ed5 fc3h: retire backup + rollback from migrate API/task — moved to /api/system/backup/* per FC-3h
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:11 -04:00
bvandeusen 7d42cddb11 fc3h: /api/system/backup blueprint — trigger, list, get, patch, restore, delete, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:57:25 -04:00
bvandeusen 1f01c4819a fc3h: celery_app — register backup tasks (include + maintenance route + Beat hourly tick + daily prune)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:56:06 -04:00
bvandeusen 06d527cb92 fc3h: prune_backups (daily retention) + backup_db_nightly (hourly tick, settings-gated)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:55:19 -04:00
bvandeusen e9ea376aed fc3h: restore_db_task + restore_images_task — restore creates 'restoring' marker row linked via restored_from_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:50 -04:00
bvandeusen 882cb491ba fc3h: backup_db_task + backup_images_task Celery tasks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:26 -04:00
bvandeusen 319e7de547 fc3h: backup_service.py — DB + images backup/restore + unlink helpers (relocated, split per-kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:35 -04:00
bvandeusen e43312a129 fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:00 -04:00
bvandeusen 8f2732a56f fc3h: alembic 0017 — backup_run table with indexes + partial-tag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:51:22 -04:00
bvandeusen c3e855bd9b fc3h: BackupRun model — artifact record for backup/restore runs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:50:59 -04:00
58 changed files with 5429 additions and 474 deletions
+82
View File
@@ -0,0 +1,82 @@
"""fc3h: backup_run table
Revision ID: 0017
Revises: 0016
Create Date: 2026-05-24
Additive. New table records every backup/restore attempt with artifact
metadata. Lifecycle tracking lives in task_run from FC-3i; this is
artifact-only (paths, sizes, tag, restore lineage).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0017"
down_revision: Union[str, None] = "0016"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"backup_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("tag", sa.String(length=64), nullable=True),
sa.Column("triggered_by", sa.String(length=32), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("sql_path", sa.Text(), nullable=True),
sa.Column("tar_path", sa.Text(), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"manifest", sa.JSON(), nullable=False, server_default="{}",
),
sa.Column(
"restored_from_id", sa.Integer(),
sa.ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
),
)
# Single-column indexes (matches Mapped[...].index=True).
op.create_index("ix_backup_run_kind", "backup_run", ["kind"])
op.create_index("ix_backup_run_status", "backup_run", ["status"])
op.create_index("ix_backup_run_tag", "backup_run", ["tag"])
op.create_index("ix_backup_run_started_at", "backup_run", ["started_at"])
op.create_index("ix_backup_run_finished_at", "backup_run", ["finished_at"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_backup_run_kind_started",
"backup_run", ["kind", sa.text("started_at DESC")],
)
op.create_index(
"ix_backup_run_status_finished",
"backup_run", ["status", sa.text("finished_at DESC")],
)
# Partial index: only tagged rows participate in retention-exempt query.
op.create_index(
"ix_backup_run_tag_partial",
"backup_run", ["tag"],
postgresql_where=sa.text("tag IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_backup_run_tag_partial", table_name="backup_run")
op.drop_index("ix_backup_run_status_finished", table_name="backup_run")
op.drop_index("ix_backup_run_kind_started", table_name="backup_run")
op.drop_index("ix_backup_run_finished_at", table_name="backup_run")
op.drop_index("ix_backup_run_started_at", table_name="backup_run")
op.drop_index("ix_backup_run_tag", table_name="backup_run")
op.drop_index("ix_backup_run_status", table_name="backup_run")
op.drop_index("ix_backup_run_kind", table_name="backup_run")
op.drop_table("backup_run")
@@ -0,0 +1,62 @@
"""fc3h: backup_* knobs on import_settings
Revision ID: 0018
Revises: 0017
Create Date: 2026-05-24
Adds four columns to the singleton import_settings row:
- backup_db_nightly_enabled (default False — opt-in)
- backup_db_nightly_hour_utc (default 3)
- backup_db_keep_last_n (default 14)
- backup_images_keep_last_n (default 3)
server_default ensures the singleton row is backfilled in place
without an UPDATE statement.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0018"
down_revision: Union[str, None] = "0017"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_enabled", sa.Boolean(),
nullable=False, server_default=sa.false(),
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_hour_utc", sa.Integer(),
nullable=False, server_default="3",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_keep_last_n", sa.Integer(),
nullable=False, server_default="14",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_images_keep_last_n", sa.Integer(),
nullable=False, server_default="3",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "backup_images_keep_last_n")
op.drop_column("import_settings", "backup_db_keep_last_n")
op.drop_column("import_settings", "backup_db_nightly_hour_utc")
op.drop_column("import_settings", "backup_db_nightly_enabled")
+4
View File
@@ -14,6 +14,7 @@ api_bp.add_url_rule("/health", view_func=health.get_health, methods=["GET"])
def all_blueprints() -> list[Blueprint]:
from .admin import admin_bp
from .aliases import aliases_bp
from .allowlist import allowlist_bp
from .artist import artist_bp
@@ -34,6 +35,7 @@ def all_blueprints() -> list[Blueprint]:
from .sources import sources_bp
from .suggestions import suggestions_bp
from .system_activity import system_activity_bp
from .system_backup import system_backup_bp
from .tags import tags_bp
return [
api_bp,
@@ -46,6 +48,8 @@ def all_blueprints() -> list[Blueprint]:
showcase_bp,
settings_bp,
system_activity_bp,
system_backup_bp,
admin_bp,
import_admin_bp,
migrate_bp,
suggestions_bp,
+200
View File
@@ -0,0 +1,200 @@
"""FC-3k: /api/admin — destructive admin actions.
Five action surfaces:
POST /api/admin/artists/<slug>/cascade-delete (Tier C)
POST /api/admin/images/bulk-delete (Tier C)
DELETE /api/admin/tags/<int:tag_id> (Tier B)
POST /api/admin/tags/<int:dest_id>/merge (Tier B)
POST /api/admin/tags/prune-unused (Tier A)
GET /api/admin/tags/<int:tag_id>/usage-count (helper)
Tier-C ops take a dry_run body flag (returns projection inline,
no dispatch) and a confirm body field (server-recomputed token).
Long-running ops dispatch a maintenance-queue Celery task; the UI
tails FC-3i's /api/system/activity/runs to surface progress.
"""
from __future__ import annotations
import hashlib
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import Artist
from ..services.cleanup_service import project_artist_cascade, project_bulk_image_delete
admin_bp = Blueprint("admin", __name__, url_prefix="/api/admin")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _bulk_image_confirm_token(image_ids: list[int]) -> str:
"""Stable 8-hex token derived from the sorted id list. Mutates
when the selection changes; stays the same across modal opens of
the same selection so the operator can paste without confusion."""
canon = ",".join(str(i) for i in sorted(image_ids))
digest = hashlib.sha256(canon.encode("utf-8")).hexdigest()
return digest[:8]
@admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"])
async def artist_cascade_delete(slug: str):
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
artist = (await session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
return _bad("not_found", status=404)
artist_id = artist.id
projected = await session.run_sync(
lambda sync_sess: project_artist_cascade(sync_sess, slug=slug)
)
if dry_run:
return jsonify(projected)
expected = f"delete-artist-{artist_id}"
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import delete_artist_cascade_task
async_result = delete_artist_cascade_task.delay(artist_id=artist_id)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/images/bulk-delete", methods=["POST"])
async def images_bulk_delete():
body = await request.get_json(silent=True) or {}
image_ids = body.get("image_ids")
if not isinstance(image_ids, list) or not image_ids:
return _bad("invalid_image_ids", detail="image_ids must be non-empty list of int")
try:
image_ids = [int(i) for i in image_ids]
except (TypeError, ValueError):
return _bad("invalid_image_ids", detail="image_ids must contain only ints")
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
projected = await session.run_sync(
lambda sync_sess: project_bulk_image_delete(
sync_sess, image_ids=image_ids,
)
)
if dry_run:
return jsonify(projected)
sha8 = _bulk_image_confirm_token(image_ids)
expected = f"delete-images-{sha8}"
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import bulk_delete_images_task
async_result = bulk_delete_images_task.delay(image_ids=image_ids)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/tags/<int:tag_id>", methods=["DELETE"])
async def tag_delete(tag_id: int):
"""Tier-B sync delete. UI yes/no modal is the only confirmation."""
from ..services.cleanup_service import delete_tag
async with get_session() as session:
try:
result = await session.run_sync(
lambda sync_sess: delete_tag(sync_sess, tag_id=tag_id)
)
except LookupError:
return _bad("not_found", status=404)
return jsonify(result)
@admin_bp.route("/tags/<int:dest_id>/merge", methods=["POST"])
async def tag_merge(dest_id: int):
"""Wraps TagService.merge. Source repoints to dest, dest survives,
source row deleted, protective alias auto-created if source was
ML-applied or allowlisted."""
from ..services.tag_service import TagMergeConflict, TagService, TagValidationError
body = await request.get_json(silent=True) or {}
source_id = body.get("source_id")
if not isinstance(source_id, int) or source_id == dest_id:
return _bad("invalid_source_id", detail="source_id must be int and differ from dest")
async with get_session() as session:
try:
result = await TagService(session).merge(
source_id=source_id, target_id=dest_id,
)
except TagMergeConflict as exc:
return _bad("merge_conflict", status=409, detail=str(exc))
except TagValidationError as exc:
return _bad("tag_kind_mismatch", detail=str(exc))
except LookupError:
return _bad("not_found", status=404)
# MergeResult is a frozen dataclass — flatten to dict.
return jsonify({
"result": {
"target_id": result.target_id,
"target_name": result.target_name,
"target_kind": result.target_kind,
"merged_count": result.merged_count,
"alias_created": result.alias_created,
"source_deleted": result.source_deleted,
},
})
@admin_bp.route("/tags/<int:tag_id>/usage-count", methods=["GET"])
async def tag_usage_count(tag_id: int):
"""Helper for the Tier-B yes/no prompt; surfaces "N associations"
in the dialog so the operator knows what they're nuking."""
from ..services.cleanup_service import count_tag_associations
async with get_session() as session:
count = await session.run_sync(
lambda sync_sess: count_tag_associations(
sync_sess, tag_id=tag_id,
)
)
return jsonify({"count": count})
@admin_bp.route("/tags/prune-unused", methods=["POST"])
async def tags_prune_unused():
"""Tier-A: dry-run preview list IS the prompt. UI calls with
dry_run=true first, shows the list, operator clicks button to
re-call with dry_run=false."""
from ..services.cleanup_service import prune_unused_tags
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: prune_unused_tags(
sync_sess, dry_run=dry_run,
)
)
return jsonify(result)
+2
View File
@@ -42,6 +42,8 @@ async def scroll():
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+77
View File
@@ -120,6 +120,83 @@ async def retry_failed():
return jsonify({"retried": len(failed_ids)})
@import_admin_bp.route("/clear-stuck", methods=["POST"])
async def clear_stuck():
"""Force any non-terminal ImportTask (status in pending/queued/
processing) to 'failed' AND finalize any ImportBatch that ends up
with no active children. Escape hatch for the operator when the
automatic recover_interrupted_tasks sweep keeps re-queueing the
same stuck row forever (e.g., underlying file is genuinely broken
and the import keeps OSError-looping at PIL load).
Idempotent + non-destructive: rows survive as 'failed' so the
Retry-Failed button can re-attempt them once whatever was broken
is fixed. Banked 2026-05-25 — operator hit 3 large PNGs that
autoretry-looped for 2 days after a corrupt-data PIL OSError.
"""
async with get_session() as session:
stuck_ids = (
await session.execute(
select(ImportTask.id).where(
ImportTask.status.in_(["pending", "queued", "processing"])
)
)
).scalars().all()
if stuck_ids:
await session.execute(
update(ImportTask)
.where(ImportTask.id.in_(stuck_ids))
.values(
status="failed",
finished_at=datetime.now(UTC),
error=(
"manually cleared via /api/import/clear-stuck "
"— stuck in non-terminal state; retry once "
"underlying cause (corrupt file, missing model, "
"etc.) is resolved"
),
)
)
# Finalize any 'running' ImportBatch that no longer has any
# active children. The "Scanning..." banner is driven by
# /api/import/status finding a running batch; left untouched,
# it would persist forever after the stuck-task clear.
running_batches = (
await session.execute(
select(ImportBatch.id).where(ImportBatch.status == "running")
)
).scalars().all()
finalized_batches = 0
for batch_id in running_batches:
still_active = (
await session.execute(
select(ImportTask.id)
.where(ImportTask.batch_id == batch_id)
.where(ImportTask.status.in_(
["pending", "queued", "processing"]
))
.limit(1)
)
).scalar_one_or_none()
if still_active is None:
await session.execute(
update(ImportBatch)
.where(ImportBatch.id == batch_id)
.values(
status="complete",
finished_at=datetime.now(UTC),
)
)
finalized_batches += 1
await session.commit()
return jsonify({
"tasks_failed": len(stuck_ids),
"batches_finalized": finalized_batches,
})
@import_admin_bp.route("/clear-completed", methods=["POST"])
async def clear_completed():
body = await request.get_json(silent=True) or {}
+5 -39
View File
@@ -1,14 +1,11 @@
"""FC-5: /api/migrate — trigger and poll migration runs.
Ingest kinds (gs_ingest, ir_ingest) accept multipart/form-data with an
`export_file` field. All other kinds accept JSON. Apply-without-backup
guard rejects non-dry-run ingests unless a pre_migration-tagged backup
exists in the last 24h (override with body.force=true).
`export_file` field. All other kinds accept JSON. Backup + rollback
were retired in FC-3h (2026-05-24); use /api/system/backup/* instead.
"""
import json
from datetime import UTC, datetime, timedelta
from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
@@ -19,17 +16,12 @@ from ..tasks.migration import run_migration
migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate")
# 'backup' + 'rollback' retired 2026-05-24 (FC-3h); see /api/system/backup/*.
_VALID_KINDS = frozenset({
"backup", "gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "rollback", "cleanup",
"gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "cleanup",
})
_INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"})
# Backup gate retired 2026-05-24 — operator-flagged the speculative-safety
# requirement was actively blocking the UI ingest path (backup itself is
# unreliable on large NFS-backed image libraries) and FC-3h will rewrite
# the backup surface as a first-class feature with its own scheduling
# + recovery. Leaving the constant for historical grep.
_APPLY_KINDS: frozenset[str] = frozenset()
def _bad(error: str, *, status: int = 400, **extra):
@@ -38,19 +30,6 @@ def _bad(error: str, *, status: int = 400, **extra):
return jsonify(body), status
def _has_recent_pre_migration_backup() -> bool:
from ..services.migrators import backup as backup_mod
images_root = Path("/images")
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
return False
created_at_str = manifest.get("created_at")
if not created_at_str:
return False
created_at = datetime.fromisoformat(created_at_str)
return (datetime.now(UTC) - created_at) < timedelta(hours=24)
def _run_to_dict(run: MigrationRun) -> dict:
return {
"id": run.id,
@@ -83,7 +62,6 @@ async def create_run(kind: str):
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
return _bad("invalid_export_file", detail=str(exc))
dry_run = str(form.get("dry_run", "false")).lower() in ("true", "1", "yes")
force = str(form.get("force", "false")).lower() in ("true", "1", "yes")
params: dict = {"data": data, "dry_run": dry_run}
else:
body = await request.get_json()
@@ -92,20 +70,8 @@ async def create_run(kind: str):
if not isinstance(body, dict):
return _bad("invalid_body")
dry_run = bool(body.get("dry_run", False))
force = bool(body.get("force", False))
params = dict(body)
is_apply = (kind in _APPLY_KINDS) and not dry_run
if is_apply and not force and not _has_recent_pre_migration_backup():
return _bad(
"no_backup",
detail="apply action requires a pre_migration-tagged backup "
"in the last 24h (or force=true).",
)
if kind == "backup":
params.setdefault("tag", "pre_migration")
async with get_session() as session:
run = MigrationRun(kind=kind, status="pending", dry_run=dry_run)
session.add(run)
+264
View File
@@ -0,0 +1,264 @@
"""FC-3h: /api/system/backup — create/list/restore/delete/tag for
DB + image backups.
Read endpoints are public on FC (operator-facing internal API; same
posture as /api/system/activity). Write endpoints take a typed
`confirm` body field that must match a server-generated token for
that backup row, to prevent click-to-destroy by stale browser tabs
or accidental cURL.
"""
from __future__ import annotations
from quart import Blueprint, jsonify, request
from sqlalchemy import desc, select
from ..extensions import get_session
from ..models import BackupRun, ImportSettings
system_backup_bp = Blueprint(
"system_backup", __name__, url_prefix="/api/system/backup",
)
_KINDS = frozenset({"db", "images"})
_TAG_MAX_LEN = 64
_BACKUP_SETTINGS_FIELDS = (
"backup_db_nightly_enabled",
"backup_db_nightly_hour_utc",
"backup_db_keep_last_n",
"backup_images_keep_last_n",
)
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _row_to_dict(r: BackupRun) -> dict:
return {
"id": r.id,
"kind": r.kind,
"status": r.status,
"tag": r.tag,
"triggered_by": r.triggered_by,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": (
int((r.finished_at - r.started_at).total_seconds())
if r.finished_at and r.started_at else None
),
"sql_path": r.sql_path,
"tar_path": r.tar_path,
"size_bytes": r.size_bytes,
"error": r.error,
"restored_from_id": r.restored_from_id,
"manifest": r.manifest or {},
}
def _validate_tag(tag):
if tag is None:
return None
if not isinstance(tag, str):
return _bad("invalid_tag", detail="tag must be string or null")
tag = tag.strip()
if not tag:
return None
if len(tag) > _TAG_MAX_LEN:
return _bad("invalid_tag", detail=f"tag too long (max {_TAG_MAX_LEN})")
return tag
def _validate_backup_settings_patch(body: dict):
if "backup_db_nightly_enabled" in body and not isinstance(
body["backup_db_nightly_enabled"], bool,
):
return _bad("invalid_value", detail="backup_db_nightly_enabled must be bool")
if "backup_db_nightly_hour_utc" in body:
v = body["backup_db_nightly_hour_utc"]
if not isinstance(v, int) or isinstance(v, bool) or not (0 <= v <= 23):
return _bad("invalid_value", detail="backup_db_nightly_hour_utc must be 0..23")
if "backup_db_keep_last_n" in body:
v = body["backup_db_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 365):
return _bad("invalid_value", detail="backup_db_keep_last_n must be 1..365")
if "backup_images_keep_last_n" in body:
v = body["backup_images_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 100):
return _bad("invalid_value", detail="backup_images_keep_last_n must be 1..100")
return None
@system_backup_bp.route("/db", methods=["POST"])
async def trigger_db_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_db_task
backup_db_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/images", methods=["POST"])
async def trigger_images_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_images_task
backup_images_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/runs", methods=["GET"])
async def list_runs():
try:
limit = min(int(request.args.get("limit", "50")), 200)
except ValueError:
return _bad("invalid_limit")
if limit < 1:
return _bad("invalid_limit")
kind = request.args.get("kind")
if kind is not None and kind not in _KINDS:
return _bad("invalid_kind", detail=f"kind must be one of {sorted(_KINDS)}")
before_id_raw = request.args.get("before_id")
before_id = int(before_id_raw) if before_id_raw else None
async with get_session() as session:
stmt = select(BackupRun).order_by(desc(BackupRun.id))
if kind:
stmt = stmt.where(BackupRun.kind == kind)
if before_id is not None:
stmt = stmt.where(BackupRun.id < before_id)
stmt = stmt.limit(limit + 1)
rows = (await session.execute(stmt)).scalars().all()
has_more = len(rows) > limit
rows = rows[:limit]
return jsonify({
"runs": [_row_to_dict(r) for r in rows],
"next_cursor": rows[-1].id if has_more and rows else None,
})
@system_backup_bp.route("/runs/<int:run_id>", methods=["GET"])
async def get_run(run_id: int):
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>", methods=["PATCH"])
async def patch_run(run_id: int):
body = await request.get_json(silent=True) or {}
if "tag" not in body:
return _bad("invalid_body", detail="tag required")
tag = _validate_tag(body["tag"])
if isinstance(tag, tuple):
return tag
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
row.tag = tag
await session.commit()
await session.refresh(row)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>/restore", methods=["POST"])
async def trigger_restore(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
if row.status != "ok":
return _bad(
"not_restorable",
detail=f"source backup status={row.status!r}; only 'ok' rows are restorable",
)
expected = f"restore-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
kind = row.kind
if kind == "db":
from ..tasks.backup import restore_db_task
restore_db_task.delay(source_backup_run_id=run_id)
else: # 'images' (the only other value _KINDS allows via the trigger path)
from ..tasks.backup import restore_images_task
restore_images_task.delay(source_backup_run_id=run_id)
return jsonify({"status": "dispatched", "kind": kind}), 202
@system_backup_bp.route("/runs/<int:run_id>", methods=["DELETE"])
async def delete_run(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
expected = f"delete-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..services import backup_service
backup_service.unlink_artifact_files(
sql_path=row.sql_path, tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
await session.delete(row)
await session.commit()
return "", 204
@system_backup_bp.route("/settings", methods=["GET"])
async def get_settings():
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
return jsonify({
"backup_db_nightly_enabled": row.backup_db_nightly_enabled,
"backup_db_nightly_hour_utc": row.backup_db_nightly_hour_utc,
"backup_db_keep_last_n": row.backup_db_keep_last_n,
"backup_images_keep_last_n": row.backup_images_keep_last_n,
})
@system_backup_bp.route("/settings", methods=["PATCH"])
async def patch_settings():
body = await request.get_json(silent=True)
if not isinstance(body, dict):
return _bad("invalid_body", detail="body must be a JSON object")
err = _validate_backup_settings_patch(body)
if err is not None:
return err
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
for field in _BACKUP_SETTINGS_FIELDS:
if field in body:
setattr(row, field, body[field])
await session.commit()
return await get_settings()
+12
View File
@@ -31,6 +31,8 @@ def make_celery() -> Celery:
"backend.app.tasks.migration",
"backend.app.tasks.ml",
"backend.app.tasks.download",
"backend.app.tasks.backup",
"backend.app.tasks.admin",
],
)
app.conf.update(
@@ -43,6 +45,8 @@ def make_celery() -> Celery:
"backend.app.tasks.scan.*": {"queue": "scan"},
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
"backend.app.tasks.migration.*": {"queue": "maintenance"},
"backend.app.tasks.backup.*": {"queue": "maintenance"},
"backend.app.tasks.admin.*": {"queue": "maintenance"},
},
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True,
@@ -89,6 +93,14 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.prune_task_runs",
"schedule": 86400.0, # daily
},
"fc3h-backup-db-nightly": {
"task": "backend.app.tasks.backup.backup_db_nightly",
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
},
"fc3h-prune-backups": {
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
},
timezone="UTC",
)
+2
View File
@@ -2,6 +2,7 @@
from .app_setting import AppSetting
from .artist import Artist
from .backup_run import BackupRun
from .base import Base
from .credential import Credential
from .download_event import DownloadEvent
@@ -27,6 +28,7 @@ __all__ = [
"Base",
"AppSetting",
"Artist",
"BackupRun",
"Source",
"Credential",
"Post",
+55
View File
@@ -0,0 +1,55 @@
"""FC-3h: backup_run — operator-facing artifact record for a backup run.
One row per backup attempt (kind='db' or 'images'). Lifecycle
tracking (started_at/finished_at/duration_ms/exception text) lives
in task_run from FC-3i — this row records artifact metadata: file
paths, sizes, tag (retention protection), and restore lineage via
restored_from_id.
Status values (String, not Postgres ENUM — per
feedback_check_existing_enums):
pending — created but task hasn't started yet (rare; usually
status starts as 'running' from the task body).
running — backup task is in flight.
ok — artifact successfully written.
error — task raised; error column populated.
restoring — this row represents a restore attempt (kind = restored
kind); linked to source via restored_from_id.
restored — restore completed successfully.
"""
from datetime import datetime
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class BackupRun(Base):
__tablename__ = "backup_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True,
)
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True,
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True,
)
sql_path: Mapped[str | None] = mapped_column(Text, nullable=True)
tar_path: Mapped[str | None] = mapped_column(Text, nullable=True)
size_bytes: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
manifest: Mapped[dict] = mapped_column(
JSON, nullable=False, default=dict, server_default="{}",
)
restored_from_id: Mapped[int | None] = mapped_column(
ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
)
+14
View File
@@ -49,3 +49,17 @@ class ImportSettings(Base):
download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5
)
# FC-3h backup knobs.
backup_db_nightly_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False,
)
backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
backup_db_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=14,
)
backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
+21 -2
View File
@@ -25,12 +25,31 @@ def _snapshot(repo_id: str, dest: Path, allow_patterns: list[str] | None) -> Non
def ensure_camie() -> None:
"""Fetch Camie v2 weights + metadata.
v2 layout (HuggingFace Camais03/camie-tagger-v2): the ONNX file is
named camie-tagger-v2.onnx (not model.onnx) and tags ship inside
camie-tagger-v2-metadata.json (not selected_tags.csv). Both at root.
The repo also contains app/, game/, training/, images/ subdirs full
of setup/demo files we don't need — allow_patterns scopes the fetch
to just the inference essentials (~790 MB instead of ~2 GB).
"""
dest = MODEL_ROOT / "camie"
if (dest / "model.onnx").is_file() and (dest / "selected_tags.csv").is_file():
model_file = dest / "camie-tagger-v2.onnx"
meta_file = dest / "camie-tagger-v2-metadata.json"
if model_file.is_file() and meta_file.is_file():
print(f"[download_models] Camie present at {dest}")
return
print(f"[download_models] Fetching {CAMIE_REPO} -> {dest}")
_snapshot(CAMIE_REPO, dest, ["model.onnx", "selected_tags.csv", "*.json"])
_snapshot(
CAMIE_REPO, dest,
[
"camie-tagger-v2.onnx",
"camie-tagger-v2-metadata.json",
"config.json",
"config.yaml",
],
)
def ensure_siglip() -> None:
+196
View File
@@ -0,0 +1,196 @@
"""FC-3h: first-class backup/restore service for FC.
Two independent backup kinds:
- 'db' — pg_dump only; fast; nightly via Beat (settings-gated)
- 'images' — tar+zstd of /images; slow; manual trigger only
Files live under <images_root>/_backups/. Each backup writes:
fc_<kind>_<ts>.{sql|tar.zst} — the artifact
fc_<kind>_<ts>.json — manifest (kind/tag/triggered_by)
Service functions are sync (subprocess-bound). Celery tasks in
backend.app.tasks.backup wrap each one with task_run-tracked
lifecycle + soft/hard time limits + retention bookkeeping.
"""
from __future__ import annotations
import json
import subprocess
from datetime import UTC, datetime
from pathlib import Path
_BACKUPS_DIRNAME = "_backups"
# Subprocess-level guardrails BEYOND the Celery soft_time_limit. The
# Celery soft limit signals the Python process; subprocess.Popen in a
# blocking syscall ignores that signal. These bound the worst case.
_DB_SUBPROCESS_TIMEOUT_S = 12 * 60 # 12 min (Celery soft is 10 min)
_IMAGES_SUBPROCESS_TIMEOUT_S = 7 * 60 * 60 # 7 hr (Celery soft is 6 hr)
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy +psycopg/+asyncpg driver suffix for pg_dump/psql."""
for driver in (
"postgresql+psycopg",
"postgresql+asyncpg",
"postgresql+psycopg2",
):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path) -> Path:
p = images_root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
def _now_ts() -> str:
return datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
def _file_size_or_none(path: Path) -> int | None:
try:
return path.stat().st_size
except OSError:
return None
def _write_manifest(
out_dir: Path, *, kind: str, ts: str,
tag: str | None, triggered_by: str,
artifact_path: Path,
) -> Path:
manifest = {
"kind": kind,
"backup_id": f"fc_{kind}_{ts}",
"tag": tag,
"triggered_by": triggered_by,
"created_at": datetime.now(UTC).isoformat(),
"artifact_path": str(artifact_path),
}
mf = out_dir / f"fc_{kind}_{ts}.json"
mf.write_text(json.dumps(manifest, indent=2))
return mf
def backup_db(
*, db_url: str, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run pg_dump; write .sql + manifest; return dict for the caller
to persist into BackupRun. Raises on subprocess failure."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_db_{ts}.sql"
subprocess.run(
[
"pg_dump", "--no-owner", "--no-acl",
"-f", str(sql_path), _libpq_url(db_url),
],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="db", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=sql_path,
)
return {
"kind": "db",
"ts": ts,
"sql_path": str(sql_path),
"tar_path": None,
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(sql_path),
}
def backup_images(
*, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run tar --zstd over images_root; write .tar.zst + manifest."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
subprocess.run(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=tar_path,
)
return {
"kind": "images",
"ts": ts,
"sql_path": None,
"tar_path": str(tar_path),
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(tar_path),
}
def restore_db(*, db_url: str, sql_path: Path) -> None:
"""Wipe public schema, then load from .sql. Raises on subprocess
failure; partial-restore state is the caller's concern."""
libpq = _libpq_url(db_url)
subprocess.run(
[
"psql", libpq, "-c",
"DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public;",
],
capture_output=True, check=True, timeout=120,
)
subprocess.run(
["psql", libpq, "-f", str(sql_path)],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
def restore_images(*, images_root: Path, tar_path: Path) -> None:
"""Untar over images_root.parent. Additive — files NOT in the
tarball are NOT removed. Caller wipes first if a clean restore
is needed."""
subprocess.run(
[
"tar", "--zstd", "-xf", str(tar_path),
"-C", str(images_root.parent),
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
def unlink_artifact_files(
*,
sql_path: str | None,
tar_path: str | None,
manifest_path: str | None,
) -> dict:
"""Best-effort unlink of all on-disk files for a BackupRun row.
Returns dict keyed by label with True/False per file. Missing
files count as success (missing_ok semantics)."""
deleted: dict = {}
for label, p in (
("sql", sql_path),
("tar", tar_path),
("manifest", manifest_path),
):
if not p:
continue
path = Path(p)
try:
path.unlink(missing_ok=True)
deleted[label] = True
except OSError:
deleted[label] = False
return deleted
+367
View File
@@ -0,0 +1,367 @@
"""FC-3k: first-class admin destructive operations.
Projections are pure SELECTs used by both dry-run preview endpoints
and Tier-B count prompts. Mutations (Task 2) are called from sync
HTTP handlers (small ops) and from Celery tasks in
backend.app.tasks.admin (long ops).
This module is the PERMANENT home of artist-cascade + image-unlink
logic. The legacy copy at backend/app/services/migrators/cleanup.py
stays in place until FC-3j; FC-3j will replace its body with thin
re-exports from this module and then delete the wrapper.
"""
from __future__ import annotations
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from ..models import Artist, ImageRecord, Tag
from ..models.series_page import SeriesPage
from ..models.tag import image_tag
def project_artist_cascade(session: Session, *, slug: str) -> dict:
"""Read-only projection of what delete_artist_cascade would touch.
Returns:
{
"artist": {"id": int, "name": str, "slug": str},
"projected": {
"images": int,
"sources": int,
"thumbs": int, # images with a thumbnail_path set
"import_tasks": int, # ImportTask rows referencing the artist's images
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
},
}
Raises LookupError if slug not found. No mutations.
"""
from ..models.import_task import ImportTask
from ..models.source import Source
artist = session.execute(
select(Artist).where(Artist.slug == slug)
).scalar_one_or_none()
if artist is None:
raise LookupError(f"artist slug not found: {slug!r}")
images_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
sources_count = session.execute(
select(func.count(Source.id))
.where(Source.artist_id == artist.id)
).scalar_one()
thumbs_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
.where(ImageRecord.thumbnail_path.is_not(None))
).scalar_one()
import_tasks_count = session.execute(
select(func.count(ImportTask.id))
.where(
ImportTask.result_image_id.in_(
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
)
)
).scalar_one()
bytes_on_disk = session.execute(
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
return {
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"projected": {
"images": images_count,
"sources": sources_count,
"thumbs": thumbs_count,
"import_tasks": import_tasks_count,
"bytes_on_disk": int(bytes_on_disk),
},
}
def project_bulk_image_delete(
session: Session, *, image_ids: list[int],
) -> dict:
"""Read-only projection of what delete_images would touch.
Returns:
{
"images_found": int,
"thumbs_to_unlink": int,
"bytes_on_disk": int,
"missing_ids": list[int], # ids passed in that don't exist
}
No mutations.
"""
if not image_ids:
return {
"images_found": 0,
"thumbs_to_unlink": 0,
"bytes_on_disk": 0,
"missing_ids": [],
}
rows = session.execute(
select(
ImageRecord.id,
ImageRecord.thumbnail_path,
ImageRecord.size_bytes,
).where(ImageRecord.id.in_(image_ids))
).all()
found_ids = {r.id for r in rows}
missing = sorted(set(image_ids) - found_ids)
return {
"images_found": len(rows),
"thumbs_to_unlink": sum(1 for r in rows if r.thumbnail_path),
"bytes_on_disk": sum(r.size_bytes for r in rows),
"missing_ids": missing,
}
def count_tag_associations(session: Session, *, tag_id: int) -> int:
"""COUNT(*) FROM image_tag WHERE tag_id=?. For Tier-B prompt."""
return session.execute(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id == tag_id)
).scalar_one()
def find_unused_tags(
session: Session, *, limit: int | None = None,
) -> list[Tag]:
"""Tags with no image_tag rows AND no series_page rows.
Sorted by name. Used by both dry-run preview and the live prune.
A tag is "unused" iff it has zero rows in image_tag AND zero rows
in series_page (so we don't accidentally prune a series tag that
happens to have no images yet).
"""
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
stmt = (
select(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
.order_by(Tag.name)
)
if limit is not None:
stmt = stmt.limit(limit)
return list(session.execute(stmt).scalars().all())
def unlink_image_files(
image: ImageRecord, images_root: Path,
) -> dict:
"""Best-effort unlink of all on-disk files for an ImageRecord.
Targets: image.path (original), image.thumbnail_path (cached
thumbnail), and the computed thumbs path at
/images/thumbs/<sha256[:3]>/<sha256>.(jpg|png|webp) (tries all
three extensions; missing extension is silently OK).
Returns {"original": bool, "thumbnail": bool}. Missing files
count as success (missing_ok semantics). OSErrors are swallowed
and reported as False so the calling DB delete still proceeds.
"""
out = {"original": False, "thumbnail": False}
if image.path:
try:
Path(image.path).unlink(missing_ok=True)
out["original"] = True
except OSError:
out["original"] = False
# Custom thumbnail_path (when set) — try it first.
if image.thumbnail_path:
try:
Path(image.thumbnail_path).unlink(missing_ok=True)
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
)
except OSError:
pass
return out
def delete_artist_cascade(
session: Session, *, artist_id: int, images_root: Path,
) -> dict:
"""Batched delete of an artist's images + the artist row.
Mirrors the cleanup_artist_async pattern: 500-row batches,
commit between batches so partial progress survives a worker
kill. Idempotent on missing artist (returns zeroed counts).
Postgres cascades handle image_tag / image_provenance /
series_page / tag_suggestion_rejection from ImageRecord delete,
and source / post / download_event / etc. from Artist delete
(via Artist.sources cascade="all, delete-orphan").
"""
artist = session.get(Artist, artist_id)
if artist is None:
return {
"artist": None,
"summary": {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"import_tasks_nulled": 0,
"files_failed": 0,
},
}
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
while True:
rows = session.execute(
select(ImageRecord)
.where(ImageRecord.artist_id == artist.id)
.limit(500)
).scalars().all()
if not rows:
break
for img in rows:
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
# ImportTask.result_image_id FK is SET NULL on image delete (Postgres
# handles this in the cascade above). We don't separately count those
# in FC-3k — the legacy cleanup_artist_async did it via
# source_path_prefix matching that's out of scope here.
import_tasks_nulled = 0
session.delete(artist)
session.commit()
return {
"artist": artist_info,
"summary": {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_nulled": import_tasks_nulled,
"files_failed": files_failed,
},
}
def delete_images(
session: Session, *, image_ids: list[int], images_root: Path,
) -> dict:
"""Delete a list of images in 500-row batches with commit between.
Postgres CASCADE on image_tag / image_provenance / series_page /
tag_suggestion_rejection / post_attachment(FK SET NULL) handles
the DB side; this function handles file unlinks first then row
deletes. Idempotent on missing IDs (returned as missing_ids;
no error). On partial OSError, the row is still deleted and
files_failed is incremented.
"""
if not image_ids:
return {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"files_failed": 0,
"missing_ids": [],
}
seen_ids: set[int] = set()
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
pending = list(image_ids)
while pending:
batch_ids = pending[:500]
pending = pending[500:]
rows = session.execute(
select(ImageRecord).where(ImageRecord.id.in_(batch_ids))
).scalars().all()
for img in rows:
seen_ids.add(img.id)
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
missing = sorted(set(image_ids) - seen_ids)
return {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"files_failed": files_failed,
"missing_ids": missing,
}
def delete_tag(session: Session, *, tag_id: int) -> dict:
"""Simple DELETE FROM tag WHERE id=?.
Postgres cascades the rest (image_tag, tag_alias, tag_allowlist,
tag_reference_embedding, tag_suggestion_rejection, series_page).
Returns counts BEFORE delete so the caller can surface them.
Raises LookupError if tag_id not found.
"""
tag = session.get(Tag, tag_id)
if tag is None:
raise LookupError(f"tag id not found: {tag_id}")
associations_count = count_tag_associations(session, tag_id=tag_id)
info = {"id": tag.id, "name": tag.name, "kind": tag.kind.value}
session.delete(tag)
session.commit()
return {"deleted": info, "associations_removed": associations_count}
def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
"""Find tags with zero references and (unless dry_run) delete them.
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
+113 -54
View File
@@ -1,26 +1,34 @@
"""Cursor-paginated gallery queries.
Cursor format: opaque base64-encoded "<iso8601_created_at>:<image_id>".
Pagination key is (created_at DESC, id DESC) so we don't drift when new
imports arrive between page loads. Decoding rejects malformed cursors with
a ValueError; the API layer translates that to HTTP 400.
Cursor format: opaque base64-encoded "<iso8601_effective_date>:<image_id>".
Pagination key is (effective_date DESC, id DESC) where effective_date is
COALESCE(post.post_date, image_record.created_at) so the gallery surfaces
images by ORIGINAL publish date when known, falling back to FC's scan
date. Important for migrated content: ~57k IR images scanned in a single
week would otherwise all share the same created_at and pile up in one
month bucket. The effective_date spreads them across the years they
were originally published.
Decoding rejects malformed cursors with a ValueError; the API layer
translates that to HTTP 400.
"""
import base64
from dataclasses import dataclass
from datetime import datetime
from sqlalchemy import and_, exists, func, or_, select
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ImageProvenance, ImageRecord, Source, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
def encode_cursor(created_at: datetime, image_id: int) -> str:
raw = f"{created_at.isoformat()}{CURSOR_SEPARATOR}{image_id}"
def encode_cursor(effective_date: datetime, image_id: int) -> str:
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
return base64.urlsafe_b64encode(raw.encode()).decode()
@@ -33,6 +41,26 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
raise ValueError(f"invalid cursor: {cursor!r}") from exc
def _effective_date_col():
"""SQL expression: COALESCE(post.post_date, image_record.created_at).
Used as the canonical sort/group/filter key across the gallery so
images backfilled with primary_post_id (e.g. via tag_apply phase 4)
surface at their original publish date, not their FC import date.
Images without a Post (or with Post.post_date NULL) fall back to
image_record.created_at and still order coherently against
post-attached ones.
"""
return func.coalesce(Post.post_date, ImageRecord.created_at)
def _outer_join_primary_post(stmt: Select) -> Select:
"""LEFT JOIN Post on ImageRecord.primary_post_id so the COALESCE
above sees Post.post_date when available. Images without a post
survive the join as NULL on the Post side; COALESCE handles it."""
return stmt.outerjoin(Post, Post.id == ImageRecord.primary_post_id)
@dataclass(frozen=True)
class GalleryImage:
id: int
@@ -41,7 +69,9 @@ class GalleryImage:
mime: str
width: int | None
height: int | None
created_at: datetime
created_at: datetime # FC's row-insert time
effective_date: datetime # COALESCE(post.post_date, created_at)
posted_at: datetime | None # post.post_date if known, else None
thumbnail_url: str
artist: dict | None = None
@@ -78,7 +108,7 @@ def _require_single_filter(tag_id, post_id, artist_id) -> None:
def _provenance_clause(post_id, artist_id):
"""Correlated EXISTS clause (NOT a join) so an image with multiple
matching provenance rows is returned exactly once and the
(created_at DESC, id DESC) cursor ordering is unaffected."""
(effective_date DESC, id DESC) cursor ordering is unaffected."""
if post_id is not None:
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
@@ -125,7 +155,9 @@ class GalleryService:
raise ValueError("limit must be between 1 and 200")
_require_single_filter(tag_id, post_id, artist_id)
stmt = select(ImageRecord)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
@@ -138,34 +170,38 @@ class GalleryService:
cur_ts, cur_id = decode_cursor(cursor)
stmt = stmt.where(
or_(
ImageRecord.created_at < cur_ts,
and_(ImageRecord.created_at == cur_ts, ImageRecord.id < cur_id),
eff < cur_ts,
and_(eff == cur_ts, ImageRecord.id < cur_id),
)
)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).scalars().all()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).all()
next_cursor = None
if len(rows) > limit:
last = rows[limit - 1]
next_cursor = encode_cursor(last.created_at, last.id)
last_record, _last_posted_at, last_eff = rows[limit - 1]
next_cursor = encode_cursor(last_eff, last_record.id)
rows = rows[:limit]
artists = await _artists_for(self.session, [r.id for r in rows])
artists = await _artists_for(
self.session, [r[0].id for r in rows]
)
images = [
GalleryImage(
id=r.id,
path=r.path,
sha256=r.sha256,
mime=r.mime,
width=r.width,
height=r.height,
created_at=r.created_at,
thumbnail_url=thumbnail_url(r.sha256, r.mime),
artist=artists.get(r.id),
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.sha256, record.mime),
artist=artists.get(record.id),
)
for r in rows
for record, posted_at, eff_date in rows
]
return GalleryPage(
images=images,
@@ -179,11 +215,13 @@ class GalleryService:
post_id: int | None = None,
artist_id: int | None = None,
) -> list[TimelineBucket]:
year_col = func.date_part("year", ImageRecord.created_at).label("yr")
month_col = func.date_part("month", ImageRecord.created_at).label("mo")
eff = _effective_date_col()
year_col = func.date_part("year", eff).label("yr")
month_col = func.date_part("month", eff).label("mo")
stmt = select(
year_col, month_col, func.count(ImageRecord.id).label("cnt")
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -201,14 +239,17 @@ class GalleryService:
post_id: int | None = None, artist_id: int | None = None,
) -> str | None:
"""Returns a cursor that, when passed to scroll(), positions at the
first image of the given year-month. None if the bucket is empty.
first image of the given year-month (by effective_date, not
created_at). None if the bucket is empty.
"""
from sqlalchemy import extract
stmt = select(ImageRecord).where(
extract("year", ImageRecord.created_at) == year,
extract("month", ImageRecord.created_at) == month,
eff = _effective_date_col()
stmt = select(ImageRecord, eff.label("eff")).where(
extract("year", eff) == year,
extract("month", eff) == month,
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -217,13 +258,14 @@ class GalleryService:
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).scalar_one_or_none()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).first()
if first is None:
return None
record, eff_date = first
# Cursor is exclusive; we encode a cursor with id+1 so the row itself
# is the first result in the next scroll().
return encode_cursor(first.created_at, first.id + 1)
return encode_cursor(eff_date, record.id + 1)
async def get_image_with_tags(self, image_id: int) -> dict | None:
record = await self.session.get(ImageRecord, image_id)
@@ -236,6 +278,13 @@ class GalleryService:
.order_by(Tag.kind.asc(), Tag.name.asc())
)
tags = (await self.session.execute(tag_stmt)).scalars().all()
# Fetch the canonical post.post_date for this image (if any) so
# the modal can show "Posted on <date>" alongside import date.
posted_at = None
if record.primary_post_id is not None:
posted_at = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
neighbors = await self._neighbors(record)
# Direct artist FK — used by the modal's ProvenancePanel as a
# fallback when ImageProvenance is empty (i.e., filesystem-
@@ -256,6 +305,7 @@ class GalleryService:
"size_bytes": record.size_bytes,
"integrity_status": record.integrity_status,
"created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.sha256, record.mime),
"image_url": f"/images/{record.path.split('/images/', 1)[-1]}",
"artist": (
@@ -275,34 +325,41 @@ class GalleryService:
}
async def _neighbors(self, record: ImageRecord) -> dict:
prev_stmt = (
select(ImageRecord.id)
.where(
# Compute the boundary image's effective_date in Python (one query
# below + the SELECT we already have on `record`) and use it for
# the neighbor comparison. Cheaper than re-deriving in SQL via
# correlated subquery.
boundary_eff = record.created_at
if record.primary_post_id is not None:
post_date = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
if post_date is not None:
boundary_eff = post_date
eff = _effective_date_col()
prev_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at > record.created_at,
eff > boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id > record.id,
),
)
)
.order_by(ImageRecord.created_at.asc(), ImageRecord.id.asc())
.limit(1)
)
next_stmt = (
select(ImageRecord.id)
.where(
).order_by(eff.asc(), ImageRecord.id.asc()).limit(1)
next_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at < record.created_at,
eff < boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id < record.id,
),
)
)
.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc())
.limit(1)
)
).order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
prev_id = (await self.session.execute(prev_stmt)).scalar_one_or_none()
next_id = (await self.session.execute(next_stmt)).scalar_one_or_none()
return {"prev_id": prev_id, "next_id": next_id}
@@ -311,9 +368,11 @@ class GalleryService:
def _group_by_year_month(
images: list[GalleryImage],
) -> list[tuple[int, int, list[int]]]:
"""Group by effective_date's year/month so migrated content surfaces
in the publish-date buckets, not the FC-scan-date bucket."""
groups: list[tuple[int, int, list[int]]] = []
for img in images:
y, m = img.created_at.year, img.created_at.month
y, m = img.effective_date.year, img.effective_date.month
if groups and groups[-1][0] == y and groups[-1][1] == m:
groups[-1][2].append(img.id)
else:
+22 -3
View File
@@ -280,7 +280,18 @@ class Importer:
)
if self.settings.skip_transparent and has_alpha:
pct = self._transparency_pct(source)
try:
pct = self._transparency_pct(source)
except OSError as exc:
# PIL.verify() at line 263 only validates header structure;
# truncated/corrupt pixel data only surfaces when load()
# actually decodes (here via getchannel('A')). Convert to
# invalid_image skip so the Celery autoretry loop doesn't
# bounce the same broken file forever.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during transparency check: {exc}",
)
if pct >= self.settings.transparency_threshold:
return ImportResult(
status="skipped", skip_reason=SkipReason.too_transparent,
@@ -302,8 +313,16 @@ class Importer:
# Perceptual near-dup (images only; videos keep phash NULL).
phash = None
if not is_video(source):
with Image.open(source) as im:
phash = compute_phash(im)
try:
with Image.open(source) as im:
phash = compute_phash(im)
except OSError as exc:
# Same rationale as the transparency-check guard above:
# broken-pixel-data files pass verify() but blow up here.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during phash compute: {exc}",
)
if phash is not None:
cand_rows = self.session.execute(
select(
+5 -1
View File
@@ -1,6 +1,10 @@
"""FC-5 migration tooling.
One module per concern (backup/rollback/gs/ir/overlap/ml_queue/verify).
One module per concern (gs/ir/overlap/ml_queue/verify/cleanup).
Each migrator returns a counts dict; the run_migration task wires
that dict into MigrationRun.counts so the UI polling shows progress.
backup + rollback were retired in FC-3h (2026-05-24); first-class
backup lives at backend/app/services/backup_service.py and exposes
its own /api/system/backup/* surface.
"""
-146
View File
@@ -1,146 +0,0 @@
"""pg_dump + tar.zst-based backup, restorable via pair of subprocess calls.
Backups live under <images_root>/_backups/. Each backup is two files
(SQL + tarball) plus a manifest JSON. Tagged backups (e.g. tag='pre_migration')
are how rollback.py finds the most recent restorable snapshot.
"""
from __future__ import annotations
import json
import shutil
import subprocess
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
_BACKUPS_DIRNAME = "_backups"
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy driver suffix so pg_dump/psql accept the URL.
SQLAlchemy uses URLs like `postgresql+psycopg://...` or
`postgresql+asyncpg://...`. libpq tools (pg_dump, psql) only know
the plain `postgresql://` scheme.
"""
for driver in ("postgresql+psycopg", "postgresql+asyncpg", "postgresql+psycopg2"):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path | None = None) -> Path:
# Overridable for tests via monkeypatch.
root = images_root if images_root is not None else Path("/images")
p = root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
_DEFAULT_SUBPROCESS_TIMEOUT_S = 30 * 60 # 30 minutes
def _run_subprocess(cmd: list[str], **kwargs: Any):
# Overridable for tests via monkeypatch. Hard wall-clock timeout
# guards against pg_dump / tar / zstd hangs on NFS — without it the
# task pretends to be 'running' forever (operator hit this 2026-05-
# 23 with two backups stuck in MigrationRun). On timeout
# subprocess.run raises TimeoutExpired which the caller surfaces as
# a task error.
return subprocess.run(
cmd,
capture_output=True,
check=True,
timeout=_DEFAULT_SUBPROCESS_TIMEOUT_S,
**{k: v for k, v in kwargs.items() if not k.startswith("_")},
)
def create_backup(
*, db_url: str, images_root: Path, tag: str = "manual",
) -> dict:
"""Create a backup: pg_dump SQL + tar.zst of images.
Returns a manifest dict. Writes <ts>.sql, <ts>.tar.zst, <ts>.json into
<images_root>/_backups/.
"""
ts = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_{ts}.sql"
tar_path = out_dir / f"fc_{ts}.tar.zst"
manifest_path = out_dir / f"fc_{ts}.json"
_run_subprocess(
["pg_dump", "--no-owner", "--no-acl", "-f", str(sql_path), _libpq_url(db_url)],
_test_ts=ts,
)
_run_subprocess(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
_test_ts=ts,
)
manifest = {
"backup_id": ts,
"tag": tag,
"created_at": datetime.now(UTC).isoformat(),
"sql_path": str(sql_path),
"tar_path": str(tar_path),
}
manifest_path.write_text(json.dumps(manifest, indent=2))
return manifest
def list_backups(images_root: Path) -> list[dict]:
out_dir = _backups_dir(images_root)
items = []
for mf in sorted(out_dir.glob("fc_*.json"), reverse=True):
try:
items.append(json.loads(mf.read_text()))
except Exception:
continue
return items
def find_latest_backup(images_root: Path, *, tag: str) -> dict | None:
for mf in list_backups(images_root):
if mf.get("tag") == tag:
return mf
return None
def restore_backup(
*, manifest: dict, db_url: str, images_root: Path,
) -> dict:
"""Restore from a backup manifest.
1. Replay the .sql via psql.
2. Wipe /images/ contents (except _backups/, which holds the file we're using).
3. Untar the .tar.zst into /images/.
"""
sql_path = Path(manifest["sql_path"])
tar_path = Path(manifest["tar_path"])
_run_subprocess(
["psql", "-d", _libpq_url(db_url), "-f", str(sql_path)],
)
# Wipe everything in images_root EXCEPT _backups/ (we'd delete the backup
# we're restoring from!).
for entry in images_root.iterdir():
if entry.name == _BACKUPS_DIRNAME:
continue
if entry.is_dir():
shutil.rmtree(entry)
else:
entry.unlink()
_run_subprocess(
["tar", "--zstd", "-xf", str(tar_path), "-C", str(images_root.parent)],
)
return {"restored_from": manifest["backup_id"]}
@@ -1,21 +0,0 @@
"""Restore from the most recent 'pre_migration'-tagged backup."""
from __future__ import annotations
from pathlib import Path
from . import backup as backup_mod
class NoBackupFoundError(Exception):
"""Raised when rollback() is called with no pre_migration backup on disk."""
def rollback_to_pre_migration(*, db_url: str, images_root: Path) -> dict:
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
raise NoBackupFoundError(
"no pre_migration-tagged backup found under <images_root>/_backups/"
)
return backup_mod.restore_backup(
manifest=manifest, db_url=db_url, images_root=images_root,
)
+35 -4
View File
@@ -37,13 +37,25 @@ from ...utils.slug import slugify
from .ir_ingest import manifest_path
# Per-platform artist-profile URL — used as Source.url when restoring
# IR PostMetadata into FC. Keep this table in sync with
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS and
# extension/lib/platforms.js.
# IR PostMetadata into FC. Must cover every platform that
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS
# recognizes; an entry missing here silently drops ALL PostMetadata for
# that platform during phase 4 (operator hit this 2026-05-25:
# DeviantArt + Pixiv posts in the IR migration produced empty
# ImageProvenance because they fell through this table).
#
# Pixiv caveat: the real profile URL takes a numeric user_id
# (https://www.pixiv.net/users/12345), but IR's PostMetadata.artist
# stores the display name not the id. We use the slugified name here
# so we preserve the artist→post→image linkage; the resulting Source.url
# won't resolve in a browser and the operator may want to manually fix
# it via Settings → Subscriptions once the migration lands.
_PLATFORM_PROFILE_URL = {
"patreon": "https://www.patreon.com/{slug}",
"subscribestar": "https://www.subscribestar.com/{slug}",
"hentaifoundry": "https://www.hentai-foundry.com/user/{slug}",
"deviantart": "https://www.deviantart.com/{slug}",
"pixiv": "https://www.pixiv.net/users/{slug}",
}
@@ -110,7 +122,14 @@ async def _ensure_provenance(
db: AsyncSession, *,
image_id: int, post_id: int, source_id: int, dry_run: bool,
) -> bool:
"""Returns True if a new ImageProvenance row was inserted."""
"""Returns True if a new ImageProvenance row was inserted.
Also sets ImageRecord.primary_post_id to this post if the image
doesn't already have one — preserves any primary_post_id already
assigned at download time by the importer (don't clobber). This is
the linkage gallery_service.py uses to surface Post.post_date as
the image's effective date for sort/group/jump/neighbor nav.
"""
existing = (await db.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == image_id,
@@ -118,6 +137,18 @@ async def _ensure_provenance(
ImageProvenance.source_id == source_id,
)
)).scalar_one_or_none()
# Whether-or-not the provenance row already exists, ensure the
# image's primary_post_id is set so the gallery date-coalesce works.
# Idempotent: only writes when currently NULL.
if not dry_run:
await db.execute(
ImageRecord.__table__.update()
.where(ImageRecord.id == image_id)
.where(ImageRecord.primary_post_id.is_(None))
.values(primary_post_id=post_id)
)
if existing is not None:
return False
if dry_run:
+82 -44
View File
@@ -4,12 +4,14 @@ CPU-only, single-image at a time. Loaded lazily inside the ml-worker
process; NOT thread-safe — the ml queue worker must run --concurrency=1
(set by the FC-1 entrypoint).
Camie's selected_tags.csv columns: tag_id,name,category,count
where category is a string: general|character|copyright|artist|meta|rating|year
(unlike WD14's integer Danbooru category ids).
v2 layout reference: HuggingFace Camais03/camie-tagger-v2 root has
camie-tagger-v2.onnx (789 MB) + camie-tagger-v2-metadata.json (7.77 MB)
+ config.json. Tags ship as nested JSON, not CSV. Preprocessing and
output handling follow the published onnx_inference.py reference:
ImageNet normalize, NCHW layout, sigmoid on refined logits (output[1]).
"""
import csv
import json
import os
from dataclasses import dataclass
from pathlib import Path
@@ -28,6 +30,8 @@ ImageFile.LOAD_TRUNCATED_IMAGES = True
MODEL_NAME = os.environ.get("CAMIE_MODEL_NAME", "camie-tagger-v2")
_MODEL_DIR = Path(os.environ.get("ML_MODEL_DIR", "/models")) / "camie"
_MODEL_FILE = f"{MODEL_NAME}.onnx"
_METADATA_FILE = f"{MODEL_NAME}-metadata.json"
# Below this confidence, predictions aren't stored (keeps the JSON compact).
STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
@@ -39,6 +43,12 @@ STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
# stored at STORE_FLOOR but artist never surfaces.
SURFACED_CATEGORIES = {"character", "copyright", "general"}
# ImageNet preprocessing constants (per Camie v2 onnx_inference.py).
_IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
_IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
# Square-pad color ≈ ImageNet mean × 255 (matches reference inference).
_PAD_COLOR = (124, 116, 104)
@dataclass(frozen=True)
class TagPrediction:
@@ -51,34 +61,48 @@ class Tagger:
def __init__(self, model_dir: Path | None = None):
self._model_dir = model_dir or _MODEL_DIR
self._session = None # onnxruntime.InferenceSession once load()ed
self._tag_meta: list[dict] | None = None
self._tag_names: list[str] | None = None
self._tag_categories: list[str] | None = None
self._input_name: str | None = None
self._output_name: str | None = None
self._input_size: int = 448
self._input_size: int = 512
def load(self) -> None:
if self._session is not None:
return
model_path = self._model_dir / "model.onnx"
tags_path = self._model_dir / "selected_tags.csv"
model_path = self._model_dir / _MODEL_FILE
meta_path = self._model_dir / _METADATA_FILE
if not model_path.is_file():
raise RuntimeError(
f"Camie model.onnx missing at {model_path}. "
f"Camie {_MODEL_FILE} missing at {model_path}. "
f"Populate /models via the ml-worker downloader."
)
if not tags_path.is_file():
if not meta_path.is_file():
raise RuntimeError(
f"Camie selected_tags.csv missing at {tags_path}. "
f"Camie {_METADATA_FILE} missing at {meta_path}. "
f"Populate /models via the ml-worker downloader."
)
tag_meta: list[dict] = []
with open(tags_path, newline="") as f:
reader = csv.DictReader(f)
for row in reader:
tag_meta.append(
{"name": row["name"], "category": row["category"]}
)
with open(meta_path) as f:
metadata = json.load(f)
# Per Camie v2 onnx_inference.py: idx_to_tag is keyed by str(idx);
# tag_to_category maps tag_name -> category. Project to two parallel
# lists indexed by output position for O(1) lookup in the hot path.
ds = metadata["dataset_info"]
idx_to_tag = ds["tag_mapping"]["idx_to_tag"]
tag_to_category = ds["tag_mapping"]["tag_to_category"]
total = ds["total_tags"]
names: list[str] = []
cats: list[str] = []
for i in range(total):
name = idx_to_tag.get(str(i), f"unknown-{i}")
names.append(name)
cats.append(tag_to_category.get(name, "general"))
# Input size from metadata; fall back to 512 (the v2 default).
self._input_size = int(
metadata.get("model_info", {}).get("img_size", 512)
)
# Lazy import — kept after the file-existence checks so the
# missing-model RuntimeError still fires first in environments
@@ -89,51 +113,65 @@ class Tagger:
str(model_path), providers=["CPUExecutionProvider"]
)
self._input_name = session.get_inputs()[0].name
self._output_name = session.get_outputs()[0].name
input_shape = session.get_inputs()[0].shape
for dim in input_shape:
if isinstance(dim, int) and dim > 1:
self._input_size = dim
break
# Assign sentinels last so a partial load isn't observable.
self._tag_meta = tag_meta
self._tag_names = names
self._tag_categories = cats
self._session = session
def _preprocess(self, image_path: Path) -> np.ndarray:
img = Image.open(image_path)
# Camie handles RGBA natively but we still composite onto white so
# transparency doesn't bias the model (same as IR's WD14 path).
if img.mode != "RGBA":
img = img.convert("RGBA")
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
# Composite RGBA onto neutral so transparency doesn't bias the model.
if img.mode == "RGBA":
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
elif img.mode != "RGB":
img = img.convert("RGB")
# Pad to square with ImageNet-mean color, then bicubic resize.
w, h = img.size
side = max(w, h)
square = Image.new("RGB", (side, side), (255, 255, 255))
square = Image.new("RGB", (side, side), _PAD_COLOR)
square.paste(img, ((side - w) // 2, (side - h) // 2))
square = square.resize(
(self._input_size, self._input_size), Image.BICUBIC
)
arr = np.array(square, dtype=np.float32)
return arr[np.newaxis, :, :, :] # NHWC
arr = np.array(square, dtype=np.float32) / 255.0 # HWC, [0,1]
arr = (arr - _IMAGENET_MEAN) / _IMAGENET_STD # ImageNet normalize
arr = arr.transpose(2, 0, 1) # HWC -> CHW
return arr[np.newaxis, :, :, :] # NCHW
def infer(self, image_path: Path) -> dict[str, TagPrediction]:
"""Run Camie on one image. Returns {name: TagPrediction}, only
entries with confidence >= STORE_FLOOR (across all categories —
the suggestion service does category filtering later)."""
"""Run Camie v2 on one image. Returns {name: TagPrediction} with
confidence >= STORE_FLOOR (across all categories — the suggestion
service does category filtering later).
v2 emits multiple outputs; we use the refined predictions
(output[1] per onnx_inference.py). Sigmoid is applied to raw
logits to produce [0,1] confidence scores.
"""
self.load()
x = self._preprocess(image_path)
out = self._session.run([self._output_name], {self._input_name: x})[0][0]
outputs = self._session.run(None, {self._input_name: x})
# Refined predictions if present (v2 emits initial + refined),
# fall back to initial for single-output forks.
logits = outputs[1] if len(outputs) > 1 else outputs[0]
# Squeeze batch dim, apply sigmoid.
probs = 1.0 / (1.0 + np.exp(-logits[0]))
results: dict[str, TagPrediction] = {}
for idx, score in enumerate(out):
names = self._tag_names
cats = self._tag_categories
for idx, score in enumerate(probs):
conf = float(score)
if conf < STORE_FLOOR:
continue
meta = self._tag_meta[idx]
results[meta["name"]] = TagPrediction(
name=meta["name"], category=meta["category"], confidence=conf
if idx >= len(names):
# Output longer than metadata declared — shouldn't happen but
# don't crash the import pipeline if v2 metadata desynchronizes.
continue
results[names[idx]] = TagPrediction(
name=names[idx], category=cats[idx], confidence=conf
)
return results
+57
View File
@@ -0,0 +1,57 @@
"""FC-3k: admin destructive Celery tasks.
Two long-running ops on the maintenance queue. task_run lifecycle is
captured automatically by FC-3i signals — these tasks just return
their summary dict so it lands in task_run.metadata (via Celery's
result backend) for the dashboard to surface.
Soft/hard time limits inherit the FC-3i recovery sweep: a runaway
task gets killed and flipped to status='timeout' by
recover_stalled_task_runs.
"""
from __future__ import annotations
import logging
from pathlib import Path
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..services import cleanup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
@celery.task(
name="backend.app.tasks.admin.delete_artist_cascade_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
def delete_artist_cascade_task(self, *, artist_id: int) -> dict:
"""Wraps cleanup_service.delete_artist_cascade. Returns the
service's summary dict for FC-3i task_run.metadata capture."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_artist_cascade(
session, artist_id=artist_id, images_root=IMAGES_ROOT,
)
@celery.task(
name="backend.app.tasks.admin.bulk_delete_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=900, time_limit=1200, # 15 min / 20 min
)
def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
"""Wraps cleanup_service.delete_images."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_images(
session, image_ids=image_ids, images_root=IMAGES_ROOT,
)
+300
View File
@@ -0,0 +1,300 @@
"""FC-3h: backup/restore Celery tasks.
All tasks live on the maintenance queue (per celery_app.task_routes).
task_run lifecycle tracking is automatic via FC-3i signals — these
tasks just record the operator-facing artifact metadata into
BackupRun.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from pathlib import Path
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import BackupRun, ImportSettings
from ..services import backup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
"""Flip a BackupRun row from running/restoring to error with a
truncated error message and finished_at. Caller already holds the
session open."""
row.status = "error"
row.error = f"{type(exc).__name__}: {exc}"[:2000]
row.finished_at = datetime.now(UTC)
session.add(row)
session.commit()
@celery.task(
name="backend.app.tasks.backup.backup_db_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=10, retry_backoff_max=120, max_retries=2,
soft_time_limit=600, time_limit=720,
)
def backup_db_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one DB backup. Returns {'backup_run_id': N}."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="db", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_db(
db_url=cfg.database_url_sync, images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.sql_path = result["sql_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.backup_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=30, retry_backoff_max=300, max_retries=1,
soft_time_limit=21600, time_limit=23400,
)
def backup_images_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one images backup. Same shape as backup_db_task; uses
tar_path instead of sql_path."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="images", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_images(
images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.tar_path = result["tar_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.restore_db_task",
bind=True,
max_retries=0, # NEVER auto-retry a half-applied restore.
soft_time_limit=1200, time_limit=1800,
)
def restore_db_task(self, *, source_backup_run_id: int) -> dict:
"""Restore from a previous DB backup. Inserts a NEW BackupRun row
(kind='db', status='restoring') linked to the source via
restored_from_id; flips to 'restored' on success or 'error' on
failure. Operator sees the restore as a row in the dashboard."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "db" or not src.sql_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid DB backup"
)
marker = BackupRun(
kind="db", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_sql_path": src.sql_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
sql_path = src.sql_path
try:
backup_service.restore_db(
db_url=cfg.database_url_sync, sql_path=Path(sql_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.restore_images_task",
bind=True, max_retries=0,
soft_time_limit=21600, time_limit=23400,
)
def restore_images_task(self, *, source_backup_run_id: int) -> dict:
"""Mirrors restore_db_task; uses backup_service.restore_images."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "images" or not src.tar_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid images backup"
)
marker = BackupRun(
kind="images", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_tar_path": src.tar_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
tar_path = src.tar_path
try:
backup_service.restore_images(
images_root=IMAGES_ROOT, tar_path=Path(tar_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.prune_backups",
soft_time_limit=300, time_limit=600,
)
def prune_backups() -> dict:
"""Daily Beat. Per-kind retention from ImportSettings.
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
for kind, keep in (
("db", s.backup_db_keep_last_n),
("images", s.backup_images_keep_last_n),
):
candidates = session.execute(
select(BackupRun)
.where(BackupRun.kind == kind)
.where(BackupRun.tag.is_(None))
.where(BackupRun.status.in_(["ok", "error"]))
.order_by(BackupRun.started_at.desc())
.offset(keep)
).scalars().all()
for row in candidates:
result = backup_service.unlink_artifact_files(
sql_path=row.sql_path,
tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
counts["files_unlinked"] += sum(
1 for v in result.values() if v
)
session.delete(row)
counts[f"{kind}_deleted"] += 1
session.commit()
return counts
@celery.task(
name="backend.app.tasks.backup.backup_db_nightly",
soft_time_limit=60, time_limit=120,
)
def backup_db_nightly() -> dict:
"""Hourly tick. Dispatches a real backup ONLY if the configured
UTC hour matches and the nightly setting is enabled. Returns
either {'skipped': '<reason>'} or {'dispatched': '<task_id>'}."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
nightly_enabled = s.backup_db_nightly_enabled
configured_hour = s.backup_db_nightly_hour_utc
if not nightly_enabled:
return {"skipped": "nightly disabled"}
now_hour = datetime.now(UTC).hour
if now_hour != configured_hour:
return {"skipped": f"hour={now_hour} != configured={configured_hour}"}
res = backup_db_task.delay(triggered_by="nightly")
return {"dispatched": res.id}
+58 -20
View File
@@ -16,6 +16,7 @@ from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
STUCK_THRESHOLD_MINUTES = 5
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7
PHASH_PAGE = 500
VERIFY_PAGE = 200
@@ -26,40 +27,77 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
@celery.task(name="backend.app.tasks.maintenance.recover_interrupted_tasks")
def recover_interrupted_tasks() -> int:
"""Find ImportTask rows stuck in 'processing' for >5 min and re-queue them.
"""Recover stuck ImportTask rows. Two distinct stuck states:
Why 5 min: import_media_file is sub-second for the vast majority of
files; even a large-video transcode caps at the per-task soft_time_limit
(5 min) defined on the task itself. Anything still 'processing' after
that window is a confirmed crash (worker died, DB disconnect mid-flush,
OOM) and must be recycled. Was 30 min historically; tightened
2026-05-24 after operator hit a 2224-row zombie pile during the IR
migration scan.
1. 'processing' > 5 min — worker crash mid-import. Re-queue via
.delay() and let the import retry. Was 30 min historically;
tightened 2026-05-24 after operator hit a 2224-row zombie pile.
import_media_file is sub-second for the vast majority of files and
capped at the per-task soft_time_limit (5 min), so anything still
'processing' after that window is a confirmed crash.
2. 'pending' or 'queued' > 30 min — enqueue-phase crash. scan_directory
creates rows with status='pending' (commit), then in a second pass
transitions to 'queued' and calls .delay() (commit). If the scanner
crashes between those two commits, rows are orphaned in 'pending'
(never enqueued) with no recovery path — invisible to the
'processing' sweep above. Flagged 2026-05-25 by operator hitting a
5490-row orphan pile. Flip these to 'failed' (not re-enqueue) so
the operator drains them via /api/import/retry-failed at their own
pace; bulk-re-enqueueing 5000+ rows would thundering-herd the
import worker.
Returns total rows touched (recovered + marked failed).
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
now = datetime.now(UTC)
processing_cutoff = now - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
orphan_cutoff = now - timedelta(minutes=ORPHAN_PENDING_THRESHOLD_MINUTES)
with SessionLocal() as session:
stuck_ids = session.execute(
select(ImportTask.id)
.where(ImportTask.status == "processing")
.where(ImportTask.started_at < cutoff)
.where(ImportTask.started_at < processing_cutoff)
).scalars().all()
if not stuck_ids:
orphan_ids = session.execute(
select(ImportTask.id)
.where(ImportTask.status.in_(["pending", "queued"]))
.where(ImportTask.created_at < orphan_cutoff)
).scalars().all()
if not stuck_ids and not orphan_ids:
return 0
session.execute(
update(ImportTask)
.where(ImportTask.id.in_(stuck_ids))
.values(status="queued", started_at=None, error="recovered from stuck state")
)
if stuck_ids:
session.execute(
update(ImportTask)
.where(ImportTask.id.in_(stuck_ids))
.values(status="queued", started_at=None, error="recovered from stuck state")
)
if orphan_ids:
session.execute(
update(ImportTask)
.where(ImportTask.id.in_(orphan_ids))
.values(
status="failed",
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
"/api/import/retry-failed"
),
)
)
session.commit()
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
if stuck_ids:
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
return len(stuck_ids)
return len(stuck_ids) + len(orphan_ids)
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_tasks")
+6 -32
View File
@@ -4,7 +4,8 @@ Dispatches to the right migrator based on `kind`. Updates MigrationRun
row's status/counts/finished_at as it runs. Failures set status='error'
with the error message preserved.
kinds: backup, gs_ingest, ir_ingest, tag_apply, ml_queue, verify, rollback, cleanup
kinds: gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup
(backup + rollback retired 2026-05-24 → see /api/system/backup/*)
"""
from __future__ import annotations
@@ -20,10 +21,8 @@ from ..celery_app import celery
from ..config import get_config
from ..models import MigrationRun
from ..services.credential_crypto import CredentialCrypto
from ..services.migrators import backup as backup_mod
from ..services.migrators import cleanup as cleanup_mod
from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify
from ..services.migrators import rollback as rollback_mod
log = logging.getLogger(__name__)
@@ -65,21 +64,11 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
async with factory() as db:
await _update_run(db, run_id, status="running")
try:
if kind == "backup":
manifest = backup_mod.create_backup(
db_url=get_config().database_url_sync,
images_root=IMAGES_ROOT,
tag=params.get("tag", "manual"),
if kind in ("backup", "rollback"):
raise ValueError(
f"kind {kind!r} retired in FC-3h; "
"use /api/system/backup/* instead"
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"manifest": manifest},
)
return manifest
elif kind == "gs_ingest":
fc_crypto = CredentialCrypto(_KEY_PATH)
@@ -166,21 +155,6 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
)
return result
elif kind == "rollback":
result = rollback_mod.rollback_to_pre_migration(
db_url=get_config().database_url_sync,
images_root=IMAGES_ROOT,
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"rollback_result": result},
)
return result
else:
raise ValueError(f"unknown kind: {kind}")
@@ -0,0 +1,95 @@
<template>
<v-card class="fc-danger-zone mt-8" variant="outlined">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-alert-octagon" color="error" size="small" />
<span>Danger zone</span>
</v-card-title>
<v-card-text>
<p class="text-body-2 fc-muted mb-4">
Cascade-delete this artist and every image, source, post, and
attachment associated with them. This cannot be undone.
Recoverable only from an FC-3h backup.
</p>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-forever"
:loading="loading"
@click="onClick"
>Delete artist &amp; cascade</v-btn>
<DestructiveConfirmModal
v-model="modalOpen"
action="delete"
kind="artist"
:run-id="artistId"
tier="C"
:projected-counts="projectedCounts"
:description="modalDescription"
@confirm="onConfirm"
/>
</v-card-text>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useRouter } from 'vue-router'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useAdminStore } from '../../stores/admin.js'
const props = defineProps({
slug: { type: String, required: true },
artistId: { type: Number, required: true },
artistName: { type: String, required: true },
})
const router = useRouter()
const store = useAdminStore()
const loading = ref(false)
const modalOpen = ref(false)
const projected = ref(null)
const projectedCounts = computed(() => projected.value?.projected || null)
const modalDescription = computed(
() => projected.value
? `Artist “${props.artistName}” — `
+ `${projected.value.projected.images} images, `
+ `${projected.value.projected.sources} sources, `
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
async function onClick() {
loading.value = true
try {
projected.value = await store.projectArtistCascade(props.slug)
modalOpen.value = true
} finally {
loading.value = false
}
}
async function onConfirm(token) {
loading.value = true
try {
const result = await store.dispatchArtistCascade(props.slug, token)
const taskId = result.task_id
router.push('/artists')
if (taskId) {
store.pollTaskUntilDone(taskId).catch(() => {})
}
} finally {
loading.value = false
}
}
</script>
<style scoped>
.fc-danger-zone {
border-color: rgb(var(--v-theme-error));
border-radius: 8px;
}
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -38,9 +38,18 @@ function onCardClick() {
.fc-artistcard { cursor: pointer; }
.fc-artistcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute. */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-artistcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-artistcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-artistcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
+47 -4
View File
@@ -44,7 +44,31 @@
</div>
<div class="fc-tagcard__meta">
<v-chip size="x-small" label>{{ card.kind }}</v-chip>
<span class="fc-tagcard__count">{{ card.image_count }}</span>
<div class="fc-tagcard__meta-right">
<span class="fc-tagcard__count">{{ card.image_count }}</span>
<v-menu>
<template #activator="{ props: act }">
<v-btn
class="fc-tagcard__menu"
icon="mdi-dots-vertical" size="x-small" variant="text"
v-bind="act" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item
title="Merge with…"
prepend-icon="mdi-call-merge"
@click="$emit('merge-with', card)"
/>
<v-list-item
title="Delete tag"
prepend-icon="mdi-delete"
base-color="error"
@click="$emit('delete', card)"
/>
</v-list>
</v-menu>
</div>
</div>
</v-card-text>
</v-card>
@@ -54,7 +78,7 @@
import { ref } from 'vue'
const props = defineProps({ card: { type: Object, required: true } })
const emit = defineEmits(['open', 'rename', 'manage', 'read'])
const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
const editing = ref(false)
const draft = ref('')
@@ -82,9 +106,19 @@ function submit() {
.fc-tagcard { cursor: pointer; }
.fc-tagcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute
(older Safari, embedded webviews). */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-tagcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-tagcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-tagcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
@@ -106,4 +140,13 @@ function submit() {
}
.fc-tagcard:hover .fc-tagcard__edit { opacity: .6; }
.fc-tagcard__edit:hover { opacity: 1; }
.fc-tagcard__meta-right {
display: flex; align-items: center; gap: 4px;
}
.fc-tagcard__menu {
opacity: 0;
transition: opacity .15s ease;
}
.fc-tagcard:hover .fc-tagcard__menu { opacity: .6; }
.fc-tagcard__menu:hover { opacity: 1; }
</style>
@@ -55,19 +55,44 @@
</template>
</div>
<div class="fc-bulk-panel__section">
<h4>Destructive</h4>
<v-btn
color="error" variant="flat" rounded="pill" block
prepend-icon="mdi-delete-forever"
:disabled="!sel.count"
:loading="deleting"
@click="onDeleteClick"
>Delete {{ sel.count }} selected</v-btn>
</div>
<div class="fc-bulk-panel__foot">
<v-btn variant="text" block @click="sel.clear()">Clear selection</v-btn>
</div>
<DestructiveConfirmModal
v-model="deleteModalOpen"
action="delete"
kind="images-selection"
:run-id="bulkToken"
tier="C"
:projected-counts="bulkProjectedCounts"
:description="bulkDescription"
@confirm="onDeleteConfirm"
/>
</aside>
</template>
<script setup>
import { ref, watch } from 'vue'
import { computed, ref, watch } from 'vue'
import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
import { useAdminStore } from '../../stores/admin.js'
import { useApi } from '../../composables/useApi.js'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
const sel = useGallerySelectionStore()
const api = useApi()
const adminStore = useAdminStore()
const addModel = ref(null)
const addHits = ref([])
@@ -99,6 +124,66 @@ async function onAddPick(id) {
watch(() => sel.order.length, () => {
if (sel.isSelectMode) sel.refresh()
})
// --- FC-3k bulk delete -----------------------------------------------
const deleting = ref(false)
const deleteModalOpen = ref(false)
const bulkProjected = ref(null)
const bulkToken = ref('')
const bulkProjectedCounts = computed(() => bulkProjected.value
? {
images: bulkProjected.value.images_found,
thumbnails: bulkProjected.value.thumbs_to_unlink,
bytes: bulkProjected.value.bytes_on_disk,
}
: null,
)
const bulkDescription = computed(
() => bulkProjected.value
? `${bulkProjected.value.images_found} images, `
+ `${Math.round(bulkProjected.value.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
async function _computeSha8(ids) {
const canon = [...ids].sort((a, b) => a - b).join(',')
const buf = new TextEncoder().encode(canon)
const hashBuf = await crypto.subtle.digest('SHA-256', buf)
const bytes = new Uint8Array(hashBuf)
let hex = ''
for (let i = 0; i < 4; i++) {
hex += bytes[i].toString(16).padStart(2, '0')
}
return hex
}
async function onDeleteClick() {
if (!sel.order.length) return
deleting.value = true
try {
bulkProjected.value = await adminStore.projectBulkImageDelete(sel.order)
bulkToken.value = await _computeSha8(sel.order)
deleteModalOpen.value = true
} finally {
deleting.value = false
}
}
async function onDeleteConfirm(token) {
deleting.value = true
try {
const result = await adminStore.dispatchBulkImageDelete(sel.order, token)
const taskId = result.task_id
if (taskId) {
adminStore.pollTaskUntilDone(taskId).catch(() => {})
}
sel.clear()
} finally {
deleting.value = false
}
}
</script>
<style scoped>
@@ -0,0 +1,128 @@
<template>
<v-dialog
:model-value="modelValue"
max-width="520" persistent
@update:model-value="$emit('update:modelValue', $event)"
>
<v-card>
<v-card-title>
{{ titleVerb }}
{{ kindLabel }}<span v-if="runId"> #{{ runId }}</span>
</v-card-title>
<v-card-text>
<v-alert
:type="alertType"
variant="tonal" density="compact" class="mb-3"
>
<strong>{{ warningText }}</strong>
<div v-if="description" class="mt-1">{{ description }}</div>
</v-alert>
<div v-if="projectedCounts" class="fc-counts mb-3">
<div v-for="(v, k) in projectedCounts" :key="k">
<span class="fc-counts-key">{{ k }}:</span>
<span class="fc-counts-val">{{ v }}</span>
</div>
</div>
<template v-if="tier === 'C'">
<div class="text-body-2 mb-2">Type the following to confirm:</div>
<div class="fc-token mb-3">{{ expectedToken }}</div>
<v-text-field
v-model="typed"
variant="outlined" density="compact" hide-details
autofocus
placeholder="paste the token above"
/>
</template>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="onCancel">Cancel</v-btn>
<v-btn
:color="confirmColor"
variant="flat" rounded="pill"
:disabled="!canConfirm"
@click="onConfirm"
>
{{ titleVerb }}
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
const props = defineProps({
modelValue: { type: Boolean, required: true },
action: { type: String, required: true }, // 'restore' | 'delete'
kind: { type: String, required: true }, // 'db' | 'images' | 'artist' | 'tag' | 'images-selection'
runId: { type: [Number, String], default: '' }, // numeric id or sha8 string
description: { type: String, default: '' },
tier: { type: String, default: 'C' }, // 'B' | 'C'
projectedCounts: { type: Object, default: null },
})
const emit = defineEmits(['update:modelValue', 'confirm'])
const typed = ref('')
const expectedToken = computed(
() => `${props.action}-${props.kind}-${props.runId}`,
)
const titleVerb = computed(
() => props.action === 'restore' ? 'Restore' : 'Delete',
)
const kindLabel = computed(() => ({
db: 'database backup',
images: 'images backup',
artist: 'artist',
tag: 'tag',
'images-selection': 'image selection',
}[props.kind] || props.kind))
const alertType = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const confirmColor = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const warningText = computed(() => (
props.action === 'restore'
? 'This replaces current state with the backup. There is no undo.'
: 'This permanently deletes the listed items. Cannot be recovered.'
))
const canConfirm = computed(
() => props.tier === 'B'
? true
: typed.value === expectedToken.value,
)
watch(() => props.modelValue, (open) => {
if (open) typed.value = ''
})
function onCancel() {
emit('update:modelValue', false)
}
function onConfirm() {
emit('confirm', expectedToken.value)
emit('update:modelValue', false)
}
</script>
<style scoped>
.fc-token {
font-family: 'JetBrains Mono', monospace;
background: rgb(var(--v-theme-surface-light));
padding: 6px 10px; border-radius: 4px;
font-size: 14px; word-break: break-all;
}
.fc-counts {
display: grid; grid-template-columns: max-content auto;
gap: 4px 12px;
font-size: 13px;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-counts-key { font-weight: 500; text-transform: capitalize; }
.fc-counts-val { font-variant-numeric: tabular-nums; }
</style>
@@ -0,0 +1,205 @@
<template>
<v-card class="fc-backup-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-database-export" size="small" />
<span>Backups</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-4">
pg_dump for the database (fast, nightly-schedulable);
tar+zstd for the images (slow, manual only). Files live in
<code>/images/_backups/</code>. Tag a backup to protect it
from autoprune.
</p>
<v-alert
v-if="store.lastError"
type="warning" variant="tonal" density="compact" class="mb-3"
>
{{ store.lastError }}
</v-alert>
<!-- Database section -->
<h3 class="fc-section-title">Database</h3>
<div class="fc-settings-row mb-3">
<v-switch
:model-value="settings.backup_db_nightly_enabled"
color="accent" density="compact" hide-details
label="Nightly"
@update:model-value="onSettingChange('backup_db_nightly_enabled', $event)"
/>
<v-text-field
:model-value="settings.backup_db_nightly_hour_utc"
label="Hour (UTC)" type="number" min="0" max="23"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_db_nightly_hour_utc', Number($event))"
/>
<v-text-field
:model-value="settings.backup_db_keep_last_n"
label="Keep last" type="number" min="1" max="365"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_db_keep_last_n', Number($event))"
/>
</div>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-database-arrow-up" class="mb-3"
:loading="dbTriggering"
@click="onTrigger('db')"
>Run DB backup now</v-btn>
<BackupRunsTable
:runs="store.dbRuns"
@restore="onRestore"
@delete="onDelete"
@tag="onTag"
/>
<v-divider class="my-4" />
<!-- Images section -->
<h3 class="fc-section-title">Images</h3>
<div class="fc-settings-row mb-3">
<v-text-field
:model-value="settings.backup_images_keep_last_n"
label="Keep last" type="number" min="1" max="100"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_images_keep_last_n', Number($event))"
/>
</div>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-folder-zip" class="mb-3"
:loading="imagesTriggering"
@click="onTrigger('images')"
>Run images backup now</v-btn>
<span class="text-caption fc-muted ml-2">
Hours for a 200GB+ library.
</span>
<BackupRunsTable
:runs="store.imagesRuns"
@restore="onRestore"
@delete="onDelete"
@tag="onTag"
/>
</v-card-text>
<BackupConfirmModal
v-model="confirmOpen"
:action="confirmAction"
:kind="confirmKind"
:run-id="confirmRunId"
:description="confirmDescription"
@confirm="onConfirmSubmit"
/>
</v-card>
</template>
<script setup>
import { computed, onMounted, onUnmounted, ref } from 'vue'
import { useBackupStore } from '../../stores/backup.js'
import BackupConfirmModal from '../modal/DestructiveConfirmModal.vue'
import BackupRunsTable from './BackupRunsTable.vue'
const store = useBackupStore()
const settings = computed(() => store.settings || {
backup_db_nightly_enabled: false,
backup_db_nightly_hour_utc: 3,
backup_db_keep_last_n: 14,
backup_images_keep_last_n: 3,
})
const dbTriggering = ref(false)
const imagesTriggering = ref(false)
const confirmOpen = ref(false)
const confirmAction = ref('restore')
const confirmKind = ref('db')
const confirmRunId = ref(0)
const confirmDescription = ref('')
let pollId = null
function pollOnce() {
if (document.hidden) return
store.loadRuns('db')
store.loadRuns('images')
}
onMounted(() => {
store.loadSettings()
pollOnce()
pollId = setInterval(pollOnce, 5000)
})
onUnmounted(() => {
if (pollId) { clearInterval(pollId); pollId = null }
})
async function onTrigger(kind) {
const triggeringRef = kind === 'db' ? dbTriggering : imagesTriggering
triggeringRef.value = true
try {
await store.triggerBackup(kind)
pollOnce()
} finally {
triggeringRef.value = false
}
}
async function onSettingChange(field, value) {
await store.patchSettings({ [field]: value })
}
function _openConfirm(action, run) {
confirmAction.value = action
confirmKind.value = run.kind
confirmRunId.value = run.id
confirmDescription.value = (
action === 'restore'
? `Source artifact: ${run.kind === 'db' ? run.sql_path : run.tar_path}`
: 'Removes the BackupRun row and unlinks the artifact files.'
)
confirmOpen.value = true
}
function onRestore(run) { _openConfirm('restore', run) }
function onDelete(run) { _openConfirm('delete', run) }
async function onTag(run) {
const next = prompt(
`Tag for backup #${run.id} (blank to clear):`,
run.tag || '',
)
if (next === null) return
await store.setTag(run.id, next.trim() || null)
pollOnce()
}
async function onConfirmSubmit(token) {
try {
if (confirmAction.value === 'restore') {
await store.restore(confirmRunId.value, token)
} else {
await store.deleteRun(confirmRunId.value, token)
}
pollOnce()
} catch {
// store.lastError already updated; surfaced in v-alert above.
}
}
</script>
<style scoped>
.fc-backup-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-title {
font-family: 'Fraunces', Georgia, serif;
font-size: 16px; font-weight: 500;
color: rgb(var(--v-theme-on-surface));
margin-bottom: 8px;
}
.fc-settings-row {
display: flex; gap: 12px; align-items: center; flex-wrap: wrap;
}
</style>
@@ -0,0 +1,109 @@
<template>
<v-table density="compact" class="fc-backup-table">
<thead>
<tr>
<th>When</th>
<th>Status</th>
<th class="text-right">Duration</th>
<th class="text-right">Size</th>
<th>Tag</th>
<th class="text-right">Actions</th>
</tr>
</thead>
<tbody>
<tr v-for="r in runs" :key="r.id">
<td class="fc-tabular" :title="r.started_at">
{{ formatRelative(r.started_at) }}
</td>
<td>
<v-icon size="small" :color="statusColor(r.status)">
{{ statusIcon(r.status) }}
</v-icon>
{{ r.status }}
</td>
<td class="text-right fc-tabular">{{ formatDuration(r) }}</td>
<td class="text-right fc-tabular">{{ formatBytes(r.size_bytes) }}</td>
<td>
<v-chip
v-if="r.tag" size="x-small" color="accent" variant="tonal"
>{{ r.tag }}</v-chip>
<span v-else class="fc-muted"></span>
</td>
<td class="text-right">
<v-menu>
<template #activator="{ props: act }">
<v-btn icon="mdi-dots-vertical" size="x-small" variant="text" v-bind="act" />
</template>
<v-list density="compact">
<v-list-item
:title="r.tag ? 'Untag' : 'Tag…'"
@click="$emit('tag', r)"
/>
<v-list-item
title="Restore…" :disabled="r.status !== 'ok'"
@click="$emit('restore', r)"
/>
<v-list-item
title="Delete…"
@click="$emit('delete', r)"
/>
</v-list>
</v-menu>
</td>
</tr>
<tr v-if="!runs.length">
<td colspan="6" class="text-center fc-muted py-4">No backups yet.</td>
</tr>
</tbody>
</v-table>
</template>
<script setup>
defineProps({ runs: { type: Array, default: () => [] } })
defineEmits(['restore', 'delete', 'tag'])
function statusIcon(s) {
return {
ok: 'mdi-check-circle', error: 'mdi-close-circle',
running: 'mdi-timer-sand', restoring: 'mdi-restore-clock',
restored: 'mdi-restore', pending: 'mdi-clock-outline',
}[s] || 'mdi-help-circle'
}
function statusColor(s) {
return {
ok: 'success', error: 'error',
running: 'accent', restoring: 'info',
restored: 'info', pending: 'on-surface-variant',
}[s] || 'on-surface-variant'
}
function formatDuration(r) {
if (r.duration_seconds == null) return '—'
const s = r.duration_seconds
if (s < 60) return `${s}s`
if (s < 3600) return `${Math.floor(s / 60)}m ${s % 60}s`
const h = Math.floor(s / 3600); const m = Math.floor((s % 3600) / 60)
return `${h}h ${m}m`
}
function formatBytes(b) {
if (b == null) return '—'
const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
let i = 0; let v = b
while (v >= 1024 && i < units.length - 1) { v /= 1024; i++ }
return `${v.toFixed(i === 0 ? 0 : 1)} ${units[i]}`
}
function formatRelative(iso) {
if (!iso) return '—'
const then = new Date(iso).getTime()
const diff = Math.max(0, (Date.now() - then) / 1000)
if (diff < 60) return `${Math.floor(diff)}s ago`
if (diff < 3600) return `${Math.floor(diff / 60)}m ago`
if (diff < 86400) return `${Math.floor(diff / 3600)}h ago`
return `${Math.floor(diff / 86400)}d ago`
}
</script>
<style scoped>
.fc-backup-table { background: transparent; }
.fc-tabular { font-variant-numeric: tabular-nums; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -17,6 +17,12 @@
>
Retry failed
</v-btn>
<v-btn
variant="text" rounded="pill" size="small" color="warning"
:disabled="!hasStuck" @click="onClearStuckOpen"
>
Clear stuck
</v-btn>
<v-btn
variant="text" rounded="pill" size="small" color="error"
@click="onClearOpen"
@@ -69,6 +75,31 @@
</v-card-actions>
</v-card>
</v-dialog>
<v-dialog v-model="clearStuckDialog" max-width="480">
<v-card>
<v-card-title>Clear stuck tasks</v-card-title>
<v-card-text>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
Force every <strong>pending / queued / processing</strong> task to
<strong>failed</strong> and finalize any active batch that
has no remaining work. Use this when the automatic recovery
sweep keeps re-queueing the same row (e.g., corrupt file in
an autoretry loop, or worker model missing).
</v-alert>
<p class="text-body-2">
Tasks remain in the database with status=<code>failed</code>;
click <em>Retry failed</em> once the underlying cause is
resolved to re-queue them.
</p>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn @click="clearStuckDialog = false">Cancel</v-btn>
<v-btn color="warning" rounded="pill" @click="onClearStuckConfirm">Clear stuck</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</v-card>
</template>
@@ -80,6 +111,7 @@ const store = useImportStore()
const statusFilter = ref(null)
const clearDialog = ref(false)
const clearAgeDays = ref(7)
const clearStuckDialog = ref(false)
const statusOptions = [
{ title: 'All', value: null },
@@ -100,6 +132,9 @@ const headers = [
]
const hasFailed = computed(() => store.tasks.some(t => t.status === 'failed'))
const hasStuck = computed(() => store.tasks.some(
t => t.status === 'pending' || t.status === 'queued' || t.status === 'processing'
))
function statusColor(s) {
return {
@@ -138,4 +173,9 @@ async function onClearConfirm() {
await store.clearCompleted(clearAgeDays.value)
clearDialog.value = false
}
function onClearStuckOpen() { clearStuckDialog.value = true }
async function onClearStuckConfirm() {
await store.clearStuck()
clearStuckDialog.value = false
}
</script>
@@ -2,7 +2,7 @@
<v-card>
<v-card-title>Trigger scan</v-card-title>
<v-card-text>
<div v-if="store.activeBatch" class="d-flex align-center" style="gap: 12px;">
<div v-if="store.activeBatch" class="d-flex align-center mb-3" style="gap: 12px;">
<v-progress-circular
indeterminate color="accent" size="20"
/>
@@ -13,20 +13,40 @@
failed {{ store.activeBatch.failed }} /
{{ store.activeBatch.total_files }} files
</span>
<v-spacer />
<v-btn
variant="text" rounded="pill" size="small" color="warning"
:loading="clearing" @click="onClearStuck"
>
Clear stuck
</v-btn>
</div>
<div v-else>
<p class="text-body-2 mb-3">
<p class="text-body-2 mb-3">
<span v-if="!store.activeBatch">
Run a quick scan of the import directory. Deep scan (pHash dedup,
archives) lands in FC-2d.
</p>
<v-btn color="primary" rounded="pill" @click="trigger" :loading="busy">
<v-icon start>mdi-magnify-scan</v-icon>
Quick scan
</v-btn>
<v-alert v-if="store.triggerError" type="error" variant="tonal" class="mt-3" closable>
{{ store.triggerError }}
</v-alert>
</div>
</span>
<span v-else>
An active batch is in progress. Wait for it to finish, or click
<em>Clear stuck</em> above if it has been wedged with no
measurable progress.
</span>
</p>
<v-btn
color="primary" rounded="pill"
:disabled="!!store.activeBatch"
:loading="busy"
@click="trigger"
>
<v-icon start>mdi-magnify-scan</v-icon>
Quick scan
</v-btn>
<v-alert v-if="store.triggerError" type="error" variant="tonal" class="mt-3" closable>
{{ store.triggerError }}
</v-alert>
</v-card-text>
</v-card>
</template>
@@ -37,9 +57,21 @@ import { useImportStore } from '../../stores/import.js'
const store = useImportStore()
const busy = ref(false)
const clearing = ref(false)
async function trigger() {
busy.value = true
try { await store.triggerScan() } catch {} finally { busy.value = false }
}
async function onClearStuck() {
clearing.value = true
try {
await store.clearStuck()
} catch {
// store surfaces error via triggerError if needed
} finally {
clearing.value = false
}
}
</script>
@@ -12,6 +12,8 @@
<MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" />
<AliasTable class="mt-4" />
<BackupCard class="mt-6" />
<TagMaintenanceCard class="mt-6" />
<BrowserExtensionCard class="mt-6" />
<LegacyMigrationCard class="mt-6" />
</div>
@@ -23,6 +25,8 @@ import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue'
import BackupCard from './BackupCard.vue'
import TagMaintenanceCard from './TagMaintenanceCard.vue'
import BrowserExtensionCard from './BrowserExtensionCard.vue'
import LegacyMigrationCard from './LegacyMigrationCard.vue'
</script>
@@ -0,0 +1,95 @@
<template>
<v-card class="fc-tag-maint">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-tag-remove" size="small" />
<span>Tag maintenance</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-4">
Remove tags with zero image associations and zero series-page
references. Auto-created tag rows that never got applied get
swept here.
</p>
<v-alert
v-if="store.lastError"
type="warning" variant="tonal" density="compact" class="mb-3"
>{{ store.lastError }}</v-alert>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="loadingPreview"
class="mb-3"
@click="onPreview"
>Preview unused tags</v-btn>
<div v-if="preview">
<p class="text-body-2 mb-2">
<strong>{{ preview.count }}</strong> unused tag(s).
<span v-if="preview.count > 50" class="fc-muted">
Showing first 50 names.
</span>
</p>
<div v-if="preview.sample_names?.length" class="fc-name-grid mb-3">
<span v-for="n in preview.sample_names" :key="n" class="fc-name">
{{ n }}
</span>
</div>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-sweep"
:disabled="!preview.count"
:loading="committing"
@click="onCommit"
>Delete {{ preview.count }} unused tag(s)</v-btn>
</div>
</v-card-text>
</v-card>
</template>
<script setup>
import { ref } from 'vue'
import { useAdminStore } from '../../stores/admin.js'
const store = useAdminStore()
const preview = ref(null)
const loadingPreview = ref(false)
const committing = ref(false)
async function onPreview() {
loadingPreview.value = true
try {
preview.value = await store.pruneUnusedTags({ dryRun: true })
} finally {
loadingPreview.value = false
}
}
async function onCommit() {
committing.value = true
try {
const result = await store.pruneUnusedTags({ dryRun: false })
preview.value = { count: 0, sample_names: result.sample_names || [] }
} finally {
committing.value = false
}
}
</script>
<style scoped>
.fc-tag-maint { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-name-grid {
display: flex; flex-wrap: wrap; gap: 4px 8px;
max-height: 200px; overflow-y: auto;
padding: 8px; border-radius: 4px;
background: rgb(var(--v-theme-surface-light));
}
.fc-name {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: rgb(var(--v-theme-on-surface-variant));
}
</style>
+153
View File
@@ -0,0 +1,153 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useAdminStore = defineStore('admin', () => {
const api = useApi()
const lastError = ref(null)
// --- Tier-C: artist cascade ---------------------------------------
async function projectArtistCascade(slug) {
lastError.value = null
try {
return await api.post(
`/api/admin/artists/${encodeURIComponent(slug)}/cascade-delete`,
{ body: { dry_run: true } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function dispatchArtistCascade(slug, confirm) {
lastError.value = null
try {
return await api.post(
`/api/admin/artists/${encodeURIComponent(slug)}/cascade-delete`,
{ body: { dry_run: false, confirm } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-C: bulk image delete ------------------------------------
async function projectBulkImageDelete(imageIds) {
lastError.value = null
try {
return await api.post(
'/api/admin/images/bulk-delete',
{ body: { image_ids: imageIds, dry_run: true } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function dispatchBulkImageDelete(imageIds, confirm) {
lastError.value = null
try {
return await api.post(
'/api/admin/images/bulk-delete',
{ body: { image_ids: imageIds, dry_run: false, confirm } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-B: tag delete + merge -----------------------------------
async function deleteTag(tagId) {
lastError.value = null
try {
return await api.delete(`/api/admin/tags/${tagId}`)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function mergeTags(destId, sourceId) {
lastError.value = null
try {
return await api.post(
`/api/admin/tags/${destId}/merge`,
{ body: { source_id: sourceId } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function tagUsageCount(tagId) {
lastError.value = null
try {
const body = await api.get(`/api/admin/tags/${tagId}/usage-count`)
return body.count
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-A: prune unused -----------------------------------------
async function pruneUnusedTags({ dryRun = true } = {}) {
lastError.value = null
try {
return await api.post(
'/api/admin/tags/prune-unused',
{ body: { dry_run: dryRun } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Task progress polling (taps FC-3i activity dashboard) --------
/**
* Polls /api/system/activity/runs?queue=maintenance every 3s,
* resolves when a task_run row with the given celery task_id
* reaches a terminal status (ok / error / timeout). Returns the
* row. Times out after 30 min by default.
*/
async function pollTaskUntilDone(taskId, { timeoutMs = 1_800_000 } = {}) {
const deadline = Date.now() + timeoutMs
while (Date.now() < deadline) {
const body = await api.get(
'/api/system/activity/runs',
{ params: { queue: 'maintenance', limit: 20 } },
)
const row = (body.runs || []).find(r => r.celery_task_id === taskId)
if (row && ['ok', 'error', 'timeout'].includes(row.status)) {
return row
}
await new Promise((r) => setTimeout(r, 3000))
}
throw new Error(`task ${taskId} did not finish within ${timeoutMs}ms`)
}
return {
lastError,
projectArtistCascade,
dispatchArtistCascade,
projectBulkImageDelete,
dispatchBulkImageDelete,
deleteTag,
mergeTags,
tagUsageCount,
pruneUnusedTags,
pollTaskUntilDone,
}
})
+109
View File
@@ -0,0 +1,109 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useBackupStore = defineStore('backup', () => {
const api = useApi()
const dbRuns = ref([])
const imagesRuns = ref([])
const settings = ref(null)
const loading = ref({ dbRuns: false, imagesRuns: false, settings: false })
const lastError = ref(null)
async function loadRuns(kind) {
const targetRef = kind === 'db' ? dbRuns : imagesRuns
const loadingKey = kind === 'db' ? 'dbRuns' : 'imagesRuns'
loading.value[loadingKey] = true
lastError.value = null
try {
const body = await api.get('/api/system/backup/runs', {
params: { kind, limit: 50 },
})
targetRef.value = body.runs || []
} catch (e) {
lastError.value = e.message
} finally {
loading.value[loadingKey] = false
}
}
async function triggerBackup(kind, tag = null) {
lastError.value = null
try {
await api.post(`/api/system/backup/${kind}`, {
body: tag ? { tag } : {},
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function restore(runId, confirm) {
lastError.value = null
try {
await api.post(`/api/system/backup/runs/${runId}/restore`, {
body: { confirm },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function deleteRun(runId, confirm) {
lastError.value = null
try {
await api.delete(`/api/system/backup/runs/${runId}`, {
body: { confirm },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function setTag(runId, tag) {
lastError.value = null
try {
await api.patch(`/api/system/backup/runs/${runId}`, {
body: { tag },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function loadSettings() {
loading.value.settings = true
lastError.value = null
try {
settings.value = await api.get('/api/system/backup/settings')
} catch (e) {
lastError.value = e.message
} finally {
loading.value.settings = false
}
}
async function patchSettings(patch) {
lastError.value = null
try {
settings.value = await api.patch('/api/system/backup/settings', {
body: patch,
})
} catch (e) {
lastError.value = e.message
throw e
}
}
return {
dbRuns, imagesRuns, settings, loading, lastError,
loadRuns, triggerBackup, restore, deleteRun, setTag,
loadSettings, patchSettings,
}
})
+8 -1
View File
@@ -92,6 +92,13 @@ export const useImportStore = defineStore('import', () => {
await loadTasks(true)
}
async function clearStuck() {
const body = await api.post('/api/import/clear-stuck')
await loadTasks(true)
await refreshStatus()
return body
}
const hasMore = computed(() => tasksNextCursor.value !== null)
return {
@@ -101,6 +108,6 @@ export const useImportStore = defineStore('import', () => {
triggerError,
loadSettings, patchSettings,
refreshStatus, triggerScan,
loadTasks, setStatusFilter, retryFailed, clearCompleted
loadTasks, setStatusFilter, retryFailed, clearCompleted, clearStuck
}
})
+7
View File
@@ -96,6 +96,12 @@
@open="openImage"
/>
</section>
<ArtistDangerZone
:slug="slug"
:artist-id="store.overview.id"
:artist-name="store.overview.name"
/>
</template>
</v-container>
</template>
@@ -106,6 +112,7 @@ import { useRoute, useRouter, RouterLink } from 'vue-router'
import { useArtistStore } from '../stores/artist.js'
import { useModalStore } from '../stores/modal.js'
import MasonryGrid from '../components/discovery/MasonryGrid.vue'
import ArtistDangerZone from '../components/artist/ArtistDangerZone.vue'
const route = useRoute()
const router = useRouter()
+1 -1
View File
@@ -84,7 +84,7 @@ onUnmounted(() => observer && observer.disconnect())
}
.fc-artists__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
gap: 12px;
}
.fc-artists__sentinel {
+122 -1
View File
@@ -27,6 +27,7 @@
<TagCard
v-for="c in store.cards" :key="c.id" :card="c"
@open="openTag" @rename="onRename" @manage="onManage" @read="onRead"
@merge-with="onMergeWith" @delete="onDeleteTag"
/>
</div>
@@ -41,6 +42,49 @@
@confirm="confirmMerge"
@cancel="pendingMerge = null"
/>
<v-dialog v-model="mergePickerOpen" max-width="520">
<v-card>
<v-card-title>Merge {{ mergeSource?.name }} into</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3" style="opacity: 0.7;">
Pick the target tag. Source tag will be deleted; all its
image associations will move to the target. Must be same
kind ({{ mergeSource?.kind }}).
</p>
<v-autocomplete
v-model="mergeTargetId"
:items="mergeHits" item-title="name" item-value="id"
:loading="mergeLoading"
density="compact" variant="outlined" hide-details
placeholder="Search target tag…" no-filter
@update:search="onMergeSearch"
/>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="mergePickerOpen = false">Cancel</v-btn>
<v-btn
color="warning" variant="flat" rounded="pill"
:disabled="!mergeTargetId"
@click="onMergeConfirm"
>Merge</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<DestructiveConfirmModal
v-model="deleteTagModalOpen"
action="delete"
kind="tag"
:run-id="deleteTagTarget?.id || 0"
tier="B"
:projected-counts="{ associations: deleteTagUsage }"
:description="deleteTagTarget
? `Delete tag “${deleteTagTarget.name}” and remove all image associations.`
: ''"
@confirm="onDeleteTagConfirm"
/>
</v-container>
</template>
@@ -48,8 +92,11 @@
import { ref, watch, onMounted, onUnmounted } from 'vue'
import { useRouter } from 'vue-router'
import { useTagDirectoryStore } from '../stores/tagDirectory.js'
import { useAdminStore } from '../stores/admin.js'
import { useApi } from '../composables/useApi.js'
import TagCard from '../components/discovery/TagCard.vue'
import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue'
import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue'
// Must stay a subset of the backend TagKind enum (character, fandom,
// general, series, archive, post, meta, rating). 'fandom' is this
@@ -106,6 +153,80 @@ function onManage(id) {
function onRead(id) {
router.push({ name: 'series-read', params: { tagId: id } })
}
// --- FC-3k tag merge + delete ----------------------------------------
const adminStore = useAdminStore()
const api = useApi()
// Tag merge via dots-menu (separate from inline-rename collision flow above)
const mergePickerOpen = ref(false)
const mergeSource = ref(null)
const mergeTargetId = ref(null)
const mergeHits = ref([])
const mergeLoading = ref(false)
let mergeDebounce = null
function onMergeWith(card) {
mergeSource.value = card
mergeTargetId.value = null
mergeHits.value = []
mergePickerOpen.value = true
}
function onMergeSearch(q) {
if (mergeDebounce) clearTimeout(mergeDebounce)
if (!q) { mergeHits.value = []; return }
mergeDebounce = setTimeout(async () => {
mergeLoading.value = true
try {
const hits = await api.get('/api/tags/autocomplete', {
params: { q, kind: mergeSource.value?.kind, limit: 20 },
})
mergeHits.value = (hits || []).filter(
(h) => h.id !== mergeSource.value?.id,
)
} finally {
mergeLoading.value = false
}
}, 250)
}
async function onMergeConfirm() {
if (!mergeSource.value || !mergeTargetId.value) return
try {
await adminStore.mergeTags(mergeTargetId.value, mergeSource.value.id)
mergePickerOpen.value = false
store.reset()
} catch {
// adminStore.lastError already set; UI surfaces it elsewhere.
}
}
// Tag delete via dots-menu (Tier B)
const deleteTagModalOpen = ref(false)
const deleteTagTarget = ref(null)
const deleteTagUsage = ref(0)
async function onDeleteTag(card) {
deleteTagTarget.value = card
try {
deleteTagUsage.value = await adminStore.tagUsageCount(card.id)
} catch {
deleteTagUsage.value = 0
}
deleteTagModalOpen.value = true
}
async function onDeleteTagConfirm() {
if (!deleteTagTarget.value) return
try {
await adminStore.deleteTag(deleteTagTarget.value.id)
store.reset()
} catch {
// adminStore.lastError already set.
}
}
</script>
<style scoped>
@@ -115,7 +236,7 @@ function onRead(id) {
}
.fc-tags__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
gap: 12px;
}
.fc-tags__sentinel {
+361
View File
@@ -0,0 +1,361 @@
"""FC-3k: /api/admin/* endpoint integration tests.
Monkeypatch `.delay()` for Tier-C tasks to capture dispatch payloads
without actually queuing. Tier-B/A endpoints run synchronously
through the real service.
"""
import hashlib
import pytest
from backend.app import create_app
from backend.app.models import Artist, ImageRecord, Tag, TagKind
from backend.app.models.tag import image_tag
pytestmark = pytest.mark.integration
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
# --- Tier-C: POST /artists/<slug>/cascade-delete --------------------
@pytest.mark.asyncio
async def test_artist_cascade_dry_run_returns_projection(client, db):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={"dry_run": True},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["artist"]["slug"] == "aria"
assert "projected" in body
assert body["projected"]["images"] == 0
@pytest.mark.asyncio
async def test_artist_cascade_unknown_slug_404(client):
resp = await client.post(
"/api/admin/artists/no-such/cascade-delete",
json={"dry_run": True},
)
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_artist_cascade_wrong_confirm_400(client, db):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
artist_id = a.id
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={"dry_run": False, "confirm": "wrong"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"] == f"delete-artist-{artist_id}"
@pytest.mark.asyncio
async def test_artist_cascade_correct_confirm_dispatches(
client, db, monkeypatch,
):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
artist_id = a.id
dispatched = []
class _Result:
id = "fake-task-id"
def _delay(**kw):
dispatched.append(kw)
return _Result()
monkeypatch.setattr(
"backend.app.tasks.admin.delete_artist_cascade_task.delay", _delay,
)
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={
"dry_run": False,
"confirm": f"delete-artist-{artist_id}",
},
)
assert resp.status_code == 202
body = await resp.get_json()
assert body["task_id"] == "fake-task-id"
assert dispatched == [{"artist_id": artist_id}]
# --- Tier-C: POST /images/bulk-delete -------------------------------
@pytest.mark.asyncio
async def test_bulk_delete_dry_run_returns_counts(client, db, tmp_path):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
i1 = ImageRecord(
artist_id=a.id, path=str(tmp_path / "1.jpg"),
sha256="1" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
i2 = ImageRecord(
artist_id=a.id, path=str(tmp_path / "2.jpg"),
sha256="2" * 64, size_bytes=20, mime="image/jpeg",
origin="imported_filesystem",
)
db.add_all([i1, i2])
await db.commit()
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [i1.id, i2.id, 9_999_999],
"dry_run": True,
},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["images_found"] == 2
assert body["bytes_on_disk"] == 30
assert body["missing_ids"] == [9_999_999]
@pytest.mark.asyncio
async def test_bulk_delete_empty_list_400(client):
resp = await client.post(
"/api/admin/images/bulk-delete",
json={"image_ids": [], "dry_run": True},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_image_ids"
@pytest.mark.asyncio
async def test_bulk_delete_non_int_400(client):
resp = await client.post(
"/api/admin/images/bulk-delete",
json={"image_ids": ["foo", 2], "dry_run": True},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_bulk_delete_wrong_confirm_400_with_expected_token(
client, db, tmp_path,
):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "x.jpg"),
sha256="c" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
await db.commit()
img_id = img.id
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [img_id], "dry_run": False, "confirm": "nope",
},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"].startswith("delete-images-")
@pytest.mark.asyncio
async def test_bulk_delete_correct_confirm_dispatches(
client, db, tmp_path, monkeypatch,
):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "x.jpg"),
sha256="d" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
await db.commit()
img_id = img.id
sha8 = hashlib.sha256(str(img_id).encode("utf-8")).hexdigest()[:8]
dispatched = []
class _Result:
id = "fake-bulk-id"
def _delay(**kw):
dispatched.append(kw)
return _Result()
monkeypatch.setattr(
"backend.app.tasks.admin.bulk_delete_images_task.delay", _delay,
)
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [img_id],
"dry_run": False,
"confirm": f"delete-images-{sha8}",
},
)
assert resp.status_code == 202
body = await resp.get_json()
assert body["task_id"] == "fake-bulk-id"
assert dispatched == [{"image_ids": [img_id]}]
# --- Tier-B: DELETE /tags/<id> + POST /tags/<dest>/merge -----------
@pytest.mark.asyncio
async def test_tag_delete_200_after_cascade(client, db):
t = Tag(name="doomed-tag", kind=TagKind.general)
db.add(t)
await db.commit()
tag_id = t.id
resp = await client.delete(f"/api/admin/tags/{tag_id}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["deleted"]["id"] == tag_id
assert body["associations_removed"] == 0
@pytest.mark.asyncio
async def test_tag_delete_404_unknown(client):
resp = await client.delete("/api/admin/tags/9999999")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_tag_merge_succeeds_for_same_kind(client, db):
src = Tag(name="src-tag", kind=TagKind.general)
dest = Tag(name="dest-tag", kind=TagKind.general)
db.add_all([src, dest])
await db.commit()
src_id = src.id
dest_id = dest.id
resp = await client.post(
f"/api/admin/tags/{dest_id}/merge",
json={"source_id": src_id},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["result"]["target_id"] == dest_id
@pytest.mark.asyncio
async def test_tag_merge_rejects_self_merge(client, db):
t = Tag(name="x", kind=TagKind.general)
db.add(t)
await db.commit()
resp = await client.post(
f"/api/admin/tags/{t.id}/merge",
json={"source_id": t.id},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_tag_merge_returns_400_on_kind_mismatch(client, db):
src = Tag(name="src", kind=TagKind.general)
dest = Tag(name="dest", kind=TagKind.artist)
db.add_all([src, dest])
await db.commit()
resp = await client.post(
f"/api/admin/tags/{dest.id}/merge",
json={"source_id": src.id},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "tag_kind_mismatch"
# --- Tier-B helper: GET /tags/<id>/usage-count ----------------------
@pytest.mark.asyncio
async def test_tag_usage_count_returns_zero_for_unused(client, db):
t = Tag(name="lonely", kind=TagKind.general)
db.add(t)
await db.commit()
resp = await client.get(f"/api/admin/tags/{t.id}/usage-count")
assert resp.status_code == 200
body = await resp.get_json()
assert body["count"] == 0
# --- Tier-A: POST /tags/prune-unused --------------------------------
@pytest.mark.asyncio
async def test_prune_unused_dry_run_lists_unused(client, db, tmp_path):
a = Artist(name="P", slug="p")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "p.jpg"),
sha256="e" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
used = Tag(name="kept", kind=TagKind.general)
unused = Tag(name="prune", kind=TagKind.general)
db.add_all([used, unused])
await db.flush()
await db.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
await db.commit()
resp = await client.post(
"/api/admin/tags/prune-unused", json={"dry_run": True},
)
body = await resp.get_json()
assert body["count"] == 1
assert "prune" in body["sample_names"]
@pytest.mark.asyncio
async def test_prune_unused_commit_deletes_and_returns_count(client, db):
db.add(Tag(name="byebye", kind=TagKind.general))
await db.commit()
resp = await client.post(
"/api/admin/tags/prune-unused", json={"dry_run": False},
)
body = await resp.get_json()
assert body["deleted"] >= 1
assert "byebye" in body["sample_names"]
+55
View File
@@ -86,6 +86,61 @@ async def test_clear_completed(client, db):
assert body["deleted"] == 1
@pytest.mark.asyncio
async def test_clear_stuck_fails_non_terminal_and_finalizes_orphan_batch(client, db):
"""Operator-flagged 2026-05-25: 3 large PNGs got stuck in 'processing'
for 2 days, the active ImportBatch never finalized, and the UI's
'Scanning...' banner persisted with 0/0 files. /api/import/clear-stuck
is the escape hatch to break the autoretry loop manually."""
from sqlalchemy import select as _select
batch = ImportBatch(triggered_by="manual", source_path="/import", scan_mode="quick")
db.add(batch)
await db.flush()
# Three stuck rows in mixed non-terminal states.
db.add(ImportTask(
batch_id=batch.id, source_path="/p1", task_type="media", status="processing",
))
db.add(ImportTask(
batch_id=batch.id, source_path="/p2", task_type="media", status="queued",
))
db.add(ImportTask(
batch_id=batch.id, source_path="/p3", task_type="media", status="pending",
))
# One already-complete row should be untouched.
db.add(ImportTask(
batch_id=batch.id, source_path="/done", task_type="media",
status="complete", finished_at=datetime.now(UTC),
))
await db.commit()
resp = await client.post("/api/import/clear-stuck")
body = await resp.get_json()
assert resp.status_code == 200
assert body["tasks_failed"] == 3
assert body["batches_finalized"] == 1
statuses = {
row.status for row in
(await db.execute(_select(ImportTask).where(ImportTask.batch_id == batch.id)))
.scalars().all()
}
assert statuses == {"failed", "complete"}
batch_status = (await db.execute(
_select(ImportBatch.status).where(ImportBatch.id == batch.id)
)).scalar_one()
assert batch_status == "complete"
@pytest.mark.asyncio
async def test_clear_stuck_no_op_when_nothing_stuck(client, db):
resp = await client.post("/api/import/clear-stuck")
body = await resp.get_json()
assert resp.status_code == 200
assert body == {"tasks_failed": 0, "batches_finalized": 0}
@pytest.mark.asyncio
async def test_trigger_accepts_deep(client, monkeypatch):
# Stub the task dispatch — assert the API accepts 'deep' and forwards
+7 -19
View File
@@ -31,17 +31,10 @@ async def client(app):
yield c
@pytest.mark.asyncio
async def test_post_backup_returns_202_and_id(client, monkeypatch):
monkeypatch.setattr(
"backend.app.api.migrate.run_migration",
type("F", (), {"delay": lambda self, *a, **k: None})(),
)
resp = await client.post("/api/migrate/backup", json={})
assert resp.status_code == 202
body = await resp.get_json()
assert "run_id" in body
assert body["status"] == "pending"
# Retired 2026-05-24 (FC-3h): `test_post_backup_returns_202_and_id`
# asserted the /api/migrate/backup endpoint, which was retired in FC-3h.
# Backup is now a first-class feature at /api/system/backup/*;
# coverage lives in tests/test_api_system_backup.py.
@pytest.mark.asyncio
@@ -62,10 +55,6 @@ async def test_post_ingest_rejects_missing_file(client):
@pytest.mark.asyncio
async def test_post_ingest_accepts_multipart_file(client, monkeypatch):
monkeypatch.setattr(
"backend.app.api.migrate._has_recent_pre_migration_backup",
lambda: True, # pretend backup exists
)
monkeypatch.setattr(
"backend.app.api.migrate.run_migration",
type("F", (), {"delay": lambda self, *a, **k: None})(),
@@ -92,10 +81,9 @@ async def test_post_ingest_accepts_multipart_file(client, monkeypatch):
@pytest.mark.asyncio
async def test_post_dry_run_allowed_without_backup(client, monkeypatch):
monkeypatch.setattr(
"backend.app.api.migrate._has_recent_pre_migration_backup",
lambda: False,
)
# FC-3h: the _has_recent_pre_migration_backup gate was retired; dry-run
# ingests were always allowed and now non-dry-run ingests are too.
# This test stays as regression coverage that dry_run ingest still works.
monkeypatch.setattr(
"backend.app.api.migrate.run_migration",
type("F", (), {"delay": lambda self, *a, **k: None})(),
+336
View File
@@ -0,0 +1,336 @@
"""FC-3h: /api/system/backup/* endpoint integration tests."""
from datetime import UTC, datetime, timedelta
import pytest
import pytest_asyncio
from sqlalchemy import select
from backend.app import create_app
from backend.app.models import BackupRun
pytestmark = pytest.mark.integration
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
@pytest_asyncio.fixture
async def _seed_runs(db):
"""Insert 4 BackupRun rows for paging/filter tests."""
now = datetime.now(UTC)
for i in range(4):
db.add(BackupRun(
kind="db" if i % 2 == 0 else "images",
status="ok",
tag=None,
triggered_by="manual",
started_at=now - timedelta(seconds=10 - i),
finished_at=now - timedelta(seconds=9 - i),
sql_path="/tmp/fake.sql" if i % 2 == 0 else None,
tar_path=None if i % 2 == 0 else "/tmp/fake.tar.zst",
size_bytes=100,
manifest={},
))
await db.commit()
# --- POST /db, /images ----------------------------------------------
@pytest.mark.asyncio
async def test_post_db_dispatches_and_returns_202(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post("/api/system/backup/db", json={})
assert resp.status_code == 202
body = await resp.get_json()
assert body["status"] == "dispatched"
assert dispatched == [{"tag": None, "triggered_by": "manual"}]
@pytest.mark.asyncio
async def test_post_db_persists_tag(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post(
"/api/system/backup/db", json={"tag": "pre-cutover"},
)
assert resp.status_code == 202
assert dispatched[0]["tag"] == "pre-cutover"
@pytest.mark.asyncio
async def test_post_db_rejects_tag_too_long(client):
resp = await client.post(
"/api/system/backup/db", json={"tag": "x" * 65},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_tag"
@pytest.mark.asyncio
async def test_post_images_dispatches(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_images_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post("/api/system/backup/images", json={})
assert resp.status_code == 202
# --- GET /runs -------------------------------------------------------
@pytest.mark.asyncio
async def test_runs_paginated_descending(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?limit=2")
body = await resp.get_json()
assert len(body["runs"]) == 2
ids = [r["id"] for r in body["runs"]]
assert ids == sorted(ids, reverse=True)
assert body["next_cursor"] is not None
@pytest.mark.asyncio
async def test_runs_filter_by_kind_db(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?kind=db")
body = await resp.get_json()
assert all(r["kind"] == "db" for r in body["runs"])
@pytest.mark.asyncio
async def test_runs_filter_by_kind_images(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?kind=images")
body = await resp.get_json()
assert all(r["kind"] == "images" for r in body["runs"])
@pytest.mark.asyncio
async def test_runs_rejects_invalid_kind(client):
resp = await client.get("/api/system/backup/runs?kind=bogus")
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_runs_rejects_invalid_limit(client):
resp = await client.get("/api/system/backup/runs?limit=not-int")
assert resp.status_code == 400
# --- GET /runs/<id> --------------------------------------------------
@pytest.mark.asyncio
async def test_get_run_returns_row(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.get(f"/api/system/backup/runs/{rid}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["id"] == rid
@pytest.mark.asyncio
async def test_get_run_404(client):
resp = await client.get("/api/system/backup/runs/999999")
assert resp.status_code == 404
# --- PATCH /runs/<id> ------------------------------------------------
@pytest.mark.asyncio
async def test_patch_tag_sets_value(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.patch(
f"/api/system/backup/runs/{rid}",
json={"tag": "monthly"},
)
assert resp.status_code == 200
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == rid)
).scalar_one()
assert tag == "monthly"
@pytest.mark.asyncio
async def test_patch_tag_null_clears(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
await client.patch(f"/api/system/backup/runs/{rid}", json={"tag": "x"})
await client.patch(f"/api/system/backup/runs/{rid}", json={"tag": None})
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == rid)
).scalar_one()
assert tag is None
# --- POST /runs/<id>/restore ----------------------------------------
@pytest.mark.asyncio
async def test_restore_wrong_confirm_400(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": "wrong"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"] == f"restore-db-{rid}"
@pytest.mark.asyncio
async def test_restore_correct_confirm_dispatches(client, _seed_runs, monkeypatch):
list_resp = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.restore_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": f"restore-db-{rid}"},
)
assert resp.status_code == 202
assert dispatched == [{"source_backup_run_id": rid}]
@pytest.mark.asyncio
async def test_restore_rejects_non_ok_status(client, db):
# Seed an error-status row directly.
db.add(BackupRun(
kind="db", status="error", tag=None, triggered_by="manual",
started_at=datetime.now(UTC), finished_at=datetime.now(UTC),
sql_path="/tmp/fake.sql", manifest={},
))
await db.commit()
rid_q = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await rid_q.get_json())["runs"][0]["id"]
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": f"restore-db-{rid}"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "not_restorable"
# --- DELETE /runs/<id> ----------------------------------------------
@pytest.mark.asyncio
async def test_delete_wrong_confirm_400(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.delete(
f"/api/system/backup/runs/{rid}",
json={"confirm": "wrong"},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_delete_correct_confirm_204(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
row = (await list_resp.get_json())["runs"][0]
rid = row["id"]
kind = row["kind"]
resp = await client.delete(
f"/api/system/backup/runs/{rid}",
json={"confirm": f"delete-{kind}-{rid}"},
)
assert resp.status_code == 204
surviving = db_sync.execute(
select(BackupRun.id).where(BackupRun.id == rid)
).scalar_one_or_none()
assert surviving is None
# --- /settings ------------------------------------------------------
@pytest.mark.asyncio
async def test_get_settings_returns_defaults(client):
resp = await client.get("/api/system/backup/settings")
assert resp.status_code == 200
body = await resp.get_json()
assert body["backup_db_nightly_enabled"] is False
assert body["backup_db_nightly_hour_utc"] == 3
assert body["backup_db_keep_last_n"] == 14
assert body["backup_images_keep_last_n"] == 3
@pytest.mark.asyncio
async def test_patch_settings_updates_values(client):
resp = await client.patch(
"/api/system/backup/settings",
json={
"backup_db_nightly_enabled": True,
"backup_db_nightly_hour_utc": 5,
"backup_db_keep_last_n": 30,
},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["backup_db_nightly_enabled"] is True
assert body["backup_db_nightly_hour_utc"] == 5
assert body["backup_db_keep_last_n"] == 30
@pytest.mark.asyncio
async def test_patch_settings_rejects_invalid_bool(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_nightly_enabled": "yes"},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_patch_settings_rejects_hour_out_of_range(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_nightly_hour_utc": 24},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_patch_settings_rejects_keep_n_below_min(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_keep_last_n": 0},
)
assert resp.status_code == 400
+200
View File
@@ -0,0 +1,200 @@
"""FC-3h: backup_service unit tests.
Subprocess calls (pg_dump, tar, psql) are monkeypatched so tests run
without external binaries. The real subprocess behavior is exercised
implicitly via the Celery task tests in test_tasks_backup.py.
"""
import json
from pathlib import Path
import pytest
from backend.app.services import backup_service
pytestmark = pytest.mark.integration
@pytest.fixture
def fake_subprocess(monkeypatch):
"""Replace subprocess.run with a fake that writes a sentinel to
the target path (for pg_dump's -f, for tar's -cf). Captures all
calls in a list."""
calls = []
class _FakeProc:
returncode = 0
stdout = b""
stderr = b""
def _fake_run(cmd, **kwargs):
calls.append(list(cmd))
if cmd[0] == "pg_dump":
i = cmd.index("-f")
Path(cmd[i + 1]).write_bytes(b"-- fake pg_dump\n")
elif cmd[0] == "tar" and "-cf" in cmd:
i = cmd.index("-cf")
Path(cmd[i + 1]).write_bytes(b"fake tar payload")
return _FakeProc()
monkeypatch.setattr("subprocess.run", _fake_run)
return calls
# --- backup_db -------------------------------------------------------
def test_backup_db_writes_sql_and_manifest(tmp_path, fake_subprocess):
result = backup_service.backup_db(
db_url="postgresql://test@x/db",
images_root=tmp_path,
tag="pre-cutover",
triggered_by="manual",
)
sql = Path(result["sql_path"])
manifest = Path(result["manifest_path"])
assert sql.is_file() and sql.suffix == ".sql"
assert manifest.is_file() and manifest.suffix == ".json"
assert result["kind"] == "db"
assert result["tar_path"] is None
assert result["size_bytes"] == len(b"-- fake pg_dump\n")
parsed = json.loads(manifest.read_text())
assert parsed["kind"] == "db"
assert parsed["tag"] == "pre-cutover"
assert parsed["triggered_by"] == "manual"
assert parsed["artifact_path"] == str(sql)
def test_backup_db_strips_sqlalchemy_psycopg_driver(tmp_path, fake_subprocess):
backup_service.backup_db(
db_url="postgresql+psycopg://u:p@h/d", images_root=tmp_path,
)
cmd = fake_subprocess[0]
assert cmd[0] == "pg_dump"
assert cmd[-1].startswith("postgresql://")
assert "+psycopg" not in cmd[-1]
def test_backup_db_strips_asyncpg_driver(tmp_path, fake_subprocess):
backup_service.backup_db(
db_url="postgresql+asyncpg://u:p@h/d", images_root=tmp_path,
)
assert "+asyncpg" not in fake_subprocess[0][-1]
def test_backup_db_default_tag_is_none(tmp_path, fake_subprocess):
result = backup_service.backup_db(
db_url="postgresql://u@h/d", images_root=tmp_path,
)
parsed = json.loads(Path(result["manifest_path"]).read_text())
assert parsed["tag"] is None
# --- backup_images ---------------------------------------------------
def test_backup_images_writes_tar_and_manifest(tmp_path, fake_subprocess):
result = backup_service.backup_images(
images_root=tmp_path, tag="monthly", triggered_by="manual",
)
tar = Path(result["tar_path"])
assert tar.is_file() and tar.name.endswith(".tar.zst")
assert result["sql_path"] is None
assert result["size_bytes"] == len(b"fake tar payload")
def test_backup_images_excludes_backups_and_quarantine(tmp_path, fake_subprocess):
backup_service.backup_images(images_root=tmp_path)
cmd = fake_subprocess[0]
excludes = [arg for arg in cmd if arg.startswith("--exclude=")]
assert any("_backups" in e for e in excludes)
assert any("_quarantine" in e for e in excludes)
# --- restore_db ------------------------------------------------------
def test_restore_db_drops_schema_then_loads(tmp_path, fake_subprocess):
sql_path = tmp_path / "fake.sql"
sql_path.write_text("SELECT 1;")
backup_service.restore_db(
db_url="postgresql://u@h/d", sql_path=sql_path,
)
# Two psql calls: one with -c (DROP SCHEMA), one with -f (load).
assert len(fake_subprocess) == 2
assert "-c" in fake_subprocess[0]
assert "DROP SCHEMA IF EXISTS public CASCADE" in fake_subprocess[0][-1]
assert "-f" in fake_subprocess[1]
assert str(sql_path) in fake_subprocess[1]
# --- restore_images --------------------------------------------------
def test_restore_images_untar_to_parent(tmp_path, fake_subprocess):
tar_path = tmp_path / "fake.tar.zst"
tar_path.write_bytes(b"fake")
backup_service.restore_images(images_root=tmp_path, tar_path=tar_path)
cmd = fake_subprocess[0]
assert cmd[:3] == ["tar", "--zstd", "-xf"]
assert str(tar_path) in cmd
assert "-C" in cmd
# --- unlink ----------------------------------------------------------
def test_unlink_removes_present_files_and_reports(tmp_path):
sql = tmp_path / "x.sql"
sql.write_bytes(b"x")
tar = tmp_path / "x.tar.zst"
tar.write_bytes(b"x")
manifest = tmp_path / "x.json"
manifest.write_text("{}")
result = backup_service.unlink_artifact_files(
sql_path=str(sql), tar_path=str(tar), manifest_path=str(manifest),
)
assert result == {"sql": True, "tar": True, "manifest": True}
assert not sql.exists() and not tar.exists() and not manifest.exists()
def test_unlink_missing_files_returns_true(tmp_path):
"""missing_ok semantics: a non-existent file isn't an error."""
result = backup_service.unlink_artifact_files(
sql_path=str(tmp_path / "nope.sql"),
tar_path=None, manifest_path=None,
)
assert result == {"sql": True}
def test_unlink_skips_none_paths():
"""A None path is skipped — not added to the result dict."""
result = backup_service.unlink_artifact_files(
sql_path=None, tar_path=None, manifest_path=None,
)
assert result == {}
# --- helpers ---------------------------------------------------------
def test_libpq_url_strips_each_known_driver():
assert backup_service._libpq_url(
"postgresql+psycopg://u@h/d"
) == "postgresql://u@h/d"
assert backup_service._libpq_url(
"postgresql+asyncpg://u@h/d"
) == "postgresql://u@h/d"
assert backup_service._libpq_url(
"postgresql+psycopg2://u@h/d"
) == "postgresql://u@h/d"
# Plain URL passes through unchanged.
assert backup_service._libpq_url(
"postgresql://u@h/d"
) == "postgresql://u@h/d"
def test_backups_dir_created_on_first_use(tmp_path):
d = backup_service._backups_dir(tmp_path)
assert d.is_dir()
assert d.name == "_backups"
+335
View File
@@ -0,0 +1,335 @@
"""FC-3k: cleanup_service unit tests.
Mutations go against real Postgres (db_sync fixture). File-system
side effects use tmp_path. Assertions on mutated rows use COLUMN
SELECTS per reference_async_coredml_test_assertions — never
re-read ORM attributes after a service mutates and re-fetches.
"""
import pytest
from sqlalchemy import func, select
from backend.app.models import Artist, ImageRecord, Tag, TagKind
from backend.app.models.tag import image_tag
from backend.app.services import cleanup_service
pytestmark = pytest.mark.integration
def _make_image(db_sync, *, artist, path, sha256, size=1000, thumb=None):
img = ImageRecord(
artist_id=artist.id,
path=path,
sha256=sha256,
size_bytes=size,
mime="image/jpeg",
origin="imported_filesystem",
thumbnail_path=thumb,
)
db_sync.add(img)
db_sync.flush()
return img
def _make_artist(db_sync, *, slug="aria", name="Aria"):
a = Artist(name=name, slug=slug)
db_sync.add(a)
db_sync.flush()
return a
def _make_tag(db_sync, *, name, kind=TagKind.general):
t = Tag(name=name, kind=kind)
db_sync.add(t)
db_sync.flush()
return t
# --- project_artist_cascade -----------------------------------------
def test_project_artist_cascade_returns_zeroes_for_empty_artist(db_sync):
_make_artist(db_sync, slug="empty")
db_sync.commit()
result = cleanup_service.project_artist_cascade(db_sync, slug="empty")
assert result["artist"]["slug"] == "empty"
assert result["projected"] == {
"images": 0,
"sources": 0,
"thumbs": 0,
"import_tasks": 0,
"bytes_on_disk": 0,
}
def test_project_artist_cascade_counts_images_and_thumbs_and_bytes(db_sync, tmp_path):
a = _make_artist(db_sync, slug="counted")
_make_image(
db_sync, artist=a, path=str(tmp_path / "a.jpg"),
sha256="a" * 64, size=1000, thumb=str(tmp_path / "a.thumb"),
)
_make_image(
db_sync, artist=a, path=str(tmp_path / "b.jpg"),
sha256="b" * 64, size=2500, thumb=None,
)
db_sync.commit()
result = cleanup_service.project_artist_cascade(db_sync, slug="counted")
assert result["projected"]["images"] == 2
assert result["projected"]["thumbs"] == 1
assert result["projected"]["bytes_on_disk"] == 3500
def test_project_artist_cascade_raises_on_unknown_slug(db_sync):
with pytest.raises(LookupError):
cleanup_service.project_artist_cascade(db_sync, slug="nope")
# --- project_bulk_image_delete --------------------------------------
def test_project_bulk_image_delete_separates_found_from_missing(db_sync, tmp_path):
a = _make_artist(db_sync, slug="bd")
i1 = _make_image(
db_sync, artist=a, path=str(tmp_path / "1.jpg"),
sha256="1" * 64, size=10, thumb="t1",
)
i2 = _make_image(
db_sync, artist=a, path=str(tmp_path / "2.jpg"),
sha256="2" * 64, size=20, thumb=None,
)
db_sync.commit()
result = cleanup_service.project_bulk_image_delete(
db_sync, image_ids=[i1.id, i2.id, 9_999_999],
)
assert result["images_found"] == 2
assert result["thumbs_to_unlink"] == 1
assert result["bytes_on_disk"] == 30
assert result["missing_ids"] == [9_999_999]
def test_project_bulk_image_delete_empty_input(db_sync):
result = cleanup_service.project_bulk_image_delete(db_sync, image_ids=[])
assert result == {
"images_found": 0,
"thumbs_to_unlink": 0,
"bytes_on_disk": 0,
"missing_ids": [],
}
# --- count_tag_associations / find_unused_tags ----------------------
def test_count_tag_associations_counts_image_tag_rows(db_sync, tmp_path):
a = _make_artist(db_sync, slug="ta")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "x.jpg"), sha256="c" * 64,
)
tag = _make_tag(db_sync, name="cyberpunk")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=tag.id,
))
db_sync.commit()
assert cleanup_service.count_tag_associations(db_sync, tag_id=tag.id) == 1
def test_find_unused_tags_returns_only_unreferenced(db_sync, tmp_path):
a = _make_artist(db_sync, slug="fu")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "y.jpg"), sha256="d" * 64,
)
used = _make_tag(db_sync, name="used")
_make_tag(db_sync, name="aaa-unused")
_make_tag(db_sync, name="zzz-unused")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.find_unused_tags(db_sync)
names = [t.name for t in result]
assert "used" not in names
assert "aaa-unused" in names
assert "zzz-unused" in names
# Sorted by name ascending.
assert names.index("aaa-unused") < names.index("zzz-unused")
# --- unlink_image_files ---------------------------------------------
def test_unlink_image_files_removes_original_and_thumbnail(db_sync, tmp_path):
original = tmp_path / "orig.jpg"
original.write_bytes(b"x")
custom_thumb = tmp_path / "custom.thumb"
custom_thumb.write_bytes(b"x")
conv_thumb = tmp_path / "thumbs" / "aaa" / ("a" * 64 + ".jpg")
conv_thumb.parent.mkdir(parents=True)
conv_thumb.write_bytes(b"x")
img = ImageRecord(
artist_id=None, path=str(original), sha256="a" * 64,
size_bytes=1, thumbnail_path=str(custom_thumb),
)
result = cleanup_service.unlink_image_files(img, tmp_path)
assert result == {"original": True, "thumbnail": True}
assert not original.exists()
assert not custom_thumb.exists()
assert not conv_thumb.exists()
def test_unlink_image_files_missing_files_count_as_success(tmp_path):
img = ImageRecord(
artist_id=None, path=str(tmp_path / "nope.jpg"),
sha256="b" * 64, size_bytes=1, thumbnail_path=None,
)
result = cleanup_service.unlink_image_files(img, tmp_path)
assert result == {"original": True, "thumbnail": False}
# --- delete_artist_cascade ------------------------------------------
def test_delete_artist_cascade_removes_images_and_artist_row(db_sync, tmp_path):
a = _make_artist(db_sync, slug="cas")
for i in range(3):
f = tmp_path / f"img{i}.jpg"
f.write_bytes(b"x")
_make_image(
db_sync, artist=a, path=str(f),
sha256=f"{i:064x}", size=10,
)
db_sync.commit()
artist_id = a.id
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=artist_id, images_root=tmp_path,
)
assert result["artist"]["slug"] == "cas"
assert result["summary"]["images_deleted"] == 3
assert result["summary"]["files_deleted"] == 3
# Column-select assertions per reference_async_coredml_test_assertions.
surviving_artist = db_sync.execute(
select(func.count(Artist.id)).where(Artist.id == artist_id)
).scalar_one()
surviving_images = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist_id)
).scalar_one()
assert surviving_artist == 0
assert surviving_images == 0
def test_delete_artist_cascade_idempotent_on_missing(db_sync, tmp_path):
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=9_999_999, images_root=tmp_path,
)
assert result["artist"] is None
assert result["summary"]["images_deleted"] == 0
# --- delete_images --------------------------------------------------
def test_delete_images_removes_rows_and_files(db_sync, tmp_path):
a = _make_artist(db_sync, slug="di")
f1 = tmp_path / "1.jpg"
f1.write_bytes(b"x")
f2 = tmp_path / "2.jpg"
f2.write_bytes(b"x")
i1 = _make_image(db_sync, artist=a, path=str(f1), sha256="1" * 64)
i2 = _make_image(db_sync, artist=a, path=str(f2), sha256="2" * 64)
db_sync.commit()
result = cleanup_service.delete_images(
db_sync, image_ids=[i1.id, i2.id, 9_999_999],
images_root=tmp_path,
)
assert result["images_deleted"] == 2
assert result["files_deleted"] == 2
assert result["missing_ids"] == [9_999_999]
surviving = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.id.in_([i1.id, i2.id]))
).scalar_one()
assert surviving == 0
# --- delete_tag -----------------------------------------------------
def test_delete_tag_cascades_associations(db_sync, tmp_path):
a = _make_artist(db_sync, slug="dt")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "x.jpg"), sha256="d" * 64,
)
tag = _make_tag(db_sync, name="doomed")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=tag.id,
))
db_sync.commit()
tag_id = tag.id
result = cleanup_service.delete_tag(db_sync, tag_id=tag_id)
assert result["deleted"]["id"] == tag_id
assert result["associations_removed"] == 1
surviving_tag = db_sync.execute(
select(func.count(Tag.id)).where(Tag.id == tag_id)
).scalar_one()
surviving_assoc = db_sync.execute(
select(func.count())
.select_from(image_tag).where(image_tag.c.tag_id == tag_id)
).scalar_one()
assert surviving_tag == 0
assert surviving_assoc == 0
def test_delete_tag_raises_on_unknown_id(db_sync):
with pytest.raises(LookupError):
cleanup_service.delete_tag(db_sync, tag_id=9_999_999)
# --- prune_unused_tags ----------------------------------------------
def test_prune_unused_tags_dry_run_returns_count_and_names(db_sync, tmp_path):
a = _make_artist(db_sync, slug="pu")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "z.jpg"), sha256="e" * 64,
)
used = _make_tag(db_sync, name="kept")
_make_tag(db_sync, name="prune-me-1")
_make_tag(db_sync, name="prune-me-2")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.prune_unused_tags(db_sync, dry_run=True)
assert result["count"] == 2
assert "prune-me-1" in result["sample_names"]
assert "prune-me-2" in result["sample_names"]
# Nothing actually deleted.
surviving = db_sync.execute(select(func.count(Tag.id))).scalar_one()
assert surviving == 3
def test_prune_unused_tags_commit_deletes_them(db_sync, tmp_path):
a = _make_artist(db_sync, slug="pu2")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "k.jpg"), sha256="f" * 64,
)
used = _make_tag(db_sync, name="kept")
_make_tag(db_sync, name="bye")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.prune_unused_tags(db_sync, dry_run=False)
assert result["deleted"] == 1
surviving_names = db_sync.execute(select(Tag.name)).scalars().all()
assert "kept" in surviving_names
assert "bye" not in surviving_names
+7 -2
View File
@@ -9,11 +9,16 @@ from backend.app.scripts import download_models as dm
def test_ensure_camie_skips_when_present(tmp_path, monkeypatch):
"""v2 layout (HF Camais03/camie-tagger-v2): the ONNX file is named
camie-tagger-v2.onnx (not model.onnx) and tags ship inside
camie-tagger-v2-metadata.json (not selected_tags.csv). Both at root.
Updated 2026-05-25 after the actual repo layout was confirmed via
WebFetch — the old assertion pinned the v1 filenames."""
monkeypatch.setattr(dm, "MODEL_ROOT", tmp_path)
camie = tmp_path / "camie"
camie.mkdir(parents=True)
(camie / "model.onnx").write_bytes(b"x")
(camie / "selected_tags.csv").write_text("tag_id,name,category,count\n")
(camie / "camie-tagger-v2.onnx").write_bytes(b"x")
(camie / "camie-tagger-v2-metadata.json").write_text("{}")
with patch.object(dm, "_snapshot") as snap:
dm.ensure_camie()
snap.assert_not_called()
+132
View File
@@ -133,3 +133,135 @@ async def test_get_image_with_tags_includes_integrity_status(db):
svc = GalleryService(db)
payload = await svc.get_image_with_tags(img.id)
assert payload["integrity_status"] == "ok"
async def _seed_image_with_post(
db, *, sha: str, image_created_at, post_date, artist_name="test-artist",
platform="patreon", external_post_id="42",
):
"""Helper: seed an Artist + Source + Post and one ImageRecord whose
primary_post_id points at that Post. Used for date-coalesce tests."""
from backend.app.models import Artist, Post, Source
artist = Artist(name=artist_name, slug=artist_name.lower().replace(" ", "-"))
db.add(artist)
await db.flush()
source = Source(
artist_id=artist.id, platform=platform,
url=f"https://www.{platform}.com/{artist.slug}",
)
db.add(source)
await db.flush()
post = Post(
source_id=source.id, external_post_id=external_post_id,
post_title="A Post", post_date=post_date,
)
db.add(post)
await db.flush()
img = ImageRecord(
path=f"/images/test/{sha[:8]}.jpg",
sha256=sha, size_bytes=1000, mime="image/jpeg",
width=100, height=100,
origin="imported_filesystem", integrity_status="unknown",
primary_post_id=post.id,
)
img.created_at = image_created_at
db.add(img)
await db.flush()
return img, post
@pytest.mark.asyncio
async def test_scroll_sorts_by_post_date_when_available(db):
"""Operator-flagged 2026-05-25: ~57k IR images all imported in the
same week sort by image.created_at and pile up in one month bucket.
Once primary_post_id is wired (via tag_apply phase 4), the gallery
should sort by Post.post_date instead, spreading them across the
actual publish years."""
base_import = _now()
# Image A: imported NOW, but post was made 2 years ago.
img_a, _ = await _seed_image_with_post(
db, sha="a" * 64,
image_created_at=base_import,
post_date=base_import - timedelta(days=730),
artist_name="Aria", external_post_id="A-1",
)
# Image B: imported NOW (1 min later), post made YESTERDAY.
img_b, _ = await _seed_image_with_post(
db, sha="b" * 64,
image_created_at=base_import - timedelta(minutes=1),
post_date=base_import - timedelta(days=1),
artist_name="Bea", external_post_id="B-1",
)
# Image C: filesystem-imported, no primary_post_id, created 5 days ago.
img_c = ImageRecord(
path="/images/test/c.jpg", sha256="c" * 64,
size_bytes=1000, mime="image/jpeg",
width=100, height=100,
origin="imported_filesystem", integrity_status="unknown",
)
img_c.created_at = base_import - timedelta(days=5)
db.add(img_c)
await db.flush()
svc = GalleryService(db)
page = await svc.scroll(cursor=None, limit=10)
# Effective-date order: B (yesterday) > C (5 days ago) > A (2 years ago)
assert [i.id for i in page.images] == [img_b.id, img_c.id, img_a.id]
# API exposes both fields explicitly so the UI can show "Posted X / Imported Y".
a_payload = next(i for i in page.images if i.id == img_a.id)
assert a_payload.posted_at is not None
assert a_payload.posted_at < a_payload.created_at
c_payload = next(i for i in page.images if i.id == img_c.id)
assert c_payload.posted_at is None
assert c_payload.effective_date == c_payload.created_at
@pytest.mark.asyncio
async def test_timeline_buckets_use_post_date_when_available(db):
"""Timeline group-by must follow the same effective_date rule so the
UI's year/month navigation surfaces publish-date buckets, not the
single FC-scan bucket all migrated images share."""
base = datetime(2026, 6, 15, 12, 0, tzinfo=UTC)
await _seed_image_with_post(
db, sha="1" * 64,
image_created_at=base,
post_date=datetime(2024, 3, 10, tzinfo=UTC),
artist_name="Carl", external_post_id="C-1",
)
await _seed_image_with_post(
db, sha="2" * 64,
image_created_at=base,
post_date=datetime(2024, 3, 11, tzinfo=UTC),
artist_name="Dee", external_post_id="D-1",
)
await _seed_image_with_post(
db, sha="3" * 64,
image_created_at=base,
post_date=datetime(2025, 9, 1, tzinfo=UTC),
artist_name="Eli", external_post_id="E-1",
)
svc = GalleryService(db)
buckets = await svc.timeline()
bucket_keys = {(b.year, b.month, b.count) for b in buckets}
# Two posts in 2024-03, one in 2025-09 — even though all imported in 2026-06.
assert (2024, 3, 2) in bucket_keys
assert (2025, 9, 1) in bucket_keys
# The FC-import bucket should NOT appear since all 3 images have post_date.
assert not any(b.year == 2026 and b.month == 6 for b in buckets)
@pytest.mark.asyncio
async def test_get_image_with_tags_includes_posted_at_when_present(db):
base = _now()
img, _ = await _seed_image_with_post(
db, sha="f" * 64,
image_created_at=base,
post_date=base - timedelta(days=365),
artist_name="Fred", external_post_id="F-1",
)
svc = GalleryService(db)
payload = await svc.get_image_with_tags(img.id)
assert payload["posted_at"] is not None
# Image's own created_at is still surfaced separately.
assert payload["created_at"] != payload["posted_at"]
+55
View File
@@ -134,3 +134,58 @@ def test_root_level_file_has_no_artist(importer, import_layout):
importer.import_one(src)
artists = importer.session.execute(select(Artist)).scalars().all()
assert artists == []
def test_pil_load_oserror_in_transparency_check_skips_not_raises(
importer, import_layout, monkeypatch,
):
"""PIL.verify() only validates header structure — broken pixel data
only surfaces when load() actually decodes. The importer must catch
the OSError and return a skipped: invalid_image result so the Celery
autoretry loop doesn't bounce the same broken file forever.
Operator hit this 2026-05-25 with a corrupt JPEG in the IR set."""
import_root, _ = import_layout
src = import_root / "Bob" / "corrupt.png"
# Make a real RGBA PNG so the has_alpha path engages.
_make_png_rgba(src, (100, 100), alpha=128)
importer.settings.skip_transparent = True
importer.settings.transparency_threshold = 0.5
# Force the next _transparency_pct call to raise as if PIL's load()
# blew up on truncated pixel data.
def _boom(_self, _src):
raise OSError("broken data stream when reading image file")
monkeypatch.setattr(
type(importer), "_transparency_pct", _boom,
)
result = importer.import_one(src)
assert result.status == "skipped"
assert result.skip_reason == SkipReason.invalid_image
assert "transparency check" in (result.error or "")
def test_pil_load_oserror_in_phash_compute_skips_not_raises(
importer, import_layout, monkeypatch,
):
"""Same shape as the transparency-check guard, but for the phash
compute block — the OTHER place PIL.load() runs implicitly during
the dedup pipeline."""
import_root, _ = import_layout
src = import_root / "Carol" / "corrupt.jpg"
_make_jpeg(src)
# Disable transparency check so we reach the phash compute block.
importer.settings.skip_transparent = False
from backend.app.services import importer as importer_module
def _boom(_im):
raise OSError("broken data stream when reading image file")
monkeypatch.setattr(importer_module, "compute_phash", _boom)
result = importer.import_one(src)
assert result.status == "skipped"
assert result.skip_reason == SkipReason.invalid_image
assert "phash compute" in (result.error or "")
+96
View File
@@ -68,6 +68,102 @@ def test_recover_interrupted_only_old(db_sync, monkeypatch):
assert dispatched == [stale.id]
def test_recover_interrupted_sweeps_pending_orphans_to_failed(db_sync, monkeypatch):
"""A scan that creates ImportTask rows but crashes before the second
pass (transition to 'queued' + .delay()) leaves rows orphaned at
status='pending'. The sweep flips them to 'failed' so the operator
can drain via /api/import/retry-failed without thundering-herding.
Banked 2026-05-25 after operator hit 5490 stuck pending rows.
"""
from backend.app.tasks import import_file
monkeypatch.setattr(import_file.import_media_file, "delay", lambda *_: None)
batch_id = _make_batch(db_sync)
now = datetime.now(UTC)
fresh_pending = ImportTask(
batch_id=batch_id, source_path="/import/fresh.jpg", task_type="media",
status="pending",
)
db_sync.add(fresh_pending)
db_sync.flush()
# created_at defaults to now() server-side; fresh row stays untouched.
# Two stale rows simulating the orphan pile: one 'pending', one
# 'queued' (scanner crashed AFTER transitioning some rows but
# before all). Both should sweep.
stale_pending = ImportTask(
batch_id=batch_id, source_path="/import/stale1.jpg", task_type="media",
status="pending",
)
stale_queued = ImportTask(
batch_id=batch_id, source_path="/import/stale2.jpg", task_type="media",
status="queued",
)
db_sync.add_all([stale_pending, stale_queued])
db_sync.flush()
# Backdate created_at past the orphan cutoff (30 min).
from sqlalchemy import update as _upd
db_sync.execute(
_upd(ImportTask)
.where(ImportTask.id.in_([stale_pending.id, stale_queued.id]))
.values(created_at=now - timedelta(hours=2))
)
db_sync.commit()
from backend.app.tasks.maintenance import recover_interrupted_tasks
touched = recover_interrupted_tasks.apply().get()
assert touched == 2
db_sync.refresh(fresh_pending)
db_sync.refresh(stale_pending)
db_sync.refresh(stale_queued)
assert fresh_pending.status == "pending" # fresh row untouched
assert stale_pending.status == "failed"
assert stale_queued.status == "failed"
assert "orphan" in (stale_pending.error or "")
def test_recover_interrupted_handles_both_stuck_and_orphans(db_sync, monkeypatch):
"""One sweep tick handles both 'processing' crashes AND
'pending'/'queued' orphans in a single pass."""
from backend.app.tasks import import_file
dispatched: list[int] = []
monkeypatch.setattr(
import_file.import_media_file, "delay", dispatched.append
)
batch_id = _make_batch(db_sync)
now = datetime.now(UTC)
stuck = ImportTask(
batch_id=batch_id, source_path="/import/stuck.jpg", task_type="media",
status="processing", started_at=now - timedelta(hours=2),
)
orphan = ImportTask(
batch_id=batch_id, source_path="/import/orphan.jpg", task_type="media",
status="pending",
)
db_sync.add_all([stuck, orphan])
db_sync.flush()
from sqlalchemy import update as _upd
db_sync.execute(
_upd(ImportTask).where(ImportTask.id == orphan.id)
.values(created_at=now - timedelta(hours=2))
)
db_sync.commit()
from backend.app.tasks.maintenance import recover_interrupted_tasks
touched = recover_interrupted_tasks.apply().get()
assert touched == 2
db_sync.refresh(stuck)
db_sync.refresh(orphan)
assert stuck.status == "queued"
assert orphan.status == "failed"
assert dispatched == [stuck.id] # stuck rows re-enqueue; orphans don't
def test_cleanup_old_deletes_finished_old(db_sync):
batch_id = _make_batch(db_sync)
now = datetime.now(UTC)
-64
View File
@@ -1,64 +0,0 @@
"""FC-5: backup + rollback service tests.
Uses tmp_path to avoid touching real /images/_backups/. Subprocess
calls (pg_dump, tar) are monkeypatched — the real shell-out is exercised
in operator-side smoke testing on the homelab, not in unit tests.
"""
import pytest
from backend.app.services.migrators import backup as backup_mod
pytestmark = pytest.mark.integration
@pytest.mark.asyncio
async def test_create_backup_writes_sql_and_tar(monkeypatch, tmp_path):
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
# Touch the expected output file so existence checks pass downstream.
if "pg_dump" in cmd[0]:
outpath = tmp_path / "_backups" / f"fc_{kwargs['_test_ts']}.sql"
outpath.write_text("-- fake pg_dump output")
else:
outpath = tmp_path / "_backups" / f"fc_{kwargs['_test_ts']}.tar.zst"
outpath.write_bytes(b"\x28\xb5\x2f\xfd") # zstd magic
from types import SimpleNamespace
return SimpleNamespace(returncode=0, stdout=b"", stderr=b"")
monkeypatch.setattr(backup_mod, "_run_subprocess", fake_run)
result = backup_mod.create_backup(
db_url="postgresql://x", images_root=tmp_path, tag="pre_migration",
)
assert "sql_path" in result
assert "tar_path" in result
assert result["tag"] == "pre_migration"
assert any("pg_dump" in c[0] for c in calls)
assert any(c[0] == "tar" for c in calls)
@pytest.mark.asyncio
async def test_find_latest_backup_by_tag(monkeypatch, tmp_path):
# Create two manifests with different tags.
backups_dir = tmp_path / "_backups"
backups_dir.mkdir()
import json
(backups_dir / "fc_20260101T000000Z.json").write_text(json.dumps({
"backup_id": "20260101T000000Z", "tag": "manual",
"created_at": "2026-01-01T00:00:00+00:00",
"sql_path": "/x/a.sql", "tar_path": "/x/a.tar.zst",
}))
(backups_dir / "fc_20260202T000000Z.json").write_text(json.dumps({
"backup_id": "20260202T000000Z", "tag": "pre_migration",
"created_at": "2026-02-02T00:00:00+00:00",
"sql_path": "/x/b.sql", "tar_path": "/x/b.tar.zst",
}))
found = backup_mod.find_latest_backup(tmp_path, tag="pre_migration")
assert found is not None
assert found["backup_id"] == "20260202T000000Z"
missing = backup_mod.find_latest_backup(tmp_path, tag="nonexistent")
assert missing is None
+4 -1
View File
@@ -40,5 +40,8 @@ def test_get_tagger_singleton():
def test_load_raises_when_model_missing(tmp_path):
t = Tagger(model_dir=tmp_path / "nonexistent")
with pytest.raises(RuntimeError, match="model.onnx missing"):
# Match the trailing "missing at <path>" rather than the specific
# filename, so a future model-version bump (camie-tagger-v3.onnx, etc.)
# doesn't bounce this test.
with pytest.raises(RuntimeError, match=r"\.onnx missing at "):
t.load()
+97
View File
@@ -227,6 +227,67 @@ async def test_image_posts_creates_source_post_provenance(db, tmp_path):
)).scalar_one()
assert prov_count == 1
# Phase 4 must also set ImageRecord.primary_post_id so the gallery's
# effective_date COALESCE can surface Post.post_date. Operator-flagged
# 2026-05-25: without this, IR-migrated images keep sorting by FC's
# scan date instead of the original publish date.
primary_post_id = (await db.execute(
select(ImageRecord.primary_post_id).where(ImageRecord.id == img_id)
)).scalar_one()
canonical_post_id = (await db.execute(
select(Post.id).where(Post.external_post_id == "10001")
)).scalar_one()
assert primary_post_id == canonical_post_id
@pytest.mark.asyncio
async def test_image_posts_primary_post_id_not_clobbered(db, tmp_path):
"""If the importer already set primary_post_id (e.g. a downloaded
image with a known provenance), phase 4 must NOT overwrite it when
re-running tag_apply against the IR migration. The existing
download-time linkage is the source of truth."""
sha = "9" * 64
await _seed_image(db, sha, suffix="9")
# Pre-set primary_post_id to a sentinel Post so we can detect a clobber.
img_id = (await db.execute(
select(ImageRecord.id).where(ImageRecord.sha256 == sha)
)).scalar_one()
# Build an existing Source + Post for the sentinel.
art = Artist(name="Pre-existing", slug="pre-existing")
db.add(art)
await db.flush()
src = Source(
artist_id=art.id, platform="patreon",
url="https://www.patreon.com/pre-existing",
)
db.add(src)
await db.flush()
sentinel_post = Post(
source_id=src.id, external_post_id="sentinel-99",
post_title="Pre-existing",
)
db.add(sentinel_post)
await db.flush()
await db.execute(
ImageRecord.__table__.update()
.where(ImageRecord.id == img_id)
.values(primary_post_id=sentinel_post.id)
)
await db.commit()
_write_manifest(tmp_path, image_posts=[
{**_POST_ENTRY, "image_sha256s": [sha]},
])
await tag_apply.apply_async(db, images_root=tmp_path, dry_run=False)
# The migration created a NEW Post (external_post_id="10001") and a
# new ImageProvenance, but primary_post_id must still point at the
# original sentinel.
primary_post_id = (await db.execute(
select(ImageRecord.primary_post_id).where(ImageRecord.id == img_id)
)).scalar_one()
assert primary_post_id == sentinel_post.id
@pytest.mark.asyncio
async def test_image_posts_idempotent_on_rerun(db, tmp_path):
@@ -281,6 +342,42 @@ async def test_image_posts_unknown_platform_skipped(db, tmp_path):
assert src_count == 0
@pytest.mark.parametrize("platform,expected_url", [
("deviantart", "https://www.deviantart.com/maewix"),
("pixiv", "https://www.pixiv.net/users/maewix"),
])
@pytest.mark.asyncio
async def test_image_posts_extended_platforms_create_source(
db, tmp_path, platform, expected_url,
):
"""Regression for 2026-05-25 operator-reported bug: phase 4's
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry,
silently dropping deviantart + pixiv PostMetadata from the IR migration."""
sha = f"{platform[0]}" * 64
await _seed_image(db, sha, suffix=f"_{platform}")
await db.commit()
_write_manifest(tmp_path, image_posts=[
{**_POST_ENTRY, "platform": platform, "image_sha256s": [sha]},
])
result = await tag_apply.apply_async(db, images_root=tmp_path, dry_run=False)
assert result["counts"]["rows_inserted"] >= 1
src_url = (await db.execute(
select(Source.url).where(Source.platform == platform)
)).scalar_one()
assert src_url == expected_url
# ImageProvenance row was created.
prov_count = (await db.execute(
select(func.count(ImageProvenance.id))
.join(Source, Source.id == ImageProvenance.source_id)
.where(Source.platform == platform)
)).scalar_one()
assert prov_count == 1
@pytest.mark.asyncio
async def test_image_posts_dry_run_makes_no_writes(db, tmp_path):
sha = "2" * 64
+158
View File
@@ -0,0 +1,158 @@
"""FC-3k: admin Celery task integration tests.
task_always_eager + signal handlers from FC-3i populate task_run.
We assert the wrapper passes args through correctly and that
task_run lifecycle status flips as expected.
"""
import pytest
from sqlalchemy import func, select
import backend.app.tasks.admin # noqa: F401 — register tasks
from backend.app.celery_app import celery
from backend.app.models import Artist, ImageRecord, TaskRun
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def _eager_celery(monkeypatch):
monkeypatch.setattr(celery.conf, "task_always_eager", True)
monkeypatch.setattr(celery.conf, "task_eager_propagates", False)
@pytest.fixture(autouse=True)
def fake_images_root(monkeypatch, tmp_path):
monkeypatch.setattr("backend.app.tasks.admin.IMAGES_ROOT", tmp_path)
# --- registration ----------------------------------------------------
def test_delete_artist_cascade_task_registered():
assert (
"backend.app.tasks.admin.delete_artist_cascade_task"
in celery.tasks
)
def test_bulk_delete_images_task_registered():
assert (
"backend.app.tasks.admin.bulk_delete_images_task"
in celery.tasks
)
# --- delete_artist_cascade_task -------------------------------------
@pytest.mark.asyncio
async def test_delete_artist_cascade_task_removes_artist_and_records_ok(
db_sync, tmp_path,
):
from backend.app.tasks.admin import delete_artist_cascade_task
a = Artist(name="Doomed", slug="doomed")
db_sync.add(a)
db_sync.flush()
for i in range(2):
f = tmp_path / f"d{i}.jpg"
f.write_bytes(b"x")
db_sync.add(ImageRecord(
artist_id=a.id, path=str(f),
sha256=f"{i:064x}", size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
))
db_sync.commit()
artist_id = a.id
result = delete_artist_cascade_task.delay(artist_id=artist_id).get()
assert result["summary"]["images_deleted"] == 2
surviving = db_sync.execute(
select(func.count(Artist.id)).where(Artist.id == artist_id)
).scalar_one()
assert surviving == 0
# FC-3i task_run lifecycle: task should have an 'ok' row.
status = db_sync.execute(
select(TaskRun.status)
.where(TaskRun.task_name.endswith(".delete_artist_cascade_task"))
.order_by(TaskRun.id.desc()).limit(1)
).scalar_one()
assert status == "ok"
@pytest.mark.asyncio
async def test_delete_artist_cascade_task_records_failure(
db_sync, monkeypatch,
):
from backend.app.tasks.admin import delete_artist_cascade_task
def _boom(*a, **kw):
raise RuntimeError("synthetic cascade fail")
monkeypatch.setattr(
"backend.app.services.cleanup_service.delete_artist_cascade", _boom,
)
with pytest.raises(RuntimeError):
delete_artist_cascade_task.delay(artist_id=1).get()
row = db_sync.execute(
select(TaskRun.status, TaskRun.error_message)
.where(TaskRun.task_name.endswith(".delete_artist_cascade_task"))
.order_by(TaskRun.id.desc()).limit(1)
).one()
assert row.status == "error"
assert "synthetic" in (row.error_message or "")
# --- bulk_delete_images_task ----------------------------------------
@pytest.mark.asyncio
async def test_bulk_delete_images_task_removes_listed_images(
db_sync, tmp_path,
):
from backend.app.tasks.admin import bulk_delete_images_task
a = Artist(name="B", slug="b")
db_sync.add(a)
db_sync.flush()
ids = []
for i in range(3):
f = tmp_path / f"b{i}.jpg"
f.write_bytes(b"x")
img = ImageRecord(
artist_id=a.id, path=str(f),
sha256=f"a{i:063x}", size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db_sync.add(img)
db_sync.flush()
ids.append(img.id)
db_sync.commit()
result = bulk_delete_images_task.delay(image_ids=ids).get()
assert result["images_deleted"] == 3
surviving = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.id.in_(ids))
).scalar_one()
assert surviving == 0
@pytest.mark.asyncio
async def test_bulk_delete_images_task_idempotent_on_empty_list(db_sync):
from backend.app.tasks.admin import bulk_delete_images_task
result = bulk_delete_images_task.delay(image_ids=[]).get()
assert result["images_deleted"] == 0
@pytest.mark.asyncio
async def test_bulk_delete_images_task_reports_missing_ids(db_sync):
from backend.app.tasks.admin import bulk_delete_images_task
result = bulk_delete_images_task.delay(
image_ids=[9_999_998, 9_999_999],
).get()
assert result["images_deleted"] == 0
assert sorted(result["missing_ids"]) == [9_999_998, 9_999_999]
+281
View File
@@ -0,0 +1,281 @@
"""FC-3h: backup/restore Celery task integration tests.
Uses task_always_eager for synchronous in-test execution.
Subprocess + IMAGES_ROOT are monkeypatched so tests don't touch
real /images or shell out to pg_dump.
"""
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from sqlalchemy import select
from backend.app.celery_app import celery
from backend.app.models import BackupRun, ImportSettings
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def _eager_celery(monkeypatch):
monkeypatch.setattr(celery.conf, "task_always_eager", True)
monkeypatch.setattr(celery.conf, "task_eager_propagates", False)
@pytest.fixture(autouse=True)
def fake_subprocess_and_images_root(monkeypatch, tmp_path):
monkeypatch.setattr("backend.app.tasks.backup.IMAGES_ROOT", tmp_path)
monkeypatch.setattr(
"backend.app.services.backup_service._DB_SUBPROCESS_TIMEOUT_S", 5,
)
class _FakeProc:
returncode = 0
stdout = b""
stderr = b""
def _fake_run(cmd, **kwargs):
if cmd[0] == "pg_dump":
i = cmd.index("-f")
Path(cmd[i + 1]).write_bytes(b"-- fake pg_dump\n")
elif cmd[0] == "tar" and "-cf" in cmd:
i = cmd.index("-cf")
Path(cmd[i + 1]).write_bytes(b"fake tar payload")
return _FakeProc()
monkeypatch.setattr("subprocess.run", _fake_run)
def _seed_backup(db_sync, *, kind, status, started_at, tag=None,
finished_at=None):
row = BackupRun(
kind=kind, status=status, tag=tag,
triggered_by="manual", started_at=started_at,
finished_at=finished_at or (started_at + timedelta(seconds=10)),
sql_path="/tmp/fake.sql" if kind == "db" else None,
tar_path="/tmp/fake.tar.zst" if kind == "images" else None,
manifest={},
)
db_sync.add(row)
db_sync.flush()
return row.id
# --- backup_db_task --------------------------------------------------
@pytest.mark.asyncio
async def test_backup_db_task_creates_backup_run_row_status_ok(db_sync):
from backend.app.tasks.backup import backup_db_task
result = backup_db_task.delay(tag=None, triggered_by="manual").get()
run_id = result["backup_run_id"]
row = db_sync.execute(
select(
BackupRun.kind, BackupRun.status, BackupRun.sql_path,
BackupRun.size_bytes, BackupRun.finished_at, BackupRun.error,
).where(BackupRun.id == run_id)
).one()
assert row.kind == "db"
assert row.status == "ok"
assert row.sql_path and row.sql_path.endswith(".sql")
assert row.size_bytes is not None and row.size_bytes > 0
assert row.finished_at is not None
assert row.error is None
@pytest.mark.asyncio
async def test_backup_db_task_records_failure_on_subprocess_error(db_sync, monkeypatch):
from backend.app.tasks.backup import backup_db_task
def _boom(*a, **kw):
raise RuntimeError("synthetic pg_dump fail")
monkeypatch.setattr("subprocess.run", _boom)
with pytest.raises(RuntimeError):
backup_db_task.delay().get()
row = db_sync.execute(
select(BackupRun.status, BackupRun.error)
.where(BackupRun.kind == "db")
.order_by(BackupRun.id.desc()).limit(1)
).one()
assert row.status == "error"
assert "synthetic" in (row.error or "")
@pytest.mark.asyncio
async def test_backup_db_task_persists_tag(db_sync):
from backend.app.tasks.backup import backup_db_task
result = backup_db_task.delay(tag="pre-cutover").get()
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == result["backup_run_id"])
).scalar_one()
assert tag == "pre-cutover"
# --- backup_images_task ---------------------------------------------
@pytest.mark.asyncio
async def test_backup_images_task_creates_backup_run(db_sync):
from backend.app.tasks.backup import backup_images_task
result = backup_images_task.delay().get()
row = db_sync.execute(
select(BackupRun.kind, BackupRun.status, BackupRun.tar_path)
.where(BackupRun.id == result["backup_run_id"])
).one()
assert row.kind == "images"
assert row.status == "ok"
assert row.tar_path and row.tar_path.endswith(".tar.zst")
# --- restore_db_task ------------------------------------------------
@pytest.mark.asyncio
async def test_restore_db_task_creates_restoring_marker_then_restored(db_sync):
from backend.app.tasks.backup import backup_db_task, restore_db_task
src = backup_db_task.delay().get()
src_id = src["backup_run_id"]
restore_db_task.delay(source_backup_run_id=src_id).get()
rows = db_sync.execute(
select(BackupRun.status, BackupRun.triggered_by)
.where(BackupRun.restored_from_id == src_id)
).all()
assert len(rows) == 1
assert rows[0].status == "restored"
assert rows[0].triggered_by == "restore"
@pytest.mark.asyncio
async def test_restore_db_task_rejects_non_db_source(db_sync):
from backend.app.tasks.backup import restore_db_task
now = datetime.now(UTC)
img_id = _seed_backup(db_sync, kind="images", status="ok", started_at=now)
db_sync.commit()
with pytest.raises(ValueError):
restore_db_task.delay(source_backup_run_id=img_id).get()
# --- prune_backups --------------------------------------------------
def test_prune_backups_keeps_last_n_untagged_per_kind(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 2
s.backup_images_keep_last_n = 1
db_sync.commit()
now = datetime.now(UTC)
for i in range(4):
_seed_backup(db_sync, kind="db", status="ok",
started_at=now - timedelta(hours=i))
for i in range(3):
_seed_backup(db_sync, kind="images", status="ok",
started_at=now - timedelta(hours=i))
db_sync.commit()
result = prune_backups.apply().get()
assert result["db_deleted"] == 2
assert result["images_deleted"] == 2
def test_prune_backups_protects_tagged_rows(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 1
db_sync.commit()
now = datetime.now(UTC)
_seed_backup(db_sync, kind="db", status="ok", started_at=now)
tagged_id = _seed_backup(
db_sync, kind="db", status="ok",
started_at=now - timedelta(days=30), tag="forever",
)
_seed_backup(db_sync, kind="db", status="ok",
started_at=now - timedelta(days=1))
db_sync.commit()
prune_backups.apply().get()
surviving_tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == tagged_id)
).scalar_one_or_none()
assert surviving_tag == "forever"
def test_prune_backups_never_deletes_running_or_restoring(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 0
db_sync.commit()
ancient = datetime.now(UTC) - timedelta(days=30)
running_id = _seed_backup(db_sync, kind="db", status="running", started_at=ancient)
restoring_id = _seed_backup(db_sync, kind="db", status="restoring", started_at=ancient)
db_sync.commit()
prune_backups.apply().get()
statuses = db_sync.execute(
select(BackupRun.status).where(BackupRun.id.in_([running_id, restoring_id]))
).scalars().all()
assert set(statuses) == {"running", "restoring"}
# --- backup_db_nightly ----------------------------------------------
def test_nightly_skips_when_disabled(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = False
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "skipped" in result and "disabled" in result["skipped"]
def test_nightly_skips_when_hour_mismatch(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = True
s.backup_db_nightly_hour_utc = (datetime.now(UTC).hour + 12) % 24
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "skipped" in result and "hour=" in result["skipped"]
def test_nightly_dispatches_when_enabled_at_configured_hour(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = True
s.backup_db_nightly_hour_utc = datetime.now(UTC).hour
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "dispatched" in result