Compare commits

..

42 Commits

Author SHA1 Message Date
bvandeusen b8dce6c483 Merge pull request 'FC-3h + FC-3k: backup first-class + admin destructive actions' (#10) from dev into main 2026-05-25 01:41:53 -04:00
bvandeusen 832345a245 fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:28:54 -04:00
bvandeusen a0136fa30d fix(fc3k): add mime=image/jpeg to test ImageRecord ctors (NOT NULL) + reorder admin import after stdlib/3rd-party (ruff I001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 01:12:52 -04:00
bvandeusen de1a4b64b7 fc3k(ui): TagMaintenanceCard — preview-then-commit prune-unused under Settings → Maintenance
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:52:30 -04:00
bvandeusen 3f500e592e fc3k(ui): per-tag dots-menu with Merge + Delete actions; Tier-B count-surfacing modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:51:51 -04:00
bvandeusen d97e3f9b59 fc3k(ui): bulk-delete action in BulkEditorPanel with sha8 confirm token + projected counts modal
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:49:45 -04:00
bvandeusen 035c49f675 fc3k(ui): ArtistDangerZone card + slot at bottom of ArtistView with cascade-delete flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:48:17 -04:00
bvandeusen e41ab1cca5 fc3k(ui): Pinia admin store — six endpoints + task_run polling for Tier-C dispatched ops
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:47:09 -04:00
bvandeusen 42c6b642c2 fc3k(ui): rename + relocate BackupConfirmModal → modal/DestructiveConfirmModal; add tier + projectedCounts props
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:46:37 -04:00
bvandeusen ad3d34a1fc fc3k: /api/admin endpoint integration tests — dry-run, confirm-mismatch, dispatch, Tier-A/B/C paths
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:45:31 -04:00
bvandeusen b5289ed372 fc3k: admin Celery task tests — registration, success, failure, missing-id idempotency
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:44:33 -04:00
bvandeusen f6aa805725 fc3k: cleanup_service unit tests — projections + mutations + file unlinks against real Postgres
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:43:51 -04:00
bvandeusen f096c9a5fb fc3k: register admin_bp in api/__init__ all_blueprints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:40 -04:00
bvandeusen 676a86b514 fc3k: /api/admin tags prune-unused endpoint — Tier-A preview-then-commit flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:20 -04:00
bvandeusen 44cc625d4a fc3k: /api/admin tag endpoints — Tier-B delete + merge + usage-count helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:03 -04:00
bvandeusen f7ee122243 fc3k: /api/admin blueprint — Tier-C artist cascade + bulk image delete with sha8-keyed confirm tokens
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:53 -04:00
bvandeusen 7c6f11964a fc3k: celery_app — register admin tasks on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:23 -04:00
bvandeusen 94c60c0af2 fc3k: admin Celery tasks — delete_artist_cascade_task + bulk_delete_images_task on maintenance queue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:03 -04:00
bvandeusen 6df102b83d fc3k: cleanup_service mutations — unlink primitive, artist cascade, bulk image delete, tag delete, prune unused
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:39:45 -04:00
bvandeusen 2ae01d27e3 fc3k: cleanup_service projections — artist cascade, bulk delete, tag usage, unused tags
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:38:57 -04:00
bvandeusen 718cc79905 fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:37:06 -04:00
bvandeusen e78a35d333 fc3h: collapse multi-line sqlalchemy import in backup_run.py — fits under line-length=100, ruff I001 would bounce
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:12:06 -04:00
bvandeusen 83bd3b4b2d fc3h(ui): slot BackupCard into Maintenance panel above migration card
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:22 -04:00
bvandeusen aecedd9fe4 fc3h(ui): BackupCard.vue + BackupRunsTable.vue — combined card with DB + Images sub-sections
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:06:00 -04:00
bvandeusen 1e34b1b428 fc3h(ui): BackupConfirmModal.vue — typed-token confirmation for restore + delete
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:05:04 -04:00
bvandeusen 102c21feaa fc3h(ui): Pinia backup store — runs, triggers, restore, delete, tag, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:04:37 -04:00
bvandeusen 57a338f7e6 fc3h(tests): drop pinned migration-backup tests (retired surface; coverage moved to test_backup_service.py + test_api_system_backup.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:03:57 -04:00
bvandeusen d04983138a fc3h: /api/system/backup endpoint integration tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:02:32 -04:00
bvandeusen 9ec6fdb596 fc3h: Celery task integration tests (backup, restore, prune, nightly)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:01:39 -04:00
bvandeusen 86ad9b80e9 fc3h: backup_service unit tests (subprocess monkeypatched)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:40 -04:00
bvandeusen 2b05f147f4 fc3h: migrators docstring — note backup/rollback retired to backup_service.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 23:00:01 -04:00
bvandeusen 70e1e010d1 fc3h: remove backend/app/services/migrators/backup.py + rollback.py (relocated to backup_service.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:18 -04:00
bvandeusen d3d4320ed5 fc3h: retire backup + rollback from migrate API/task — moved to /api/system/backup/* per FC-3h
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:59:11 -04:00
bvandeusen 7d42cddb11 fc3h: /api/system/backup blueprint — trigger, list, get, patch, restore, delete, settings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:57:25 -04:00
bvandeusen 1f01c4819a fc3h: celery_app — register backup tasks (include + maintenance route + Beat hourly tick + daily prune)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:56:06 -04:00
bvandeusen 06d527cb92 fc3h: prune_backups (daily retention) + backup_db_nightly (hourly tick, settings-gated)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:55:19 -04:00
bvandeusen e9ea376aed fc3h: restore_db_task + restore_images_task — restore creates 'restoring' marker row linked via restored_from_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:50 -04:00
bvandeusen 882cb491ba fc3h: backup_db_task + backup_images_task Celery tasks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:53:26 -04:00
bvandeusen 319e7de547 fc3h: backup_service.py — DB + images backup/restore + unlink helpers (relocated, split per-kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:35 -04:00
bvandeusen e43312a129 fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:52:00 -04:00
bvandeusen 8f2732a56f fc3h: alembic 0017 — backup_run table with indexes + partial-tag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:51:22 -04:00
bvandeusen c3e855bd9b fc3h: BackupRun model — artifact record for backup/restore runs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 22:50:59 -04:00
38 changed files with 4456 additions and 325 deletions
+82
View File
@@ -0,0 +1,82 @@
"""fc3h: backup_run table
Revision ID: 0017
Revises: 0016
Create Date: 2026-05-24
Additive. New table records every backup/restore attempt with artifact
metadata. Lifecycle tracking lives in task_run from FC-3i; this is
artifact-only (paths, sizes, tag, restore lineage).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0017"
down_revision: Union[str, None] = "0016"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"backup_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("tag", sa.String(length=64), nullable=True),
sa.Column("triggered_by", sa.String(length=32), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("sql_path", sa.Text(), nullable=True),
sa.Column("tar_path", sa.Text(), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"manifest", sa.JSON(), nullable=False, server_default="{}",
),
sa.Column(
"restored_from_id", sa.Integer(),
sa.ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
),
)
# Single-column indexes (matches Mapped[...].index=True).
op.create_index("ix_backup_run_kind", "backup_run", ["kind"])
op.create_index("ix_backup_run_status", "backup_run", ["status"])
op.create_index("ix_backup_run_tag", "backup_run", ["tag"])
op.create_index("ix_backup_run_started_at", "backup_run", ["started_at"])
op.create_index("ix_backup_run_finished_at", "backup_run", ["finished_at"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_backup_run_kind_started",
"backup_run", ["kind", sa.text("started_at DESC")],
)
op.create_index(
"ix_backup_run_status_finished",
"backup_run", ["status", sa.text("finished_at DESC")],
)
# Partial index: only tagged rows participate in retention-exempt query.
op.create_index(
"ix_backup_run_tag_partial",
"backup_run", ["tag"],
postgresql_where=sa.text("tag IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_backup_run_tag_partial", table_name="backup_run")
op.drop_index("ix_backup_run_status_finished", table_name="backup_run")
op.drop_index("ix_backup_run_kind_started", table_name="backup_run")
op.drop_index("ix_backup_run_finished_at", table_name="backup_run")
op.drop_index("ix_backup_run_started_at", table_name="backup_run")
op.drop_index("ix_backup_run_tag", table_name="backup_run")
op.drop_index("ix_backup_run_status", table_name="backup_run")
op.drop_index("ix_backup_run_kind", table_name="backup_run")
op.drop_table("backup_run")
@@ -0,0 +1,62 @@
"""fc3h: backup_* knobs on import_settings
Revision ID: 0018
Revises: 0017
Create Date: 2026-05-24
Adds four columns to the singleton import_settings row:
- backup_db_nightly_enabled (default False — opt-in)
- backup_db_nightly_hour_utc (default 3)
- backup_db_keep_last_n (default 14)
- backup_images_keep_last_n (default 3)
server_default ensures the singleton row is backfilled in place
without an UPDATE statement.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0018"
down_revision: Union[str, None] = "0017"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_enabled", sa.Boolean(),
nullable=False, server_default=sa.false(),
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_hour_utc", sa.Integer(),
nullable=False, server_default="3",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_keep_last_n", sa.Integer(),
nullable=False, server_default="14",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_images_keep_last_n", sa.Integer(),
nullable=False, server_default="3",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "backup_images_keep_last_n")
op.drop_column("import_settings", "backup_db_keep_last_n")
op.drop_column("import_settings", "backup_db_nightly_hour_utc")
op.drop_column("import_settings", "backup_db_nightly_enabled")
+4
View File
@@ -14,6 +14,7 @@ api_bp.add_url_rule("/health", view_func=health.get_health, methods=["GET"])
def all_blueprints() -> list[Blueprint]: def all_blueprints() -> list[Blueprint]:
from .admin import admin_bp
from .aliases import aliases_bp from .aliases import aliases_bp
from .allowlist import allowlist_bp from .allowlist import allowlist_bp
from .artist import artist_bp from .artist import artist_bp
@@ -34,6 +35,7 @@ def all_blueprints() -> list[Blueprint]:
from .sources import sources_bp from .sources import sources_bp
from .suggestions import suggestions_bp from .suggestions import suggestions_bp
from .system_activity import system_activity_bp from .system_activity import system_activity_bp
from .system_backup import system_backup_bp
from .tags import tags_bp from .tags import tags_bp
return [ return [
api_bp, api_bp,
@@ -46,6 +48,8 @@ def all_blueprints() -> list[Blueprint]:
showcase_bp, showcase_bp,
settings_bp, settings_bp,
system_activity_bp, system_activity_bp,
system_backup_bp,
admin_bp,
import_admin_bp, import_admin_bp,
migrate_bp, migrate_bp,
suggestions_bp, suggestions_bp,
+200
View File
@@ -0,0 +1,200 @@
"""FC-3k: /api/admin — destructive admin actions.
Five action surfaces:
POST /api/admin/artists/<slug>/cascade-delete (Tier C)
POST /api/admin/images/bulk-delete (Tier C)
DELETE /api/admin/tags/<int:tag_id> (Tier B)
POST /api/admin/tags/<int:dest_id>/merge (Tier B)
POST /api/admin/tags/prune-unused (Tier A)
GET /api/admin/tags/<int:tag_id>/usage-count (helper)
Tier-C ops take a dry_run body flag (returns projection inline,
no dispatch) and a confirm body field (server-recomputed token).
Long-running ops dispatch a maintenance-queue Celery task; the UI
tails FC-3i's /api/system/activity/runs to surface progress.
"""
from __future__ import annotations
import hashlib
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import Artist
from ..services.cleanup_service import project_artist_cascade, project_bulk_image_delete
admin_bp = Blueprint("admin", __name__, url_prefix="/api/admin")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _bulk_image_confirm_token(image_ids: list[int]) -> str:
"""Stable 8-hex token derived from the sorted id list. Mutates
when the selection changes; stays the same across modal opens of
the same selection so the operator can paste without confusion."""
canon = ",".join(str(i) for i in sorted(image_ids))
digest = hashlib.sha256(canon.encode("utf-8")).hexdigest()
return digest[:8]
@admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"])
async def artist_cascade_delete(slug: str):
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
artist = (await session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if artist is None:
return _bad("not_found", status=404)
artist_id = artist.id
projected = await session.run_sync(
lambda sync_sess: project_artist_cascade(sync_sess, slug=slug)
)
if dry_run:
return jsonify(projected)
expected = f"delete-artist-{artist_id}"
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import delete_artist_cascade_task
async_result = delete_artist_cascade_task.delay(artist_id=artist_id)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/images/bulk-delete", methods=["POST"])
async def images_bulk_delete():
body = await request.get_json(silent=True) or {}
image_ids = body.get("image_ids")
if not isinstance(image_ids, list) or not image_ids:
return _bad("invalid_image_ids", detail="image_ids must be non-empty list of int")
try:
image_ids = [int(i) for i in image_ids]
except (TypeError, ValueError):
return _bad("invalid_image_ids", detail="image_ids must contain only ints")
dry_run = bool(body.get("dry_run", False))
supplied_confirm = body.get("confirm", "")
async with get_session() as session:
projected = await session.run_sync(
lambda sync_sess: project_bulk_image_delete(
sync_sess, image_ids=image_ids,
)
)
if dry_run:
return jsonify(projected)
sha8 = _bulk_image_confirm_token(image_ids)
expected = f"delete-images-{sha8}"
if supplied_confirm != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..tasks.admin import bulk_delete_images_task
async_result = bulk_delete_images_task.delay(image_ids=image_ids)
return jsonify({"task_id": async_result.id}), 202
@admin_bp.route("/tags/<int:tag_id>", methods=["DELETE"])
async def tag_delete(tag_id: int):
"""Tier-B sync delete. UI yes/no modal is the only confirmation."""
from ..services.cleanup_service import delete_tag
async with get_session() as session:
try:
result = await session.run_sync(
lambda sync_sess: delete_tag(sync_sess, tag_id=tag_id)
)
except LookupError:
return _bad("not_found", status=404)
return jsonify(result)
@admin_bp.route("/tags/<int:dest_id>/merge", methods=["POST"])
async def tag_merge(dest_id: int):
"""Wraps TagService.merge. Source repoints to dest, dest survives,
source row deleted, protective alias auto-created if source was
ML-applied or allowlisted."""
from ..services.tag_service import TagMergeConflict, TagService, TagValidationError
body = await request.get_json(silent=True) or {}
source_id = body.get("source_id")
if not isinstance(source_id, int) or source_id == dest_id:
return _bad("invalid_source_id", detail="source_id must be int and differ from dest")
async with get_session() as session:
try:
result = await TagService(session).merge(
source_id=source_id, target_id=dest_id,
)
except TagMergeConflict as exc:
return _bad("merge_conflict", status=409, detail=str(exc))
except TagValidationError as exc:
return _bad("tag_kind_mismatch", detail=str(exc))
except LookupError:
return _bad("not_found", status=404)
# MergeResult is a frozen dataclass — flatten to dict.
return jsonify({
"result": {
"target_id": result.target_id,
"target_name": result.target_name,
"target_kind": result.target_kind,
"merged_count": result.merged_count,
"alias_created": result.alias_created,
"source_deleted": result.source_deleted,
},
})
@admin_bp.route("/tags/<int:tag_id>/usage-count", methods=["GET"])
async def tag_usage_count(tag_id: int):
"""Helper for the Tier-B yes/no prompt; surfaces "N associations"
in the dialog so the operator knows what they're nuking."""
from ..services.cleanup_service import count_tag_associations
async with get_session() as session:
count = await session.run_sync(
lambda sync_sess: count_tag_associations(
sync_sess, tag_id=tag_id,
)
)
return jsonify({"count": count})
@admin_bp.route("/tags/prune-unused", methods=["POST"])
async def tags_prune_unused():
"""Tier-A: dry-run preview list IS the prompt. UI calls with
dry_run=true first, shows the list, operator clicks button to
re-call with dry_run=false."""
from ..services.cleanup_service import prune_unused_tags
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: prune_unused_tags(
sync_sess, dry_run=dry_run,
)
)
return jsonify(result)
+5 -39
View File
@@ -1,14 +1,11 @@
"""FC-5: /api/migrate — trigger and poll migration runs. """FC-5: /api/migrate — trigger and poll migration runs.
Ingest kinds (gs_ingest, ir_ingest) accept multipart/form-data with an Ingest kinds (gs_ingest, ir_ingest) accept multipart/form-data with an
`export_file` field. All other kinds accept JSON. Apply-without-backup `export_file` field. All other kinds accept JSON. Backup + rollback
guard rejects non-dry-run ingests unless a pre_migration-tagged backup were retired in FC-3h (2026-05-24); use /api/system/backup/* instead.
exists in the last 24h (override with body.force=true).
""" """
import json import json
from datetime import UTC, datetime, timedelta
from pathlib import Path
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
from sqlalchemy import select from sqlalchemy import select
@@ -19,17 +16,12 @@ from ..tasks.migration import run_migration
migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate") migrate_bp = Blueprint("migrate", __name__, url_prefix="/api/migrate")
# 'backup' + 'rollback' retired 2026-05-24 (FC-3h); see /api/system/backup/*.
_VALID_KINDS = frozenset({ _VALID_KINDS = frozenset({
"backup", "gs_ingest", "ir_ingest", "tag_apply", "gs_ingest", "ir_ingest", "tag_apply",
"ml_queue", "verify", "rollback", "cleanup", "ml_queue", "verify", "cleanup",
}) })
_INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"}) _INGEST_KINDS = frozenset({"gs_ingest", "ir_ingest"})
# Backup gate retired 2026-05-24 — operator-flagged the speculative-safety
# requirement was actively blocking the UI ingest path (backup itself is
# unreliable on large NFS-backed image libraries) and FC-3h will rewrite
# the backup surface as a first-class feature with its own scheduling
# + recovery. Leaving the constant for historical grep.
_APPLY_KINDS: frozenset[str] = frozenset()
def _bad(error: str, *, status: int = 400, **extra): def _bad(error: str, *, status: int = 400, **extra):
@@ -38,19 +30,6 @@ def _bad(error: str, *, status: int = 400, **extra):
return jsonify(body), status return jsonify(body), status
def _has_recent_pre_migration_backup() -> bool:
from ..services.migrators import backup as backup_mod
images_root = Path("/images")
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
return False
created_at_str = manifest.get("created_at")
if not created_at_str:
return False
created_at = datetime.fromisoformat(created_at_str)
return (datetime.now(UTC) - created_at) < timedelta(hours=24)
def _run_to_dict(run: MigrationRun) -> dict: def _run_to_dict(run: MigrationRun) -> dict:
return { return {
"id": run.id, "id": run.id,
@@ -83,7 +62,6 @@ async def create_run(kind: str):
except (UnicodeDecodeError, json.JSONDecodeError) as exc: except (UnicodeDecodeError, json.JSONDecodeError) as exc:
return _bad("invalid_export_file", detail=str(exc)) return _bad("invalid_export_file", detail=str(exc))
dry_run = str(form.get("dry_run", "false")).lower() in ("true", "1", "yes") dry_run = str(form.get("dry_run", "false")).lower() in ("true", "1", "yes")
force = str(form.get("force", "false")).lower() in ("true", "1", "yes")
params: dict = {"data": data, "dry_run": dry_run} params: dict = {"data": data, "dry_run": dry_run}
else: else:
body = await request.get_json() body = await request.get_json()
@@ -92,20 +70,8 @@ async def create_run(kind: str):
if not isinstance(body, dict): if not isinstance(body, dict):
return _bad("invalid_body") return _bad("invalid_body")
dry_run = bool(body.get("dry_run", False)) dry_run = bool(body.get("dry_run", False))
force = bool(body.get("force", False))
params = dict(body) params = dict(body)
is_apply = (kind in _APPLY_KINDS) and not dry_run
if is_apply and not force and not _has_recent_pre_migration_backup():
return _bad(
"no_backup",
detail="apply action requires a pre_migration-tagged backup "
"in the last 24h (or force=true).",
)
if kind == "backup":
params.setdefault("tag", "pre_migration")
async with get_session() as session: async with get_session() as session:
run = MigrationRun(kind=kind, status="pending", dry_run=dry_run) run = MigrationRun(kind=kind, status="pending", dry_run=dry_run)
session.add(run) session.add(run)
+264
View File
@@ -0,0 +1,264 @@
"""FC-3h: /api/system/backup — create/list/restore/delete/tag for
DB + image backups.
Read endpoints are public on FC (operator-facing internal API; same
posture as /api/system/activity). Write endpoints take a typed
`confirm` body field that must match a server-generated token for
that backup row, to prevent click-to-destroy by stale browser tabs
or accidental cURL.
"""
from __future__ import annotations
from quart import Blueprint, jsonify, request
from sqlalchemy import desc, select
from ..extensions import get_session
from ..models import BackupRun, ImportSettings
system_backup_bp = Blueprint(
"system_backup", __name__, url_prefix="/api/system/backup",
)
_KINDS = frozenset({"db", "images"})
_TAG_MAX_LEN = 64
_BACKUP_SETTINGS_FIELDS = (
"backup_db_nightly_enabled",
"backup_db_nightly_hour_utc",
"backup_db_keep_last_n",
"backup_images_keep_last_n",
)
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _row_to_dict(r: BackupRun) -> dict:
return {
"id": r.id,
"kind": r.kind,
"status": r.status,
"tag": r.tag,
"triggered_by": r.triggered_by,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": (
int((r.finished_at - r.started_at).total_seconds())
if r.finished_at and r.started_at else None
),
"sql_path": r.sql_path,
"tar_path": r.tar_path,
"size_bytes": r.size_bytes,
"error": r.error,
"restored_from_id": r.restored_from_id,
"manifest": r.manifest or {},
}
def _validate_tag(tag):
if tag is None:
return None
if not isinstance(tag, str):
return _bad("invalid_tag", detail="tag must be string or null")
tag = tag.strip()
if not tag:
return None
if len(tag) > _TAG_MAX_LEN:
return _bad("invalid_tag", detail=f"tag too long (max {_TAG_MAX_LEN})")
return tag
def _validate_backup_settings_patch(body: dict):
if "backup_db_nightly_enabled" in body and not isinstance(
body["backup_db_nightly_enabled"], bool,
):
return _bad("invalid_value", detail="backup_db_nightly_enabled must be bool")
if "backup_db_nightly_hour_utc" in body:
v = body["backup_db_nightly_hour_utc"]
if not isinstance(v, int) or isinstance(v, bool) or not (0 <= v <= 23):
return _bad("invalid_value", detail="backup_db_nightly_hour_utc must be 0..23")
if "backup_db_keep_last_n" in body:
v = body["backup_db_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 365):
return _bad("invalid_value", detail="backup_db_keep_last_n must be 1..365")
if "backup_images_keep_last_n" in body:
v = body["backup_images_keep_last_n"]
if not isinstance(v, int) or isinstance(v, bool) or not (1 <= v <= 100):
return _bad("invalid_value", detail="backup_images_keep_last_n must be 1..100")
return None
@system_backup_bp.route("/db", methods=["POST"])
async def trigger_db_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_db_task
backup_db_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/images", methods=["POST"])
async def trigger_images_backup():
body = await request.get_json(silent=True) or {}
tag = _validate_tag(body.get("tag"))
if isinstance(tag, tuple):
return tag
from ..tasks.backup import backup_images_task
backup_images_task.delay(tag=tag, triggered_by="manual")
return jsonify({"status": "dispatched"}), 202
@system_backup_bp.route("/runs", methods=["GET"])
async def list_runs():
try:
limit = min(int(request.args.get("limit", "50")), 200)
except ValueError:
return _bad("invalid_limit")
if limit < 1:
return _bad("invalid_limit")
kind = request.args.get("kind")
if kind is not None and kind not in _KINDS:
return _bad("invalid_kind", detail=f"kind must be one of {sorted(_KINDS)}")
before_id_raw = request.args.get("before_id")
before_id = int(before_id_raw) if before_id_raw else None
async with get_session() as session:
stmt = select(BackupRun).order_by(desc(BackupRun.id))
if kind:
stmt = stmt.where(BackupRun.kind == kind)
if before_id is not None:
stmt = stmt.where(BackupRun.id < before_id)
stmt = stmt.limit(limit + 1)
rows = (await session.execute(stmt)).scalars().all()
has_more = len(rows) > limit
rows = rows[:limit]
return jsonify({
"runs": [_row_to_dict(r) for r in rows],
"next_cursor": rows[-1].id if has_more and rows else None,
})
@system_backup_bp.route("/runs/<int:run_id>", methods=["GET"])
async def get_run(run_id: int):
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>", methods=["PATCH"])
async def patch_run(run_id: int):
body = await request.get_json(silent=True) or {}
if "tag" not in body:
return _bad("invalid_body", detail="tag required")
tag = _validate_tag(body["tag"])
if isinstance(tag, tuple):
return tag
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
row.tag = tag
await session.commit()
await session.refresh(row)
return jsonify(_row_to_dict(row))
@system_backup_bp.route("/runs/<int:run_id>/restore", methods=["POST"])
async def trigger_restore(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
if row.status != "ok":
return _bad(
"not_restorable",
detail=f"source backup status={row.status!r}; only 'ok' rows are restorable",
)
expected = f"restore-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
kind = row.kind
if kind == "db":
from ..tasks.backup import restore_db_task
restore_db_task.delay(source_backup_run_id=run_id)
else: # 'images' (the only other value _KINDS allows via the trigger path)
from ..tasks.backup import restore_images_task
restore_images_task.delay(source_backup_run_id=run_id)
return jsonify({"status": "dispatched", "kind": kind}), 202
@system_backup_bp.route("/runs/<int:run_id>", methods=["DELETE"])
async def delete_run(run_id: int):
body = await request.get_json(silent=True) or {}
supplied = body.get("confirm", "")
async with get_session() as session:
row = await session.get(BackupRun, run_id)
if row is None:
return _bad("not_found", status=404)
expected = f"delete-{row.kind}-{row.id}"
if supplied != expected:
return _bad(
"confirm_mismatch",
detail=f"confirm must equal {expected!r}",
expected=expected,
)
from ..services import backup_service
backup_service.unlink_artifact_files(
sql_path=row.sql_path, tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
await session.delete(row)
await session.commit()
return "", 204
@system_backup_bp.route("/settings", methods=["GET"])
async def get_settings():
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
return jsonify({
"backup_db_nightly_enabled": row.backup_db_nightly_enabled,
"backup_db_nightly_hour_utc": row.backup_db_nightly_hour_utc,
"backup_db_keep_last_n": row.backup_db_keep_last_n,
"backup_images_keep_last_n": row.backup_images_keep_last_n,
})
@system_backup_bp.route("/settings", methods=["PATCH"])
async def patch_settings():
body = await request.get_json(silent=True)
if not isinstance(body, dict):
return _bad("invalid_body", detail="body must be a JSON object")
err = _validate_backup_settings_patch(body)
if err is not None:
return err
async with get_session() as session:
row = (await session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
)).scalar_one()
for field in _BACKUP_SETTINGS_FIELDS:
if field in body:
setattr(row, field, body[field])
await session.commit()
return await get_settings()
+12
View File
@@ -31,6 +31,8 @@ def make_celery() -> Celery:
"backend.app.tasks.migration", "backend.app.tasks.migration",
"backend.app.tasks.ml", "backend.app.tasks.ml",
"backend.app.tasks.download", "backend.app.tasks.download",
"backend.app.tasks.backup",
"backend.app.tasks.admin",
], ],
) )
app.conf.update( app.conf.update(
@@ -43,6 +45,8 @@ def make_celery() -> Celery:
"backend.app.tasks.scan.*": {"queue": "scan"}, "backend.app.tasks.scan.*": {"queue": "scan"},
"backend.app.tasks.maintenance.*": {"queue": "maintenance"}, "backend.app.tasks.maintenance.*": {"queue": "maintenance"},
"backend.app.tasks.migration.*": {"queue": "maintenance"}, "backend.app.tasks.migration.*": {"queue": "maintenance"},
"backend.app.tasks.backup.*": {"queue": "maintenance"},
"backend.app.tasks.admin.*": {"queue": "maintenance"},
}, },
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent. # Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True, task_acks_late=True,
@@ -89,6 +93,14 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.prune_task_runs", "task": "backend.app.tasks.maintenance.prune_task_runs",
"schedule": 86400.0, # daily "schedule": 86400.0, # daily
}, },
"fc3h-backup-db-nightly": {
"task": "backend.app.tasks.backup.backup_db_nightly",
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
},
"fc3h-prune-backups": {
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
}, },
timezone="UTC", timezone="UTC",
) )
+2
View File
@@ -2,6 +2,7 @@
from .app_setting import AppSetting from .app_setting import AppSetting
from .artist import Artist from .artist import Artist
from .backup_run import BackupRun
from .base import Base from .base import Base
from .credential import Credential from .credential import Credential
from .download_event import DownloadEvent from .download_event import DownloadEvent
@@ -27,6 +28,7 @@ __all__ = [
"Base", "Base",
"AppSetting", "AppSetting",
"Artist", "Artist",
"BackupRun",
"Source", "Source",
"Credential", "Credential",
"Post", "Post",
+55
View File
@@ -0,0 +1,55 @@
"""FC-3h: backup_run — operator-facing artifact record for a backup run.
One row per backup attempt (kind='db' or 'images'). Lifecycle
tracking (started_at/finished_at/duration_ms/exception text) lives
in task_run from FC-3i — this row records artifact metadata: file
paths, sizes, tag (retention protection), and restore lineage via
restored_from_id.
Status values (String, not Postgres ENUM — per
feedback_check_existing_enums):
pending — created but task hasn't started yet (rare; usually
status starts as 'running' from the task body).
running — backup task is in flight.
ok — artifact successfully written.
error — task raised; error column populated.
restoring — this row represents a restore attempt (kind = restored
kind); linked to source via restored_from_id.
restored — restore completed successfully.
"""
from datetime import datetime
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class BackupRun(Base):
__tablename__ = "backup_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True,
)
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True,
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True,
)
sql_path: Mapped[str | None] = mapped_column(Text, nullable=True)
tar_path: Mapped[str | None] = mapped_column(Text, nullable=True)
size_bytes: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
manifest: Mapped[dict] = mapped_column(
JSON, nullable=False, default=dict, server_default="{}",
)
restored_from_id: Mapped[int | None] = mapped_column(
ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
)
+14
View File
@@ -49,3 +49,17 @@ class ImportSettings(Base):
download_failure_warning_threshold: Mapped[int] = mapped_column( download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5 Integer, nullable=False, default=5
) )
# FC-3h backup knobs.
backup_db_nightly_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False,
)
backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
backup_db_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=14,
)
backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3,
)
+196
View File
@@ -0,0 +1,196 @@
"""FC-3h: first-class backup/restore service for FC.
Two independent backup kinds:
- 'db' — pg_dump only; fast; nightly via Beat (settings-gated)
- 'images' — tar+zstd of /images; slow; manual trigger only
Files live under <images_root>/_backups/. Each backup writes:
fc_<kind>_<ts>.{sql|tar.zst} — the artifact
fc_<kind>_<ts>.json — manifest (kind/tag/triggered_by)
Service functions are sync (subprocess-bound). Celery tasks in
backend.app.tasks.backup wrap each one with task_run-tracked
lifecycle + soft/hard time limits + retention bookkeeping.
"""
from __future__ import annotations
import json
import subprocess
from datetime import UTC, datetime
from pathlib import Path
_BACKUPS_DIRNAME = "_backups"
# Subprocess-level guardrails BEYOND the Celery soft_time_limit. The
# Celery soft limit signals the Python process; subprocess.Popen in a
# blocking syscall ignores that signal. These bound the worst case.
_DB_SUBPROCESS_TIMEOUT_S = 12 * 60 # 12 min (Celery soft is 10 min)
_IMAGES_SUBPROCESS_TIMEOUT_S = 7 * 60 * 60 # 7 hr (Celery soft is 6 hr)
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy +psycopg/+asyncpg driver suffix for pg_dump/psql."""
for driver in (
"postgresql+psycopg",
"postgresql+asyncpg",
"postgresql+psycopg2",
):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path) -> Path:
p = images_root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
def _now_ts() -> str:
return datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
def _file_size_or_none(path: Path) -> int | None:
try:
return path.stat().st_size
except OSError:
return None
def _write_manifest(
out_dir: Path, *, kind: str, ts: str,
tag: str | None, triggered_by: str,
artifact_path: Path,
) -> Path:
manifest = {
"kind": kind,
"backup_id": f"fc_{kind}_{ts}",
"tag": tag,
"triggered_by": triggered_by,
"created_at": datetime.now(UTC).isoformat(),
"artifact_path": str(artifact_path),
}
mf = out_dir / f"fc_{kind}_{ts}.json"
mf.write_text(json.dumps(manifest, indent=2))
return mf
def backup_db(
*, db_url: str, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run pg_dump; write .sql + manifest; return dict for the caller
to persist into BackupRun. Raises on subprocess failure."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_db_{ts}.sql"
subprocess.run(
[
"pg_dump", "--no-owner", "--no-acl",
"-f", str(sql_path), _libpq_url(db_url),
],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="db", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=sql_path,
)
return {
"kind": "db",
"ts": ts,
"sql_path": str(sql_path),
"tar_path": None,
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(sql_path),
}
def backup_images(
*, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run tar --zstd over images_root; write .tar.zst + manifest."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
subprocess.run(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=tar_path,
)
return {
"kind": "images",
"ts": ts,
"sql_path": None,
"tar_path": str(tar_path),
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(tar_path),
}
def restore_db(*, db_url: str, sql_path: Path) -> None:
"""Wipe public schema, then load from .sql. Raises on subprocess
failure; partial-restore state is the caller's concern."""
libpq = _libpq_url(db_url)
subprocess.run(
[
"psql", libpq, "-c",
"DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public;",
],
capture_output=True, check=True, timeout=120,
)
subprocess.run(
["psql", libpq, "-f", str(sql_path)],
capture_output=True, check=True,
timeout=_DB_SUBPROCESS_TIMEOUT_S,
)
def restore_images(*, images_root: Path, tar_path: Path) -> None:
"""Untar over images_root.parent. Additive — files NOT in the
tarball are NOT removed. Caller wipes first if a clean restore
is needed."""
subprocess.run(
[
"tar", "--zstd", "-xf", str(tar_path),
"-C", str(images_root.parent),
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
def unlink_artifact_files(
*,
sql_path: str | None,
tar_path: str | None,
manifest_path: str | None,
) -> dict:
"""Best-effort unlink of all on-disk files for a BackupRun row.
Returns dict keyed by label with True/False per file. Missing
files count as success (missing_ok semantics)."""
deleted: dict = {}
for label, p in (
("sql", sql_path),
("tar", tar_path),
("manifest", manifest_path),
):
if not p:
continue
path = Path(p)
try:
path.unlink(missing_ok=True)
deleted[label] = True
except OSError:
deleted[label] = False
return deleted
+367
View File
@@ -0,0 +1,367 @@
"""FC-3k: first-class admin destructive operations.
Projections are pure SELECTs used by both dry-run preview endpoints
and Tier-B count prompts. Mutations (Task 2) are called from sync
HTTP handlers (small ops) and from Celery tasks in
backend.app.tasks.admin (long ops).
This module is the PERMANENT home of artist-cascade + image-unlink
logic. The legacy copy at backend/app/services/migrators/cleanup.py
stays in place until FC-3j; FC-3j will replace its body with thin
re-exports from this module and then delete the wrapper.
"""
from __future__ import annotations
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from ..models import Artist, ImageRecord, Tag
from ..models.series_page import SeriesPage
from ..models.tag import image_tag
def project_artist_cascade(session: Session, *, slug: str) -> dict:
"""Read-only projection of what delete_artist_cascade would touch.
Returns:
{
"artist": {"id": int, "name": str, "slug": str},
"projected": {
"images": int,
"sources": int,
"thumbs": int, # images with a thumbnail_path set
"import_tasks": int, # ImportTask rows referencing the artist's images
"bytes_on_disk": int, # SUM(image_record.size_bytes) — column is NOT NULL
},
}
Raises LookupError if slug not found. No mutations.
"""
from ..models.import_task import ImportTask
from ..models.source import Source
artist = session.execute(
select(Artist).where(Artist.slug == slug)
).scalar_one_or_none()
if artist is None:
raise LookupError(f"artist slug not found: {slug!r}")
images_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
sources_count = session.execute(
select(func.count(Source.id))
.where(Source.artist_id == artist.id)
).scalar_one()
thumbs_count = session.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist.id)
.where(ImageRecord.thumbnail_path.is_not(None))
).scalar_one()
import_tasks_count = session.execute(
select(func.count(ImportTask.id))
.where(
ImportTask.result_image_id.in_(
select(ImageRecord.id).where(ImageRecord.artist_id == artist.id)
)
)
).scalar_one()
bytes_on_disk = session.execute(
select(func.coalesce(func.sum(ImageRecord.size_bytes), 0))
.where(ImageRecord.artist_id == artist.id)
).scalar_one()
return {
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"projected": {
"images": images_count,
"sources": sources_count,
"thumbs": thumbs_count,
"import_tasks": import_tasks_count,
"bytes_on_disk": int(bytes_on_disk),
},
}
def project_bulk_image_delete(
session: Session, *, image_ids: list[int],
) -> dict:
"""Read-only projection of what delete_images would touch.
Returns:
{
"images_found": int,
"thumbs_to_unlink": int,
"bytes_on_disk": int,
"missing_ids": list[int], # ids passed in that don't exist
}
No mutations.
"""
if not image_ids:
return {
"images_found": 0,
"thumbs_to_unlink": 0,
"bytes_on_disk": 0,
"missing_ids": [],
}
rows = session.execute(
select(
ImageRecord.id,
ImageRecord.thumbnail_path,
ImageRecord.size_bytes,
).where(ImageRecord.id.in_(image_ids))
).all()
found_ids = {r.id for r in rows}
missing = sorted(set(image_ids) - found_ids)
return {
"images_found": len(rows),
"thumbs_to_unlink": sum(1 for r in rows if r.thumbnail_path),
"bytes_on_disk": sum(r.size_bytes for r in rows),
"missing_ids": missing,
}
def count_tag_associations(session: Session, *, tag_id: int) -> int:
"""COUNT(*) FROM image_tag WHERE tag_id=?. For Tier-B prompt."""
return session.execute(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id == tag_id)
).scalar_one()
def find_unused_tags(
session: Session, *, limit: int | None = None,
) -> list[Tag]:
"""Tags with no image_tag rows AND no series_page rows.
Sorted by name. Used by both dry-run preview and the live prune.
A tag is "unused" iff it has zero rows in image_tag AND zero rows
in series_page (so we don't accidentally prune a series tag that
happens to have no images yet).
"""
used_via_image_tag = select(image_tag.c.tag_id).distinct()
used_via_series = select(SeriesPage.series_tag_id).where(
SeriesPage.series_tag_id.is_not(None)
).distinct()
stmt = (
select(Tag)
.where(Tag.id.not_in(used_via_image_tag))
.where(Tag.id.not_in(used_via_series))
.order_by(Tag.name)
)
if limit is not None:
stmt = stmt.limit(limit)
return list(session.execute(stmt).scalars().all())
def unlink_image_files(
image: ImageRecord, images_root: Path,
) -> dict:
"""Best-effort unlink of all on-disk files for an ImageRecord.
Targets: image.path (original), image.thumbnail_path (cached
thumbnail), and the computed thumbs path at
/images/thumbs/<sha256[:3]>/<sha256>.(jpg|png|webp) (tries all
three extensions; missing extension is silently OK).
Returns {"original": bool, "thumbnail": bool}. Missing files
count as success (missing_ok semantics). OSErrors are swallowed
and reported as False so the calling DB delete still proceeds.
"""
out = {"original": False, "thumbnail": False}
if image.path:
try:
Path(image.path).unlink(missing_ok=True)
out["original"] = True
except OSError:
out["original"] = False
# Custom thumbnail_path (when set) — try it first.
if image.thumbnail_path:
try:
Path(image.thumbnail_path).unlink(missing_ok=True)
out["thumbnail"] = True
except OSError:
out["thumbnail"] = False
# Convention thumbs dir — try all extensions; missing OK.
if image.sha256:
bucket = image.sha256[:3]
for ext in ("jpg", "png", "webp"):
try:
(images_root / "thumbs" / bucket / f"{image.sha256}.{ext}").unlink(
missing_ok=True,
)
except OSError:
pass
return out
def delete_artist_cascade(
session: Session, *, artist_id: int, images_root: Path,
) -> dict:
"""Batched delete of an artist's images + the artist row.
Mirrors the cleanup_artist_async pattern: 500-row batches,
commit between batches so partial progress survives a worker
kill. Idempotent on missing artist (returns zeroed counts).
Postgres cascades handle image_tag / image_provenance /
series_page / tag_suggestion_rejection from ImageRecord delete,
and source / post / download_event / etc. from Artist delete
(via Artist.sources cascade="all, delete-orphan").
"""
artist = session.get(Artist, artist_id)
if artist is None:
return {
"artist": None,
"summary": {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"import_tasks_nulled": 0,
"files_failed": 0,
},
}
artist_info = {"id": artist.id, "name": artist.name, "slug": artist.slug}
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
while True:
rows = session.execute(
select(ImageRecord)
.where(ImageRecord.artist_id == artist.id)
.limit(500)
).scalars().all()
if not rows:
break
for img in rows:
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
# ImportTask.result_image_id FK is SET NULL on image delete (Postgres
# handles this in the cascade above). We don't separately count those
# in FC-3k — the legacy cleanup_artist_async did it via
# source_path_prefix matching that's out of scope here.
import_tasks_nulled = 0
session.delete(artist)
session.commit()
return {
"artist": artist_info,
"summary": {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"import_tasks_nulled": import_tasks_nulled,
"files_failed": files_failed,
},
}
def delete_images(
session: Session, *, image_ids: list[int], images_root: Path,
) -> dict:
"""Delete a list of images in 500-row batches with commit between.
Postgres CASCADE on image_tag / image_provenance / series_page /
tag_suggestion_rejection / post_attachment(FK SET NULL) handles
the DB side; this function handles file unlinks first then row
deletes. Idempotent on missing IDs (returned as missing_ids;
no error). On partial OSError, the row is still deleted and
files_failed is incremented.
"""
if not image_ids:
return {
"images_deleted": 0,
"files_deleted": 0,
"thumbs_deleted": 0,
"files_failed": 0,
"missing_ids": [],
}
seen_ids: set[int] = set()
images_deleted = 0
files_deleted = 0
thumbs_deleted = 0
files_failed = 0
pending = list(image_ids)
while pending:
batch_ids = pending[:500]
pending = pending[500:]
rows = session.execute(
select(ImageRecord).where(ImageRecord.id.in_(batch_ids))
).scalars().all()
for img in rows:
seen_ids.add(img.id)
unlinked = unlink_image_files(img, images_root)
if unlinked["original"]:
files_deleted += 1
else:
files_failed += 1
if unlinked["thumbnail"]:
thumbs_deleted += 1
session.delete(img)
images_deleted += 1
session.commit()
missing = sorted(set(image_ids) - seen_ids)
return {
"images_deleted": images_deleted,
"files_deleted": files_deleted,
"thumbs_deleted": thumbs_deleted,
"files_failed": files_failed,
"missing_ids": missing,
}
def delete_tag(session: Session, *, tag_id: int) -> dict:
"""Simple DELETE FROM tag WHERE id=?.
Postgres cascades the rest (image_tag, tag_alias, tag_allowlist,
tag_reference_embedding, tag_suggestion_rejection, series_page).
Returns counts BEFORE delete so the caller can surface them.
Raises LookupError if tag_id not found.
"""
tag = session.get(Tag, tag_id)
if tag is None:
raise LookupError(f"tag id not found: {tag_id}")
associations_count = count_tag_associations(session, tag_id=tag_id)
info = {"id": tag.id, "name": tag.name, "kind": tag.kind.value}
session.delete(tag)
session.commit()
return {"deleted": info, "associations_removed": associations_count}
def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
"""Find tags with zero references and (unless dry_run) delete them.
Returns:
dry_run=True: {"count": N, "sample_names": [first 50]}
dry_run=False: {"deleted": N, "sample_names": [first 50]}
"""
unused = find_unused_tags(session)
sample = [t.name for t in unused[:50]]
if dry_run:
return {"count": len(unused), "sample_names": sample}
ids = [t.id for t in unused]
if ids:
session.execute(
Tag.__table__.delete().where(Tag.id.in_(ids))
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
+5 -1
View File
@@ -1,6 +1,10 @@
"""FC-5 migration tooling. """FC-5 migration tooling.
One module per concern (backup/rollback/gs/ir/overlap/ml_queue/verify). One module per concern (gs/ir/overlap/ml_queue/verify/cleanup).
Each migrator returns a counts dict; the run_migration task wires Each migrator returns a counts dict; the run_migration task wires
that dict into MigrationRun.counts so the UI polling shows progress. that dict into MigrationRun.counts so the UI polling shows progress.
backup + rollback were retired in FC-3h (2026-05-24); first-class
backup lives at backend/app/services/backup_service.py and exposes
its own /api/system/backup/* surface.
""" """
-146
View File
@@ -1,146 +0,0 @@
"""pg_dump + tar.zst-based backup, restorable via pair of subprocess calls.
Backups live under <images_root>/_backups/. Each backup is two files
(SQL + tarball) plus a manifest JSON. Tagged backups (e.g. tag='pre_migration')
are how rollback.py finds the most recent restorable snapshot.
"""
from __future__ import annotations
import json
import shutil
import subprocess
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
_BACKUPS_DIRNAME = "_backups"
def _libpq_url(sa_url: str) -> str:
"""Strip SQLAlchemy driver suffix so pg_dump/psql accept the URL.
SQLAlchemy uses URLs like `postgresql+psycopg://...` or
`postgresql+asyncpg://...`. libpq tools (pg_dump, psql) only know
the plain `postgresql://` scheme.
"""
for driver in ("postgresql+psycopg", "postgresql+asyncpg", "postgresql+psycopg2"):
if sa_url.startswith(driver + "://"):
return "postgresql://" + sa_url[len(driver) + 3:]
return sa_url
def _backups_dir(images_root: Path | None = None) -> Path:
# Overridable for tests via monkeypatch.
root = images_root if images_root is not None else Path("/images")
p = root / _BACKUPS_DIRNAME
p.mkdir(parents=True, exist_ok=True)
return p
_DEFAULT_SUBPROCESS_TIMEOUT_S = 30 * 60 # 30 minutes
def _run_subprocess(cmd: list[str], **kwargs: Any):
# Overridable for tests via monkeypatch. Hard wall-clock timeout
# guards against pg_dump / tar / zstd hangs on NFS — without it the
# task pretends to be 'running' forever (operator hit this 2026-05-
# 23 with two backups stuck in MigrationRun). On timeout
# subprocess.run raises TimeoutExpired which the caller surfaces as
# a task error.
return subprocess.run(
cmd,
capture_output=True,
check=True,
timeout=_DEFAULT_SUBPROCESS_TIMEOUT_S,
**{k: v for k, v in kwargs.items() if not k.startswith("_")},
)
def create_backup(
*, db_url: str, images_root: Path, tag: str = "manual",
) -> dict:
"""Create a backup: pg_dump SQL + tar.zst of images.
Returns a manifest dict. Writes <ts>.sql, <ts>.tar.zst, <ts>.json into
<images_root>/_backups/.
"""
ts = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
out_dir = _backups_dir(images_root)
sql_path = out_dir / f"fc_{ts}.sql"
tar_path = out_dir / f"fc_{ts}.tar.zst"
manifest_path = out_dir / f"fc_{ts}.json"
_run_subprocess(
["pg_dump", "--no-owner", "--no-acl", "-f", str(sql_path), _libpq_url(db_url)],
_test_ts=ts,
)
_run_subprocess(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
f"--exclude={images_root.name}/_backups",
f"--exclude={images_root.name}/_quarantine",
],
_test_ts=ts,
)
manifest = {
"backup_id": ts,
"tag": tag,
"created_at": datetime.now(UTC).isoformat(),
"sql_path": str(sql_path),
"tar_path": str(tar_path),
}
manifest_path.write_text(json.dumps(manifest, indent=2))
return manifest
def list_backups(images_root: Path) -> list[dict]:
out_dir = _backups_dir(images_root)
items = []
for mf in sorted(out_dir.glob("fc_*.json"), reverse=True):
try:
items.append(json.loads(mf.read_text()))
except Exception:
continue
return items
def find_latest_backup(images_root: Path, *, tag: str) -> dict | None:
for mf in list_backups(images_root):
if mf.get("tag") == tag:
return mf
return None
def restore_backup(
*, manifest: dict, db_url: str, images_root: Path,
) -> dict:
"""Restore from a backup manifest.
1. Replay the .sql via psql.
2. Wipe /images/ contents (except _backups/, which holds the file we're using).
3. Untar the .tar.zst into /images/.
"""
sql_path = Path(manifest["sql_path"])
tar_path = Path(manifest["tar_path"])
_run_subprocess(
["psql", "-d", _libpq_url(db_url), "-f", str(sql_path)],
)
# Wipe everything in images_root EXCEPT _backups/ (we'd delete the backup
# we're restoring from!).
for entry in images_root.iterdir():
if entry.name == _BACKUPS_DIRNAME:
continue
if entry.is_dir():
shutil.rmtree(entry)
else:
entry.unlink()
_run_subprocess(
["tar", "--zstd", "-xf", str(tar_path), "-C", str(images_root.parent)],
)
return {"restored_from": manifest["backup_id"]}
@@ -1,21 +0,0 @@
"""Restore from the most recent 'pre_migration'-tagged backup."""
from __future__ import annotations
from pathlib import Path
from . import backup as backup_mod
class NoBackupFoundError(Exception):
"""Raised when rollback() is called with no pre_migration backup on disk."""
def rollback_to_pre_migration(*, db_url: str, images_root: Path) -> dict:
manifest = backup_mod.find_latest_backup(images_root, tag="pre_migration")
if manifest is None:
raise NoBackupFoundError(
"no pre_migration-tagged backup found under <images_root>/_backups/"
)
return backup_mod.restore_backup(
manifest=manifest, db_url=db_url, images_root=images_root,
)
+57
View File
@@ -0,0 +1,57 @@
"""FC-3k: admin destructive Celery tasks.
Two long-running ops on the maintenance queue. task_run lifecycle is
captured automatically by FC-3i signals — these tasks just return
their summary dict so it lands in task_run.metadata (via Celery's
result backend) for the dashboard to surface.
Soft/hard time limits inherit the FC-3i recovery sweep: a runaway
task gets killed and flipped to status='timeout' by
recover_stalled_task_runs.
"""
from __future__ import annotations
import logging
from pathlib import Path
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..services import cleanup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
@celery.task(
name="backend.app.tasks.admin.delete_artist_cascade_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
def delete_artist_cascade_task(self, *, artist_id: int) -> dict:
"""Wraps cleanup_service.delete_artist_cascade. Returns the
service's summary dict for FC-3i task_run.metadata capture."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_artist_cascade(
session, artist_id=artist_id, images_root=IMAGES_ROOT,
)
@celery.task(
name="backend.app.tasks.admin.bulk_delete_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=900, time_limit=1200, # 15 min / 20 min
)
def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
"""Wraps cleanup_service.delete_images."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.delete_images(
session, image_ids=image_ids, images_root=IMAGES_ROOT,
)
+300
View File
@@ -0,0 +1,300 @@
"""FC-3h: backup/restore Celery tasks.
All tasks live on the maintenance queue (per celery_app.task_routes).
task_run lifecycle tracking is automatic via FC-3i signals — these
tasks just record the operator-facing artifact metadata into
BackupRun.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from pathlib import Path
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import BackupRun, ImportSettings
from ..services import backup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
"""Flip a BackupRun row from running/restoring to error with a
truncated error message and finished_at. Caller already holds the
session open."""
row.status = "error"
row.error = f"{type(exc).__name__}: {exc}"[:2000]
row.finished_at = datetime.now(UTC)
session.add(row)
session.commit()
@celery.task(
name="backend.app.tasks.backup.backup_db_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=10, retry_backoff_max=120, max_retries=2,
soft_time_limit=600, time_limit=720,
)
def backup_db_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one DB backup. Returns {'backup_run_id': N}."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="db", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_db(
db_url=cfg.database_url_sync, images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.sql_path = result["sql_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.backup_images_task",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=30, retry_backoff_max=300, max_retries=1,
soft_time_limit=21600, time_limit=23400,
)
def backup_images_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one images backup. Same shape as backup_db_task; uses
tar_path instead of sql_path."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
row = BackupRun(
kind="images", status="running", tag=tag,
triggered_by=triggered_by, started_at=now, manifest={},
)
session.add(row)
session.commit()
session.refresh(row)
run_id = row.id
try:
result = backup_service.backup_images(
images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.tar_path = result["tar_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}
@celery.task(
name="backend.app.tasks.backup.restore_db_task",
bind=True,
max_retries=0, # NEVER auto-retry a half-applied restore.
soft_time_limit=1200, time_limit=1800,
)
def restore_db_task(self, *, source_backup_run_id: int) -> dict:
"""Restore from a previous DB backup. Inserts a NEW BackupRun row
(kind='db', status='restoring') linked to the source via
restored_from_id; flips to 'restored' on success or 'error' on
failure. Operator sees the restore as a row in the dashboard."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "db" or not src.sql_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid DB backup"
)
marker = BackupRun(
kind="db", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_sql_path": src.sql_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
sql_path = src.sql_path
try:
backup_service.restore_db(
db_url=cfg.database_url_sync, sql_path=Path(sql_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.restore_images_task",
bind=True, max_retries=0,
soft_time_limit=21600, time_limit=23400,
)
def restore_images_task(self, *, source_backup_run_id: int) -> dict:
"""Mirrors restore_db_task; uses backup_service.restore_images."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
src = session.get(BackupRun, source_backup_run_id)
if src is None or src.kind != "images" or not src.tar_path:
raise ValueError(
f"BackupRun id={source_backup_run_id} is not a valid images backup"
)
marker = BackupRun(
kind="images", status="restoring",
triggered_by="restore", started_at=now,
restored_from_id=src.id,
manifest={"source_tar_path": src.tar_path},
)
session.add(marker)
session.commit()
session.refresh(marker)
marker_id = marker.id
tar_path = src.tar_path
try:
backup_service.restore_images(
images_root=IMAGES_ROOT, tar_path=Path(tar_path),
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
if row is not None:
_mark_failed(session, row, exc)
raise
with SessionLocal() as session:
row = session.get(BackupRun, marker_id)
row.status = "restored"
row.finished_at = datetime.now(UTC)
session.commit()
return {"backup_run_id": marker_id}
@celery.task(
name="backend.app.tasks.backup.prune_backups",
soft_time_limit=300, time_limit=600,
)
def prune_backups() -> dict:
"""Daily Beat. Per-kind retention from ImportSettings.
Returns {"db_deleted": N, "images_deleted": M, "files_unlinked": K}.
Tagged rows (tag IS NOT NULL) are never pruned.
Status='running' / 'restoring' rows are never pruned (recovery
sweep from FC-3i handles those via task_run).
"""
SessionLocal = _sync_session_factory()
counts = {"db_deleted": 0, "images_deleted": 0, "files_unlinked": 0}
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
for kind, keep in (
("db", s.backup_db_keep_last_n),
("images", s.backup_images_keep_last_n),
):
candidates = session.execute(
select(BackupRun)
.where(BackupRun.kind == kind)
.where(BackupRun.tag.is_(None))
.where(BackupRun.status.in_(["ok", "error"]))
.order_by(BackupRun.started_at.desc())
.offset(keep)
).scalars().all()
for row in candidates:
result = backup_service.unlink_artifact_files(
sql_path=row.sql_path,
tar_path=row.tar_path,
manifest_path=(row.manifest or {}).get("manifest_path"),
)
counts["files_unlinked"] += sum(
1 for v in result.values() if v
)
session.delete(row)
counts[f"{kind}_deleted"] += 1
session.commit()
return counts
@celery.task(
name="backend.app.tasks.backup.backup_db_nightly",
soft_time_limit=60, time_limit=120,
)
def backup_db_nightly() -> dict:
"""Hourly tick. Dispatches a real backup ONLY if the configured
UTC hour matches and the nightly setting is enabled. Returns
either {'skipped': '<reason>'} or {'dispatched': '<task_id>'}."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
s = session.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
nightly_enabled = s.backup_db_nightly_enabled
configured_hour = s.backup_db_nightly_hour_utc
if not nightly_enabled:
return {"skipped": "nightly disabled"}
now_hour = datetime.now(UTC).hour
if now_hour != configured_hour:
return {"skipped": f"hour={now_hour} != configured={configured_hour}"}
res = backup_db_task.delay(triggered_by="nightly")
return {"dispatched": res.id}
+6 -32
View File
@@ -4,7 +4,8 @@ Dispatches to the right migrator based on `kind`. Updates MigrationRun
row's status/counts/finished_at as it runs. Failures set status='error' row's status/counts/finished_at as it runs. Failures set status='error'
with the error message preserved. with the error message preserved.
kinds: backup, gs_ingest, ir_ingest, tag_apply, ml_queue, verify, rollback, cleanup kinds: gs_ingest, ir_ingest, tag_apply, ml_queue, verify, cleanup
(backup + rollback retired 2026-05-24 → see /api/system/backup/*)
""" """
from __future__ import annotations from __future__ import annotations
@@ -20,10 +21,8 @@ from ..celery_app import celery
from ..config import get_config from ..config import get_config
from ..models import MigrationRun from ..models import MigrationRun
from ..services.credential_crypto import CredentialCrypto from ..services.credential_crypto import CredentialCrypto
from ..services.migrators import backup as backup_mod
from ..services.migrators import cleanup as cleanup_mod from ..services.migrators import cleanup as cleanup_mod
from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify from ..services.migrators import gs_ingest, ir_ingest, ml_queue, tag_apply, verify
from ..services.migrators import rollback as rollback_mod
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
@@ -65,21 +64,11 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
async with factory() as db: async with factory() as db:
await _update_run(db, run_id, status="running") await _update_run(db, run_id, status="running")
try: try:
if kind == "backup": if kind in ("backup", "rollback"):
manifest = backup_mod.create_backup( raise ValueError(
db_url=get_config().database_url_sync, f"kind {kind!r} retired in FC-3h; "
images_root=IMAGES_ROOT, "use /api/system/backup/* instead"
tag=params.get("tag", "manual"),
) )
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"manifest": manifest},
)
return manifest
elif kind == "gs_ingest": elif kind == "gs_ingest":
fc_crypto = CredentialCrypto(_KEY_PATH) fc_crypto = CredentialCrypto(_KEY_PATH)
@@ -166,21 +155,6 @@ async def _run_async(run_id: int, kind: str, params: dict) -> dict:
) )
return result return result
elif kind == "rollback":
result = rollback_mod.rollback_to_pre_migration(
db_url=get_config().database_url_sync,
images_root=IMAGES_ROOT,
)
await _update_run(
db, run_id, status="ok",
counts={"rows_processed": 0, "rows_inserted": 0,
"rows_skipped": 0, "files_copied": 0,
"bytes_copied": 0, "conflicts": 0},
finished_at=datetime.now(UTC),
metadata_patch={"rollback_result": result},
)
return result
else: else:
raise ValueError(f"unknown kind: {kind}") raise ValueError(f"unknown kind: {kind}")
@@ -0,0 +1,95 @@
<template>
<v-card class="fc-danger-zone mt-8" variant="outlined">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-alert-octagon" color="error" size="small" />
<span>Danger zone</span>
</v-card-title>
<v-card-text>
<p class="text-body-2 fc-muted mb-4">
Cascade-delete this artist and every image, source, post, and
attachment associated with them. This cannot be undone.
Recoverable only from an FC-3h backup.
</p>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-forever"
:loading="loading"
@click="onClick"
>Delete artist &amp; cascade</v-btn>
<DestructiveConfirmModal
v-model="modalOpen"
action="delete"
kind="artist"
:run-id="artistId"
tier="C"
:projected-counts="projectedCounts"
:description="modalDescription"
@confirm="onConfirm"
/>
</v-card-text>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useRouter } from 'vue-router'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useAdminStore } from '../../stores/admin.js'
const props = defineProps({
slug: { type: String, required: true },
artistId: { type: Number, required: true },
artistName: { type: String, required: true },
})
const router = useRouter()
const store = useAdminStore()
const loading = ref(false)
const modalOpen = ref(false)
const projected = ref(null)
const projectedCounts = computed(() => projected.value?.projected || null)
const modalDescription = computed(
() => projected.value
? `Artist “${props.artistName}” — `
+ `${projected.value.projected.images} images, `
+ `${projected.value.projected.sources} sources, `
+ `${Math.round(projected.value.projected.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
async function onClick() {
loading.value = true
try {
projected.value = await store.projectArtistCascade(props.slug)
modalOpen.value = true
} finally {
loading.value = false
}
}
async function onConfirm(token) {
loading.value = true
try {
const result = await store.dispatchArtistCascade(props.slug, token)
const taskId = result.task_id
router.push('/artists')
if (taskId) {
store.pollTaskUntilDone(taskId).catch(() => {})
}
} finally {
loading.value = false
}
}
</script>
<style scoped>
.fc-danger-zone {
border-color: rgb(var(--v-theme-error));
border-radius: 8px;
}
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
+35 -2
View File
@@ -44,7 +44,31 @@
</div> </div>
<div class="fc-tagcard__meta"> <div class="fc-tagcard__meta">
<v-chip size="x-small" label>{{ card.kind }}</v-chip> <v-chip size="x-small" label>{{ card.kind }}</v-chip>
<span class="fc-tagcard__count">{{ card.image_count }}</span> <div class="fc-tagcard__meta-right">
<span class="fc-tagcard__count">{{ card.image_count }}</span>
<v-menu>
<template #activator="{ props: act }">
<v-btn
class="fc-tagcard__menu"
icon="mdi-dots-vertical" size="x-small" variant="text"
v-bind="act" @click.stop
/>
</template>
<v-list density="compact">
<v-list-item
title="Merge with…"
prepend-icon="mdi-call-merge"
@click="$emit('merge-with', card)"
/>
<v-list-item
title="Delete tag"
prepend-icon="mdi-delete"
base-color="error"
@click="$emit('delete', card)"
/>
</v-list>
</v-menu>
</div>
</div> </div>
</v-card-text> </v-card-text>
</v-card> </v-card>
@@ -54,7 +78,7 @@
import { ref } from 'vue' import { ref } from 'vue'
const props = defineProps({ card: { type: Object, required: true } }) const props = defineProps({ card: { type: Object, required: true } })
const emit = defineEmits(['open', 'rename', 'manage', 'read']) const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
const editing = ref(false) const editing = ref(false)
const draft = ref('') const draft = ref('')
@@ -106,4 +130,13 @@ function submit() {
} }
.fc-tagcard:hover .fc-tagcard__edit { opacity: .6; } .fc-tagcard:hover .fc-tagcard__edit { opacity: .6; }
.fc-tagcard__edit:hover { opacity: 1; } .fc-tagcard__edit:hover { opacity: 1; }
.fc-tagcard__meta-right {
display: flex; align-items: center; gap: 4px;
}
.fc-tagcard__menu {
opacity: 0;
transition: opacity .15s ease;
}
.fc-tagcard:hover .fc-tagcard__menu { opacity: .6; }
.fc-tagcard__menu:hover { opacity: 1; }
</style> </style>
@@ -55,19 +55,44 @@
</template> </template>
</div> </div>
<div class="fc-bulk-panel__section">
<h4>Destructive</h4>
<v-btn
color="error" variant="flat" rounded="pill" block
prepend-icon="mdi-delete-forever"
:disabled="!sel.count"
:loading="deleting"
@click="onDeleteClick"
>Delete {{ sel.count }} selected</v-btn>
</div>
<div class="fc-bulk-panel__foot"> <div class="fc-bulk-panel__foot">
<v-btn variant="text" block @click="sel.clear()">Clear selection</v-btn> <v-btn variant="text" block @click="sel.clear()">Clear selection</v-btn>
</div> </div>
<DestructiveConfirmModal
v-model="deleteModalOpen"
action="delete"
kind="images-selection"
:run-id="bulkToken"
tier="C"
:projected-counts="bulkProjectedCounts"
:description="bulkDescription"
@confirm="onDeleteConfirm"
/>
</aside> </aside>
</template> </template>
<script setup> <script setup>
import { ref, watch } from 'vue' import { computed, ref, watch } from 'vue'
import { useGallerySelectionStore } from '../../stores/gallerySelection.js' import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
import { useAdminStore } from '../../stores/admin.js'
import { useApi } from '../../composables/useApi.js' import { useApi } from '../../composables/useApi.js'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
const sel = useGallerySelectionStore() const sel = useGallerySelectionStore()
const api = useApi() const api = useApi()
const adminStore = useAdminStore()
const addModel = ref(null) const addModel = ref(null)
const addHits = ref([]) const addHits = ref([])
@@ -99,6 +124,66 @@ async function onAddPick(id) {
watch(() => sel.order.length, () => { watch(() => sel.order.length, () => {
if (sel.isSelectMode) sel.refresh() if (sel.isSelectMode) sel.refresh()
}) })
// --- FC-3k bulk delete -----------------------------------------------
const deleting = ref(false)
const deleteModalOpen = ref(false)
const bulkProjected = ref(null)
const bulkToken = ref('')
const bulkProjectedCounts = computed(() => bulkProjected.value
? {
images: bulkProjected.value.images_found,
thumbnails: bulkProjected.value.thumbs_to_unlink,
bytes: bulkProjected.value.bytes_on_disk,
}
: null,
)
const bulkDescription = computed(
() => bulkProjected.value
? `${bulkProjected.value.images_found} images, `
+ `${Math.round(bulkProjected.value.bytes_on_disk / 1_048_576)} MiB on disk`
: '',
)
async function _computeSha8(ids) {
const canon = [...ids].sort((a, b) => a - b).join(',')
const buf = new TextEncoder().encode(canon)
const hashBuf = await crypto.subtle.digest('SHA-256', buf)
const bytes = new Uint8Array(hashBuf)
let hex = ''
for (let i = 0; i < 4; i++) {
hex += bytes[i].toString(16).padStart(2, '0')
}
return hex
}
async function onDeleteClick() {
if (!sel.order.length) return
deleting.value = true
try {
bulkProjected.value = await adminStore.projectBulkImageDelete(sel.order)
bulkToken.value = await _computeSha8(sel.order)
deleteModalOpen.value = true
} finally {
deleting.value = false
}
}
async function onDeleteConfirm(token) {
deleting.value = true
try {
const result = await adminStore.dispatchBulkImageDelete(sel.order, token)
const taskId = result.task_id
if (taskId) {
adminStore.pollTaskUntilDone(taskId).catch(() => {})
}
sel.clear()
} finally {
deleting.value = false
}
}
</script> </script>
<style scoped> <style scoped>
@@ -0,0 +1,128 @@
<template>
<v-dialog
:model-value="modelValue"
max-width="520" persistent
@update:model-value="$emit('update:modelValue', $event)"
>
<v-card>
<v-card-title>
{{ titleVerb }}
{{ kindLabel }}<span v-if="runId"> #{{ runId }}</span>
</v-card-title>
<v-card-text>
<v-alert
:type="alertType"
variant="tonal" density="compact" class="mb-3"
>
<strong>{{ warningText }}</strong>
<div v-if="description" class="mt-1">{{ description }}</div>
</v-alert>
<div v-if="projectedCounts" class="fc-counts mb-3">
<div v-for="(v, k) in projectedCounts" :key="k">
<span class="fc-counts-key">{{ k }}:</span>
<span class="fc-counts-val">{{ v }}</span>
</div>
</div>
<template v-if="tier === 'C'">
<div class="text-body-2 mb-2">Type the following to confirm:</div>
<div class="fc-token mb-3">{{ expectedToken }}</div>
<v-text-field
v-model="typed"
variant="outlined" density="compact" hide-details
autofocus
placeholder="paste the token above"
/>
</template>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="onCancel">Cancel</v-btn>
<v-btn
:color="confirmColor"
variant="flat" rounded="pill"
:disabled="!canConfirm"
@click="onConfirm"
>
{{ titleVerb }}
</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
const props = defineProps({
modelValue: { type: Boolean, required: true },
action: { type: String, required: true }, // 'restore' | 'delete'
kind: { type: String, required: true }, // 'db' | 'images' | 'artist' | 'tag' | 'images-selection'
runId: { type: [Number, String], default: '' }, // numeric id or sha8 string
description: { type: String, default: '' },
tier: { type: String, default: 'C' }, // 'B' | 'C'
projectedCounts: { type: Object, default: null },
})
const emit = defineEmits(['update:modelValue', 'confirm'])
const typed = ref('')
const expectedToken = computed(
() => `${props.action}-${props.kind}-${props.runId}`,
)
const titleVerb = computed(
() => props.action === 'restore' ? 'Restore' : 'Delete',
)
const kindLabel = computed(() => ({
db: 'database backup',
images: 'images backup',
artist: 'artist',
tag: 'tag',
'images-selection': 'image selection',
}[props.kind] || props.kind))
const alertType = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const confirmColor = computed(
() => props.action === 'restore' ? 'warning' : 'error',
)
const warningText = computed(() => (
props.action === 'restore'
? 'This replaces current state with the backup. There is no undo.'
: 'This permanently deletes the listed items. Cannot be recovered.'
))
const canConfirm = computed(
() => props.tier === 'B'
? true
: typed.value === expectedToken.value,
)
watch(() => props.modelValue, (open) => {
if (open) typed.value = ''
})
function onCancel() {
emit('update:modelValue', false)
}
function onConfirm() {
emit('confirm', expectedToken.value)
emit('update:modelValue', false)
}
</script>
<style scoped>
.fc-token {
font-family: 'JetBrains Mono', monospace;
background: rgb(var(--v-theme-surface-light));
padding: 6px 10px; border-radius: 4px;
font-size: 14px; word-break: break-all;
}
.fc-counts {
display: grid; grid-template-columns: max-content auto;
gap: 4px 12px;
font-size: 13px;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-counts-key { font-weight: 500; text-transform: capitalize; }
.fc-counts-val { font-variant-numeric: tabular-nums; }
</style>
@@ -0,0 +1,205 @@
<template>
<v-card class="fc-backup-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-database-export" size="small" />
<span>Backups</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-4">
pg_dump for the database (fast, nightly-schedulable);
tar+zstd for the images (slow, manual only). Files live in
<code>/images/_backups/</code>. Tag a backup to protect it
from autoprune.
</p>
<v-alert
v-if="store.lastError"
type="warning" variant="tonal" density="compact" class="mb-3"
>
{{ store.lastError }}
</v-alert>
<!-- Database section -->
<h3 class="fc-section-title">Database</h3>
<div class="fc-settings-row mb-3">
<v-switch
:model-value="settings.backup_db_nightly_enabled"
color="accent" density="compact" hide-details
label="Nightly"
@update:model-value="onSettingChange('backup_db_nightly_enabled', $event)"
/>
<v-text-field
:model-value="settings.backup_db_nightly_hour_utc"
label="Hour (UTC)" type="number" min="0" max="23"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_db_nightly_hour_utc', Number($event))"
/>
<v-text-field
:model-value="settings.backup_db_keep_last_n"
label="Keep last" type="number" min="1" max="365"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_db_keep_last_n', Number($event))"
/>
</div>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-database-arrow-up" class="mb-3"
:loading="dbTriggering"
@click="onTrigger('db')"
>Run DB backup now</v-btn>
<BackupRunsTable
:runs="store.dbRuns"
@restore="onRestore"
@delete="onDelete"
@tag="onTag"
/>
<v-divider class="my-4" />
<!-- Images section -->
<h3 class="fc-section-title">Images</h3>
<div class="fc-settings-row mb-3">
<v-text-field
:model-value="settings.backup_images_keep_last_n"
label="Keep last" type="number" min="1" max="100"
density="compact" hide-details style="max-width: 110px;"
@update:model-value="onSettingChange('backup_images_keep_last_n', Number($event))"
/>
</div>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-folder-zip" class="mb-3"
:loading="imagesTriggering"
@click="onTrigger('images')"
>Run images backup now</v-btn>
<span class="text-caption fc-muted ml-2">
Hours for a 200GB+ library.
</span>
<BackupRunsTable
:runs="store.imagesRuns"
@restore="onRestore"
@delete="onDelete"
@tag="onTag"
/>
</v-card-text>
<BackupConfirmModal
v-model="confirmOpen"
:action="confirmAction"
:kind="confirmKind"
:run-id="confirmRunId"
:description="confirmDescription"
@confirm="onConfirmSubmit"
/>
</v-card>
</template>
<script setup>
import { computed, onMounted, onUnmounted, ref } from 'vue'
import { useBackupStore } from '../../stores/backup.js'
import BackupConfirmModal from '../modal/DestructiveConfirmModal.vue'
import BackupRunsTable from './BackupRunsTable.vue'
const store = useBackupStore()
const settings = computed(() => store.settings || {
backup_db_nightly_enabled: false,
backup_db_nightly_hour_utc: 3,
backup_db_keep_last_n: 14,
backup_images_keep_last_n: 3,
})
const dbTriggering = ref(false)
const imagesTriggering = ref(false)
const confirmOpen = ref(false)
const confirmAction = ref('restore')
const confirmKind = ref('db')
const confirmRunId = ref(0)
const confirmDescription = ref('')
let pollId = null
function pollOnce() {
if (document.hidden) return
store.loadRuns('db')
store.loadRuns('images')
}
onMounted(() => {
store.loadSettings()
pollOnce()
pollId = setInterval(pollOnce, 5000)
})
onUnmounted(() => {
if (pollId) { clearInterval(pollId); pollId = null }
})
async function onTrigger(kind) {
const triggeringRef = kind === 'db' ? dbTriggering : imagesTriggering
triggeringRef.value = true
try {
await store.triggerBackup(kind)
pollOnce()
} finally {
triggeringRef.value = false
}
}
async function onSettingChange(field, value) {
await store.patchSettings({ [field]: value })
}
function _openConfirm(action, run) {
confirmAction.value = action
confirmKind.value = run.kind
confirmRunId.value = run.id
confirmDescription.value = (
action === 'restore'
? `Source artifact: ${run.kind === 'db' ? run.sql_path : run.tar_path}`
: 'Removes the BackupRun row and unlinks the artifact files.'
)
confirmOpen.value = true
}
function onRestore(run) { _openConfirm('restore', run) }
function onDelete(run) { _openConfirm('delete', run) }
async function onTag(run) {
const next = prompt(
`Tag for backup #${run.id} (blank to clear):`,
run.tag || '',
)
if (next === null) return
await store.setTag(run.id, next.trim() || null)
pollOnce()
}
async function onConfirmSubmit(token) {
try {
if (confirmAction.value === 'restore') {
await store.restore(confirmRunId.value, token)
} else {
await store.deleteRun(confirmRunId.value, token)
}
pollOnce()
} catch {
// store.lastError already updated; surfaced in v-alert above.
}
}
</script>
<style scoped>
.fc-backup-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-title {
font-family: 'Fraunces', Georgia, serif;
font-size: 16px; font-weight: 500;
color: rgb(var(--v-theme-on-surface));
margin-bottom: 8px;
}
.fc-settings-row {
display: flex; gap: 12px; align-items: center; flex-wrap: wrap;
}
</style>
@@ -0,0 +1,109 @@
<template>
<v-table density="compact" class="fc-backup-table">
<thead>
<tr>
<th>When</th>
<th>Status</th>
<th class="text-right">Duration</th>
<th class="text-right">Size</th>
<th>Tag</th>
<th class="text-right">Actions</th>
</tr>
</thead>
<tbody>
<tr v-for="r in runs" :key="r.id">
<td class="fc-tabular" :title="r.started_at">
{{ formatRelative(r.started_at) }}
</td>
<td>
<v-icon size="small" :color="statusColor(r.status)">
{{ statusIcon(r.status) }}
</v-icon>
{{ r.status }}
</td>
<td class="text-right fc-tabular">{{ formatDuration(r) }}</td>
<td class="text-right fc-tabular">{{ formatBytes(r.size_bytes) }}</td>
<td>
<v-chip
v-if="r.tag" size="x-small" color="accent" variant="tonal"
>{{ r.tag }}</v-chip>
<span v-else class="fc-muted"></span>
</td>
<td class="text-right">
<v-menu>
<template #activator="{ props: act }">
<v-btn icon="mdi-dots-vertical" size="x-small" variant="text" v-bind="act" />
</template>
<v-list density="compact">
<v-list-item
:title="r.tag ? 'Untag' : 'Tag…'"
@click="$emit('tag', r)"
/>
<v-list-item
title="Restore…" :disabled="r.status !== 'ok'"
@click="$emit('restore', r)"
/>
<v-list-item
title="Delete…"
@click="$emit('delete', r)"
/>
</v-list>
</v-menu>
</td>
</tr>
<tr v-if="!runs.length">
<td colspan="6" class="text-center fc-muted py-4">No backups yet.</td>
</tr>
</tbody>
</v-table>
</template>
<script setup>
defineProps({ runs: { type: Array, default: () => [] } })
defineEmits(['restore', 'delete', 'tag'])
function statusIcon(s) {
return {
ok: 'mdi-check-circle', error: 'mdi-close-circle',
running: 'mdi-timer-sand', restoring: 'mdi-restore-clock',
restored: 'mdi-restore', pending: 'mdi-clock-outline',
}[s] || 'mdi-help-circle'
}
function statusColor(s) {
return {
ok: 'success', error: 'error',
running: 'accent', restoring: 'info',
restored: 'info', pending: 'on-surface-variant',
}[s] || 'on-surface-variant'
}
function formatDuration(r) {
if (r.duration_seconds == null) return '—'
const s = r.duration_seconds
if (s < 60) return `${s}s`
if (s < 3600) return `${Math.floor(s / 60)}m ${s % 60}s`
const h = Math.floor(s / 3600); const m = Math.floor((s % 3600) / 60)
return `${h}h ${m}m`
}
function formatBytes(b) {
if (b == null) return '—'
const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB']
let i = 0; let v = b
while (v >= 1024 && i < units.length - 1) { v /= 1024; i++ }
return `${v.toFixed(i === 0 ? 0 : 1)} ${units[i]}`
}
function formatRelative(iso) {
if (!iso) return '—'
const then = new Date(iso).getTime()
const diff = Math.max(0, (Date.now() - then) / 1000)
if (diff < 60) return `${Math.floor(diff)}s ago`
if (diff < 3600) return `${Math.floor(diff / 60)}m ago`
if (diff < 86400) return `${Math.floor(diff / 3600)}h ago`
return `${Math.floor(diff / 86400)}d ago`
}
</script>
<style scoped>
.fc-backup-table { background: transparent; }
.fc-tabular { font-variant-numeric: tabular-nums; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -12,6 +12,8 @@
<MLThresholdSliders class="mt-4" /> <MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" /> <AllowlistTable class="mt-4" />
<AliasTable class="mt-4" /> <AliasTable class="mt-4" />
<BackupCard class="mt-6" />
<TagMaintenanceCard class="mt-6" />
<BrowserExtensionCard class="mt-6" /> <BrowserExtensionCard class="mt-6" />
<LegacyMigrationCard class="mt-6" /> <LegacyMigrationCard class="mt-6" />
</div> </div>
@@ -23,6 +25,8 @@ import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue' import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue' import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue' import AliasTable from './AliasTable.vue'
import BackupCard from './BackupCard.vue'
import TagMaintenanceCard from './TagMaintenanceCard.vue'
import BrowserExtensionCard from './BrowserExtensionCard.vue' import BrowserExtensionCard from './BrowserExtensionCard.vue'
import LegacyMigrationCard from './LegacyMigrationCard.vue' import LegacyMigrationCard from './LegacyMigrationCard.vue'
</script> </script>
@@ -0,0 +1,95 @@
<template>
<v-card class="fc-tag-maint">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-tag-remove" size="small" />
<span>Tag maintenance</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-4">
Remove tags with zero image associations and zero series-page
references. Auto-created tag rows that never got applied get
swept here.
</p>
<v-alert
v-if="store.lastError"
type="warning" variant="tonal" density="compact" class="mb-3"
>{{ store.lastError }}</v-alert>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="loadingPreview"
class="mb-3"
@click="onPreview"
>Preview unused tags</v-btn>
<div v-if="preview">
<p class="text-body-2 mb-2">
<strong>{{ preview.count }}</strong> unused tag(s).
<span v-if="preview.count > 50" class="fc-muted">
Showing first 50 names.
</span>
</p>
<div v-if="preview.sample_names?.length" class="fc-name-grid mb-3">
<span v-for="n in preview.sample_names" :key="n" class="fc-name">
{{ n }}
</span>
</div>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-sweep"
:disabled="!preview.count"
:loading="committing"
@click="onCommit"
>Delete {{ preview.count }} unused tag(s)</v-btn>
</div>
</v-card-text>
</v-card>
</template>
<script setup>
import { ref } from 'vue'
import { useAdminStore } from '../../stores/admin.js'
const store = useAdminStore()
const preview = ref(null)
const loadingPreview = ref(false)
const committing = ref(false)
async function onPreview() {
loadingPreview.value = true
try {
preview.value = await store.pruneUnusedTags({ dryRun: true })
} finally {
loadingPreview.value = false
}
}
async function onCommit() {
committing.value = true
try {
const result = await store.pruneUnusedTags({ dryRun: false })
preview.value = { count: 0, sample_names: result.sample_names || [] }
} finally {
committing.value = false
}
}
</script>
<style scoped>
.fc-tag-maint { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-name-grid {
display: flex; flex-wrap: wrap; gap: 4px 8px;
max-height: 200px; overflow-y: auto;
padding: 8px; border-radius: 4px;
background: rgb(var(--v-theme-surface-light));
}
.fc-name {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: rgb(var(--v-theme-on-surface-variant));
}
</style>
+153
View File
@@ -0,0 +1,153 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useAdminStore = defineStore('admin', () => {
const api = useApi()
const lastError = ref(null)
// --- Tier-C: artist cascade ---------------------------------------
async function projectArtistCascade(slug) {
lastError.value = null
try {
return await api.post(
`/api/admin/artists/${encodeURIComponent(slug)}/cascade-delete`,
{ body: { dry_run: true } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function dispatchArtistCascade(slug, confirm) {
lastError.value = null
try {
return await api.post(
`/api/admin/artists/${encodeURIComponent(slug)}/cascade-delete`,
{ body: { dry_run: false, confirm } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-C: bulk image delete ------------------------------------
async function projectBulkImageDelete(imageIds) {
lastError.value = null
try {
return await api.post(
'/api/admin/images/bulk-delete',
{ body: { image_ids: imageIds, dry_run: true } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function dispatchBulkImageDelete(imageIds, confirm) {
lastError.value = null
try {
return await api.post(
'/api/admin/images/bulk-delete',
{ body: { image_ids: imageIds, dry_run: false, confirm } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-B: tag delete + merge -----------------------------------
async function deleteTag(tagId) {
lastError.value = null
try {
return await api.delete(`/api/admin/tags/${tagId}`)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function mergeTags(destId, sourceId) {
lastError.value = null
try {
return await api.post(
`/api/admin/tags/${destId}/merge`,
{ body: { source_id: sourceId } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
async function tagUsageCount(tagId) {
lastError.value = null
try {
const body = await api.get(`/api/admin/tags/${tagId}/usage-count`)
return body.count
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Tier-A: prune unused -----------------------------------------
async function pruneUnusedTags({ dryRun = true } = {}) {
lastError.value = null
try {
return await api.post(
'/api/admin/tags/prune-unused',
{ body: { dry_run: dryRun } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Task progress polling (taps FC-3i activity dashboard) --------
/**
* Polls /api/system/activity/runs?queue=maintenance every 3s,
* resolves when a task_run row with the given celery task_id
* reaches a terminal status (ok / error / timeout). Returns the
* row. Times out after 30 min by default.
*/
async function pollTaskUntilDone(taskId, { timeoutMs = 1_800_000 } = {}) {
const deadline = Date.now() + timeoutMs
while (Date.now() < deadline) {
const body = await api.get(
'/api/system/activity/runs',
{ params: { queue: 'maintenance', limit: 20 } },
)
const row = (body.runs || []).find(r => r.celery_task_id === taskId)
if (row && ['ok', 'error', 'timeout'].includes(row.status)) {
return row
}
await new Promise((r) => setTimeout(r, 3000))
}
throw new Error(`task ${taskId} did not finish within ${timeoutMs}ms`)
}
return {
lastError,
projectArtistCascade,
dispatchArtistCascade,
projectBulkImageDelete,
dispatchBulkImageDelete,
deleteTag,
mergeTags,
tagUsageCount,
pruneUnusedTags,
pollTaskUntilDone,
}
})
+109
View File
@@ -0,0 +1,109 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useBackupStore = defineStore('backup', () => {
const api = useApi()
const dbRuns = ref([])
const imagesRuns = ref([])
const settings = ref(null)
const loading = ref({ dbRuns: false, imagesRuns: false, settings: false })
const lastError = ref(null)
async function loadRuns(kind) {
const targetRef = kind === 'db' ? dbRuns : imagesRuns
const loadingKey = kind === 'db' ? 'dbRuns' : 'imagesRuns'
loading.value[loadingKey] = true
lastError.value = null
try {
const body = await api.get('/api/system/backup/runs', {
params: { kind, limit: 50 },
})
targetRef.value = body.runs || []
} catch (e) {
lastError.value = e.message
} finally {
loading.value[loadingKey] = false
}
}
async function triggerBackup(kind, tag = null) {
lastError.value = null
try {
await api.post(`/api/system/backup/${kind}`, {
body: tag ? { tag } : {},
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function restore(runId, confirm) {
lastError.value = null
try {
await api.post(`/api/system/backup/runs/${runId}/restore`, {
body: { confirm },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function deleteRun(runId, confirm) {
lastError.value = null
try {
await api.delete(`/api/system/backup/runs/${runId}`, {
body: { confirm },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function setTag(runId, tag) {
lastError.value = null
try {
await api.patch(`/api/system/backup/runs/${runId}`, {
body: { tag },
})
} catch (e) {
lastError.value = e.message
throw e
}
}
async function loadSettings() {
loading.value.settings = true
lastError.value = null
try {
settings.value = await api.get('/api/system/backup/settings')
} catch (e) {
lastError.value = e.message
} finally {
loading.value.settings = false
}
}
async function patchSettings(patch) {
lastError.value = null
try {
settings.value = await api.patch('/api/system/backup/settings', {
body: patch,
})
} catch (e) {
lastError.value = e.message
throw e
}
}
return {
dbRuns, imagesRuns, settings, loading, lastError,
loadRuns, triggerBackup, restore, deleteRun, setTag,
loadSettings, patchSettings,
}
})
+7
View File
@@ -96,6 +96,12 @@
@open="openImage" @open="openImage"
/> />
</section> </section>
<ArtistDangerZone
:slug="slug"
:artist-id="store.overview.id"
:artist-name="store.overview.name"
/>
</template> </template>
</v-container> </v-container>
</template> </template>
@@ -106,6 +112,7 @@ import { useRoute, useRouter, RouterLink } from 'vue-router'
import { useArtistStore } from '../stores/artist.js' import { useArtistStore } from '../stores/artist.js'
import { useModalStore } from '../stores/modal.js' import { useModalStore } from '../stores/modal.js'
import MasonryGrid from '../components/discovery/MasonryGrid.vue' import MasonryGrid from '../components/discovery/MasonryGrid.vue'
import ArtistDangerZone from '../components/artist/ArtistDangerZone.vue'
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
+121
View File
@@ -27,6 +27,7 @@
<TagCard <TagCard
v-for="c in store.cards" :key="c.id" :card="c" v-for="c in store.cards" :key="c.id" :card="c"
@open="openTag" @rename="onRename" @manage="onManage" @read="onRead" @open="openTag" @rename="onRename" @manage="onManage" @read="onRead"
@merge-with="onMergeWith" @delete="onDeleteTag"
/> />
</div> </div>
@@ -41,6 +42,49 @@
@confirm="confirmMerge" @confirm="confirmMerge"
@cancel="pendingMerge = null" @cancel="pendingMerge = null"
/> />
<v-dialog v-model="mergePickerOpen" max-width="520">
<v-card>
<v-card-title>Merge {{ mergeSource?.name }} into</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3" style="opacity: 0.7;">
Pick the target tag. Source tag will be deleted; all its
image associations will move to the target. Must be same
kind ({{ mergeSource?.kind }}).
</p>
<v-autocomplete
v-model="mergeTargetId"
:items="mergeHits" item-title="name" item-value="id"
:loading="mergeLoading"
density="compact" variant="outlined" hide-details
placeholder="Search target tag…" no-filter
@update:search="onMergeSearch"
/>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn variant="text" @click="mergePickerOpen = false">Cancel</v-btn>
<v-btn
color="warning" variant="flat" rounded="pill"
:disabled="!mergeTargetId"
@click="onMergeConfirm"
>Merge</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<DestructiveConfirmModal
v-model="deleteTagModalOpen"
action="delete"
kind="tag"
:run-id="deleteTagTarget?.id || 0"
tier="B"
:projected-counts="{ associations: deleteTagUsage }"
:description="deleteTagTarget
? `Delete tag “${deleteTagTarget.name}” and remove all image associations.`
: ''"
@confirm="onDeleteTagConfirm"
/>
</v-container> </v-container>
</template> </template>
@@ -48,8 +92,11 @@
import { ref, watch, onMounted, onUnmounted } from 'vue' import { ref, watch, onMounted, onUnmounted } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { useTagDirectoryStore } from '../stores/tagDirectory.js' import { useTagDirectoryStore } from '../stores/tagDirectory.js'
import { useAdminStore } from '../stores/admin.js'
import { useApi } from '../composables/useApi.js'
import TagCard from '../components/discovery/TagCard.vue' import TagCard from '../components/discovery/TagCard.vue'
import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue' import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue'
import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue'
// Must stay a subset of the backend TagKind enum (character, fandom, // Must stay a subset of the backend TagKind enum (character, fandom,
// general, series, archive, post, meta, rating). 'fandom' is this // general, series, archive, post, meta, rating). 'fandom' is this
@@ -106,6 +153,80 @@ function onManage(id) {
function onRead(id) { function onRead(id) {
router.push({ name: 'series-read', params: { tagId: id } }) router.push({ name: 'series-read', params: { tagId: id } })
} }
// --- FC-3k tag merge + delete ----------------------------------------
const adminStore = useAdminStore()
const api = useApi()
// Tag merge via dots-menu (separate from inline-rename collision flow above)
const mergePickerOpen = ref(false)
const mergeSource = ref(null)
const mergeTargetId = ref(null)
const mergeHits = ref([])
const mergeLoading = ref(false)
let mergeDebounce = null
function onMergeWith(card) {
mergeSource.value = card
mergeTargetId.value = null
mergeHits.value = []
mergePickerOpen.value = true
}
function onMergeSearch(q) {
if (mergeDebounce) clearTimeout(mergeDebounce)
if (!q) { mergeHits.value = []; return }
mergeDebounce = setTimeout(async () => {
mergeLoading.value = true
try {
const hits = await api.get('/api/tags/autocomplete', {
params: { q, kind: mergeSource.value?.kind, limit: 20 },
})
mergeHits.value = (hits || []).filter(
(h) => h.id !== mergeSource.value?.id,
)
} finally {
mergeLoading.value = false
}
}, 250)
}
async function onMergeConfirm() {
if (!mergeSource.value || !mergeTargetId.value) return
try {
await adminStore.mergeTags(mergeTargetId.value, mergeSource.value.id)
mergePickerOpen.value = false
store.reset()
} catch {
// adminStore.lastError already set; UI surfaces it elsewhere.
}
}
// Tag delete via dots-menu (Tier B)
const deleteTagModalOpen = ref(false)
const deleteTagTarget = ref(null)
const deleteTagUsage = ref(0)
async function onDeleteTag(card) {
deleteTagTarget.value = card
try {
deleteTagUsage.value = await adminStore.tagUsageCount(card.id)
} catch {
deleteTagUsage.value = 0
}
deleteTagModalOpen.value = true
}
async function onDeleteTagConfirm() {
if (!deleteTagTarget.value) return
try {
await adminStore.deleteTag(deleteTagTarget.value.id)
store.reset()
} catch {
// adminStore.lastError already set.
}
}
</script> </script>
<style scoped> <style scoped>
+361
View File
@@ -0,0 +1,361 @@
"""FC-3k: /api/admin/* endpoint integration tests.
Monkeypatch `.delay()` for Tier-C tasks to capture dispatch payloads
without actually queuing. Tier-B/A endpoints run synchronously
through the real service.
"""
import hashlib
import pytest
from backend.app import create_app
from backend.app.models import Artist, ImageRecord, Tag, TagKind
from backend.app.models.tag import image_tag
pytestmark = pytest.mark.integration
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
# --- Tier-C: POST /artists/<slug>/cascade-delete --------------------
@pytest.mark.asyncio
async def test_artist_cascade_dry_run_returns_projection(client, db):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={"dry_run": True},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["artist"]["slug"] == "aria"
assert "projected" in body
assert body["projected"]["images"] == 0
@pytest.mark.asyncio
async def test_artist_cascade_unknown_slug_404(client):
resp = await client.post(
"/api/admin/artists/no-such/cascade-delete",
json={"dry_run": True},
)
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_artist_cascade_wrong_confirm_400(client, db):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
artist_id = a.id
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={"dry_run": False, "confirm": "wrong"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"] == f"delete-artist-{artist_id}"
@pytest.mark.asyncio
async def test_artist_cascade_correct_confirm_dispatches(
client, db, monkeypatch,
):
a = Artist(name="Aria", slug="aria")
db.add(a)
await db.commit()
artist_id = a.id
dispatched = []
class _Result:
id = "fake-task-id"
def _delay(**kw):
dispatched.append(kw)
return _Result()
monkeypatch.setattr(
"backend.app.tasks.admin.delete_artist_cascade_task.delay", _delay,
)
resp = await client.post(
"/api/admin/artists/aria/cascade-delete",
json={
"dry_run": False,
"confirm": f"delete-artist-{artist_id}",
},
)
assert resp.status_code == 202
body = await resp.get_json()
assert body["task_id"] == "fake-task-id"
assert dispatched == [{"artist_id": artist_id}]
# --- Tier-C: POST /images/bulk-delete -------------------------------
@pytest.mark.asyncio
async def test_bulk_delete_dry_run_returns_counts(client, db, tmp_path):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
i1 = ImageRecord(
artist_id=a.id, path=str(tmp_path / "1.jpg"),
sha256="1" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
i2 = ImageRecord(
artist_id=a.id, path=str(tmp_path / "2.jpg"),
sha256="2" * 64, size_bytes=20, mime="image/jpeg",
origin="imported_filesystem",
)
db.add_all([i1, i2])
await db.commit()
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [i1.id, i2.id, 9_999_999],
"dry_run": True,
},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["images_found"] == 2
assert body["bytes_on_disk"] == 30
assert body["missing_ids"] == [9_999_999]
@pytest.mark.asyncio
async def test_bulk_delete_empty_list_400(client):
resp = await client.post(
"/api/admin/images/bulk-delete",
json={"image_ids": [], "dry_run": True},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_image_ids"
@pytest.mark.asyncio
async def test_bulk_delete_non_int_400(client):
resp = await client.post(
"/api/admin/images/bulk-delete",
json={"image_ids": ["foo", 2], "dry_run": True},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_bulk_delete_wrong_confirm_400_with_expected_token(
client, db, tmp_path,
):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "x.jpg"),
sha256="c" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
await db.commit()
img_id = img.id
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [img_id], "dry_run": False, "confirm": "nope",
},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"].startswith("delete-images-")
@pytest.mark.asyncio
async def test_bulk_delete_correct_confirm_dispatches(
client, db, tmp_path, monkeypatch,
):
a = Artist(name="B", slug="b")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "x.jpg"),
sha256="d" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
await db.commit()
img_id = img.id
sha8 = hashlib.sha256(str(img_id).encode("utf-8")).hexdigest()[:8]
dispatched = []
class _Result:
id = "fake-bulk-id"
def _delay(**kw):
dispatched.append(kw)
return _Result()
monkeypatch.setattr(
"backend.app.tasks.admin.bulk_delete_images_task.delay", _delay,
)
resp = await client.post(
"/api/admin/images/bulk-delete",
json={
"image_ids": [img_id],
"dry_run": False,
"confirm": f"delete-images-{sha8}",
},
)
assert resp.status_code == 202
body = await resp.get_json()
assert body["task_id"] == "fake-bulk-id"
assert dispatched == [{"image_ids": [img_id]}]
# --- Tier-B: DELETE /tags/<id> + POST /tags/<dest>/merge -----------
@pytest.mark.asyncio
async def test_tag_delete_200_after_cascade(client, db):
t = Tag(name="doomed-tag", kind=TagKind.general)
db.add(t)
await db.commit()
tag_id = t.id
resp = await client.delete(f"/api/admin/tags/{tag_id}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["deleted"]["id"] == tag_id
assert body["associations_removed"] == 0
@pytest.mark.asyncio
async def test_tag_delete_404_unknown(client):
resp = await client.delete("/api/admin/tags/9999999")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_tag_merge_succeeds_for_same_kind(client, db):
src = Tag(name="src-tag", kind=TagKind.general)
dest = Tag(name="dest-tag", kind=TagKind.general)
db.add_all([src, dest])
await db.commit()
src_id = src.id
dest_id = dest.id
resp = await client.post(
f"/api/admin/tags/{dest_id}/merge",
json={"source_id": src_id},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["result"]["target_id"] == dest_id
@pytest.mark.asyncio
async def test_tag_merge_rejects_self_merge(client, db):
t = Tag(name="x", kind=TagKind.general)
db.add(t)
await db.commit()
resp = await client.post(
f"/api/admin/tags/{t.id}/merge",
json={"source_id": t.id},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_tag_merge_returns_400_on_kind_mismatch(client, db):
src = Tag(name="src", kind=TagKind.general)
dest = Tag(name="dest", kind=TagKind.artist)
db.add_all([src, dest])
await db.commit()
resp = await client.post(
f"/api/admin/tags/{dest.id}/merge",
json={"source_id": src.id},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "tag_kind_mismatch"
# --- Tier-B helper: GET /tags/<id>/usage-count ----------------------
@pytest.mark.asyncio
async def test_tag_usage_count_returns_zero_for_unused(client, db):
t = Tag(name="lonely", kind=TagKind.general)
db.add(t)
await db.commit()
resp = await client.get(f"/api/admin/tags/{t.id}/usage-count")
assert resp.status_code == 200
body = await resp.get_json()
assert body["count"] == 0
# --- Tier-A: POST /tags/prune-unused --------------------------------
@pytest.mark.asyncio
async def test_prune_unused_dry_run_lists_unused(client, db, tmp_path):
a = Artist(name="P", slug="p")
db.add(a)
await db.flush()
img = ImageRecord(
artist_id=a.id, path=str(tmp_path / "p.jpg"),
sha256="e" * 64, size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db.add(img)
used = Tag(name="kept", kind=TagKind.general)
unused = Tag(name="prune", kind=TagKind.general)
db.add_all([used, unused])
await db.flush()
await db.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
await db.commit()
resp = await client.post(
"/api/admin/tags/prune-unused", json={"dry_run": True},
)
body = await resp.get_json()
assert body["count"] == 1
assert "prune" in body["sample_names"]
@pytest.mark.asyncio
async def test_prune_unused_commit_deletes_and_returns_count(client, db):
db.add(Tag(name="byebye", kind=TagKind.general))
await db.commit()
resp = await client.post(
"/api/admin/tags/prune-unused", json={"dry_run": False},
)
body = await resp.get_json()
assert body["deleted"] >= 1
assert "byebye" in body["sample_names"]
+7 -19
View File
@@ -31,17 +31,10 @@ async def client(app):
yield c yield c
@pytest.mark.asyncio # Retired 2026-05-24 (FC-3h): `test_post_backup_returns_202_and_id`
async def test_post_backup_returns_202_and_id(client, monkeypatch): # asserted the /api/migrate/backup endpoint, which was retired in FC-3h.
monkeypatch.setattr( # Backup is now a first-class feature at /api/system/backup/*;
"backend.app.api.migrate.run_migration", # coverage lives in tests/test_api_system_backup.py.
type("F", (), {"delay": lambda self, *a, **k: None})(),
)
resp = await client.post("/api/migrate/backup", json={})
assert resp.status_code == 202
body = await resp.get_json()
assert "run_id" in body
assert body["status"] == "pending"
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -62,10 +55,6 @@ async def test_post_ingest_rejects_missing_file(client):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_post_ingest_accepts_multipart_file(client, monkeypatch): async def test_post_ingest_accepts_multipart_file(client, monkeypatch):
monkeypatch.setattr(
"backend.app.api.migrate._has_recent_pre_migration_backup",
lambda: True, # pretend backup exists
)
monkeypatch.setattr( monkeypatch.setattr(
"backend.app.api.migrate.run_migration", "backend.app.api.migrate.run_migration",
type("F", (), {"delay": lambda self, *a, **k: None})(), type("F", (), {"delay": lambda self, *a, **k: None})(),
@@ -92,10 +81,9 @@ async def test_post_ingest_accepts_multipart_file(client, monkeypatch):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_post_dry_run_allowed_without_backup(client, monkeypatch): async def test_post_dry_run_allowed_without_backup(client, monkeypatch):
monkeypatch.setattr( # FC-3h: the _has_recent_pre_migration_backup gate was retired; dry-run
"backend.app.api.migrate._has_recent_pre_migration_backup", # ingests were always allowed and now non-dry-run ingests are too.
lambda: False, # This test stays as regression coverage that dry_run ingest still works.
)
monkeypatch.setattr( monkeypatch.setattr(
"backend.app.api.migrate.run_migration", "backend.app.api.migrate.run_migration",
type("F", (), {"delay": lambda self, *a, **k: None})(), type("F", (), {"delay": lambda self, *a, **k: None})(),
+336
View File
@@ -0,0 +1,336 @@
"""FC-3h: /api/system/backup/* endpoint integration tests."""
from datetime import UTC, datetime, timedelta
import pytest
import pytest_asyncio
from sqlalchemy import select
from backend.app import create_app
from backend.app.models import BackupRun
pytestmark = pytest.mark.integration
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
@pytest_asyncio.fixture
async def _seed_runs(db):
"""Insert 4 BackupRun rows for paging/filter tests."""
now = datetime.now(UTC)
for i in range(4):
db.add(BackupRun(
kind="db" if i % 2 == 0 else "images",
status="ok",
tag=None,
triggered_by="manual",
started_at=now - timedelta(seconds=10 - i),
finished_at=now - timedelta(seconds=9 - i),
sql_path="/tmp/fake.sql" if i % 2 == 0 else None,
tar_path=None if i % 2 == 0 else "/tmp/fake.tar.zst",
size_bytes=100,
manifest={},
))
await db.commit()
# --- POST /db, /images ----------------------------------------------
@pytest.mark.asyncio
async def test_post_db_dispatches_and_returns_202(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post("/api/system/backup/db", json={})
assert resp.status_code == 202
body = await resp.get_json()
assert body["status"] == "dispatched"
assert dispatched == [{"tag": None, "triggered_by": "manual"}]
@pytest.mark.asyncio
async def test_post_db_persists_tag(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post(
"/api/system/backup/db", json={"tag": "pre-cutover"},
)
assert resp.status_code == 202
assert dispatched[0]["tag"] == "pre-cutover"
@pytest.mark.asyncio
async def test_post_db_rejects_tag_too_long(client):
resp = await client.post(
"/api/system/backup/db", json={"tag": "x" * 65},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_tag"
@pytest.mark.asyncio
async def test_post_images_dispatches(client, monkeypatch):
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.backup_images_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post("/api/system/backup/images", json={})
assert resp.status_code == 202
# --- GET /runs -------------------------------------------------------
@pytest.mark.asyncio
async def test_runs_paginated_descending(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?limit=2")
body = await resp.get_json()
assert len(body["runs"]) == 2
ids = [r["id"] for r in body["runs"]]
assert ids == sorted(ids, reverse=True)
assert body["next_cursor"] is not None
@pytest.mark.asyncio
async def test_runs_filter_by_kind_db(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?kind=db")
body = await resp.get_json()
assert all(r["kind"] == "db" for r in body["runs"])
@pytest.mark.asyncio
async def test_runs_filter_by_kind_images(client, _seed_runs):
resp = await client.get("/api/system/backup/runs?kind=images")
body = await resp.get_json()
assert all(r["kind"] == "images" for r in body["runs"])
@pytest.mark.asyncio
async def test_runs_rejects_invalid_kind(client):
resp = await client.get("/api/system/backup/runs?kind=bogus")
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_runs_rejects_invalid_limit(client):
resp = await client.get("/api/system/backup/runs?limit=not-int")
assert resp.status_code == 400
# --- GET /runs/<id> --------------------------------------------------
@pytest.mark.asyncio
async def test_get_run_returns_row(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.get(f"/api/system/backup/runs/{rid}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["id"] == rid
@pytest.mark.asyncio
async def test_get_run_404(client):
resp = await client.get("/api/system/backup/runs/999999")
assert resp.status_code == 404
# --- PATCH /runs/<id> ------------------------------------------------
@pytest.mark.asyncio
async def test_patch_tag_sets_value(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.patch(
f"/api/system/backup/runs/{rid}",
json={"tag": "monthly"},
)
assert resp.status_code == 200
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == rid)
).scalar_one()
assert tag == "monthly"
@pytest.mark.asyncio
async def test_patch_tag_null_clears(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
await client.patch(f"/api/system/backup/runs/{rid}", json={"tag": "x"})
await client.patch(f"/api/system/backup/runs/{rid}", json={"tag": None})
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == rid)
).scalar_one()
assert tag is None
# --- POST /runs/<id>/restore ----------------------------------------
@pytest.mark.asyncio
async def test_restore_wrong_confirm_400(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": "wrong"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
assert body["expected"] == f"restore-db-{rid}"
@pytest.mark.asyncio
async def test_restore_correct_confirm_dispatches(client, _seed_runs, monkeypatch):
list_resp = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
dispatched = []
monkeypatch.setattr(
"backend.app.tasks.backup.restore_db_task.delay",
lambda **kw: dispatched.append(kw),
)
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": f"restore-db-{rid}"},
)
assert resp.status_code == 202
assert dispatched == [{"source_backup_run_id": rid}]
@pytest.mark.asyncio
async def test_restore_rejects_non_ok_status(client, db):
# Seed an error-status row directly.
db.add(BackupRun(
kind="db", status="error", tag=None, triggered_by="manual",
started_at=datetime.now(UTC), finished_at=datetime.now(UTC),
sql_path="/tmp/fake.sql", manifest={},
))
await db.commit()
rid_q = await client.get("/api/system/backup/runs?kind=db&limit=1")
rid = (await rid_q.get_json())["runs"][0]["id"]
resp = await client.post(
f"/api/system/backup/runs/{rid}/restore",
json={"confirm": f"restore-db-{rid}"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "not_restorable"
# --- DELETE /runs/<id> ----------------------------------------------
@pytest.mark.asyncio
async def test_delete_wrong_confirm_400(client, _seed_runs):
list_resp = await client.get("/api/system/backup/runs?limit=1")
rid = (await list_resp.get_json())["runs"][0]["id"]
resp = await client.delete(
f"/api/system/backup/runs/{rid}",
json={"confirm": "wrong"},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_delete_correct_confirm_204(client, _seed_runs, db_sync):
list_resp = await client.get("/api/system/backup/runs?limit=1")
row = (await list_resp.get_json())["runs"][0]
rid = row["id"]
kind = row["kind"]
resp = await client.delete(
f"/api/system/backup/runs/{rid}",
json={"confirm": f"delete-{kind}-{rid}"},
)
assert resp.status_code == 204
surviving = db_sync.execute(
select(BackupRun.id).where(BackupRun.id == rid)
).scalar_one_or_none()
assert surviving is None
# --- /settings ------------------------------------------------------
@pytest.mark.asyncio
async def test_get_settings_returns_defaults(client):
resp = await client.get("/api/system/backup/settings")
assert resp.status_code == 200
body = await resp.get_json()
assert body["backup_db_nightly_enabled"] is False
assert body["backup_db_nightly_hour_utc"] == 3
assert body["backup_db_keep_last_n"] == 14
assert body["backup_images_keep_last_n"] == 3
@pytest.mark.asyncio
async def test_patch_settings_updates_values(client):
resp = await client.patch(
"/api/system/backup/settings",
json={
"backup_db_nightly_enabled": True,
"backup_db_nightly_hour_utc": 5,
"backup_db_keep_last_n": 30,
},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["backup_db_nightly_enabled"] is True
assert body["backup_db_nightly_hour_utc"] == 5
assert body["backup_db_keep_last_n"] == 30
@pytest.mark.asyncio
async def test_patch_settings_rejects_invalid_bool(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_nightly_enabled": "yes"},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_patch_settings_rejects_hour_out_of_range(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_nightly_hour_utc": 24},
)
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_patch_settings_rejects_keep_n_below_min(client):
resp = await client.patch(
"/api/system/backup/settings",
json={"backup_db_keep_last_n": 0},
)
assert resp.status_code == 400
+200
View File
@@ -0,0 +1,200 @@
"""FC-3h: backup_service unit tests.
Subprocess calls (pg_dump, tar, psql) are monkeypatched so tests run
without external binaries. The real subprocess behavior is exercised
implicitly via the Celery task tests in test_tasks_backup.py.
"""
import json
from pathlib import Path
import pytest
from backend.app.services import backup_service
pytestmark = pytest.mark.integration
@pytest.fixture
def fake_subprocess(monkeypatch):
"""Replace subprocess.run with a fake that writes a sentinel to
the target path (for pg_dump's -f, for tar's -cf). Captures all
calls in a list."""
calls = []
class _FakeProc:
returncode = 0
stdout = b""
stderr = b""
def _fake_run(cmd, **kwargs):
calls.append(list(cmd))
if cmd[0] == "pg_dump":
i = cmd.index("-f")
Path(cmd[i + 1]).write_bytes(b"-- fake pg_dump\n")
elif cmd[0] == "tar" and "-cf" in cmd:
i = cmd.index("-cf")
Path(cmd[i + 1]).write_bytes(b"fake tar payload")
return _FakeProc()
monkeypatch.setattr("subprocess.run", _fake_run)
return calls
# --- backup_db -------------------------------------------------------
def test_backup_db_writes_sql_and_manifest(tmp_path, fake_subprocess):
result = backup_service.backup_db(
db_url="postgresql://test@x/db",
images_root=tmp_path,
tag="pre-cutover",
triggered_by="manual",
)
sql = Path(result["sql_path"])
manifest = Path(result["manifest_path"])
assert sql.is_file() and sql.suffix == ".sql"
assert manifest.is_file() and manifest.suffix == ".json"
assert result["kind"] == "db"
assert result["tar_path"] is None
assert result["size_bytes"] == len(b"-- fake pg_dump\n")
parsed = json.loads(manifest.read_text())
assert parsed["kind"] == "db"
assert parsed["tag"] == "pre-cutover"
assert parsed["triggered_by"] == "manual"
assert parsed["artifact_path"] == str(sql)
def test_backup_db_strips_sqlalchemy_psycopg_driver(tmp_path, fake_subprocess):
backup_service.backup_db(
db_url="postgresql+psycopg://u:p@h/d", images_root=tmp_path,
)
cmd = fake_subprocess[0]
assert cmd[0] == "pg_dump"
assert cmd[-1].startswith("postgresql://")
assert "+psycopg" not in cmd[-1]
def test_backup_db_strips_asyncpg_driver(tmp_path, fake_subprocess):
backup_service.backup_db(
db_url="postgresql+asyncpg://u:p@h/d", images_root=tmp_path,
)
assert "+asyncpg" not in fake_subprocess[0][-1]
def test_backup_db_default_tag_is_none(tmp_path, fake_subprocess):
result = backup_service.backup_db(
db_url="postgresql://u@h/d", images_root=tmp_path,
)
parsed = json.loads(Path(result["manifest_path"]).read_text())
assert parsed["tag"] is None
# --- backup_images ---------------------------------------------------
def test_backup_images_writes_tar_and_manifest(tmp_path, fake_subprocess):
result = backup_service.backup_images(
images_root=tmp_path, tag="monthly", triggered_by="manual",
)
tar = Path(result["tar_path"])
assert tar.is_file() and tar.name.endswith(".tar.zst")
assert result["sql_path"] is None
assert result["size_bytes"] == len(b"fake tar payload")
def test_backup_images_excludes_backups_and_quarantine(tmp_path, fake_subprocess):
backup_service.backup_images(images_root=tmp_path)
cmd = fake_subprocess[0]
excludes = [arg for arg in cmd if arg.startswith("--exclude=")]
assert any("_backups" in e for e in excludes)
assert any("_quarantine" in e for e in excludes)
# --- restore_db ------------------------------------------------------
def test_restore_db_drops_schema_then_loads(tmp_path, fake_subprocess):
sql_path = tmp_path / "fake.sql"
sql_path.write_text("SELECT 1;")
backup_service.restore_db(
db_url="postgresql://u@h/d", sql_path=sql_path,
)
# Two psql calls: one with -c (DROP SCHEMA), one with -f (load).
assert len(fake_subprocess) == 2
assert "-c" in fake_subprocess[0]
assert "DROP SCHEMA IF EXISTS public CASCADE" in fake_subprocess[0][-1]
assert "-f" in fake_subprocess[1]
assert str(sql_path) in fake_subprocess[1]
# --- restore_images --------------------------------------------------
def test_restore_images_untar_to_parent(tmp_path, fake_subprocess):
tar_path = tmp_path / "fake.tar.zst"
tar_path.write_bytes(b"fake")
backup_service.restore_images(images_root=tmp_path, tar_path=tar_path)
cmd = fake_subprocess[0]
assert cmd[:3] == ["tar", "--zstd", "-xf"]
assert str(tar_path) in cmd
assert "-C" in cmd
# --- unlink ----------------------------------------------------------
def test_unlink_removes_present_files_and_reports(tmp_path):
sql = tmp_path / "x.sql"
sql.write_bytes(b"x")
tar = tmp_path / "x.tar.zst"
tar.write_bytes(b"x")
manifest = tmp_path / "x.json"
manifest.write_text("{}")
result = backup_service.unlink_artifact_files(
sql_path=str(sql), tar_path=str(tar), manifest_path=str(manifest),
)
assert result == {"sql": True, "tar": True, "manifest": True}
assert not sql.exists() and not tar.exists() and not manifest.exists()
def test_unlink_missing_files_returns_true(tmp_path):
"""missing_ok semantics: a non-existent file isn't an error."""
result = backup_service.unlink_artifact_files(
sql_path=str(tmp_path / "nope.sql"),
tar_path=None, manifest_path=None,
)
assert result == {"sql": True}
def test_unlink_skips_none_paths():
"""A None path is skipped — not added to the result dict."""
result = backup_service.unlink_artifact_files(
sql_path=None, tar_path=None, manifest_path=None,
)
assert result == {}
# --- helpers ---------------------------------------------------------
def test_libpq_url_strips_each_known_driver():
assert backup_service._libpq_url(
"postgresql+psycopg://u@h/d"
) == "postgresql://u@h/d"
assert backup_service._libpq_url(
"postgresql+asyncpg://u@h/d"
) == "postgresql://u@h/d"
assert backup_service._libpq_url(
"postgresql+psycopg2://u@h/d"
) == "postgresql://u@h/d"
# Plain URL passes through unchanged.
assert backup_service._libpq_url(
"postgresql://u@h/d"
) == "postgresql://u@h/d"
def test_backups_dir_created_on_first_use(tmp_path):
d = backup_service._backups_dir(tmp_path)
assert d.is_dir()
assert d.name == "_backups"
+335
View File
@@ -0,0 +1,335 @@
"""FC-3k: cleanup_service unit tests.
Mutations go against real Postgres (db_sync fixture). File-system
side effects use tmp_path. Assertions on mutated rows use COLUMN
SELECTS per reference_async_coredml_test_assertions — never
re-read ORM attributes after a service mutates and re-fetches.
"""
import pytest
from sqlalchemy import func, select
from backend.app.models import Artist, ImageRecord, Tag, TagKind
from backend.app.models.tag import image_tag
from backend.app.services import cleanup_service
pytestmark = pytest.mark.integration
def _make_image(db_sync, *, artist, path, sha256, size=1000, thumb=None):
img = ImageRecord(
artist_id=artist.id,
path=path,
sha256=sha256,
size_bytes=size,
mime="image/jpeg",
origin="imported_filesystem",
thumbnail_path=thumb,
)
db_sync.add(img)
db_sync.flush()
return img
def _make_artist(db_sync, *, slug="aria", name="Aria"):
a = Artist(name=name, slug=slug)
db_sync.add(a)
db_sync.flush()
return a
def _make_tag(db_sync, *, name, kind=TagKind.general):
t = Tag(name=name, kind=kind)
db_sync.add(t)
db_sync.flush()
return t
# --- project_artist_cascade -----------------------------------------
def test_project_artist_cascade_returns_zeroes_for_empty_artist(db_sync):
_make_artist(db_sync, slug="empty")
db_sync.commit()
result = cleanup_service.project_artist_cascade(db_sync, slug="empty")
assert result["artist"]["slug"] == "empty"
assert result["projected"] == {
"images": 0,
"sources": 0,
"thumbs": 0,
"import_tasks": 0,
"bytes_on_disk": 0,
}
def test_project_artist_cascade_counts_images_and_thumbs_and_bytes(db_sync, tmp_path):
a = _make_artist(db_sync, slug="counted")
_make_image(
db_sync, artist=a, path=str(tmp_path / "a.jpg"),
sha256="a" * 64, size=1000, thumb=str(tmp_path / "a.thumb"),
)
_make_image(
db_sync, artist=a, path=str(tmp_path / "b.jpg"),
sha256="b" * 64, size=2500, thumb=None,
)
db_sync.commit()
result = cleanup_service.project_artist_cascade(db_sync, slug="counted")
assert result["projected"]["images"] == 2
assert result["projected"]["thumbs"] == 1
assert result["projected"]["bytes_on_disk"] == 3500
def test_project_artist_cascade_raises_on_unknown_slug(db_sync):
with pytest.raises(LookupError):
cleanup_service.project_artist_cascade(db_sync, slug="nope")
# --- project_bulk_image_delete --------------------------------------
def test_project_bulk_image_delete_separates_found_from_missing(db_sync, tmp_path):
a = _make_artist(db_sync, slug="bd")
i1 = _make_image(
db_sync, artist=a, path=str(tmp_path / "1.jpg"),
sha256="1" * 64, size=10, thumb="t1",
)
i2 = _make_image(
db_sync, artist=a, path=str(tmp_path / "2.jpg"),
sha256="2" * 64, size=20, thumb=None,
)
db_sync.commit()
result = cleanup_service.project_bulk_image_delete(
db_sync, image_ids=[i1.id, i2.id, 9_999_999],
)
assert result["images_found"] == 2
assert result["thumbs_to_unlink"] == 1
assert result["bytes_on_disk"] == 30
assert result["missing_ids"] == [9_999_999]
def test_project_bulk_image_delete_empty_input(db_sync):
result = cleanup_service.project_bulk_image_delete(db_sync, image_ids=[])
assert result == {
"images_found": 0,
"thumbs_to_unlink": 0,
"bytes_on_disk": 0,
"missing_ids": [],
}
# --- count_tag_associations / find_unused_tags ----------------------
def test_count_tag_associations_counts_image_tag_rows(db_sync, tmp_path):
a = _make_artist(db_sync, slug="ta")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "x.jpg"), sha256="c" * 64,
)
tag = _make_tag(db_sync, name="cyberpunk")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=tag.id,
))
db_sync.commit()
assert cleanup_service.count_tag_associations(db_sync, tag_id=tag.id) == 1
def test_find_unused_tags_returns_only_unreferenced(db_sync, tmp_path):
a = _make_artist(db_sync, slug="fu")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "y.jpg"), sha256="d" * 64,
)
used = _make_tag(db_sync, name="used")
_make_tag(db_sync, name="aaa-unused")
_make_tag(db_sync, name="zzz-unused")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.find_unused_tags(db_sync)
names = [t.name for t in result]
assert "used" not in names
assert "aaa-unused" in names
assert "zzz-unused" in names
# Sorted by name ascending.
assert names.index("aaa-unused") < names.index("zzz-unused")
# --- unlink_image_files ---------------------------------------------
def test_unlink_image_files_removes_original_and_thumbnail(db_sync, tmp_path):
original = tmp_path / "orig.jpg"
original.write_bytes(b"x")
custom_thumb = tmp_path / "custom.thumb"
custom_thumb.write_bytes(b"x")
conv_thumb = tmp_path / "thumbs" / "aaa" / ("a" * 64 + ".jpg")
conv_thumb.parent.mkdir(parents=True)
conv_thumb.write_bytes(b"x")
img = ImageRecord(
artist_id=None, path=str(original), sha256="a" * 64,
size_bytes=1, thumbnail_path=str(custom_thumb),
)
result = cleanup_service.unlink_image_files(img, tmp_path)
assert result == {"original": True, "thumbnail": True}
assert not original.exists()
assert not custom_thumb.exists()
assert not conv_thumb.exists()
def test_unlink_image_files_missing_files_count_as_success(tmp_path):
img = ImageRecord(
artist_id=None, path=str(tmp_path / "nope.jpg"),
sha256="b" * 64, size_bytes=1, thumbnail_path=None,
)
result = cleanup_service.unlink_image_files(img, tmp_path)
assert result == {"original": True, "thumbnail": False}
# --- delete_artist_cascade ------------------------------------------
def test_delete_artist_cascade_removes_images_and_artist_row(db_sync, tmp_path):
a = _make_artist(db_sync, slug="cas")
for i in range(3):
f = tmp_path / f"img{i}.jpg"
f.write_bytes(b"x")
_make_image(
db_sync, artist=a, path=str(f),
sha256=f"{i:064x}", size=10,
)
db_sync.commit()
artist_id = a.id
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=artist_id, images_root=tmp_path,
)
assert result["artist"]["slug"] == "cas"
assert result["summary"]["images_deleted"] == 3
assert result["summary"]["files_deleted"] == 3
# Column-select assertions per reference_async_coredml_test_assertions.
surviving_artist = db_sync.execute(
select(func.count(Artist.id)).where(Artist.id == artist_id)
).scalar_one()
surviving_images = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.artist_id == artist_id)
).scalar_one()
assert surviving_artist == 0
assert surviving_images == 0
def test_delete_artist_cascade_idempotent_on_missing(db_sync, tmp_path):
result = cleanup_service.delete_artist_cascade(
db_sync, artist_id=9_999_999, images_root=tmp_path,
)
assert result["artist"] is None
assert result["summary"]["images_deleted"] == 0
# --- delete_images --------------------------------------------------
def test_delete_images_removes_rows_and_files(db_sync, tmp_path):
a = _make_artist(db_sync, slug="di")
f1 = tmp_path / "1.jpg"
f1.write_bytes(b"x")
f2 = tmp_path / "2.jpg"
f2.write_bytes(b"x")
i1 = _make_image(db_sync, artist=a, path=str(f1), sha256="1" * 64)
i2 = _make_image(db_sync, artist=a, path=str(f2), sha256="2" * 64)
db_sync.commit()
result = cleanup_service.delete_images(
db_sync, image_ids=[i1.id, i2.id, 9_999_999],
images_root=tmp_path,
)
assert result["images_deleted"] == 2
assert result["files_deleted"] == 2
assert result["missing_ids"] == [9_999_999]
surviving = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.id.in_([i1.id, i2.id]))
).scalar_one()
assert surviving == 0
# --- delete_tag -----------------------------------------------------
def test_delete_tag_cascades_associations(db_sync, tmp_path):
a = _make_artist(db_sync, slug="dt")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "x.jpg"), sha256="d" * 64,
)
tag = _make_tag(db_sync, name="doomed")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=tag.id,
))
db_sync.commit()
tag_id = tag.id
result = cleanup_service.delete_tag(db_sync, tag_id=tag_id)
assert result["deleted"]["id"] == tag_id
assert result["associations_removed"] == 1
surviving_tag = db_sync.execute(
select(func.count(Tag.id)).where(Tag.id == tag_id)
).scalar_one()
surviving_assoc = db_sync.execute(
select(func.count())
.select_from(image_tag).where(image_tag.c.tag_id == tag_id)
).scalar_one()
assert surviving_tag == 0
assert surviving_assoc == 0
def test_delete_tag_raises_on_unknown_id(db_sync):
with pytest.raises(LookupError):
cleanup_service.delete_tag(db_sync, tag_id=9_999_999)
# --- prune_unused_tags ----------------------------------------------
def test_prune_unused_tags_dry_run_returns_count_and_names(db_sync, tmp_path):
a = _make_artist(db_sync, slug="pu")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "z.jpg"), sha256="e" * 64,
)
used = _make_tag(db_sync, name="kept")
_make_tag(db_sync, name="prune-me-1")
_make_tag(db_sync, name="prune-me-2")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.prune_unused_tags(db_sync, dry_run=True)
assert result["count"] == 2
assert "prune-me-1" in result["sample_names"]
assert "prune-me-2" in result["sample_names"]
# Nothing actually deleted.
surviving = db_sync.execute(select(func.count(Tag.id))).scalar_one()
assert surviving == 3
def test_prune_unused_tags_commit_deletes_them(db_sync, tmp_path):
a = _make_artist(db_sync, slug="pu2")
img = _make_image(
db_sync, artist=a, path=str(tmp_path / "k.jpg"), sha256="f" * 64,
)
used = _make_tag(db_sync, name="kept")
_make_tag(db_sync, name="bye")
db_sync.execute(image_tag.insert().values(
image_record_id=img.id, tag_id=used.id,
))
db_sync.commit()
result = cleanup_service.prune_unused_tags(db_sync, dry_run=False)
assert result["deleted"] == 1
surviving_names = db_sync.execute(select(Tag.name)).scalars().all()
assert "kept" in surviving_names
assert "bye" not in surviving_names
-64
View File
@@ -1,64 +0,0 @@
"""FC-5: backup + rollback service tests.
Uses tmp_path to avoid touching real /images/_backups/. Subprocess
calls (pg_dump, tar) are monkeypatched — the real shell-out is exercised
in operator-side smoke testing on the homelab, not in unit tests.
"""
import pytest
from backend.app.services.migrators import backup as backup_mod
pytestmark = pytest.mark.integration
@pytest.mark.asyncio
async def test_create_backup_writes_sql_and_tar(monkeypatch, tmp_path):
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
# Touch the expected output file so existence checks pass downstream.
if "pg_dump" in cmd[0]:
outpath = tmp_path / "_backups" / f"fc_{kwargs['_test_ts']}.sql"
outpath.write_text("-- fake pg_dump output")
else:
outpath = tmp_path / "_backups" / f"fc_{kwargs['_test_ts']}.tar.zst"
outpath.write_bytes(b"\x28\xb5\x2f\xfd") # zstd magic
from types import SimpleNamespace
return SimpleNamespace(returncode=0, stdout=b"", stderr=b"")
monkeypatch.setattr(backup_mod, "_run_subprocess", fake_run)
result = backup_mod.create_backup(
db_url="postgresql://x", images_root=tmp_path, tag="pre_migration",
)
assert "sql_path" in result
assert "tar_path" in result
assert result["tag"] == "pre_migration"
assert any("pg_dump" in c[0] for c in calls)
assert any(c[0] == "tar" for c in calls)
@pytest.mark.asyncio
async def test_find_latest_backup_by_tag(monkeypatch, tmp_path):
# Create two manifests with different tags.
backups_dir = tmp_path / "_backups"
backups_dir.mkdir()
import json
(backups_dir / "fc_20260101T000000Z.json").write_text(json.dumps({
"backup_id": "20260101T000000Z", "tag": "manual",
"created_at": "2026-01-01T00:00:00+00:00",
"sql_path": "/x/a.sql", "tar_path": "/x/a.tar.zst",
}))
(backups_dir / "fc_20260202T000000Z.json").write_text(json.dumps({
"backup_id": "20260202T000000Z", "tag": "pre_migration",
"created_at": "2026-02-02T00:00:00+00:00",
"sql_path": "/x/b.sql", "tar_path": "/x/b.tar.zst",
}))
found = backup_mod.find_latest_backup(tmp_path, tag="pre_migration")
assert found is not None
assert found["backup_id"] == "20260202T000000Z"
missing = backup_mod.find_latest_backup(tmp_path, tag="nonexistent")
assert missing is None
+158
View File
@@ -0,0 +1,158 @@
"""FC-3k: admin Celery task integration tests.
task_always_eager + signal handlers from FC-3i populate task_run.
We assert the wrapper passes args through correctly and that
task_run lifecycle status flips as expected.
"""
import pytest
from sqlalchemy import func, select
import backend.app.tasks.admin # noqa: F401 — register tasks
from backend.app.celery_app import celery
from backend.app.models import Artist, ImageRecord, TaskRun
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def _eager_celery(monkeypatch):
monkeypatch.setattr(celery.conf, "task_always_eager", True)
monkeypatch.setattr(celery.conf, "task_eager_propagates", False)
@pytest.fixture(autouse=True)
def fake_images_root(monkeypatch, tmp_path):
monkeypatch.setattr("backend.app.tasks.admin.IMAGES_ROOT", tmp_path)
# --- registration ----------------------------------------------------
def test_delete_artist_cascade_task_registered():
assert (
"backend.app.tasks.admin.delete_artist_cascade_task"
in celery.tasks
)
def test_bulk_delete_images_task_registered():
assert (
"backend.app.tasks.admin.bulk_delete_images_task"
in celery.tasks
)
# --- delete_artist_cascade_task -------------------------------------
@pytest.mark.asyncio
async def test_delete_artist_cascade_task_removes_artist_and_records_ok(
db_sync, tmp_path,
):
from backend.app.tasks.admin import delete_artist_cascade_task
a = Artist(name="Doomed", slug="doomed")
db_sync.add(a)
db_sync.flush()
for i in range(2):
f = tmp_path / f"d{i}.jpg"
f.write_bytes(b"x")
db_sync.add(ImageRecord(
artist_id=a.id, path=str(f),
sha256=f"{i:064x}", size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
))
db_sync.commit()
artist_id = a.id
result = delete_artist_cascade_task.delay(artist_id=artist_id).get()
assert result["summary"]["images_deleted"] == 2
surviving = db_sync.execute(
select(func.count(Artist.id)).where(Artist.id == artist_id)
).scalar_one()
assert surviving == 0
# FC-3i task_run lifecycle: task should have an 'ok' row.
status = db_sync.execute(
select(TaskRun.status)
.where(TaskRun.task_name.endswith(".delete_artist_cascade_task"))
.order_by(TaskRun.id.desc()).limit(1)
).scalar_one()
assert status == "ok"
@pytest.mark.asyncio
async def test_delete_artist_cascade_task_records_failure(
db_sync, monkeypatch,
):
from backend.app.tasks.admin import delete_artist_cascade_task
def _boom(*a, **kw):
raise RuntimeError("synthetic cascade fail")
monkeypatch.setattr(
"backend.app.services.cleanup_service.delete_artist_cascade", _boom,
)
with pytest.raises(RuntimeError):
delete_artist_cascade_task.delay(artist_id=1).get()
row = db_sync.execute(
select(TaskRun.status, TaskRun.error_message)
.where(TaskRun.task_name.endswith(".delete_artist_cascade_task"))
.order_by(TaskRun.id.desc()).limit(1)
).one()
assert row.status == "error"
assert "synthetic" in (row.error_message or "")
# --- bulk_delete_images_task ----------------------------------------
@pytest.mark.asyncio
async def test_bulk_delete_images_task_removes_listed_images(
db_sync, tmp_path,
):
from backend.app.tasks.admin import bulk_delete_images_task
a = Artist(name="B", slug="b")
db_sync.add(a)
db_sync.flush()
ids = []
for i in range(3):
f = tmp_path / f"b{i}.jpg"
f.write_bytes(b"x")
img = ImageRecord(
artist_id=a.id, path=str(f),
sha256=f"a{i:063x}", size_bytes=10, mime="image/jpeg",
origin="imported_filesystem",
)
db_sync.add(img)
db_sync.flush()
ids.append(img.id)
db_sync.commit()
result = bulk_delete_images_task.delay(image_ids=ids).get()
assert result["images_deleted"] == 3
surviving = db_sync.execute(
select(func.count(ImageRecord.id))
.where(ImageRecord.id.in_(ids))
).scalar_one()
assert surviving == 0
@pytest.mark.asyncio
async def test_bulk_delete_images_task_idempotent_on_empty_list(db_sync):
from backend.app.tasks.admin import bulk_delete_images_task
result = bulk_delete_images_task.delay(image_ids=[]).get()
assert result["images_deleted"] == 0
@pytest.mark.asyncio
async def test_bulk_delete_images_task_reports_missing_ids(db_sync):
from backend.app.tasks.admin import bulk_delete_images_task
result = bulk_delete_images_task.delay(
image_ids=[9_999_998, 9_999_999],
).get()
assert result["images_deleted"] == 0
assert sorted(result["missing_ids"]) == [9_999_998, 9_999_999]
+281
View File
@@ -0,0 +1,281 @@
"""FC-3h: backup/restore Celery task integration tests.
Uses task_always_eager for synchronous in-test execution.
Subprocess + IMAGES_ROOT are monkeypatched so tests don't touch
real /images or shell out to pg_dump.
"""
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from sqlalchemy import select
from backend.app.celery_app import celery
from backend.app.models import BackupRun, ImportSettings
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def _eager_celery(monkeypatch):
monkeypatch.setattr(celery.conf, "task_always_eager", True)
monkeypatch.setattr(celery.conf, "task_eager_propagates", False)
@pytest.fixture(autouse=True)
def fake_subprocess_and_images_root(monkeypatch, tmp_path):
monkeypatch.setattr("backend.app.tasks.backup.IMAGES_ROOT", tmp_path)
monkeypatch.setattr(
"backend.app.services.backup_service._DB_SUBPROCESS_TIMEOUT_S", 5,
)
class _FakeProc:
returncode = 0
stdout = b""
stderr = b""
def _fake_run(cmd, **kwargs):
if cmd[0] == "pg_dump":
i = cmd.index("-f")
Path(cmd[i + 1]).write_bytes(b"-- fake pg_dump\n")
elif cmd[0] == "tar" and "-cf" in cmd:
i = cmd.index("-cf")
Path(cmd[i + 1]).write_bytes(b"fake tar payload")
return _FakeProc()
monkeypatch.setattr("subprocess.run", _fake_run)
def _seed_backup(db_sync, *, kind, status, started_at, tag=None,
finished_at=None):
row = BackupRun(
kind=kind, status=status, tag=tag,
triggered_by="manual", started_at=started_at,
finished_at=finished_at or (started_at + timedelta(seconds=10)),
sql_path="/tmp/fake.sql" if kind == "db" else None,
tar_path="/tmp/fake.tar.zst" if kind == "images" else None,
manifest={},
)
db_sync.add(row)
db_sync.flush()
return row.id
# --- backup_db_task --------------------------------------------------
@pytest.mark.asyncio
async def test_backup_db_task_creates_backup_run_row_status_ok(db_sync):
from backend.app.tasks.backup import backup_db_task
result = backup_db_task.delay(tag=None, triggered_by="manual").get()
run_id = result["backup_run_id"]
row = db_sync.execute(
select(
BackupRun.kind, BackupRun.status, BackupRun.sql_path,
BackupRun.size_bytes, BackupRun.finished_at, BackupRun.error,
).where(BackupRun.id == run_id)
).one()
assert row.kind == "db"
assert row.status == "ok"
assert row.sql_path and row.sql_path.endswith(".sql")
assert row.size_bytes is not None and row.size_bytes > 0
assert row.finished_at is not None
assert row.error is None
@pytest.mark.asyncio
async def test_backup_db_task_records_failure_on_subprocess_error(db_sync, monkeypatch):
from backend.app.tasks.backup import backup_db_task
def _boom(*a, **kw):
raise RuntimeError("synthetic pg_dump fail")
monkeypatch.setattr("subprocess.run", _boom)
with pytest.raises(RuntimeError):
backup_db_task.delay().get()
row = db_sync.execute(
select(BackupRun.status, BackupRun.error)
.where(BackupRun.kind == "db")
.order_by(BackupRun.id.desc()).limit(1)
).one()
assert row.status == "error"
assert "synthetic" in (row.error or "")
@pytest.mark.asyncio
async def test_backup_db_task_persists_tag(db_sync):
from backend.app.tasks.backup import backup_db_task
result = backup_db_task.delay(tag="pre-cutover").get()
tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == result["backup_run_id"])
).scalar_one()
assert tag == "pre-cutover"
# --- backup_images_task ---------------------------------------------
@pytest.mark.asyncio
async def test_backup_images_task_creates_backup_run(db_sync):
from backend.app.tasks.backup import backup_images_task
result = backup_images_task.delay().get()
row = db_sync.execute(
select(BackupRun.kind, BackupRun.status, BackupRun.tar_path)
.where(BackupRun.id == result["backup_run_id"])
).one()
assert row.kind == "images"
assert row.status == "ok"
assert row.tar_path and row.tar_path.endswith(".tar.zst")
# --- restore_db_task ------------------------------------------------
@pytest.mark.asyncio
async def test_restore_db_task_creates_restoring_marker_then_restored(db_sync):
from backend.app.tasks.backup import backup_db_task, restore_db_task
src = backup_db_task.delay().get()
src_id = src["backup_run_id"]
restore_db_task.delay(source_backup_run_id=src_id).get()
rows = db_sync.execute(
select(BackupRun.status, BackupRun.triggered_by)
.where(BackupRun.restored_from_id == src_id)
).all()
assert len(rows) == 1
assert rows[0].status == "restored"
assert rows[0].triggered_by == "restore"
@pytest.mark.asyncio
async def test_restore_db_task_rejects_non_db_source(db_sync):
from backend.app.tasks.backup import restore_db_task
now = datetime.now(UTC)
img_id = _seed_backup(db_sync, kind="images", status="ok", started_at=now)
db_sync.commit()
with pytest.raises(ValueError):
restore_db_task.delay(source_backup_run_id=img_id).get()
# --- prune_backups --------------------------------------------------
def test_prune_backups_keeps_last_n_untagged_per_kind(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 2
s.backup_images_keep_last_n = 1
db_sync.commit()
now = datetime.now(UTC)
for i in range(4):
_seed_backup(db_sync, kind="db", status="ok",
started_at=now - timedelta(hours=i))
for i in range(3):
_seed_backup(db_sync, kind="images", status="ok",
started_at=now - timedelta(hours=i))
db_sync.commit()
result = prune_backups.apply().get()
assert result["db_deleted"] == 2
assert result["images_deleted"] == 2
def test_prune_backups_protects_tagged_rows(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 1
db_sync.commit()
now = datetime.now(UTC)
_seed_backup(db_sync, kind="db", status="ok", started_at=now)
tagged_id = _seed_backup(
db_sync, kind="db", status="ok",
started_at=now - timedelta(days=30), tag="forever",
)
_seed_backup(db_sync, kind="db", status="ok",
started_at=now - timedelta(days=1))
db_sync.commit()
prune_backups.apply().get()
surviving_tag = db_sync.execute(
select(BackupRun.tag).where(BackupRun.id == tagged_id)
).scalar_one_or_none()
assert surviving_tag == "forever"
def test_prune_backups_never_deletes_running_or_restoring(db_sync):
from backend.app.tasks.backup import prune_backups
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_keep_last_n = 0
db_sync.commit()
ancient = datetime.now(UTC) - timedelta(days=30)
running_id = _seed_backup(db_sync, kind="db", status="running", started_at=ancient)
restoring_id = _seed_backup(db_sync, kind="db", status="restoring", started_at=ancient)
db_sync.commit()
prune_backups.apply().get()
statuses = db_sync.execute(
select(BackupRun.status).where(BackupRun.id.in_([running_id, restoring_id]))
).scalars().all()
assert set(statuses) == {"running", "restoring"}
# --- backup_db_nightly ----------------------------------------------
def test_nightly_skips_when_disabled(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = False
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "skipped" in result and "disabled" in result["skipped"]
def test_nightly_skips_when_hour_mismatch(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = True
s.backup_db_nightly_hour_utc = (datetime.now(UTC).hour + 12) % 24
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "skipped" in result and "hour=" in result["skipped"]
def test_nightly_dispatches_when_enabled_at_configured_hour(db_sync):
from backend.app.tasks.backup import backup_db_nightly
s = db_sync.execute(
select(ImportSettings).where(ImportSettings.id == 1)
).scalar_one()
s.backup_db_nightly_enabled = True
s.backup_db_nightly_hour_utc = datetime.now(UTC).hour
db_sync.commit()
result = backup_db_nightly.apply().get()
assert "dispatched" in result