CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 1m1s
Build images / build-web (push) Successful in 1m34s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m26s
Build images / promote (push) Skipped
CI / integration (push) Failing after 3m5s
Three Celery tasks wrapping the 3a service, and the endpoints that drive them. Routed to `maintenance_long` alongside backups: 33k renames on NFS have no business in the quick lane where the self-healing sweeps live (the 2026-06-07 starvation). A durability bug in 3a, found by thinking about what a crash costs rather than by a failing test: `apply_run` wrote its ledger only at the end, so a worker dying at row 30,000 of 33,789 would have taken the undo information for the first 29,999 with it — and that ledger is the ONLY record of where those files came from. It now persists every 200 moves. Two things fell out of writing that: - `_persist` reassigns `run.moves`, so the loop had to snapshot the plan first rather than iterate the attribute it rewrites. - the reassignment is itself load-bearing: SQLAlchemy does not track in-place mutation of a JSONB list, so an `.append()` alone would never reach the database and the ledger would have stayed silently empty. Re-running a partially-applied plan is safe — the moved rows no longer match their `from` and refuse as "row moved since planning" — but `apply_placement` deliberately has NO autoretry: re-entering a half-applied plan should be the operator's call after reading what happened, not the queue's. Endpoints gate on run state as well as the service does, so a stray POST cannot re-apply an applied run. The list response omits `moves` (an applied whole-library run carries tens of thousands of entries); the detail endpoint includes them, because that detail IS the preview read before agreeing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
309 lines
12 KiB
Python
309 lines
12 KiB
Python
"""FC-Cleanup: /api/cleanup/* — retroactive enforcement of import filters.
|
|
|
|
Endpoints:
|
|
POST /min-dimension/preview synchronous SQL audit
|
|
POST /min-dimension/delete synchronous SQL delete (Tier-C token)
|
|
POST /audit async transparency / single_color start
|
|
GET /audit list recent audit_run rows
|
|
GET /audit/<id> single audit_run row
|
|
POST /audit/<id>/apply apply matched_ids deletes (Tier-C token)
|
|
POST /audit/<id>/cancel flip running audit to cancelled
|
|
|
|
Unused-tags retroactive prune intentionally NOT in this namespace —
|
|
TagMaintenanceCard (Maintenance tab → moved to Cleanup tab in v26.05.25.7)
|
|
uses the existing /api/admin/tags/prune-unused endpoint via the admin
|
|
store. No duplicate route here.
|
|
|
|
Confirm-token format matches modal/DestructiveConfirmModal.vue convention:
|
|
`delete-min-dim-<sha8(w,h)>` for min-dim delete
|
|
`delete-audit-<id>` for audit apply
|
|
(Modal hardcodes action ∈ {'restore', 'delete'}; "apply audit" is semantically a delete of the matched images, so we use `delete-audit-<id>`.)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
from pathlib import Path
|
|
|
|
from quart import Blueprint, jsonify, request
|
|
from sqlalchemy import select
|
|
|
|
from ..extensions import get_session
|
|
from ..models import LibraryAuditRun, LibraryPlacementRun
|
|
from ..services import cleanup_service, library_layout
|
|
from ._responses import error_response as _bad
|
|
|
|
cleanup_bp = Blueprint("cleanup", __name__, url_prefix="/api/cleanup")
|
|
|
|
IMAGES_ROOT = Path("/images")
|
|
|
|
|
|
def _min_dim_token(min_w: int, min_h: int) -> str:
|
|
# SHA-256 (not MD5) — Web Crypto's subtle.digest rejects MD5; both
|
|
# sides use SHA-256 truncated to 8 hex chars.
|
|
canon = f"{min_w}x{min_h}"
|
|
return f"delete-min-dim-{hashlib.sha256(canon.encode()).hexdigest()[:8]}"
|
|
|
|
|
|
def _serialize_audit_run(audit: LibraryAuditRun) -> dict:
|
|
return {
|
|
"id": audit.id,
|
|
"rule": audit.rule,
|
|
"params": audit.params,
|
|
"status": audit.status,
|
|
"started_at": audit.started_at.isoformat() if audit.started_at else None,
|
|
"finished_at": audit.finished_at.isoformat() if audit.finished_at else None,
|
|
"scanned_count": audit.scanned_count,
|
|
"matched_count": audit.matched_count,
|
|
"matched_ids": audit.matched_ids,
|
|
"error": audit.error,
|
|
}
|
|
|
|
|
|
@cleanup_bp.route("/min-dimension/preview", methods=["POST"])
|
|
async def min_dim_preview():
|
|
body = await request.get_json(silent=True) or {}
|
|
try:
|
|
min_w = int(body.get("min_width", 0))
|
|
min_h = int(body.get("min_height", 0))
|
|
except (TypeError, ValueError):
|
|
return _bad("invalid_dimensions")
|
|
if min_w < 0 or min_h < 0:
|
|
return _bad("invalid_dimensions")
|
|
async with get_session() as session:
|
|
projection = await session.run_sync(
|
|
lambda s: cleanup_service.project_min_dimension_violations(
|
|
s, min_width=min_w, min_height=min_h,
|
|
)
|
|
)
|
|
# Hand the canonical Tier-C delete token back with the preview so
|
|
# the frontend doesn't have to recompute SHA-256 client-side.
|
|
# window.crypto.subtle is Secure-Context-gated and undefined on
|
|
# plain-HTTP origins (homelab posture); without this the Delete
|
|
# button silently swallowed the TypeError and never opened the
|
|
# confirm modal. Operator-flagged 2026-05-27.
|
|
projection["confirm_token"] = _min_dim_token(min_w, min_h)
|
|
return jsonify(projection)
|
|
|
|
|
|
@cleanup_bp.route("/min-dimension/delete", methods=["POST"])
|
|
async def min_dim_delete():
|
|
body = await request.get_json(silent=True) or {}
|
|
try:
|
|
min_w = int(body.get("min_width", 0))
|
|
min_h = int(body.get("min_height", 0))
|
|
except (TypeError, ValueError):
|
|
return _bad("invalid_dimensions")
|
|
if min_w < 0 or min_h < 0:
|
|
return _bad("invalid_dimensions")
|
|
supplied = body.get("confirm", "")
|
|
expected = _min_dim_token(min_w, min_h)
|
|
if supplied != expected:
|
|
return _bad("confirm_mismatch", expected=expected)
|
|
async with get_session() as session:
|
|
deleted = await session.run_sync(
|
|
lambda s: cleanup_service.delete_min_dimension_violations(
|
|
s, min_width=min_w, min_height=min_h, images_root=IMAGES_ROOT,
|
|
)
|
|
)
|
|
await session.commit()
|
|
return jsonify({"deleted": deleted})
|
|
|
|
|
|
@cleanup_bp.route("/audit", methods=["POST"])
|
|
async def audit_create():
|
|
body = await request.get_json(silent=True) or {}
|
|
rule = body.get("rule")
|
|
params = body.get("params") or {}
|
|
if rule not in ("transparency", "single_color"):
|
|
return _bad("invalid_rule")
|
|
if not isinstance(params, dict):
|
|
return _bad("invalid_params")
|
|
async with get_session() as session:
|
|
try:
|
|
audit_id = await session.run_sync(
|
|
lambda s: cleanup_service.start_audit_run(
|
|
s, rule=rule, params=params,
|
|
)
|
|
)
|
|
except cleanup_service.AuditAlreadyRunning as running_id:
|
|
return _bad(
|
|
"audit_already_running", status=409,
|
|
running_id=int(str(running_id)),
|
|
)
|
|
except ValueError as exc:
|
|
return _bad(str(exc))
|
|
await session.commit()
|
|
return jsonify({"audit_id": audit_id, "status": "running"}), 202
|
|
|
|
|
|
@cleanup_bp.route("/audit/<int:audit_id>", methods=["GET"])
|
|
async def audit_get(audit_id: int):
|
|
async with get_session() as session:
|
|
audit = (await session.execute(
|
|
select(LibraryAuditRun).where(LibraryAuditRun.id == audit_id)
|
|
)).scalar_one_or_none()
|
|
if audit is None:
|
|
return _bad("not_found", status=404)
|
|
return jsonify(_serialize_audit_run(audit))
|
|
|
|
|
|
@cleanup_bp.route("/audit", methods=["GET"])
|
|
async def audit_history():
|
|
try:
|
|
limit = min(int(request.args.get("limit", "20")), 100)
|
|
except ValueError:
|
|
return _bad("invalid_limit")
|
|
# Optional rule filter so a card can reconnect to ITS latest run on mount
|
|
# (?rule=transparency&limit=1) — the audit survives navigation; the UI
|
|
# rehydrates from this rather than losing the in-flight scan.
|
|
rule = request.args.get("rule") or None
|
|
async with get_session() as session:
|
|
stmt = select(LibraryAuditRun).order_by(LibraryAuditRun.id.desc())
|
|
if rule is not None:
|
|
stmt = stmt.where(LibraryAuditRun.rule == rule)
|
|
rows = (await session.execute(stmt.limit(limit))).scalars().all()
|
|
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
|
|
|
|
|
|
@cleanup_bp.route("/audit/<int:audit_id>/apply", methods=["POST"])
|
|
async def audit_apply(audit_id: int):
|
|
body = await request.get_json(silent=True) or {}
|
|
confirm = body.get("confirm", "")
|
|
async with get_session() as session:
|
|
try:
|
|
deleted = await session.run_sync(
|
|
lambda s: cleanup_service.apply_audit_run(
|
|
s, audit_id=audit_id, confirm_token=confirm,
|
|
images_root=IMAGES_ROOT,
|
|
)
|
|
)
|
|
except cleanup_service.AuditNotReady as exc:
|
|
return _bad("audit_not_ready", current_status=str(exc))
|
|
except cleanup_service.ConfirmTokenMismatch as exc:
|
|
return _bad("confirm_mismatch", expected=str(exc))
|
|
except ValueError as exc:
|
|
return _bad("not_found", status=404, detail=str(exc))
|
|
await session.commit()
|
|
return jsonify({"deleted": deleted})
|
|
|
|
|
|
@cleanup_bp.route("/audit/<int:audit_id>/cancel", methods=["POST"])
|
|
async def audit_cancel(audit_id: int):
|
|
async with get_session() as session:
|
|
await session.run_sync(
|
|
lambda s: cleanup_service.cancel_audit_run(s, audit_id=audit_id)
|
|
)
|
|
await session.commit()
|
|
return jsonify({"cancelled": True})
|
|
|
|
|
|
@cleanup_bp.route("/layout", methods=["GET"])
|
|
async def layout_survey():
|
|
"""Milestone #421 blast radius: which ImageRecord rows sit outside their
|
|
artist's canonical slug directory, per artist.
|
|
|
|
Read-only. `?check_disk=1` additionally stats every destination to find
|
|
collisions with a file already there and sources that have gone missing —
|
|
the numbers the apply refuses on, at the cost of one stat per misplaced
|
|
row over NFS. It is OFF by default because a count-only pass answers "how
|
|
big is this" in seconds where the disk pass can run for minutes and time
|
|
the request out.
|
|
"""
|
|
check_disk = request.args.get("check_disk", "").lower() in ("1", "true", "yes")
|
|
async with get_session() as session:
|
|
report = await session.run_sync(
|
|
lambda s: library_layout.survey_layout(
|
|
s, IMAGES_ROOT, check_disk=check_disk,
|
|
)
|
|
)
|
|
return jsonify({**report.as_dict(), "checked_disk": check_disk})
|
|
|
|
|
|
def _serialize_placement_run(run: LibraryPlacementRun, *, moves: bool = False) -> dict:
|
|
"""`moves` is opt-in: an applied whole-library run carries tens of
|
|
thousands of entries, which is a fine thing to hold in Postgres and a
|
|
poor thing to put in every list response."""
|
|
out = {
|
|
"id": run.id,
|
|
"status": run.status,
|
|
"artist_id": run.artist_id,
|
|
"started_at": run.started_at.isoformat() if run.started_at else None,
|
|
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
|
|
"planned_count": run.planned_count,
|
|
"moved_count": run.moved_count,
|
|
"refused_count": run.refused_count,
|
|
"refusals": run.refusals or [],
|
|
"error": run.error,
|
|
}
|
|
if moves:
|
|
out["moves"] = run.moves or []
|
|
return out
|
|
|
|
|
|
@cleanup_bp.route("/placement/runs", methods=["GET"])
|
|
async def placement_runs():
|
|
"""Newest first. Without `moves`, so the list stays small."""
|
|
try:
|
|
limit = min(int(request.args.get("limit", "25")), 100)
|
|
except ValueError:
|
|
return _bad("invalid_limit")
|
|
async with get_session() as session:
|
|
rows = (await session.execute(
|
|
select(LibraryPlacementRun)
|
|
.order_by(LibraryPlacementRun.id.desc()).limit(limit)
|
|
)).scalars().all()
|
|
return jsonify({"runs": [_serialize_placement_run(r) for r in rows]})
|
|
|
|
|
|
@cleanup_bp.route("/placement/runs/<int:run_id>", methods=["GET"])
|
|
async def placement_run(run_id: int):
|
|
"""One run WITH its moves — this is the preview the operator reads before
|
|
agreeing, and the record of what happened afterwards."""
|
|
async with get_session() as session:
|
|
run = await session.get(LibraryPlacementRun, run_id)
|
|
if run is None:
|
|
return _bad("not_found", status=404)
|
|
return jsonify(_serialize_placement_run(run, moves=True))
|
|
|
|
|
|
@cleanup_bp.route("/placement/plan", methods=["POST"])
|
|
async def placement_plan():
|
|
"""Queue a planning run. `artist_id` scopes it to one artist, which is the
|
|
intended use: do one, look at the gallery, then continue or revert."""
|
|
body = await request.get_json(silent=True) or {}
|
|
artist_id = body.get("artist_id")
|
|
if artist_id is not None and not isinstance(artist_id, int):
|
|
return _bad("invalid_artist_id")
|
|
from ..tasks.library_placement import plan_placement
|
|
plan_placement.delay(artist_id)
|
|
return jsonify({"status": "dispatched"}), 202
|
|
|
|
|
|
@cleanup_bp.route("/placement/runs/<int:run_id>/apply", methods=["POST"])
|
|
async def placement_apply(run_id: int):
|
|
"""Execute a ready run. This renames files and rewrites rows."""
|
|
async with get_session() as session:
|
|
run = await session.get(LibraryPlacementRun, run_id)
|
|
if run is None:
|
|
return _bad("not_found", status=404)
|
|
if run.status != "ready":
|
|
return _bad("not_ready", detail=f"run is {run.status}")
|
|
from ..tasks.library_placement import apply_placement
|
|
apply_placement.delay(run_id)
|
|
return jsonify({"status": "dispatched"}), 202
|
|
|
|
|
|
@cleanup_bp.route("/placement/runs/<int:run_id>/revert", methods=["POST"])
|
|
async def placement_revert(run_id: int):
|
|
"""Put an applied run's files back. The reason the ledger is kept."""
|
|
async with get_session() as session:
|
|
run = await session.get(LibraryPlacementRun, run_id)
|
|
if run is None:
|
|
return _bad("not_found", status=404)
|
|
if run.status != "applied":
|
|
return _bad("not_applied", detail=f"run is {run.status}")
|
|
from ..tasks.library_placement import revert_placement
|
|
revert_placement.delay(run_id)
|
|
return jsonify({"status": "dispatched"}), 202
|