revert: remove the placement reconciler — it manufactured the problem it solved
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 59s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m55s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m22s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 59s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m55s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m22s
Milestone #421 built a sweep that compared each image's `artist_id` to the name of the directory holding its file, and called every mismatch a misplaced image. It reported 33,789 of 63,605 as wrongly filed. That number described the comparison, not the library. What it actually was: 32,475 (97.1%) one artist's own folder, spelled differently — Telepurte/ vs telepurte/. Same artist, same art. 657 ( 2.0%) loose at the images root 328 ( 1.0%) in a folder named after a different artist And the 1% did not mean what the tool assumed either. `ImageProvenance` records the post and source every file was downloaded from — the authoritative answer, which the tool never consulted. Querying it for all 328: 144 provenance agrees with the record (move would be right) 87 provenance agrees with the FOLDER (the record is wrong; move wrong) 53 provenance names SEVERAL artists (no single correct folder) 41 no provenance at all 3 agrees with neither So the sweep would have misfiled or arbitrarily picked for ~41% of the only set it was really needed for. The system already knew where each file came from; the tool inferred it from a column and a directory name instead. Operator, 2026-09-21: *"the current system consistently records where items are and where they came from this is just complicating something works and doesn't need fixing."* Correct on both counts. Removed: the service, the tasks, the model and migration 0099's table, the /api/cleanup/layout and /placement/* endpoints, the Maintenance card and its store actions, and the tests. 0100 drops the table (rule #22 — no legacy). KEPT deliberately, per the operator: - `utils.paths.canonical_subdir` — new filesystem imports derive their directory from the artist's slug, matching what the downloader always did. Not part of this tool; removing it would be churn that fixes nothing. - The 327 files run 1 moved (InsoUwu/ -> insouwu/). Same artist either way, and the gallery renders them correctly. - Everything from #4223 (three-gate dedup, 256-bit pHash) and #4234 (backup credential exclusion). Those fixed problems that were actually reported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -1,125 +0,0 @@
|
||||
"""Placement reconciler tasks — plan, apply, revert (milestone #421).
|
||||
|
||||
The service (`services.library_layout`) holds the decisions; this module is
|
||||
only the async wrapper, matching `tasks.library_audit`: run on the
|
||||
maintenance queue, mark the run `error` with a traceback if anything escapes,
|
||||
and return a small summary dict so eager-mode tests can assert on it.
|
||||
|
||||
Applying is a long run — 33,789 renames on the operator's library at the time
|
||||
of writing — so `apply_placement` persists its ledger in chunks rather than
|
||||
at the end. That ledger is the only record of where each file came from, and
|
||||
a worker that dies two thirds of the way through must not take the undo
|
||||
information for the first two thirds with it.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import traceback
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.exc import DBAPIError, OperationalError
|
||||
|
||||
from ..celery_app import celery
|
||||
from ..models import LibraryPlacementRun
|
||||
from ..services import library_layout
|
||||
from ._sync_engine import sync_session_factory as _sync_session_factory
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
IMAGES_ROOT = Path("/images")
|
||||
|
||||
# Commit the ledger every this many moves. Small enough that a crash loses
|
||||
# seconds of work, large enough not to make a COMMIT per rename.
|
||||
_APPLY_CHUNK = 200
|
||||
|
||||
|
||||
def _fail(session, run_id: int, message: str) -> None:
|
||||
run = session.get(LibraryPlacementRun, run_id)
|
||||
if run is not None:
|
||||
run.status = "error"
|
||||
run.error = message
|
||||
run.finished_at = datetime.now(UTC)
|
||||
session.commit()
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.library_placement.plan_placement",
|
||||
autoretry_for=(OperationalError, DBAPIError),
|
||||
retry_backoff=5, retry_backoff_max=60, retry_jitter=True, max_retries=3,
|
||||
soft_time_limit=900, time_limit=1000,
|
||||
)
|
||||
def plan_placement(artist_id: int | None = None) -> dict:
|
||||
"""Build a move plan and leave it `ready` for the operator to read.
|
||||
|
||||
Reads rows and stats destinations; moves nothing.
|
||||
"""
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
run = library_layout.plan_placement(
|
||||
session, IMAGES_ROOT, artist_id=artist_id,
|
||||
)
|
||||
session.commit()
|
||||
return {
|
||||
"run_id": run.id, "status": run.status,
|
||||
"planned_count": run.planned_count,
|
||||
}
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.library_placement.apply_placement",
|
||||
soft_time_limit=7200, time_limit=7500,
|
||||
)
|
||||
def apply_placement(run_id: int) -> dict:
|
||||
"""Execute a `ready` run's stored plan. Renames files and rewrites rows.
|
||||
|
||||
No autoretry: a retry would re-enter a half-applied plan on a schedule
|
||||
nobody asked for. Re-running IS safe (the applied rows refuse as "row
|
||||
moved since planning"), but that should be the operator's decision after
|
||||
reading what happened, not the queue's.
|
||||
"""
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
run = session.get(LibraryPlacementRun, run_id)
|
||||
if run is None:
|
||||
return {"run_id": run_id, "status": "missing"}
|
||||
if run.status != "ready":
|
||||
return {"run_id": run_id, "status": run.status, "skipped": True}
|
||||
try:
|
||||
library_layout.apply_run(session, run, chunk=_APPLY_CHUNK)
|
||||
session.commit()
|
||||
except Exception:
|
||||
log.exception("placement apply failed for run %s", run_id)
|
||||
session.rollback()
|
||||
_fail(session, run_id, traceback.format_exc())
|
||||
return {"run_id": run_id, "status": "error"}
|
||||
return {
|
||||
"run_id": run_id, "status": run.status,
|
||||
"moved": run.moved_count, "refused": run.refused_count,
|
||||
}
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.library_placement.revert_placement",
|
||||
soft_time_limit=7200, time_limit=7500,
|
||||
)
|
||||
def revert_placement(run_id: int) -> dict:
|
||||
"""Put an applied run's files back where they came from."""
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
run = session.get(LibraryPlacementRun, run_id)
|
||||
if run is None:
|
||||
return {"run_id": run_id, "status": "missing"}
|
||||
if run.status != "applied":
|
||||
return {"run_id": run_id, "status": run.status, "skipped": True}
|
||||
try:
|
||||
library_layout.revert_run(session, run)
|
||||
session.commit()
|
||||
except Exception:
|
||||
log.exception("placement revert failed for run %s", run_id)
|
||||
session.rollback()
|
||||
_fail(session, run_id, traceback.format_exc())
|
||||
return {"run_id": run_id, "status": "error"}
|
||||
return {
|
||||
"run_id": run_id, "status": run.status,
|
||||
"refused": run.refused_count,
|
||||
}
|
||||
Reference in New Issue
Block a user