revert: remove the placement reconciler — it manufactured the problem it solved
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 59s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m55s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m22s

Milestone #421 built a sweep that compared each image's `artist_id` to the
name of the directory holding its file, and called every mismatch a misplaced
image. It reported 33,789 of 63,605 as wrongly filed. That number described
the comparison, not the library.

What it actually was:

  32,475  (97.1%)  one artist's own folder, spelled differently
                   — Telepurte/ vs telepurte/. Same artist, same art.
     657  ( 2.0%)  loose at the images root
     328  ( 1.0%)  in a folder named after a different artist

And the 1% did not mean what the tool assumed either. `ImageProvenance`
records the post and source every file was downloaded from — the
authoritative answer, which the tool never consulted. Querying it for all 328:

    144  provenance agrees with the record  (move would be right)
     87  provenance agrees with the FOLDER  (the record is wrong; move wrong)
     53  provenance names SEVERAL artists   (no single correct folder)
     41  no provenance at all
      3  agrees with neither

So the sweep would have misfiled or arbitrarily picked for ~41% of the only
set it was really needed for. The system already knew where each file came
from; the tool inferred it from a column and a directory name instead.

Operator, 2026-09-21: *"the current system consistently records where items
are and where they came from this is just complicating something works and
doesn't need fixing."* Correct on both counts.

Removed: the service, the tasks, the model and migration 0099's table, the
/api/cleanup/layout and /placement/* endpoints, the Maintenance card and its
store actions, and the tests. 0100 drops the table (rule #22 — no legacy).

KEPT deliberately, per the operator:
- `utils.paths.canonical_subdir` — new filesystem imports derive their
  directory from the artist's slug, matching what the downloader always did.
  Not part of this tool; removing it would be churn that fixes nothing.
- The 327 files run 1 moved (InsoUwu/ -> insouwu/). Same artist either way,
  and the gallery renders them correctly.
- Everything from #4223 (three-gate dedup, 256-bit pHash) and #4234 (backup
  credential exclusion). Those fixed problems that were actually reported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-21 18:17:13 -04:00
co-authored by Claude Opus 5
parent 2dd9b956d5
commit 11a01a9686
14 changed files with 111 additions and 1850 deletions
-125
View File
@@ -1,125 +0,0 @@
"""Placement reconciler tasks — plan, apply, revert (milestone #421).
The service (`services.library_layout`) holds the decisions; this module is
only the async wrapper, matching `tasks.library_audit`: run on the
maintenance queue, mark the run `error` with a traceback if anything escapes,
and return a small summary dict so eager-mode tests can assert on it.
Applying is a long run — 33,789 renames on the operator's library at the time
of writing — so `apply_placement` persists its ledger in chunks rather than
at the end. That ledger is the only record of where each file came from, and
a worker that dies two thirds of the way through must not take the undo
information for the first two thirds with it.
"""
import logging
import traceback
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..models import LibraryPlacementRun
from ..services import library_layout
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
# Commit the ledger every this many moves. Small enough that a crash loses
# seconds of work, large enough not to make a COMMIT per rename.
_APPLY_CHUNK = 200
def _fail(session, run_id: int, message: str) -> None:
run = session.get(LibraryPlacementRun, run_id)
if run is not None:
run.status = "error"
run.error = message
run.finished_at = datetime.now(UTC)
session.commit()
@celery.task(
name="backend.app.tasks.library_placement.plan_placement",
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=5, retry_backoff_max=60, retry_jitter=True, max_retries=3,
soft_time_limit=900, time_limit=1000,
)
def plan_placement(artist_id: int | None = None) -> dict:
"""Build a move plan and leave it `ready` for the operator to read.
Reads rows and stats destinations; moves nothing.
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = library_layout.plan_placement(
session, IMAGES_ROOT, artist_id=artist_id,
)
session.commit()
return {
"run_id": run.id, "status": run.status,
"planned_count": run.planned_count,
}
@celery.task(
name="backend.app.tasks.library_placement.apply_placement",
soft_time_limit=7200, time_limit=7500,
)
def apply_placement(run_id: int) -> dict:
"""Execute a `ready` run's stored plan. Renames files and rewrites rows.
No autoretry: a retry would re-enter a half-applied plan on a schedule
nobody asked for. Re-running IS safe (the applied rows refuse as "row
moved since planning"), but that should be the operator's decision after
reading what happened, not the queue's.
"""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = session.get(LibraryPlacementRun, run_id)
if run is None:
return {"run_id": run_id, "status": "missing"}
if run.status != "ready":
return {"run_id": run_id, "status": run.status, "skipped": True}
try:
library_layout.apply_run(session, run, chunk=_APPLY_CHUNK)
session.commit()
except Exception:
log.exception("placement apply failed for run %s", run_id)
session.rollback()
_fail(session, run_id, traceback.format_exc())
return {"run_id": run_id, "status": "error"}
return {
"run_id": run_id, "status": run.status,
"moved": run.moved_count, "refused": run.refused_count,
}
@celery.task(
name="backend.app.tasks.library_placement.revert_placement",
soft_time_limit=7200, time_limit=7500,
)
def revert_placement(run_id: int) -> dict:
"""Put an applied run's files back where they came from."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = session.get(LibraryPlacementRun, run_id)
if run is None:
return {"run_id": run_id, "status": "missing"}
if run.status != "applied":
return {"run_id": run_id, "status": run.status, "skipped": True}
try:
library_layout.revert_run(session, run)
session.commit()
except Exception:
log.exception("placement revert failed for run %s", run_id)
session.rollback()
_fail(session, run_id, traceback.format_exc())
return {"run_id": run_id, "status": "error"}
return {
"run_id": run_id, "status": run.status,
"refused": run.refused_count,
}