feat: placement reconciler — plan, apply, revert (4246, slice 3a)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 11s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 1m1s
Build images / build-web (push) Successful in 1m11s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m56s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m43s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 11s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 1m1s
Build images / build-web (push) Successful in 1m11s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m56s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m43s
Milestone #421 step 3, reframed on the operator's steer: not a one-off migration but the system that keeps the tree true. The 33,789 misplaced rows the survey found are just its first run. The placement half was already done, verified by reading each writer rather than assuming: downloads have always written `<root>/<slug>/<platform>/` (gallery_dl.py:523), attach_in_place leaves files where the downloader put them, and `_copy_to_library` / `_supersede` became canonical in #4244. So nothing is written off-canon today; what remains is the backlog and a standing check for future drift. `LibraryPlacementRun` (migration 0099) holds the plan as JSONB, and that one structure does three jobs: it is the PREVIEW the operator reads, the list the APPLY executes (rather than re-deriving the set, so the two cannot disagree), and — because `from` is retained — the UNDO. The undo is the point. It makes a 33,789-file operation something to do one artist at a time, look at in the gallery, and reverse if it reads wrong. That settles whether artist_id or the folder held the truth (spike #4257) by doing rather than by arguing it from a 50-row sample. An applied run is therefore HISTORY, not state — lesson #4226's trap, since it is the only record of where those files used to be. The model and the migration both say so: any future retention here may prune ready/cancelled/ error runs, never an applied one. Everything fails closed. The apply re-checks each row against what the plan recorded — source still there, destination still free, row still pointing where the plan said — because a download or a supersede can land in between. A refusal is recorded with its reason and the run continues; one stale row is not a reason to abandon the other 33,788. The row is updated only after its rename lands, so a failed move can never leave `path` naming a file that is not there. Writing the collision test caught the code disagreeing with its own comment: it claimed the first of two rows wanting one destination and skipped the second, silently picking a winner by iteration order. Now it counts first and filters after, so genuinely neither is planned. Thumbnails are sha-addressed, not path-keyed, so they do not move — pinned by a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -213,3 +213,191 @@ def test_survey_is_read_only(db_sync, tmp_path):
|
||||
assert db_sync.execute(
|
||||
select(ImageRecord.path).where(ImageRecord.id == rec.id)
|
||||
).scalar_one() == before
|
||||
|
||||
|
||||
# --- plan / apply / revert (#4246) ------------------------------------------
|
||||
|
||||
|
||||
def _staged(db, tmp_path, slug, stray, name="x.png", n=100):
|
||||
"""An artist with one file sitting in `stray`'s directory."""
|
||||
artist = _artist(db, slug.title(), slug)
|
||||
src = tmp_path / stray / name
|
||||
src.parent.mkdir(parents=True, exist_ok=True)
|
||||
src.write_bytes(b"pixels")
|
||||
rec = _image(db, str(src), artist, n)
|
||||
return artist, rec, src
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_plan_records_where_each_file_came_from(db_sync, tmp_path):
|
||||
from backend.app.services.library_layout import plan_placement
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=20)
|
||||
run = plan_placement(db_sync, tmp_path)
|
||||
|
||||
assert run.status == "ready"
|
||||
assert run.planned_count == 1
|
||||
assert run.moves == [{
|
||||
"image_id": rec.id,
|
||||
"from": str(src),
|
||||
"to": str(tmp_path / "conto" / "x.png"),
|
||||
}]
|
||||
# Planning touches nothing.
|
||||
assert src.exists()
|
||||
assert db_sync.get(ImageRecord, rec.id).path == str(src)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_apply_moves_file_and_row_together(db_sync, tmp_path):
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=21)
|
||||
run = apply_run(db_sync, plan_placement(db_sync, tmp_path))
|
||||
|
||||
dest = tmp_path / "conto" / "x.png"
|
||||
assert run.status == "applied"
|
||||
assert run.moved_count == 1 and run.refused_count == 0
|
||||
assert dest.exists() and not src.exists()
|
||||
db_sync.expire_all()
|
||||
assert db_sync.get(ImageRecord, rec.id).path == str(dest)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_revert_puts_it_back(db_sync, tmp_path):
|
||||
"""The whole reason `from` is retained: do one artist, look, undo."""
|
||||
from backend.app.services.library_layout import (
|
||||
apply_run,
|
||||
plan_placement,
|
||||
revert_run,
|
||||
)
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=22)
|
||||
run = revert_run(db_sync, apply_run(db_sync, plan_placement(db_sync, tmp_path)))
|
||||
|
||||
assert run.status == "reverted"
|
||||
assert src.exists()
|
||||
assert not (tmp_path / "conto" / "x.png").exists()
|
||||
db_sync.expire_all()
|
||||
assert db_sync.get(ImageRecord, rec.id).path == str(src)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_apply_refuses_a_row_that_moved_since_planning(db_sync, tmp_path):
|
||||
"""A supersede or an earlier run can rewrite a path between plan and
|
||||
apply. The stale entry is declined, not forced."""
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=23)
|
||||
run = plan_placement(db_sync, tmp_path)
|
||||
|
||||
elsewhere = tmp_path / "conto" / "already-here.png"
|
||||
elsewhere.parent.mkdir(parents=True, exist_ok=True)
|
||||
elsewhere.write_bytes(b"pixels")
|
||||
rec.path = str(elsewhere)
|
||||
db_sync.flush()
|
||||
|
||||
run = apply_run(db_sync, run)
|
||||
|
||||
assert run.moved_count == 0 and run.refused_count == 1
|
||||
assert run.refusals[0]["reason"] == "row moved since planning"
|
||||
assert src.exists() # untouched
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_apply_never_overwrites_an_occupied_destination(db_sync, tmp_path):
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=24)
|
||||
run = plan_placement(db_sync, tmp_path)
|
||||
|
||||
squatter = tmp_path / "conto" / "x.png"
|
||||
squatter.parent.mkdir(parents=True, exist_ok=True)
|
||||
squatter.write_bytes(b"someone else")
|
||||
|
||||
run = apply_run(db_sync, run)
|
||||
|
||||
assert run.refused_count == 1
|
||||
assert run.refusals[0]["reason"] == "destination occupied"
|
||||
assert squatter.read_bytes() == b"someone else"
|
||||
db_sync.expire_all()
|
||||
assert db_sync.get(ImageRecord, rec.id).path == str(src)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_apply_leaves_the_row_alone_when_the_source_is_gone(db_sync, tmp_path):
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
_, rec, src = _staged(db_sync, tmp_path, "conto", "Conto", n=25)
|
||||
run = plan_placement(db_sync, tmp_path)
|
||||
src.unlink()
|
||||
|
||||
run = apply_run(db_sync, run)
|
||||
|
||||
assert run.refusals[0]["reason"] == "source missing"
|
||||
db_sync.expire_all()
|
||||
# The row still points at the missing file rather than at a file that
|
||||
# was never created — a broken row is recoverable, a lying one is not.
|
||||
assert db_sync.get(ImageRecord, rec.id).path == str(src)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_plan_scopes_to_one_artist(db_sync, tmp_path):
|
||||
"""Per-artist scope is what makes this incremental instead of one
|
||||
irreversible sweep."""
|
||||
from backend.app.services.library_layout import plan_placement
|
||||
|
||||
conto, _, _ = _staged(db_sync, tmp_path, "conto", "Conto", n=26)
|
||||
_staged(db_sync, tmp_path, "maewix", "Maewix", name="y.png", n=27)
|
||||
|
||||
run = plan_placement(db_sync, tmp_path, artist_id=conto.id)
|
||||
|
||||
assert run.planned_count == 1
|
||||
assert run.artist_id == conto.id
|
||||
assert "Conto" in run.moves[0]["from"]
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_plan_skips_both_rows_when_two_want_one_destination(db_sync, tmp_path):
|
||||
"""Which of two colliding rows 'wins' is not this sweep's call."""
|
||||
from backend.app.services.library_layout import plan_placement
|
||||
|
||||
artist = _artist(db_sync, "Sticky", "sticky")
|
||||
for stray, n in (("StickySpoodge", 28), ("Stickyspoodge", 29)):
|
||||
p = tmp_path / stray / "dup.png"
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_bytes(b"pixels")
|
||||
_image(db_sync, str(p), artist, n)
|
||||
|
||||
run = plan_placement(db_sync, tmp_path)
|
||||
|
||||
assert run.planned_count == 0
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_thumbnails_do_not_move(db_sync, tmp_path):
|
||||
"""Thumbs are sha-addressed (`thumbs/<xx>/<sha>.jpg`), not path-keyed, so
|
||||
a placement move must not touch them. Pinned so nobody 'fixes' it."""
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
artist, rec, _ = _staged(db_sync, tmp_path, "conto", "Conto", n=30)
|
||||
thumb = tmp_path / "thumbs" / "ab" / "abc.jpg"
|
||||
thumb.parent.mkdir(parents=True, exist_ok=True)
|
||||
thumb.write_bytes(b"thumb")
|
||||
rec.thumbnail_path = str(thumb)
|
||||
db_sync.flush()
|
||||
|
||||
apply_run(db_sync, plan_placement(db_sync, tmp_path))
|
||||
|
||||
db_sync.expire_all()
|
||||
assert thumb.exists()
|
||||
assert db_sync.get(ImageRecord, rec.id).thumbnail_path == str(thumb)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_apply_refuses_a_run_that_is_not_ready(db_sync, tmp_path):
|
||||
from backend.app.services.library_layout import apply_run, plan_placement
|
||||
|
||||
_staged(db_sync, tmp_path, "conto", "Conto", n=31)
|
||||
run = apply_run(db_sync, plan_placement(db_sync, tmp_path))
|
||||
with pytest.raises(ValueError):
|
||||
apply_run(db_sync, run)
|
||||
|
||||
Reference in New Issue
Block a user