fix: a post record is marked seen only after it is upserted, and an hourly sweep dates the posts killed walks left undated (#4436)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Failing after 2m25s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Failing after 2m25s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
ingest_core marked a post's record key seen when it wrote _post.json, but the record reaches the database (and dates the post and its images) only in phase 3. A walk killed before phase 3 left posts the ledger called recorded and the database never dated; ticks early-out long before reaching them again. Keys now join the media in mark_seen_after_import. date_posts_from_records finds each undated native post's record under its artist's folder and upserts it with the post's own source. Hourly on maintenance_long; an empty query once everything is dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -943,8 +943,10 @@ async def test_a_run_that_dies_before_import_leaves_its_media_unmarked(
|
||||
_FakeDownloader(tmp_path))
|
||||
ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest",
|
||||
url="https://patreon.com/ingest", mode="tick")
|
||||
# Phase 3 never ran, so `mark_seen_after_import` never did: only the post key.
|
||||
assert _count_ledger(sync_engine, source_id) == 1
|
||||
# Phase 3 never ran, so `mark_seen_after_import` never did: neither the
|
||||
# media nor the post record is marked (#4436 — the record, marked at write
|
||||
# time, left the post undated for good once the run died before phase 3).
|
||||
assert _count_ledger(sync_engine, source_id) == 0
|
||||
|
||||
# The next walk finds the file on disk with no record, and imports it.
|
||||
ing2 = _ingester(sync_engine, tmp_path, _FakeClient([(None, [("p1", [m1])])]),
|
||||
@@ -954,6 +956,7 @@ async def test_a_run_that_dies_before_import_leaves_its_media_unmarked(
|
||||
assert result.written_paths == [str(tmp_path / "p1_1.jpg")]
|
||||
assert result.files_downloaded == 0 # not fetched again
|
||||
assert "on disk but never imported: p1_1.jpg" in result.stdout
|
||||
assert len(result.post_record_paths) == 1 # the record is written again
|
||||
result.mark_seen_after_import()
|
||||
assert _count_ledger(sync_engine, source_id) == 2
|
||||
|
||||
|
||||
Reference in New Issue
Block a user