refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m18s

Two corrections from operator review:
1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI
   section — the native ingester now writes a per-post line into the run stdout
   (parity with gallery-dl's per-file stdout), so the per-post handling shows in
   the panel the operator already uses.
2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record
   now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring
   the download_post -> MediaOutcome contract — and the downloader owns the read;
   ingest_core only formats the outcome into the log line.

Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture'
section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+
' — EMPTY' when 0), so an empty body is self-explanatory by post_type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 23:27:38 -04:00
parent bcc7266021
commit eb811e11f6
8 changed files with 71 additions and 162 deletions
+1 -24
View File
@@ -60,11 +60,10 @@ def _make_fake_dl_result(
*, success=True, written_paths=None, quarantined_paths=None,
files_downloaded=0, error_type=None, error_message=None,
stdout="", stderr="", cursor=None, run_stats=None, posts_processed=0,
relink_source_paths=None, post_diagnostics=None,
relink_source_paths=None,
):
return SimpleNamespace(
relink_source_paths=relink_source_paths or [],
post_diagnostics=post_diagnostics or [],
success=success,
url="https://patreon.com/alice",
artist_slug="alice",
@@ -821,28 +820,6 @@ async def test_recapture_mode_selected_and_flag_cleared_on_complete(
assert "_backfill_recapture" not in co
@pytest.mark.asyncio
async def test_post_diagnostics_written_to_event_metadata(
db, db_sync, tmp_path, seed_artist_and_source,
):
"""#842: per-post body diagnostics land on the DownloadEvent.metadata_ so the
UI can show how each post was handled (post_type + body length)."""
_artist, source = seed_artist_and_source
diag = [
{"post_id": "p1", "title": "Has body", "post_type": "image_file", "body_chars": 42},
{"post_id": "p2", "title": "Empty poll", "post_type": "poll", "body_chars": 0},
]
svc, _ = _backfill_svc(db, db_sync, tmp_path, _make_fake_dl_result(
success=True, post_diagnostics=diag,
))
await svc.download_source(source.id)
ev = (await db.execute(
select(DownloadEvent).order_by(DownloadEvent.id.desc()).limit(1)
)).scalar_one()
assert ev.metadata_["post_diagnostics"] == diag
@pytest.mark.asyncio
async def test_partial_error_type_maps_to_ok_status(
db, db_sync, tmp_path, seed_artist_and_source,