refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m18s

Two corrections from operator review:
1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI
   section — the native ingester now writes a per-post line into the run stdout
   (parity with gallery-dl's per-file stdout), so the per-post handling shows in
   the panel the operator already uses.
2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record
   now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring
   the download_post -> MediaOutcome contract — and the downloader owns the read;
   ingest_core only formats the outcome into the log line.

Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture'
section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+
' — EMPTY' when 0), so an empty body is self-explanatory by post_type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 23:27:38 -04:00
parent bcc7266021
commit eb811e11f6
8 changed files with 71 additions and 162 deletions
+13 -22
View File
@@ -143,11 +143,6 @@ class Ingester:
# media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
# re-downloading or unlinking the file. Empty outside recapture mode.
relink: list[tuple[str, str]] = []
# #842: per-post body-capture diagnostics surfaced on the DownloadEvent so
# the operator can see IN THE UI how each post was handled — crucially the
# post_type + final body length, which explains an empty body (e.g. a poll
# whose body the API never returns) without digging through worker logs.
post_diag: list[dict] = []
downloaded = 0
errors = 0
quarantined = 0
@@ -185,7 +180,6 @@ class Ingester:
written_paths=written,
post_record_paths=list(post_records),
relink_source_paths=list(relink),
post_diagnostics=list(post_diag),
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
@@ -266,23 +260,20 @@ class Ingester:
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
rec_path = write_post_record(post, artist_slug)
if rec_path is not None:
post_records.append(str(rec_path))
rec = write_post_record(post, artist_slug)
if rec.path is not None:
post_records.append(str(rec.path))
self._mark_seen(source_id, [(pkey, ppid)])
# Read the FINAL body (write_post_record memoized any
# detail-fetched content onto the post) + post_type
# (already in the feed fieldset) for the per-post UI
# diagnostic. A body_chars of 0 with the post_type is
# the operator's "why is this one empty" answer.
pattrs = post.get("attributes") or {}
pbody = pattrs.get("content")
post_diag.append({
"post_id": ppid,
"title": pattrs.get("title") or None,
"post_type": pattrs.get("post_type") or None,
"body_chars": len(pbody) if isinstance(pbody, str) else 0,
})
# Per-post handling line in the run stdout (the existing
# "Raw stdout" panel) — the downloader already read the
# post; we only format its outcome here. post_type beside
# a 0-char body is the "why is this one empty" answer.
log_lines.append(
f" post {ppid} [{rec.post_type or '?'}] "
f"body: {rec.body_chars} chars"
+ ("" if rec.body_chars else " — EMPTY")
+ (f"{rec.title}" if rec.title else "")
)
media = self.client.extract_media(post, included)
if not media: