refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)
Two corrections from operator review: 1. Reuse the existing 'Raw stdout' panel instead of a bespoke structured UI section — the native ingester now writes a per-post line into the run stdout (parity with gallery-dl's per-file stdout), so the per-post handling shows in the panel the operator already uses. 2. DRY: stop re-reading post['attributes'] inline in ingest_core. write_post_record now returns a PostRecordOutcome (path, post_type, title, body_chars) — mirroring the download_post -> MediaOutcome contract — and the downloader owns the read; ingest_core only formats the outcome into the log line. Reverts the post_diagnostics metadata field + DownloadDetailModal 'Post capture' section added earlier. Per-post line: 'post <id> [<post_type>] body: N chars' (+ ' — EMPTY' when 0), so an empty body is self-explanatory by post_type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -143,11 +143,6 @@ class Ingester:
|
||||
# media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
|
||||
# re-downloading or unlinking the file. Empty outside recapture mode.
|
||||
relink: list[tuple[str, str]] = []
|
||||
# #842: per-post body-capture diagnostics surfaced on the DownloadEvent so
|
||||
# the operator can see IN THE UI how each post was handled — crucially the
|
||||
# post_type + final body length, which explains an empty body (e.g. a poll
|
||||
# whose body the API never returns) without digging through worker logs.
|
||||
post_diag: list[dict] = []
|
||||
downloaded = 0
|
||||
errors = 0
|
||||
quarantined = 0
|
||||
@@ -185,7 +180,6 @@ class Ingester:
|
||||
written_paths=written,
|
||||
post_record_paths=list(post_records),
|
||||
relink_source_paths=list(relink),
|
||||
post_diagnostics=list(post_diag),
|
||||
stdout="\n".join(log_lines),
|
||||
stderr="",
|
||||
return_code=return_code,
|
||||
@@ -266,23 +260,20 @@ class Ingester:
|
||||
else self._seen_keys(source_id, [pkey])
|
||||
)
|
||||
if pkey not in already:
|
||||
rec_path = write_post_record(post, artist_slug)
|
||||
if rec_path is not None:
|
||||
post_records.append(str(rec_path))
|
||||
rec = write_post_record(post, artist_slug)
|
||||
if rec.path is not None:
|
||||
post_records.append(str(rec.path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
# Read the FINAL body (write_post_record memoized any
|
||||
# detail-fetched content onto the post) + post_type
|
||||
# (already in the feed fieldset) for the per-post UI
|
||||
# diagnostic. A body_chars of 0 with the post_type is
|
||||
# the operator's "why is this one empty" answer.
|
||||
pattrs = post.get("attributes") or {}
|
||||
pbody = pattrs.get("content")
|
||||
post_diag.append({
|
||||
"post_id": ppid,
|
||||
"title": pattrs.get("title") or None,
|
||||
"post_type": pattrs.get("post_type") or None,
|
||||
"body_chars": len(pbody) if isinstance(pbody, str) else 0,
|
||||
})
|
||||
# Per-post handling line in the run stdout (the existing
|
||||
# "Raw stdout" panel) — the downloader already read the
|
||||
# post; we only format its outcome here. post_type beside
|
||||
# a 0-char body is the "why is this one empty" answer.
|
||||
log_lines.append(
|
||||
f" post {ppid} [{rec.post_type or '?'}] "
|
||||
f"body: {rec.body_chars} chars"
|
||||
+ ("" if rec.body_chars else " — EMPTY")
|
||||
+ (f" — {rec.title}" if rec.title else "")
|
||||
)
|
||||
|
||||
media = self.client.extract_media(post, included)
|
||||
if not media:
|
||||
|
||||
Reference in New Issue
Block a user