feat(downloads): per-post body-capture diagnostics in the event UI (#842)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m32s

Operator can't (and shouldn't have to) hunt worker logs to see why a recapture
left a post body empty. Surface per-post handling ON THE EVENT, in the UI.

The feed already requests post_type (in _FIELDS_POST), so ingest_core builds a
per-post diagnostic {post_id, title, post_type, body_chars} with zero extra
fetching — a 0-char body next to its post_type explains an empty post at a
glance (e.g. polls/embeds whose body the API never returns).

- ingest_core: accumulate post_diagnostics; thread via DownloadResult
- download_service: write to DownloadEvent.metadata_['post_diagnostics']
- DownloadDetailModal: 'Post capture' section — totals + empty-body table
  (post_type + chars, flagged) + all-posts table; included in Copy-all
- tests: ingester diag (post_type + body_chars), download_service metadata

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 23:09:37 -04:00
parent 3df191e255
commit bcc7266021
6 changed files with 154 additions and 2 deletions
+19
View File
@@ -143,6 +143,11 @@ class Ingester:
# media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
# re-downloading or unlinking the file. Empty outside recapture mode.
relink: list[tuple[str, str]] = []
# #842: per-post body-capture diagnostics surfaced on the DownloadEvent so
# the operator can see IN THE UI how each post was handled — crucially the
# post_type + final body length, which explains an empty body (e.g. a poll
# whose body the API never returns) without digging through worker logs.
post_diag: list[dict] = []
downloaded = 0
errors = 0
quarantined = 0
@@ -180,6 +185,7 @@ class Ingester:
written_paths=written,
post_record_paths=list(post_records),
relink_source_paths=list(relink),
post_diagnostics=list(post_diag),
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
@@ -264,6 +270,19 @@ class Ingester:
if rec_path is not None:
post_records.append(str(rec_path))
self._mark_seen(source_id, [(pkey, ppid)])
# Read the FINAL body (write_post_record memoized any
# detail-fetched content onto the post) + post_type
# (already in the feed fieldset) for the per-post UI
# diagnostic. A body_chars of 0 with the post_type is
# the operator's "why is this one empty" answer.
pattrs = post.get("attributes") or {}
pbody = pattrs.get("content")
post_diag.append({
"post_id": ppid,
"title": pattrs.get("title") or None,
"post_type": pattrs.get("post_type") or None,
"body_chars": len(pbody) if isinstance(pbody, str) else 0,
})
media = self.client.extract_media(post, included)
if not media: