fix: a post record is marked seen only after it is upserted, and an hourly sweep dates the posts killed walks left undated (#4436)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Failing after 2m25s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 27s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Failing after 2m25s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
ingest_core marked a post's record key seen when it wrote _post.json, but the record reaches the database (and dates the post and its images) only in phase 3. A walk killed before phase 3 left posts the ledger called recorded and the database never dated; ticks early-out long before reaching them again. Keys now join the media in mark_seen_after_import. date_posts_from_records finds each undated native post's record under its artist's folder and upserts it with the post's own source. Hourly on maintenance_long; an empty query once everything is dated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -289,6 +289,11 @@ class Ingester:
|
||||
# Media handed to phase 3 for import. Marked seen by phase 3 once the
|
||||
# import has run (`mark_seen_after_import`), not here — see there.
|
||||
fetched: list[tuple[str, str]] = []
|
||||
# Post-record keys written this walk. Marked with the media, after
|
||||
# phase 3 has upserted the records — marking them at write time left a
|
||||
# walk killed before phase 3 with posts the ledger calls recorded that
|
||||
# the database never dated, and no later tick walks back to them (#4436).
|
||||
recorded: list[tuple[str, str]] = []
|
||||
downloaded = 0
|
||||
errors = 0
|
||||
quarantined = 0
|
||||
@@ -342,7 +347,9 @@ class Ingester:
|
||||
written_paths=written,
|
||||
post_record_paths=list(post_records),
|
||||
relink_source_paths=list(relink),
|
||||
mark_seen_after_import=lambda: self._mark_seen(source_id, fetched),
|
||||
mark_seen_after_import=lambda: self._mark_seen(
|
||||
source_id, fetched + recorded,
|
||||
),
|
||||
stdout="\n".join(log_lines),
|
||||
stderr="",
|
||||
return_code=return_code,
|
||||
@@ -511,7 +518,7 @@ class Ingester:
|
||||
posts_with_body += 1
|
||||
if rec.path is not None:
|
||||
post_records.append(str(rec.path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
recorded.append((pkey, ppid))
|
||||
# Per-post handling line in the run stdout (the existing
|
||||
# "Raw stdout" panel) — the downloader already read the
|
||||
# post; we only format its outcome here. post_type beside
|
||||
|
||||
Reference in New Issue
Block a user