feat: a tick keeps looking back 30 days, so an EDITED post is reached (4386)
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 40s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 40s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator, 2026-09-23, on a Floppystack post: "this post has been updated as he implements hot fixes — any chance we have a way to scan for or see updated posts so we can update ours to match and pull the new attachments and pictures etc." The download half already worked: extract_media reads the media list off the LIVE feed response every walk, so a newly attached hotfix build is a ledger key we have never seen. Only REACHING the post was missing — a tick stopped after 20 contiguous already-have-it items, and a post edited three days after publication sits well below twenty. Not a bug in the early-out; a count cannot express "recent". The early-out now needs BOTH conditions: the run of seen items AND a post published before the horizon. Strictly a widening — window 0 is exactly the old behaviour, and no window can make a tick stop EARLIER than it used to, so a source paused for months still walks its whole unseen backlog. The horizon is a floor on how far to look, never a ceiling. Inside the window the post-record gate is bypassed too (write_post_record revisit=True): the body is re-read from the feed response already in hand, so a revisit costs zero requests, and a body that comes back empty writes NOTHING rather than blanking one a detail-fetch had filled. Revisits are kept out of the #862 body-drift canary's sample for the same reason — an empty revisit is healthy, and counting it would walk the alarm toward firing on good ticks. The run summary names what changed ("3 post(s) updated (5 new file(s))") with a line per post; the ask was to SEE updated posts, not only to end up with their bytes. download_revisit_days is a settings row, not a constant (rule 25) — how long a creator keeps editing is a property of the creator. Default 30, 0 turns it off. Migration 0108. Also corrects two stale docstrings: both clients described post_meta as feeding an Ingester.preview that no longer calls it. It had no consumer at all until this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -340,7 +340,7 @@ class PatreonDownloader(BaseNativeDownloader):
|
||||
|
||||
def _write_sidecar_data(
|
||||
self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
|
||||
minimal: bool = False,
|
||||
minimal: bool = False, detail_fetch: bool = True,
|
||||
) -> Path:
|
||||
"""Serialize the post's metadata to `sidecar_path`. The post-only record
|
||||
(`write_post_record`) writes the FULL post (body/title/date/url); the
|
||||
@@ -364,7 +364,13 @@ class PatreonDownloader(BaseNativeDownloader):
|
||||
# dict — so a multi-image post fetches detail at most once, the post-record
|
||||
# body-length read reuses it, and a fully-seen post (no fresh download → no
|
||||
# sidecar write) never pays the extra GET.
|
||||
if (not content or not content.strip()) and self._content_fetcher:
|
||||
# `detail_fetch=False` on a REVISIT (a post inside the tick's revisit
|
||||
# window that we already captured): re-read the body from the feed
|
||||
# response we are holding and pay nothing. Without this a 30-day window
|
||||
# would buy one detail GET per body-less post per tick, forever — a
|
||||
# per-creator cost that grows with how prolific they are, to re-fetch a
|
||||
# body we already stored.
|
||||
if (not content or not content.strip()) and self._content_fetcher and detail_fetch:
|
||||
fetched = self._content_fetcher(str(post.get("id") or ""))
|
||||
if fetched:
|
||||
content = fetched
|
||||
@@ -386,7 +392,9 @@ class PatreonDownloader(BaseNativeDownloader):
|
||||
sidecar_path.write_text(json.dumps(data, indent=2))
|
||||
return sidecar_path
|
||||
|
||||
def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
|
||||
def write_post_record(
|
||||
self, post: dict, artist_slug: str, *, revisit: bool = False,
|
||||
) -> PostRecordOutcome:
|
||||
"""Write a post-ONLY sidecar (no media file) for a media-less post, so
|
||||
the importer can still upsert the Post + its body — text posts often hold
|
||||
the only copy of an external <a href> link. Named `_post.json`: the
|
||||
@@ -397,6 +405,18 @@ class PatreonDownloader(BaseNativeDownloader):
|
||||
Returns a PostRecordOutcome (path None when the post has no id) carrying
|
||||
the captured body's shape — post_type + final char count — so the engine
|
||||
can log per-post handling without re-reading the post itself.
|
||||
|
||||
`revisit=True` is the tick re-reading a post it already captured
|
||||
(ingest_core's revisit window, #...). Two differences, both about not
|
||||
making an update cost more than it is worth:
|
||||
|
||||
* no detail-fetch — the body comes from the feed response already in
|
||||
hand, so a revisit costs zero requests;
|
||||
* a body that comes back empty writes NOTHING and returns `path=None`.
|
||||
On a first capture an empty body is the truth about the post; on a
|
||||
revisit it usually just means this post's body only ever came from
|
||||
the detail endpoint we just declined to call, and writing it would
|
||||
blank a stored body to say something we never learned.
|
||||
"""
|
||||
attrs = post.get("attributes") or {}
|
||||
title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
|
||||
@@ -406,9 +426,17 @@ class PatreonDownloader(BaseNativeDownloader):
|
||||
return PostRecordOutcome(
|
||||
path=None, post_type=post_type, title=title, body_chars=0,
|
||||
)
|
||||
if revisit:
|
||||
feed_body = post_body_html(attrs)
|
||||
if not (isinstance(feed_body, str) and feed_body.strip()):
|
||||
return PostRecordOutcome(
|
||||
path=None, post_type=post_type, title=title, body_chars=0,
|
||||
)
|
||||
post_dir = self.images_root / artist_slug / "patreon" / post_dir_name(post)
|
||||
post_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = self._write_sidecar_data(post, post_dir / "_post.json")
|
||||
path = self._write_sidecar_data(
|
||||
post, post_dir / "_post.json", detail_fetch=not revisit,
|
||||
)
|
||||
# _write_sidecar_data has by now memoized any detail-fetched body onto
|
||||
# post["attributes"]["content"], so re-read it for the FINAL char count.
|
||||
body = (post.get("attributes") or {}).get("content")
|
||||
|
||||
Reference in New Issue
Block a user