feat: a tick keeps looking back 30 days, so an EDITED post is reached (4386)
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 40s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / extension-version (push) Successful in 3s
CI and images / lint (push) Successful in 3s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 40s
CI and images / integration (push) Failing after 2m17s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator, 2026-09-23, on a Floppystack post: "this post has been updated as he implements hot fixes — any chance we have a way to scan for or see updated posts so we can update ours to match and pull the new attachments and pictures etc." The download half already worked: extract_media reads the media list off the LIVE feed response every walk, so a newly attached hotfix build is a ledger key we have never seen. Only REACHING the post was missing — a tick stopped after 20 contiguous already-have-it items, and a post edited three days after publication sits well below twenty. Not a bug in the early-out; a count cannot express "recent". The early-out now needs BOTH conditions: the run of seen items AND a post published before the horizon. Strictly a widening — window 0 is exactly the old behaviour, and no window can make a tick stop EARLIER than it used to, so a source paused for months still walks its whole unseen backlog. The horizon is a floor on how far to look, never a ceiling. Inside the window the post-record gate is bypassed too (write_post_record revisit=True): the body is re-read from the feed response already in hand, so a revisit costs zero requests, and a body that comes back empty writes NOTHING rather than blanking one a detail-fetch had filled. Revisits are kept out of the #862 body-drift canary's sample for the same reason — an empty revisit is healthy, and counting it would walk the alarm toward firing on good ticks. The run summary names what changed ("3 post(s) updated (5 new file(s))") with a line per post; the ask was to SEE updated posts, not only to end up with their bytes. download_revisit_days is a settings row, not a constant (rule 25) — how long a creator keeps editing is a property of the creator. Default 30, 0 turns it off. Migration 0108. Also corrects two stale docstrings: both clients described post_meta as feeding an Ingester.preview that no longer calls it. It had no consumer at all until this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -658,6 +658,53 @@ def test_write_post_record_writes_enriched_post_only_sidecar(tmp_path):
|
||||
assert rec.body_chars == len("<p>text post body</p>")
|
||||
|
||||
|
||||
# The revisit contract (2026-09-23): a tick re-reading a post it already
|
||||
# captured, so an edit made after first capture reaches us. Both halves exist to
|
||||
# stop an update costing more than it is worth.
|
||||
|
||||
|
||||
def test_a_revisit_re_reads_the_body_without_paying_for_a_detail_fetch(tmp_path):
|
||||
"""A 30-day window would otherwise buy one detail GET per body-less post per
|
||||
tick, forever — a per-creator cost that grows with how prolific they are, to
|
||||
re-fetch a body we already stored."""
|
||||
calls: list[str] = []
|
||||
|
||||
def _fetcher(post_id: str) -> str:
|
||||
calls.append(post_id)
|
||||
return "<p>detail body</p>"
|
||||
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
session=_FakeSession(), content_fetcher=_fetcher,
|
||||
)
|
||||
post = _post()
|
||||
post["attributes"]["content"] = "<p>feed body</p>"
|
||||
rec = dl.write_post_record(post, "artist-x", revisit=True)
|
||||
|
||||
assert calls == []
|
||||
assert json.loads(rec.path.read_text())["content"] == "<p>feed body</p>"
|
||||
|
||||
|
||||
def test_a_revisit_with_an_empty_body_writes_nothing_at_all(tmp_path):
|
||||
"""The data-loss guard. On a FIRST capture an empty body is the truth about
|
||||
the post; on a revisit it usually means the body only ever came from the
|
||||
detail endpoint we just declined to call. Writing it would blank a stored
|
||||
body to say something we never learned."""
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
session=_FakeSession(), content_fetcher=lambda _pid: "<p>detail</p>",
|
||||
)
|
||||
post = _post()
|
||||
post["attributes"]["content"] = ""
|
||||
rec = dl.write_post_record(post, "artist-x", revisit=True)
|
||||
|
||||
assert rec.path is None
|
||||
assert rec.body_chars == 0
|
||||
# Not "wrote an empty file" — nothing was written, so a record captured on
|
||||
# an earlier walk is still exactly what it was.
|
||||
assert not list(tmp_path.rglob("_post.json"))
|
||||
|
||||
|
||||
def test_write_post_record_none_without_post_id(tmp_path):
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
|
||||
Reference in New Issue
Block a user