feat(agent): survive + auto-recover when curator is unreachable
For redeploying curator while away with nobody to restart the agent: - _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx, 401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job fault. On a transport error it hands the job back (best effort) and signals the loop to back off — instead of calling fail(), which would burn the job's server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't re-lease forever. - lease loop retries with capped exponential backoff (poll_idle → 60s) and resets on the first successful lease, so a long outage is gentle and recovery is automatic within ≤60s of curator returning. Sleeps are interruptible so Stop / pool-shrink stays responsive. - AUTO_START env (default on in compose) resumes the worker on container start, so a host reboot / crash-restart (restart: unless-stopped) self-heals with nobody at the desktop. - control UI shows a "waited out" counter + an "curator unreachable, holding work" banner so the recovering state reads as recovery, not failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -16,6 +16,8 @@ class Config:
|
||||
embed_dtype: str # torch dtype for the crop embedder: float16|float32
|
||||
embed_model_override: str # force a SigLIP-family model ("" → use the one
|
||||
# the server announces in the lease)
|
||||
auto_start: bool # start the worker pool on boot (so a container restart
|
||||
# resumes processing without anyone clicking Start)
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "Config":
|
||||
@@ -30,4 +32,5 @@ class Config:
|
||||
poll_idle_seconds=float(os.environ.get("POLL_IDLE_SECONDS", "10")),
|
||||
embed_dtype=os.environ.get("SIGLIP_DTYPE", "float16"),
|
||||
embed_model_override=os.environ.get("EMBED_MODEL_NAME", ""),
|
||||
auto_start=os.environ.get("AUTO_START", "").lower() in ("1", "true", "yes"),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user