When a journaled model step fails, Restate correctly uses SdkInternalException for replay control flow. The wrapper then does:
except SdkInternalBaseException as e:
raise Exception("Internal error during model call") from e
This conflicts with Restate’s Python error-handling guidance (re-raise internal exceptions; don’t catch them as generic Exception). Integrators must walk __cause__ chains to recover semantics.
For LLM provider non-retryable 4xx (e.g. invalid tool schema → HTTP 400), the failure should be TerminalError inside the journaled step, not a retryable transient. Otherwise Restate retries permanently invalid requests—contrary to the AI integration guide (“handle terminal errors”, bound LLM retries).
Proposed behavior:
except SdkInternalBaseException as e:
if isinstance(e, SuspendedException):
raise
# Optional: inspect e.__cause__ for provider HTTP errors
if is_non_retryable_provider_4xx(e.__cause__):
raise TerminalError("Model request rejected by provider", status_code=400) from e.__cause__
raise # re-raise SdkInternalBaseException unchanged — do NOT wrap in Exception(...)
Policy suggestion for upstream defaults:
| Provider status |
Treatment |
| 400, 401, 403, 404, 422, … |
TerminalError (non-retryable) |
| 408, 429 |
Retryable (existing transient path) |
| 5xx |
Retryable per RunOptions |
Proposals:
- Stop wrapping SdkInternalBaseException in generic Exception.
- Either adopt a default provider-status policy or expose a hook on RestateAgent / RestateModelWrapper for classifying model failures before journaling.
- Document recommended RunOptions(max_attempts=…) for LLM journal steps.
When a journaled model step fails, Restate correctly uses SdkInternalException for replay control flow. The wrapper then does:
This conflicts with Restate’s Python error-handling guidance (re-raise internal exceptions; don’t catch them as generic Exception). Integrators must walk __cause__ chains to recover semantics.
For LLM provider non-retryable 4xx (e.g. invalid tool schema → HTTP 400), the failure should be TerminalError inside the journaled step, not a retryable transient. Otherwise Restate retries permanently invalid requests—contrary to the AI integration guide (“handle terminal errors”, bound LLM retries).
Proposed behavior:
Policy suggestion for upstream defaults:
Proposals: