From ea3ae4884dcd84d9c1d47eb03121c61c437f7f6e Mon Sep 17 00:00:00 2001 From: Ayushi Ahjolia Date: Mon, 31 Aug 2026 16:28:02 -0700 Subject: [PATCH 1/3] docs: add 1.x to 2.x migration guide --- docs/migration-1.x-to-2.x.md | 367 ++++++++++++++++++ .../aws_durable_execution_sdk_python/waits.py | 12 +- 2 files changed, 378 insertions(+), 1 deletion(-) create mode 100644 docs/migration-1.x-to-2.x.md diff --git a/docs/migration-1.x-to-2.x.md b/docs/migration-1.x-to-2.x.md new file mode 100644 index 00000000..3b8ad36b --- /dev/null +++ b/docs/migration-1.x-to-2.x.md @@ -0,0 +1,367 @@ +# Migrating from 1.x to 2.x + +`2.x` is a breaking major release. Every change is a bug fix or brings Python to +parity with the JavaScript and Java SDKs. The changes most likely to touch your +code are the typed, per-operation **error hierarchy**, the +**serialize/deserialize round trip on the first run**, and the new fail-fast +**default `map` / `parallel` completion** (which completes after the first +observed failure and stops scheduling pending items; already-started items are +not cancelled, so with unlimited concurrency all items may already be running - +set `max_concurrency` if you need to bound that). Opt back into process-all with +`CompletionConfig.all_completed()`. + +There is no compatibility shim: removed names (for example `CallableRuntimeError`) +are gone with no alias. If you are not ready to migrate, stay on `1.x`. + +> Instrumentation plugins are out of scope here. The experimental `plugins=` hook +> already existed in `1.x`; `2.x` adds opt-in auto-discovery and `PluginLoadError`. +> The plugin interface itself also changed incompatibly (hook signatures changed, +> enums moved, and `InvocationEndInfo.status` is now required), so plugin authors +> should expect to update. Because plugins are opt-in and still evolving, they are +> documented with the plugin/OpenTelemetry feature rather than in this guide. + +## Porting to 2.0 + +Each change below lists what changed and what you must do. The ones most likely +to touch your code are the [typed error hierarchy](#callableruntimeerror-and-friends-removed), +the [first-run serialize/deserialize round trip](#first-run-serializedeserialize-round-trip), +and the [fail-fast `map` / `parallel` default](#completionconfigall_completed-tolerates-all-failures). +See [Finding affected code](#finding-affected-code) for a grep checklist. + +### `CallableRuntimeError` and friends removed + +`CallableRuntimeError`, `UserlandError`, and +`CallableRuntimeErrorSerializableDetails` are gone; typed per-operation errors +replace them. + +Catch `StepError`, `InvokeError`, `ChildContextError`, or `WaitForConditionError` +(or the base `DurableOperationError`) instead of `CallableRuntimeError`. See +[Error handling](#error-handling-the-biggest-change) for the full hierarchy. + +### `CallbackError` moved out of the termination tree + +`CallbackError` is no longer a termination reason, and graded subtypes were +added. + +Remove any `termination_reason == TerminationReason.CALLBACK_ERROR` check (the +enum member is gone). Optionally catch `CallbackTimeoutError`, +`CallbackExternalError`, or `CallbackSubmitterError`. See +[Callbacks](#callbacks). + +### `BatchResult.throw_if_error()` raises typed errors + +It no longer raises `CallableRuntimeError`. + +Replace `except CallableRuntimeError` with `ChildContextError` (ordinary item +failure), `SerDesError` (item serialize/deserialize failure), and +`BatchCompletionError` (custom `should_complete` failed the batch with no item +error). `ChildContextError` and `BatchCompletionError` share the base +`DurableOperationError`, but `SerDesError` does not, so catch +`(DurableOperationError, SerDesError)` or list all three: + +```python +# 1.x +except CallableRuntimeError: + ... + +# 2.x +except (DurableOperationError, SerDesError): + ... +``` + +See [map / parallel](#map--parallel) for a full example. + +### Serdes failures surface as `SerDesError`, not `ExecutionError` + +`SerDesError` is a direct child of `DurableExecutionsError`, not `ExecutionError`. + +If you caught serdes failures with `except ExecutionError`, catch `SerDesError` +(or `DurableExecutionsError`) instead: + +```python +# 1.x +except ExecutionError: + ... + +# 2.x +except SerDesError: + ... +``` + +### First-run serialize/deserialize round trip + +`step`, child contexts, `map`/`parallel`, and `wait_for_condition` now round-trip +their result through the serdes on the first run, returning +`deserialize(serialize(x))` - the same canonical value replay returns. + +If you relied on the raw pre-serialization object, use the deserialized shape +instead (or make your `SerDes` round-trip identity). Ensure `wait_for_condition` +`initial_state` is serializable by the configured serdes. For a transient serdes +failure, raise the new `RetryableSerDesError` (retries) instead of `SerDesError` +(permanent). See [Serialize/Deserialize round trip](#serializedeserialize-round-trip). + +### Empty-string payloads are preserved + +`1.x` dropped empty-string (`""`) payloads when serializing; `2.x` keeps them. + +A step, invoke, or child result of `""` that surfaced as `None` (dropped payload) +in `1.x` now surfaces as `""`. If you treated an absent payload as `None`, handle +`""` explicitly. + +### `InvokeConfig.timeout` / `timeout_seconds` removed + +Both fields are gone. + +Remove them. Enforce any timeout inside the invoked function or as a separate +timer. + +### `map` / child batching names removed + +Removed: `ItemBatcher`, `ItemsPerBatchUnit`, `BatchedInput`, `TerminationMode`, +`StepFuture`, `MapConfig.item_batcher`, `ChildConfig.item_serdes`; also +`ChainedInvokeFailedToStartType`, `ChainedInvokeTimeoutType`, and +`ChainedInvokeStopType` (from `lambda_service`). + +Remove all uses. Replace `ChildConfig.item_serdes` with `ChildConfig.serdes`. + +### Config validation moved to construction and call time + +`MapConfig`, `ParallelConfig`, and `CompletionConfig` now validate arguments at +construction (for example `max_concurrency=0` or `min_successful=0` raise +`ValidationError`). `min_successful > total` is validated at the +`map()`/`parallel()` call, not at construction. + +Wrap config construction, and the `map()`/`parallel()` call, in +`try/except ValidationError` when inputs are external. + +### `CompletionConfig.all_completed()` tolerates all failures + +It now actually tolerates every failure, and the default `map` / `parallel` +completion config is now fail-fast. + +If you hand-built the old all-`None` config, use the factory instead. To preserve +1.x process-all behavior, pass it explicitly: + +```python +# 2.x - keep processing every item even when some fail +MapConfig(completion_config=CompletionConfig.all_completed()) +ParallelConfig(completion_config=CompletionConfig.all_completed()) +``` + +### `BatchResult.all` omits never-started branches + +`total_count` and positional iteration differ for early-completed batches, +because never-started branches are no longer included. + +If you index `.all` by original position or expect `total_count` to include +unstarted branches, update that logic. + +### `summary_generator` output moved into an envelope + +For `map`/`parallel`, a custom `summary_generator` output is now stored under a +`"summary"` key in an SDK-owned envelope; it no longer replaces the checkpoint +payload. `ChildConfig.summary_generator` is unchanged: its output is still +checkpointed verbatim. + +If you parse `map`/`parallel` summary payloads from execution history, read the +`"summary"` key from the envelope. Child-context summary consumers need no change. + +### `WaitDecision` and wait timeouts removed + +`WaitDecision` is gone, along with `WaitStrategyConfig.timeout` and +`WaitStrategyConfig.timeout_seconds`. + +Use `WaitForConditionDecision` (`stop_polling()` / `continue_waiting(delay)`). + +### `wait_for_condition` raises on exhaustion + +It raises `WaitForConditionError` when it exhausts `max_attempts`. + +Catch `WaitForConditionError` instead of inspecting the returned state. + +### Finding affected code + +Grep for the removed and changed names before upgrading: + +```bash +rg -n "CallableRuntimeError|UserlandError|CallableRuntimeErrorSerializableDetails" . +rg -n "CallbackError|CALLBACK_ERROR" . +rg -n "InvokeConfig\(|\.timeout_seconds" . +rg -n "WaitDecision|WaitStrategyConfig\(|item_batcher|ItemBatcher|ItemsPerBatchUnit" . +rg -n "TerminationMode|BatchedInput|StepFuture|ChildConfig\(" . +rg -n "ChainedInvoke|except ExecutionError" . +``` + +## Error Handling (the biggest change) + +In `1.x` nearly every user-land failure surfaced as one `CallableRuntimeError`, +so a failed step was indistinguishable from a failed invoke or child branch. `2.x` +raises a specific type per operation, all under a new base `DurableOperationError`. +Inspect the failure through its fields, not `__cause__`: `error_type`, `message`, +`data`, and `stack_trace`. Do not rely on `__cause__` being the original +exception - the SDK reconstructs a `DurableOperationError` stand-in carrying +those same fields (on both the first run and replay, for determinism), so the +original type is not preserved (a `ValueError` does not stay a `ValueError`) and +custom attributes are lost. + +For `StepError`, `InvokeError`, `ChildContextError`, and `WaitForConditionError`, +`error_type` is the name of the error that escaped your code (e.g. `"ValueError"`). +The graded callback errors are different: `CallbackTimeoutError`, +`CallbackExternalError`, and `CallbackSubmitterError` are constructed without the +originating `error_type`, so `error_type` is the callback class name, not the +underlying cause. Use the specific callback exception type (and `message` / +`data` / `stack_trace`) to distinguish those. + +```python +# 1.x +from aws_durable_execution_sdk_python.exceptions import CallableRuntimeError +try: + result = context.step(charge_card, name="charge") +except CallableRuntimeError as e: + context.logger.error("something failed: %s", e.message) + +# 2.x +from aws_durable_execution_sdk_python import StepError, DurableOperationError +try: + result = context.step(charge_card, name="charge") +except StepError as e: # or `except DurableOperationError` to catch any operation + context.logger.error("charge step failed: %s", e.message) +``` + +New types, all exported from the package root: `DurableOperationError` (base), +`StepError`, `InvokeError`, `ChildContextError`, `WaitForConditionError`, +`CallbackError` (+ `CallbackExternalError`, `CallbackTimeoutError`, +`CallbackSubmitterError`), plus `SerDesError` (now exported) and +`RetryableSerDesError`. `SerDesError` stays a direct child of +`DurableExecutionsError`; `RetryableSerDesError` is a retryable `InvocationError`. + +### Callbacks + +`context.wait_for_callback(...)` returns the payload directly and raises the +callback error from the call itself (there is no `callback.result()`): + +```python +from aws_durable_execution_sdk_python import ( + CallbackError, CallbackTimeoutError, CallbackSubmitterError, +) +try: + payload = context.wait_for_callback(submit_approval, name="approval") +except CallbackTimeoutError: + ... # timeout / heartbeat expiry +except CallbackSubmitterError: + ... # the submitter step failed +except CallbackError as e: # external + internal + context.logger.error("callback failed: %s", e.message) +``` + +### map / parallel + +`throw_if_error()` can raise three types: `ChildContextError` for the ordinary +item/branch failure, `SerDesError` if an item result failed to serialize or +deserialize, and `BatchCompletionError` when a custom `should_complete` predicate +marked the batch failed with no failed item (see the completion-predicate section +below). `ChildContextError` and `BatchCompletionError` share the base +`DurableOperationError`, but `SerDesError` does not, so catch +`(DurableOperationError, SerDesError)` or list all three explicitly. + +```python +from aws_durable_execution_sdk_python import ( + ChildContextError, SerDesError, BatchCompletionError, +) + +result = context.map(items, process_item) +try: + result.throw_if_error() +except (ChildContextError, SerDesError, BatchCompletionError): + for err in result.get_errors(): # every failed item's ErrorObject + context.logger.error("%s: %s", err.type, err.message) +``` + +## Serialize/Deserialize Round Trip + +`1.x` returned the raw in-memory result on the first run but the deserialized +result on replay, so a non-identity custom `SerDes` produced different values. +`2.x` round-trips (`serialize` then `deserialize`) on the first run for `step`, +child contexts, `map`/`parallel`, and `wait_for_condition` (which also feeds the +deserialized state to the wait strategy). This makes first-run behavior match +replay for deterministic, reversible custom serdes. If your code depended on +the raw pre-serialization object on the first run, switch to the deserialized +shape (or make the serdes round-trip identity). This also surfaces genuine +serialization bugs on the first run instead of later on replay. + +`invoke` and `wait` are unaffected. `wait_for_callback` is implemented via a +child context, so its result is serialized and deserialized before it returns: +do not rely on callback-result object identity. The enclosing child context uses +the default (extended-type) serdes, not `WaitForCallbackConfig.serdes`, so the +value your callback deserializer returns must itself be serializable by the +default serdes; otherwise the child raises `SerDesError`. + +`wait_for_condition` also round-trips `initial_state` through the serdes before +the first check, so `initial_state` must now be serializable by the configured +serdes. Custom serdes should serialize polling state to a non-empty string; an +empty string checkpoint payload is currently treated as no stored polling state +on resume, so the operation may restart from `initial_state`. + +## New in 2.x: Custom Completion Predicate (Optional) + +`2.x` adds a `should_complete` predicate to `CompletionConfig`, giving `map` and +`parallel` full control over when a batch completes early. This is a new feature, +not a breaking change - no action is required unless you adopt it. + +```python +from aws_durable_execution_sdk_python import complete_batch, continue_batch +from aws_durable_execution_sdk_python.config import CompletionConfig + +config = CompletionConfig( + should_complete=lambda status: ( + complete_batch() if status.success_count >= 2 else continue_batch() + ) +) +``` + +The predicate receives a `CompletionStatus` snapshot (counts plus per-item +statuses) and returns a `CompletionDecision` - `continue_batch()`, or +`complete_batch(CompletionOutcome.SUCCEEDED)` / `complete_batch(CompletionOutcome.FAILED)` +(the outcome defaults to `SUCCEEDED`). A `FAILED` outcome marks the whole batch +failed; `throw_if_error()` then raises `BatchCompletionError` (a +`DurableOperationError` subtype) even when no individual item failed. Individual +item/branch failures still surface as `ChildContextError`. Notes: + +- It cannot be combined with `min_successful` or the `tolerated_failure_*` + fields; doing so raises `ValidationError` at construction. +- The predicate runs before any branch is scheduled (`completed_count == 0`) + and on suspension state changes. At those points, unscheduled item statuses + are `None`; handle missing statuses explicitly so the predicate does not + fail the batch or complete it before useful work starts. +- The predicate must be deterministic, side-effect-free, and monotonic: once a + progress snapshot returns `complete_batch(outcome)`, every later snapshot + containing that progress must return `complete_batch(outcome)` with the same + `CompletionOutcome`. Replaying an already-completed batch uses the + checkpointed decision, but a mid-run resume re-runs the batch live and + re-evaluates the predicate as completed branches replay, possibly in a + different order. +- New exports: `complete_batch`, `continue_batch`, `CompletionStatus`, + `CompletionDecision`, `CompletionOutcome`, `CompletionItemStatus`, + `BatchItemStatus`, `BatchCompletionError`. + +## New in 2.x: Attempt Number in Contexts (Optional) + +`StepContext` and `WaitForConditionCheckContext` now expose an `attempt` field +(the current attempt number, starting at 1). Read it inside a step or a +`wait_for_condition` check to branch on the retry count. The SDK injects these +contexts, so normal usage needs no change. But `attempt` is a required +dataclass field with no default: if you construct these contexts directly (in +tests, fixtures, or wrappers), you must now pass `attempt` or construction fails +with `TypeError`. + +## Recommended Validation After Upgrading + +1. Build and run your test suite against `2.x`, and grep for the removed names above. +2. Trigger a failure in a `step`, an `invoke`, and a `map`/`parallel` branch; + confirm you catch `StepError`, `InvokeError`, and `ChildContextError`. +3. Exercise a `wait_for_callback` timeout and a submitter-step failure + (`CallbackTimeoutError`, `CallbackSubmitterError`). +4. Exercise a `wait_for_condition` that exhausts its attempts (`WaitForConditionError`). +5. If you use a custom `SerDes`, run a workflow that checkpoints a result, an + error payload, and `wait_for_condition` polling state; confirm first-run + output equals replay output. diff --git a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py index 3753a1be..b191d201 100644 --- a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py +++ b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py @@ -99,7 +99,17 @@ def wait_strategy(result: T, attempts_made: int) -> WaitForConditionDecision: @dataclass(frozen=True) class WaitForConditionConfig(Generic[T]): - """Configuration for wait_for_condition.""" + """Configuration for wait_for_condition. + + Attributes: + wait_strategy: Called after each poll with (state, attempts_made) and + returns a WaitForConditionDecision (continue_waiting or stop_polling). + initial_state: State passed to the first poll. It is round-tripped + through serdes (serialize then deserialize) before the first check, + so it must be serializable by the configured serdes. + serdes: SerDes used to serialize and deserialize the polled state at + each checkpoint. Defaults to the SDK's extended-type serdes when None. + """ wait_strategy: Callable[[T, int], WaitForConditionDecision] initial_state: T From c26de4d91a7af99c4b0b458a751e35497de6afc3 Mon Sep 17 00:00:00 2001 From: yaythomas Date: Thu, 3 Sep 2026 22:30:06 +0000 Subject: [PATCH 2/3] docs: finalize 1.x to 2.0 migration guide - Rename to migration-1.x-to-2.0.md; name the release 2.0, not 2.x - Add a Why upgrade to 2.0 section listing only 2.0 additions - Rewrite in active voice; remove filler and passive constructions - Rename batching heading to Removed 1.x-only names - Add max_concurrency in-flight semantics section - Add replay operation identity validation (#698) - Add wait_for_condition unreadable-polling-state failure - Scope RetryableSerDesError guidance to invocation replay and step semantics; scope exhaustion to create_wait_strategy - Correct callbacks note: create_callback().result() still exists - Move the experimental plugin note to its own linked section - Mark the grep checklist as non-exhaustive --- docs/migration-1.x-to-2.0.md | 451 +++++++++++++++++++++++++++++++++++ docs/migration-1.x-to-2.x.md | 367 ---------------------------- 2 files changed, 451 insertions(+), 367 deletions(-) create mode 100644 docs/migration-1.x-to-2.0.md delete mode 100644 docs/migration-1.x-to-2.x.md diff --git a/docs/migration-1.x-to-2.0.md b/docs/migration-1.x-to-2.0.md new file mode 100644 index 00000000..a966f763 --- /dev/null +++ b/docs/migration-1.x-to-2.0.md @@ -0,0 +1,451 @@ +# Migrating from 1.x to 2.0 + +`2.0` is a major release. It contains breaking changes. The changes +most likely to touch your code are the typed, per-operation +[error hierarchy](#error-handling), the +[first-run serialize and deserialize round trip](#serialize-and-deserialize-round-trip), +and the new fail-fast [default `map` and `parallel` completion](#completionconfigall_completed-tolerates-all-failures). + +## Why upgrade to 2.0 + +- **Typed errors tell you which operation failed.** `1.x` collapsed almost + every failure into one `CallableRuntimeError`. `2.0` raises `StepError`, + `InvokeError`, `ChildContextError`, `WaitForConditionError`, or a graded + `CallbackError` subtype, so you branch on the operation that failed instead of + parsing a message. See [Error handling](#error-handling). +- **`should_complete` gives `map` and `parallel` custom completion.** A + predicate decides when a batch finishes early, so you express quorum and + dependency rules directly. See + [Custom completion predicate](#custom-completion-predicate). +- **First-run output matches replay.** A step returns the same value the first + time it runs and on every replay, so a non-identity custom serdes no longer + produces two different values. See + [Serialize and deserialize round trip](#serialize-and-deserialize-round-trip). +- **`attempt` is available inside steps and condition checks.** Read the current + attempt number to branch on retry count. See + [Attempt number in contexts](#attempt-number-in-contexts). +- **Replay catches non-determinism early.** The SDK validates each operation's + identity against its checkpoint on replay and fails fast on drift instead of + consuming the wrong checkpoint. See + [Replay validates operation identity](#replay-validates-operation-identity). + +## Porting to 2.0 + +The changes +most likely to touch your code are the +[typed error hierarchy](#callableruntimeerror-and-friends-removed), the +[first-run round trip](#first-run-serialize-and-deserialize-round-trip), and the +[fail-fast `map` and `parallel` default](#completionconfigall_completed-tolerates-all-failures). +[Finding affected code](#finding-affected-code) lists a grep checklist. + +### `CallableRuntimeError` and friends removed + +`2.0` removes `CallableRuntimeError`, `UserlandError`, and +`CallableRuntimeErrorSerializableDetails`, and replaces them with typed +per-operation errors. + +Catch `StepError`, `InvokeError`, `ChildContextError`, or `WaitForConditionError` +(or the base `DurableOperationError`) instead of `CallableRuntimeError`. +[Error handling](#error-handling) describes the full hierarchy. + +### `CallbackError` moved out of the termination tree + +`CallbackError` no longer names a termination reason, and `2.0` adds graded +subtypes. + +Remove any `termination_reason == TerminationReason.CALLBACK_ERROR` check, +because the enum member no longer exists. Catch `CallbackTimeoutError`, +`CallbackExternalError`, or `CallbackSubmitterError` when you need the specific +mode. See [Callbacks](#callbacks). + +### `BatchResult.throw_if_error()` raises typed errors + +`throw_if_error()` no longer raises `CallableRuntimeError`. + +Replace `except CallableRuntimeError` with three types: `ChildContextError` for +an ordinary item failure, `SerDesError` for an item that failed to serialize or +deserialize, and `BatchCompletionError` when a custom `should_complete` predicate +fails the batch with no item error. `ChildContextError` and `BatchCompletionError` +share the base `DurableOperationError`, but `SerDesError` does not, so catch +`(DurableOperationError, SerDesError)` or list all three. + +```python +# 1.x +except CallableRuntimeError: + ... + +# 2.0 +except (DurableOperationError, SerDesError): + ... +``` + +[map and parallel](#map-and-parallel) shows a full example. + +### Serdes failures surface as `SerDesError`, not `ExecutionError` + +`SerDesError` descends directly from `DurableExecutionsError`, not from +`ExecutionError`. + +Catch `SerDesError` (or `DurableExecutionsError`) where you caught +`ExecutionError` for a serdes failure. + +```python +# 1.x +except ExecutionError: + ... + +# 2.0 +except SerDesError: + ... +``` + +### First-run serialize and deserialize round trip + +`step`, child contexts, `map`, `parallel`, and `wait_for_condition` now +round-trip their result through the serdes on the first run and return +`deserialize(serialize(x))`, the same canonical value replay returns. + +Switch to the deserialized shape if your code depended on the raw +pre-serialization object, or make your `SerDes` round-trip to an identical value. +`wait_for_condition` also round-trips `initial_state`, so the configured serdes +must serialize it. Raise the new `RetryableSerDesError` for a transient serdes +failure, which replays the invocation. Read +[Serialize and deserialize round trip](#serialize-and-deserialize-round-trip) +before you adopt `RetryableSerDesError`, because an invocation replay re-runs a +non-idempotent step body. + +### 2.0 preserves empty-string payloads + +`1.x` dropped an empty-string (`""`) payload when it serialized. `2.0` keeps it. + +A step, invoke, or child result of `""` that surfaced as `None` in `1.x` now +surfaces as `""`. Handle `""` explicitly if you treated an absent payload as +`None`. + +### `InvokeConfig.timeout` and `timeout_seconds` removed + +`2.0` removes both fields. + +Remove them. Enforce a timeout inside the invoked function or with a separate +timer. + +### Removed 1.x-only names + +`2.0` removes `ItemBatcher`, `ItemsPerBatchUnit`, `BatchedInput`, +`TerminationMode`, `StepFuture`, `MapConfig.item_batcher`, and +`ChildConfig.item_serdes`. It also removes `ChainedInvokeFailedToStartType`, +`ChainedInvokeTimeoutType`, and `ChainedInvokeStopType` from `lambda_service`. + +Remove all uses. Replace `ChildConfig.item_serdes` with `ChildConfig.serdes`. + +### Config validation moved to construction and call time + +`MapConfig`, `ParallelConfig`, and `CompletionConfig` now validate their +arguments at construction. `max_concurrency=0` and `min_successful=0` raise +`ValidationError`. The `map()` and `parallel()` call validates +`min_successful > total`, because that check needs the item count. + +Wrap config construction, and the `map()` or `parallel()` call, in +`try/except ValidationError` when the inputs come from outside your code. + +### `max_concurrency` bounds in-flight branches + +`1.x` used `max_concurrency` to cap worker threads. `2.0` caps in-flight +branches. A branch that suspends while it awaits an invoke result or a callback +holds its slot until it reaches a terminal state, and a new branch starts only +when a slot frees up. When every in-flight branch suspends and no slot is free, +the parent suspends too. + +Revisit the value if you sized `max_concurrency` around thread count rather than +concurrent in-flight work. + +### `CompletionConfig.all_completed()` tolerates all failures + +`all_completed()` now tolerates every failure, and the default `map` and +`parallel` completion config runs fail-fast: the batch completes after the first +observed failure and stops scheduling pending items. The SDK does not cancel +already-started items, so with unlimited concurrency every item may already be +running. Set `max_concurrency` to bound that. + +Call the factory instead of hand-building the old all-`None` config. Pass +`all_completed()` explicitly to keep the `1.x` process-all behavior. + +```python +# 2.0: keep processing every item even when some fail +MapConfig(completion_config=CompletionConfig.all_completed()) +ParallelConfig(completion_config=CompletionConfig.all_completed()) +``` + +### `BatchResult.all` omits never-started branches + +`BatchResult.all` no longer includes never-started branches, so `total_count` +and positional iteration differ for an early-completed batch. + +Update any logic that indexes `.all` by original position or expects +`total_count` to include unstarted branches. + +### `summary_generator` output moved into an envelope + +For `map` and `parallel`, the SDK now stores a custom `summary_generator` output +under a `"summary"` key inside an SDK-owned envelope instead of using it as the +checkpoint payload. `ChildConfig.summary_generator` behaves as before and +checkpoints its output verbatim. + +Read the `"summary"` key from the envelope if you parse `map` or `parallel` +summary payloads from the execution history. A child-context summary consumer +needs no change. + +### `WaitDecision` and wait timeouts removed + +`2.0` removes `WaitDecision`, `WaitStrategyConfig.timeout`, and +`WaitStrategyConfig.timeout_seconds`. + +Use `WaitForConditionDecision` (`stop_polling()` or `continue_waiting(delay)`). + +### `create_wait_strategy` raises on exhaustion + +A strategy from `create_wait_strategy(WaitStrategyConfig(...))` raises +`WaitForConditionError` when it exhausts `max_attempts`. `WaitForConditionConfig` +has no `max_attempts` of its own, so a custom wait strategy must enforce its own +limit or it polls indefinitely. + +Catch `WaitForConditionError` instead of inspecting the returned state. + +### Replay validates operation identity + +On replay, the SDK validates each operation's checkpoint against the current +code by `type`, `sub_type`, `name`, and `parent_id`. Any mismatch raises +`NonDeterministicExecutionError` and fails the execution. `1.x` let a renamed or +reordered operation consume a neighboring checkpoint silently. `2.0` fails fast. +A `map` or `parallel` batch also validates FLAT against NESTED nesting drift. + +`NonDeterministicExecutionError` descends from `ExecutionError`, so it is +unrecoverable and you should not catch it. Treat this change as a deployment +constraint rather than a code change. Do not rename an operation (the `name=` +argument, or a step function's name when you omit `name`), change a batch's +`nesting_type`, or reparent an operation while executions are in flight. Drain +in-flight executions before you deploy such a change. + +### `wait_for_condition` fails on unreadable polling state + +When a custom serdes fails to deserialize checkpointed polling state on a +resumed invocation, `2.0` fails the operation with a typed error. `1.x` caught +the failure and silently restarted from `initial_state`, so the loop could +succeed with a result computed from the wrong state. + +If your custom serdes can fail while restoring state, expect the operation to +fail instead of restarting. + +### Finding affected code + +Grep for the removed and changed names before you upgrade. This list covers the +renamed and removed identifiers, not the behavioral changes above, so treat it as +a starting point rather than a complete check: + +```bash +rg -n "CallableRuntimeError|UserlandError|CallableRuntimeErrorSerializableDetails" . +rg -n "CallbackError|CALLBACK_ERROR" . +rg -n "InvokeConfig\(|\.timeout_seconds" . +rg -n "WaitDecision|WaitStrategyConfig\(|item_batcher|ItemBatcher|ItemsPerBatchUnit" . +rg -n "TerminationMode|BatchedInput|StepFuture|ChildConfig\(" . +rg -n "ChainedInvoke|except ExecutionError" . +``` + +## Error Handling + +In `1.x` nearly every user-land failure surfaced as one `CallableRuntimeError`, +so a failed step looked identical to a failed invoke or child branch. `2.0` +raises a specific type per operation under a new base `DurableOperationError`. +Inspect the failure through its fields, `error_type`, `message`, `data`, and +`stack_trace`, not `__cause__`. The SDK reconstructs a `DurableOperationError` +stand-in that carries those fields on both the first run and replay, so +`__cause__` does not hold the original exception. A `ValueError` does not stay a +`ValueError`, and custom attributes do not survive. + +For `StepError`, `InvokeError`, `ChildContextError`, and `WaitForConditionError`, +`error_type` holds the name of the error that escaped your code, for example +`"ValueError"`. The graded callback errors work differently. The SDK constructs +`CallbackTimeoutError`, `CallbackExternalError`, and `CallbackSubmitterError` +without the originating `error_type`, so `error_type` holds the callback class +name rather than the underlying cause. Branch on the specific callback exception +type, and read `message`, `data`, and `stack_trace`, to tell those apart. + +```python +# 1.x +from aws_durable_execution_sdk_python.exceptions import CallableRuntimeError +try: + result = context.step(charge_card, name="charge") +except CallableRuntimeError as e: + context.logger.error("something failed: %s", e.message) + +# 2.0 +from aws_durable_execution_sdk_python import StepError, DurableOperationError +try: + result = context.step(charge_card, name="charge") +except StepError as e: # or `except DurableOperationError` for any operation + context.logger.error("charge step failed: %s", e.message) +``` + +The package root exports every new type: `DurableOperationError` (base), +`StepError`, `InvokeError`, `ChildContextError`, `WaitForConditionError`, +`CallbackError` (with `CallbackExternalError`, `CallbackTimeoutError`, and +`CallbackSubmitterError`), and now `SerDesError` and `RetryableSerDesError`. +`SerDesError` descends directly from `DurableExecutionsError`. +`RetryableSerDesError` is a retryable `InvocationError`. + +### Callbacks + +`context.wait_for_callback(...)` returns the payload directly, so do not call +`.result()` on its return value. It raises the callback error from the call +itself. The separate `context.create_callback()` API still returns a `Callback` +whose `.result()` waits for completion, so keep that wait in a manual callback +flow. + +```python +from aws_durable_execution_sdk_python import ( + CallbackError, CallbackTimeoutError, CallbackSubmitterError, +) +try: + payload = context.wait_for_callback(submit_approval, name="approval") +except CallbackTimeoutError: + ... # timeout or heartbeat expiry +except CallbackSubmitterError: + ... # the submitter step failed +except CallbackError as e: # external and internal + context.logger.error("callback failed: %s", e.message) +``` + +### map and parallel + +`throw_if_error()` raises one of three types: `ChildContextError` for an ordinary +item or branch failure, `SerDesError` for an item that failed to serialize or +deserialize, and `BatchCompletionError` when a custom `should_complete` predicate +fails the batch with no failed item (see +[Custom completion predicate](#custom-completion-predicate)). `ChildContextError` +and `BatchCompletionError` share the base `DurableOperationError`, but +`SerDesError` does not, so catch `(DurableOperationError, SerDesError)` or list +all three. + +```python +from aws_durable_execution_sdk_python import ( + ChildContextError, SerDesError, BatchCompletionError, +) + +result = context.map(items, process_item) +try: + result.throw_if_error() +except (ChildContextError, SerDesError, BatchCompletionError): + for err in result.get_errors(): # every failed item's ErrorObject + context.logger.error("%s: %s", err.type, err.message) +``` + +## Serialize and Deserialize Round Trip + +`1.x` returned the raw in-memory result on the first run but the deserialized +result on replay, so a non-identity custom `SerDes` returned two different +values. `2.0` round-trips the result, running `serialize` then `deserialize`, on +the first run for `step`, child contexts, `map`, `parallel`, and +`wait_for_condition`, which also feeds the deserialized state to the wait +strategy. First-run output now matches replay for a deterministic, reversible +custom serdes. Switch to the deserialized shape if your code depended on the raw +pre-serialization object, or make the serdes round-trip to an identical value. +The round trip also surfaces a genuine serialization bug on the first run rather +than later on replay. + +`invoke` and `wait` do not round-trip. `wait_for_callback` runs in a child +context, so the child context serializes and deserializes its result before it +returns. Do not rely on callback-result object identity. That child context uses +the default extended-type serdes, not `WaitForCallbackConfig.serdes`, so the +default serdes must serialize whatever your callback deserializer returns, or the +child raises `SerDesError`. + +`wait_for_condition` round-trips `initial_state` through the serdes before the +first check, so the configured serdes must serialize `initial_state`. On a +resumed invocation, a serdes that fails to deserialize the stored polling state +fails the operation. `1.x` silently restarted from `initial_state` in that case. +A custom serdes should serialize polling state to a non-empty string. The SDK +currently treats an empty-string checkpoint payload as no stored polling state +on resume, so the operation can restart from `initial_state`. + +`RetryableSerDesError` does more than retry serialization. An executor re-raises +it before it writes a success checkpoint, so the backend re-invokes the whole +execution. An `AT_LEAST_ONCE_PER_RETRY` step re-runs its body and repeats its +side effects on that re-invocation, so raise `RetryableSerDesError` only from an +idempotent step or accept the duplicate work. An `AT_MOST_ONCE_PER_RETRY` step +already wrote its START checkpoint, so the retried invocation treats the step as +interrupted and applies the step retry strategy instead of re-running the body. +Raise `SerDesError` for a permanent failure. + +## Custom Completion Predicate + +`2.0` adds a `should_complete` predicate to `CompletionConfig` that gives `map` +and `parallel` control over when a batch completes early. This is a new feature, +not a breaking change, and requires no action unless you adopt it. + +```python +from aws_durable_execution_sdk_python import complete_batch, continue_batch +from aws_durable_execution_sdk_python.config import CompletionConfig + +config = CompletionConfig( + should_complete=lambda status: ( + complete_batch() if status.success_count >= 2 else continue_batch() + ) +) +``` + +The predicate receives a `CompletionStatus` snapshot, which holds the counts and +the per-item statuses, and returns a `CompletionDecision`. Return +`continue_batch()` to keep going, `complete_batch(CompletionOutcome.SUCCEEDED)` +to finish successfully, or `complete_batch(CompletionOutcome.FAILED)` to finish +with a failure. The outcome defaults to `SUCCEEDED`. A `FAILED` outcome marks the +whole batch failed, and `throw_if_error()` then raises `BatchCompletionError`, a +`DurableOperationError` subtype, even when no individual item failed. An +individual item or branch failure still surfaces as `ChildContextError`. + +Three constraints govern the predicate: + +- It cannot combine with `min_successful` or the `tolerated_failure_*` fields. + Combining them raises `ValidationError` at construction. +- It runs before the SDK schedules any branch (`completed_count == 0`) and on + each suspension state change. At those points an unscheduled item has status + `None`, so handle a missing status explicitly, or the predicate can fail the + batch or complete it before useful work starts. +- It must stay deterministic, side-effect-free, and monotonic. Once a progress + snapshot returns `complete_batch(outcome)`, every later snapshot that contains + that progress must return `complete_batch(outcome)` with the same + `CompletionOutcome`. Replaying an already-completed batch uses the checkpointed + decision, but a mid-run resume re-runs the batch live and re-evaluates the + predicate as completed branches replay, possibly in a different order. + +The package root exports `complete_batch`, `continue_batch`, `CompletionStatus`, +`CompletionDecision`, `CompletionOutcome`, `CompletionItemStatus`, +`BatchItemStatus`, and `BatchCompletionError`. + +## Attempt Number in Contexts + +`StepContext` and `WaitForConditionCheckContext` now expose an `attempt` field, +the current attempt number starting at 1. Read it inside a step or a +`wait_for_condition` check to branch on the retry count. The SDK injects these +contexts, so normal usage needs no change. `attempt` is a required dataclass +field with no default, so a context you construct directly, in a test, fixture, +or wrapper, must now pass `attempt` or construction raises `TypeError`. + +## Instrumentation plugins + +Instrumentation plugins remain experimental in this release, and the plugin +interface changed incompatibly. See the +[observability and plugin documentation](https://docs.aws.amazon.com/durable-execution/sdk-reference/observability/logging/) +for details. + +## Recommended Validation After Upgrading + +1. Build your project against `2.0`, run your test suite, and grep for the + removed names above. +2. Fail a `step`, an `invoke`, and a `map` or `parallel` branch, and confirm you + catch `StepError`, `InvokeError`, and `ChildContextError`. +3. Time out a `wait_for_callback` and fail its submitter step, and confirm you + catch `CallbackTimeoutError` and `CallbackSubmitterError`. +4. Exhaust a `wait_for_condition` and confirm you catch `WaitForConditionError`. +5. Run a workflow that checkpoints a result, an error payload, and + `wait_for_condition` polling state through your custom `SerDes`, and confirm + the first-run output equals the replay output. diff --git a/docs/migration-1.x-to-2.x.md b/docs/migration-1.x-to-2.x.md deleted file mode 100644 index 3b8ad36b..00000000 --- a/docs/migration-1.x-to-2.x.md +++ /dev/null @@ -1,367 +0,0 @@ -# Migrating from 1.x to 2.x - -`2.x` is a breaking major release. Every change is a bug fix or brings Python to -parity with the JavaScript and Java SDKs. The changes most likely to touch your -code are the typed, per-operation **error hierarchy**, the -**serialize/deserialize round trip on the first run**, and the new fail-fast -**default `map` / `parallel` completion** (which completes after the first -observed failure and stops scheduling pending items; already-started items are -not cancelled, so with unlimited concurrency all items may already be running - -set `max_concurrency` if you need to bound that). Opt back into process-all with -`CompletionConfig.all_completed()`. - -There is no compatibility shim: removed names (for example `CallableRuntimeError`) -are gone with no alias. If you are not ready to migrate, stay on `1.x`. - -> Instrumentation plugins are out of scope here. The experimental `plugins=` hook -> already existed in `1.x`; `2.x` adds opt-in auto-discovery and `PluginLoadError`. -> The plugin interface itself also changed incompatibly (hook signatures changed, -> enums moved, and `InvocationEndInfo.status` is now required), so plugin authors -> should expect to update. Because plugins are opt-in and still evolving, they are -> documented with the plugin/OpenTelemetry feature rather than in this guide. - -## Porting to 2.0 - -Each change below lists what changed and what you must do. The ones most likely -to touch your code are the [typed error hierarchy](#callableruntimeerror-and-friends-removed), -the [first-run serialize/deserialize round trip](#first-run-serializedeserialize-round-trip), -and the [fail-fast `map` / `parallel` default](#completionconfigall_completed-tolerates-all-failures). -See [Finding affected code](#finding-affected-code) for a grep checklist. - -### `CallableRuntimeError` and friends removed - -`CallableRuntimeError`, `UserlandError`, and -`CallableRuntimeErrorSerializableDetails` are gone; typed per-operation errors -replace them. - -Catch `StepError`, `InvokeError`, `ChildContextError`, or `WaitForConditionError` -(or the base `DurableOperationError`) instead of `CallableRuntimeError`. See -[Error handling](#error-handling-the-biggest-change) for the full hierarchy. - -### `CallbackError` moved out of the termination tree - -`CallbackError` is no longer a termination reason, and graded subtypes were -added. - -Remove any `termination_reason == TerminationReason.CALLBACK_ERROR` check (the -enum member is gone). Optionally catch `CallbackTimeoutError`, -`CallbackExternalError`, or `CallbackSubmitterError`. See -[Callbacks](#callbacks). - -### `BatchResult.throw_if_error()` raises typed errors - -It no longer raises `CallableRuntimeError`. - -Replace `except CallableRuntimeError` with `ChildContextError` (ordinary item -failure), `SerDesError` (item serialize/deserialize failure), and -`BatchCompletionError` (custom `should_complete` failed the batch with no item -error). `ChildContextError` and `BatchCompletionError` share the base -`DurableOperationError`, but `SerDesError` does not, so catch -`(DurableOperationError, SerDesError)` or list all three: - -```python -# 1.x -except CallableRuntimeError: - ... - -# 2.x -except (DurableOperationError, SerDesError): - ... -``` - -See [map / parallel](#map--parallel) for a full example. - -### Serdes failures surface as `SerDesError`, not `ExecutionError` - -`SerDesError` is a direct child of `DurableExecutionsError`, not `ExecutionError`. - -If you caught serdes failures with `except ExecutionError`, catch `SerDesError` -(or `DurableExecutionsError`) instead: - -```python -# 1.x -except ExecutionError: - ... - -# 2.x -except SerDesError: - ... -``` - -### First-run serialize/deserialize round trip - -`step`, child contexts, `map`/`parallel`, and `wait_for_condition` now round-trip -their result through the serdes on the first run, returning -`deserialize(serialize(x))` - the same canonical value replay returns. - -If you relied on the raw pre-serialization object, use the deserialized shape -instead (or make your `SerDes` round-trip identity). Ensure `wait_for_condition` -`initial_state` is serializable by the configured serdes. For a transient serdes -failure, raise the new `RetryableSerDesError` (retries) instead of `SerDesError` -(permanent). See [Serialize/Deserialize round trip](#serializedeserialize-round-trip). - -### Empty-string payloads are preserved - -`1.x` dropped empty-string (`""`) payloads when serializing; `2.x` keeps them. - -A step, invoke, or child result of `""` that surfaced as `None` (dropped payload) -in `1.x` now surfaces as `""`. If you treated an absent payload as `None`, handle -`""` explicitly. - -### `InvokeConfig.timeout` / `timeout_seconds` removed - -Both fields are gone. - -Remove them. Enforce any timeout inside the invoked function or as a separate -timer. - -### `map` / child batching names removed - -Removed: `ItemBatcher`, `ItemsPerBatchUnit`, `BatchedInput`, `TerminationMode`, -`StepFuture`, `MapConfig.item_batcher`, `ChildConfig.item_serdes`; also -`ChainedInvokeFailedToStartType`, `ChainedInvokeTimeoutType`, and -`ChainedInvokeStopType` (from `lambda_service`). - -Remove all uses. Replace `ChildConfig.item_serdes` with `ChildConfig.serdes`. - -### Config validation moved to construction and call time - -`MapConfig`, `ParallelConfig`, and `CompletionConfig` now validate arguments at -construction (for example `max_concurrency=0` or `min_successful=0` raise -`ValidationError`). `min_successful > total` is validated at the -`map()`/`parallel()` call, not at construction. - -Wrap config construction, and the `map()`/`parallel()` call, in -`try/except ValidationError` when inputs are external. - -### `CompletionConfig.all_completed()` tolerates all failures - -It now actually tolerates every failure, and the default `map` / `parallel` -completion config is now fail-fast. - -If you hand-built the old all-`None` config, use the factory instead. To preserve -1.x process-all behavior, pass it explicitly: - -```python -# 2.x - keep processing every item even when some fail -MapConfig(completion_config=CompletionConfig.all_completed()) -ParallelConfig(completion_config=CompletionConfig.all_completed()) -``` - -### `BatchResult.all` omits never-started branches - -`total_count` and positional iteration differ for early-completed batches, -because never-started branches are no longer included. - -If you index `.all` by original position or expect `total_count` to include -unstarted branches, update that logic. - -### `summary_generator` output moved into an envelope - -For `map`/`parallel`, a custom `summary_generator` output is now stored under a -`"summary"` key in an SDK-owned envelope; it no longer replaces the checkpoint -payload. `ChildConfig.summary_generator` is unchanged: its output is still -checkpointed verbatim. - -If you parse `map`/`parallel` summary payloads from execution history, read the -`"summary"` key from the envelope. Child-context summary consumers need no change. - -### `WaitDecision` and wait timeouts removed - -`WaitDecision` is gone, along with `WaitStrategyConfig.timeout` and -`WaitStrategyConfig.timeout_seconds`. - -Use `WaitForConditionDecision` (`stop_polling()` / `continue_waiting(delay)`). - -### `wait_for_condition` raises on exhaustion - -It raises `WaitForConditionError` when it exhausts `max_attempts`. - -Catch `WaitForConditionError` instead of inspecting the returned state. - -### Finding affected code - -Grep for the removed and changed names before upgrading: - -```bash -rg -n "CallableRuntimeError|UserlandError|CallableRuntimeErrorSerializableDetails" . -rg -n "CallbackError|CALLBACK_ERROR" . -rg -n "InvokeConfig\(|\.timeout_seconds" . -rg -n "WaitDecision|WaitStrategyConfig\(|item_batcher|ItemBatcher|ItemsPerBatchUnit" . -rg -n "TerminationMode|BatchedInput|StepFuture|ChildConfig\(" . -rg -n "ChainedInvoke|except ExecutionError" . -``` - -## Error Handling (the biggest change) - -In `1.x` nearly every user-land failure surfaced as one `CallableRuntimeError`, -so a failed step was indistinguishable from a failed invoke or child branch. `2.x` -raises a specific type per operation, all under a new base `DurableOperationError`. -Inspect the failure through its fields, not `__cause__`: `error_type`, `message`, -`data`, and `stack_trace`. Do not rely on `__cause__` being the original -exception - the SDK reconstructs a `DurableOperationError` stand-in carrying -those same fields (on both the first run and replay, for determinism), so the -original type is not preserved (a `ValueError` does not stay a `ValueError`) and -custom attributes are lost. - -For `StepError`, `InvokeError`, `ChildContextError`, and `WaitForConditionError`, -`error_type` is the name of the error that escaped your code (e.g. `"ValueError"`). -The graded callback errors are different: `CallbackTimeoutError`, -`CallbackExternalError`, and `CallbackSubmitterError` are constructed without the -originating `error_type`, so `error_type` is the callback class name, not the -underlying cause. Use the specific callback exception type (and `message` / -`data` / `stack_trace`) to distinguish those. - -```python -# 1.x -from aws_durable_execution_sdk_python.exceptions import CallableRuntimeError -try: - result = context.step(charge_card, name="charge") -except CallableRuntimeError as e: - context.logger.error("something failed: %s", e.message) - -# 2.x -from aws_durable_execution_sdk_python import StepError, DurableOperationError -try: - result = context.step(charge_card, name="charge") -except StepError as e: # or `except DurableOperationError` to catch any operation - context.logger.error("charge step failed: %s", e.message) -``` - -New types, all exported from the package root: `DurableOperationError` (base), -`StepError`, `InvokeError`, `ChildContextError`, `WaitForConditionError`, -`CallbackError` (+ `CallbackExternalError`, `CallbackTimeoutError`, -`CallbackSubmitterError`), plus `SerDesError` (now exported) and -`RetryableSerDesError`. `SerDesError` stays a direct child of -`DurableExecutionsError`; `RetryableSerDesError` is a retryable `InvocationError`. - -### Callbacks - -`context.wait_for_callback(...)` returns the payload directly and raises the -callback error from the call itself (there is no `callback.result()`): - -```python -from aws_durable_execution_sdk_python import ( - CallbackError, CallbackTimeoutError, CallbackSubmitterError, -) -try: - payload = context.wait_for_callback(submit_approval, name="approval") -except CallbackTimeoutError: - ... # timeout / heartbeat expiry -except CallbackSubmitterError: - ... # the submitter step failed -except CallbackError as e: # external + internal - context.logger.error("callback failed: %s", e.message) -``` - -### map / parallel - -`throw_if_error()` can raise three types: `ChildContextError` for the ordinary -item/branch failure, `SerDesError` if an item result failed to serialize or -deserialize, and `BatchCompletionError` when a custom `should_complete` predicate -marked the batch failed with no failed item (see the completion-predicate section -below). `ChildContextError` and `BatchCompletionError` share the base -`DurableOperationError`, but `SerDesError` does not, so catch -`(DurableOperationError, SerDesError)` or list all three explicitly. - -```python -from aws_durable_execution_sdk_python import ( - ChildContextError, SerDesError, BatchCompletionError, -) - -result = context.map(items, process_item) -try: - result.throw_if_error() -except (ChildContextError, SerDesError, BatchCompletionError): - for err in result.get_errors(): # every failed item's ErrorObject - context.logger.error("%s: %s", err.type, err.message) -``` - -## Serialize/Deserialize Round Trip - -`1.x` returned the raw in-memory result on the first run but the deserialized -result on replay, so a non-identity custom `SerDes` produced different values. -`2.x` round-trips (`serialize` then `deserialize`) on the first run for `step`, -child contexts, `map`/`parallel`, and `wait_for_condition` (which also feeds the -deserialized state to the wait strategy). This makes first-run behavior match -replay for deterministic, reversible custom serdes. If your code depended on -the raw pre-serialization object on the first run, switch to the deserialized -shape (or make the serdes round-trip identity). This also surfaces genuine -serialization bugs on the first run instead of later on replay. - -`invoke` and `wait` are unaffected. `wait_for_callback` is implemented via a -child context, so its result is serialized and deserialized before it returns: -do not rely on callback-result object identity. The enclosing child context uses -the default (extended-type) serdes, not `WaitForCallbackConfig.serdes`, so the -value your callback deserializer returns must itself be serializable by the -default serdes; otherwise the child raises `SerDesError`. - -`wait_for_condition` also round-trips `initial_state` through the serdes before -the first check, so `initial_state` must now be serializable by the configured -serdes. Custom serdes should serialize polling state to a non-empty string; an -empty string checkpoint payload is currently treated as no stored polling state -on resume, so the operation may restart from `initial_state`. - -## New in 2.x: Custom Completion Predicate (Optional) - -`2.x` adds a `should_complete` predicate to `CompletionConfig`, giving `map` and -`parallel` full control over when a batch completes early. This is a new feature, -not a breaking change - no action is required unless you adopt it. - -```python -from aws_durable_execution_sdk_python import complete_batch, continue_batch -from aws_durable_execution_sdk_python.config import CompletionConfig - -config = CompletionConfig( - should_complete=lambda status: ( - complete_batch() if status.success_count >= 2 else continue_batch() - ) -) -``` - -The predicate receives a `CompletionStatus` snapshot (counts plus per-item -statuses) and returns a `CompletionDecision` - `continue_batch()`, or -`complete_batch(CompletionOutcome.SUCCEEDED)` / `complete_batch(CompletionOutcome.FAILED)` -(the outcome defaults to `SUCCEEDED`). A `FAILED` outcome marks the whole batch -failed; `throw_if_error()` then raises `BatchCompletionError` (a -`DurableOperationError` subtype) even when no individual item failed. Individual -item/branch failures still surface as `ChildContextError`. Notes: - -- It cannot be combined with `min_successful` or the `tolerated_failure_*` - fields; doing so raises `ValidationError` at construction. -- The predicate runs before any branch is scheduled (`completed_count == 0`) - and on suspension state changes. At those points, unscheduled item statuses - are `None`; handle missing statuses explicitly so the predicate does not - fail the batch or complete it before useful work starts. -- The predicate must be deterministic, side-effect-free, and monotonic: once a - progress snapshot returns `complete_batch(outcome)`, every later snapshot - containing that progress must return `complete_batch(outcome)` with the same - `CompletionOutcome`. Replaying an already-completed batch uses the - checkpointed decision, but a mid-run resume re-runs the batch live and - re-evaluates the predicate as completed branches replay, possibly in a - different order. -- New exports: `complete_batch`, `continue_batch`, `CompletionStatus`, - `CompletionDecision`, `CompletionOutcome`, `CompletionItemStatus`, - `BatchItemStatus`, `BatchCompletionError`. - -## New in 2.x: Attempt Number in Contexts (Optional) - -`StepContext` and `WaitForConditionCheckContext` now expose an `attempt` field -(the current attempt number, starting at 1). Read it inside a step or a -`wait_for_condition` check to branch on the retry count. The SDK injects these -contexts, so normal usage needs no change. But `attempt` is a required -dataclass field with no default: if you construct these contexts directly (in -tests, fixtures, or wrappers), you must now pass `attempt` or construction fails -with `TypeError`. - -## Recommended Validation After Upgrading - -1. Build and run your test suite against `2.x`, and grep for the removed names above. -2. Trigger a failure in a `step`, an `invoke`, and a `map`/`parallel` branch; - confirm you catch `StepError`, `InvokeError`, and `ChildContextError`. -3. Exercise a `wait_for_callback` timeout and a submitter-step failure - (`CallbackTimeoutError`, `CallbackSubmitterError`). -4. Exercise a `wait_for_condition` that exhausts its attempts (`WaitForConditionError`). -5. If you use a custom `SerDes`, run a workflow that checkpoints a result, an - error payload, and `wait_for_condition` polling state; confirm first-run - output equals replay output. From 25de7cc4de8dae664f8f561985d5d4e3aabc9125 Mon Sep 17 00:00:00 2001 From: yaythomas Date: Fri, 4 Sep 2026 01:10:54 +0000 Subject: [PATCH 3/3] docs: correct guide claims against 1.7.0 behavior Address review feedback from ParidelPooya and Codex: - all_completed(): 1.x already failed fast by default and 1.x all_completed() hit the same fail-fast path, so the factory change is the breaking change, not the default. Advise 1.x all_completed() users to switch to all_successful() or a bare CompletionConfig() to keep fail-fast behavior. - Drop the default-completion claim from the intro and porting summary; promote replay identity validation to the headline. - SerDesError: the raise site changed (1.x wrapped serdes failures in ExecutionError), not the class hierarchy. - should_complete runs on every branch state change, including completion and failure, not only scheduling and suspension. - Identity validation cannot catch swaps of operations with identical identities; prohibit reordering under in-flight executions regardless. - Scope RetryableSerDesError body-replay to pre-SUCCEED first-run failures; checkpoint-read retries never re-run the step body. - Point the plugin note at the plugins page, not the logging page. - Fix two SDK docstrings the guide contradicted: MapConfig's completion_config default is fail-fast, and __cause__ holds a reconstructed stand-in on both first run and replay. --- docs/migration-1.x-to-2.0.md | 71 ++++++++++++------- .../config.py | 5 +- .../exceptions.py | 7 +- 3 files changed, 53 insertions(+), 30 deletions(-) diff --git a/docs/migration-1.x-to-2.0.md b/docs/migration-1.x-to-2.0.md index a966f763..cd9fb64e 100644 --- a/docs/migration-1.x-to-2.0.md +++ b/docs/migration-1.x-to-2.0.md @@ -4,7 +4,7 @@ most likely to touch your code are the typed, per-operation [error hierarchy](#error-handling), the [first-run serialize and deserialize round trip](#serialize-and-deserialize-round-trip), -and the new fail-fast [default `map` and `parallel` completion](#completionconfigall_completed-tolerates-all-failures). +and [replay operation identity validation](#replay-validates-operation-identity). ## Why upgrade to 2.0 @@ -34,8 +34,8 @@ and the new fail-fast [default `map` and `parallel` completion](#completionconfi The changes most likely to touch your code are the [typed error hierarchy](#callableruntimeerror-and-friends-removed), the -[first-run round trip](#first-run-serialize-and-deserialize-round-trip), and the -[fail-fast `map` and `parallel` default](#completionconfigall_completed-tolerates-all-failures). +[first-run round trip](#first-run-serialize-and-deserialize-round-trip), and +[replay operation identity validation](#replay-validates-operation-identity). [Finding affected code](#finding-affected-code) lists a grep checklist. ### `CallableRuntimeError` and friends removed @@ -83,8 +83,11 @@ except (DurableOperationError, SerDesError): ### Serdes failures surface as `SerDesError`, not `ExecutionError` -`SerDesError` descends directly from `DurableExecutionsError`, not from -`ExecutionError`. +In `1.x`, the SDK wrapped serialization and deserialization failures in +`ExecutionError`. In `2.0` it raises `SerDesError` instead. The class hierarchy +did not change: `SerDesError` descends from `DurableExecutionsError` in both +versions, and it is not an `ExecutionError` subclass, so an +`except ExecutionError` handler no longer catches serdes failures. Catch `SerDesError` (or `DurableExecutionsError`) where you caught `ExecutionError` for a serdes failure. @@ -159,21 +162,27 @@ the parent suspends too. Revisit the value if you sized `max_concurrency` around thread count rather than concurrent in-flight work. -### `CompletionConfig.all_completed()` tolerates all failures +### `CompletionConfig.all_completed()` now actually tolerates all failures -`all_completed()` now tolerates every failure, and the default `map` and -`parallel` completion config runs fail-fast: the batch completes after the first -observed failure and stops scheduling pending items. The SDK does not cancel -already-started items, so with unlimited concurrency every item may already be -running. Set `max_concurrency` to bound that. +In `1.x`, `all_completed()` returned an all-`None` config, which hit the +fail-fast path: the batch failed on the first item failure, the same as the +default. In `2.0` it returns `tolerated_failure_percentage=100` and tolerates +every failure, as its name always promised. The default `map` and `parallel` +completion behavior is unchanged: both versions fail fast on the first observed +failure, stop scheduling pending items, and do not cancel already-started items. -Call the factory instead of hand-building the old all-`None` config. Pass -`all_completed()` explicitly to keep the `1.x` process-all behavior. +If you called `all_completed()` in `1.x`, you got fail-fast behavior. After you +upgrade, the same call processes every item even when some fail. To keep the +fail-fast behavior you had, switch to `CompletionConfig.all_successful()` or a +bare `CompletionConfig()`. To process every item even when some fail, keep +`all_completed()`: ```python -# 2.0: keep processing every item even when some fail +# 1.x all_completed() behaved like this; keep fail-fast explicitly: +MapConfig(completion_config=CompletionConfig.all_successful()) + +# 2.0 all_completed() now means what it says; process every item: MapConfig(completion_config=CompletionConfig.all_completed()) -ParallelConfig(completion_config=CompletionConfig.all_completed()) ``` ### `BatchResult.all` omits never-started branches @@ -216,15 +225,19 @@ Catch `WaitForConditionError` instead of inspecting the returned state. On replay, the SDK validates each operation's checkpoint against the current code by `type`, `sub_type`, `name`, and `parent_id`. Any mismatch raises `NonDeterministicExecutionError` and fails the execution. `1.x` let a renamed or -reordered operation consume a neighboring checkpoint silently. `2.0` fails fast. +reordered operation consume a neighboring checkpoint silently. `2.0` fails fast +when the drift changes any of those four fields. Validation cannot catch a swap +of operations with identical identities, for example two unnamed steps under the +same parent, so it narrows the silent-mismatch window rather than closing it. A `map` or `parallel` batch also validates FLAT against NESTED nesting drift. `NonDeterministicExecutionError` descends from `ExecutionError`, so it is unrecoverable and you should not catch it. Treat this change as a deployment constraint rather than a code change. Do not rename an operation (the `name=` argument, or a step function's name when you omit `name`), change a batch's -`nesting_type`, or reparent an operation while executions are in flight. Drain -in-flight executions before you deploy such a change. +`nesting_type`, reorder operations, or reparent an operation while executions +are in flight, even when the reordered operations carry identical identities. +Drain in-flight executions before you deploy such a change. ### `wait_for_condition` fails on unreadable polling state @@ -368,13 +381,18 @@ currently treats an empty-string checkpoint payload as no stored polling state on resume, so the operation can restart from `initial_state`. `RetryableSerDesError` does more than retry serialization. An executor re-raises -it before it writes a success checkpoint, so the backend re-invokes the whole -execution. An `AT_LEAST_ONCE_PER_RETRY` step re-runs its body and repeats its -side effects on that re-invocation, so raise `RetryableSerDesError` only from an +it without writing a checkpoint, so the invocation fails and the backend +re-invokes the whole execution. What happens next depends on the phase in which +the error was raised. When the first-run result round trip fails before the +success checkpoint exists, an `AT_LEAST_ONCE_PER_RETRY` step re-runs its body +and repeats its side effects, so raise `RetryableSerDesError` there only from an idempotent step or accept the duplicate work. An `AT_MOST_ONCE_PER_RETRY` step already wrote its START checkpoint, so the retried invocation treats the step as interrupted and applies the step retry strategy instead of re-running the body. -Raise `SerDesError` for a permanent failure. +When the error is raised while deserializing an already-succeeded checkpoint, +the success checkpoint exists, so the next invocation retries only the +deserialization and never re-runs the step body. Raise `SerDesError` for a +permanent failure. ## Custom Completion Predicate @@ -406,9 +424,10 @@ Three constraints govern the predicate: - It cannot combine with `min_successful` or the `tolerated_failure_*` fields. Combining them raises `ValidationError` at construction. -- It runs before the SDK schedules any branch (`completed_count == 0`) and on - each suspension state change. At those points an unscheduled item has status - `None`, so handle a missing status explicitly, or the predicate can fail the +- It runs on every branch state change: scheduling, completion, failure, and + suspension. The first evaluation happens before the SDK schedules any branch + (`completed_count == 0`), and an unscheduled item has status `None` at that + point, so handle a missing status explicitly, or the predicate can fail the batch or complete it before useful work starts. - It must stay deterministic, side-effect-free, and monotonic. Once a progress snapshot returns `complete_batch(outcome)`, every later snapshot that contains @@ -434,7 +453,7 @@ or wrapper, must now pass `attempt` or construction raises `TypeError`. Instrumentation plugins remain experimental in this release, and the plugin interface changed incompatibly. See the -[observability and plugin documentation](https://docs.aws.amazon.com/durable-execution/sdk-reference/observability/logging/) +[plugin documentation](https://docs.aws.amazon.com/durable-execution/sdk-reference/observability/plugins/) for details. ## Recommended Validation After Upgrading diff --git a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/config.py b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/config.py index bc9253a6..aa5a2b32 100644 --- a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/config.py +++ b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/config.py @@ -550,8 +550,9 @@ class MapConfig(Generic[T]): completion_config: Defines when the map operation should complete. Controls success/failure criteria for the overall map operation. - Default allows any number of failures. Use CompletionConfig.all_successful() - to require all items to succeed. + The default fails fast: the first failed item fails the batch. + Use CompletionConfig.all_completed() to process every item + regardless of failures. serdes: Custom serialization/deserialization configuration for BatchResult. Applied at the handler level to serialize the entire BatchResult object. diff --git a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/exceptions.py b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/exceptions.py index d36cf080..d51ab961 100644 --- a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/exceptions.py +++ b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/exceptions.py @@ -265,8 +265,11 @@ class DurableOperationError(DurableExecutionsError): Wraps a failure that escaped a Durable Function operation (step, invoke, child context, wait_for_condition). The concrete class identifies the - operation kind (so callers can ``except StepError``); the escaping error is - preserved as ``__cause__`` on the first run and reconstructed on replay. + operation kind (so callers can ``except StepError``). ``__cause__`` holds a + ``DurableOperationError`` stand-in rebuilt from the checkpointed error + fields on both the first run and replay, for determinism; it does not hold + the original exception object. Inspect failures through ``error_type``, + ``message``, ``data``, and ``stack_trace``. Attributes: message: Human-readable failure message.