[zhongkechen]

Issue Link, if available

Partial fix for #726, stacked on #733. Arbitrarily nested bounded-pool starvation remains unresolved.

Description

Concurrent LMI invocations sharing a bounded executor can occupy every worker with roots that wait for queued steps. withInvocationExecutorFactory(...) gives each invocation a fresh executor for its root, steps, child contexts, and coordinators. Existing withExecutorService(...) executors remain caller-owned and shared.

The stack now uses the 3.x plugin factory API and preserves main's handler-owned Start/End lifecycle. The runtime caller waits for actual admitted work and End before shutting down an invocation-owned executor, including after cancellation. Non-cooperative work can delay shutdown beyond the policy budget; executor termination is observed with any cleanup time that remains.

Startup rejected by the invocation deadline leaves handler/plugin code undispatched and closes an owned executor exactly once. Lifecycle fencing also surrounds operation/task registration during draining, so cancellation cannot strand a late replayed pending operation or an undispatched task. The latest #733 lifecycle fixes also preserve captured input fatals and report cancellation during terminal serialization accurately to End hooks and both OTel views.

End hooks, cancellation notifications, and the caller retain the first recorded lifecycle failure when owner drain is interrupted or later cleanup also fails for lifecycle reasons. The SDK-created owner-drain diagnostic and later lifecycle cleanup failures remain attached, including owned-executor termination timeouts, without creating suppression cycles. Standalone owner-drain errors retain their original shape. JVM fatal and unrelated delivery error precedence stays unchanged.

Owned executor cleanup preserves the first JVM fatal and retains later executor failures, including standard asynchronous wrappers and setup failures before handler dispatch. The shutdown/termination attempt is retained before querying executor state, preserving both failures when that query also fails. Separate close observations prevent try-with-resources from replacing the original fatal with a self-suppression error. Fatal cleanup failures retain priority after terminal ACK and caller interruption.

If graceful shutdown throws, normal finalization attempts forced stop and waits within the same remaining executor deadline. Post-manager setup also enters explicit finalization, so a later cleanup JVM fatal cannot be hidden beneath an ordinary setup error.

Confirmed executor termination releases its identity lease even if a subsequent state query fails. The query failure remains visible, including its existing fatal priority. A false or throwing termination wait alone does not release the lease; existing closed-state handling remains in place.

A fatal reported only by root completion, such as asynchronous MDC restoration after End, is observed after the root actually exits and before owned executor shutdown. Later cleanup failures remain attached to that original fatal. This observation stays local to cleanup, preserving the selected End result and checkpoint/cancellation control. Directly attached diagnostics are compared by identity to avoid adding the same error twice.

When cleanup wins over a nonfatal delivery failure, the original delivery error remains attached as a diagnostic. The caller keeps the existing fatal, cutoff, ACK and standalone-completion rules. An identical error already present directly or as a known SDK cleanup cause is not attached again.

Initialization claims the executor identity lease before inspecting its state. If validation or constructor setup fails after a fresh executor is acquired, the runtime cleans that executor and releases its lease while preserving the original failure, later cleanup diagnostics, and caller interruption. An executor already leased to another invocation is rejected without querying or shutting it down.

DurableConfig.toBuilder() preserves the executor factory and resolved plugin factories. The local runner obtains a new executor on replay/resume without rediscovering providers. The factory must return a fresh, non-null, usable executor; overlapping reuse and configuring both executor modes are rejected.

The LMI fixed2 fixture uses per-invocation fixed(2) pools. The shared nested2 fixture remains a diagnostic fixture outside the 12 gating cloud cases.

Validation

Validated source tree: a7acf3c34a6106fd78fa789080adc72f25e2413f, including upstream main 953c3771d4374a8bfeaa8653b133a8fa83d624dd and the latest #733 implementation.

• Java 17: clean Spotless/install reactor passed all 10 modules, with 3,823 tests passed and 36 conditional/cloud tests skipped.
• Java 25: the initial core run hit one timer-sensitive batch-count assertion. The unchanged method passed a targeted retry; the tested core was packaged and the remaining clean reactor completed. Across distinct tests, 3,828 passed and 34 skipped, with no outstanding failures. The retry is counted once.
• Installed Core3/plugin/testing artifacts passed all four OTel API compatibility cases across APIs 1.49.0 and 1.66.0. Artifact hashes match the final reactor outputs.
• All three opt-in local LMI lifecycle controls and all 203 Python harness tests passed. The final harness sources match those tested.
• All 12 final targeted Java17 controls passed, including serializer failures with ordinary/fatal cleanup, ACK delivery failures, caller cutoff, and repeated SDK error aliases. Independent Java25 review passed the four adjacent ACK/cutoff/alias controls. The final full reactors include the existing ownership, MDC, fatal, replay and lifecycle regression suites.
• No new deployed cloud acceptance run is claimed. [Bug]: LMI invocation cleanup can hang or outlive the response; shared fixed pools can deadlock #726's shared nested-pool limitation remains open. Merge and retarget the stack after fix: enforce invocation deadlines while awaiting owner cleanup #733.

Checklist

• Shared executors remain caller-owned.
• Invocation-owned executors are shut down by the runtime caller after actual work and End.
• Executor selection, factory identity, and configuration-copy/replay behavior are covered.
• Cleanup preserves fatal priority and diagnostic evidence.
• The partial-mitigation boundary is retained.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

Codex AI review · Finding arf_v1_5yqbeiesxrmt6cny7n7mpdtywh

[P2] Retain cleanup diagnostics on the recorded delivery fatal

If output serialization throws an InternalError after the handler returns successfully, getInvocationFatal() is already set, so this branch skips the completed fatal. finishActualInvocationCleanup() returns a CompletionException wrapping that fatal from checkpoint-batcher shutdown. If owned shutdown() then throws an ordinary exception, its diagnostic is attached to that wrapper. DurableExecutor.execute() throws the raw completed fatal before inspecting finalFailure, losing the shutdown diagnostic. Aggregate owned cleanup failures onto the recorded fatal while preserving wrapper diagnostics without creating a cause/suppression cycle. Add a regression covering fatal serialization followed by ordinary shutdown failure.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review

One P2 finding at sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java:1722: later cleanup diagnostics can be lost after a fatal serialization error. Tests were inspected but not executed under the read-only review constraints.

Reviewed commit ee70d9fc94613f652ea25c70a192b9d6570e567d. Workflow run