feat: add context-aware filesystem SerDes

[zhongkechen]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Closes #463.

Related to #509.

This is a smaller explicit-context alternative to #648 and #649.

Description

• Keep the original SerDes methods unchanged and add backward-compatible default overloads that receive
  SerDesContext explicitly.
• Route SDK-managed SerDes calls through one invocation-scoped SerDesRunner that runs inline by default, optionally
  uses a dedicated executor, passes context directly, shares in-flight deserializations, and keeps a bounded
  weak-reference cache keyed by SerDes identity, execution/entity, serialization generation, type, and payload hash.
• Add FileSystemSerDes to the core SDK with ALWAYS and OVERFLOW modes, URI/hash path encodings, a configurable checkpoint-envelope limit, structured and custom previews, content-addressed immutable files, SecureDirectoryStream traversal, NOFOLLOW_LINKS file I/O, digest verification, and strict marker-only envelope recognition.
• Keep value encoding independently configurable through FileSystemSerDes.delegate(...), while existing operation configs select the SerDes used for step, invoke payload/result, callback, child-context, map, parallel, and wait-for-condition boundaries.
• Classify filesystem read/write failures as RetryableSerDesException and add RetrySerDes, a non-pipeline decorator that applies existing bounded RetryStrategy implementations only to transient failures.
• Publish each filesystem payload to a unique immutable file and remove partially created files when publication fails.
• Keep initial invocation input as normal delegate JSON when no durable execution context exists.
• Apply the runner to root input/output, operation results/state, invoke payloads, and serialized exceptions.
• Propagate SerDesRunner and durable contexts through local/cloud test results, operation inspection, history processing, and async execution snapshots, with an operation resolver for per-operation SerDes overrides.
• Replace ADR-005 with the accepted explicit-context/executor/cache design and add a dedicated
  serialization/filesystem/retry guide, exact filesystem wire-format reference, invoke-boundary guidance, and runnable
  example.
• Align applicable test and E2E coverage with feat: add filesystem-backed SerDes stage (Approach A) #648: expanded filesystem, retry, cache, replay, raw-input, and cross-invoke tests; generated persistent EFS/VPC infrastructure; one filesystem Lambda stack per Java version; and cloud assertions for offloaded operation/output envelopes and structured previews.
• Add SHA-256 integrity metadata to both inline and file envelopes and verify it before delegate deserialization.
• Require external-storage SerDes references to be immutable or versioned, and keep filesystem payload files as direct
  children of a pre-provisioned base directory so all directory traversal remains relative to held secure handles.

Applicable parity with #648

This PR now carries over the #648 coverage that applies to an explicit-context SerDes:

• cache identity, eviction, concurrent misses, null results, fatal errors, retries, delay/interruption handling, and invalid retry decisions;
• inline/file integrity, envelope limits, malformed/unsupported envelopes, custom delegate encoding, previews, immutable files, provider capability, and symlink/traversal protection;
• root input/output, operation overrides, step/wait/child/map replay, callback/invoke boundaries, caller/callee filesystem exchange, cache reuse, and serialized exceptions;
• local-runner large-output replay, a runnable filesystem example, exact wire-format documentation, SAM generator tests, and EFS-backed cloud E2E coverage.

Pipeline composition, binary stages, chained-invoke source framing/JavaScript compatibility, and pipeline-specific
producer/attempt metadata remain intentionally out of scope for this explicit-context design.

Demo/Screenshots

Not applicable. Configuration examples are included in the documentation.

Checklist

• I have filled out every section of the PR template
• I have thoroughly tested this change

Testing

• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 clean install
• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 -pl sdk -Dtest=SerDesRunnerTest,RetrySerDesTest,FileSystemSerDesTest test
• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 -pl sdk-integration-tests -am -Dtest=FileSystemSerDesIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test
• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 -pl sdk-testing -am -Dtest=LocalDurableTestRunnerTest -Dsurefire.failIfNoSpecifiedTests=false test
• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 -pl examples -am -Dtest=FileSystemSerDesExampleTest -Dsurefire.failIfNoSpecifiedTests=false test
• python3 -m unittest examples/test_generate_template.py
• mvn -Dmaven.repo.local=/home/frankchn/.codex-tmp/m2 spotless:check
• git diff --check

The full eight-module reactor passed, including 1,229 SDK tests. The EFS-backed cloud test is opt-in locally and is
enabled by the updated E2E workflow.

Unit Tests

Yes. The focused SDK parity set contains 53 tests across SerDesRunnerTest, RetrySerDesTest, and
FileSystemSerDesTest. Coverage includes default-method compatibility, explicit context propagation, inline and
configured-executor dispatch, cache
identity/eviction/concurrency/generation invalidation, retry selection/exhaustion/delay/interruption, retryable
filesystem I/O, filesystem modes, immutable unique files, configurable envelope limits, delegate encoding, structured
previews, strict marker-only envelope validation, digest verification, secure-directory provider enforcement, and
non-retryable structural, permission, invalid-path, and symlink rejection for base paths, directories, and files. The
envelope limit cannot exceed the safe 255 KiB checkpoint ceiling, and raw boundaries remain context-free even when the
delegate implements contextual encoding.

Integration Tests

Yes. Eleven FileSystemSerDesIntegrationTest scenarios cover raw root input and offloaded output, operation-level
selection, step/wait/child/map replay, raw callback/invoke results, standard JSON invoke overrides, caller/callee
offloaded payload/result exchange, repeated-get caching, custom exception payloads, and deterministic external-storage
keys across state/exception replay. The operation override is also inspected through its effective filesystem SerDes,
with equivalent resolver coverage for cloud history. Checkpoint-failure replay verifies old and new immutable
filesystem references remain independently readable. Local-runner coverage verifies large-output replay without
duplicate operation execution.

Examples

Yes. FileSystemSerDesExample demonstrates EFS-style configuration, structured previews and masking, bounded
filesystem retries, forced replay, and checksum verification. It has a local test and an opt-in cloud test backed by
the same persistent EFS/VPC SAM infrastructure pattern used by #648.

[github-actions]

Codex AI review · Finding arf_v1_ijtikdiu4z2pfc5y5degiqq7ru

[P1] Preserve context-free decoding for initial input. Invocation input is serialized before a durable ARN exists through SerDes.serialize(Object), but this invokes the context-aware deserializer. A valid SerDes using distinct raw and contextual encodings will therefore fail every initial invocation. Add a context-free dispatch method to SerDesRunner and use it here—and for other externally produced raw boundaries—with a regression test covering distinct encodings.

[github-actions]

Codex AI review · Finding arf_v1_zrfjhwzrlpr5feemsk7pt3ypuu

[P2] Bound generation bookkeeping with the cache. The 256-entry limit applies only to completed values; every unique SerDes/entity context is retained here for the entire invocation. Large maps or long workflows can consequently exhaust Lambda memory. Evict inactive generation state alongside completed/in-flight entries, and test that all runner bookkeeping remains bounded across many entities.

[github-actions]

Codex AI review · Finding arf_v1_fphsma5t5ymczje74zssfgxx6q

[P2] Treat a missing configured base directory as permanent. NoSuchFileException while traversing a missing base-path component reaches this retryable fallback, so RetrySerDes repeatedly delays for a pre-provisioning/configuration error. Distinguish base-directory traversal failures from a missing payload file, wrap the former in SerDesException, and update the missing-base test accordingly.

[github-actions]

Codex AI review

Three actionable issues remain. The main risk is incompatibility at the initial-input boundary for context-aware SerDes implementations; cache bookkeeping and filesystem retry classification also need correction. Static review only; tests were not run per constraints.

Reviewed commit 3e353306c1b8ea5bfae1cd97b720923d2032a491. Workflow run