Affected area
Rust provider codecs, Python binding, Node.js binding, buffered LLM execution, and streaming LLM execution.
Problem or opportunity
The Hermes integration exposed a lossy Anthropic Messages round trip: structured tool_use and tool_result history could disappear, and block-form system content could lose cache_control. #509 fixed the known codec behavior, but the review also identified that codec coverage is not yet systematic across structured provider payloads.
This follow-up comes from the Hermes integration review in NousResearch/hermes-agent#67607 and was requested by @jquesnelle. Codec correctness is a Relay contract and should be proven centrally rather than recreated in every harness.
Proposed enhancement
Add a table-driven conformance suite for every built-in provider request and response codec. The suite should exercise no-op round trips, scalar edits, structured tool history, multimodal or attachment content, provider-specific extension fields, and streaming finalization.
Rust should own the complete conformance matrix. Python and Node.js should include focused binding tests proving that the same lossless behavior reaches callbacks through public managed-execution APIs.
Runtime contract and binding impact
- A no-op intercept preserves the provider payload exactly.
- An intercept that returns no replacement leaves the original field untouched.
- A scalar edit changes only the requested field and preserves unrelated provider-specific data.
- Buffered and streaming paths apply the same preservation rules.
- The contract covers Anthropic Messages, OpenAI Chat Completions, and OpenAI Responses without expanding public content enums solely for test convenience.
Alternatives considered
Relying only on framework integration tests leaves gaps, duplicates fixtures, and catches regressions after publication. Keeping one Anthropic regression prevents recurrence of the known bug but does not establish a reusable contract for the other codecs.
Acceptance criteria
- Anthropic coverage includes block-form
system with cache_control, tool_use, tool_result, nested text content, image/document content where supported, and unknown provider extensions.
- OpenAI Chat coverage includes tool calls/results, multimodal message parts, attachments or provider extensions where supported, and unknown message fields.
- OpenAI Responses coverage includes structured input/output items, tool calls/results, reasoning or provider extensions, and multimodal content where supported.
- Every codec has exact no-op round-trip assertions and targeted scalar-edit assertions.
- Buffered and streaming execution paths are covered.
- Rust contains the complete matrix; Python and Node.js each prove representative public-binding paths.
- Tests preserve the existing meaning of
None and avoid a source-level breaking expansion of exhaustive public enums.
Affected area
Rust provider codecs, Python binding, Node.js binding, buffered LLM execution, and streaming LLM execution.
Problem or opportunity
The Hermes integration exposed a lossy Anthropic Messages round trip: structured
tool_useandtool_resulthistory could disappear, and block-formsystemcontent could losecache_control. #509 fixed the known codec behavior, but the review also identified that codec coverage is not yet systematic across structured provider payloads.This follow-up comes from the Hermes integration review in NousResearch/hermes-agent#67607 and was requested by @jquesnelle. Codec correctness is a Relay contract and should be proven centrally rather than recreated in every harness.
Proposed enhancement
Add a table-driven conformance suite for every built-in provider request and response codec. The suite should exercise no-op round trips, scalar edits, structured tool history, multimodal or attachment content, provider-specific extension fields, and streaming finalization.
Rust should own the complete conformance matrix. Python and Node.js should include focused binding tests proving that the same lossless behavior reaches callbacks through public managed-execution APIs.
Runtime contract and binding impact
Alternatives considered
Relying only on framework integration tests leaves gaps, duplicates fixtures, and catches regressions after publication. Keeping one Anthropic regression prevents recurrence of the known bug but does not establish a reusable contract for the other codecs.
Acceptance criteria
systemwithcache_control,tool_use,tool_result, nested text content, image/document content where supported, and unknown provider extensions.Noneand avoid a source-level breaking expansion of exhaustive public enums.