Skip to content

[Enhancement]: Add lossless provider-codec conformance coverage #520

Description

@afourniernv

Affected area

Rust provider codecs, Python binding, Node.js binding, buffered LLM execution, and streaming LLM execution.

Problem or opportunity

The Hermes integration exposed a lossy Anthropic Messages round trip: structured tool_use and tool_result history could disappear, and block-form system content could lose cache_control. #509 fixed the known codec behavior, but the review also identified that codec coverage is not yet systematic across structured provider payloads.

This follow-up comes from the Hermes integration review in NousResearch/hermes-agent#67607 and was requested by @jquesnelle. Codec correctness is a Relay contract and should be proven centrally rather than recreated in every harness.

Proposed enhancement

Add a table-driven conformance suite for every built-in provider request and response codec. The suite should exercise no-op round trips, scalar edits, structured tool history, multimodal or attachment content, provider-specific extension fields, and streaming finalization.

Rust should own the complete conformance matrix. Python and Node.js should include focused binding tests proving that the same lossless behavior reaches callbacks through public managed-execution APIs.

Runtime contract and binding impact

  • A no-op intercept preserves the provider payload exactly.
  • An intercept that returns no replacement leaves the original field untouched.
  • A scalar edit changes only the requested field and preserves unrelated provider-specific data.
  • Buffered and streaming paths apply the same preservation rules.
  • The contract covers Anthropic Messages, OpenAI Chat Completions, and OpenAI Responses without expanding public content enums solely for test convenience.

Alternatives considered

Relying only on framework integration tests leaves gaps, duplicates fixtures, and catches regressions after publication. Keeping one Anthropic regression prevents recurrence of the known bug but does not establish a reusable contract for the other codecs.

Acceptance criteria

  • Anthropic coverage includes block-form system with cache_control, tool_use, tool_result, nested text content, image/document content where supported, and unknown provider extensions.
  • OpenAI Chat coverage includes tool calls/results, multimodal message parts, attachments or provider extensions where supported, and unknown message fields.
  • OpenAI Responses coverage includes structured input/output items, tool calls/results, reasoning or provider extensions, and multimodal content where supported.
  • Every codec has exact no-op round-trip assertions and targeted scalar-edit assertions.
  • Buffered and streaming execution paths are covered.
  • Rust contains the complete matrix; Python and Node.js each prove representative public-binding paths.
  • Tests preserve the existing meaning of None and avoid a source-level breaking expansion of exhaustive public enums.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Improvementimprovement to existing functionalityTestTest relatedlang:jsPR changes/introduces Javascript/Typescript codelang:pythonPR changes/introduces Python codelang:rustPR changes/introduces Rust code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions