Skip to content

Fix streamed Chat Completions aggregation#654

Draft
taking-lying-flat wants to merge 1 commit into
OpenBMB:mainfrom
taking-lying-flat:agent/fix-streamed-chat-completions
Draft

Fix streamed Chat Completions aggregation#654
taking-lying-flat wants to merge 1 commit into
OpenBMB:mainfrom
taking-lying-flat:agent/fix-streamed-chat-completions

Conversation

@taking-lying-flat

@taking-lying-flat taking-lying-flat commented Jul 19, 2026

Copy link
Copy Markdown

Summary

  • route both Chat Completions branches through the OpenAI SDK streaming helper
  • aggregate streamed deltas back into a complete ChatCompletion-shaped response
  • request usage in the terminal usage-only chunk and retry without stream_options for compatible gateways that reject it
  • preserve legacy gateways through explicit stream: false and a narrow fallback when streaming is explicitly unsupported
  • prevent internal protocol, stream, and stream_options controls from leaking into incompatible request paths

Root cause

The existing Chat Completions branches called client.chat.completions.create() without streaming and then assumed a complete response at response.choices[0].message. Gateways that require streaming rejected long-running requests. Simply setting stream: true was also insufficient because it returned an iterator that the existing timeline, token accounting, and tool-call deserialization code could not consume.

Impact

Chat Completions now use streaming transport by default while downstream ChatDev code still receives an aggregated completion. Token tracking, timeline handling, and tool-call parsing keep their existing interfaces. Legacy non-streaming gateways can opt out with params.stream: false, and explicit unsupported-streaming responses downgrade narrowly without masking authentication, quota, timeout, or server failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant