Non-streaming chat completions fail against Anthropic-backed OpenAI-compatible endpoints on long-running turns (HTTP 400 "Streaming is required")

[CyberSeppi]

Summary

runtime/node/agent/providers/openai_provider.py issues every client.chat.completions.create(**payload) call without setting stream=True. The OpenAI Python SDK defaults to stream=False in that case, and the full response is awaited over a single non-streaming HTTP call.

When ChatDev is pointed at an OpenAI-compatible endpoint that is backed by Anthropic Claude models (either Anthropic's own API or an intermediary such as AWS Bedrock or an enterprise LLM gateway), long-running turns hit an upstream policy limit and fail with:

Error code: 400 — {
  "type": "invalid_request",
  "status": 400,
  "message": "Streaming is required for operations that may take longer than 10 minutes",
  "instance": "/v1/chat/completions"
}

The 10-minute non-streaming cap is enforced by Anthropic (documented at https://github.com/anthropics/anthropic-sdk-python#long-requests). Any workflow that produces a large output in a single turn — big JSON aggregations, full-page renders, long structured summaries — will trip it.

Environment

• ChatDev branch: main (commit 4fd4da6 at time of report)
• Provider: openai (built-in OpenAIProvider)
• Upstream: OpenAI-compatible API serving claude-sonnet-4-6 / claude-opus-4-7
• Python: 3.12
• openai SDK: shipped with ChatDev's uv.lock

Reproduction

A workflow with a single agent node that:

1. Uses provider: openai with a Claude model (model: claude-sonnet-4-6 or similar) against an Anthropic-backed OpenAI-compatible endpoint.
2. Receives a large input via edges: from upstream nodes (a few hundred KB of JSON is enough).
3. Emits a large output in one turn (e.g. rendering full-page Confluence storage XML, ~15 KB / ~5–10k output tokens, ~3–5 min wall-time at typical Claude Sonnet speeds).

Result: HTTP 400 with the message above. The node's final_message becomes the error text and the workflow exits completed with the error surfaced instead of the intended output.

The same workflow succeeds against GPT-family models under identical conditions, because Azure OpenAI does not enforce the 10-minute non-streaming cap.

Root cause

OpenAIProvider._build_chat_payload() and _build_request_payload() do not set stream=True, and call_model() calls client.chat.completions.create(**payload) / client.responses.create(**payload) directly. The OpenAI Python SDK defaults stream to False, so ChatDev's LLM traffic is entirely non-streaming.

For GPT-shaped upstreams this is fine — for Anthropic-shaped ones (either Anthropic Cloud, Bedrock, or any OpenAI-compat gateway that proxies to Anthropic and inherits its policies) it is a hard failure on any turn that could exceed 10 min.

Suggested solution

Force stream=True at the provider level and aggregate the chunks back into an object with the same attribute shape used by the existing deserializer, so that the rest of the provider code (_track_token_usage, _deserialize_chat_response, _append_chat_response_output) is unchanged.

Two concrete places to change in runtime/node/agent/providers/openai_provider.py:

1. In call_model(), replace the two direct client.chat.completions.create(**request_payload) calls with a helper self._chat_completion(client, payload).
2. Implement _chat_completion() as:
   • Set payload[\"stream\"] = True and payload.setdefault(\"stream_options\", {\"include_usage\": True}).
   • Iterate the returned stream, concatenating delta.content and reassembling delta.tool_calls[] by index.
   • Track the final usage object from the last chunk (OpenAI streams it when stream_options.include_usage=true).
   • Return a lightweight object graph exposing choices[0].message.{content, tool_calls}, .id, .model, .usage — the exact shape the rest of the file consumes.

I have a working patch running against an Anthropic-backed OpenAI-compatible LLM gateway (Bedrock Claude Sonnet 4-6 + Claude Opus 4-7). It reduces a 10+ min hanging turn to a ~4 min streamed completion with no behavior change for GPT models. Happy to open a PR with the patch + a small unit test that mocks the streamed chunk shape, if the maintainers are open to it.

Impact / severity

• Blocking for any ChatDev workflow using Claude via an OpenAI-compat gateway on turns > ~10 min.
• Silent for smaller turns — the same code path works and gives no warning that it will break at scale.
• Not caught by existing tests since the streaming vs. non-streaming distinction is upstream-policy-driven, not detected client-side.

Related upstream references

• Anthropic long-request policy: https://github.com/anthropics/anthropic-sdk-python#long-requests
• OpenAI SDK stream_options.include_usage: https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options