fix(chatcmpl): surface content-filter refusals when buffering streamed tool calls#3897
fix(chatcmpl): surface content-filter refusals when buffering streamed tool calls#3897gunjanjaswal wants to merge 3 commits into
Conversation
The tool-call buffering layer only forwarded a choice when its delta carried output, so a terminal chunk with an empty delta was dropped before reaching the stream handler. With buffer_streamed_tool_calls enabled, a provider that signals a safety block via finish_reason == "content_filter" and an empty delta produced a silently empty turn instead of a ResponseOutputRefusal, and "length" was dropped the same way, making a truncated turn indistinguishable from a short answer. Forward a delta-stripped copy of a terminal choice so the finish_reason reaches the handler while buffering semantics stay unchanged. "tool_calls" is excluded because the synthesized buffered chunk already carries that finish_reason.
seratch
left a comment
seratch
left a comment
There was a problem hiding this comment.
Before merging, please narrow the passthrough condition and related tests to finish_reason == "content_filter". The current patch also forwards length and stop, but handle_stream() does not interpret either value, so those cases do not change the normalized events or completed response. Please remove those claims and cases while retaining the end-to-end content-filter coverage and the tool_calls non-duplication test. After that focused change, this should be ready for another review.
handle_stream() does not interpret finish_reason "length" or "stop", so forwarding those terminal choices changed nothing observable while widening the surface of the fix. Restrict the passthrough to "content_filter", which is the only value the handler acts on, and drop the corresponding test cases.
|
Good call, narrowed it to Dropped the two parametrized cases along with them and removed the |
…-content-filter-refusal
2 participants
#3769 made a content-filtered stream surface a
ResponseOutputRefusalinstead of an empty turn, but only on the non-buffered path. Withbuffer_streamed_tool_calls=Truethe refusal never fires.buffer_tool_call_streamforwards a choice only when its delta carries output. A terminal chunk likeChoice(index=0, delta=ChoiceDelta(), finish_reason="content_filter")has an empty delta, so it gets dropped before the handler ever sees it.saw_content_filteris never set and the synthesis block added in #3769 never runs. The caller gets a silently empty turn, which is the exact failure mode that PR set out to fix.The fix forwards a delta-stripped copy of a content-filtered terminal choice, so the
finish_reasonreaches the handler while buffering semantics stay unchanged.Reachable through
OpenAIChatCompletionsModel, and also viaMultiProvider(openai_buffer_streamed_tool_calls=True)andOpenAIProvider.Tests mirror the content-filter cases from
test_litellm_chatcompletions_stream.pywith buffering on, plus coverage of the buffering layer itself and a guard against duplicating thetool_callsfinish. The gap went untested because LiteLLM never buffers. Checked againstupstream/mainthat the new tests fail without the source change and pass with it. Full suite green (5354 passed), ruff and mypy clean.