Instrument LiteLLM Anthropic messages acreate for GenAI traces - #28
Conversation
…hrough calls emit GenAI spans. acompletion wrapping misses Bedrock/Vertex Claude HTTP pass-through, which never re-enters acompletion. Co-authored-by: Cursor <cursoragent@cursor.com>
…t usage and stop_reason. OpenAI stream_chunk_builder never sees id/tokens/stop_reason on a single last chunk for SSE or message_start/message_delta events. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
| target[key] = value | ||
|
|
||
|
|
||
| def _aggregate_anthropic_messages_stream(chunks: list[Any]) -> dict[str, Any]: |
There was a problem hiding this comment.
Can this be turned into more generic function handling any sse chunks, not only from this Litellm antrhopic? How do we handle chunks from other instrumentation, are they also processed separately?
| return isinstance(event_type, str) and event_type in _ANTHROPIC_STREAM_EVENT_TYPES | ||
|
|
||
|
|
||
| def _looks_like_anthropic_messages_stream(chunks: list[Any]) -> bool: |
There was a problem hiding this comment.
The raw data: substring check is a reasonable fast path for native SSE byte streams, but it could theoretically false-positive if streamed text content contains data:. In practice this is unlikely because OpenAI-shaped LiteLLM chunks are dict/ModelResponse objects, not raw SSE strings — so I think this is fine as-is. Worth a brief docstring noting the trade-off for future readers.
|
Follow-up on the stream-detection note: docstring to add on |
3 participants
Summary
litellm.anthropic.messages.acreate(Anthropic Messages pass-through used by llm-model-service) produced nolitellm_requestspans.create/acreateon the LiteLLM Anthropic messages interface, reuse the existing span/stream/policy path, and keep a single span when acreate falls back to acompletion.Test plan
pytest test/instrumentation/litellm/(or./scripts/run-unit-tests.shscoped) passesMade with Cursor