Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions packages/agent-sessions/src/session-summary.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1566,6 +1566,40 @@ describe("per-model cost, tools and failure groups", () => {
])
})

// The tests above read spans ingested before the gateway stamped them. On a
// stamped span its `maple_ai.tool_call` verdict decides, as on the list.
it("counts a stamped tool span by the gateway's verdict alone", () => {
const stamped = (spanId: string, startMs: number, mapleToolCall: number, toolCallResult?: string) =>
toolSpan({
spanId,
traceId: `trace-${spanId}`,
toolName: "delete_file",
startMs,
durationMs: 1,
genAi: { mapleLlmCall: 0, mapleToolCall, toolCallId: "call_a", toolCallResult },
})
const summary = summarize([
// Google ADK's confirmation request: no call.
stamped(
"paused",
0,
0,
'{"error": "This tool call requires confirmation, please approve or reject."}',
),
// LlamaIndex ends a step waiting for a human in error, which the
// gateway stamped neither a call nor a failure.
{ ...stamped("waiting", 500, 0), statusCode: "Error" },
// No result and no framework mark: a call, not merged into the one
// that follows under the same id.
stamped("no-result", 1_000, 1),
stamped("approved", 2_000, 1, "deleted /tmp/scratch-notes.txt"),
])

expect(summary.work.toolCalls).toBe(2)
expect(summary.tools[0]!.events.map((event) => event.spanId)).toEqual(["no-result", "approved"])
expect(summary.failures.errors).toBe(0)
})

it("keeps two calls that share an id when both returned, and every call captured without payloads", () => {
const lane = (spanId: string, traceId: string, startMs: number, result?: string) =>
toolSpan({
Expand Down
26 changes: 14 additions & 12 deletions packages/agent-sessions/src/session-summary.ts
Original file line number Diff line number Diff line change
Expand Up @@ -875,18 +875,19 @@ function modelUsage(
}

/**
* The tool calls the session made, in start order, a call paused for a human
* counted once. The interrupted call leaves a tool span that recorded no
* result and did not fail, and the resumed turn opens another under the same
* `gen_ai.tool.call.id` that carries the result on its attributes (Strands)
* — so the paused copy is dropped when a later copy with a result exists.
* Not covered, and still counted twice: Google ADK (the outcome is only in
* `gcp.vertex.agent.tool_response`, and the paused copy's confirmation
* request reads as a result), Strands versions that record results in span
* events, and OpenAI Agents (no call id). Nothing else is merged: two calls
* that merely share an id (parallel lanes, a provider numbering its calls per
* turn) both carry results, and a session captured without payloads keeps
* every span.
* The tool calls the session made, in start order. On a span the ingest
* gateway stamped, its verdict alone (`maple_ai.tool_call`, what the list
* sums): the copy a call paused for a human's approval leaves is stamped no
* call by its framework's explicit mark, and nothing is merged.
*
* The spans ingested before the gateway stamped them keep the old merge until
* they age out of the 30-day TTL: a call paused for a human leaves a tool span
* that recorded no result and did not fail, and the resumed turn opens
* another under the same `gen_ai.tool.call.id` that carries the result on its
* attributes (Strands), so the paused copy is dropped when a later copy with a
* result exists. Two calls that merely share an id (parallel lanes, a provider
* numbering its calls per turn) both carry results, and a session captured
* without payloads keeps every span.
*/
function countedToolCalls(ordered: readonly AiSessionSpan[]): readonly AiSessionSpan[] {
const tools = ordered.filter((span) => classifyAiSpan(span) === "tool")
Expand All @@ -897,6 +898,7 @@ function countedToolCalls(ordered: readonly AiSessionSpan[]): readonly AiSession
if (callId !== undefined && callId !== "" && recorded(span)) resumedAt.set(callId, spanStartMs(span))
}
return tools.filter((span) => {
if (span.genAi.mapleLlmCall !== undefined) return true
const resumed = resumedAt.get(span.genAi.toolCallId ?? "")
return resumed === undefined || resumed <= spanStartMs(span) || recorded(span) || spanFailed(span)
})
Expand Down
6 changes: 4 additions & 2 deletions packages/agent-sessions/src/session-turns.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,11 +127,13 @@ const FAILED_RESPONSE_STATUSES = new Set(["failed", "error"])
* `gen_ai.response.status` counts too. Scoped to AI spans because HTTP
* instrumentation legitimately stamps `error.type` on expected 4xx requests
* whose span status is deliberately not `Error`. On a span the ingest gateway
* stamped, its verdict (`MAPLE_AI_STAMP_ATTRS.error`), which is this rule.
* stamped, its verdict (`MAPLE_AI_STAMP_ATTRS.error`), which is this rule
* less the copy a call paused for a human's approval leaves, which some
* frameworks end in error.
*/
export function spanFailed(span: AiSessionSpan): boolean {
if (span.statusCode === "Error") return true
if (span.genAi.mapleLlmCall !== undefined) return span.genAi.mapleError === 1
if (span.statusCode === "Error") return true
if (!span.isAiSpan) return false
const errorType = span.genAi.errorType
if (errorType !== undefined && errorType !== "") return true
Expand Down
Loading
Loading