feat(lib): single telemetry row per tool call with duration and outcome - #430
ruturaj-browserstack merged 9 commits into
Conversation
…utcome Every tool today writes its MCPInstrumentation row at entry, before any work happens, so nothing in telemetry carries a duration and a tool that returns `isError: true` without throwing counts as a success. Add one wrapper in server-factory around every registered tool's handler. It times the call and emits a separate `MCPToolCompleted` event when the handler settles, with `duration_ms` and `outcome` (ok / error_result / threw). The existing entry and catch rows are untouched, and the new event uses its own event_type so no query or dashboard keyed on MCPInstrumentation changes. The wrapper is transparent (result passed through, throws rethrown), idempotent (a second call on the same map is a no-op), skips task-style handlers, and never lets a telemetry failure affect the tool call. Handler is assigned directly rather than via tool.update() to avoid a tools/list_changed notification at registration time. Exported as `instrumentToolLatency` so the remote wrapper can apply the same timing to the tools it registers itself (uploadAsset, TFA plugin). Refs AIMCP-225 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (7)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📜 Recent review details🔇 Additional comments (7)
📝 WalkthroughWalkthroughThe change adds consolidated tool-call telemetry with error classification, latency, outcomes, and call-time client information. It wraps registered tool handlers, wires instrumentation into server setup, exports the new APIs, and adds coverage for success, failure, concurrency, and telemetry faults. ChangesTool call telemetry
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant ToolHandler
participant withToolCall
participant trackMCP
participant InstrumentationAPI
ToolHandler->>withToolCall: execute tool handler
ToolHandler->>trackMCP: record call context or error
withToolCall->>withToolCall: calculate duration and outcome
withToolCall->>InstrumentationAPI: post consolidated telemetry event
withToolCall-->>ToolHandler: return result or rethrow error
Merge Risk: ⚪ Minimal · up to Each tool call now produces one telemetry row with duration, outcome and error class. Real tool calls reach this instrumentation. Telemetry failures do not affect tool results. The only visible change is that dashboards now count failed calls separately, which the PR describes. No blocking risk remains. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
A rabbit reads each line, Comment |
…s ones Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…mpleted The Rails endpoint (railsApp sdk_controller#event) allowlists event types and rejects anything else with 400 INVALID_EVENT_TYPE; a probe confirmed the MCPToolCompleted rows never reached BigQuery while the entry/catch rows did. Reuse the accepted event type and mark the row with `phase: "completed"`, omitting `success`, so queries counting success='true' / 'false' are unaffected. Verified in BigQuery: completion rows land with phase, outcome and duration_ms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Failure rows carried only the free-text error_message, so grouping failures by
cause meant regex over strings. Add `classifyError` and stamp `error_class` on
the catch-path row: auth_error, not_found, rate_limited, validation, timeout,
network, server_error, unknown.
Reads the HTTP status from the axios error object, falls back to the status
embedded in plain Error messages thrown by utils ("…: 404 Not Found",
"status code 401"), maps transport codes (ECONNABORTED, ECONNREFUSED…), Zod
errors, and the entitlement refusal message. The success row is unchanged.
Refs AIMCP-225
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the call settles Replace the entry row + completion row pair with a single row per call. The tools' existing trackMCP calls (entry and catch block) no longer post; inside an instrumented call they record into an AsyncLocalStorage context and the wrapper writes one row when the handler settles, carrying success, duration_ms, outcome (ok / error_result / threw) and, on failure, error_message / error_type / error_class. - success=false when the handler reported an error via trackMCP or threw; failure counts stay identical to today, but a failed call is now one row instead of two. - Invocation count is COUNT(*) per tool (excluding `started`), no phase filter needed. - trackMCP outside an instrumented call (the started heartbeat, tools a host registers without wrapping) behaves exactly as before. - Context is request-scoped (AsyncLocalStorage), so concurrent calls in the remote wrapper never share state; verified by test. - trackMCPCompleted and the phase field are gone; withToolCall is exported for hosts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s as "unknown"
Checked the classifier against 30 days of production failure messages rather than the
synthetic errors it was tested with. Of the 13 most common failure messages (1,718
failures), exactly one classified into a real bucket; everything else fell to "unknown".
The cause is upstream of the classifier: utilities catch the API error and re-throw a
human-readable string, discarding `code` and `response.status`. So classification falls
to message matching, and the patterns ("unauthorized", "forbidden", "timed out") do not
match the product's actual vocabulary ("is required", "you must provide", "you do not
have access to").
Shipping a field that is 99% one value is worse than not shipping it. The row keeps
error_message and error_type, which is what readers use today. Revisit either by tuning
patterns to the real vocabulary, or by preserving the status at the throw sites so the
classifier gets structured data instead of prose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Telemetry currently writes one row when a tool starts and a second row if it fails, so no row carries how long the call took, and a tool that returns an error result without throwing is recorded as a success.
This makes it one row per call, written when the call settles.
instrumentToolLatencywraps each registered handler. The existingtrackMCPcalls at entry and in catch blocks record into a request-scoped context instead of posting, and the wrapper writes the single row when the handler finishes. No tool files change.duration_msandoutcome(ok/error_result/threw) to the existing fields.successkeeps its meaning: false when the handler reported an error or threw. Failure counts are unchanged; a failed call is now one row instead of two.trackMCPoutside a wrapped call behaves as before.withToolCallandinstrumentToolLatencyare exported for hosts that register their own tools.Tests cover ok / error-result / throw paths, concurrency, idempotency, and that telemetry failures never affect a tool call.