Skip to content

Classify Kernel API failures on tool errors - #133

Open
masnwilliams wants to merge 1 commit into
mainfrom
hypeship/classify-tool-errors
Open

Classify Kernel API failures on tool errors#133
masnwilliams wants to merge 1 commit into
mainfrom
hypeship/classify-tool-errors

Conversation

@masnwilliams

@masnwilliams masnwilliams commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Every failed MCP tool call currently records $mcp_error_type = "Error", so a stale session id, an org hitting its concurrency limit, and a fault on our side are indistinguishable in analytics. This names them.

toolErrorResponse becomes throwToolError, which throws instead of returning an isError result. The thrown error's name carries the Kernel API status — KernelApiError404, KernelApiError429, KernelApiError502 — or the SDK error class for transport failures (APIConnectionTimeoutError).

Why this works

The analytics SDK reads the error category from a thrown error's name. A returned isError result has no error object, so it coerces from the result text and always lands as a generic Error. That's why the existing data has exactly one error type.

Because throwToolError is the single funnel for caught API errors, the split falls out with no per-call-site logic:

  • KernelApiError<status> — the Kernel API rejected the request.
  • Error — the tool itself rejected the call (missing session_id, conflicting arguments, invalid config), which never reaches the API.

Agents see no change

The MCP SDK converts a thrown error back into the same isError text result, and the message string is byte-identical to what the helper produced before. Verified against the real API:

  • manage_browsers get with an unknown session id → Error in manage_browsers (get): 404 browser session '...' not found, isError: true
  • manage_browsers get with no session id → Error: session_id is required for get action., isError: true

Privacy

Status codes only. No message, parameters, or response are captured — confirmed on the captured events — and no $exception events are emitted. $mcp_error_type is already on the send allow-list, so nothing new is added to what leaves the server.

Verification

Ran the route locally against the production API and checked the captured events:

call $mcp_error_type duration message / params / response captured
API 404 on unknown session KernelApiError404 1983 ms none
input guard, no session_id Error 2 ms none

tsc --noEmit and prettier --check pass on every touched file.

Follow-up, deliberately not here

The generic Error bucket still mixes several kinds of tool-side rejection (missing argument vs conflicting arguments vs invalid config). Naming those too is mechanical but touches ~95 call sites, so it belongs in its own change if the split turns out to be worth it.


Note

Low Risk
Mechanical error-handling refactor across MCP tools; agent-visible behavior is intended to stay the same, with analytics-only classification added.

Overview
MCP tool handlers now throw classified errors on Kernel API failures instead of returning isError results, so analytics can record distinct $mcp_error_type values (e.g. KernelApiError404, KernelApiError429) instead of a generic Error.

toolErrorResponse is replaced by throwToolError, which throws a ToolCallError whose name comes from the SDK APIError HTTP status (KernelApiError{status}) or the underlying error class for non-API failures. The user-facing message is unchanged; the MCP SDK still surfaces the same isError text to agents.

All MCP tool catch blocks that handled API errors are updated to call throwToolError. Input validation paths still return errorResponse(...) and remain in the generic Error bucket.

Reviewed by Cursor Bugbot for commit b9aa34d. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mcp Ready Ready Preview Jul 31, 2026 3:39pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant