[BUG] Incorrect context length while using with LM studio

[navroop005]

Problem (one or two sentences)

There is huge mismatch in context length shown by Roo code extention and the actual context length in roo code.
As shown by Roo code:

Logs in LM studio:

Context (who is affected and when)

Happens while using it with LM studio and Qwen 3.5 35B-A3B

Reproduction steps

While using the Roo code as intended, slowly the difference between the estimated tokens start getting more and more.

Expected result

The token calculation in extension should match the actual token count

Actual result

Different token count in Roo code and LM studio

Variations tried (optional)

No response

App Version

Version: 3.51.1 (7c9722b)

API Provider (optional)

LM Studio

Model Used (optional)

Qwen 3.5 35B-A3B

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

[roomote-v0]

I have analyzed this issue and identified the root cause. Here is my proposed implementation plan:

Root Cause

The LM Studio provider (src/api/providers/lm-studio.ts) is the only OpenAI-compatible provider that does NOT request usage data from the API. Instead, it manually estimates token counts using tiktoken (an OpenAI tokenizer) with a 1.5x "fudge factor" applied on top. This causes two compounding sources of inaccuracy:

1. Wrong tokenizer: The o200k_base tokenizer is designed for OpenAI models, not Qwen models. Different models use different tokenizers with different vocabularies, so the token counts will inherently differ.
2. 1.5x fudge factor: After the already-inaccurate tiktoken count, a 50% increase is applied as a safety margin (TOKEN_FUDGE_FACTOR = 1.5 in src/utils/tiktoken.ts), inflating the estimate further.

Every other OpenAI-compatible provider (OpenAI, DeepSeek, xAI, OpenRouter, Requesty, LiteLLM, etc.) requests actual usage data from the API using stream_options: { include_usage: true } and reads the real token counts from chunk.usage in the streaming response. LM Studio's OpenAI-compatible API supports this same mechanism.

Proposed Fix

1. Add stream_options: { include_usage: true } to the streaming request params in LmStudioHandler.createMessage()
2. Read chunk.usage from the streaming response to capture server-reported prompt_tokens and completion_tokens
3. Use the server-reported values for the final usage yield, falling back to the current tiktoken estimation only if the server does not report usage (for backward compatibility with older LM Studio versions)
4. Remove the manual pre-counting of input tokens (lines 75-81) and the post-counting of output tokens (lines 153-159) when server-reported values are available
5. Add tests covering both the new path (server-reported usage) and the fallback path

This aligns the LM Studio provider with the pattern used by all other OpenAI-compatible providers and will give users accurate token counts as reported by the actual inference server.

Please mention @roomote in a comment saying that you approve this plan, and I'll proceed with the implementation.