feat: add X-Session-Id header for proxy cache routing affinity#31511
Open
songchaow wants to merge 1 commit into
Open
feat: add X-Session-Id header for proxy cache routing affinity#31511songchaow wants to merge 1 commit into
songchaow wants to merge 1 commit into
Conversation
songchaow
force-pushed
the
feat/x-session-id-header
branch
from
35e9346 to
9b67f1c
Compare
Providers like LLM++ (TimiAI) use X-Session-Id to route requests from the same session to the same upstream account, maximizing KV-Cache hit rate. Without this header, multi-turn conversations get scattered across different upstream accounts, causing frequent cache misses. The header is sent alongside the existing x-session-affinity header for all non-opencode providers.
songchaow
force-pushed
the
feat/x-session-id-header
branch
from
9b67f1c to
e1e97f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes N/A (improvement — no existing issue)
Type of change
What does this PR do?
Adds an
X-Session-Idheader (set to the session ID) alongside the existingx-session-affinityheader for all non-opencode providers.Why: Anthropic API proxies (e.g. enterprise gateways that front multiple upstream Anthropic accounts) use
X-Session-Idto implement routing affinity — pinning all requests from the same session to the same upstream account. This maximizes KV-Cache hit rate on the upstream side because the cached prompt prefix lives on that specific account.Without this header, multi-turn conversations get scattered across different upstream accounts by the load balancer, causing frequent prompt cache misses and significantly higher latency + cost.
The header name
X-Session-Idis the convention used by Anthropic-compatible proxy providers (documented in their API specs). It is sent unconditionally for non-opencode providers — providers that do not recognize it will simply ignore an unknown header, so there is no behavioral change for direct Anthropic API usage or other providers.The change is a single line addition in
packages/opencode/src/session/llm/request.ts.How did you verify your code works?
bun run build --skip-embed-web-ui --singleX-Session-Idis present in all outgoing requestscache_creation_input_tokens: 1538on request 1, thencache_read_input_tokens: 1538on request 2 — confirming the routing affinity enables cache hitsScreenshots / recordings
N/A — backend-only change, no UI impact.
Checklist