Feature/multi provider routing policy changes#2244
Open
Aakashwije wants to merge 8 commits into
Open
Conversation
Aakashwije
requested review from
AnuGayan,
Arshardh,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Krishanx92,
PasanT9,
RakhithaRR,
Tharsanan1,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub and
tharindu1st
as code owners
Contributor
📝 WalkthroughAdded multi-provider routing for LLM proxies, allowing a single proxy to select between multiple upstream LLM providers. Key updates include:
WalkthroughThis PR adds Suggested reviewers
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (3)
gateway/examples/llm-proxy.yaml (1)
29-32: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winUse safer credential placeholders in proxy examples.
Line 32 hardcodes a provider loopback key. Prefer an explicit placeholder (or a secret-backed reference) so this manifest is less likely to be reused with fixed credentials.
Suggested update
- value: provider_loopback_key_1234567890abcdef1234567890 + value: REPLACE_WITH_PROVIDER_LOOPBACK_KEY🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/llm-proxy.yaml` around lines 29 - 32, The auth section in the llm-proxy.yaml example file contains a hardcoded provider loopback key in the value field that should not be used as-is in production. Replace this hardcoded credential value with an explicit placeholder string (such as ${PROVIDER_LOOPBACK_KEY} or similar reference syntax) that makes it obvious this is an example requiring substitution with actual credentials, preventing accidental reuse of the example manifest with fixed credentials.gateway/examples/openai-multi-provider-proxy.yaml (1)
29-48: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winUse secret-backed or placeholder auth values in the multi-provider example.
The primary and additional provider auth blocks embed fixed loopback keys. Prefer secret references or explicit placeholders to promote safer defaults when users copy this manifest.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/openai-multi-provider-proxy.yaml` around lines 29 - 48, Replace the hardcoded loopback key values in the auth blocks for the primary provider and all additionalProviders (anthropic-provider, azure-openai-provider, and mistral-provider) with either secret references (e.g., valueFrom.secretKeyRef or environment variable references) or explicit placeholder values (e.g., "${PROVIDER_API_KEY}" or "YOUR_API_KEY_HERE") that clearly indicate users must substitute their own credentials before deploying the manifest.gateway/gateway-controller/pkg/utils/llm_transformer_test.go (1)
307-313: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick winMake auth-policy assertions order-independent.
The test currently ties provider checks to fixed slice indexes. Please match by
ExecutionCondition(or provider key) before asserting header values so valid reorderings don't create false failures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/gateway-controller/pkg/utils/llm_transformer_test.go` around lines 307 - 313, The test assertions in the diff are order-dependent, checking fixed indexes [0] and [1] for specific providers which will fail if the authPolicies slice is reordered. Instead of relying on index positions, iterate through the authPolicies slice and match each policy by its ExecutionCondition content (looking for "openai-provider" and "anthropic-provider"), then assert the corresponding header value using firstRequestHeaderValue against the matched policy's Params. This makes the test resilient to any reordering of the authPolicies slice.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gateway/examples/anthropic-openai-proxy.yaml`:
- Around line 9-12: The header comment in the Single-provider mode section
contains outdated parameter references. Update the comment to accurately reflect
that the configuration uses provider.id (set at line 30) and the translator
policy parameters are provider/upstreamName, rather than referring to omitting
"id" and metadata["selected_provider"]. This clarifies the actual configuration
pattern being used and removes the ambiguity in the parameter naming.
In `@gateway/examples/azure-openai-provider.yaml`:
- Around line 32-50: The api-key-auth authentication plugin is currently
configured to only protect the /openai/deployments/{deployment}/chat/completions
endpoint, but the accessControl.exceptions section allows four different paths.
To align these, add the three missing paths to the api-key-auth paths array:
/openai/deployments/{deployment}/completions (POST),
/openai/deployments/{deployment}/embeddings (POST), and /openai/models (GET).
This ensures all allowed endpoints enforce the same authentication rule.
- Line 26: The upstream.url field on line 26 is missing a value and only
contains a comment, which causes it to parse as null and makes the YAML
structurally invalid for users. Replace the empty value with a concrete
placeholder string (such as a sample URL or a descriptive placeholder text) so
the configuration file remains valid YAML before users edit it with their actual
endpoint URL.
In `@gateway/examples/azure-openai-proxy.yaml`:
- Around line 27-29: The proxy configuration references the
`azure-openai-provider` but is missing the `provider.auth` configuration
required for upstream authorization. Add `provider.auth` configuration to the
provider section that specifies the appropriate authentication method to satisfy
the `X-API-Key` requirement of the target provider, ensuring the proxy can
properly authenticate requests to the upstream service.
In `@gateway/examples/gemini-provider.yaml`:
- Around line 26-41: The api-key-auth policy in the policies section only
protects the /v1beta/models/{model}:generateContent endpoint, but the
accessControl.exceptions section allows both
/v1beta/models/{model}:generateContent and
/v1beta/models/{model}:streamGenerateContent endpoints. Add a second entry in
the api-key-auth policy paths array to also protect the
/v1beta/models/{model}:streamGenerateContent endpoint with the same
authentication requirement (key: X-API-Key in header), ensuring consistent auth
coverage across all allowed endpoints.
In `@gateway/examples/mistral-provider.yaml`:
- Around line 25-28: In the mistral-provider.yaml file's auth section, the value
for the Authorization header should follow the Bearer token format instead of
the plain placeholder currently shown. Replace the value field (currently set to
$dxxxd) with a Bearer-formatted placeholder that clearly demonstrates proper
authentication format for API key authentication, such as using "Bearer" prefix
with an appropriate token placeholder to make the example valid and immediately
usable.
- Around line 29-46: The api-key-auth policy only covers the
/v1/chat/completions endpoint in the policies section, but the
accessControl.exceptions configuration allows unauthenticated access to
/v1/embeddings and /v1/models as well. To fix this inconsistency, add the
missing /v1/embeddings and /v1/models paths to the api-key-auth policy's paths
list with appropriate methods (POST for /v1/embeddings and GET for /v1/models),
ensuring all endpoints listed in accessControl.exceptions have matching
authentication policy coverage.
In `@gateway/examples/openai-provider.yaml`:
- Around line 30-50: The api-key-auth plugin only protects the /chat/completions
endpoint, but accessControl.exceptions allows access to five endpoints:
/chat/completions, /completions, /embeddings, /models, and /models/{modelId}.
Add all four additional paths (/completions, /embeddings, /models, and
/models/{modelId}) to the api-key-auth plugin's paths section with their
respective HTTP methods (POST for /completions and /embeddings, GET for /models
and /models/{modelId}) to ensure consistent authentication protection across all
allowed endpoints.
In `@mistral-proxy.yaml`:
- Around line 217-233: The translator parameter definitions in the
openai-to-mistral and openai-to-anthropic translator blocks use `id` but should
use `provider` to match the translation policy contract. In both translator
blocks (identified by their name fields), replace the `id` parameter with
`provider` in the params section. Additionally, add an explicit `upstreamName`
parameter to the openai-to-anthropic translator block to prevent falling back to
the default upstream, ensuring correct routing behavior as intended by the
configuration.
In `@platform-api/src/api/generated.go`:
- Around line 1868-1876: The LLMProxyAdditionalProvider struct is missing an
`auth` field that is needed to represent provider-level authentication
configuration in the multi-provider payload. Add an `auth` field to the
LLMProxyAdditionalProvider struct with appropriate JSON and YAML tags (likely
omitempty) to enable end-to-end representation of auth configuration through the
platform API types, maintaining parity with the full provider specification.
In `@platform-api/src/internal/model/llm.go`:
- Around line 221-227: The LLMProxyAdditionalProvider struct definition is
missing an Auth field which prevents per-provider authentication configuration
from being represented and propagated through deployment mapping. Add an Auth
field to the LLMProxyAdditionalProvider struct to capture optional
authentication details, and then wire this field through the corresponding DTO
layer and mapper functions to ensure the auth configuration is properly
serialized, transferred, and applied throughout the system.
In `@platform-api/src/internal/service/llm.go`:
- Around line 633-639: The Create and Update methods in the llm.go service
validate only the primary provider ID (req.Provider.Id) but fail to validate the
provider IDs referenced in AdditionalProviders before mapping them with
mapAdditionalProvidersAPIToModel. Add validation logic before the mapping that
checks each provider ID in req.AdditionalProviders exists in the system, similar
to how the primary provider is validated. This validation should occur in both
the Create and Update methods (referenced around lines 633-639 and 846-852) to
prevent invalid provider references from being persisted to the database.
In `@platform-api/src/resources/openapi.yaml`:
- Around line 10382-10403: The LLMProxyAdditionalProvider schema definition is
missing an auth property that is required to keep the API contract aligned with
other layers in this PR. Add an optional auth property to the
LLMProxyAdditionalProvider object schema in openapi.yaml that allows
configuration of authentication for additional providers. This property should
follow the same pattern and structure used for auth configuration in related
provider schemas to maintain consistency across the API contract.
---
Nitpick comments:
In `@gateway/examples/llm-proxy.yaml`:
- Around line 29-32: The auth section in the llm-proxy.yaml example file
contains a hardcoded provider loopback key in the value field that should not be
used as-is in production. Replace this hardcoded credential value with an
explicit placeholder string (such as ${PROVIDER_LOOPBACK_KEY} or similar
reference syntax) that makes it obvious this is an example requiring
substitution with actual credentials, preventing accidental reuse of the example
manifest with fixed credentials.
In `@gateway/examples/openai-multi-provider-proxy.yaml`:
- Around line 29-48: Replace the hardcoded loopback key values in the auth
blocks for the primary provider and all additionalProviders (anthropic-provider,
azure-openai-provider, and mistral-provider) with either secret references
(e.g., valueFrom.secretKeyRef or environment variable references) or explicit
placeholder values (e.g., "${PROVIDER_API_KEY}" or "YOUR_API_KEY_HERE") that
clearly indicate users must substitute their own credentials before deploying
the manifest.
In `@gateway/gateway-controller/pkg/utils/llm_transformer_test.go`:
- Around line 307-313: The test assertions in the diff are order-dependent,
checking fixed indexes [0] and [1] for specific providers which will fail if the
authPolicies slice is reordered. Instead of relying on index positions, iterate
through the authPolicies slice and match each policy by its ExecutionCondition
content (looking for "openai-provider" and "anthropic-provider"), then assert
the corresponding header value using firstRequestHeaderValue against the matched
policy's Params. This makes the test resilient to any reordering of the
authPolicies slice.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 164d0672-ed28-4b5e-986a-d90317dec193
📒 Files selected for processing (41)
event-gateway/default-policies/openai-header-router.yamlevent-gateway/default-policies/openai-to-anthropic.yamlevent-gateway/default-policies/openai-to-azure-openai.yamlevent-gateway/default-policies/openai-to-gemini.yamlevent-gateway/default-policies/openai-to-mistral.yamlgateway/build-manifest.yamlgateway/build.yamlgateway/examples/anthropic-openai-proxy.yamlgateway/examples/anthropic-provider.yamlgateway/examples/azure-openai-provider.yamlgateway/examples/azure-openai-proxy.yamlgateway/examples/gemini-provider.yamlgateway/examples/llm-provider.yamlgateway/examples/llm-proxy.yamlgateway/examples/mistral-openai-proxy.yamlgateway/examples/mistral-provider.yamlgateway/examples/openai-multi-provider-proxy.yamlgateway/examples/openai-provider.yamlgateway/gateway-controller/api/management-openapi.yamlgateway/gateway-controller/pkg/api/management/generated.gogateway/gateway-controller/pkg/config/llm_validator.gogateway/gateway-controller/pkg/utils/llm_transformer.gogateway/gateway-controller/pkg/utils/llm_transformer_test.gogateway/sample-policies/slugify-body/policy-definition.yamlgateway/system-policies/analytics/policy-definition.yamlkubernetes/gateway-operator/api/v1alpha1/llmproxy_types.gokubernetes/gateway-operator/api/v1alpha1/zz_generated.deepcopy.gokubernetes/gateway-operator/config/crd/bases/gateway.api-platform.wso2.com_llmproxies.yamlkubernetes/gateway-operator/internal/controller/llmproxy_controller.gokubernetes/gateway-operator/internal/controller/llmproxy_controller_test.gokubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.gokubernetes/gateway-operator/internal/controller/management_valuefrom_enqueue.gokubernetes/gateway-operator/internal/controller/management_valuefrom_fingerprint.gokubernetes/helm/operator-helm-chart/crds/gateway.api-platform.wso2.com_llmproxies.yamlmistral-proxy.yamlplatform-api/src/api/generated.goplatform-api/src/internal/dto/llm_deployment.goplatform-api/src/internal/model/llm.goplatform-api/src/internal/service/llm.goplatform-api/src/internal/service/llm_deployment.goplatform-api/src/resources/openapi.yaml
… in provider examples All allowed paths in accessControl.exceptions now have matching api-key-auth entries so loopback callers must present an API key for every exposed endpoint, not just chat/completions.
…access control exceptions - azure-openai-proxy: add provider.auth so the proxy can authenticate against the provider's api-key-auth policy on the loopback hop - gemini-provider: add streamGenerateContent to api-key-auth paths to match the accessControl exceptions list
… auth, align gemini api-key-auth paths - azure-openai-provider: replace null url comment with a valid placeholder - azure-openai-proxy: add missing provider.auth for loopback authentication - gemini-provider: add streamGenerateContent to api-key-auth to match accessControl exceptions
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
gateway/examples/openai-multi-provider-proxy.yaml (1)
33-43: 🎯 Functional Correctness | 🟠 MajorRemove the unsupported
transformerblock.The
transformerfield is not defined in theadditionalProvidersschema; valid fields are limited toid,as, andauth. This block will likely cause a validation error or be ignored. The intended translation logic is already defined in theopenai-to-anthropicpolicy further down in the file.Incorrect snippet to remove
transformer: type: openai-to-anthropic version: v1 params: model: claude-sonnet-4-5-20250929🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/openai-multi-provider-proxy.yaml` around lines 33 - 43, The additionalProviders entry for anthropic-provider includes an unsupported transformer block that is not part of the schema. Remove the transformer section from this provider and keep only the allowed fields on the additionalProviders item (such as id, as, and auth); the openai-to-anthropic logic should remain in the separate policy definition later in the file.
🧹 Nitpick comments (1)
gateway/examples/openai-multi-provider-proxy.yaml (1)
102-139: 🎯 Functional Correctness | 🔵 TrivialRemove redundant
executionConditionfromopenai-to-anthropictranslator.The
openai-to-anthropictranslator (lines 102-111) includes an explicitexecutionCondition, whileopenai-to-azure-openai,openai-to-mistral, andopenai-to-geminirely solely on theidparameter in theirparams(lines 120, 129, 138). Per the translator policy schema, setting theprovider(mapped to theidhere) automatically gates execution whenmetadata["selected_provider"]matches. Remove line 103 to ensure consistency across all translators.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/openai-multi-provider-proxy.yaml` around lines 102 - 139, Remove the redundant executionCondition from the openai-to-anthropic translator so it follows the same provider-gated behavior as openai-to-azure-openai, openai-to-mistral, and openai-to-gemini. In the openai-to-anthropic block, keep the id value in params as the provider selector and delete the explicit metadata["selected_provider"] check; the translator policy schema already uses the provider/id field to gate execution consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@gateway/examples/openai-multi-provider-proxy.yaml`:
- Around line 33-43: The additionalProviders entry for anthropic-provider
includes an unsupported transformer block that is not part of the schema. Remove
the transformer section from this provider and keep only the allowed fields on
the additionalProviders item (such as id, as, and auth); the openai-to-anthropic
logic should remain in the separate policy definition later in the file.
---
Nitpick comments:
In `@gateway/examples/openai-multi-provider-proxy.yaml`:
- Around line 102-139: Remove the redundant executionCondition from the
openai-to-anthropic translator so it follows the same provider-gated behavior as
openai-to-azure-openai, openai-to-mistral, and openai-to-gemini. In the
openai-to-anthropic block, keep the id value in params as the provider selector
and delete the explicit metadata["selected_provider"] check; the translator
policy schema already uses the provider/id field to gate execution consistently.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 54dfde27-92bb-460e-b570-e74a36f4340f
📒 Files selected for processing (2)
gateway/examples/mistral-provider.yamlgateway/examples/openai-multi-provider-proxy.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
- gateway/examples/mistral-provider.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
LLM Gateway users are increasingly building AI applications that need to target
multiple LLM providers (OpenAI, Anthropic, Azure OpenAI, Mistral, Gemini) through
a single, OpenAI-compatible endpoint. Previously, each
LLMProxywas hard-wiredto a single upstream provider, which forced users to deploy separate proxies or
handle provider selection in their application code.
This PR introduces multi-provider routing for LLM proxies together with a set
of built-in OpenAI-format translation policies that convert outgoing requests
and incoming responses between the OpenAI Chat Completions format and each
provider's native API.
Goals
LLMProxyto declare multiple LLM backend providers as namedselectable upstreams (
additionalProviders).openai-header-router) that selects thetarget provider per request and publishes the selection into shared metadata.
OpenAI Chat Completions schema and each provider's native wire format:
openai-to-anthropic→ Anthropic Messages APIopenai-to-azure-openai→ Azure OpenAI REST APIopenai-to-mistral→ Mistral AI APIopenai-to-gemini→ Google GeminigenerateContentAPIprovider via conditional execution (
ExecutionCondition).Approach
additionalProvidersfield onLLMProxyA new optional
additionalProviderslist is added toLLMProxyConfigDataacrossall layers (Kubernetes CRD, management API, platform API). Each entry references
a deployed
LlmProviderbyid, with an optionalasalias used by downstreampolicies to address the upstream cluster, and an optional
authblock forprovider-level API key injection.
Transformer changes (
llm_transformer.go)The gateway controller transformer now:
UpstreamDefinition.ExecutionCondition) so that each provider'sAuthorization/X-Api-Keyheader is only injected when
request.Metadata['selected_provider']matchesthat provider's name. The primary provider's auth policy fires when the
metadata key is absent (default) or matches the primary id.
Validator changes (
llm_validator.go)Validates
additionalProvidersfor:idvaluesasaliases,or
idvalues)type,header,value) whenauthis setRouting and translation policy architecture
Client (OpenAI format)
│
▼
openai-header-router ← reads X-Provider header, sets selected_provider
│
├── openai-to-anthropic ← runs only when selected_provider == "anthropic-"
├── openai-to-azure-openai ← runs only when selected_provider == "azure-"
├── openai-to-mistral ← runs only when selected_provider == "mistral-"
└── openai-to-gemini ← runs only when selected_provider == "gemini-"
Each translator policy operates in two modes:
providerparameter is set; the policyevaluates the
selected_providermetadata and is a no-op for non-matchingrequests.
provideris omitted; the translatorruns on every request.
Translation policies handle:
temperature,top_p,max_tokens, etc.)/v1/messagesfor Anthropic,/{apiVersion}/models/{model}:generateContentfor Gemini)ChatCompletionshapeExample multi-provider proxy manifest
User stories
Documentation
gateway/examples/ serve as the primary reference for this feature.
Automation tests
Unit tests
- Each additional provider is registered as a named UpstreamDefinition.
- Two upstream-auth policies are emitted for the two providers.
- Each policy's ExecutionCondition references only its own provider name.
- API key values are not crossed between providers.
gateway/gateway-controller/pkg/config/llm_validator_test.go (existing suite) — additional additionalProviders validation paths covered.
Integration tests
Security checks
Samples
gateway/examples/contains ready-to-deploy YAML manifests for each provider combination:openai-multi-provider-proxy.yamlanthropic-openai-proxy.yamlazure-openai-proxy.yamlmistral-openai-proxy.yamlanthropic-provider.yamlLlmProviderfor Anthropicazure-openai-provider.yamlLlmProviderfor Azure OpenAIgemini-provider.yamlLlmProviderfor Google Geminimistral-provider.yamlLlmProviderfor Mistralopenai-provider.yamlLlmProviderfor OpenAIllm-provider.yaml/llm-proxy.yamlRelated PRs
Test Environment