Add LiteLLM as a new LLM provider - #909
Open
prodmanpd wants to merge 1 commit into
Open
Conversation
Add LiteLLM as a first-class provider. LiteLLM is a self-hosted AI gateway that exposes a single OpenAI-compatible endpoint in front of 100+ models across providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, and more). The provider mirrors the existing OpenAI-compatible providers: it uses the shared StreamOpenAICompatibleChatCompletion helper for streaming chat and LoadModelsResponse against /v1/models for automatic model discovery. The user supplies the base URL of their proxy; it is normalized and the OpenAI-compatible /v1/ path is appended. It uses the self-hosted trust policy and confidence handling, since the gateway owner decides which downstream providers requests are routed to. Wired through the standard plumbing: the LLMProviders enum, LLMProvidersExtensions (name, confidence, factory, hostname/API-key requirements), ProviderExtensions (capabilities + reasoning), and SettingsManager (per-scheme confidence levels). Hostname and API key are both required. Added a user-facing changelog entry and updated the configuration-plugin docs to include the new LITE_LLM confidence key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds LiteLLM as a first class LLM provider. LiteLLM is a self hosted AI gateway that exposes a single OpenAI compatible endpoint in front of 100+ models (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Mistral, and many more), with central key management, budgets, and fallbacks. Users point AI Studio at their LiteLLM proxy, models are auto discovered via
/v1/models, and chat is streamed through the existing OpenAI compatible path.Changes
Provider/LiteLLM/ProviderLiteLLM.cs(new): first class provider. MirrorsProviderDeepSeek(OpenAI compatible streaming chat plus/v1/modelsmodel loading), but takes a user supplied hostname (likeSelf-hosted), normalized to{host}/v1/, and uses the self hosted TLS trust policy so private or self signed proxies work.Provider/LLMProviders.cs: newLITE_LLMenum member.Provider/LLMProvidersExtensions.cs: display name, secret id, confidence, provider factory (passes the hostname), API key and hostname requirements. Embeddings and transcription are not claimed (chat only, same as DeepSeek).Settings/ProviderExtensions.cs: model capabilities via the generic open source detector (the gateway serves arbitrary models).Settings/ProviderExtensions.Reasoning.cs: LiteLLM joins the multi detector reasoning group so reasoning parameters are recognized regardless of the downstream model.Settings/SettingsManager.cs: default confidence level per scheme, mirroringSelf-hosted(user operated infrastructure).Plugins/configuration/plugin.lua: enterprise config docs list the newLITE_LLMprovider key.wwwroot/changelog/v26.8.1.md: user facing changelog entry.Confidence handling
LiteLLM is a self operated gateway: the user runs the proxy and decides which downstream providers it routes to, so the destination cannot be known in advance. It is therefore treated like a self hosted endpoint (
Confidence.SELF_HOSTED), and the user assigns the trust level themselves.Tests
1. Build: the repo's canonical
cd app/Build && dotnet run buildcompiled the full solution (including the new provider, which is part of themindworkAIStudioassembly) with 0 errors on .NET 9, producing the runnablemindworkAIStudio.dlland the Tauri release binary.2. Live E2E through a real LiteLLM proxy to Azure OpenAI. This exercises the exact surface
ProviderLiteLLMrelies on: model discovery (GET {host}/v1/models, used by the "Load models" button, deserialized intoModelsResponse) and streaming chat (POST {host}/v1/chat/completions,stream=true, SSEdata:lines parsed by the sharedStreamOpenAICompatibleChatCompletionandChatCompletionDeltaStreamLinepath).Model discovery (
GET /v1/models):{"data":[ {"id":"azure/gpt-4.1"}, {"id":"azure/gpt-4o"}, {"id":"azure/gpt-4o-mini"}, {"id":"anthropic/claude-sonnet-4-6"}, {"id":"anthropic/claude-opus-4-6"}, {"id":"anthropic/claude-haiku-4-5"} ... (10 models total) ]}Streaming chat completion (
POST /v1/chat/completions,model=azure/gpt-4o,stream=true):The stream assembles to
LITELLM_OK, confirming the full chain: AI Studio's OpenAI compatible request shape, then the LiteLLM proxy, then Azure OpenAI, then SSE deltas parsed back throughProviderLiteLLM. This was validated at the transport layer against a local LiteLLM proxy using the exact endpoints and request and response shapes the provider builds and consumes.Risk / Compatibility
Self-hostedprovider.Example usage
In the app: Settings, then Providers, then Add.
http://localhost:4000(AI Studio appends/v1/)./v1/models), pick a model, and chat.In code, the provider is used exactly like the other providers (it implements
IProvider). Construct it with the proxy hostname (normalized to{host}/v1/), discover models, and stream a chat completion: