Skip to content

Add LiteLLM as a new LLM provider - #909

Open
prodmanpd wants to merge 1 commit into
MindWorkAI:mainfrom
prodmanpd:feat/add-litellm-provider
Open

Add LiteLLM as a new LLM provider#909
prodmanpd wants to merge 1 commit into
MindWorkAI:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

Copy link
Copy Markdown

Summary

This PR adds LiteLLM as a first class LLM provider. LiteLLM is a self hosted AI gateway that exposes a single OpenAI compatible endpoint in front of 100+ models (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Mistral, and many more), with central key management, budgets, and fallbacks. Users point AI Studio at their LiteLLM proxy, models are auto discovered via /v1/models, and chat is streamed through the existing OpenAI compatible path.

Changes

  • Provider/LiteLLM/ProviderLiteLLM.cs (new): first class provider. Mirrors ProviderDeepSeek (OpenAI compatible streaming chat plus /v1/models model loading), but takes a user supplied hostname (like Self-hosted), normalized to {host}/v1/, and uses the self hosted TLS trust policy so private or self signed proxies work.
  • Provider/LLMProviders.cs: new LITE_LLM enum member.
  • Provider/LLMProvidersExtensions.cs: display name, secret id, confidence, provider factory (passes the hostname), API key and hostname requirements. Embeddings and transcription are not claimed (chat only, same as DeepSeek).
  • Settings/ProviderExtensions.cs: model capabilities via the generic open source detector (the gateway serves arbitrary models).
  • Settings/ProviderExtensions.Reasoning.cs: LiteLLM joins the multi detector reasoning group so reasoning parameters are recognized regardless of the downstream model.
  • Settings/SettingsManager.cs: default confidence level per scheme, mirroring Self-hosted (user operated infrastructure).
  • Plugins/configuration/plugin.lua: enterprise config docs list the new LITE_LLM provider key.
  • wwwroot/changelog/v26.8.1.md: user facing changelog entry.

Confidence handling

LiteLLM is a self operated gateway: the user runs the proxy and decides which downstream providers it routes to, so the destination cannot be known in advance. It is therefore treated like a self hosted endpoint (Confidence.SELF_HOSTED), and the user assigns the trust level themselves.

Tests

1. Build: the repo's canonical cd app/Build && dotnet run build compiled the full solution (including the new provider, which is part of the mindworkAIStudio assembly) with 0 errors on .NET 9, producing the runnable mindworkAIStudio.dll and the Tauri release binary.

2. Live E2E through a real LiteLLM proxy to Azure OpenAI. This exercises the exact surface ProviderLiteLLM relies on: model discovery (GET {host}/v1/models, used by the "Load models" button, deserialized into ModelsResponse) and streaming chat (POST {host}/v1/chat/completions, stream=true, SSE data: lines parsed by the shared StreamOpenAICompatibleChatCompletion and ChatCompletionDeltaStreamLine path).

Model discovery (GET /v1/models):

{"data":[
  {"id":"azure/gpt-4.1"}, {"id":"azure/gpt-4o"}, {"id":"azure/gpt-4o-mini"},
  {"id":"anthropic/claude-sonnet-4-6"}, {"id":"anthropic/claude-opus-4-6"}, {"id":"anthropic/claude-haiku-4-5"}
  ... (10 models total)
]}

Streaming chat completion (POST /v1/chat/completions, model=azure/gpt-4o, stream=true):

data: {"model":"azure/gpt-4o","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","role":"assistant"}}]}
data: {"...","choices":[{"index":0,"delta":{"content":"L"}}]}
data: {"...","choices":[{"index":0,"delta":{"content":"IT"}}]}
data: {"...","choices":[{"index":0,"delta":{"content":"EL"}}]}
data: {"...","choices":[{"index":0,"delta":{"content":"LM"}}]}
data: {"...","choices":[{"index":0,"delta":{"content":"_OK"}}]}
data: {"...","choices":[{"finish_reason":"stop","index":0,"delta":{}}]}
data: [DONE]

The stream assembles to LITELLM_OK, confirming the full chain: AI Studio's OpenAI compatible request shape, then the LiteLLM proxy, then Azure OpenAI, then SSE deltas parsed back through ProviderLiteLLM. This was validated at the transport layer against a local LiteLLM proxy using the exact endpoints and request and response shapes the provider builds and consumes.

Risk / Compatibility

  • Additive only. No existing provider is touched; the base install is unaffected.
  • Reuses the shared OpenAI compatible streaming and model loading helpers, so behavior and error handling match the other OpenAI compatible providers.
  • The self hosted trust policy allows private or self signed proxy endpoints (whitelisted host), consistent with the Self-hosted provider.

Example usage

In the app: Settings, then Providers, then Add.

  1. Provider: LiteLLM.
  2. Hostname: your proxy base URL, for example http://localhost:4000 (AI Studio appends /v1/).
  3. API key: your LiteLLM virtual or master key.
  4. Click Load models (fetched from /v1/models), pick a model, and chat.

In code, the provider is used exactly like the other providers (it implements IProvider). Construct it with the proxy hostname (normalized to {host}/v1/), discover models, and stream a chat completion:

using AIStudio.Provider;
using AIStudio.Provider.LiteLLM;

// Construct the LiteLLM provider pointed at your proxy.
IProvider provider = new ProviderLiteLLM("http://localhost:4000")
{
    InstanceName = "Company LiteLLM gateway",
};

// Discover the models the proxy serves (GET /v1/models):
ModelLoadResult models = await provider.GetTextModels();

// Stream a chat completion (POST /v1/chat/completions, stream=true).
// chatModel comes from the discovered models; chatThread and settingsManager
// are the app's current chat context.
await foreach (ContentStreamChunk chunk in provider.StreamChatCompletion(chatModel, chatThread, settingsManager))
    Console.Write(chunk.Content);

Add LiteLLM as a first-class provider. LiteLLM is a self-hosted AI
gateway that exposes a single OpenAI-compatible endpoint in front of
100+ models across providers (OpenAI, Anthropic, Google, Azure, AWS
Bedrock, and more).

The provider mirrors the existing OpenAI-compatible providers: it uses
the shared StreamOpenAICompatibleChatCompletion helper for streaming
chat and LoadModelsResponse against /v1/models for automatic model
discovery. The user supplies the base URL of their proxy; it is
normalized and the OpenAI-compatible /v1/ path is appended. It uses the
self-hosted trust policy and confidence handling, since the gateway
owner decides which downstream providers requests are routed to.

Wired through the standard plumbing: the LLMProviders enum,
LLMProvidersExtensions (name, confidence, factory, hostname/API-key
requirements), ProviderExtensions (capabilities + reasoning), and
SettingsManager (per-scheme confidence levels). Hostname and API key
are both required. Added a user-facing changelog entry and updated the
configuration-plugin docs to include the new LITE_LLM confidence key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant