Skip to content

Commit 93b9913

Browse files
MadzionatorMadzionator
authored
feat: InferenceParams specific for backend (#126)
* Add provider inference params and adapt services Introduce IProviderInferenceParams and provider-specific parameter types (LocalInferenceParams, AnthropicParams, OpenAiParams, GeminiParams, GroqCloudParams, DeepSeekParams, OllamaParams, XaiParams). Replace usages of the old InferenceParams with IProviderInferenceParams/LocalInferenceParams across the codebase: Chat now stores ProviderParams (with LocalParams and InferenceGrammar helpers), mappers updated to handle LocalInferenceParams, interfaces and AgentService adjusted to accept IProviderInferenceParams, and tests updated accordingly. LLM and OpenAI-compatible services now apply provider-specific settings (temperature, max_tokens, top_p, etc.) via ApplyProviderParams, and grammar handling was unified to use Chat.InferenceGrammar. Examples updated to pass LocalInferenceParams. These changes enable multi-backend provider support and provider-specific configuration handling. * Validate provider params and add exception Introduce InvalidProviderParamsException for clearer bad-request errors when provider params types mismatch. Add an ExpectedParamsType contract to OpenAiCompatibleService and implement it in several providers (Gemini, GroqCloud, Ollama, OpenAi, Xai, DeepSeek). Perform runtime type checks in OpenAiCompatibleService, LLMService (local inference), and AnthropicService to throw the new exception when chat.ProviderParams is the wrong type. Misc: remove TopK from GeminiParams and stop emitting top_k in Gemini requests; preserve Chat.Backend if already set; minor cleanup (pragmas/using/whitespace). * Add provider params integration tests Add ProviderParamsTests integration test suite that verifies provider-specific inference parameters for OpenAI, Anthropic, Gemini, DeepSeek, GroqCloud, Xai, local (Gemma2), and Ollama models. Each passing test checks the model returns the expected answer when custom params are supplied; additional negative tests assert InvalidProviderParamsException is thrown when wrong provider params are used. Add Xunit.SkippableFact package reference to enable conditional skipping based on environment (API keys, local model file, or Ollama availability). * Add ProviderParamsFactory and wire inference params Introduce ProviderParamsFactory to create IProviderInferenceParams based on BackendType. Update Home.razor to set the backend and inference params on the new chat context (casting to IChatConfigurationBuilder) before preserving message history, ensuring the correct provider settings are applied when switching models. Also simplify ChatService by using the null-coalescing assignment (chat.Backend ??= settings.BackendType) to default the backend. * Rename provider to backend * fix: don't send default parameters; Params may vary by model * Adding additional parameters * Refactor chat helper and AnhropicService Centralize message, image and tool handling into ChatHelper and update services to use it. Added ServiceConstants properties for ToolCalls/ToolCallId/ToolName and replaced hardcoded property keys in LLMService/OpenAiCompatibleService with ServiceConstants.Properties. Moved image extraction, message merging and message-array building logic out of Anthropic/OpenAi services into ChatHelper, made BuildAnthropicRequestBody asynchronous and now use ChatHelper.BuildMessagesArray. Removed duplicated helper methods and consolidated image MIME detection and message content construction. * post merge fixes * versioning
1 parent e65a512 commit 93b9913

42 files changed

Lines changed: 965 additions & 428 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Examples/Examples/Chat/ChatCustomGrammarExample.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ public async Task Start()
2424
await AIHub.Chat()
2525
.WithModel(Models.Local.Gemma2_2b)
2626
.WithMessage("Generate random person")
27-
.WithInferenceParams(new InferenceParams
27+
.WithInferenceParams(new LocalInferenceParams
2828
{
2929
Grammar = personGrammar
3030
})

‎Examples/Examples/Chat/ChatExampleOpenAi.cs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
using Examples.Utils;
22
using MaIN.Core.Hub;
3+
using MaIN.Domain.Configuration.BackendInferenceParams;
34
using MaIN.Domain.Models;
45

56
namespace Examples.Chat;
@@ -15,6 +16,14 @@ public async Task Start()
1516
await AIHub.Chat()
1617
.WithModel(Models.OpenAi.Gpt5Nano)
1718
.WithMessage("What do you consider to be the greatest invention in history?")
19+
.WithInferenceParams(new OpenAiInferenceParams // We could override some inference params
20+
{
21+
ResponseFormat = "text",
22+
AdditionalParams = new Dictionary<string, object>
23+
{
24+
["max_completion_tokens"] = 2137
25+
}
26+
})
1827
.CompleteAsync(interactive: true);
1928
}
2029
}

‎Examples/Examples/Chat/ChatGrammarExampleGemini.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ public async Task Start()
4040
await AIHub.Chat()
4141
.WithModel(Models.Gemini.Gemini2_5Flash)
4242
.WithMessage("Generate random person")
43-
.WithInferenceParams(new InferenceParams
43+
.WithInferenceParams(new LocalInferenceParams
4444
{
4545
Grammar = new Grammar(grammarValue, GrammarFormat.JSONSchema)
4646
})

‎MaIN.Core.IntegrationTests/BackendParamsTests.cs‎

Lines changed: 320 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,320 @@
1+
using MaIN.Core.Hub;
2+
using MaIN.Domain.Configuration;
3+
using MaIN.Domain.Entities;
4+
using MaIN.Domain.Configuration.BackendInferenceParams;
5+
using MaIN.Domain.Exceptions;
6+
using MaIN.Domain.Models;
7+
using MaIN.Domain.Models.Concrete;
8+
9+
namespace MaIN.Core.IntegrationTests;
10+
11+
public class BackendParamsTests : IntegrationTestBase
12+
{
13+
private const string TestQuestion = "What is 2+2? Answer with just the number.";
14+
15+
[SkippableFact]
16+
public async Task OpenAi_Should_RespondWithParams()
17+
{
18+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.OpenAi)?.ApiKeyEnvName!);
19+
20+
var result = await AIHub.Chat()
21+
.WithModel(Models.OpenAi.Gpt4oMini)
22+
.WithMessage(TestQuestion)
23+
.WithInferenceParams(new OpenAiInferenceParams
24+
{
25+
Temperature = 0.3f,
26+
MaxTokens = 100,
27+
TopP = 0.9f
28+
})
29+
.CompleteAsync();
30+
31+
Assert.True(result.Done);
32+
Assert.NotNull(result.Message);
33+
Assert.NotEmpty(result.Message.Content);
34+
Assert.Contains("4", result.Message.Content);
35+
}
36+
37+
[SkippableFact]
38+
public async Task Anthropic_Should_RespondWithParams()
39+
{
40+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.Anthropic)?.ApiKeyEnvName!);
41+
42+
var result = await AIHub.Chat()
43+
.WithModel(Models.Anthropic.ClaudeSonnet4)
44+
.WithMessage(TestQuestion)
45+
.WithInferenceParams(new AnthropicInferenceParams
46+
{
47+
Temperature = 0.3f,
48+
MaxTokens = 100,
49+
TopP = 0.9f
50+
})
51+
.CompleteAsync();
52+
53+
Assert.True(result.Done);
54+
Assert.NotNull(result.Message);
55+
Assert.NotEmpty(result.Message.Content);
56+
Assert.Contains("4", result.Message.Content);
57+
}
58+
59+
[SkippableFact]
60+
public async Task Gemini_Should_RespondWithParams()
61+
{
62+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.Gemini)?.ApiKeyEnvName!);
63+
64+
var result = await AIHub.Chat()
65+
.WithModel(Models.Gemini.Gemini2_0Flash)
66+
.WithMessage(TestQuestion)
67+
.WithInferenceParams(new GeminiInferenceParams
68+
{
69+
Temperature = 0.3f,
70+
MaxTokens = 100,
71+
TopP = 0.9f
72+
})
73+
.CompleteAsync();
74+
75+
Assert.True(result.Done);
76+
Assert.NotNull(result.Message);
77+
Assert.NotEmpty(result.Message.Content);
78+
Assert.Contains("4", result.Message.Content);
79+
}
80+
81+
[SkippableFact]
82+
public async Task DeepSeek_Should_RespondWithParams()
83+
{
84+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.DeepSeek)?.ApiKeyEnvName!);
85+
86+
var result = await AIHub.Chat()
87+
.WithModel(Models.DeepSeek.Reasoner)
88+
.WithMessage(TestQuestion)
89+
.WithInferenceParams(new DeepSeekInferenceParams
90+
{
91+
Temperature = 0.3f,
92+
MaxTokens = 100,
93+
TopP = 0.9f
94+
})
95+
.CompleteAsync();
96+
97+
Assert.True(result.Done);
98+
Assert.NotNull(result.Message);
99+
Assert.NotEmpty(result.Message.Content);
100+
Assert.Contains("4", result.Message.Content);
101+
}
102+
103+
[SkippableFact]
104+
public async Task GroqCloud_Should_RespondWithParams()
105+
{
106+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.GroqCloud)?.ApiKeyEnvName!);
107+
108+
var result = await AIHub.Chat()
109+
.WithModel(Models.Groq.Llama3_1_8bInstant)
110+
.WithMessage(TestQuestion)
111+
.WithInferenceParams(new GroqCloudInferenceParams
112+
{
113+
Temperature = 0.3f,
114+
MaxTokens = 100,
115+
TopP = 0.9f
116+
})
117+
.CompleteAsync();
118+
119+
Assert.True(result.Done);
120+
Assert.NotNull(result.Message);
121+
Assert.NotEmpty(result.Message.Content);
122+
Assert.Contains("4", result.Message.Content);
123+
}
124+
125+
[SkippableFact]
126+
public async Task Xai_Should_RespondWithParams()
127+
{
128+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.Xai)?.ApiKeyEnvName!);
129+
130+
var result = await AIHub.Chat()
131+
.WithModel(Models.Xai.Grok3Beta)
132+
.WithMessage(TestQuestion)
133+
.WithInferenceParams(new XaiInferenceParams
134+
{
135+
Temperature = 0.3f,
136+
MaxTokens = 100,
137+
TopP = 0.9f
138+
})
139+
.CompleteAsync();
140+
141+
Assert.True(result.Done);
142+
Assert.NotNull(result.Message);
143+
Assert.NotEmpty(result.Message.Content);
144+
Assert.Contains("4", result.Message.Content);
145+
}
146+
147+
[SkippableFact]
148+
public async Task Self_Should_RespondWithParams()
149+
{
150+
Skip.If(!File.Exists("C:/Models/gemma2-2b.gguf"), "Local model not found at C:/Models/gemma2-2b.gguf");
151+
152+
var result = await AIHub.Chat()
153+
.WithModel(Models.Local.Gemma2_2b)
154+
.WithMessage(TestQuestion)
155+
.WithInferenceParams(new LocalInferenceParams
156+
{
157+
Temperature = 0.3f,
158+
ContextSize = 8192,
159+
MaxTokens = 100,
160+
TopK = 40,
161+
TopP = 0.9f
162+
})
163+
.CompleteAsync();
164+
165+
Assert.True(result.Done);
166+
Assert.NotNull(result.Message);
167+
Assert.NotEmpty(result.Message.Content);
168+
Assert.Contains("4", result.Message.Content);
169+
}
170+
171+
[SkippableFact]
172+
public async Task LocalOllama_Should_RespondWithParams()
173+
{
174+
SkipIfOllamaNotRunning();
175+
176+
var result = await AIHub.Chat()
177+
.WithModel(Models.Ollama.Gemma3_4b)
178+
.WithMessage(TestQuestion)
179+
.WithInferenceParams(new OllamaInferenceParams
180+
{
181+
Temperature = 0.3f,
182+
MaxTokens = 100,
183+
TopK = 40,
184+
TopP = 0.9f,
185+
NumCtx = 2048
186+
})
187+
.CompleteAsync();
188+
189+
Assert.True(result.Done);
190+
Assert.NotNull(result.Message);
191+
Assert.NotEmpty(result.Message.Content);
192+
Assert.Contains("4", result.Message.Content);
193+
}
194+
195+
[SkippableFact]
196+
public async Task ClaudOllama_Should_RespondWithParams()
197+
{
198+
SkipIfMissingKey(LLMApiRegistry.GetEntry(BackendType.Ollama)?.ApiKeyEnvName!);
199+
200+
var result = await AIHub.Chat()
201+
.WithModel(Models.Ollama.Gemma3_4b)
202+
.WithMessage(TestQuestion)
203+
.WithInferenceParams(new OllamaInferenceParams
204+
{
205+
Temperature = 0.3f,
206+
MaxTokens = 100,
207+
TopK = 40,
208+
TopP = 0.9f,
209+
NumCtx = 2048
210+
})
211+
.CompleteAsync();
212+
213+
Assert.True(result.Done);
214+
Assert.NotNull(result.Message);
215+
Assert.NotEmpty(result.Message.Content);
216+
Assert.Contains("4", result.Message.Content);
217+
}
218+
219+
// --- Params mismatch validation (no API key required) ---
220+
221+
[Fact]
222+
public async Task Self_Should_ThrowWhenGivenWrongParams()
223+
{
224+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
225+
AIHub.Chat()
226+
.WithModel(Models.Local.Gemma2_2b)
227+
.WithMessage(TestQuestion)
228+
.WithInferenceParams(new OpenAiInferenceParams())
229+
.CompleteAsync());
230+
}
231+
232+
[Fact]
233+
public async Task OpenAi_Should_ThrowWhenGivenWrongParams()
234+
{
235+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
236+
AIHub.Chat()
237+
.WithModel(Models.OpenAi.Gpt4oMini)
238+
.WithMessage(TestQuestion)
239+
.WithInferenceParams(new DeepSeekInferenceParams())
240+
.CompleteAsync());
241+
}
242+
243+
[Fact]
244+
public async Task Anthropic_Should_ThrowWhenGivenWrongParams()
245+
{
246+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
247+
AIHub.Chat()
248+
.WithModel(Models.Anthropic.ClaudeSonnet4)
249+
.WithMessage(TestQuestion)
250+
.WithInferenceParams(new OpenAiInferenceParams())
251+
.CompleteAsync());
252+
}
253+
254+
[Fact]
255+
public async Task Gemini_Should_ThrowWhenGivenWrongParams()
256+
{
257+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
258+
AIHub.Chat()
259+
.WithModel(Models.Gemini.Gemini2_0Flash)
260+
.WithMessage(TestQuestion)
261+
.WithInferenceParams(new AnthropicInferenceParams())
262+
.CompleteAsync());
263+
}
264+
265+
[Fact]
266+
public async Task DeepSeek_Should_ThrowWhenGivenWrongParams()
267+
{
268+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
269+
AIHub.Chat()
270+
.WithModel(Models.DeepSeek.Reasoner)
271+
.WithMessage(TestQuestion)
272+
.WithInferenceParams(new GeminiInferenceParams())
273+
.CompleteAsync());
274+
}
275+
276+
[Fact]
277+
public async Task GroqCloud_Should_ThrowWhenGivenWrongParams()
278+
{
279+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
280+
AIHub.Chat()
281+
.WithModel(Models.Groq.Llama3_1_8bInstant)
282+
.WithMessage(TestQuestion)
283+
.WithInferenceParams(new OpenAiInferenceParams())
284+
.CompleteAsync());
285+
}
286+
287+
[Fact]
288+
public async Task Xai_Should_ThrowWhenGivenWrongParams()
289+
{
290+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
291+
AIHub.Chat()
292+
.WithModel(Models.Xai.Grok3Beta)
293+
.WithMessage(TestQuestion)
294+
.WithInferenceParams(new AnthropicInferenceParams())
295+
.CompleteAsync());
296+
}
297+
298+
[Fact]
299+
public async Task Ollama_Should_ThrowWhenGivenWrongParams()
300+
{
301+
await Assert.ThrowsAsync<InvalidBackendParamsException>(() =>
302+
AIHub.Chat()
303+
.WithModel(Models.Ollama.Gemma3_4b)
304+
.WithMessage(TestQuestion)
305+
.WithInferenceParams(new DeepSeekInferenceParams())
306+
.CompleteAsync());
307+
}
308+
309+
private static void SkipIfMissingKey(string envName)
310+
{
311+
Skip.If(string.IsNullOrEmpty(Environment.GetEnvironmentVariable(envName)),
312+
$"{envName} environment variable not set");
313+
}
314+
315+
private static void SkipIfOllamaNotRunning()
316+
{
317+
Skip.If(!Helpers.NetworkHelper.PingHost("127.0.0.1", 11434, 3),
318+
"Ollama is not running on localhost:11434");
319+
}
320+
}

‎MaIN.Core.IntegrationTests/MaIN.Core.IntegrationTests.csproj‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
<PrivateAssets>all</PrivateAssets>
1616
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
1717
</PackageReference>
18+
<PackageReference Include="Xunit.SkippableFact" Version="1.5.61" />
1819
</ItemGroup>
1920

2021
<ItemGroup>

‎Releases/0.10.2.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# 0.10.2 release
2+
3+
Inference parameters are now backend-specific — each AI provider has its own typed params class where only explicitly set values are sent to the API, with an AdditionalParams dictionary for custom fields.

‎src/MaIN.Core.UnitTests/AgentContextTests.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ public async Task CreateAsync_ShouldCallAgentServiceCreateAgent()
139139
It.IsAny<Agent>(),
140140
It.IsAny<bool>(),
141141
It.IsAny<bool>(),
142-
It.IsAny<InferenceParams>(),
142+
It.IsAny<IBackendInferenceParams>(),
143143
It.IsAny<MemoryParams>(),
144144
It.IsAny<bool>()))
145145
.ReturnsAsync(agent);
@@ -153,7 +153,7 @@ public async Task CreateAsync_ShouldCallAgentServiceCreateAgent()
153153
It.IsAny<Agent>(),
154154
It.Is<bool>(f => f == true),
155155
It.Is<bool>(r => r == false),
156-
It.IsAny<InferenceParams>(),
156+
It.IsAny<IBackendInferenceParams>(),
157157
It.IsAny<MemoryParams>(),
158158
It.IsAny<bool>()),
159159
Times.Once);

0 commit comments

Comments
 (0)