Check MockGeneratorPlugin sequential LLM request mock usability

[waldekmastykarz]

Evaluate if MockGeneratorPlugin can record and replay mocks for sequential LLM requests without requiring user modifications. Clarify whether recorded mocks are reusable as-is or if additional steps are needed for correct behavior. Summarize findings and any recommended workflow adjustments.

[waldekmastykarz]

Looked into the codebase to answer this. Here's what I found:

Can MockGeneratorPlugin record sequential LLM requests?

Yes, but with a caveat. MockGeneratorPlugin records all intercepted API responses in order during a recording session and saves them to a mocks-{timestamp}.json file. However, it does not automatically tag requests with sequential markers.

Are recorded mocks reusable as-is?

It depends on the scenario:

• Different URLs per request: If each LLM request hits a different URL (or uses a different body), the recorded mocks will replay correctly because the mock matching is URL+method+body-based.
• Same URL, different responses (e.g., multi-turn chat completions to the same endpoint): The recorded mocks are not directly reusable as-is. MockResponsePlugin uses first-match-wins logic (FirstOrDefault), so it will return the same mock for every request to that URL.

Workaround: the nth property

MockResponsePlugin supports an nth property on mock requests that enables sequential matching. It tracks how many times a URL has been matched and only returns a mock when the count matches nth. However, MockGeneratorPlugin does not auto-populate nth during recording. You'd need to manually add it to the generated mocks file:

{ "request": { "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "nth": 1 }, "response": { "..." : "first response" } }
{ "request": { "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "nth": 2 }, "response": { "..." : "second response" } }

Recommendation

We could improve this by having MockGeneratorPlugin auto-detect duplicate URL+method combinations during recording and automatically assign nth values. This would make recorded mocks for sequential LLM requests (and any repeated-endpoint scenarios) work out of the box without manual edits.