请问在RL阶段，训练时模型每次的rollout都是实际调用了工具并且把结果返回给下一轮的推理吗？还是只模拟tool response，没有实际获取工具调用的实际结果呢？

[alanshaoTT]

LongVT是非常好的工作！！作为小白，想要复现训练流程，关于RL阶段的每一次rollout对于工具调用的实现不太清楚~我用的是ms-swift框架

[mwxely]

Thanks for the kind words and for the great question!

Short answer: Yes, during RL training, every tool call in each rollout is actually executed via an MCP server, and the real results are fed back into the model's context for the next reasoning step. There is no simulation or mocked tool response.

Below is a walkthrough of the key code path.

---

1. Multi-turn rollout state machine

The core rollout loop lives in sglang_rollout.py. Each request goes through a state machine:

PENDING  →  RUNNING  →  (tool call detected)  →  TOOL_CALLING  →  RUNNING  →  …  →  COMPLETED

When the model generates a response containing a tool call, the parser detects it and transitions to TOOL_CALLING:

if self._function_call_parser and self._function_call_parser.has_tool_call(content):
    finish_reason_type = FinishReasonTypeEnum.TOOL_CALL
    _req.state = AsyncRolloutRequestStateEnum.TOOL_CALLING

2. Actual tool execution (not simulated)

In the TOOL_CALLING state, tools are executed in parallel via asyncio.gather:

tool_call_results = await asyncio.gather(
    *[
        self._tool_map[tool_call.function.name].execute(
            _req.request_id,
            tool_call.function.arguments,
            **_req.tools_kwargs[tool_call.function.name].get("execute_kwargs", {}),
        )
        for tool_call in parsed_tool_calls
    ]
)

For our video tools specifically, VideoTools.execute() calls self._call_tool(), which is inherited from MCPBaseTool and ultimately invokes a real MCP server through ClientManager.call_tool():

async def call_tool(self, tool_name, parameters, timeout):
    while not self.rate_limiter.acquire():
        await asyncio.sleep(0.1)
    client = self.get_client_with_tool_name(tool_name)
    async with client:
        return await client.call_tool_mcp(tool_name, parameters)

This is a real MCP client call to a running MCP server — not a mock. The server processes the video (e.g., crop_video), and returns actual image frames as base64-encoded data, which VideoTools._parse_tool_result() decodes back into PIL images.

3. Tool results fed back into the next inference step

After execution, the real tool outputs are added to the message history and the state returns to RUNNING for the next generation:

success = _req.add_tool_response_messages(
    self.processing_class,
    [resp for resp, _, _ in tool_call_results],
    max_model_len=self.config.max_model_len,
)
# ...
_req.state = AsyncRolloutRequestStateEnum.RUNNING  # continue to next generation

add_tool_response_messages() handles both text and multimodal returns — for video tools, the returned image frames are appended to the model's multimodal input, tokenized via the chat template, and concatenated to input_ids. The next call to SGLang's engine then generates with this updated context (tool response included).

4. Summary of the full loop

Model generates response
   ↓
Parser detects tool_call (e.g. crop_video)          # L931-L933
   ↓
Actual MCP server call via ClientManager             # L847-L856 → mcp_base_tool.py L63-L77
   ↓
Real frames returned, decoded to PIL images          # video_tools.py L57-L60, L88-L97
   ↓
Tool response added to message history & input_ids   # schemas.py L409+
   ↓
State → RUNNING, model generates next response       # L896, L923
   ↓
Repeat until max_tool_calls or max_assistant_turns

So to directly answer your question: every rollout during RL training actually invokes the tool and gets real results back — the MCP server runs crop_video (or other registered tools), returns the processed frames, and those frames become part of the model's visual input for the next reasoning step.

Hope this helps! Feel free to ask follow-up questions.

[alanshaoTT]

Thank you very much for your detailed response！！ It has been very helpful to me and serves as an excellent learning resource！！！！

[mwxely]

We are closing this issue as it appears resolved and we have not received further feedback. Please feel free to reopen or file a new issue if needed.