feat(tools): add LLMSandboxTool for self-hosted code execution - #6785
feat(tools): add LLMSandboxTool for self-hosted code execution#6785vndee wants to merge 1 commit into
Conversation
Runs agent-authored code in a container on infrastructure the user already operates, via llm-sandbox. Docker, Podman and Kubernetes backends, seven languages. The existing sandbox tools both depend on a hosted service: E2B requires E2B_API_KEY, Daytona likewise. This adds the self-hosted option -- no API key, no per-execution cost, and code never leaves the user's machines. That matters for data-governance constraints, air-gapped evaluation, and high-volume batch work where per-call pricing dominates. Hardened by default: no network, capped memory and pids, every Linux capability dropped except DAC_OVERRIDE, no-new-privileges set. Verified in a running container -- CapEff 0000000000000002, outbound connections fail. Three deliberate choices: DAC_OVERRIDE is kept because llm-sandbox copies the source file into the container and cannot read it otherwise; read_only is unsupported because Docker rejects that copy against a read-only rootfs; and the schema exposes only code, since a package-installation argument would let a model pick arbitrary PyPI packages, which runs setup.py at install time. 11 tests, no container required.
📝 WalkthroughWalkthroughChangesLLM sandbox tool
Sequence Diagram(s)sequenceDiagram
participant Agent
participant LLMSandboxTool
participant LLMSandbox
participant Container
Agent->>LLMSandboxTool: Submit source code
LLMSandboxTool->>LLMSandbox: Create configured session
LLMSandbox->>Container: Execute code
Container-->>LLMSandbox: Return exit code and output
LLMSandbox-->>LLMSandboxTool: Return execution result
LLMSandboxTool-->>Agent: Return stdout or error
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a new LLMSandboxTool to crewai-tools that executes agent-authored code in a self-hosted container environment via llm-sandbox, aiming to provide a non-hosted alternative to existing sandbox tools (E2B/Daytona) and ship with hardened default runtime configs.
Changes:
- Introduces
LLMSandboxToolimplementation, runtime hardening defaults, and public exports. - Adds documentation for the tool and registers an optional dependency extra (
llm-sandbox). - Adds unit tests that mock
SandboxSessionto validate output handling and config forwarding.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py | New tool implementation, defaults, and error handling for llm-sandbox execution |
| lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/README.md | New tool documentation and installation guidance |
| lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/init.py | Exposes LLMSandboxTool and DEFAULT_RUNTIME_CONFIGS from the tool package |
| lib/crewai-tools/src/crewai_tools/init.py | Registers new tool for top-level import (from crewai_tools import LLMSandboxTool) |
| lib/crewai-tools/pyproject.toml | Adds llm-sandbox optional extra dependency |
| lib/crewai-tools/tests/tools/test_llm_sandbox_tool.py | Adds mocked unit tests for tool behavior and config propagation |
Suppressed comments (1)
lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py:125
- The repo runs strict mypy with
disallow_any_unimported = true(pyproject.toml). Ifllm-sandboxis not marked as typed (nopy.typed), mypy will fail on this import unless it is explicitly ignored (consistent with other optional deps likee2b_code_interpreter).
from llm_sandbox.exceptions import SandboxError
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| runtime_configs: dict[str, Any] = Field( | ||
| default_factory=lambda: dict(DEFAULT_RUNTIME_CONFIGS), | ||
| description="Container settings passed to the backend. Defaults to a hardened set.", | ||
| ) |
| try: | ||
| from llm_sandbox import SandboxSession | ||
| except ImportError as exc: |
| description=( | ||
| "Source to execute, complete and self-contained. Print anything you " | ||
| "want returned -- only stdout comes back." | ||
| ), |
| backend: str = Field( | ||
| default="docker", | ||
| description="Container backend: docker, podman, kubernetes or micromamba.", | ||
| ) |
| uv add crewai-tools --extra llm-sandbox | ||
| ``` | ||
|
|
||
| Requires a container runtime; Docker by default. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai-tools/pyproject.toml`:
- Around line 29-31: Update the llm-sandbox dependency declaration in
pyproject.toml to include the extras required for the documented and tested
Kubernetes and Podman backends, alongside the existing docker extra. Preserve
the advertised backend support and ensure each listed runtime installs its
corresponding dependencies.
In
`@lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py`:
- Around line 97-99: Update the runtime_configs default_factory in the
LlmSandboxTool configuration to deep-copy a private immutable template,
preventing nested lists such as cap_drop from being shared across tool instances
or with DEFAULT_RUNTIME_CONFIGS. Add a regression test that mutates a nested
value on one instance and verifies later instances retain the original defaults.
- Around line 76-79: Update Kubernetes handling around the backend field,
_session_kwargs(), and _run() so it uses a fixed hardened pod_manifest covering
network, resources, writable paths, and security context instead of
runtime_configs; reject backend="kubernetes" until that manifest is available if
it cannot be safely added. Add Kubernetes to the README and _run() import hint,
while preserving the existing Docker/Podman behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 299a699e-15b3-4fa6-8e40-72400a66affe
📒 Files selected for processing (6)
lib/crewai-tools/pyproject.tomllib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.pylib/crewai-tools/tests/tools/test_llm_sandbox_tool.py
| llm-sandbox = [ | ||
| "llm-sandbox[docker]>=0.3.43", | ||
| ] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
curl -fsSL https://pypi.org/pypi/llm-sandbox/json |
jq '{
latest: .info.version,
recent_releases: (.releases | keys | sort | reverse | .[:10]),
backend_requirements: [.info.requires_dist[] | select(test("extra =="))]
}'Repository: crewAIInc/crewAI
Length of output: 871
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "pyproject optional extra:"
sed -n '24,34p' lib/crewai-tools/pyproject.toml || true
echo
echo "References to llm-sandbox backend/extras in repository:"
rg -n 'llm-sandbox|mcp-(docker|k8s|podman)|docker|podman|k8s|kubernetes|Kubernetes|Podman' lib/crewai-tools README.md pyproject.toml 2>/dev/null | head -200Repository: crewAIInc/crewAI
Length of output: 5257
Declare llm-sandbox backend extras to match the advertised runtimes.
llm-sandbox[docker]>=0.3.43 is an installable dependency, but it only installs Docker support. The library documents and tests Podman and Kubernetes backends, and the extra currently only advertises Kubernetes documentation without the Kubernetes dependencies. Add the Kubernetes/Podman extras, or drop them from the supported-backend surface and update the README/code examples.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/pyproject.toml` around lines 29 - 31, Update the llm-sandbox
dependency declaration in pyproject.toml to include the extras required for the
documented and tested Kubernetes and Podman backends, alongside the existing
docker extra. Preserve the advertised backend support and ensure each listed
runtime installs its corresponding dependencies.
| backend: str = Field( | ||
| default="docker", | ||
| description="Container backend: docker, podman, kubernetes or micromamba.", | ||
| ) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate target file:"
fd -a 'llm_sandbox_tool.py' . || true
echo
echo "Target file outline:"
ast-grep outline lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py || true
echo
echo "Relevant target lines:"
sed -n '1,150p' lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py
echo
echo "Search llm-sandbox/package references:"
rg -n "llm-sandbox|llm_sandbox|LLMSandbox|runtime_configs|pod_manifest|backend.*kubernetes|Kubernetes" -S .Repository: crewAIInc/crewAI
Length of output: 13687
🌐 Web query:
llm-sandbox configuration runtime_configs pod_manifest Kubernetes
💡 Result:
In LLM Sandbox, the configuration approach depends on the chosen backend. For the Kubernetes backend, runtime configurations are not supported via the standard runtime_configs parameter [1]. Instead, you must define infrastructure requirements using a Kubernetes Pod manifest provided via the pod_manifest parameter [1][2]. Key details regarding this configuration: 1. Kubernetes vs. Other Backends: While backends like Docker and Podman support the runtime_configs parameter for items like resource limits (CPU/memory), security options (capabilities, privileged mode), and network settings, these do not apply to the Kubernetes backend [1][3]. 2. Using pod_manifest: When using the Kubernetes backend, you pass a dictionary representing a Kubernetes Pod manifest (as you would define in a YAML file) to the pod_manifest parameter in the SandboxSession [1][2]. This allows you to specify: - Resource requirements (requests and limits) [2]. - Security contexts (both pod-level and container-level) [2][4]. - Volume mounts (e.g., for writable directories like /sandbox or /tmp when using read-only filesystems) [2]. - Environment variables, images, and other standard Kubernetes spec fields [2][4]. 3. Critical Requirements: When defining a custom manifest, ensure you include: - "tty": True: This is essential for maintaining an active connection to the container [1][4]. - SecurityContext: Both pod-level and container-level security contexts are required to ensure proper permissions and execution [4]. - Writable Volumes: If you enable a read-only root filesystem in your security context, you must explicitly define and mount volumes (e.g., emptyDir) for critical writable paths like /sandbox and /tmp [2]. If you are using the LLM Sandbox MCP (Model Context Protocol) server, note that environment-based configuration (SANDBOX_*) also does not apply to the Kubernetes backend [3]. You must provide a custom pod manifest directly through your code or a custom wrapper [3].
Citations:
- 1: https://vndee.github.io/llm-sandbox/configuration/
- 2: https://vndee.github.io/llm-sandbox/backends/
- 3: https://github.com/vndee/llm-sandbox/blob/main/docs/mcp-integration.md
- 4: https://vndee.github.io/llm-sandbox/getting-started/
Use a hardened pod_manifest for Kubernetes instead of runtime_configs.
With backend="kubernetes", _session_kwargs() only forwards runtime_configs. Kubernetes does not apply those Docker/Podman settings; it needs a fixed hardened pod_manifest for network, resources, writable paths, and security context. Reject Kubernetes support until that manifest is installed, or add and forward one with coverage. Also update the README and _run() import hint to include Kubernetes, because Docker/Podman cannot produce a Kubernetes manifest.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py`
around lines 76 - 79, Update Kubernetes handling around the backend field,
_session_kwargs(), and _run() so it uses a fixed hardened pod_manifest covering
network, resources, writable paths, and security context instead of
runtime_configs; reject backend="kubernetes" until that manifest is available if
it cannot be safely added. Add Kubernetes to the README and _run() import hint,
while preserving the existing Docker/Podman behavior.
Source: Coding guidelines
| runtime_configs: dict[str, Any] = Field( | ||
| default_factory=lambda: dict(DEFAULT_RUNTIME_CONFIGS), | ||
| description="Container settings passed to the backend. Defaults to a hardened set.", |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Deep-copy the nested default configuration.
dict(DEFAULT_RUNTIME_CONFIGS) copies only the outer dictionary. The nested lists remain shared with DEFAULT_RUNTIME_CONFIGS and every future tool instance. For example, clearing one instance’s cap_drop list removes capability dropping from later default instances. Use a deep copy from a private immutable template, and add a regression test that mutates nested values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/llm_sandbox_tool/llm_sandbox_tool.py`
around lines 97 - 99, Update the runtime_configs default_factory in the
LlmSandboxTool configuration to deep-copy a private immutable template,
preventing nested lists such as cap_drop from being shared across tool instances
or with DEFAULT_RUNTIME_CONFIGS. Add a regression test that mutates a nested
value on one instance and verifies later instances retain the original defaults.
Adds a sandbox tool that runs agent-authored code on container infrastructure the user already operates, via llm-sandbox. Docker, Podman and Kubernetes backends; Python, JavaScript, Java, C++, Go, R and Ruby.
Why
crewai_toolscurrently has two sandbox options and both depend on a hosted service —E2BPythonToolrequiresE2B_API_KEY, Daytona likewise. This adds the self-hosted case: no API key, no per-execution cost, and code never leaves the user's machines.That matters for data-governance constraints, air-gapped evaluation, and high-volume batch work where per-call pricing dominates.
Usage
Hardened by default
{ "network_mode": "none", "mem_limit": "512m", "pids_limit": 128, "cap_drop": ["ALL"], "cap_add": ["DAC_OVERRIDE"], "security_opt": ["no-new-privileges:true"], }Verified in a running container:
CapEffis0000000000000002(DAC_OVERRIDE only) and outbound connections fail.Three choices that may look odd on review, all deliberate:
DAC_OVERRIDEis kept. llm-sandbox copies the source file into the container; dropping it makes that file unreadable and every run fails with[Errno 13] Permission denied.read_only: Trueis not used. Docker rejects the code copy against a read-only rootfs (container rootfs is marked read-only), with or without a tmpfs on the workdir.code. A package-installation argument would let a model choose arbitrary PyPI packages, which executessetup.pyat install time — and the default network isolation would block it anyway. Useimage=to pre-bake dependencies.The README states plainly that this is container isolation, not VM isolation, and points at gVisor/Kata for adversarial workloads.
Tests
11 tests, no container required —
SandboxSessionis mocked. They cover metadata, output handling for success/failure/empty, that the hardening reaches the session, thatkeep_templateis set (without it the image is re-pulled every call), config forwarding, and two regression guards: the schema stays single-parameter, and aSandboxErrordoes not leak theDOCKER_HOSTsocket path back to the model.Also verified end to end against real Docker: executes correctly, egress blocked, capabilities as above.
Registered in
crewai_tools/__init__.pyand added as an optional extra inlib/crewai-tools/pyproject.toml.I maintain llm-sandbox and will maintain this tool.