Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ coverage.out
coverage.html
.task/
pkg/infra/vm/initbin/waggle-init
pkg/infra/vm/runtimebin/propolis-runner
pkg/infra/vm/runtimebin/go-microvm-runner
pkg/infra/vm/runtimebin/libkrun.*
pkg/infra/vm/runtimebin/libkrunfw.*
pkg/infra/vm/runtimebin/sha256sums.txt
pkg/infra/vm/runtimebin/VERSION
pkg/infra/vm/runtimebin/LICENSE-GPL

Expand Down
2 changes: 1 addition & 1 deletion .ko.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ builds:
- -X main.buildDate={{.Env.CREATION_TIME}}
labels:
org.opencontainers.image.created: "{{.Env.CREATION_TIME}}"
org.opencontainers.image.description: "Waggle - MCP server for isolated code execution via propolis microVMs"
org.opencontainers.image.description: "Waggle - MCP server for isolated code execution via go-microvm microVMs"
org.opencontainers.image.licenses: "Apache-2.0"
org.opencontainers.image.revision: "{{.Env.GITHUB_SHA}}"
org.opencontainers.image.source: "{{.Env.GITHUB_SERVER_URL}}/{{.Env.GITHUB_REPOSITORY}}"
Expand Down
10 changes: 5 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# waggle

MCP server for isolated code execution via propolis microVMs. Go + mcp-go, Streamable HTTP transport. Module: `github.com/stacklok/waggle`.
MCP server for isolated code execution via go-microvm microVMs. Go + mcp-go, Streamable HTTP transport. Module: `github.com/stacklok/waggle`.

## Commands

```bash
task build # Build self-contained binary with embedded propolis runtime
task build # Build self-contained binary with embedded go-microvm runtime
task test # go test -v -race ./...
task lint # golangci-lint run ./...
task verify # fmt + lint + test (CI gate)
Expand All @@ -20,19 +20,19 @@ Run a single test: `go test -v -race -run TestName ./pkg/path/to/package`

- `pkg/domain/` — Pure types and interfaces, no external deps. `environment/` is the aggregate root with a state machine (Creating→Running→Destroying→Destroyed|Error)
- `pkg/service/` — Orchestration: `EnvironmentService`, `ExecutionService`, `FilesystemService`
- `pkg/infra/` — Adapters: `vm/` (propolis), `ssh/` (executor+filesystem), `store/` (in-memory repo)
- `pkg/infra/` — Adapters: `vm/` (go-microvm), `ssh/` (executor+filesystem), `store/` (in-memory repo)
- `pkg/mcp/` — 8 MCP tool definitions + handlers + server assembly
- `pkg/cleanup/` — Background reaper for expired environments
- Entry point: `cmd/waggle/main.go` wires everything with DI, handles signals

## Things That Will Bite You

- **propolis is a tagged dependency (v0.0.16)**: `task build` embeds propolis-runner, libkrun, and libkrunfw into the binary (downloaded via `task fetch-runtime`/`task fetch-firmware`, verified with sha256sums). Use `build-dev-system` for the system libkrun-devel path.
- **go-microvm is a tagged dependency (v0.0.22)**: `task build` embeds go-microvm-runner, libkrun, and libkrunfw into the binary (downloaded via `task fetch-runtime`/`task fetch-firmware`, verified with sha256sums). Use `build-dev-system` for the system libkrun-devel path.
- **Image cache and log level**: `WAGGLE_IMAGE_CACHE_DIR` (default `~/.cache/waggle/images/`) enables shared OCI image cache with layer-level caching and COW rootfs cloning. `WAGGLE_IMAGE_CACHE_MAX_AGE` (default `168h`/7d) controls eviction. `WAGGLE_LOG_LEVEL` (0-5, default 0) sets libkrun verbosity; levels > 3 leak hypervisor internals.
- **Layered images**: Runtime images (python/node/shell) inherit from `images/base/` via `ARG BASE_IMAGE`. Build base first: `task build-image-base`. All runtime image tasks depend on it automatically.
- **MCP error handling has two paths**: Return `mcp.NewToolResultError("msg"), nil` for user-facing errors (bad input, not found). Return `nil, err` only for internal server failures. Mixing these up breaks the MCP protocol.
- **Code execution uses temp files, not `-c`**: Multi-line code is written to `/tmp/waggle_<uuid>.<ext>` in the VM via heredoc, executed, then cleaned up. Using `python3 -c` or `node -e` breaks on complex code.
- **Shell escaping is mandatory**: Always use `propolis/ssh.ShellEscape()` for any user-provided string passed to SSH commands. Missing this is a command injection vulnerability.
- **Shell escaping is mandatory**: Always use `go-microvm/ssh.ShellEscape()` for any user-provided string passed to SSH commands. Missing this is a command injection vulnerability.
- **MemoryStore returns copies**: Both `Save()` and `FindByID()` copy the struct to prevent aliasing bugs. Mutating a returned `*Environment` does not affect the store — you must call `Save()` again.
- **Port allocator verifies with net.Listen**: It doesn't just track allocations — it probes each port. Tests must call `portAlloc.SetListenCheck(func(_ uint16) error { return nil })` to skip real binding.
- **SPDX headers on every file**: Every `.go` and `.yaml` file needs both `SPDX-FileCopyrightText: Copyright 2025 Stacklok, Inc.` and `SPDX-License-Identifier: Apache-2.0`. Linting will fail without them.
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ All file operations take `environment_id` and `path`. Files persist within the e

## Why MicroVMs

Most code execution MCP servers use containers or V8 isolates. Waggle uses microVMs ([propolis](https://github.com/stacklok/propolis) + [libkrun](https://github.com/containers/libkrun)):
Most code execution MCP servers use containers or V8 isolates. Waggle uses microVMs ([go-microvm](https://github.com/stacklok/go-microvm) + [libkrun](https://github.com/containers/libkrun)):

| | Containers | V8 Isolates | Waggle (microVMs) |
|---|---|---|---|
Expand All @@ -118,23 +118,23 @@ Most code execution MCP servers use containers or V8 isolates. Waggle uses micro

The trade-off is startup time (seconds vs milliseconds), but you get a real Linux environment where `pip install numpy` and `apt-get install ffmpeg` just work.

**Ecosystem**: [ToolHive](https://github.com/stacklok/toolhive) (platform) → [Propolis](https://github.com/stacklok/propolis) (VM substrate) → **Waggle** (MCP interface)
**Ecosystem**: [ToolHive](https://github.com/stacklok/toolhive) (platform) → [go-microvm](https://github.com/stacklok/go-microvm) (VM substrate) → **Waggle** (MCP interface)

## Quick Start

### Prerequisites

- [Go](https://go.dev/dl/) 1.26+
- [Task](https://taskfile.dev/) (`go install github.com/go-task/task/v3/cmd/task@latest`)
- [GitHub CLI](https://cli.github.com/) (`gh`) — used to download the propolis runtime
- [GitHub CLI](https://cli.github.com/) (`gh`) — used to download the go-microvm runtime
- KVM access on Linux (`/dev/kvm`) or Hypervisor.framework on macOS

### Build and Run

```bash
git clone https://github.com/stacklok/waggle.git
cd waggle
task build # Downloads propolis runtime + firmware, builds a self-contained binary
task build # Downloads go-microvm runtime + firmware, builds a self-contained binary
task run # Starts the MCP server
```

Expand Down Expand Up @@ -215,7 +215,7 @@ Runtime command selection order:
2) Probed commands inside the VM (if available)
3) Built-in fallbacks (`python3`, `pip install`, `node`, `npm install -g`, `sh`, `apk add --no-cache`)

No SSH server is needed — `waggle-init` is injected into the VM at boot time by propolis and handles all communication.
No SSH server is needed — `waggle-init` is injected into the VM at boot time by go-microvm and handles all communication.

## Development

Expand Down
62 changes: 31 additions & 31 deletions Taskfile.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ vars:
sh: git rev-parse --short HEAD 2>/dev/null || echo "unknown"
BUILD_DATE:
sh: date -u +"%Y-%m-%dT%H:%M:%SZ"
PROPOLIS_VERSION:
sh: go list -m github.com/stacklok/propolis | awk '{print $2}'
MICROVM_VERSION:
sh: go list -m github.com/stacklok/go-microvm | awk '{print $2}'
HOST_ARCH:
sh: uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/'
HOST_OS:
Expand All @@ -32,13 +32,13 @@ vars:
-X main.version={{.VERSION}}
-X main.commit={{.COMMIT}}
-X main.buildDate={{.BUILD_DATE}}
-X github.com/stacklok/waggle/pkg/infra/vm/runtimebin.Version={{.PROPOLIS_VERSION}}
-X github.com/stacklok/waggle/pkg/infra/vm/runtimebin.Version={{.MICROVM_VERSION}}

# Quick reference:
# task build → Build self-contained waggle with embedded propolis runtime
# task build-dev-system → Build waggle + propolis-runner from system libkrun
# task fetch-runtime → Download pre-built propolis runtime from GitHub Release
# task fetch-firmware → Download pre-built propolis firmware from GitHub Release
# task build → Build self-contained waggle with embedded go-microvm runtime
# task build-dev-system → Build waggle + go-microvm-runner from system libkrun
# task fetch-runtime → Download pre-built go-microvm runtime from GitHub Release
# task fetch-firmware → Download pre-built go-microvm firmware from GitHub Release
# task test → Run tests with race detector
# task lint → Run linter
# task verify → fmt + lint + test
Expand All @@ -50,7 +50,7 @@ tasks:
- task --list

build:
desc: Build self-contained waggle with embedded propolis runtime
desc: Build self-contained waggle with embedded go-microvm runtime
deps: [build-init, fetch-runtime, fetch-firmware]
env:
CGO_ENABLED: "0"
Expand All @@ -65,65 +65,65 @@ tasks:
- '{{.BUILD_DIR}}/{{.BINARY_NAME}}'

build-dev-system:
desc: Build waggle + propolis-runner from system libkrun (requires libkrun-devel)
desc: Build waggle + go-microvm-runner from system libkrun (requires libkrun-devel)
platforms: [linux]
cmds:
- task: build
- mkdir -p {{.BUILD_DIR}}
- CGO_ENABLED=1 go build -o {{.BUILD_DIR}}/propolis-runner github.com/stacklok/propolis/runner/cmd/propolis-runner
- CGO_ENABLED=1 go build -o {{.BUILD_DIR}}/go-microvm-runner github.com/stacklok/go-microvm/runner/cmd/go-microvm-runner

build-dev-system-darwin:
desc: Build waggle + propolis-runner from system libkrun (macOS, requires Homebrew libkrun)
desc: Build waggle + go-microvm-runner from system libkrun (macOS, requires Homebrew libkrun)
platforms: [darwin]
cmds:
- task: build
- mkdir -p {{.BUILD_DIR}}
- CGO_ENABLED=1 go build -o {{.BUILD_DIR}}/propolis-runner github.com/stacklok/propolis/runner/cmd/propolis-runner
- codesign --entitlements assets/entitlements.plist --force -s - {{.BUILD_DIR}}/propolis-runner
- CGO_ENABLED=1 go build -o {{.BUILD_DIR}}/go-microvm-runner github.com/stacklok/go-microvm/runner/cmd/go-microvm-runner
- codesign --entitlements assets/entitlements.plist --force -s - {{.BUILD_DIR}}/go-microvm-runner

fetch-runtime:
desc: Download pre-built propolis runtime from GitHub Release
desc: Download pre-built go-microvm runtime from GitHub Release
status:
- test -f pkg/infra/vm/runtimebin/propolis-runner
- test -f pkg/infra/vm/runtimebin/go-microvm-runner
cmds:
- mkdir -p pkg/infra/vm/runtimebin
- >-
gh release download {{.PROPOLIS_VERSION}}
--repo stacklok/propolis
--pattern "propolis-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz"
gh release download {{.MICROVM_VERSION}}
--repo stacklok/go-microvm
--pattern "go-microvm-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz"
--dir pkg/infra/vm/runtimebin/ --clobber
- >-
gh release download {{.PROPOLIS_VERSION}}
--repo stacklok/propolis
gh release download {{.MICROVM_VERSION}}
--repo stacklok/go-microvm
--pattern "sha256sums.txt"
--dir pkg/infra/vm/runtimebin/ --clobber
- cd pkg/infra/vm/runtimebin/ && sha256sum --check --ignore-missing sha256sums.txt
- >-
tar -xzf pkg/infra/vm/runtimebin/propolis-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
tar -xzf pkg/infra/vm/runtimebin/go-microvm-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
-C pkg/infra/vm/runtimebin/ --strip-components=1
- rm -f pkg/infra/vm/runtimebin/propolis-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
- rm -f pkg/infra/vm/runtimebin/go-microvm-runtime-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz

fetch-firmware:
desc: Download pre-built propolis firmware from GitHub Release
desc: Download pre-built go-microvm firmware from GitHub Release
status:
- ls pkg/infra/vm/runtimebin/libkrunfw.* >/dev/null 2>&1
cmds:
- mkdir -p pkg/infra/vm/runtimebin
- >-
gh release download {{.PROPOLIS_VERSION}}
--repo stacklok/propolis
--pattern "propolis-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz"
gh release download {{.MICROVM_VERSION}}
--repo stacklok/go-microvm
--pattern "go-microvm-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz"
--dir pkg/infra/vm/runtimebin/ --clobber
- >-
gh release download {{.PROPOLIS_VERSION}}
--repo stacklok/propolis
gh release download {{.MICROVM_VERSION}}
--repo stacklok/go-microvm
--pattern "sha256sums.txt"
--dir pkg/infra/vm/runtimebin/ --clobber
- cd pkg/infra/vm/runtimebin/ && sha256sum --check --ignore-missing sha256sums.txt
- >-
tar -xzf pkg/infra/vm/runtimebin/propolis-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
tar -xzf pkg/infra/vm/runtimebin/go-microvm-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
-C pkg/infra/vm/runtimebin/ --strip-components=1
- rm -f pkg/infra/vm/runtimebin/propolis-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz
- rm -f pkg/infra/vm/runtimebin/go-microvm-firmware-{{.HOST_OS}}-{{.HOST_ARCH}}.tar.gz

build-init:
desc: Build the waggle-init binary (guest VM init)
Expand Down Expand Up @@ -202,7 +202,7 @@ tasks:
- rm -rf {{.BUILD_DIR}}/
- rm -f coverage.out coverage.html
- rm -f pkg/infra/vm/initbin/waggle-init
- rm -f pkg/infra/vm/runtimebin/propolis-runner
- rm -f pkg/infra/vm/runtimebin/go-microvm-runner
- rm -f pkg/infra/vm/runtimebin/libkrun.so.1 pkg/infra/vm/runtimebin/libkrun.1.dylib
- rm -f pkg/infra/vm/runtimebin/libkrunfw.so.5 pkg/infra/vm/runtimebin/libkrunfw.5.dylib
- rm -f pkg/infra/vm/runtimebin/VERSION pkg/infra/vm/runtimebin/LICENSE-GPL
Expand Down
2 changes: 1 addition & 1 deletion cmd/waggle-init/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
// Package main provides the entry point for waggle-init, a minimal init
// process that runs as PID 1 inside guest VMs. It starts a zombie reaper,
// configures the system (mounts, network, hardening), and launches an
// embedded SSH server via propolis guest/boot.
// embedded SSH server via go-microvm guest/boot.
package main
4 changes: 2 additions & 2 deletions cmd/waggle-init/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import (
"os/signal"
"syscall"

"github.com/stacklok/propolis/guest/boot"
"github.com/stacklok/propolis/guest/reaper"
"github.com/stacklok/go-microvm/guest/boot"
"github.com/stacklok/go-microvm/guest/reaper"
)

func main() {
Expand Down
6 changes: 3 additions & 3 deletions cmd/waggle/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (

"github.com/adrg/xdg"
"github.com/mark3labs/mcp-go/server"
"github.com/stacklok/propolis/image"
"github.com/stacklok/go-microvm/image"

"github.com/stacklok/waggle/pkg/cleanup"
"github.com/stacklok/waggle/pkg/config"
Expand Down Expand Up @@ -114,10 +114,10 @@ func run(logFile string) error {
vm.WithRuntimeSource(runtimebin.RuntimeSource()),
vm.WithFirmwareSource(runtimebin.FirmwareSource()),
)
slog.Info("using embedded propolis runtime", "version", runtimebin.Version)
slog.Info("using embedded go-microvm runtime", "version", runtimebin.Version)
}

provider := vm.NewPropolisProvider(providerOpts...)
provider := vm.NewMicroVMProvider(providerOpts...)
portAlloc := vm.NewPortAllocator(cfg.SSHPortBase, cfg.SSHPortMax)

// Create domain adapters.
Expand Down
18 changes: 9 additions & 9 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This document describes Waggle's internal architecture, design decisions, and ex

## System Overview

Waggle is an MCP (Model Context Protocol) server that provides AI agents with isolated code execution environments. Each environment is a lightweight microVM powered by [propolis](https://github.com/stacklok/propolis)/[libkrun](https://github.com/containers/libkrun), giving true VM-level isolation with near-container startup times.
Waggle is an MCP (Model Context Protocol) server that provides AI agents with isolated code execution environments. Each environment is a lightweight microVM powered by [go-microvm](https://github.com/stacklok/go-microvm)/[libkrun](https://github.com/containers/libkrun), giving true VM-level isolation with near-container startup times.

```
AI Agent (Claude, etc.)
Expand All @@ -28,7 +28,7 @@ Waggle is an MCP (Model Context Protocol) server that provides AI agents with is
| |
+-------v------+ +-------v------+
| VMProvider | | SSH | pkg/infra/
| (Propolis) | | Executor + | vm/propolis.go, ssh/executor.go,
| (go-microvm)| | Executor + | vm/microvm.go, ssh/executor.go,
| PortAlloc | | FileSystem | ssh/filesystem.go
+-------+------+ +-------+------+
| |
Expand Down Expand Up @@ -85,7 +85,7 @@ Application services orchestrate domain objects and infrastructure adapters.
1. Validate runtime, check capacity (`MaxEnvironments`)
2. Allocate SSH port from `PortAllocator`
3. Create `Environment` in `Creating` state
4. Call `VMProvider.CreateVM()` (SSH key gen, propolis.Run, SSH readiness wait)
4. Call `VMProvider.CreateVM()` (SSH key gen, microvm.Run, SSH readiness wait)
5. Transition to `Running` (or `Error` on failure, releasing the port)

**ExecutionService** -- Code execution:
Expand All @@ -103,8 +103,8 @@ Application services orchestrate domain objects and infrastructure adapters.

Adapters implementing domain interfaces using concrete technologies.

**PropolisProvider** (`pkg/infra/vm/propolis.go`):
- Wraps `propolis.Run()` with SSH key injection and readiness wait
**MicroVMProvider** (`pkg/infra/vm/microvm.go`):
- Wraps `microvm.Run()` with SSH key injection and readiness wait
- Maintains in-memory map of `envID -> vmEntry{vm, sshKeyPath}`
- Uses `WithRootFSHook` to inject authorized_keys before boot
- Uses `WithPostBoot` to wait for SSH via `ssh.Client.WaitForReady()`
Expand Down Expand Up @@ -165,7 +165,7 @@ This is the critical path -- how user code goes from MCP tool call to execution
f. Delegates to Executor.Execute()
4. SSHExecutor:
a. Looks up environment SSH port and key path
b. Creates propolis SSH client
b. Creates go-microvm SSH client
c. Calls RunStream() with separate stdout/stderr buffers
d. Extracts exit code from SSH error (if non-zero)
e. Returns ExecResult{Stdout, Stderr, ExitCode, DurationMs}
Expand All @@ -176,15 +176,15 @@ This is the critical path -- how user code goes from MCP tool call to execution

- **New runtimes**: Add to `Runtime` enum in `pkg/domain/environment/runtime.go`, add image config
- **Persistent storage**: Implement `environment.Repository` backed by SQLite/Postgres
- **Network policies**: Use propolis `WithEgressPolicy()` or `WithFirewallRules()` in PropolisProvider
- **VirtioFS mounts**: Add shared host directories via `propolis.WithVirtioFS()`
- **Network policies**: Use go-microvm `WithEgressPolicy()` or `WithFirewallRules()` in MicroVMProvider
- **VirtioFS mounts**: Add shared host directories via `microvm.WithVirtioFS()`
- **Custom images**: Build OCI images with pre-installed tools, configure via `WAGGLE_IMAGE_*`

## Concurrency Model

- MCP server handles concurrent requests via Go's HTTP goroutine model
- `MemoryStore` uses `sync.RWMutex` for concurrent environment access
- `PortAllocator` uses `sync.Mutex` for allocation/release
- `PropolisProvider` uses `sync.RWMutex` for VM handle map
- `MicroVMProvider` uses `sync.RWMutex` for VM handle map
- Each environment is an independent VM -- no shared state between environments
- Reaper runs in its own goroutine, accesses environments through the store
Loading
Loading