feat(sandbox-adapter): Kubernetes provider built on Agent Sandbox #378

Summary

We'd like to contribute a SandboxProvider for Kubernetes, built on Kubernetes Agent Sandbox (SandboxClaim / SandboxTemplate / SandboxWarmPool), at packages/sandbox-adapter/src/kubernetes.ts.

The main thing we need from you is which runtime to implement: how the provider runs commands and file operations inside the sandbox pod. There are three candidates:

1. pods/exec: the Kubernetes exec API, as the Docker provider uses docker exec. Nothing extra in the image.
2. sandboxd: Agent Sandbox's own in-pod runtime daemon.
3. Sandbox Agent: the sandbox-agent server running in the pod.

We recommend pods/exec, so that, like the other providers, the sandbox image needs nothing installed. We'll build whichever you prefer. The claim lifecycle (create, connect, suspend, destroy) is the same for all three. Because this provider depends on cluster-side setup, we'd like to agree on the design before writing the tests and docs.

Motivation

• Teams that run their own Kubernetes clusters (for data residency, cost, or GPU access) have no sandbox-adapter provider today. Docker covers local development only.
• Agent Sandbox is the Kubernetes SIG Apps project for exactly this workload. It reached v1.0 and gives claims, warm pools, per-sandbox Services, managed NetworkPolicies, and suspend/resume. It works with any runtime class, so gVisor (GKE Sandbox) or Kata provide the isolation.
• We're building toward running piDurable() sessions this way, with the Rivet engine and the sandboxes in the same cluster.

Proposed API

import { agentSandboxProvider } from "@rivet-dev/sandbox-adapter/kubernetes";

export const agent = piDurable({
  // ...
  sandbox: agentSandboxProvider({
    namespace: "sandbox",
    warmPool: "coding-agent",        // SandboxWarmPool to claim from
    cwd: "/workspace",
    container: "agent",              // container to exec into; defaults to the first
    readyTimeoutMs: 300_000,         // a cold claim includes node scale-up
    suspend: "pod",                  // "pod" | "none"
    claim: { labels, podLabels, volumeClaimTemplates, shutdownAfterSeconds },
    kubeConfig,                      // default: in-cluster ServiceAccount, else ~/.kube/config
  }),
});

The runtime options

Option 1 (recommended): pods/exec

The Sandbox operations run through the Kubernetes exec API (@kubernetes/client-node's Exec), the same way the Docker provider uses docker exec:

• exec runs /bin/sh -c, with the working directory, environment and command passed as positional arguments rather than interpolated into the script. It streams stdout and stderr to onData and reads the exit code from the exec status channel.
• Each command starts in its own process group and records its PID in the sandbox. Abort and timeout kill that group explicitly with a second exec. The provider doesn't rely on a closed exec stream killing the process: with non-TTY exec, Kubernetes can leave it running after the client disconnects. On connect, the provider kills any group left over from a previous actor generation, so a resumed run never shares the workspace with an orphaned command.
• readFile streams cat. writeFile pipes content through stdin. mkdir, stat, readdir and exists use mkdir -p, test and a POSIX-sh directory listing, as the Docker provider does.

The only image requirement is /bin/sh with cat and mkdir. Exec goes through the API server, so every command is authenticated, authorized and audit-logged by Kubernetes. It also works on gVisor and Kata pods, where port-forwarding does not.

Options 2 and 3: sandboxd or Sandbox Agent

Both connect directly to the pod through the per-sandbox Service (status.serviceFQDN), so the API server is off the data path. They require a daemon in the sandbox image.

Comparison

• sandboxd is Agent Sandbox's standard runtime (KEP-539.2). It suits clusters that want the direct path and already lock sandbox ingress down with NetworkPolicy.
• Sandbox Agent suits teams already using it in their sandboxes. It's also the most tolerant of long commands over unreliable connections.

Whichever you choose, we'd keep the runtime behind an internal interface, so another one could be added later behind a runtime option without changing the provider's API.

What we've verified

Tested on GKE 1.36 with a GKE Sandbox (gVisor) node pool and Agent Sandbox v1.0.7:

1. Exec works on gVisor sandbox pods through the Kubernetes API (kubectl exec), including running commands and reading files.
2. Suspend and resume. Patching the Sandbox to Suspended removed the pod in about 4 seconds, and the claim reported Ready=False (SandboxSuspended). The claim controller didn't revert the patch. Resuming took about 9 seconds with a node already available. With a PVC from the template's volumeClaimTemplates, the workspace survived. With an emptyDir it was wiped, so suspend needs a persistent workspace volume.
3. Cleanup. Deleting the claim removed the Sandbox, pod, Service and PVC.
4. Cold start. A claim from an empty warm pool, including a node scale-up from zero, was ready in about 2 minutes.
5. Sandbox Agent runtime (0.5.3): processes survive the client disconnecting. Client A started a 40-second process, followed its logs to sequence 6, and disconnected. Client B, from another pod, found it still running 15 seconds later. logs?since=6 returned 7, 8, 9…, all 40 entries were contiguous, and exitCode was reported.

Still to verify before the PR:

• Exec from an in-cluster ServiceAccount with namespace-scoped RBAC.
• Whether a dropped exec connection kills the process under runc and gVisor, which determines how much the explicit kill matters.
• Binary-safe writeFile and large readFile round-trips.

Cluster requirements (would go in the docs)

• Agent Sandbox v1.0+ with the extensions (SandboxClaim, SandboxTemplate, SandboxWarmPool).
• A SandboxTemplate and SandboxWarmPool. For suspend, the template also needs a volumeClaimTemplates entry mounted at the workspace.
• RBAC for the actor's ServiceAccount in the sandbox namespace:
  • sandboxclaims: create, get, list, watch, patch, delete;
  • sandboxes: get, watch, patch;
  • with pods/exec: pods get and pods/exec create.
• With sandboxd or Sandbox Agent: service: true on the template, the daemon in the sandbox image, and a NetworkPolicy (through the template's networkPolicy) that admits the actor's namespace on the daemon's ports.

We'd include example YAML for all of the above. The runtime class (gVisor, Kata or the default) is left to the user.

Tests

• Unit tests against a fake Kubernetes API, plus a fake exec stream or a fake runtime server depending on the chosen runtime, like docker.test.ts.
• An integration test against a kind cluster with Agent Sandbox installed and the default runtime class. Happy to wire it into CI, or keep it opt-in, whichever you prefer.

Questions for maintainers

1. Which runtime should we implement: pods/exec, sandboxd, or Sandbox Agent? We recommend pods/exec, but this is your call, and it's the one answer we need before starting.
2. Should the export be ./kubernetes or ./agent-sandbox?
3. Is @kubernetes/client-node acceptable as an optional peer dependency?
4. suspend on a template without a workspace PVC: should it refuse, warn, or silently delete the pod?
5. Should the kind integration test run in CI?

If the direction looks right, we'll open the PR with the provider, tests, a section in docs/content/docs/sandboxes.mdx, and an example under examples/docs/sandboxes/.